Repository navigation
feat: let a video processor change camera and desktop frames - #342
Merged
Merged
Conversation
VideoTrackSource.setVideoProcessor(VideoProcessor) runs the frames of a camera or desktop source through a processor before they reach its tracks, as Android's VideoSource does. The source hands each frame to onFrameCaptured() after adapting it to what its tracks ask for; the processor delivers the frames to send to the sink it got in setSink(): the same frame, a new one, or none. null removes the processor, and a CustomVideoSource rejects one, since its frames come from Java anyway. The frame passed to the processor is released once onFrameCaptured() returns; a processor that keeps it retains it, as with other frames. The sink drops frames once the processor is replaced or the source is disposed, so a late delivery from another thread does no harm. Natively, VideoProcessorHost holds the processor and turns frames into Java frames and back. VideoTrackDeviceSource, VideoTrackDeviceSourceMac and VideoTrackDesktopSource derive from it and pass their final frames through ProcessFrame(); the Java base class reaches the host through its own handle field. An exception thrown by the processor goes to the capture thread's uncaught exception handler. VideoDeviceSource.dispose() and VideoDesktopSource.dispose() are now Java methods that detach the sink before the native dispose, which removes the processor before releasing the source.
On headless Linux, listing the screens throws instead of returning an empty list, which failed the two tests that capture the screen rather than skipping them.
SendableMetatype
added a commit
to EduGeyser/webrtc-java
that referenced
this pull request
Oct 10, 2026
Brings in the work upstream added on October 10: the missing peer connection signaling and control calls (devopvoid#334), callback exceptions kept from unwinding through WebRTC (devopvoid#335), first-packet observers, sender and receiver IDs and track volume (devopvoid#338), video processors for camera and desktop frames (devopvoid#342), and fixes to audio device changes, camera hotplug and media stream views (devopvoid#333, devopvoid#337, devopvoid#341, devopvoid#343). Fork-side adaptations: - setAudioPlayout and setAudioRecording reach the voice engine, so the data channels variant leaves them out, like the other audio natives. - The new tests that need audio or video carry the media tag. - The Maven publish action stays deleted.
SendableMetatype
added a commit
to EduGeyser/webrtc-java
that referenced
this pull request
Oct 10, 2026
Brings in the work upstream added on October 10: the missing peer connection signaling and control calls (devopvoid#334), callback exceptions kept from unwinding through WebRTC (devopvoid#335), first-packet observers, sender and receiver IDs and track volume (devopvoid#338), video processors for camera and desktop frames (devopvoid#342), and fixes to audio device changes, camera hotplug and media stream views (devopvoid#333, devopvoid#337, devopvoid#341, devopvoid#343). Fork-side adaptations: - setBitrate, setAudioPlayout and setAudioRecording reach the call and the voice engine, which a factory without a media engine does not create, so the data channels variant leaves them out, like the other media natives. - The new tests that need audio or video carry the media tag. - The Maven publish action stays deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The counterpart of Android's
VideoSource.setVideoProcessor: a hook between a camera or desktop source and its tracks, to blur a background, draw an overlay, mirror the picture, or drop frames.API
dev.onvoid.webrtc.media.video.VideoProcessor, shaped like Android's:VideoTrackSource.setVideoProcessor(VideoProcessor)sets it, replacing any previous one;nullremoves it. The source hands each frame toonFrameCaptured()after adapting it to what its tracks ask for (the encoder's resolution and frame rate requests, the desktop source's maximum size), so the processor only works on frames that will be sent, at their final size. The processor delivers the frames to send to its sink: the same frame, a new one, or none; it may deliver later and from another thread.VideoDeviceSource) and desktop (VideoDesktopSource) sources take a processor.CustomVideoSourcethrowsUnsupportedOperationException: its frames come from Java anyway, and a frame replaced in Java would lose the NTP time and frame ID the custom source sets.onFrameCaptured()is released when it returns; a processor that keeps or forwards it later retains it, as with frames elsewhere. The sink does not take over a delivered frame's reference.Lifetime
The sink a processor gets is a small Java object that forwards to its source until it is detached: when the processor is replaced, or before the source is disposed. Delivery and detaching synchronize on the sink, so a frame delivered from another thread never reaches a disposed source. To detach first,
VideoDeviceSource.dispose()andVideoDesktopSource.dispose()are now Java methods around the nativedisposeInternal(), which also removes the processor before releasing the source, so a source that a track keeps alive stops calling into Java.Native
VideoProcessorHost(own file pair) holds the Java processor and turns frames into Java frames, holding a buffer reference for the duration of the call. Setting a processor swaps ashared_ptrunder a mutex, so a frame being processed keeps its processor.VideoTrackDeviceSource,VideoTrackDeviceSourceMacandVideoTrackDesktopSourcederive from it and pass their final frame throughProcessFrame(), delivering it themselves when no processor is set.DeliverProcessedFrame()sends to the broadcaster orAdaptedVideoTrackSource::OnFrame.processorHostHandlefield, set by each source'sinitialize, so no cast of the native source type is needed.JNI_VideoTrackSourceholdssetVideoProcessorNativeanddeliverProcessedFrame, which accepts native and Java I420 buffers throughVideoFrame::toNativeBuffer.The macOS camera source was changed the same way but could not be compiled here; CI will tell.
Docs
camera-capture.mdgains "Changing Frames Before They Are Sent" with an example and the ownership rules;desktop-capture.mdpoints to it.Tests
VideoProcessorTests:customSourceRejectsProcessor,setAndRemove(camera and desktop sources),staleSinkDropsFrames(a replaced processor's sink and sinks after dispose drop frames without failing).processedDesktopFramesReachTrack: real screen capture through a processor that replaces every frame with a grey one; the track receives the grey frames.processorCanDropFrames: a processor that delivers nothing; the track receives nothing.The two screen tests skip themselves where there is no screen or no frames. The full
webrtcsuite passes on Windows (266 run, 0 failures, 2 skipped), also under-Pjni-checkwith no JNI errors; both screen tests ran.