Skip to content

feat: let a video processor change camera and desktop frames - #342

Merged
devopvoid merged 2 commits into
mainfrom
feat/video-processor
Oct 10, 2026
Merged

devopvoid merged 2 commits into
mainfrom
feat/video-processor

Conversation

@devopvoid

Copy link
Copy Markdown
Owner

The counterpart of Android's VideoSource.setVideoProcessor: a hook between a camera or desktop source and its tracks, to blur a background, draw an overlay, mirror the picture, or drop frames.

API

dev.onvoid.webrtc.media.video.VideoProcessor, shaped like Android's:

public interface VideoProcessor {
    void setSink(VideoTrackSink sink);
    void onFrameCaptured(VideoFrame frame);
}

VideoTrackSource.setVideoProcessor(VideoProcessor) sets it, replacing any previous one; null removes it. The source hands each frame to onFrameCaptured() after adapting it to what its tracks ask for (the encoder's resolution and frame rate requests, the desktop source's maximum size), so the processor only works on frames that will be sent, at their final size. The processor delivers the frames to send to its sink: the same frame, a new one, or none; it may deliver later and from another thread.

  • Camera (VideoDeviceSource) and desktop (VideoDesktopSource) sources take a processor. CustomVideoSource throws UnsupportedOperationException: its frames come from Java anyway, and a frame replaced in Java would lose the NTP time and frame ID the custom source sets.
  • The frame passed to onFrameCaptured() is released when it returns; a processor that keeps or forwards it later retains it, as with frames elsewhere. The sink does not take over a delivered frame's reference.
  • An exception from the processor goes to the capture thread's uncaught exception handler (as in fix: stop exceptions from callbacks unwinding through WebRTC #335), and the frame is dropped.

Lifetime

The sink a processor gets is a small Java object that forwards to its source until it is detached: when the processor is replaced, or before the source is disposed. Delivery and detaching synchronize on the sink, so a frame delivered from another thread never reaches a disposed source. To detach first, VideoDeviceSource.dispose() and VideoDesktopSource.dispose() are now Java methods around the native disposeInternal(), which also removes the processor before releasing the source, so a source that a track keeps alive stops calling into Java.

Native

  • VideoProcessorHost (own file pair) holds the Java processor and turns frames into Java frames, holding a buffer reference for the duration of the call. Setting a processor swaps a shared_ptr under a mutex, so a frame being processed keeps its processor.
  • VideoTrackDeviceSource, VideoTrackDeviceSourceMac and VideoTrackDesktopSource derive from it and pass their final frame through ProcessFrame(), delivering it themselves when no processor is set. DeliverProcessedFrame() sends to the broadcaster or AdaptedVideoTrackSource::OnFrame.
  • The Java base class reaches the host through its own processorHostHandle field, set by each source's initialize, so no cast of the native source type is needed. JNI_VideoTrackSource holds setVideoProcessorNative and deliverProcessedFrame, which accepts native and Java I420 buffers through VideoFrame::toNativeBuffer.

The macOS camera source was changed the same way but could not be compiled here; CI will tell.

Docs

camera-capture.md gains "Changing Frames Before They Are Sent" with an example and the ownership rules; desktop-capture.md points to it.

Tests

VideoProcessorTests:

  • customSourceRejectsProcessor, setAndRemove (camera and desktop sources), staleSinkDropsFrames (a replaced processor's sink and sinks after dispose drop frames without failing).
  • processedDesktopFramesReachTrack: real screen capture through a processor that replaces every frame with a grey one; the track receives the grey frames.
  • processorCanDropFrames: a processor that delivers nothing; the track receives nothing.

The two screen tests skip themselves where there is no screen or no frames. The full webrtc suite passes on Windows (266 run, 0 failures, 2 skipped), also under -Pjni-check with no JNI errors; both screen tests ran.

VideoTrackSource.setVideoProcessor(VideoProcessor) runs the frames of a
camera or desktop source through a processor before they reach its
tracks, as Android's VideoSource does. The source hands each frame to
onFrameCaptured() after adapting it to what its tracks ask for; the
processor delivers the frames to send to the sink it got in setSink():
the same frame, a new one, or none. null removes the processor, and a
CustomVideoSource rejects one, since its frames come from Java anyway.

The frame passed to the processor is released once onFrameCaptured()
returns; a processor that keeps it retains it, as with other frames.
The sink drops frames once the processor is replaced or the source is
disposed, so a late delivery from another thread does no harm.

Natively, VideoProcessorHost holds the processor and turns frames into
Java frames and back. VideoTrackDeviceSource, VideoTrackDeviceSourceMac
and VideoTrackDesktopSource derive from it and pass their final frames
through ProcessFrame(); the Java base class reaches the host through
its own handle field. An exception thrown by the processor goes to the
capture thread's uncaught exception handler.

VideoDeviceSource.dispose() and VideoDesktopSource.dispose() are now
Java methods that detach the sink before the native dispose, which
removes the processor before releasing the source.
On headless Linux, listing the screens throws instead of returning an empty list, which failed the two tests that capture the screen rather than skipping them.
@devopvoid
devopvoid merged commit b0008ff into main Oct 10, 2026
24 checks passed
@devopvoid
devopvoid deleted the feat/video-processor branch October 10, 2026 16:26
SendableMetatype added a commit to EduGeyser/webrtc-java that referenced this pull request Oct 10, 2026
Brings in the work upstream added on October 10: the missing peer
connection signaling and control calls (devopvoid#334), callback exceptions kept
from unwinding through WebRTC (devopvoid#335), first-packet observers, sender and
receiver IDs and track volume (devopvoid#338), video processors for camera and
desktop frames (devopvoid#342), and fixes to audio device changes, camera
hotplug and media stream views (devopvoid#333, devopvoid#337, devopvoid#341, devopvoid#343).

Fork-side adaptations:
- setAudioPlayout and setAudioRecording reach the voice engine, so the
  data channels variant leaves them out, like the other audio natives.
- The new tests that need audio or video carry the media tag.
- The Maven publish action stays deleted.
SendableMetatype added a commit to EduGeyser/webrtc-java that referenced this pull request Oct 10, 2026
Brings in the work upstream added on October 10: the missing peer
connection signaling and control calls (devopvoid#334), callback exceptions kept
from unwinding through WebRTC (devopvoid#335), first-packet observers, sender and
receiver IDs and track volume (devopvoid#338), video processors for camera and
desktop frames (devopvoid#342), and fixes to audio device changes, camera
hotplug and media stream views (devopvoid#333, devopvoid#337, devopvoid#341, devopvoid#343).

Fork-side adaptations:
- setBitrate, setAudioPlayout and setAudioRecording reach the call and
  the voice engine, which a factory without a media engine does not
  create, so the data channels variant leaves them out, like the other
  media natives.
- The new tests that need audio or video carry the media tag.
- The Maven publish action stays deleted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant