Skip to content

fix: mark encoded frames as the end of their temporal unit - #356

Merged
devopvoid merged 1 commit into
mainfrom
fix/encoder-end-of-temporal-unit
Oct 11, 2026
Merged

devopvoid merged 1 commit into
mainfrom
fix/encoder-end-of-temporal-unit

Conversation

@devopvoid

Copy link
Copy Markdown
Owner

Follows up on the discuss-webrtc PSA An update to frame drop handling in VideoEncoder and EncodedImageCallback, which applies to M152 (branch-heads/7977). WebRTC now expects encoders to:

  • set EncodedImage::is_end_of_temporal_unit on every output frame (true for encoders without spatial or simulcast layers), and
  • report every dropped frame through EncodedImageCallback::OnFrameDropped.

The flag is an unset std::optional<bool> by default. Without it, WebRTC holds what it keeps of a frame longer than needed, for example the captured frame held for corruption detection. The PSA says this may become a hard error later.

Changes

  • Hardware encoders (Media Foundation, NVENC, VA-API H.264/H.265/VP9, VideoToolbox H.265): EncoderOutputProcessor, which builds every frame they output, sets the flag. They all produce a single layer.
  • Java encoders (VideoEncoderWrapper): the flag is set on each frame, and the frames the wrapper gives up on are now reported through OnFrameDropped instead of disappearing silently:
    • frames the Java encoder produced no output for, found once a later frame comes out;
    • the oldest pending frame when too many are pending;
    • frames that cannot be converted for the Java encoder.
  • Media Foundation and VideoToolbox already reported their drops. NVENC and VA-API drop none (Encode() returns an error code instead).

Tests

  • New VideoCodecFactoryTests.javaEncoderDroppingFramesKeepsSending: a Java encoder that drops every other frame keeps a call sending, which exercises the new drop reporting. Encoder drops and the flag don't reach getStats in M152, so the test can't check them directly.
  • mvn -pl webrtc-jni,webrtc package on windows-x86_64: 285 tests, 0 failures, 3 skipped.
  • The Linux and macOS encoders change only through the shared EncoderOutputProcessor; CI covers those builds.

Since M152 WebRTC expects encoders to set EncodedImage's end-of-temporal-unit
flag, and to report every frame they drop through
EncodedImageCallback::OnFrameDropped, so that it can release what it keeps of
a frame (the captured frame held for corruption detection, for one) as soon
as nothing more of it will come out.

All encoders here produce a single layer, so every frame ends its temporal
unit. EncoderOutputProcessor sets the flag for the hardware encoders, and the
Java encoder wrapper sets it for the frames of Java encoders. The wrapper
also reports the frames it gives up on: frames a Java encoder produced no
output for, the oldest frame when too many are pending, and frames that
cannot be converted for the Java encoder. The Media Foundation and
VideoToolbox encoders report their drops already; NVENC and VA-API drop none.
@devopvoid
devopvoid merged commit 26070d8 into main Oct 11, 2026
22 checks passed
@devopvoid
devopvoid deleted the fix/encoder-end-of-temporal-unit branch October 11, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant