Repository navigation
fix: mark encoded frames as the end of their temporal unit - #356
Merged
Merged
Conversation
Since M152 WebRTC expects encoders to set EncodedImage's end-of-temporal-unit flag, and to report every frame they drop through EncodedImageCallback::OnFrameDropped, so that it can release what it keeps of a frame (the captured frame held for corruption detection, for one) as soon as nothing more of it will come out. All encoders here produce a single layer, so every frame ends its temporal unit. EncoderOutputProcessor sets the flag for the hardware encoders, and the Java encoder wrapper sets it for the frames of Java encoders. The wrapper also reports the frames it gives up on: frames a Java encoder produced no output for, the oldest frame when too many are pending, and frames that cannot be converted for the Java encoder. The Media Foundation and VideoToolbox encoders report their drops already; NVENC and VA-API drop none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on the discuss-webrtc PSA An update to frame drop handling in VideoEncoder and EncodedImageCallback, which applies to M152 (
branch-heads/7977). WebRTC now expects encoders to:EncodedImage::is_end_of_temporal_uniton every output frame (truefor encoders without spatial or simulcast layers), andEncodedImageCallback::OnFrameDropped.The flag is an unset
std::optional<bool>by default. Without it, WebRTC holds what it keeps of a frame longer than needed, for example the captured frame held for corruption detection. The PSA says this may become a hard error later.Changes
EncoderOutputProcessor, which builds every frame they output, sets the flag. They all produce a single layer.VideoEncoderWrapper): the flag is set on each frame, and the frames the wrapper gives up on are now reported throughOnFrameDroppedinstead of disappearing silently:Encode()returns an error code instead).Tests
VideoCodecFactoryTests.javaEncoderDroppingFramesKeepsSending: a Java encoder that drops every other frame keeps a call sending, which exercises the new drop reporting. Encoder drops and the flag don't reachgetStatsin M152, so the test can't check them directly.mvn -pl webrtc-jni,webrtc packageon windows-x86_64: 285 tests, 0 failures, 3 skipped.EncoderOutputProcessor; CI covers those builds.