Repository navigation
Conversation
The hardware codec factories now use VideoToolbox for the codecs WebRTC's macOS factories lack, where the Mac has the hardware for them: - H.265 (Main profile), encoded and decoded on the media engine. The sessions are required to be hardware ones and checked to be, so H.265 is never encoded or decoded in software; a Mac without a hardware HEVC encoder or decoder does not offer it. The encoder works in real time, without frame reordering, with a key frame only on request, and takes CVPixelBuffers as they are and converts other frames to NV12. Its output goes through EncoderOutputProcessor, which puts the parameter sets in front of each key frame. The decoder makes its session from the VPS, SPS and PPS of key frames. - AV1 (profile 0, 8 bit), decoded on Macs with an AV1 decoder (Apple M3 and newer). The session is made from the sequence header, which is parsed for the AV1 configuration box; high bit depth, monochrome and spatial layers go to dav1d, as does a decoder that keeps failing. The VP9 decoder shares a new VTVideoDecoder base with them, which holds the session, the hardware check and the output. The session limit of the process applies only to codecs with a software decoder to go to. H.265 is offered after the default codecs, which keep their order, with the SDP format and the EncoderOutputProcessor support of #347, taken over as they are there. Tests that a call in H.265 is encoded and decoded in hardware run on macOS now, check that the call is still in H.265 rather than in a codec WebRTC switched to, and cover odd sizes, a resolution change, a bitrate cap and restarts; the software factories are checked to offer no H.265.
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #346 (H.265 in the WebRTC build), and based on its branch until that is merged.
The hardware codec factories now use VideoToolbox for the codecs WebRTC's macOS factories lack, where the Mac has the hardware for them:
RequireHardwareAccelerated…) and are checked to be, so H.265 is never encoded or decoded in software; a Mac without a hardware HEVC encoder or decoder does not offer it. The encoder works in real time, without frame reordering, with a key frame only on request. CVPixelBuffers (camera, VideoToolbox decoders) go in as they are, other frames are converted to NV12. Its output is turned from length-prefixed NAL units into Annex B and goes throughEncoderOutputProcessor, which puts the parameter sets in front of each key frame. The decoder makes its session from the VPS, SPS and PPS of key frames. With no software codec to fall back to, a failing encoder makes WebRTC switch to the next negotiated codec, and a failing decoder asks for key frames.av1C). High bit depth, monochrome and spatial layers go to dav1d, as does a decoder that keeps failing.VTVideoDecoderbase with them: session, hardware check, range-matched NV12 output, failure handling. The process-wide session limit applies only to codecs that have a software decoder to go to.H265Format) and the H.265 handling inEncoderOutputProcessorare taken over from feat: encode and decode H.265 in hardware on Windows #347 unchanged, as are its test helpers, so that the two merge cleanly; the docs, the Javadoc and a few test lines will conflict with feat: encode and decode H.265 in hardware on Windows #347.Testing
On an Apple M2, macOS 14.5, after WebRTC was rebuilt with
rtc_use_h265(incremental, about 2 minutes), withwebrtc.test.hardwareH265Encoder,webrtc.test.hardwareH265Decoderandwebrtc.test.hardwareDecoderset so that the tests fail rather than skip:hardwareEncodesH265,hardwareDecodesH265,hardwareCropsPaddedH265,hardwareKeepsCodecs.h265EncodesOddSize,h265FollowsResolutionChange,h265FollowsBitrateCap,h265RestartsCleanly, andsoftwareOffersNoH265(the default factories never offer H.265). The H.265 call tests also check that the call is still in H.265 (TestMediaCall.getSentVideoCodec()), since on macOS a fallback to H.264 would also report a VideoToolbox implementation.webrtctest suite: 280 tests, no failures. The 12 skipped are Windows/Linux-only or need an AV1 decoder.Not verified
-Dwebrtc.test.hardwareAv1Decoder=truethere.webrtc-java-mediatests: the configure step stops before them because this branch is behindmainby feat: decode with Vulkan Video on Linux in the media player #345, and the FFmpeg install in~/ffmpegwas built bymainwith other components. Nothing in that module changes here.