diff --git a/docs/guide/advanced/video-codecs.md b/docs/guide/advanced/video-codecs.md index 4cf1d994d..cbf35a950 100644 --- a/docs/guide/advanced/video-codecs.md +++ b/docs/guide/advanced/video-codecs.md @@ -16,7 +16,7 @@ for (VideoCodecInfo codec : new DefaultVideoEncoderFactory().getSupportedCodecs( ### Hardware Encoding -`HardwareVideoEncoderFactory` offers the same codecs as `DefaultVideoEncoderFactory`, but encodes on the GPU where the platform supports it: +`HardwareVideoEncoderFactory` offers the codecs of `DefaultVideoEncoderFactory`, and H.265 where the hardware encodes it, and encodes on the GPU where the platform supports it: ```java PeerConnectionFactory factory = PeerConnectionFactory.builder() @@ -24,19 +24,19 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder() .build(); ``` -| Platform | H.264 | AV1 | -|---|---|---| -| Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) | -| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | NVENC on NVIDIA RTX 40 and newer | -| macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | Software | +| Platform | H.264 | H.265 | AV1 | +|---|---|---|---| +| Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | Not offered | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) | +| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | Not offered | NVENC on NVIDIA RTX 40 and newer | +| macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | VideoToolbox, on Macs with a hardware HEVC encoder, such as all with Apple silicon | Software | Hardware encoders produce a single layer. A stream that asks for SVC, such as an AV1 stream with a scalability mode like `L1T3`, is encoded in software. NVENC needs an NVIDIA driver of version 522 or newer on Windows, 520 or newer on Linux. VA-API needs libva 2 and a driver that encodes H.264, such as Intel's `intel-media-va-driver` (iHD) or Mesa's `mesa-va-drivers` for AMD, and access to a render node in `/dev/dri`. All of them are loaded at run time, so nothing needs to be installed on machines without them. -The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`. +The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding these in hardware never changes what is negotiated. H.265 (Main profile) has no software encoder: it is offered after the other codecs, only where the hardware encodes it, so the codec a peer connection prefers stays the same, and a call uses H.265 only if it is preferred (e.g. with `RTCRtpTransceiver.setCodecPreferences()`) and the other end decodes it. On macOS, VideoToolbox is required to encode H.265 in hardware, and never falls back to its software encoder; a Mac without a hardware HEVC encoder does not offer H.265. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`. -Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `OpenH264` or `libaom`. +Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `VideoToolbox (H.265)`, `OpenH264` or `libaom`. ### Hardware Decoding @@ -53,7 +53,7 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder() |---|---| | Windows | H.264, AV1 and VP9 (profile 0), on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) | | Linux | H.264 and VP9 (profile 0) on NVIDIA GPUs, through NVDEC; VA-API, for Intel and AMD GPUs, is not there yet | -| macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`, and VP9 (profile 0) through VideoToolbox's VP9 decoder, on Macs that have one | +| macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`; VP9 (profile 0) and H.265 (Main profile) through the decoders of VideoToolbox, on Macs that have them, and AV1 (profile 0) on Apple M3 and newer | On Linux, NVDEC needs an NVIDIA driver, which provides `libcuda.so.1` and `libnvcuvid.so.1`; both are loaded when the factory is made, so a machine without the driver is not affected. Streams with spatial layers are decoded in software. @@ -61,7 +61,9 @@ AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes, and V On macOS, VP9 decoding in hardware saves CPU: in a measurement on an Apple M2 it took a fraction of the processor time libvpx needs, but multi-threaded libvpx was about as fast on the clock. VideoToolbox decodes VP9 in hardware only where the Mac has a decoder for it, which Apple silicon does; a Mac without one decodes VP9 with libvpx as before. Profile 2 (10 bit), sizes below 64x64 or above 4096x4096, and streams with spatial layers (SVC) are decoded with libvpx too. -Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)`, `VideoToolbox (VP9)` or `NVDEC (NVIDIA GeForce RTX 4070)`. +AV1 is decoded by VideoToolbox on Macs with an AV1 decoder, which are those with Apple M3 and newer, and by dav1d elsewhere. High bit depth, monochrome streams and streams with spatial layers go to dav1d as well. H.265 is decoded by VideoToolbox in hardware only, never in software: a Mac without a hardware HEVC decoder does not offer H.265, and an H.265 stream the hardware cannot decode is not decoded. + +Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)`, `VideoToolbox (VP9)`, `VideoToolbox (H.265)`, `VideoToolbox (AV1)` or `NVDEC (NVIDIA GeForce RTX 4070)`. ### Native Codecs @@ -114,7 +116,7 @@ The list is in order of preference. A codec name WebRTC does not know, such as ` ### H.265 -The library is built with WebRTC's H.265 support: H.265 is negotiated with its SDP parameters (`profile-id`, `tier-flag`, `level-id`, `tx-mode`) and sent with the H.265 RTP payload format (RFC 7798). WebRTC has no H.265 encoder or decoder of its own, and this library adds none in software, so H.265 is offered only by a factory that brings one, such as a hardware encoder reached through another library: +The library is built with WebRTC's H.265 support: H.265 is negotiated with its SDP parameters (`profile-id`, `tier-flag`, `level-id`, `tx-mode`) and sent with the H.265 RTP payload format (RFC 7798). WebRTC has no H.265 encoder or decoder of its own, and this library adds none in software, so H.265 is offered only by a factory that brings one: the hardware factories on Macs that encode or decode it in hardware (see above), or one of your own, such as a hardware encoder reached through another library: ```java Map parameters = new HashMap<>(); diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h b/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h index 8a4d31415..56720ca04 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h @@ -22,6 +22,7 @@ #include "api/video_codecs/sdp_video_format.h" #include "api/video_codecs/video_encoder.h" #include "common_video/h264/h264_bitstream_parser.h" +#include "common_video/h265/h265_bitstream_parser.h" #include "modules/video_coding/codecs/h264/include/h264_globals.h" #include "modules/video_coding/include/video_codec_interface.h" #include "modules/video_coding/svc/scalable_video_controller_no_layering.h" @@ -33,15 +34,16 @@ namespace jni { // Turns what a hardware encoder produces for a frame into what WebRTC - // sends, the same way for every hardware encoder: H.264 as Annex B NAL - // units, AV1 as OBUs in the low overhead bitstream format. + // sends, the same way for every hardware encoder: H.264 and H.265 as + // Annex B NAL units, AV1 as OBUs in the low overhead bitstream format. // // A receiver can start decoding only from a key frame that carries the - // parameter sets, the SPS and PPS of H.264 or the sequence header of AV1. - // Encoders put them in front of the first key frame, not necessarily in - // front of every one, so they are kept and put back where missing. The - // frame dependencies WebRTC needs for AV1 are those of a stream without - // layers, which is all hardware encoders produce here. + // parameter sets: the SPS and PPS of H.264, the VPS, SPS and PPS of H.265, + // or the sequence header of AV1. Encoders put them in front of the first + // key frame, not necessarily in front of every one, so they are kept and + // put back where missing. The frame dependencies WebRTC needs for AV1 and + // H.265 are those of a stream without layers, which is all hardware + // encoders produce here. class EncoderOutputProcessor { public: @@ -54,19 +56,21 @@ namespace jni // specific information WebRTC needs to send it. The caller sets // the size of the image before, which the AV1 frame dependencies // refer to, and its timestamps. The frame is a key - // frame if the encoder says so or, for H.264, if it holds an IDR - // slice. Returns false if the bitstream cannot be parsed, or a key + // frame if the encoder says so or if it holds an IDR slice of + // H.264 or an IRAP picture of H.265. Returns false if the + // bitstream cannot be parsed, or a key // frame lacks parameter sets and there are none to put back. bool Process(std::span bitstream, bool keyFrame, webrtc::EncodedImage & image, webrtc::CodecSpecificInfo & info); // The quality scaling settings that suit the codec: the thresholds - // of WebRTC's H.264 encoder, and none for AV1, whose QP is not - // parsed from the bitstream. + // of WebRTC's H.264 encoder, for H.265 as well, and none for AV1, + // whose QP is not parsed from the bitstream. webrtc::VideoEncoder::ScalingSettings GetScalingSettings() const; private: bool ProcessH264(std::span bitstream, bool & keyFrame, std::vector & output); + bool ProcessH265(std::span bitstream, bool & keyFrame, std::vector & output); bool ProcessAv1(std::span bitstream, bool keyFrame, std::vector & output); void FillCodecSpecificInfo(const webrtc::EncodedImage & image, bool keyFrame, @@ -76,10 +80,12 @@ namespace jni const webrtc::VideoCodecType codec; webrtc::H264PacketizationMode packetizationMode; - // The SPS and PPS, or the sequence header OBU, last seen. + // The parameter sets of H.264 or H.265, or the sequence header + // OBU, last seen. std::vector parameterSets; webrtc::H264BitstreamParser h264Parser; + webrtc::H265BitstreamParser h265Parser; webrtc::ScalableVideoControllerNoLayering svcController; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/H265Format.h b/webrtc-jni/src/main/cpp/include/media/video/codec/H265Format.h new file mode 100644 index 000000000..108105d19 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/H265Format.h @@ -0,0 +1,31 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_H265_FORMAT_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_H265_FORMAT_H_ + +#include "api/video_codecs/sdp_video_format.h" + +namespace jni +{ + // The H.265 format the hardware codec factories offer: Main profile, main + // tier, level 3.1, one RTP stream (tx-mode SRST). The same level as the + // H.264 they offer; WebRTC negotiates the level down to what both ends + // take. + webrtc::SdpVideoFormat CreateH265MainFormat(); +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTAv1Decoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTAv1Decoder.h new file mode 100644 index 000000000..91c00566c --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTAv1Decoder.h @@ -0,0 +1,66 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_AV1_DECODER_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_AV1_DECODER_H_ + +#include "media/video/codec/macos/VTVideoDecoder.h" + +#include "api/video/encoded_image.h" + +#include +#include +#include + +namespace jni +{ + // Decodes AV1 profile 0, 8 bit, on the media engine of Macs that have an + // AV1 decoder, Apple M3 and newer, through VideoToolbox. + // + // The session is created from the sequence header of a key frame, which + // goes into the AV1 configuration box ("av1C") of the format description, + // and again whenever a key frame brings one the session does not take. + // The OBUs of a temporal unit go to VideoToolbox as one sample, without + // temporal delimiters, as AV1 is stored in ISOBMFF. + // + // Whatever VideoToolbox cannot decode goes to dav1d, by returning + // WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE: another profile or bit depth, + // monochrome, a stream with spatial layers, and a decoder that keeps + // failing. + class VTAv1Decoder : public VTVideoDecoder + { + public: + VTAv1Decoder(); + ~VTAv1Decoder() override = default; + + using webrtc::VideoDecoder::Decode; + + bool Configure(const Settings & settings) override; + int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override; + + private: + // Makes the format description of a stream from its sequence + // header OBU, header and size field included. Returns null for a + // stream the hardware decoder does not take. + static CMVideoFormatDescriptionRef CreateFormat(std::span sequenceHeaderObu); + + private: + // The sample, rebuilt for each frame. + std::vector sample; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTCodecUtils.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTCodecUtils.h new file mode 100644 index 000000000..3480e9a7a --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTCodecUtils.h @@ -0,0 +1,70 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_CODEC_UTILS_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_CODEC_UTILS_H_ + +#include +#include +#include + +#include + +namespace jni +{ + // Releases a Core Foundation object when it goes out of scope. + template + class ScopedCF + { + public: + ScopedCF() : ref(nullptr) {} + explicit ScopedCF(T ref) : ref(ref) {} + ~ScopedCF() { if (ref) CFRelease(ref); } + + ScopedCF(const ScopedCF &) = delete; + ScopedCF & operator=(const ScopedCF &) = delete; + + T get() const { return ref; } + T * receive() { return &ref; } + explicit operator bool() const { return ref != nullptr; } + + // Gives up ownership. + T release() { T result = ref; ref = nullptr; return result; } + + private: + T ref; + }; + + ScopedCF CreateCFDictionary(); + + // Sets an integer value in a dictionary. Returns false if the number + // cannot be created. + bool SetCFNumber(CFMutableDictionaryRef dictionary, CFStringRef key, int64_t value); + + // Writes the colour of a stream, given as the code points of ISO/IEC + // 23091-2 (H.273) that H.265, VP9 and AV1 share, into the extensions of a + // format description. Code points without a Core Media counterpart, and + // unspecified ones, are left out. + void SetColorExtensions(CFMutableDictionaryRef extensions, uint8_t primaries, uint8_t transfer, + uint8_t matrix); + + // Whether a VideoToolbox session works in hardware, as the session reports + // it under the given property. Requiring hardware when the session is + // created is not relied on alone. + bool IsUsingHardware(VTSessionRef session, CFStringRef property); +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Decoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Decoder.h new file mode 100644 index 000000000..9fcff8c53 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Decoder.h @@ -0,0 +1,55 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_H265_DECODER_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_H265_DECODER_H_ + +#include "media/video/codec/macos/VTVideoDecoder.h" + +#include "api/video/encoded_image.h" + +#include +#include + +namespace jni +{ + // Decodes H.265 Main profile on the media engine of a Mac, through + // VideoToolbox, and only there: there is no software decoder for H.265, + // and the session is required to be a hardware one. + // + // WebRTC hands over Annex B NAL units. The session is created from the + // VPS, SPS and PPS of a key frame, and again whenever a key frame brings + // parameter sets the session does not take. The slices go to VideoToolbox + // with length prefixes, as the format description has it, without the + // parameter sets. + class VTH265Decoder : public VTVideoDecoder + { + public: + VTH265Decoder(); + ~VTH265Decoder() override = default; + + using webrtc::VideoDecoder::Decode; + + bool Configure(const Settings & settings) override; + int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override; + + private: + // The sample, rebuilt for each frame. + std::vector sample; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Encoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Encoder.h new file mode 100644 index 000000000..655a1c081 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Encoder.h @@ -0,0 +1,108 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_H265_ENCODER_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_H265_ENCODER_H_ + +#include "media/video/codec/EncoderOutputProcessor.h" + +#include "api/video/video_frame.h" +#include "api/video_codecs/sdp_video_format.h" +#include "api/video_codecs/video_codec.h" +#include "api/video_codecs/video_encoder.h" + +#include +#include +#include + +#include +#include +#include + +namespace jni +{ + // Encodes H.265 Main profile on the media engine of a Mac, through + // VideoToolbox, and only there: the compression session is required to + // be a hardware one, there being no software encoder for H.265. + // + // The stream is made for real-time use: no frame reordering, a key frame + // only when WebRTC asks for one, a single layer. VideoToolbox hands out + // the frames on a thread of its own, with length prefixed NAL units and + // the parameter sets in the format description; they are turned into + // Annex B there, with the parameter sets in front of each key frame, and + // passed to WebRTC. + // + // Frames that come as a CVPixelBuffer, as from the camera, go to + // VideoToolbox as they are; others are converted to NV12 first. + class VTH265Encoder : public webrtc::VideoEncoder + { + public: + explicit VTH265Encoder(const webrtc::SdpVideoFormat & format); + ~VTH265Encoder() override; + + int32_t InitEncode(const webrtc::VideoCodec * codecSettings, const Settings & settings) override; + int32_t RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * callback) override; + int32_t Release() override; + int32_t Encode(const webrtc::VideoFrame & frame, const std::vector * frameTypes) override; + void SetRates(const RateControlParameters & parameters) override; + EncoderInfo GetEncoderInfo() const override; + + private: + // What becomes of an input frame once its output arrives. + struct PendingFrame + { + VTH265Encoder * encoder; + uint32_t rtpTimestamp; + int64_t captureTimeMs; + int64_t ntpTimeMs; + webrtc::VideoRotation rotation; + }; + + static void OnOutput(void * encoder, void * frame, OSStatus status, VTEncodeInfoFlags flags, + CMSampleBufferRef sample); + + bool CreateSession(); + void DestroySession(); + void ConfigureRates(); + + // Returns a pixel buffer of the frame for the session, or null. + CVPixelBufferRef CreatePixelBuffer(const webrtc::VideoFrame & frame); + + void DeliverOutput(const PendingFrame & pending, CMSampleBufferRef sample); + + private: + webrtc::VideoCodec codecSettings; + uint32_t bitrateBps; + uint32_t framerate; + + VTCompressionSessionRef session; + + std::atomic callback; + + // Set once the session fails, from any thread. + std::atomic failed; + + // Used on the encoder thread only. + int64_t lastTimestampUs; + std::vector scaleBuffer; + + // Used on the output thread of the session only. + EncoderOutputProcessor outputProcessor; + std::vector annexB; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoder.h new file mode 100644 index 000000000..afdcc647d --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoder.h @@ -0,0 +1,119 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_DECODER_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_DECODER_H_ + +#include "api/video/encoded_image.h" +#include "api/video_codecs/video_decoder.h" + +#include +#include +#include + +#include +#include + +namespace jni +{ + // What the VideoToolbox decoders of the codecs have in common: a + // decompression session on the media engine of the Mac, made for the + // format description of the stream, which the decoder of each codec reads + // from its key frames. + // + // Decoding is synchronous: each frame comes out of VideoToolbox before + // Decode returns. A frame is handed on as the CVPixelBuffer VideoToolbox + // made, without a copy, in 8 bit 4:2:0 and in the range of the stream. + // + // The session is made in hardware only, and checked to be. Where it cannot + // be, or the decoder keeps failing, a codec that has a software decoder + // goes to it, by returning WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE. One that + // has none, H.265, is never decoded in software: its frames fail, and + // WebRTC asks for key frames, until the hardware decodes again. + class VTVideoDecoder : public webrtc::VideoDecoder + { + public: + ~VTVideoDecoder() override; + + int32_t RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * callback) override; + int32_t Release() override; + DecoderInfo GetDecoderInfo() const override; + const char * ImplementationName() const override; + + protected: + // The name is reported as the implementation. With a software + // fallback, a decoder that keeps failing gives way to it. + VTVideoDecoder(const char * name, bool softwareFallback); + + // Makes sure there is a session for the format, which this takes + // over, keeping the session where the stream did not change, or + // changed in a way the session accepts. Returns false if there is + // no session for it, which leaves none. + bool SetFormat(CMVideoFormatDescriptionRef format); + + bool HasSession() const; + + // The size of the format the session decodes. + CMVideoDimensions GetDimensions() const; + + // Decodes one sample, in the form the format description says, + // and hands the picture to WebRTC with the timing of the image. + int32_t DecodeSample(std::span sample, const webrtc::EncodedImage & image, + int64_t renderTimeMs); + + // What to return for a frame that cannot be decoded. + int32_t Fail(OSStatus status); + + // What to return when the hardware cannot take a stream at all. + int32_t Unsupported(); + + // Whether the decoder needs a key frame before it can go on, once + // the session or a frame failed. + bool requireKeyFrame; + + private: + // The result of decoding one frame, filled in by the callback of + // the session. + struct Output + { + OSStatus status = noErr; + CVImageBufferRef image = nullptr; + }; + + static void OnOutput(void * decoder, void * frame, OSStatus status, VTDecodeInfoFlags flags, + CVImageBufferRef image, CMTime presentationTime, CMTime duration); + + bool CreateSession(CMVideoFormatDescriptionRef newFormat); + void DestroySession(); + + void Deliver(const webrtc::EncodedImage & image, int64_t renderTimeMs, CVImageBufferRef pixels); + + private: + const char * const name; + const bool softwareFallback; + + webrtc::DecodedImageCallback * callback; + + VTDecompressionSessionRef session; + CMVideoFormatDescriptionRef format; + bool fullRange; + + int consecutiveErrors; + int64_t sampleCount; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoderFactory.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoderFactory.h index 3e8d18c6f..aded6ff53 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoderFactory.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoderFactory.h @@ -28,15 +28,17 @@ namespace jni { // Creates the VideoToolbox decoders that WebRTC's own decoders for macOS - // do not offer: VP9, in profile 0, the format the software factory - // offers first. H.264 is decoded through VideoToolbox by the default - // decoders already. + // do not offer, for the codecs this Mac decodes in hardware: VP9 in + // profile 0, the format the software factory offers first, AV1 in + // profile 0 (Apple M3 and newer), and H.265 in the Main profile, which + // has no software decoder. H.264 is decoded through VideoToolbox by the + // default decoders already. class VTVideoDecoderFactory : public webrtc::VideoDecoderFactory { public: // Returns a factory, or null if this Mac has no hardware decoder - // for VP9. The decoder VideoToolbox has for VP9 has to be - // registered first; this does it, once for the process. + // for any of these codecs. The decoder VideoToolbox has for VP9 + // has to be registered first; this does it, once for the process. static std::unique_ptr Create(); ~VTVideoDecoderFactory() override = default; @@ -46,7 +48,10 @@ namespace jni const webrtc::SdpVideoFormat & format) override; private: - VTVideoDecoderFactory() = default; + explicit VTVideoDecoderFactory(std::vector formats); + + private: + const std::vector formats; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoEncoderFactory.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoEncoderFactory.h new file mode 100644 index 000000000..622f31673 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoEncoderFactory.h @@ -0,0 +1,52 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_ENCODER_FACTORY_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_ENCODER_FACTORY_H_ + +#include "api/environment/environment.h" +#include "api/video_codecs/sdp_video_format.h" +#include "api/video_codecs/video_encoder.h" +#include "api/video_codecs/video_encoder_factory.h" + +#include +#include + +namespace jni +{ + // Creates the VideoToolbox encoders that WebRTC's own encoders for macOS + // do not offer: H.265 in the Main profile, on Macs that encode it in + // hardware. H.264 is encoded through VideoToolbox by the default encoders + // already, and VideoToolbox has no encoder for VP8, VP9 or AV1. + class VTVideoEncoderFactory : public webrtc::VideoEncoderFactory + { + public: + // Returns a factory, or null if this Mac has no hardware encoder + // for H.265. + static std::unique_ptr Create(); + + ~VTVideoEncoderFactory() override = default; + + std::vector GetSupportedFormats() const override; + std::unique_ptr Create(const webrtc::Environment & env, + const webrtc::SdpVideoFormat & format) override; + + private: + VTVideoEncoderFactory() = default; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVp9Decoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVp9Decoder.h index d5361817e..87a0c3ed8 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVp9Decoder.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVp9Decoder.h @@ -17,14 +17,11 @@ #ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VP9_DECODER_H_ #define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VP9_DECODER_H_ +#include "media/video/codec/macos/VTVideoDecoder.h" + #include "api/video/encoded_image.h" -#include "api/video_codecs/video_decoder.h" #include "modules/video_coding/utility/vp9_uncompressed_header_parser.h" -#include -#include -#include - #include namespace jni @@ -34,83 +31,28 @@ namespace jni // // The decompression session needs the properties of the stream, so it is // created on the first key frame, from the header the key frame carries, - // and again whenever a key frame changes the size or the range. Decoding - // is synchronous: each frame comes out of VideoToolbox before Decode - // returns. A frame is handed on as the CVPixelBuffer VideoToolbox made, - // without a copy. + // and again whenever a key frame changes the size or the range. // // Whatever VideoToolbox cannot decode in place goes to the software // decoder, by returning WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE: a profile // other than 0, a size outside what the hardware decodes, frames with // spatial layers, and a decoder that keeps failing. - class VTVp9Decoder : public webrtc::VideoDecoder + class VTVp9Decoder : public VTVideoDecoder { public: VTVp9Decoder(); - ~VTVp9Decoder() override; + ~VTVp9Decoder() override = default; using webrtc::VideoDecoder::Decode; bool Configure(const Settings & settings) override; int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override; - int32_t RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * callback) override; - int32_t Release() override; - DecoderInfo GetDecoderInfo() const override; - const char * ImplementationName() const override; - - private: - // What a session is created for. A key frame that differs from it - // needs a new format description, and often a new session. - struct StreamConfig - { - int width = 0; - int height = 0; - bool fullRange = false; - webrtc::Vp9ColorSpace colorSpace = webrtc::Vp9ColorSpace::CS_UNKNOWN; - - bool operator==(const StreamConfig & other) const; - }; - - // The result of decoding one frame, filled in by the callback of - // the session. - struct Output - { - OSStatus status = noErr; - CVImageBufferRef image = nullptr; - }; - - static void OnOutput(void * decoder, void * frame, OSStatus status, VTDecodeInfoFlags flags, - CVImageBufferRef image, CMTime presentationTime, CMTime duration); - - // Reads the stream properties from a key frame header. Returns - // false for a stream the hardware decoder does not take. - static bool ReadConfig(const webrtc::Vp9UncompressedHeader & header, StreamConfig & config); - - // Makes sure there is a session for the stream, creating or - // replacing it where the stream changed. - bool EnsureSession(const StreamConfig & config); - bool CreateSession(const StreamConfig & config, CMVideoFormatDescriptionRef format); - void DestroySession(); - - // Decodes one encoded frame as a single sample. - OSStatus DecodeSample(const webrtc::EncodedImage & image, Output & output); - - // Counts a failure and tells WebRTC what to do about it. - int32_t Fail(OSStatus status); - - void Deliver(const webrtc::EncodedImage & image, int64_t renderTimeMs, CVImageBufferRef pixels); private: - webrtc::DecodedImageCallback * callback; - - VTDecompressionSessionRef session; - CMVideoFormatDescriptionRef format; - StreamConfig active; - - // Set after a failure, until the next key frame. - bool requireKeyFrame; - int consecutiveErrors; - int64_t sampleCount; + // Makes the format description of a stream from the header of a + // key frame. Returns null for a stream the hardware decoder does + // not take. + static CMVideoFormatDescriptionRef CreateFormat(const webrtc::Vp9UncompressedHeader & header); }; } diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp index ae94a39de..e4306191e 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp @@ -19,6 +19,7 @@ #include "api/video/render_resolution.h" #include "api/video_codecs/scalability_mode.h" #include "common_video/h264/h264_common.h" +#include "common_video/h265/h265_common.h" #include "modules/video_coding/codecs/interface/common_constants.h" #include "rtc_base/logging.h" @@ -131,10 +132,19 @@ namespace jni webrtc::EncodedImage & image, webrtc::CodecSpecificInfo & info) { std::vector output; - - const bool processed = codec == webrtc::kVideoCodecAV1 - ? ProcessAv1(bitstream, keyFrame, output) - : ProcessH264(bitstream, keyFrame, output); + bool processed; + + switch (codec) { + case webrtc::kVideoCodecAV1: + processed = ProcessAv1(bitstream, keyFrame, output); + break; + case webrtc::kVideoCodecH265: + processed = ProcessH265(bitstream, keyFrame, output); + break; + default: + processed = ProcessH264(bitstream, keyFrame, output); + break; + } if (!processed) { return false; @@ -145,10 +155,16 @@ namespace jni ? webrtc::VideoFrameType::kVideoFrameKey : webrtc::VideoFrameType::kVideoFrameDelta); + const std::span encoded(output.data(), output.size()); + if (codec == webrtc::kVideoCodecH264) { - h264Parser.ParseBitstream(std::span(output.data(), output.size())); + h264Parser.ParseBitstream(encoded); image.qp_ = h264Parser.GetLastSliceQp().value_or(-1); } + else if (codec == webrtc::kVideoCodecH265) { + h265Parser.ParseBitstream(encoded); + image.qp_ = h265Parser.GetLastSliceQp().value_or(-1); + } else { image.qp_ = -1; } @@ -204,6 +220,57 @@ namespace jni return true; } + bool EncoderOutputProcessor::ProcessH265(std::span bitstream, bool & keyFrame, + std::vector & output) + { + std::vector parameterSetsFound; + bool hasVps = false; + bool hasSps = false; + bool hasPps = false; + + const std::vector nalus = webrtc::H265::FindNaluIndices(bitstream); + + if (nalus.empty()) { + return false; + } + + for (const webrtc::H265::NaluIndex & nalu : nalus) { + const size_t end = nalu.payload_start_offset + nalu.payload_size; + const uint8_t type = webrtc::H265::ParseNaluType(bitstream[nalu.payload_start_offset]); + + if (type == webrtc::H265::NaluType::kVps || type == webrtc::H265::NaluType::kSps + || type == webrtc::H265::NaluType::kPps) + { + hasVps |= type == webrtc::H265::NaluType::kVps; + hasSps |= type == webrtc::H265::NaluType::kSps; + hasPps |= type == webrtc::H265::NaluType::kPps; + + parameterSetsFound.insert(parameterSetsFound.end(), + bitstream.begin() + nalu.start_offset, bitstream.begin() + end); + } + else if (type >= webrtc::H265::NaluType::kBlaWLp && type <= webrtc::H265::NaluType::kRsvIrapVcl23) { + // An IRAP picture (IDR, CRA, BLA), which a decoder can start at. + keyFrame = true; + } + } + + if (hasVps && hasSps && hasPps) { + parameterSets = std::move(parameterSetsFound); + } + else if (keyFrame) { + if (parameterSets.empty()) { + RTC_LOG(LS_WARNING) << "H.265 encoder produced a key frame without VPS, SPS and PPS"; + return false; + } + + output.insert(output.end(), parameterSets.begin(), parameterSets.end()); + } + + output.insert(output.end(), bitstream.begin(), bitstream.end()); + + return true; + } + bool EncoderOutputProcessor::ProcessAv1(std::span bitstream, bool keyFrame, std::vector & output) { @@ -260,8 +327,8 @@ namespace jni return; } - // AV1 has no codec specific information; WebRTC sends its frame - // dependencies from these, in the dependency descriptor. + // AV1 and H.265 have no codec specific information; WebRTC sends + // their frame dependencies from these, in the dependency descriptor. auto layerFrames = svcController.NextFrameConfig(/*restart=*/keyFrame); info.generic_frame_info = svcController.OnEncodeDone(layerFrames[0]); info.scalability_mode = webrtc::ScalabilityMode::kL1T1; @@ -276,7 +343,9 @@ namespace jni webrtc::VideoEncoder::ScalingSettings EncoderOutputProcessor::GetScalingSettings() const { - if (codec == webrtc::kVideoCodecH264) { + // H.265 has no thresholds of its own in WebRTC yet and uses those of + // H.264 (bugs.webrtc.org/13485). + if (codec == webrtc::kVideoCodecH264 || codec == webrtc::kVideoCodecH265) { return webrtc::VideoEncoder::ScalingSettings(kLowH264QpThreshold, kHighH264QpThreshold); } diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/H265Format.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/H265Format.cpp new file mode 100644 index 000000000..399417322 --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/H265Format.cpp @@ -0,0 +1,33 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/H265Format.h" + +#include "media/base/media_constants.h" + +namespace jni +{ + webrtc::SdpVideoFormat CreateH265MainFormat() + { + // level-id is 30 times the level: 93 is level 3.1. + return webrtc::SdpVideoFormat(webrtc::kH265CodecName, { + { "profile-id", "1" }, + { "tier-flag", "0" }, + { "level-id", "93" }, + { "tx-mode", "SRST" } + }); + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/MacHardwareVideoCodecFactories.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/MacHardwareVideoCodecFactories.cpp index 08bc0ca71..77efb96f8 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/MacHardwareVideoCodecFactories.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/MacHardwareVideoCodecFactories.cpp @@ -17,14 +17,21 @@ #include "media/video/codec/HardwareVideoDecoderFactory.h" #include "media/video/codec/HardwareVideoEncoderFactory.h" #include "media/video/codec/macos/VTVideoDecoderFactory.h" +#include "media/video/codec/macos/VTVideoEncoderFactory.h" namespace jni { std::vector> CreatePlatformHardwareVideoEncoderFactories() { - // H.264 is encoded through VideoToolbox by the default encoders, and - // VideoToolbox has no encoder for the other codecs. - return {}; + std::vector> factories; + + // H.264 is encoded through VideoToolbox by the default encoders; + // H.265 is the codec they lack. + if (auto videoToolbox = VTVideoEncoderFactory::Create()) { + factories.push_back(std::move(videoToolbox)); + } + + return factories; } std::vector> CreatePlatformHardwareVideoDecoderFactories() @@ -32,7 +39,8 @@ namespace jni std::vector> factories; // H.264 is decoded through VideoToolbox by the default decoders; - // VP9 is the codec they decode in software. + // VP9 and AV1 are the codecs they decode in software, and H.265 one + // they lack. if (auto videoToolbox = VTVideoDecoderFactory::Create()) { factories.push_back(std::move(videoToolbox)); } diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTAv1Decoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTAv1Decoder.cpp new file mode 100644 index 000000000..bdc29d0aa --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTAv1Decoder.cpp @@ -0,0 +1,485 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/macos/VTAv1Decoder.h" +#include "media/video/codec/macos/VTCodecUtils.h" + +#include "modules/video_coding/include/video_error_codes.h" +#include "rtc_base/bitstream_reader.h" +#include "rtc_base/logging.h" + +#include + +namespace jni +{ + namespace + { + // OBU types, section 6.2.2 of the AV1 specification. + constexpr uint8_t kObuSequenceHeader = 1; + constexpr uint8_t kObuTemporalDelimiter = 2; + constexpr uint8_t kObuTileList = 8; + constexpr uint8_t kObuPadding = 15; + + struct Obu + { + uint8_t type; + uint8_t spatialId; + // The header, with the extension if there is one, without the + // size field. + std::span header; + std::span payload; + }; + + bool ReadLeb128(std::span data, size_t & position, uint64_t & value) + { + value = 0; + + for (int i = 0; i < 8; i++) { + if (position >= data.size()) { + return false; + } + + const uint8_t byte = data[position++]; + + value |= static_cast(byte & 0x7F) << (i * 7); + + if (!(byte & 0x80)) { + return true; + } + } + + return false; + } + + void WriteLeb128(uint64_t value, std::vector & output) + { + do { + uint8_t byte = value & 0x7F; + value >>= 7; + + output.push_back(value != 0 ? (byte | 0x80) : byte); + } + while (value != 0); + } + + // Splits a temporal unit into its OBUs, section 5.3 of the AV1 + // specification. An OBU without a size field runs to the end. + bool ParseObus(std::span data, std::vector & obus) + { + size_t position = 0; + + while (position < data.size()) { + const size_t offset = position; + const uint8_t header = data[position++]; + + // The forbidden bit. + if (header & 0x80) { + return false; + } + + const uint8_t type = (header >> 3) & 0x0F; + const bool extension = header & 0x04; + const bool hasSize = header & 0x02; + uint8_t spatialId = 0; + + if (extension) { + if (position >= data.size()) { + return false; + } + + spatialId = (data[position++] >> 3) & 0x03; + } + + const size_t headerSize = position - offset; + uint64_t payloadSize = 0; + + if (hasSize) { + if (!ReadLeb128(data, position, payloadSize)) { + return false; + } + } + else { + payloadSize = data.size() - position; + } + + if (payloadSize > data.size() - position) { + return false; + } + + obus.push_back(Obu { + type, + spatialId, + data.subspan(offset, headerSize), + data.subspan(position, static_cast(payloadSize)) + }); + + position += static_cast(payloadSize); + } + + return true; + } + + // Writes an OBU with its size field, which ISOBMFF wants on every OBU + // of a sample. + void WriteObu(const Obu & obu, std::vector & output) + { + output.push_back(obu.header[0] | 0x02); + output.insert(output.end(), obu.header.begin() + 1, obu.header.end()); + + WriteLeb128(obu.payload.size(), output); + + output.insert(output.end(), obu.payload.begin(), obu.payload.end()); + } + + // What the configuration box and the format description need of a + // sequence header, section 5.5 of the AV1 specification. + struct SequenceHeader + { + uint8_t profile = 0; + uint8_t level = 0; + uint8_t tier = 0; + bool highBitDepth = false; + bool twelveBit = false; + bool monochrome = false; + bool subsamplingX = true; + bool subsamplingY = true; + uint8_t chromaSamplePosition = 0; + bool fullRange = false; + uint8_t primaries = 2; + uint8_t transfer = 2; + uint8_t matrix = 2; + uint32_t maxWidth = 0; + uint32_t maxHeight = 0; + // The spatial layers of the first operating point. + int spatialLayers = 1; + }; + + uint32_t ReadUvlc(webrtc::BitstreamReader & reader) + { + int leadingZeros = 0; + + while (reader.Ok() && reader.ReadBit() == 0) { + if (++leadingZeros >= 32) { + reader.Invalidate(); + return 0; + } + } + + return leadingZeros == 0 ? 0 : static_cast(reader.ReadBits(leadingZeros) + (1ull << leadingZeros) - 1); + } + + bool ParseSequenceHeader(std::span payload, SequenceHeader & sequence) + { + webrtc::BitstreamReader reader(payload); + + sequence.profile = reader.ReadBits(3); + reader.ConsumeBits(1); // still_picture + + const bool reducedStillPictureHeader = reader.ReadBit(); + bool decoderModelInfoPresent = false; + int bufferDelayLength = 0; + + if (reducedStillPictureHeader) { + sequence.level = reader.ReadBits(5); + } + else { + const bool timingInfoPresent = reader.ReadBit(); + + if (timingInfoPresent) { + // num_units_in_display_tick, time_scale + reader.ConsumeBits(64); + + if (reader.ReadBit()) { + // num_ticks_per_picture_minus_1 + (void) ReadUvlc(reader); + } + + decoderModelInfoPresent = reader.ReadBit(); + + if (decoderModelInfoPresent) { + bufferDelayLength = reader.ReadBits(5) + 1; + // num_units_in_decoding_tick, buffer_removal_time_length_minus_1, + // frame_presentation_time_length_minus_1 + reader.ConsumeBits(32 + 5 + 5); + } + } + + const bool initialDisplayDelayPresent = reader.ReadBit(); + const int operatingPoints = reader.ReadBits(5) + 1; + + for (int i = 0; i < operatingPoints; i++) { + const uint32_t idc = reader.ReadBits(12); + const uint8_t level = reader.ReadBits(5); + const uint8_t tier = level > 7 ? reader.ReadBit() : 0; + + if (decoderModelInfoPresent && reader.ReadBit()) { + // decoder_buffer_delay, encoder_buffer_delay, low_delay_mode_flag + reader.ConsumeBits(2 * bufferDelayLength + 1); + } + if (initialDisplayDelayPresent && reader.ReadBit()) { + reader.ConsumeBits(4); + } + + if (i == 0) { + sequence.level = level; + sequence.tier = tier; + // The spatial layers are bits 8 to 11 of the idc. + sequence.spatialLayers = std::max(1, __builtin_popcount((idc >> 8) & 0x0F)); + } + } + } + + const int widthBits = reader.ReadBits(4) + 1; + const int heightBits = reader.ReadBits(4) + 1; + + sequence.maxWidth = static_cast(reader.ReadBits(widthBits)) + 1; + sequence.maxHeight = static_cast(reader.ReadBits(heightBits)) + 1; + + if (!reducedStillPictureHeader && reader.ReadBit()) { + // delta_frame_id_length_minus_2, additional_frame_id_length_minus_1 + reader.ConsumeBits(4 + 3); + } + + // use_128x128_superblock, enable_filter_intra, enable_intra_edge_filter + reader.ConsumeBits(3); + + if (!reducedStillPictureHeader) { + // enable_interintra_compound, enable_masked_compound, + // enable_warped_motion, enable_dual_filter + reader.ConsumeBits(4); + + const bool enableOrderHint = reader.ReadBit(); + + if (enableOrderHint) { + // enable_jnt_comp, enable_ref_frame_mvs + reader.ConsumeBits(2); + } + + const bool chooseScreenContentTools = reader.ReadBit(); + int forceScreenContentTools = 2; + + if (!chooseScreenContentTools) { + forceScreenContentTools = reader.ReadBit(); + } + if (forceScreenContentTools > 0) { + // seq_choose_integer_mv, and seq_force_integer_mv without it + if (!reader.ReadBit()) { + reader.ConsumeBits(1); + } + } + if (enableOrderHint) { + reader.ConsumeBits(3); + } + } + + // enable_superres, enable_cdef, enable_restoration + reader.ConsumeBits(3); + + // color_config + sequence.highBitDepth = reader.ReadBit(); + + if (sequence.profile == 2 && sequence.highBitDepth) { + sequence.twelveBit = reader.ReadBit(); + } + + sequence.monochrome = sequence.profile == 1 ? false : reader.ReadBit(); + + if (reader.ReadBit()) { + sequence.primaries = reader.ReadBits(8); + sequence.transfer = reader.ReadBits(8); + sequence.matrix = reader.ReadBits(8); + } + + if (sequence.monochrome) { + sequence.fullRange = reader.ReadBit(); + } + else if (sequence.primaries == 1 && sequence.transfer == 13 && sequence.matrix == 0) { + // sRGB: full range 4:4:4. + sequence.fullRange = true; + sequence.subsamplingX = false; + sequence.subsamplingY = false; + } + else { + sequence.fullRange = reader.ReadBit(); + + if (sequence.profile == 1) { + sequence.subsamplingX = false; + sequence.subsamplingY = false; + } + else if (sequence.profile == 2) { + if (sequence.twelveBit) { + sequence.subsamplingX = reader.ReadBit(); + sequence.subsamplingY = sequence.subsamplingX ? reader.ReadBit() : false; + } + else { + sequence.subsamplingX = true; + sequence.subsamplingY = false; + } + } + + if (sequence.subsamplingX && sequence.subsamplingY) { + sequence.chromaSamplePosition = reader.ReadBits(2); + } + } + + return reader.Ok(); + } + } + + VTAv1Decoder::VTAv1Decoder() : + VTVideoDecoder("VideoToolbox (AV1)", true) + { + } + + bool VTAv1Decoder::Configure(const Settings & settings) + { + if (settings.codec_type() != webrtc::kVideoCodecAV1) { + return false; + } + + Release(); + + // The session waits for the first key frame, whose sequence header + // tells what to create it for. + return true; + } + + int32_t VTAv1Decoder::Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) + { + if (image.size() == 0) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } + + std::vector obus; + + if (!ParseObus(std::span(image.data(), image.size()), obus) || obus.empty()) { + return Fail(kVTVideoDecoderBadDataErr); + } + + const bool keyFrame = image.FrameType() == webrtc::VideoFrameType::kVideoFrameKey; + std::vector sequenceHeader; + + sample.clear(); + + for (const Obu & obu : obus) { + if (obu.type == kObuTemporalDelimiter || obu.type == kObuTileList || obu.type == kObuPadding) { + continue; + } + // The frames of the higher spatial layers of a temporal unit are + // left to dav1d. + if (obu.spatialId > 0) { + return Unsupported(); + } + if (obu.type == kObuSequenceHeader && sequenceHeader.empty()) { + WriteObu(obu, sequenceHeader); + } + + WriteObu(obu, sample); + } + + if (keyFrame) { + // A key frame without a sequence header goes on with the session + // there is. + if (!sequenceHeader.empty()) { + CMVideoFormatDescriptionRef format = CreateFormat(sequenceHeader); + + if (format == nullptr || !SetFormat(format)) { + return Unsupported(); + } + } + if (HasSession()) { + requireKeyFrame = false; + } + } + + if (!HasSession() || requireKeyFrame) { + // Nothing to build on: WebRTC asks the sender for a key frame. + return WEBRTC_VIDEO_CODEC_ERROR; + } + + return DecodeSample(sample, image, renderTimeMs); + } + + CMVideoFormatDescriptionRef VTAv1Decoder::CreateFormat(std::span sequenceHeaderObu) + { + std::vector obus; + SequenceHeader sequence; + + if (!ParseObus(sequenceHeaderObu, obus) || obus.size() != 1 + || !ParseSequenceHeader(obus[0].payload, sequence)) + { + RTC_LOG(LS_WARNING) << "VideoToolbox AV1 decoder: the sequence header cannot be parsed"; + return nullptr; + } + + // The negotiated format is profile 0. Check, rather than trust it, and + // leave what the output format does not hold to dav1d. + if (sequence.profile != 0 || sequence.highBitDepth || sequence.monochrome) { + RTC_LOG(LS_INFO) << "VideoToolbox AV1 decoder: profile " << static_cast(sequence.profile) + << (sequence.highBitDepth ? ", high bit depth" : "") + << (sequence.monochrome ? ", monochrome" : "") << " is not decoded in hardware"; + return nullptr; + } + if (sequence.spatialLayers > 1) { + RTC_LOG(LS_INFO) << "VideoToolbox AV1 decoder: spatial layers are not decoded in hardware"; + return nullptr; + } + + // The AV1 codec configuration box, section 2.3 of the AV1 ISOBMFF + // binding: marker and version 1, the profile and level, the tier, + // bit depth and chroma format, no initial presentation delay, and the + // sequence header as its configuration OBU. + std::vector box = { + 0x81, + static_cast((sequence.profile << 5) | (sequence.level & 0x1F)), + static_cast((sequence.tier << 7) | (sequence.highBitDepth << 6) | (sequence.twelveBit << 5) + | (sequence.monochrome << 4) | (sequence.subsamplingX << 3) | (sequence.subsamplingY << 2) + | (sequence.chromaSamplePosition & 0x03)), + 0 + }; + box.insert(box.end(), sequenceHeaderObu.begin(), sequenceHeaderObu.end()); + + ScopedCF boxData(CFDataCreate(kCFAllocatorDefault, box.data(), box.size())); + ScopedCF atoms = CreateCFDictionary(); + ScopedCF extensions = CreateCFDictionary(); + + if (!boxData || !atoms || !extensions) { + return nullptr; + } + + CFDictionarySetValue(atoms.get(), CFSTR("av1C"), boxData.get()); + CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms, + atoms.get()); + CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_FullRangeVideo, + sequence.fullRange ? kCFBooleanTrue : kCFBooleanFalse); + + SetColorExtensions(extensions.get(), sequence.primaries, sequence.transfer, sequence.matrix); + + // Frames may be smaller than the largest size the sequence allows, + // which is the size the format describes. + CMVideoFormatDescriptionRef format = nullptr; + OSStatus status = CMVideoFormatDescriptionCreate(kCFAllocatorDefault, kCMVideoCodecType_AV1, + static_cast(sequence.maxWidth), static_cast(sequence.maxHeight), extensions.get(), + &format); + + if (status != noErr) { + RTC_LOG(LS_WARNING) << "VideoToolbox AV1 format description failed, status " << status; + return nullptr; + } + + return format; + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTCodecUtils.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTCodecUtils.cpp new file mode 100644 index 000000000..63ea4434f --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTCodecUtils.cpp @@ -0,0 +1,108 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/macos/VTCodecUtils.h" + +namespace jni +{ + namespace + { + CFStringRef GetPrimaries(uint8_t primaries) + { + switch (primaries) { + case 1: return kCMFormatDescriptionColorPrimaries_ITU_R_709_2; + case 5: return kCMFormatDescriptionColorPrimaries_EBU_3213; + case 6: + case 7: return kCMFormatDescriptionColorPrimaries_SMPTE_C; + case 9: return kCMFormatDescriptionColorPrimaries_ITU_R_2020; + default: return nullptr; + } + } + + CFStringRef GetTransfer(uint8_t transfer) + { + switch (transfer) { + // BT.601 has the transfer function of BT.709. + case 1: + case 6: return kCMFormatDescriptionTransferFunction_ITU_R_709_2; + case 7: return kCMFormatDescriptionTransferFunction_SMPTE_240M_1995; + case 14: + case 15: return kCMFormatDescriptionTransferFunction_ITU_R_2020; + case 16: return kCMFormatDescriptionTransferFunction_SMPTE_ST_2084_PQ; + case 18: return kCMFormatDescriptionTransferFunction_ITU_R_2100_HLG; + default: return nullptr; + } + } + + CFStringRef GetMatrix(uint8_t matrix) + { + switch (matrix) { + case 1: return kCMFormatDescriptionYCbCrMatrix_ITU_R_709_2; + case 5: + case 6: return kCMFormatDescriptionYCbCrMatrix_ITU_R_601_4; + case 7: return kCMFormatDescriptionYCbCrMatrix_SMPTE_240M_1995; + case 9: return kCMFormatDescriptionYCbCrMatrix_ITU_R_2020; + default: return nullptr; + } + } + } + + ScopedCF CreateCFDictionary() + { + return ScopedCF(CFDictionaryCreateMutable(kCFAllocatorDefault, 0, + &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks)); + } + + bool SetCFNumber(CFMutableDictionaryRef dictionary, CFStringRef key, int64_t value) + { + ScopedCF number(CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt64Type, &value)); + + if (!number) { + return false; + } + + CFDictionarySetValue(dictionary, key, number.get()); + + return true; + } + + void SetColorExtensions(CFMutableDictionaryRef extensions, uint8_t primaries, uint8_t transfer, + uint8_t matrix) + { + if (CFStringRef value = GetPrimaries(primaries)) { + CFDictionarySetValue(extensions, kCMFormatDescriptionExtension_ColorPrimaries, value); + } + if (CFStringRef value = GetTransfer(transfer)) { + CFDictionarySetValue(extensions, kCMFormatDescriptionExtension_TransferFunction, value); + } + if (CFStringRef value = GetMatrix(matrix)) { + CFDictionarySetValue(extensions, kCMFormatDescriptionExtension_YCbCrMatrix, value); + } + } + + bool IsUsingHardware(VTSessionRef session, CFStringRef property) + { + CFBooleanRef hardware = nullptr; + bool usingHardware = false; + + if (VTSessionCopyProperty(session, property, kCFAllocatorDefault, &hardware) == noErr && hardware != nullptr) { + usingHardware = CFBooleanGetValue(hardware); + CFRelease(hardware); + } + + return usingHardware; + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTH265Decoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTH265Decoder.cpp new file mode 100644 index 000000000..d328371ff --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTH265Decoder.cpp @@ -0,0 +1,149 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/macos/VTH265Decoder.h" + +#include "common_video/h265/h265_common.h" +#include "modules/video_coding/include/video_error_codes.h" +#include "rtc_base/logging.h" + +#include + +namespace jni +{ + namespace + { + // The length prefix of each NAL unit in a sample. + constexpr int kNaluLengthSize = 4; + + bool IsParameterSet(uint8_t type) + { + return type == webrtc::H265::NaluType::kVps || type == webrtc::H265::NaluType::kSps + || type == webrtc::H265::NaluType::kPps; + } + + // An IRAP picture (IDR, CRA, BLA), which a decoder can start at. + bool IsIrap(uint8_t type) + { + return type >= webrtc::H265::NaluType::kBlaWLp && type <= webrtc::H265::NaluType::kRsvIrapVcl23; + } + } + + VTH265Decoder::VTH265Decoder() : + VTVideoDecoder("VideoToolbox (H.265)", false) + { + } + + bool VTH265Decoder::Configure(const Settings & settings) + { + if (settings.codec_type() != webrtc::kVideoCodecH265) { + return false; + } + + Release(); + + // The session waits for the first key frame, whose parameter sets + // tell what to create it for. + return true; + } + + int32_t VTH265Decoder::Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) + { + if (image.size() == 0) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } + + const std::span data(image.data(), image.size()); + const std::vector nalus = webrtc::H265::FindNaluIndices(data); + + if (nalus.empty()) { + return WEBRTC_VIDEO_CODEC_ERROR; + } + + std::vector parameterSets; + std::vector parameterSetSizes; + bool hasVps = false; + bool hasSps = false; + bool hasPps = false; + bool keyFrame = image.FrameType() == webrtc::VideoFrameType::kVideoFrameKey; + + sample.clear(); + + for (const webrtc::H265::NaluIndex & nalu : nalus) { + if (nalu.payload_size == 0) { + continue; + } + + const uint8_t * payload = data.data() + nalu.payload_start_offset; + const uint8_t type = webrtc::H265::ParseNaluType(payload[0]); + + if (IsParameterSet(type)) { + hasVps |= type == webrtc::H265::NaluType::kVps; + hasSps |= type == webrtc::H265::NaluType::kSps; + hasPps |= type == webrtc::H265::NaluType::kPps; + + parameterSets.push_back(payload); + parameterSetSizes.push_back(nalu.payload_size); + + // The format description holds them; the sample does not. + continue; + } + if (type == webrtc::H265::NaluType::kAud) { + continue; + } + + keyFrame |= IsIrap(type); + + const uint32_t size = static_cast(nalu.payload_size); + + for (int shift = (kNaluLengthSize - 1) * 8; shift >= 0; shift -= 8) { + sample.push_back(static_cast(size >> shift)); + } + + sample.insert(sample.end(), payload, payload + nalu.payload_size); + } + + if (hasVps && hasSps && hasPps) { + CMVideoFormatDescriptionRef format = nullptr; + OSStatus status = CMVideoFormatDescriptionCreateFromHEVCParameterSets(kCFAllocatorDefault, + parameterSets.size(), parameterSets.data(), parameterSetSizes.data(), kNaluLengthSize, nullptr, + &format); + + if (status != noErr) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 format description failed, status " << status; + return Unsupported(); + } + if (!SetFormat(format)) { + return Unsupported(); + } + } + + if (!HasSession() || (requireKeyFrame && !keyFrame)) { + // Nothing to build on: WebRTC asks the sender for a key frame. + requireKeyFrame = true; + return WEBRTC_VIDEO_CODEC_ERROR; + } + + requireKeyFrame = false; + + if (sample.empty()) { + // Parameter sets alone have no picture. + return WEBRTC_VIDEO_CODEC_OK; + } + + return DecodeSample(sample, image, renderTimeMs); + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTH265Encoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTH265Encoder.cpp new file mode 100644 index 000000000..f548c227b --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTH265Encoder.cpp @@ -0,0 +1,580 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/macos/VTH265Encoder.h" +#include "media/video/codec/macos/VTCodecUtils.h" + +#include "api/video_codecs/scalability_mode.h" +#include "modules/video_coding/include/video_error_codes.h" +#include "rtc_base/logging.h" +#include "rtc_base/time_utils.h" +#include "sdk/objc/components/video_frame_buffer/RTCCVPixelBuffer.h" +#include "sdk/objc/native/src/objc_frame_buffer.h" +#include "third_party/libyuv/include/libyuv/convert_from.h" + +#include +#include +#include + +namespace jni +{ + namespace + { + // The rate may go above the average by this much within a second, as + // WebRTC allows its own VideoToolbox H.264 encoder. + constexpr double kDataRateLimitFactor = 1.5; + + // Key frames come when WebRTC asks for them; these only bound how + // long a stream goes without one, as for WebRTC's H.264 encoder. + constexpr int kMaxKeyFrameInterval = 7200; + constexpr int kMaxKeyFrameIntervalSeconds = 240; + + constexpr uint8_t kStartCode[] = { 0, 0, 0, 1 }; + + void SetProperty(VTCompressionSessionRef session, CFStringRef key, CFTypeRef value) + { + OSStatus status = VTSessionSetProperty(session, key, value); + + if (status != noErr) { + RTC_LOG(LS_INFO) << "VideoToolbox H.265 encoder: a property is not supported, status " << status; + } + } + + void SetIntProperty(VTCompressionSessionRef session, CFStringRef key, int64_t value) + { + ScopedCF number(CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt64Type, &value)); + + if (number) { + SetProperty(session, key, number.get()); + } + } + + bool IsKeyFrame(CMSampleBufferRef sample) + { + CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sample, false); + + if (attachments == nullptr || CFArrayGetCount(attachments) == 0) { + return true; + } + + auto attachment = static_cast(CFArrayGetValueAtIndex(attachments, 0)); + + return !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync); + } + } + + VTH265Encoder::VTH265Encoder(const webrtc::SdpVideoFormat & format) : + codecSettings(), + bitrateBps(0), + framerate(30), + session(nullptr), + callback(nullptr), + failed(false), + lastTimestampUs(-1), + outputProcessor(webrtc::kVideoCodecH265, format) + { + } + + VTH265Encoder::~VTH265Encoder() + { + Release(); + } + + int32_t VTH265Encoder::InitEncode(const webrtc::VideoCodec * settings, const Settings & encoderSettings) + { + if (settings == nullptr || settings->width == 0 || settings->height == 0) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } + if (settings->numberOfSimulcastStreams > 1) { + return WEBRTC_VIDEO_CODEC_ERR_SIMULCAST_PARAMETERS_NOT_SUPPORTED; + } + // A single layer; there is no software encoder for anything else. + if (settings->GetScalabilityMode().value_or(webrtc::ScalabilityMode::kL1T1) != webrtc::ScalabilityMode::kL1T1) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } + // 4:2:0 has chroma at half the resolution in both directions. The + // first frame of a source can be odd, since WebRTC asks the source + // for requested_resolution_alignment only once the encoder has been + // set up, so the encoder takes the even size below it and drops the + // last row and column of such frames. + if (settings->width < 2 || settings->height < 2) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } + + Release(); + + codecSettings = *settings; + codecSettings.width &= ~1; + codecSettings.height &= ~1; + bitrateBps = std::max(1u, codecSettings.startBitrate) * 1000; + framerate = std::max(1u, codecSettings.maxFramerate); + + if (!CreateSession()) { + Release(); + + return WEBRTC_VIDEO_CODEC_ERROR; + } + + RTC_LOG(LS_INFO) << "VideoToolbox H.265 encoder initialized: " << codecSettings.width << "x" + << codecSettings.height << " at " << bitrateBps << " bps"; + + return WEBRTC_VIDEO_CODEC_OK; + } + + bool VTH265Encoder::CreateSession() + { + ScopedCF specification = CreateCFDictionary(); + ScopedCF attributes = CreateCFDictionary(); + ScopedCF surfaceProperties(CFDictionaryCreate(kCFAllocatorDefault, nullptr, nullptr, 0, + &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks)); + + if (!specification || !attributes || !surfaceProperties) { + return false; + } + + // Only the hardware encoder: where there is none, or it is busy, the + // session fails to create, and WebRTC goes on with the next codec. + CFDictionarySetValue(specification.get(), kVTVideoEncoderSpecification_EnableHardwareAcceleratedVideoEncoder, + kCFBooleanTrue); + CFDictionarySetValue(specification.get(), kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder, + kCFBooleanTrue); + + // The buffers of the pool, which frames that are not in a + // CVPixelBuffer already are converted into. Studio range, as WebRTC's + // I420 frames are. + if (!SetCFNumber(attributes.get(), kCVPixelBufferPixelFormatTypeKey, + kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) + || !SetCFNumber(attributes.get(), kCVPixelBufferWidthKey, codecSettings.width) + || !SetCFNumber(attributes.get(), kCVPixelBufferHeightKey, codecSettings.height)) + { + return false; + } + + CFDictionarySetValue(attributes.get(), kCVPixelBufferIOSurfacePropertiesKey, surfaceProperties.get()); + + OSStatus status = VTCompressionSessionCreate(kCFAllocatorDefault, codecSettings.width, codecSettings.height, + kCMVideoCodecType_HEVC, specification.get(), attributes.get(), nullptr, OnOutput, this, &session); + + if (status != noErr) { + RTC_LOG(LS_INFO) << "VideoToolbox H.265 encoder: no session, status " << status; + session = nullptr; + + return false; + } + + if (!IsUsingHardware(session, kVTCompressionPropertyKey_UsingHardwareAcceleratedVideoEncoder)) { + RTC_LOG(LS_INFO) << "VideoToolbox H.265 encoder: the session does not encode in hardware"; + return false; + } + + SetProperty(session, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue); + SetProperty(session, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_HEVC_Main_AutoLevel); + SetProperty(session, kVTCompressionPropertyKey_AllowFrameReordering, kCFBooleanFalse); + SetIntProperty(session, kVTCompressionPropertyKey_MaxKeyFrameInterval, kMaxKeyFrameInterval); + SetIntProperty(session, kVTCompressionPropertyKey_MaxKeyFrameIntervalDuration, kMaxKeyFrameIntervalSeconds); + + ConfigureRates(); + + status = VTCompressionSessionPrepareToEncodeFrames(session); + + if (status != noErr) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 encoder failed to prepare, status " << status; + return false; + } + + return true; + } + + void VTH265Encoder::ConfigureRates() + { + SetIntProperty(session, kVTCompressionPropertyKey_AverageBitRate, bitrateBps); + SetIntProperty(session, kVTCompressionPropertyKey_ExpectedFrameRate, framerate); + + // Bytes over a duration in seconds. + const int64_t bytes = static_cast(bitrateBps * kDataRateLimitFactor / 8); + const double seconds = 1.0; + + ScopedCF bytesNumber(CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt64Type, &bytes)); + ScopedCF secondsNumber(CFNumberCreate(kCFAllocatorDefault, kCFNumberDoubleType, &seconds)); + + if (bytesNumber && secondsNumber) { + const void * values[] = { bytesNumber.get(), secondsNumber.get() }; + ScopedCF limits(CFArrayCreate(kCFAllocatorDefault, values, 2, &kCFTypeArrayCallBacks)); + + if (limits) { + SetProperty(session, kVTCompressionPropertyKey_DataRateLimits, limits.get()); + } + } + } + + int32_t VTH265Encoder::RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * encodeCallback) + { + callback = encodeCallback; + + return WEBRTC_VIDEO_CODEC_OK; + } + + int32_t VTH265Encoder::Release() + { + DestroySession(); + + failed = false; + lastTimestampUs = -1; + + // No output arrives any more. + outputProcessor.Reset(); + + return WEBRTC_VIDEO_CODEC_OK; + } + + void VTH265Encoder::DestroySession() + { + if (session == nullptr) { + return; + } + + // The frames still in the session come out, and are handed on, before + // it goes; their pending entries go with them. + VTCompressionSessionCompleteFrames(session, kCMTimeInvalid); + VTCompressionSessionInvalidate(session); + CFRelease(session); + + session = nullptr; + } + + int32_t VTH265Encoder::Encode(const webrtc::VideoFrame & frame, + const std::vector * frameTypes) + { + webrtc::EncodedImageCallback * encodeCallback = callback.load(); + + if (session == nullptr || encodeCallback == nullptr) { + return WEBRTC_VIDEO_CODEC_UNINITIALIZED; + } + // There is no software encoder to go on with: an error makes WebRTC + // switch to the next codec. + if (failed) { + return WEBRTC_VIDEO_CODEC_ERROR; + } + + bool keyFrame = false; + + if (frameTypes != nullptr) { + for (webrtc::VideoFrameType type : *frameTypes) { + keyFrame |= type == webrtc::VideoFrameType::kVideoFrameKey; + } + } + + ScopedCF pixelBuffer(CreatePixelBuffer(frame)); + + if (!pixelBuffer) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 encoder: no pixel buffer for a frame"; + + encodeCallback->OnFrameDropped(frame.rtp_timestamp(), 0, true); + + return WEBRTC_VIDEO_CODEC_OK; + } + + ScopedCF frameProperties(keyFrame ? CreateCFDictionary().release() : nullptr); + + if (frameProperties) { + CFDictionarySetValue(frameProperties.get(), kVTEncodeFrameOptionKey_ForceKeyFrame, kCFBooleanTrue); + } + + // Rate control goes by the presentation times, which have to rise. + int64_t timestampUs = frame.timestamp_us(); + + if (timestampUs <= lastTimestampUs) { + timestampUs = lastTimestampUs + 1; + } + + lastTimestampUs = timestampUs; + + // The output callback owns it. Should the session reject the frame + // without calling back, it is lost, as with WebRTC's H.264 encoder, + // rather than freed twice. + auto * pending = new PendingFrame { + this, + frame.rtp_timestamp(), + frame.render_time_ms(), + frame.ntp_time_ms(), + frame.rotation() + }; + + OSStatus status = VTCompressionSessionEncodeFrame(session, pixelBuffer.get(), + CMTimeMake(timestampUs, webrtc::kNumMicrosecsPerSec), kCMTimeInvalid, frameProperties.get(), + pending, nullptr); + + if (status != noErr) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 encoder rejected a frame, status " << status; + + failed = true; + + return WEBRTC_VIDEO_CODEC_ERROR; + } + + return WEBRTC_VIDEO_CODEC_OK; + } + + CVPixelBufferRef VTH265Encoder::CreatePixelBuffer(const webrtc::VideoFrame & frame) + { + const int width = static_cast(codecSettings.width); + const int height = static_cast(codecSettings.height); + + webrtc::scoped_refptr buffer = frame.video_frame_buffer(); + + CVPixelBufferPoolRef pool = VTCompressionSessionGetPixelBufferPool(session); + + if (pool == nullptr) { + return nullptr; + } + + // The native buffers on macOS are those of the camera and of the + // VideoToolbox decoders. + if (buffer->type() == webrtc::VideoFrameBuffer::Type::kNative) { + id objcBuffer = + static_cast(buffer.get())->wrapped_frame_buffer(); + + if ([objcBuffer isKindOfClass:[RTC_OBJC_TYPE(RTCCVPixelBuffer) class]]) { + RTC_OBJC_TYPE(RTCCVPixelBuffer) * pixelBuffer = (RTC_OBJC_TYPE(RTCCVPixelBuffer) *) objcBuffer; + + // The session scales a buffer of another size itself. + if (![pixelBuffer requiresCropping]) { + return CVPixelBufferRetain(pixelBuffer.pixelBuffer); + } + + CVPixelBufferRef output = nullptr; + + if (CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pool, &output) != kCVReturnSuccess) { + return nullptr; + } + + if ([pixelBuffer requiresScalingToWidth:width height:height]) { + scaleBuffer.resize([pixelBuffer bufferSizeForCroppingAndScalingToWidth:width height:height]); + } + + if (![pixelBuffer cropAndScaleTo:output withTempBuffer:scaleBuffer.data()]) { + CVPixelBufferRelease(output); + return nullptr; + } + + return output; + } + } + + // An odd frame is cut to the even size of the session, anything else + // of another size is scaled to it. + if (buffer->width() < width || buffer->height() < height + || buffer->width() > width + 1 || buffer->height() > height + 1) + { + buffer = buffer->Scale(width, height); + } + + webrtc::scoped_refptr i420 = buffer->ToI420(); + + if (!i420) { + return nullptr; + } + + CVPixelBufferRef output = nullptr; + + if (CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pool, &output) != kCVReturnSuccess) { + return nullptr; + } + if (CVPixelBufferLockBaseAddress(output, 0) != kCVReturnSuccess) { + CVPixelBufferRelease(output); + return nullptr; + } + + int result = libyuv::I420ToNV12(i420->DataY(), i420->StrideY(), i420->DataU(), i420->StrideU(), + i420->DataV(), i420->StrideV(), + static_cast(CVPixelBufferGetBaseAddressOfPlane(output, 0)), + static_cast(CVPixelBufferGetBytesPerRowOfPlane(output, 0)), + static_cast(CVPixelBufferGetBaseAddressOfPlane(output, 1)), + static_cast(CVPixelBufferGetBytesPerRowOfPlane(output, 1)), + width, height); + + CVPixelBufferUnlockBaseAddress(output, 0); + + if (result != 0) { + CVPixelBufferRelease(output); + return nullptr; + } + + return output; + } + + void VTH265Encoder::SetRates(const RateControlParameters & parameters) + { + const uint32_t bitrate = parameters.bitrate.get_sum_bps(); + + // A zero bitrate pauses the stream; the encoder gets no frames then. + if (bitrate == 0) { + return; + } + + bitrateBps = bitrate; + + if (parameters.framerate_fps >= 1.0) { + framerate = static_cast(parameters.framerate_fps + 0.5); + } + + if (session != nullptr) { + ConfigureRates(); + } + } + + webrtc::VideoEncoder::EncoderInfo VTH265Encoder::GetEncoderInfo() const + { + EncoderInfo info; + info.implementation_name = "VideoToolbox (H.265)"; + info.is_hardware_accelerated = true; + // CVPixelBuffers are taken as they are. + info.supports_native_handle = true; + info.supports_simulcast = false; + info.scaling_settings = outputProcessor.GetScalingSettings(); + // 4:2:0 needs even dimensions. + info.requested_resolution_alignment = 2; + + return info; + } + + void VTH265Encoder::OnOutput(void * encoder, void * frame, OSStatus status, VTEncodeInfoFlags flags, + CMSampleBufferRef sample) + { + std::unique_ptr pending(static_cast(frame)); + + if (pending == nullptr) { + return; + } + + VTH265Encoder * self = pending->encoder; + + if (status != noErr) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 encoder failed, status " << status; + + self->failed = true; + return; + } + + webrtc::EncodedImageCallback * encodeCallback = self->callback.load(); + + if ((flags & kVTEncodeInfo_FrameDropped) || sample == nullptr) { + if (encodeCallback != nullptr) { + encodeCallback->OnFrameDropped(pending->rtpTimestamp, 0, true); + } + return; + } + + self->DeliverOutput(*pending, sample); + } + + void VTH265Encoder::DeliverOutput(const PendingFrame & pending, CMSampleBufferRef sample) + { + CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sample); + CMBlockBufferRef block = CMSampleBufferGetDataBuffer(sample); + + if (format == nullptr || block == nullptr) { + failed = true; + return; + } + + const bool keyFrame = IsKeyFrame(sample); + size_t parameterSetCount = 0; + int lengthSize = 0; + + OSStatus status = CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(format, 0, nullptr, nullptr, + ¶meterSetCount, &lengthSize); + + if (status != noErr || lengthSize < 1 || lengthSize > 4) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 encoder: no parameter sets, status " << status; + failed = true; + return; + } + + annexB.clear(); + + // The parameter sets are in the format description only; a receiver + // needs them in front of each key frame. + if (keyFrame) { + for (size_t i = 0; i < parameterSetCount; i++) { + const uint8_t * parameterSet = nullptr; + size_t size = 0; + + status = CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(format, i, ¶meterSet, &size, + nullptr, nullptr); + + if (status != noErr) { + failed = true; + return; + } + + annexB.insert(annexB.end(), std::begin(kStartCode), std::end(kStartCode)); + annexB.insert(annexB.end(), parameterSet, parameterSet + size); + } + } + + // The NAL units come with length prefixes, which become start codes. + const size_t length = CMBlockBufferGetDataLength(block); + std::vector data(length); + + if (CMBlockBufferCopyDataBytes(block, 0, length, data.data()) != kCMBlockBufferNoErr) { + failed = true; + return; + } + + size_t position = 0; + + while (position + lengthSize <= length) { + size_t naluSize = 0; + + for (int i = 0; i < lengthSize; i++) { + naluSize = (naluSize << 8) | data[position++]; + } + + if (naluSize > length - position) { + RTC_LOG(LS_WARNING) << "VideoToolbox H.265 encoder: a NAL unit runs past the frame"; + failed = true; + return; + } + + annexB.insert(annexB.end(), std::begin(kStartCode), std::end(kStartCode)); + annexB.insert(annexB.end(), data.begin() + position, data.begin() + position + naluSize); + + position += naluSize; + } + + webrtc::EncodedImage image; + image._encodedWidth = codecSettings.width; + image._encodedHeight = codecSettings.height; + image.SetRtpTimestamp(pending.rtpTimestamp); + image.capture_time_ms_ = pending.captureTimeMs; + image.ntp_time_ms_ = pending.ntpTimeMs; + image.rotation_ = pending.rotation; + + webrtc::CodecSpecificInfo info; + + if (!outputProcessor.Process(annexB, keyFrame, image, info)) { + failed = true; + return; + } + + webrtc::EncodedImageCallback * encodeCallback = callback.load(); + + if (encodeCallback != nullptr) { + encodeCallback->OnEncodedImage(image, &info); + } + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoder.cpp new file mode 100644 index 000000000..b8250324a --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoder.cpp @@ -0,0 +1,362 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/macos/VTVideoDecoder.h" +#include "media/video/codec/macos/VTCodecUtils.h" + +#include "api/make_ref_counted.h" +#include "api/video/video_frame.h" +#include "modules/video_coding/include/video_error_codes.h" +#include "rtc_base/logging.h" +#include "sdk/objc/components/video_frame_buffer/RTCCVPixelBuffer.h" +#include "sdk/objc/native/src/objc_frame_buffer.h" + +#include + +namespace jni +{ + namespace + { + // A decoder that fails this often in a row, each time on a frame + // WebRTC then replaced by a key frame, is given up on. + constexpr int kMaxConsecutiveErrors = 3; + + // How many sessions of codecs with a software decoder the process + // keeps at a time. The media engine serves a limited number of + // streams; a stream past the limit starts in software. The number is a + // cautious guess. A codec without a software decoder has nowhere to + // go, and asks VideoToolbox regardless. + constexpr int kMaxSessions = 8; + + std::atomic sessionCount(0); + + bool IsFullRange(CMVideoFormatDescriptionRef format) + { + CFTypeRef value = CMFormatDescriptionGetExtension(format, kCMFormatDescriptionExtension_FullRangeVideo); + + return value != nullptr && CFGetTypeID(value) == CFBooleanGetTypeID() + && CFBooleanGetValue(static_cast(value)); + } + } + + VTVideoDecoder::VTVideoDecoder(const char * name, bool softwareFallback) : + requireKeyFrame(false), + name(name), + softwareFallback(softwareFallback), + callback(nullptr), + session(nullptr), + format(nullptr), + fullRange(false), + consecutiveErrors(0), + sampleCount(0) + { + } + + VTVideoDecoder::~VTVideoDecoder() + { + DestroySession(); + } + + int32_t VTVideoDecoder::RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * decodeCallback) + { + callback = decodeCallback; + + return WEBRTC_VIDEO_CODEC_OK; + } + + int32_t VTVideoDecoder::Release() + { + DestroySession(); + + requireKeyFrame = false; + consecutiveErrors = 0; + + return WEBRTC_VIDEO_CODEC_OK; + } + + webrtc::VideoDecoder::DecoderInfo VTVideoDecoder::GetDecoderInfo() const + { + DecoderInfo info; + info.implementation_name = ImplementationName(); + info.is_hardware_accelerated = true; + + return info; + } + + const char * VTVideoDecoder::ImplementationName() const + { + return name; + } + + bool VTVideoDecoder::SetFormat(CMVideoFormatDescriptionRef newFormat) + { + ScopedCF scopedFormat(newFormat); + + if (session != nullptr && CMFormatDescriptionEqual(newFormat, format)) { + return true; + } + + // The pixel format of the output follows the range, so a session does + // not outlast a change of it. Another size is not accepted in place + // either, but a change of the colour space may be. + if (session != nullptr && IsFullRange(newFormat) == fullRange + && VTDecompressionSessionCanAcceptFormatDescription(session, newFormat)) + { + CFRelease(format); + format = scopedFormat.release(); + + return true; + } + + // Whatever is still decoding finishes before the session goes. + DestroySession(); + + if (!CreateSession(newFormat)) { + return false; + } + + format = scopedFormat.release(); + + return true; + } + + bool VTVideoDecoder::HasSession() const + { + return session != nullptr; + } + + CMVideoDimensions VTVideoDecoder::GetDimensions() const + { + if (format == nullptr) { + return CMVideoDimensions { 0, 0 }; + } + + return CMVideoFormatDescriptionGetDimensions(format); + } + + bool VTVideoDecoder::CreateSession(CMVideoFormatDescriptionRef newFormat) + { + if (softwareFallback && sessionCount.load() >= kMaxSessions) { + RTC_LOG(LS_INFO) << name << ": too many sessions"; + return false; + } + + const bool newFullRange = IsFullRange(newFormat); + + ScopedCF specification = CreateCFDictionary(); + ScopedCF attributes = CreateCFDictionary(); + ScopedCF surfaceProperties(CFDictionaryCreate(kCFAllocatorDefault, nullptr, nullptr, 0, + &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks)); + + if (!specification || !attributes || !surfaceProperties) { + return false; + } + + // The output is 8 bit 4:2:0, in the range of the stream, so that the + // samples reach WebRTC as the software decoder would give them. + const int pixelFormat = newFullRange + ? kCVPixelFormatType_420YpCbCr8BiPlanarFullRange + : kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange; + + if (!SetCFNumber(attributes.get(), kCVPixelBufferPixelFormatTypeKey, pixelFormat)) { + return false; + } + + // Only the hardware decoder: where there is none, or it is busy, the + // session fails to create. + CFDictionarySetValue(specification.get(), kVTVideoDecoderSpecification_EnableHardwareAcceleratedVideoDecoder, + kCFBooleanTrue); + CFDictionarySetValue(specification.get(), kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder, + kCFBooleanTrue); + + CFDictionarySetValue(attributes.get(), kCVPixelBufferIOSurfacePropertiesKey, surfaceProperties.get()); + + VTDecompressionOutputCallbackRecord record = { OnOutput, this }; + OSStatus status = VTDecompressionSessionCreate(kCFAllocatorDefault, newFormat, specification.get(), + attributes.get(), &record, &session); + + if (status != noErr) { + RTC_LOG(LS_INFO) << name << ": no session, status " << status; + session = nullptr; + + return false; + } + + if (!IsUsingHardware(session, kVTDecompressionPropertyKey_UsingHardwareAcceleratedVideoDecoder)) { + RTC_LOG(LS_INFO) << name << ": the session does not decode in hardware"; + + VTDecompressionSessionInvalidate(session); + CFRelease(session); + session = nullptr; + + return false; + } + + if (softwareFallback) { + sessionCount++; + } + + fullRange = newFullRange; + + const CMVideoDimensions size = CMVideoFormatDescriptionGetDimensions(newFormat); + + RTC_LOG(LS_INFO) << name << ": " << size.width << "x" << size.height + << (fullRange ? ", full range" : ", studio range"); + + return true; + } + + void VTVideoDecoder::DestroySession() + { + if (session != nullptr) { + VTDecompressionSessionWaitForAsynchronousFrames(session); + VTDecompressionSessionInvalidate(session); + CFRelease(session); + session = nullptr; + + if (softwareFallback) { + sessionCount--; + } + } + if (format != nullptr) { + CFRelease(format); + format = nullptr; + } + + fullRange = false; + } + + int32_t VTVideoDecoder::DecodeSample(std::span data, const webrtc::EncodedImage & image, + int64_t renderTimeMs) + { + if (callback == nullptr) { + return WEBRTC_VIDEO_CODEC_UNINITIALIZED; + } + if (session == nullptr) { + return WEBRTC_VIDEO_CODEC_ERROR; + } + + ScopedCF block; + size_t size = data.size(); + + OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, nullptr, size, + kCFAllocatorDefault, nullptr, 0, size, kCMBlockBufferAssureMemoryNowFlag, block.receive()); + + if (status == noErr) { + status = CMBlockBufferReplaceDataBytes(data.data(), block.get(), 0, size); + } + + ScopedCF sample; + + if (status == noErr) { + CMSampleTimingInfo timing = { CMTimeMake(1, 30), CMTimeMake(sampleCount++, 30), kCMTimeInvalid }; + + status = CMSampleBufferCreateReady(kCFAllocatorDefault, block.get(), format, 1, 1, &timing, 1, &size, + sample.receive()); + } + + Output output; + + if (status == noErr) { + VTDecodeInfoFlags info = 0; + + status = VTDecompressionSessionDecodeFrame(session, sample.get(), kVTDecodeFrame_1xRealTimePlayback, + &output, &info); + } + if (status == noErr) { + // The frame is out before this returns, unless VideoToolbox + // decides to work asynchronously after all. + status = VTDecompressionSessionWaitForAsynchronousFrames(session); + } + if (status == noErr) { + status = output.status; + } + + ScopedCF pixels(output.image); + + if (status != noErr) { + return Fail(status); + } + + consecutiveErrors = 0; + + // A frame that is not shown, or that VideoToolbox dropped, has no + // picture. + if (pixels) { + Deliver(image, renderTimeMs, pixels.get()); + } + + return WEBRTC_VIDEO_CODEC_OK; + } + + int32_t VTVideoDecoder::Fail(OSStatus status) + { + RTC_LOG(LS_WARNING) << name << " failed, status " << status; + + requireKeyFrame = true; + + if (status == kVTInvalidSessionErr) { + DestroySession(); + } + if (++consecutiveErrors >= kMaxConsecutiveErrors && softwareFallback) { + return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; + } + + // WebRTC asks the sender for a key frame. + return WEBRTC_VIDEO_CODEC_ERROR; + } + + int32_t VTVideoDecoder::Unsupported() + { + requireKeyFrame = true; + + return softwareFallback ? WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE : WEBRTC_VIDEO_CODEC_ERROR; + } + + void VTVideoDecoder::OnOutput(void * decoder, void * frame, OSStatus status, VTDecodeInfoFlags flags, + CVImageBufferRef image, CMTime presentationTime, CMTime duration) + { + Output * output = static_cast(frame); + output->status = status; + + if (status == noErr && image != nullptr) { + output->image = static_cast(const_cast(CFRetain(image))); + } + } + + void VTVideoDecoder::Deliver(const webrtc::EncodedImage & image, int64_t renderTimeMs, CVImageBufferRef pixels) + { + // The buffer holds on to the pixels; WebRTC's frame holds on to the + // buffer, and converts to I420 only where a consumer needs it. + RTC_OBJC_TYPE(RTCCVPixelBuffer) * pixelBuffer = + [[RTC_OBJC_TYPE(RTCCVPixelBuffer) alloc] initWithPixelBuffer:pixels]; + + webrtc::scoped_refptr buffer = + webrtc::make_ref_counted(pixelBuffer); + + [pixelBuffer release]; + + webrtc::VideoFrame frame = webrtc::VideoFrame::Builder() + .set_video_frame_buffer(buffer) + .set_rtp_timestamp(image.RtpTimestamp()) + .set_timestamp_ms(renderTimeMs) + .set_ntp_time_ms(image.ntp_time_ms_) + .set_rotation(image.rotation_) + .build(); + + callback->Decoded(frame, std::nullopt, std::nullopt); + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoderFactory.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoderFactory.cpp index e7a2afedc..53b7a429e 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoderFactory.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoderFactory.cpp @@ -15,8 +15,13 @@ */ #include "media/video/codec/macos/VTVideoDecoderFactory.h" +#include "media/video/codec/macos/VTAv1Decoder.h" +#include "media/video/codec/macos/VTH265Decoder.h" #include "media/video/codec/macos/VTVp9Decoder.h" +#include "media/video/codec/H265Format.h" +#include "api/video/video_codec_type.h" +#include "api/video_codecs/video_codec.h" #include "rtc_base/logging.h" #include @@ -25,20 +30,39 @@ namespace jni { namespace { - // Whether this Mac decodes VP9 in hardware. VideoToolbox has the VP9 - // decoder only after it is registered, so that comes first. The - // answer does not change while the process runs, and is asked once. + // Whether this Mac decodes the codec in hardware. The answer does not + // change while the process runs, and is asked once per codec. + bool HasHardwareDecoder(CMVideoCodecType codec, const char * name) + { + const bool supported = VTIsHardwareDecodeSupported(codec); + + RTC_LOG(LS_INFO) << "VideoToolbox hardware decoder for " << name << ": " << supported; + + return supported; + } + bool HasHardwareVp9Decoder() { + // VideoToolbox has the VP9 decoder only after it is registered. static const bool available = [] { VTRegisterSupplementalVideoDecoderIfAvailable(kCMVideoCodecType_VP9); - const bool supported = VTIsHardwareDecodeSupported(kCMVideoCodecType_VP9); + return HasHardwareDecoder(kCMVideoCodecType_VP9, "VP9"); + }(); + + return available; + } - RTC_LOG(LS_INFO) << "VideoToolbox hardware decoder for VP9: " << supported; + bool HasHardwareAv1Decoder() + { + static const bool available = HasHardwareDecoder(kCMVideoCodecType_AV1, "AV1"); - return supported; - }(); + return available; + } + + bool HasHardwareH265Decoder() + { + static const bool available = HasHardwareDecoder(kCMVideoCodecType_HEVC, "H.265"); return available; } @@ -46,21 +70,47 @@ namespace jni std::unique_ptr VTVideoDecoderFactory::Create() { - if (!HasHardwareVp9Decoder()) { + std::vector formats; + + if (HasHardwareVp9Decoder()) { + formats.push_back(webrtc::SdpVideoFormat::VP9Profile0()); + } + if (HasHardwareAv1Decoder()) { + formats.push_back(webrtc::SdpVideoFormat::AV1Profile0()); + } + if (HasHardwareH265Decoder()) { + formats.push_back(CreateH265MainFormat()); + } + + if (formats.empty()) { return nullptr; } - return std::unique_ptr(new VTVideoDecoderFactory()); + return std::unique_ptr(new VTVideoDecoderFactory(std::move(formats))); + } + + VTVideoDecoderFactory::VTVideoDecoderFactory(std::vector formats) : + formats(std::move(formats)) + { } std::vector VTVideoDecoderFactory::GetSupportedFormats() const { - return { webrtc::SdpVideoFormat::VP9Profile0() }; + return formats; } std::unique_ptr VTVideoDecoderFactory::Create(const webrtc::Environment & env, const webrtc::SdpVideoFormat & format) { - return std::make_unique(); + switch (webrtc::PayloadStringToCodecType(format.name)) { + case webrtc::kVideoCodecVP9: + return std::make_unique(); + case webrtc::kVideoCodecAV1: + return std::make_unique(); + case webrtc::kVideoCodecH265: + return std::make_unique(); + default: + return nullptr; + } } } diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoEncoderFactory.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoEncoderFactory.cpp new file mode 100644 index 000000000..cc5ef91f0 --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoEncoderFactory.cpp @@ -0,0 +1,81 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/macos/VTVideoEncoderFactory.h" +#include "media/video/codec/macos/VTCodecUtils.h" +#include "media/video/codec/macos/VTH265Encoder.h" +#include "media/video/codec/H265Format.h" + +#include "rtc_base/logging.h" + +#include + +namespace jni +{ + namespace + { + // Whether VideoToolbox lists a hardware encoder for H.265. The answer + // does not change while the process runs, and is asked once. + bool HasHardwareH265Encoder() + { + static const bool available = [] { + ScopedCF encoders; + bool found = false; + + if (VTCopyVideoEncoderList(nullptr, encoders.receive()) == noErr && encoders) { + for (CFIndex i = 0; i < CFArrayGetCount(encoders.get()) && !found; i++) { + auto encoder = static_cast(CFArrayGetValueAtIndex(encoders.get(), i)); + auto codec = static_cast(CFDictionaryGetValue(encoder, + kVTVideoEncoderList_CodecType)); + auto hardware = static_cast(CFDictionaryGetValue(encoder, + kVTVideoEncoderList_IsHardwareAccelerated)); + int32_t type = 0; + + found = codec != nullptr && CFNumberGetValue(codec, kCFNumberSInt32Type, &type) + && type == kCMVideoCodecType_HEVC + && hardware != nullptr && CFBooleanGetValue(hardware); + } + } + + RTC_LOG(LS_INFO) << "VideoToolbox hardware encoder for H.265: " << found; + + return found; + }(); + + return available; + } + } + + std::unique_ptr VTVideoEncoderFactory::Create() + { + if (!HasHardwareH265Encoder()) { + return nullptr; + } + + return std::unique_ptr(new VTVideoEncoderFactory()); + } + + std::vector VTVideoEncoderFactory::GetSupportedFormats() const + { + return { CreateH265MainFormat() }; + } + + std::unique_ptr VTVideoEncoderFactory::Create(const webrtc::Environment & env, + const webrtc::SdpVideoFormat & format) + { + return std::make_unique(format); + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVp9Decoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVp9Decoder.cpp index e3eb1f298..bbfc1495f 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVp9Decoder.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVp9Decoder.cpp @@ -15,15 +15,11 @@ */ #include "media/video/codec/macos/VTVp9Decoder.h" +#include "media/video/codec/macos/VTCodecUtils.h" -#include "api/make_ref_counted.h" -#include "api/video/video_frame.h" #include "modules/video_coding/include/video_error_codes.h" #include "rtc_base/logging.h" -#include "sdk/objc/components/video_frame_buffer/RTCCVPixelBuffer.h" -#include "sdk/objc/native/src/objc_frame_buffer.h" -#include #include namespace jni @@ -36,160 +32,43 @@ namespace jni constexpr int kMinSize = 64; constexpr int kMaxSize = 4096; - // A decoder that fails this often in a row, each time on a frame - // WebRTC then replaced by a key frame, is given up on. - constexpr int kMaxConsecutiveErrors = 3; - - // How many sessions the process keeps at a time. The media engine - // serves a limited number of streams; a stream past the limit - // starts in software. The number is a cautious guess. - constexpr int kMaxSessions = 8; - - std::atomic sessionCount(0); - - // Releases a Core Foundation object when it goes out of scope. - template - class ScopedCF - { - public: - ScopedCF() : ref(nullptr) {} - explicit ScopedCF(T ref) : ref(ref) {} - ~ScopedCF() { if (ref) CFRelease(ref); } - - ScopedCF(const ScopedCF &) = delete; - ScopedCF & operator=(const ScopedCF &) = delete; - - T get() const { return ref; } - T * receive() { return &ref; } - explicit operator bool() const { return ref != nullptr; } - - // Gives up ownership. - T release() { T result = ref; ref = nullptr; return result; } - - private: - T ref; - }; - - ScopedCF CreateDictionary() - { - return ScopedCF(CFDictionaryCreateMutable(kCFAllocatorDefault, 0, - &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks)); - } - - // How a colour space is written into the VP9 configuration box, as - // its code points, and into the format description. - struct ColorInfo + // The colour code points of a VP9 colour space, for the VP9 + // configuration box and the format description. + struct ColorCodePoints { uint8_t primaries; uint8_t transfer; uint8_t matrix; - CFStringRef cmPrimaries; - CFStringRef cmTransfer; - CFStringRef cmMatrix; }; - ColorInfo GetColorInfo(webrtc::Vp9ColorSpace colorSpace) + ColorCodePoints GetColorCodePoints(webrtc::Vp9ColorSpace colorSpace) { switch (colorSpace) { case webrtc::Vp9ColorSpace::CS_BT_601: case webrtc::Vp9ColorSpace::CS_SMPTE_170: - return { 6, 6, 6, kCMFormatDescriptionColorPrimaries_SMPTE_C, - kCMFormatDescriptionTransferFunction_ITU_R_709_2, kCMFormatDescriptionYCbCrMatrix_ITU_R_601_4 }; + return { 6, 6, 6 }; case webrtc::Vp9ColorSpace::CS_BT_709: - return { 1, 1, 1, kCMFormatDescriptionColorPrimaries_ITU_R_709_2, - kCMFormatDescriptionTransferFunction_ITU_R_709_2, kCMFormatDescriptionYCbCrMatrix_ITU_R_709_2 }; + return { 1, 1, 1 }; case webrtc::Vp9ColorSpace::CS_SMPTE_240: - return { 7, 7, 7, kCMFormatDescriptionColorPrimaries_SMPTE_C, - kCMFormatDescriptionTransferFunction_SMPTE_240M_1995, - kCMFormatDescriptionYCbCrMatrix_SMPTE_240M_1995 }; + return { 7, 7, 7 }; case webrtc::Vp9ColorSpace::CS_BT_2020: - return { 9, 14, 9, kCMFormatDescriptionColorPrimaries_ITU_R_2020, - kCMFormatDescriptionTransferFunction_ITU_R_2020, kCMFormatDescriptionYCbCrMatrix_ITU_R_2020 }; + return { 9, 14, 9 }; default: // Not signalled in the stream: unspecified. - return { 2, 2, 2, nullptr, nullptr, nullptr }; + return { 2, 2, 2 }; } } - - // The format description of a VP9 stream: its dimensions, the VP9 - // configuration box ("vpcC") and the colour properties. - CMVideoFormatDescriptionRef CreateFormat(int width, int height, bool fullRange, - webrtc::Vp9ColorSpace colorSpace) - { - const ColorInfo color = GetColorInfo(colorSpace); - - // Version 1 and flags 0 come first: VideoToolbox rejects the - // box without them. Then the profile, a level that covers the - // sizes in use, the bit depth, 4:2:0 co-located with luma (1) - // and the range, the colour code points, and no initialization - // data. - const uint8_t box[] = { - 1, 0, 0, 0, - 0, - 51, - static_cast((8 << 4) | (1 << 1) | (fullRange ? 1 : 0)), - color.primaries, color.transfer, color.matrix, - 0, 0 - }; - - ScopedCF boxData(CFDataCreate(kCFAllocatorDefault, box, sizeof(box))); - ScopedCF atoms = CreateDictionary(); - ScopedCF extensions = CreateDictionary(); - - if (!boxData || !atoms || !extensions) { - return nullptr; - } - - CFDictionarySetValue(atoms.get(), CFSTR("vpcC"), boxData.get()); - CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms, - atoms.get()); - CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_FullRangeVideo, - fullRange ? kCFBooleanTrue : kCFBooleanFalse); - - if (color.cmPrimaries != nullptr) { - CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_ColorPrimaries, color.cmPrimaries); - CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_TransferFunction, color.cmTransfer); - CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_YCbCrMatrix, color.cmMatrix); - } - - CMVideoFormatDescriptionRef format = nullptr; - OSStatus status = CMVideoFormatDescriptionCreate(kCFAllocatorDefault, kCMVideoCodecType_VP9, width, - height, extensions.get(), &format); - - if (status != noErr) { - RTC_LOG(LS_WARNING) << "VideoToolbox VP9 format description failed, status " << status; - return nullptr; - } - - return format; - } - } - - bool VTVp9Decoder::StreamConfig::operator==(const StreamConfig & other) const - { - return width == other.width && height == other.height && fullRange == other.fullRange - && colorSpace == other.colorSpace; } VTVp9Decoder::VTVp9Decoder() : - callback(nullptr), - session(nullptr), - format(nullptr), - requireKeyFrame(false), - consecutiveErrors(0), - sampleCount(0) + VTVideoDecoder("VideoToolbox (VP9)", true) { } - VTVp9Decoder::~VTVp9Decoder() - { - Release(); - } - bool VTVp9Decoder::Configure(const Settings & settings) { if (settings.codec_type() != webrtc::kVideoCodecVP9) { @@ -205,9 +84,6 @@ namespace jni int32_t VTVp9Decoder::Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) { - if (callback == nullptr) { - return WEBRTC_VIDEO_CODEC_UNINITIALIZED; - } if (image.size() == 0) { return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; } @@ -219,24 +95,25 @@ namespace jni return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; } + const std::span data(image.data(), image.size()); + // The parser gives a header only for a frame that states its size: // key frames, and inter frames that do not take it from a reference. // Most inter frames take it, and have none. - const std::optional header = - webrtc::ParseUncompressedVp9Header(std::span(image.data(), image.size())); + const std::optional header = webrtc::ParseUncompressedVp9Header(data); if (image.FrameType() == webrtc::VideoFrameType::kVideoFrameKey) { - StreamConfig config; - // A key frame has to state what the session is made for. - if (!header || !ReadConfig(*header, config) || !EnsureSession(config)) { + CMVideoFormatDescriptionRef format = header ? CreateFormat(*header) : nullptr; + + if (format == nullptr || !SetFormat(format)) { return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; } requireKeyFrame = false; } else { - if (session == nullptr || requireKeyFrame) { + if (!HasSession() || requireKeyFrame) { // Nothing to build on: WebRTC asks the sender for a key frame. return WEBRTC_VIDEO_CODEC_ERROR; } @@ -244,316 +121,86 @@ namespace jni // A frame that states another size without being a key frame is // reference scaling, which the session does not take. One that // takes its size from a reference has no header to check. + const CMVideoDimensions size = GetDimensions(); + if (header && !header->show_existing_frame - && (header->frame_width != active.width || header->frame_height != active.height)) + && (static_cast(header->frame_width) != size.width + || static_cast(header->frame_height) != size.height)) { return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; } } - Output output; - OSStatus status = DecodeSample(image, output); - - if (status == noErr) { - status = output.status; - } - - ScopedCF pixels(output.image); - - if (status != noErr) { - return Fail(status); - } - - consecutiveErrors = 0; - - // A frame that is not shown, or that VideoToolbox dropped, has no - // picture. - if (pixels) { - Deliver(image, renderTimeMs, pixels.get()); - } - - return WEBRTC_VIDEO_CODEC_OK; - } - - int32_t VTVp9Decoder::RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * decodeCallback) - { - callback = decodeCallback; - - return WEBRTC_VIDEO_CODEC_OK; - } - - int32_t VTVp9Decoder::Release() - { - DestroySession(); - - requireKeyFrame = false; - consecutiveErrors = 0; - - return WEBRTC_VIDEO_CODEC_OK; - } - - webrtc::VideoDecoder::DecoderInfo VTVp9Decoder::GetDecoderInfo() const - { - DecoderInfo info; - info.implementation_name = ImplementationName(); - info.is_hardware_accelerated = true; - - return info; - } - - const char * VTVp9Decoder::ImplementationName() const - { - return "VideoToolbox (VP9)"; - } - - void VTVp9Decoder::OnOutput(void * decoder, void * frame, OSStatus status, VTDecodeInfoFlags flags, - CVImageBufferRef image, CMTime presentationTime, CMTime duration) - { - Output * output = static_cast(frame); - output->status = status; - - if (status == noErr && image != nullptr) { - output->image = static_cast(const_cast(CFRetain(image))); - } + // The whole encoded image is one sample, hidden frames and all: a + // frame that is not shown, given to VideoToolbox on its own, would + // come out as a picture. + return DecodeSample(data, image, renderTimeMs); } - bool VTVp9Decoder::ReadConfig(const webrtc::Vp9UncompressedHeader & header, StreamConfig & config) + CMVideoFormatDescriptionRef VTVp9Decoder::CreateFormat(const webrtc::Vp9UncompressedHeader & header) { // The negotiated format is profile 0, 8 bit 4:2:0. Check, rather than // trust it: a stream that is something else is not for this decoder. if (header.profile != 0 || header.bit_detph != webrtc::Vp9BitDept::k8Bit) { RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: profile " << header.profile << " is not decoded in hardware"; - return false; + return nullptr; } if (header.sub_sampling && *header.sub_sampling != webrtc::Vp9YuvSubsampling::k420) { - return false; - } - if (header.frame_width < kMinSize || header.frame_height < kMinSize - || header.frame_width > kMaxSize || header.frame_height > kMaxSize) - { - RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: " << header.frame_width << "x" << header.frame_height - << " is not decoded in hardware"; - return false; - } - - config.width = header.frame_width; - config.height = header.frame_height; - // The range a stream does not state is the studio range, as in libvpx. - config.fullRange = header.color_range && *header.color_range == webrtc::Vp9ColorRange::kFull; - config.colorSpace = header.color_space.value_or(webrtc::Vp9ColorSpace::CS_UNKNOWN); - - return true; - } - - bool VTVp9Decoder::EnsureSession(const StreamConfig & config) - { - if (session != nullptr && config == active) { - return true; - } - - CMVideoFormatDescriptionRef newFormat = CreateFormat(config.width, config.height, config.fullRange, - config.colorSpace); - - if (newFormat == nullptr) { - return false; - } - - // The pixel format of the output follows the range, so a session does - // not outlast a change of it. Another size is not accepted in place - // either, but a change of the colour space may be. - if (session != nullptr && active.fullRange == config.fullRange - && VTDecompressionSessionCanAcceptFormatDescription(session, newFormat)) - { - CFRelease(format); - format = newFormat; - active = config; - - return true; - } - - // Whatever is still decoding finishes before the session goes. - DestroySession(); - - if (!CreateSession(config, newFormat)) { - CFRelease(newFormat); - - return false; - } - - format = newFormat; - active = config; - - return true; - } - - bool VTVp9Decoder::CreateSession(const StreamConfig & config, CMVideoFormatDescriptionRef newFormat) - { - if (sessionCount.load() >= kMaxSessions) { - RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: too many sessions, using the software decoder"; - return false; - } - - ScopedCF specification = CreateDictionary(); - ScopedCF attributes = CreateDictionary(); - ScopedCF surfaceProperties(CFDictionaryCreate(kCFAllocatorDefault, nullptr, nullptr, 0, - &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks)); - - // The output is 8 bit 4:2:0, in the range of the stream, so that the - // samples reach WebRTC as the software decoder would give them. - const int pixelFormat = config.fullRange - ? kCVPixelFormatType_420YpCbCr8BiPlanarFullRange - : kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange; - ScopedCF pixelFormatNumber(CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &pixelFormat)); - - if (!specification || !attributes || !surfaceProperties || !pixelFormatNumber) { - return false; - } - - // Only the hardware decoder: where there is none, or it is busy, the - // session fails to create and the software decoder takes over. - CFDictionarySetValue(specification.get(), kVTVideoDecoderSpecification_EnableHardwareAcceleratedVideoDecoder, - kCFBooleanTrue); - CFDictionarySetValue(specification.get(), kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder, - kCFBooleanTrue); - - CFDictionarySetValue(attributes.get(), kCVPixelBufferPixelFormatTypeKey, pixelFormatNumber.get()); - CFDictionarySetValue(attributes.get(), kCVPixelBufferIOSurfacePropertiesKey, surfaceProperties.get()); - - VTDecompressionOutputCallbackRecord record = { OnOutput, this }; - OSStatus status = VTDecompressionSessionCreate(kCFAllocatorDefault, newFormat, specification.get(), - attributes.get(), &record, &session); - - if (status != noErr) { - RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: no session, status " << status; - session = nullptr; - - return false; - } - - // Do not rely on the requirement alone. - CFBooleanRef hardware = nullptr; - bool usingHardware = false; - - if (VTSessionCopyProperty(session, kVTDecompressionPropertyKey_UsingHardwareAcceleratedVideoDecoder, - kCFAllocatorDefault, &hardware) == noErr && hardware != nullptr) - { - usingHardware = CFBooleanGetValue(hardware); - CFRelease(hardware); + return nullptr; } - if (!usingHardware) { - RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: the session does not decode in hardware"; - VTDecompressionSessionInvalidate(session); - CFRelease(session); - session = nullptr; + const int width = header.frame_width; + const int height = header.frame_height; - return false; + if (width < kMinSize || height < kMinSize || width > kMaxSize || height > kMaxSize) { + RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: " << width << "x" << height + << " is not decoded in hardware"; + return nullptr; } - sessionCount++; - - RTC_LOG(LS_INFO) << "VideoToolbox VP9 decoder: " << config.width << "x" << config.height - << (config.fullRange ? ", full range" : ", studio range"); - - return true; - } + // The range a stream does not state is the studio range, as in libvpx. + const bool fullRange = header.color_range && *header.color_range == webrtc::Vp9ColorRange::kFull; + const ColorCodePoints color = GetColorCodePoints(header.color_space.value_or( + webrtc::Vp9ColorSpace::CS_UNKNOWN)); + + // Version 1 and flags 0 come first: VideoToolbox rejects the box + // without them. Then the profile, a level that covers the sizes in + // use, the bit depth, 4:2:0 co-located with luma (1) and the range, + // the colour code points, and no initialization data. + const uint8_t box[] = { + 1, 0, 0, 0, + 0, + 51, + static_cast((8 << 4) | (1 << 1) | (fullRange ? 1 : 0)), + color.primaries, color.transfer, color.matrix, + 0, 0 + }; - void VTVp9Decoder::DestroySession() - { - if (session != nullptr) { - VTDecompressionSessionWaitForAsynchronousFrames(session); - VTDecompressionSessionInvalidate(session); - CFRelease(session); - session = nullptr; + ScopedCF boxData(CFDataCreate(kCFAllocatorDefault, box, sizeof(box))); + ScopedCF atoms = CreateCFDictionary(); + ScopedCF extensions = CreateCFDictionary(); - sessionCount--; - } - if (format != nullptr) { - CFRelease(format); - format = nullptr; + if (!boxData || !atoms || !extensions) { + return nullptr; } - active = StreamConfig(); - } - - OSStatus VTVp9Decoder::DecodeSample(const webrtc::EncodedImage & image, Output & output) - { - // The whole encoded image is one sample, hidden frames and all: a - // frame that is not shown, given to VideoToolbox on its own, would - // come out as a picture. - ScopedCF block; - size_t size = image.size(); - - OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, nullptr, size, - kCFAllocatorDefault, nullptr, 0, size, kCMBlockBufferAssureMemoryNowFlag, block.receive()); + CFDictionarySetValue(atoms.get(), CFSTR("vpcC"), boxData.get()); + CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms, + atoms.get()); + CFDictionarySetValue(extensions.get(), kCMFormatDescriptionExtension_FullRangeVideo, + fullRange ? kCFBooleanTrue : kCFBooleanFalse); - if (status == noErr) { - status = CMBlockBufferReplaceDataBytes(image.data(), block.get(), 0, size); - } + SetColorExtensions(extensions.get(), color.primaries, color.transfer, color.matrix); - ScopedCF sample; + CMVideoFormatDescriptionRef format = nullptr; + OSStatus status = CMVideoFormatDescriptionCreate(kCFAllocatorDefault, kCMVideoCodecType_VP9, width, + height, extensions.get(), &format); - if (status == noErr) { - CMSampleTimingInfo timing = { CMTimeMake(1, 30), CMTimeMake(sampleCount++, 30), kCMTimeInvalid }; - - status = CMSampleBufferCreateReady(kCFAllocatorDefault, block.get(), format, 1, 1, &timing, 1, &size, - sample.receive()); - } if (status != noErr) { - return status; + RTC_LOG(LS_WARNING) << "VideoToolbox VP9 format description failed, status " << status; + return nullptr; } - VTDecodeInfoFlags info = 0; - - status = VTDecompressionSessionDecodeFrame(session, sample.get(), kVTDecodeFrame_1xRealTimePlayback, - &output, &info); - - if (status == noErr) { - // The frame is out before this returns, unless VideoToolbox - // decides to work asynchronously after all. - status = VTDecompressionSessionWaitForAsynchronousFrames(session); - } - - return status; - } - - int32_t VTVp9Decoder::Fail(OSStatus status) - { - RTC_LOG(LS_WARNING) << ImplementationName() << " failed, status " << status; - - requireKeyFrame = true; - - if (status == kVTInvalidSessionErr) { - DestroySession(); - } - if (++consecutiveErrors >= kMaxConsecutiveErrors) { - return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; - } - - // WebRTC asks the sender for a key frame. - return WEBRTC_VIDEO_CODEC_ERROR; - } - - void VTVp9Decoder::Deliver(const webrtc::EncodedImage & image, int64_t renderTimeMs, CVImageBufferRef pixels) - { - // The buffer holds on to the pixels; WebRTC's frame holds on to the - // buffer, and converts to I420 only where a consumer needs it. - RTC_OBJC_TYPE(RTCCVPixelBuffer) * pixelBuffer = - [[RTC_OBJC_TYPE(RTCCVPixelBuffer) alloc] initWithPixelBuffer:pixels]; - - webrtc::scoped_refptr buffer = - webrtc::make_ref_counted(pixelBuffer); - - [pixelBuffer release]; - - webrtc::VideoFrame frame = webrtc::VideoFrame::Builder() - .set_video_frame_buffer(buffer) - .set_rtp_timestamp(image.RtpTimestamp()) - .set_timestamp_ms(renderTimeMs) - .set_ntp_time_ms(image.ntp_time_ms_) - .set_rotation(image.rotation_) - .build(); - - callback->Decoded(frame, std::nullopt, std::nullopt); + return format; } } diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoDecoderFactory.java b/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoDecoderFactory.java index 8f5e465d3..3129f28d0 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoDecoderFactory.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoDecoderFactory.java @@ -44,21 +44,23 @@ * decoder that fails to start, or fails while decoding, is replaced by the * software decoder of the same codec, which starts with the next key frame. * On macOS, H.264 is decoded through VideoToolbox by the {@link - * DefaultVideoDecoderFactory} already, and this factory adds VP9 (profile 0) - * through the VP9 decoder of VideoToolbox, on Macs that have one; streams it - * cannot decode, such as those with spatial layers, go to libvpx. Where there - * is no hardware decoder, this factory decodes like a {@link - * DefaultVideoDecoderFactory}. + * DefaultVideoDecoderFactory} already. This factory adds the hardware + * decoders of VideoToolbox for VP9 (profile 0) and H.265 (Main profile), on + * Macs that have them, and for AV1 (profile 0), on Apple M3 and newer. VP9 + * and AV1 streams they cannot decode, such as those with spatial layers, go + * to libvpx and dav1d. Where there is no hardware decoder, this factory + * decodes like a {@link DefaultVideoDecoderFactory}. *

* On Linux, H.264 and VP9 (profile 0) are decoded on NVIDIA GPUs with NVDEC, * loaded from the NVIDIA driver when the factory is made; a machine without * the driver is not affected. Streams with spatial layers go to libvpx. * Decoding with VA-API, on the GPUs of Intel and AMD, is not there yet. *

- * The hardware decoders take over only codecs the software decoders have too, - * so the supported codecs are the same as those of a {@link - * DefaultVideoDecoderFactory}, and what a peer connection negotiates does not - * depend on the GPU. + * The supported codecs are those of a {@link DefaultVideoDecoderFactory}, in + * the same order. H.265 comes after them, only where the hardware decodes it: + * there is no software decoder for it to fall back to, and it is never + * decoded in software, so an H.265 stream the hardware cannot decode is not + * decoded at all. * * @author Alex Andres */ diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java b/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java index 365c6a9e6..e879e9219 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java @@ -43,13 +43,15 @@ * encoder sessions left, or fails while encoding, is replaced by the next one * in line, and finally by the software encoder of the same codec, so a stream * keeps going. Where there is no hardware encoder, this factory encodes like a - * {@link DefaultVideoEncoderFactory}. On macOS, that already uses - * VideoToolbox. + * {@link DefaultVideoEncoderFactory}. On macOS, that already encodes H.264 + * with VideoToolbox, and this factory adds H.265 (Main profile), encoded by + * the hardware HEVC encoder of VideoToolbox on Macs that have one. *

- * The hardware encoders take over only codecs the software encoders have too, - * so the supported codecs are the same as those of a {@link - * DefaultVideoEncoderFactory}, and what a peer connection negotiates does not - * depend on the GPU. + * The supported codecs are those of a {@link DefaultVideoEncoderFactory}, in + * the same order, so the codec a peer connection prefers does not depend on + * the hardware. H.265 comes after them, only where the hardware encodes it: + * there is no software encoder for it to fall back to, and it is never + * encoded in software. * * @author Alex Andres */ diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoDecoderIntegrationTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoDecoderIntegrationTest.java index fd1a30380..a2062b26c 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoDecoderIntegrationTest.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoDecoderIntegrationTest.java @@ -27,8 +27,11 @@ import dev.onvoid.webrtc.media.video.VideoTrackSink; import dev.onvoid.webrtc.media.video.codec.DefaultVideoDecoderFactory; import dev.onvoid.webrtc.media.video.codec.HardwareVideoDecoderFactory; +import dev.onvoid.webrtc.media.video.codec.HardwareVideoEncoderFactory; +import dev.onvoid.webrtc.media.video.codec.VideoCodecInfo; import java.nio.ByteBuffer; +import java.util.List; import java.util.Locale; import java.util.Map; import java.util.concurrent.CountDownLatch; @@ -48,8 +51,11 @@ * A machine without a hardware decoder skips the tests that need one, as CI * runners do. Set the system property {@code webrtc.test.hardwareDecoder} to * {@code true} on a machine that has one, to make those tests fail instead, - * and {@code webrtc.test.hardwareAv1Decoder} for a GPU that decodes AV1, and - * {@code webrtc.test.hardwareVp9Decoder} for one that decodes VP9 on Windows or Linux. + * and {@code webrtc.test.hardwareAv1Decoder} for a GPU that decodes AV1 (on + * macOS, Apple M3 and newer), + * {@code webrtc.test.hardwareVp9Decoder} for one that decodes VP9 on Windows or Linux, + * and {@code webrtc.test.hardwareH265Decoder} for one that decodes and encodes + * H.265, which a call in H.265 needs at both ends, having no software codec. * On macOS the VP9 tests follow {@code webrtc.test.hardwareDecoder}. */ @Execution(ExecutionMode.SAME_THREAD) @@ -63,6 +69,8 @@ class HardwareVideoDecoderIntegrationTest extends TestBase { private static final boolean HARDWARE_VP9_REQUIRED = Boolean.getBoolean("webrtc.test.hardwareVp9Decoder"); + private static final boolean HARDWARE_H265_REQUIRED = Boolean.getBoolean("webrtc.test.hardwareH265Decoder"); + private static final String OS = System.getProperty("os.name").toLowerCase(Locale.ROOT); private static final Predicate H264 = codec -> @@ -76,12 +84,88 @@ class HardwareVideoDecoderIntegrationTest extends TestBase { "VP9".equalsIgnoreCase(codec.getName()) && "0".equals(codec.getSDPFmtp().getOrDefault("profile-id", "0")); + private static final Predicate H265 = codec -> "H265".equalsIgnoreCase(codec.getName()); + @Test void hardwareKeepsCodecs() { - // Hardware decoding must not change what is negotiated. - assertEquals(new DefaultVideoDecoderFactory().getSupportedCodecs(), - new HardwareVideoDecoderFactory().getSupportedCodecs()); + // Hardware decoding must not change what is negotiated otherwise: the + // codecs of the default decoders come first, in their order. After + // them comes H.265 where the GPU decodes it, which has no software + // decoder. + List software = new DefaultVideoDecoderFactory().getSupportedCodecs(); + List hardware = new HardwareVideoDecoderFactory().getSupportedCodecs(); + + assertTrue(hardware.size() >= software.size(), hardware.toString()); + assertEquals(software, hardware.subList(0, software.size())); + + for (VideoCodecInfo added : hardware.subList(software.size(), hardware.size())) { + assertEquals("H265", added.getName(), hardware.toString()); + } + } + + @Test + void hardwareDecodesH265() throws Exception { + assumeH265(); + + PeerConnectionFactory hardware = hardwareCallFactory(); + String implementation; + + try { + // 640x360: hardware H.265 encoders have a minimum size, and AMD's + // Media Foundation encoder refuses widths up to 352. WebRTC then + // switches to the next codec, there being no software H.265. + implementation = decoderImplementation(hardware, H265, 640, 360, "video/H265"); + } + finally { + hardware.dispose(); + } + + assertTrue(isHardware(implementation), implementation); + } + + @Test + void hardwareCropsPaddedH265() throws Exception { + // HEVC codes in blocks of up to 64, so 360 rows come in a larger + // surface. The encoder is a hardware one too, there being no software + // H.265, and AMD's pads the coded picture below the 360 rows with + // neutral chroma itself; its loop filters pull the last chroma row + // towards that, by up to about 20. That row is left out of the check: + // a crop at the wrong offset would show across the whole plane. + assumeH265(); + + assertCropsPaddedFrames(hardwareCallFactory(), H265, "video/H265", 1, + implementation -> assertTrue(isHardware(implementation), implementation)); + } + + /** + * Skips the test where the GPU does not both encode and decode H.265, which + * a call in it needs at either end, there being no software codec; fails + * it where that is required. + */ + private static void assumeH265() { + boolean available = offersH265(new HardwareVideoEncoderFactory().getSupportedCodecs()) + && offersH265(new HardwareVideoDecoderFactory().getSupportedCodecs()); + + if (HARDWARE_H265_REQUIRED) { + assertTrue(available, "no hardware H.265 encoder and decoder"); + } + else { + assumeTrue(available, "no hardware H.265 encoder and decoder"); + } + } + + private static boolean offersH265(List codecs) { + return codecs.stream().anyMatch(codec -> "H265".equalsIgnoreCase(codec.getName())); + } + + /** A factory that encodes as well as decodes in hardware. */ + private PeerConnectionFactory hardwareCallFactory() { + return PeerConnectionFactory.builder() + .setAudioDeviceModule(audioDevModule) + .setVideoEncoderFactory(new HardwareVideoEncoderFactory()) + .setVideoDecoderFactory(new HardwareVideoDecoderFactory()) + .build(); } @Test @@ -121,10 +205,20 @@ void hardwareCropsPaddedVp9() throws Exception { */ private void assertCropsPaddedFrames(Predicate codec, Consumer expectHardware) throws Exception { + assertCropsPaddedFrames(hardwareFactory(), codec, null, 0, expectHardware); + } + + /** + * Does what {@link #assertCropsPaddedFrames(Predicate, Consumer)} does, on + * a call of the given factory, which this disposes of, leaving the given + * number of chroma rows at the bottom of the picture out of the check. + * With a MIME type, checks that the call sent that codec. + */ + private void assertCropsPaddedFrames(PeerConnectionFactory hardware, Predicate codec, + String mimeType, int exemptChromaRows, Consumer expectHardware) throws Exception { final int width = 640; final int height = 360; - PeerConnectionFactory hardware = hardwareFactory(); CountDownLatch received = new CountDownLatch(30); AtomicReference wrong = new AtomicReference<>(); AtomicInteger padded = new AtomicInteger(); @@ -145,7 +239,7 @@ private void assertCropsPaddedFrames(Predicate codec, Con wrong.compareAndSet(null, "a frame of " + frameWidth + "x" + frameHeight); } else { - String error = checkPicture(frame.buffer.toI420()); + String error = checkPicture(frame.buffer.toI420(), exemptChromaRows); if (error != null) { wrong.compareAndSet(null, "a " + frameWidth + "x" + frameHeight + " frame with " + error); @@ -167,6 +261,11 @@ private void assertCropsPaddedFrames(Predicate codec, Con implementation = decoderImplementationOf(call); + if (mimeType != null) { + // A call goes on in another codec when the encoder fails. + assertEquals(mimeType, call.getSentVideoCodec()); + } + track.removeSink(sink); receiver.dispose(); } @@ -182,9 +281,10 @@ private void assertCropsPaddedFrames(Predicate codec, Con /** * Returns what is wrong with the picture of a frame the call sent, or - * {@code null}. + * {@code null}. The given number of chroma rows at the bottom is not + * checked. */ - private static String checkPicture(I420Buffer buffer) { + private static String checkPicture(I420Buffer buffer, int exemptChromaRows) { ByteBuffer y = buffer.getDataY(); int lastRow = (buffer.getHeight() - 1) * buffer.getStrideY(); @@ -196,7 +296,7 @@ private static String checkPicture(I420Buffer buffer) { } int chromaWidth = (buffer.getWidth() + 1) / 2; - int chromaHeight = (buffer.getHeight() + 1) / 2; + int chromaHeight = (buffer.getHeight() + 1) / 2 - exemptChromaRows; if (!isFlat(buffer.getDataU(), buffer.getStrideU(), chromaWidth, chromaHeight, TestMediaCall.CHROMA_U)) { return "U not at " + TestMediaCall.CHROMA_U; @@ -247,9 +347,6 @@ private static void expectHardware(String implementation, boolean required) { } private void assertHardwareDecodes(Predicate codec, boolean required) throws Exception { - assumeTrue(OS.contains("win") || OS.contains("linux"), - "hardware decoders are implemented on Windows and Linux only"); - PeerConnectionFactory hardware = PeerConnectionFactory.builder() .setAudioDeviceModule(audioDevModule) .setVideoDecoderFactory(new HardwareVideoDecoderFactory()) @@ -549,23 +646,34 @@ private static long count(Map stats, String name) { */ private static String decoderImplementation(PeerConnectionFactory factory, Predicate codec) throws Exception { + return decoderImplementation(factory, codec, 320, 240, null); + } + + /** + * Does what {@link #decoderImplementation(PeerConnectionFactory, Predicate)} + * does, with video of the given size; 320x240 is the call's default. With + * a MIME type, checks that the call sent that codec. + */ + private static String decoderImplementation(PeerConnectionFactory factory, + Predicate codec, int sentWidth, int sentHeight, String mimeType) throws Exception { CountDownLatch received = new CountDownLatch(10); AtomicReference wrongSize = new AtomicReference<>(); AtomicReference blank = new AtomicReference<>(); String implementation; try (TestMediaCall call = new TestMediaCall(factory, true, false, codec)) { + call.setVideoSize(sentWidth, sentHeight); call.negotiate(); RTCRtpReceiver receiver = call.getReceiver("video"); VideoTrack track = (VideoTrack) receiver.getTrack(); VideoTrackSink sink = frame -> { - // The call sends 320x240; a decoder that returned the padded - // frame rather than the picture would be larger. + // A decoder that returned the padded frame rather than the + // picture would be larger than what was sent. int width = frame.buffer.getWidth(); int height = frame.buffer.getHeight(); - if (width != 320 || height != 240) { + if (width != sentWidth || height != sentHeight) { wrongSize.compareAndSet(null, width + "x" + height); } else if (!hasPicture(frame.buffer.toI420())) { @@ -584,6 +692,11 @@ else if (!hasPicture(frame.buffer.toI420())) { implementation = decoderImplementationOf(call); + if (mimeType != null) { + // A call goes on in another codec when the encoder fails. + assertEquals(mimeType, call.getSentVideoCodec()); + } + track.removeSink(sink); receiver.dispose(); } diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java index b4b38c29a..d6a422299 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java @@ -17,14 +17,18 @@ package dev.onvoid.webrtc; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assumptions.assumeFalse; import static org.junit.jupiter.api.Assumptions.assumeTrue; import dev.onvoid.webrtc.media.video.VideoTrack; import dev.onvoid.webrtc.media.video.VideoTrackSink; +import dev.onvoid.webrtc.media.video.codec.DefaultVideoDecoderFactory; import dev.onvoid.webrtc.media.video.codec.DefaultVideoEncoderFactory; +import dev.onvoid.webrtc.media.video.codec.HardwareVideoDecoderFactory; import dev.onvoid.webrtc.media.video.codec.HardwareVideoEncoderFactory; +import dev.onvoid.webrtc.media.video.codec.VideoCodecInfo; import java.util.ArrayList; import java.util.List; @@ -45,7 +49,9 @@ * A machine without a hardware encoder skips the tests that need one, as CI * runners do. Set the system property {@code webrtc.test.hardwareEncoder} to * {@code true} on a machine that has one, to make those tests fail instead, - * and {@code webrtc.test.hardwareAv1Encoder} for a GPU that encodes AV1. + * {@code webrtc.test.hardwareAv1Encoder} for a GPU that encodes AV1, and + * {@code webrtc.test.hardwareH265Encoder} for one that encodes and decodes + * H.265, which a call in H.265 needs at both ends, having no software codec. */ @Execution(ExecutionMode.SAME_THREAD) class HardwareVideoEncoderIntegrationTest extends TestBase { @@ -56,6 +62,8 @@ class HardwareVideoEncoderIntegrationTest extends TestBase { private static final boolean HARDWARE_AV1_REQUIRED = Boolean.getBoolean("webrtc.test.hardwareAv1Encoder"); + private static final boolean HARDWARE_H265_REQUIRED = Boolean.getBoolean("webrtc.test.hardwareH265Encoder"); + private static final String OS = System.getProperty("os.name").toLowerCase(Locale.ROOT); /** @@ -69,12 +77,179 @@ class HardwareVideoEncoderIntegrationTest extends TestBase { private static final Predicate AV1 = codec -> "AV1".equalsIgnoreCase(codec.getName()); + private static final Predicate H265 = codec -> "H265".equalsIgnoreCase(codec.getName()); + @Test void hardwareKeepsCodecs() { - // Hardware encoding must not change what is negotiated. - assertEquals(new DefaultVideoEncoderFactory().getSupportedCodecs(), - new HardwareVideoEncoderFactory().getSupportedCodecs()); + // Hardware encoding must not change which codec is preferred: the + // codecs of the default encoders come first, in their order. After + // them comes H.265 where the GPU encodes it, which has no software + // encoder. + List software = new DefaultVideoEncoderFactory().getSupportedCodecs(); + List hardware = new HardwareVideoEncoderFactory().getSupportedCodecs(); + + assertTrue(hardware.size() >= software.size(), hardware.toString()); + assertEquals(software, hardware.subList(0, software.size())); + + for (VideoCodecInfo added : hardware.subList(software.size(), hardware.size())) { + assertEquals("H265", added.getName(), hardware.toString()); + } + } + + @Test + void softwareOffersNoH265() { + // H.265 is encoded and decoded in hardware only, never in software. + assertFalse(offersH265(new DefaultVideoEncoderFactory().getSupportedCodecs())); + assertFalse(offersH265(new DefaultVideoDecoderFactory().getSupportedCodecs())); + } + + @Test + void hardwareEncodesH265() throws Exception { + assumeH265(); + + PeerConnectionFactory hardware = hardwareCallFactory(); + + // 640x360: hardware H.265 encoders have a minimum size, and AMD's + // Media Foundation encoder refuses widths up to 352. WebRTC then + // switches to the next codec, there being no software H.265. + try (Run run = new Run(hardware, H265, 640, 360)) { + run.awaitFrames(10); + + assertH265(run); + } + finally { + hardware.dispose(); + } + } + + @Test + void h265EncodesOddSize() throws Exception { + assumeH265(); + + PeerConnectionFactory hardware = hardwareCallFactory(); + + // 4:2:0 needs even dimensions, which the encoder asks for, rather than + // refusing the size and ending the H.265 stream. + try (Run run = new Run(hardware, H265, 641, 361)) { + run.awaitFrames(10); + + assertH265(run); + assertEquals(0, run.width.get() % 2, "width " + run.width.get()); + } + finally { + hardware.dispose(); + } + } + + @Test + void h265FollowsResolutionChange() throws Exception { + assumeH265(); + + PeerConnectionFactory hardware = hardwareCallFactory(); + + try (Run run = new Run(hardware, H265, 640, 360)) { + run.awaitFrames(10); + + // A new size starts the encoder over, with a new session, and the + // decoder with new parameter sets. + run.call.setVideoSize(960, 540); + run.awaitWidth(960); + + run.call.setVideoSize(640, 360); + run.awaitWidth(640); + + assertH265(run); + } + finally { + hardware.dispose(); + } + } + + @Test + void h265FollowsBitrateCap() throws Exception { + assumeH265(); + + PeerConnectionFactory hardware = hardwareCallFactory(); + + try (Run run = new Run(hardware, H265, 640, 360)) { + run.awaitFrames(10); + + // The encoder is reconfigured in place, without a new session or + // a lost stream. + run.setMaxBitrate(150_000); + run.awaitTargetBitrate(150_000); + run.awaitMoreFrames(10); + + run.setMaxBitrate(2_000_000); + run.awaitMoreFrames(10); + + assertH265(run); + } + finally { + hardware.dispose(); + } + } + + @Test + void h265RestartsCleanly() throws Exception { + assumeH265(); + + PeerConnectionFactory hardware = hardwareCallFactory(); + + try { + // A session left behind by each call would run the hardware out of + // sessions, and the later calls would not be in H.265. + for (int i = 0; i < 10; i++) { + try (Run run = new Run(hardware, H265, 640, 360)) { + run.awaitFrames(5); + + assertH265(run); + } + } + } + finally { + hardware.dispose(); + } + } + + /** + * Skips the test where the hardware does not both encode and decode H.265, + * which a call in it needs at either end, there being no software codec; + * fails it where that is required. + */ + private static void assumeH265() { + boolean available = offersH265(new HardwareVideoEncoderFactory().getSupportedCodecs()) + && offersH265(new HardwareVideoDecoderFactory().getSupportedCodecs()); + + if (HARDWARE_H265_REQUIRED) { + assertTrue(available, "no hardware H.265 encoder and decoder"); + } + else { + assumeTrue(available, "no hardware H.265 encoder and decoder"); + } + } + + /** + * Checks that the call is still in H.265, which it would not be after an + * encoder failure, and is encoded in hardware. + */ + private static void assertH265(Run run) throws InterruptedException { + assertEquals("video/H265", run.codec()); + assertHardware(run.implementation(), true); + } + + /** A factory that encodes as well as decodes in hardware. */ + private PeerConnectionFactory hardwareCallFactory() { + return PeerConnectionFactory.builder() + .setAudioDeviceModule(audioDevModule) + .setVideoEncoderFactory(new HardwareVideoEncoderFactory()) + .setVideoDecoderFactory(new HardwareVideoDecoderFactory()) + .build(); + } + + private static boolean offersH265(List codecs) { + return codecs.stream().anyMatch(codec -> "H265".equalsIgnoreCase(codec.getName())); } @Test @@ -268,8 +443,8 @@ void macEncodesH264WithVideoToolbox() throws Exception { // The default encoders use VideoToolbox on macOS. assertTrue(encoderImplementation(factory, H264).contains("VideoToolbox")); - // The hardware factory has nothing of its own there, and hands over to - // the default encoders. + // The hardware factory has nothing of its own for H.264 there, and + // hands over to the default encoders. PeerConnectionFactory hardware = PeerConnectionFactory.builder() .setAudioDeviceModule(audioDevModule) .setVideoEncoderFactory(new HardwareVideoEncoderFactory()) @@ -298,7 +473,8 @@ private PeerConnectionFactory hardwareFactory() { private static void assertHardware(String implementation, boolean required) { boolean hardwareUsed = implementation.startsWith("NVENC") || implementation.startsWith("VA-API") - || implementation.contains("MediaFoundation"); + || implementation.contains("MediaFoundation") + || implementation.contains("VideoToolbox"); if (required) { assertTrue(hardwareUsed, implementation); @@ -415,7 +591,19 @@ void awaitTargetBitrate(double atMost) throws InterruptedException { }, "target bitrate not at or below " + atMost); } - String implementation() throws InterruptedException { + String codec() throws InterruptedException { + String[] codec = new String[1]; + + await(() -> { + codec[0] = call.getSentVideoCodec(); + + return codec[0] != null; + }, "no codec in the statistics"); + + return codec[0]; + } + + String implementation() throws InterruptedException { // The statistics catch up with the encoder shortly after. Object[] implementation = new Object[1]; diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/TestMediaCall.java b/webrtc/src/test/java/dev/onvoid/webrtc/TestMediaCall.java index 149f9e339..e9c90ab3b 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/TestMediaCall.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/TestMediaCall.java @@ -185,6 +185,36 @@ Map getOutboundVideoStats() throws InterruptedException { return outbound.get(); } + /** + * Returns the MIME type of the codec the caller sends video in, such as + * {@code video/H265}, or null if the statistics do not tell yet. The codec + * preferred is not always the one sent: a call goes on with the next + * negotiated codec when the encoder of the preferred one fails. + */ + String getSentVideoCodec() throws InterruptedException { + CountDownLatch done = new CountDownLatch(1); + AtomicReference mimeType = new AtomicReference<>(); + + caller.getPeerConnection().getStats(videoSender, report -> { + Map all = report.getStats(); + + for (RTCStats stats : all.values()) { + if (stats.getType() == RTCStatsType.OUTBOUND_RTP) { + RTCStats codec = all.get(String.valueOf(stats.getAttributes().get("codecId"))); + + if (codec != null) { + mimeType.set(String.valueOf(codec.getAttributes().get("mimeType"))); + } + } + } + done.countDown(); + }); + + done.await(5, TimeUnit.SECONDS); + + return mimeType.get(); + } + /** * Returns the callee's statistics of its inbound video stream, or null if * there are none yet.