feat: encode and decode H.265, and decode AV1, in hardware on macOS

docs/guide/advanced/video-codecs.md

@@ -16,27 +16,27 @@ for (VideoCodecInfo codec : new DefaultVideoEncoderFactory().getSupportedCodecs(
 
 ### Hardware Encoding
 
-`HardwareVideoEncoderFactory` offers the same codecs as `DefaultVideoEncoderFactory`, but encodes on the GPU where the platform supports it:
+`HardwareVideoEncoderFactory` offers the codecs of `DefaultVideoEncoderFactory`, and H.265 where the hardware encodes it, and encodes on the GPU where the platform supports it:
 
 ```java
 PeerConnectionFactory factory = PeerConnectionFactory.builder()
         .setVideoEncoderFactory(new HardwareVideoEncoderFactory())
         .build();
 ```
 
-| Platform | H.264 | AV1 |
-|---|---|---|
-| Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) |
-| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | NVENC on NVIDIA RTX 40 and newer |
-| macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | Software |
+| Platform | H.264 | H.265 | AV1 |
+|---|---|---|---|
+| Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | Not offered | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) |
+| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | Not offered | NVENC on NVIDIA RTX 40 and newer |
+| macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | VideoToolbox, on Macs with a hardware HEVC encoder, such as all with Apple silicon | Software |
 
 Hardware encoders produce a single layer. A stream that asks for SVC, such as an AV1 stream with a scalability mode like `L1T3`, is encoded in software.
 
 NVENC needs an NVIDIA driver of version 522 or newer on Windows, 520 or newer on Linux. VA-API needs libva 2 and a driver that encodes H.264, such as Intel's `intel-media-va-driver` (iHD) or Mesa's `mesa-va-drivers` for AMD, and access to a render node in `/dev/dri`. All of them are loaded at run time, so nothing needs to be installed on machines without them.
 
-The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`.
+The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding these in hardware never changes what is negotiated. H.265 (Main profile) has no software encoder: it is offered after the other codecs, only where the hardware encodes it, so the codec a peer connection prefers stays the same, and a call uses H.265 only if it is preferred (e.g. with `RTCRtpTransceiver.setCodecPreferences()`) and the other end decodes it. On macOS, VideoToolbox is required to encode H.265 in hardware, and never falls back to its software encoder; a Mac without a hardware HEVC encoder does not offer H.265. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`.
 
-Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `OpenH264` or `libaom`.
+Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `VideoToolbox (H.265)`, `OpenH264` or `libaom`.
 
 ### Hardware Decoding
 
@@ -53,15 +53,17 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
 |---|---|
 | Windows | H.264, AV1 and VP9 (profile 0), on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
 | Linux | H.264 and VP9 (profile 0) on NVIDIA GPUs, through NVDEC; VA-API, for Intel and AMD GPUs, is not there yet |
-| macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`, and VP9 (profile 0) through VideoToolbox's VP9 decoder, on Macs that have one |
+| macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`; VP9 (profile 0) and H.265 (Main profile) through the decoders of VideoToolbox, on Macs that have them, and AV1 (profile 0) on Apple M3 and newer |
 
 On Linux, NVDEC needs an NVIDIA driver, which provides `libcuda.so.1` and `libnvcuvid.so.1`; both are loaded when the factory is made, so a machine without the driver is not affected. Streams with spatial layers are decoded in software.
 
 AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes, and VP9 the *VP9 Video Extensions*, a free component of the Microsoft Store that a system may not have; without it, VP9 is decoded in software. Streams with spatial layers (SVC) are decoded in software too, since their layers reach a decoder without the index that says where each ends. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.
 
 On macOS, VP9 decoding in hardware saves CPU: in a measurement on an Apple M2 it took a fraction of the processor time libvpx needs, but multi-threaded libvpx was about as fast on the clock. VideoToolbox decodes VP9 in hardware only where the Mac has a decoder for it, which Apple silicon does; a Mac without one decodes VP9 with libvpx as before. Profile 2 (10 bit), sizes below 64x64 or above 4096x4096, and streams with spatial layers (SVC) are decoded with libvpx too.
 
-Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)`, `VideoToolbox (VP9)` or `NVDEC (NVIDIA GeForce RTX 4070)`.
+AV1 is decoded by VideoToolbox on Macs with an AV1 decoder, which are those with Apple M3 and newer, and by dav1d elsewhere. High bit depth, monochrome streams and streams with spatial layers go to dav1d as well. H.265 is decoded by VideoToolbox in hardware only, never in software: a Mac without a hardware HEVC decoder does not offer H.265, and an H.265 stream the hardware cannot decode is not decoded.
+
+Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)`, `VideoToolbox (VP9)`, `VideoToolbox (H.265)`, `VideoToolbox (AV1)` or `NVDEC (NVIDIA GeForce RTX 4070)`.
 
 ### Native Codecs
 
@@ -114,7 +116,7 @@ The list is in order of preference. A codec name WebRTC does not know, such as `
 
 ### H.265
 
-The library is built with WebRTC's H.265 support: H.265 is negotiated with its SDP parameters (`profile-id`, `tier-flag`, `level-id`, `tx-mode`) and sent with the H.265 RTP payload format (RFC 7798). WebRTC has no H.265 encoder or decoder of its own, and this library adds none in software, so H.265 is offered only by a factory that brings one, such as a hardware encoder reached through another library:
+The library is built with WebRTC's H.265 support: H.265 is negotiated with its SDP parameters (`profile-id`, `tier-flag`, `level-id`, `tx-mode`) and sent with the H.265 RTP payload format (RFC 7798). WebRTC has no H.265 encoder or decoder of its own, and this library adds none in software, so H.265 is offered only by a factory that brings one: the hardware factories on Macs that encode or decode it in hardware (see above), or one of your own, such as a hardware encoder reached through another library:
 
 ```java
 Map<String, String> parameters = new HashMap<>();

webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h

@@ -22,6 +22,7 @@
 #include "api/video_codecs/sdp_video_format.h"
 #include "api/video_codecs/video_encoder.h"
 #include "common_video/h264/h264_bitstream_parser.h"
+#include "common_video/h265/h265_bitstream_parser.h"
 #include "modules/video_coding/codecs/h264/include/h264_globals.h"
 #include "modules/video_coding/include/video_codec_interface.h"
 #include "modules/video_coding/svc/scalable_video_controller_no_layering.h"
@@ -33,15 +34,16 @@
 namespace jni
 {
 	// Turns what a hardware encoder produces for a frame into what WebRTC
-	// sends, the same way for every hardware encoder: H.264 as Annex B NAL
-	// units, AV1 as OBUs in the low overhead bitstream format.
+	// sends, the same way for every hardware encoder: H.264 and H.265 as
+	// Annex B NAL units, AV1 as OBUs in the low overhead bitstream format.
 	//
 	// A receiver can start decoding only from a key frame that carries the
-	// parameter sets, the SPS and PPS of H.264 or the sequence header of AV1.
-	// Encoders put them in front of the first key frame, not necessarily in
-	// front of every one, so they are kept and put back where missing. The
-	// frame dependencies WebRTC needs for AV1 are those of a stream without
-	// layers, which is all hardware encoders produce here.
+	// parameter sets: the SPS and PPS of H.264, the VPS, SPS and PPS of H.265,
+	// or the sequence header of AV1. Encoders put them in front of the first
+	// key frame, not necessarily in front of every one, so they are kept and
+	// put back where missing. The frame dependencies WebRTC needs for AV1 and
+	// H.265 are those of a stream without layers, which is all hardware
+	// encoders produce here.
 	class EncoderOutputProcessor
 	{
 		public:
@@ -54,19 +56,21 @@ namespace jni
 			// specific information WebRTC needs to send it. The caller sets
 			// the size of the image before, which the AV1 frame dependencies
 			// refer to, and its timestamps. The frame is a key
-			// frame if the encoder says so or, for H.264, if it holds an IDR
-			// slice. Returns false if the bitstream cannot be parsed, or a key
+			// frame if the encoder says so or if it holds an IDR slice of
+			// H.264 or an IRAP picture of H.265. Returns false if the
+			// bitstream cannot be parsed, or a key
 			// frame lacks parameter sets and there are none to put back.
 			bool Process(std::span<const uint8_t> bitstream, bool keyFrame, webrtc::EncodedImage & image,
 				webrtc::CodecSpecificInfo & info);
 
 			// The quality scaling settings that suit the codec: the thresholds
-			// of WebRTC's H.264 encoder, and none for AV1, whose QP is not
-			// parsed from the bitstream.
+			// of WebRTC's H.264 encoder, for H.265 as well, and none for AV1,
+			// whose QP is not parsed from the bitstream.
 			webrtc::VideoEncoder::ScalingSettings GetScalingSettings() const;
 
 		private:
 			bool ProcessH264(std::span<const uint8_t> bitstream, bool & keyFrame, std::vector<uint8_t> & output);
+			bool ProcessH265(std::span<const uint8_t> bitstream, bool & keyFrame, std::vector<uint8_t> & output);
 			bool ProcessAv1(std::span<const uint8_t> bitstream, bool keyFrame, std::vector<uint8_t> & output);
 
 			void FillCodecSpecificInfo(const webrtc::EncodedImage & image, bool keyFrame,
@@ -76,10 +80,12 @@ namespace jni
 			const webrtc::VideoCodecType codec;
 			webrtc::H264PacketizationMode packetizationMode;
 
-			// The SPS and PPS, or the sequence header OBU, last seen.
+			// The parameter sets of H.264 or H.265, or the sequence header
+			// OBU, last seen.
 			std::vector<uint8_t> parameterSets;
 
 			webrtc::H264BitstreamParser h264Parser;
+			webrtc::H265BitstreamParser h265Parser;
 			webrtc::ScalableVideoControllerNoLayering svcController;
 	};
 }

webrtc-jni/src/main/cpp/include/media/video/codec/H265Format.h

@@ -0,0 +1,31 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_H265_FORMAT_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_H265_FORMAT_H_
+
+#include "api/video_codecs/sdp_video_format.h"
+
+namespace jni
+{
+	// The H.265 format the hardware codec factories offer: Main profile, main
+	// tier, level 3.1, one RTP stream (tx-mode SRST). The same level as the
+	// H.264 they offer; WebRTC negotiates the level down to what both ends
+	// take.
+	webrtc::SdpVideoFormat CreateH265MainFormat();
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTAv1Decoder.h

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_AV1_DECODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_AV1_DECODER_H_
+
+#include "media/video/codec/macos/VTVideoDecoder.h"
+
+#include "api/video/encoded_image.h"
+
+#include <cstdint>
+#include <span>
+#include <vector>
+
+namespace jni
+{
+	// Decodes AV1 profile 0, 8 bit, on the media engine of Macs that have an
+	// AV1 decoder, Apple M3 and newer, through VideoToolbox.
+	//
+	// The session is created from the sequence header of a key frame, which
+	// goes into the AV1 configuration box ("av1C") of the format description,
+	// and again whenever a key frame brings one the session does not take.
+	// The OBUs of a temporal unit go to VideoToolbox as one sample, without
+	// temporal delimiters, as AV1 is stored in ISOBMFF.
+	//
+	// Whatever VideoToolbox cannot decode goes to dav1d, by returning
+	// WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE: another profile or bit depth,
+	// monochrome, a stream with spatial layers, and a decoder that keeps
+	// failing.
+	class VTAv1Decoder : public VTVideoDecoder
+	{
+		public:
+			VTAv1Decoder();
+			~VTAv1Decoder() override = default;
+
+			using webrtc::VideoDecoder::Decode;
+
+			bool Configure(const Settings & settings) override;
+			int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override;
+
+		private:
+			// Makes the format description of a stream from its sequence
+			// header OBU, header and size field included. Returns null for a
+			// stream the hardware decoder does not take.
+			static CMVideoFormatDescriptionRef CreateFormat(std::span<const uint8_t> sequenceHeaderObu);
+
+		private:
+			// The sample, rebuilt for each frame.
+			std::vector<uint8_t> sample;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTCodecUtils.h

@@ -0,0 +1,70 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_CODEC_UTILS_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_CODEC_UTILS_H_
+
+#include <CoreFoundation/CoreFoundation.h>
+#include <CoreMedia/CoreMedia.h>
+#include <VideoToolbox/VideoToolbox.h>
+
+#include <cstdint>
+
+namespace jni
+{
+	// Releases a Core Foundation object when it goes out of scope.
+	template <typename T>
+	class ScopedCF
+	{
+		public:
+			ScopedCF() : ref(nullptr) {}
+			explicit ScopedCF(T ref) : ref(ref) {}
+			~ScopedCF() { if (ref) CFRelease(ref); }
+
+			ScopedCF(const ScopedCF &) = delete;
+			ScopedCF & operator=(const ScopedCF &) = delete;
+
+			T get() const { return ref; }
+			T * receive() { return &ref; }
+			explicit operator bool() const { return ref != nullptr; }
+
+			// Gives up ownership.
+			T release() { T result = ref; ref = nullptr; return result; }
+
+		private:
+			T ref;
+	};
+
+	ScopedCF<CFMutableDictionaryRef> CreateCFDictionary();
+
+	// Sets an integer value in a dictionary. Returns false if the number
+	// cannot be created.
+	bool SetCFNumber(CFMutableDictionaryRef dictionary, CFStringRef key, int64_t value);
+
+	// Writes the colour of a stream, given as the code points of ISO/IEC
+	// 23091-2 (H.273) that H.265, VP9 and AV1 share, into the extensions of a
+	// format description. Code points without a Core Media counterpart, and
+	// unspecified ones, are left out.
+	void SetColorExtensions(CFMutableDictionaryRef extensions, uint8_t primaries, uint8_t transfer,
+		uint8_t matrix);
+
+	// Whether a VideoToolbox session works in hardware, as the session reports
+	// it under the given property. Requiring hardware when the session is
+	// created is not relied on alone.
+	bool IsUsingHardware(VTSessionRef session, CFStringRef property);
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTH265Decoder.h

@@ -0,0 +1,55 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_H265_DECODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_H265_DECODER_H_
+
+#include "media/video/codec/macos/VTVideoDecoder.h"
+
+#include "api/video/encoded_image.h"
+
+#include <cstdint>
+#include <vector>
+
+namespace jni
+{
+	// Decodes H.265 Main profile on the media engine of a Mac, through
+	// VideoToolbox, and only there: there is no software decoder for H.265,
+	// and the session is required to be a hardware one.
+	//
+	// WebRTC hands over Annex B NAL units. The session is created from the
+	// VPS, SPS and PPS of a key frame, and again whenever a key frame brings
+	// parameter sets the session does not take. The slices go to VideoToolbox
+	// with length prefixes, as the format description has it, without the
+	// parameter sets.
+	class VTH265Decoder : public VTVideoDecoder
+	{
+		public:
+			VTH265Decoder();
+			~VTH265Decoder() override = default;
+
+			using webrtc::VideoDecoder::Decode;
+
+			bool Configure(const Settings & settings) override;
+			int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override;
+
+		private:
+			// The sample, rebuilt for each frame.
+			std::vector<uint8_t> sample;
+	};
+}
+
+#endif