feat: encode H.265 and VP9 and decode VP9 with VA-API on Linux, when turned on

docs/guide/advanced/video-codecs.md

@@ -27,19 +27,27 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
 | Platform | H.264 | H.265 | AV1 |
 |---|---|---|---|
 | Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | NVENC on NVIDIA GPUs, otherwise Media Foundation on GPUs that encode H.265 | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) |
-| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | NVENC on NVIDIA GPUs | NVENC on NVIDIA RTX 40 and newer |
+| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | NVENC on NVIDIA GPUs; VA-API (Intel, AMD) when turned on, see below | NVENC on NVIDIA RTX 40 and newer |
 | macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | VideoToolbox, on Macs with a hardware HEVC encoder, such as all with Apple silicon | Software |
 
 Hardware encoders produce a single layer. A stream that asks for SVC, such as an AV1 stream with a scalability mode like `L1T3`, is encoded in software.
 
-VP9 can be encoded in hardware too, on Windows with the Media Foundation VP9 encoder of the GPU driver, which Intel GPUs have. It is off by default, since it has not been tested on such a GPU yet; turn it on with a system property before the factory is created:
+VP9 can be encoded in hardware too, with the VP9 encoder of the GPU driver, which Intel GPUs have: through Media Foundation on Windows and VA-API on Linux. It is off by default, since it has not been tested on such a GPU yet; turn it on with a system property before the factory is created:
 
 ```java
 System.setProperty("webrtc.hardware.vp9Encoder", "true");
 ```
 
 Where the GPU has no VP9 encoder, or the stream asks for SVC, VP9 is still encoded with libvpx.
 
+On Linux, H.265 can be encoded with the VA-API encoder of Intel and AMD drivers, also off by default until it has been tested on such GPUs:
+
+```java
+System.setProperty("webrtc.hardware.vaapiExperimental", "true");
+```
+
+It is offered only where the driver encodes H.265 Main, and only for sending: Intel and AMD GPUs do not decode H.265 here yet, so the other end has to.
+
 NVENC needs an NVIDIA driver of version 522 or newer on Windows, 520 or newer on Linux. VA-API needs libva 2 and a driver that encodes H.264, such as Intel's `intel-media-va-driver` (iHD) or Mesa's `mesa-va-drivers` for AMD, and access to a render node in `/dev/dri`. All of them are loaded at run time, so nothing needs to be installed on machines without them.
 
 The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding these in hardware never changes what is negotiated. H.265 (Main profile) has no software encoder: it is offered after the other codecs, only where the hardware encodes it, so the codec a peer connection prefers stays the same, and a call uses H.265 only if it is preferred (e.g. with `RTCRtpTransceiver.setCodecPreferences()`) and the other end decodes it. On macOS, VideoToolbox is required to encode H.265 in hardware, and never falls back to its software encoder; a Mac without a hardware HEVC encoder does not offer H.265. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`.
@@ -60,7 +68,7 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
 | Platform | Hardware decoding |
 |---|---|
 | Windows | H.264, H.265 (Main profile), AV1 and VP9 (profile 0), on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
-| Linux | H.264 and VP9 (profile 0) on NVIDIA GPUs, through NVDEC; VA-API, for Intel and AMD GPUs, is not there yet |
+| Linux | H.264 and VP9 (profile 0) on NVIDIA GPUs, through NVDEC; VP9 (profile 0) on Intel and AMD GPUs through VA-API, when turned on with `webrtc.hardware.vaapiExperimental` |
 | macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`; VP9 (profile 0) and H.265 (Main profile) through the decoders of VideoToolbox, on Macs that have them, and AV1 (profile 0) on Apple M3 and newer |
 
 On Linux, NVDEC needs an NVIDIA driver, which provides `libcuda.so.1` and `libnvcuvid.so.1`; both are loaded when the factory is made, so a machine without the driver is not affected. Streams with spatial layers are decoded in software.

webrtc-jni/src/main/cpp/include/media/video/codec/HardwareCodecOptions.h

@@ -24,10 +24,18 @@ namespace jni
 	// tested on one.
 	constexpr const char * kHardwareVp9EncoderProperty = "webrtc.hardware.vp9Encoder";
 
+	// The Java system property that turns on the VA-API codecs beyond H.264
+	// encoding, which are off by default: they have not been tested on an
+	// Intel or AMD GPU under Linux yet.
+	constexpr const char * kExperimentalVaapiProperty = "webrtc.hardware.vaapiExperimental";
+
 	// Whether hardware VP9 encoding is turned on: the Java system property
 	// above is "true", as Boolean.getBoolean has it. Callable from any thread;
 	// false where Java cannot be asked, for one with an exception pending.
 	bool IsHardwareVp9EncoderEnabled();
+
+	// Whether the experimental VA-API codecs are turned on, the same way.
+	bool IsExperimentalVaapiEnabled();
 }
 
 #endif

webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiEncodeSession.h

@@ -0,0 +1,94 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_ENCODE_SESSION_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_ENCODE_SESSION_H_
+
+#include "media/video/codec/linux/VaapiLibrary.h"
+
+#include "api/video/video_frame.h"
+
+#include <va/va.h>
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+namespace jni
+{
+	// What every VA-API encoder of a single codec needs: a configuration and
+	// a context with an input surface and two reconstructed ones (the frame
+	// being encoded and the one it refers to), the output buffer, the
+	// parameter buffers of a frame, uploading a frame as NV12, encoding it
+	// and reading what came out. The codec's encoder fills in the parameters.
+	//
+	// Used by the H.265 and VP9 encoders; the H.264 one keeps its own, older
+	// copy of the same.
+	class VaapiEncodeSession
+	{
+		public:
+			// The name says whose operations failed, in the log.
+			VaapiEncodeSession(VaapiLibrary & library, std::string name);
+			~VaapiEncodeSession();
+
+			VaapiEncodeSession(const VaapiEncodeSession &) = delete;
+			VaapiEncodeSession & operator=(const VaapiEncodeSession &) = delete;
+
+			// Creates the session for 4:2:0 at constant bitrate, with surfaces
+			// of the given size, which is that of the frame aligned as the
+			// codec needs.
+			bool Create(VAProfile profile, VAEntrypoint entrypoint, uint32_t surfaceWidth, uint32_t surfaceHeight);
+			void Destroy();
+
+			VASurfaceID InputSurface() const;
+			VASurfaceID Reconstructed(int index) const;
+			VABufferID CodedBuffer() const;
+
+			// Puts the frame into the input surface as NV12, at the given size.
+			// A frame up to tolerance pixels larger is cropped, which needs no
+			// copy; one of another size, which raced with a change of size, is
+			// scaled.
+			bool Upload(const webrtc::VideoFrame & frame, int width, int height, int tolerance);
+
+			bool AddBuffer(VABufferType type, void * data, size_t size);
+			bool AddMiscParameter(VAEncMiscParameterType type, const void * data, size_t size);
+			// The rate control, frame rate and HRD parameters, for constant
+			// bitrate with the given initial quantizer.
+			bool AddRateControl(uint32_t bitrateBps, uint32_t framerate, uint32_t initialQp);
+
+			// Encodes the input surface with the parameter buffers added, and
+			// waits for it.
+			bool Encode();
+			bool ReadOutput(std::vector<uint8_t> & output);
+			// Drops the parameter buffers of the frame.
+			void DropBuffers();
+
+			bool Check(VAStatus status, const char * operation) const;
+
+		private:
+			VaapiLibrary & library;
+			const VaapiFunctions & va;
+			const std::string name;
+
+			VAConfigID config;
+			VAContextID context;
+			VASurfaceID surfaces[3];
+			VABufferID codedBuffer;
+			std::vector<VABufferID> frameBuffers;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiH265Encoder.h

@@ -0,0 +1,115 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_H265_ENCODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_H265_ENCODER_H_
+
+#include "media/video/codec/EncoderOutputProcessor.h"
+#include "media/video/codec/linux/VaapiEncodeSession.h"
+#include "media/video/codec/linux/VaapiLibrary.h"
+
+#include "api/video/video_frame.h"
+#include "api/video_codecs/sdp_video_format.h"
+#include "api/video_codecs/video_codec.h"
+#include "api/video_codecs/video_encoder.h"
+
+#include <va/va.h>
+#include <va/va_enc_hevc.h>
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+namespace jni
+{
+	// Encodes H.265 Main with the GPU driver's VA-API encoder, as Intel and
+	// AMD drivers provide it on Linux. Experimental: it has not run on such a
+	// GPU yet, and is used only where it is turned on.
+	//
+	// It works as VaapiH264Encoder does: synchronous, one reference frame,
+	// P-frames only, the driver writing the parameter sets and slice headers
+	// from the parameters it is given. The block sizes and coding tools are
+	// those the driver says it takes, as FFmpeg's VA-API encoder chooses them.
+	// The frame size is kept to a multiple of the smallest coding block, so
+	// the coded picture is the frame and needs no cropping. There is no
+	// software H.265 to fall back to: anything that fails, including a driver
+	// that writes no parameter sets into key frames, is an error, upon which
+	// WebRTC switches to the next codec it negotiated.
+	class VaapiH265Encoder : public webrtc::VideoEncoder
+	{
+		public:
+			VaapiH265Encoder(VaapiLibrary & library, VAEntrypoint entrypoint, const webrtc::SdpVideoFormat & format);
+			~VaapiH265Encoder() override;
+
+			int32_t InitEncode(const webrtc::VideoCodec * codecSettings, const Settings & settings) override;
+			int32_t RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * callback) override;
+			int32_t Release() override;
+			int32_t Encode(const webrtc::VideoFrame & frame, const std::vector<webrtc::VideoFrameType> * frameTypes) override;
+			void SetRates(const RateControlParameters & parameters) override;
+			EncoderInfo GetEncoderInfo() const override;
+
+		private:
+			// What the driver says about the HEVC encoder it has.
+			void QueryCapabilities();
+			bool Submit(bool idr);
+
+			void FillSequence(VAEncSequenceParameterBufferHEVC & sequence) const;
+			void FillPicture(VAEncPictureParameterBufferHEVC & picture, bool idr) const;
+			void FillSlice(VAEncSliceParameterBufferHEVC & slice, bool idr) const;
+			// The picture the current one refers to.
+			VAPictureHEVC Reference() const;
+
+		private:
+			VaapiLibrary & library;
+			const VAEntrypoint entrypoint;
+			const std::string implementationName;
+			VaapiEncodeSession session;
+
+			// The sizes, in log2 of luma samples, and the tools the driver
+			// takes.
+			uint32_t log2CtbSize;
+			uint32_t log2MinCbSize;
+			uint32_t log2MinTbSize;
+			uint32_t log2MaxTbSize;
+			uint32_t maxTransformDepthInter;
+			uint32_t maxTransformDepthIntra;
+			bool amp;
+			bool sao;
+			bool signDataHiding;
+			bool transformSkip;
+			bool cuQpDelta;
+
+			bool initialized;
+			webrtc::VideoCodec codecSettings;
+			uint32_t widthInCtbs;
+			uint32_t heightInCtbs;
+			uint32_t bitrateBps;
+			uint32_t framerate;
+			bool ratesChanged;
+
+			// The picture order count since the last IDR picture, and which of
+			// the reconstructed surfaces holds the current picture.
+			int32_t pictureOrder;
+			int current;
+			bool referenceValid;
+
+			webrtc::EncodedImageCallback * callback;
+
+			EncoderOutputProcessor outputProcessor;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiLibrary.h

@@ -56,6 +56,7 @@ namespace jni
 		decltype(&::vaDeriveImage) DeriveImage = nullptr;
 		decltype(&::vaCreateImage) CreateImage = nullptr;
 		decltype(&::vaPutImage) PutImage = nullptr;
+		decltype(&::vaGetImage) GetImage = nullptr;
 		decltype(&::vaDestroyImage) DestroyImage = nullptr;
 	};
 
@@ -77,9 +78,17 @@ namespace jni
 
 			VADisplay Display() const;
 
-			// The encode entrypoint the driver offers, full or low-power.
+			// The encode entrypoint the driver offers for H.264, full or
+			// low-power.
 			VAEntrypoint Entrypoint() const;
 
+			// Whether the driver encodes the profile, 4:2:0 at constant
+			// bitrate, and with which entrypoint, full or low-power.
+			bool EncodeEntrypoint(VAProfile profile, VAEntrypoint * entrypoint) const;
+
+			// Whether the driver decodes the profile into 4:2:0 surfaces.
+			bool Decodes(VAProfile profile) const;
+
 			// The driver, e.g. "Intel iHD driver for Intel(R) Gen Graphics".
 			const std::string & Vendor() const;
 
@@ -90,7 +99,7 @@ namespace jni
 
 			bool Load();
 			bool OpenDisplay();
-			bool SupportsEncoding(VADisplay display, VAEntrypoint * entrypoint) const;
+			bool SupportsEncoding(VADisplay display, VAProfile profile, VAEntrypoint * entrypoint) const;
 
 		private:
 			DynamicLibrary va;

webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiVideoDecoderFactory.h

@@ -0,0 +1,56 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_VIDEO_DECODER_FACTORY_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_VIDEO_DECODER_FACTORY_H_
+
+#include "media/video/codec/linux/VaapiLibrary.h"
+
+#include "api/environment/environment.h"
+#include "api/video_codecs/sdp_video_format.h"
+#include "api/video_codecs/video_decoder.h"
+#include "api/video_codecs/video_decoder_factory.h"
+
+#include <memory>
+#include <vector>
+
+namespace jni
+{
+	// Creates VA-API decoders on the GPUs of Intel and AMD, for VP9 profile 0
+	// where the driver decodes it, the format WebRTC's software decoder
+	// offers too.
+	class VaapiVideoDecoderFactory : public webrtc::VideoDecoderFactory
+	{
+		public:
+			// Returns a factory, or null if there is no VA-API driver, or one
+			// that decodes no codec offered here.
+			static std::unique_ptr<VaapiVideoDecoderFactory> Create();
+
+			~VaapiVideoDecoderFactory() override = default;
+
+			std::vector<webrtc::SdpVideoFormat> GetSupportedFormats() const override;
+			std::unique_ptr<webrtc::VideoDecoder> Create(const webrtc::Environment & env,
+				const webrtc::SdpVideoFormat & format) override;
+
+		private:
+			explicit VaapiVideoDecoderFactory(VaapiLibrary & library);
+
+		private:
+			VaapiLibrary & library;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiVideoEncoderFactory.h

@@ -32,13 +32,15 @@ namespace jni
 	// Creates VA-API encoders on the GPU whose driver encodes H.264, as Intel
 	// and AMD drivers do. It offers H.264 in the profiles WebRTC's software
 	// encoder offers too, and only with packetization mode 1: mode 0 needs
-	// each NAL unit to fit a packet.
+	// each NAL unit to fit a packet. With the experimental VA-API codecs
+	// turned on, it offers H.265 Main as well, and with hardware VP9 encoding
+	// turned on VP9 profile 0, where the driver encodes them.
 	class VaapiVideoEncoderFactory : public webrtc::VideoEncoderFactory
 	{
 		public:
 			// Returns a factory, or null if no GPU driver encodes H.264
 			// through VA-API.
-			static std::unique_ptr<VaapiVideoEncoderFactory> Create();
+			static std::unique_ptr<VaapiVideoEncoderFactory> Create(bool experimental, bool vp9Enabled);
 
 			~VaapiVideoEncoderFactory() override = default;
 
@@ -47,10 +49,15 @@ namespace jni
 				const webrtc::SdpVideoFormat & format) override;
 
 		private:
-			explicit VaapiVideoEncoderFactory(VaapiLibrary & library);
+			VaapiVideoEncoderFactory(VaapiLibrary & library, bool h265, VAEntrypoint h265Entrypoint, bool vp9,
+				VAEntrypoint vp9Entrypoint);
 
 		private:
 			VaapiLibrary & library;
+			const bool h265;
+			const VAEntrypoint h265Entrypoint;
+			const bool vp9;
+			const VAEntrypoint vp9Entrypoint;
 	};
 }