Skip to content

feat: list the hardware codecs of the hardware video codec factories - #359

Merged
devopvoid merged 4 commits into
mainfrom
feat/hardware-codec-list
Oct 12, 2026
Merged

devopvoid merged 4 commits into
mainfrom
feat/hardware-codec-list

Conversation

@devopvoid

Copy link
Copy Markdown
Owner

Summary

Lets applications tell which video codecs the GPU handles, and makes the hardware codec factories cheaper and more consistent to use.

  • getHardwareCodecs() on HardwareVideoEncoderFactory and HardwareVideoDecoderFactory: the supported codecs, in their order, that a hardware encoder or decoder of the machine takes first (on macOS, H.264 through VideoToolbox included). Natively, the composite hardware factories implement WebRTC's QueryCodecSupport and report a format as power efficient when a hardware sub-factory supports it.
  • CodecListExample builds its PeerConnectionFactory with the hardware factories and marks every video codec as HARDWARE or SOFTWARE.
  • H.264 High 4:4:4 stays off the GPU decoders. NVDEC and Media Foundation offered every H.264 format WebRTC decodes, though GPU decoders take 4:2:0 only, so a 4:4:4 stream failed on the GPU and lost frames until the next key frame reached the software decoder. Both now offer the 4:2:0 profiles only.
  • The GPU is probed once per process. The platform's hardware factories are made once per setting of the hardware codec options (webrtc.hardware.vp9Encoder, webrtc.hardware.vaapiExperimental) and shared. A PeerConnectionFactory built with the hardware factories makes them natively, as it does the default ones, instead of going through Java for every codec; before, it probed twice, and the two probes could disagree. A Java factory of one's own makes the built-in factories only when it first hands out a native codec.

On a Windows machine with an AMD GPU, after the first probe of the process, creating both hardware factories takes about 4 ms and the PeerConnectionFactory about 1.3 ms; before, each setup probed twice, about 40 ms per decoder probe.

One behaviour change: a GPU, driver or codec extension installed or removed while the process runs is seen by a new process only. The Javadoc and the video codecs guide say so.

Testing

  • mvn -pl webrtc-jni,webrtc package on Windows x64 (AMD GPU): 289 tests, 0 failures, 3 skipped (two macOS-only, the opt-in VP9 encoder).
  • New tests: hardware codecs are supported codecs in the same order and include every codec without a software implementation; no H.264 High 4:4:4 among the hardware decoders; a factory built with the hardware factories offers exactly their codecs; a Java factory delegating to HardwareVideoEncoderFactory still encodes H.264 on the GPU; the H.264 and AV1 hardware tests check that the encoded codec is listed as hardware.
  • Ran CodecListExample: H.264 Constrained Baseline/Baseline (packetization mode 1), AV1 and H.265 encode in hardware; VP9 profile 0, H.264 4:2:0, AV1 profile 0 and H.265 decode in hardware.
  • Not run locally: the NVDEC change (Linux) and the macOS H.264 marking, which CI builds.

The example builds its PeerConnectionFactory with the hardware video codec
factories, so the RTP capabilities include the codecs only the GPU handles,
and compares the codecs of the default and the hardware factories, marking
those only available in hardware.
HardwareVideoEncoderFactory and HardwareVideoDecoderFactory get
getHardwareCodecs(): the supported codecs, in their order, that a hardware
encoder or decoder of the machine takes first, and on macOS H.264, which
the default factories handle with VideoToolbox. Both lists come from one
native factory, so the hardware is probed once.

Natively, the composite hardware factories implement WebRTC's
QueryCodecSupport and report a format as power efficient when a hardware
sub-factory supports it.

CodecListExample marks each video codec as hardware or software with it.
NVDEC and the Media Foundation decoders offered every H.264 format WebRTC
decodes, High 4:4:4 Predictive among them, though GPU decoders take 4:2:0
only. A 4:4:4 stream started on the GPU, failed, and lost frames until the
next key frame reached the software decoder, and getHardwareCodecs() listed
the profile as hardware. Both now offer the 4:2:0 profiles only, so 4:4:4
streams go to the software decoder from the start.
Every hardware codec factory probed the GPU anew, about 40 ms for the
decoders on Windows, and a PeerConnectionFactory built with the hardware
factories probed twice: once for their Java objects, and once more in the
wrappers that took them through Java, where the two probes could also
disagree.

The platform's hardware factories, which hold nothing but what probing
found, are now made once per process and setting of the hardware codec
options, and shared. PeerConnectionFactory makes the hardware factories
natively, as it does the default ones, without a detour through Java for
every codec it creates. A Java factory of one's own makes the built-in
factories only when it first hands out a native codec, so one that never
does probes no hardware.
@devopvoid
devopvoid merged commit 64c487d into main Oct 12, 2026
24 checks passed
@devopvoid
devopvoid deleted the feat/hardware-codec-list branch October 12, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant