Repository navigation
feat: decode HEVC and AV1 in hardware in the media player - #331
Merged
Merged
Conversation
The player decoded only H.264 and VP9 on the GPU. FFmpeg is now built with the hardware decoders of HEVC and AV1 as well: Direct3D 11 and 12 on Windows, NVDEC on Linux, VideoToolbox on macOS. FFmpeg's own AV1 decoder has no software path, so AV1 gets dav1d as its software decoder. It is a new submodule (1.5.4), built with Meson before FFmpeg into a static library that is linked into libavcodec. On Windows that is with the static C runtime, which is the one FFmpeg is built with. Its version and options are recorded in components.txt, so changing them rebuilds on its own. Meson and Ninja are new build prerequisites, installed by the CI preparation steps. Cross targets get a Meson cross file made from the same variables FFmpeg is configured with; only windows-x86_64 has been built so far. FFmpeg prefers libdav1d to its native AV1 decoder, and only the native one has hardware configurations, so VideoDecoder picks a decoder that has one when hardware decoding is asked for, and otherwise keeps the preferred one. The licence of dav1d and a notice go into the platform jar. Tests compare every frame decoded in hardware with the software decoder for HEVC and AV1, at 320x240 and 1080p. AV1 has its own property, webrtc.test.hardwareAv1Decoding, since older GPUs cannot decode it.
Meson finds the compiler by its path inside the Xcode toolchain, which skips
the /usr/bin/cc shim that points SDKROOT at an SDK, so on the macOS ARM64
runner it reported that the compiler cannot compile programs ("library
'System' not found"). The SDK is named for the Meson run now. FFmpeg's own
configure gets it from the shim, so SDKROOT is unset again before it runs.
Matroska has no codec ID of its own for G.711, so avformat_query_codec said no and the recorder left PCMU and PCMA tracks out of .mkv files, although the docs, the error message and a comment all promise that Matroska holds every codec WebRTC sends. The Matroska muxer writes G.711 all the same, in its compatibility mode for Windows audio codecs (A_MS/ACM), as "ffmpeg -c copy" does, so the recorder now lets it. The recorder tests can now send the call's audio in a chosen codec, and record PCMU and PCMA into Matroska, reading them back as such.
The player decoded HEVC in software unless asked for hardware, and fell back to software wherever the hardware had no decoder, did not take a stream, or failed. HEVC is now decoded on the hardware only: a player sends it there whether hardwareDecoding is set or not, and where the hardware cannot decode it, creating the player fails with "Decoder not found", or playback stops with an error. No path opens a software decoder for it. FFmpeg's hevc decoder stays in the build, because the HEVC hwaccels run inside it (configure selects it for every one of them); it parses the headers and hands the slices to the hardware, but VideoDecoder never lets it decode a picture: no software open, no software pixel format, no fall back. The tests compared hardware HEVC with software decoding, which no longer exists. They now compare with luma checksums the software decoder gave before this change, which every conforming decoder matches. A new test fault hides the hardware devices, so that a machine without a GPU decoder is tested on every machine, CI included: HEVC does not open there, while H.264 still falls back to software.
HEVC is decoded through the hardware decoders of the platform only, and the project bundles no software HEVC encoder. The disclaimer says so, and that the FFmpeg libraries of the webrtc-java-media platform jars contain FFmpeg's HEVC decoder, which the hardware decoding runs through, and leaves the license for HEVC playback and encoding to end-users and distributors. It goes into the root NOTICE and into the FFmpeg notice each webrtc-java-media platform jar carries.
--enable-decoder=hevc put FFmpeg's HEVC decoder into every build, but each HEVC hwaccel selects that decoder itself, so where there is one the decoder comes along without it, and the line only made a difference where there is none: on 32-bit ARM Linux it shipped an HEVC decoder that VideoDecoder never lets decode. Without the line, that build has no HEVC decoder at all, and the others are as before (CONFIG_HEVC_DECODER is still 1 on Windows, through the D3D11 and D3D12 hwaccels). The HEVC disclaimer in the NOTICE files now says which platform jars contain the decoder: those that decode HEVC in hardware, all but 32-bit ARM Linux.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The media player decoded only H.264 and VP9 on the GPU. This adds HEVC and AV1:
libdav1dto its native AV1 decoder, and only the native one has hardware configurations.VideoDecodertherefore picks a decoder that has one when hardware decoding is asked for, and otherwise keeps the preferred one.components.txt, so changing them rebuilds FFmpeg on its own.HEVC in hardware only, never in software
HEVC is now decoded on the hardware only. The player sends it to the hardware decoder whether or not
hardwareDecodingis set, soisHardwareDecoding()is alwaystruefor it. Without a hardware decoder for HEVC, creating the player fails with anIOException("Decoder not found"). If the hardware does not take a stream, or fails while playing, playback stops with an error. Every path that could reach software is closed inVideoDecoder: software open, a software pixel format inget_format, and the fallback after a hardware failure. Other codecs keep their software fallback.hevcdecoder can't be left out where HEVC is decoded in hardware: every HEVC hwaccel runs inside it (configureselects it), parsing the parameter sets and slice headers and handing each slice to the hardware. It never decodes a picture itself. The explicit--enable-decoder=hevcis gone, so the decoder only comes in through the hwaccels. Windows, macOS and Linux x64/ARM64 are unchanged (CONFIG_HEVC_DECODERis still 1 on Windows), and 32-bit ARM Linux, which has no HEVC hwaccel, now ships no HEVC decoder at all.NOTICEand the FFmpeg notice in each media platform jar carry an HEVC/H.265 disclaimer. It says HEVC is decoded through hardware acceleration only and no software HEVC encoder is bundled. It also says that the jars that decode HEVC in hardware (all but 32-bit ARM Linux) contain FFmpeg's HEVC decoder for that reason, and it leaves the HEVC license to end-users and distributors. The compiled WebRTC library has no HEVC decoder (only libavcodec's table of HEVC profile names, checked on Windows).media-files.md, theMediaPlayerandMediaFileSourceJavadoc) describe the new behaviour.Also: G.711 in Matroska recordings
MediaRecorderleft PCMU and PCMA tracks out of.mkvfiles with a warning, although the docs, the error message and a code comment all say Matroska holds every codec WebRTC sends. Matroska has no codec ID of its own for G.711, soavformat_query_codecsays no. The Matroska muxer writes it all the same, in its compatibility mode for Windows audio codecs (A_MS/ACMwith a WAVEFORMATEX), which is whatffmpeg -c copyproduces too.RecorderTrack'sCanHoldnow lets G.711 into Matroska; WebM is unchanged. The recording guide notes that players built on FFmpeg read this mode and some others do not.Testing
On Windows x64 (AMD Radeon RX 9070 XT), with the hardware decoders required (
webrtc.test.hardwareDecodingand the newwebrtc.test.hardwareAv1Decoding), all 81 tests of the media module pass.HardwareDecodingTestnow compares every frame decoded in hardware with the software decoder for HEVC and AV1, at 320x240 and 1080p (four new test assets of 19 to 362 KB).With HEVC hardware only, there is no software decoding to compare HEVC with, so
HardwareDecodingTestcompares it with luma checksums that FFmpeg's software decoder gave before the change (media-test-hevc*.luma.txt; HEVC decoding is exact, so every conforming decoder matches them). A new test fault,HardwareFault.hideDevices(), makes any machine look as if it had no GPU decoder, so CI runs the case too: HEVC does not open, and H.264 still falls back to software. Each HEVC test also checks that no software decoder was opened. All 88 media module tests pass on Windows x64 with hardware required and none skipped, before and after FFmpeg was rebuilt without--enable-decoder=hevc. With HEVC allowed back into software, the four tests of the hardware-only behaviour fail.For G.711,
TestCallcan now send the call's audio in a chosen codec, and two new tests (recordsMuLawIntoMatroska,recordsALawIntoMatroska) record PCMU and PCMA into.mkvand read them back aspcm_mulaw/pcm_alawat 8 kHz mono. All 9MediaRecorderTesttests pass on Windows x64, and with theCanHoldchange undone both new tests fail with the old "cannot hold" warning.Not verified
Only windows-x86_64 has been built. The Meson cross files for the other targets (Linux x64/ARM64/ARM32, macOS x86-64 and ARM64, Windows ARM64) are generated from the toolchain files and the variables FFmpeg is configured with, but have not been run. The CI builds are the first to exercise them. 10-bit and other profiles were not tested, nor were streams other than the synthetic test assets. G.711 recordings in Matroska were checked by reading them back through FFmpeg, not in other players. That the 32-bit ARM Linux build has no HEVC decoder follows from FFmpeg's configure logic, but has not been built; neither has HEVC hardware decoding on macOS or Linux, where it now has no software fallback.