feat: decode with Vulkan Video on Linux in the media player

NOTICE

@@ -24,14 +24,15 @@ HEVC/H.265
 Disclaimer regarding HEVC/H.265:
 This project decodes HEVC/H.265 video through hardware acceleration only, using
 the hardware decoders of the operating system or GPU (VideoToolbox, Direct3D
-11/12, NVDEC). It does not include, bundle, or distribute any software-based
-HEVC encoder (such as x265), and it never decodes HEVC in software. The FFmpeg
-libraries in the webrtc-java-media platform jars that decode HEVC in hardware
-(all but 32-bit ARM Linux) contain FFmpeg's HEVC decoder, because FFmpeg's
-hardware decoding runs through it: it reads the stream's headers and passes the
-coded data to the hardware decoder, which decodes the pictures. End-users and distributors are responsible for ensuring that the
-underlying operating system or hardware platform holds a valid license for HEVC
-playback and encoding.
+11/12, NVDEC, Vulkan Video). It does not include, bundle, or distribute any
+software-based HEVC encoder (such as x265), and it never decodes HEVC in
+software. The FFmpeg libraries in the webrtc-java-media platform jars that
+decode HEVC in hardware (all but 32-bit ARM Linux) contain FFmpeg's HEVC
+decoder, because FFmpeg's hardware decoding runs through it: it reads the
+stream's headers and passes the coded data to the hardware decoder, which
+decodes the pictures. End-users and distributors are responsible for ensuring
+that the underlying operating system or hardware platform holds a valid license
+for HEVC playback and encoding.
 
 DOCUMENTATION
 

docs/guide/media/media-files.md

@@ -224,7 +224,7 @@ The pictures are the same ones software decoding gives; in the tests, every fram
 
 **HEVC is decoded in hardware only.** A player decodes it on the media engine or GPU whether or not the flag is set, and never in software: `isHardwareDecoding()` is always `true` for it. Where the machine has no hardware decoder for HEVC, creating the player fails with an `IOException`; where the hardware does not take a stream (a profile it cannot decode) or fails while playing, playback stops with an error. The fallback below does not apply to it.
 
-- **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux through NVDEC on NVIDIA GPUs (x86-64 and ARM64; the driver provides `libcuda.so.1` and `libnvcuvid.so.1`, which are loaded when a player asks for hardware). The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM, VA-API on Intel and AMD GPUs), the video is decoded in software, and `isHardwareDecoding()` says so.
+- **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux (x86-64 and ARM64) through NVDEC on NVIDIA GPUs (the driver provides `libcuda.so.1` and `libnvcuvid.so.1`) and through Vulkan Video on the GPUs of Intel and AMD (the Vulkan loader `libvulkan.so.1` and a driver with the video extensions of the codec: Mesa's ANV or RADV, recent enough for each codec, AV1 and VP9 needing the newest). These libraries are loaded when a player asks for hardware, so a machine without them is not affected. The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM), the video is decoded in software, except HEVC, and `isHardwareDecoding()` says so.
 - **Codecs:** H.264, H.265/HEVC, VP9 and AV1, where the GPU decodes them: a GPU that is a few years old may lack AV1, and on a Mac only an M3 or later has it. VP8, MPEG-4 and MJPEG are decoded in software, and so is AV1 where there is no GPU decoder for it (it is decoded by dav1d, which is built into the FFmpeg libraries).
 - **Fallback** (every codec but HEVC): a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream.
 - **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it.

webrtc-java-media/src/main/cpp/CMakeLists.txt

@@ -201,3 +201,17 @@ if(FFMPEG_NVDEC)
                 "${FFNVCODEC_LICENSES_DIR}/LICENSE"
         COMMENT "Collecting the license of the NVDEC headers for the platform jar")
 endif()
+
+# So do the Vulkan headers it was built against for Vulkan Video.
+if(FFMPEG_VULKAN)
+    set(VULKAN_LICENSES_DIR "${NATIVES_DIR}/META-INF/licenses/vulkan")
+
+    add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
+        COMMAND ${CMAKE_COMMAND} -E make_directory "${VULKAN_LICENSES_DIR}"
+        COMMAND ${CMAKE_COMMAND} -E copy_if_different
+                "${CMAKE_CURRENT_SOURCE_DIR}/dependencies/vulkan/LICENSE.md"
+                "${CMAKE_CURRENT_SOURCE_DIR}/dependencies/vulkan/LICENSES/Apache-2.0.txt"
+                "${CMAKE_CURRENT_SOURCE_DIR}/dependencies/vulkan/LICENSES/MIT.txt"
+                "${VULKAN_LICENSES_DIR}"
+        COMMENT "Collecting the licenses of the Vulkan headers for the platform jar")
+endif()

webrtc-java-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt

@@ -198,10 +198,29 @@ elseif(LINUX AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|AMD64|amd64|aarch64|ar
         --enable-hwaccel=vp9_nvdec
         --enable-hwaccel=av1_nvdec
     )
+    # Vulkan Video, for the GPUs of Intel and AMD, which have no NVDEC: the
+    # player tries it after NVDEC. FFmpeg loads the Vulkan loader
+    # (libvulkan.so.1) with dlopen when a device is created, so, as with NVDEC,
+    # the libraries load on a machine without Vulkan. The build needs the
+    # Vulkan headers, newer than distributions have (VP9 needs 1.4.317), which
+    # dependencies/vulkan carries, found through a pkg-config file made below.
+    # The decoding itself is the driver's: Mesa (ANV for Intel, RADV for AMD)
+    # or NVIDIA's, recent enough to have the video extensions of each codec.
+    set(FFMPEG_VULKAN TRUE)
+
+    list(APPEND FFMPEG_COMPONENTS
+        --enable-vulkan
+        --enable-hwaccel=h264_vulkan
+        --enable-hwaccel=hevc_vulkan
+        --enable-hwaccel=vp9_vulkan
+        --enable-hwaccel=av1_vulkan
+    )
 endif()
 
-# The parent adds the license of the NVDEC headers to the platform jar.
+# The parent adds the licenses of the NVDEC and Vulkan headers to the platform
+# jar.
 set(FFMPEG_NVDEC "${FFMPEG_NVDEC}" PARENT_SCOPE)
+set(FFMPEG_VULKAN "${FFMPEG_VULKAN}" PARENT_SCOPE)
 
 # Written next to an install once it is built, naming the components it was
 # built with. Without it, an install from before a component was added would
@@ -875,26 +894,36 @@ fi
         set(FFMPEG_MAKE "${MAKE_EXECUTABLE}")
     endif()
 
-    if(FFMPEG_NVDEC)
-        # configure looks for the NVDEC headers through pkg-config, so that
-        # needs to be there, and to know where the vendored headers are.
+    if(FFMPEG_NVDEC OR FFMPEG_VULKAN)
+        # configure looks for the NVDEC and Vulkan headers through pkg-config,
+        # so that needs to be there, and to know where the vendored headers
+        # are.
         find_program(PKG_CONFIG_EXECUTABLE NAMES pkg-config pkgconf)
 
         if(NOT PKG_CONFIG_EXECUTABLE)
             message(FATAL_ERROR
-                "No pkg-config, which FFmpeg's configure uses to find the NVDEC headers. "
+                "No pkg-config, which FFmpeg's configure uses to find the NVDEC and Vulkan headers. "
                 "Install it, for example with: sudo apt-get install pkg-config")
         endif()
 
-        get_filename_component(FFNVCODEC_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../ffnvcodec" ABSOLUTE)
-        set(FFNVCODEC_PC_DIR "${CMAKE_CURRENT_BINARY_DIR}/ffnvcodec-pc")
+        set(HEADERS_PC_DIR "${CMAKE_CURRENT_BINARY_DIR}/headers-pc")
+
+        if(FFMPEG_NVDEC)
+            get_filename_component(FFNVCODEC_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../ffnvcodec" ABSOLUTE)
 
-        configure_file("${FFNVCODEC_DIR}/ffnvcodec.pc.in" "${FFNVCODEC_PC_DIR}/ffnvcodec.pc" @ONLY)
+            configure_file("${FFNVCODEC_DIR}/ffnvcodec.pc.in" "${HEADERS_PC_DIR}/ffnvcodec.pc" @ONLY)
+        endif()
+        if(FFMPEG_VULKAN)
+            get_filename_component(VULKAN_HEADERS_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../vulkan" ABSOLUTE)
+
+            configure_file("${VULKAN_HEADERS_DIR}/vulkan.pc.in" "${HEADERS_PC_DIR}/vulkan.pc" @ONLY)
+        endif()
 
         # Only our directory: a sysroot or library path of the host would put
-        # the includedir somewhere it is not.
+        # the includedir somewhere it is not, and the host's own vulkan.pc
+        # would name headers too old for VP9.
         set(FFMPEG_SHELL_PREAMBLE
-"${FFMPEG_SHELL_PREAMBLE}export PKG_CONFIG_PATH=\"${FFNVCODEC_PC_DIR}:$PKG_CONFIG_PATH\"
+"${FFMPEG_SHELL_PREAMBLE}export PKG_CONFIG_PATH=\"${HEADERS_PC_DIR}:$PKG_CONFIG_PATH\"
 unset PKG_CONFIG_SYSROOT_DIR PKG_CONFIG_LIBDIR
 ")
     endif()

webrtc-java-media/src/main/cpp/dependencies/ffmpeg/FFmpeg-NOTICE.txt.in

@@ -57,11 +57,12 @@ HEVC/H.265
 Disclaimer regarding HEVC/H.265:
 This project decodes HEVC/H.265 video through hardware acceleration only, using
 the hardware decoders of the operating system or GPU (VideoToolbox, Direct3D
-11/12, NVDEC). It does not include, bundle, or distribute any software-based
-HEVC encoder (such as x265), and it never decodes HEVC in software. The FFmpeg
-libraries in the webrtc-java-media platform jars that decode HEVC in hardware
-(all but 32-bit ARM Linux) contain FFmpeg's HEVC decoder, because FFmpeg's
-hardware decoding runs through it: it reads the stream's headers and passes the
-coded data to the hardware decoder, which decodes the pictures. End-users and distributors are responsible for ensuring that the
-underlying operating system or hardware platform holds a valid license for HEVC
-playback and encoding.
+11/12, NVDEC, Vulkan Video). It does not include, bundle, or distribute any
+software-based HEVC encoder (such as x265), and it never decodes HEVC in
+software. The FFmpeg libraries in the webrtc-java-media platform jars that
+decode HEVC in hardware (all but 32-bit ARM Linux) contain FFmpeg's HEVC
+decoder, because FFmpeg's hardware decoding runs through it: it reads the
+stream's headers and passes the coded data to the hardware decoder, which
+decodes the pictures. End-users and distributors are responsible for ensuring
+that the underlying operating system or hardware platform holds a valid license
+for HEVC playback and encoding.

webrtc-java-media/src/main/cpp/dependencies/vulkan/LICENSE.md

@@ -0,0 +1,18 @@
+Copyright 2015-2023 The Khronos Group Inc.
+
+Files in this repository fall under one of these licenses:
+
+- `Apache-2.0`
+- `MIT`
+
+Note: With the exception of `parse_dependency.py` the files using `MIT` license
+also fall under `Apache-2.0`. Example:
+
+```
+SPDX-License-Identifier: Apache-2.0 OR MIT
+```
+
+Full license text of these licenses is available at:
+
+  * Apache-2.0: https://opensource.org/licenses/Apache-2.0
+  * MIT: https://opensource.org/licenses/MIT