fix: fall back from a failed hardware video decoder without losing or damaging pictures

docs/guide/media/media-files.md

@@ -224,7 +224,7 @@ The pictures are the same ones software decoding gives; in the tests, every fram
 
 - **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux through NVDEC on NVIDIA GPUs (x86-64 and ARM64; the driver provides `libcuda.so.1` and `libnvcuvid.so.1`, which are loaded when a player asks for hardware). The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM, VA-API on Intel and AMD GPUs), the video is decoded in software, and `isHardwareDecoding()` says so.
 - **Codecs:** H.264 and VP9. VP8, MPEG-4, MJPEG and H.265/HEVC are decoded in software.
-- **Fallback:** a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails after it has produced a picture, decoding goes on in software from the next key frame.
+- **Fallback:** a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream.
 - **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it.
 
 ## Playing a Live Stream

webrtc-java-media/src/main/cpp/include/JNI_HardwareFault.h

@@ -0,0 +1,60 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <jni.h>
+/* Header for class dev_onvoid_webrtc_media_player_HardwareFault */
+
+#ifndef _Included_dev_onvoid_webrtc_media_player_HardwareFault
+#define _Included_dev_onvoid_webrtc_media_player_HardwareFault
+#ifdef __cplusplus
+extern "C" {
+#endif
+	/*
+	 * Class:     dev_onvoid_webrtc_media_player_HardwareFault
+	 * Method:    failSend
+	 * Signature: (ZI)V
+	 */
+	JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failSend
+	(JNIEnv *, jclass, jboolean, jint);
+
+	/*
+	 * Class:     dev_onvoid_webrtc_media_player_HardwareFault
+	 * Method:    failDrain
+	 * Signature: ()V
+	 */
+	JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failDrain
+	(JNIEnv *, jclass);
+
+	/*
+	 * Class:     dev_onvoid_webrtc_media_player_HardwareFault
+	 * Method:    disarm
+	 * Signature: ()V
+	 */
+	JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_disarm
+	(JNIEnv *, jclass);
+
+	/*
+	 * Class:     dev_onvoid_webrtc_media_player_HardwareFault
+	 * Method:    softwareThreads
+	 * Signature: ()I
+	 */
+	JNIEXPORT jint JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_softwareThreads
+	(JNIEnv *, jclass);
+
+#ifdef __cplusplus
+}
+#endif
+#endif

webrtc-java-media/src/main/cpp/include/media/HardwareFault.h

@@ -0,0 +1,63 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef WEBRTC_JAVA_MEDIA_HARDWARE_FAULT_H_
+#define WEBRTC_JAVA_MEDIA_HARDWARE_FAULT_H_
+
+namespace ffmpeg
+{
+	// Makes the hardware video decoder fail on purpose, so that the fall back
+	// to software can be tested: a GPU cannot be made to fail on demand.
+	//
+	// Only the tests arm it, through JNI_HardwareFault, and a decoder that is
+	// not armed pays a single atomic load per packet. A fault applies to the
+	// hardware decoder only, never to the software one that takes over, and
+	// is armed for the next decoder that runs into it.
+	class HardwareFault
+	{
+		public:
+			// Fails the hardware decoder's send of the packet that follows the
+			// given number of them, once, with the given AVERROR.
+			static void FailSend(int error, int after_packets);
+
+			// Makes the hardware decoder produce nothing, and fail with the
+			// given AVERROR once it is asked for pictures after it was told
+			// to drain: the failure of a decoder that never delivered.
+			static void FailDrain(int error);
+
+			// Forgets any fault, and the software decoder noted last.
+			static void Disarm();
+
+			// Notes how many threads a software decoder of the video decoder
+			// got, for a test to see that one that took over from the hardware
+			// has all it can use.
+			static void NoteSoftwareDecoder(int threads);
+
+			// The thread count noted last, or 0.
+			static int SoftwareThreads();
+
+			// The error to fail the next send with, or 0.
+			static int NextSendError();
+
+			// Whether the hardware decoder's pictures are held back.
+			static bool HoldsPictures();
+
+			// The error to fail the receive of a picture with once draining.
+			static int DrainError();
+	};
+}
+
+#endif

webrtc-java-media/src/main/cpp/include/media/VideoDecoder.h

@@ -91,6 +91,10 @@ namespace ffmpeg
 			// frame, to be decoded again in software if it fails by then.
 			static constexpr size_t kMaxKeptPackets = 32;
 
+			// How many packets may be passed over while waiting for a key
+			// frame, for a stream whose packets do not say which ones are.
+			static constexpr size_t kMaxSkippedPackets = 250;
+
 			// Opens the codec context of the stream, in hardware or not.
 			int OpenContext(bool hardware);
 
@@ -106,11 +110,27 @@ namespace ffmpeg
 			// into system memory first.
 			int ReceiveFromCodec(AVFrame ** frame, int64_t * timestamp_us);
 
-			// Opens a software decoder in place of the hardware one that
-			// failed with the given error, and decodes the packets kept
-			// since the start again. Returns the AVERROR the caller gets:
-			// EAGAIN once decoding goes on, or an error if it cannot.
-			int FallBackToSoftware(int error);
+			// Whether a result of the hardware decoder means that it failed.
+			// A stream that cannot be decoded, which is how corrupt data shows,
+			// fails a software decoder as well and says nothing of the
+			// hardware, except before its first picture, where the software
+			// decoder is the one to tell.
+			bool IsHardwareFailure(int result) const;
+
+			// Opens a software decoder in place of the hardware one, which
+			// failed with the given error, or if that is 0 did not take the
+			// stream, and decodes the packets kept since the start again.
+			// The packet being sent when it happened, if any, is the current
+			// one. Returns 0, or the error that kept a decoder from opening.
+			int FallBackToSoftware(int error, const AVPacket * current);
+
+			// Hands a packet, or null to drain, to the codec, taking its
+			// pictures aside while it asks for that.
+			void SendToSoftware(const AVPacket * packet);
+
+			// Whether a packet is passed over because the decoder is waiting
+			// for a key frame, which it is after it lost pictures it needs.
+			bool SkipUntilKeyFrame(const AVPacket * packet);
 
 			void KeepPacket(const AVPacket * packet);
 			void DropKeptPackets();
@@ -153,6 +173,19 @@ namespace ffmpeg
 			// sent to it are kept.
 			bool probation_ = false;
 			std::vector<AVPacket *> kept_;
+
+			// Set by ChooseFormat when the decoder went for a software format.
+			bool software_selected_ = false;
+
+			// Whether the end of the stream was announced, which a decoder that
+			// takes over has to be told as well.
+			bool draining_ = false;
+
+			// After a software decoder took over from a hardware one that had
+			// delivered pictures, it has none of the ones the next refer to,
+			// and starts again at a key frame.
+			bool awaiting_key_frame_ = false;
+			size_t skipped_ = 0;
 			std::deque<Pending> pending_;
 	};
 }

webrtc-java-media/src/main/cpp/src/JNI_HardwareFault.cpp

@@ -0,0 +1,51 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "JNI_HardwareFault.h"
+#include "media/HardwareFault.h"
+
+extern "C" {
+#include <libavutil/error.h>
+}
+
+// The Java class this belongs to exists in the tests only, so that nothing but
+// them can reach the native side of a fault.
+
+JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failSend
+(JNIEnv * env, jclass caller, jboolean invalidData, jint afterPackets)
+{
+	// Bad data is the stream's fault; anything else the hardware's.
+	ffmpeg::HardwareFault::FailSend(invalidData == JNI_TRUE ? AVERROR_INVALIDDATA : AVERROR_EXTERNAL,
+			afterPackets);
+}
+
+JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failDrain
+(JNIEnv * env, jclass caller)
+{
+	ffmpeg::HardwareFault::FailDrain(AVERROR_EXTERNAL);
+}
+
+JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_disarm
+(JNIEnv * env, jclass caller)
+{
+	ffmpeg::HardwareFault::Disarm();
+}
+
+JNIEXPORT jint JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_softwareThreads
+(JNIEnv * env, jclass caller)
+{
+	return ffmpeg::HardwareFault::SoftwareThreads();
+}

webrtc-java-media/src/main/cpp/src/media/HardwareFault.cpp

@@ -0,0 +1,94 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "media/HardwareFault.h"
+
+#include <atomic>
+
+namespace
+{
+	enum Mode
+	{
+		kNone,
+		kSend,
+		kDrain
+	};
+
+	std::atomic<int> mode{ kNone };
+	std::atomic<int> error_code{ 0 };
+	std::atomic<int> packets_left{ 0 };
+	std::atomic<int> software_threads{ 0 };
+}
+
+namespace ffmpeg
+{
+	void HardwareFault::FailSend(int error, int after_packets)
+	{
+		error_code.store(error);
+		packets_left.store(after_packets);
+		mode.store(kSend);
+	}
+
+	void HardwareFault::FailDrain(int error)
+	{
+		error_code.store(error);
+		mode.store(kDrain);
+	}
+
+	void HardwareFault::Disarm()
+	{
+		mode.store(kNone);
+		software_threads.store(0);
+	}
+
+	void HardwareFault::NoteSoftwareDecoder(int threads)
+	{
+		software_threads.store(threads);
+	}
+
+	int HardwareFault::SoftwareThreads()
+	{
+		return software_threads.load();
+	}
+
+	int HardwareFault::NextSendError()
+	{
+		if (mode.load() != kSend) {
+			return 0;
+		}
+
+		if (packets_left.fetch_sub(1) > 0) {
+			return 0;
+		}
+
+		// Once only: the decoder that takes over must not meet it again.
+		mode.store(kNone);
+
+		return error_code.load();
+	}
+
+	bool HardwareFault::HoldsPictures()
+	{
+		return mode.load() == kDrain;
+	}
+
+	int HardwareFault::DrainError()
+	{
+		mode.store(kNone);
+
+		return error_code.load();
+	}
+}