From 8440877cad58364f9193cddd775a680a98b0e8ab Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Mon, 5 Oct 2026 16:38:04 +0200 Subject: [PATCH 1/2] fix: fall back from a failed hardware video decoder without losing or damaging pictures Several ways the media player's hardware decoder gives way to software did not hold up: - A stream the hardware does not take (a profile or a sampling it has no format for) was decoded by the context made for hardware, which has one thread. The decoder now notices, before the first picture, that it went for a software format, and decodes again in a context made for software, which has the cores of the machine. - A hardware decoder that failed while the stream was being drained handed its replacement neither the end of the stream nor the pictures it had queued, so the last pictures of a short stream were lost. The replacement is told the stream has ended, and the pictures of the packets it takes up again are returned at once rather than with a later call. - A packet that could not be decoded was taken for a failure of the hardware after its first picture, so one damaged packet gave up the hardware for good while software decoding stops with an error. It is now left to fail as it does there. Before the first picture the software decoder still takes over, since it decodes the same packets and reports the error if the stream is at fault. - After a failure past the first picture the replacement was sent packets whose reference pictures it does not have. H.264 decoders skip them, VP9 decoders fail on them and ended playback. It now waits for a key frame, within a bound for streams that do not mark them. "No hardware video decoder" is logged verbosely: it is the ordinary case of a codec without one, and went to stderr for every player. The tests make the hardware fail on purpose, which no GPU does on demand, through a small native fault the tests arm (HardwareFault, with a Java class in the tests only). They cover a failure before the first picture, bad data before and after it, a failure past it for H.264 and VP9, a failure while draining, and the thread count of the decoder that takes over. Each was seen to fail with its fix taken out. --- docs/guide/media/media-files.md | 2 +- .../src/main/cpp/include/JNI_HardwareFault.h | 60 ++++++ .../main/cpp/include/media/HardwareFault.h | 63 ++++++ .../src/main/cpp/include/media/VideoDecoder.h | 43 +++- .../src/main/cpp/src/JNI_HardwareFault.cpp | 51 +++++ .../src/main/cpp/src/media/HardwareFault.cpp | 94 +++++++++ .../src/main/cpp/src/media/VideoDecoder.cpp | 183 ++++++++++++++---- .../media/player/HardwareDecodingTest.java | 141 ++++++++++++++ .../webrtc/media/player/HardwareFault.java | 67 +++++++ 9 files changed, 660 insertions(+), 44 deletions(-) create mode 100644 webrtc-java-media/src/main/cpp/include/JNI_HardwareFault.h create mode 100644 webrtc-java-media/src/main/cpp/include/media/HardwareFault.h create mode 100644 webrtc-java-media/src/main/cpp/src/JNI_HardwareFault.cpp create mode 100644 webrtc-java-media/src/main/cpp/src/media/HardwareFault.cpp create mode 100644 webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareFault.java diff --git a/docs/guide/media/media-files.md b/docs/guide/media/media-files.md index a0dd35fd5..1c6fa501f 100644 --- a/docs/guide/media/media-files.md +++ b/docs/guide/media/media-files.md @@ -224,7 +224,7 @@ The pictures are the same ones software decoding gives; in the tests, every fram - **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux through NVDEC on NVIDIA GPUs (x86-64 and ARM64; the driver provides `libcuda.so.1` and `libnvcuvid.so.1`, which are loaded when a player asks for hardware). The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM, VA-API on Intel and AMD GPUs), the video is decoded in software, and `isHardwareDecoding()` says so. - **Codecs:** H.264 and VP9. VP8, MPEG-4, MJPEG and H.265/HEVC are decoded in software. -- **Fallback:** a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails after it has produced a picture, decoding goes on in software from the next key frame. +- **Fallback:** a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream. - **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it. ## Playing a Live Stream diff --git a/webrtc-java-media/src/main/cpp/include/JNI_HardwareFault.h b/webrtc-java-media/src/main/cpp/include/JNI_HardwareFault.h new file mode 100644 index 000000000..7a7f3aba0 --- /dev/null +++ b/webrtc-java-media/src/main/cpp/include/JNI_HardwareFault.h @@ -0,0 +1,60 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +/* Header for class dev_onvoid_webrtc_media_player_HardwareFault */ + +#ifndef _Included_dev_onvoid_webrtc_media_player_HardwareFault +#define _Included_dev_onvoid_webrtc_media_player_HardwareFault +#ifdef __cplusplus +extern "C" { +#endif + /* + * Class: dev_onvoid_webrtc_media_player_HardwareFault + * Method: failSend + * Signature: (ZI)V + */ + JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failSend + (JNIEnv *, jclass, jboolean, jint); + + /* + * Class: dev_onvoid_webrtc_media_player_HardwareFault + * Method: failDrain + * Signature: ()V + */ + JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failDrain + (JNIEnv *, jclass); + + /* + * Class: dev_onvoid_webrtc_media_player_HardwareFault + * Method: disarm + * Signature: ()V + */ + JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_disarm + (JNIEnv *, jclass); + + /* + * Class: dev_onvoid_webrtc_media_player_HardwareFault + * Method: softwareThreads + * Signature: ()I + */ + JNIEXPORT jint JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_softwareThreads + (JNIEnv *, jclass); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/webrtc-java-media/src/main/cpp/include/media/HardwareFault.h b/webrtc-java-media/src/main/cpp/include/media/HardwareFault.h new file mode 100644 index 000000000..85f0f17f5 --- /dev/null +++ b/webrtc-java-media/src/main/cpp/include/media/HardwareFault.h @@ -0,0 +1,63 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef WEBRTC_JAVA_MEDIA_HARDWARE_FAULT_H_ +#define WEBRTC_JAVA_MEDIA_HARDWARE_FAULT_H_ + +namespace ffmpeg +{ + // Makes the hardware video decoder fail on purpose, so that the fall back + // to software can be tested: a GPU cannot be made to fail on demand. + // + // Only the tests arm it, through JNI_HardwareFault, and a decoder that is + // not armed pays a single atomic load per packet. A fault applies to the + // hardware decoder only, never to the software one that takes over, and + // is armed for the next decoder that runs into it. + class HardwareFault + { + public: + // Fails the hardware decoder's send of the packet that follows the + // given number of them, once, with the given AVERROR. + static void FailSend(int error, int after_packets); + + // Makes the hardware decoder produce nothing, and fail with the + // given AVERROR once it is asked for pictures after it was told + // to drain: the failure of a decoder that never delivered. + static void FailDrain(int error); + + // Forgets any fault, and the software decoder noted last. + static void Disarm(); + + // Notes how many threads a software decoder of the video decoder + // got, for a test to see that one that took over from the hardware + // has all it can use. + static void NoteSoftwareDecoder(int threads); + + // The thread count noted last, or 0. + static int SoftwareThreads(); + + // The error to fail the next send with, or 0. + static int NextSendError(); + + // Whether the hardware decoder's pictures are held back. + static bool HoldsPictures(); + + // The error to fail the receive of a picture with once draining. + static int DrainError(); + }; +} + +#endif diff --git a/webrtc-java-media/src/main/cpp/include/media/VideoDecoder.h b/webrtc-java-media/src/main/cpp/include/media/VideoDecoder.h index 5c1a22a20..30d4c1dcd 100644 --- a/webrtc-java-media/src/main/cpp/include/media/VideoDecoder.h +++ b/webrtc-java-media/src/main/cpp/include/media/VideoDecoder.h @@ -91,6 +91,10 @@ namespace ffmpeg // frame, to be decoded again in software if it fails by then. static constexpr size_t kMaxKeptPackets = 32; + // How many packets may be passed over while waiting for a key + // frame, for a stream whose packets do not say which ones are. + static constexpr size_t kMaxSkippedPackets = 250; + // Opens the codec context of the stream, in hardware or not. int OpenContext(bool hardware); @@ -106,11 +110,27 @@ namespace ffmpeg // into system memory first. int ReceiveFromCodec(AVFrame ** frame, int64_t * timestamp_us); - // Opens a software decoder in place of the hardware one that - // failed with the given error, and decodes the packets kept - // since the start again. Returns the AVERROR the caller gets: - // EAGAIN once decoding goes on, or an error if it cannot. - int FallBackToSoftware(int error); + // Whether a result of the hardware decoder means that it failed. + // A stream that cannot be decoded, which is how corrupt data shows, + // fails a software decoder as well and says nothing of the + // hardware, except before its first picture, where the software + // decoder is the one to tell. + bool IsHardwareFailure(int result) const; + + // Opens a software decoder in place of the hardware one, which + // failed with the given error, or if that is 0 did not take the + // stream, and decodes the packets kept since the start again. + // The packet being sent when it happened, if any, is the current + // one. Returns 0, or the error that kept a decoder from opening. + int FallBackToSoftware(int error, const AVPacket * current); + + // Hands a packet, or null to drain, to the codec, taking its + // pictures aside while it asks for that. + void SendToSoftware(const AVPacket * packet); + + // Whether a packet is passed over because the decoder is waiting + // for a key frame, which it is after it lost pictures it needs. + bool SkipUntilKeyFrame(const AVPacket * packet); void KeepPacket(const AVPacket * packet); void DropKeptPackets(); @@ -153,6 +173,19 @@ namespace ffmpeg // sent to it are kept. bool probation_ = false; std::vector kept_; + + // Set by ChooseFormat when the decoder went for a software format. + bool software_selected_ = false; + + // Whether the end of the stream was announced, which a decoder that + // takes over has to be told as well. + bool draining_ = false; + + // After a software decoder took over from a hardware one that had + // delivered pictures, it has none of the ones the next refer to, + // and starts again at a key frame. + bool awaiting_key_frame_ = false; + size_t skipped_ = 0; std::deque pending_; }; } diff --git a/webrtc-java-media/src/main/cpp/src/JNI_HardwareFault.cpp b/webrtc-java-media/src/main/cpp/src/JNI_HardwareFault.cpp new file mode 100644 index 000000000..006ae3ac0 --- /dev/null +++ b/webrtc-java-media/src/main/cpp/src/JNI_HardwareFault.cpp @@ -0,0 +1,51 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "JNI_HardwareFault.h" +#include "media/HardwareFault.h" + +extern "C" { +#include +} + +// The Java class this belongs to exists in the tests only, so that nothing but +// them can reach the native side of a fault. + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failSend +(JNIEnv * env, jclass caller, jboolean invalidData, jint afterPackets) +{ + // Bad data is the stream's fault; anything else the hardware's. + ffmpeg::HardwareFault::FailSend(invalidData == JNI_TRUE ? AVERROR_INVALIDDATA : AVERROR_EXTERNAL, + afterPackets); +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_failDrain +(JNIEnv * env, jclass caller) +{ + ffmpeg::HardwareFault::FailDrain(AVERROR_EXTERNAL); +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_disarm +(JNIEnv * env, jclass caller) +{ + ffmpeg::HardwareFault::Disarm(); +} + +JNIEXPORT jint JNICALL Java_dev_onvoid_webrtc_media_player_HardwareFault_softwareThreads +(JNIEnv * env, jclass caller) +{ + return ffmpeg::HardwareFault::SoftwareThreads(); +} diff --git a/webrtc-java-media/src/main/cpp/src/media/HardwareFault.cpp b/webrtc-java-media/src/main/cpp/src/media/HardwareFault.cpp new file mode 100644 index 000000000..cecbfa608 --- /dev/null +++ b/webrtc-java-media/src/main/cpp/src/media/HardwareFault.cpp @@ -0,0 +1,94 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/HardwareFault.h" + +#include + +namespace +{ + enum Mode + { + kNone, + kSend, + kDrain + }; + + std::atomic mode{ kNone }; + std::atomic error_code{ 0 }; + std::atomic packets_left{ 0 }; + std::atomic software_threads{ 0 }; +} + +namespace ffmpeg +{ + void HardwareFault::FailSend(int error, int after_packets) + { + error_code.store(error); + packets_left.store(after_packets); + mode.store(kSend); + } + + void HardwareFault::FailDrain(int error) + { + error_code.store(error); + mode.store(kDrain); + } + + void HardwareFault::Disarm() + { + mode.store(kNone); + software_threads.store(0); + } + + void HardwareFault::NoteSoftwareDecoder(int threads) + { + software_threads.store(threads); + } + + int HardwareFault::SoftwareThreads() + { + return software_threads.load(); + } + + int HardwareFault::NextSendError() + { + if (mode.load() != kSend) { + return 0; + } + + if (packets_left.fetch_sub(1) > 0) { + return 0; + } + + // Once only: the decoder that takes over must not meet it again. + mode.store(kNone); + + return error_code.load(); + } + + bool HardwareFault::HoldsPictures() + { + return mode.load() == kDrain; + } + + int HardwareFault::DrainError() + { + mode.store(kNone); + + return error_code.load(); + } +} diff --git a/webrtc-java-media/src/main/cpp/src/media/VideoDecoder.cpp b/webrtc-java-media/src/main/cpp/src/media/VideoDecoder.cpp index 120976ae1..b91084fc5 100644 --- a/webrtc-java-media/src/main/cpp/src/media/VideoDecoder.cpp +++ b/webrtc-java-media/src/main/cpp/src/media/VideoDecoder.cpp @@ -15,6 +15,7 @@ */ #include "media/VideoDecoder.h" +#include "media/HardwareFault.h" extern "C" { #include @@ -82,7 +83,9 @@ namespace ffmpeg int result = OpenContext(hardware); if (result < 0 && hardware) { - av_log(nullptr, AV_LOG_INFO, "No hardware video decoder (%s), decoding in software\n", + // Verbose: a codec with no hardware decoder is nothing to report, + // and IsHardware tells a caller that wants to know. + av_log(nullptr, AV_LOG_VERBOSE, "No hardware video decoder (%s), decoding in software\n", ErrorString(result).c_str()); result = OpenContext(false); @@ -113,6 +116,8 @@ namespace ffmpeg int VideoDecoder::OpenContext(bool hardware) { + software_selected_ = false; + const AVCodec * codec = avcodec_find_decoder(stream_->codecpar->codec_id); if (codec == nullptr) { @@ -165,6 +170,10 @@ namespace ffmpeg hardware_ = hardware; + if (!hardware) { + HardwareFault::NoteSoftwareDecoder(codec_context_->thread_count); + } + return 0; } @@ -203,7 +212,7 @@ namespace ffmpeg AVPixelFormat VideoDecoder::ChooseFormat(AVCodecContext * context, const AVPixelFormat * formats) { - const VideoDecoder * decoder = static_cast(context->opaque); + VideoDecoder * decoder = static_cast(context->opaque); for (const AVPixelFormat * format = formats; *format != AV_PIX_FMT_NONE; format++) { if (*format == decoder->hardware_format_) { @@ -212,7 +221,11 @@ namespace ffmpeg } // The decoder does not offer the hardware format for this stream: - // a profile or a size the hardware does not take. Software it is. + // a profile or a size the hardware does not take. Software it is, and + // the context made for hardware, which has one thread, is replaced by + // one made for software. + decoder->software_selected_ = true; + return avcodec_default_get_format(context, formats); } @@ -248,6 +261,10 @@ namespace ffmpeg hardware_ = false; hardware_format_ = AV_PIX_FMT_NONE; probation_ = false; + software_selected_ = false; + draining_ = false; + awaiting_key_frame_ = false; + skipped_ = 0; time_base_ = { 0, 1 }; } @@ -257,9 +274,34 @@ namespace ffmpeg avcodec_flush_buffers(codec_context_); } - // What was kept belongs to the position that was left. + // What was kept belongs to the position that was left, and a seek + // lands on a key frame anyway. DropKeptPackets(); DropPending(); + + draining_ = false; + awaiting_key_frame_ = false; + skipped_ = 0; + } + + bool VideoDecoder::IsHardwareFailure(int result) const + { + return IsFailure(result) && (probation_ || result != AVERROR_INVALIDDATA); + } + + bool VideoDecoder::SkipUntilKeyFrame(const AVPacket * packet) + { + if (!awaiting_key_frame_) { + return false; + } + + if ((packet->flags & AV_PKT_FLAG_KEY) != 0 || ++skipped_ > kMaxSkippedPackets) { + awaiting_key_frame_ = false; + + return false; + } + + return true; } int VideoDecoder::SendPacket(const AVPacket * packet) @@ -268,18 +310,29 @@ namespace ffmpeg return AVERROR(EINVAL); } + if (packet == nullptr) { + draining_ = true; + } + else if (SkipUntilKeyFrame(packet)) { + return 0; + } + if (probation_ && packet != nullptr) { KeepPacket(packet); } - int result = avcodec_send_packet(codec_context_, packet); + int result = hardware_ ? HardwareFault::NextSendError() : 0; - if (hardware_ && IsFailure(result)) { - result = FallBackToSoftware(result); + if (result == 0) { + result = avcodec_send_packet(codec_context_, packet); + } - if (packet == nullptr && result == AVERROR(EAGAIN)) { - // The drain the caller started was the hardware decoder's. - result = avcodec_send_packet(codec_context_, nullptr); + if (hardware_) { + if (probation_ && software_selected_) { + return FallBackToSoftware(0, packet); + } + if (IsHardwareFailure(result)) { + return FallBackToSoftware(result, packet); } } @@ -303,10 +356,14 @@ namespace ffmpeg int result = ReceiveFromCodec(frame, timestamp_us); - if (hardware_ && IsFailure(result)) { - // Decoding goes on in software; the pictures of the packets it - // takes up again come with the calls that follow. - return FallBackToSoftware(result); + const bool software_chosen = hardware_ && probation_ && software_selected_; + + if (software_chosen || (hardware_ && IsHardwareFailure(result))) { + // Decoding goes on in software, and the pictures of the packets it + // takes up again are the next ones. + int error = FallBackToSoftware(software_chosen ? 0 : result, nullptr); + + return error < 0 ? error : ReceiveFrame(frame, timestamp_us); } return result; @@ -314,8 +371,27 @@ namespace ffmpeg int VideoDecoder::ReceiveFromCodec(AVFrame ** frame, int64_t * timestamp_us) { + if (hardware_ && HardwareFault::HoldsPictures()) { + while (avcodec_receive_frame(codec_context_, decoded_) >= 0) { + av_frame_unref(decoded_); + } + + return draining_ ? HardwareFault::DrainError() : AVERROR(EAGAIN); + } + int result = avcodec_receive_frame(codec_context_, decoded_); + if (hardware_ && probation_ && software_selected_) { + // The picture, if there is one, is of no use: ReceiveFrame replaces + // this decoder, which has the one thread of a hardware decoder, + // and decodes the packets again. + if (result >= 0) { + av_frame_unref(decoded_); + } + + return AVERROR(EAGAIN); + } + if (result < 0) { return result; } @@ -395,14 +471,17 @@ namespace ffmpeg av_frame_unref(decoded_); } - if (result >= 0 && probation_) { - // The first picture is out: either the hardware decoded it, or the - // decoder offered no hardware format and it was software that did. - probation_ = false; + if (result >= 0) { + if (probation_) { + // The first picture is out, and the hardware decoded it. + probation_ = false; - DropKeptPackets(); + DropKeptPackets(); + } if (!hardware_frame) { + // The decoder went for a software format part way through, for + // a size or a profile the hardware does not take. hardware_ = false; } } @@ -410,10 +489,15 @@ namespace ffmpeg return result; } - int VideoDecoder::FallBackToSoftware(int error) + int VideoDecoder::FallBackToSoftware(int error, const AVPacket * current) { - av_log(nullptr, AV_LOG_WARNING, "Hardware video decoding failed (%s), decoding in software\n", - ErrorString(error).c_str()); + if (error < 0) { + av_log(nullptr, AV_LOG_WARNING, "Hardware video decoding failed (%s), decoding in software\n", + ErrorString(error).c_str()); + } + else { + av_log(nullptr, AV_LOG_VERBOSE, "The hardware does not take the stream, decoding in software\n"); + } avcodec_free_context(&codec_context_); @@ -428,35 +512,58 @@ namespace ffmpeg return result; } - // Nothing has come out of the hardware yet: the packets it was sent - // are decoded again, so that none of them is lost. After a picture - // the stream goes on from its next key frame. + // Nothing has come out of the hardware yet: the packets it was sent, + // the current one among them, are decoded again, so that none of them + // is lost. + const bool replay = probation_; + std::vector kept; kept.swap(kept_); probation_ = false; for (AVPacket * packet : kept) { - int sent = avcodec_send_packet(codec_context_, packet); - - while (sent == AVERROR(EAGAIN)) { - // The decoder wants its pictures taken before it takes more. - AVFrame * frame = nullptr; - int64_t timestamp_us = 0; + SendToSoftware(packet); - if (ReceiveFromCodec(&frame, ×tamp_us) < 0) { - break; - } + av_packet_free(&packet); + } - pending_.push_back({ frame, timestamp_us }); + if (!replay) { + // After a picture the new decoder has none of the ones that the + // next refer to, so the stream goes on from its next key frame, + // the current packet included. + awaiting_key_frame_ = true; + skipped_ = 0; - sent = avcodec_send_packet(codec_context_, packet); + if (current != nullptr && !SkipUntilKeyFrame(current)) { + SendToSoftware(current); } + } - av_packet_free(&packet); + if (draining_) { + SendToSoftware(nullptr); } - return AVERROR(EAGAIN); + return 0; + } + + void VideoDecoder::SendToSoftware(const AVPacket * packet) + { + int sent = avcodec_send_packet(codec_context_, packet); + + while (sent == AVERROR(EAGAIN)) { + // The decoder wants its pictures taken before it takes more. + AVFrame * frame = nullptr; + int64_t timestamp_us = 0; + + if (ReceiveFromCodec(&frame, ×tamp_us) < 0) { + break; + } + + pending_.push_back({ frame, timestamp_us }); + + sent = avcodec_send_packet(codec_context_, packet); + } } void VideoDecoder::KeepPacket(const AVPacket * packet) diff --git a/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java index 807fc0532..3a29fc042 100644 --- a/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java +++ b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java @@ -18,6 +18,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assumptions.assumeTrue; @@ -44,6 +45,7 @@ import dev.onvoid.webrtc.media.video.VideoTrackSink; import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestInstance; @@ -183,6 +185,135 @@ void fileSourceTakesTheOption() throws Exception { } } + @AfterEach + void disarmFaults() { + HardwareFault.disarm(); + } + + @Test + void failureBeforeTheFirstPictureIsDecodedAgain() throws Exception { + requireHardwareDecoder(); + + Result software = play(H264, false, 0); + + HardwareFault.failSend(false, 0); + + Result hardware = play(H264, true, 0); + + // The packets the hardware was sent are decoded again in software, so + // nothing is lost. + assertNull(hardware.error); + assertFalse(hardware.hardware); + assertEquals(software.frames, hardware.frames); + } + + @Test + void badDataBeforeTheFirstPictureIsLeftToSoftware() throws Exception { + requireHardwareDecoder(); + + Result software = play(H264, false, 0); + + // Before it has delivered anything, a decoder that fails is not told + // from the stream failing: software decodes the same packets, and says + // if it is the stream. + HardwareFault.failSend(true, 0); + + Result hardware = play(H264, true, 0); + + assertNull(hardware.error); + assertFalse(hardware.hardware); + assertEquals(software.frames, hardware.frames); + } + + @Test + void failureAfterPicturesGoesOnAtTheNextKeyFrame() throws Exception { + assertGoesOnAtTheNextKeyFrame(H264); + } + + @Test + void failureAfterPicturesGoesOnAtTheNextKeyFrameVp9() throws Exception { + // Where H.264 decoders skip what they cannot decode, a VP9 decoder + // fails on a picture whose references it does not have, so the new + // decoder must not be sent any before the key frame. + assertGoesOnAtTheNextKeyFrame(VP9); + } + + private void assertGoesOnAtTheNextKeyFrame(String asset) throws Exception { + requireHardwareDecoder(); + + Result software = play(asset, false, 0); + + // Five packets in, the second of the two key frames is ten packets away. + HardwareFault.failSend(false, 5); + + Result hardware = play(asset, true, 0); + + assertNull(hardware.error); + assertFalse(hardware.hardware); + + // What the hardware delivered, then the second second of the media + // from its key frame, as software decodes it. Pictures that need ones + // lost with the hardware decoder are not delivered at all. + int second = FRAMES / 2; + int before = hardware.frames.size() - second; + + assertTrue(before > 0 && before < second, "frames: " + hardware.frames.size()); + assertEquals(software.frames.subList(0, before), hardware.frames.subList(0, before)); + assertEquals(software.frames.subList(second, FRAMES), hardware.frames.subList(before, hardware.frames.size())); + } + + @Test + void badDataAfterPicturesIsNotAHardwareFailure() throws Exception { + requireHardwareDecoder(); + + HardwareFault.failSend(true, 5); + + Result hardware = play(H264, true, 0); + + // As with software decoding, playback stops with the error, and the + // hardware keeps the stream: a damaged packet says nothing of it. + assertNotNull(hardware.error); + assertTrue(hardware.hardware, "the player gave up the hardware over bad data"); + } + + @Test + void failureWhileDrainingKeepsThePictures() throws Exception { + requireHardwareDecoder(); + + Result software = play(H264, false, 0); + + // The hardware delivers nothing until the end of the stream, where it + // fails: every picture comes from the software decoder that takes over + // and is told that the stream has ended. + HardwareFault.failDrain(); + + Result hardware = play(H264, true, 0); + + assertNull(hardware.error); + assertFalse(hardware.hardware); + assertEquals(FRAMES, hardware.frames.size()); + assertEquals(software.frames, hardware.frames); + } + + @Test + void softwareThatTakesOverUsesEveryCore() throws Exception { + assumeTrue(Runtime.getRuntime().availableProcessors() > 1, "a single core"); + + requireHardwareDecoder(); + + // VP9 in 4:4:4: the hardware decoder offers no format for it, and the + // stream is decoded in software. The decoder made for the hardware has + // one thread, so it is replaced by one that has the cores of the + // machine. + HardwareFault.disarm(); + + Result hardware = play(VP9_444, true, 0); + + assertNull(hardware.error); + assertFalse(hardware.hardware); + assertTrue(HardwareFault.softwareThreads() > 1, "threads: " + HardwareFault.softwareThreads()); + } + @Test void seeksInHardware() throws Exception { Result software = play(H264, false, 1_000_000); @@ -513,6 +644,16 @@ private static void requireHardware(Result result) { requireHardware(result.hardware); } + /** + * Skips the test where this machine has no hardware decoder to make fail, + * and fails it where one is required. + */ + private void requireHardwareDecoder() throws Exception { + try (MediaFileSource source = new MediaFileSource(path(H264), true)) { + requireHardware(source.getPlayer().isHardwareDecoding()); + } + } + private static void requireHardware(boolean hardware) { if (HARDWARE_REQUIRED) { assertTrue(hardware, "the player did not decode in hardware"); diff --git a/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareFault.java b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareFault.java new file mode 100644 index 000000000..b1b086a1d --- /dev/null +++ b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareFault.java @@ -0,0 +1,67 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.media.player; + +/** + * Makes the hardware video decoder of the native library fail on purpose, so + * that the tests can see what a player does when a GPU fails, which no GPU + * does on demand. It lives in the tests so that nothing else can reach it. + *

+ * A fault is armed for the next hardware decoder that runs into it, fires + * once, and has to be {@link #disarm() disarmed} when a test is done. + * + * @author Alex Andres + */ +final class HardwareFault { + + static { + FFmpeg.load(); + } + + + private HardwareFault() { + // Static access only. + } + + /** + * Fails the send of the packet that follows the given number of them to + * the hardware decoder. + * + * @param invalidData True to fail as bad data does, which says nothing + * of the hardware, false to fail as the hardware does. + * @param afterPackets How many packets are sent before the one that fails. + */ + static native void failSend(boolean invalidData, int afterPackets); + + /** + * Keeps the hardware decoder from delivering any picture, and fails it + * when it is asked for them at the end of the stream. + */ + static native void failDrain(); + + /** + * Forgets any fault, and the thread count of {@link #softwareThreads()}. + */ + static native void disarm(); + + /** + * Returns the number of threads the last software video decoder to be + * opened got, or 0 if there was none since the last {@link #disarm()}. + */ + static native int softwareThreads(); + +} From 4c8de5dfdb83ceb8872c9ef5b882788729036d7c Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Mon, 5 Oct 2026 19:43:36 +0200 Subject: [PATCH 2/2] test: require that the hardware decoder decodes before making it fail The fault tests took a hardware decoder that opens for one that decodes. On the macOS runners, which are virtual machines, VideoToolbox opens a device and then cannot set up the decoder ("hwaccel initialisation returned error"), so the stream is decoded in software from its first packet and a fault meant for the hardware never arrives. The tests that fail the hardware after its first pictures then saw no failure and no error. A stream is now played once in hardware before the first of them runs, and the tests are skipped, or fail where hardware is required, unless the hardware delivered its pictures. The test of the software decoder's threads needs only the hardware decoder to be set up, since it is about the decoder that replaces it, and keeps that check. --- .../media/player/HardwareDecodingTest.java | 25 +++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java index 3a29fc042..9dd527738 100644 --- a/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java +++ b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/player/HardwareDecodingTest.java @@ -117,6 +117,9 @@ class HardwareDecodingTest { private AudioDeviceModule audioModule; private PeerConnectionFactory factory; + /** Whether the hardware decoder of this machine decodes, once it is known. */ + private Boolean hardwareDecodes; + @BeforeAll void initFactory() { @@ -299,7 +302,7 @@ void failureWhileDrainingKeepsThePictures() throws Exception { void softwareThatTakesOverUsesEveryCore() throws Exception { assumeTrue(Runtime.getRuntime().availableProcessors() > 1, "a single core"); - requireHardwareDecoder(); + requireHardwareContext(); // VP9 in 4:4:4: the hardware decoder offers no format for it, and the // stream is decoded in software. The decoder made for the hardware has @@ -645,10 +648,28 @@ private static void requireHardware(Result result) { } /** - * Skips the test where this machine has no hardware decoder to make fail, + * Skips the test where this machine has no hardware decoder that decodes, * and fails it where one is required. + *

+ * A decoder that opens is not one that decodes: a virtual machine, as CI + * runs on, can have the device and no engine behind it, and the stream is + * decoded in software from its first packet, where a fault meant for the + * hardware never arrives. So a stream is played first, once, and the + * hardware has to have delivered its pictures. */ private void requireHardwareDecoder() throws Exception { + if (hardwareDecodes == null) { + hardwareDecodes = play(H264, true, 0).hardware; + } + + requireHardware(hardwareDecodes); + } + + /** + * Skips the test where this machine has no hardware decoder to set up, + * and fails it where one is required. + */ + private void requireHardwareContext() throws Exception { try (MediaFileSource source = new MediaFileSource(path(H264), true)) { requireHardware(source.getPlayer().isHardwareDecoding()); }