diff --git a/docs/guide/advanced/encoded-transforms.md b/docs/guide/advanced/encoded-transforms.md index 507cae98..d78300b2 100644 --- a/docs/guide/advanced/encoded-transforms.md +++ b/docs/guide/advanced/encoded-transforms.md @@ -9,6 +9,7 @@ This guide explains how to read and change the encoded frames of a call as they - Asking for key frames - End-to-end encryption as a worked example - Threading and performance +- Sending frames that are already encoded through an injector A sender's transform sees every frame after the encoder and before the packetizer; a receiver's sees every frame after the depacketizer and before the decoder. Everything between the two, whether that is the network, a TURN server or an SFU, only ever carries what the transform produced. That is what makes end-to-end encryption possible, and also frame metadata, watermarks, or analysis of the encoded stream. @@ -104,3 +105,33 @@ The transform never runs on a thread that carries media. Each sender and receive - a transform that falls several seconds behind has frames dropped until it catches up, rather than having them queue up in memory. Per frame, the cost is one small Java object and, only if the payload is read, one copy into a buffer that is reused from frame to frame. Senders and receivers without a transform are not touched at all. + +## Injecting Encoded Frames + +A sender can also send frames that are already encoded, such as a camera's H.264 stream or a recording, without WebRTC encoding anything. An injector takes the place of the sender's track, so it is created for a sender without one, for example from a transceiver added by media type: + +```java +RTCRtpTransceiverInit init = new RTCRtpTransceiverInit(); +init.direction = RTCRtpTransceiverDirection.SEND_ONLY; + +RTCRtpTransceiver transceiver = peerConnection.addTransceiver(MediaType.VIDEO, init); +RTCEncodedVideoFrameInjector injector = transceiver.getSender() + .createEncodedVideoFrameInjector(new RTCEncodedVideoFrameInjectorObserver() { + @Override + public void onKeyFrameRequest() { + // Inject a key frame next. + } + + @Override + public void onBitrateChange(int allocatedBitrate, int availableOutgoingBitrate) { + // Adapt the source's bitrate. + } + }); + +// After negotiation, for every frame: +injector.injectFrame(frameData, rtpTimestamp, keyFrame, width, height); +``` + +The frames must be in the codec the sender negotiated, which `setCodecPreferences` on the transceiver can make sure of, and the RTP timestamp runs on the 90 kHz video clock. An audio sender takes an `RTCEncodedAudioFrameInjector` the same way, whose frames carry the payload type of their codec, from the sender's `getParameters().codecs`. Do not inject audio through a factory that sends audio captured by its audio device module, as WebRTC feeds captured audio to every audio sender. + +Giving the sender a track through `replaceTrack` ends the injection, and `dispose()` releases the injector. The observer runs on a callback thread shared with other WebRTC observers, so it may call back into WebRTC but should not block. diff --git a/docs/guide/data/data-channels.md b/docs/guide/data/data-channels.md index e39b5ff4..614c5972 100644 --- a/docs/guide/data/data-channels.md +++ b/docs/guide/data/data-channels.md @@ -203,7 +203,7 @@ dataChannel.sendAsync(binaryChannelBuffer, new RTCDataChannelSendObserver() { }); ``` -The observer is called exactly once, with a failure if the operation is discarded while the channel shuts down. Success means the message was queued locally, not that the peer received it. Callbacks run on a native callback thread shared with other send and DTLS transport observers; they may call other WebRTC methods, but should not block, so dispatch long work to an executor of your own. Without an observer, `sendAsync` logs failures and reports nothing to the caller. +The observer is called exactly once, with a failure if the operation is discarded while the channel shuts down. Success means the message was queued locally, not that the peer received it. Callbacks run on a native callback thread shared with other WebRTC observers; they may call other WebRTC methods, but should not block, so dispatch long work to an executor of your own. Without an observer, `sendAsync` logs failures and reports nothing to the caller. ### Receiving Data diff --git a/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt b/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt index 543bd764..467693a5 100644 --- a/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt +++ b/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt @@ -171,13 +171,29 @@ elseif(LINUX) endif() endif() +# Targets outside the default build graph whose objects the JNI code needs, +# built and linked explicitly: the api/video utility classes +# (AdaptedVideoTrackSource, VideoAdapter, VideoBroadcaster), which fell out of +# it when they moved out of media/base, and the frame factory that encoded +# frame injectors are fed through. On Linux and Windows the shared library link +# would leave their symbols undefined and loading would fail at runtime. The +# Apple archive is assembled from an object glob and only needs them built. +set(WEBRTC_EXTRA_TARGETS + api/video:adapted_video_track_source + api/video:video_adapter + api/video:video_broadcaster + api:frame_transformer_factory +) + # Written next to a compiled WebRTC, in the build tree and in the install -# directory, naming the GN args it was built with. A build without it, or with -# other args, is not used and WebRTC is built again: it would lack what the -# args enable (H.265 for one), or have defines, and with them struct layouts, -# that differ from those the JNI code is compiled with below. Changing an arg -# here thus rebuilds WebRTC on its own; CI's cache key covers this file. +# directory, naming the GN args and the extra targets it was built with. A +# build without it, or with other args or targets, is not used and WebRTC is +# built again: it would lack what the args enable (H.265 for one) or a target's +# library, or have defines, and with them struct layouts, that differ from those +# the JNI code is compiled with below. Changing an arg or a target here thus +# rebuilds WebRTC on its own; CI's cache key covers this file. set(WEBRTC_ARGS_FILE "webrtc-java-gn-args.txt") +set(WEBRTC_BUILD_RECORD "${COMPILE_ARGS} extra_targets=\"${WEBRTC_EXTRA_TARGETS}\"") function(webrtc_args_match DIRECTORY RESULT) set(MATCHES FALSE) @@ -185,7 +201,7 @@ function(webrtc_args_match DIRECTORY RESULT) if(EXISTS "${DIRECTORY}/${WEBRTC_ARGS_FILE}") file(READ "${DIRECTORY}/${WEBRTC_ARGS_FILE}" RECORDED_ARGS) - if(RECORDED_ARGS STREQUAL COMPILE_ARGS) + if(RECORDED_ARGS STREQUAL WEBRTC_BUILD_RECORD) set(MATCHES TRUE) endif() endif() @@ -199,23 +215,22 @@ webrtc_args_match("${WEBRTC_SRC}/${WEBRTC_BUILD}" WEBRTC_BUILT_ARGS_MATCH) if(EXISTS "${WEBRTC_LIB_PATH_INSTALLED}" AND NOT WEBRTC_INSTALLED_ARGS_MATCH) # Out of the way, so that nothing links it before the new build replaces # it on install. - message(STATUS "WebRTC: the installed build has other GN args, building again") + message(STATUS "WebRTC: the installed build has other GN args or targets, building again") file(REMOVE "${WEBRTC_LIB_PATH_INSTALLED}") endif() +# The libraries of the extra targets, re-archived flat under obj/ by the build +# below. +set(EXTRA_LIB_NAMES "") +foreach(EXTRA_TARGET ${WEBRTC_EXTRA_TARGETS}) + string(REGEX REPLACE "^.*:" "" EXTRA_TARGET_NAME "${EXTRA_TARGET}") -# The api/video utility classes (AdaptedVideoTrackSource, VideoAdapter, -# VideoBroadcaster) fell out of the default build graph when they moved out -# of media/base, so their objects no longer reach the monolithic archive. -# On Linux and Windows the shared library link leaves their symbols -# undefined and loading fails at runtime; they are built and linked -# explicitly. The Apple archive is assembled from an object glob and only -# needs them built. -if(WIN32) - set(VIDEO_UTIL_LIB_NAMES adapted_video_track_source.lib video_adapter.lib video_broadcaster.lib) -else() - set(VIDEO_UTIL_LIB_NAMES libadapted_video_track_source.a libvideo_adapter.a libvideo_broadcaster.a) -endif() + if(WIN32) + list(APPEND EXTRA_LIB_NAMES "${EXTRA_TARGET_NAME}.lib") + else() + list(APPEND EXTRA_LIB_NAMES "lib${EXTRA_TARGET_NAME}.a") + endif() +endforeach() if(EXISTS "${WEBRTC_LIB_PATH_INSTALLED}") set(TARGET_INC_DIR ${WEBRTC_INSTALL_DIR}/include) @@ -224,7 +239,7 @@ if(EXISTS "${WEBRTC_LIB_PATH_INSTALLED}") set(TARGET_LIBCPP_BUILDTOOLS_INC_DIR ${TARGET_INC_DIR}/third_party/libc++/) set(TARGET_LIBCPP_INC_DIR ${TARGET_INC_DIR}/third_party/libc++/include) set(TARGET_LIBCPP_ABI_INC_DIR ${TARGET_INC_DIR}/third_party/libc++abi/include) - set(VIDEO_UTIL_LIB_DIR ${WEBRTC_INSTALL_DIR}/lib) + set(EXTRA_LIB_DIR ${WEBRTC_INSTALL_DIR}/lib) else() set(TARGET_INC_DIR ${WEBRTC_SRC}) set(TARGET_LINK_LIB ${WEBRTC_LIB_PATH}) @@ -234,13 +249,13 @@ else() set(TARGET_LIBCPP_ABI_INC_DIR ${WEBRTC_SRC}/third_party/libc++abi/src/include) # The flat re-archives; gn's originals are thin archives that break # when copied away from their object files. - set(VIDEO_UTIL_LIB_DIR ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj) + set(EXTRA_LIB_DIR ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj) endif() -set(VIDEO_UTIL_LINK_LIBS "") +set(EXTRA_LINK_LIBS "") if(NOT APPLE) - foreach(VIDEO_UTIL_LIB ${VIDEO_UTIL_LIB_NAMES}) - list(APPEND VIDEO_UTIL_LINK_LIBS "${VIDEO_UTIL_LIB_DIR}/${VIDEO_UTIL_LIB}") + foreach(EXTRA_LIB ${EXTRA_LIB_NAMES}) + list(APPEND EXTRA_LINK_LIBS "${EXTRA_LIB_DIR}/${EXTRA_LIB}") endforeach() endif() @@ -268,7 +283,7 @@ target_include_directories(${PROJECT_NAME} ${TARGET_INC_DIR}/third_party/abseil-cpp ${TARGET_INC_DIR}/third_party/libyuv/include ) -target_link_libraries(${PROJECT_NAME} ${TARGET_LINK_LIB} ${VIDEO_UTIL_LINK_LIBS}) +target_link_libraries(${PROJECT_NAME} ${TARGET_LINK_LIB} ${EXTRA_LINK_LIBS}) # rtc_use_h265=true makes gn define RTC_ENABLE_H265 for WebRTC; the JNI code # needs it as well, for the H.265 parts of WebRTC's headers it uses (the H.265 @@ -460,7 +475,7 @@ message(STATUS "WebRTC: compile") # output directory that Siso built. if(APPLE) execute_command( - COMMAND autoninja -C "${WEBRTC_BUILD}" :default api/audio_codecs:builtin_audio_decoder_factory api/task_queue:default_task_queue_factory sdk:native_api sdk:default_codec_factory_objc pc:peer_connection sdk:videocapture_objc api/video:adapted_video_track_source api/video:video_adapter api/video:video_broadcaster + COMMAND autoninja -C "${WEBRTC_BUILD}" :default api/audio_codecs:builtin_audio_decoder_factory api/task_queue:default_task_queue_factory sdk:native_api sdk:default_codec_factory_objc pc:peer_connection sdk:videocapture_objc ${WEBRTC_EXTRA_TARGETS} WORKING_DIRECTORY "${WEBRTC_SRC}" RESULT_VARIABLE NINJA_RESULT ) @@ -478,11 +493,10 @@ else() RESULT_VARIABLE NINJA_RESULT ) - # The api/video utility targets are not reachable from the default - # graph since their move out of media/base; build them explicitly so - # their symbols exist for the shared library link. + # The extra targets are not reachable from the default graph; build them + # explicitly so their symbols exist for the shared library link. execute_command( - COMMAND autoninja -C "${WEBRTC_BUILD}" api/video:adapted_video_track_source api/video:video_adapter api/video:video_broadcaster + COMMAND autoninja -C "${WEBRTC_BUILD}" ${WEBRTC_EXTRA_TARGETS} WORKING_DIRECTORY "${WEBRTC_SRC}" RESULT_VARIABLE NINJA_UTIL_RESULT ) @@ -494,12 +508,14 @@ if(WIN32) # without the object tree; a cold build never notices because the # objects still sit beside the archives. Re-archive them in regular # form with the toolchain librarian, mirroring the Linux path below. - foreach(VIDEO_UTIL_TARGET adapted_video_track_source video_adapter video_broadcaster) - file(GLOB_RECURSE VIDEO_UTIL_OBJS - ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/api/video/${VIDEO_UTIL_TARGET}/*.obj + foreach(EXTRA_TARGET ${WEBRTC_EXTRA_TARGETS}) + string(REPLACE ":" "/" EXTRA_TARGET_DIR "${EXTRA_TARGET}") + string(REGEX REPLACE "^.*:" "" EXTRA_TARGET_NAME "${EXTRA_TARGET}") + file(GLOB_RECURSE EXTRA_TARGET_OBJS + ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${EXTRA_TARGET_DIR}/*.obj ) execute_command( - COMMAND ${WEBRTC_SRC}/third_party/llvm-build/Release+Asserts/bin/lld-link.exe /lib /OUT:${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${VIDEO_UTIL_TARGET}.lib ${VIDEO_UTIL_OBJS} + COMMAND ${WEBRTC_SRC}/third_party/llvm-build/Release+Asserts/bin/lld-link.exe /lib /OUT:${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${EXTRA_TARGET_NAME}.lib ${EXTRA_TARGET_OBJS} ) endforeach() endif() @@ -516,13 +532,15 @@ if(LINUX) # gn emits thin archives whose members reference objects by relative # path, so they break once copied to the install tree. Re-archive the - # api/video utility objects into regular archives, like libc++ below. - foreach(VIDEO_UTIL_TARGET adapted_video_track_source video_adapter video_broadcaster) - file(GLOB_RECURSE VIDEO_UTIL_OBJS - ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/api/video/${VIDEO_UTIL_TARGET}/*.o + # extra targets' objects into regular archives, like libc++ below. + foreach(EXTRA_TARGET ${WEBRTC_EXTRA_TARGETS}) + string(REPLACE ":" "/" EXTRA_TARGET_DIR "${EXTRA_TARGET}") + string(REGEX REPLACE "^.*:" "" EXTRA_TARGET_NAME "${EXTRA_TARGET}") + file(GLOB_RECURSE EXTRA_TARGET_OBJS + ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${EXTRA_TARGET_DIR}/*.o ) execute_command( - COMMAND ${CMAKE_AR} rcs ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/lib${VIDEO_UTIL_TARGET}.a ${VIDEO_UTIL_OBJS} + COMMAND ${CMAKE_AR} rcs ${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/lib${EXTRA_TARGET_NAME}.a ${EXTRA_TARGET_OBJS} ) endforeach() @@ -564,16 +582,16 @@ if(LINUX) endif() if(LINUX) - foreach(VIDEO_UTIL_LIB ${VIDEO_UTIL_LIB_NAMES}) + foreach(EXTRA_LIB ${EXTRA_LIB_NAMES}) install( - FILES "${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${VIDEO_UTIL_LIB}" + FILES "${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${EXTRA_LIB}" DESTINATION "${WEBRTC_INSTALL_DIR}/lib" ) endforeach() elseif(WIN32) - foreach(VIDEO_UTIL_LIB ${VIDEO_UTIL_LIB_NAMES}) + foreach(EXTRA_LIB ${EXTRA_LIB_NAMES}) install( - FILES "${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${VIDEO_UTIL_LIB}" + FILES "${WEBRTC_SRC}/${WEBRTC_BUILD}/obj/${EXTRA_LIB}" DESTINATION "${WEBRTC_INSTALL_DIR}/lib" ) endforeach() @@ -582,7 +600,7 @@ endif() # The record of the GN args, once the build has succeeded, in the build tree # and, with the library, in the install directory. if(GN_RESULT EQUAL 0 AND NINJA_RESULT EQUAL 0 AND NINJA_UTIL_RESULT EQUAL 0 AND EXISTS "${WEBRTC_LIB_PATH}") - file(WRITE "${WEBRTC_SRC}/${WEBRTC_BUILD}/${WEBRTC_ARGS_FILE}" "${COMPILE_ARGS}") + file(WRITE "${WEBRTC_SRC}/${WEBRTC_BUILD}/${WEBRTC_ARGS_FILE}" "${WEBRTC_BUILD_RECORD}") install(FILES "${WEBRTC_SRC}/${WEBRTC_BUILD}/${WEBRTC_ARGS_FILE}" DESTINATION "${WEBRTC_INSTALL_DIR}") else() diff --git a/webrtc-jni/src/main/cpp/include/JNI_RTCEncodedAudioFrameInjector.h b/webrtc-jni/src/main/cpp/include/JNI_RTCEncodedAudioFrameInjector.h new file mode 100644 index 00000000..6063ac53 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/JNI_RTCEncodedAudioFrameInjector.h @@ -0,0 +1,36 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#ifndef _Included_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector +#define _Included_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector +#ifdef __cplusplus +extern "C" { +#endif +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector_injectDirectBuffer + (JNIEnv *, jobject, jobject, jint, jint, jint, jlong, jint, jboolean, jlong, jboolean); + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector_injectByteArray + (JNIEnv *, jobject, jbyteArray, jint, jlong, jint, jboolean, jlong, jboolean); + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector_dispose + (JNIEnv *, jobject); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/webrtc-jni/src/main/cpp/include/JNI_RTCEncodedVideoFrameInjector.h b/webrtc-jni/src/main/cpp/include/JNI_RTCEncodedVideoFrameInjector.h new file mode 100644 index 00000000..81a40c89 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/JNI_RTCEncodedVideoFrameInjector.h @@ -0,0 +1,36 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#ifndef _Included_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector +#define _Included_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector +#ifdef __cplusplus +extern "C" { +#endif +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector_injectDirectBuffer + (JNIEnv *, jobject, jobject, jint, jint, jlong, jboolean, jint, jint, jlong, jboolean); + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector_injectByteArray + (JNIEnv *, jobject, jbyteArray, jlong, jboolean, jint, jint, jlong, jboolean); + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector_dispose + (JNIEnv *, jobject); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h b/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h index a6d626da..0e0feee7 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h +++ b/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h @@ -55,6 +55,14 @@ extern "C" { JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiverInternal (JNIEnv *, jobject, jobject, jobject); + /* + * Class: dev_onvoid_webrtc_RTCPeerConnection + * Method: addTransceiverOfType + * Signature: (Ldev/onvoid/webrtc/media/MediaType;Ldev/onvoid/webrtc/RTCRtpTransceiverInit;)Ldev/onvoid/webrtc/RTCRtpTransceiver; + */ + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiverOfType + (JNIEnv *, jobject, jobject, jobject); + /* * Class: dev_onvoid_webrtc_RTCPeerConnection * Method: createDataChannel diff --git a/webrtc-jni/src/main/cpp/include/JNI_RTCRtpSender.h b/webrtc-jni/src/main/cpp/include/JNI_RTCRtpSender.h index 26a0437e..1c23215c 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_RTCRtpSender.h +++ b/webrtc-jni/src/main/cpp/include/JNI_RTCRtpSender.h @@ -79,6 +79,22 @@ extern "C" { JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_generateKeyFrame (JNIEnv *, jobject); + /* + * Class: dev_onvoid_webrtc_RTCRtpSender + * Method: createVideoFrameInjector + * Signature: (Ldev/onvoid/webrtc/RTCEncodedVideoFrameInjectorObserver;)Ldev/onvoid/webrtc/RTCEncodedVideoFrameInjector; + */ + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_createVideoFrameInjector + (JNIEnv *, jobject, jobject); + + /* + * Class: dev_onvoid_webrtc_RTCRtpSender + * Method: createAudioFrameInjector + * Signature: (Ldev/onvoid/webrtc/RTCEncodedAudioFrameInjectorObserver;)Ldev/onvoid/webrtc/RTCEncodedAudioFrameInjector; + */ + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_createAudioFrameInjector + (JNIEnv *, jobject, jobject); + /* * Class: dev_onvoid_webrtc_RTCRtpSender * Method: getId diff --git a/webrtc-jni/src/main/cpp/include/api/EncodedFrameInjectorObserver.h b/webrtc-jni/src/main/cpp/include/api/EncodedFrameInjectorObserver.h new file mode 100644 index 00000000..8c37ae6f --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/api/EncodedFrameInjectorObserver.h @@ -0,0 +1,101 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_API_ENCODED_FRAME_INJECTOR_OBSERVER_H_ +#define JNI_WEBRTC_API_ENCODED_FRAME_INJECTOR_OBSERVER_H_ + +#include "JavaClass.h" +#include "JavaRef.h" + +#include "api/encoded_audio_frame_injector_interface.h" +#include "api/encoded_video_frame_injector_interface.h" + +#include +#include +#include + +namespace jni +{ + // Forwards the requests of an encoded frame injector to its Java observer. + // + // WebRTC makes them on its encoder threads; the Java observer is called on + // the JavaCallbackThread instead, so that it may call back into WebRTC. + // WebRTC keeps the callbacks for as long as the injector lives, which can + // be longer than this object, so they only share its state. + class EncodedFrameInjectorObserver + { + public: + // For an RTCEncodedVideoFrameInjectorObserver. + static std::unique_ptr ForVideo(JNIEnv * env, const JavaRef & observer); + + // For an RTCEncodedAudioFrameInjectorObserver. + static std::unique_ptr ForAudio(JNIEnv * env, const JavaRef & observer); + + // Waits for a Java call in progress, so the Java observer hears + // nothing more once this returns. + ~EncodedFrameInjectorObserver(); + + EncodedFrameInjectorObserver(const EncodedFrameInjectorObserver &) = delete; + EncodedFrameInjectorObserver & operator=(const EncodedFrameInjectorObserver &) = delete; + + webrtc::KeyFrameCallback CreateKeyFrameCallback() const; + webrtc::BitrateInfoCallback CreateBitrateInfoCallback() const; + webrtc::TargetBitrateCallback CreateTargetBitrateCallback() const; + + private: + class JavaVideoObserverClass : public JavaClass + { + public: + explicit JavaVideoObserverClass(JNIEnv * env); + + jmethodID onKeyFrameRequest; + jmethodID onBitrateChange; + }; + + class JavaAudioObserverClass : public JavaClass + { + public: + explicit JavaAudioObserverClass(JNIEnv * env); + + jmethodID onTargetBitrateChange; + }; + + struct Target + { + Target(JNIEnv * env, const JavaRef & observer); + + // Recursive, so that a Java observer may dispose its injector. + std::recursive_mutex mutex; + bool active; + const JavaGlobalRef observer; + jmethodID onKeyFrameRequest; + jmethodID onBitrateChange; + jmethodID onTargetBitrateChange; + }; + + explicit EncodedFrameInjectorObserver(std::shared_ptr target); + + // Calls the Java observer on the JavaCallbackThread, unless it is + // no longer active by then. + template + static void Post(const std::shared_ptr & target, Call call); + + private: + const std::shared_ptr target; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/src/JNI_RTCEncodedAudioFrameInjector.cpp b/webrtc-jni/src/main/cpp/src/JNI_RTCEncodedAudioFrameInjector.cpp new file mode 100644 index 00000000..a1061fb1 --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/JNI_RTCEncodedAudioFrameInjector.cpp @@ -0,0 +1,124 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "JNI_RTCEncodedAudioFrameInjector.h" +#include "api/EncodedFrameInjectorObserver.h" +#include "JavaError.h" +#include "JavaUtils.h" + +#include "api/encoded_audio_frame_injector_interface.h" +#include "api/frame_transformer_factory.h" + +#include +#include +#include + +static void injectFrame(webrtc::EncodedAudioFrameInjectorInterface * injector, + std::span data, jint payloadType, jlong rtpTimestamp, + jint audioLevel, jboolean hasAudioLevel, jlong captureTimeMs, jboolean hasCaptureTime) +{ + std::optional level; + std::optional captureTime; + + if (hasAudioLevel) { + level = static_cast(audioLevel); + } + if (hasCaptureTime) { + captureTime = static_cast(captureTimeMs); + } + + // The injector sends on the sender's SSRC, and takes neither the MIME type + // nor the sequence number from the frame. + auto frame = webrtc::CreateOutgoingAudioFrame( + webrtc::TransformableAudioFrameInterface::FrameType::kAudioFrameSpeech, + webrtc::PayloadType(payloadType), static_cast(rtpTimestamp), data.data(), data.size(), + captureTime, 0, {}, "", std::nullopt, level); + + injector->InjectFrame(std::move(frame)); +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector_injectDirectBuffer +(JNIEnv * env, jobject caller, jobject jBuffer, jint position, jint length, jint payloadType, + jlong rtpTimestamp, jint audioLevel, jboolean hasAudioLevel, jlong captureTimeMs, jboolean hasCaptureTime) +{ + webrtc::EncodedAudioFrameInjectorInterface * injector = GetHandle(env, caller); + CHECK_HANDLE(injector); + + try { + const uint8_t * address = static_cast(env->GetDirectBufferAddress(jBuffer)); + + if (address == nullptr) { + env->Throw(jni::JavaError(env, "Non-direct buffer provided")); + return; + } + + jlong capacity = env->GetDirectBufferCapacity(jBuffer); + + if (position < 0 || length < 0 || static_cast(position) + length > capacity) { + env->Throw(jni::JavaError(env, "Buffer position/length out of bounds")); + return; + } + + // The frame copies the data, so the caller may reuse the buffer. + injectFrame(injector, std::span(address + position, static_cast(length)), + payloadType, rtpTimestamp, audioLevel, hasAudioLevel, captureTimeMs, hasCaptureTime); + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector_injectByteArray +(JNIEnv * env, jobject caller, jbyteArray jData, jint payloadType, jlong rtpTimestamp, + jint audioLevel, jboolean hasAudioLevel, jlong captureTimeMs, jboolean hasCaptureTime) +{ + webrtc::EncodedAudioFrameInjectorInterface * injector = GetHandle(env, caller); + CHECK_HANDLE(injector); + + try { + auto releaseArray = [env, jData](jbyte * bytes) { + env->ReleaseByteArrayElements(jData, bytes, JNI_ABORT); + }; + std::unique_ptr bytes(env->GetByteArrayElements(jData, nullptr), releaseArray); + + if (!bytes) { + return; + } + + auto data = std::span(reinterpret_cast(bytes.get()), + static_cast(env->GetArrayLength(jData))); + + injectFrame(injector, data, payloadType, rtpTimestamp, audioLevel, hasAudioLevel, captureTimeMs, hasCaptureTime); + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedAudioFrameInjector_dispose +(JNIEnv * env, jobject caller) +{ + webrtc::EncodedAudioFrameInjectorInterface * injector = GetHandle(env, caller); + CHECK_HANDLE(injector); + + // The sender keeps the injector and its callbacks; only the Java observer + // stops hearing from them here. + ClearNativeObserver(env, caller, "observerHandle"); + + injector->Release(); + + SetHandle(env, caller, nullptr); +} diff --git a/webrtc-jni/src/main/cpp/src/JNI_RTCEncodedVideoFrameInjector.cpp b/webrtc-jni/src/main/cpp/src/JNI_RTCEncodedVideoFrameInjector.cpp new file mode 100644 index 00000000..397648e9 --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/JNI_RTCEncodedVideoFrameInjector.cpp @@ -0,0 +1,120 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "JNI_RTCEncodedVideoFrameInjector.h" +#include "api/EncodedFrameInjectorObserver.h" +#include "JavaError.h" +#include "JavaUtils.h" + +#include "api/encoded_video_frame_injector_interface.h" +#include "api/frame_transformer_factory.h" + +#include +#include +#include + +static void injectFrame(webrtc::EncodedVideoFrameInjectorInterface * injector, + std::span data, jlong rtpTimestamp, jboolean keyFrame, jint width, jint height, + jlong captureTimeMs, jboolean hasCaptureTime) +{ + std::optional captureTime; + + if (hasCaptureTime) { + captureTime = captureTimeMs; + } + + // The injector sends with the codec the sender negotiated, so the payload + // type and the codec type of the frame are not used. + auto frame = webrtc::CreateOutgoingVideoFrame( + keyFrame ? webrtc::VideoFrameType::kVideoFrameKey : webrtc::VideoFrameType::kVideoFrameDelta, + webrtc::PayloadType(0), static_cast(rtpTimestamp), data, captureTime, {}, + webrtc::kVideoCodecGeneric, std::nullopt, static_cast(width), static_cast(height)); + + injector->InjectFrame(std::move(frame)); +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector_injectDirectBuffer +(JNIEnv * env, jobject caller, jobject jBuffer, jint position, jint length, jlong rtpTimestamp, + jboolean keyFrame, jint width, jint height, jlong captureTimeMs, jboolean hasCaptureTime) +{ + webrtc::EncodedVideoFrameInjectorInterface * injector = GetHandle(env, caller); + CHECK_HANDLE(injector); + + try { + const uint8_t * address = static_cast(env->GetDirectBufferAddress(jBuffer)); + + if (address == nullptr) { + env->Throw(jni::JavaError(env, "Non-direct buffer provided")); + return; + } + + jlong capacity = env->GetDirectBufferCapacity(jBuffer); + + if (position < 0 || length < 0 || static_cast(position) + length > capacity) { + env->Throw(jni::JavaError(env, "Buffer position/length out of bounds")); + return; + } + + // The frame copies the data, so the caller may reuse the buffer. + injectFrame(injector, std::span(address + position, static_cast(length)), + rtpTimestamp, keyFrame, width, height, captureTimeMs, hasCaptureTime); + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector_injectByteArray +(JNIEnv * env, jobject caller, jbyteArray jData, jlong rtpTimestamp, jboolean keyFrame, + jint width, jint height, jlong captureTimeMs, jboolean hasCaptureTime) +{ + webrtc::EncodedVideoFrameInjectorInterface * injector = GetHandle(env, caller); + CHECK_HANDLE(injector); + + try { + auto releaseArray = [env, jData](jbyte * bytes) { + env->ReleaseByteArrayElements(jData, bytes, JNI_ABORT); + }; + std::unique_ptr bytes(env->GetByteArrayElements(jData, nullptr), releaseArray); + + if (!bytes) { + return; + } + + auto data = std::span(reinterpret_cast(bytes.get()), + static_cast(env->GetArrayLength(jData))); + + injectFrame(injector, data, rtpTimestamp, keyFrame, width, height, captureTimeMs, hasCaptureTime); + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCEncodedVideoFrameInjector_dispose +(JNIEnv * env, jobject caller) +{ + webrtc::EncodedVideoFrameInjectorInterface * injector = GetHandle(env, caller); + CHECK_HANDLE(injector); + + // The sender keeps the injector and its callbacks; only the Java observer + // stops hearing from them here. + ClearNativeObserver(env, caller, "observerHandle"); + + injector->Release(); + + SetHandle(env, caller, nullptr); +} diff --git a/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp b/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp index 2fd24bde..21550ef0 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp @@ -195,6 +195,46 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceive return nullptr; } +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiverOfType +(JNIEnv * env, jobject caller, jobject jMediaType, jobject jTransceiverInit) +{ + if (jMediaType == nullptr) { + env->Throw(jni::JavaNullPointerException(env, "MediaType must not be null")); + return nullptr; + } + + webrtc::PeerConnectionInterface * pc = GetHandle(env, caller); + CHECK_HANDLEV(pc, nullptr); + + try { + auto mediaType = jni::JavaEnums::toNative(env, jMediaType); + + webrtc::RTCErrorOr> result; + + if (jTransceiverInit != nullptr) { + auto init = jni::RTCRtpTransceiverInit::toNative(env, jni::JavaLocalRef(env, jTransceiverInit)); + + result = pc->AddTransceiver(mediaType, init); + } + else { + result = pc->AddTransceiver(mediaType); + } + + if (result.ok()) { + auto transceiver = result.MoveValue(); + + return jni::JavaFactories::create(env, transceiver.release()).release(); + } + + env->Throw(jni::JavaRuntimeException(env, jni::RTCErrorToString(result.error()).c_str())); + } + catch (...) { + ThrowCxxJavaException(env); + } + + return nullptr; +} + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_createDataChannel (JNIEnv * env, jobject caller, jstring jLabel, jobject jDict) { diff --git a/webrtc-jni/src/main/cpp/src/JNI_RTCRtpSender.cpp b/webrtc-jni/src/main/cpp/src/JNI_RTCRtpSender.cpp index 55a81cd9..ff29148f 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_RTCRtpSender.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_RTCRtpSender.cpp @@ -15,6 +15,7 @@ */ #include "JNI_RTCRtpSender.h" +#include "api/EncodedFrameInjectorObserver.h" #include "api/RTCRtpSenderObserver.h" #include "api/RTCRtpSendParameters.h" #include "api/WebRTCUtils.h" @@ -160,6 +161,80 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_generateKeyFrame } } +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_createVideoFrameInjector +(JNIEnv * env, jobject caller, jobject jObserver) +{ + webrtc::RtpSenderInterface * sender = GetHandle(env, caller); + CHECK_HANDLEV(sender, nullptr); + + try { + std::unique_ptr observer; + webrtc::KeyFrameCallback keyFrameCallback; + webrtc::BitrateInfoCallback bitrateCallback; + + if (jObserver != nullptr) { + observer = jni::EncodedFrameInjectorObserver::ForVideo(env, jni::JavaLocalRef(env, jObserver)); + keyFrameCallback = observer->CreateKeyFrameCallback(); + bitrateCallback = observer->CreateBitrateInfoCallback(); + } + + auto injector = sender->CreateEncodedVideoFrameInjector(std::move(keyFrameCallback), std::move(bitrateCallback)); + + // WebRTC refuses the injector for a stopped sender, a sender with a + // track, an audio sender, and a sender that has one already. + if (!injector) { + return nullptr; + } + + auto jInjector = jni::JavaFactories::create(env, injector.release()); + + SetHandle(env, jInjector.get(), "observerHandle", observer.release()); + + return jInjector.release(); + } + catch (...) { + ThrowCxxJavaException(env); + } + + return nullptr; +} + +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_createAudioFrameInjector +(JNIEnv * env, jobject caller, jobject jObserver) +{ + webrtc::RtpSenderInterface * sender = GetHandle(env, caller); + CHECK_HANDLEV(sender, nullptr); + + try { + std::unique_ptr observer; + webrtc::TargetBitrateCallback bitrateCallback; + + if (jObserver != nullptr) { + observer = jni::EncodedFrameInjectorObserver::ForAudio(env, jni::JavaLocalRef(env, jObserver)); + bitrateCallback = observer->CreateTargetBitrateCallback(); + } + + auto injector = sender->CreateEncodedAudioFrameInjector(std::move(bitrateCallback)); + + // WebRTC refuses the injector for a stopped sender, a sender with a + // track, a video sender, and a sender that has one already. + if (!injector) { + return nullptr; + } + + auto jInjector = jni::JavaFactories::create(env, injector.release()); + + SetHandle(env, jInjector.get(), "observerHandle", observer.release()); + + return jInjector.release(); + } + catch (...) { + ThrowCxxJavaException(env); + } + + return nullptr; +} + JNIEXPORT jstring JNICALL Java_dev_onvoid_webrtc_RTCRtpSender_getId (JNIEnv * env, jobject caller) { diff --git a/webrtc-jni/src/main/cpp/src/WebRTCContext.cpp b/webrtc-jni/src/main/cpp/src/WebRTCContext.cpp index cbadf2a2..700d2b5b 100644 --- a/webrtc-jni/src/main/cpp/src/WebRTCContext.cpp +++ b/webrtc-jni/src/main/cpp/src/WebRTCContext.cpp @@ -26,6 +26,8 @@ #include "JavaUtils.h" #include "JNI_WebRTC.h" +#include "api/encoded_audio_frame_injector_interface.h" +#include "api/encoded_video_frame_injector_interface.h" #include "api/environment/environment_factory.h" #include "api/peer_connection_interface.h" #include "api/priority.h" @@ -112,6 +114,8 @@ namespace jni JavaFactories::add(env, PKG_MEDIA"MediaStream"); JavaFactories::add(env, PKG"RTCDataChannel"); JavaFactories::add(env, PKG"RTCDtlsTransport"); + JavaFactories::add(env, PKG"RTCEncodedAudioFrameInjector"); + JavaFactories::add(env, PKG"RTCEncodedVideoFrameInjector"); JavaFactories::add(env, PKG"RTCIceTransport"); JavaFactories::add(env, PKG"RTCPeerConnection"); JavaFactories::add(env, PKG"RTCRtpReceiver"); diff --git a/webrtc-jni/src/main/cpp/src/api/EncodedFrameInjectorObserver.cpp b/webrtc-jni/src/main/cpp/src/api/EncodedFrameInjectorObserver.cpp new file mode 100644 index 00000000..c5d5b25f --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/api/EncodedFrameInjectorObserver.cpp @@ -0,0 +1,129 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "api/EncodedFrameInjectorObserver.h" +#include "api/JavaCallbackThread.h" +#include "api/WebRTCUtils.h" +#include "JavaClasses.h" +#include "JavaUtils.h" +#include "JNI_WebRTC.h" + +#include + +namespace jni +{ + std::unique_ptr EncodedFrameInjectorObserver::ForVideo(JNIEnv * env, const JavaRef & observer) + { + auto target = std::make_shared(env, observer); + const auto javaClass = JavaClasses::get(env); + + target->onKeyFrameRequest = javaClass->onKeyFrameRequest; + target->onBitrateChange = javaClass->onBitrateChange; + + return std::unique_ptr(new EncodedFrameInjectorObserver(std::move(target))); + } + + std::unique_ptr EncodedFrameInjectorObserver::ForAudio(JNIEnv * env, const JavaRef & observer) + { + auto target = std::make_shared(env, observer); + const auto javaClass = JavaClasses::get(env); + + target->onTargetBitrateChange = javaClass->onTargetBitrateChange; + + return std::unique_ptr(new EncodedFrameInjectorObserver(std::move(target))); + } + + EncodedFrameInjectorObserver::EncodedFrameInjectorObserver(std::shared_ptr target) : + target(std::move(target)) + { + } + + EncodedFrameInjectorObserver::~EncodedFrameInjectorObserver() + { + std::lock_guard lock(target->mutex); + + target->active = false; + } + + template + void EncodedFrameInjectorObserver::Post(const std::shared_ptr & target, Call call) + { + JavaCallbackThread::Post([target, call = std::move(call)](JNIEnv * env) { + std::lock_guard lock(target->mutex); + + if (!target->active) { + return; + } + + call(env, *target); + + // A Java exception must not escape into the next callback. + ReportPendingException(env); + }); + } + + webrtc::KeyFrameCallback EncodedFrameInjectorObserver::CreateKeyFrameCallback() const + { + return [target = target]() { + Post(target, [](JNIEnv * env, Target & target) { + env->CallVoidMethod(target.observer, target.onKeyFrameRequest); + }); + }; + } + + webrtc::BitrateInfoCallback EncodedFrameInjectorObserver::CreateBitrateInfoCallback() const + { + return [target = target](int32_t allocatedBitrate, int32_t availableOutgoingBitrate) { + Post(target, [allocatedBitrate, availableOutgoingBitrate](JNIEnv * env, Target & target) { + env->CallVoidMethod(target.observer, target.onBitrateChange, + static_cast(allocatedBitrate), static_cast(availableOutgoingBitrate)); + }); + }; + } + + webrtc::TargetBitrateCallback EncodedFrameInjectorObserver::CreateTargetBitrateCallback() const + { + return [target = target](int32_t targetBitrate) { + Post(target, [targetBitrate](JNIEnv * env, Target & target) { + env->CallVoidMethod(target.observer, target.onTargetBitrateChange, static_cast(targetBitrate)); + }); + }; + } + + EncodedFrameInjectorObserver::Target::Target(JNIEnv * env, const JavaRef & observer) : + active(true), + observer(env, observer.get()), + onKeyFrameRequest(nullptr), + onBitrateChange(nullptr), + onTargetBitrateChange(nullptr) + { + } + + EncodedFrameInjectorObserver::JavaVideoObserverClass::JavaVideoObserverClass(JNIEnv * env) + { + jclass cls = FindClass(env, PKG"RTCEncodedVideoFrameInjectorObserver"); + + onKeyFrameRequest = GetMethod(env, cls, "onKeyFrameRequest", "()V"); + onBitrateChange = GetMethod(env, cls, "onBitrateChange", "(II)V"); + } + + EncodedFrameInjectorObserver::JavaAudioObserverClass::JavaAudioObserverClass(JNIEnv * env) + { + jclass cls = FindClass(env, PKG"RTCEncodedAudioFrameInjectorObserver"); + + onTargetBitrateChange = GetMethod(env, cls, "onTargetBitrateChange", "(I)V"); + } +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCDataChannel.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCDataChannel.java index 5fbbdc6d..23674e29 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCDataChannel.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCDataChannel.java @@ -199,7 +199,7 @@ public void send(RTCDataChannelBuffer buffer) throws Exception { * limit, an array offset) must never reach the wire. Reads through a * duplicate, so the caller's position is left untouched. */ - private static byte[] copyWindow(ByteBuffer data) { + static byte[] copyWindow(ByteBuffer data) { if (data.hasArray() && data.arrayOffset() == 0 && data.position() == 0 && data.remaining() == data.array().length) { return data.array(); @@ -249,8 +249,8 @@ public void sendAsync(RTCDataChannelBuffer buffer) { * returns, without changing its position or limit. *

* The observer is called once when the operation completes or is discarded - * during channel shutdown. It runs on a native callback thread that all - * send and DTLS transport observers share, in the order the results arrive, + * during channel shutdown. It runs on a native callback thread shared with + * other WebRTC observers, in the order the results arrive, * and may run before this method returns. It may call other WebRTC methods, * but should not block, since that holds up the other observers. No * callback is guaranteed during JVM shutdown. diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCDtlsTransportObserver.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCDtlsTransportObserver.java index 7270ed78..6f5adafb 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCDtlsTransportObserver.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCDtlsTransportObserver.java @@ -19,8 +19,8 @@ /** * Receives events from an {@link RTCDtlsTransport}. *

- * The events arrive in order on a native callback thread that all DTLS - * transport and data channel send observers share. An observer may call other + * The events arrive in order on a native callback thread shared with other + * WebRTC observers. An observer may call other * WebRTC methods, but should not block, since that holds up the other * observers. Once {@link RTCDtlsTransport#unregisterObserver()} returns, the * observer receives no more events. diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedAudioFrameInjector.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedAudioFrameInjector.java new file mode 100644 index 00000000..27f4906f --- /dev/null +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedAudioFrameInjector.java @@ -0,0 +1,129 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc; + +import java.nio.ByteBuffer; +import java.util.Objects; + +import dev.onvoid.webrtc.internal.DisposableNativeObject; + +/** + * Sends audio frames that are already encoded on an {@link RTCRtpSender}, + * in place of audio from a track: WebRTC packetizes and sends them as they + * are, without encoding anything itself. + *

+ * The frames must be in a codec the sender negotiated, named by the payload + * type each frame is injected with; the sender's + * {@link RTCRtpSender#getParameters() parameters} list them. The injector takes + * the place of the sender's track: it can only be created for a sender without + * one, and {@link RTCRtpSender#replaceTrack(dev.onvoid.webrtc.media.MediaStreamTrack) + * giving the sender a track} ends the injection. + *

+ * Frames can be injected from any thread. + * + * @author Alex Andres + * + * @see RTCRtpSender#createEncodedAudioFrameInjector(RTCEncodedAudioFrameInjectorObserver) + */ +public class RTCEncodedAudioFrameInjector extends DisposableNativeObject { + + /* + * The native observer set through this object, owned by it and freed when + * disposed. + */ + @SuppressWarnings("unused") + private long observerHandle; + + + /** + * Constructor to be used by the native api. + */ + private RTCEncodedAudioFrameInjector() { + + } + + /** + * Sends an encoded frame. The bytes between the position and the limit of + * the buffer are copied before this method returns, so the buffer may be + * reused right away; its position and limit stay as they are. + * + * @param data The encoded frame. + * @param payloadType The payload type of the frame's codec. + * @param rtpTimestamp The RTP timestamp of the frame, on the codec's + * clock, without the sender's random offset. + */ + public void injectFrame(ByteBuffer data, int payloadType, long rtpTimestamp) { + injectFrame(data, payloadType, rtpTimestamp, null, null); + } + + /** + * Sends an encoded frame, along with its audio level and capture time. + * The bytes between the position and the limit of the buffer are copied + * before this method returns, so the buffer may be reused right away; its + * position and limit stay as they are. + * + * @param data The encoded frame. + * @param payloadType The payload type of the frame's codec. + * @param rtpTimestamp The RTP timestamp of the frame, on the codec's + * clock, without the sender's random offset. + * @param audioLevel The audio level in -dBov, from 0 (loudest) to 127 + * (silence), or null to send none. + * @param captureTimeMs The absolute time the frame was captured, in + * milliseconds, or null to send none. + */ + public void injectFrame(ByteBuffer data, int payloadType, long rtpTimestamp, Integer audioLevel, + Long captureTimeMs) { + Objects.requireNonNull(data, "data"); + + if (payloadType < 0 || payloadType > 127) { + throw new IllegalArgumentException("Payload type out of range: " + payloadType); + } + if (rtpTimestamp < 0 || rtpTimestamp > 0xFFFFFFFFL) { + throw new IllegalArgumentException("RTP timestamp out of the 32-bit range: " + rtpTimestamp); + } + if (audioLevel != null && (audioLevel < 0 || audioLevel > 127)) { + throw new IllegalArgumentException("Audio level out of range: " + audioLevel); + } + + int level = audioLevel != null ? audioLevel : 0; + long captureTime = captureTimeMs != null ? captureTimeMs : 0; + + if (data.isDirect()) { + injectDirectBuffer(data, data.position(), data.remaining(), payloadType, rtpTimestamp, + level, audioLevel != null, captureTime, captureTimeMs != null); + } + else { + injectByteArray(RTCDataChannel.copyWindow(data), payloadType, rtpTimestamp, + level, audioLevel != null, captureTime, captureTimeMs != null); + } + } + + /** + * Releases the native injector. The sender keeps sending nothing until it + * gets a track again, and the observer is not called any more once this + * returns. + */ + @Override + public native void dispose(); + + private native void injectDirectBuffer(ByteBuffer data, int position, int length, int payloadType, + long rtpTimestamp, int audioLevel, boolean hasAudioLevel, long captureTimeMs, boolean hasCaptureTime); + + private native void injectByteArray(byte[] data, int payloadType, long rtpTimestamp, + int audioLevel, boolean hasAudioLevel, long captureTimeMs, boolean hasCaptureTime); + +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedAudioFrameInjectorObserver.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedAudioFrameInjectorObserver.java new file mode 100644 index 00000000..8ef102fc --- /dev/null +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedAudioFrameInjectorObserver.java @@ -0,0 +1,39 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc; + +/** + * Receives the requests of an {@link RTCEncodedAudioFrameInjector}'s sender: + * what the encoder would otherwise have answered. + *

+ * The calls arrive in order on a native callback thread shared with other + * WebRTC observers. An observer may call + * other WebRTC methods, but should not block, since that holds up the other + * observers. + * + * @author Alex Andres + */ +public interface RTCEncodedAudioFrameInjectorObserver { + + /** + * The bitrate the sender allows for the injected frames has changed. + * + * @param targetBitrate The target bitrate, in bits per second. + */ + void onTargetBitrateChange(int targetBitrate); + +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedVideoFrameInjector.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedVideoFrameInjector.java new file mode 100644 index 00000000..789b21ce --- /dev/null +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedVideoFrameInjector.java @@ -0,0 +1,129 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc; + +import java.nio.ByteBuffer; +import java.util.Objects; + +import dev.onvoid.webrtc.internal.DisposableNativeObject; + +/** + * Sends video frames that are already encoded on an {@link RTCRtpSender}, + * in place of frames from a track: WebRTC packetizes and sends them as they + * are, without encoding anything itself. + *

+ * The frames must be in the codec the sender negotiated, which its + * {@link RTCRtpSender#getParameters() parameters} name, and the sender sends + * them with that codec's payload type. The injector takes the place of the + * sender's track: it can only be created for a sender without one, and + * {@link RTCRtpSender#replaceTrack(dev.onvoid.webrtc.media.MediaStreamTrack) + * giving the sender a track} ends the injection. + *

+ * Frames can be injected from any thread. + * + * @author Alex Andres + * + * @see RTCRtpSender#createEncodedVideoFrameInjector(RTCEncodedVideoFrameInjectorObserver) + */ +public class RTCEncodedVideoFrameInjector extends DisposableNativeObject { + + /* + * The native observer set through this object, owned by it and freed when + * disposed. + */ + @SuppressWarnings("unused") + private long observerHandle; + + + /** + * Constructor to be used by the native api. + */ + private RTCEncodedVideoFrameInjector() { + + } + + /** + * Sends an encoded frame. The bytes between the position and the limit of + * the buffer are copied before this method returns, so the buffer may be + * reused right away; its position and limit stay as they are. + * + * @param data The encoded frame. + * @param rtpTimestamp The RTP timestamp of the frame, on the 90 kHz video + * clock, without the sender's random offset. + * @param keyFrame Whether the frame is a key frame. + * @param width The width of the frame in pixels. + * @param height The height of the frame in pixels. + */ + public void injectFrame(ByteBuffer data, long rtpTimestamp, boolean keyFrame, int width, int height) { + inject(data, rtpTimestamp, keyFrame, width, height, 0, false); + } + + /** + * Sends an encoded frame, along with the time it was captured. The bytes + * between the position and the limit of the buffer are copied before this + * method returns, so the buffer may be reused right away; its position and + * limit stay as they are. + * + * @param data The encoded frame. + * @param rtpTimestamp The RTP timestamp of the frame, on the 90 kHz video + * clock, without the sender's random offset. + * @param keyFrame Whether the frame is a key frame. + * @param width The width of the frame in pixels. + * @param height The height of the frame in pixels. + * @param captureTimeMs The time the frame was captured, in milliseconds. + */ + public void injectFrame(ByteBuffer data, long rtpTimestamp, boolean keyFrame, int width, int height, + long captureTimeMs) { + inject(data, rtpTimestamp, keyFrame, width, height, captureTimeMs, true); + } + + /** + * Releases the native injector. The sender keeps sending nothing until it + * gets a track again, and the observer is not called any more once this + * returns. + */ + @Override + public native void dispose(); + + private void inject(ByteBuffer data, long rtpTimestamp, boolean keyFrame, int width, int height, + long captureTimeMs, boolean hasCaptureTime) { + Objects.requireNonNull(data, "data"); + + if (rtpTimestamp < 0 || rtpTimestamp > 0xFFFFFFFFL) { + throw new IllegalArgumentException("RTP timestamp out of the 32-bit range: " + rtpTimestamp); + } + if (width < 1 || width > 0xFFFF || height < 1 || height > 0xFFFF) { + throw new IllegalArgumentException("Frame size out of range: " + width + "x" + height); + } + + if (data.isDirect()) { + injectDirectBuffer(data, data.position(), data.remaining(), rtpTimestamp, keyFrame, + width, height, captureTimeMs, hasCaptureTime); + } + else { + injectByteArray(RTCDataChannel.copyWindow(data), rtpTimestamp, keyFrame, + width, height, captureTimeMs, hasCaptureTime); + } + } + + private native void injectDirectBuffer(ByteBuffer data, int position, int length, long rtpTimestamp, + boolean keyFrame, int width, int height, long captureTimeMs, boolean hasCaptureTime); + + private native void injectByteArray(byte[] data, long rtpTimestamp, boolean keyFrame, + int width, int height, long captureTimeMs, boolean hasCaptureTime); + +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedVideoFrameInjectorObserver.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedVideoFrameInjectorObserver.java new file mode 100644 index 00000000..44f577e8 --- /dev/null +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCEncodedVideoFrameInjectorObserver.java @@ -0,0 +1,48 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc; + +/** + * Receives the requests of an {@link RTCEncodedVideoFrameInjector}'s sender: + * what the encoder would otherwise have answered. + *

+ * The calls arrive in order on a native callback thread shared with other + * WebRTC observers. An observer may call + * other WebRTC methods, but should not block, since that holds up the other + * observers. + * + * @author Alex Andres + */ +public interface RTCEncodedVideoFrameInjectorObserver { + + /** + * The sender needs a key frame, for example because the remote peer asked + * for one to recover from loss. The next injected frame should be one. + */ + void onKeyFrameRequest(); + + /** + * The bitrate the sender allows for the injected frames has changed. + * + * @param allocatedBitrate The bitrate allocated to the frames, + * in bits per second. + * @param availableOutgoingBitrate The bitrate the connection is estimated + * to carry, in bits per second. + */ + void onBitrateChange(int allocatedBitrate, int availableOutgoingBitrate); + +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java index ce0ab52a..01e5624a 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java @@ -21,6 +21,7 @@ import dev.onvoid.webrtc.internal.NativeObject; import dev.onvoid.webrtc.media.MediaStreamTrack; +import dev.onvoid.webrtc.media.MediaType; import java.util.List; @@ -161,6 +162,23 @@ public RTCRtpTransceiver addTransceiver(MediaStreamTrack track, return addTransceiverInternal(track, init); } + /** + * Create a new RTCRtpTransceiver of the given media type, without a track, + * and add it to the set of transceivers. Its sender sends nothing until it + * gets a track through {@link RTCRtpSender#replaceTrack} or frames through + * an encoded frame injector. + * + * @param mediaType The media type of the transceiver, audio or video. + * @param init The transceiver configuration options, or null for the + * defaults. + * + * @return The RTCRtpTransceiver which will be used to transmit and receive + * the media data. + */ + public RTCRtpTransceiver addTransceiver(MediaType mediaType, RTCRtpTransceiverInit init) { + return addTransceiverOfType(mediaType, init); + } + /** * Reports a track that is about to be sent to the factory that created this * connection, which rejects a track whose audio is pushed while the factory @@ -521,6 +539,9 @@ public int hashCode() { private native RTCRtpSender addTrackInternal(MediaStreamTrack track, List streamIds); + private native RTCRtpTransceiver addTransceiverOfType(MediaType mediaType, + RTCRtpTransceiverInit init); + private native RTCRtpTransceiver addTransceiverInternal( MediaStreamTrack track, RTCRtpTransceiverInit init); diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java index 3fd044b8..55f34f1d 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java @@ -183,6 +183,64 @@ private RTCRtpSender() { */ public native void generateKeyFrame(); + /** + * Creates an injector that sends video frames that are already encoded + * on this sender, in place of a track. + * + * @param observer The observer for the sender's key frame requests and + * bitrate changes, or {@code null} for none. + * + * @return The injector, to be disposed when no longer used. + * + * @throws IllegalStateException If this is not a video sender, or it is + * stopped, has a track or has an injector. + */ + public RTCEncodedVideoFrameInjector createEncodedVideoFrameInjector( + RTCEncodedVideoFrameInjectorObserver observer) { + RTCEncodedVideoFrameInjector injector = createVideoFrameInjector(observer); + + if (injector == null) { + throw new IllegalStateException("An encoded video frame injector needs a video sender that is " + + "not stopped and has no track and no injector"); + } + + return injector; + } + + /** + * Creates an injector that sends audio frames that are already encoded + * on this sender, in place of a track. + *

+ * Do not inject audio through a factory that sends audio captured by its + * audio device module: WebRTC passes captured audio to every audio sender + * that sends, and it would drive the injecting sender as well. + * + * @param observer The observer for the sender's target bitrate changes, + * or {@code null} for none. + * + * @return The injector, to be disposed when no longer used. + * + * @throws IllegalStateException If this is not an audio sender, or it is + * stopped, has a track or has an injector. + */ + public RTCEncodedAudioFrameInjector createEncodedAudioFrameInjector( + RTCEncodedAudioFrameInjectorObserver observer) { + RTCEncodedAudioFrameInjector injector = createAudioFrameInjector(observer); + + if (injector == null) { + throw new IllegalStateException("An encoded audio frame injector needs an audio sender that is " + + "not stopped and has no track and no injector"); + } + + return injector; + } + + private native RTCEncodedVideoFrameInjector createVideoFrameInjector( + RTCEncodedVideoFrameInjectorObserver observer); + + private native RTCEncodedAudioFrameInjector createAudioFrameInjector( + RTCEncodedAudioFrameInjectorObserver observer); + /** * Releases the native reference held by this RTCRtpSender instance. *

diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/RTCEncodedFrameInjectorTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/RTCEncodedFrameInjectorTests.java new file mode 100644 index 00000000..6a48790f --- /dev/null +++ b/webrtc/src/test/java/dev/onvoid/webrtc/RTCEncodedFrameInjectorTests.java @@ -0,0 +1,344 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import dev.onvoid.webrtc.media.MediaType; +import dev.onvoid.webrtc.media.video.VideoTrack; +import dev.onvoid.webrtc.media.video.VideoTrackSink; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.parallel.Execution; +import org.junit.jupiter.api.parallel.ExecutionMode; + +/** + * Tests sending encoded frames through injectors. The frames are taken from a + * regular call first, so that they are real encoder output. + */ +@Execution(ExecutionMode.SAME_THREAD) +class RTCEncodedFrameInjectorTests extends TestBase { + + private static final long TIMEOUT_SECONDS = 10; + + + @Test + void injectedVideoIsDecoded() throws Exception { + List frames = captureVideoFrames(30); + + TestPeerConnection caller = new TestPeerConnection(factory); + TestPeerConnection callee = new TestPeerConnection(factory); + AtomicInteger bitrate = new AtomicInteger(); + AtomicReference size = new AtomicReference<>(); + CountDownLatch decoded = new CountDownLatch(10); + + try { + RTCRtpTransceiver transceiver = addSendOnlyTransceiver(caller, MediaType.VIDEO, "VP8"); + RTCRtpSender sender = transceiver.getSender(); + + RTCEncodedVideoFrameInjector injector = sender.createEncodedVideoFrameInjector( + new RTCEncodedVideoFrameInjectorObserver() { + @Override + public void onKeyFrameRequest() { } + + @Override + public void onBitrateChange(int allocatedBitrate, int availableOutgoingBitrate) { + bitrate.set(allocatedBitrate); + } + }); + + negotiate(caller, callee); + + RTCRtpReceiver receiver = getReceiver(callee); + VideoTrack track = (VideoTrack) receiver.getTrack(); + VideoTrackSink sink = frame -> { + size.set(frame.buffer.getWidth() + "x" + frame.buffer.getHeight()); + frame.release(); + decoded.countDown(); + }; + track.addSink(sink); + + // The frames are sent over and over, each round starting with the + // key frame they begin with, at 30 frames per second. + long rtpTimestamp = 0; + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); + + while (decoded.getCount() > 0 && System.nanoTime() < deadline) { + for (CapturedFrame frame : frames) { + injector.injectFrame(ByteBuffer.wrap(frame.data), rtpTimestamp, frame.keyFrame, + frame.width, frame.height); + rtpTimestamp += 3000; + Thread.sleep(33); + } + } + + assertEquals(0, decoded.getCount(), "too few injected frames decoded"); + assertEquals(frames.get(0).width + "x" + frames.get(0).height, size.get()); + assertTrue(bitrate.get() > 0, "no bitrate reported"); + + track.removeSink(sink); + receiver.dispose(); + injector.dispose(); + sender.dispose(); + transceiver.dispose(); + } + finally { + caller.close(); + callee.close(); + } + } + + @Test + void injectedAudioIsReceived() throws Exception { + List frames = captureAudioFrames(50); + + TestPeerConnection caller = new TestPeerConnection(factory); + TestPeerConnection callee = new TestPeerConnection(factory); + AtomicReference firstReceived = new AtomicReference<>(); + CountDownLatch received = new CountDownLatch(10); + + try { + RTCRtpTransceiver transceiver = addSendOnlyTransceiver(caller, MediaType.AUDIO, "opus"); + RTCRtpSender sender = transceiver.getSender(); + RTCEncodedAudioFrameInjector injector = sender.createEncodedAudioFrameInjector(null); + + negotiate(caller, callee); + + int payloadType = -1; + + for (RTCRtpCodecParameters codec : sender.getParameters().codecs) { + if ("opus".equalsIgnoreCase(codec.codecName)) { + payloadType = codec.payloadType; + } + } + + assertTrue(payloadType >= 0, "opus was not negotiated"); + + RTCRtpReceiver receiver = getReceiver(callee); + receiver.setTransform(frame -> { + ByteBuffer data = frame.getData(); + byte[] copy = new byte[data.remaining()]; + data.get(copy); + firstReceived.compareAndSet(null, copy); + received.countDown(); + }); + + // Opus frames of 20 ms on its 48 kHz clock. + long rtpTimestamp = 0; + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); + + while (received.getCount() > 0 && System.nanoTime() < deadline) { + for (CapturedFrame frame : frames) { + ByteBuffer data = ByteBuffer.allocateDirect(frame.data.length); + data.put(frame.data).flip(); + + injector.injectFrame(data, payloadType, rtpTimestamp); + rtpTimestamp += 960; + Thread.sleep(20); + } + } + + assertEquals(0, received.getCount(), "too few injected frames received"); + assertNotNull(firstReceived.get()); + assertTrue(frames.stream().anyMatch(frame -> Arrays.equals(frame.data, firstReceived.get())), + "the received frame is not one that was injected"); + + receiver.setTransform(null); + receiver.dispose(); + injector.dispose(); + sender.dispose(); + transceiver.dispose(); + } + finally { + caller.close(); + callee.close(); + } + } + + @Test + void injectorNeedsTracklessSender() throws Exception { + try (TestMediaCall call = new TestMediaCall(factory, true, false)) { + RTCRtpSender sender = call.getVideoSender(); + + // The sender has a track. + assertThrows(IllegalStateException.class, () -> sender.createEncodedVideoFrameInjector(null)); + // The sender sends video. + assertThrows(IllegalStateException.class, () -> sender.createEncodedAudioFrameInjector(null)); + } + } + + @Test + void injectorRejectsInvalidFrames() throws Exception { + TestPeerConnection caller = new TestPeerConnection(factory); + + try { + RTCRtpTransceiver transceiver = addSendOnlyTransceiver(caller, MediaType.VIDEO, "VP8"); + RTCRtpSender sender = transceiver.getSender(); + RTCEncodedVideoFrameInjector injector = sender.createEncodedVideoFrameInjector(null); + ByteBuffer data = ByteBuffer.allocate(1); + + // A sender takes one injector only. + assertThrows(IllegalStateException.class, () -> sender.createEncodedVideoFrameInjector(null)); + + assertThrows(NullPointerException.class, () -> injector.injectFrame(null, 0, true, 320, 240)); + assertThrows(IllegalArgumentException.class, () -> injector.injectFrame(data, -1, true, 320, 240)); + assertThrows(IllegalArgumentException.class, () -> injector.injectFrame(data, 0, true, 0, 240)); + assertThrows(IllegalArgumentException.class, () -> injector.injectFrame(data, 0, true, 320, 70000)); + + injector.dispose(); + sender.dispose(); + transceiver.dispose(); + } + finally { + caller.close(); + } + } + + /** + * Takes the given number of encoded video frames from a call, starting + * with the key frame a stream begins with. + */ + private List captureVideoFrames(int count) throws Exception { + List frames = Collections.synchronizedList(new ArrayList<>()); + CountDownLatch captured = new CountDownLatch(count); + + try (TestMediaCall call = new TestMediaCall(factory, true, false)) { + call.getVideoSender().setTransform(frame -> { + RTCEncodedVideoFrame videoFrame = (RTCEncodedVideoFrame) frame; + + if (captured.getCount() > 0 && (!frames.isEmpty() || videoFrame.isKeyFrame())) { + frames.add(new CapturedFrame(frame, videoFrame.isKeyFrame(), + videoFrame.getWidth(), videoFrame.getHeight())); + captured.countDown(); + } + }); + + call.negotiate(); + call.awaitConnected(); + call.startMedia(); + + assertTrue(captured.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), "too few video frames encoded"); + } + + return new ArrayList<>(frames); + } + + /** + * Takes the given number of encoded audio frames from a call. + */ + private List captureAudioFrames(int count) throws Exception { + List frames = Collections.synchronizedList(new ArrayList<>()); + CountDownLatch captured = new CountDownLatch(count); + + try (TestMediaCall call = new TestMediaCall(factory, false, true)) { + call.getAudioSender().setTransform(frame -> { + if (captured.getCount() > 0 && frame.getSize() > 0) { + frames.add(new CapturedFrame(frame, false, 0, 0)); + captured.countDown(); + } + }); + + call.negotiate(); + call.awaitConnected(); + call.startMedia(); + + assertTrue(captured.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), "too few audio frames encoded"); + } + + return new ArrayList<>(frames); + } + + /** + * Adds a transceiver that sends without a track, preferring the named codec. + */ + private RTCRtpTransceiver addSendOnlyTransceiver(TestPeerConnection peer, MediaType mediaType, String codec) { + RTCRtpTransceiverInit init = new RTCRtpTransceiverInit(); + init.direction = RTCRtpTransceiverDirection.SEND_ONLY; + + RTCRtpTransceiver transceiver = peer.getPeerConnection().addTransceiver(mediaType, init); + + List codecs = new ArrayList<>( + factory.getRtpSenderCapabilities(mediaType).getCodecs()); + codecs.sort(Comparator.comparing(capability -> !codec.equalsIgnoreCase(capability.getName()))); + transceiver.setCodecPreferences(codecs); + + return transceiver; + } + + private static void negotiate(TestPeerConnection caller, TestPeerConnection callee) throws Exception { + caller.setRemotePeerConnection(callee); + callee.setRemotePeerConnection(caller); + + callee.setRemoteDescription(caller.createOffer()); + caller.setRemoteDescription(callee.createAnswer()); + + caller.waitUntilConnected(); + callee.waitUntilConnected(); + } + + /** + * Returns the receiver of the callee's only transceiver, the caller's to + * dispose. + */ + private static RTCRtpReceiver getReceiver(TestPeerConnection callee) { + RTCRtpTransceiver[] transceivers = callee.getPeerConnection().getTransceivers(); + RTCRtpReceiver receiver = transceivers[0].getReceiver(); + + for (RTCRtpTransceiver transceiver : transceivers) { + transceiver.dispose(); + } + + return receiver; + } + + + + private static class CapturedFrame { + + final byte[] data; + final boolean keyFrame; + final int width; + final int height; + + + CapturedFrame(RTCEncodedFrame frame, boolean keyFrame, int width, int height) { + ByteBuffer payload = frame.getData(); + + this.data = new byte[payload.remaining()]; + this.keyFrame = keyFrame; + this.width = width; + this.height = height; + + payload.get(data); + } + } + +} diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/RTCPeerConnectionTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/RTCPeerConnectionTests.java index df80bf21..10255d05 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/RTCPeerConnectionTests.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/RTCPeerConnectionTests.java @@ -18,6 +18,8 @@ import static org.junit.jupiter.api.Assertions.*; +import dev.onvoid.webrtc.media.MediaStreamTrack; +import dev.onvoid.webrtc.media.MediaType; import dev.onvoid.webrtc.media.audio.AudioOptions; import dev.onvoid.webrtc.media.audio.AudioTrackSource; import dev.onvoid.webrtc.media.audio.AudioTrack; @@ -211,7 +213,10 @@ void removeTrack() { @Test void addTransceiverNullParams() { assertThrows(NullPointerException.class, () -> { - peerConnection.addTransceiver(null, null); + peerConnection.addTransceiver((MediaStreamTrack) null, null); + }); + assertThrows(NullPointerException.class, () -> { + peerConnection.addTransceiver((MediaType) null, null); }); }