Repository navigation
audio: ffmpeg_dec: add MP3 and AAC-LC encoder mode (libshine, vo-aacenc) and capture topologies - #11414
Draft
lgirdwood wants to merge 4 commits into
Draft
audio: ffmpeg_dec: add MP3 and AAC-LC encoder mode (libshine, vo-aacenc) and capture topologies#11414lgirdwood wants to merge 4 commits into
lgirdwood wants to merge 4 commits into
Conversation
Add the SOF forks of libshine (fixed-point MP3 encoder) and vo-aacenc (VisualOn fixed-point AAC-LC encoder) under modules/audio/shine and modules/audio/vo-aacenc, pinned to their sof-hifi branches on thesofproject. shine (sof-hifi): ab594a0 shine: add Xtensa HiFi accelerated multipliers and optimize quantize loop a531d18 shine: work around clang/Xtensa backend miscompile in rate-control loop b917fc4 shine: precompute init tables to avoid runtime pow/sqrt on the DSP vo-aacenc (sof-hifi): c823d30 xtensa: optimize basic operators, band energy, and autocorrelation Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…capabilities Advertise SND_AUDIOCODEC_MP3 (guarded by !CONFIG_CADENCE_CODEC_MP3_ENC) and SND_AUDIOCODEC_AAC for SOF_IPC_STREAM_CAPTURE in IPC4_SOF_CODEC_INFO when CONFIG_FFMPEG_ENC_MP3 or CONFIG_FFMPEG_ENC_AAC is enabled. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
lgirdwood
commented
Nov 6, 2026
lgirdwood
left a comment
lgirdwood
left a comment
Member
Author
There was a problem hiding this comment.
Lets also split the media format encoding logic into separate files like we do with decode.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical encoder defects can cause out-of-bounds reads, exhaust the MP3 capture heap, and carry prior-recording audio into restarted AAC streams.
8 open issues
What changed in this PR
This PR extends SOF’s FFmpeg module with a build-time encoder mode for MP3 and AAC-LC, adding compressed HDA capture alongside the existing decoder infrastructure.
Changes:
- Adds fixed-point encoding through pinned libshine and vo-aacenc dependencies.
- Introduces encoder configuration, processing callbacks, and capture capability reporting.
- Adds HDA capture routing, reusable topology classes, and development targets.
| File | Description |
|---|---|
west.yml |
Pins libshine and vo-aacenc dependencies. |
tools/topology/topology2/sof-hda-generic.conf |
Includes capture pipeline classes. |
tools/topology/topology2/platform/intel/compr-default.conf |
Defines encoder IDs and resource budgets. |
tools/topology/topology2/platform/intel/compr-capture-ffmpeg.conf |
Instantiates compressed capture and routing. |
tools/topology/topology2/include/pipelines/cavs/mixout-gain-host-copier-capture.conf |
Removes the class-level gain-to-copier route. |
tools/topology/topology2/include/pipelines/cavs/mixout-capture.conf |
Adds a standalone timer-scheduled mixout. |
tools/topology/topology2/include/pipelines/cavs/compr-capture-ffmpeg.conf |
Defines the encoder-to-host capture pipeline. |
tools/topology/topology2/include/components/ffmpeg_enc.conf |
Defines the encoder widget using the existing module UUID. |
tools/topology/topology2/development/tplg-targets.cmake |
Adds MP3 and AAC capture targets. |
tools/topology/topology2/cavs-mixin-mixout-hda.conf |
Adds conditional mixer-based capture fan-out. |
src/kernel/audio/base_fw.c |
Advertises MP3 and AAC capture capabilities. |
src/audio/ffmpeg_dec/llext/CMakeLists.txt |
Adds encoder library build dependencies. |
src/audio/ffmpeg_dec/Kconfig |
Adds encoder selections and build mode. |
src/audio/ffmpeg_dec/ffmpeg.cmake |
Cross-builds the encoder libraries. |
src/audio/ffmpeg_dec/ffmpeg_dec.h |
Declares encoder callbacks and shared copy helpers. |
src/audio/ffmpeg_dec/ffmpeg_dec.c |
Selects encoder callbacks and shares buffer helpers. |
src/audio/ffmpeg_dec/ffmpeg_dec-encode.c |
Implements PCM conversion and compressed encoding. |
src/audio/ffmpeg_dec/CMakeLists.txt |
Integrates encoder sources, headers, and libraries. |
Review effort: Balanced
lgirdwood
force-pushed
the
topic/ffmpeg-encoders
branch
2 times, most recently
from
November 7, 2026 14:21
c8f3e90 to
d5281ae
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Encoder paths silently accept requests they cannot honor for format, bitrate, sample rate, channel count, and channel mode, while the HDA routing bypasses configured microphone processing.
9 open issues
4 resolved since last review
Review effort: Balanced
lgirdwood
commented
Nov 7, 2026
lgirdwood
left a comment
lgirdwood
left a comment
Member
Author
There was a problem hiding this comment.
We need to simplify the naming so the encoders use the same file naming conventions as the decoders. See comments.
lgirdwood
force-pushed
the
topic/ffmpeg-encoders
branch
from
November 7, 2026 14:55
d5281ae to
9aef131
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical regressions prevent passthrough-stub initialization and attenuate IPC4 24-bit recordings by about 48 dB.
3 open issues
9 resolved since last review
Review effort: Balanced
|
|
||
| cd->out_channels = source_get_channels(src); | ||
| cd->out_rate = source_get_rate(src); | ||
| e->src_fmt = source_get_frm_fmt(src); |
Member
Author
There was a problem hiding this comment.
Fixed: ffmpeg_enc_configure() now uses source_get_frm_fmt(src) for the container byte-width check and source_get_valid_fmt(src) for e->src_fmt sample conversion.
| } | ||
| #endif | ||
|
|
||
| if (!cd->codec_ops) { |
Member
Author
There was a problem hiding this comment.
Fixed: included defined(CONFIG_COMP_FFMPEG_DEC_STUB) in FFMPEG_DEC_DEFAULT_CODEC and guarded the !cd->codec_ops check with #if !defined(CONFIG_COMP_FFMPEG_DEC_STUB).
| mp3->avctx->sample_rate = cd->out_rate; | ||
| av_channel_layout_default(&mp3->avctx->ch_layout, cd->out_channels); | ||
| mp3->avctx->sample_fmt = AV_SAMPLE_FMT_S16P; | ||
| mp3->avctx->bit_rate = e->bit_rate; |
Member
Author
There was a problem hiding this comment.
Fixed: ffmpeg_enc_mp3_validate_params() and ffmpeg_enc_mp3_configure() now reject bitrates that are not whole multiples of 1000 bps (bit_rate % 1000 != 0).
…acenc) Add compress capture encoder support (CONFIG_FFMPEG_ENC) to the ffmpeg_dec module, split into a generic encode core (ffmpeg-encode.c) and per-codec encoder files behind struct ffmpeg_enc_codec_ops: - Fixed-point MP3 encoding via libshine (CONFIG_FFMPEG_ENC_MP3, ffmpeg-enc-mp3.c, 1152-sample MPEG-1 Layer III frames). - Fixed-point AAC-LC encoding via vo-aacenc (CONFIG_FFMPEG_ENC_AAC, ffmpeg-enc-aac.c, 1024-sample ADTS frames). Key details: - Dispatches each module instance between playback (decode) and capture (encode) at runtime from the IPC4 compress stream direction (SOF_IPC_STREAM_PLAYBACK vs SOF_IPC_STREAM_CAPTURE), allowing a single firmware binary to include both decoders and encoders simultaneously. - Selects the encoder codec_ops (SND_AUDIOCODEC_MP3 vs SND_AUDIOCODEC_AAC), validates stream format/profile/mode, rate_control, sample_rate, channel count, and bitrate from IPC4 snd_codec parameters at runtime. - Validates and converts S16_LE, S24_4LE, and S32_LE input PCM formats and recreates encoder handles across stream resets to avoid leaking inter-stream state. - Cross-builds libshine and vo-aacenc archives in ffmpeg.cmake with explicit source/header dependency tracking for both built-in (=y) and LLEXT (=m) builds. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add compressed audio capture (encode) topology support for the FFmpeg encoder widget (ffmpeg_enc): - include/components/ffmpeg_enc.conf: encoder widget class sharing the ffmpeg_dec UUID with DAPM widget type 'encoder'. - include/pipelines/cavs/compr-capture-ffmpeg.conf: DP encoder + aif_out host-copier capture pipeline. - include/pipelines/cavs/mixout-capture.conf: standalone timer-scheduled mixout pipeline for cross-pipeline LL->DP capture fan-out. - platform/intel/compr-capture-ffmpeg.conf & compr-default.conf: COMPR_FFMPEG_ENC capture PCM (id 51, pipeline id 91, mixout pipeline id 93) and per-instance stack/heap sizing (64 KiB stack, 384 KiB heap). - cavs-mixin-mixout-hda.conf & development/tplg-targets.cmake: mixin capture fan-out when COMPR_FFMPEG_ENC is enabled, plus sof-hda-generic-ffmpeg-mp3-enc-compr and sof-hda-generic-ffmpeg-aac-enc-compr development topology targets. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
lgirdwood
force-pushed
the
topic/ffmpeg-encoders
branch
from
November 7, 2026 15:46
9aef131 to
9c95b7a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR (Phase 3 of the FFmpeg integration series, following #11308, #11334, and #11390) adds fixed-point audio encode support (
CONFIG_FFMPEG_DEC_ENCODE_MODE) and compressed capture topologies to theffmpeg_decmodule:west: manifest: add shine and vo-aacenc projectsshine(thesofproject/shine,sof-hifibranchab594a0) andvo-aacenc(thesofproject/vo-aacenc,sof-hifibranchc823d30) undermodules/audio/shineandmodules/audio/vo-aacenc.audio: base_fw: advertise FFmpeg MP3 and AAC compress capture encode capabilitiesSND_AUDIOCODEC_MP3(guarded by!CONFIG_CADENCE_CODEC_MP3_ENC) andSND_AUDIOCODEC_AACforSOF_IPC_STREAM_CAPTUREinIPC4_SOF_CODEC_INFOwhenCONFIG_FFMPEG_ENC_MP3/CONFIG_FFMPEG_ENC_AACare enabled.audio: ffmpeg_dec: add MP3 and AAC-LC encoder mode (libshine, vo-aacenc)src/audio/ffmpeg_dec/ffmpeg_dec-encode.cdriving fixed-point MP3 encoding (libshine, 1152-sample frames) and fixed-point AAC-LC ADTS encoding (vo-aacenc, 1024-sample frames) over the DP sink/source.processinterface.struct snd_codecparameters when provided, rejecting unsupported/disabled codec IDs with-ENOTSUP, and falls back to Kconfig defaults (CONFIG_FFMPEG_ENC_DEFAULT_AAC/CONFIG_FFMPEG_ENC_DEFAULT_MP3) when no IPC codec ID is specified.libshine(shine_ext) andvo-aacenc(vo_aacenc_ext) inffmpeg.cmakefor both built-in (=y) and LLEXT (=m) builds.topology2: add compressed audio capture pipelines for FFmpeg encodersffmpeg_enc.conf,compr-capture-ffmpeg.conf, andmixout-capture.confto tap the HDA analog capture BE via a standalone timer-scheduledmixoutinto the DPffmpeg_encpipeline (PCM ID 51, pipeline ID 91, mixout pipeline ID 93).sof-hda-generic-ffmpeg-mp3-enc-comprandsof-hda-generic-ffmpeg-aac-enc-comprdevelopment topology targets.