Fix AudioFrameV3Wrapper::get_data() to return correct float32 audio data - #45
Conversation
## Problem
AudioFrameV3Wrapper::get_data() wraps NDI's float32 planar audio data
as a uint8 numpy array with incorrect strides, making all audio
received via framesync_capture_audio_v2() unusable in Python.
Specifically:
- The returned numpy array has dtype=uint8 and shape (channels, samples)
- tobytes() returns only 1/4 of the actual audio data (1024 bytes
instead of 4096 for a 1024-sample mono frame)
- The array's internal buffer is a COPY that is too small; reading
past it via ctypes returns garbage (uninitialized memory)
- All samples appear as NaN or silent corruption
## Root Cause
AudioFrameV3Wrapper::get_data() uses sizeof(uint8_t) as the element
size and format_descriptor<uint8_t> as the numpy format, but the
underlying NDI audio data is always FLTP (float planar, 32-bit).
Meanwhile, AudioFrameV2Wrapper::get_data() correctly uses
sizeof(float) and format_descriptor<float> — V3 is a copy-paste error.
## The Fix
Change AudioFrameV3Wrapper::get_data() to match AudioFrameV2Wrapper:
Before (buggy):
size = sizeof(uint8_t)
format_descriptor<uint8_t>
stride: {col * size * 4, size} // col * 4 bytes, 1 byte
After (fixed):
size = sizeof(float)
format_descriptor<float>
stride: {col * size, size} // col * 4 bytes, 4 bytes
This changes the numpy array dtype from uint8 to float32 and fixes
the stride calculation so the full audio data is accessible.
## Why This Is Correct
1. NDI SDK guarantees FLTP format: the FourCC field on AudioFrameV3
is always FOURCC_AUDIO_TYPE_FLTP (float planar). There are no
other audio FourCC types in the current SDK.
2. Matches AudioFrameV2Wrapper: the V2 wrapper already handles this
correctly. V3 should match.
3. Matches DistroAV (OBS NDI plugin): DistroAV's C++ code accesses
p_data directly as float* with channel_stride_in_bytes offsets —
confirming the data is float32.
4. Verified on real hardware: tested with NDI Camera Android app
sending mono 48kHz audio. After the fix, framesync_capture_audio_v2()
returns clean float32 data with shape (channels, samples),
dtype=float32, no NaN values, and correct amplitude range.
## Reproduction
Before fix:
import NDIlib as ndi
import numpy as np
ndi.initialize()
recv = ndi.recv_create_v3()
# ... connect to source ...
fs = ndi.framesync_create(recv)
af = ndi.framesync_capture_audio_v2(fs, 0, 0, 1024)
print(af.data.dtype) # uint8 (WRONG)
print(af.data.shape) # (1, 1024) — misleading, only 256 float32 samples fit
print(af.data.nbytes) # 1024 (WRONG, should be 4096)
print(np.any(np.isnan(af.data))) # True (WRONG)
After fix:
print(af.data.dtype) # float32 (CORRECT)
print(af.data.shape) # (1, 1024)
print(af.data.nbytes) # 4096 (CORRECT)
print(np.any(np.isnan(af.data))) # False (CORRECT)
print(af.data.max()) # ~0.002 (real audio from phone mic)
## Impact
- Breaking change: No. The array shape stays (channels, samples).
Only the dtype changes from uint8 to float32, which is the
correct representation of the data.
- Backward compatibility: Code that treated the array as raw bytes
via tobytes() will get different output, but that was already
broken (only returned 1/4 of the data).
- Performance: No change. Same memory allocation, same data copy.
Fixed incorrect parenthesis count in py::array(py::buffer_info(...)) call (extra closing paren) and corrected size to sizeof(float) to match the float format descriptor, ensuring proper stride and item size for float32 audio data.
|
@ichux Thank you for your contribution! |
I'm happy to help. Thanks for blazing the trail in the lib as it's helpful. I just got on NDI and found it doing all I needed. |
|
@ichux Thanks for tracking this down and for the clear fix.
Given that change, there are two related places that should also be updated to keep the wrapper consistent. Would you mind including the following small changes in this PR before I merge it? First, please update the setter side to use - py::array_t<uint8_t> data_array;
+ py::array_t<float> data_array;
- void set_data(py::array_t<uint8_t> arr) {
+ void set_data(py::array_t<float> arr) {
- [](AudioFrameV3Wrapper &s, py::array_t<uint8_t> arr) {
+ [](AudioFrameV3Wrapper &s, py::array_t<float> arr) {Otherwise, assigning Second, please make the empty-frame return type consistent as well: - return py::array_t<uint8_t>();
+ return py::array_t<float>();While reviewing this code, I also found a couple of other related issues. They are outside the scope of this PR, so after this is merged, I’ll take care of those in a follow-up before making the next release. Thanks again for the contribution! |
I'm glad I could help! I have done the fixes. Please, whatever you need me doing, just let me know and I will do it. |
Changed AudioFrameV3Wrapper to consistently use float32 for the public FLTP payload: data_array type, set_data parameter and static_cast, get_data return type, and the py::class_ binding. Fixed extra closing parenthesis in py::array(py::buffer_info(...)) call that broke compilation. sizeof(float) now matches the float format descriptor for correct stride calculation.
Changed AudioFrameV3Wrapper to consistently use float32 for the public FLTP payload: data_array type, set_data parameter and static_cast, get_data return type, and the py::class_ binding. Fixed extra closing parenthesis in py::array(py::buffer_info(...)) call that broke compilation. sizeof(float) now matches the float format descriptor for correct stride calculation.
|
@buresu Heads up, the full build is broken before this PR. pybind11's remove_class fails on every m.def lambda that takes a py::capsule parameter with additional args. I hit it when trying to compile locally. Specifically:
I isolated my changes in a standalone test (if you want it, please let me know and I will zip it to you) as it compiles and runs fine. The regression is in the existing binding code, not in this PR. I guess that's what you might have discovered and intend to address. |
|
@ichux Thank you for the fix! Since I’m unable to reproduce the issue with building the main branch either in CI or in my local environment, I suspect it may be environment-specific. Could you let me know what environment you are using to build the project? |
|
@ichux I’ve released v6.3.2.4, which includes this PR and the audio-related fixes. You can update to the latest version with: |
I work with Python 3.14.4. I don't know if that helps. Like you said, it might be my end because the CI passed. I noticed you compiled on Python 3.12. It doesn't exist there! |
|
@ichux If the build issue requires a fix, please open a separate issue for it. |
@buresu The build issue was a local environment problem: my NDI SDK was outdated and I've since installed the correct version (6.3.2.0) and the project builds and imports cleanly. No separate issue needed. |
2 participants
Problem
AudioFrameV3Wrapper::get_data() wraps NDI's float32 planar audio data as a uint8 numpy array with incorrect strides, making all audio received via framesync_capture_audio_v2() unusable in Python.
Specifically:
Root Cause
AudioFrameV3Wrapper::get_data() uses sizeof(uint8_t) as the element size and format_descriptor<uint8_t> as the numpy format, but the underlying NDI audio data is always FLTP (float planar, 32-bit).
Meanwhile, AudioFrameV2Wrapper::get_data() correctly uses sizeof(float) and format_descriptor: V3 is a copy-paste error.
The Fix
Change AudioFrameV3Wrapper::get_data() to match AudioFrameV2Wrapper:
Before (buggy):
size = sizeof(uint8_t)
format_descriptor<uint8_t>
stride: {col * size * 4, size} // col * 4 bytes, 1 byte
After (fixed):
size = sizeof(float)
format_descriptor
stride: {col * size, size} // col * 4 bytes, 4 bytes
This changes the numpy array dtype from uint8 to float32 and fixes the stride calculation so the full audio data is accessible.
Why This Is Correct
NDI SDK guarantees FLTP format: the FourCC field on AudioFrameV3 is always FOURCC_AUDIO_TYPE_FLTP (float planar). There are no other audio FourCC types in the current SDK.
Matches AudioFrameV2Wrapper: the V2 wrapper already handles this correctly. V3 should match.
Matches DistroAV (OBS NDI plugin): DistroAV's C++ code accesses p_data directly as float* with channel_stride_in_bytes offsets: confirming the data is float32.
Verified on real hardware: tested with NDI Camera Android app sending mono 48kHz audio. After the fix, framesync_capture_audio_v2() returns clean float32 data with shape (channels, samples), dtype=float32, no NaN values, and correct amplitude range.
Reproduction
Before fix:
import NDIlib as ndi
import numpy as np
ndi.initialize()
recv = ndi.recv_create_v3()
... connect to source ...
fs = ndi.framesync_create(recv)
af = ndi.framesync_capture_audio_v2(fs, 0, 0, 1024)
print(af.data.dtype) # uint8 (WRONG)
print(af.data.shape) # (1, 1024) — misleading, only 256 float32 samples fit
print(af.data.nbytes) # 1024 (WRONG, should be 4096)
print(np.any(np.isnan(af.data))) # True (WRONG)
After fix:
print(af.data.dtype) # float32 (CORRECT)
print(af.data.shape) # (1, 1024)
print(af.data.nbytes) # 4096 (CORRECT)
print(np.any(np.isnan(af.data))) # False (CORRECT)
print(af.data.max()) # ~0.002 (real audio from phone mic)
Impact