[alden-sift]

What was changed

CSV, Parquet, TDMS and HDF5 import configs can now define bit field channels. Before this, DataColumnBase had no bit_field_elements field, so a config from detect_config couldn't describe a BIT_FIELD channel.

What's a bit field

A bit field is one integer that packs several small values into fixed bit ranges. Hardware status words are the common case. In this 8-bit status value, bit 0 says whether the vehicle is armed, bits 1 to 3 hold an error code, and bits 4 to 7 hold a mode:

bit:   7 6 5 4 | 3 2 1 | 0
       mode    | error | armed

The raw value 53 (0b0011_0101) decodes to armed=1, error_code=2, mode=3. Sift splits a BIT_FIELD channel into one sub-channel per element (status.armed, status.error_code, status.mode), but it needs the layout to do that. Each element is a ChannelBitFieldElement with a name, a starting bit index and a bit_count width. detect_config can't infer the layout, so the user supplies it:

status = config["status"]
status.data_type = ChannelDataType.BIT_FIELD
status.bit_field_elements = [
    ChannelBitFieldElement(name="armed", index=0, bit_count=1),
    ChannelBitFieldElement(name="error_code", index=1, bit_count=3),
    ChannelBitFieldElement(name="mode", index=4, bit_count=4),
]

Value formats

In CSV and Parquet, a bit field value must be raw bytes, a string of 0s and 1s, or URL-safe base64. Bytes are little-endian. Integers aren't accepted, and Parquet has no bit field type, so the client honors the config and expects the data in one of those formats. It doesn't rewrite files.

TDMS and HDF5 store typed integers, and an integer channel imports as its little-endian bytes, so those files need no conversion.

import_from_path checks CSV and Parquet BIT_FIELD columns before uploading, so a wrong format fails before the file is sent:

• A Parquet column must be a binary or string type. Integer, list and other types raise, naming the column and type and showing how to convert. This reads only the schema.
• For CSV columns and Parquet string columns, a sample of the first 1,000 values must be in one of the accepted formats. Any value that parses passes, including all-digit ones like 10 (bit string) or 1234 (base64), since the client can't tell those from intended input.
• Values past the sample aren't checked client side.

Other changes

• DataColumnBase gets bit_field_elements. A validator requires it when data_type is BIT_FIELD and rejects it otherwise. It also rejects duplicate names, negative indexes, zero-width elements and overlapping ranges.
• ULog and MCAP columns reject BIT_FIELD, since those imports don't support bit field overrides. The field is on DataColumnBase, so UlogDataColumn and McapDataColumn opt out through a _supports_bit_fields class flag, and the validator raises when they're set to BIT_FIELD.
• _channel_config_proto() writes the elements, and every _from_proto reads them back, including the embedded Parquet config path.
• _channel_config_proto() re-runs the enum and bit field checks. Pydantic doesn't validate on assignment, so an in-place edit like the one above could otherwise send a BIT_FIELD channel with no elements.
• MCAP configs no longer drop enum_types. MCAP built its channel config by hand. It now uses the shared helper and reads enums back.
• The Parquet flat dataset example has a commented bit field snippet.

Multi-channel single-channel-per-row Parquet still doesn't support bit fields, same as enums, because that layout derives channel configs per row.

Testing

• Unit tests cover config round trips for CSV, Parquet, TDMS and HDF5, ULog and MCAP rejecting BIT_FIELD at construction and on in-place edits, each validation error, in-place edits caught at import, and the pre-upload checks: accepted and rejected Parquet types, nested fields, sampled CSV and Parquet string values, unpadded bit strings and digit-only base64 passing, and a failing check stopping the import before anything is created or uploaded.
• Integration tests ran against a local stack. Parquet binary, fixed_size_binary, bit string and base64 string columns, a 16-bit Parquet status word, and CSV bit strings and base64, and TDMS and HDF5 uint8, uint16 and int16 channels all import and decode to the expected element values. The 16-bit cases decode 0x1234 to low=0x34, high=0x12, so multi-byte values are little-endian.

Verification on a local stack

Each case wrote a synthetic file, imported it into a local stack and read the elements back:

config = client.data_import.detect_config(path, data_type=DataTypeKey.PARQUET_FLATDATASET)  # or detect_config(path) for CSV
status = config["status"]
status.data_type = ChannelDataType.BIT_FIELD
status.bit_field_elements = [
    ChannelBitFieldElement(name="armed", index=0, bit_count=1),
    ChannelBitFieldElement(name="error_code", index=1, bit_count=3),
    ChannelBitFieldElement(name="mode", index=4, bit_count=4),
]

job = client.data_import.import_from_path(path, config=config)
job.wait_until_complete()
run = client.data_import.get_run(job.get_data_import().id_)
channel = client.channels.list_(run=run, name="status")[0]
data = client.channels.get_data(channels=[channel], run=run)  # status.armed, status.error_code, status.mode

Accepted formats. All six imports succeeded and created a BIT_FIELD channel with the three elements.

Status word	Parquet binary	Parquet fixed_size_binary[1]	Parquet string, bits	Parquet string, base64	CSV, bits	CSV, base64	Returned (armed, error_code, mode)
53	0x35	0x35	00110101	NQ==	00110101	NQ==	1, 2, 3
0	0x00	0x00	00000000	AA==	00000000	AA==	0, 0, 0
255	0xff	0xff	11111111	_w==	11111111	_w==	1, 7, 15
1	0x01	0x01	00000001	AQ==	00000001	AQ==	1, 0, 0

All six returned identical values, matching the hand-decoded expectation.

16-bit status word.

values = [0x1234, 0x00FF, 0xFF00, 0x0001]
status = pa.array([v.to_bytes(2, "little") for v in values], pa.binary())

config["status"].bit_field_elements = [
    ChannelBitFieldElement(name="low", index=0, bit_count=8),
    ChannelBitFieldElement(name="high", index=8, bit_count=8),
]

Status word	Bytes written	Returned (low, high)
0x1234	34 12	52, 18
0x00ff	ff 00	255, 0
0xff00	00 ff	0, 255
0x0001	01 00	1, 0

TDMS and HDF5 integer channels. Each file held one integer status channel, set to BIT_FIELD after detect_config:

values = np.array([0x1234, -1, -256, 1], np.int16)  # also uint8 [53, 0, 255, 1] and uint16 [0x1234, 0x00FF, 0xFF00, 1]

# TDMS
properties = {"wf_start_time": np.datetime64(start_ns, "ns"), "wf_increment": 1.0, "wf_start_offset": 0.0}
writer.write_segment([RootObject(), GroupObject("g"), ChannelObject("g", "status", values, properties=properties)])
config = client.data_import.detect_config(path)  # column "g.status"

# HDF5
f["time"] = np.array([start_ns + i * 1_000_000_000 for i in range(len(values))])
f["status"] = values
config = client.data_import.detect_config(path, data_type=DataTypeKey.HDF5_ONE_D)
config.time_format = TimeFormat.ABSOLUTE_UNIX_NANOSECONDS

TDMS and HDF5 returned identical values:

Channel	Values written	Returned
uint8	53, 0, 255, 1	armed 1, 0, 1, 1; error_code 2, 0, 7, 0; mode 3, 0, 15, 0
uint16	0x1234, 0x00ff, 0xff00, 0x0001	low 52, 255, 0, 1; high 18, 0, 255, 0
int16	0x1234, -1, -256, 1	low 52, 255, 0, 1; high 18, 255, 255, 0

Rejected before upload. Each raised a ValueError before an import was created. A bit field value is never read as a decimal integer, so a CSV cell 53 isn't a bit string or valid base64. It has to be written as 00110101, NQ== or the byte 0x35.

File	Values
Parquet uint8	53, 0, 255, 1
Parquet list<uint8>	[53], [0], [255], [1]
CSV	53, 0, 255, 1

Parquet uint8:

bit field column 'status' is uint8, but bit field values must be raw bytes, a string of 0s and 1s like '00110101', or URL-safe base64. Store the column as binary, for example pa.array([int(v).to_bytes(2, 'little') for v in values], pa.binary()) for a 16-bit status word.

Parquet list<uint8>:

bit field column 'status' is list<element: uint8>, but bit field values must be raw bytes, a string of 0s and 1s like '00110101', or URL-safe base64. Store the column as binary, for example pa.array([int(v).to_bytes(2, 'little') for v in values], pa.binary()) for a 16-bit status word.

CSV:

bit field column 'status' has value '53' in row 2, which isn't raw bytes, a string of 0s and 1s like '00110101', or URL-safe base64. To convert integers, use format(value, '08b') for an 8-bit status word.

[github-actions]

Python docs preview: https://sift-stack.github.io/sift/python/pr-861/

Deployed from 93022f5. The link may take up to a minute to become live as GitHub Pages propagates.

[alexluck-sift]

is this the right place for this?

[alden-sift]

I believe so, but it can be better prefaced with something like "For files containing bit field datatype columns, …". Otherwise, users using the import_from_path method for such files won't discover this rule without digging into bit field parsing code.

[alden-sift]

improved wording in in cb935df.

[alexluck-sift]

Not blocking: do you expect users to need units, a description or signedness per element? Elements only carry a name, index and width, so that would be a proto change to ChannelBitFieldElement. Worth tracking separately if so?

[alexluck-sift]

Elements only carry a name, index and width, so units and description here describe the whole status word, and from your tables each element comes back as an unsigned int. Does the import service put the parent's units on status.armed and the other sub-channels? Either way, I'd say both things in this docstring and the changelog, since a user setting units on a bit field column will expect it per element.

[alden-sift]

Tested on dev - the parent's units and descriptions do show up for each sub-channel. Docstring and changelog is updated accordingly in cb935df

[alden-sift]

I can see a need for each element to have its own unit and description. Can't make an informed comment on signedness, but quick google says that it is implementable. I can make the follow up issue in linear.

[alexluck-sift]

UlogDataColumn rejects bit fields, so this read can only return None or make _from_proto raise on a proto that carries elements. Same at the MCAP read below. I'd drop it in both places and keep the enum_types read.

Suggested change

bit_field_elements=_bit_field_elements_from_proto(d.channel_config),

[alden-sift]

applied in dbc5c41.

[alexluck-sift]

Same here, since McapDataColumn rejects bit fields too.

Suggested change

bit_field_elements=_bit_field_elements_from_proto(d.channel_config),

[alden-sift]

applied in dbc5c41.

[alexluck-sift]

These Parquet helpers re-import pyarrow per function and copy the missing-pyarrow message from detect_config. _internal/util/parquet.py already imports pyarrow at the top and owns PATH_SEPARATOR, so I'd move _check_parquet, _field_type and _column_values there and keep one guarded import here, the same way detect_config imports detect_flat_parquet_config.

[alden-sift]

applied in a865d0c.