Skip to content

Feat/q decode cols - #22

Merged
vbanas merged 3 commits into
masterfrom
feat/q-decode-cols
Oct 20, 2026
Merged

vbanas merged 3 commits into
masterfrom
feat/q-decode-cols

Conversation

@vbanas

@vbanas vbanas commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

Move the vendored rayforce-q pin from df8ced9 (2.4.0) to c7582d2, the
merge of its PR #23, whose tree is the "Release 2.5.0" commit 0f0dfd2.
rayforce-q has pushed no 2.5.0 tag, so it is pinned by commit as
before. The core stays at v2.11.5.

2.5.0 adds q_decode_cols: a table message decoded keeping only the
named columns, in the order named, as q's cols#t. The others are
skipped by their serialized length, so no STR is allocated per string
cell and no symbol is interned. rayforce-sys declares it, and
q::decode_response_cols wraps it with the same header checks as
q::decode_response, which now share them. It takes &[S: AsRef<str>],
so &[&str] and &[String] both work. A name the table lacks is an Err
naming it, as are a repeated name and one holding a NUL. A keyed table
keeps its key table whole; a non-table message or an empty name list
decodes as decode_response does.

The rest of q.c's change is a fix and two refactors. q_build_table no
longer releases a table ray_table_add_col already consumed, a double
release reachable from the wire on every decode path. q_decode and the
dict construction move into helpers both decoders share.

build.rs refuses a RAYFORCE_Q_SRC checkout whose q.h lacks
q_decode_cols, naming 2.5.0, where a caller would have failed to link.
decode_response's error buffer is c_char now; it was i8, which does
not type-check where c_char is unsigned (aarch64 Linux).

Through the binding, on a synthetic 100 000-row, 142-column table kept
to 23 columns (release build, best of 5): 32 ms, against 155 ms for
decode_response of the whole message and 25 ms for a message holding
only those columns.

cargo test --workspace passes on the local checkouts, on the vendored
pins, and on the debug core with RAY_DFD=1, each with the IPC tests
against a v2.11.5 server (RAYFORCE_REQUIRE_SERVER=1). clippy is clean
at -D warnings, fmt and rustdoc report nothing, and the packaged
rayforce-sys carries q.c and q.h.
@morsel-ray

morsel-ray commented Oct 20, 2026 •

Copy link
Copy Markdown

Follow-up: the locator fix in4447d7d7 is verified in actual CI logs. Both jobs now load /usr/lib/llvm-18/lib/libclang.so and set the correct C-API directory; all seven new projected-decode tests explicitly pass in each flavour. The debug/release archive checks report2/0 DFD symbols respectively, and the server-required environment is set. No failed test summary appears in either job. These are inspected remote results, not a local Rust run or live-q interoperability certification.

I also checked the source update: relative to5d72 it changes the CI/release locator plus README/changelog, not the binding implementation. The maintainer has now merged this as b54f5ae; its complete tree matches checked4447d7d7. I did not perform that merge or submit a retrospective approval.


Initial5d72 triage, retained for the failure's provenance:

I read both failed CI job logs on5d72a0fc. This is currently a build-environment/libclang-discovery failure, not a failing decoder/IPC test:

  • debug job118720889708 and release job118720890251 both set LIBCLANG_PATH=/usr/lib/llvm-18/lib/clang/18/lib/linux;
  • bindgen0.70.1 then reports no usable libclang.so/libclang-*.so there and the build script exits101, before the test suite runs.

The workflow already installs libclang-dev. The problem is the broad find /usr/lib -name 'libclang*.so*' | head -1: it can select a compiler-runtime library (libclang_rt.*) rather than the libclang C API. The selected nested runtime directory in both logs fits that diagnosis.

Please resolve the actual C-API library (e.g. an appropriate llvm-config --libdir, or filter libclang.so* / libclang-[0-9]*.so* explicitly), and verify it can load/export clang_createIndex before invoking cargo. A fresh build on both flavours is still required; rerunning the same broad search until it picks another file would not repair the setup. I haven't run the Rust suite locally or reviewed/approved the new binding yet.

For scope: master remains7633c118/1.3.1;5d72 is this feature PR with proposed1.3.2. This red PR is not evidence that the existing default release has become broken.

The CI and release workflows set LIBCLANG_PATH to the directory of the
first file `find /usr/lib -name 'libclang*.so*'` returned. That
pattern also matches libclang-cpp and LLVM's compiler runtime,
llvm-<v>/lib/clang/<v>/lib/linux/libclang_rt.*.so. On the runners of
2026-10-20 the runtime came first, so both CI legs pointed bindgen at
/usr/lib/llvm-18/lib/clang/18/lib/linux, which holds no libclang, and
the rayforce-sys build script panicked with "Unable to find libclang"
before any test ran.

The step now matches only libclang.so and libclang-<version>.so names,
tries the installed clang's /usr/lib/llvm-<major>/lib/libclang.so
first and then the rest newest first, and takes the first one that
loads and exports clang_createIndex. With none, it fails with an
error naming that. The README's Linux example had the same search; it
now uses the same pattern.

Reproduced locally: LIBCLANG_PATH set to that runtime directory gives
the same bindgen panic. The new step picks /usr/lib/llvm-18/lib, and
/usr/lib/x86_64-linux-gnu when the preferred path is absent; a
libclang_rt or libclang-cpp library fails its load check.
@vbanas
vbanas merged commit b54f5ae into master Oct 20, 2026
2 checks passed
@vbanas
vbanas deleted the feat/q-decode-cols branch October 20, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants