Repository navigation
Feat/q decode cols - #23
Conversation
A caller that keeps a few columns of a wide table paid q_decode for all of them: one STR allocation per string cell and one intern per symbol cell, then freed what it did not want. q_decode_cols takes the names to keep and returns those columns in that order, as q's cols#t does. The other columns are skipped by their serialized length: symbol terminators are counted eight bytes at a time, string cells are stepped over inline, and nothing is allocated or interned. Skipping reads only lengths, so a timespan column, which q_decode rejects, can be dropped. For a keyed table the key comes back whole and the names pick among the value columns. A name the table lacks is a RAY_ERROR naming it; a NULL or repeated name returns NULL with err set. With no names, or for a message that is not a table, it is q_decode. Kept columns go through the same q_des_obj call q_decode makes. On a synthetic 100 000-row, 142-column window kept to 23 columns it takes ~43 ms, against ~182 ms for q_decode of the whole message and ~37 ms for a message holding only those 23 columns. Tested without q on encoded and hand-built tables: every truncation of them, a symbol column ending at each offset in a word, and skipped names that stay uninterned. Tested against q on a table of every supported type plus a timespan, with p#/s# attributes, a keyed table, and -18! output. Release 2.5.0.
q_build_table released the table again when ray_table_add_col failed, but add_col consumes the table on every path, failure included: in core v2.11.0 and later on all of them, and in earlier cores on every failure q_build_table can reach. The second ray_release read the freed block's refcount, which ray_free leaves at 1, and freed it again. A table whose column is an atom reaches this from the wire. There the double release is masked: add_col's ray_error reuses the freed block, and ray_release ignores errors. add_col's out-of-memory paths create their error before releasing the table, so there the block really was freed twice. The new codec check decodes such a table and expects the error. On the current allocator it passes with or without the fix, because the allocator hides the double release; it keeps the path covered.
morsel-ray
left a comment
morsel-ray
left a comment
There was a problem hiding this comment.
Reviewed 0f0dfd2. No further requested changes from my scoped review of the bounded skip walk, requested-order/keyed projection, cleanup and legacy-decode compatibility. I also checked the consumed-table failure contract directly in core v2.11.5 /5d5f47c2; removing the extra release is correct.
Local checks against that core: instrumented adapter/driver codec, exchange and CLI selftests plus33 server assertions and malformed/fragmented-frame checks pass. Real q is absent here and its legs explicitly skipped. The existing core archive was uninstrumented; this is not a whole-core sanitizer claim.
I additionally ran36 fresh-process synthetic raw/compressed projection cases with type/order/every-cell assertions: multiple empty/control-byte symbols around word boundaries and at input end, key/value request interleaving, key-only selection, and actual two-byte compression backreferences. Skipped control-byte symbols remain uninterned. Adapter/harness ASan/UBSan runs passed. These are synthetic protocol fixtures, not real-q interoperability or independent confirmation of the latency figures.
Optional follow-up coverage: capture a compact positive compressed fixture in the q-free codec tests, since the new in-tree compressed-projection leg currently requires q. Not a blocker for this change.
Closes #22
Adds
q_decode_cols(resp, resp_len, compressed, cols, ncols, err, errlen): asq_decode, but a table keepsonly the named columns, in the order named (as q's
cols#t). The other columns are skipped by theirserialized length, with no allocation per string cell and no symbol interning.
q_decode_cols(full, names)equalsq_decode(names#t)whatever order the names are in.q_decoderejects) can be dropped.RAY_ERRORnaming it; a NULL or repeated name returns NULL witherrset.Synthetic 100 000-row, 142-column window with 23 columns kept: ~43 ms, against ~182 ms for
q_decodeof thewhole message and ~37 ms for a message holding only those 23 columns.
A separate commit fixes
q_build_table, which released a table thatray_table_add_colhad alreadyreleased on failure.
Tests: q-free codec checks (requested order, keyed table, timespan skip, skipped names never interned, every
truncation, a symbol column ending at each offset in an 8-byte word) and a
--cols-selftestleg againstreal q (every type plus a timespan with
p#/s#, a keyed table, compressed-18!output).make recheckpasses against core v2.11.5 (
5d5f47c2) with q.