Skip to content

Parse plain decimals without copying the lexeme - #74

Draft
davesnx wants to merge 1 commit into
perf/hot-pathsfrom
perf/float-parsing
Draft

davesnx wants to merge 1 commit into
perf/hot-pathsfrom
perf/float-parsing

Conversation

@davesnx

@davesnx davesnx commented Sep 21, 2026

Copy link
Copy Markdown
Owner

Stacked on #72 (perf/hot-paths); the diff here is one reader change plus tests.

Parsing an array of one million decimals took 246 ms on perf/hot-paths against 146 ms for jq 1.7. A stage-by-stage harness over the same file showed where the time went: lexing plus list building alone costs about 117 ms and 7 words per number, decimal construction (Decimal.of_lexeme_opt) only about 24 ms, and the remaining hundred milliseconds is allocation. The real path allocates 26 words per number, 3.6x the stub, mostly from Lexing.lexeme copying every literal into a fresh string kept as repr for round-trip printing, plus the record and its Some box. A larger minor heap did not help (s=4M was 17 % slower), and Z.of_int allocates nothing for values in range, so a small-coefficient variant would gain nothing.

Change. Read.mll gains try_fast_decimal: for a literal of the shape -?digits.digits with no exponent, at most 18 significant digits and no trailing zero in the fraction, it accumulates the digits straight from lexbuf.lex_buffer and builds { coeff; scale; repr = None } with no string allocation. The grammar's positive_int already forbids leading zeros, so for that shape canonical printing is byte-identical to the source text; the trailing-zero case (1.50) is the only one where it is not, and it falls back. Exponents, longer mantissas and trailing zeros take the unchanged slow path with repr preserved. No type or consumer changes.

Measured, hyperfine -N --warmup 3 --min-runs 15, release builds, Linux x86-64, -c length on the 1M-float array (0f5ca93 is main, ecae5a8 is the base of this PR):

Binary mean ± σ
0f5ca93 (main) 286.6 ± 11.9 ms
ecae5a8 (perf/hot-paths) 245.7 ± 6.0 ms
this branch 199.4 ± 7.9 ms
jq 1.7 146.4 ± 3.0 ms

On a 200k-record array with three decimal fields each, 264 to 213 ms (jq 183). Integer parsing is untouched (1M ints: within noise). Arithmetic-heavy map(. + 10) | add gains about 6 %, since compute dominates there.

Parity. cmp of -c output against a 0f5ca93 build for ., map(. + 10) | length, add, max and sort | .[0:3] on the 1M-float array, 1M ints, the 21 MB object array, small.json, big.json and huge.json: 30 of 30 identical. Literal round trips identical for 1.0, 1e5, -0, -0.0, 1.50, 1E-7, 1.5e-7, 0.005, 100.00, -0.5 and a 21-digit mantissa. Structural-equality cases ([1.5, 1.50] | unique, [1.5, 0.5 + 1.0] | unique, group_by(.)) unchanged. Exact decimal arithmetic unchanged.

Checks at this head: dune build @runtest @fmt exit 0 (847 runtime tests, 104 reader tests incl. 4 new for both sides of the fast path, Cram), release @all exit 0, git diff --check clean.

Remaining gap. The rest of the distance to jq and jaq is the list-of-boxed-values representation and cons-then-reverse array building in the reader, which affects every type, not decimals; that is a representation redesign and out of scope here.

Risk: two-way door. A merge revert restores the previous reader path; no data, API or output bytes change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant