[davesnx]

A comparison against jq 1.7/1.8.1, jaq 3.1.1 and gojq 0.12.19 on 2026-09-19 showed query-json losing on several common operations because of work repeated per element, not because of the language design. This PR removes that repeated work in ten places, one commit each, with output kept byte-identical to main except for the two behavior changes listed at the end.

Area	Cause on main	Fix
sub, gsub	pattern recompiled on every input value	bounded (64-entry) compile cache keyed by pattern
split, contains, inside, index, rindex, find_indices	Re.compile (Re.str needle) per call for literal needles	plain byte search (Literal_str)
sort_by, min_by, max_by, unique, unique_by	key expression re-run on both operands of every comparison; List.mem scan for seen values	decorate once, List.stable_sort, hash set
range, limit, first, last, nth	whole range materialised before yielding, non-tail recursion	yield per element from a tail-recursive loop; last walks instead of reversing
object + and *	two passes plus @ copy; List.assoc per key was quadratic when both sides are large	one [@tail_mod_cons] pass; hash index above 32 right-hand keys
integer + and -	every Int + Int went through the exact Decimal/Z path	native fast paths with the same overflow promotion Int -> Int64 -> Big_int
decimal literals	3 substrings and Z.of_string per number with a fraction	accumulate up to 18 digits in a native int
pretty printer	a closure allocated per array/object item; each result rendered to a string then all concatenated; buffer grown from 4 KB	named item writers; one shared buffer sized from the input; %.6g via the primitive
colored output	closures and an always-true if colorize per token	direct ANSI writers for the colored path
string lexing	investigated; no change measured as a win (one attempt regressed 5 %)	six new reader tests only

Measurements. hyperfine -N --warmup 2 --min-runs 10, release builds, Linux x86-64 (AMD EPYC 9755), OCaml 5.5.0~beta1, -c (no color) unless stated. Base is 0f5ca93; each row was measured on this branch right after its commit landed. jq is 1.7 from Debian.

Query and fixture	base	this branch	jq 1.7
[.[] | .name.english | gsub("a"; "4")] | length, 21 MB array	703 ± 5 ms	434 ± 4 ms	1090 ± 14 ms
map(.name.english | contains("an")) | length, 21 MB array	692 ± 12 ms	445 ± 7 ms	459 ± 8 ms
sort_by(.base.HP) | length, 21 MB array (97,080 objects)	836 ± 21 ms	455 ± 10 ms	724 ± 19 ms
[range(1000000)] | length	193 ± 3 ms	107 ± 2 ms	118 ± 6 ms
first(range(100000000))	stack overflow	4.2 ± 0.6 ms	1.6 ± 0.1 ms
[limit(10; range(100000000))]	stack overflow	4.2 ± 0.4 ms	1.6 ± 0.2 ms
. + {"z": 1} | length, 100k-key object	35.6 ± 2.2 ms	31.8 ± 3.3 ms	24.5 ± 1.9 ms
. + . | length, 100k-key object	86.5 ± 2.6 s	60 ± 3 ms	not measured
add, array of 1M ints	147 ± 9 ms	132 ± 2 ms	302 ± 5 ms
length, array of 1M floats	271 ± 4 ms	241 ± 8 ms	136 ± 7 ms
., 97 MB benchmarks/huge.json	1.99 ± 0.04 s	1.58 ± 0.02 s	2.76 ± 0.01 s
.[] | .name.english, 21 MB array (97,080 results)	695 ± 12 ms	437 ± 10 ms	465 ± 9 ms
. colored (-C on this branch), 97 MB	2.16 ± 0.02 s	1.89 ± 0.02 s	n/a

On the repository's own benchmarks/bench.sh cases (10 runs, warmup 3) the rendering-heavy rows move and the parse-dominated rows do not: big . 16.4 -> 14.3 ms (jq 16.8), huge . 2146 -> 1644 ms (jq 2767), huge keys 1208 -> 1273 ms (jq 1222, within run-to-run noise). Float parsing stays about 1.8x behind jq: the remaining cost is the Decimal record per number and cons-then-reverse array building, which is a representation change and out of scope here.

Parity. For every query above and the edge cases in the commits, cmp of -c output against a 0f5ca93 build is identical, including decimal results (12.99 + 10 is still 22.99), integer overflow boundaries, mixed-type sort, duplicate keys in merges, and the empty-separator forms of split and /.

Behavior changes, each in its own commit so it can be dropped:

• Colorize only on a TTY and write ANSI codes directly: color is on only when stdout is a terminal or -C/--color is given, off when piped or when NO_COLOR is non-empty; -c/--no-color still wins. Documented in README and CHANGES; Cram covers both the piped default and -C.
• Search and split strings without compiling a regex per call: index(""), rindex("") and find_indices("") return null/null/[] as jq does. On main, index("") returned 0 and rindex("")/find_indices("") looped forever.

Regression coverage added: lazy limit(0; ...), first(empty), nth out of range, 10k-deep descend/paths; invalid-pattern passthrough and cache-key collision for sub/gsub; empty-needle index/rindex/find_indices; from_entries and + with duplicate keys; overflow boundaries for +/-; float formatting and nested empty containers in the writer; escapes, surrogates and a 100k-char string in the reader.

Checks at 7eef191: dune build @runtest @fmt (842 runtime tests, 99 reader tests, CLI Cram) exit 0, release @all exit 0, git diff --check clean; the same three checks were run on every commit in the range.

Where to start reading: source/Interpreter.ml for the interpreter changes (Literal_str, Operators.assoc_merge, yield_range_list, sort_by), source/jotason/Write.ml and source/Core.ml for rendering, cli/cli.ml for the color decision.

Risk: two-way door. A merge revert restores every previous behavior, including color-on-pipe and the index("") result; no data, published API or release artifact changes.

[davesnx]

Independent review of 7eef191 against main (0f5ca93), commit by commit. Method: code reading of every changed function, a 0f5ca93 release build for differential checks, 50+ hand-built comparison queries plus a 53-query sweep over benchmarks/{small,medium,big}.json and 1M-int, 1M-float, 21 MB-array and 100k-key fixtures, and dune build @runtest @fmt on source/ and cli/ (842 runtime tests, 99 reader tests, Cram; exit 0).

Result of the sweep: 50 of 53 queries byte-identical to main. The three differences are exactly the documented index("")/rindex("")/find_indices("") change; on main the last two loop forever, so this is a hang fix as much as a value change.

Findings (inline): three should-fix, two nits. No wrong results or data loss found.

Checked and clean

• Literal_str and all callers: boundary math, needle longer than haystack, start > limit, overlapping matches in find_indices/rindex (same as the old Re semantics), empty needle per caller, byte-level UTF-8 behavior unchanged (café/münchen/日本語 fixture).
• assoc_merge/merge_left: first-duplicate-wins on the right in both paths, left duplicates kept in place, append order, threshold boundary at 31/32/33/34/65 keys, * through deep_merge, [@tail_mod_cons] stack safety on a 100k-key self-merge (main exceeded a 90 s timeout).
• Integer fast paths: promotion Int -> Int64 -> Big_int at every boundary; the old json_of_integer_z also normalized in-range Int64 back to Int, so type/unique results are unchanged.
• Decimal.of_lexeme_opt fast path: 18-digit bound, sign, scale >= 0 gate, trailing-zero trim, repr, -0, 0.000, 1e-7, 19-digit slow path; round trips identical.
• sort_by/unique/unique_by/min_by/max_by: Hashtbl vs List.mem equality on nan, -0.0, differing-repr decimals; stability; key evaluated once; error propagation.
• yield_range_list/last_*: step 0, negative steps, huge ranges under first/limit (no overflow now, correct values), last on empty; the assert false is unreachable by construction.
• Regex cache: invalid pattern memoized as None with the old passthrough behavior; compiled Re.re safely reused.
• Write.ml/Core.ml: float formatting identical across floats.json, extreme magnitudes and negative zero; NaN/Infinity still null; colored -C output identical per constructor; the summarize && not colorize branch is reachable via Error.ml context rendering and is byte-identical.
• cli.ml color precedence -c > -C > NO_COLOR > isatty, incl. error messages.

[davesnx]

should-fix. The PR carries two behavior changes but only this one has a CHANGES entry. The other, from "Search and split strings without compiling a regex per call": index("") returns null (was 0), and rindex("")/find_indices("") return null/[] where main looped forever. Release notes built from this file would miss a user-visible change and a hang fix. Suggest a second bullet in the same style.

[davesnx]

Fixed in ecae5a8: added a second [BREAKING] bullet under Unreleased for index("")/rindex("")/find_indices(""), noting that the last two previously looped forever.

[davesnx]

should-fix. The hash-indexed branch below this threshold check has no unit coverage: the merge tests in Test_runtime.ml use objects of two or three keys, so every test runs the linear path. Verified by hand that 31/32/33/34/65-key right-hand sides with duplicate keys on both sides match main, so it is correct today, but nothing protects it. Add a + and a * case with a right-hand object of exactly 32 and of more than 32 keys, including a duplicate key.

[davesnx]

Fixed in ecae5a8: object_merge_threshold_boundary builds right-hand objects of exactly 32 and 40 entries (a key shared with the left, a key duplicated on the right) and checks + and * against the expected merged order and values, so both the linear and the hash-indexed paths are covered. 847 runtime tests now.

[davesnx]

should-fix. The 64-entry Hashtbl.reset eviction is never exercised: the new tests use two distinct patterns. Add a case that runs 65+ distinct literal patterns through sub/gsub and then reuses the first one, asserting the match still succeeds after the reset.

[davesnx]

Fixed in ecae5a8: regex_cache_eviction runs 65 distinct literal patterns through gsub in one query and then reuses the first one, forcing the 64-entry reset and a recompile.

[davesnx]

nit. First use of od in this repo's Cram tests, and make test also runs on the windows-latest job. That job runs under Cygwin, whose coreutils include od, so this should be fine; worth confirming on the first CI run rather than assuming.

[davesnx]

Fixed in ecae5a8: the Cram case now uses grep -c "$(printf '\033')" and expects 1, so it no longer depends on od.

[davesnx]

nit. This cache only grows. Harmless for the one-shot CLI, but Write.ml is also compiled into the JS library, where a long-lived process rendering arbitrary depths keeps the largest indent string array forever. Rendering a 10,000-deep array is byte-identical to main, so this is about memory, not output. A cap (for example, compute directly above depth 256) would remove the concern.

[davesnx]

Fixed in ecae5a8: indent_for caches depths up to 64 and builds deeper indents directly with String.make. A new writer test renders a 100-deep nesting and a 200-deep array is byte-identical to main with color neutralised.