fix(query): COUNT over a novelty-only or overlay-only predicate returned 0

[aaj3f]

SELECT (COUNT(*) AS ?n) WHERE { ?s <p> ?o } returns 0 whenever <p> exists only in an overlay — silently, with no error, on SPARQL, JSON-LD and Cypher alike.

The issue is filed as a datalog problem. It isn't. The headline case is:

insert a property, count it, get 0.

No reasoning involved. Any newly-introduced predicate is overlay-only for the whole window between its first commit and the next index, and on a write-active server that window is permanently open for whatever predicate was introduced most recently. The datalog and OWL2-RL cases are the same defect with a window that never closes, because a derived fact never enters the persisted dictionary.

GROUP BY is unaffected, and so is the non-aggregate form of the same query — which is why it reads as a reasoning bug rather than a planner one.

Live since b9fbfe35c (2026-06-02), i.e. this predates and is unrelated to #1823.

Cause

count_rows_operator resolved the predicate through sid_to_p_id above its own base/overlay lane split and read a miss as COUNT = 0:

let Some(p_id) = store.sid_to_p_id(&pred_sid) else {
    return Ok(Some(build_count_batch(out_var, 0)?)); // predicate absent => 0
};

sid_to_p_id resolves against the persisted index dictionary only. "Absent ⇒ 0" is sound under the metadata lane, which requires no overlay and to_t == max_t, and false under the base-plus-novelty-delta lane. b9fbfe35c replaced a single overlay-excluding gate with the two-lane structure for a perf win and left the check above the split, where its precondition no longer holds — a gate-relaxation seam.

Fix

Move the lookup into each lane: the metadata lane keeps ⇒ 0, the overlay lane defers to the planned pipeline. Lane conditions are untouched; only the position of the lookup moves.

This is the idiom already used by the numeric-compare and encoded-filter operators in the same file and by fast_predicate_scalar_agg, fast_union_star_count_all and fast_group_count_firsts. fast_count.rs:66 was the only site in the COUNT / scalar-aggregate family that got it wrong — and the differential harness addition below confirms that empirically, not just by inspection. That scope qualifier is load-bearing: @bplatz audited every sid_to_p_id caller in fluree-db-query/src and fluree-db-api/src while checking this claim and found one more site with the same shape outside the family, at binary_range.rs:855. It is pre-existing, unrelated to this change, and being filed separately — see the thread on fast_path_common.rs:3192 for why it is not folded in here.

A second hunk documents the caller gate on count_predicate_overlay_delta, which is unreachable for an overlay-only predicate by construction (its touched-leaf partition and overlay cursor are both range-bounded by p_id), so the lookup does not get re-hoisted later.

Perf — positive, not merely neutral

Counting sid_to_p_id probes per path:

Path	Before	After
Metadata lane (overlay-free, at HEAD)	1	1
Overlay lane (to_t >= max_t)	1	1
Neither lane (to_t < max_t, time travel below head)	1	0

No path does more work than before, and the time-travel path does one probe fewer. overlay_free was already computed unconditionally, so nothing else moved.

One behavioural note so a timing difference is not misread as a regression: for time travel below the head (to_t < max_t) against a predicate absent from the persisted dictionary, the operator previously returned an instant 0 and now falls through to the generic pipeline. The old answer was also correct in that case, so this is not a correctness fix — it is the same deferral applied uniformly rather than special-cased, and it trades an instant constant for a generic-path walk on a shape that is both rare and cheap (the base index holds no rows for that predicate).

The only behavioral delta is the deferral, and it can fire only when sid_to_p_id misses — i.e. only when the base index holds zero rows for that predicate. No query that previously did real base-index work is slowed, because the predicate that triggers the deferral has no base-index work to do.

Worth stating precisely, since it is easy to over-claim: the deferral's own cost is a walk of the graph's overlay. overlay_only_flakes (binary_range.rs:1514) passes None/None bounds and filters per flake, so it is O(all overlay flakes), not O(rows for that predicate). That is bounded by novelty (capped by the reindex thresholds), not by ledger size, and it is the cost the already-correct non-aggregate form of the identical query pays today. The fix makes the aggregate form as expensive as the projection form, not more.

Testing

New fluree-db-api/tests/it_fastpath_1863_regression.rs: 13 cases over four file-backed, explicitly-reindexed ledgers, on SPARQL, JSON-LD and Cypher (all three confirmed broken). Each case asserts three things — the fast-lane answer, the kill-switch (generic pipeline) answer, and the COUNT rows routing stamp.

#1863's "owl2rl is not affected" is false. The issue's probe used a symmetric-property axiom, which derives extra flakes for a predicate that already has a base-index entry, so it resolved a p_id and escaped. An owl:inverseOf axiom minting a predicate with zero base assertions fails identically; that case is in the file.

Why these tests catch what 3.5 months of green datalog suites did not

Two reasons, and both are designed against explicitly:

1. Every datalog suite is memory-backed. With no binary store the operator returns at its first line, the generic pipeline answers correctly, and the suite is green vacuously. Every ledger here is file-backed with index points pinned by the test (IndexConfig at 5 GB thresholds plus explicit reindex), so an auto-index trigger cannot quietly make them vacuous either.

2. A MustNotFire assertion is satisfied both by a correct deferral and by a fixture that never reached the fast path at all. So every ledger also carries a MustFire canary over an indexed predicate under the same conditions. The canaries are what turn "the lane declined" into "the lane was live and declined". If one ever regresses to MustNotFire, the canary is the bug report — don't "fix" the canary.

Non-vacuity. Reverting only the fast_count.rs change (mutation needle taken from post-cargo fmt source; the mutated tree differed from origin/main by a comment block only) produces 16 failures: 8 on the value assertion and 8 on the routing stamp, with all five canaries still passing:

SPARQL COUNT(*) over a novelty-only predicate: fast lane counted 0, expected 7
SPARQL COUNT(?s) over a novelty-only predicate: fast lane counted 0, expected 7
JSON-LD count over a novelty-only predicate: fast lane counted 0, expected 7
Cypher count(*) over a novelty-only relationship: fast lane counted 0, expected 7
SPARQL COUNT(*) over a datalog-derived predicate: fast lane counted 0, expected 3
SPARQL COUNT(?a) over a datalog-derived predicate: fast lane counted 0, expected 3
JSON-LD count over a datalog-derived predicate: fast lane counted 0, expected 3
owl2rl_inverse_of SPARQL COUNT(*) over a derived predicate: fast lane counted 0, expected 3

The value failures are the point. A suite that failed only on stamps would be pinning routing, not correctness.

Differential harness

it_differential_fastpath.rs gains a tail commit introducing a predicate absent from every base commit. Its Overlay condition previously added only new subjects under predicates the base index already knew, so no condition ever produced an overlay-only predicate — the exact hole this defect lived in. With the fix reverted it reports fast=[0] against generic=[16].

The SUM, MIN/MAX, GROUP BY and plain SELECT cases added over the same predicate agree across both lanes in all three conditions.

To be precise about what that does and does not show, since it is easy to over-read: the siblings do not handle an overlay-only predicate — they decline at their gate. With outcomes recorded, COUNT rows, COUNT rows numeric compare, COUNT(DISTINCT), SUM(?o) and MIN/MAX string each emit fallback:gate_declined and fused_chain serves the answer; fast_predicate_scalar_agg.rs:516-525 returns Ok(None) on exactly this shape. So the two lanes agreeing means both were taking the generic path, not that a fast lane computed it correctly.

Declining is the behaviour you want here, and it is what makes the ungated site stand out: every other sid_to_p_id caller in the COUNT / scalar-aggregate family sits behind an overlay gate, and fast_count.rs:66 was the lone one within that family that answered instead of deferring.

Third commit is separable

{"opts": {"reasoning": "datalog"}} was silently ignored on the JSON-LD surface — no error, no reasoning, an empty result set. opts is a deliberately open bag carrying several genuine query-level knobs (objectVarParsing, includeSystemFacts, t, maxFuel), and parse_object_var_parsing even reads opts first, so guessing that reasoning lives there is a reasonable mistake with an unhelpful failure mode.

All four keys ReasoningModes::from_query_json consumes (reasoning, rules, ontology, reasoningBudget) are now accepted from opts as an alias, with the top level canonical and the sole source when present. No privilege change: ledger-config override control is applied downstream to the parsed modes (config_resolver::merge_reasoning → GraphDb::effective_reasoning), where Force wins whatever the source. Rejecting unknown opts keys instead is not viable — the server injects into opts itself (identity, policy-values, policyClass).

Drop this commit if you want the PR narrower; nothing else depends on it.

Gates

Local toolchain matches CI exactly (rust-toolchain.toml pins 1.97.0; CI pins dtolnay/rust-toolchain@1.97.0), so these are comparable rather than merely suggestive.

Ran locally, green:

• cargo fmt --all -- --check
• cargo clippy --all --all-features --all-targets --locked -- -D warnings — the full CI clippy gate, exit 0, zero warnings
• cargo check --workspace --all-targets --locked
• cargo clippy --all-targets -- -D warnings in testsuite-sparql/ (it has a path dep on fluree-db-api and no --no-deps, so the changed crates really are linted there) and its cargo fmt --all -- --check
• both wasm32 clippy invocations
• cargo test --workspace --all-features --doc --locked
• every fluree-db-api test target under --features native (1,458 group-suite tests plus the three standalone kill-switch binaries)

cargo nextest run --workspace --all-features --no-fail-fast --locked: 12,839 passed, zero assertion failures, 38 skipped, and 10 hard-killed at exactly the 360 s ceiling (slow-timeout 120 s × 3) — still running, not failing. I chased all 10 rather than wave them off, since both stragglers are COUNT fast-path tests on indexed ledgers and it_query_sparql_indexed.rs alone holds 106 COUNT-shaped queries. Every one passes: grp_query_sparql 388/388 and it_iceberg_local_fs 4/4 in isolation, the grp_index/grp_query ones already green standalone, and the last two under cargo test (no 360 s kill) at 89 passed / 41 s and 1 passed / 529 s. The 41 s figure is the tell — this box was shared with six sibling agent worktrees at load 18–26 on 16 cores. CI's dedicated runner should not hit the cap.

Not run locally, stated so their absence does not read as a pass:

• W3C SPARQL suites (cargo test in testsuite-sparql/) — the rdf-tests submodule is not initialized in this checkout. Its clippy sibling did compile the changed crates; the SPARQL surface is covered by the new regression file and grp_query.
• sql-bridge fmt/clippy/tests — separate workspace, untouched, no path dep on the changed crates; its live tests need MySQL/Postgres services.
• wasm-smoke (headless Chrome, npm packaging) and the wasm32 Node runtime smoke — unaffected by this change.

CI result at ad655371b: all jobs green — clippy, fmt, test, testsuite-sparql, sql-bridge, wasm32, wasm-smoke, workflow-lint. That closes the three gaps listed above: the W3C SPARQL suites, sql-bridge and wasm-smoke all ran and passed in CI even though they could not run locally.

Provenance

The mechanism, the b9fbfe35c provenance, and the refutation of the issue's two narrowing claims (that this is datalog-specific, and that owl2rl is unaffected) are set out in the correction comment on #1863 — not restated here. This body covers the fix and the tests.

Fixes #1863

[bplatz]

Approving. The fix is correct and the tests are strong.

What I verified:

• Lane 1's absent => 0 really is sound. overlay_has_novelty (fast_path_common.rs:3654) is the family's standard gate, and combined with to_t == store.max_t() it matches fast_path_store's structural conditions, with the O1 gate having already discharged policy.
• Lane 2's Ok(None) routes to the fallback, the same contract the numeric-compare twin uses.
• The cited precedent is accurate. fast_count.rs:201-237 is structured exactly as described — HEAD lane behind fast_path_store_policy_cleared answers 0, overlay lane defers with a comment saying why.
• The probe-count table is right. Lane 1 returns unconditionally, so only one lookup happens per call on any path.
• The differential-harness addition fills a real hole — no prior condition produced an overlay-only predicate, only new subjects under already-indexed ones.

One note on framing, for anyone auditing from this description later: allow_cursor_fast_path — count_plan_exec's entry gate — does not guarantee overlay-free. It is single-ledger + no from_t + root-or-no policy only. Overlay-freeness there is established per-lane. It is done consistently, so this is a documentation observation rather than a defect.

The test file is also the right pattern for the fast-path kill switch — programmatic set_fast_paths_disabled with a Drop guard, an assertion that FLUREE_DISABLE_QUERY_FAST_PATHS is unset, and its own [[test]] binary so the process-global toggle cannot contaminate siblings.

One inline comment, on a second site with the same defect that I found while checking the "only site in the family" claim. It is pre-existing and outside this family, so it is genuinely optional for this PR — fold it in or file it as a follow-up.

[bplatz]

The same defect exists at one more site, outside this family — pre-existing, not introduced here.

I audited every sid_to_p_id call site in fluree-db-query/src and fluree-db-api/src that returns a substantive answer on a miss, to check the "only site in the family that got it wrong" claim. Within the COUNT / scalar-aggregate family it holds: every other site is protected. (Including all nine count_plan_exec.rs sites I was suspicious of — those are guarded per-lane by the flag at count_plan_exec.rs:82-89, which is stricter than overlay_has_novelty: raw epoch() != 0 plus to_t != max_t.)

One site outside the family has exactly the shape this caller-gate note describes — binary_range.rs:855, in binary_lookup_subject_predicate_refs_batched_v3:

let p_id = match store.sid_to_p_id(predicate) {
    Some(id) => id,
    None => return Ok(HashMap::new()), // unknown predicate → no results
};

No overlay guard, and it sits above the branch_for_order check (~line 908) whose None arm routes to batched_refs_overlay_only(...) — so it short-circuits the very overlay path that exists for this case. Line 426 of the same file gets it right, with the comment "Overlay may still contain this predicate (novelty), so return overlay-only."

Three things suggest it is worth tracking rather than leaving implicit:

1. Policy reaches it. fluree-db-policy/src/class_lookup.rs:41-50 calls lookup_subject_predicate_refs_batched as its preferred path and treats Ok(map) as authoritative — only ErrorKind::Unsupported falls through to the per-subject SPOT lookup. On a ledger where rdf:type has no persisted p_id (written but never indexed, or a named graph living only in novelty), every subject looks classless and f:onClass rules silently never match.

2. The indexer's class/property stats use the same provider.

3. It is already known and worked around rather than fixed. fluree-db-api/tests/it_batched_refs_overlay_lifecycle.rs:47-50 seeds its fixture specifically to dodge it:

   Carries rdf:type so the predicate is in the persisted dictionary — without that the batched lookup short-circuits on an unknown predicate and never reaches the overlay path at all.

The shape of the fix looks like the one in this PR: move the p_id resolution below the branch check and, on a miss, delegate to batched_refs_overlay_only(...), which already takes predicate: &Sid and needs no p_id.

Either fold it in here (it is the same defect and the same reasoning) or file it as a Follow-up: — your call. I have not traced whether "no class matches" fails open or closed for a given rule shape, so someone closer to policy should set the severity before it is scheduled.

[aaj3f]

Thanks @bplatz — I went and read this one rather than taking it on faith, and you're right on every part of it. binary_range.rs:855 is exactly as you describe:

let p_id = match store.sid_to_p_id(predicate) {
    Some(id) => id,
    None => return Ok(HashMap::new()), // unknown predicate → no results
};

…and it does sit above the branch_for_order check at :907, whose None arm already delegates to batched_refs_overlay_only(...) — so the early return short-circuits the very path that exists for this case. Same distinction :426 in the same file gets right. I also confirmed the two things that make it reachable rather than theoretical: fluree-db-policy/src/class_lookup.rs:39-51 takes Ok(map) as authoritative and only falls through on ErrorKind::Unsupported, so an empty map reads as a successful "nothing here has a class"; and it_batched_refs_overlay_lifecycle.rs:47-50 is already seeding around it, in a comment that says so in as many words.

One thing in your favor that I think makes it even smaller than you sketched: I don't think the p_id resolution has to move below the branch check at all. batched_refs_overlay_only takes predicate: &Sid and no p_id, and every other argument it wants is already in scope at :855 — so it looks like it's just the None arm changing, with the ordering left alone:

None => {
    // Unknown in the persisted dictionary: the BASE scan cannot match, but the
    // overlay may still carry this predicate. Same distinction as `:426`.
    return batched_refs_overlay_only(
        store, dict_novelty, g_id, predicate, subjects, opts, overlay,
    );
}

Which is the same shape as the fast_count.rs change in this PR, with the same perf story — it can only fire when the predicate has no persisted entry, so the base had nothing to contribute either way, and against an empty overlay the walk visits nothing.

So I'm going to file it rather than fold it in — and I want to be explicit that it isn't because it's hard. It's the one flavor of thing I don't think I should decide unilaterally: the entire effect of the fix is that f:onClass rules start matching on ledgers where they currently, silently, don't. For an allow-shaped class rule that means data that was hidden becomes visible; for a deny-shaped one it means the behavior today is failing open. You said you hadn't traced which way that lands for a given rule shape, and I haven't either — and an untraced change to policy enforcement seems like the wrong passenger for a P0 whose title is about COUNT, on a review several people have already signed off on. If it is a fail-open, it wants its own severity and its own set of eyes, not a footnote in this thread.

I'll get it filed with your evidence carried over — the class_lookup.rs reachability, the fixture that works around it, and the :426 contrast — and flag the fail-open question up front so whoever picks it up prices it properly instead of reading it as a wrong-results nit.

If you think I'm over-weighting the policy angle and it should just go in here, say so and I'll fold it — you're closer to that code than I am.

Separately, your framing note is a fair hit. "the lone ungated one" is only true within the COUNT/scalar-aggregate family and I'd written it as though it were absolute; I've fixed the sentence to name the scope it's actually true within. Same for the allow_cursor_fast_path observation — you're right that it's single-ledger + no from_t + root-or-no-policy only, and that overlay-freeness is established per-lane rather than by that gate.