Make the export → import round trip faithful (breaking: export's default output changes)

[aaj3f]

fluree export's default output changes. Annotated ledgers now emit RDF 1.2 annotation syntax
instead of the reserved f:reifies* system triples, and --all-graphs no longer emits the ledger's
own #txn-meta and #config. --raw-reifies restores the old annotation bytes and
--system-graphs restores the old --all-graphs payload, so nothing consuming today's exports is
without a path forward. Both old forms were unreadable in ways this PR is about: the first is a
shape Fluree's own write surfaces reject, and the second does not re-import into any ledger.

fluree export produced a file you could not read back. Three defects made that true in different
ways, and fixing them turns the round trip into an operation people will actually use — which is
what makes the fourth and fifth worth fixing in the same PR, because a normal round trip is how you
reach them.

fluree export (#1574, #1847, #1859)

#1574 — export a never-indexed ledger, and stop writing Turtle into .flpack

Every RDF format refused on a ledger that had been committed to but never indexed:
no binary index available for export (is the ledger indexed?). That requirement was structural,
not informational — write_to attaches a novelty overlay two lines after the bail-out, and on such
a ledger that overlay holds the whole ledger.

The writers now get an empty BinaryIndexStore, seeded with the snapshot's namespace table so
ID→IRI resolution completes through DictNovelty, and they stop bailing when a graph has no SPOT
branch — BinaryCursor exhausts a zero-length leaf range and falls through to its overlay-only
tail, which emits exactly those rows. That second half independently fixes a narrower case: on an
indexed ledger, a named graph first written after the last index build exported nothing.

Export stays a read. Auto-indexing was the other candidate and it is the wrong shape here: it
triples the ledger's on-disk footprint (6.1 M → 34 M at 1 M triples) as a side effect of a
read-shaped command, with nothing that would ever clean it up.

One consequence of opening this path needed fixing with it, and review caught it. Overlay rows
that miss V3 translation bypass the cursor's sorted merge, and a never-indexed ledger maximizes
them — with no persisted dictionary to encode against, a plain string, a language tag and a decimal
all miss. They were emitted after the stream, which reopened a subject block that had already
closed: ex:bob appeared twice with "Bob"@en stranded at the end of the file. Valid Turtle,
every triple present, and re-imports correctly — but export's output shape then depended on
whether the ledger happened to be indexed, which is the one property a faithful-round-trip change
cannot afford to add. UntranslatedBySubject indexes those rows by subject so each writer folds
them into the block they belong to as it closes, and emits whatever the base stream never reached
as its own blocks. Memory is unchanged — they were already held for the whole export as a Vec.

After the fix, subject grouping is index-independent in Turtle, TriG and JSON-LD, pinned by
export_subject_grouping_is_the_same_indexed_or_not. Intra-block predicate order still differs
(untranslated rows append rather than sorting into p_id position) and is deliberately not
asserted: it carries no meaning in Turtle, and the triple set is already pinned exactly.

The command in the report had a second defect, and auto-indexing would have made it silent
rather than loud. fluree export mydb -o mydb.flpack passes no --format; the default is turtle,
and nothing consulted the output name — so it ran the Turtle writer into a file named .flpack.
--format becomes optional: absent plus a .flpack output name infers ledger, and an explicit
RDF format contradicting that extension is refused naming both sides rather than guessed.

#1847 — the filed premise is wrong; three real defects sit behind it

--all-graphs works, and has since the v4 baseline. It is documented, and --graph <IRI> works
too; the repro simply did not pass the flag (and would not have run as written — fluree export
takes the ledger positionally and rejects -l). A reviewer should know that going in, because the
title sends you looking for a bug that is not there. What is there:

is_system_graph was never called. fluree-db-api/src/export.rs:62-69 defines it under the
comment "System graph IDs excluded from dataset exports", and git log -L puts it at the v4
baseline, added and never wired. So --all-graphs emitted the ledger's own #txn-meta and
#config alongside user graphs — a file that, re-imported into a ledger of the same name, lands
that commit metadata on the target's reserved graph ids, and into a differently-named one lands a
foreign ledger's commit history in an ordinary user graph. docs/cli/export.md had been written to
describe the unfiltered behaviour as intended. The filter now runs at the two call sites it was
written for. --system-graphs keeps the old bytes available for diagnostics, as a modifier on
--all-graphs rather than a selector, and its help text and the docs say plainly that its output
does not re-import cleanly anywhere.

The CLI discarded ExportStats at both call sites, so a --format trig export that dropped
every named graph in the ledger printed exactly what a complete one printed. The literal complaint
in the issue was "nothing in the output to suggest anything is missing", and that was accurate.
Export now summarizes to stderr — stdout stays pure RDF, so redirecting still yields a clean file —
and names the flag that would have included them. Turtle and N-Triples get a different remedy in
that message, because --all-graphs is rejected for them: there the fix is the format, not the
flag.

Retitle suggestion, as a note rather than an edit: "fluree export leaks system graphs into
--all-graphs, and says nothing when it drops user graphs". The reporter's underlying worry —
"export → import silently loses rows" — is correct, by two mechanisms he did not identify, and that
title names both.

#1859 — export emitted a form Fluree refuses to ingest

All five RDF formats emitted the seven reserved f:reifies* system facts as ordinary triples. No
other RDF 1.2 tool reads that, and Fluree's own insert and update surfaces reject it: hand-written
f:reifies* is refused as a system-controlled predicate. Fluree exported a form Fluree will not let
you write.

The new syntax is the default, with --raw-reifies preserving today's bytes — not behind a
flag, not a clean break. Per format: s p o ~ <r1> ~ <r2> in Turtle and TriG,
<r> rdf:reifies <<( s p o )>> in N-Triples and N-Quads, "@annotation": {"@id": …} in JSON-LD.
All five suppress rows whose predicate satisfies is_reserved_reifies_predicate — all seven, not
the three the issue happened to show; a lang-tagged object adds reifiesLang and a plain literal
adds reifiesDatatype.

The reifier marker is emitted inline and the reifier's own properties are left where the scan
already puts them, later in the stream as an ordinary subject. Inlining a {| … |} body instead
would mean a random SPOT seek per reifier, out of scan order, at the exact moment the base edge is
written — the difference between a one-pass fix and a two-pass one. Both spellings are the same RDF
1.2, and the target syntax was verified to round-trip byte-identically against a released binary
before any of this was written, so it was known to be ingestible going in.

Where "which reifiers point at this edge" comes from is decided once per export, three ways,
and the cost of each is the reason the design is shaped this way:

snapshot state	source	cost
no f:reifies* flake ever observed	none	two boolean reads, then the scan runs exactly as before
sealed arena, empty attachment overlay	current_annotations_batch	one sorted merge-scan per ColumnBatch, O(edges·log edges + covered leaves)
sealed arena + attachments committed since	current_annotations_merged, per edge	the batched read is arena-only by contract and cannot see a novelty retract of an indexed attachment
no arena	base-index PSOT scan over the seven reserved predicates	O(annotations), once at export start

That last row was going to be a refusal with a fluree reindex hint. It is not, because the hint
does not work: fluree index leaves an annotated ledger at
has_annotations = true, annotation_index = None, had_annotation_arena = true, and that sticky bit
permanently blocks the bootstrap at indexer_attachment_provider.rs:304-306, so no later reindex
recovers it. A refusal whose remedy sends the user in a circle is worse than no refusal — and
refusing would have made export the only surface in the system that cannot read a ledger everything
else reads, since SPARQL-star returns those annotations correctly through its own fallback. That
indexer defect is pre-existing, is being written up separately, and is the reason this decision
changed. The scan reuses scan_base_index_for_attachment_events_in — the routine the arena's own
seal pass uses, decoding through the canonical EdgeKey::from_reifies_facts — rather than adding a
second scanner. The refusal survives for the one genuinely unreadable case: an arena root present
with no content store to read it from.

The perf commitment is intact, and measured rather than asserted. Common export (no
annotations, no named graphs, indexed ledger), 200 000 triples, release binaries, this branch
against origin/main @ 3d89e8f71:

	--format ntriples	--format turtle
origin/main	0.127 s	0.110 s
this branch	0.128 s	0.110 s

Both binaries were timed in the same window on a shared machine, so the
comparison holds even though the absolutes would move on a quiet one — the
pairing is the claim, not the milliseconds. The byte-identical result below is
not time-dependent at all.

and the output is byte-identical on both formats. Structurally, not by luck: a ledger with no
annotations short-circuits on two booleans, never opens an arena, never builds the suppression id
list, and batch_reifiers returns before touching the batch — so the second decode of each row,
which is the real cost of annotation emission, happens only on ledgers that carry annotations. Per
row the writers gain one if let Some(…) on a None.

That measurement is on an indexed, annotation-free ledger, which review correctly noted is the one
shape that cannot exhibit the subject-splitting above. Measured separately on a never-indexed
ledger after the grouping fix: identical subject grouping and an identical triple set against the
same ledger post-fluree index, with only intra-block predicate order differing. A never-indexed
export is not a hot path and was not timed; the claim above is scoped to the indexed one.

Known gap, reported rather than dropped. Annotations written inside a named graph do not reach
the output: both forward lookups are blind to them, because the PSOT range that feeds the arena's
seal comes back empty for a named graph and the arena is built from that same scan. The rows are in
the ledger and SPARQL reads them fine. Rather than suppress the bundle and emit nothing, export
counts it and says so:

  warning: 1 edge annotations could not be resolved and are NOT in the output;
           re-run with --raw-reifies to emit them as f:reifies* triples

That is a fidelity tradeoff worth a reviewer's eye: for that one shape the output moves from "wrong
syntax but complete" to "right syntax, incomplete, and loud, with a working escape hatch". The clean
fix is upstream in the seal scan, not in export, which is why it is not in this PR.

The test coverage is the structural half of this

Every export test in fluree-db-cli/tests/integration.rs asserted a failure. That is how a
filter designed, written and documented at the v4 baseline stayed uncalled for three years, and how
an output format Fluree itself rejects shipped on five surfaces: nothing ever looked at what export
produced. --all-graphs had no positive coverage anywhere in the repo.

The block is now positive-path and round-trip: export → re-ingest → query, per format, per
annotation object shape, and across all three annotation sources (never-indexed, fluree index,
create --from). One of those tests caught a vacuity in another: the --all-graphs round trip
originally imported into a ledger of the same name, where a leaked #txn-meta collides and
vanishes — so it passed with the system-graph fix reverted. It now also imports into a differently
named ledger, where a leak is visible.

Import and delete (#1846, #1861)

These are here because the export fixes above are what make them reachable. A round trip through
fluree export → fluree create --from was not an operation anyone could complete; now it is, and
it is the normal way to produce a TriG file with GRAPH blocks in it.

#1846 — reserved graphs, refused at import and dropped at load

The filed mechanism is not the one. The issue proposes running stage()'s IRI-shape check over
graph_delta before apply_delta. That check is validate_absolute_graph_iri, and
urn:fluree:imp:main#txn-meta passes it cleanly — it is a perfectly well-formed absolute IRI. The
check that matters is the reserved-graph guard, and apply_delta is the wrong layer anyway: by then
the flakes are encoded and the blob is written.

What actually happened was a record that is correct and invisible. Bulk import bypasses
stage(), so #1838's reserved-graph guard never saw a GRAPH <urn:fluree:{ledger}#txn-meta> block.
The import path allocated the reserved g_id correctly and the commit writer encoded the flakes
faithfully; the loss came afterwards, in the graph-scoped index builder, where g_id 1 is built
only from the synthetic commit-metadata chunk and g_id 2 has no pass at all. A correctly-routed
record from a data chunk was filtered out of every index segment while staying durable in the blob.
An indexed-only test reports the benign answer here, which is part of why this survived.

Both reserved IRIs are now refused at the top of the named-graph loop, by literal IRI and again by
the g_id the block resolves to. #txn-meta reuses ReservedGraphTarget so the message matches every
other refusing surface; #config gets a distinct ConfigGraphImportUnsupported, because ordinary
transactions may write ledger configuration and refusing it at import is a capability gap rather
than a policy — the two must not read alike. The <#txn-meta> sentinel spelling is untouched and
pinned by a test, since parse_trig_phase1 routes it to the commit envelope where legitimate commit
metadata belongs.

The consumption side is closed too, and that is the half that matters for a ledger received from
somewhere else. Genuine commit records are derived, never carried: generate_commit_flakes rebuilds
them from the envelope on every load with g: None, and they acquire the txn-meta graph Sid only
from stamp_graph_on_commit_flakes afterwards. So a flake arriving in a commit blob already
carrying that graph Sid and already claiming a FLUREE_COMMIT-namespace subject did not come from
any writer — it is forged by construction. Those are dropped, at the five sites where a blob's
flakes enter novelty, and logged at warn with the ledger and t so an affected ledger is
diagnosable rather than silently altered.

The drop is keyed on the conjunction, deliberately: either half alone is ordinary data, and a drop
generalised to "reserved graphs" would delete ledger config and freeze the resolved-config cache's
invalidation key. Config is the deliberate opposite case and is never touched. Tests pin both halves
and the config case.

Backward compatibility was checked rather than assumed — no writer in this tree produces such a
flake, and the ordering makes that structural: commit_txn writes and publishes the blob before any
metadata is generated. Cost is nil: at the two sites replaying blob and generated flakes together,
one traversal now does both jobs where there were two, and a non-commit flake — essentially all of a
ledger — still pays the single u16 comparison it paid before.

These two halves meet the export half in one place, and it is worth checking deliberately. The
transact audit asked that #1847's fix exclude reserved graphs from the emitted dataset, so that a
--all-graphs export does not produce exactly the file #1846 refuses, and that #1846's refusal be
scoped to reserved graphs only — never to user named graphs, which #1847's fix now emits in volume.
Both hold, and export_all_graphs_round_trips_into_a_same_named_ledger exercises the seam: it
imports a --all-graphs export containing a user named graph, so an over-broad import guard would
fail it.

#1861 — what each spelling of an annotation delete actually removes

The docs explained base-edge retraction, and separately showed the annotation form for inserts, and
never connected the two. The surprise is two rules meeting:

1. The annotation form asserts the base triple, so deleting it retracts the base edge. RDF 1.2
   Turtle 2.11.1 defines s p o ~ :r {| … |} as syntax that both reifies and asserts, and SPARQL 1.2
   Update 3.1.2 admits the same production into DELETE DATA. This half is spec-mandated — a store
   that kept the edge here would be the one diverging.
2. Retracting a base edge cascades to every reifier attached to it, including ones the delete
   never named. This half is Fluree's own; neither spec entails it. It is now said out loud, so a
   reader who checks the spec and finds Fluree deleting more than it names does not conclude there is
   a second bug and file one.

The table covers three spellings the issue does not mention, two of which change the guidance —
including DELETE DATA { s p o ~ :claim1 }, a bare reifier with no body block, which reads as
"detach claim1" and is the worst outcome available: the edge goes, both claims are detached, and
both bodies are left standing as orphans. It also says what to do instead, which is the reason
someone arrives at the section. The matrix is measured rather than read off the code, and pinned by
tests so the documented table cannot drift.

The issue's second ask — a commit-time warning on the cascade — is not built, and that is why
#1861 stays open below. There is no advisory channel on a successful transaction today; adding one
means a new field on the stage result threaded through four crates for one string. It is also aimed
at the wrong moment: a warning printed after the commit reports a loss that already happened. If it
is worth engineering against rather than documenting, the instrument is a pre-commit one.

Merge order with #1871

No longer a hazard: this PR now carries #1871's text verbatim for the three places the two
collide on wording — docs/cli/server-integration.md's at row and Required-semantics item 3, and
docs/cli/export.md's --at row. Whichever lands first, the other's merge is a no-op on those
lines instead of a conflict.

That is worth the duplication because of what the conflict would have looked like rather than
because conflicts are expensive. This branch inserts a system_graphs row immediately above the
at row and edits Required-semantics items 2 and 5, which bracket item 3 — so both hunks carried
the old time-spec text as context, and "take #1875's side" would have been the natural-looking
resolution while silently reinstating a normative spec of the #1805 bug. That section is what
third-party POST /export implementors build against; someone built this bug from that page
already. A resolution note in a body relies on the resolver reading it and then correctly
identifying which side is which mid-conflict. Copying the text removes the decision.

What ran, and what did not

Locally, green: cargo fmt --all (after the last edit); cargo clippy --all-targets --no-deps on
each crate the export half touches — fluree-db-api, fluree-db-cli, fluree-db-binary-index,
fluree-db-server; cargo test -p fluree-db-cli (383 tests); cargo test -p fluree-db-binary-index (412). On the merged tree specifically: cargo fmt --all --check,
cargo check --workspace --all-targets, and the cross-half seam test — run at the workspace level
rather than per crate on purpose, since the two halves touch different crates and a zero-conflict
merge can still be a broken tree.

cargo test -p fluree-db-api ran on the transact half: 61 suites, 0 failures. But 17 of that
crate's 77 [[test]] targets never built, because they carry required-features that are off by
default — iceberg (9), aws-testcontainers (3), sql (2), credential, residency. None sits
on a commit-replay or ingest path, but they did not run, and "irrelevant" is a judgement rather
than a result. CI's test job is nextest --workspace --all-features, so those 17 first execute
there, not locally.

Not run locally, and not to be read as passing: the workspace-wide test and clippy runs.
Workspace clippy is skipped deliberately — the environment's clippy is newer than this repo's
baseline and fires pre-existing lints everywhere, so it was isolated per crate. testsuite-sparql
is untouched by the export half, so its separate fmt/clippy pair does not apply there; note it is
excluded from the workspace, so the clean workspace check does not cover it, and the new
TransactError::ConfigGraphImportUnsupported variant is on a public enum that is not
#[non_exhaustive] — that crate's only reference to TransactError:: is inside a doc comment
(query_handler.rs:675), checked by hand, and CI's testsuite-sparql job is the real gate. CI is
the gate for all of these.

Eight mutations were run to prove the new tests non-vacuous — each fix reverted in turn, tests
confirmed failing, fix restored. Details in the commit messages.

Issues

Fixes #1574
Fixes #1847
Fixes #1846
Partially addresses #1859 — annotation syntax ships on all five RDF formats by default, for every
object shape, with --raw-reifies as the escape hatch. What remains is annotations written inside a
named graph, and the distinction matters: they are reported, not silently dropped, and the
defect that blocks them is in the arena's seal scan, not in export. The PSOT range that feeds
the seal comes back empty for a named graph, so the bundles never reach the arena and neither
forward lookup can see them; export cannot fix that from the read side. This is the boundary drawn
where the defect actually is, rather than work left unfinished — the seal-scan defect is written up
separately.
Partially addresses #1861 — the documentation half is done, including three spellings the issue does
not mention. The commit-time cascade warning is not built; see above for why, and for what would be
the better instrument if it is wanted.

[bplatz]

Approving. The analysis here is unusually good — several of the linked issues get their premises correctly overturned rather than taken at face value, and the is_system_graph find, the "every export test asserted a failure" observation, and the decision not to ship a refusal whose remedy sends the user in a circle are all right calls.

That said, I'd ask you to make your own assessment of the items below before merging. I think three of them are load-bearing for the PR's headline claim. I built the branch and reproduced all three, so these aren't reading-the-diff guesses — but you know this code better than I do and may see a reason they're fine or handled elsewhere.

Worth looking at first

1. Untranslated overlay rows bypass f:reifies* suppression (export.rs:1988). On a never-indexed ledger with a literal-object annotated edge, the default export emits a partial f:reifies* bundle — so #1859's defect is back on exactly the path #1574 opens, and the round trip plants a reserved predicate into the target ledger as ordinary user data.

2. xsd:decimal objects lose their annotation on every path, including a sealed arena (export.rs:417). Ref, string, lang, integer, boolean and date all keep their marker; decimal alone drops it and leaves an orphan reifier. export_annotation_object_shapes covers Ref/Lit/Lang but no numeric literal, which is why it's green.

3. Export is byte-nondeterministic run to run (export.rs:1928). Same ledger, same command, six distinct outputs in six runs, in both Turtle and N-Triples. The triple set is identical — it's purely ordering out of surviving_untranslated's HashMap. The body sets intra-block order aside as meaningless in Turtle, which is right for indexed-vs-not but doesn't cover diffing, checksumming or reproducible backups.

Also worth a look, lower stakes

4. run_remote_rdf discards the stats and the new headers, so #1847's fix doesn't reach the CLI whenever a local server is running (commands/export.rs:452).
5. The HTTP surface emits x-fluree-export-named-graphs-omitted on a single-graph request where the CLI correctly stays quiet (routes/export.rs:156).
6. annotations_out_of_scope has a CLI warning but no header (routes/export.rs:168).
7. FLUREE_EXPORT_ANNOTATION_SCAN is read with .is_ok(), so =0 enables it (export_annotations.rs:167).
8. raw_reifies is missing from the POST /export field table that names it as the remedy (server-integration.md:2066).
9. Perf: batch_reifiers is O(dataset) rather than O(annotations) (export.rs:417), and the arena reader is rebuilt per ColumnBatch (export_annotations.rs:218).
10. graph_sid_of skips the find_subject_sid round-trip its sibling documents as required (export_builder.rs:267), and --at resolves annotations at HEAD on the arena-less path (export_annotations.rs:259).

None of this argues against the shape of the PR — the export half is clearly the right design, and the test restructuring is the most valuable part of it. Items 1-3 are all about object shapes the new tests don't happen to cover, which is the same failure mode the PR itself diagnoses in the old suite.

[bplatz]

Untranslated overlay rows bypass f:reifies* suppression.

These three emitters — write_raw_po_turtle, write_raw_flake_ntriples (:1894) and merge_untranslated_jsonld (:983) — resolve flake.p directly and never call ann.is_reifies_row. Only write_batch does. So any reifies flake that misses V3 translation is emitted raw, which puts #1859's defect back on exactly the path #1574 opens.

Reproduced on this branch:

fluree create la
fluree insert la --format turtle -e \
  'ex:alice ex:label "Alice"@en ~ ex:claimL {| ex:src ex:a |} .'
fluree export la --format turtle        # never indexed

<http://example.org/claimL>
    <http://example.org/src> <http://example.org/a> ;
    <https://ns.flur.ee/db#reifiesObject> "Alice"@en .    # leaked
<http://example.org/alice>
    <http://example.org/label> "Alice"@en .               # no ~ marker

Two things make this worse than a leak:

• It is a partial bundle. The other six facts translated fine and were suppressed by is_reifies_row, so the output is neither RDF 1.2 annotation syntax nor the old f:reifies* form.
• Round-tripping it lands f:reifiesObject in the target ledger as ordinary user data, and the annotation is gone. create --from on that file warns Annotation arena was not sealed (no attachment events resolved). Bulk TriG import persists reserved-graph triples into an unreachable graph id #1846's guard doesn't catch it — that guards reserved graphs, not reserved predicates.

The base row also loses its marker: none of the untranslated emitters take a reifiers slice, so an annotated edge whose own base row is untranslated can never get ~ <r>.

The comment in export_never_indexed_groups_each_subject_once notes that a lang tag and a decimal both miss translation without a persisted dictionary — and all three never-indexed annotation tests use ex:knows ex:bob, the one object shape that translates.

[aaj3f]

Confirmed, and reproduced in all three formats rather than Turtle alone — the same partial bundle reaches N-Triples and JSON-LD output. Fixed in d36b71196.

The obvious fix is a trap, and it is worth saying why before the fix itself. Filtering f:reifies* out of the untranslated set at its single collection point is one line, and config.annotations is already in scope there. It would also have made this strictly worse: the unresolved counter only moves where the translated path calls note_bundle_in_scope, so a silent filter converts a visible wrong answer into an invisible one — no marker, no bundle, no number. A leaked reserved triple is at least evidence. An annotation that vanishes without trace is not.

So suppression and accounting had to be the same change. And that is also what makes your second paragraph — the base row never getting its marker — the same defect rather than a separate one: emitting the marker needs the reifier, and reporting the omission needs to know there was one. Both want the untranslated path to consult the probe rather than merely be filtered by it. They are fixed together; 1b is not deferred.

resolve_untranslated partitions the set once, notes every suppressed bundle, and resolves every reifier in one live_reifiers call. Deliberately not one probe per row — your finding below about batch_reifiers being O(dataset) is the reason, and a second per-row probe would have been the wrong direction for a cost already under review. When that one is remediated this call site should extend rather than need rework.

Each format emits through the path its translated rows already use: Turtle ~ <r>, N-Triples a triple term under rdf:reifies, JSON-LD @annotation via the existing annotated_jsonld_values.

is_any_reifies(&Sid) now sits with the seven individual checks in namespaces.rs, and EdgeKey::from_reifies_facts's inline seven-way disjunction calls it — a seven-way || written twice is where an eighth predicate gets added to one copy.

On your diagnosis of why the old tests missed this: you are right and it is the part I would not want softened. All three never-indexed annotation tests used ex:knows ex:bob, the one object shape that translates. That is the same failure this PR diagnoses in the suite it replaced, repeated by me in the replacement. A green suite over a homogeneous fixture is exactly as uninformative as the one it replaced.

[aaj3f]

Correcting one thing in my reply above. I said is_any_reifies(&Sid) "now sits with the seven individual checks". It should never have existed: is_reserved_reifies_predicate already covered exactly that set — re-exported from fluree_db_core, unit-tested in namespaces.rs, used by the indexer, and named in annotation_arena/bundle.rs as the canonical check. I found it by reading AnnotationContext::new, which already calls it by name.

So my commit argued that a seven-way disjunction written twice is where an eighth predicate gets added to only one copy, and in making that argument I created a third copy. Removed in fdd377729; both call sites now use the existing helper. No behaviour change — the sets were identical — but the claim in the reply was wrong and the duplication was the exact hazard it named.

[bplatz]

Same gap as write_raw_po_turtle (:1988) — this path resolves flake.p with no is_reifies_row check, so a reifies flake that missed V3 translation reaches the JSON-LD output as an ordinary property. See that comment for the repro and the round-trip consequence.

[aaj3f]

Same defect, same fix — see the reply on export.rs:1988. The JSON-LD path now routes untranslated rows through annotated_jsonld_values, the same helper the translated rows use, so the @annotation shape is identical rather than reimplemented.

Confirmed reproduced here before fixing: "https://ns.flur.ee/db#reifiesObject": {"@value":"Alice","@language":"en"} in the node body, and no @annotation on the edge.

[bplatz]

xsd:decimal objects lose their annotation on every lookup path, including a sealed arena.

The EdgeKey built here is the seek key, and decimal appears not to match what the arena stored. Measured on create --from (sealed arena, the best case):

ex:pRef  ex:bob                    ~ ex:cRef    ok
ex:pStr  "plain"                   ~ ex:cStr    ok
ex:pLang "hi"@en                   ~ ex:cLang   ok
ex:pInt  "42"^^xsd:integer         ~ ex:cInt    ok
ex:pDec  "1.5"^^xsd:decimal                     marker dropped
ex:pBool "true"^^xsd:boolean       ~ ex:cBool   ok
ex:pDate "2020-01-01"^^xsd:date    ~ ex:cDate   ok

ex:cDec is then emitted as an orphan reifier that nothing points at. It is loud (annotations_unresolved fires), but the documented known gap is named-graph annotations — this is a plain default-graph edge. export_annotation_object_shapes covers Ref/Lit/Lang but no numeric literal, which is why it passes. Worth checking whether it is the same decimal representation mismatch that has bitten typed equality elsewhere.

---

Separately: the cost here is O(dataset), not O(annotations).

An EdgeKey is built for every non-reifies row of the whole ledger whenever annotations are present — a second full decode plus a Sid clone, a FlakeValue clone and an owned lang String per row. On a 10M-triple ledger with three annotations that is 10M constructions.

The comment frames this as deliberate path separation scoped to annotated ledgers, which is true, but the perf table measures only the annotation-free ledger, which short-circuits before reaching here. Driving from the set the annotation source already knows it covers (the arena's forward branch carries first_edge/last_edge per leaf, scanned is already a HashMap<EdgeKey, _>, the overlay is a BTreeMap) would make it proportional to the feature.

[aaj3f]

Confirmed, reproduced, and wider than reported. Fixed in 0ca5d05c3.

The mechanism is not a decimal representation mismatch — it is upstream of any comparison. The seek key needs a datatype Sid, and batch_reifiers asked resolve_datatype_sid(o_type):

let Some(dt) = resolver.store.resolve_datatype_sid(o_type) else {
    continue;
};

That returns None for the NUM_BIG_OVERFLOW arena, because the arena holds both overflow xsd:integer and xsd:decimal and the o_type alone cannot say which. So the continue skipped the row before a key was ever built — which is why it fails on every lookup path including a sealed arena, and why the reifier survives as an orphan.

resolve_datatype_sid_for_value exists for exactly this ambiguity and falls back to the decoded value's variant. It was added for #1329, where the same gap rendered arena-served big numerics with an empty @type. This call site had not adopted it. That is the whole fix.

Two ways it is wider than the report, both now in the fixture:

• Overflow-magnitude xsd:integer drops its annotation identically, since it shares that arena. Your fixture used "42"^^xsd:integer, small enough to miss the arena, so this half was invisible from it.
• A bare Turtle numeric parses as xsd:decimal. So the exposure is not people writing explicit decimal datatypes — it is anyone writing ex:score 0.9 the ordinary way, with nothing in the source to hint at why the annotation vanished.

Measured before and after, on create --from with a sealed arena:

before:  pRef ok   pStr ok   pLang ok   pInt ok   pBool ok   pDate ok   pDec DROPPED
         (and a separate fixture: bare 1.5 DROPPED, big xsd:integer DROPPED)
after:   all shapes keep their marker, counter silent

The regression test covers all three arena-served shapes and keeps a ref and a small integer as controls, so a regression that broke everything could not satisfy it.

[bplatz]

Export is byte-nondeterministic run to run whenever untranslated rows exist.

UntranslatedBySubject fixes which block a row lands in, but not the order inside it: the Vec<Flake> per subject keeps the iteration order of surviving_untranslated's HashMap<Flake, Flake> (latest.into_values(), default randomly-seeded hasher). The N-Triples path bypasses this struct entirely and iterates that Vec directly (:1482).

Same ledger, same command, no writes in between:

6x  fluree export nt --format ntriples | md5   ->  6 distinct outputs
6x  fluree export nt --format turtle   | md5   ->  6 distinct outputs
    sorted triple-set hash                      ->  identical (pure ordering)

The PR body sets intra-block predicate order aside as carrying no meaning in Turtle. That holds for comparing an indexed export against a non-indexed one, which is what it was written about. It doesn't cover diffing two exports of the same ledger, checksumming or content-addressing an export, or reproducible backups — all of which this breaks.

export_never_indexed_matches_indexed sorts before comparing, so it can't see it. A sort_unstable_by on (p, o) in surviving_untranslated would close it for all three formats at once.

[aaj3f]

Confirmed and fixed. surviving_untranslated now sorts before returning.

One change from your suggestion: I keyed the sort on the whole fact identity — s, p, o, dt, lang, list-index — rather than (p, o). Flake's hand-written Eq keys on (s, p, o, dt, m), so two surviving rows can differ in nothing but a language tag; a (p, o) key would leave those tied, and sort_unstable_by gives ties an unspecified order. Sorting on the full identity is total over the map's own key, so nothing can tie.

Pinned by a test that exports the same never-indexed ledger five times in five processes and compares bytes, with a mixed-shape fixture (lang tag and decimal, both of which miss V3 translation without a persisted dictionary). Removing the sort fails it at run 2, not run 5 — the reordering is strong rather than marginal, which is consistent with your six-for-six.

The part worth more than the fix: your byte-identical result could not reliably have shown otherwise. The PR body leans on an indexed-vs-never-indexed export coming out byte-identical at 633 bytes, cmp clean — and on this code that comparison had a random component on both sides. Two runs agreeing was luck, not evidence, and the conclusion drawn from it was not licensed. The same applies to the earlier reviewer's measurement of the same thing.

What the fixture can support is set equality, which is what the N-Triples comparison actually established. The byte-identity claim needs restating, and after this fix it becomes true rather than lucky — but it should be re-measured and re-worded rather than left standing on the old run. I am rewriting that paragraph in the body.

[bplatz]

#1847's fix does not reach the CLI's default path.

run_remote_rdf ends here: report_rdf_stats is never called, and RemoteClient::export_rdf returns resp.bytes() (remote_client.rs:1934) — the three new x-fluree-export-* headers the server just learned to send are discarded before they get here.

Whenever a local server is running, if !direct { if let Some(client) = context::try_server_route_client(dirs) { ... } } (:118-137) takes this branch. So fluree export mydb --format trig drops every named graph and prints nothing at all — the literal complaint in #1847, on what is the common path for anyone running a server.

Every new CLI test in this PR runs against a temp dir with no server configured, so only the local path is covered.

[bplatz]

.is_ok() tests for presence, so FLUREE_EXPORT_ANNOTATION_SCAN=0 enables the scan. docs/cli/export.md documents =1 as the way to force it, which implies =0 turns it off.

On a ledger with a sealed arena that silently trades a correct arena read for the lossy fallback plus an annotations_unresolved warning. The FLUREE_DISABLE_* flags in this tree read by presence and are named so presence-only is correct; a positively-named flag wants its value parsed.

[aaj3f]

Confirmed and fixed. FLUREE_EXPORT_ANNOTATION_SCAN=0 enabled the scan.

Your framing decided the fix: the FLUREE_DISABLE_* flags read by presence and are named so that presence-only is correct, and this one is not one of those. indexer_attachment_provider::force_annotation_bootstrap is the crate's other positively-named flag and already parses its value — so this is now the same shape rather than a new convention.

One deliberate structural choice: the parse is split from the env read, so env_flag_enabled(Option<&str>) is unit-testable without mutating a process-global. That is not only tidiness. An env var set inside a test is precisely the shared-process-state hazard behind a family of flakes we have been measuring today — threshold tests that pass isolated and under cargo nextest (one process per test) but fail under cargo test (threads in one process). Testing this one by setting the variable would have added an instance of the problem in order to check a fix.

The assertions pin =0, false and set-but-empty as off, and 1/true/TRUE as on. Reverting the parse to value.is_some() fails it.

The wire-level test that legitimately does need the variable set (export_scan_source) already lives in its own [[test]] target for the same reason, alongside telemetry_test — and it still selects the scan after this change, which is the other half of the check.

[bplatz]

Two costs stack in this arm, and edges here is every non-reifies row in the batch rather than only the annotated ones:

• AnnotationArenaReader::new is constructed inside live_reifiers, which runs once per ColumnBatch — so the branch/leaf caches are rebuilt and thrown away for every batch of the export.
• current_annotations_merged is then called per edge, walking the forward branch each time.

The trigger is ordinary: a sealed arena plus any attachment committed since the last index build. The perf table measures the annotation-free ledger, which never reaches here. Hoisting the reader to the export (the source is already chosen once per export) would at least stop the per-batch rebuild.

[bplatz]

scan_base_index_for_attachment_events_in computes let to_t = t.max(snapshot.t) (indexer_attachment_provider.rs:366), so this resolves annotations at HEAD rather than at the requested --at. The filter below only drops forward (t > as_of_t) and never restores backward.

For --at t:5 on a ledger where an annotation was asserted at t=3 and retracted at t=7, the range scan at to_t = snapshot.t no longer returns the bundle, so scanned has nothing — while the export cursor runs at to_t = 5, where the bundle rows are live and is_reifies_row suppresses them. The annotation disappears from a point-in-time export that should contain it, and since no edge is known to be annotated it doesn't even bump annotations_unresolved.

Narrow (arena-less + --at + a retracted annotation), but silent.

[aaj3f]

Confirmed and fixed in 13ccb5f35. Your reading of the mechanism was exact — t.max(snapshot.t) raising the bound to HEAD, and the filter below only able to drop forward, never restore.

The fix makes the bound the caller's rather than the function's. A seal pass genuinely does want every bundle the base carries, so the two seal callers now clamp it themselves and keep today's behaviour; the export passes the time it was actually asked for. That seemed better than a flag, since the two callers want different things for good reasons rather than one of them being wrong.

Reproduced and mutation-checked, because the fixture has three requirements and getting any of them wrong gives a test that passes without touching the bug:

• the ledger must be indexed past the requested t, or snapshot.t is 0 and the clamp is a no-op — my first attempt at this fixture had exactly that flaw and passed before the fix as well as after;
• the annotation must be retracted after the requested t, or it is live at HEAD anyway;
• the scan must be the annotation source, since that is the path the bound belongs to.

With all three: --at 1 emits the marker with the change, emits nothing with the clamp reinstated, and HEAD correctly shows neither.

Worth adding to your last sentence — "narrow, but silent". It is slightly worse than silent: because no edge is known to be annotated, annotations_unresolved does not move either, so the one signal designed to catch a dropped annotation is also absent. A point-in-time export came back clean and short.

[bplatz]

This builds the g component of the same seek key that ExportResolver::resolve_subject_sid builds the s component of, but with a bare encode_iri — skipping the find_subject_sid round-trip. That function's own doc comment says why it matters: "a Sid that differs in any position lands on the wrong span and reports 'no annotations' rather than failing."

Wherever a stored graph Sid differs from a freshly encoded one (namespace split, overflow/EMPTY namespace), every named-graph annotation probe returns empty — indistinguishable from the known named-graph gap, so the two couldn't be told apart once the seal-scan defect is fixed upstream.

[aaj3f]

Taking the round-trip here — find_subject_sid(&iri)?.unwrap_or_else(|| encode_iri(&iri)), matching what resolve_subject_sid already does — and in one more place you could not have seen: the same bare encode_iri is how I resolve the graph Sid in the stacked fix for the named-graph seal defect. Correct whether or not a fixture exists, so it goes in regardless of the rest of this comment.

You are right about the construct, and the divergence condition is narrower than "a stored Sid differs from a freshly encoded one". The two functions disagree in exactly one situation:

// encode_iri — split form whenever the PREFIX is registered
if let Some(&code) = namespace_reverse.get(canonical_prefix) {
    return Sid::new(code, canonical_suffix);
}

// find_subject_sid — additionally requires the SUBJECT KEY to exist
if let Some(&ns_code) = namespace_reverse.get(canonical_prefix) {
    if reverse_lookup_subject_key(ns_code, canonical_suffix)?.is_some() {
        return Ok(Some(Sid::new(ns_code, canonical_suffix)));
    }
}
Ok(find_full_iri_subject_fallback(iri)?.map(|(ns, _)| Sid::new(ns, iri)))

So it needs an IRI stored full-IRI while its prefix is separately registered. encode_iri then returns (code, "g1") against stored (full_iri_code, "http://…/g1") — different position under the derived Ord, landing on the wrong span and reporting "no annotations" rather than failing, exactly as resolve_subject_sid warns.

I could not construct that for a graph IRI, and I would rather ask than assert it is unconstructible. Instrumenting graph_sid_of to compute both and compare, then running five deliberately varied graph IRIs through create --from:

http://example.org/g1                            agree
urn:uuid:5f4d3c2b-1a09-4876-b5e4-3d2c1b0a9f8e    agree
http://example.org/deeply/nested/path/.../g3     agree
https://a.example/x#frag                         agree
tag:example.org,2026:g5                          agree

All five stored in split form under a freshly registered namespace (codes 13–17). The two shapes I can see reaching it are namespace-table overflow, and an IRI written before its namespace existed and registered by a later write. Have you seen either in practice? You found the construct, so you may know the shape that produces it — that is a question you can answer and I cannot, and it decides whether this is a latent class or a live defect.

Either way the fix lands, and I am describing it as removing the class rather than as recovering anything measured.

Incidentally, that same five-graph fixture reported Annotation arena was not sealed (no attachment events resolved) and emitted zero ~ markers — an independent reproduction of the named-graph gap at a scale I had not tried, which is useful evidence for the stacked fix.

[aaj3f]

Correcting myself: I said above I could not construct the divergence. I found it, and it is the namespace-overflow shape. Your finding is confirmed rather than merely conceded.

It is in the write path. NamespaceRegistry::sid_for_iri (fluree-db-transact/src/namespace.rs:492) is how every graph Sid is derived on transact:

let code = self.get_or_allocate(prefix);
if code == OVERFLOW {
    Sid::new(OVERFLOW, iri)   // full IRI as name
} else {
    Sid::new(code, suffix)
}

Read-side encode_iri has no OVERFLOW branch — an unregistered prefix falls to Sid::new(EMPTY, iri) — and ensure_code documents OVERFLOW as "a sentinel, never register it". So once the namespace table overflows, a graph is stored (OVERFLOW, iri) and read back (EMPTY, iri): identical name, different namespace code, wrong span under the derived Ord, silent "no annotations" exactly as you described.

My five-IRI fixture missed it because every namespace in it was allocated a real code; nothing I varied about IRI shape could reach it, because the trigger is table pressure rather than the IRI.

One methodological note, since it nearly went the other way. I had already committed to the round-trip here when I read LedgerSnapshot::build_reverse_graph, which derives graph Sids with a bare encode_iri and treats it as authoritative. That reads as "encode_iri is the graph-Sid convention, so the round-trip is wrong", and I came close to reversing on it. What settled it was going to where the value is written rather than arbitrating between two read sites — convention at a read site is not evidence about storage.

That turned up something adjacent I am filing separately rather than fixing here: build_reverse_graph has the same mismatch, keying graph Sid → GraphId with encode_iri on the novelty routing path, where an overflowed graph would be keyed (EMPTY, iri) against a stored (OVERFLOW, iri) and never match. Its ok_or_else error branch is also dead — LedgerSnapshot::encode_iri returns Some(..) unconditionally, so the "no matching namespace prefix" error it is written to raise cannot fire. Unmeasured, outside this PR's surface, and bigger than the corner you flagged.

Fix here is find_subject_sid(iri).ok().flatten().unwrap_or_else(|| encode_iri(iri)), matching ExportResolver::resolve_subject_sid.

[aaj3f]

Filed the core half as #1891 — outside your twelve, and unmeasured, so not a finding you missed.

The part worth thirty seconds of your time is the dead branch. build_reverse_graph does:

let sid = self.encode_iri(iri).ok_or_else(|| {
    Error::invalid_index("graph IRI ... has no matching namespace prefix")
})?;

and LedgerSnapshot::encode_iri is Some(self.encode_iri_inner(iri)) — unconditionally. The ok_or_else cannot fire. An unregistered graph IRI does not raise that error; it silently becomes (EMPTY, iri) and the routing lookup misses. encode_iri_strict is the variant that returns None, which is the tell that this call meant to use it.

Underneath it is the same OVERFLOW mismatch as here, on the novelty routing path. Mechanism read from source; symptom not reproduced. Detail and three candidate fixes are in #1891 — I deliberately recommended none, since it is a LedgerSnapshot API question rather than an export one.

Fix for this comment is landed in c8cbaf951, and in the stacked seal PR as 3d465481d under the same commit message so the two halves read as one change.

[bplatz]

raw_reifies is missing from this table and from the example body above, though ExportRequest accepts it (routes/export.rs:25) and the response-header section at :1066 tells clients to "re-request with raw_reifies". An implementor building against this page can't implement the escape hatch the page itself points at.

The breaking default change isn't stated here either — POST /export response bodies now carry RDF 1.2 annotation syntax instead of f:reifies* triples. Given this section is what third-party implementors build against, that seems like a Required-semantics item.

[aaj3f]

All three gaps confirmed and fixed in 6f7c9d972.

raw_reifies is now in the request field table, cross-referenced to the header that names it as the remedy — you were right that an implementor could not implement the escape hatch the page points at.

The new x-fluree-export-annotations-out-of-scope header is in the response table too.

And the breaking default is stated, as its own paragraph under the field table rather than buried in a row:

Breaking change in 4.2. Response bodies now carry RDF 1.2 annotation syntax for edge annotations — s p o ~ <r> in Turtle and TriG, a triple term under rdf:reifies in N-Triples and N-Quads, @annotation in JSON-LD — where previous versions emitted the underlying f:reifies* triples. A consumer that parsed those triples directly will not find them. Set raw_reifies to keep the old bytes.

Your reasoning for why it belongs on this page specifically — that it is what third-party implementors build against — is why it is stated in the reference rather than only in the PR description and the CLI guide.