From 2975ceddcef744a1c4aa5cd92431e7431c4342ab Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:01:54 +0200 Subject: [PATCH 1/8] cas: classify transient GC round failures and stream emulated blob publication Two defects surfaced by the `content_addressed_garbage_collection_log` scenario cards for issue #2233. Any `S3_ERROR` timeout during a GC round was recorded as an indistinguishable `Failed` outcome with a free-text error, and a failed round zeroed out the real counters and cleared `i_am_leader`, suppressing the heartbeat and provoking leadership ping-pong on a flaky backend. Transient error codes (`S3_ERROR`, `NETWORK_ERROR`, `ABORTED`, timeouts, `MEMORY_LIMIT_EXCEEDED`) now produce an `Aborted` outcome while keeping leadership, and `system.cas_gc_log` gains an `error_code` column alongside the `Aborted` outcome. Separately, the emulated blob-publication path materialized the whole blob body in memory (about 1 GiB for a 512 MiB blob) under a global mutex; it now streams the body instead. Related: https://github.com/Altinity/ClickHouse/issues/2233 Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../en/operations/system-tables/cas_gc_log.md | 5 +- .../Backend/CasObjectStorageBackend.cpp | 76 +++---- .../Backend/CasObjectStorageBackend.h | 6 +- .../ContentAddressedMetadataStorage.cpp | 4 + .../ContentAddressed/Gc/CasGc.cpp | 7 +- .../ContentAddressed/Gc/CasGc.h | 9 +- .../ContentAddressed/Gc/CasGcScheduler.cpp | 82 +++++-- .../ContentAddressed/Gc/CasGcScheduler.h | 15 +- src/Disks/tests/gtest_cas_gc_log.cpp | 211 +++++++++++++++++- .../ContentAddressedGarbageCollectionLog.cpp | 8 +- .../ContentAddressedGarbageCollectionLog.h | 6 +- 11 files changed, 363 insertions(+), 66 deletions(-) diff --git a/docs/en/operations/system-tables/cas_gc_log.md b/docs/en/operations/system-tables/cas_gc_log.md index 5fd04b4fb11d..eb422cddc4d1 100644 --- a/docs/en/operations/system-tables/cas_gc_log.md +++ b/docs/en/operations/system-tables/cas_gc_log.md @@ -38,7 +38,7 @@ specified (it is enabled by default in the shipped `config.xml`). - `gc_id` ([String](/sql-reference/data-types/string)) — The GC scheduler instance id (which mounter ran the round). - `trigger` ([Enum8](/sql-reference/data-types/enum)) — `Scheduled` (background tick) or `Manual` (`SYSTEM` command). - `round` ([UInt64](/sql-reference/data-types/int-uint)) — The GC round number (`0` on a `Start` row). -- `outcome` ([Enum8](/sql-reference/data-types/enum)) — `Unknown` (on a `Start` row), `Success` (led, folded, and completed), `NotALeader` (another replica holds the GC lease), `Deferred` (led but took the skip-unchanged fast path — no fold ran, because no changed shard reached the fold threshold and no graduation was due), or `Error` (the round threw). +- `outcome` ([Enum8](/sql-reference/data-types/enum)) — `Unknown` (on a `Start` row), `Success` (led, folded, and completed), `NotALeader` (another replica holds the GC lease), `Deferred` (led but took the skip-unchanged fast path — no fold ran, because no changed shard reached the fold threshold and no graduation was due), `Aborted` (the round threw a transient error — backend unavailability, a lost lease, a concurrent leader; the next scheduled round retries it), or `Error` (the round threw a non-transient error — investigate). - `candidates_marked` ([UInt64](/sql-reference/data-types/int-uint)) — Objects retired (marked) this round. - `objects_deleted` ([UInt64](/sql-reference/data-types/int-uint)) — Objects physically deleted this round. - `objects_absent` ([UInt64](/sql-reference/data-types/int-uint)) — Retire candidates found already absent. @@ -51,7 +51,8 @@ specified (it is enabled by default in the shipped `config.xml`). - `fence_outs` ([UInt64](/sql-reference/data-types/int-uint)) — Expired mounts fenced out by this round's heartbeat floor. - `anomalies` ([UInt64](/sql-reference/data-types/int-uint)) — Fold clamps surfaced (and survived) this round. A steady non-zero value warrants a look at the round log details. - `duration_ms` ([UInt64](/sql-reference/data-types/int-uint)) — The round wall-clock duration (on a `Finish` row). -- `error` ([String](/sql-reference/data-types/string)) — The exception text when `outcome = 'Error'`. +- `error` ([String](/sql-reference/data-types/string)) — The exception text when `outcome = 'Aborted'` or `'Error'`. +- `error_code` ([Int32](/sql-reference/data-types/int-uint)) — The exception code when `outcome = 'Aborted'` or `'Error'`; `0` otherwise. Key monitoring on this column rather than on the `error` text. On an `Aborted` or `Error` row the counters still report everything the round completed before it threw, and `round != 0` on such a row means the round's closing compare-and-swap committed and the failure hit only post-commit cleanup. - `ProfileEvents` ([Map(LowCardinality(String), UInt64)](/sql-reference/data-types/map)) — On a `Start`/`Finish` row, the per-round `ProfileEvents` delta (the `CAS*` counters and S3/disk events for this round). On a `Phase` row, **that phase's** delta, so `GROUP BY phase` over `ProfileEvents['S3ListObjects']` attributes the round's `LIST` budget to the phase that spent it. - `round_id` ([String](/sql-reference/data-types/string)) — The correlator for every row of one round attempt: its `Start`, each of its `Phase` rows, and its `Finish`. Minted per attempt, so unlike `round` it exists even for a round that never committed and for a round that never led. Group by this column to reconstruct one round. - `phase` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — The GC phase this row describes; empty on `Start`/`Finish`. See [Per-phase rows](#per-phase-rows) for the phase list. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index b7b04fdb401b..d27e01b412b0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -252,13 +252,6 @@ PutResult ObjectStorageBackend::nativeConditionalPut(const String & key, const S namespace { -/// Keep the emulated backend's publication memory bound to one materialized body at a time. -std::mutex & emulatedBlobPublicationMutex() -{ - static std::mutex mutex; - return mutex; -} - } /// True when an exception from `IObjectStorage::readObject` means "the object is simply not there". @@ -485,7 +478,7 @@ Token ObjectStorageBackend::emuWrite(const String & key, const String & bytes, c return emuMintToken(key, metadata ? metadata->etag : String{}, /*just_wrote=*/true); } -void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const String & bytes) +void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const String & envelope, ReadBuffer & payload, uint64_t payload_size) { if (object_storage->getType() != ObjectStorageType::Local) throw Exception( @@ -497,14 +490,30 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St const String root = object_storage->getCommonKeyPrefix(); const String destination_path = resolvePathRelativelyToBase(destination_object, root); const String temporary_path = resolvePathRelativelyToBase(temporary_object, root); - const auto existing_token_state = emu_token_state.find(key); + /// The body is STREAMED into the temporary file -- envelope, then a bounded copy of the payload -- + /// never materialized in memory. (An earlier revision accumulated envelope+payload in one String, + /// whose growth doubling made the peak allocation up to 2x the payload, and serialized every + /// publication behind a dedicated mutex just to bound that peak to one body at a time; streaming + /// removes both.) The destination stays untouched until the byte count has been validated: a short + /// or long source aborts on the temporary file, which is then removed. try { auto out = object_storage->writeObject(StoredObject(temporary_object), WriteMode::Rewrite); - out->write(bytes.data(), bytes.size()); + out->write(envelope.data(), envelope.size()); + const auto copy_result = blob_publication_detail::copyBlobPayloadBounded(payload, *out, payload_size); + if (!copy_result.exact(payload_size)) + { + out->cancel(); + throw Exception( + ErrorCodes::CORRUPTED_DATA, + "ObjectStorageBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", + copy_result.has_excess ? "more than " : "", + copy_result.copied, + key, + payload_size); + } out->finalize(); - std::filesystem::rename(temporary_path, destination_path); } catch (...) { @@ -517,7 +526,21 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St /// existing disambiguator is sufficient: if the next observation sees the same ETag, it returns /// a token distinct from the old incarnation; if the ETag changed, emuMintToken resets the state /// to that new ETag. With no existing state, this backend has issued no same-process stale token - /// that needs fencing. The post-rename increment cannot allocate or throw. + /// that needs fencing. The post-rename increment cannot allocate or throw. `emu_mutex` spans the + /// rename and the bump so a concurrent emulated observation never sees the new incarnation with + /// the old disambiguator. + std::lock_guard lock(emu_mutex); + const auto existing_token_state = emu_token_state.find(key); + try + { + std::filesystem::rename(temporary_path, destination_path); + } + catch (...) + { + std::error_code cleanup_error; + std::filesystem::remove(temporary_path, cleanup_error); + throw; + } if (existing_token_state != emu_token_state.end()) ++existing_token_state->second.second; } @@ -872,32 +895,9 @@ void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) if (mode != Mode::Native) { - /// The emulated adapter's writes are whole-body operations. Serialize materialization so - /// concurrent publications retain the existing one-body peak-memory bound. - std::lock_guard publish_lock(emulatedBlobPublicationMutex()); - - String body = streaming->fresh_envelope; - blob_publication_detail::BlobPayloadCopyResult copy_result; - { - WriteBufferFromString out(body, AppendModeTag{}); - copy_result = blob_publication_detail::copyBlobPayloadBounded(*payload, out, streaming->payload_size); - if (copy_result.exact(streaming->payload_size)) - out.finalize(); - else - out.cancel(); - } - - if (!copy_result.exact(streaming->payload_size)) - throw Exception( - ErrorCodes::CORRUPTED_DATA, - "ObjectStorageBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", - copy_result.has_excess ? "more than " : "", - copy_result.copied, - request.destination_key, - streaming->payload_size); - - std::lock_guard lock(emu_mutex); - emuPublishBlobAtomically(request.destination_key, body); + /// Streams straight into the temporary file and renames -- see emuPublishBlobAtomically. + emuPublishBlobAtomically( + request.destination_key, streaming->fresh_envelope, *payload, streaming->payload_size); return; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index bd369d4e3603..7344c8fbede4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -264,7 +264,11 @@ class ObjectStorageBackend final : public Backend /// Write a complete blob body to a sibling temporary local object, then atomically replace `key` /// and advance any existing same-ETag disambiguator. A failure before the rename leaves the old /// destination and its token state untouched and cleans the temporary. - void emuPublishBlobAtomically(const String & key, const String & bytes); + /// Streams `envelope` + exactly `payload_size` bytes of `payload` into a temporary sibling of + /// `key`, then renames it into place -- nothing is visible at the destination until the byte count + /// has been validated, and the rename keeps publication atomic. Takes `emu_mutex` itself (for the + /// rename + token-state bump only); the caller must NOT hold it. + void emuPublishBlobAtomically(const String & key, const String & envelope, ReadBuffer & payload, uint64_t payload_size); /// Return the current emulated token for a key we just read/HEAD'd, reflecting its on-disk etag — /// does NOT advance the same-etag disambiguator (that only applies to a just-completed write). Token emuObserveToken(const String & key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index c9cf2b166389..447fe908e45e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -547,6 +547,9 @@ Cas::GcRoundLogger ContentAddressedMetadataStorage::makeGcRoundLogger() const case Cas::GcRoundLogRecord::Outcome::Deferred: e.outcome = ContentAddressedGarbageCollectionLogElement::DEFERRED; break; + case Cas::GcRoundLogRecord::Outcome::Aborted: + e.outcome = ContentAddressedGarbageCollectionLogElement::ABORTED; + break; } e.round = r.round; e.candidates_marked = r.candidates_marked; @@ -562,6 +565,7 @@ Cas::GcRoundLogger ContentAddressedMetadataStorage::makeGcRoundLogger() const e.anomalies = r.anomalies; e.duration_ms = r.duration_ms; e.error = r.error; + e.error_code = r.error_code; e.profile_events = r.profile_events; e.round_id = r.round_id; e.phase = r.phase; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 0a0e7cf2725c..358f619d5b34 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -348,9 +348,12 @@ void Gc::runNamespaceJanitorPage( t.metric("leaked", janitor_result.leaked); } -RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool allow_steal, UniversePolicy policy) +RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool allow_steal, UniversePolicy policy, + RoundReport * progress) { - RoundReport report; + RoundReport local_report; + RoundReport & report = progress ? *progress : local_report; + report = RoundReport{}; GcState state; Token state_token; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index 2754fe433c19..93209b70a3ff 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -429,8 +429,15 @@ class Gc /// `policy` is the destructive gate's universe seam — see `UniversePolicy`. Production passes /// nothing; a test whose subject is the suppressed gate passes `StageA_Suppressed` here, which is /// the only way to reach that posture. + /// `progress` (optional) is the caller's window into a round that THROWS: the round accumulates its + /// report directly in `*progress` (reset at entry) as each phase completes, so on an exception the + /// caller still sees everything the round durably did before it died -- `round` is stamped only + /// after the round's single `gc/state` CAS commits, so `progress->round != 0` on a failed round + /// proves the round committed and died in the post-CAS tail. On the success path `*progress` equals + /// the returned report. RoundReport runRegularRound(std::function on_lease_acquired = {}, bool allow_steal = true, - UniversePolicy policy = UniversePolicy::kDefault); + UniversePolicy policy = UniversePolicy::kDefault, + RoundReport * progress = nullptr); /// Advisory heartbeat: bump /gc/hb to {gc_id, hb_seq+1}. Best-effort (a lost CAS is /// harmless — the next pulse retries). Touches NO Gc instance state. Static by design. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp index f265444576a8..0ae8351de5ba 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp @@ -12,9 +12,35 @@ #include #include +namespace DB::ErrorCodes +{ + extern const int S3_ERROR; + extern const int NETWORK_ERROR; + extern const int ABORTED; + extern const int TIMEOUT_EXCEEDED; + extern const int SOCKET_TIMEOUT; + extern const int MEMORY_LIMIT_EXCEEDED; +} + namespace DB::Cas { +bool isTransientGcRoundError(int code) +{ + /// Codes that name a condition which clears without intervention: the backend refused or timed out + /// (`S3_ERROR`, `NETWORK_ERROR`, `TIMEOUT_EXCEEDED`, `SOCKET_TIMEOUT`), another actor legitimately + /// moved shared state (`ABORTED` -- the round CAS's own "another leader advanced it"), or memory + /// pressure hit a manual round running on a budgeted query thread (`MEMORY_LIMIT_EXCEEDED`). + /// Everything else -- notably `LOGICAL_ERROR`, `CORRUPTED_DATA`, `BAD_ARGUMENTS` -- stays + /// non-transient BY OMISSION: an unrecognised code must read as a real failure, never as noise. + return code == ErrorCodes::S3_ERROR + || code == ErrorCodes::NETWORK_ERROR + || code == ErrorCodes::ABORTED + || code == ErrorCodes::TIMEOUT_EXCEEDED + || code == ErrorCodes::SOCKET_TIMEOUT + || code == ErrorCodes::MEMORY_LIMIT_EXCEEDED; +} + namespace { /// Non-zero events of a per-round snapshot, keyed by event name. The snapshot is already a @@ -192,9 +218,29 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe Rec fin = start; fin.event_type = Rec::EventType::Finish; + /// Lives OUTSIDE the try and is filled progressively by the round (the `progress` out-parameter of + /// `runRegularRound`), so the Finish row of a THROWING round still carries everything the round + /// durably did before it died -- `round != 0` on such a row proves the round's `gc/state` CAS + /// committed and the failure hit the post-CAS tail. + Cas::RoundReport rep; + const auto fill_counters = [&fin](const Cas::RoundReport & r) + { + fin.round = r.round; + fin.candidates_marked = r.candidates; + fin.objects_deleted = r.deleted; + fin.objects_absent = r.absent; + fin.objects_replaced = r.replaced; + fin.objects_spared = r.spared; + fin.manifests_deleted = r.manifests_deleted; + fin.entries_condemned = r.condemned; + fin.entries_graduated = r.graduated; + fin.entries_redeleted = r.redeleted; + fin.fence_outs = r.fence_outs; + fin.anomalies = r.anomalies.size(); + }; try { - const Cas::RoundReport rep = round_gc.runRegularRound(std::move(on_lease_acquired), allow_steal); + (void)round_gc.runRegularRound(std::move(on_lease_acquired), allow_steal, Cas::UniversePolicy::kDefault, &rep); if (rep.acquired_lease) { /// Keep health state per scheduler. Process-global gauges cannot distinguish multiple @@ -210,18 +256,7 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe fin.outcome = !rep.acquired_lease ? Rec::Outcome::NotALeader : rep.deferred ? Rec::Outcome::Deferred : Rec::Outcome::Success; - fin.round = rep.round; - fin.candidates_marked = rep.candidates; - fin.objects_deleted = rep.deleted; - fin.objects_absent = rep.absent; - fin.objects_replaced = rep.replaced; - fin.objects_spared = rep.spared; - fin.manifests_deleted = rep.manifests_deleted; - fin.entries_condemned = rep.condemned; - fin.entries_graduated = rep.graduated; - fin.entries_redeleted = rep.redeleted; - fin.fence_outs = rep.fence_outs; - fin.anomalies = rep.anomalies.size(); + fill_counters(rep); fin.duration_ms = std::chrono::duration_cast( std::chrono::steady_clock::now() - t0).count(); fin.profile_events = collect_profile_events(); @@ -230,8 +265,10 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe } catch (...) { - fin.outcome = Rec::Outcome::Failed; + fin.error_code = getCurrentExceptionCode(); + fin.outcome = isTransientGcRoundError(fin.error_code) ? Rec::Outcome::Aborted : Rec::Outcome::Failed; fin.error = getCurrentExceptionMessage(false); + fill_counters(rep); fin.duration_ms = std::chrono::duration_cast( std::chrono::steady_clock::now() - t0).count(); fin.profile_events = collect_profile_events(); @@ -342,8 +379,21 @@ void CasGcScheduler::loop() catch (...) { /// Idempotent round - the next tick retries; failures must never kill the pacing thread. - /// runRoundLogged already emitted the Aborted Finish row before rethrowing. - i_am_leader.store(false, std::memory_order_relaxed); + /// runRoundLogged already emitted the classified (Aborted/Failed) Finish row before rethrowing. + /// + /// Leadership is dropped only on a NON-transient failure. Dropping it on every failure + /// silenced the advisory heartbeat for a whole interval (`heartbeatLoop` gates its pulses on + /// `i_am_leader`), and when the failure was itself a backend outage the durable lease + /// `(owner, seq)` was frozen too -- together exactly the two-of-two dead-leader signature + /// `acquireOrRenewLease` steals on. A live leader blocked on a flaky store was then deposed, + /// and every handover forces the successor into a full fold: more single-attempt conditional + /// writes against the same flaky backend, a self-reinforcing loop. Keeping the flag keeps the + /// pulses; the lease protocol stays authoritative -- a mounter that really died stops pulsing + /// with or without this flag. A non-transient failure still clears it: a logic-broken leader + /// must stay depositable, and with the flag held its heartbeat would keep beating and no + /// follower could ever steal a lease whose holder cannot complete a round. + if (!isTransientGcRoundError(getCurrentExceptionCode())) + i_am_leader.store(false, std::memory_order_relaxed); tryLogCurrentException(log, "CA GC round failed (will retry next tick)"); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h index 2cce48399b66..a01186507074 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h @@ -27,7 +27,11 @@ struct GcRoundLogRecord /// -- no fold, no pre-CAS deletes, no `gc/state` CAS. Distinct from `Success` so a reader of /// `system.cas_gc_log` (or this scheduler's own log line) can tell a round /// that genuinely folded and found nothing apart from one that never folded at all. - enum class Outcome { Unknown, Success, NotALeader, Failed, Deferred }; + /// `Aborted`: the round threw an exception whose code names a transient condition (backend + /// unavailability, a lost lease, a concurrent leader) -- the next scheduled round retries it and + /// nothing durable is wrong. `Failed` is reserved for everything else (a logic error, corrupted + /// data, an unclassified code): fail-closed, an unrecognised failure reads as real. + enum class Outcome { Unknown, Success, NotALeader, Failed, Deferred, Aborted }; enum class Trigger { Scheduled, Manual }; EventType event_type = EventType::Start; @@ -51,6 +55,9 @@ struct GcRoundLogRecord UInt64 anomalies = 0; /// fold clamps surfaced (never wedging) this round UInt64 duration_ms = 0; String error; + /// `getCurrentExceptionCode()` of the failure on an `Aborted`/`Failed` Finish row; 0 otherwise. + /// The structured twin of `error`: oracles and operators key on this, never on message wording. + Int32 error_code = 0; /// On a `Start`/`Finish` row: the whole round's delta. On a `Phase` row: THAT PHASE's delta. std::map profile_events; @@ -75,6 +82,12 @@ struct GcRoundLogRecord using GcRoundLogger = std::function; +/// True when an exception code names a condition that clears by itself -- the backend was unreachable +/// or slow, or another actor legitimately moved shared state -- so the next scheduled round is the +/// retry. False for everything else, deliberately including any code not on the list: an unrecognised +/// failure must read as a real one. +bool isTransientGcRoundError(int code); + /// Paces regular content-addressed garbage-collection rounds for one pool. The scheduler does not /// implement the GC protocol: `Cas::Gc` owns lease acquisition, work deduplication, and the /// split-brain-safe round operations, so schedulers on different mounters may run independently. diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index 5dbd654343d9..e48ad0e2ff1f 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -8,6 +8,8 @@ #include +#include +#include #include #include @@ -26,6 +28,7 @@ namespace DB::ErrorCodes { extern const int BAD_ARGUMENTS; + extern const int NETWORK_ERROR; } using namespace DB::Cas; @@ -289,7 +292,10 @@ TEST(CASGCLog, AbortedFinishOnThrowingRound) ASSERT_EQ(round_rows.size(), 2u) << "a throwing round still emits a Start and a (Aborted) Finish"; EXPECT_EQ(round_rows[0].event_type, Rec::EventType::Start); EXPECT_EQ(round_rows[1].event_type, Rec::EventType::Finish); - EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Failed); + EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Failed) + << "BAD_ARGUMENTS is not on the transient list, so the row must read as a real failure"; + EXPECT_EQ(round_rows[1].error_code, DB::ErrorCodes::BAD_ARGUMENTS) + << "the Finish row must carry the structured exception code, not only the message text"; EXPECT_FALSE(round_rows[1].error.empty()) << "a failed Finish must carry the exception text"; EXPECT_EQ(round_rows[1].disk_name, "ca"); EXPECT_FALSE(round_rows[1].gc_id.empty()); @@ -299,6 +305,209 @@ TEST(CASGCLog, AbortedFinishOnThrowingRound) << "every row of a FAILED round must still correlate through round_id"; } +/// A round that dies with a TRANSIENT code -- the backend was unreachable, timed out, or another +/// actor moved shared state -- must be classified `Aborted`, not `Failed`: the next scheduled round +/// is the retry and nothing durable is wrong. The classifier keys on the exception CODE +/// (`isTransientGcRoundError`), never on message wording. +class NetworkThrowingBackend : public InMemoryBackend +{ +public: + ListPage list(const String & prefix, const String & cursor, size_t limit) override + { + if (arm) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected backend outage"); + return InMemoryBackend::list(prefix, cursor, limit); + } + std::atomic arm{false}; +}; + +TEST(CASGCLog, TransientThrowIsClassifiedAborted) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + + std::vector rows; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) { rows.push_back(r); }); + + backend->arm.store(true); + EXPECT_THROW(sched.runOneRoundNow(Rec::Trigger::Manual), DB::Exception); + + const std::vector round_rows = roundRowsOnly(rows); + ASSERT_EQ(round_rows.size(), 2u); + EXPECT_EQ(round_rows[1].event_type, Rec::EventType::Finish); + EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Aborted) + << "NETWORK_ERROR names a transient condition; the row must not read as a GC defect"; + EXPECT_EQ(round_rows[1].error_code, DB::ErrorCodes::NETWORK_ERROR); + EXPECT_FALSE(round_rows[1].error.empty()); +} + +/// The classifier itself, pinned direct: the transient list is exact and everything else fails closed. +TEST(CASGCLog, TransientErrorClassifierFailsClosed) +{ + EXPECT_TRUE(DB::Cas::isTransientGcRoundError(DB::ErrorCodes::NETWORK_ERROR)); + EXPECT_FALSE(DB::Cas::isTransientGcRoundError(DB::ErrorCodes::BAD_ARGUMENTS)); + EXPECT_FALSE(DB::Cas::isTransientGcRoundError(0)); + EXPECT_FALSE(DB::Cas::isTransientGcRoundError(-1)); +} + +/// A backend that lets the round's FIRST `gc/state` CAS (the lease acquire/renew) through and throws +/// a transient error on the SECOND (the round-closing commit). The round therefore does all of its +/// pre-CAS work -- including condemning the dropped part -- and dies at `round_commit`. +class StateCommitThrowingBackend : public InMemoryBackend +{ +public: + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override + { + if (arm && key.ends_with("gc/state") && ++state_puts_since_arm >= 2) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected outage on the round-closing CAS"); + return InMemoryBackend::casPut(key, bytes, expected, meta); + } + std::atomic arm{false}; + std::atomic state_puts_since_arm{0}; +}; + +/// The Finish row of a THROWING round must still carry the counters of everything the round did +/// before it died. Before this existed, the exception path emitted a row with `round = 0` and every +/// counter zero, so a round that condemned entries and then lost its commit CAS was +/// indistinguishable from a round that never got past the lease. +TEST(CASGCLog, AbortedFinishCarriesProgressiveCounters) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, + PoolConfig{.pool_prefix = "p", .server_root_id = "test", .gc_fold_max_defer_rounds = 0}); + const RootNamespace ns{"srv1/tbl"}; + + publishPart(store, ns.string(), "all_0_0_0", "hello-progressive-counters"); + store->dropRef(ns, "all_0_0_0"); + store->renewWatermarkOnce(); + + std::vector rows; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) { rows.push_back(r); }); + + backend->arm.store(true); + EXPECT_THROW(sched.runOneRoundNow(Rec::Trigger::Manual), DB::Exception); + + const std::vector round_rows = roundRowsOnly(rows); + ASSERT_EQ(round_rows.size(), 2u); + const Rec & fin = round_rows[1]; + EXPECT_EQ(fin.outcome, Rec::Outcome::Aborted); + EXPECT_EQ(fin.error_code, DB::ErrorCodes::NETWORK_ERROR); + EXPECT_EQ(fin.round, 0u) << "the commit CAS never landed, so the round number must stay unstamped"; + EXPECT_GT(fin.candidates_marked + fin.entries_condemned + fin.entries_graduated + + fin.entries_redeleted + fin.objects_deleted + fin.fence_outs, 0u) + << "the pre-CAS work the round performed must survive into its failure row"; +} + +/// The pacing loop drops leadership only on a NON-transient round failure. A transient failure +/// (backend outage class) keeps `i_am_leader` set, so the advisory heartbeat keeps pulsing and a +/// live leader blocked on a flaky store is not deposed -- dropping the flag on every failure was +/// half of the dead-leader signature (`!incumbent_renewed && !hb_alive`) and produced leadership +/// ping-pong under backend fault windows. A non-transient failure must still clear the flag: a +/// logic-broken leader has to stay depositable. +class ModalThrowingBackend : public InMemoryBackend +{ +public: + enum Mode : int { Off = 0, Transient = 1, Logic = 2 }; + ListPage list(const String & prefix, const String & cursor, size_t limit) override + { + const int m = mode.load(); + if (m == Transient) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected backend outage"); + if (m == Logic) + throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected logic failure"); + return InMemoryBackend::list(prefix, cursor, limit); + } + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override + { + if (key.ends_with("gc/hb")) + ++hb_puts; + return InMemoryBackend::casPut(key, bytes, expected, meta); + } + std::atomic mode{Off}; + std::atomic hb_puts{0}; +}; + +TEST(CASGCScheduler, TransientRoundFailureKeepsLeadershipAndHeartbeat) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + + std::mutex rows_mutex; + std::condition_variable rows_cv; + std::vector finishes; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) + { + if (r.event_type != Rec::EventType::Finish) + return; + std::lock_guard g(rows_mutex); + finishes.push_back(r); + rows_cv.notify_all(); + }); + + const auto wait_for_finish = [&](size_t count) -> Rec + { + std::unique_lock lock(rows_mutex); + const bool ok = rows_cv.wait_for(lock, std::chrono::seconds(30), [&] { return finishes.size() >= count; }); + EXPECT_TRUE(ok) << "timed out waiting for Finish row #" << count; + return finishes.at(count - 1); + }; + /// Bounded poll for an ASYNC flag change. The loop stores `i_am_leader` after `runRoundLogged` + /// returns (after the Finish row was emitted), so the row alone is not a happens-before for the + /// flag -- poll to the expected value instead of asserting a racy instantaneous read. + const auto poll_leader = [&](bool expected) -> bool + { + for (int i = 0; i < 3000; ++i) + { + if (sched.gcHealth().is_leader == expected) + return true; + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + return sched.gcHealth().is_leader == expected; + }; + + sched.start(); + sched.requestRoundSoon(); + const Rec first = wait_for_finish(1); + EXPECT_TRUE(first.outcome == Rec::Outcome::Success || first.outcome == Rec::Outcome::Deferred) + << "outcome=" << static_cast(first.outcome); + EXPECT_TRUE(poll_leader(true)) << "a successful round must establish leadership"; + + backend->mode.store(ModalThrowingBackend::Transient); + sched.requestRoundSoon(); + const Rec aborted = wait_for_finish(2); + EXPECT_EQ(aborted.outcome, Rec::Outcome::Aborted); + /// Leadership kept => the advisory heartbeat keeps pulsing. Waiting for a NEW pulse after the + /// failed round is the happens-after proof that the flag survived; with the flag dropped the + /// heartbeat loop skips every pulse until the next successful round, and this wait times out. + const uint64_t hb_before = backend->hb_puts.load(); + bool pulsed = false; + for (int i = 0; i < 3000 && !pulsed; ++i) + { + pulsed = backend->hb_puts.load() > hb_before; + if (!pulsed) + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + EXPECT_TRUE(pulsed) << "a transient round failure must not silence the advisory heartbeat"; + EXPECT_TRUE(sched.gcHealth().is_leader) << "a transient round failure must not drop leadership"; + + backend->mode.store(ModalThrowingBackend::Logic); + sched.requestRoundSoon(); + const Rec failed = wait_for_finish(3); + EXPECT_EQ(failed.outcome, Rec::Outcome::Failed); + EXPECT_TRUE(poll_leader(false)) << "a non-transient round failure must still surrender leadership"; + + backend->mode.store(ModalThrowingBackend::Off); + sched.stop(); +} + /// Every row of one round -- its Start, each of its Phase rows, and its Finish -- carries the SAME /// non-empty `round_id`, and two rounds carry DIFFERENT ones. That is the property the column exists /// for: `round` is 0 on Start, is only known after the round's single `gc/state` CAS, and is absent on a diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp index fde269c4c999..2438cab2801f 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp @@ -21,7 +21,7 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri auto outcome_enum = std::make_shared(DataTypeEnum8::Values{ {"Unknown", static_cast(UNKNOWN)}, {"Success", static_cast(SUCCESS)}, {"NotALeader", static_cast(NOT_A_LEADER)}, {"Error", static_cast(FAILED)}, - {"Deferred", static_cast(DEFERRED)}}); + {"Deferred", static_cast(DEFERRED)}, {"Aborted", static_cast(ABORTED)}}); auto trigger_enum = std::make_shared(DataTypeEnum8::Values{ {"Scheduled", static_cast(SCHEDULED)}, {"Manual", static_cast(MANUAL)}}); auto lc_string = std::make_shared(std::make_shared()); @@ -38,7 +38,7 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"gc_id", std::make_shared(), "GC scheduler instance id (which mounter)."}, {"trigger", trigger_enum, "Scheduled (background tick) or Manual (SYSTEM command)."}, {"round", std::make_shared(), "GC round number (0 on Start)."}, - {"outcome", outcome_enum, "Unknown (Start) / Success (led, folded, and completed) / NotALeader (another replica holds the GC lease) / Deferred (led but took the skip-unchanged fast path -- no fold ran) / Error (the round threw)."}, + {"outcome", outcome_enum, "Unknown (Start) / Success (led, folded, and completed) / NotALeader (another replica holds the GC lease) / Deferred (led but took the skip-unchanged fast path -- no fold ran) / Aborted (the round threw a transient error -- backend unavailability, a lost lease, a concurrent leader -- and the next scheduled round retries) / Error (the round threw a non-transient error)."}, {"candidates_marked", std::make_shared(), "Objects retired (marked) this round."}, {"objects_deleted", std::make_shared(), "Objects physically deleted this round."}, {"objects_absent", std::make_shared(), "Retire candidates found already absent."}, @@ -51,7 +51,8 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"fence_outs", std::make_shared(), "Expired mounts fenced out by this round's heartbeat floor."}, {"anomalies", std::make_shared(), "Fold clamps surfaced (and survived) this round; steady >0 warrants a look at the round log details."}, {"duration_ms", std::make_shared(), "Round wall-clock duration (Finish)."}, - {"error", std::make_shared(), "Exception text when outcome = Error."}, + {"error", std::make_shared(), "Exception text when outcome = Aborted or Error."}, + {"error_code", std::make_shared(), "Exception code when outcome = Aborted or Error; 0 otherwise. The structured twin of `error`: key monitoring on this column, not on message text."}, {"ProfileEvents", std::make_shared(lc_string, std::make_shared()), "On a Start/Finish row: the per-round ProfileEvents delta (the Cas* counters and S3 events for this round). On a Phase row: THAT PHASE's delta, so `GROUP BY phase` over `ProfileEvents['S3ListObjects']` attributes the round's LIST budget to the phase that spent it. Empty on the `meta_pool_wait` row by construction — that phase's work runs on other threads (read its `phase_metrics` instead)."}, {"round_id", std::make_shared(), @@ -92,6 +93,7 @@ void ContentAddressedGarbageCollectionLogElement::appendToBlock(MutableColumns & columns[i++]->insert(anomalies); columns[i++]->insert(duration_ms); columns[i++]->insert(error); + columns[i++]->insert(error_code); { Map map; map.reserve(profile_events.size()); diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.h b/src/Interpreters/ContentAddressedGarbageCollectionLog.h index 9cbdbd3525f6..b7ffdc734c75 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.h +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.h @@ -15,7 +15,10 @@ struct ContentAddressedGarbageCollectionLogElement /// `DEFERRED`: the round acquired the GC lease and took the skip-unchanged fast path -- no fold, no /// pre-CAS deletes, no `gc/state` CAS. Kept distinct from `SUCCESS` so a query against this table can /// tell a round that genuinely folded and found nothing apart from one that never folded at all. - enum Outcome : int8_t { UNKNOWN = 1, SUCCESS = 2, NOT_A_LEADER = 3, FAILED = 4, DEFERRED = 5 }; + /// `ABORTED`: the round threw an exception whose code names a transient condition (backend + /// unavailability, a lost lease, a concurrent leader); the next scheduled round retries it. + /// `FAILED` is everything else -- fail-closed, an unclassified error reads as real. + enum Outcome : int8_t { UNKNOWN = 1, SUCCESS = 2, NOT_A_LEADER = 3, FAILED = 4, DEFERRED = 5, ABORTED = 6 }; enum Trigger : int8_t { SCHEDULED = 1, MANUAL = 2 }; time_t event_time = 0; @@ -42,6 +45,7 @@ struct ContentAddressedGarbageCollectionLogElement UInt64 anomalies = 0; /// fold clamps surfaced this round UInt64 duration_ms = 0; String error; + Int32 error_code = 0; /// exception code on an Aborted/Error FINISH; 0 otherwise std::map profile_events; /// per-round delta (FINISH); per-phase delta (PHASE) String round_id; /// correlator for every row of one round attempt From bf77615fe0a2ae1cb1bc89ce25cdb82dab9c9f85 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 26 Aug 2026 16:30:23 +0200 Subject: [PATCH 2/8] Classify a relink-confirm refusal as NO_REPLICA_HAS_PART, not NETWORK_ERROR A fetch-by-relink that loses the offer-to-confirm race -- the source's ref moved (a merge, a mutation, an outdated-part drop) between the offer and the confirm -- is a designed, fail-closed outcome: the receiver abandons the relink and the replication queue retries, re-selecting the source and the covering part. It was thrown as `NETWORK_ERROR`, which misdescribes it three ways: - both queue executors (`processQueueEntry`, `ReplicatedMergeTreeQueue`-driven `ReplicatedMergeMutateTaskBase`) treat `NETWORK_ERROR` as an unclassified failure, so every refusal printed an Error-level log line with a full stack trace; issue #2219 records a multi-hour false triage chasing a network fault that was never there (up to 53% of relink proofs refuse under small-part load); - stateless `part_log` hygiene checks tolerate the fetch-transient class under the code upstream fetches use for it, `NO_REPLICA_HAS_PART` (e.g. `02265_column_ttl` whitelists exactly that code), so a refusal landing in `part_log` as `NETWORK_ERROR` fails them -- this is what broke `02265_column_ttl` in the CAS lanes on PR #2159 (13/14 reruns under `prefer_fetch_merged_part_size_threshold=1`); - the label suggests retrying the transport, while the one recovery that is unsound here is a byte re-request to the same source. Both relink retry-later throw sites (taxonomy row 3, the confirm refusal, and row 5b, the unresolved promote) now throw `NO_REPLICA_HAS_PART`. The queue behavior is unchanged -- the exception is stored on the entry, backed off, and re-executed -- but both executors demote it to INFO with no stack trace. Unlike `ABORTED` (the other demoted code), it keeps `need_to_save_exception`, so a refusal storm stays visible in `system.replication_queue`; `ABORTED`'s save-nothing shape is the known pathology where a refusal loop runs invisibly with no backoff accounting. `test_confirm_refuses_when_source_dropped_in_window` now pins the classification: the refusal must not appear at Error level, must appear at Information level, and must reach `part_log` only as `NO_REPLICA_HAS_PART`. No message text changed; no generic queue code changed. Closes: https://github.com/Altinity/ClickHouse/issues/2219 Related: https://github.com/Altinity/ClickHouse/pull/2159 Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- src/Storages/MergeTree/DataPartsExchange.cpp | 25 +++++++++++----- .../test_cas_replicated_relink/test.py | 30 ++++++++++++++++++- 2 files changed, 46 insertions(+), 9 deletions(-) diff --git a/src/Storages/MergeTree/DataPartsExchange.cpp b/src/Storages/MergeTree/DataPartsExchange.cpp index 00f27191db7b..7a8ae2e54c42 100644 --- a/src/Storages/MergeTree/DataPartsExchange.cpp +++ b/src/Storages/MergeTree/DataPartsExchange.cpp @@ -72,7 +72,7 @@ namespace ErrorCodes extern const int CHECKSUM_DOESNT_MATCH; extern const int INSECURE_PATH; extern const int LOGICAL_ERROR; - extern const int NETWORK_ERROR; + extern const int NO_REPLICA_HAS_PART; extern const int S3_ERROR; extern const int ZERO_COPY_REPLICATION_ERROR; } @@ -1314,8 +1314,14 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// 3. THE CONFIRM DID NOT PROVE THE SOURCE: an `unproven` answer, an absent answer cookie, a transport /// failure, a timeout. All one outcome, deliberately (`CasConfirmAnswer`: only `yes` authorizes). /// `+1`: durable, then released by `abort`. Action: THROW a locally generated retry-later -/// `NETWORK_ERROR` naming the source and the part -- never `nullptr`, because a byte re-request goes -/// back to the very source whose state is in doubt. +/// `NO_REPLICA_HAS_PART` naming the source and the part -- never `nullptr`, because a byte +/// re-request goes back to the very source whose state is in doubt. That code, deliberately: +/// both queue executors (`processQueueEntry`, `ReplicatedMergeMutateTaskBase::executeStep`) +/// demote it to INFO with no stack trace -- a refusal is the designed outcome of racing a source +/// whose ref moved on, not a network fault (issue #2219 records a multi-hour false triage chasing +/// that label) -- yet, unlike `ABORTED`, it still records the exception on the queue entry, so a +/// refusal storm stays visible in `system.replication_queue`. It is also the one fetch-transient +/// code the stateless corpus already tolerates in `part_log` checks (e.g. `02265_column_ttl`). /// Lose a part? No -- the queue stores the exception, backs off, and re-executes the entry, which /// recomputes the source and the covering-part discovery. The fetch is postponed, not dropped. /// Double-promote? No -- `abort` appends the exact precommit removal and no committed ref exists. @@ -1339,7 +1345,8 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// `+1`: still owed -- the handle attempts its abandon, which is REJECTED by the state machine if /// the promote in fact landed (a promoted binding is no longer a precommit), so no committed ref is /// ever undone here. -/// Action: THROW the retry-later `NETWORK_ERROR`, as row 3 -- returning `nullptr` is the one thing +/// Action: THROW the retry-later `NO_REPLICA_HAS_PART`, as row 3 -- returning `nullptr` is the +/// one thing /// that must not happen, because a byte fetch would publish the part a SECOND time over a relink /// that may already be committed. /// Lose a part? No -- retry-later, as row 3. Double-promote? No -- nothing is published on this exit. @@ -1544,9 +1551,11 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( { /// Taxonomy row 3. Locally generated on purpose — nothing here is the source's error to report — /// and thrown rather than returned, because the one recovery that is NOT sound after this is a - /// byte re-request to the same source. `NETWORK_ERROR` puts it in the retry-later class, so the - /// queue stores it, backs off, and re-selects on re-execution. - throw Exception(ErrorCodes::NETWORK_ERROR, + /// byte re-request to the same source. `NO_REPLICA_HAS_PART` puts it in the retry-later class + /// (the queue stores it, backs off, and re-selects on re-execution) and both queue executors + /// demote it to INFO without a stack trace -- see the taxonomy, row 3, for why this refusal is + /// an ordinary outcome rather than a fault. + throw Exception(ErrorCodes::NO_REPLICA_HAS_PART, "Source {} did not prove it still holds the manifest it offered for part {} by relink; " "the relink is abandoned and the fetch will be retried later", fetch_uri.getHost(), part_name); @@ -1565,7 +1574,7 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( /// second time over a relink that may already be committed. Thrown in the retry-later class /// instead, exactly as an unproven confirm is (row 3) -- the queue stores it, backs off, and /// re-executes, by which time the ref lane has resolved the ambiguity one way or the other. - throw Exception(ErrorCodes::NETWORK_ERROR, + throw Exception(ErrorCodes::NO_REPLICA_HAS_PART, "Relink of part {} from {} could not be resolved: the promotion may or may not have " "committed, so the bytes must NOT be fetched; the fetch will be retried later", part_name, fetch_uri.getHost()); diff --git a/tests/integration/test_cas_replicated_relink/test.py b/tests/integration/test_cas_replicated_relink/test.py index 83d8d239ba92..234cbbfbe7a4 100644 --- a/tests/integration/test_cas_replicated_relink/test.py +++ b/tests/integration/test_cas_replicated_relink/test.py @@ -769,7 +769,10 @@ def test_confirm_refuses_when_source_dropped_in_window(): """Task 16 step 1 — the race the confirm exists to lose safely. Taxonomy row 3: the source cannot prove it still holds the offered manifest, so the receiver aborts - its durable `+1` and throws a retry-later `NETWORK_ERROR` INSTEAD of falling back to bytes. The two + its durable `+1` and throws a retry-later `NO_REPLICA_HAS_PART` INSTEAD of falling back to bytes. + That code is part of the contract (issue #2219): both queue executors demote it to INFO with no + stack trace, it stays recorded on the queue entry, and it is the one fetch-transient code the + stateless corpus already tolerates in `part_log` checks. The two assertions that matter are (a) the queue recovers by re-selecting — here, onto the covering part — and (b) NO byte re-request ever went to the source whose state was in doubt. (b) is the entire reason row 3 throws where rows 2 and 5 return `nullptr`. @@ -810,6 +813,31 @@ def test_confirm_refuses_when_source_dropped_in_window(): assert not log_lines(node2, relink_finished_pattern(table, part)) assert any_state_part_count(node2, table, part) == 0 + # (c) the refusal's CLASSIFICATION -- the contract pinned after issue #2219. The refusal must reach + # the operator as the tolerated fetch-transient `NO_REPLICA_HAS_PART` (both queue executors + # demote it to INFO, no stack trace; every stateless `part_log` hygiene check that whitelists + # that code -- e.g. `02265_column_ttl` -- stays green), never as an Error-level `NETWORK_ERROR` + # with a stack trace, which reads as a network fault and once cost a multi-hour false triage. + refusal_error_pattern = r".*did not prove it still holds the manifest" + refusal_info_pattern = r".*did not prove it still holds the manifest" + assert not log_lines(node2, refusal_error_pattern), ( + "the relink refusal is a designed outcome and must not be logged at Error level" + ) + assert log_lines(node2, refusal_info_pattern), ( + "the demoted refusal must still be visible at Information level -- silence would be worse than " + "the old noise" + ) + node2.query("SYSTEM FLUSH LOGS part_log") + stray_codes = node2.query( + "SELECT DISTINCT errorCodeToName(error) FROM system.part_log " + "WHERE table = '{}' AND error != 0 AND errorCodeToName(error) != 'NO_REPLICA_HAS_PART'".format( + table + ) + ).split() + assert stray_codes == [], ( + "a relink refusal must reach part_log only as NO_REPLICA_HAS_PART, got: {}".format(stray_codes) + ) + drop_everywhere(table) From b9140d458ec7380d95ad8d321f7b1f927bc358ab Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:02:32 +0200 Subject: [PATCH 3/8] =?UTF-8?q?cas:=20wire-keys=20phase=201=20=E2=80=94=20?= =?UTF-8?q?move=20every=20codec=20onto=20WireKey=20carriers=20(no=20keys?= =?UTF-8?q?=20renamed=20yet)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Behavior-preserving groundwork for an atomic rename of the CAS wire-format JSON keys from abstract letters (`t`, `k`, `s`, ...) to semantic names (`kind`, `outcome`, `state`, ...), landed as its own phase so the rename itself is a single reviewable diff. Adds `WireKey` and per-encoding field write helpers, and `EnumWireTable` — a table pairing each enum value with its wire word, proven complete against the enum by a set-equality coverage check with a failing witness for every member. `kMinBlobHeaderLen` gets one compile-time owner instead of several hand-kept constants. `TokenType`, `ObjectKind`, and `BlobHashAlgo` move onto `EnumWireTable`, and the blob-meta, pool-meta, GC state/heartbeat/ maintenance, server-root, blob-envelope, ref-log/ref-ckpt/ref-snapshot/ ref-catalog, run, fold-seal, and gc-outcomes codecs are all migrated onto the carriers — every one of them still writing its existing wire spelling. `RunMarker` becomes a typed enum, and the format test battery is closed out with a set-equality check over the codec registry. No wire-format bytes change in this phase; the follow-up phase (next commit) performs the actual key cut. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../Formats/CasBlobEnvelopeFormat.cpp | 92 ++++----- .../Formats/CasBlobEnvelopeFormat.h | 3 + .../Formats/CasBlobMetaFormat.cpp | 47 +++-- .../Formats/CasBlobMetaFormat.h | 2 + .../Formats/CasEnvelopeLimits.h | 14 ++ .../Formats/CasFoldSealFormat.cpp | 176 ++++++++++-------- .../Formats/CasFoldSealFormat.h | 4 +- .../ContentAddressed/Formats/CasFormat.cpp | 14 ++ .../ContentAddressed/Formats/CasFormat.h | 1 + .../Formats/CasGcMaintenanceStateFormat.cpp | 10 +- .../Formats/CasGcOutcomesFormat.cpp | 70 +++---- .../Formats/CasGcOutcomesFormat.h | 3 + .../Formats/CasGcStateFormat.cpp | 72 ++++--- .../Formats/CasPartManifestFormat.cpp | 104 +++++------ .../Formats/CasPartManifestFormat.h | 3 + .../Formats/CasPoolMetaFormat.cpp | 51 ++--- .../Formats/CasRecordStreamFormat.cpp | 89 +++++---- .../Formats/CasRecordStreamFormat.h | 41 +++- .../Formats/CasRefCatalogFormat.cpp | 85 +++++---- .../Formats/CasRefCkptFormat.cpp | 38 ++-- .../Formats/CasRefLogFormat.cpp | 176 +++++++++--------- .../Formats/CasRefLogFormat.h | 4 + .../Formats/CasRefSnapshotFormat.cpp | 89 ++++----- .../Formats/CasRefWireVocab.cpp | 42 +++-- .../Formats/CasRefWireVocab.h | 6 +- .../Formats/CasServerRootFormats.cpp | 92 +++++---- .../Formats/CasTextFormat.cpp | 3 +- .../ContentAddressed/Formats/CasTextFormat.h | 71 +++++-- .../ContentAddressed/Formats/CasWireVocab.cpp | 78 ++++---- .../ContentAddressed/Formats/CasWireVocab.h | 135 +++++++++++++- .../ContentAddressed/Gc/CasBlobInDegree.cpp | 68 +++---- .../ContentAddressed/Gc/CasBlobInDegree.h | 10 +- .../ContentAddressed/Gc/CasGc.cpp | 32 ++-- .../Primitives/CasBlobDigest.cpp | 14 +- .../Primitives/CasBlobDigest.h | 11 +- .../Primitives/CasEnumWireTable.h | 73 ++++++++ .../Primitives/CasEnumWireTableAsserts.h | 37 ++++ .../ContentAddressed/Tools/CasFsck.cpp | 8 +- .../ContentAddressed/Tools/CasInspect.cpp | 122 ++---------- src/Disks/tests/cas_format_test_battery.h | 18 ++ src/Disks/tests/cas_test_helpers.h | 14 +- .../tests/gtest_cas_blob_envelope_format.cpp | 2 + src/Disks/tests/gtest_cas_blob_indegree.cpp | 76 ++++++-- .../tests/gtest_cas_blob_meta_format.cpp | 29 ++- src/Disks/tests/gtest_cas_encoding_pins.cpp | 21 ++- src/Disks/tests/gtest_cas_enum_wire_table.cpp | 131 +++++++++++++ src/Disks/tests/gtest_cas_event_log.cpp | 2 +- .../tests/gtest_cas_fold_seal_format.cpp | 22 ++- src/Disks/tests/gtest_cas_format_battery.cpp | 25 +++ src/Disks/tests/gtest_cas_gc_attempt.cpp | 2 +- src/Disks/tests/gtest_cas_gc_fold.cpp | 4 +- src/Disks/tests/gtest_cas_gc_leak.cpp | 2 +- .../gtest_cas_gc_maintenance_state_format.cpp | 12 ++ .../tests/gtest_cas_gc_outcomes_format.cpp | 38 ++++ src/Disks/tests/gtest_cas_gc_rebuild.cpp | 2 +- src/Disks/tests/gtest_cas_gc_resume.cpp | 2 +- src/Disks/tests/gtest_cas_gc_round.cpp | 16 +- src/Disks/tests/gtest_cas_gc_shard_plan.cpp | 2 +- src/Disks/tests/gtest_cas_gc_state_format.cpp | 4 + src/Disks/tests/gtest_cas_inspect.cpp | 98 +++++++++- src/Disks/tests/gtest_cas_json_writer.cpp | 30 ++- src/Disks/tests/gtest_cas_observability.cpp | 50 ++++- .../tests/gtest_cas_orphan_nomination.cpp | 4 +- .../tests/gtest_cas_part_manifest_format.cpp | 2 + src/Disks/tests/gtest_cas_pluggable_hash.cpp | 4 +- .../gtest_cas_rebuild_condemn_nothing.cpp | 4 +- .../tests/gtest_cas_record_stream_format.cpp | 44 ++++- src/Disks/tests/gtest_cas_ref_catalog.cpp | 7 +- src/Disks/tests/gtest_cas_ref_ckpt.cpp | 16 ++ .../tests/gtest_cas_ref_epoch_seal_format.cpp | 2 + src/Disks/tests/gtest_cas_ref_log_format.cpp | 49 +++++ .../tests/gtest_cas_ref_snapshot_format.cpp | 2 + .../tests/gtest_cas_server_root_format.cpp | 8 + src/Disks/tests/gtest_cas_text_format.cpp | 10 + .../tests/gtest_cas_truncate_reclaim.cpp | 2 +- src/Disks/tests/gtest_cas_wire_vocab.cpp | 143 +++++++++++++- 76 files changed, 1997 insertions(+), 892 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h create mode 100644 src/Disks/tests/gtest_cas_enum_wire_table.cpp diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index 65176572e896..a4933ff2ccb8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include @@ -20,30 +21,29 @@ namespace constexpr std::string_view kBlobType = "cas_blob"; -std::string_view opToWord(ProvenanceOp op) +namespace EnvelopeWire { - switch (op) - { - case ProvenanceOp::Other: return "other"; - case ProvenanceOp::Insert: return "insert"; - case ProvenanceOp::Merge: return "merge"; - case ProvenanceOp::Mutation: return "mutation"; - case ProvenanceOp::Attach: return "attach"; - case ProvenanceOp::Repack: return "repack"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown ProvenanceOp {}", static_cast(op)); + constexpr WireKey type{"type"}; + constexpr WireKey version{"v"}; + constexpr WireKey tag{"tag"}; + constexpr WireKey build{"bld"}; + constexpr WireKey time_ms{"ts"}; + constexpr WireKey creator{"by"}; + constexpr WireKey op{"op"}; + constexpr WireKey chver{"ch"}; + constexpr WireKey ref{"ref"}; } -ProvenanceOp opFromWord(std::string_view w) -{ - if (w == "other") return ProvenanceOp::Other; - if (w == "insert") return ProvenanceOp::Insert; - if (w == "merge") return ProvenanceOp::Merge; - if (w == "mutation") return ProvenanceOp::Mutation; - if (w == "attach") return ProvenanceOp::Attach; - if (w == "repack") return ProvenanceOp::Repack; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown op '{}'", w); -} +constexpr EnumWireTable kProvenanceOpWords{{{ + {ProvenanceOp::Other, "other"}, + {ProvenanceOp::Insert, "insert"}, + {ProvenanceOp::Merge, "merge"}, + {ProvenanceOp::Mutation, "mutation"}, + {ProvenanceOp::Attach, "attach"}, + {ProvenanceOp::Repack, "repack"}, +}}}; + +static_assert(casEnumTableCoversEnum()); /// The escaped byte-length of one raw ref char under the frozen envelope alphabet (see writeEnvelopeRefField). size_t escapedLen(char c) @@ -96,6 +96,11 @@ void writeEnvelopeRefField(String & json, size_t budget, std::string_view raw_re } +std::string_view provenanceOpToWireWord(ProvenanceOp op) +{ + return kProvenanceOpWords.toWord(op, "CAS blob envelope"); +} + String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { if (header.kind != ObjectKind::Blob) @@ -108,16 +113,16 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { CasJsonWriter buf(256); bool first = true; - writeKey(buf, "type", first); writeStringValue(buf, kBlobType); - writeKey(buf, "v", first); writeIntText(currentCompatibilityVersion(), buf); - writeKey(buf, "tag", first); writeHex128Value(buf, header.incarnation_tag); - writeKey(buf, "bld", first); writeHex128Value(buf, header.build_id); + writeKey(buf, EnvelopeWire::type, first); writeStringValue(buf, kBlobType); + writeKey(buf, EnvelopeWire::version, first); writeIntText(currentCompatibilityVersion(), buf); + writeKey(buf, EnvelopeWire::tag, first); writeHex128Value(buf, header.incarnation_tag); + writeKey(buf, EnvelopeWire::build, first); writeHex128Value(buf, header.build_id); if (header.provenance) { - writeKey(buf, "ts", first); writeIntText(header.provenance->created_at_ms, buf); - writeKey(buf, "by", first); writeHex128Value(buf, header.provenance->creator_server_id); - writeKey(buf, "op", first); writeStringValue(buf, opToWord(header.provenance->op)); - writeKey(buf, "ch", first); writeIntText(header.provenance->ch_version, buf); + writeKey(buf, EnvelopeWire::time_ms, first); writeIntText(header.provenance->created_at_ms, buf); + writeKey(buf, EnvelopeWire::creator, first); writeHex128Value(buf, header.provenance->creator_server_id); + writeKey(buf, EnvelopeWire::op, first); writeStringValue(buf, provenanceOpToWireWord(header.provenance->op)); + writeKey(buf, EnvelopeWire::chver, first); writeIntText(header.provenance->ch_version, buf); } /// Test-only critical extension: an unknown `!`-key BEFORE `ref`. if (header.emit_unknown_critical_key) @@ -132,15 +137,18 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) /// (byte blob_header_len-1 is reserved for '\n'; the pad zone fills the gap with spaces). if (header.intended_ref) { - static constexpr std::string_view ref_key = ",\"ref\":"; + /// 4 = the `,"` before and `":` after the key text — the `,"ref":` framing minus the key itself. + constexpr size_t ref_key_size = 4 + EnvelopeWire::ref.text.size(); /// +3 = opening quote + closing quote + closing brace. - const size_t fixed = json.size() + ref_key.size() + 3; + const size_t fixed = json.size() + ref_key_size + 3; if (blob_header_len < 1 || fixed > static_cast(blob_header_len) - 1) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS blob envelope: non-ref fields ({} bytes) do not fit blob_header_len {} before the ref", fixed, blob_header_len); const size_t budget = (static_cast(blob_header_len) - 1) - fixed; - json += ref_key; + json += ",\""; + json += EnvelopeWire::ref.text; + json += "\":"; writeEnvelopeRefField(json, budget, *header.intended_ref); } json += '}'; @@ -173,7 +181,7 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje String key; while (r.nextKey(key)) { - if (key == "type") + if (key == EnvelopeWire::type) { const String t = r.readString(); if (t != kBlobType) @@ -181,37 +189,37 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje "CAS blob envelope: object is a '{}', not a '{}'", t, kBlobType); saw_type = true; } - else if (key == "v") + else if (key == EnvelopeWire::version) { h.compatibility_version = r.readU32Number(); checkCompatibility(h.compatibility_version, "blob envelope"); saw_v = true; } - else if (key == "tag") + else if (key == EnvelopeWire::tag) h.incarnation_tag = r.readHex128(); - else if (key == "bld") + else if (key == EnvelopeWire::build) h.build_id = r.readHex128(); - else if (key == "ts") + else if (key == EnvelopeWire::time_ms) { prov.created_at_ms = r.readU64Number(); have_prov = true; } - else if (key == "by") + else if (key == EnvelopeWire::creator) { prov.creator_server_id = r.readHex128(); have_prov = true; } - else if (key == "op") + else if (key == EnvelopeWire::op) { - prov.op = opFromWord(r.readString()); + prov.op = kProvenanceOpWords.fromWord(r.readString(), "CAS blob envelope"); have_prov = true; } - else if (key == "ch") + else if (key == EnvelopeWire::chver) { prov.ch_version = static_cast(r.readU64Number()); have_prov = true; } - else if (key == "ref") + else if (key == EnvelopeWire::ref) h.intended_ref = r.readString(); else r.skipUnknown(key); /// `!`-key -> UNKNOWN_FORMAT_VERSION; unknown plain key -> skipped (tolerant) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 19250fe69ddd..68c473a70c3d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -32,6 +32,9 @@ enum class ProvenanceOp : uint8_t Repack = 5, }; +/// Returns the persisted wire word for a validated provenance operation. +std::string_view provenanceOpToWireWord(ProvenanceOp op); + /// Optional diagnostic metadata recorded with an envelope. The fields identify when and where the /// incarnation was created, the ClickHouse build that wrote it, and the operation that produced it; /// none of them participates in object identity or a protocol decision. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp index b62fd3b82424..ab98b8858693 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include @@ -17,26 +18,25 @@ namespace DB::Cas namespace { -std::string_view metaStateToWord(MetaState s) +namespace BlobMetaWire { - switch (s) - { - case MetaState::Clean: return "clean"; - case MetaState::Condemned: return "condemned"; - } - // The enum is persisted as a closed vocabulary. Do not silently invent a spelling for a value - // added without a corresponding format decision: that would make the writer emit data older - // readers cannot classify. - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: unknown MetaState {}", static_cast(s)); + constexpr WireKey state{"st"}; + constexpr WireKey condemn_round{"cr"}; + constexpr WireKey size{"sz"}; } -MetaState metaStateFromWord(std::string_view w) -{ - if (w == "clean") return MetaState::Clean; - if (w == "condemned") return MetaState::Condemned; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: unknown state '{}'", w); +constexpr EnumWireTable kMetaStateWords{{{ + {MetaState::Clean, "clean"}, + {MetaState::Condemned, "condemned"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + } +std::string_view metaStateToWireWord(MetaState state) +{ + return kMetaStateWords.toWord(state, "CAS blob meta"); } String encodeBlobMeta(const BlobMeta & meta) @@ -46,12 +46,9 @@ String encodeBlobMeta(const BlobMeta & meta) // `version` is represented by the header line. The JSON body contains only fields that describe // the current marker and its accounting data. bool first = true; - writeKey(out, "st", first); - writeStringValue(out, metaStateToWord(meta.state)); - writeKey(out, "cr", first); - writeU64StringValue(out, meta.condemn_round); - writeKey(out, "sz", first); - writeU64StringValue(out, meta.size); + writeWordField(out, BlobMetaWire::state, metaStateToWireWord(meta.state), first); + writeU64StringField(out, BlobMetaWire::condemn_round, meta.condemn_round, first); + writeU64StringField(out, BlobMetaWire::size, meta.size, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -72,14 +69,14 @@ BlobMeta decodeBlobMeta(std::string_view bytes) String key; while (r.nextKey(key)) { - if (key == "st") + if (key == BlobMetaWire::state) { - m.state = metaStateFromWord(r.readString()); + m.state = kMetaStateWords.fromWord(r.readString(), "CAS blob meta"); saw_state = true; } - else if (key == "cr") + else if (key == BlobMetaWire::condemn_round) m.condemn_round = r.readU64String(); - else if (key == "sz") + else if (key == BlobMetaWire::size) m.size = r.readU64String(); else r.skipUnknown(key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h index 6694fbafeb33..5290e7c831db 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h @@ -19,6 +19,8 @@ enum class MetaState : uint8_t /// so a writer may republish it by replacing the body and updating this marker. }; +std::string_view metaStateToWireWord(MetaState state); + /// The durable per-hash meta record. Its text representation consists of a format header followed by /// one JSON object with the state word, the GC condemnation round, and the raw body size. `size` is /// retained for introspection, fsck, and GC accounting; reads of the blob never consult the meta. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h new file mode 100644 index 000000000000..df8cee44bc47 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h @@ -0,0 +1,14 @@ +#pragma once + +#include + +namespace DB::Cas +{ + +/// The pool-wide floor for `blob_header_len`. One compile-time owner, read by BOTH +/// `validatePoolBlobHeaderLen` (pool creation / decode) and the blob-envelope codec, so the +/// mandatory-descriptor worst-case proof and the enforced floor can never guard different numbers. +/// The derivation of the floor lives in `CasPoolMetaFormat.cpp` next to the worst-case table. +inline constexpr uint64_t kMinBlobHeaderLen = 240; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index b4bba2bff08f..2324cc2c037a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -20,32 +21,52 @@ namespace ErrorCodes namespace DB::Cas { -std::string_view holdReasonToWord(HoldReason r) +namespace { - switch (r) - { - case HoldReason::GapBelowWitness: return "gap_below_witness"; - case HoldReason::UnconsumedSealCrossing: return "unconsumed_seal_crossing"; - case HoldReason::WitnessDisappeared: return "witness_disappeared"; - case HoldReason::BodyUndecodable: return "body_undecodable"; - case HoldReason::ManifestBodyMissing: return "manifest_body_missing"; - case HoldReason::CheckpointUndecodable: return "checkpoint_undecodable"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown hold reason {}", static_cast(r)); -} -namespace +namespace FoldSealWire { + constexpr WireKey generation{"g"}; + constexpr WireKey parent_generation{"pg"}; + constexpr WireKey kind{"k"}; + constexpr WireKey run_key{"key"}; + constexpr WireKey checksum{"ck"}; + constexpr WireKey shard{"shard"}; + constexpr WireKey key_generation{"gen"}; + constexpr WireKey life{"life"}; + constexpr WireKey classification{"cls"}; + constexpr WireKey fold_epoch{"lfe"}; + constexpr WireKey fold_seq{"lfs"}; + constexpr WireKey hold_reason{"hr"}; + constexpr WireKey hold_epoch{"hpe"}; + constexpr WireKey hold_seq{"hps"}; + constexpr WireKey retries{"hrc"}; + constexpr WireKey retry_round{"hnr"}; + constexpr WireKey remove_epoch{"rte"}; + constexpr WireKey remove_seq{"rts"}; + constexpr WireKey condemned_total{"ct"}; + constexpr WireKey pending_total{"pt"}; + constexpr WireKey oldest_round{"ocr"}; +} + +constexpr std::string_view kRefLifeTag = "rfl"; +constexpr std::string_view kBlobRunTag = "btr"; +constexpr std::string_view kCondemnedTag = "cnd"; + +constexpr EnumWireTable kHoldReasonWords{{{ + {HoldReason::GapBelowWitness, "gap_below_witness"}, + {HoldReason::UnconsumedSealCrossing, "unconsumed_seal_crossing"}, + {HoldReason::WitnessDisappeared, "witness_disappeared"}, + {HoldReason::BodyUndecodable, "body_undecodable"}, + {HoldReason::ManifestBodyMissing, "manifest_body_missing"}, + {HoldReason::CheckpointUndecodable, "checkpoint_undecodable"}, +}}}; + +static_assert(casEnumTableCoversEnum()); HoldReason holdReasonFromWord(std::string_view w) { - if (w == "gap_below_witness") return HoldReason::GapBelowWitness; - if (w == "unconsumed_seal_crossing") return HoldReason::UnconsumedSealCrossing; - if (w == "witness_disappeared") return HoldReason::WitnessDisappeared; - if (w == "body_undecodable") return HoldReason::BodyUndecodable; - if (w == "manifest_body_missing") return HoldReason::ManifestBodyMissing; - if (w == "checkpoint_undecodable") return HoldReason::CheckpointUndecodable; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown hold reason '{}'", w); + return kHoldReasonWords.fromWord(w, "CAS fold seal hold reason"); } /// The classification set is CLOSED. Every consumer of a coverage row branches on exact values — the @@ -88,11 +109,11 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { bool first = true; - writeKey(out, "k", first); writeStringValue(out, kind); - writeKey(out, "key", first); writeStringValue(out, r.key); - writeKey(out, "ck", first); writeHex128Value(out, r.checksum); - writeKey(out, "shard", first); writeIntText(r.shard, out); - writeKey(out, "gen", first); writeU64StringValue(out, r.generation); + writeStringField(out, FoldSealWire::kind, kind, first); + writeStringField(out, FoldSealWire::run_key, r.key, first); + writeHex128Field(out, FoldSealWire::checksum, r.checksum, first); + writeNumberField(out, FoldSealWire::shard, r.shard, first); + writeU64StringField(out, FoldSealWire::key_generation, r.key_generation, first); closeObject(out, first); } @@ -106,7 +127,7 @@ void validateFoldSealStructure( std::vector run_seen(gc_shards, false); for (const RunRef & run : seal.blob_target_runs) { - if (run.key.empty() || run.generation == 0) + if (run.key.empty() || run.key_generation == 0) throw Exception(error_code, "CAS fold seal {}: blob-target run requires a nonempty key and nonzero physical generation", source); @@ -121,10 +142,10 @@ void validateFoldSealStructure( run_seen[run.shard] = true; const auto parsed = layout.parseBlobTargetRunKey(run.key); - if (!parsed || parsed->generation != run.generation || parsed->shard != run.shard || parsed->seq != 0) + if (!parsed || parsed->generation != run.key_generation || parsed->shard != run.shard || parsed->seq != 0) throw Exception(error_code, "CAS fold seal {}: blob-target run key '{}' is not canonical for generation {}, shard {}, sequence 0", - source, run.key, run.generation, run.shard); + source, run.key, run.key_generation, run.shard); } if (seal.condemned_summary.size() != gc_shards) @@ -154,6 +175,11 @@ void validateFoldSealStructure( } +std::string_view holdReasonToWord(HoldReason r) +{ + return kHoldReasonWords.toWord(r, "CAS fold seal hold reason"); +} + FoldSealCaps foldSealCaps() { const FormatTraits & t = traitsFor(FormatId::FoldSeal); @@ -205,8 +231,8 @@ String encodeFoldSeal(const CasFoldSeal & seal) /// meta line { bool first = true; - writeKey(out, "g", first); writeU64StringValue(out, seal.generation); - writeKey(out, "pg", first); writeU64StringValue(out, seal.parent_generation); + writeU64StringField(out, FoldSealWire::generation, seal.generation, first); + writeU64StringField(out, FoldSealWire::parent_generation, seal.parent_generation, first); closeObject(out, first); closeLine("meta"); } @@ -263,25 +289,23 @@ String encodeFoldSeal(const CasFoldSeal & seal) life_state.cleanup_evidence->remove_txn_id.ref_sequence); bool first = true; - writeKey(out, "k", first); writeStringValue(out, "rfl"); - writeKey(out, "life", first); writeHex128Value(out, life_id); - writeKey(out, "cls", first); writeIntText(static_cast(cov.classification), out); - writeKey(out, "lfe", first); writeU64StringValue(out, cov.last_folded_ref_id.writer_epoch); - writeKey(out, "lfs", first); writeU64StringValue(out, cov.last_folded_ref_id.ref_sequence); + writeStringField(out, FoldSealWire::kind, kRefLifeTag, first); + writeHex128Field(out, FoldSealWire::life, life_id, first); + writeNumberField(out, FoldSealWire::classification, static_cast(cov.classification), first); + writeU64StringField(out, FoldSealWire::fold_epoch, cov.last_folded_ref_id.writer_epoch, first); + writeU64StringField(out, FoldSealWire::fold_seq, cov.last_folded_ref_id.ref_sequence, first); if (cov.hold) { - writeKey(out, "hr", first); writeStringValue(out, holdReasonToWord(cov.hold->reason)); - writeKey(out, "hpe", first); writeU64StringValue(out, cov.hold->offending_position.writer_epoch); - writeKey(out, "hps", first); writeU64StringValue(out, cov.hold->offending_position.ref_sequence); - writeKey(out, "hrc", first); writeIntText(cov.hold->retry_count, out); - writeKey(out, "hnr", first); writeU64StringValue(out, cov.hold->next_retry_round); + writeStringField(out, FoldSealWire::hold_reason, holdReasonToWord(cov.hold->reason), first); + writeU64StringField(out, FoldSealWire::hold_epoch, cov.hold->offending_position.writer_epoch, first); + writeU64StringField(out, FoldSealWire::hold_seq, cov.hold->offending_position.ref_sequence, first); + writeNumberField(out, FoldSealWire::retries, cov.hold->retry_count, first); + writeU64StringField(out, FoldSealWire::retry_round, cov.hold->next_retry_round, first); } if (life_state.cleanup_evidence) { - writeKey(out, "rte", first); - writeU64StringValue(out, life_state.cleanup_evidence->remove_txn_id.writer_epoch); - writeKey(out, "rts", first); - writeU64StringValue(out, life_state.cleanup_evidence->remove_txn_id.ref_sequence); + writeU64StringField(out, FoldSealWire::remove_epoch, life_state.cleanup_evidence->remove_txn_id.writer_epoch, first); + writeU64StringField(out, FoldSealWire::remove_seq, life_state.cleanup_evidence->remove_txn_id.ref_sequence, first); } closeObject(out, first); closeLine("rfl"); @@ -293,7 +317,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) std::sort(runs.begin(), runs.end(), [](const RunRef & a, const RunRef & b) { return a.key < b.key; }); for (const RunRef & r : runs) { - writeRun(out, "btr", r); + writeRun(out, kBlobRunTag, r); closeLine("btr"); } } @@ -303,11 +327,11 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const auto & [shard, s] : seal.condemned_summary) { bool first = true; - writeKey(out, "k", first); writeStringValue(out, "cnd"); - writeKey(out, "shard", first); writeIntText(shard, out); - writeKey(out, "ct", first); writeIntText(s.condemned_total, out); - writeKey(out, "pt", first); writeIntText(s.pending_total, out); - writeKey(out, "ocr", first); writeU64StringValue(out, s.oldest_nonpending_condemn_round); + writeStringField(out, FoldSealWire::kind, kCondemnedTag, first); + writeNumberField(out, FoldSealWire::shard, shard, first); + writeNumberField(out, FoldSealWire::condemned_total, s.condemned_total, first); + writeNumberField(out, FoldSealWire::pending_total, s.pending_total, first); + writeU64StringField(out, FoldSealWire::oldest_round, s.oldest_nonpending_condemn_round, first); closeObject(out, first); closeLine("cnd"); ++n; @@ -338,8 +362,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect String key; while (r.nextKey(key)) { - if (key == "g") seal.generation = r.readU64String(); - else if (key == "pg") seal.parent_generation = r.readU64String(); + if (key == FoldSealWire::generation) seal.generation = r.readU64String(); + else if (key == FoldSealWire::parent_generation) seal.parent_generation = r.readU64String(); else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } } @@ -370,11 +394,11 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect seal.generation, *expected_generation); return seal; } - if (key != "k") + if (key != FoldSealWire::kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"k\""); const String kind = r.readString(); - if (kind == "rfl") + if (kind == kRefLifeTag) { std::optional life_id; RefCoverage cov; @@ -395,17 +419,17 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional remove_txn_sequence; while (r.nextKey(key)) { - if (key == "life") life_id = r.readHex128(); - else if (key == "cls") classification = r.readU64Number(); - else if (key == "lfe") cov.last_folded_ref_id.writer_epoch = r.readU64String(); - else if (key == "lfs") cov.last_folded_ref_id.ref_sequence = r.readU64String(); - else if (key == "hr") hold_reason = holdReasonFromWord(r.readString()); - else if (key == "hpe") hold_epoch = r.readU64String(); - else if (key == "hps") hold_sequence = r.readU64String(); - else if (key == "hrc") hold_retry_count = r.readU32Number(); - else if (key == "hnr") hold_next_retry_round = r.readU64String(); - else if (key == "rte") remove_txn_epoch = r.readU64String(); - else if (key == "rts") remove_txn_sequence = r.readU64String(); + if (key == FoldSealWire::life) life_id = r.readHex128(); + else if (key == FoldSealWire::classification) classification = r.readU64Number(); + else if (key == FoldSealWire::fold_epoch) cov.last_folded_ref_id.writer_epoch = r.readU64String(); + else if (key == FoldSealWire::fold_seq) cov.last_folded_ref_id.ref_sequence = r.readU64String(); + else if (key == FoldSealWire::hold_reason) hold_reason = holdReasonFromWord(r.readString()); + else if (key == FoldSealWire::hold_epoch) hold_epoch = r.readU64String(); + else if (key == FoldSealWire::hold_seq) hold_sequence = r.readU64String(); + else if (key == FoldSealWire::retries) hold_retry_count = r.readU32Number(); + else if (key == FoldSealWire::retry_round) hold_next_retry_round = r.readU64String(); + else if (key == FoldSealWire::remove_epoch) remove_txn_epoch = r.readU64String(); + else if (key == FoldSealWire::remove_seq) remove_txn_sequence = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown rfl key '{}'", key); } @@ -478,7 +502,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect "CAS fold seal: a second ref-life record for '{}' -- a life id appears at most once", life_hex); } - else if (kind == "btr") + else if (kind == kBlobRunTag) { std::optional run_key; std::optional checksum; @@ -486,19 +510,19 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional generation; while (r.nextKey(key)) { - if (key == "key") run_key = r.readString(); - else if (key == "ck") checksum = r.readHex128(); - else if (key == "shard") shard = r.readU64Number(); - else if (key == "gen") generation = r.readU64String(); + if (key == FoldSealWire::run_key) run_key = r.readString(); + else if (key == FoldSealWire::checksum) checksum = r.readHex128(); + else if (key == FoldSealWire::shard) shard = r.readU64Number(); + else if (key == FoldSealWire::key_generation) generation = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown run key '{}'", key); } if (!run_key || !checksum || !shard || !generation) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: btr requires key, ck, shard, and gen"); seal.blob_target_runs.push_back(RunRef{ - .key = std::move(*run_key), .checksum = *checksum, .shard = *shard, .generation = *generation}); + .key = std::move(*run_key), .checksum = *checksum, .shard = *shard, .key_generation = *generation}); } - else if (kind == "cnd") + else if (kind == kCondemnedTag) { std::optional shard; std::optional condemned_total; @@ -506,10 +530,10 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional oldest_nonpending_condemn_round; while (r.nextKey(key)) { - if (key == "shard") shard = r.readU64Number(); - else if (key == "ct") condemned_total = r.readU64Number(); - else if (key == "pt") pending_total = r.readU64Number(); - else if (key == "ocr") oldest_nonpending_condemn_round = r.readU64String(); + if (key == FoldSealWire::shard) shard = r.readU64Number(); + else if (key == FoldSealWire::condemned_total) condemned_total = r.readU64Number(); + else if (key == FoldSealWire::pending_total) pending_total = r.readU64Number(); + else if (key == FoldSealWire::oldest_round) oldest_nonpending_condemn_round = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown cnd key '{}'", key); } if (!shard || !condemned_total || !pending_total || !oldest_nonpending_condemn_round) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h index f6d00a6e3b50..ea4200e29eea 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h @@ -27,7 +27,7 @@ struct RunRef String key; UInt128 checksum{}; uint64_t shard = 0; /// gc-shard this run belongs to (REQUIRED for blob_target_runs) - uint64_t generation = 0; /// generation whose key namespace physically holds the object (for retention) + uint64_t key_generation = 0; /// generation whose key namespace physically holds the object (for retention) bool operator==(const RunRef &) const = default; }; @@ -147,7 +147,7 @@ struct RefLifeFoldState /// must not be interpreted as zero. struct CondemnedSummary { - uint64_t condemned_total = 0; /// count of `kCondemned` rows in this shard's sealed run + uint64_t condemned_total = 0; /// count of `RunMarker::Condemned` rows in this shard's sealed run uint64_t pending_total = 0; /// how many of those are `delete_pending` (a graduation is due) uint64_t oldest_nonpending_condemn_round = UINT64_MAX; /// min condemn_round over non-pending; UINT64_MAX = none bool operator==(const CondemnedSummary &) const = default; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp index 362306691473..dc9665aff837 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp @@ -1,6 +1,8 @@ #include #include +#include + namespace DB { namespace ErrorCodes @@ -197,6 +199,18 @@ const FormatTraits * traitsForType(std::string_view type) return nullptr; } +std::span allRegisteredFormatIds() +{ + static const auto ids = [] + { + std::array out{}; + for (size_t i = 0; i < std::size(TRAITS); ++i) + out[i] = TRAITS[i].id; + return out; + }(); + return ids; +} + std::string_view storedSuffix(FormatId id) { return traitsFor(id).compression == CompressionPolicy::Always ? ".zst" : ""; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h index 1acc2b3925de..1c98e4f283f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h @@ -202,6 +202,7 @@ const FormatTraits & traitsFor(FormatId id); /// Looks up a header-line `type` string. Returns nullptr for an unregistered type; it does not throw /// because callers use this result to classify the input before decoding it. const FormatTraits * traitsForType(std::string_view type); +std::span allRegisteredFormatIds(); /// Returns the storage-key suffix for `id`: `.zst` for `Always`, and an empty suffix otherwise. /// Key builders use this policy directly so a point lookup never has to inspect the object body or /// try multiple keys. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp index c5dda3286ad6..bc0720bf6f27 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp @@ -13,6 +13,11 @@ namespace DB::ErrorCodes namespace DB::Cas { +namespace GcMaintenanceWire +{ + constexpr WireKey janitor_cursor{"cur"}; +} + String encodeGcMaintenanceState(const GcMaintenanceState & state) { if (state.janitor_cursor.size() > kMaxGcMaintenanceCursorBytes) @@ -22,8 +27,7 @@ String encodeGcMaintenanceState(const GcMaintenanceState & state) CasJsonWriter out; writeHeaderLine(out, FormatId::GcMaintenanceState); bool first = true; - writeKey(out, "cur", first); - writeStringValue(out, state.janitor_cursor); + writeStringField(out, GcMaintenanceWire::janitor_cursor, state.janitor_cursor, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -46,7 +50,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data) String key; while (reader.nextKey(key)) { - if (key == "cur") + if (key == GcMaintenanceWire::janitor_cursor) { result.janitor_cursor = reader.readString(); has_cursor = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index e69ea98a799e..9ee65ebafa78 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -18,29 +19,33 @@ namespace DB::Cas namespace { -std::string_view outcomeKindToWord(OutcomeKind o) +namespace GcOutcomesWire { - switch (o) - { - case OutcomeKind::Deleted: return "deleted"; - case OutcomeKind::Absent: return "absent"; - case OutcomeKind::Replaced: return "replaced"; - case OutcomeKind::Spared: return "spared"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: unknown OutcomeKind {}", static_cast(o)); + constexpr WireKey kind{"k"}; + constexpr WireKey outcome{"oc"}; } +constexpr EnumWireTable kOutcomeKindWords{{{ + {OutcomeKind::Deleted, "deleted"}, + {OutcomeKind::Absent, "absent"}, + {OutcomeKind::Replaced, "replaced"}, + {OutcomeKind::Spared, "spared"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + OutcomeKind outcomeKindFromWord(std::string_view w) { - if (w == "deleted") return OutcomeKind::Deleted; - if (w == "absent") return OutcomeKind::Absent; - if (w == "replaced") return OutcomeKind::Replaced; - if (w == "spared") return OutcomeKind::Spared; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: unknown outcome '{}'", w); + return kOutcomeKindWords.fromWord(w, "CAS outcome log outcome kind"); } } +std::string_view outcomeKindToWireWord(OutcomeKind outcome) +{ + return kOutcomeKindWords.toWord(outcome, "CAS outcome log outcome kind"); +} + String encodeOutcomeLog(const OutcomeLog & log) { CasJsonWriter out(256); @@ -48,12 +53,10 @@ String encodeOutcomeLog(const OutcomeLog & log) for (const OutcomeEntry & e : log.entries) { bool first = true; - writeKey(out, "k", first); - writeStringValue(out, objectKindToWord(e.kind)); + writeStringField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); writeBlobRefFields(out, first, e.ref); /// ha + h writeTokenFields(out, first, e.token); /// tt + tv - writeKey(out, "oc", first); - writeStringValue(out, outcomeKindToWord(e.outcome)); + writeStringField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); closeObject(out, first); writeChar('\n', out); } @@ -92,34 +95,21 @@ OutcomeLog decodeOutcomeLog(std::string_view data) } OutcomeEntry e; - String ha; - String hhex; - String tv; - bool have_ha = false; - bool have_h = false; - bool have_tt = false; - TokenType tt{}; + BlobRefFields blob_ref_fields; + TokenFields token_fields; do { - if (key == "k") e.kind = objectKindFromWord(r.readString(), "outcome log"); - else if (key == "ha") { ha = r.readString(); have_ha = true; } - else if (key == "h") { hhex = r.readString(); have_h = true; } - else if (key == "tt") { tt = tokenTypeFromWord(r.readString(), "outcome log"); have_tt = true; } - else if (key == "tv") tv = r.readString(); - else if (key == "oc") e.outcome = outcomeKindFromWord(r.readString()); + if (key == GcOutcomesWire::kind) e.kind = objectKindFromWord(r.readString(), "outcome log"); + else if (matchBlobRefFields(key, r, blob_ref_fields)) {} + else if (matchTokenFields(key, r, token_fields)) {} + else if (key == GcOutcomesWire::outcome) e.outcome = outcomeKindFromWord(r.readString()); else r.skipUnknown(key); } while (r.nextKey(key)); - if (!have_ha || !have_h || !have_tt) + if (!blob_ref_fields.algo_word || !blob_ref_fields.digest_hex || !token_fields.type_word) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: record missing ha/h/tt"); - const BlobHashAlgo algo = blobHashAlgoFromWord(ha, "outcome log"); - /// Validate the digest width before `fromHex`: a width mismatch must surface as the - /// CORRUPTED_DATA required for malformed serialized input, not fromHex's BAD_ARGUMENTS. - if (hhex.size() != blobHashLenFor(algo) * 2) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS outcome log: digest width {} does not match algo '{}'", hhex.size(), ha); - e.ref = BlobRef{algo, codecFor(algo).fromHex(hhex)}; - e.token = Token{tv, tt}; + e.ref = blob_ref_fields.build("outcome log"); + e.token = Token{token_fields.value.value_or(""), tokenTypeFromWord(*token_fields.type_word, "outcome log")}; if (!line_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: junk after record"); log.entries.push_back(std::move(e)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index 09a850ee66ff..474c3c136a0b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -27,6 +27,9 @@ enum class OutcomeKind : uint8_t Spared = 4, /// The merge found a positive in-degree, so the candidate was kept alive. }; +/// Canonical wire word for one `OutcomeKind`. +std::string_view outcomeKindToWireWord(OutcomeKind outcome); + /// One observation about a blob incarnation considered by GC. `token` identifies the exact /// incarnation that GC examined, while `ref` identifies the content address; retaining both lets /// replay and inspection distinguish an absent object from a replacement that won a race with GC. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp index 7012c6787f70..b70f016cbae0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp @@ -16,6 +16,24 @@ namespace ErrorCodes namespace DB::Cas { +namespace GcStateWire +{ + constexpr WireKey round{"rnd"}; + constexpr WireKey gc_shards{"gcs"}; + constexpr WireKey snap_generation{"sg"}; + constexpr WireKey snap_pruned_through{"spt"}; + constexpr WireKey snap_attempt{"sa"}; + constexpr WireKey manifest_sweep_cursor{"msc"}; + constexpr WireKey lease_owner{"lo"}; + constexpr WireKey lease_seq{"ls"}; +} + +namespace GcHeartbeatWire +{ + constexpr WireKey owner{"by"}; + constexpr WireKey hb_seq{"seq"}; +} + String encodeGcState(const GcState & state) { if (state.gc_shards < 1) @@ -23,14 +41,14 @@ String encodeGcState(const GcState & state) CasJsonWriter out(256); writeHeaderLine(out, FormatId::GcState); bool first = true; - writeKey(out, "rnd", first); writeU64StringValue(out, state.round); - writeKey(out, "gcs", first); writeIntText(state.gc_shards, out); - writeKey(out, "sg", first); writeU64StringValue(out, state.snap_generation); - writeKey(out, "spt", first); writeU64StringValue(out, state.snap_pruned_through); - writeKey(out, "sa", first); writeU64StringValue(out, state.snap_attempt); - writeKey(out, "msc", first); writeStringValue(out, state.manifest_sweep_cursor); - writeKey(out, "lo", first); writeHex128Value(out, state.lease.owner); - writeKey(out, "ls", first); writeU64StringValue(out, state.lease.seq); + writeU64StringField(out, GcStateWire::round, state.round, first); + writeNumberField(out, GcStateWire::gc_shards, state.gc_shards, first); + writeU64StringField(out, GcStateWire::snap_generation, state.snap_generation, first); + writeU64StringField(out, GcStateWire::snap_pruned_through, state.snap_pruned_through, first); + writeU64StringField(out, GcStateWire::snap_attempt, state.snap_attempt, first); + writeStringField(out, GcStateWire::manifest_sweep_cursor, state.manifest_sweep_cursor, first); + writeHex128Field(out, GcStateWire::lease_owner, state.lease.owner, first); + writeU64StringField(out, GcStateWire::lease_seq, state.lease.seq, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -49,15 +67,27 @@ GcState decodeGcState(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "rnd") state.round = r.readU64String(); - else if (key == "gcs") { state.gc_shards = r.readU64Number(); saw_gcs = true; } - else if (key == "sg") state.snap_generation = r.readU64String(); - else if (key == "spt") state.snap_pruned_through = r.readU64String(); - else if (key == "sa") state.snap_attempt = r.readU64String(); - else if (key == "msc") state.manifest_sweep_cursor = r.readString(); - else if (key == "lo") state.lease.owner = r.readHex128(); - else if (key == "ls") state.lease.seq = r.readU64String(); - else r.skipUnknown(key); + if (key == GcStateWire::round) + state.round = r.readU64String(); + else if (key == GcStateWire::gc_shards) + { + state.gc_shards = r.readU64Number(); + saw_gcs = true; + } + else if (key == GcStateWire::snap_generation) + state.snap_generation = r.readU64String(); + else if (key == GcStateWire::snap_pruned_through) + state.snap_pruned_through = r.readU64String(); + else if (key == GcStateWire::snap_attempt) + state.snap_attempt = r.readU64String(); + else if (key == GcStateWire::manifest_sweep_cursor) + state.manifest_sweep_cursor = r.readString(); + else if (key == GcStateWire::lease_owner) + state.lease.owner = r.readHex128(); + else if (key == GcStateWire::lease_seq) + state.lease.seq = r.readU64String(); + else + r.skipUnknown(key); } /// Fail closed on an absent gcs: the writer always emits it, so a missing key means a corrupt object. /// Do NOT silently keep the struct default (1) — that would hide corruption (no-fallback principle). @@ -75,8 +105,8 @@ String encodeGcHeartbeat(const GcHeartbeat & hb) CasJsonWriter out(256); writeHeaderLine(out, FormatId::GcHeartbeat); bool first = true; - writeKey(out, "by", first); writeHex128Value(out, hb.owner); - writeKey(out, "seq", first); writeU64StringValue(out, hb.hb_seq); + writeHex128Field(out, GcHeartbeatWire::owner, hb.owner, first); + writeU64StringField(out, GcHeartbeatWire::hb_seq, hb.hb_seq, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -96,12 +126,12 @@ GcHeartbeat decodeGcHeartbeat(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "by") + if (key == GcHeartbeatWire::owner) { hb.owner = r.readHex128(); saw_by = true; } - else if (key == "seq") + else if (key == GcHeartbeatWire::hb_seq) { hb.hb_seq = r.readU64String(); saw_seq = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index 5e7f9ff5ffbf..3e406ad01203 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -22,41 +23,37 @@ namespace DB::Cas namespace { -std::string_view placementToWord(EntryPlacement p) +namespace PartManifestWire { - switch (p) - { - case EntryPlacement::Inline: return "inline"; - case EntryPlacement::Blob: return "blob"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: unknown placement {}", static_cast(p)); + constexpr WireKey ns{"ns"}; + constexpr WireKey payload_digest{"pd"}; + constexpr WireKey path{"p"}; + constexpr WireKey place{"pm"}; + constexpr WireKey size{"sz"}; + constexpr WireKey inline_size{"il"}; } -EntryPlacement placementFromWord(std::string_view w) -{ - if (w == "inline") return EntryPlacement::Inline; - if (w == "blob") return EntryPlacement::Blob; - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: unknown placement '{}'", w); -} +constexpr EnumWireTable kEntryPlacementWords{{{ + {EntryPlacement::Inline, "inline"}, + {EntryPlacement::Blob, "blob"}, +}}}; + +static_assert(casEnumTableCoversEnum()); /// One entry-record line: {"p","pm", then either the Blob's "ha"/"h"/"sz" or the Inline's "il"}. void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) { bool first = true; - writeKey(out, "p", first); - writeStringValue(out, e.path); - writeKey(out, "pm", first); - writeStringValue(out, placementToWord(e.placement)); + writeStringField(out, PartManifestWire::path, e.path, first); + writeWordField(out, PartManifestWire::place, entryPlacementToWireWord(e.placement), first); if (e.placement == EntryPlacement::Blob) { writeBlobRefFields(out, first, e.ref); /// ha + h - writeKey(out, "sz", first); - writeIntText(e.blob_size, out); + writeNumberField(out, PartManifestWire::size, e.blob_size, first); } else { - writeKey(out, "il", first); - writeIntText(e.inline_bytes.size(), out); + writeNumberField(out, PartManifestWire::inline_size, e.inline_bytes.size(), first); } closeObject(out, first); writeChar('\n', out); @@ -80,6 +77,11 @@ String bannerFor(std::string_view path, uint64_t n) } +std::string_view entryPlacementToWireWord(EntryPlacement placement) +{ + return kEntryPlacementWords.toWord(placement, "PartManifest: EntryPlacement"); +} + String encodePartManifest(const PartManifest & m) { /// Canonical path order plus duplicate-path rejection makes the encoded record sequence @@ -101,11 +103,9 @@ String encodePartManifest(const PartManifest & m) /// namespace + payload digest. { bool first = true; - writeManifestRefFields(out, first, "", m.ref); - writeKey(out, "ns", first); - writeStringValue(out, m.root_namespace_id.string()); - writeKey(out, "pd", first); - writeHex128Value(out, m.payload_digest); + writeManifestRefFields(out, first, kBareManifestRefKeys, m.ref); + writeStringField(out, PartManifestWire::ns, m.root_namespace_id.string(), first); + writeHex128Field(out, PartManifestWire::payload_digest, m.payload_digest, first); closeObject(out, first); writeChar('\n', out); } @@ -144,28 +144,22 @@ PartManifest decodePartManifest(std::string_view data) const String meta = readLine(in, line_cap, "cas_part_manifest"); ReadBufferFromMemory mm(meta.data(), meta.size()); JsonObjectReader r(mm, KeyStrictness::Tolerant, "cas_part_manifest"); - std::optional me; - std::optional mb; - std::optional mo; + ManifestRefFields fields; std::optional ns; std::optional pd; String key; while (r.nextKey(key)) { - if (key == "me") me = r.readU64String(); - else if (key == "mb") mb = r.readU64String(); - else if (key == "mo") mo = r.readU64Number(); - else if (key == "ns") ns = r.readString(); - else if (key == "pd") pd = r.readHex128(); + if (matchManifestRefFields(key, r, kBareManifestRefKeys, fields)) {} + else if (key == PartManifestWire::ns) ns = r.readString(); + else if (key == PartManifestWire::payload_digest) pd = r.readHex128(); else r.skipUnknown(key); } - if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing me/mb/mo"); if (!ns) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing ns"); if (!pd) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing pd"); - m.ref = manifestRefFromFields(*me, *mb, *mo, "PartManifest", "descriptor"); + m.ref = fields.buildRef("PartManifest", "descriptor"); m.root_namespace_id = RootNamespace(*ns); m.payload_digest = *pd; if (!mm.eof()) @@ -176,6 +170,7 @@ PartManifest decodePartManifest(std::string_view data) /// the payload zone below can read exactly that many raw bytes back into `inline_bytes`. /// Index-aligned with `m.entries` (Blob entries push an unused 0 placeholder). std::vector inline_lens; + String blob_ref_what; /// reused across Blob entries so the error context does not allocate per row while (true) { const String line = readLine(in, line_cap, "cas_part_manifest"); @@ -198,7 +193,7 @@ PartManifest decodePartManifest(std::string_view data) break; } - if (key != "p") + if (key != PartManifestWire::path) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"p\""); ManifestEntry e; e.path = r.readString(); @@ -218,40 +213,31 @@ PartManifest decodePartManifest(std::string_view data) } std::optional pm; - std::optional ha; - std::optional h; + BlobRefFields blob_ref; std::optional sz; std::optional il; while (r.nextKey(key)) { - if (key == "pm") pm = r.readString(); - else if (key == "ha") ha = r.readString(); - else if (key == "h") h = r.readString(); - else if (key == "sz") sz = r.readU64Number(); - else if (key == "il") il = r.readU64Number(); + if (key == PartManifestWire::place) pm = r.readString(); + else if (matchBlobRefFields(key, r, blob_ref)) {} + else if (key == PartManifestWire::size) sz = r.readU64Number(); + else if (key == PartManifestWire::inline_size) il = r.readU64Number(); else r.skipUnknown(key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after record"); if (!pm) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing pm", e.path); - e.placement = placementFromWord(*pm); + e.placement = kEntryPlacementWords.fromWord(*pm, "PartManifest"); if (e.placement == EntryPlacement::Blob) { - if (!ha || !h || !sz) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing ha/h/sz", e.path); - const BlobHashAlgo algo = blobHashAlgoFromWord(*ha, "PartManifest entry"); - /// Validate the digest width before calling `fromHex`. A width mismatch otherwise - /// produces `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed - /// serialized input, allowing an invalid manifest to escape the decoder's fail-closed - /// error contract. - const uint64_t expected_hex_len = blobHashLenFor(algo) * 2; - if (h->size() != expected_hex_len) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "PartManifest: entry '{}' digest hex width {} does not match algo width {}", - e.path, h->size(), expected_hex_len); - e.ref = BlobRef{algo, codecFor(algo).fromHex(*h)}; + if (!sz) + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing sz", e.path); + blob_ref_what.assign("PartManifest entry '"); + blob_ref_what += e.path; + blob_ref_what += '\''; + e.ref = blob_ref.build(blob_ref_what); e.blob_size = *sz; inline_lens.push_back(0); /// unused for Blob; keeps inline_lens index-aligned with entries } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h index f2a416d15743..e02e1ce6a917 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h @@ -42,6 +42,9 @@ enum class EntryPlacement : uint8_t Blob = 2, /// bytes stored as a content-addressed blob at `blobKey` }; +/// Canonical wire word for one manifest entry placement. +std::string_view entryPlacementToWireWord(EntryPlacement placement); + /// One file entry inside a part manifest. `ref` is meaningful only for `Blob`; `inline_bytes` only /// for `Inline`. `blob_size` is the raw `Blob` byte count (0 for `Inline` — decode never fills it for /// an inline entry, since the wire format carries no redundant size for inline bytes). Use `size()` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp index 50e9843e9254..4e46131779cb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp @@ -1,3 +1,4 @@ +#include #include #include #include @@ -16,6 +17,15 @@ namespace ErrorCodes namespace DB::Cas { +namespace PoolMetaWire +{ + constexpr WireKey pool_id{"pid"}; + constexpr WireKey blob_header_len{"hln"}; + constexpr WireKey gc_shards{"gcs"}; + constexpr WireKey min_reader_generation{"mrg"}; + constexpr WireKey algos_used{"alg"}; +} + /// Minimum `blob_header_len` that provably fits the v3 `cas_blob` JSON envelope's mandatory (always- /// written) non-ref fields, computed at type maxima from `encodeEnvelopeHeader` (CasBlobEnvelopeFormat.cpp): /// {"type":"cas_blob" 18 @@ -33,7 +43,6 @@ namespace DB::Cas /// that used to mask this is gone). We floor at 240 (a multiple of 8 comfortably above 225, leaving /// >= 15 bytes for the diagnostic ref even at type maxima, and well under the 256 default) so a /// misconfigured pool fails at CREATION with BAD_ARGUMENTS, not at first write with LOGICAL_ERROR. -static constexpr uint64_t kMinBlobHeaderLen = 240; void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what) { @@ -52,14 +61,16 @@ void validatePoolAlgosUsed(const std::vector & algos_used, int error_co throw Exception(error_code, "CAS {}: algos_used must be non-empty", what); for (size_t i = 0; i < algos_used.size(); ++i) { - try - { - blobHashAlgoName(static_cast(algos_used[i])); - } - catch (const Exception &) - { + /// A direct membership scan, not `blobHashAlgoName`: that throws `LOGICAL_ERROR`, which + /// aborts at construction under a sanitizer/debug build before any catch can run, but a + /// persisted `algos_used` byte is exactly the unvalidated input this function must reject + /// cleanly instead. + bool known = false; + for (const auto & entry : kBlobHashAlgoWords.entries) + if (static_cast(entry.value) == algos_used[i]) + known = true; + if (!known) throw Exception(error_code, "CAS {}: algos_used contains an unknown algo {}", what, algos_used[i]); - } if (i > 0 && algos_used[i] <= algos_used[i - 1]) throw Exception(error_code, "CAS {}: algos_used must be strictly sorted with no duplicates, got {} at index {} not after {}", @@ -73,15 +84,11 @@ String encodePoolMeta(const PoolMeta & pm) writeHeaderLine(out, FormatId::PoolMeta); bool first = true; - writeKey(out, "pid", first); - writeHex128Value(out, pm.pool_id); - writeKey(out, "hln", first); - writeIntText(pm.blob_header_len, out); - writeKey(out, "gcs", first); - writeIntText(pm.gc_shards, out); - writeKey(out, "mrg", first); - writeIntText(pm.min_reader_generation, out); - writeKey(out, "alg", first); + writeHex128Field(out, PoolMetaWire::pool_id, pm.pool_id, first); + writeNumberField(out, PoolMetaWire::blob_header_len, pm.blob_header_len, first); + writeNumberField(out, PoolMetaWire::gc_shards, pm.gc_shards, first); + writeNumberField(out, PoolMetaWire::min_reader_generation, pm.min_reader_generation, first); + writeKey(out, PoolMetaWire::algos_used, first); { /// Comma-joined algo words (tiny list, <=3): "ch128" or "ch128,sha256". String joined; @@ -127,21 +134,21 @@ PoolMeta decodePoolMeta(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "pid") + if (key == PoolMetaWire::pool_id) { pm.pool_id = r.readHex128(); saw_pid = true; } - else if (key == "hln") + else if (key == PoolMetaWire::blob_header_len) pm.blob_header_len = r.readU64Number(); - else if (key == "gcs") + else if (key == PoolMetaWire::gc_shards) { pm.gc_shards = r.readU64Number(); saw_gc_shards = true; } - else if (key == "mrg") + else if (key == PoolMetaWire::min_reader_generation) pm.min_reader_generation = r.readU64Number(); - else if (key == "alg") + else if (key == PoolMetaWire::algos_used) { const String joined = r.readString(); size_t start = 0; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index b21458aaf6e2..985c1964e9e5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -19,6 +20,25 @@ namespace DB::Cas namespace { +namespace RunWire +{ + constexpr WireKey ref{"b"}; + constexpr WireKey src{"s"}; + constexpr WireKey mark{"m"}; + constexpr WireKey pending{"pend"}; + constexpr WireKey size{"sz"}; + constexpr WireKey condemn_round{"cr"}; + constexpr WireKey confirmed{"mc"}; +} + +constexpr EnumWireTable kRunMarkerWords{{{ + {RunMarker::Zero, "zero"}, + {RunMarker::Edge, "edge"}, + {RunMarker::Condemned, "condemned"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + UInt128 toWideChecksum(CityHash_v1_0_2::uint128 h) { /// Keep the high and low halves in the same order for the write-side helper and the streaming @@ -72,32 +92,20 @@ BlobRef parseB(std::string_view b) if (digest_hex.size() != static_cast(blobHashLenFor(algo)) * 2) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: digest hex width {} does not match algo width {}", digest_hex.size(), blobHashLenFor(algo) * 2); + for (const char c : digest_hex) + if (!isLowercaseHexChar(c)) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: non-lowercase-hex digest in record key"); BlobRef ref; ref.algo = algo; ref.digest = codecFor(algo).fromHex(String(digest_hex)); return ref; } -std::string_view markerToWord(char m) -{ - switch (m) - { - case kEdgeActive: return "edge"; - case kZeroMarker: return "zero"; - case kCondemned: return "condemned"; - default: - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: unknown row marker 0x{:02x}", static_cast(m)); - } } -char markerFromWord(std::string_view w) +std::string_view runMarkerToWireWord(RunMarker marker) { - if (w == "edge") return kEdgeActive; - if (w == "zero") return kZeroMarker; - if (w == "condemned") return kCondemned; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: unknown row marker '{}'", w); -} - + return kRunMarkerWords.toWord(marker, "CAS cas_run: RunMarker"); } void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) @@ -174,23 +182,16 @@ void SourceEdgeRunWriter::append(const SourceEdgeRecord & rec) scratch.clear(); bool first = true; - writeKey(scratch, "b", first); - writeStringValue(scratch, renderB(rec.ref)); - writeKey(scratch, "s", first); - writeHex128Value(scratch, rec.source_id); - writeKey(scratch, "m", first); - writeStringValue(scratch, markerToWord(rec.marker)); - if (rec.marker == kCondemned) + writeStringField(scratch, RunWire::ref, renderB(rec.ref), first); + writeHex128Field(scratch, RunWire::src, rec.source_id, first); + writeWordField(scratch, RunWire::mark, runMarkerToWireWord(rec.marker), first); + if (rec.marker == RunMarker::Condemned) { - writeKey(scratch, "pend", first); - writeBoolValue(scratch, rec.delete_pending); + writeBoolField(scratch, RunWire::pending, rec.delete_pending, first); writeTokenFields(scratch, first, rec.token); /// tt + tv - writeKey(scratch, "sz", first); - writeIntText(rec.size, scratch); - writeKey(scratch, "cr", first); - writeU64StringValue(scratch, rec.condemn_round); - writeKey(scratch, "mc", first); - writeBoolValue(scratch, rec.marker_confirmed); + writeNumberField(scratch, RunWire::size, rec.size, first); + writeU64StringField(scratch, RunWire::condemn_round, rec.condemn_round, first); + writeBoolField(scratch, RunWire::confirmed, rec.marker_confirmed, first); } closeObject(scratch, first); writeChar('\n', scratch); @@ -263,7 +264,7 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) SourceEdgeRecord out; String b; - String tv; + TokenFields token_fields; bool have_b = false; bool have_s = false; bool have_m = false; @@ -273,29 +274,27 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) bool have_sz = false; bool have_cr = false; bool have_mc = false; - TokenType tt{}; do { - if (key == "b") { b = r.readString(); have_b = true; } - else if (key == "s") { out.source_id = r.readHex128(); have_s = true; } - else if (key == "m") { out.marker = markerFromWord(r.readString()); have_m = true; } - else if (key == "pend") { out.delete_pending = r.readBool(); have_pend = true; } - else if (key == "tt") { tt = tokenTypeFromWord(r.readString(), "cas_run"); have_tt = true; } - else if (key == "tv") { tv = r.readString(); have_tv = true; } - else if (key == "sz") { out.size = r.readU64Number(); have_sz = true; } - else if (key == "cr") { out.condemn_round = r.readU64String(); have_cr = true; } - else if (key == "mc") { out.marker_confirmed = r.readBool(); have_mc = true; } + if (key == RunWire::ref) { b = r.readString(); have_b = true; } + else if (key == RunWire::src) { out.source_id = r.readHex128(); have_s = true; } + else if (key == RunWire::mark) { out.marker = kRunMarkerWords.fromWord(r.readString(), "CAS cas_run"); have_m = true; } + else if (key == RunWire::pending) { out.delete_pending = r.readBool(); have_pend = true; } + else if (matchTokenFields(key, r, token_fields)) { have_tt = token_fields.type_word.has_value(); have_tv = token_fields.value.has_value(); } + else if (key == RunWire::size) { out.size = r.readU64Number(); have_sz = true; } + else if (key == RunWire::condemn_round) { out.condemn_round = r.readU64String(); have_cr = true; } + else if (key == RunWire::confirmed) { out.marker_confirmed = r.readBool(); have_mc = true; } else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } while (r.nextKey(key)); if (!have_b || !have_s || !have_m) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing b/s/m"); out.ref = parseB(b); - if (out.marker == kCondemned) + if (out.marker == RunMarker::Condemned) { if (!have_pend || !have_tt || !have_tv || !have_sz || !have_cr || !have_mc) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pend/tt/tv/sz/cr/mc"); - out.token = Token{tv, tt}; + out.token = Token{*token_fields.value, tokenTypeFromWord(*token_fields.type_word, "cas_run")}; } else if (have_pend || have_tt || have_tv || have_sz || have_cr || have_mc) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: non-condemned record carries condemned fields"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index d5f9a4801caf..0d9c6dfdad4b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -11,6 +12,11 @@ #include #include +namespace DB::ErrorCodes +{ + extern const int CORRUPTED_DATA; +} + namespace DB::Cas { @@ -18,14 +24,30 @@ namespace DB::Cas /// format, shared by this codec and the GC fold that interprets the rows. /// /// Source-edge rows use `source_id == 0` as a sentinel key. A real active edge must never use that key; -/// both sentinel tags are restricted to it. `kZeroMarker` describes a zero transition for the current -/// generation and is dropped when the row is carried forward. `kCondemned` carries the condemned +/// both sentinel tags are restricted to it. `RunMarker::Zero` describes a zero transition for the current +/// generation and is dropped when the row is carried forward. `RunMarker::Condemned` carries the condemned /// incarnation at the sentinel key across generations until settlement; its payload contains the full /// deletion token and other condemned-row state. A condemned row subsumes the zero marker for that /// generation. -constexpr char kEdgeActive = 0x01; -constexpr char kZeroMarker = 0x00; -constexpr char kCondemned = 0x02; +enum class RunMarker : char +{ + Zero = 0x00, + Edge = 0x01, + Condemned = 0x02, +}; + +constexpr char runMarkerByte(RunMarker marker) +{ + return static_cast(marker); +} + +inline RunMarker runMarkerFromByte(char byte, std::string_view what) +{ + if (byte != runMarkerByte(RunMarker::Zero) && byte != runMarkerByte(RunMarker::Edge) + && byte != runMarkerByte(RunMarker::Condemned)) + throw Exception(ErrorCodes::CORRUPTED_DATA, "{}: unknown marker byte {}", what, static_cast(byte)); + return static_cast(byte); +} /// The `cas_run` codec represents the GC source-edge in-degree data plane as sorted NDJSON. This is /// the `RecordStream` family @@ -49,18 +71,18 @@ constexpr char kCondemned = 0x02; /// algo's width; `s` is the 32-hex source id. String-sorting records by (b, s) reproduces the current /// `(algorithm, digest, source_id)` byte order (lowercase hex preserves unsigned byte order and the /// algorithm byte is emitted first) — the invariant the fold's two-cursor merge depends on. The row-tag word -/// `m` maps to the `kEdgeActive`/`kZeroMarker`/`kCondemned` bytes; a `condemned` row additionally +/// `m` maps to the `RunMarker` bytes; a `condemned` row additionally /// carries the retired incarnation (`pend`/`tt`/`tv`/`sz`/`cr`) and the durable condemn-marker /// confirmation bit (`mc`). /// One decoded source-edge row. All fields are identifier-layer types so the codec stays backend-free. /// The condemned-only fields (`delete_pending`/`token`/`size`/`condemn_round`/`marker_confirmed`) are -/// meaningful only when `marker == kCondemned`. +/// meaningful only when `marker == RunMarker::Condemned`. struct SourceEdgeRecord { BlobRef ref{}; UInt128 source_id{}; - char marker = kEdgeActive; + RunMarker marker = RunMarker::Edge; bool delete_pending = false; Token token{}; uint64_t size = 0; @@ -71,6 +93,9 @@ struct SourceEdgeRecord /// The header-line `kind` word for the only live `cas_run` kind. inline constexpr std::string_view kSourceEdgeKindWord = "source_edge"; +/// Canonical wire word for one source-edge run marker. +std::string_view runMarkerToWireWord(RunMarker marker); + /// Write the typed header line `{"type":"cas_run","v":G_BUILD,"kind":""}\n` with a fixed key /// order for byte-determinism. The `kind` field distinguishes the record schema within the run /// family, so a reader can reject a valid run of the wrong kind before interpreting any records. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index c5b119ba44ca..66606ef5b632 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -20,30 +21,30 @@ namespace ErrorCodes namespace DB::Cas { -std::string_view nsStateToWord(NsState s) +namespace { - switch (s) - { - case NsState::Creating: return "creating"; - case NsState::Live: return "live"; - case NsState::Removing: return "removing"; - } - /// Every value reaching here came from a live `NsState` or from `nsStateFromWord`, which already - /// validated it on decode -- so this is a bug in THIS process, not corruption arriving from a - /// store. - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS ref catalog: unknown ns state {}", static_cast(s)); -} -NsState nsStateFromWord(std::string_view w) +namespace RefCatalogWire { - if (w == "creating") return NsState::Creating; - if (w == "live") return NsState::Live; - if (w == "removing") return NsState::Removing; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ns state '{}'", w); + constexpr WireKey kind{"k"}; + constexpr WireKey ns{"ns"}; + constexpr WireKey state{"st"}; + constexpr WireKey life{"inc"}; + constexpr WireKey remove_round{"rsr"}; + constexpr WireKey creator{"csr"}; + constexpr WireKey creator_epoch{"cwe"}; + constexpr WireKey creator_fence{"cfg"}; } -namespace -{ +constexpr std::string_view kEntryTag = "ent"; + +constexpr EnumWireTable kNsStateWords{{{ + {NsState::Creating, "creating"}, + {NsState::Live, "live"}, + {NsState::Removing, "removing"}, +}}}; + +static_assert(casEnumTableCoversEnum()); /// `creator` is required iff `state == Creating`, forbidden otherwise -- one predicate, used by both /// directions of the codec, so the writer's self-check and the reader's fail-close can never disagree. @@ -70,6 +71,16 @@ bool isCanonicalCatalogOrder(const std::vector & entries) } +std::string_view nsStateToWord(NsState s) +{ + return kNsStateWords.toWord(s, "CAS ref catalog"); +} + +NsState nsStateFromWord(std::string_view w) +{ + return kNsStateWords.fromWord(w, "CAS ref catalog ns state"); +} + String encodeRefCatalog(const RefCatalog & catalog) { const uint64_t line_cap = traitsFor(FormatId::RefCatalog).line_cap; @@ -136,19 +147,19 @@ String encodeRefCatalog(const RefCatalog & catalog) e.ns.string(), nsStateToWord(e.state), e.removal_started_round ? "carries" : "lacks"); bool first = true; - writeKey(out, "k", first); writeStringValue(out, "ent"); - writeKey(out, "ns", first); writeStringValue(out, e.ns.string()); - writeKey(out, "st", first); writeStringValue(out, nsStateToWord(e.state)); - writeKey(out, "inc", first); writeHex128Value(out, e.incarnation); + writeKey(out, RefCatalogWire::kind, first); writeStringValue(out, kEntryTag); + writeKey(out, RefCatalogWire::ns, first); writeStringValue(out, e.ns.string()); + writeKey(out, RefCatalogWire::state, first); writeStringValue(out, nsStateToWord(e.state)); + writeKey(out, RefCatalogWire::life, first); writeHex128Value(out, e.incarnation); if (e.removal_started_round) { - writeKey(out, "rsr", first); writeU64StringValue(out, *e.removal_started_round); + writeKey(out, RefCatalogWire::remove_round, first); writeU64StringValue(out, *e.removal_started_round); } if (e.creator) { - writeKey(out, "csr", first); writeStringValue(out, e.creator->server_root_id); - writeKey(out, "cwe", first); writeU64StringValue(out, e.creator->writer_epoch); - writeKey(out, "cfg", first); writeU64StringValue(out, e.creator->fence_generation); + writeKey(out, RefCatalogWire::creator, first); writeStringValue(out, e.creator->server_root_id); + writeKey(out, RefCatalogWire::creator_epoch, first); writeU64StringValue(out, e.creator->writer_epoch); + writeKey(out, RefCatalogWire::creator_fence, first); writeU64StringValue(out, e.creator->fence_generation); } closeObject(out, first); closeLine("ent"); @@ -190,10 +201,10 @@ RefCatalog decodeRefCatalog(std::string_view data) "CAS ref catalog: trailer count {} != {} records", n, seen); return catalog; } - if (key != "k") + if (key != RefCatalogWire::kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"k\""); const String kind = r.readString(); - if (kind != "ent") + if (kind != kEntryTag) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown record kind '{}'", kind); String ns_str; @@ -205,13 +216,13 @@ RefCatalog decodeRefCatalog(std::string_view data) std::optional removal_started_round; while (r.nextKey(key)) { - if (key == "ns") ns_str = r.readString(); - else if (key == "st") st_word = r.readString(); - else if (key == "inc") inc = r.readHex128(); - else if (key == "csr") csr = r.readString(); - else if (key == "cwe") cwe = r.readU64String(); - else if (key == "cfg") cfg = r.readU64String(); - else if (key == "rsr") removal_started_round = r.readU64String(); + if (key == RefCatalogWire::ns) ns_str = r.readString(); + else if (key == RefCatalogWire::state) st_word = r.readString(); + else if (key == RefCatalogWire::life) inc = r.readHex128(); + else if (key == RefCatalogWire::creator) csr = r.readString(); + else if (key == RefCatalogWire::creator_epoch) cwe = r.readU64String(); + else if (key == RefCatalogWire::creator_fence) cfg = r.readU64String(); + else if (key == RefCatalogWire::remove_round) removal_started_round = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ent key '{}'", key); } if (!l.eof()) @@ -340,7 +351,7 @@ uint64_t widestBlobTargetRunReservationBytes(const Layout & layout, uint64_t gc_ .key = layout.blobTargetRunKey(max, max, gc_shards - 1, 0), .checksum = std::numeric_limits::max(), .shard = gc_shards - 1, - .generation = max}); + .key_generation = max}); return encodeFoldSeal(seal).size() - encodeFoldSeal(CasFoldSeal{}).size(); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index 6ff7fa5dda43..cc344f015ce2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -15,6 +15,22 @@ namespace ErrorCodes namespace DB::Cas { +namespace +{ + +namespace RefCkptWire +{ + constexpr WireKey life_epoch{"le"}; + constexpr WireKey committed_epoch{"cte"}; + constexpr WireKey committed_seq{"cts"}; + constexpr WireKey snapshot_epoch{"cse"}; + constexpr WireKey snapshot_seq{"css"}; + constexpr WireKey seal_epoch{"lse"}; + constexpr WireKey seal_seq{"lss"}; +} + +} + void checkRefCkptInvariants(const RefCkpt & ckpt, std::string_view what) { /// PRESENT means REAL. `life_epoch` may be absent (no writer of this object knew the namespace's @@ -90,15 +106,15 @@ String encodeRefCkpt(const RefCkpt & ckpt) /// three ref formats cannot disagree on the encoding. if (ckpt.life_epoch) { - writeKey(out, "le", first); + writeKey(out, RefCkptWire::life_epoch, first); writeU64StringValue(out, *ckpt.life_epoch); } if (ckpt.committed_through) - writeRefTxnIdFields(out, first, "cte", "cts", *ckpt.committed_through); + writeRefTxnIdFields(out, first, RefCkptWire::committed_epoch, RefCkptWire::committed_seq, *ckpt.committed_through); if (ckpt.checkpoint_snapshot_id) - writeRefTxnIdFields(out, first, "cse", "css", *ckpt.checkpoint_snapshot_id); + writeRefTxnIdFields(out, first, RefCkptWire::snapshot_epoch, RefCkptWire::snapshot_seq, *ckpt.checkpoint_snapshot_id); if (ckpt.last_epoch_seal) - writeRefTxnIdFields(out, first, "lse", "lss", *ckpt.last_epoch_seal); + writeRefTxnIdFields(out, first, RefCkptWire::seal_epoch, RefCkptWire::seal_seq, *ckpt.last_epoch_seal); closeObject(out, first); writeChar('\n', out); @@ -136,13 +152,13 @@ RefCkpt decodeRefCkpt(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "le") ckpt.life_epoch = r.readU64String(); - else if (key == "cte") cte = r.readU64String(); - else if (key == "cts") cts = r.readU64String(); - else if (key == "cse") cse = r.readU64String(); - else if (key == "css") css = r.readU64String(); - else if (key == "lse") lse = r.readU64String(); - else if (key == "lss") lss = r.readU64String(); + if (key == RefCkptWire::life_epoch) ckpt.life_epoch = r.readU64String(); + else if (key == RefCkptWire::committed_epoch) cte = r.readU64String(); + else if (key == RefCkptWire::committed_seq) cts = r.readU64String(); + else if (key == RefCkptWire::snapshot_epoch) cse = r.readU64String(); + else if (key == RefCkptWire::snapshot_seq) css = r.readU64String(); + else if (key == RefCkptWire::seal_epoch) lse = r.readU64String(); + else if (key == RefCkptWire::seal_seq) lss = r.readU64String(); else r.skipUnknown(key); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index be7ee5567575..c40fe9cfb7cc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -20,27 +21,31 @@ namespace DB::Cas namespace { -std::string_view opKindToWord(RefOpKind k) +namespace RefLogWire { - switch (k) - { - case RefOpKind::NamespaceBirth: return "namespace_birth"; - case RefOpKind::OwnerTransition: return "owner_transition"; - case RefOpKind::SetPublishedAt: return "set_published_at"; - case RefOpKind::RemoveNamespace: return "remove_namespace"; - case RefOpKind::EpochSeal: return "epoch_seal"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown op kind {}", static_cast(k)); + constexpr WireKey ns{"ns"}; + constexpr WireKey txn_epoch{"we"}; + constexpr WireKey txn_seq{"rs"}; + constexpr WireKey prev_epoch{"!pse"}; + constexpr WireKey prev_seq{"!pss"}; + constexpr WireKey op{"op"}; + constexpr WireKey ref{"rn"}; + constexpr WireKey published_ms{"ts"}; } +constexpr EnumWireTable kRefOpWords{{{ + {RefOpKind::NamespaceBirth, "namespace_birth"}, + {RefOpKind::OwnerTransition, "owner_transition"}, + {RefOpKind::SetPublishedAt, "set_published_at"}, + {RefOpKind::RemoveNamespace, "remove_namespace"}, + {RefOpKind::EpochSeal, "epoch_seal"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + RefOpKind opKindFromWord(std::string_view w) { - if (w == "namespace_birth") return RefOpKind::NamespaceBirth; - if (w == "owner_transition") return RefOpKind::OwnerTransition; - if (w == "set_published_at") return RefOpKind::SetPublishedAt; - if (w == "remove_namespace") return RefOpKind::RemoveNamespace; - if (w == "epoch_seal") return RefOpKind::EpochSeal; - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown op kind '{}'", w); + return kRefOpWords.fromWord(w, "RefLogTxn"); } /// Byte budget over the encoded text. A removal-class transaction uses the larger complete-table @@ -69,22 +74,10 @@ void checkBudget(const std::vector & ops, size_t encoded_bytes) } } -void writeBindingFields(CasJsonWriter & out, bool & first, std::string_view prefix, const RefOwnerBinding & b) -{ - checkCanonicalRefName(b.ref_name, "RefLogTxn", "owner binding ref_name"); - checkManifestRef(b.manifest_ref, "RefLogTxn", "owner binding manifest_ref"); - out.key(prefix, "bk", first); - writeStringValue(out, refOwnerKindToWord(b.kind)); - out.key(prefix, "rn", first); - writeStringValue(out, b.ref_name); - writeManifestRefFields(out, first, prefix, b.manifest_ref); -} - void writeOp(CasJsonWriter & out, const RefOp & op) { bool first = true; - writeKey(out, "op", first); - writeStringValue(out, opKindToWord(op.kind)); + writeWordField(out, RefLogWire::op, refOpKindToWireWord(op.kind), first); switch (op.kind) { case RefOpKind::NamespaceBirth: @@ -93,57 +86,39 @@ void writeOp(CasJsonWriter & out, const RefOp & op) break; case RefOpKind::OwnerTransition: if (op.old_binding) - writeBindingFields(out, first, "o", *op.old_binding); + writeBindingFields(out, first, kOldBindingKeys, *op.old_binding); if (op.new_binding) - writeBindingFields(out, first, "n", *op.new_binding); + writeBindingFields(out, first, kNewBindingKeys, *op.new_binding); break; case RefOpKind::SetPublishedAt: checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); checkManifestRef(op.expected_manifest_ref, "RefLogTxn", "set_published_at manifest_ref"); - writeKey(out, "rn", first); - writeStringValue(out, op.ref_name); - writeManifestRefFields(out, first, "", op.expected_manifest_ref); - writeKey(out, "ts", first); - writeIntText(op.published_at_ms, out); + writeStringField(out, RefLogWire::ref, op.ref_name, first); + writeManifestRefFields(out, first, kBareManifestRefKeys, op.expected_manifest_ref); + writeNumberField(out, RefLogWire::published_ms, op.published_at_ms, first); break; } closeObject(out, first); writeChar('\n', out); } -/// Collector for a ManifestRef's three flat fields under an optional prefix. -struct ManifestFields -{ - std::optional me; - std::optional mb; - std::optional mo; - - bool any() const { return me || mb || mo; } - ManifestRef build(std::string_view what) const - { - if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} manifest_ref missing me/mb/mo", what); - return manifestRefFromFields(*me, *mb, *mo, "RefLogTxn", what); - } -}; - /// Collector for one binding (old/new) under a prefix. struct BindingFields { - std::optional bk; - std::optional rn; - ManifestFields mf; + std::optional kind; + std::optional ref; + ManifestRefFields manifest_fields; - bool any() const { return bk || rn || mf.any(); } + bool any() const { return kind || ref || manifest_fields.any(); } RefOwnerBinding build(std::string_view what) const { - if (!bk || !rn) + if (!kind || !ref) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing bk/rn", what); RefOwnerBinding b; - b.kind = refOwnerKindFromWord(*bk, "RefLogTxn owner binding"); - b.ref_name = *rn; + b.kind = refOwnerKindFromWord(*kind, "RefLogTxn owner binding"); + b.ref_name = *ref; checkCanonicalRefName(b.ref_name, "RefLogTxn", "owner binding ref_name"); - b.manifest_ref = mf.build(what); + b.manifest_ref = manifest_fields.buildRef("RefLogTxn", what); return b; } }; @@ -160,11 +135,10 @@ struct BindingFields void writeLogMeta(CasJsonWriter & out, const String & ns, const RefTxnId & txn_id, const std::optional & prev_epoch_seal) { bool first = true; - writeKey(out, "ns", first); - writeStringValue(out, ns); - writeRefTxnIdFields(out, first, "we", "rs", txn_id); + writeStringField(out, RefLogWire::ns, ns, first); + writeRefTxnIdFields(out, first, RefLogWire::txn_epoch, RefLogWire::txn_seq, txn_id); if (prev_epoch_seal) - writeRefTxnIdFields(out, first, "!pse", "!pss", *prev_epoch_seal); + writeRefTxnIdFields(out, first, RefLogWire::prev_epoch, RefLogWire::prev_seq, *prev_epoch_seal); closeObject(out, first); writeChar('\n', out); } @@ -176,7 +150,7 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) /// set_published_at fields std::optional sp_rn; - ManifestFields sp_mf; + ManifestRefFields sp_manifest_fields; std::optional sp_ts; /// owner_transition bindings BindingFields ob; @@ -185,21 +159,27 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) String key; while (r.nextKey(key)) { - if (key == "rn") sp_rn = r.readString(); - else if (key == "me") sp_mf.me = r.readU64String(); - else if (key == "mb") sp_mf.mb = r.readU64String(); - else if (key == "mo") sp_mf.mo = r.readU64Number(); - else if (key == "ts") sp_ts = r.readU64Number(); - else if (key == "obk") ob.bk = r.readString(); - else if (key == "orn") ob.rn = r.readString(); - else if (key == "ome") ob.mf.me = r.readU64String(); - else if (key == "omb") ob.mf.mb = r.readU64String(); - else if (key == "omo") ob.mf.mo = r.readU64Number(); - else if (key == "nbk") nb.bk = r.readString(); - else if (key == "nrn") nb.rn = r.readString(); - else if (key == "nme") nb.mf.me = r.readU64String(); - else if (key == "nmb") nb.mf.mb = r.readU64String(); - else if (key == "nmo") nb.mf.mo = r.readU64Number(); + if (key == RefLogWire::ref) + sp_rn = r.readString(); + else if (matchManifestRefFields(key, r, kBareManifestRefKeys, sp_manifest_fields)) + { + } + else if (key == RefLogWire::published_ms) + sp_ts = r.readU64Number(); + else if (key == kOldBindingKeys.kind) + ob.kind = r.readString(); + else if (key == kOldBindingKeys.ref) + ob.ref = r.readString(); + else if (matchManifestRefFields(key, r, kOldBindingKeys.manifest, ob.manifest_fields)) + { + } + else if (key == kNewBindingKeys.kind) + nb.kind = r.readString(); + else if (key == kNewBindingKeys.ref) + nb.ref = r.readString(); + else if (matchManifestRefFields(key, r, kNewBindingKeys.manifest, nb.manifest_fields)) + { + } else if (key == "pl") /// `"pl"` (payload) was removed from the op wire in stage-1 T12 (the `set_payload` op became /// `set_published_at`). The retired op WORD is already rejected by `opKindFromWord`, but this @@ -228,7 +208,7 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing rn/ts"); op.ref_name = *sp_rn; checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); - op.expected_manifest_ref = sp_mf.build("set_published_at manifest_ref"); + op.expected_manifest_ref = sp_manifest_fields.buildRef("RefLogTxn", "set_published_at"); op.published_at_ms = *sp_ts; break; } @@ -237,6 +217,11 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) } +std::string_view refOpKindToWireWord(RefOpKind kind) +{ + return kRefOpWords.toWord(kind, "RefLogTxn"); +} + bool refLogTxnIsEpochSeal(const RefLogTxn & txn) { return txn.ops.size() == 1 && txn.ops.front().kind == RefOpKind::EpochSeal; @@ -332,12 +317,27 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con String key; while (r.nextKey(key)) { - if (key == "ns") { txn.ns = r.readString(); saw_ns = true; } - else if (key == "we") { txn.txn_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { txn.txn_id.ref_sequence = r.readU64String(); saw_rs = true; } - else if (key == "!pse") pse = r.readU64String(); - else if (key == "!pss") pss = r.readU64String(); - else r.skipUnknown(key); + if (key == RefLogWire::ns) + { + txn.ns = r.readString(); + saw_ns = true; + } + else if (key == RefLogWire::txn_epoch) + { + txn.txn_id.writer_epoch = r.readU64String(); + saw_we = true; + } + else if (key == RefLogWire::txn_seq) + { + txn.txn_id.ref_sequence = r.readU64String(); + saw_rs = true; + } + else if (key == RefLogWire::prev_epoch) + pse = r.readU64String(); + else if (key == RefLogWire::prev_seq) + pss = r.readU64String(); + else + r.skipUnknown(key); } if (!saw_ns || !saw_we || !saw_rs) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing ns/we/rs"); @@ -385,7 +385,7 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con "RefLogTxn: trailer count {} != {} ops", n, txn.ops.size()); break; } - if (key != "op") + if (key != RefLogWire::op) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: record must start with \"op\""); const RefOpKind kind = opKindFromWord(r.readString()); txn.ops.push_back(readOpRecord(r, kind)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h index 34347fb97aaf..c61e11275e98 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h @@ -41,6 +41,10 @@ enum class RefOpKind : uint8_t EpochSeal = 5, }; +/// Convert a ref-log operation discriminator to its canonical wire word. Throws `LOGICAL_ERROR` if +/// `kind` is not represented by this format. +std::string_view refOpKindToWireWord(RefOpKind kind); + /// One operation inside a `RefLogTxn`. Only the fields documented next to `kind` are meaningful for /// that kind, and the codec never reads or writes the others. `OwnerTransition` optionally removes /// `old_binding` and/or installs `new_binding`; `SetPublishedAt` carries the expected manifest and the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index f31e9fed4ca2..8ed19c9a1ac3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -20,6 +20,20 @@ namespace DB::Cas namespace { +namespace RefSnapWire +{ + constexpr WireKey ns{"ns"}; + constexpr WireKey snapshot_epoch{"we"}; + constexpr WireKey snapshot_seq{"rs"}; + constexpr WireKey lifecycle{"lc"}; + constexpr WireKey kind{"k"}; + constexpr WireKey ref{"rn"}; + constexpr WireKey published_ms{"ts"}; +} + +constexpr std::string_view kCommittedTag = "c"; +constexpr std::string_view kPrecommitTag = "p"; + void checkCommittedSorted(const std::vector & rows) { for (size_t i = 1; i < rows.size(); ++i) @@ -59,13 +73,10 @@ void writeCommittedRow(CasJsonWriter & out, const RefCommittedRow & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "committed"); bool first = true; - writeKey(out, "k", first); - writeStringValue(out, "c"); - writeKey(out, "rn", first); - writeStringValue(out, row.ref_name); - writeManifestRefFields(out, first, "", row.manifest_ref); - writeKey(out, "ts", first); - writeIntText(row.published_at_ms, out); + writeWordField(out, RefSnapWire::kind, kCommittedTag, first); + writeStringField(out, RefSnapWire::ref, row.ref_name, first); + writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); + writeNumberField(out, RefSnapWire::published_ms, row.published_at_ms, first); closeObject(out, first); writeChar('\n', out); } @@ -79,11 +90,9 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "precommit"); bool first = true; - writeKey(out, "k", first); - writeStringValue(out, "p"); - writeKey(out, "rn", first); - writeStringValue(out, row.ref_name); - writeManifestRefFields(out, first, "", row.manifest_ref); + writeWordField(out, RefSnapWire::kind, kPrecommitTag, first); + writeStringField(out, RefSnapWire::ref, row.ref_name, first); + writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); closeObject(out, first); writeChar('\n', out); } @@ -95,33 +104,13 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) { bool first = true; - writeKey(out, "ns", first); - writeStringValue(out, snapshot.ns); - writeRefTxnIdFields(out, first, "we", "rs", snapshot.snapshot_id); - writeKey(out, "lc", first); - writeStringValue(out, "live"); + writeStringField(out, RefSnapWire::ns, snapshot.ns, first); + writeRefTxnIdFields(out, first, RefSnapWire::snapshot_epoch, RefSnapWire::snapshot_seq, snapshot.snapshot_id); + writeStringField(out, RefSnapWire::lifecycle, "live", first); closeObject(out, first); writeChar('\n', out); } -/// Collector for a ManifestRef's three flat fields (bare "me"/"mb"/"mo"). -struct ManifestFields -{ - std::optional me; - std::optional mb; - std::optional mo; - - /// Reconstruct a manifest reference after the tolerant reader has collected all three flat - /// fields. Missing fields are malformed input; `manifestRefFromFields` performs the remaining - /// range checks and reports the same corruption context as the row decoder. - ManifestRef build(std::string_view what) const - { - if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: {} manifest_ref missing me/mb/mo", what); - return manifestRefFromFields(*me, *mb, *mo, "RefTableSnapshot", what); - } -}; - } String encodeRefTableSnapshot(const RefTableSnapshot & snapshot) @@ -166,10 +155,10 @@ RefTableSnapshot decodeRefTableSnapshot( String key; while (r.nextKey(key)) { - if (key == "ns") { snapshot.ns = r.readString(); saw_ns = true; } - else if (key == "we") { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } - else if (key == "lc") + if (key == RefSnapWire::ns) { snapshot.ns = r.readString(); saw_ns = true; } + else if (key == RefSnapWire::snapshot_epoch) { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } + else if (key == RefSnapWire::snapshot_seq) { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } + else if (key == RefSnapWire::lifecycle) { const String lifecycle = r.readString(); if (lifecycle != "live") @@ -210,20 +199,20 @@ RefTableSnapshot decodeRefTableSnapshot( "RefTableSnapshot: trailer count {} != {} rows", n, snapshot.committed.size() + snapshot.precommits.size()); break; } - if (key != "k") + if (key != RefSnapWire::kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"k\""); const String k = r.readString(); std::optional rn; - ManifestFields mf; + ManifestRefFields mf; std::optional ts; while (r.nextKey(key)) { - if (key == "rn") rn = r.readString(); - else if (key == "me") mf.me = r.readU64String(); - else if (key == "mb") mf.mb = r.readU64String(); - else if (key == "mo") mf.mo = r.readU64Number(); - else if (key == "ts") ts = r.readU64Number(); + if (key == RefSnapWire::ref) rn = r.readString(); + else if (matchManifestRefFields(key, r, kBareManifestRefKeys, mf)) + { + } + else if (key == RefSnapWire::published_ms) ts = r.readU64Number(); else if (key == "pl") /// `"pl"` (payload) was removed from the row wire in stage-1 T12. It is a KNOWN-removed /// field, not a genuinely-unknown future one the tolerant reader may skip -- silently @@ -235,18 +224,18 @@ RefTableSnapshot decodeRefTableSnapshot( if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after record"); - if (k == "c") + if (k == kCommittedTag) { if (!rn || !ts) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing rn/ts"); RefCommittedRow row; row.ref_name = *rn; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); - row.manifest_ref = mf.build("committed"); + row.manifest_ref = mf.buildRef("RefTableSnapshot", "committed"); row.published_at_ms = *ts; snapshot.committed.push_back(std::move(row)); } - else if (k == "p") + else if (k == kPrecommitTag) { if (!rn) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing rn"); @@ -254,7 +243,7 @@ RefTableSnapshot decodeRefTableSnapshot( row.kind = RefOwnerKind::Precommit; row.ref_name = *rn; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); - row.manifest_ref = mf.build("precommit"); + row.manifest_ref = mf.buildRef("RefTableSnapshot", "precommit"); snapshot.precommits.push_back(std::move(row)); } else diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp index bf1a5445e4df..c184f99d4f4c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp @@ -1,4 +1,6 @@ #include +#include +#include #include namespace DB @@ -12,21 +14,26 @@ namespace ErrorCodes namespace DB::Cas { +namespace +{ + +constexpr EnumWireTable kRefOwnerKindWords{{{ + {RefOwnerKind::Committed, "committed"}, + {RefOwnerKind::Precommit, "precommit"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + +} + std::string_view refOwnerKindToWord(RefOwnerKind k) { - switch (k) - { - case RefOwnerKind::Committed: return "committed"; - case RefOwnerKind::Precommit: return "precommit"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref wire: unknown RefOwnerKind {}", static_cast(k)); + return kRefOwnerKindWords.toWord(k, "CAS ref wire: RefOwnerKind"); } RefOwnerKind refOwnerKindFromWord(std::string_view w, std::string_view what) { - if (w == "committed") return RefOwnerKind::Committed; - if (w == "precommit") return RefOwnerKind::Precommit; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown owner kind '{}'", what, w); + return kRefOwnerKindWords.fromWord(w, what); } void checkRefTxnIdNonzero(const RefTxnId & id, std::string_view format, std::string_view field) @@ -36,12 +43,19 @@ void checkRefTxnIdNonzero(const RefTxnId & id, std::string_view format, std::str "{}: {} fields must both be nonzero, got {}-{}", format, field, id.writer_epoch, id.ref_sequence); } -void writeRefTxnIdFields(CasJsonWriter & out, bool & first, std::string_view epoch_key, std::string_view seq_key, const RefTxnId & id) +void writeRefTxnIdFields(CasJsonWriter & out, bool & first, WireKey epoch_key, WireKey seq_key, const RefTxnId & id) +{ + writeU64StringField(out, epoch_key, id.writer_epoch, first); + writeU64StringField(out, seq_key, id.ref_sequence, first); +} + +void writeBindingFields(CasJsonWriter & out, bool & first, const BindingWireKeys & keys, const RefOwnerBinding & binding) { - writeKey(out, epoch_key, first); - writeU64StringValue(out, id.writer_epoch); - writeKey(out, seq_key, first); - writeU64StringValue(out, id.ref_sequence); + checkCanonicalRefName(binding.ref_name, "RefLogTxn", "owner binding ref_name"); + checkManifestRef(binding.manifest_ref, "RefLogTxn", "owner binding manifest_ref"); + writeWordField(out, keys.kind, refOwnerKindToWord(binding.kind), first); + writeStringField(out, keys.ref, binding.ref_name, first); + writeManifestRefFields(out, first, keys.manifest, binding.manifest_ref); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h index fdccef8f7fdb..e13fd116331e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h @@ -50,7 +50,11 @@ RefOwnerKind refOwnerKindFromWord(std::string_view w, std::string_view what); /// letting each format distinguish its primary id from any secondary id it embeds (for example, /// `cas_ref_log`'s `we`/`rs` versus its `prev_epoch_seal` pair) while sharing one writer so the /// formats can never disagree on the representation. -void writeRefTxnIdFields(CasJsonWriter & out, bool & first, std::string_view epoch_key, std::string_view seq_key, const RefTxnId & id); +void writeRefTxnIdFields(CasJsonWriter & out, bool & first, WireKey epoch_key, WireKey seq_key, const RefTxnId & id); + +/// Append one owner binding's flat fields named by `keys` to a ref-log `owner_transition` object. +/// The binding's ref name and manifest reference are validated before writing. +void writeBindingFields(CasJsonWriter & out, bool & first, const BindingWireKeys & keys, const RefOwnerBinding & binding); /// `RefTxnId`'s validity rule applied to ONE field of a decoded or about-to-be-encoded record: both /// components nonzero. `renderRefTxnId` refuses to build a key from anything else, so a half-zero id diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp index f523553271b4..68c26d47e0e6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp @@ -14,6 +14,31 @@ namespace ErrorCodes namespace DB::Cas { +namespace OwnerWire +{ + constexpr WireKey server_uuid{"su"}; + constexpr WireKey retired_at_ms{"rt"}; +} + +namespace ServerEpochWire +{ + constexpr WireKey next_writer_epoch{"nwe"}; +} + +namespace MountLeaseWire +{ + constexpr WireKey server_uuid{"su"}; + constexpr WireKey writer_epoch{"we"}; + constexpr WireKey hostname{"hn"}; + constexpr WireKey pid{"pid"}; + constexpr WireKey started_at_ms{"sat"}; + constexpr WireKey seq{"seq"}; + constexpr WireKey expires_at_ms{"eat"}; + constexpr WireKey min_active{"ma"}; + constexpr WireKey gc_fenced{"fen"}; + constexpr WireKey write_attempt_id{"write_attempt_id"}; +} + namespace { @@ -32,13 +57,9 @@ String encodeOwner(const OwnerObject & o) CasJsonWriter out(256); writeHeaderLine(out, FormatId::Owner); bool first = true; - writeKey(out, "su", first); - writeHex128Value(out, o.server_uuid); + writeHex128Field(out, OwnerWire::server_uuid, o.server_uuid, first); if (o.retired_at_ms) - { - writeKey(out, "rt", first); - writeIntText(*o.retired_at_ms, out); - } + writeNumberField(out, OwnerWire::retired_at_ms, *o.retired_at_ms, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -58,12 +79,12 @@ OwnerObject decodeOwner(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "su") + if (key == OwnerWire::server_uuid) { o.server_uuid = r.readHex128(); saw = true; } - else if (key == "rt") + else if (key == OwnerWire::retired_at_ms) rt = r.readU64Number(); else r.skipUnknown(key); @@ -81,8 +102,7 @@ String encodeServerEpoch(const ServerEpoch & e) CasJsonWriter out(256); writeHeaderLine(out, FormatId::ServerEpoch); bool first = true; - writeKey(out, "nwe", first); - writeU64StringValue(out, e.next_writer_epoch); + writeU64StringField(out, ServerEpochWire::next_writer_epoch, e.next_writer_epoch, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -101,7 +121,7 @@ ServerEpoch decodeServerEpoch(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "nwe") + if (key == ServerEpochWire::next_writer_epoch) { e.next_writer_epoch = r.readU64String(); saw = true; @@ -121,16 +141,16 @@ String encodeMountLease(const MountLease & m) CasJsonWriter out(256); writeHeaderLine(out, FormatId::MountLease); bool first = true; - writeKey(out, "su", first); writeHex128Value(out, m.server_uuid); - writeKey(out, "we", first); writeU64StringValue(out, m.writer_epoch); - writeKey(out, "hn", first); writeStringValue(out, m.hostname); - writeKey(out, "pid", first); writeIntText(m.pid, out); - writeKey(out, "sat", first); writeIntText(m.started_at_ms, out); - writeKey(out, "seq", first); writeU64StringValue(out, m.seq); - writeKey(out, "eat", first); writeIntText(m.expires_at_ms, out); - writeKey(out, "ma", first); writeU64StringValue(out, m.min_active); - writeKey(out, "fen", first); writeBoolValue(out, m.gc_fenced); - writeKey(out, "write_attempt_id", first); writeHex128Value(out, m.write_attempt_id); + writeHex128Field(out, MountLeaseWire::server_uuid, m.server_uuid, first); + writeU64StringField(out, MountLeaseWire::writer_epoch, m.writer_epoch, first); + writeStringField(out, MountLeaseWire::hostname, m.hostname, first); + writeNumberField(out, MountLeaseWire::pid, m.pid, first); + writeNumberField(out, MountLeaseWire::started_at_ms, m.started_at_ms, first); + writeU64StringField(out, MountLeaseWire::seq, m.seq, first); + writeNumberField(out, MountLeaseWire::expires_at_ms, m.expires_at_ms, first); + writeU64StringField(out, MountLeaseWire::min_active, m.min_active, first); + writeBoolField(out, MountLeaseWire::gc_fenced, m.gc_fenced, first); + writeHex128Field(out, MountLeaseWire::write_attempt_id, m.write_attempt_id, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -151,29 +171,37 @@ MountLease decodeMountLease(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "su") + if (key == MountLeaseWire::server_uuid) { m.server_uuid = r.readHex128(); saw_su = true; } - else if (key == "we") + else if (key == MountLeaseWire::writer_epoch) { m.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "hn") m.hostname = r.readString(); - else if (key == "pid") m.pid = r.readU64Number(); - else if (key == "sat") m.started_at_ms = r.readU64Number(); - else if (key == "seq") m.seq = r.readU64String(); - else if (key == "eat") m.expires_at_ms = r.readU64Number(); - else if (key == "ma") m.min_active = r.readU64String(); - else if (key == "fen") m.gc_fenced = r.readBool(); - else if (key == "write_attempt_id") + else if (key == MountLeaseWire::hostname) + m.hostname = r.readString(); + else if (key == MountLeaseWire::pid) + m.pid = r.readU64Number(); + else if (key == MountLeaseWire::started_at_ms) + m.started_at_ms = r.readU64Number(); + else if (key == MountLeaseWire::seq) + m.seq = r.readU64String(); + else if (key == MountLeaseWire::expires_at_ms) + m.expires_at_ms = r.readU64Number(); + else if (key == MountLeaseWire::min_active) + m.min_active = r.readU64String(); + else if (key == MountLeaseWire::gc_fenced) + m.gc_fenced = r.readBool(); + else if (key == MountLeaseWire::write_attempt_id) { m.write_attempt_id = r.readHex128(); saw_write_attempt_id = true; } - else r.skipUnknown(key); + else + r.skipUnknown(key); } if (!saw_su || !saw_we || !saw_write_attempt_id || m.write_attempt_id == UInt128{}) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing or zero identity field"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 9814d5d14811..8a53273956ee 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -192,8 +192,7 @@ UInt128 JsonObjectReader::readHex128() return guarded([&] { const String hex = readString(); - if (hex.size() != 32 - || std::any_of(hex.begin(), hex.end(), [](char c) { return unhex(c) == 0xff || (c >= 'A' && c <= 'F'); })) + if (hex.size() != 32 || std::any_of(hex.begin(), hex.end(), [](char c) { return !isLowercaseHexChar(c); })) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: expected 32 lowercase hex chars, got '{}'", what, hex); return unhexUInt(hex.data()); }); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 50edd33bd4ba..2ba6df5a1c5e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -59,18 +59,6 @@ class CasJsonWriter append("\":"); } - /// Same, for the prefixed key vocabulary ("o"/"n" + "me"/"mb"/"mo"/"bk"/"rn") — the - /// prefix and name are appended back to back, no composed temporary. - void key(std::string_view prefix, std::string_view name, bool & first) - { - appendChar(first ? '{' : ','); - first = false; - appendChar('"'); - append(prefix); - append(name); - append("\":"); - } - /// Quoted JSON string with full escaping (bulk-run scan). Defined in CasTextFormat.cpp. void stringValue(std::string_view s); @@ -154,6 +142,65 @@ inline void writeIntText(uint64_t v, CasJsonWriter & out) { out.u64Number(v); } void writeHeaderLine(CasJsonWriter & out, FormatId id); void writeTrailerLine(CasJsonWriter & out, uint64_t n); +/// A wire-key carrier. The explicit constructor keeps raw string literals out of writer call +/// sites: a codec passes its named constant, and an inline `WireKey{"..."}` is deliberately loud. +struct WireKey +{ + std::string_view text; + + explicit constexpr WireKey(std::string_view text_) : text(text_) {} + + friend constexpr bool operator==(std::string_view s, const WireKey & k) { return s == k.text; } +}; + +inline void writeKey(CasJsonWriter & out, WireKey key, bool & first) +{ + writeKey(out, key.text, first); +} + +inline void writeWordField(CasJsonWriter & out, WireKey key, std::string_view word, bool & first) +{ + writeKey(out, key, first); + writeStringValue(out, word); +} + +inline void writeStringField(CasJsonWriter & out, WireKey key, std::string_view value, bool & first) +{ + writeKey(out, key, first); + writeStringValue(out, value); +} + +inline void writeU64StringField(CasJsonWriter & out, WireKey key, uint64_t value, bool & first) +{ + writeKey(out, key, first); + writeU64StringValue(out, value); +} + +inline void writeNumberField(CasJsonWriter & out, WireKey key, uint64_t value, bool & first) +{ + writeKey(out, key, first); + out.u64Number(value); +} + +inline void writeHex128Field(CasJsonWriter & out, WireKey key, const UInt128 & value, bool & first) +{ + writeKey(out, key, first); + writeHex128Value(out, value); +} + +inline void writeBoolField(CasJsonWriter & out, WireKey key, bool value, bool & first) +{ + writeKey(out, key, first); + writeBoolValue(out, value); +} + +/// True iff `c` is one lowercase hexadecimal digit. Persisted CAS digests deliberately reject +/// uppercase spellings so each digest has one canonical textual representation. +constexpr bool isLowercaseHexChar(char c) +{ + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'); +} + /// Pull cursor over one canonical JSON object. /// /// The reader borrows the input buffer and records the object name for exception messages. It diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 31d44f260121..e44bd062c33d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -1,7 +1,10 @@ #include #include #include +#include #include +#include +#include namespace DB { @@ -14,73 +17,57 @@ namespace ErrorCodes namespace DB::Cas { +static_assert(casEnumTableCoversEnum()); +static_assert(casEnumTableCoversEnum()); + std::string_view tokenTypeToWord(TokenType t) { - switch (t) - { - case TokenType::ETag: return "etag"; - case TokenType::Generation: return "generation"; - case TokenType::Emulated: return "emulated"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS wire: unknown TokenType {}", static_cast(t)); + return kTokenTypeWords.toWord(t, "CAS wire: TokenType"); } TokenType tokenTypeFromWord(std::string_view w, std::string_view what) { - if (w == "etag") return TokenType::ETag; - if (w == "generation") return TokenType::Generation; - if (w == "emulated") return TokenType::Emulated; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown token type '{}'", what, w); + return kTokenTypeWords.fromWord(w, what); } BlobHashAlgo blobHashAlgoFromWord(std::string_view w, std::string_view what) { - if (w == "ch128") return BlobHashAlgo::CityHash128; - if (w == "xxh3") return BlobHashAlgo::XXH3_128; - if (w == "sha256") return BlobHashAlgo::Sha256; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown blob hash algo '{}'", what, w); + return kBlobHashAlgoWords.fromWord(w, what); } std::string_view objectKindToWord(ObjectKind k) { - switch (k) - { - case ObjectKind::Blob: return "blob"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS wire: unknown ObjectKind {}", static_cast(k)); + return kObjectKindWords.toWord(k, "CAS wire: ObjectKind"); } ObjectKind objectKindFromWord(std::string_view w, std::string_view what) { - if (w == "blob") return ObjectKind::Blob; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown object kind '{}'", what, w); + return kObjectKindWords.fromWord(w, what); } void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t) { - writeKey(out, "tt", first); + writeKey(out, SharedWire::token_type, first); writeStringValue(out, tokenTypeToWord(t.type)); - writeKey(out, "tv", first); + writeKey(out, SharedWire::token, first); writeStringValue(out, t.value); } void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r) { - writeKey(out, "ha", first); + writeKey(out, SharedWire::algo, first); writeStringValue(out, blobHashAlgoName(r.algo)); - writeKey(out, "h", first); + writeKey(out, SharedWire::digest, first); writeStringValue(out, codecFor(r.algo).toHex(r.digest)); } -void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view prefix, const ManifestRef & r) +void writeManifestRefFields(CasJsonWriter & out, bool & first, const ManifestRefWireKeys & keys, const ManifestRef & r) { - /// Unlike the WriteBuffer overload, the two-part key() form appends the prefix and name back - /// to back with no composed String(prefix) + "..." temporary. - out.key(prefix, "me", first); + writeKey(out, keys.epoch, first); out.u64StringValue(r.writer_epoch); - out.key(prefix, "mb", first); + writeKey(out, keys.build, first); out.u64StringValue(r.build_sequence); - out.key(prefix, "mo", first); + writeKey(out, keys.ord, first); out.u64Number(r.manifest_ordinal); } @@ -100,4 +87,31 @@ ManifestRef manifestRefFromFields(uint64_t writer_epoch, uint64_t build_sequence return r; } +ManifestRef ManifestRefFields::buildRef(std::string_view what, std::string_view context) const +{ + if (!epoch || !build || !ord) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: {} manifest_ref missing epoch/build/ord", what, context); + return manifestRefFromFields(*epoch, *build, *ord, what, context); +} + +BlobRef BlobRefFields::build(std::string_view what) const +{ + if (!algo_word || !digest_hex) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: blob ref missing ha/h", what); + const BlobHashAlgo algo = blobHashAlgoFromWord(*algo_word, what); + /// Validate the digest width before calling `fromHex`. A width mismatch otherwise produces + /// `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed serialized input, + /// allowing an invalid record to escape the decoder's fail-closed error contract. + const uint64_t expected_hex_len = blobHashLenFor(algo) * 2; + if (digest_hex->size() != expected_hex_len) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS {}: digest hex width {} does not match algo width {}", what, digest_hex->size(), expected_hex_len); + /// Same fence, same reason as the width check above: a right-width but non-lowercase-hex digest + /// must also surface as CORRUPTED_DATA rather than `DigestCodec::fromHex`'s BAD_ARGUMENTS. Mirrors + /// `JsonObjectReader::readHex128`'s lowercase-hex predicate. + if (std::any_of(digest_hex->begin(), digest_hex->end(), [](char c) { return !isLowercaseHexChar(c); })) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: digest is not lowercase hex, got '{}'", what, *digest_hex); + return BlobRef{algo, codecFor(algo).fromHex(*digest_hex)}; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h index 8727e6e219b6..eea099e7711e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h @@ -1,9 +1,11 @@ #pragma once +#include #include #include #include #include #include +#include #include namespace DB::Cas @@ -15,6 +17,18 @@ namespace DB::Cas /// unrecognized value with `CORRUPTED_DATA`; silently choosing a default would turn malformed /// persisted data into a different valid-looking record. +/// The `TokenType` wire vocabulary; coverage is proven in `CasWireVocab.cpp`. +inline constexpr EnumWireTable kTokenTypeWords{{{ + {TokenType::ETag, "etag"}, + {TokenType::Generation, "generation"}, + {TokenType::Emulated, "emulated"}, +}}}; + +/// The `ObjectKind` wire vocabulary; coverage is proven in `CasWireVocab.cpp`. +inline constexpr EnumWireTable kObjectKindWords{{{ + {ObjectKind::Blob, "blob"}, +}}}; + /// Convert a token discriminator to its canonical wire word. Throws `CORRUPTED_DATA` if `t` is not /// one of the token types understood by this build. std::string_view tokenTypeToWord(TokenType t); @@ -44,12 +58,53 @@ void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t); /// lowercase digest are canonical, and the digest is rendered at the width required by `r.algo`. void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r); -/// Append the three flat `ManifestRef` fields `me`, `mb`, and `mo` to an in-progress JSON object. -/// `prefix` is prepended to each key, allowing the ref codecs to distinguish old and new owner -/// bindings (`ome`/`omb`/`omo` and `nme`/`nmb`/`nmo`) while part manifests and ordinary rows use an -/// empty prefix. The two unbounded `uint64_t` values are decimal JSON strings; the bounded ordinal -/// is a JSON number. All consumers use this exact spelling and representation. -void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view prefix, const ManifestRef & r); +/// The `ha`/`h` and `tt`/`tv` key spellings, named once so `writeBlobRefFields`/`writeTokenFields` +/// and the `match*Fields` collectors below can never drift apart on the literal. +namespace SharedWire +{ + inline constexpr WireKey algo{"ha"}; + inline constexpr WireKey digest{"h"}; + inline constexpr WireKey token_type{"tt"}; + inline constexpr WireKey token{"tv"}; +} + +/// One `ManifestRef`'s three flat key names. Every bundle spells the SAME wire representation +/// (two decimal-string `uint64_t`s and one JSON-number ordinal); only the key names vary per +/// binding role. Member names carry the semantic role the ref plays (`epoch`/`build`/`ord`); the +/// bundle constants below carry the CURRENT wire spelling for each role. +struct ManifestRefWireKeys +{ + WireKey epoch; + WireKey build; + WireKey ord; +}; + +/// The unprefixed `me`/`mb`/`mo` spelling used by part manifests, snapshot rows, and the +/// `set_published_at` ref-log op. +inline constexpr ManifestRefWireKeys kBareManifestRefKeys{WireKey{"me"}, WireKey{"mb"}, WireKey{"mo"}}; +/// The `ome`/`omb`/`omo` spelling for a ref-log owner_transition's OLD binding. +inline constexpr ManifestRefWireKeys kOldManifestRefKeys{WireKey{"ome"}, WireKey{"omb"}, WireKey{"omo"}}; +/// The `nme`/`nmb`/`nmo` spelling for a ref-log owner_transition's NEW binding. +inline constexpr ManifestRefWireKeys kNewManifestRefKeys{WireKey{"nme"}, WireKey{"nmb"}, WireKey{"nmo"}}; + +/// One owner binding's key names: the owner-kind word, the ref name, and its nested `ManifestRef` +/// bundle. Only the ref-log owner_transition op uses this bundle (old/new binding sides). +struct BindingWireKeys +{ + WireKey kind; + WireKey ref; + ManifestRefWireKeys manifest; +}; + +/// The `obk`/`orn`/`ome`/`omb`/`omo` spelling for the OLD binding side. +inline constexpr BindingWireKeys kOldBindingKeys{WireKey{"obk"}, WireKey{"orn"}, kOldManifestRefKeys}; +/// The `nbk`/`nrn`/`nme`/`nmb`/`nmo` spelling for the NEW binding side. +inline constexpr BindingWireKeys kNewBindingKeys{WireKey{"nbk"}, WireKey{"nrn"}, kNewManifestRefKeys}; + +/// Append the three flat `ManifestRef` fields named by `keys` to an in-progress JSON object. The +/// two unbounded `uint64_t` values are decimal JSON strings; the bounded ordinal is a JSON number. +/// All consumers use this exact representation; only the key spelling varies by `keys`. +void writeManifestRefFields(CasJsonWriter & out, bool & first, const ManifestRefWireKeys & keys, const ManifestRef & r); /// Construct a `ManifestRef` from decoded field values and validate the complete domain range: /// nonzero `writer_epoch` and `build_sequence`, and `manifest_ordinal` in @@ -59,4 +114,72 @@ void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view ManifestRef manifestRefFromFields(uint64_t writer_epoch, uint64_t build_sequence, uint64_t manifest_ordinal, std::string_view caller, std::string_view what); +/// Collector for one `ManifestRef`'s three flat fields, filled in by repeated calls to +/// `matchManifestRefFields` as a tolerant reader walks an object's keys. `buildRef` checks that the +/// group is all-or-nothing complete, then delegates the completed group to `manifestRefFromFields`, +/// which performs the nonzero and range checks. +struct ManifestRefFields +{ + std::optional epoch; + std::optional build; + std::optional ord; + + bool any() const { return epoch || build || ord; } + + /// `what` names the codec (passed through to `manifestRefFromFields` as its `caller`); `context` + /// names the field being reconstructed (e.g. "descriptor", "committed"). Throws `CORRUPTED_DATA` + /// if the group is not all-or-nothing complete. + ManifestRef buildRef(std::string_view what, std::string_view context) const; +}; + +/// Collector for one `BlobRef`'s two flat fields (`ha`/`h`), filled in by `matchBlobRefFields`. +struct BlobRefFields +{ + std::optional algo_word; + std::optional digest_hex; + + /// Requires both fields, parses the algorithm word, and checks the digest hex width against the + /// algorithm's width BEFORE calling `fromHex` -- a width mismatch must surface as `CORRUPTED_DATA` + /// (malformed persisted input), not `DigestCodec::fromHex`'s `BAD_ARGUMENTS` (a caller-contract + /// violation). `what` identifies the field in the exception. + BlobRef build(std::string_view what) const; +}; + +/// Collector for one `Token`'s two flat fields (`tt`/`tv`), filled in by `matchTokenFields`. Phase 1 +/// deliberately has no `build`: callers keep their own local requiredness checks until the unified +/// both-required build is introduced. +struct TokenFields +{ + std::optional type_word; + std::optional value; +}; + +/// Each `match*Fields` helper tests `key` against the one or two field names it owns, consumes the +/// value on a match via `r`, and reports whether it recognized the key. None of them loop over an +/// object's keys or validate a completed group -- that is the caller's (tolerant-reader loop) and +/// the collector's `build`/`buildRef` job respectively. Defined inline: a decoder's per-key dispatch +/// is a hot path and must not gain a function-call boundary here. + +inline bool matchManifestRefFields(std::string_view key, JsonObjectReader & r, const ManifestRefWireKeys & keys, ManifestRefFields & fields) +{ + if (key == keys.epoch) { fields.epoch = r.readU64String(); return true; } + if (key == keys.build) { fields.build = r.readU64String(); return true; } + if (key == keys.ord) { fields.ord = r.readU64Number(); return true; } + return false; +} + +inline bool matchBlobRefFields(std::string_view key, JsonObjectReader & r, BlobRefFields & fields) +{ + if (key == SharedWire::algo) { fields.algo_word = r.readString(); return true; } + if (key == SharedWire::digest) { fields.digest_hex = r.readString(); return true; } + return false; +} + +inline bool matchTokenFields(std::string_view key, JsonObjectReader & r, TokenFields & fields) +{ + if (key == SharedWire::token_type) { fields.type_word = r.readString(); return true; } + if (key == SharedWire::token) { fields.value = r.readString(); return true; } + return false; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp index d018228750da..2700aec0291a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp @@ -40,11 +40,11 @@ const UInt128 kZeroSourceId{0}; /// Streams a shard's prior source-edge run at O(one block) resident memory: chains the run SEGMENTS the /// caller resolved from the parent seal (`blob_target_runs` filtered to one shard) and exposes a one-row /// lookahead for the fold merge. The prior run carries -/// BOTH surviving edges (`kEdgeActive`) AND the retired `kCondemned` sentinel rows at the zero source id, +/// BOTH surviving edges (`RunMarker::Edge`) AND the retired `RunMarker::Condemned` sentinel rows at the zero source id, /// so the cursor stops at edges AND at condemned rows (exposing the type via `rowType`), while zero-marker /// sentinels are dropped on carry (per-generation, never carried forward). Row/key invariants are enforced -/// while streaming: `kEdgeActive` never at `source_id = 0`; sentinel rows (`kZeroMarker` / -/// `kCondemned`) ONLY at `source_id = 0`; at most one sentinel per blob; an unknown value byte or an empty +/// while streaming: `RunMarker::Edge` never at `source_id = 0`; sentinel rows (`RunMarker::Zero` / +/// `RunMarker::Condemned`) ONLY at `source_id = 0`; at most one sentinel per blob; an unknown value byte or an empty /// payload is `CORRUPTED_DATA`. Resolution uses the exact object references supplied by the caller, so a run /// sealed for generation G that physically lives under an older generation's key is reached /// without key construction. An empty `segments` is the fresh-pool / empty baseline. The row stream is @@ -61,10 +61,10 @@ class PriorEdgeCursor bool valid() const { return has_current; } const String & key() const { return current_key; } - /// The value byte of the current row: `kEdgeActive` (a surviving edge) or `kCondemned` (a retired + /// The value byte of the current row: `RunMarker::Edge` (a surviving edge) or `RunMarker::Condemned` (a retired /// sentinel row). Zero markers are never surfaced (dropped on carry). - char rowType() const { return current_type; } - /// The decoded retired sentinel for the current row (only valid when `rowType() == kCondemned`). + RunMarker rowType() const { return current_type; } + /// The decoded retired sentinel for the current row (only valid when `rowType() == RunMarker::Condemned`). const CondemnedRow & condemnedRow() const { return current_condemned; } /// Advance to the next surviving edge OR retired sentinel, dropping zero markers, enforcing the @@ -87,40 +87,37 @@ class PriorEdgeCursor SourceEdgeKeyCodec::parse(k, bh, sid); if (p.empty()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: empty row payload"); - const char v = p[0]; + const RunMarker v = runMarkerFromByte(p[0], "CAS source-edge run"); const bool sentinel_key = (sid == kZeroSourceId); if (sentinel_key) { /// A sentinel key carries exactly one row per blob and never an edge. - if (v == kEdgeActive) + if (v == RunMarker::Edge) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: active edge at the reserved sentinel source_id 0"); - if (v != kZeroMarker && v != kCondemned) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS source-edge run: unknown sentinel row type 0x{:02x}", static_cast(v)); if (have_sentinel_blob && sentinel_blob == bh) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: duplicate sentinel row for one blob"); have_sentinel_blob = true; sentinel_blob = bh; - if (v == kZeroMarker) + if (v == RunMarker::Zero) continue; // A zero marker is per-generation and is dropped on carry. /// A retired sentinel: decode and surface it (settled at close-out, not an edge). current_condemned = decodeCondemnedRow(p); current_key = k; - current_type = kCondemned; + current_type = RunMarker::Condemned; has_current = true; return; } /// A non-sentinel key must carry a surviving edge and nothing else. - if (v != kEdgeActive) + if (v != RunMarker::Edge) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: sentinel row type 0x{:02x} at a non-sentinel key", static_cast(v)); current_key = k; - current_type = kEdgeActive; + current_type = RunMarker::Edge; has_current = true; return; } @@ -151,7 +148,7 @@ class PriorEdgeCursor size_t seg_idx = 0; std::optional reader; String current_key; - char current_type = kEdgeActive; + RunMarker current_type = RunMarker::Edge; CondemnedRow current_condemned; bool has_current = false; @@ -191,7 +188,7 @@ void assertValidSourceEdgeId(const UInt128 & source_id) String encodeCondemnedRow(const CondemnedRow & row) { String out; - out.push_back(kCondemned); + out.push_back(runMarkerByte(RunMarker::Condemned)); out.push_back(static_cast((row.delete_pending ? 1 : 0) | (row.marker_confirmed ? 2 : 0))); out.push_back(static_cast(row.token.type)); auto beU64 = [&](uint64_t v) { for (int i = 7; i >= 0; --i) out += static_cast((v >> (8 * i)) & 0xFF); }; @@ -209,7 +206,7 @@ CondemnedRow decodeCondemnedRow(std::string_view p) { /// [0]=0x02 [1]=flags [2]=token_type [3..10]=round [11..18]=size [19..20]=len [21..]=value constexpr size_t kFixed = 21; - if (p.size() < kFixed || p[0] != kCondemned) + if (p.size() < kFixed || runMarkerFromByte(p[0], "CAS condemned row") != RunMarker::Condemned) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS condemned row: malformed header"); CondemnedRow row; const uint8_t flags = static_cast(p[1]); @@ -248,19 +245,16 @@ bool SourceEdgeRunView::next(String & key, String & payload) key = SourceEdgeKeyCodec::key(rec.ref, rec.source_id); switch (rec.marker) { - case kEdgeActive: - case kZeroMarker: - payload = String(1, rec.marker); + case RunMarker::Edge: + case RunMarker::Zero: + payload = String(1, runMarkerByte(rec.marker)); break; - case kCondemned: + case RunMarker::Condemned: payload = encodeCondemnedRow(CondemnedRow{.delete_pending = rec.delete_pending, .token = rec.token, .size = rec.size, .condemn_round = rec.condemn_round, .marker_confirmed = rec.marker_confirmed}); break; - default: - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS source-edge run: unknown row marker 0x{:02x}", static_cast(rec.marker)); } return true; } @@ -394,7 +388,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); // sorted NDJSON; byte-deterministic for write-once adoption - // Streaming two-cursor merge over the prior run (surviving edges AND retired kCondemned + // Streaming two-cursor merge over the prior run (surviving edges AND retired RunMarker::Condemned // sentinel rows at the zero source id) and this round's edge deltas (by (blob_hash, source_id)). All // rows for one blob are adjacent in both inputs; the sentinel key (source_id 0) sorts first. We resolve // final presence per edge locally (idempotent: prior present + activate => present; any remove => @@ -572,23 +566,23 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, } } - /// Emit at most one sentinel row per blob: the `kCondemned` row when the + /// Emit at most one sentinel row per blob: the `RunMarker::Condemned` row when the /// blob is condemned/carried/graduated this pass (still_retired grew for it), else a per-generation - /// `kZeroMarker` when it transitioned to zero this pass but was not condemned (redelete-dropped or + /// `RunMarker::Zero` when it transitioned to zero this pass but was not condemned (redelete-dropped or /// absent-at-condemn). A blob with surviving edges (cur_edges > 0) emits neither — its edge rows /// were appended inline, and a condemned/zeroed blob has NO surviving edges, so appending the /// sentinel now (its key sorts first for the blob, and no edge rows precede it) keeps the run - /// sorted. `still_retired` therefore mirrors exactly the emitted `kCondemned` rows, in order. + /// sorted. `still_retired` therefore mirrors exactly the emitted `RunMarker::Condemned` rows, in order. if (rmr.still_retired.size() > retired_before) { const RetiredEntry & e = rmr.still_retired.back(); - writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = kCondemned, + writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = RunMarker::Condemned, .delete_pending = e.delete_pending, .token = e.token, .size = e.size, .condemn_round = e.condemn_round, .marker_confirmed = e.marker_confirmed}); } else if (cur_edges == 0 && cur_touched) - writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = kZeroMarker}); + writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = RunMarker::Zero}); }; auto openBlobIfNeeded = [&](const BlobRef & b) { @@ -622,9 +616,9 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, openBlobIfNeeded(blob_ref); /// A retired sentinel row from the prior run: stash it for close-out settlement. It is not an edge - /// and NEVER a touch — a carried kCondemned row must not force a zero-marker or a peek_head HEAD + /// and NEVER a touch — a carried RunMarker::Condemned row must not force a zero-marker or a peek_head HEAD /// and never a touch. Deltas never key the zero source id, so no delta merges at this key. - if (from_prior && cursor.rowType() == kCondemned) + if (from_prior && cursor.rowType() == RunMarker::Condemned) { cur_condemned = cursor.condemnedRow(); cursor.advance(); @@ -673,7 +667,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, if (present) { - writer.append(SourceEdgeRecord{.ref = blob_ref, .source_id = source_id, .marker = kEdgeActive}); + writer.append(SourceEdgeRecord{.ref = blob_ref, .source_id = source_id, .marker = RunMarker::Edge}); ++cur_edges; } } @@ -689,7 +683,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, const String run_key = layout.blobTargetRunKey(new_generation, attempt, shard, 0); putDeterministicArtifact(backend, run_key, run_bytes); out_runs.push_back(RunRef{.key = run_key, .checksum = run_checksum, - .shard = shard, .generation = new_generation}); + .shard = shard, .key_generation = new_generation}); } std::vector zeroInDegree(Backend & backend, const std::vector & runs) @@ -700,12 +694,12 @@ std::vector zeroInDegree(Backend & backend, const std::vector redelete if d = 0 (the caller executes the exact-token @@ -342,9 +342,9 @@ struct GcRoundWorkBudget /// `confirm_condemned_marker` below): an unconfirmed /// entry is carried unchanged instead; /// d = 0 otherwise -> still_retired, carried byte-unchanged. -/// A carried `kCondemned` row is SETTLEMENT-ONLY: it never sets the blob's `cur_touched` bit, so a +/// A carried `RunMarker::Condemned` row is SETTLEMENT-ONLY: it never sets the blob's `cur_touched` bit, so a /// generation that only carries the row emits no zero-marker and pays no `peek_head` HEAD. The surviving -/// `still_retired` entries are re-emitted as `kCondemned` sentinel rows into the OUTPUT run (one sentinel +/// `still_retired` entries are re-emitted as `RunMarker::Condemned` sentinel rows into the OUTPUT run (one sentinel /// per blob, emitted before the blob's edges since the sentinel key sorts first), so the next generation /// reads them back — `still_retired` mirrors exactly those rows, in the same order. /// When the pass is clamped on any shard, landed-before-cut events may remain unfolded behind the clamp, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 358f619d5b34..f9c9ca6f72a5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -901,7 +901,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al } /// Retired-in-snapshot — there is NO separate retired-list object to publish anymore. The - /// round's surviving condemned entries were already sealed as `kCondemned` rows inside the fold's + /// round's surviving condemned entries were already sealed as `RunMarker::Condemned` rows inside the fold's /// `blob_target_runs` (durable before this CAS, via `putDeterministicArtifact`), and the per-shard /// `condemned_summary` the seal carries makes the next round's graduation/carry decisions zero-I/O. @@ -923,13 +923,13 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// current shard's run back at an older generation's key). Retention must never reclaim these. std::set referenced_generations; for (const RunRef & r : folded.fold_seal.blob_target_runs) - referenced_generations.insert(r.generation); + referenced_generations.insert(r.key_generation); /// ALSO protect every generation the PARENT (currently-adopted, pre-fold) seal references /// (`parent_seal_runs`, captured above): this prune runs BEFORE the round's own gc/state CAS below, so /// a losing leader must not destroy what the winning leader's already-adopted seal still points at — /// pre-CAS destructive actions may only rely on PREVIOUSLY PUBLISHED state (triage #5). for (const RunRef & r : parent_seal_runs) - referenced_generations.insert(r.generation); + referenced_generations.insert(r.key_generation); /// Retention floor uses THIS round's (post-fold) `generation`, so `gc_snapshot_generations_to_keep` /// keeps exactly that many generations back from the current one. If this round's `gc/state` CAS /// then LOSES, the prune reclaimed one generation deeper than the durably-adopted generation would @@ -980,7 +980,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al uint64_t objects_reclaimed = 0; std::set new_referenced_generations; for (const RunRef & r : folded.fold_seal.blob_target_runs) - new_referenced_generations.insert(r.generation); + new_referenced_generations.insert(r.key_generation); std::set handed_off; /// dedupe: multiple parent refs can share one generation /// GATED like every other destructive site, and it is also the FIRST destructive site of the @@ -1003,11 +1003,11 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al for (const RunRef & old_ref : handoff_candidates) { /// Only generations the wholesale prune already passed AND that no live ref still pins. - if (old_ref.generation > state.snap_pruned_through) + if (old_ref.key_generation > state.snap_pruned_through) continue; /// not yet pruned-through: the normal prune will reclaim it when it ages out - if (new_referenced_generations.contains(old_ref.generation)) + if (new_referenced_generations.contains(old_ref.key_generation)) continue; /// still referenced by a (possibly different-shard) live ref: keep it - if (!handed_off.insert(old_ref.generation).second) + if (!handed_off.insert(old_ref.key_generation).second) continue; /// already reclaimed this round via another shard's ref /// `bounded_remaining` draws from the hand-off's OWN reserve, never `UINT64_MAX` and never /// `pruneSupersededGenerations`' shared remainder: this hand-off is a ONE-SHOT event (see the @@ -1021,13 +1021,13 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al if (remaining == 0) break; const uint64_t reclaimed = deletePrefixWholesale( - backend, layout.gcGenPrefix(old_ref.generation), remaining); + backend, layout.gcGenPrefix(old_ref.key_generation), remaining); round_work_budget.handoff_prefix_wholesale_objects_used += reclaimed; objects_reclaimed += reclaimed; LOG_TRACE(logger, "CAS GC hand-off: generation {} moved out of the live seal below the retention cursor " "({} objects) — post-CAS wholesale reclaim (the prune had skipped it while referenced)", - old_ref.generation, reclaimed); + old_ref.key_generation, reclaimed); } t.metric("generations_reclaimed", handed_off.size()); t.metric("objects_reclaimed", objects_reclaimed); @@ -1665,7 +1665,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & result.fold_seal.ref_lives = walk_plan.successorFoldStates(); /// Retired-in-snapshot: the prior generation's condemned entries RIDE the source-edge run as - /// `kCondemned` sentinel rows, so the round no longer reads any separate retired-list object — + /// `RunMarker::Condemned` sentinel rows, so the round no longer reads any separate retired-list object — /// the parent seal's `blob_target_runs` ARE the retired input. The per-gc-shard `condemned_summary` /// the seal carries below is distilled from the `still_retired` rows each shard re-emits, making the /// next round's `graduationDue` / pure-carry decisions zero-I/O. @@ -2821,7 +2821,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// is deterministic (same refs for the same inputs), so seal determinism / crash-replay adoption hold. /// An empty delta with a NON-EMPTY retired list still runs the merge: settlement must happen every /// pass (carried/graduated/redeleted entries), and that pass reads the run to recompute in-degrees. - /// Distill one shard's `condemned_summary` entry from the `kCondemned` rows it re-emitted this pass + /// Distill one shard's `condemned_summary` entry from the `RunMarker::Condemned` rows it re-emitted this pass /// (`still_retired` mirrors those rows exactly). Folding shards call this; it makes the next /// round's `graduationDue` and pure-carry decisions read only the seal, never a run. auto summarize = [](const std::vector & still) -> CondemnedSummary @@ -3021,7 +3021,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & else { /// Either a real delta or a non-empty retired input: run the merge (empty deltas still settle - /// the kCondemned rows riding the parent run). The prior runs are the parent seal's shard-0 refs. + /// the RunMarker::Condemned rows riding the parent run). The prior runs are the parent seal's shard-0 refs. foldDeltasIntoGeneration(backend, layout, priorRunsFor(0), new_generation, attempt, /*shard*/0, std::move(deltas), result.fold_seal.blob_target_runs, @@ -3606,7 +3606,7 @@ bool Gc::graduationDue(const GcState & state, uint64_t current_round) { /// Retired-in-snapshot: the graduation signal is read from the adopted fold seal's per-shard /// `condemned_summary` — ZERO backend I/O beyond the single seal read. A summary distilled from this - /// generation's `kCondemned` rows says, per shard, how many entries are `delete_pending` (a graduation + /// generation's `RunMarker::Condemned` rows says, per shard, how many entries are `delete_pending` (a graduation /// is already published) and the oldest non-pending condemn round (one crosses the floor once /// `condemn_round < current_round`). if (state.snap_generation == 0) @@ -3980,7 +3980,7 @@ RebuildReport Gc::rebuildBaseline(bool force) std::vector attempt_of(gc_shards, 0); /// The fold is EDGE-ONLY here: a rebuild condemns nothing (spec §7, and the deletion below), so no /// condemn round is stamped and no head source is supplied. `current_round` 0 graduates nothing and - /// `condemn_round` 0 with an empty `head_blob` mints no `kCondemned` row -- this call is + /// `condemn_round` 0 with an empty `head_blob` mints no `RunMarker::Condemned` row -- this call is /// `foldDeltasIntoGeneration`'s pure edge form. auto flush_shard = [&](uint64_t shard) { @@ -4315,7 +4315,7 @@ std::vector Gc::previewDeletes() out.push_back(std::move(e)); } - /// Retired-in-snapshot: stream the SAME adopted seal runs and emit every `kCondemned` + /// Retired-in-snapshot: stream the SAME adopted seal runs and emit every `RunMarker::Condemned` /// sentinel row. The stored token IS the authority — NO HEAD here (a HEAD would defeat the point /// and cost I/O). `delete_pending` rows are deleted next fold; the rest await graduation. Preview /// stays WRITE-FREE (`openSourceEdgeRun` is a pure reader). Output is a superset of the above. @@ -4326,7 +4326,7 @@ std::vector Gc::previewDeletes() String payload; while (reader.next(key, payload)) { - if (payload.empty() || payload[0] != kCondemned) + if (payload.empty() || runMarkerFromByte(payload[0], "CAS source-edge run") != RunMarker::Condemned) continue; BlobRef ref; UInt128 source_id; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp index ae34e530bb1c..90a695349916 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp @@ -1,20 +1,14 @@ #include +#include namespace DB::Cas { +static_assert(casEnumTableCoversEnum()); + std::string_view blobHashAlgoName(BlobHashAlgo algo) { - switch (algo) - { - case BlobHashAlgo::CityHash128: - return "ch128"; - case BlobHashAlgo::XXH3_128: - return "xxh3"; - case BlobHashAlgo::Sha256: - return "sha256"; - } - throw Exception(ErrorCodes::BAD_ARGUMENTS, "blobHashAlgoName: unknown BlobHashAlgo {}", static_cast(algo)); + return kBlobHashAlgoWords.toWord(algo, "blobHashAlgoName"); } uint64_t blobHashLenFor(BlobHashAlgo algo) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h index cc557fd753e1..54360403b8d7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -42,8 +43,16 @@ enum class BlobHashAlgo : uint8_t Sha256 = 3, }; +/// The `BlobHashAlgo` wire vocabulary (also the blob PATH SEGMENT, e.g. +/// `/blobs///`); coverage is proven in `CasBlobDigest.cpp`. +inline constexpr EnumWireTable kBlobHashAlgoWords{{{ + {BlobHashAlgo::CityHash128, "ch128"}, + {BlobHashAlgo::XXH3_128, "xxh3"}, + {BlobHashAlgo::Sha256, "sha256"}, +}}}; + /// The blob PATH SEGMENT for `algo`, e.g. `/blobs///`: `"ch128"` | `"xxh3"` | -/// `"sha256"`. Throws `BAD_ARGUMENTS` for an out-of-range enum value. +/// `"sha256"`. Throws `LOGICAL_ERROR` for an out-of-range enum value. std::string_view blobHashAlgoName(BlobHashAlgo algo); /// Returns the digest byte width for `algo`: 16 for `CityHash128` and `XXH3_128`, or 32 for diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h new file mode 100644 index 000000000000..31d76eee0bf8 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h @@ -0,0 +1,73 @@ +#pragma once + +#include +#include + +#include +#include + +namespace DB::ErrorCodes +{ + extern const int CORRUPTED_DATA; + extern const int LOGICAL_ERROR; +} + +namespace DB::Cas +{ + +/// One persisted enum <-> wire-word vocabulary: the single carrier the encoder, the decoder, the +/// introspection renderer, and the tests all read. Every persisted enum is dense, so `toWord` is a +/// direct indexed lookup; `fromWord` is a linear pass over a handful of words. Coverage is proven +/// at each table's definition site by `casEnumTableCoversEnum` (CasEnumWireTableAsserts.h — .cpp +/// and tests only) together with the `denseAndOrdered`/`wordsUnique` predicates below. +template +struct EnumWireTable +{ + struct Entry + { + Enum value; + std::string_view word; + }; + + std::array entries; + + /// An empty table would make `denseAndOrdered`/`wordsUnique` vacuously true and `toWord`'s + /// index arithmetic read past the array — no wire vocabulary is empty, so reject at compile time. + static_assert(N > 0, "EnumWireTable must hold at least one entry"); + + constexpr bool denseAndOrdered() const + { + for (size_t i = 0; i < N; ++i) + if (static_cast(entries[i].value) != static_cast(entries[0].value) + i) + return false; + return true; + } + + constexpr bool wordsUnique() const + { + for (size_t i = 0; i < N; ++i) + for (size_t j = i + 1; j < N; ++j) + if (entries[i].word == entries[j].word) + return false; + return true; + } + + std::string_view toWord(Enum value, std::string_view what) const + { + const uint64_t index = static_cast(value) - static_cast(entries.front().value); + if (index >= entries.size() || entries[index].value != value) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "{}: value {} is outside the wire vocabulary", what, static_cast(value)); + return entries[index].word; + } + + Enum fromWord(std::string_view word, std::string_view what) const + { + for (const auto & entry : entries) + if (entry.word == word) + return entry.value; + throw Exception(ErrorCodes::CORRUPTED_DATA, "{}: unknown word '{}'", what, word); + } +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h new file mode 100644 index 000000000000..e59e13fcef6b --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h @@ -0,0 +1,37 @@ +#pragma once + +/// Compile-time coverage proof for EnumWireTable: SET EQUALITY with the enum's declared values. +/// Size-plus-uniqueness is not enough (an invalid casted value satisfies both while an enumerator +/// goes missing). This header pulls in magic_enum and therefore MUST be included only from .cpp +/// files and tests, never from another header. + +#include + +#include + +namespace DB::Cas +{ + +template +consteval bool casEnumTableCoversEnum() +{ + /// One assert per table carries all three obligations: a table author cannot forget density + /// or word uniqueness, because coverage subsumes them. + if (!Table.denseAndOrdered() || !Table.wordsUnique()) + return false; + constexpr auto declared = magic_enum::enum_values(); + if (declared.size() != Table.entries.size()) + return false; + for (size_t i = 0; i < declared.size(); ++i) + { + bool found = false; + for (const auto & entry : Table.entries) + if (entry.value == declared[i]) + found = true; + if (!found) + return false; + } + return true; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp index 7848ae591f27..33af5b0ec1e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp @@ -800,7 +800,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// The NON-SENTINEL source edges the snapshot still holds on each unreferenced blob, collected in /// `detail` mode only. `in_run_hashes` alone answers "does GC still see this blob at all"; the /// stale-edge cross-check below needs the edge IDENTITIES so it can ask whether their source - /// manifests still exist. Sentinel rows (`source_id == 0` — `kZeroMarker`/`kCondemned`) are not + /// manifests still exist. Sentinel rows (`source_id == 0` — `RunMarker::Zero`/`RunMarker::Condemned`) are not /// edges and are excluded. std::unordered_map, BlobRefHash> unref_edge_sources; bool have_gc_state = false; @@ -821,8 +821,8 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// The adopted fold seal names the snapshot runs; resolution is by ref, never by key /// construction. Every row whose hash is in our candidate set marks "known to GC" — /// edges still counted (drop unfolded), an explicit zero-marker mid-pipeline, or a - /// `kCondemned` sentinel row that carries the condemned state (retired-in-snapshot): - /// the `kCondemned` rows feed `retired_by_hash` (the `PendingGc` classification) in the + /// `RunMarker::Condemned` sentinel row that carries the condemned state (retired-in-snapshot): + /// the `RunMarker::Condemned` rows feed `retired_by_hash` (the `PendingGc` classification) in the /// SAME pass, replacing the removed `retired_refs`/`decodeRetiredSet` loop. /// /// These sets are keyed by the full `BlobRef`, not a narrowed digest. The run's own @@ -852,7 +852,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co in_run_hashes.insert(ref); if (detail && source_id != UInt128{0}) unref_edge_sources[ref].push_back(source_id); - if (!payload.empty() && payload[0] == kCondemned) + if (!payload.empty() && runMarkerFromByte(payload[0], "CAS source-edge run") == RunMarker::Condemned) { const CondemnedRow row = decodeCondemnedRow(payload); RetiredEntry e; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index 1428be282f30..4793eadd4e12 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -8,7 +9,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -126,20 +129,10 @@ String renderRefTxnIdObj(const RefTxnId & id) .str(); } -String refOwnerKindName(RefOwnerKind k) -{ - switch (k) - { - case RefOwnerKind::Committed: return "Committed"; - case RefOwnerKind::Precommit: return "Precommit"; - } - return "Unknown"; -} - String renderRefOwnerBinding(const RefOwnerBinding & b) { return JsonObj() - .add("kind", jsonEscape(refOwnerKindName(b.kind))) + .add("kind", jsonEscape(refOwnerKindToWord(b.kind))) .add("ref_name", jsonEscape(b.ref_name)) .add("manifest_ref", renderManifestRef(b.manifest_ref)) .str(); @@ -196,23 +189,10 @@ String renderRefCkpt(const RootNamespace & ns, const RefCkpt & c) .str(); } -String refOpKindName(RefOpKind k) -{ - switch (k) - { - case RefOpKind::NamespaceBirth: return "NamespaceBirth"; - case RefOpKind::OwnerTransition: return "OwnerTransition"; - case RefOpKind::SetPublishedAt: return "SetPublishedAt"; - case RefOpKind::RemoveNamespace: return "RemoveNamespace"; - case RefOpKind::EpochSeal: return "EpochSeal"; - } - return "Unknown"; -} - String renderRefOp(const RefOp & op) { return JsonObj() - .add("kind", jsonEscape(refOpKindName(op.kind))) + .add("kind", jsonEscape(refOpKindToWireWord(op.kind))) .add("old_binding", op.old_binding ? renderRefOwnerBinding(*op.old_binding) : "null") .add("new_binding", op.new_binding ? renderRefOwnerBinding(*op.new_binding) : "null") .add("ref_name", jsonEscape(op.ref_name)) @@ -237,16 +217,6 @@ String renderRefLogTxn(const RefLogTxn & t) .str(); } -String placementName(EntryPlacement p) -{ - switch (p) - { - case EntryPlacement::Inline: return "Inline"; - case EntryPlacement::Blob: return "Blob"; - } - return "Unknown"; -} - /// `inline_bytes` renders as its LENGTH only, not its content — an inline file's bytes are payload /// data, not part-manifest identity, and may be arbitrarily large / non-UTF8. String renderManifestEntry(const ManifestEntry & e) @@ -256,7 +226,7 @@ String renderManifestEntry(const ManifestEntry & e) /// digest widths, and each entry's own `ref.algo` determines its width. return JsonObj() .add("path", jsonEscape(e.path)) - .add("placement", jsonEscape(placementName(e.placement))) + .add("placement", jsonEscape(entryPlacementToWireWord(e.placement))) .add("blob", jsonEscape(blobIdOf(e.ref))) .add("blob_size", jsonUInt(e.blob_size)) .add("inline_bytes_size", jsonUInt(e.inline_bytes.size())) @@ -315,43 +285,23 @@ String renderGcState(const GcState & s) .str(); } -String tokenTypeName(TokenType t) -{ - switch (t) - { - case TokenType::ETag: return "ETag"; - case TokenType::Generation: return "Generation"; - case TokenType::Emulated: return "Emulated"; - } - return "Unknown"; -} - /// `Token::value` is an opaque backend-native string (e.g. an S3 ETag) — NOT a 128-bit hash — so it /// renders verbatim (escaped), not hex-converted; `type` names which backend family minted it. String renderToken(const Token & t) { return JsonObj() .add("value", jsonEscape(t.value)) - .add("type", jsonEscape(tokenTypeName(t.type))) + .add("type", jsonEscape(tokenTypeToWord(t.type))) .str(); } -String objectKindName(ObjectKind k) -{ - switch (k) - { - case ObjectKind::Blob: return "Blob"; - } - return "Unknown"; -} - String renderRunRef(const RunRef & r) { return JsonObj() .add("key", jsonEscape(r.key)) .add("checksum", jsonHex(r.checksum)) .add("shard", jsonUInt(r.shard)) - .add("generation", jsonUInt(r.generation)) + .add("generation", jsonUInt(r.key_generation)) .str(); } @@ -379,7 +329,7 @@ String renderFoldSeal(const CasFoldSeal & seal) for (const auto & r : seal.blob_target_runs) blob_target_runs.push_back(renderRunRef(r)); - /// A fold seal carries per-GC-shard totals for `kCondemned` rows in its source runs. Render the + /// A fold seal carries per-GC-shard totals for `RunMarker::Condemned` rows in its source runs. Render the /// summary from the seal itself; the older separate retired-reference object is no longer part /// of the current layout. JsonObj condemned_summary; @@ -399,40 +349,16 @@ String renderFoldSeal(const CasFoldSeal & seal) .str(); } -String provenanceOpName(ProvenanceOp op) -{ - switch (op) - { - case ProvenanceOp::Other: return "Other"; - case ProvenanceOp::Insert: return "Insert"; - case ProvenanceOp::Merge: return "Merge"; - case ProvenanceOp::Mutation: return "Mutation"; - case ProvenanceOp::Attach: return "Attach"; - case ProvenanceOp::Repack: return "Repack"; - } - return "Unknown"; -} - String renderProvenance(const Provenance & p) { return JsonObj() .add("created_at_ms", jsonUInt(p.created_at_ms)) .add("creator_server_id", jsonHex(p.creator_server_id)) .add("ch_version", jsonUInt(p.ch_version)) - .add("op", jsonEscape(provenanceOpName(p.op))) + .add("op", jsonEscape(provenanceOpToWireWord(p.op))) .str(); } -String metaStateName(MetaState s) -{ - switch (s) - { - case MetaState::Clean: return "clean"; - case MetaState::Condemned: return "condemned"; - } - return "unknown"; -} - /// The per-hash `.meta` descriptor is the blob body's sibling and records its freshness state /// (`Clean` or `Condemned`), not its payload. It is rendered separately from `renderEnvelopeHeader`: /// the body remains an enveloped object, while the descriptor has its own format. @@ -441,7 +367,7 @@ String renderBlobMeta(const BlobMeta & m) return JsonObj() .add("object", jsonEscape("blob_meta")) .add("version", jsonUInt(m.version)) - .add("state", jsonEscape(metaStateName(m.state))) + .add("state", jsonEscape(metaStateToWireWord(m.state))) .add("condemn_round", jsonUInt(m.condemn_round)) .add("size", jsonUInt(m.size)) .str(); @@ -450,7 +376,7 @@ String renderBlobMeta(const BlobMeta & m) String renderEnvelopeHeader(const EnvelopeHeader & h) { return JsonObj() - .add("kind", jsonEscape(objectKindName(h.kind))) + .add("kind", jsonEscape(objectKindToWord(h.kind))) /// The blob identity is carried by the object key, so the envelope keeps only the provenance /// fields needed for forensics (`ch` and `bld`) together with its compatibility version. .add("compatibility_version", jsonUInt(h.compatibility_version)) @@ -463,17 +389,11 @@ String renderEnvelopeHeader(const EnvelopeHeader & h) } /// The word vocabulary a row's marker byte renders as, matching the `cas_run` NDJSON's own `"m"` field -/// words (`CasRecordStreamFormat.cpp`'s private `markerToWord`) so cas-inspect speaks the same vocabulary +/// words (`runMarkerToWireWord`) so cas-inspect speaks the same vocabulary /// as the on-disk format rather than inventing a second one. -String sourceEdgeRowKindName(char marker) +String sourceEdgeRowKindName(RunMarker marker) { - switch (marker) - { - case kEdgeActive: return "edge"; - case kZeroMarker: return "zero"; - case kCondemned: return "condemned"; - default: return "unknown"; - } + return String(runMarkerToWireWord(marker)); } String renderCondemnedRow(const CondemnedRow & r) @@ -514,7 +434,7 @@ String renderBlobTargetRun(const ParsedBlobTargetRunKey & parsed, std::string_vi if (payload.empty()) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "cas-inspect: source-edge run row for blob {} has an empty payload", blobIdOf(ref)); - const char marker = payload[0]; + const RunMarker marker = runMarkerFromByte(payload[0], "cas-inspect: source-edge run row"); distinct_blobs.insert(ref); JsonObj row; @@ -527,20 +447,16 @@ String renderBlobTargetRun(const ParsedBlobTargetRunKey & parsed, std::string_vi switch (marker) { - case kEdgeActive: + case RunMarker::Edge: ++edge_count; break; - case kZeroMarker: + case RunMarker::Zero: ++zero_marker_count; break; - case kCondemned: + case RunMarker::Condemned: ++condemned_count; row.add("condemned", renderCondemnedRow(decodeCondemnedRow(payload))); // CORRUPTED_DATA on malformed (fail-closed) break; - default: - throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, - "cas-inspect: source-edge run row for blob {} has an unknown marker 0x{:02x}", - blobIdOf(ref), static_cast(marker)); } rows.push_back(row.str()); } diff --git a/src/Disks/tests/cas_format_test_battery.h b/src/Disks/tests/cas_format_test_battery.h index 173bfcc5c4df..361947968f3c 100644 --- a/src/Disks/tests/cas_format_test_battery.h +++ b/src/Disks/tests/cas_format_test_battery.h @@ -4,6 +4,7 @@ #include #include #include +#include namespace DB::ErrorCodes { @@ -53,6 +54,23 @@ void expectCode(int code, F && f, const String & context) } } +namespace DB::Cas::tests +{ +inline std::set & batteryCoveredIds() +{ + static std::set ids; + return ids; +} + +struct BatteryCoverageRegistrar +{ + explicit BatteryCoverageRegistrar(FormatId id) { batteryCoveredIds().insert(id); } +}; +} + +#define CAS_BATTERY_COVERS(format_id) \ + static const DB::Cas::tests::BatteryCoverageRegistrar battery_covers_##format_id{DB::Cas::FormatId::format_id} + inline void runFormatBattery(const FormatBatteryCase & c) { using namespace DB::Cas; diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index e7b5f6221c18..441ea58fc558 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -475,9 +475,9 @@ inline String encodeMinimalGcState(uint64_t round) /// Inject condemned bookkeeping + gc/state directly (bypassing a real GC round) so a test can seed the /// GC ledger's condemned state at an arbitrary round. Retired-in-snapshot: the condemned entries are -/// seeded the way a real round leaves them — as `kCondemned` sentinel rows inside an adopted fold seal's +/// seeded the way a real round leaves them — as `RunMarker::Condemned` sentinel rows inside an adopted fold seal's /// shard run (there is no separate retired-list object). A synthetic +edge/-edge pair nets each blob to -/// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `kCondemned` row. +/// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `RunMarker::Condemned` row. /// Also sets {round} on gc/state. Entries carry a `condemn_round` (default 0 → uses `round`); callers /// pass fresh (non-pending) condemns. An empty `entries` set just advances {round}. inline void injectRetire( @@ -621,7 +621,7 @@ inline bool runRoundsUntilAbsent( /// The CURRENT condemned entries for `shard`, read from the adopted fold seal's `blob_target_runs` /// (retired-in-snapshot T4): the round no longer writes a separate retired-list object — condemned -/// entries RIDE the source-edge run as `kCondemned` sentinel rows at the zero-sentinel key. This reads +/// entries RIDE the source-edge run as `RunMarker::Condemned` sentinel rows at the zero-sentinel key. This reads /// the seal at (snap_generation, snap_attempt), opens every run for `shard`, and reconstructs the /// `RetiredEntry` shape (hash from the run key, the rest from the decoded `CondemnedRow`). Empty when /// gc/state / the seal / the runs are absent. Used by ack-floor tests to assert pending/condemn state. @@ -649,7 +649,7 @@ inline std::vector currentRetiredSet( String p; while (r.next(k, p)) { - if (p.empty() || p[0] != DB::Cas::kCondemned) + if (p.empty() || DB::Cas::runMarkerFromByte(p[0], "CAS test source-edge run") != DB::Cas::RunMarker::Condemned) continue; DB::Cas::BlobRef ref; DB::UInt128 source_id{}; @@ -668,7 +668,7 @@ inline std::vector currentRetiredSet( return out; } -/// True iff ANY gc-shard's adopted-seal run still holds a `kCondemned` row — the ack-floor deletion +/// True iff ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row — the ack-floor deletion /// pipeline is in flight while this is true (retired-in-snapshot T4 replacement for the old /// "iterate gc/state.retired_refs" probe). `gc_shards` is read from gc/state when 0 is passed. inline bool anyCondemnedInSeal( @@ -861,7 +861,7 @@ inline std::vector runsForShard( } } -/// Stream the sealed in-degree run segments `runs` and count the active source edges (`kEdgeActive` +/// Stream the sealed in-degree run segments `runs` and count the active source edges (`RunMarker::Edge` /// rows) for `ref`. Test-side replacement for the deleted per-blob point query `inDegreeInGeneration` /// (codecs-v3 phase 5: a `cas_run` is a sequential NDJSON stream with no random access, so a blob's /// in-degree is recomputed by a full stream-and-count rather than a seek). A condemned / zero-marker @@ -877,7 +877,7 @@ inline int64_t inDegreeInRuns( String p; while (r.next(k, p)) { - if (p.empty() || p[0] != DB::Cas::kEdgeActive) + if (p.empty() || DB::Cas::runMarkerFromByte(p[0], "CAS test source-edge run") != DB::Cas::RunMarker::Edge) continue; DB::Cas::BlobRef row_ref; DB::UInt128 source_id{}; diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 1119a5be0a64..495ed57f0325 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -154,6 +154,8 @@ TEST(CASBlobEnvelopeFormat, RefEscaperAlphabetPinned) << "escaper alphabet drifted: '/' must be verbatim, quote/backslash escaped, control -> \\uXXXX"; } +CAS_BATTERY_COVERS(Blob); + TEST(CASFormatBattery, BlobEnvelope) { /// The golden is CONSTRUCTED from the hand-pinned json literal (same one FixedLengthAndPadZone diff --git a/src/Disks/tests/gtest_cas_blob_indegree.cpp b/src/Disks/tests/gtest_cas_blob_indegree.cpp index e47b7351f063..a340fde248ac 100644 --- a/src/Disks/tests/gtest_cas_blob_indegree.cpp +++ b/src/Disks/tests/gtest_cas_blob_indegree.cpp @@ -144,27 +144,27 @@ TEST(CASBlobInDegree, FoldDeltaDivergentBytesThrowsCorrupted) /// ==== two-cursor settlement merge (retired-in-snapshot T3, spec §2.1/§3) ==== /// -/// The retired input is no longer a separate `prior_retired` vector — the prior generation's `kCondemned` +/// The retired input is no longer a separate `prior_retired` vector — the prior generation's `RunMarker::Condemned` /// rows RIDE the source-edge run at the zero-sentinel key. These helpers build such a prior run directly /// (via the sorted-NDJSON `SourceEdgeRunWriter`, codecs-v3 phase 5) and decode a run for assertions. namespace { -/// A `kCondemned` sentinel record for `h` at the zero source_id, carrying the condemned incarnation. +/// A `RunMarker::Condemned` sentinel record for `h` at the zero source_id, carrying the condemned incarnation. SourceEdgeRecord condemnedRec(UInt128 h, const CondemnedRow & row) { return SourceEdgeRecord{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h)}, - .source_id = UInt128{0}, .marker = kCondemned, + .source_id = UInt128{0}, .marker = RunMarker::Condemned, .delete_pending = row.delete_pending, .token = row.token, .size = row.size, .condemn_round = row.condemn_round}; } -/// An active-edge record (`kEdgeActive`) for `h` at source `sid`. +/// An active-edge record (`RunMarker::Edge`) for `h` at source `sid`. SourceEdgeRecord edgeRec(UInt128 h, UInt128 sid) { return SourceEdgeRecord{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h)}, - .source_id = sid, .marker = kEdgeActive}; + .source_id = sid, .marker = RunMarker::Edge}; } /// head_blob / peek_head stub: present with a fixed token/size. @@ -189,7 +189,7 @@ CondemnedRow condemnedRowFor(uint64_t condemn_round, const String & tok = "t", .size = size, .condemn_round = condemn_round}; } -/// Build a source-edge run (`kSourceEdgeKeySchema128`) carrying the given `kCondemned` sentinel rows +/// Build a source-edge run (`kSourceEdgeKeySchema128`) carrying the given `RunMarker::Condemned` sentinel rows /// and surviving edges, write it under `blobTargetRunKey(gen, attempt, shard, 0)`, and return its /// `RunRef`. Rows are emitted in (blob_hash, source_id) order (sentinels at source_id 0 sort first /// per blob). @@ -224,7 +224,7 @@ RunRef writeSourceEdgeRun(InMemoryBackend & backend, const Layout & layout, const String bytes = out.str(); const String key = layout.blobTargetRunKey(gen, attempt, shard, 0); backend.putIfAbsent(key, bytes); - return RunRef{.key = key, .checksum = sourceEdgeRunChecksum(bytes), .shard = shard, .generation = gen}; + return RunRef{.key = key, .checksum = sourceEdgeRunChecksum(bytes), .shard = shard, .key_generation = gen}; } struct DecodedRun @@ -251,11 +251,11 @@ DecodedRun decodeRun(InMemoryBackend & backend, const RunRef & run) EXPECT_FALSE(p.empty()); if (p.empty()) continue; - if (p[0] == kCondemned) + if (runMarkerFromByte(p[0], "CAS test source-edge run") == RunMarker::Condemned) d.condemned.emplace_back(bh, decodeCondemnedRow(p)); - else if (p[0] == kZeroMarker) + else if (runMarkerFromByte(p[0], "CAS test source-edge run") == RunMarker::Zero) d.zero_markers.push_back(bh); - else if (p[0] == kEdgeActive) + else if (runMarkerFromByte(p[0], "CAS test source-edge run") == RunMarker::Edge) d.edges.emplace_back(bh, sid); else ADD_FAILURE() << "unknown run row type"; @@ -304,7 +304,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) InMemoryBackend backend; Layout layout{"pool"}; - /// Gen 1's run holds one unrelated surviving edge (b9) plus the carried kCondemned rows for A=b1 + /// Gen 1's run holds one unrelated surviving edge (b9) plus the carried RunMarker::Condemned rows for A=b1 /// (condemned round 2) and B=b2 (round 3); neither A nor B has any edge (in-degree 0 by definition). /// current_round = 3: strictly-below graduates, at-the-current-round stays. const RunRef gen1 = writeSourceEdgeRun(backend, layout, /*gen*/1, 0, 0, @@ -329,7 +329,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) EXPECT_TRUE(rmr.spared.empty()); EXPECT_TRUE(rmr.redelete.empty()); - /// still_retired mirrors exactly the kCondemned rows written into the output run, in order. + /// still_retired mirrors exactly the RunMarker::Condemned rows written into the output run, in order. const DecodedRun out = decodeRun(backend, runs2[0]); ASSERT_EQ(out.condemned.size(), 2u); EXPECT_EQ(out.condemned[0].first, b(1)); @@ -415,7 +415,7 @@ TEST(CASThreeCursorMerge, NewCandidateCondemned) EXPECT_TRUE(rmr.graduated.empty()); EXPECT_TRUE(rmr.spared.empty()); - /// The fresh condemn is emitted as a kCondemned row (not a zero marker) into the output run. + /// The fresh condemn is emitted as a RunMarker::Condemned row (not a zero marker) into the output run. const DecodedRun out = decodeRun(backend, runs2[0]); ASSERT_EQ(out.condemned.size(), 1u); EXPECT_EQ(out.condemned[0].first, b(3)); @@ -451,7 +451,7 @@ TEST(CASThreeCursorMerge, AbsentBlobNotCondemned) TEST(CASThreeCursorMerge, SnapshotEdgesUnperturbedByRetired) { - /// Retired-in-snapshot changes the byte-invariant: the retired machinery now WRITES kCondemned + /// Retired-in-snapshot changes the byte-invariant: the retired machinery now WRITES RunMarker::Condemned /// sentinel rows into the run, so a retired-engaged run is no longer byte-identical to a plain one. /// The preserved invariant (spec §2.1) is narrower: the retired machinery touches ONLY the sentinel /// namespace — the surviving EDGE rows are byte-identical to a plain fold of the same deltas. @@ -485,7 +485,7 @@ TEST(CASThreeCursorMerge, SnapshotEdgesUnperturbedByRetired) TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) { - /// Gen 1 condemns b (a real +edge/-edge net-to-zero with head_blob present) -> a kCondemned row. Gen 2 + /// Gen 1 condemns b (a real +edge/-edge net-to-zero with head_blob present) -> a RunMarker::Condemned row. Gen 2 /// has NO deltas at all: the carried row must (a) survive byte-identically, (b) emit no zero marker, /// (c) never call peek_head (a carried sentinel is not a touch). InMemoryBackend backend; @@ -502,7 +502,7 @@ TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) const DecodedRun g1 = decodeRun(backend, runs1[0]); ASSERT_EQ(g1.condemned.size(), 1u); EXPECT_EQ(g1.condemned[0].first, b(2)); - EXPECT_TRUE(g1.zero_markers.empty()); /// a condemned blob emits kCondemned, never a zero marker + EXPECT_TRUE(g1.zero_markers.empty()); /// a condemned blob emits RunMarker::Condemned, never a zero marker } /// Gen 2: empty deltas, current_round 1 (< 5 => b carries, does not graduate). peek_head must NOT fire. @@ -541,7 +541,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) out.finalize(); const String bytes = out.str(); const RunRef bad{.key = layout.blobTargetRunKey(1, 0, 0, 0), - .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .generation = 1}; + .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .key_generation = 1}; backend.putIfAbsent(bad.key, bytes); std::vector runs2; @@ -561,7 +561,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) out.finalize(); const String bytes = out.str(); const RunRef bad{.key = layout.blobTargetRunKey(1, 0, 0, 0), - .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .generation = 1}; + .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .key_generation = 1}; backend.putIfAbsent(bad.key, bytes); std::vector runs2; @@ -698,7 +698,7 @@ TEST(CASBlobInDegree, ZeroInDegreeStreamsBlockBounded) EXPECT_LE(backend.getCount(gen2_run_key), 2u); } -/// ==== kCondemned row codec + typed source-edge open (retired-in-snapshot T2, spec §2.1) ==== +/// ==== RunMarker::Condemned row codec + typed source-edge open (retired-in-snapshot T2, spec §2.1) ==== TEST(CASCondemnedRow, RoundTripAllTokenTypes) { @@ -711,11 +711,47 @@ TEST(CASCondemnedRow, RoundTripAllTokenTypes) row.size = 4096; row.condemn_round = 7; const auto bytes = DB::Cas::encodeCondemnedRow(row); - ASSERT_EQ(bytes[0], DB::Cas::kCondemned); + ASSERT_EQ(bytes[0], DB::Cas::runMarkerByte(DB::Cas::RunMarker::Condemned)); EXPECT_EQ(DB::Cas::decodeCondemnedRow(bytes), row); } } +TEST(CASCondemnedRow, UnknownMarkerByteFailsClosedWithCorruptedData) +{ + /// This pins the condemned-row decoder's own marker validation. + DB::Cas::CondemnedRow row; + row.token = DB::Cas::Token{.value = "t", .type = DB::Cas::TokenType::ETag}; + auto bytes = DB::Cas::encodeCondemnedRow(row); + bytes[0] = 0x03; + + try + { + static_cast(DB::Cas::decodeCondemnedRow(bytes)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + } +} + +TEST(CASRecordStream, RunMarkerByteContractFailsClosed) +{ + /// This helper is defense-in-depth; upstream word validation means no input path reaches it. + for (const auto marker : {DB::Cas::RunMarker::Zero, DB::Cas::RunMarker::Edge, DB::Cas::RunMarker::Condemned}) + EXPECT_EQ(DB::Cas::runMarkerFromByte(DB::Cas::runMarkerByte(marker), "CAS test"), marker); + + try + { + static_cast(DB::Cas::runMarkerFromByte(0x03, "CAS test")); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + } +} + TEST(CASCondemnedRow, UnknownFlagBitsFailClosed) { DB::Cas::CondemnedRow row; diff --git a/src/Disks/tests/gtest_cas_blob_meta_format.cpp b/src/Disks/tests/gtest_cas_blob_meta_format.cpp index bb055d2cef71..59d1f7205dc4 100644 --- a/src/Disks/tests/gtest_cas_blob_meta_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta_format.cpp @@ -6,6 +6,29 @@ using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } +namespace +{ +/// Same tiny inline copy as `gtest_cas_wire_vocab.cpp`'s `expectThrowsCode`: stays clear of +/// `Disks/tests/cas_test_helpers.h`'s `DB::Cas::tests::expectThrowsCode`, which would both drag +/// in the whole CAS backend/store machinery this file otherwise has no need for AND collide (same +/// namespace, same name and signature) if that header were ever included here too. +template +void expectThrowsCode(int expected_code, F && fn) +{ + try + { + fn(); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code); + } +} +} + +CAS_BATTERY_COVERS(BlobMeta); + TEST(CASFormatBattery, BlobMeta) { BlobMeta m; @@ -40,10 +63,10 @@ TEST(CASBlobMetaFormat, FailsClosedOnUnknownStateAndTruncation) /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"st\":\"zombie\",\"cr\":\"0\",\"sz\":\"0\"}\n"; - EXPECT_THROW(decodeBlobMeta(bad_state), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(bad_state); }); /// Missing state key -> CORRUPTED_DATA. const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"cr\":\"0\",\"sz\":\"0\"}\n"; - EXPECT_THROW(decodeBlobMeta(no_state), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(no_state); }); /// Truncated (header only) -> CORRUPTED_DATA. - EXPECT_THROW(decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"); }); } diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index f4c3a913fda7..01f0b99a465a 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -91,9 +91,20 @@ TEST(CASEncodingPins, SourceEdgeRunLines) SourceEdgeRecord active; active.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(2))}; active.source_id = UInt128(5); - active.marker = kEdgeActive; + active.marker = RunMarker::Edge; writer.append(active); + SourceEdgeRecord condemned; + condemned.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(3))}; + condemned.source_id = UInt128(0); + condemned.marker = RunMarker::Condemned; + condemned.delete_pending = true; + condemned.token = Token{.value = "token", .type = TokenType::ETag}; + condemned.size = 9; + condemned.condemn_round = 7; + condemned.marker_confirmed = true; + writer.append(condemned); + writer.finish(); out.finalize(); @@ -103,8 +114,10 @@ TEST(CASEncodingPins, SourceEdgeRunLines) const String header = fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()); const String expected_record = "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; - const String trailer = "{\"n\":1}\n"; - /// There is exactly one record, so the whole buffer must be byte-identical to header + record + trailer. - const String expected_full = header + expected_record + trailer; + const String expected_condemned = + "{\"b\":\"0100000000000000000000000000000003\",\"s\":\"00000000000000000000000000000000\",\"m\":\"condemned\",\"pend\":true,\"tt\":\"etag\",\"tv\":\"token\",\"sz\":9,\"cr\":\"7\",\"mc\":true}\n"; + const String trailer = "{\"n\":2}\n"; + /// Both records must remain byte-identical to their canonical stored representation. + const String expected_full = header + expected_record + expected_condemned + trailer; EXPECT_EQ(text, expected_full) << text; } diff --git a/src/Disks/tests/gtest_cas_enum_wire_table.cpp b/src/Disks/tests/gtest_cas_enum_wire_table.cpp new file mode 100644 index 000000000000..90f59c866cb2 --- /dev/null +++ b/src/Disks/tests/gtest_cas_enum_wire_table.cpp @@ -0,0 +1,131 @@ +#include +#include +#include +#include +#include + +using namespace DB::Cas; + +namespace +{ + +enum class Fruit : uint8_t +{ + Apple = 0, + Pear = 1, + Plum = 2, +}; + +constexpr EnumWireTable fruits{{{ + {Fruit::Apple, "apple"}, + {Fruit::Pear, "pear"}, + {Fruit::Plum, "plum"}, +}}}; + +static_assert(fruits.denseAndOrdered()); +static_assert(fruits.wordsUnique()); +static_assert(casEnumTableCoversEnum()); + +/// A one-based dense enum exercises the index arithmetic from the first entry's value. +enum class Grade : uint8_t +{ + Low = 1, + Mid = 2, + High = 3, +}; + +constexpr EnumWireTable grades{{{ + {Grade::Low, "low"}, + {Grade::Mid, "mid"}, + {Grade::High, "high"}, +}}}; + +static_assert(grades.denseAndOrdered()); +static_assert(casEnumTableCoversEnum()); + +} + +TEST(CASEnumWireTable, RoundTripsEveryEntryBothWays) +{ + for (const auto & e : fruits.entries) + { + EXPECT_EQ(fruits.toWord(e.value, "fruits"), e.word); + EXPECT_EQ(fruits.fromWord(e.word, "fruits"), e.value); + } + for (const auto & e : grades.entries) + EXPECT_EQ(grades.fromWord(grades.toWord(e.value, "grades"), "grades"), e.value); +} + +TEST(CASEnumWireTable, FromWordFailsClosed) +{ + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { fruits.fromWord("banana", "fruits"); }); +} + +/// `LOGICAL_ERROR` aborts the process in debug/sanitizer builds (`handle_error_code`), so the +/// defensive toWord branch needs the death-test split this test directory already uses (see +/// `gtest_cas_gc_state_format.cpp`'s `RejectsZeroGcShardsOnEncode` pair) — a bare EXPECT_THROW +/// would SIGABRT the whole gate binary on those lanes. +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASEnumWireTableDeathTest, ToWordAbortsOnOutOfRangeValue) +{ + EXPECT_DEATH(fruits.toWord(static_cast(99), "fruits"), "outside the wire vocabulary"); +} +#else +TEST(CASEnumWireTable, ToWordThrowsLogicalErrorOnOutOfRangeValue) +{ + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, + [&] { fruits.toWord(static_cast(99), "fruits"); }); +} +#endif + +/// The compile-time proofs must also be exercised in the direction where they can fail — a +/// predicate rewritten to `return true;` must break this file. All three are constexpr, so the +/// negative cases are plain static_asserts over deliberately bad tables: +namespace bad_tables +{ + +enum class Sparse : uint8_t { A = 0, B = 2 }; +constexpr EnumWireTable sparse{{{{Sparse::A, "a"}, {Sparse::B, "b"}}}}; +static_assert(!sparse.denseAndOrdered()); +/// ...and through the folded coverage proof, so deleting its density disjunct breaks the file +/// (this table is set-equal and word-unique — only density rejects it): +static_assert(!casEnumTableCoversEnum()); + +constexpr EnumWireTable dup_words{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "apple"}, {Fruit::Plum, "plum"}}}}; +static_assert(!dup_words.wordsUnique()); +/// ...and through the folded proof (dense, right-sized, set-equal — only word uniqueness rejects +/// it), so deleting the uniqueness disjunct breaks the file: +static_assert(!casEnumTableCoversEnum()); + +constexpr EnumWireTable dup_value{{{ + {Fruit::Apple, "apple"}, {Fruit::Apple, "pear"}, {Fruit::Plum, "plum"}}}}; +static_assert(!casEnumTableCoversEnum()); + +constexpr EnumWireTable invalid_value{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}, {static_cast(99), "plum"}}}}; +static_assert(!casEnumTableCoversEnum()); + +/// The two cases above fail the folded density check before the set-equality core runs, so the +/// core needs its own failing witnesses — both dense and word-unique, so they reach it. +/// Reaches the size comparison: one enumerator short. +constexpr EnumWireTable missing_enumerator{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}}}}; +static_assert(!casEnumTableCoversEnum()); + +/// Reaches the declared-values scan: right size, dense from Pear, an out-of-enum value present +/// and `Apple` missing — the asserts header's own motivating scenario. +constexpr EnumWireTable enumerator_missing{{{ + {Fruit::Pear, "pear"}, {Fruit::Plum, "plum"}, {static_cast(3), "quince"}}}}; +static_assert(!casEnumTableCoversEnum()); + +/// Guards the size comparison itself: every declared value present PLUS one out-of-enum entry — +/// the only miscoverage the declared-values scan cannot see (an enumerator deleted from the enum +/// while its table row survived). +constexpr EnumWireTable extra_entry{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}, {Fruit::Plum, "plum"}, + {static_cast(3), "quince"}}}}; +static_assert(!casEnumTableCoversEnum()); + +} diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index ae65ed27274b..808830e46a9f 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -554,7 +554,7 @@ String publishOneBlobPart(const PoolPtr & s, const String & ns, const String & r /// Whether the CURRENT retired list (any gc-shard) still holds an entry (ack-floor pipeline in flight). bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_fold_seal_format.cpp b/src/Disks/tests/gtest_cas_fold_seal_format.cpp index afe6331a4ed8..d5e2144093eb 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_format.cpp @@ -29,13 +29,15 @@ void eraseRequiredField(String & encoded, std::string_view field) } } +CAS_BATTERY_COVERS(FoldSeal); + TEST(CASFormatBattery, FoldSeal) { CasFoldSeal seal; seal.generation = 5; seal.parent_generation = 4; seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{7, 11}}; - seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(0x0f), .shard = 0, .generation = 5}); + seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(0x0f), .shard = 0, .key_generation = 5}); seal.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 4}; runFormatBattery({FormatId::FoldSeal, @@ -72,8 +74,8 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRejectsTwoBlobTargetRunsForOneShard) seal.generation = 7; seal.parent_generation = 6; seal.blob_target_runs = { - RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .generation = 7}, - RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .key_generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .key_generation = 7}, }; seal.condemned_summary[0] = CondemnedSummary{}; @@ -88,8 +90,8 @@ TEST(CASFoldSealFormatDeathTest, ProducerValidationRejectsMalformedSealBeforePut const Layout layout("p"); CasFoldSeal seal; seal.blob_target_runs = { - RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .generation = 7}, - RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .generation = 7}}; + RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .key_generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .key_generation = 7}}; seal.condemned_summary[0] = CondemnedSummary{}; EXPECT_DEATH({ validateFoldSealForWrite(seal, layout, 1); }, "duplicate blob-target shard"); } @@ -99,8 +101,8 @@ TEST(CASFoldSealFormat, ProducerValidationRejectsMalformedSealBeforePut) const Layout layout("p"); CasFoldSeal seal; seal.blob_target_runs = { - RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .generation = 7}, - RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .generation = 7}}; + RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .key_generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .key_generation = 7}}; seal.condemned_summary[0] = CondemnedSummary{}; cas_battery_detail::expectCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { validateFoldSealForWrite(seal, layout, 1); }, "duplicate blob-target shard"); @@ -117,7 +119,7 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel .key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, - .generation = 7}); + .key_generation = 7}); seal.condemned_summary[0] = CondemnedSummary{}; const String valid = encodeFoldSeal(seal); @@ -161,7 +163,7 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRejectsNoncanonicalRowsAndIncompleteS .key = layout.blobTargetRunKey(7, 1, 1, 0), .checksum = UInt128{1}, .shard = 1, - .generation = 7}); + .key_generation = 7}); seal.condemned_summary[0] = CondemnedSummary{}; cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, @@ -254,7 +256,7 @@ TEST(CASFoldSeal, FoldSealCondemnedSummaryRoundTrips) s.parent_generation = 8; s.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2}; s.blob_target_runs.push_back(RunRef{.key = "gc/gen/9/blob_target/0/0", .checksum = UInt128(0x77), - .shard = 0, .generation = 9}); + .shard = 0, .key_generation = 9}); s.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 5}; s.condemned_summary[1] = CondemnedSummary{}; /// explicit zero entry (totality over gc_shards) diff --git a/src/Disks/tests/gtest_cas_format_battery.cpp b/src/Disks/tests/gtest_cas_format_battery.cpp index f6ba9bcdbcd0..d6e9f01b3535 100644 --- a/src/Disks/tests/gtest_cas_format_battery.cpp +++ b/src/Disks/tests/gtest_cas_format_battery.cpp @@ -4,9 +4,28 @@ using namespace DB::Cas; +namespace +{ +template +void expectThrowsCode(int expected_code, F && fn) +{ + try + { + fn(); + FAIL() << "expected exception " << expected_code; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code) << e.message(); + } +} +} + /// The real cas_pool_meta case replaces the phase-1 toy proving instance. Every other control-plane /// format registers its own battery row in its own gtest_cas__format.cpp file (Tasks 3-6). +CAS_BATTERY_COVERS(PoolMeta); + TEST(CASFormatBattery, PoolMeta) { PoolMeta pm; @@ -21,3 +40,9 @@ TEST(CASFormatBattery, PoolMeta) .golden = currentFormatHeader("cas_pool_meta") + "{\"pid\":\"00112233445566778899aabbccddeeff\",\"hln\":256,\"gcs\":1,\"mrg\":3,\"alg\":\"ch128\"}\n"}); } + +TEST(CASPoolMeta, ValidateAlgosUsedRejectsUnknownByte) +{ + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [] { validatePoolAlgosUsed({7}, DB::ErrorCodes::CORRUPTED_DATA, "t"); }); +} diff --git a/src/Disks/tests/gtest_cas_gc_attempt.cpp b/src/Disks/tests/gtest_cas_gc_attempt.cpp index b919298ae24e..49e63031c6df 100644 --- a/src/Disks/tests/gtest_cas_gc_attempt.cpp +++ b/src/Disks/tests/gtest_cas_gc_attempt.cpp @@ -54,7 +54,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_fold.cpp b/src/Disks/tests/gtest_cas_gc_fold.cpp index c764aa0ac0b7..c3af2cbe4ef0 100644 --- a/src/Disks/tests/gtest_cas_gc_fold.cpp +++ b/src/Disks/tests/gtest_cas_gc_fold.cpp @@ -249,7 +249,7 @@ TEST(CASGCFold, EmptyDeltaShardCarriesParentRunRef) EXPECT_EQ(carried.key, parent_ref.key) << "carried ref points at the PARENT generation's run key"; EXPECT_EQ(carried.checksum, parent_ref.checksum); EXPECT_EQ(carried.shard, 0u); - EXPECT_EQ(carried.generation, st1.snap_generation) + EXPECT_EQ(carried.key_generation, st1.snap_generation) << "the carried ref names the generation whose key namespace physically holds the object"; } @@ -314,7 +314,7 @@ TEST(CASGCFold, PreviewResolvesCarriedRef) const auto seal2 = decodeFoldSeal( backend->get(store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes); ASSERT_EQ(seal2.blob_target_runs.size(), 1u); - ASSERT_EQ(seal2.blob_target_runs.front().generation, st1.snap_generation) + ASSERT_EQ(seal2.blob_target_runs.front().key_generation, st1.snap_generation) << "the current seal's ref physically lives at the parent generation (carried, not reconstructed)"; // The preview resolves the carried ref (a gen-1 physical key) and computes in-degree 1 => blob 1 is diff --git a/src/Disks/tests/gtest_cas_gc_leak.cpp b/src/Disks/tests/gtest_cas_gc_leak.cpp index a65b225b8f9a..36a77237e9dc 100644 --- a/src/Disks/tests/gtest_cas_gc_leak.cpp +++ b/src/Disks/tests/gtest_cas_gc_leak.cpp @@ -47,7 +47,7 @@ PoolPtr openTestPool(std::shared_ptr & out_backend) /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 158040be38c9..81a1dd46c2d3 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -1,4 +1,5 @@ #include "cas_test_helpers.h" +#include "cas_format_test_battery.h" #include #include #include @@ -24,6 +25,17 @@ class FailingMaintenanceReadBackend : public InMemoryBackend }; } +CAS_BATTERY_COVERS(GcMaintenanceState); + +TEST(CASFormatBattery, GcMaintenanceState) +{ + GcMaintenanceState state{.janitor_cursor = "cas/ns/a"}; + runFormatBattery({FormatId::GcMaintenanceState, + [&] { return sealObject(FormatId::GcMaintenanceState, encodeGcMaintenanceState(state)); }, + [](std::string_view s) { decodeGcMaintenanceState(std::string(openObject(FormatId::GcMaintenanceState, s))); }, + currentFormatHeader("cas_gc_maintenance_state") + "{\"cur\":\"cas/ns/a\"}\n"}); +} + TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) { EXPECT_EQ(static_cast(FormatId::GcMaintenanceState), 25); diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index e6530f131299..e03aa26dbaad 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -27,6 +27,8 @@ void expectThrowsCode(int expected_code, F && fn) } +CAS_BATTERY_COVERS(GcOutcomes); + TEST(CASFormatBattery, GcOutcomes) { OutcomeLog log; @@ -75,6 +77,42 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) EXPECT_EQ(encodeOutcomeLog(d), text); } +TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) +{ + OutcomeLog log; + log.entries.push_back({ObjectKind::Blob, + BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}, + Token{"e-1", TokenType::ETag}, OutcomeKind::Deleted}); + const String bytes = encodeOutcomeLog(log); + + const String token_value = R"(,"tv":"e-1")"; + const auto token_value_pos = bytes.find(token_value); + ASSERT_NE(token_value_pos, String::npos); + String missing_token_value = bytes; + missing_token_value.erase(token_value_pos, token_value.size()); + const OutcomeLog decoded = decodeOutcomeLog(missing_token_value); + ASSERT_EQ(decoded.entries.size(), 1u); + EXPECT_EQ(decoded.entries[0].token.value, ""); + + for (const String & field : {String(R"(,"ha":"ch128")"), String(R"(,"h":"00112233445566778899aabbccddeeff")"), String(R"(,"tt":"etag")")}) + { + const auto pos = bytes.find(field); + ASSERT_NE(pos, String::npos); + String incomplete = bytes; + incomplete.erase(pos, field.size()); + try + { + decodeOutcomeLog(incomplete); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), "CAS outcome log: record missing ha/h/tt"); + } + } +} + TEST(CASGCOutcomesFormat, GarbageAndUnknownWordsFailClosed) { EXPECT_THROW(decodeOutcomeLog(String("")), DB::Exception); diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index aa896dbf68df..1ad8f576809d 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -518,7 +518,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) const auto parsed = store->layout().parseBlobTargetRunKey(run.key); ASSERT_TRUE(parsed.has_value()); EXPECT_EQ(parsed->shard, run.shard); - EXPECT_EQ(parsed->generation, run.generation); + EXPECT_EQ(parsed->generation, run.key_generation); EXPECT_EQ(parsed->seq, 0u); } EXPECT_TRUE(run_seen[0]); diff --git a/src/Disks/tests/gtest_cas_gc_resume.cpp b/src/Disks/tests/gtest_cas_gc_resume.cpp index 6c707bdd60cd..55ff116b7109 100644 --- a/src/Disks/tests/gtest_cas_gc_resume.cpp +++ b/src/Disks/tests/gtest_cas_gc_resume.cpp @@ -28,7 +28,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// Whether the CURRENT retired list (any gc-shard) still holds an entry. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index b987cc39d8fb..8ea415a9f024 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -129,7 +129,7 @@ GcState readState(InMemoryBackend & b, const Pool & s) return decodeGcState(got->bytes); } -/// Whether ANY gc-shard's adopted-seal run still holds a `kCondemned` row (retired-in-snapshot T4: the +/// Whether ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row (retired-in-snapshot T4: the /// retired state rides the snapshot run, not a separate retired-list object) — the ack-floor deletion /// pipeline is still in flight while this is true. bool anyRetiredPending(InMemoryBackend & b, const Pool & s) @@ -542,7 +542,7 @@ TEST(CASGCRound, PublishDropReclaimsBlobAndManifestToFixpoint) /// retired-in-snapshot T4: after a round condemns one blob, the ADOPTED fold seal's per-shard /// condemned_summary reflects it (condemned_total == 1, pending_total == 0) — distilled zero-I/O from the -/// kCondemned rows the fold sealed into the snapshot run. +/// RunMarker::Condemned rows the fold sealed into the snapshot run. TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) { auto backend = std::make_shared(); @@ -582,7 +582,7 @@ TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) << "a non-pending condemned entry records its condemn round"; } -/// retired-in-snapshot T5: `previewDeletes` streams the adopted seal's `kCondemned` rows and reports each +/// retired-in-snapshot T5: `previewDeletes` streams the adopted seal's `RunMarker::Condemned` rows and reports each /// with the STORED condemn-time token — `awaiting_graduation` while newly condemned, then `delete_pending` /// once graduated, and NOTHING once the exact-token redelete has removed the blob. The preview performs no /// HEAD on the condemned rows (the token is durable in-run) and is WRITE-FREE throughout (spec §5 req 1). @@ -602,7 +602,7 @@ TEST(CASGCRound, PreviewReportsCondemnedRowsAndIsWriteFree) EXPECT_TRUE(gc.previewDeletes().empty()) << "a live-referenced blob is never previewed for deletion"; dropRefTransition(*backend, store->layout(), ns, "tbl", r); - runRegularRoundReclaiming(gc); /// condemning round: -1 => in-degree 0 => kCondemned row (not pending) + runRegularRoundReclaiming(gc); /// condemning round: -1 => in-degree 0 => RunMarker::Condemned row (not pending) /// Write-free contract: a full key->token snapshot must be identical across the previewDeletes call. const auto before = snapshotKeyTokens(*backend); @@ -689,7 +689,7 @@ TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) const auto parsed = store->layout().parseBlobTargetRunKey(run.key); ASSERT_TRUE(parsed.has_value()); EXPECT_EQ(parsed->shard, run.shard); - EXPECT_EQ(parsed->generation, run.generation); + EXPECT_EQ(parsed->generation, run.key_generation); EXPECT_EQ(parsed->seq, 0u); } EXPECT_TRUE(run_seen[0]); @@ -1468,7 +1468,7 @@ TEST(CASGCRetention, PruneRetainsLiveReferencedRun) backend->get(store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); ASSERT_EQ(seal1.blob_target_runs.size(), 1u); const String referenced_run_key = seal1.blob_target_runs.front().key; - ASSERT_EQ(seal1.blob_target_runs.front().generation, ref_gen); + ASSERT_EQ(seal1.blob_target_runs.front().key_generation, ref_gen); ASSERT_TRUE(backend->head(referenced_run_key).exists); /// Several idle rounds: no delta, no retired => pure ref-carry. Each round advances the generation @@ -1493,7 +1493,7 @@ TEST(CASGCRetention, PruneRetainsLiveReferencedRun) backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); ASSERT_EQ(seal_now.blob_target_runs.size(), 1u); EXPECT_EQ(seal_now.blob_target_runs.front().key, referenced_run_key); - EXPECT_EQ(seal_now.blob_target_runs.front().generation, ref_gen); + EXPECT_EQ(seal_now.blob_target_runs.front().key_generation, ref_gen); EXPECT_EQ(inDegreeOf(*backend, store->layout(), DB::UInt128(1)), 1) << "folding still resolves in-degree through the retained, carried parent ref"; @@ -1548,7 +1548,7 @@ TEST(CASGCRetention, HandOffDeletesSupersededRef) const auto seal_after = decodeFoldSeal( backend->get(store->layout().foldSealKey(st_after.snap_generation, st_after.snap_attempt))->bytes); for (const RunRef & rr : seal_after.blob_target_runs) - EXPECT_NE(rr.generation, old_gen) << "the live seal must have moved its ref off gen-1"; + EXPECT_NE(rr.key_generation, old_gen) << "the live seal must have moved its ref off gen-1"; /// ... and the post-CAS hand-off delete reclaimed gen-1's WHOLE prefix (not just the single run /// object): seal, attempt subtree, run — all gone. The ordinary prune would have leaked it because its diff --git a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp index c45cd3ff772d..c8bc92efffdf 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp @@ -564,7 +564,7 @@ TEST(CASGCShardRetireDrain, ReclaimsDroppableBlobOwnedByNonZeroShard) }; /// Whether ANY gc-shard still holds an in-flight condemned entry (the ack-floor deletion pipeline is /// in flight while this is true). Retired-in-snapshot (T4): reconstructed from the adopted fold seal's - /// kCondemned rows across all shards, not a separate retired list. + /// RunMarker::Condemned rows across all shards, not a separate retired list. auto anyRetiredPending = [&] { return anyCondemnedInSeal(*backend, layout); diff --git a/src/Disks/tests/gtest_cas_gc_state_format.cpp b/src/Disks/tests/gtest_cas_gc_state_format.cpp index aa661429f813..7e9452fdf8a7 100644 --- a/src/Disks/tests/gtest_cas_gc_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_state_format.cpp @@ -10,6 +10,8 @@ namespace DB::ErrorCodes extern const int LOGICAL_ERROR; } +CAS_BATTERY_COVERS(GcState); + TEST(CASFormatBattery, GcState) { GcState s; @@ -28,6 +30,8 @@ TEST(CASFormatBattery, GcState) "\"lo\":\"00000000000000000000000000000001\",\"ls\":\"12\"}\n"}); } +CAS_BATTERY_COVERS(GcHeartbeat); + TEST(CASFormatBattery, GcHeartbeat) { GcHeartbeat hb{UInt128(1), 1741}; diff --git a/src/Disks/tests/gtest_cas_inspect.cpp b/src/Disks/tests/gtest_cas_inspect.cpp index d807bec3f931..2ed70c293259 100644 --- a/src/Disks/tests/gtest_cas_inspect.cpp +++ b/src/Disks/tests/gtest_cas_inspect.cpp @@ -51,7 +51,7 @@ TEST(CASInspect, RendersSetPublishedAtOpWithNoPayloadSizeKey) const String bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); const String json = caInspectToJson(layout, key, bytes, DB::Cas::tests::fixture::fixtureLife(ns)); - EXPECT_NE(json.find(R"("kind":"SetPublishedAt")"), String::npos) << json; + EXPECT_NE(json.find(R"("kind":"set_published_at")"), String::npos) << json; EXPECT_EQ(json.find("payload"), String::npos) << json; } @@ -75,10 +75,99 @@ TEST(CASInspect, RendersEpochSealTxnWithPrevEpochSeal) const String bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); const String json = caInspectToJson(layout, key, bytes, DB::Cas::tests::fixture::fixtureLife(ns)); - EXPECT_NE(json.find(R"("kind":"EpochSeal")"), String::npos) << json; + EXPECT_NE(json.find(R"("kind":"epoch_seal")"), String::npos) << json; EXPECT_NE(json.find(R"("prev_epoch_seal":{"writer_epoch":2,"ref_sequence":9})"), String::npos) << json; } +/// The remaining two `RefOpKind` words this file's other tests do not exercise: a namespace's birth +/// record and its removal terminator. +TEST(CASInspect, RendersNamespaceBirthAndRemoveNamespaceOpKinds) +{ + const Layout layout("p"); + const RootNamespace ns{"srv1/db/tbl"}; + + RefLogTxn birth_txn; + birth_txn.ns = ns.string(); + birth_txn.txn_id = RefTxnId{1, 1}; + RefOp birth; + birth.kind = RefOpKind::NamespaceBirth; + birth_txn.ops.push_back(birth); + const String birth_key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), birth_txn.txn_id); + const String birth_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(birth_txn)); + const String birth_json = caInspectToJson( + layout, birth_key, birth_bytes, DB::Cas::tests::fixture::fixtureLife(ns)); + EXPECT_NE(birth_json.find(R"("kind":"namespace_birth")"), String::npos) << birth_json; + + RefLogTxn remove_txn; + remove_txn.ns = ns.string(); + remove_txn.txn_id = RefTxnId{1, 2}; + RefOp remove; + remove.kind = RefOpKind::RemoveNamespace; + remove_txn.ops.push_back(remove); + const String remove_key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), remove_txn.txn_id); + const String remove_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(remove_txn)); + const String remove_json = caInspectToJson( + layout, remove_key, remove_bytes, DB::Cas::tests::fixture::fixtureLife(ns)); + EXPECT_NE(remove_json.find(R"("kind":"remove_namespace")"), String::npos) << remove_json; +} + +/// `RefOwnerKind` renders as its full wire word (`committed`/`precommit`), not the enumerator spelling, +/// at both binding slots an `owner_transition` op carries. +TEST(CASInspect, RendersRefOwnerKindWireWords) +{ + const Layout layout("p"); + const RootNamespace ns{"srv1/db/tbl"}; + const RefTxnId id{1, 3}; + + RefLogTxn txn; + txn.ns = ns.string(); + txn.txn_id = id; + RefOp op; + op.kind = RefOpKind::OwnerTransition; + op.old_binding = RefOwnerBinding{RefOwnerKind::Committed, "all_1_1_0", manifestRef(1, 1, 1)}; + op.new_binding = RefOwnerBinding{RefOwnerKind::Precommit, "all_1_1_0", manifestRef(1, 1, 1)}; + txn.ops.push_back(op); + + const String key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id); + const String bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); + + const String json = caInspectToJson(layout, key, bytes, DB::Cas::tests::fixture::fixtureLife(ns)); + EXPECT_NE(json.find(R"("old_binding":{"kind":"committed")"), String::npos) << json; + EXPECT_NE(json.find(R"("new_binding":{"kind":"precommit")"), String::npos) << json; +} + +/// `TokenType` renders as its full wire word; the blob-target-run test below covers `emulated`, so +/// this pins the other two (`etag`/`generation`) via a second condemned-row-only run. +TEST(CASInspect, RendersTokenTypeWireWordsEtagAndGeneration) +{ + const Layout layout("p"); + + SourceEdgeRecord etag_rec; + etag_rec.ref = bh(1); + etag_rec.source_id = UInt128{0}; + etag_rec.marker = RunMarker::Condemned; + etag_rec.token = Token{.value = "v-etag", .type = TokenType::ETag}; + + SourceEdgeRecord gen_rec; + gen_rec.ref = bh(1); + gen_rec.source_id = UInt128{1}; + gen_rec.marker = RunMarker::Condemned; + gen_rec.token = Token{.value = "v-gen", .type = TokenType::Generation}; + + DB::WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(etag_rec); + writer.append(gen_rec); + writer.finish(); + out.finalize(); + const String bytes = out.str(); + + const String key = layout.blobTargetRunKey(/*generation*/3, /*attempt*/0, /*shard*/0, /*seq*/0); + const String json = caInspectToJson(layout, key, bytes); + EXPECT_NE(json.find(R"("type":"etag")"), String::npos) << json; + EXPECT_NE(json.find(R"("type":"generation")"), String::npos) << json; +} + TEST(CASInspect, RendersCommittedRowWithNoPayloadSizeKey) { const Layout layout("p"); @@ -117,7 +206,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) SourceEdgeRecord condemned_rec; condemned_rec.ref = bh(1); condemned_rec.source_id = UInt128{0}; - condemned_rec.marker = kCondemned; + condemned_rec.marker = RunMarker::Condemned; condemned_rec.delete_pending = true; condemned_rec.token = Token{.value = "etag-1", .type = TokenType::Emulated}; condemned_rec.size = 123; @@ -127,7 +216,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) SourceEdgeRecord edge_rec; edge_rec.ref = bh(2); edge_rec.source_id = UInt128(9); - edge_rec.marker = kEdgeActive; + edge_rec.marker = RunMarker::Edge; DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); @@ -147,6 +236,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) EXPECT_NE(json.find(R"("delete_pending":true)"), String::npos) << json; EXPECT_NE(json.find(R"("condemn_round":7)"), String::npos) << json; EXPECT_NE(json.find(R"("value":"etag-1")"), String::npos) << json; + EXPECT_NE(json.find(R"("type":"emulated")"), String::npos) << json; EXPECT_NE(json.find(R"("rows":2)"), String::npos) << json; EXPECT_NE(json.find(R"("distinct_blobs":2)"), String::npos) << json; EXPECT_NE(json.find(R"("edges":1)"), String::npos) << json; diff --git a/src/Disks/tests/gtest_cas_json_writer.cpp b/src/Disks/tests/gtest_cas_json_writer.cpp index f04b89255818..4eda28e717a3 100644 --- a/src/Disks/tests/gtest_cas_json_writer.cpp +++ b/src/Disks/tests/gtest_cas_json_writer.cpp @@ -21,7 +21,7 @@ TEST(CASJsonWriter, KeyValueSequenceMatchesCanonicalShape) w.u64Number(3); w.key("ok", first); w.boolValue(true); - w.key("o", "me", first); + w.key("ome", first); w.u64StringValue(1); w.closeObject(first); w.newline(); @@ -212,3 +212,31 @@ TEST(CASJsonWriterVocab, MatchesReferenceVocabulary) ref.finalize(); EXPECT_EQ(std::move(w).take(), ref.str()); } + +TEST(CASJsonWriter, WireKeyFieldHelpersMatchThePrimitivePairs) +{ + CasJsonWriter w; + bool first = true; + constexpr WireKey k_word{"st"}; + constexpr WireKey k_str{"hn"}; + constexpr WireKey k_u64s{"we"}; + constexpr WireKey k_num{"eat"}; + constexpr WireKey k_hex{"su"}; + constexpr WireKey k_bool{"fen"}; + writeWordField(w, k_word, "clean", first); + writeStringField(w, k_str, "host-1", first); + writeU64StringField(w, k_u64s, 7, first); + writeNumberField(w, k_num, 1752537630000, first); + writeHex128Field(w, k_hex, DB::UInt128{1}, first); + writeBoolField(w, k_bool, false, first); + w.closeObject(first); + w.newline(); + EXPECT_EQ(std::move(w).take(), + "{\"st\":\"clean\",\"hn\":\"host-1\",\"we\":\"7\",\"eat\":1752537630000," + "\"su\":\"00000000000000000000000000000001\",\"fen\":false}\n"); + + /// The reader-side comparison contract: a String key compares against the constant. + String key = "st"; + EXPECT_TRUE(key == k_word); + EXPECT_FALSE(key == k_str); +} diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index bf989f1e3206..d7430d321425 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -428,7 +428,7 @@ TEST(CASObservability, CaInspectDecodesRefLogToJson) const String json = caInspectToJson( layout, key, encodeRefLogTxn(txn), DB::Cas::tests::fixture::fixtureLife(ns)); EXPECT_NE(json.find("ref_log"), String::npos); - EXPECT_NE(json.find("OwnerTransition"), String::npos); + EXPECT_NE(json.find("owner_transition"), String::npos); EXPECT_NE(json.find("all_0_0_0"), String::npos); } @@ -440,11 +440,16 @@ TEST(CASObservability, CaInspectDecodesPartManifestToJson) PartManifest m; m.ref = ManifestRef{.writer_epoch = 1, .build_sequence = 2, .manifest_ordinal = 3}; m.root_namespace_id = ns; - ManifestEntry e; - e.path = "data.bin"; - e.placement = EntryPlacement::Inline; - e.inline_bytes = "hello"; - m.entries = {e}; + ManifestEntry inline_entry; + inline_entry.path = "data.bin"; + inline_entry.placement = EntryPlacement::Inline; + inline_entry.inline_bytes = "hello"; + ManifestEntry blob_entry; + blob_entry.path = "payload.bin"; + blob_entry.placement = EntryPlacement::Blob; + blob_entry.ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload.bin"))}; + blob_entry.blob_size = 5; + m.entries = {inline_entry, blob_entry}; m.payload_digest = computePayloadDigest(m); const ManifestId id{.root_namespace = ns, .ref = m.ref}; @@ -453,6 +458,9 @@ TEST(CASObservability, CaInspectDecodesPartManifestToJson) EXPECT_NE(json.find("\"root_namespace_id\""), String::npos); EXPECT_NE(json.find("data.bin"), String::npos); EXPECT_NE(json.find("\"manifest_ordinal\":3"), String::npos); + /// `EntryPlacement` renders as its full wire word (`inline`/`blob`), not the enumerator spelling. + EXPECT_NE(json.find(R"("placement":"inline")"), String::npos) << json; + EXPECT_NE(json.find(R"("placement":"blob")"), String::npos) << json; } TEST(CASObservability, CaInspectDecodesMountLeaseToJson) @@ -485,6 +493,36 @@ TEST(CASObservability, CaInspectDecodesGcStateToJson) EXPECT_NE(json.find("\"gc_shards\":4"), String::npos); } +/// `ObjectKind`/`ProvenanceOp` render as their full wire words, not the enumerator spelling. Loops +/// over every `ProvenanceOp` value so each one ends up pinned, not just whichever one a single case +/// would have picked. +TEST(CASObservability, CaInspectDecodesEnvelopeHeaderWithEveryProvenanceOpWord) +{ + Layout layout("p"); + const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of("envelope-inspect"))}; + const String key = layout.blobKey(ref); + + const std::vector> ops = { + {ProvenanceOp::Other, "other"}, + {ProvenanceOp::Insert, "insert"}, + {ProvenanceOp::Merge, "merge"}, + {ProvenanceOp::Mutation, "mutation"}, + {ProvenanceOp::Attach, "attach"}, + {ProvenanceOp::Repack, "repack"}, + }; + for (const auto & [op, word] : ops) + { + EnvelopeHeader h; + h.kind = ObjectKind::Blob; + h.provenance = Provenance{.op = op}; + const String bytes = encodeEnvelopeHeader(h, 256); + + const String json = caInspectToJson(layout, key, bytes); + EXPECT_NE(json.find(R"("kind":"blob")"), String::npos) << json; + EXPECT_NE(json.find("\"op\":\"" + word + "\""), String::npos) << json; + } +} + TEST(CASObservability, CaInspectUnknownKeyThrows) { Layout layout("p"); diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index d79f10bf73f7..43963c40e800 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -49,7 +49,7 @@ bool activeSourceExists(Backend & backend, const Layout & layout, const UInt128 String payload; while (view.next(key, payload)) { - if (payload.empty() || payload[0] != kEdgeActive) + if (payload.empty() || runMarkerFromByte(payload[0], "CAS test source-edge run") != RunMarker::Edge) continue; BlobRef ref; UInt128 row_source{}; @@ -74,7 +74,7 @@ size_t condemnedCount(Backend & backend, const Layout & layout) String key; String payload; while (view.next(key, payload)) - count += !payload.empty() && payload[0] == kCondemned; + count += !payload.empty() && runMarkerFromByte(payload[0], "CAS test source-edge run") == RunMarker::Condemned; view.verifyAgainst(run.checksum); } return count; diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index abb81c1928aa..ac14519e04c5 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -58,6 +58,8 @@ PartManifest sample() } +CAS_BATTERY_COVERS(PartManifest); + TEST(CASFormatBattery, PartManifest) { const PartManifest m = sample(); diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index dca4081c0041..c960c0547037 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -431,9 +431,9 @@ TEST(CASPluggableHash, Sha256BlobSeenByCondemnSweepAndFsckNotSilentlySkipped) ASSERT_NE(oit, frep.objects.end()) << "the sha256 blob must appear in fsck's detailed object list"; /// The fold above already condemned it into the GC snapshot, so fsck's GC-pipeline-view /// classification (not the generic Unaccounted bucket -- reachable only by width-correctly pairing - /// the fsck-side hash against the run's kCondemned row hash) must recognize it as known-to-GC. + /// the fsck-side hash against the run's RunMarker::Condemned row hash) must recognize it as known-to-GC. EXPECT_EQ(oit->cls, FsckClass::PendingGc) - << "THE CRUX: fsck must pair the sha256 blob against the GC snapshot's kCondemned row (a " + << "THE CRUX: fsck must pair the sha256 blob against the GC snapshot's RunMarker::Condemned row (a " "silent-leak regression in CasFsck.cpp's unref_hashes/in_run_hashes/retired_by_hash port " "leaves this as the generic Unaccounted bucket instead)"; } diff --git a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp index afe5ed26fb44..ee1b2a4d4d99 100644 --- a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp +++ b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp @@ -101,7 +101,7 @@ bool blobPresent(Backend & backend, const Layout & layout, const DB::UInt128 & h return backend.head(layout.blobKey(blobRefOf(hash))).exists; } -/// Whether ANY run the newest fold seal references carries a `kCondemned` row for `hash`. This is where +/// Whether ANY run the newest fold seal references carries a `RunMarker::Condemned` row for `hash`. This is where /// a rebuild used to put its zero-edge condemnations, so "nothing was condemned" is checked HERE rather /// than by watching for a deletion several rounds later. bool condemnedInSealedRuns(Backend & backend, const Layout & layout, const DB::UInt128 & hash) @@ -121,7 +121,7 @@ bool condemnedInSealedRuns(Backend & backend, const Layout & layout, const DB::U BlobRef ref; UInt128 sid; SourceEdgeKeyCodec::parse(k, ref, sid); - if (p.empty() || p[0] != kCondemned) + if (p.empty() || runMarkerFromByte(p[0], "CAS test source-edge run") != RunMarker::Condemned) continue; if (ref.digest.toU128() == hash) return true; diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 42e44585c357..2c2805f49310 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -1,4 +1,5 @@ #include +#include "cas_format_test_battery.h" #include #include #include @@ -26,17 +27,17 @@ BlobRef chRef(uint64_t n) SourceEdgeRecord edge(const BlobRef & ref, uint64_t source_id) { - return SourceEdgeRecord{.ref = ref, .source_id = UInt128(source_id), .marker = kEdgeActive}; + return SourceEdgeRecord{.ref = ref, .source_id = UInt128(source_id), .marker = RunMarker::Edge}; } SourceEdgeRecord zero(const BlobRef & ref) { - return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = kZeroMarker}; + return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Zero}; } SourceEdgeRecord condemned(const BlobRef & ref, const Token & token, uint64_t size, uint64_t round, bool pend) { - return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = kCondemned, + return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = pend, .token = token, .size = size, .condemn_round = round}; } @@ -66,6 +67,19 @@ std::vector decodeRun(const String & bytes) } +CAS_BATTERY_COVERS(RunFile); + +TEST(CASFormatBattery, RunFile) +{ + const std::vector records{edge(chRef(2), 5)}; + runFormatBattery({FormatId::RunFile, + [&] { return sealObject(FormatId::RunFile, encodeRun(records)); }, + [](std::string_view s) { decodeRun(std::string(openObject(FormatId::RunFile, s))); }, + fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()) + + "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n" + "{\"n\":1}\n"}); +} + TEST(CASRecordStream, EmptyRunRoundTripsAndChecksumMatches) { const String bytes = encodeRun({}); @@ -98,18 +112,18 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) EXPECT_EQ(back[0].ref, a); EXPECT_EQ(back[0].source_id, UInt128(10)); - EXPECT_EQ(back[0].marker, kEdgeActive); + EXPECT_EQ(back[0].marker, RunMarker::Edge); EXPECT_EQ(back[1].ref, b); EXPECT_EQ(back[1].source_id, UInt128(0)); - EXPECT_EQ(back[1].marker, kCondemned); + EXPECT_EQ(back[1].marker, RunMarker::Condemned); EXPECT_TRUE(back[1].delete_pending); EXPECT_EQ(back[1].token, (Token{"e-1", TokenType::ETag})); EXPECT_EQ(back[1].size, 4242u); EXPECT_EQ(back[1].condemn_round, 7u); EXPECT_EQ(back[2].ref, c); - EXPECT_EQ(back[2].marker, kZeroMarker); + EXPECT_EQ(back[2].marker, RunMarker::Zero); } TEST(CASRecordStream, WriterIsByteDeterministic) @@ -203,6 +217,24 @@ TEST(CASRecordStream, TrailerCountMismatchIsCorruptData) EXPECT_THROW(decodeRun(bytes), DB::Exception); } +TEST(CASRecordStream, UppercaseDigestInRecordKeyIsCorruptedData) +{ + String bytes = encodeRun({edge(chRef(10), 1)}); + const size_t digest = bytes.find("0000000000000000000000000000000a"); + ASSERT_NE(digest, String::npos); + bytes[digest + 31] = 'A'; + + try + { + static_cast(decodeRun(bytes)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + } +} + TEST(CASRecordStream, TruncationAtLineBoundaryFailsClosed) { const String bytes = encodeRun({edge(chRef(1), 10), edge(chRef(1), 20)}); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index 11a4c029f139..41d88fa57487 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -210,6 +210,8 @@ class ScopedCasGcLogCapture /// ---------- format-battery registration ---------- +CAS_BATTERY_COVERS(RefCatalog); + TEST(CASFormatBattery, RefCatalog) { RefCatalog c; @@ -261,6 +263,7 @@ TEST(CASRefCatalogFormat, RemovalStartedRoundIsRequiredExactlyForRemoving) const RefCatalog catalog{.entries = {removing}}; const String encoded = encodeRefCatalog(catalog); EXPECT_NE(encoded.find("\"rsr\":\"19\""), String::npos); + EXPECT_NE(encoded.find("\"st\":\"removing\""), String::npos); EXPECT_EQ(decodeRefCatalog(encoded), catalog); const String inc = "00000000000000000000000000000009"; @@ -566,7 +569,7 @@ TEST(CASRefCatalogFormat, NsStateToWordRaisesLogicalErrorOnImpossibleValue) #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASRefCatalogFormatDeathTest, NsStateToWordRaisesLogicalErrorOnImpossibleValueAborts) { - EXPECT_DEATH({ (void)nsStateToWord(static_cast(99)); }, "unknown ns state"); // NOLINT(clang-analyzer-optin.core.EnumCastOutOfRange): the whole point of this test is an impossible enum value + EXPECT_DEATH({ (void)nsStateToWord(static_cast(99)); }, "outside the wire vocabulary"); // NOLINT(clang-analyzer-optin.core.EnumCastOutOfRange): the whole point of this test is an impossible enum value } #endif @@ -703,7 +706,7 @@ TEST(CASRefCatalogAdmission, ReservationCoversActualWidestLegalRowsAcrossDecimal .key = layout.blobTargetRunKey(max, max, shard, 0), .checksum = std::numeric_limits::max(), .shard = shard, - .generation = max}); + .key_generation = max}); seal.condemned_summary.emplace(shard, CondemnedSummary{ .condemned_total = max, .pending_total = max, diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index 9b63fa4dfc92..b864587eff7d 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -1,6 +1,7 @@ #include #include "config.h" +#include "cas_format_test_battery.h" #include #include @@ -283,6 +284,21 @@ TEST(CASRefCheckpoint, CommittedThroughHasCanonicalExactWireEncoding) EXPECT_EQ(decodeRefCkpt(expected), ckpt); } +CAS_BATTERY_COVERS(RefCkpt); + +TEST(CASFormatBattery, RefCkpt) +{ + RefCkpt ckpt{.life_epoch = std::optional{7}, + .committed_through = RefTxnId{9, 11}, + .checkpoint_snapshot_id = RefTxnId{9, 10}, + .last_epoch_seal = RefTxnId{8, 12}}; + runFormatBattery({FormatId::RefCkpt, + [&] { return sealObject(FormatId::RefCkpt, encodeRefCkpt(ckpt)); }, + [](std::string_view s) { decodeRefCkpt(std::string(openObject(FormatId::RefCkpt, s))); }, + currentFormatHeader("cas_ref_ckpt") + + "{\"le\":\"7\",\"cte\":\"9\",\"cts\":\"11\",\"cse\":\"9\",\"css\":\"10\",\"lse\":\"8\",\"lss\":\"12\"}\n"}); +} + /// `last_epoch_seal` is chain evidence, not an arbitrary lower bound. It either names the frontier /// itself when that frontier is the terminal seal, or closes the immediately preceding numeric epoch. /// Accepting a gap or a later same-epoch frontier would manufacture a boundary that INV-2 never proved. diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index 5b6fa2070c46..cd9ce6c74291 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -471,6 +471,8 @@ TEST(CASRefEpochSealFormat, DecodeRejectsUnknownOpWordRegressionGuard) /// Shape-level failure-mode battery (truncation / v+1 gate / wrong type / leading garbage) /// =================================================================================== +static const DB::Cas::tests::BatteryCoverageRegistrar battery_covers_RefLog_seal{DB::Cas::FormatId::RefLog}; + TEST(CASRefEpochSealFormat, FormatBatteryEpochSeal) { RefLogTxn txn; diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index d5186e77aca1..ed6ed34ae599 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -285,6 +285,53 @@ TEST(CASRefCodec, RoundTripOwnerTransitionReplace) ASSERT_TRUE(decoded.ops[0].new_binding.has_value()); } +TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) +{ + RefLogTxn txn; + txn.ns = "ns"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = RefOpKind::OwnerTransition; + op.old_binding = RefOwnerBinding{RefOwnerKind::Precommit, "old", manifestRef(1, 1, 1)}; + op.new_binding = RefOwnerBinding{RefOwnerKind::Committed, "new", manifestRef(1, 1, 1)}; + txn.ops.push_back(op); + const String bytes = encodeRefLogTxn(txn); + + const String old_group = R"(,"obk":"precommit","orn":"old","ome":"1","omb":"1","omo":1)"; + const auto old_group_pos = bytes.find(old_group); + ASSERT_NE(old_group_pos, String::npos); + String old_absent = bytes; + old_absent.erase(old_group_pos, old_group.size()); + const RefLogTxn without_old = decodeRefLogTxn(old_absent, txn.ns, txn.txn_id); + ASSERT_EQ(without_old.ops.size(), 1u); + EXPECT_FALSE(without_old.ops[0].old_binding.has_value()); + EXPECT_TRUE(without_old.ops[0].new_binding.has_value()); + + const String old_ref = R"(,"orn":"old")"; + const auto old_ref_pos = bytes.find(old_ref); + ASSERT_NE(old_ref_pos, String::npos); + String incomplete_old = bytes; + incomplete_old.erase(old_ref_pos, old_ref.size()); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_old, txn.ns, txn.txn_id); }); + + const String new_group = R"(,"nbk":"committed","nrn":"new","nme":"1","nmb":"1","nmo":1)"; + const auto new_group_pos = bytes.find(new_group); + ASSERT_NE(new_group_pos, String::npos); + String new_absent = bytes; + new_absent.erase(new_group_pos, new_group.size()); + const RefLogTxn without_new = decodeRefLogTxn(new_absent, txn.ns, txn.txn_id); + ASSERT_EQ(without_new.ops.size(), 1u); + EXPECT_TRUE(without_new.ops[0].old_binding.has_value()); + EXPECT_FALSE(without_new.ops[0].new_binding.has_value()); + + const String new_ref = R"(,"nrn":"new")"; + const auto new_ref_pos = bytes.find(new_ref); + ASSERT_NE(new_ref_pos, String::npos); + String incomplete_new = bytes; + incomplete_new.erase(new_ref_pos, new_ref.size()); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_new, txn.ns, txn.txn_id); }); +} + TEST(CASRefCodec, RoundTripMultipleOpsInOneTransaction) { RefLogTxn txn; @@ -766,6 +813,8 @@ TEST(CASRefCodec, EncodeRejectsZeroManifestRefInSetPublishedAt) /// Shape-level failure-mode battery (truncation / v+1 gate / wrong type / leading garbage) /// =================================================================================== +CAS_BATTERY_COVERS(RefLog); + TEST(CASFormatBattery, RefLog) { RefLogTxn txn; diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 2293c0bc166c..2fea715879e2 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -419,6 +419,8 @@ TEST(CASRefSnapshotCodec, DecodeRejectsOversizedBufferDirectly) /// Shape-level failure-mode battery (truncation / v+1 gate / wrong type / leading garbage) /// =================================================================================== +CAS_BATTERY_COVERS(RefSnapshot); + TEST(CASFormatBattery, RefSnapshot) { const RefTableSnapshot s = makeLiveSnapshot(); diff --git a/src/Disks/tests/gtest_cas_server_root_format.cpp b/src/Disks/tests/gtest_cas_server_root_format.cpp index c2d1474dc29b..e302a4d81573 100644 --- a/src/Disks/tests/gtest_cas_server_root_format.cpp +++ b/src/Disks/tests/gtest_cas_server_root_format.cpp @@ -11,6 +11,8 @@ namespace DB::ErrorCodes extern const int CORRUPTED_DATA; } +CAS_BATTERY_COVERS(Owner); + TEST(CASFormatBattery, Owner) { OwnerObject o; @@ -31,11 +33,15 @@ TEST(CASOwnerFormat, RetiredAtRoundTrip) o.server_uuid = hexToU128("0123456789abcdeffedcba9876543210"); o.retired_at_ms = 1752537600000ULL; + EXPECT_EQ(encodeOwner(o), currentFormatHeader("cas_owner") + + "{\"su\":\"0123456789abcdeffedcba9876543210\",\"rt\":1752537600000}\n"); const OwnerObject back = decodeOwner(encodeOwner(o)); EXPECT_EQ(back.server_uuid, o.server_uuid); EXPECT_EQ(back.retired_at_ms, o.retired_at_ms); } +CAS_BATTERY_COVERS(ServerEpoch); + TEST(CASFormatBattery, ServerEpoch) { ServerEpoch e; @@ -46,6 +52,8 @@ TEST(CASFormatBattery, ServerEpoch) currentFormatHeader("cas_epoch") + "{\"nwe\":\"7\"}\n"}); } +CAS_BATTERY_COVERS(MountLease); + TEST(CASFormatBattery, MountLease) { MountLease m{hexToU128("0123456789abcdeffedcba9876543210"), 7, "host-1", 4242, diff --git a/src/Disks/tests/gtest_cas_text_format.cpp b/src/Disks/tests/gtest_cas_text_format.cpp index 4371afb3f9e8..625271435b24 100644 --- a/src/Disks/tests/gtest_cas_text_format.cpp +++ b/src/Disks/tests/gtest_cas_text_format.cpp @@ -1,4 +1,5 @@ #include +#include "cas_format_test_battery.h" #include #include #include @@ -34,6 +35,15 @@ void expectCode(int code, F && f) } } +TEST(CASFormatBattery, EveryRegisteredFormatIsBatteryCovered) +{ + std::set registered; + for (FormatId id : allRegisteredFormatIds()) + registered.insert(id); + EXPECT_EQ(registered, DB::Cas::tests::batteryCoveredIds()) + << "a registered codec is missing from the common battery (or vice versa)"; +} + /// ---- Task 2: FormatId entries for refsnaplog / blob meta / heartbeat ---- TEST(CASFormatIds, NewIdsExistWithFrozenValues) diff --git a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp index 0a76c338cdb5..522b7a738e68 100644 --- a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp +++ b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp @@ -75,7 +75,7 @@ ManifestId publishPart2( /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index efbe3da7a0ae..207ad24a295f 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -8,6 +9,43 @@ using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } +namespace +{ +/// Same tiny inline copy as `gtest_cas_part_manifest_format.cpp`'s `expectThrowsCode`: stays clear +/// of `Disks/tests/cas_test_helpers.h`'s `DB::Cas::tests::expectThrowsCode`, which would both drag +/// in the whole CAS backend/store machinery this file otherwise has no need for AND collide (same +/// namespace, same name and signature) if that header were ever included here too. +template +void expectThrowsCode(int expected_code, F && fn) +{ + try + { + fn(); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code); + } +} +} + +static_assert(DB::Cas::casEnumTableCoversEnum()); +static_assert(DB::Cas::casEnumTableCoversEnum()); +static_assert(DB::Cas::casEnumTableCoversEnum()); + +TEST(CASWireVocab, EnumTablesPinTheCurrentWords) +{ + using namespace DB::Cas; + EXPECT_EQ(kTokenTypeWords.toWord(TokenType::ETag, "t"), "etag"); + EXPECT_EQ(kTokenTypeWords.toWord(TokenType::Generation, "t"), "generation"); + EXPECT_EQ(kTokenTypeWords.toWord(TokenType::Emulated, "t"), "emulated"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::CityHash128, "t"), "ch128"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::XXH3_128, "t"), "xxh3"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::Sha256, "t"), "sha256"); + EXPECT_EQ(kObjectKindWords.toWord(ObjectKind::Blob, "t"), "blob"); +} + TEST(CASWireVocab, EnumWordsRoundTrip) { for (TokenType t : {TokenType::ETag, TokenType::Generation, TokenType::Emulated}) @@ -15,8 +53,8 @@ TEST(CASWireVocab, EnumWordsRoundTrip) for (BlobHashAlgo a : {BlobHashAlgo::CityHash128, BlobHashAlgo::XXH3_128, BlobHashAlgo::Sha256}) EXPECT_EQ(blobHashAlgoFromWord(blobHashAlgoName(a), "a"), a); EXPECT_EQ(objectKindFromWord(objectKindToWord(ObjectKind::Blob), "k"), ObjectKind::Blob); - EXPECT_THROW(tokenTypeFromWord("nope", "t"), DB::Exception); - EXPECT_THROW(blobHashAlgoFromWord("nope", "a"), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { tokenTypeFromWord("nope", "t"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { blobHashAlgoFromWord("nope", "a"); }); } TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) @@ -51,3 +89,104 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) const BlobRef back{blobHashAlgoFromWord(ha, "a"), codecFor(blobHashAlgoFromWord(ha, "a")).fromHex(h)}; EXPECT_EQ(back, ref); } + +TEST(CASWireVocab, ManifestRefBundleWritesTheOldPrefixedKeys) +{ + using namespace DB::Cas; + CasJsonWriter w; + bool first = true; + writeManifestRefFields(w, first, kOldManifestRefKeys, ManifestRef{1, 2, 3}); + w.closeObject(first); + EXPECT_EQ(std::move(w).take(), R"({"ome":"1","omb":"2","omo":3})"); +} + +TEST(CASWireVocab, MatchAndBuildRoundTripsABlobRef) +{ + using namespace DB::Cas; + const String rendered = R"({"ha":"ch128","h":"00112233445566778899aabbccddeeff"})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + BlobRefFields fields; + String key; + while (r.nextKey(key)) + { + if (matchBlobRefFields(key, r, fields)) + continue; + r.skipUnknown(key); + } + const BlobRef ref = fields.build("t"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(ref.algo, "t"), "ch128"); +} + +TEST(CASWireVocab, BlobRefBuildFailsClosedOnHalfAGroupAndOnBadWidth) +{ + using namespace DB::Cas; + BlobRefFields only_algo; + only_algo.algo_word = "ch128"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { only_algo.build("t"); }); + + BlobRefFields short_digest; + short_digest.algo_word = "ch128"; + short_digest.digest_hex = "00112233445566778899aabbccddee"; /// 30 hex chars, needs 32 + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { short_digest.build("t"); }); +} + +TEST(CASWireVocab, BlobRefBuildFailsClosedOnRightWidthNonHexDigest) +{ + using namespace DB::Cas; + BlobRefFields bad_hex; + bad_hex.algo_word = "ch128"; + bad_hex.digest_hex = "gg112233445566778899aabbccddeeff"; /// 32 chars (right width), not hex + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { bad_hex.build("t"); }); +} + +TEST(CASWireVocab, MatchManifestRefFieldsAndBuildRefRoundTripInAnyKeyOrder) +{ + using namespace DB::Cas; + /// Fed out of writer order (mo, me, mb) to pin key-order independence. `me`/`mb` are quoted + /// decimal strings and `mo` is a bare number -- a swapped read primitive between the two shapes + /// would fail to parse this literal. + const String rendered = R"({"mo":3,"me":"7","mb":"9"})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + ManifestRefFields fields; + String key; + while (r.nextKey(key)) + { + if (matchManifestRefFields(key, r, kBareManifestRefKeys, fields)) + continue; + r.skipUnknown(key); + } + EXPECT_EQ(fields.buildRef("t", "ctx"), (ManifestRef{7, 9, 3})); +} + +TEST(CASWireVocab, ManifestRefFieldsBuildRefFailsClosedOnHalfAGroup) +{ + using namespace DB::Cas; + ManifestRefFields fields; + fields.epoch = 7; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { fields.buildRef("t", "ctx"); }); +} + +TEST(CASWireVocab, MatchTokenFieldsConsumesTtTvAndLeavesUnrelatedKeyUnmatched) +{ + using namespace DB::Cas; + const String rendered = R"({"tt":"etag","tv":"abc","zz":1})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + TokenFields fields; + String key; + bool saw_unmatched = false; + while (r.nextKey(key)) + { + if (matchTokenFields(key, r, fields)) + continue; + saw_unmatched = true; + r.skipUnknown(key); + } + ASSERT_TRUE(fields.type_word.has_value()); + EXPECT_EQ(*fields.type_word, "etag"); + ASSERT_TRUE(fields.value.has_value()); + EXPECT_EQ(*fields.value, "abc"); + EXPECT_TRUE(saw_unmatched); +} From 540890f654bce0d3233bf82f2ec25a5553c9a4a6 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:03:08 +0200 Subject: [PATCH 4/8] =?UTF-8?q?cas:=20wire-keys=20phase=202=20=E2=80=94=20?= =?UTF-8?q?cut=20the=20wire-format=20JSON=20keys=20to=20semantic=20names?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The actual key rename, on top of the phase-1 carrier infrastructure. The format generation history is first reset to a `{1, 1}` baseline, since CAS has no released, persisted data yet and pre-release generations exist only to prove the evolution machinery. Every CAS wire format switches its JSON keys from single-letter/abbreviated spellings to descriptive names in one pass: the shared `BlobRef`/`Token`/ `ManifestRef`/binding fields, `cas_blob_meta`, `cas_pool_meta` (`algos_used` becomes a JSON word array instead of a bitmask), GC state/heartbeat/ maintenance state, the server-root record (`MountLease::min_active` becomes `min_active_build_sequence`), `cas_ref_ckpt`, `cas_ref_log` (the seal link becomes `!prev_epoch`/`!prev_seq`), `cas_ref_snapshot`, `cas_part_manifest`, `cas_run`, `cas_gc_outcomes` (`kind`/`outcome`), the fold-seal record and its `CoverageClass` words, the blob descriptor (with its 239-byte worst case proved at compile time against the 240-byte floor), and `cas_ref_catalog`. Golden tests are re-pinned to the new bytes throughout. Token-group requiredness is unified through `TokenFields::build`: an outcome missing its token now fails closed instead of serializing a partial group. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../cas/architecture/manifests-and-refs.md | 2 +- .../cas/architecture/mounts-and-leases.md | 12 +- .../cas/architecture/storage-layout.md | 2 +- .../ContentAddressedTransaction.cpp | 6 +- .../Formats/CasBlobEnvelopeFormat.cpp | 74 ++++- .../Formats/CasBlobEnvelopeFormat.h | 18 +- .../Formats/CasBlobMetaFormat.cpp | 15 +- .../Formats/CasBlobMetaFormat.h | 5 + .../Formats/CasEnvelopeLimits.h | 4 +- .../Formats/CasFoldSealFormat.cpp | 179 ++++++----- .../Formats/CasFoldSealFormat.h | 72 +++-- .../ContentAddressed/Formats/CasFormat.cpp | 58 +--- .../ContentAddressed/Formats/CasFormat.h | 101 +----- .../Formats/CasGcMaintenanceStateFormat.cpp | 4 +- .../Formats/CasGcOutcomesFormat.cpp | 26 +- .../Formats/CasGcOutcomesFormat.h | 3 + .../Formats/CasGcStateFormat.cpp | 24 +- .../Formats/CasGcStateFormat.h | 4 +- .../ContentAddressed/Formats/CasLayout.cpp | 4 +- .../Formats/CasPartManifestFormat.cpp | 59 ++-- .../Formats/CasPartManifestFormat.h | 13 +- .../Formats/CasPoolMetaFormat.cpp | 99 +++--- .../Formats/CasPoolMetaFormat.h | 18 +- .../Formats/CasRecordStreamFormat.cpp | 69 ++-- .../Formats/CasRecordStreamFormat.h | 19 +- .../Formats/CasRefCatalogFormat.cpp | 30 +- .../Formats/CasRefCatalogFormat.h | 8 +- .../Formats/CasRefCkptFormat.cpp | 62 ++-- .../Formats/CasRefLogFormat.cpp | 119 ++++--- .../Formats/CasRefLogFormat.h | 25 +- .../Formats/CasRefSnapshotFormat.cpp | 96 +++--- .../Formats/CasRefSnapshotFormat.h | 4 +- .../Formats/CasRefWireVocab.h | 7 +- .../Formats/CasServerRootFormats.cpp | 41 ++- .../Formats/CasServerRootFormats.h | 16 +- .../Formats/CasTextFormat.cpp | 35 ++ .../ContentAddressed/Formats/CasTextFormat.h | 17 +- .../ContentAddressed/Formats/CasWireVocab.cpp | 9 +- .../ContentAddressed/Formats/CasWireVocab.h | 54 ++-- .../ContentAddressed/Formats/README.md | 77 +++-- .../ContentAddressed/Gc/CasGc.cpp | 24 +- .../Gc/CasOrphanManifestSweep.cpp | 22 +- .../Gc/CasOrphanManifestSweep.h | 4 +- .../ContentAddressed/Pool/CasPartWriteTxn.cpp | 4 +- .../ContentAddressed/Pool/CasPool.cpp | 55 +--- .../ContentAddressed/Pool/CasPool.h | 2 +- .../ContentAddressed/Pool/CasPoolMeta.cpp | 4 +- .../ContentAddressed/Pool/CasServerRoot.cpp | 28 +- .../ContentAddressed/Pool/CasServerRoot.h | 18 +- .../Primitives/CasBlobDigest.h | 5 +- .../Primitives/CasEnumWireTableAsserts.h | 3 +- .../Tools/CasDecommission.cpp | 4 +- .../ContentAddressed/Tools/CasInspect.cpp | 10 +- src/Disks/tests/cas_test_helpers.h | 12 +- .../tests/gtest_cas_blob_envelope_format.cpp | 129 +++++++- .../tests/gtest_cas_blob_meta_format.cpp | 29 +- src/Disks/tests/gtest_cas_decommission.cpp | 6 +- src/Disks/tests/gtest_cas_encoding_pins.cpp | 302 ++++++++++++++++-- src/Disks/tests/gtest_cas_enum_wire_table.cpp | 2 +- src/Disks/tests/gtest_cas_event_log.cpp | 2 +- src/Disks/tests/gtest_cas_fold_seal_codec.cpp | 2 +- .../tests/gtest_cas_fold_seal_format.cpp | 101 +++--- src/Disks/tests/gtest_cas_forget.cpp | 8 +- src/Disks/tests/gtest_cas_format.cpp | 120 +++---- src/Disks/tests/gtest_cas_format_battery.cpp | 18 +- src/Disks/tests/gtest_cas_fsck.cpp | 4 +- .../tests/gtest_cas_gc_arithmetic_intake.cpp | 36 +-- src/Disks/tests/gtest_cas_gc_attempt.cpp | 2 +- src/Disks/tests/gtest_cas_gc_bounded_walk.cpp | 8 +- src/Disks/tests/gtest_cas_gc_fold.cpp | 2 +- .../tests/gtest_cas_gc_frontier_gate.cpp | 10 +- src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 190 +++++------ src/Disks/tests/gtest_cas_gc_leak.cpp | 2 +- .../gtest_cas_gc_maintenance_state_format.cpp | 24 +- .../tests/gtest_cas_gc_outcomes_format.cpp | 63 ++-- src/Disks/tests/gtest_cas_gc_rebuild.cpp | 6 +- src/Disks/tests/gtest_cas_gc_resume.cpp | 2 +- src/Disks/tests/gtest_cas_gc_round.cpp | 12 +- src/Disks/tests/gtest_cas_gc_shard_plan.cpp | 2 +- src/Disks/tests/gtest_cas_gc_state_format.cpp | 38 +-- src/Disks/tests/gtest_cas_heartbeat.cpp | 20 +- src/Disks/tests/gtest_cas_inspect.cpp | 27 ++ src/Disks/tests/gtest_cas_json_writer.cpp | 32 +- src/Disks/tests/gtest_cas_mount.cpp | 38 +-- .../tests/gtest_cas_ns_file_incarnation.cpp | 54 ---- .../tests/gtest_cas_ns_file_read_contract.cpp | 2 +- .../tests/gtest_cas_orphan_manifest_sweep.cpp | 10 +- .../tests/gtest_cas_orphan_nomination.cpp | 2 +- .../tests/gtest_cas_part_manifest_format.cpp | 117 +++++-- src/Disks/tests/gtest_cas_part_write.cpp | 2 +- .../gtest_cas_part_write_root_dangle.cpp | 12 +- src/Disks/tests/gtest_cas_pluggable_hash.cpp | 29 +- src/Disks/tests/gtest_cas_pool.cpp | 13 +- .../gtest_cas_rebuild_condemn_nothing.cpp | 2 +- .../tests/gtest_cas_record_stream_format.cpp | 98 +++++- .../tests/gtest_cas_recovery_grounding.cpp | 12 +- src/Disks/tests/gtest_cas_ref_catalog.cpp | 127 +++++--- .../gtest_cas_ref_catalog_birth_wiring.cpp | 4 +- src/Disks/tests/gtest_cas_ref_ckpt.cpp | 64 ++-- src/Disks/tests/gtest_cas_ref_ckpt_join.cpp | 9 +- .../tests/gtest_cas_ref_contiguous_alloc.cpp | 108 ------- .../tests/gtest_cas_ref_epoch_seal_format.cpp | 30 +- src/Disks/tests/gtest_cas_ref_log_format.cpp | 65 +++- .../tests/gtest_cas_ref_read_contract.cpp | 2 +- .../tests/gtest_cas_ref_snapshot_format.cpp | 50 ++- .../tests/gtest_cas_server_root_format.cpp | 43 ++- .../tests/gtest_cas_shutdown_context.cpp | 2 +- .../gtest_cas_sweep_deletion_premise.cpp | 16 +- src/Disks/tests/gtest_cas_text_format.cpp | 46 ++- .../tests/gtest_cas_truncate_reclaim.cpp | 2 +- src/Disks/tests/gtest_cas_wire_vocab.cpp | 90 +++++- src/Disks/tests/gtest_cas_writer_duties.cpp | 2 +- .../StorageSystemContentAddressedMounts.cpp | 2 +- tests/integration/test_cas_gc_sharded/test.py | 16 +- .../test_cas_gcs/gcs_mocks/server.py | 2 +- tests/integration/test_cas_gcs/test.py | 4 +- .../05023_cas_dropns_leaked_namespace.sh | 26 +- 117 files changed, 2350 insertions(+), 1635 deletions(-) diff --git a/docs/en/antalya/cas/architecture/manifests-and-refs.md b/docs/en/antalya/cas/architecture/manifests-and-refs.md index df86d82e575d..ba476cffe88a 100644 --- a/docs/en/antalya/cas/architecture/manifests-and-refs.md +++ b/docs/en/antalya/cas/architecture/manifests-and-refs.md @@ -101,7 +101,7 @@ swept for that root. flowchart TD A["LIST one page of cas/manifests/
freeze candidates with exact GET"] --> B{"build-prefix eligible?
durable watermark fact only"} B -->|"epoch less than lease epoch"| ELIG["eligible, old-epoch debris"] - B -->|"same epoch, min_active clears build_seq"| ELIG + B -->|"same epoch, min_active_build_sequence clears build_seq"| ELIG B -->|"no lease, or epoch ahead, or build may be live"| SKIP["skip"] ELIG --> C["protection view: committed manifests
plus live precommits
plus manifests with an unfolded minus-one"] C -->|"key protected"| SKIP2["skip"] diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index d778756ce323..66e540a5464f 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -68,7 +68,7 @@ Two failure modes this closes: One object, `gc/server-roots//mount`, carries **both** the liveness lease and the build watermark — there is no separate watermark object. `MountLease` fields: `server_uuid`, `writer_epoch`, `write_attempt_id`, `hostname`, `pid`, `started_at_ms`, renewal `seq`, -`expires_at_ms`, `min_active` (the build-watermark floor), and `gc_fenced`. +`expires_at_ms`, `min_active_build_sequence` (the build-watermark floor), and `gc_fenced`. - **Logical renewal identity.** Each holder-originated body has a fresh nonzero `write_attempt_id`. One logical renewal fixes one immutable `(key, bytes, expected token, @@ -133,9 +133,9 @@ into "not found". Global build ordering is the **pair** `(writer_epoch, build_seq)` compared lexicographically — the exact comparison GC uses for eligibility. The durable authority for both is the mount object -itself: no mount means no deletion authority means nothing is swept. `min_active`, the oldest +itself: no mount means no deletion authority means nothing is swept. `min_active_build_sequence`, the oldest in-flight `build_seq`, rides in the same mount object as the watermark floor; `UINT64_MAX` in -`min_active` is the farewell/retired sentinel, not a real build. +`min_active_build_sequence` is the farewell/retired sentinel, not a real build. ## Mount claim outcomes {#claim-outcomes} @@ -153,7 +153,7 @@ a `MountClaimResult::Kind` together with a `MountPriorState` describing which ce | `MountPriorState` | Certificate that justified the reclaim | |---|---| | `None` | no reclaim needed (fresh claim or same-epoch refresh) | -| `Clean` | the predecessor's own graceful farewell (`min_active == UINT64_MAX`) | +| `Clean` | the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) | | `Fenced` | GC's own threshold-gated fence-out (`gc_fenced`) | | `UncleanObserved` | this claimant's own token-stability observation held for the full `TTL + drift` window | @@ -168,7 +168,7 @@ stateDiagram-v2 Absent --> Live: claimMount putIfAbsent, seq=1 Live --> Live: keeper beat, putOverwrite seq+1 Live --> Fenced: GC observes a stable token past threshold, gc_fenced=1, body preserved - Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active=MAX) + Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active_build_sequence=MAX) Fenced --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim Terminated --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim Live --> Live: same-uuid claim, proven-dead token via UncleanObserved @@ -218,7 +218,7 @@ processed before renewal resumes. **Clean unmount:** request stop and join both persistent workers, drain the ref lanes, and only if the drain *certified* quiescence call `MountLeaseKeeper::release` on an `Active` keeper to write the -terminal farewell (`expires_at_ms` already expired, `min_active = UINT64_MAX`). That sentinel is what +terminal farewell (`expires_at_ms` already expired, `min_active_build_sequence = UINT64_MAX`). That sentinel is what lets a successor reclaim instantly. A `RenewalTerminal` keeper, an unresolved ref write, or a sent renewal ambiguity writes no farewell — an unearned farewell would let a successor start mutating while a stale conditional request from the predecessor is still in flight. diff --git a/docs/en/antalya/cas/architecture/storage-layout.md b/docs/en/antalya/cas/architecture/storage-layout.md index e4d20725836b..b136552acf37 100644 --- a/docs/en/antalya/cas/architecture/storage-layout.md +++ b/docs/en/antalya/cas/architecture/storage-layout.md @@ -45,7 +45,7 @@ namespace's shape and never interprets its contents. | `gc/gen//attempt//outcomes//.zst` | GC outcome log | `cas_gc_outcomes` | GC | | `gc/server-roots//owner` | server-root owner singleton | `cas_owner` | mount | | `gc/server-roots//epoch` | server-root epoch singleton | `cas_epoch` | mount | -| `gc/server-roots//mount` | mount lease (incl. `min_active` watermark) | `cas_mount_lease` | mount | +| `gc/server-roots//mount` | mount lease (incl. `min_active_build_sequence` watermark) | `cas_mount_lease` | mount | | `roots/` | loose mountpoint object, verbatim | — (never interpreted) | upper layers | | `staging//…` | S3-native upload staging scratch | — | writer, own mount only | diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp index 1216906194b4..437c4d536250 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp @@ -113,7 +113,7 @@ ContentAddressedTransaction::~ContentAddressedTransaction() /// backstop for aborted/exception-unwound transactions whose publishStaging never ran. cleanupPendingTempFiles(); - /// An uncommitted transaction's uploads become min_active-spared debris: abandon every + /// An uncommitted transaction's uploads become min_active_build_sequence-spared debris: abandon every /// still-open PartWriteTxn so its build_seq is retired. This replaces the former pin machinery. if (committed) return; @@ -759,8 +759,8 @@ std::string ContentAddressedTransaction::buildS3StagingBlobHeader( header.kind = Cas::ObjectKind::Blob; header.incarnation_tag = (static_cast(thread_local_rng()) << 64) | thread_local_rng(); header.build_id = 0; /// not known at stream time; diagnostic-only (not read by GC/read paths) - /// ch = the real ClickHouse VERSION_INTEGER (diagnostic-only; consistent with `PartWriteTxn::buildHeader`). - /// The v3 envelope drops hash_algo/domain_id/writer_version, so forensics ride on ch + bld. + /// `chver` = the real ClickHouse VERSION_INTEGER (diagnostic-only; consistent with `PartWriteTxn::buildHeader`). + /// The envelope drops hash_algo/domain_id/writer_version, so forensics ride on `chver` + `build`. header.provenance = Cas::Provenance{ /*created_at_ms*/ 0, cfg.server_id, VERSION_INTEGER, Cas::ProvenanceOp::Other}; header.intended_ref = route.ns.string() + "/" + route.ref; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index a4933ff2ccb8..129714d6b9e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -1,8 +1,11 @@ #include +#include #include #include #include #include +#include +#include namespace DB { @@ -26,11 +29,11 @@ namespace EnvelopeWire constexpr WireKey type{"type"}; constexpr WireKey version{"v"}; constexpr WireKey tag{"tag"}; - constexpr WireKey build{"bld"}; - constexpr WireKey time_ms{"ts"}; - constexpr WireKey creator{"by"}; + constexpr WireKey build{"build"}; + constexpr WireKey time_ms{"time_ms"}; + constexpr WireKey creator{"creator"}; constexpr WireKey op{"op"}; - constexpr WireKey chver{"ch"}; + constexpr WireKey chver{"chver"}; constexpr WireKey ref{"ref"}; } @@ -45,6 +48,60 @@ constexpr EnumWireTable kProvenanceOpWords{{{ static_assert(casEnumTableCoversEnum()); +/// Byte cost of one JSON key as written by `CasJsonWriter::key`: the leading `{`/`,` separator (1) +/// plus the opening quote (1), the key text, and the closing quote and colon (2). +constexpr size_t keyCost(WireKey key) +{ + return 4 + key.text.size(); +} + +/// A quoted `hex128Value` is always exactly this wide -- 2 quote bytes plus 2 hex digits per +/// `UInt128` byte -- because `writeHexUIntLowercase` zero-pads; there is no smaller or larger case. +constexpr size_t kQuotedHex128Len = 2 + sizeof(UInt128) * 2; + +/// Maximum decimal digits an unquoted `writeIntText` value can produce for each integer width the +/// envelope persists, taken from the type itself rather than re-typed as a literal. +constexpr size_t kMaxU64DecimalLen = std::numeric_limits::digits10 + 1; +constexpr size_t kMaxU32DecimalLen = std::numeric_limits::digits10 + 1; + +/// The longest persisted `op` word, found by walking the table rather than hardcoding one -- the +/// worst case must track `kProvenanceOpWords` even if a future entry outgrows "mutation". +constexpr size_t maxProvenanceOpWordLen() +{ + size_t max_len = 0; + for (const auto & entry : kProvenanceOpWords.entries) + max_len = std::max(max_len, entry.word.size()); + return max_len; +} + +/// Mandatory (always-written whenever `provenance` is set) non-`ref` fields at type maxima, in the +/// exact field order `encodeEnvelopeHeader` writes them. `CasPoolMetaFormat.cpp` records why 240 was +/// chosen as the floor above this bound. +constexpr size_t kMandatoryNonRefWorstCase = + keyCost(EnvelopeWire::type) + 2 + kBlobType.size() + + keyCost(EnvelopeWire::version) + kMaxU32DecimalLen + + keyCost(EnvelopeWire::tag) + kQuotedHex128Len + + keyCost(EnvelopeWire::build) + kQuotedHex128Len + + keyCost(EnvelopeWire::time_ms) + kMaxU64DecimalLen + + keyCost(EnvelopeWire::creator) + kQuotedHex128Len + + keyCost(EnvelopeWire::op) + 2 + maxProvenanceOpWordLen() + + keyCost(EnvelopeWire::chver) + kMaxU32DecimalLen; + +/// The encoder always frames `ref`, even when empty: the key (`,"ref":`), the empty quotes, the +/// closing `}`, and the trailing '\n' reserved at byte `blob_header_len - 1`. +constexpr size_t kRefFramingAndTerminator = keyCost(EnvelopeWire::ref) + 2 + 1 + 1; + +/// The worst-case byte count `encodeEnvelopeHeader` can ever produce before the diagnostic `ref` +/// gets any budget at all. Proven, not merely documented: the static_assert below fails the BUILD if +/// a future key or type change ever closes the margin under `kMinBlobHeaderLen`. +constexpr size_t kMandatoryDescriptorWorstCase = kMandatoryNonRefWorstCase + kRefFramingAndTerminator; + +static_assert(kMandatoryDescriptorWorstCase <= kMinBlobHeaderLen - 1, + "the mandatory blob-envelope fields plus the empty-ref framing must fit under kMinBlobHeaderLen " + "(the trailing '\\n' is already counted above, so the spare byte is the diagnostic ref's floor " + "budget, not the newline); if a field grew, either shrink it back or " + "raise kMinBlobHeaderLen (CasEnvelopeLimits.h) to match"); + /// The escaped byte-length of one raw ref char under the frozen envelope alphabet (see writeEnvelopeRefField). size_t escapedLen(char c) { @@ -101,6 +158,11 @@ std::string_view provenanceOpToWireWord(ProvenanceOp op) return kProvenanceOpWords.toWord(op, "CAS blob envelope"); } +ProvenanceOp provenanceOpFromWireWord(std::string_view w) +{ + return kProvenanceOpWords.fromWord(w, "CAS blob envelope"); +} + String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { if (header.kind != ObjectKind::Blob) @@ -129,7 +191,7 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { writeKey(buf, "!x", first); writeStringValue(buf, "1"); } - json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":3,...,"ch":26006001 (no ref, no closing brace) + json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":1,...,"chver":26006001 (no ref, no closing brace) } /// Optional `ref`, truncated to the exact remaining budget. Layout after this block: @@ -211,7 +273,7 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje } else if (key == EnvelopeWire::op) { - prov.op = kProvenanceOpWords.fromWord(r.readString(), "CAS blob envelope"); + prov.op = provenanceOpFromWireWord(r.readString()); have_prov = true; } else if (key == EnvelopeWire::chver) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 68c473a70c3d..0aa26e22e182 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -35,6 +35,9 @@ enum class ProvenanceOp : uint8_t /// Returns the persisted wire word for a validated provenance operation. std::string_view provenanceOpToWireWord(ProvenanceOp op); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +ProvenanceOp provenanceOpFromWireWord(std::string_view w); + /// Optional diagnostic metadata recorded with an envelope. The fields identify when and where the /// incarnation was created, the ClickHouse build that wrote it, and the operation that produced it; /// none of them participates in object identity or a protocol decision. @@ -58,19 +61,20 @@ struct Provenance /// algorithm and digest are already present in the object key and manifest reference, `domain_id` had /// no validating consumer, and `header_hash` had no consumer once the CityHash64 check left the /// envelope. Writer forensics are represented -/// by `ch` and `bld`, so a separate `writer_version` is unnecessary. The `v` field is the sole format -/// compatibility gate; a reader rejects a version it does not understand before interpreting the body. +/// by `chver` and `build`, so a separate `writer_version` is unnecessary. The `v` field is the sole +/// format compatibility gate; a reader rejects a version it does not understand before interpreting +/// the body. struct EnvelopeHeader { ObjectKind kind = ObjectKind::Blob; /// Set by decode from the header `v`; encode stamps `currentCompatibilityVersion`. A reader /// fails closed (UNKNOWN_FORMAT_VERSION) when `v` exceeds what this build understands. uint32_t compatibility_version = 0; - UInt128 incarnation_tag{}; /// `tag` - UInt128 build_id{}; /// `bld` - std::optional provenance; /// `ts` / `by` / `op` / `ch` - std::optional intended_ref; /// `ref` (diagnostic; truncated on encode to fit the header) - uint32_t header_len = 0; /// filled by encode/decode = blob_header_len (payload offset) + UInt128 incarnation_tag{}; /// `tag` + UInt128 build_id{}; /// `build` + std::optional provenance; /// `time_ms` / `creator` / `op` / `chver` + std::optional intended_ref; /// `ref` (diagnostic; truncated on encode to fit the header) + uint32_t header_len = 0; /// filled by encode/decode = blob_header_len (payload offset) /// Test-only knob: emit an unknown `!`-critical key. Decoding the resulting header must fail /// closed with `UNKNOWN_FORMAT_VERSION`, exercising the compatibility rule for critical extensions. bool emit_unknown_critical_key = false; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp index ab98b8858693..2b40584e0199 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp @@ -20,9 +20,9 @@ namespace namespace BlobMetaWire { - constexpr WireKey state{"st"}; - constexpr WireKey condemn_round{"cr"}; - constexpr WireKey size{"sz"}; + constexpr WireKey state{"state"}; + constexpr WireKey condemn_round{"condemn_round"}; + constexpr WireKey size{"size"}; } constexpr EnumWireTable kMetaStateWords{{{ @@ -39,6 +39,11 @@ std::string_view metaStateToWireWord(MetaState state) return kMetaStateWords.toWord(state, "CAS blob meta"); } +MetaState metaStateFromWireWord(std::string_view w) +{ + return kMetaStateWords.fromWord(w, "CAS blob meta"); +} + String encodeBlobMeta(const BlobMeta & meta) { CasJsonWriter out(256); @@ -71,7 +76,7 @@ BlobMeta decodeBlobMeta(std::string_view bytes) { if (key == BlobMetaWire::state) { - m.state = kMetaStateWords.fromWord(r.readString(), "CAS blob meta"); + m.state = metaStateFromWireWord(r.readString()); saw_state = true; } else if (key == BlobMetaWire::condemn_round) @@ -82,7 +87,7 @@ BlobMeta decodeBlobMeta(std::string_view bytes) r.skipUnknown(key); } if (!saw_state) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing st"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing state"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: trailing bytes"); return m; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h index 5290e7c831db..9176dc641203 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h @@ -19,8 +19,13 @@ enum class MetaState : uint8_t /// so a writer may republish it by replacing the body and updating this marker. }; +/// Convert a meta-state discriminator to its canonical wire word. Throws `LOGICAL_ERROR` for an +/// out-of-range enum value. std::string_view metaStateToWireWord(MetaState state); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +MetaState metaStateFromWireWord(std::string_view w); + /// The durable per-hash meta record. Its text representation consists of a format header followed by /// one JSON object with the state word, the GC condemnation round, and the raw body size. `size` is /// retained for introspection, fsck, and GC accounting; reads of the blob never consult the meta. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h index df8cee44bc47..92e6b252ab1b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h @@ -8,7 +8,9 @@ namespace DB::Cas /// The pool-wide floor for `blob_header_len`. One compile-time owner, read by BOTH /// `validatePoolBlobHeaderLen` (pool creation / decode) and the blob-envelope codec, so the /// mandatory-descriptor worst-case proof and the enforced floor can never guard different numbers. -/// The derivation of the floor lives in `CasPoolMetaFormat.cpp` next to the worst-case table. +/// The byte-for-byte worst-case derivation (`kMandatoryDescriptorWorstCase`) lives beside the +/// envelope key constants in `CasBlobEnvelopeFormat.cpp`; `CasPoolMetaFormat.cpp` records why 240 +/// (rather than the bare worst case) was chosen as the floor. inline constexpr uint64_t kMinBlobHeaderLen = 240; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index 2324cc2c037a..129c4b5c3493 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -26,32 +27,32 @@ namespace namespace FoldSealWire { - constexpr WireKey generation{"g"}; - constexpr WireKey parent_generation{"pg"}; - constexpr WireKey kind{"k"}; + constexpr WireKey generation{"generation"}; + constexpr WireKey parent_generation{"parent_generation"}; + constexpr WireKey kind{"kind"}; constexpr WireKey run_key{"key"}; - constexpr WireKey checksum{"ck"}; + constexpr WireKey checksum{"checksum"}; constexpr WireKey shard{"shard"}; - constexpr WireKey key_generation{"gen"}; + constexpr WireKey key_generation{"key_generation"}; constexpr WireKey life{"life"}; - constexpr WireKey classification{"cls"}; - constexpr WireKey fold_epoch{"lfe"}; - constexpr WireKey fold_seq{"lfs"}; - constexpr WireKey hold_reason{"hr"}; - constexpr WireKey hold_epoch{"hpe"}; - constexpr WireKey hold_seq{"hps"}; - constexpr WireKey retries{"hrc"}; - constexpr WireKey retry_round{"hnr"}; - constexpr WireKey remove_epoch{"rte"}; - constexpr WireKey remove_seq{"rts"}; - constexpr WireKey condemned_total{"ct"}; - constexpr WireKey pending_total{"pt"}; - constexpr WireKey oldest_round{"ocr"}; + constexpr WireKey classification{"class"}; + constexpr WireKey fold_epoch{"fold_epoch"}; + constexpr WireKey fold_seq{"fold_seq"}; + constexpr WireKey hold_reason{"hold_reason"}; + constexpr WireKey hold_epoch{"hold_epoch"}; + constexpr WireKey hold_seq{"hold_seq"}; + constexpr WireKey retries{"retries"}; + constexpr WireKey retry_round{"retry_round"}; + constexpr WireKey remove_epoch{"remove_epoch"}; + constexpr WireKey remove_seq{"remove_seq"}; + constexpr WireKey condemned_total{"condemned"}; + constexpr WireKey pending_total{"pending"}; + constexpr WireKey oldest_round{"oldest_round"}; } -constexpr std::string_view kRefLifeTag = "rfl"; -constexpr std::string_view kBlobRunTag = "btr"; -constexpr std::string_view kCondemnedTag = "cnd"; +constexpr std::string_view kRefLifeTag = "ref_life"; +constexpr std::string_view kBlobRunTag = "blob_run"; +constexpr std::string_view kCondemnedTag = "condemned"; constexpr EnumWireTable kHoldReasonWords{{{ {HoldReason::GapBelowWitness, "gap_below_witness"}, @@ -64,20 +65,14 @@ constexpr EnumWireTable kHoldReasonWords{{{ static_assert(casEnumTableCoversEnum()); -HoldReason holdReasonFromWord(std::string_view w) -{ - return kHoldReasonWords.fromWord(w, "CAS fold seal hold reason"); -} +constexpr EnumWireTable kCoverageClassWords{{{ + {CoverageClass::Absent, "absent"}, + {CoverageClass::Unchanged, "unchanged"}, + {CoverageClass::Folded, "folded"}, + {CoverageClass::Clamped, "clamped"}, +}}}; -/// The classification set is CLOSED. Every consumer of a coverage row branches on exact values — the -/// sweep's §6 deletion premise refuses a row by testing `== 4` and then `== 0` — so a value outside the -/// set is not an unknown variant to be tolerated forward: it is a row that passes every refusal written -/// in terms of the set and reaches the irreversible delete. One predicate, used by both directions, so -/// the writer's self-check and the reader's fail-close can never name different sets. -bool isKnownClassification(uint64_t classification) -{ - return classification == 0 || classification == 1 || classification == 2 || classification == 4; -} +static_assert(casEnumTableCoversEnum()); /// A hold names a position the fold must resolve, and both components of that id are nonzero (the /// canonical `RefTxnId` rule `renderRefTxnId` enforces for every id that becomes a key). A zero @@ -104,7 +99,7 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi what, key); } -/// Emit one run record (`k` = "btr") WITHOUT its line terminator; the caller closes (and measures) the +/// Emit one run record (`kind` = `blob_run`) WITHOUT its line terminator; the caller closes (and measures) the /// line, and sorts the vector by key first. void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { @@ -180,6 +175,36 @@ std::string_view holdReasonToWord(HoldReason r) return kHoldReasonWords.toWord(r, "CAS fold seal hold reason"); } +HoldReason holdReasonFromWord(std::string_view w) +{ + return kHoldReasonWords.fromWord(w, "CAS fold seal hold reason"); +} + +std::string_view coverageClassToWord(CoverageClass c) +{ + return kCoverageClassWords.toWord(c, "CAS fold seal classification"); +} + +CoverageClass coverageClassFromWord(std::string_view w) +{ + return kCoverageClassWords.fromWord(w, "CAS fold seal classification"); +} + +namespace +{ +/// A seal can carry thousands of `ref_life` rows, so a rejected word names the row that carries it -- +/// "unknown word" alone leaves an operator scanning the object by hand. +CoverageClass coverageClassInRow(std::string_view w, std::string_view life_hex) +{ + return kCoverageClassWords.fromWord(w, fmt::format("CAS fold seal: ref_life '{}' classification", life_hex)); +} + +HoldReason holdReasonInRow(std::string_view w, std::string_view life_hex) +{ + return kHoldReasonWords.fromWord(w, fmt::format("CAS fold seal: ref_life '{}' hold_reason", life_hex)); +} +} + FoldSealCaps foldSealCaps() { const FormatTraits & t = traitsFor(FormatId::FoldSeal); @@ -253,22 +278,19 @@ String encodeFoldSeal(const CasFoldSeal & seal) /// process, not corruption arriving from a store — and none of these shapes is repairable once /// durable, so none is ever written. /// - /// A classification outside the closed set first, because the two checks after it are stated in - /// terms of the set and a row they cannot classify makes their answers meaningless. - if (!isKnownClassification(cov.classification)) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS fold seal: coverage '{}' has classification {}, which is not one of the four the " - "fold grammar defines (0 absent, 1 unchanged, 2 folded, 4 clamped) — every consumer " - "branches on those exact values, so this row would pass refusals meant to stop it", - life_hex, cov.classification); - /// A classification-4 row whose hold was dropped is indistinguishable, once durable, from a - /// namespace that stopped for no reason — and a hold on any other classification claims a stop - /// that did not happen. - if ((cov.classification == 4) != cov.hold.has_value()) + /// A classification outside the four named values first, because the two checks after it are + /// stated in terms of those names and a row they cannot classify makes their answers meaningless. + /// `coverageClassToWord` IS that range check (it throws `LOGICAL_ERROR` for a value the table + /// does not index), so capturing its result here also gives the record its wire value below. + const std::string_view classification_word = coverageClassToWord(cov.classification); + /// A clamped row whose hold was dropped is indistinguishable, once durable, from a namespace that + /// stopped for no reason — and a hold on any other classification claims a stop that did not + /// happen. + if ((cov.classification == CoverageClass::Clamped) != cov.hold.has_value()) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS fold seal: coverage '{}' has classification {} and {} hold — the hold fields are " - "required for classification 4 and forbidden otherwise", - life_hex, cov.classification, cov.hold ? "a" : "no"); + "required for classification clamped and forbidden otherwise", + life_hex, classification_word, cov.hold ? "a" : "no"); /// A hold that names no position resolves itself on the next round (nothing sorts below /// `{0, 0}`) and cannot be rendered where the sweep reports why it retained a manifest. if (cov.hold && !isCanonicalHoldPosition(cov.hold->offending_position)) @@ -291,7 +313,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) bool first = true; writeStringField(out, FoldSealWire::kind, kRefLifeTag, first); writeHex128Field(out, FoldSealWire::life, life_id, first); - writeNumberField(out, FoldSealWire::classification, static_cast(cov.classification), first); + writeStringField(out, FoldSealWire::classification, classification_word, first); writeU64StringField(out, FoldSealWire::fold_epoch, cov.last_folded_ref_id.writer_epoch, first); writeU64StringField(out, FoldSealWire::fold_seq, cov.last_folded_ref_id.ref_sequence, first); if (cov.hold) @@ -308,7 +330,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) writeU64StringField(out, FoldSealWire::remove_seq, life_state.cleanup_evidence->remove_txn_id.ref_sequence, first); } closeObject(out, first); - closeLine("rfl"); + closeLine(kRefLifeTag); ++n; } @@ -318,7 +340,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const RunRef & r : runs) { writeRun(out, kBlobRunTag, r); - closeLine("btr"); + closeLine(kBlobRunTag); } } n += seal.blob_target_runs.size(); @@ -333,7 +355,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) writeNumberField(out, FoldSealWire::pending_total, s.pending_total, first); writeU64StringField(out, FoldSealWire::oldest_round, s.oldest_nonpending_condemn_round, first); closeObject(out, first); - closeLine("cnd"); + closeLine(kCondemnedTag); ++n; } @@ -395,21 +417,18 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect return seal; } if (key != FoldSealWire::kind) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"k\""); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"kind\""); const String kind = r.readString(); if (kind == kRefLifeTag) { std::optional life_id; RefCoverage cov; - /// Read WIDE and validated before it is narrowed to the persisted byte. `cls` is the field - /// every consumer branches on, and a plain `static_cast` maps 258 onto 2 ("all - /// records through the cursor were folded") and 256 onto 0 — a forged or damaged seal would - /// buy full coverage with an integer no reader ever sees. - std::optional classification; /// The hold fields are read individually so the grammar can be checked on WHICH of them /// arrived, not merely on how many. `JsonObjectReader` already rejects a duplicate key, so - /// a second `hr` can never quietly rewrite the reason. + /// a second `hold_reason` can never quietly rewrite the reason. + std::optional classification_word; + std::optional hold_reason_word; std::optional hold_reason; std::optional hold_epoch; std::optional hold_sequence; @@ -420,17 +439,20 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect while (r.nextKey(key)) { if (key == FoldSealWire::life) life_id = r.readHex128(); - else if (key == FoldSealWire::classification) classification = r.readU64Number(); + /// The two word-valued fields are collected as words and converted BELOW, once the row's + /// life id is known: a seal can carry thousands of rows, so an unknown word has to say + /// WHICH row carries it. + else if (key == FoldSealWire::classification) classification_word = r.readString(); else if (key == FoldSealWire::fold_epoch) cov.last_folded_ref_id.writer_epoch = r.readU64String(); else if (key == FoldSealWire::fold_seq) cov.last_folded_ref_id.ref_sequence = r.readU64String(); - else if (key == FoldSealWire::hold_reason) hold_reason = holdReasonFromWord(r.readString()); + else if (key == FoldSealWire::hold_reason) hold_reason_word = r.readString(); else if (key == FoldSealWire::hold_epoch) hold_epoch = r.readU64String(); else if (key == FoldSealWire::hold_seq) hold_sequence = r.readU64String(); else if (key == FoldSealWire::retries) hold_retry_count = r.readU32Number(); else if (key == FoldSealWire::retry_round) hold_next_retry_round = r.readU64String(); else if (key == FoldSealWire::remove_epoch) remove_txn_epoch = r.readU64String(); else if (key == FoldSealWire::remove_seq) remove_txn_sequence = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown rfl key '{}'", key); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown ref_life key '{}'", key); } if (!life_id || *life_id == 0) @@ -438,16 +460,13 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect "CAS fold seal: a ref-life row is missing a nonzero opaque life id"); const String life_hex = u128ToHex(*life_id); - /// `cls` is required, not defaulted: an absent one would read as 0 ("no round folded this - /// namespace"), which is a claim about a fold, not the absence of one. - if (!classification) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: rfl '{}' missing cls", life_hex); - if (!isKnownClassification(*classification)) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: coverage '{}' has classification {}, which is not one of the four " - "the fold grammar defines (0 absent, 1 unchanged, 2 folded, 4 clamped)", - life_hex, *classification); - cov.classification = static_cast(*classification); /// in range, so narrowing is exact + /// `class` is required, not defaulted: an absent one would read as `absent` ("no round + /// folded this namespace"), which is a claim about a fold, not the absence of one. + if (!classification_word) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: ref_life '{}' missing class", life_hex); + if (hold_reason_word) + hold_reason = holdReasonInRow(*hold_reason_word, life_hex); + cov.classification = coverageClassInRow(*classification_word, life_hex); /// The same strict grammar the encoder enforces, applied to bytes we did not write. A /// PARTIAL hold is corruption, never a hold with defaults: a hold whose offending position @@ -456,11 +475,11 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect || hold_retry_count || hold_next_retry_round; const bool every_hold_field = hold_reason && hold_epoch && hold_sequence && hold_retry_count && hold_next_retry_round; - if (cov.classification == 4) + if (cov.classification == CoverageClass::Clamped) { if (!every_hold_field) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: coverage '{}' is held (classification 4) but its hold is " + "CAS fold seal: coverage '{}' is held (classification clamped) but its hold is " "incomplete — reason, offending position, retry count and next retry round are " "all required", life_hex); /// PRESENT is not enough: the position must be one a fold can actually retry. `{0, 0}` @@ -480,8 +499,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect else if (any_hold_field) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: coverage '{}' carries hold fields at classification {} — they are " - "forbidden on anything but a held (classification 4) row", - life_hex, cov.classification); + "forbidden on anything but a held (classification clamped) row", + life_hex, coverageClassToWord(cov.classification)); const bool any_cleanup_field = remove_txn_epoch || remove_txn_sequence; const bool every_cleanup_field = remove_txn_epoch && remove_txn_sequence; @@ -518,7 +537,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect } if (!run_key || !checksum || !shard || !generation) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: btr requires key, ck, shard, and gen"); + "CAS fold seal: blob_run requires key, checksum, shard, and key_generation"); seal.blob_target_runs.push_back(RunRef{ .key = std::move(*run_key), .checksum = *checksum, .shard = *shard, .key_generation = *generation}); } @@ -534,11 +553,11 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect else if (key == FoldSealWire::condemned_total) condemned_total = r.readU64Number(); else if (key == FoldSealWire::pending_total) pending_total = r.readU64Number(); else if (key == FoldSealWire::oldest_round) oldest_nonpending_condemn_round = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown cnd key '{}'", key); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown condemned key '{}'", key); } if (!shard || !condemned_total || !pending_total || !oldest_nonpending_condemn_round) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: cnd requires shard, ct, pt, and ocr"); + "CAS fold seal: condemned requires shard, condemned, pending, and oldest_round"); insertRecordOnce(seal.condemned_summary, *shard, CondemnedSummary{ .condemned_total = *condemned_total, .pending_total = *pending_total, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h index ea4200e29eea..02a043e9b2b2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h @@ -36,10 +36,12 @@ struct RunRef /// correlating logs. Persisted as a word, so an unknown word is `CORRUPTED_DATA` rather than a silently /// reinterpreted integer. /// -/// THESE ARE WIRE VALUES, AND THEY ARE APPEND-ONLY. A durable seal written by one build is read by -/// another, so a renumbered value or a reused word makes an older seal describe a hold that is not the -/// one it recorded — and a hold's whole job is to say truthfully what stopped a namespace and where. -/// Add new reasons at the end; never renumber, never repurpose a retired word. +/// THE WORDS ARE THE WIRE, AND THE WORD VOCABULARY IS APPEND-ONLY. A durable seal written by one build +/// is read by another, so a reused or repurposed word makes an older seal describe a hold that is not +/// the one it recorded — and a hold's whole job is to say truthfully what stopped a namespace and +/// where. The enumerator NUMBERS never leave memory; they are constrained only by the wire table's +/// density-and-order proof, so inserting a value in the middle is a compile-time question, not a +/// durability one. Add new reasons freely; never reuse or repurpose a retired word. enum class HoldReason : uint8_t { GapBelowWitness = 1, /// 404 at the expected id with a durable witness above it, same epoch @@ -55,6 +57,34 @@ enum class HoldReason : uint8_t /// namespace — and a second rendering of these words elsewhere would be a second place for them to drift. std::string_view holdReasonToWord(HoldReason r); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. Paired with the renderer so a caller +/// that needs to prove the vocabulary round-trips does not have to reach for the table itself. +HoldReason holdReasonFromWord(std::string_view w); + +/// What the current round did for one life-keyed `CasFoldSeal::ref_lives` row. A BOUNDED enum: the type +/// itself is the closed set, so a producer cannot construct a fifth shape without an explicit cast, and +/// the decoder's wire-word lookup refuses anything else as `CORRUPTED_DATA` rather than silently +/// reinterpreting an integer. +/// +/// THE WORDS ARE THE WIRE, AND THE WORD VOCABULARY IS APPEND-ONLY, for the same reason `HoldReason`'s +/// is: a durable seal written by one build is read by another. The enumerator numbers never leave +/// memory. `Clamped` sits at 3, not the 4 a retired byte-valued wire used for it: nothing outside this +/// JSON ever persisted the raw byte, and a dense range is what makes the wire table's lookup a direct +/// index rather than a search. +enum class CoverageClass : uint8_t +{ + Absent = 0, /// no round has folded a ref cursor for this namespace + Unchanged = 1, /// folded, but nothing moved this round + Folded = 2, /// every record through the observed cursor was folded + Clamped = 3, /// folding stopped below the ref-log cursor; must be read again next round +}; + +/// The wire word one `CoverageClass` is persisted as; `fromWord` rejects anything else as +/// `CORRUPTED_DATA`. Exported for the same reason `holdReasonToWord` is: the classification is rendered +/// outside the codec too (`cas-inspect`, the sweep's retention messages). +std::string_view coverageClassToWord(CoverageClass c); +CoverageClass coverageClassFromWord(std::string_view w); + /// The durable hold on one namespace. It rides `RefCoverage` across rounds and across `REBUILD`, and /// clears ONLY by folding through `offending_position` and adopting the result in `gc/state` — never by /// observing another absent, because an absent is exactly the observation a lying store produces. @@ -87,21 +117,17 @@ struct RefHold bool operator==(const RefHold &) const = default; }; -/// Records what the current round did for one life-keyed `CasFoldSeal::ref_lives` row. -/// `classification` is a persisted byte: -/// 0 means absent, 1 means unchanged, 2 means all records through the observed cursor were folded, and 4 -/// means folding was clamped below the ref-log cursor. A clamped entry must be read again in the next -/// round, because an unfolded event may become foldable by then. +/// Records what the current round did for one life-keyed `CasFoldSeal::ref_lives` row. See +/// `CoverageClass` for what each value means. /// -/// THE SET {0, 1, 2, 4} IS CLOSED, and both codecs enforce it (decode `CORRUPTED_DATA`, encode -/// `LOGICAL_ERROR`). Every consumer branches on exact values — the sweep's §6 deletion premise refuses a -/// row by testing `== 4` and `== 0` — so an unrecognized byte is not a variant to tolerate: it passes -/// every refusal stated in terms of the set and reaches the delete. The decoder also validates BEFORE -/// narrowing to the byte, because a wide integer on the wire (258, say) truncates into the set and would -/// otherwise claim a coverage the fold never proved. +/// Every consumer branches on exact values — the sweep's §6 deletion premise refuses a row by testing +/// `== Clamped` and `== Absent` — so the CLOSED set matters beyond the codec, and it is now the type +/// itself: `CoverageClass` names exactly the four shapes, both codecs go through the shared wire table +/// (decode `CORRUPTED_DATA`, encode `LOGICAL_ERROR` on the one path that still reaches an out-of-range +/// value, an explicit cast), and an unrecognized wire word is refused before it ever reaches a consumer. struct RefCoverage { - uint8_t classification = 0; + CoverageClass classification = CoverageClass::Absent; /// The greatest `RefTxnId` whose owner changes have contributed their manifest-edge deltas. There is /// one ref-log stream per namespace life, so this cursor is stored in that life-keyed row. @@ -109,11 +135,11 @@ struct RefCoverage /// offending transaction so the complete transaction is retried rather than partially applied. RefTxnId last_folded_ref_id{}; - /// STRICT GRAMMAR: present if and only if `classification == 4`. Both directions enforce it — the - /// encoder refuses to write a classification-4 row without a hold (a clamp whose reason was lost is - /// indistinguishable from a clean cursor once it is durable) and refuses to write a hold on any - /// other classification (`LOGICAL_ERROR`); the decoder rejects both shapes as `CORRUPTED_DATA`. The - /// pairing lives in the type, not only in the codec, so no producer can construct the forbidden + /// STRICT GRAMMAR: present if and only if `classification == CoverageClass::Clamped`. Both directions + /// enforce it — the encoder refuses to write a clamped row without a hold (a clamp whose reason was + /// lost is indistinguishable from a clean cursor once it is durable) and refuses to write a hold on + /// any other classification (`LOGICAL_ERROR`); the decoder rejects both shapes as `CORRUPTED_DATA`. + /// The pairing lives in the type, not only in the codec, so no producer can construct the forbidden /// combination by forgetting a field. std::optional hold = std::nullopt; @@ -189,10 +215,10 @@ FoldSealCaps foldSealCaps(); void checkFoldSealObjectBytes(uint64_t encoded_bytes); /// Encodes a fold seal as a strict, raw text control object. The header and meta lines are followed by -/// tagged records in the fixed `rfl`/`btr`/`cnd` order and a record-count trailer. Map iteration and +/// tagged records in the fixed `ref_life`/`blob_run`/`condemned` order and a record-count trailer. Map iteration and /// run references are sorted so retries produce byte-identical output for write-once adoption. /// -/// Enforces the whole coverage grammar — the closed classification set, the classification-4 hold +/// Enforces the whole coverage grammar — the closed classification set, the clamped-classification hold /// pairing, and the hold's canonical offending position — and BOTH byte caps: every emitted line against /// `line_cap` — header, meta, records and trailer alike, with no exception — and the whole object /// against `object_cap`. Both PUT sites go through this function, so the gate cannot be bypassed by diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp index dc9665aff837..23b19ddc3508 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp @@ -18,57 +18,13 @@ namespace DB::Cas namespace { -/// Generation-1 baseline for every class. A future format change appends to that class's array and +/// Generation-1 baseline for every class. A future format change appends to that class's own array and /// bumps `G_BUILD`: additive changes use the previous reader floor, while breaking changes use the -/// new generation as the floor. Existing entries are immutable history. +/// new generation as the floor. Existing entries are immutable history. Every class currently shares +/// this baseline; a class that outgrows it gets its own named array again, the way the pre-reset +/// history once had. constexpr FormatChangePoint BASELINE[] = {{1, 1}}; -/// The two ref classes changed at generation 4 (INV-1, per-namespace contiguous ids) AND AGAIN at -/// generation 5 (Stage B's recreate-only "format bump B": the ref layer re-keyed under -/// `//`). Both changes are BREAKING even though not one byte of the encoding moved -/// either time -- a generation-3 stream's ids came from a pool-wide counter and legitimately skip, -/// which a generation-4 reader reports as corruption, and a generation-4 key names no incarnation at -/// all, which a generation-5 reader also reports as corruption (`Layout::parseRefObjectKey`). Each -/// floor is the change generation itself. -constexpr FormatChangePoint REF_STREAM[] = { - {1, 1}, - {kContiguousRefStreamsGeneration, kContiguousRefStreamsGeneration}, - {kNamespaceLifeKeyedGeneration, kNamespaceLifeKeyedGeneration}, - {kOpaqueNamespaceLifeLayoutGeneration, kOpaqueNamespaceLifeLayoutGeneration}, -}; - -/// `cas_ref_ckpt` is BORN at generation 4, so it has no generation-1 baseline to inherit: there is no -/// such thing as a generation-1 `_ckpt` object, and claiming one would say a generation-1 reader could -/// read it. Generation 5 re-keys it under `//` exactly like `REF_STREAM` above, for -/// the same reason and with the same floor. -constexpr FormatChangePoint REF_CKPT[] = { - {kContiguousRefStreamsGeneration, kContiguousRefStreamsGeneration}, - {kNamespaceLifeKeyedGeneration, kNamespaceLifeKeyedGeneration}, - {kOpaqueNamespaceLifeLayoutGeneration, kOpaqueNamespaceLifeLayoutGeneration}, - {kCommittedRefFrontierGeneration, kCommittedRefFrontierGeneration}, -}; - -/// `cas_ref_catalog` is BORN at generation 4, one generation BEFORE the bump that makes namespace -/// existence catalog-authoritative (Stage B's Task 4, "format bump B" -- `kNamespaceLifeKeyedGeneration`): -/// Task 2 introduced the catalog OBJECT while `G_BUILD` was still the value -/// `kContiguousRefStreamsGeneration` names, and Task 4 is the later change that actually wires -/// discovery to read it and bumps the floor. The catalog's own encoding is unaffected by that bump (it -/// reuses `kContiguousRefStreamsGeneration` as its birth generation, not a second constant named after -/// itself, for the same reason `REF_CKPT` originally did), so it carries no second change point here. -constexpr FormatChangePoint REF_CATALOG[] = {{kContiguousRefStreamsGeneration, kContiguousRefStreamsGeneration}}; -constexpr FormatChangePoint GC_MAINTENANCE_STATE[] = {{kUnifiedRefLifeFoldGeneration, kUnifiedRefLifeFoldGeneration}}; -constexpr FormatChangePoint POOL_META[] = { - {1, 1}, - {kPoolGcShardsGeneration, kPoolGcShardsGeneration}, - {kCommittedRefFrontierGeneration, kCommittedRefFrontierGeneration}, - {kMountWriteAttemptIdGeneration, kMountWriteAttemptIdGeneration}, -}; - -constexpr FormatChangePoint MOUNT_LEASE[] = { - {1, 1}, - {kMountWriteAttemptIdGeneration, kMountWriteAttemptIdGeneration}, -}; - } std::span changePoints(FormatId id) @@ -77,17 +33,11 @@ std::span changePoints(FormatId id) { case FormatId::RefLog: case FormatId::RefSnapshot: - return REF_STREAM; case FormatId::RefCkpt: - return REF_CKPT; case FormatId::RefCatalog: - return REF_CATALOG; case FormatId::GcMaintenanceState: - return GC_MAINTENANCE_STATE; case FormatId::PoolMeta: - return POOL_META; case FormatId::MountLease: - return MOUNT_LEASE; case FormatId::Blob: case FormatId::GcState: case FormatId::Roster: diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h index 1c98e4f283f6..7b287a01ae9d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h @@ -15,87 +15,10 @@ namespace DB::Cas /// compatibility_version <= G_BUILD. Bump this (and append a change-point in CasFormat.cpp) when a new /// format generation is introduced. /// -/// Generation 2 is the first generation that understands mixed-algorithm pools: the schema-3 -/// source-edge settlement key includes the algorithm prefix, so a generation-1 reader can open the -/// pool but cannot decode its GC state. Pool admission CAS-raises `min_reader_generation` to this -/// build's own floor (`G_BUILD`), and a persisted floor above `G_BUILD` fails closed. -/// -/// Generation 3 replaced mutable ref-shard objects with immutable `_log` and `_snap` objects. -/// -/// Generation 4 makes each namespace's ref-log ids per-namespace and CONTIGUOUS within a writer epoch -/// (INV-1). The bytes of a `_log`/`_snap` object did not change, but their MEANING did: a generation-3 -/// pool's ids were drawn from a pool-wide counter and are full of legitimate holes, which this build -/// reads as a truncated -- i.e. corrupt -- stream. The per-object forward gate cannot reject such a -/// pool (its version is not in the future), so pool-meta decoding applies -/// `kContiguousRefStreamsGeneration` as a backward floor. Pools below the floor must be recreated; -/// there is no migration path in the pre-release format. -/// -/// Generation 5 (Stage B's own recreate-only bump, the plan's "format bump B") re-keys the ref layer -/// under `//` (spec INV-3: the whole-pool namespace catalog mints the incarnation). -/// Again the bytes of `_log`/`_snap`/`_ckpt` objects did not change, but the KEY SHAPE they live under -/// did: a generation-4 key named a namespace directly (`cas/refs//_log/`), while this -/// generation's reader recognizes only the incarnation-qualified shape -/// (`cas/refs///_log/`) -- `Layout::parseRefObjectKey`/`parseRefCkptKey` already -/// refuse the un-incarnated shape with `CORRUPTED_DATA` (Stage B Tasks 1/1c landed that refusal ahead -/// of this bump, deliberately: the pre-release format carries zero persisted data and zero compat -/// obligation, so the key shapes and the bump that makes them the ONLY readable shape need not land in -/// the same commit). `kNamespaceLifeKeyedGeneration` is the backward floor for this change, applied the -/// same way `kContiguousRefStreamsGeneration` is. -/// -/// Generation 6 replaces that namespace-bearing grammar with opaque pool-wide life identifiers and -/// splits hot ref streams from point-read state: `cas/ns/stream//...` contains `_log`, `_snap` -/// while `cas/ns/state//...` contains `_ckpt` and `_files`. A generation-5 -/// pool must be recreated; no dual parser or copy-forward path exists. -/// -/// Generation 7 replaces the fold seal's independent namespace-keyed coverage and cleanup -/// collections with one opaque-life-keyed row and removes the retired terminal-marker object class. A generation-6 -/// pool must be recreated; there is no dual reader for the split grammar. -/// -/// Generation 8 persists the creation-time `gc_shards` authority in `_pool_meta`. Generation-7 pools -/// must be recreated because namespace admission can precede creation of `gc/state`; accepting a -/// metadata object without this field would leave different openers charging different seal bounds. -/// Generation 9 adds `_ckpt.committed_through`, the exact recovery frontier. Generation-8 pools -/// must be recreated: the absence of this field has the incompatible meaning that no transaction has -/// entered durable logical history. Generation 10 adds the required `write_attempt_id` to mount -/// leases. Generation-9 pools must be recreated because a missing attempt identity makes ambiguous -/// mount writes impossible to distinguish from a different body under the same writer incarnation. -constexpr uint32_t G_BUILD = 10; - -/// The pool-format generation at which ref-log ids became per-namespace and contiguous. Pool metadata -/// below this value cannot be opened, because its ref streams carry holes this build reports as -/// corruption; the backward-floor check is applied by `decodePoolMeta`. Named separately from `G_BUILD` -/// so a later generation that CAN still read a generation-4 pool does not silently move the floor with -/// it. -constexpr uint32_t kContiguousRefStreamsGeneration = 4; - -/// The pool-format generation at which the ref layer (and, per Stage B's Task 4b, namespace files) -/// became incarnation-scoped under `//`. Pool metadata below this value cannot be -/// opened: its ref-object keys carry no incarnation segment, which this build's parsers refuse as -/// corruption rather than read as a compatibility case (see the `G_BUILD` doc above). The backward- -/// floor check is applied by `decodePoolMeta`, exactly mirroring `kContiguousRefStreamsGeneration`; -/// named separately for the same reason that one is -- so a later generation that can still read a -/// generation-5 pool does not silently move this floor with it. Pools below the floor must be -/// recreated; there is no migration path in the pre-release format. -constexpr uint32_t kNamespaceLifeKeyedGeneration = 5; - -/// The recreate-only generation at which namespace text disappeared from physical life keys and hot -/// ref streams were separated from point-read namespace state. -constexpr uint32_t kOpaqueNamespaceLifeLayoutGeneration = 6; - -/// The recreate-only generation at which one unified ref-life row replaced the split coverage and -/// namespace-cleanup grammar. -constexpr uint32_t kUnifiedRefLifeFoldGeneration = 7; - -/// The recreate-only generation at which `_pool_meta` became the authority for `gc_shards`. -constexpr uint32_t kPoolGcShardsGeneration = 8; - -/// The recreate-only generation at which `_ckpt` gained its exact committed-transaction frontier. -constexpr uint32_t kCommittedRefFrontierGeneration = 9; - -/// The recreate-only generation at which mount leases gained their required durable holder-write -/// identity. The pool-level reader floor rejects every older pool before it can interpret a mount -/// body without this field. -constexpr uint32_t kMountWriteAttemptIdGeneration = 10; +/// The generation history was reset to this {1, 1} baseline: CAS is pre-release, carries no persisted +/// data, and so pays no compatibility cost for starting the count over. Every class's `changePoints` +/// begins at generation 1 until a future change appends a real entry. +constexpr uint32_t G_BUILD = 1; /// Stable identifiers for every self-describing persisted object class. The text registry uses the /// corresponding `type` string as the on-disk identity. Numeric values are part of the format history: @@ -151,20 +74,20 @@ void checkCompatibility(uint32_t compatibility_version, std::string_view what); /// One append-only entry in a class's format history. At `generation`, the class's ENCODING or the /// MEANING of what it encodes changed, and a reader must understand at least `min_reader` to read an /// object written at that generation. Additive changes retain the previous reader floor; breaking -/// changes set the floor to the change generation itself. Generation 4's ref-stream entry is the -/// worked example of the second kind: not one byte of `cas_ref_log` moved, but its ids became dense, -/// so an older stream is unreadable to this build and the floor is the change generation. +/// changes set the floor to the change generation itself — even when not one byte of the encoding +/// moves: a change that makes ids dense, for example, leaves the bytes readable but their MEANING +/// unreadable to an older build, so the floor is the change generation. struct FormatChangePoint { uint16_t generation; uint16_t min_reader; }; -/// Returns the append-only change-point history for `id`, oldest first. A class's history begins at -/// the generation it was BORN in, not at 1: the classes that existed from the start carry the frozen -/// `{1, 1}` baseline, while `RefCkpt` — introduced at generation 4 — begins at `{4, 4}`, because there -/// is no such thing as a generation-1 `_ckpt` and claiming one would say a generation-1 reader could -/// read it. Future changes append entries without editing old ones. +/// Returns the append-only change-point history for `id`, oldest first. After the pre-release +/// generation reset every class carries the shared `{1, 1}` baseline. A class born LATER than the +/// current baseline must begin its history at its birth generation, not at 1 — claiming an earlier +/// entry would say an older reader could read an object kind that did not yet exist. Future changes +/// append entries without editing old ones. std::span changePoints(FormatId id); /// The text-format registry has one row per decodable persisted object. Each row is the single source diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp index bc0720bf6f27..cc13021b020d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp @@ -15,7 +15,7 @@ namespace DB::Cas namespace GcMaintenanceWire { - constexpr WireKey janitor_cursor{"cur"}; + constexpr WireKey janitor_cursor{"janitor_cursor"}; } String encodeGcMaintenanceState(const GcMaintenanceState & state) @@ -59,7 +59,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data) reader.skipUnknown(key); } if (!has_cursor) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: missing cur"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: missing janitor_cursor"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: trailing bytes"); if (result.janitor_cursor.size() > kMaxGcMaintenanceCursorBytes) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index 9ee65ebafa78..1b19256d6d4b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -21,8 +21,8 @@ namespace namespace GcOutcomesWire { - constexpr WireKey kind{"k"}; - constexpr WireKey outcome{"oc"}; + constexpr WireKey kind{"kind"}; + constexpr WireKey outcome{"outcome"}; } constexpr EnumWireTable kOutcomeKindWords{{{ @@ -34,11 +34,6 @@ constexpr EnumWireTable kOutcomeKindWords{{{ static_assert(casEnumTableCoversEnum()); -OutcomeKind outcomeKindFromWord(std::string_view w) -{ - return kOutcomeKindWords.fromWord(w, "CAS outcome log outcome kind"); -} - } std::string_view outcomeKindToWireWord(OutcomeKind outcome) @@ -46,6 +41,11 @@ std::string_view outcomeKindToWireWord(OutcomeKind outcome) return kOutcomeKindWords.toWord(outcome, "CAS outcome log outcome kind"); } +OutcomeKind outcomeKindFromWireWord(std::string_view w) +{ + return kOutcomeKindWords.fromWord(w, "CAS outcome log outcome kind"); +} + String encodeOutcomeLog(const OutcomeLog & log) { CasJsonWriter out(256); @@ -54,8 +54,8 @@ String encodeOutcomeLog(const OutcomeLog & log) { bool first = true; writeStringField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); - writeBlobRefFields(out, first, e.ref); /// ha + h - writeTokenFields(out, first, e.token); /// tt + tv + writeBlobRefFields(out, first, e.ref); /// algo + digest + writeTokenFields(out, first, e.token); /// token_type + token writeStringField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); closeObject(out, first); writeChar('\n', out); @@ -78,7 +78,7 @@ OutcomeLog decodeOutcomeLog(std::string_view data) JsonObjectReader r(line_in, KeyStrictness::Tolerant, "outcome log"); String key; - /// The first key distinguishes a trailer ("n") from a record ("k"). + /// The first key distinguishes a trailer (`n`) from a record (`kind`). if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: empty line"); if (key == "n") @@ -102,14 +102,12 @@ OutcomeLog decodeOutcomeLog(std::string_view data) if (key == GcOutcomesWire::kind) e.kind = objectKindFromWord(r.readString(), "outcome log"); else if (matchBlobRefFields(key, r, blob_ref_fields)) {} else if (matchTokenFields(key, r, token_fields)) {} - else if (key == GcOutcomesWire::outcome) e.outcome = outcomeKindFromWord(r.readString()); + else if (key == GcOutcomesWire::outcome) e.outcome = outcomeKindFromWireWord(r.readString()); else r.skipUnknown(key); } while (r.nextKey(key)); - if (!blob_ref_fields.algo_word || !blob_ref_fields.digest_hex || !token_fields.type_word) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: record missing ha/h/tt"); e.ref = blob_ref_fields.build("outcome log"); - e.token = Token{token_fields.value.value_or(""), tokenTypeFromWord(*token_fields.type_word, "outcome log")}; + e.token = token_fields.build("outcome log"); if (!line_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: junk after record"); log.entries.push_back(std::move(e)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index 474c3c136a0b..edefc816cff2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -30,6 +30,9 @@ enum class OutcomeKind : uint8_t /// Canonical wire word for one `OutcomeKind`. std::string_view outcomeKindToWireWord(OutcomeKind outcome); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +OutcomeKind outcomeKindFromWireWord(std::string_view w); + /// One observation about a blob incarnation considered by GC. `token` identifies the exact /// incarnation that GC examined, while `ref` identifies the content address; retaining both lets /// replay and inspection distinguish an absent object from a replacement that won a race with GC. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp index b70f016cbae0..5cc268a4d9c3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp @@ -18,20 +18,20 @@ namespace DB::Cas namespace GcStateWire { - constexpr WireKey round{"rnd"}; - constexpr WireKey gc_shards{"gcs"}; - constexpr WireKey snap_generation{"sg"}; - constexpr WireKey snap_pruned_through{"spt"}; - constexpr WireKey snap_attempt{"sa"}; - constexpr WireKey manifest_sweep_cursor{"msc"}; - constexpr WireKey lease_owner{"lo"}; - constexpr WireKey lease_seq{"ls"}; + constexpr WireKey round{"round"}; + constexpr WireKey gc_shards{"gc_shards"}; + constexpr WireKey snap_generation{"snap_generation"}; + constexpr WireKey snap_pruned_through{"snap_pruned_through"}; + constexpr WireKey snap_attempt{"snap_attempt"}; + constexpr WireKey manifest_sweep_cursor{"manifest_sweep_cursor"}; + constexpr WireKey lease_owner{"lease_owner"}; + constexpr WireKey lease_seq{"lease_seq"}; } namespace GcHeartbeatWire { - constexpr WireKey owner{"by"}; - constexpr WireKey hb_seq{"seq"}; + constexpr WireKey owner{"owner"}; + constexpr WireKey hb_seq{"hb_seq"}; } String encodeGcState(const GcState & state) @@ -89,10 +89,10 @@ GcState decodeGcState(std::string_view data) else r.skipUnknown(key); } - /// Fail closed on an absent gcs: the writer always emits it, so a missing key means a corrupt object. + /// Fail closed on an absent gc_shards: the writer always emits it, so a missing key means a corrupt object. /// Do NOT silently keep the struct default (1) — that would hide corruption (no-fallback principle). if (!saw_gcs) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gcs"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gc_shards"); if (state.gc_shards == 0) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: gc_shards must be >= 1"); if (!body_in.eof() || !in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h index 88bce088ed13..ca38e649b82c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h @@ -45,7 +45,7 @@ String encodeGcState(const GcState & state); /// Decode a complete `cas_gc_state` text object. The header and size limits are checked before the /// body is parsed; unknown non-reserved fields are tolerated for forward evolution, but malformed -/// input, trailing bytes, a missing `gcs`, or a zero shard count raises `CORRUPTED_DATA` rather than +/// input, trailing bytes, a missing `gc_shards`, or a zero shard count raises `CORRUPTED_DATA` rather than /// falling back to a default state. GcState decodeGcState(std::string_view data); @@ -53,7 +53,7 @@ GcState decodeGcState(std::string_view data); /// independently of round progress, because its lease renewal counter can remain unchanged during a /// long fold. A follower that observes the heartbeat advance backs off from stealing the lease; this /// prevents mistaking an alive, mid-round leader for a stalled one. The value is persisted as the -/// versioned `cas_gc_hb` text object, whose body contains `by` and `seq` string values, replacing the +/// versioned `cas_gc_hb` text object, whose body contains `owner` and `hb_seq` string values, replacing the /// former unversioned 24-byte record. struct GcHeartbeat { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp index 5bd928ec01a3..9466a8af735e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp @@ -200,8 +200,8 @@ NamespaceLifePhysicalId Layout::namespaceLifePhysicalIdOf(std::string_view key, if (!incarnation) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "CasLayout: object '{}' names no life: '{}' is not 32 lower-case hex digits of a nonzero " - "life id. Generation-5 namespace-bearing pools are rejected by the pool-metadata format " - "gate before this generation-6 physical-key parser is reached", + "life id. Pools whose keys predate the opaque-life layout are rejected by the " + "pool-metadata format gate before this physical-key parser is reached", key, segment); return *incarnation; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index 3e406ad01203..e8ff121dfee7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -25,12 +25,11 @@ namespace namespace PartManifestWire { - constexpr WireKey ns{"ns"}; - constexpr WireKey payload_digest{"pd"}; - constexpr WireKey path{"p"}; - constexpr WireKey place{"pm"}; - constexpr WireKey size{"sz"}; - constexpr WireKey inline_size{"il"}; + constexpr WireKey ns{"root_namespace"}; + constexpr WireKey payload_digest{"payload_digest"}; + constexpr WireKey path{"path"}; + constexpr WireKey place{"place"}; + constexpr WireKey size{"size"}; } constexpr EnumWireTable kEntryPlacementWords{{{ @@ -40,7 +39,8 @@ constexpr EnumWireTable kEntryPlacementWords{{{ static_assert(casEnumTableCoversEnum()); -/// One entry-record line: {"p","pm", then either the Blob's "ha"/"h"/"sz" or the Inline's "il"}. +/// One entry-record line: `path`/`place`, followed by either `algo`/`digest`/`size` for a Blob or +/// `size` for Inline bytes. void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) { bool first = true; @@ -48,12 +48,12 @@ void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) writeWordField(out, PartManifestWire::place, entryPlacementToWireWord(e.placement), first); if (e.placement == EntryPlacement::Blob) { - writeBlobRefFields(out, first, e.ref); /// ha + h + writeBlobRefFields(out, first, e.ref); /// algo + digest writeNumberField(out, PartManifestWire::size, e.blob_size, first); } else { - writeNumberField(out, PartManifestWire::inline_size, e.inline_bytes.size(), first); + writeNumberField(out, PartManifestWire::size, e.inline_bytes.size(), first); } closeObject(out, first); writeChar('\n', out); @@ -69,7 +69,7 @@ String bannerFor(std::string_view path, uint64_t n) CasJsonWriter w(path.size() + 32); w.append("==> "); w.stringValue(path); - w.append(" il="); + w.append(" size="); w.u64Number(n); w.append(" <=="); return std::move(w).take(); @@ -82,6 +82,11 @@ std::string_view entryPlacementToWireWord(EntryPlacement placement) return kEntryPlacementWords.toWord(placement, "PartManifest: EntryPlacement"); } +EntryPlacement entryPlacementFromWireWord(std::string_view w) +{ + return kEntryPlacementWords.fromWord(w, "PartManifest: EntryPlacement"); +} + String encodePartManifest(const PartManifest & m) { /// Canonical path order plus duplicate-path rejection makes the encoded record sequence @@ -99,7 +104,7 @@ String encodePartManifest(const PartManifest & m) CasJsonWriter out(256); writeHeaderLine(out, FormatId::PartManifest); - /// descriptor meta line: ManifestRef (me/mb/mo, shared rendering with refsnaplog) + root + /// descriptor meta line: ManifestRef (epoch/build/ord, shared rendering with refsnaplog) + root /// namespace + payload digest. { bool first = true; @@ -156,9 +161,9 @@ PartManifest decodePartManifest(std::string_view data) else r.skipUnknown(key); } if (!ns) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing ns"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing root_namespace"); if (!pd) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing pd"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing payload_digest"); m.ref = fields.buildRef("PartManifest", "descriptor"); m.root_namespace_id = RootNamespace(*ns); m.payload_digest = *pd; @@ -166,7 +171,7 @@ PartManifest decodePartManifest(std::string_view data) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after descriptor line"); } - /// entry record lines, until the trailer. Inline entries remember their declared `il` length so + /// entry record lines, until the trailer. Inline entries remember their declared `size` so /// the payload zone below can read exactly that many raw bytes back into `inline_bytes`. /// Index-aligned with `m.entries` (Blob entries push an unused 0 placeholder). std::vector inline_lens; @@ -194,7 +199,7 @@ PartManifest decodePartManifest(std::string_view data) } if (key != PartManifestWire::path) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"p\""); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"path\""); ManifestEntry e; e.path = r.readString(); @@ -214,38 +219,36 @@ PartManifest decodePartManifest(std::string_view data) std::optional pm; BlobRefFields blob_ref; - std::optional sz; - std::optional il; + std::optional size; while (r.nextKey(key)) { if (key == PartManifestWire::place) pm = r.readString(); else if (matchBlobRefFields(key, r, blob_ref)) {} - else if (key == PartManifestWire::size) sz = r.readU64Number(); - else if (key == PartManifestWire::inline_size) il = r.readU64Number(); + else if (key == PartManifestWire::size) size = r.readU64Number(); else r.skipUnknown(key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after record"); if (!pm) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing pm", e.path); - e.placement = kEntryPlacementWords.fromWord(*pm, "PartManifest"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing place", e.path); + e.placement = entryPlacementFromWireWord(*pm); if (e.placement == EntryPlacement::Blob) { - if (!sz) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing sz", e.path); + if (!size) + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing size", e.path); blob_ref_what.assign("PartManifest entry '"); blob_ref_what += e.path; blob_ref_what += '\''; e.ref = blob_ref.build(blob_ref_what); - e.blob_size = *sz; + e.blob_size = *size; inline_lens.push_back(0); /// unused for Blob; keeps inline_lens index-aligned with entries } else { - if (!il) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: inline entry '{}' missing il", e.path); - inline_lens.push_back(*il); /// bytes filled from the payload zone below + if (!size) + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: inline entry '{}' missing size", e.path); + inline_lens.push_back(*size); /// bytes filled from the payload zone below } /// Canonical ascending-order and no-duplicate-path enforcement: compare only against the @@ -263,7 +266,7 @@ PartManifest decodePartManifest(std::string_view data) } /// payload zone: for each Inline entry, in the same order it appeared above, a banner line then - /// exactly `il` raw bytes then a terminating '\n'. + /// exactly `size` raw bytes then a terminating '\n'. for (size_t i = 0; i < m.entries.size(); ++i) { if (m.entries[i].placement != EntryPlacement::Inline) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h index e02e1ce6a917..a26597e1bebe 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h @@ -15,14 +15,14 @@ namespace DB::Cas /// stable for the surrounding CAS protocol. /// /// header line {"type":"cas_part_manifest","v":N} -/// descriptor meta line {"me","mb","mo"} (the ManifestRef, shared rendering with -/// refsnaplog, `CasWireVocab.h`) + "ns" (root namespace) + "pd" +/// descriptor meta line {"epoch","build","ord"} (the ManifestRef, shared rendering with +/// refsnaplog, `CasWireVocab.h`) + `root_namespace` + `payload_digest` /// (payload digest, 32 lowercase hex) -/// one entry-record line each {"p":path,"pm":placement-word, then either the Blob's -/// {"ha","h","sz"} or the Inline's {"il"}}, in canonical path order +/// one entry-record line each {"path":path,"place":placement-word, then either the Blob's +/// {"algo","digest","size"} or the Inline's {"size"}}, in canonical path order /// trailer line {"n":entry-count} /// PAYLOAD ZONE (raw, follows the trailer): for each Inline entry, in path order, a -/// `head -v`-style banner line `==> "" il= <==\n`, then +/// `head -v`-style banner line `==> "" size= <==\n`, then /// exactly `n` raw bytes, then `\n`. The path uses the same writer as /// the entry-record line, so decode can rebuild the banner byte-wise. /// Blob entries carry no @@ -45,6 +45,9 @@ enum class EntryPlacement : uint8_t /// Canonical wire word for one manifest entry placement. std::string_view entryPlacementToWireWord(EntryPlacement placement); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +EntryPlacement entryPlacementFromWireWord(std::string_view w); + /// One file entry inside a part manifest. `ref` is meaningful only for `Blob`; `inline_bytes` only /// for `Inline`. `blob_size` is the raw `Blob` byte count (0 for `Inline` — decode never fills it for /// an inline entry, since the wire format carries no redundant size for inline bytes). Use `size()` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp index 4e46131779cb..80f9572c4547 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp @@ -4,6 +4,7 @@ #include #include #include +#include namespace DB { @@ -19,35 +20,26 @@ namespace DB::Cas namespace PoolMetaWire { - constexpr WireKey pool_id{"pid"}; - constexpr WireKey blob_header_len{"hln"}; - constexpr WireKey gc_shards{"gcs"}; - constexpr WireKey min_reader_generation{"mrg"}; - constexpr WireKey algos_used{"alg"}; + constexpr WireKey pool_id{"pool_id"}; + constexpr WireKey blob_header_len{"blob_header_len"}; + constexpr WireKey gc_shards{"gc_shards"}; + constexpr WireKey min_reader_generation{"min_reader_generation"}; + constexpr WireKey algos_used{"algos_used"}; } -/// Minimum `blob_header_len` that provably fits the v3 `cas_blob` JSON envelope's mandatory (always- -/// written) non-ref fields, computed at type maxima from `encodeEnvelopeHeader` (CasBlobEnvelopeFormat.cpp): -/// {"type":"cas_blob" 18 -/// ,"v": 5 + 10 (currentCompatibilityVersion) 15 -/// ,"tag":"<32 hex>" 7 + 34 41 -/// ,"bld":"<32 hex>" 7 + 34 41 -/// ,"ts": 6 + 20 (created_at_ms) 26 -/// ,"by":"<32 hex>" 7 + 34 41 -/// ,"op":"" 6 + 10 (longest op word "mutation") 16 -/// ,"ch": 6 + 10 (VERSION_INTEGER) 16 -/// non-ref JSON = 214 bytes -/// The encoder then always frames the ref: `,"ref":` (7) + `""` (2) + `}` (1), and reserves byte -/// blob_header_len-1 for '\n' (1) = 11 bytes. So the mandatory content needs 214 + 11 = 225 bytes; -/// below that, encodeEnvelopeHeader throws LOGICAL_ERROR on the FIRST blob write (the old drop-and-retry -/// that used to mask this is gone). We floor at 240 (a multiple of 8 comfortably above 225, leaving -/// >= 15 bytes for the diagnostic ref even at type maxima, and well under the 256 default) so a -/// misconfigured pool fails at CREATION with BAD_ARGUMENTS, not at first write with LOGICAL_ERROR. +/// Minimum `blob_header_len` that provably fits the `cas_blob` JSON envelope's mandatory-descriptor +/// worst case. The byte-for-byte derivation (`kMandatoryDescriptorWorstCase`, currently 239 bytes) lives +/// beside the envelope key constants in `CasBlobEnvelopeFormat.cpp`, next to the compile-time proof that +/// it fits under this floor; below that bound, `encodeEnvelopeHeader` throws `LOGICAL_ERROR` on the +/// FIRST blob write (the old drop-and-retry that used to mask this is gone). We floor at 240 (a +/// multiple of 8 comfortably above the worst case, leaving at least one byte for the diagnostic `ref` +/// even at type maxima, and well under the 256 default) so a misconfigured pool fails at CREATION with +/// `BAD_ARGUMENTS`, not at first write with `LOGICAL_ERROR`. void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what) { if (blob_header_len < kMinBlobHeaderLen) - throw Exception(error_code, "CAS {}: blob_header_len must be >= {} (v3 envelope minimum), got {}", + throw Exception(error_code, "CAS {}: blob_header_len must be >= {} (blob envelope minimum), got {}", what, kMinBlobHeaderLen, blob_header_len); if (blob_header_len % 8 != 0) throw Exception(error_code, "CAS {}: blob_header_len must be a multiple of 8, got {}", what, blob_header_len); @@ -62,9 +54,8 @@ void validatePoolAlgosUsed(const std::vector & algos_used, int error_co for (size_t i = 0; i < algos_used.size(); ++i) { /// A direct membership scan, not `blobHashAlgoName`: that throws `LOGICAL_ERROR`, which - /// aborts at construction under a sanitizer/debug build before any catch can run, but a - /// persisted `algos_used` byte is exactly the unvalidated input this function must reject - /// cleanly instead. + /// aborts at construction under a sanitizer/debug build before any catch can run, but + /// this function validates a raw byte vector, so it must reject cleanly rather than abort. bool known = false; for (const auto & entry : kBlobHashAlgoWords.entries) if (static_cast(entry.value) == algos_used[i]) @@ -80,6 +71,8 @@ void validatePoolAlgosUsed(const std::vector & algos_used, int error_co String encodePoolMeta(const PoolMeta & pm) { + validatePoolAlgosUsed(pm.algos_used, ErrorCodes::CORRUPTED_DATA, "pool meta"); + CasJsonWriter out(256); writeHeaderLine(out, FormatId::PoolMeta); @@ -88,18 +81,13 @@ String encodePoolMeta(const PoolMeta & pm) writeNumberField(out, PoolMetaWire::blob_header_len, pm.blob_header_len, first); writeNumberField(out, PoolMetaWire::gc_shards, pm.gc_shards, first); writeNumberField(out, PoolMetaWire::min_reader_generation, pm.min_reader_generation, first); - writeKey(out, PoolMetaWire::algos_used, first); - { - /// Comma-joined algo words (tiny list, <=3): "ch128" or "ch128,sha256". - String joined; - for (size_t i = 0; i < pm.algos_used.size(); ++i) - { - if (i != 0) - joined += ','; - joined += blobHashAlgoName(static_cast(pm.algos_used[i])); - } - writeStringValue(out, joined); - } + /// Sized by the whole algo vocabulary and safe to index by `algos_used`: the validation above + /// admits only known algo bytes in strictly increasing order, so the vector cannot be longer + /// than the table. Relaxing that check to non-strict ordering would overrun this array. + std::array algo_words; + for (size_t i = 0; i < pm.algos_used.size(); ++i) + algo_words[i] = kBlobHashAlgoWords.toWord(static_cast(pm.algos_used[i]), "CAS pool meta"); + writeWordArrayField(out, PoolMetaWire::algos_used, std::span{algo_words}.first(pm.algos_used.size()), first); closeObject(out, first); writeChar('\n', out); @@ -111,18 +99,14 @@ PoolMeta decodePoolMeta(std::string_view data) ReadBufferFromMemory in(data.data(), data.size()); const TextHeader header = expectHeaderLine(in, FormatId::PoolMeta); - /// An older pool predates a breaking ref-layer change this build cannot reconcile, so - /// reject it before reading the metadata body. Writers always emit the current generation, while - /// `expectHeaderLine` separately rejects a future generation that this build cannot understand. - /// Generation 10 is the latest recreate-only authority floor and rejects old pools before any - /// mount lease body lacking its durable write-attempt identity can be interpreted. - if (header.v < kMountWriteAttemptIdGeneration) + /// The format-generation baseline is 1; a header below it cannot have been written by any build + /// this codec understands. `expectHeaderLine` above already rejects the symmetric FUTURE case + /// (`v > G_BUILD`); reject the backward case here, before the metadata body is read. + if (header.v < 1) throw Exception(ErrorCodes::UNKNOWN_FORMAT_VERSION, - "CAS pool format {} predates generation-10 mount-attempt-identity floor; recreate the pool. " - "This build requires the durable mount write attempt identity " - "in the generation-10 format " - "(generation {}+), and CAS is pre-release: there is no in-place migration.", - header.v, kMountWriteAttemptIdGeneration); + "CAS pool format {} predates the format-generation baseline; recreate the pool " + "(CAS is pre-release, so there is no in-place migration)", + header.v); const String body = readLine(in, traitsFor(FormatId::PoolMeta).line_cap, "pool meta"); ReadBufferFromMemory body_in(body.data(), body.size()); @@ -150,27 +134,16 @@ PoolMeta decodePoolMeta(std::string_view data) pm.min_reader_generation = r.readU64Number(); else if (key == PoolMetaWire::algos_used) { - const String joined = r.readString(); - size_t start = 0; - while (start <= joined.size()) - { - const size_t comma = joined.find(',', start); - const String word = joined.substr(start, comma == String::npos ? String::npos : comma - start); - if (word.empty()) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: empty algo word in '{}'", joined); + for (const String & word : r.readStringArray()) pm.algos_used.push_back(static_cast(blobHashAlgoFromWord(word, "pool meta algo"))); - if (comma == String::npos) - break; - start = comma + 1; - } } else r.skipUnknown(key); } if (!saw_pid) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing pid"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing pool_id"); if (!saw_gc_shards || pm.gc_shards == 0) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing or zero gcs"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing or zero gc_shards"); if (!body_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: junk after body object"); if (!in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h index 2ca0894d2f01..e25e767fb5e3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h @@ -14,8 +14,9 @@ class Backend; class Layout; /// `_pool_meta` — the pool identity and the pool-wide constants that every reader and writer must -/// agree on. The v3 text representation is a header line followed by one JSON body object: -/// {"pid":"<32hex>","hln":,"mrg":,"alg":""}. +/// agree on. The text representation is a header line followed by one JSON body object: +/// {"pool_id":"<32hex>","blob_header_len":,"gc_shards":, +/// "min_reader_generation":,"algos_used":["",...]}. /// /// The persisted object is authoritative after creation. On reopen, `createOrValidate` uses its /// `blob_header_len` and reader-generation floor rather than replacing them with local configuration; @@ -67,7 +68,7 @@ struct PoolMeta /// Serializes valid pool metadata as the versioned `_pool_meta` text object. The output includes the /// format header, one JSON body line, and its terminating newline; it is suitable for a conditional -/// backend write and preserves the sorted algorithm set as comma-separated vocabulary words. +/// backend write and preserves the sorted algorithm set as a JSON array of vocabulary words. String encodePoolMeta(const PoolMeta &); /// Parses and validates a persisted `_pool_meta` object. Unknown JSON keys are tolerated for additive @@ -76,11 +77,12 @@ String encodePoolMeta(const PoolMeta &); /// corruption or compatibility error code. PoolMeta decodePoolMeta(std::string_view); -/// Checks the fixed blob-envelope size invariant. The length must be 8-byte aligned, at most 16 KiB, -/// and at least 240 bytes: v3's mandatory envelope fields, framing, and newline consume 225 bytes at -/// type maxima, while 240 leaves room for a diagnostic `ref`. The caller supplies the error code so -/// persisted violations can be reported as `CORRUPTED_DATA` and bad creation arguments as -/// `BAD_ARGUMENTS`. +/// Checks the fixed blob-envelope size invariant: 8-byte aligned, at most 16 KiB, and at least +/// `kMinBlobHeaderLen`. That floor and the worst case it must clear are derived once beside the +/// envelope encoder, which also proves the relation at compile time — no number is restated here, +/// because a second copy is exactly what a single owner exists to prevent. The caller supplies the +/// error code so persisted violations can be reported as `CORRUPTED_DATA` and bad creation arguments +/// as `BAD_ARGUMENTS`. void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what); /// Checks that every admitted hash algorithm is known, that the set is non-empty, and that its numeric diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index 985c1964e9e5..857e428538f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -22,13 +22,13 @@ namespace namespace RunWire { - constexpr WireKey ref{"b"}; - constexpr WireKey src{"s"}; - constexpr WireKey mark{"m"}; - constexpr WireKey pending{"pend"}; - constexpr WireKey size{"sz"}; - constexpr WireKey condemn_round{"cr"}; - constexpr WireKey confirmed{"mc"}; + constexpr WireKey ref{"ref"}; + constexpr WireKey src{"src"}; + constexpr WireKey mark{"mark"}; + constexpr WireKey pending{"pending"}; + constexpr WireKey size{"size"}; + constexpr WireKey condemn_round{"condemn_round"}; + constexpr WireKey confirmed{"confirmed"}; } constexpr EnumWireTable kRunMarkerWords{{{ @@ -66,8 +66,8 @@ BlobHashAlgo algoFromByte(uint8_t b, std::string_view what) } } -/// `b` = the algo byte as two lowercase hex chars, then the digest hex at the algo's width. The algo -/// byte leads so that string-sorting `b` reproduces the binary (algo, digest) byte order. +/// `ref` = the algo byte as two lowercase hex chars, then the digest hex at the algo's width. The +/// algo byte leads so that string-sorting `ref` reproduces the binary (algo, digest) byte order. String renderB(const BlobRef & ref) { static constexpr char H[] = "0123456789abcdef"; @@ -108,6 +108,11 @@ std::string_view runMarkerToWireWord(RunMarker marker) return kRunMarkerWords.toWord(marker, "CAS cas_run: RunMarker"); } +RunMarker runMarkerFromWireWord(std::string_view w) +{ + return kRunMarkerWords.fromWord(w, "CAS cas_run: RunMarker"); +} + void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) { const FormatTraits & t = traitsFor(FormatId::RunFile); @@ -188,7 +193,7 @@ void SourceEdgeRunWriter::append(const SourceEdgeRecord & rec) if (rec.marker == RunMarker::Condemned) { writeBoolField(scratch, RunWire::pending, rec.delete_pending, first); - writeTokenFields(scratch, first, rec.token); /// tt + tv + writeTokenFields(scratch, first, rec.token); /// token_type + token writeNumberField(scratch, RunWire::size, rec.size, first); writeU64StringField(scratch, RunWire::condemn_round, rec.condemn_round, first); writeBoolField(scratch, RunWire::confirmed, rec.marker_confirmed, first); @@ -265,38 +270,36 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) SourceEdgeRecord out; String b; TokenFields token_fields; - bool have_b = false; - bool have_s = false; - bool have_m = false; - bool have_pend = false; - bool have_tt = false; - bool have_tv = false; - bool have_sz = false; - bool have_cr = false; - bool have_mc = false; + bool have_ref = false; + bool have_src = false; + bool have_mark = false; + bool have_pending = false; + bool have_size = false; + bool have_condemn_round = false; + bool have_confirmed = false; do { - if (key == RunWire::ref) { b = r.readString(); have_b = true; } - else if (key == RunWire::src) { out.source_id = r.readHex128(); have_s = true; } - else if (key == RunWire::mark) { out.marker = kRunMarkerWords.fromWord(r.readString(), "CAS cas_run"); have_m = true; } - else if (key == RunWire::pending) { out.delete_pending = r.readBool(); have_pend = true; } - else if (matchTokenFields(key, r, token_fields)) { have_tt = token_fields.type_word.has_value(); have_tv = token_fields.value.has_value(); } - else if (key == RunWire::size) { out.size = r.readU64Number(); have_sz = true; } - else if (key == RunWire::condemn_round) { out.condemn_round = r.readU64String(); have_cr = true; } - else if (key == RunWire::confirmed) { out.marker_confirmed = r.readBool(); have_mc = true; } + if (key == RunWire::ref) { b = r.readString(); have_ref = true; } + else if (key == RunWire::src) { out.source_id = r.readHex128(); have_src = true; } + else if (key == RunWire::mark) { out.marker = runMarkerFromWireWord(r.readString()); have_mark = true; } + else if (key == RunWire::pending) { out.delete_pending = r.readBool(); have_pending = true; } + else if (matchTokenFields(key, r, token_fields)) {} + else if (key == RunWire::size) { out.size = r.readU64Number(); have_size = true; } + else if (key == RunWire::condemn_round) { out.condemn_round = r.readU64String(); have_condemn_round = true; } + else if (key == RunWire::confirmed) { out.marker_confirmed = r.readBool(); have_confirmed = true; } else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } while (r.nextKey(key)); - if (!have_b || !have_s || !have_m) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing b/s/m"); + if (!have_ref || !have_src || !have_mark) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing ref/src/mark"); out.ref = parseB(b); if (out.marker == RunMarker::Condemned) { - if (!have_pend || !have_tt || !have_tv || !have_sz || !have_cr || !have_mc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pend/tt/tv/sz/cr/mc"); - out.token = Token{*token_fields.value, tokenTypeFromWord(*token_fields.type_word, "cas_run")}; + if (!have_pending || !have_size || !have_condemn_round || !have_confirmed) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pending/size/condemn_round/confirmed"); + out.token = token_fields.build("cas_run"); } - else if (have_pend || have_tt || have_tv || have_sz || have_cr || have_mc) + else if (have_pending || token_fields.type_word || token_fields.value || have_size || have_condemn_round || have_confirmed) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: non-condemned record carries condemned fields"); if (!line_in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index 0d9c6dfdad4b..fcc216b121ee 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -62,18 +62,18 @@ inline RunMarker runMarkerFromByte(char byte, std::string_view what) /// leaking into the format implementation. /// /// File shape: -/// {"type":"cas_run","v":3,"kind":"source_edge"} header line (type + v + kind gate) -/// {"b":"01","s":"<32hex>","m":"edge"} an active-edge / zero-marker row -/// {"b":"01","s":"00000000000000000000000000000000","m":"condemned","pend":false,"tt":"etag","tv":"...","sz":123,"cr":"7","mc":false} +/// {"type":"cas_run","v":1,"kind":"source_edge"} header line (type + v + kind gate) +/// {"ref":"01","src":"<32hex>","mark":"edge"} an active-edge / zero-marker row +/// {"ref":"01","src":"00000000000000000000000000000000","mark":"condemned","pending":false,"token_type":"etag","token":"...","size":123,"condemn_round":"7","confirmed":false} /// {"n":184267} trailer: record count /// -/// The record key `b` is the algo BYTE as two lowercase hex chars followed by the digest hex at the -/// algo's width; `s` is the 32-hex source id. String-sorting records by (b, s) reproduces the current +/// The record key `ref` is the algo BYTE as two lowercase hex chars followed by the digest hex at the +/// algo's width; `src` is the 32-hex source id. String-sorting records by (`ref`, `src`) reproduces the current /// `(algorithm, digest, source_id)` byte order (lowercase hex preserves unsigned byte order and the /// algorithm byte is emitted first) — the invariant the fold's two-cursor merge depends on. The row-tag word -/// `m` maps to the `RunMarker` bytes; a `condemned` row additionally -/// carries the retired incarnation (`pend`/`tt`/`tv`/`sz`/`cr`) and the durable condemn-marker -/// confirmation bit (`mc`). +/// `mark` maps to the `RunMarker` bytes; a `condemned` row additionally +/// carries the retired incarnation (`pending`/`token_type`/`token`/`size`/`condemn_round`) and the durable condemn-marker +/// confirmation bit (`confirmed`). /// One decoded source-edge row. All fields are identifier-layer types so the codec stays backend-free. /// The condemned-only fields (`delete_pending`/`token`/`size`/`condemn_round`/`marker_confirmed`) are @@ -96,6 +96,9 @@ inline constexpr std::string_view kSourceEdgeKindWord = "source_edge"; /// Canonical wire word for one source-edge run marker. std::string_view runMarkerToWireWord(RunMarker marker); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +RunMarker runMarkerFromWireWord(std::string_view w); + /// Write the typed header line `{"type":"cas_run","v":G_BUILD,"kind":""}\n` with a fixed key /// order for byte-determinism. The `kind` field distinguishes the record schema within the run /// family, so a reader can reject a valid run of the wrong kind before interpreting any records. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index 66606ef5b632..f2cea307342c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -26,17 +26,17 @@ namespace namespace RefCatalogWire { - constexpr WireKey kind{"k"}; + constexpr WireKey kind{"kind"}; constexpr WireKey ns{"ns"}; - constexpr WireKey state{"st"}; - constexpr WireKey life{"inc"}; - constexpr WireKey remove_round{"rsr"}; - constexpr WireKey creator{"csr"}; - constexpr WireKey creator_epoch{"cwe"}; - constexpr WireKey creator_fence{"cfg"}; + constexpr WireKey state{"state"}; + constexpr WireKey life{"life"}; + constexpr WireKey remove_round{"remove_round"}; + constexpr WireKey creator{"creator"}; + constexpr WireKey creator_epoch{"creator_epoch"}; + constexpr WireKey creator_fence{"creator_fence"}; } -constexpr std::string_view kEntryTag = "ent"; +constexpr std::string_view kEntryTag = "entry"; constexpr EnumWireTable kNsStateWords{{{ {NsState::Creating, "creating"}, @@ -162,7 +162,7 @@ String encodeRefCatalog(const RefCatalog & catalog) writeKey(out, RefCatalogWire::creator_fence, first); writeU64StringValue(out, e.creator->fence_generation); } closeObject(out, first); - closeLine("ent"); + closeLine("entry"); } const size_t trailer_start = out.size(); @@ -202,7 +202,7 @@ RefCatalog decodeRefCatalog(std::string_view data) return catalog; } if (key != RefCatalogWire::kind) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"k\""); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"kind\""); const String kind = r.readString(); if (kind != kEntryTag) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown record kind '{}'", kind); @@ -223,13 +223,13 @@ RefCatalog decodeRefCatalog(std::string_view data) else if (key == RefCatalogWire::creator_epoch) cwe = r.readU64String(); else if (key == RefCatalogWire::creator_fence) cfg = r.readU64String(); else if (key == RefCatalogWire::remove_round) removal_started_round = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ent key '{}'", key); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown entry key '{}'", key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: junk after record"); if (!st_word) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing st", ns_str); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing state", ns_str); const NsState state = nsStateFromWord(*st_word); /// throws CORRUPTED_DATA on an unknown word /// A missing "ns" key reads as the same empty string a present-but-empty one would, and both @@ -245,7 +245,7 @@ RefCatalog decodeRefCatalog(std::string_view data) ns_str, ns_str.size(), kMaxNamespaceBytes); if (!inc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing inc", ns_str); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing life", ns_str); if (*inc == 0) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: namespace '{}' has a zero incarnation -- 0 never names a life", ns_str); @@ -326,7 +326,7 @@ uint64_t worstCaseEntryFoldReservationBytes() /// coverage record plus terminal cleanup evidence, all numeric fields at maximum width. seal.ref_lives[std::numeric_limits::max()] = RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{kU64Max, kU64Max}, .hold = RefHold{.reason = HoldReason::UnconsumedSealCrossing, .offending_position = RefTxnId{kU64Max, kU64Max}, @@ -379,7 +379,7 @@ void checkFoldSealReservation( /// wrap to a remainder far smaller than the true reservation, which would answer "fits" for an /// `entry_count` that plainly does not. const uint64_t ref_lives = mulByteBudget(entry_count, worstCaseEntryFoldReservationBytes()); - /// `validateFoldSealStructure` permits at most one canonical seq-0 `btr` per shard, so charging + /// `validateFoldSealStructure` permits at most one canonical seq-0 `blob_run` per shard, so charging /// one widest row for every shard covers the full legal run domain without per-entry arithmetic. const uint64_t blob_target_runs = mulByteBudget( gc_shards, widestBlobTargetRunReservationBytes(layout, gc_shards)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h index ca1fbe5a6ddc..9515b00b425f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h @@ -16,7 +16,7 @@ class Layout; /// The byte bound every namespace name admitted into `ref_catalog` must satisfy (spec INV-3: /// "namespace names get a byte bound"). It keeps the catalog's operator-visible row and line grammar /// bounded, and both directions of the codec enforce it. Logical namespace bytes do NOT enter -/// predicate (2): fold-seal `rfl` rows are keyed only by the fixed-width opaque life id. +/// predicate (2): fold-seal `ref_life` rows are keyed only by the fixed-width opaque life id. constexpr size_t kMaxNamespaceBytes = 512; /// One namespace's catalog lifecycle state (spec INV-3, §3). `Creating` blocks publication and @@ -44,7 +44,7 @@ std::string_view nsStateToWord(NsState s); /// Inverse of `nsStateToWord`; throws `CORRUPTED_DATA` for anything but the three registered words. NsState nsStateFromWord(std::string_view w); -/// The fence identity of the mounted writer CREATING one namespace (spec §3): the server root plus +/// The fence identity of the mounted writer CREATING one namespace: the server root plus /// the writer epoch and admission fence generation captured at the moment `Creating` was minted. It /// is what a reconciler compares against `CasServerRoot`'s liveness/fence machinery before a stalled /// `Creating` entry may be CAS-reconciled away (INV-3: "stalled creators occupy entries until @@ -92,7 +92,7 @@ struct RefCatalog bool operator==(const RefCatalog &) const = default; }; -/// Encodes `catalog` as the canonical `cas_ref_catalog` text object: a header line, one "ent" record +/// Encodes `catalog` as the canonical `cas_ref_catalog` text object: a header line, one "entry" record /// per entry in canonical (ns-sorted) order, and a record-count trailer -- the same tagged-record /// container `encodeFoldSeal` uses. Enforces the FULL strict grammar on the way out: canonical order /// and no duplicate namespace, a non-empty namespace within the `kMaxNamespaceBytes` bound, nonzero @@ -144,7 +144,7 @@ uint64_t widestCondemnedSummaryReservationBytes(uint64_t gc_shards); /// PRE-PUT GATE, predicate (2) of INV-3's additive admission. Reserves the widest fixed frame, one /// widest ref-life row per candidate catalog entry, and one widest blob-target plus condemned-summary -/// row per authoritative GC shard. The `btr` multiplier follows the authoritative fold-seal grammar: +/// row per authoritative GC shard. The `blob_run` multiplier follows the authoritative fold-seal grammar: /// at most one canonical sequence-0 run is legal for each shard. Equality is accepted; refuses /// (`LIMIT_EXCEEDED`, naming `ns`) one entry over. Every multiplication and addition saturates, so an /// unreachable-in-practice count can never wrap into something that reads as "fits". diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index cc344f015ce2..dbc93315a99a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -20,13 +20,13 @@ namespace namespace RefCkptWire { - constexpr WireKey life_epoch{"le"}; - constexpr WireKey committed_epoch{"cte"}; - constexpr WireKey committed_seq{"cts"}; - constexpr WireKey snapshot_epoch{"cse"}; - constexpr WireKey snapshot_seq{"css"}; - constexpr WireKey seal_epoch{"lse"}; - constexpr WireKey seal_seq{"lss"}; + constexpr WireKey life_epoch{"life_epoch"}; + constexpr WireKey committed_epoch{"committed_epoch"}; + constexpr WireKey committed_seq{"committed_seq"}; + constexpr WireKey snapshot_epoch{"snapshot_epoch"}; + constexpr WireKey snapshot_seq{"snapshot_seq"}; + constexpr WireKey seal_epoch{"seal_epoch"}; + constexpr WireKey seal_seq{"seal_seq"}; } } @@ -143,22 +143,22 @@ RefCkpt decodeRefCkpt(std::string_view data) JsonObjectReader r(body_in, KeyStrictness::Strict, "cas_ref_ckpt"); RefCkpt ckpt; - std::optional cse; - std::optional css; - std::optional lse; - std::optional lss; - std::optional cte; - std::optional cts; + std::optional snapshot_epoch; + std::optional snapshot_seq; + std::optional seal_epoch; + std::optional seal_seq; + std::optional committed_epoch; + std::optional committed_seq; String key; while (r.nextKey(key)) { if (key == RefCkptWire::life_epoch) ckpt.life_epoch = r.readU64String(); - else if (key == RefCkptWire::committed_epoch) cte = r.readU64String(); - else if (key == RefCkptWire::committed_seq) cts = r.readU64String(); - else if (key == RefCkptWire::snapshot_epoch) cse = r.readU64String(); - else if (key == RefCkptWire::snapshot_seq) css = r.readU64String(); - else if (key == RefCkptWire::seal_epoch) lse = r.readU64String(); - else if (key == RefCkptWire::seal_seq) lss = r.readU64String(); + else if (key == RefCkptWire::committed_epoch) committed_epoch = r.readU64String(); + else if (key == RefCkptWire::committed_seq) committed_seq = r.readU64String(); + else if (key == RefCkptWire::snapshot_epoch) snapshot_epoch = r.readU64String(); + else if (key == RefCkptWire::snapshot_seq) snapshot_seq = r.readU64String(); + else if (key == RefCkptWire::seal_epoch) seal_epoch = r.readU64String(); + else if (key == RefCkptWire::seal_seq) seal_seq = r.readU64String(); else r.skipUnknown(key); } @@ -167,23 +167,23 @@ RefCkpt decodeRefCkpt(std::string_view data) /// deletable" today and as "recovery has no base" tomorrow -- both of which a reader would trust. /// Fail closed instead. (A missing whole field is a legitimate absence, not truncation: every field /// of this object is optional, so there is nothing to miss.) - if (cse || css) + if (snapshot_epoch || snapshot_seq) { - if (!cse || !css) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: checkpoint_snapshot_id needs both cse and css"); - ckpt.checkpoint_snapshot_id = RefTxnId{*cse, *css}; + if (!snapshot_epoch || !snapshot_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: checkpoint_snapshot_id needs both snapshot_epoch and snapshot_seq"); + ckpt.checkpoint_snapshot_id = RefTxnId{*snapshot_epoch, *snapshot_seq}; } - if (cte || cts) + if (committed_epoch || committed_seq) { - if (!cte || !cts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: committed_through needs both cte and cts"); - ckpt.committed_through = RefTxnId{*cte, *cts}; + if (!committed_epoch || !committed_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: committed_through needs both committed_epoch and committed_seq"); + ckpt.committed_through = RefTxnId{*committed_epoch, *committed_seq}; } - if (lse || lss) + if (seal_epoch || seal_seq) { - if (!lse || !lss) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: last_epoch_seal needs both lse and lss"); - ckpt.last_epoch_seal = RefTxnId{*lse, *lss}; + if (!seal_epoch || !seal_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: last_epoch_seal needs both seal_epoch and seal_seq"); + ckpt.last_epoch_seal = RefTxnId{*seal_epoch, *seal_seq}; } if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: trailing bytes"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index c40fe9cfb7cc..4c6dc9fdf053 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -23,14 +23,14 @@ namespace namespace RefLogWire { - constexpr WireKey ns{"ns"}; - constexpr WireKey txn_epoch{"we"}; - constexpr WireKey txn_seq{"rs"}; - constexpr WireKey prev_epoch{"!pse"}; - constexpr WireKey prev_seq{"!pss"}; + constexpr WireKey ns{"namespace"}; + constexpr WireKey txn_epoch{"txn_epoch"}; + constexpr WireKey txn_seq{"txn_seq"}; + constexpr WireKey prev_epoch{"!prev_epoch"}; + constexpr WireKey prev_seq{"!prev_seq"}; constexpr WireKey op{"op"}; - constexpr WireKey ref{"rn"}; - constexpr WireKey published_ms{"ts"}; + constexpr WireKey ref{"ref"}; + constexpr WireKey published_ms{"published_ms"}; } constexpr EnumWireTable kRefOpWords{{{ @@ -43,11 +43,6 @@ constexpr EnumWireTable kRefOpWords{{{ static_assert(casEnumTableCoversEnum()); -RefOpKind opKindFromWord(std::string_view w) -{ - return kRefOpWords.fromWord(w, "RefLogTxn"); -} - /// Byte budget over the encoded text. A removal-class transaction uses the larger complete-table /// budget and has neither an op-count nor a per-op cap; normal transactions are bounded by /// `ref_txn_max_ops` and, per op, by `ref_op_max_bytes` (checked via `encodedOpSize`, one op at a @@ -113,7 +108,7 @@ struct BindingFields RefOwnerBinding build(std::string_view what) const { if (!kind || !ref) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing bk/rn", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing kind/ref", what); RefOwnerBinding b; b.kind = refOwnerKindFromWord(*kind, "RefLogTxn owner binding"); b.ref_name = *ref; @@ -123,10 +118,10 @@ struct BindingFields } }; -/// The log transaction's header-object meta line (ns + txn_id + the optional `prev_epoch_seal` +/// The log transaction's header-object meta line (`namespace` + txn_id + the optional `prev_epoch_seal` /// chain). Shared by `encodeRefLogTxn` and `removalFramingSize` so the two never disagree by a byte; /// `removalFramingSize` always passes `std::nullopt` -- a removal transaction is never a sequence-1 -/// epoch-transition record. Additive: the `"!pse"`/`"!pss"` pair is emitted only when +/// epoch-transition record. Additive: the `"!prev_epoch"`/`"!prev_seq"` pair is emitted only when /// `prev_epoch_seal` is set, so a body without it is byte-identical to the pre-EpochSeal wire shape. /// `!`-prefixed: `prev_epoch_seal` is INV-2 chain evidence, not cosmetic metadata -- a decoder that /// doesn't understand it must refuse the object rather than silently drop the chain link while @@ -149,9 +144,9 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) op.kind = kind; /// set_published_at fields - std::optional sp_rn; + std::optional sp_ref; ManifestRefFields sp_manifest_fields; - std::optional sp_ts; + std::optional sp_published_ms; /// owner_transition bindings BindingFields ob; BindingFields nb; @@ -160,12 +155,12 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) while (r.nextKey(key)) { if (key == RefLogWire::ref) - sp_rn = r.readString(); + sp_ref = r.readString(); else if (matchManifestRefFields(key, r, kBareManifestRefKeys, sp_manifest_fields)) { } else if (key == RefLogWire::published_ms) - sp_ts = r.readU64Number(); + sp_published_ms = r.readU64Number(); else if (key == kOldBindingKeys.kind) ob.kind = r.readString(); else if (key == kOldBindingKeys.ref) @@ -181,8 +176,8 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) { } else if (key == "pl") - /// `"pl"` (payload) was removed from the op wire in stage-1 T12 (the `set_payload` op became - /// `set_published_at`). The retired op WORD is already rejected by `opKindFromWord`, but this + /// `"pl"` (payload) was removed from the op wire when the `set_payload` op became + /// `set_published_at`. The retired op WORD is already rejected by `refOpKindFromWireWord`, but this /// generic reader reads field keys before switching on kind, so a `"pl"` field paired with a /// still-recognized op word would otherwise be `skipUnknown`'d. It is a KNOWN-removed field, /// not a genuinely-unknown one -- reject it explicitly rather than silently discard it. @@ -204,12 +199,12 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) op.new_binding = nb.build("new"); break; case RefOpKind::SetPublishedAt: - if (!sp_rn || !sp_ts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing rn/ts"); - op.ref_name = *sp_rn; + if (!sp_ref || !sp_published_ms) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing ref/published_ms"); + op.ref_name = *sp_ref; checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); op.expected_manifest_ref = sp_manifest_fields.buildRef("RefLogTxn", "set_published_at"); - op.published_at_ms = *sp_ts; + op.published_at_ms = *sp_published_ms; break; } return op; @@ -222,6 +217,11 @@ std::string_view refOpKindToWireWord(RefOpKind kind) return kRefOpWords.toWord(kind, "RefLogTxn"); } +RefOpKind refOpKindFromWireWord(std::string_view w) +{ + return kRefOpWords.fromWord(w, "RefLogTxn"); +} + bool refLogTxnIsEpochSeal(const RefLogTxn & txn) { return txn.ops.size() == 1 && txn.ops.front().kind == RefOpKind::EpochSeal; @@ -310,10 +310,10 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con ReadBufferFromMemory m(line.data(), line.size()); JsonObjectReader r(m, KeyStrictness::Tolerant, "cas_ref_log"); bool saw_ns = false; - bool saw_we = false; - bool saw_rs = false; - std::optional pse; - std::optional pss; + bool saw_txn_epoch = false; + bool saw_txn_seq = false; + std::optional prev_epoch; + std::optional prev_seq; String key; while (r.nextKey(key)) { @@ -325,29 +325,29 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con else if (key == RefLogWire::txn_epoch) { txn.txn_id.writer_epoch = r.readU64String(); - saw_we = true; + saw_txn_epoch = true; } else if (key == RefLogWire::txn_seq) { txn.txn_id.ref_sequence = r.readU64String(); - saw_rs = true; + saw_txn_seq = true; } else if (key == RefLogWire::prev_epoch) - pse = r.readU64String(); + prev_epoch = r.readU64String(); else if (key == RefLogWire::prev_seq) - pss = r.readU64String(); + prev_seq = r.readU64String(); else r.skipUnknown(key); } - if (!saw_ns || !saw_we || !saw_rs) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing ns/we/rs"); - /// Both-or-neither: `nextKey` already rejects a repeated "!pse"/"!pss" (duplicate-key check), so + if (!saw_ns || !saw_txn_epoch || !saw_txn_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing namespace/txn_epoch/txn_seq"); + /// Both-or-neither: `nextKey` already rejects a repeated "!prev_epoch"/"!prev_seq" (duplicate-key check), so /// this only guards against a body carrying exactly one of the pair. - if (pse || pss) + if (prev_epoch || prev_seq) { - if (!pse || !pss) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: prev_epoch_seal needs both !pse and !pss"); - txn.prev_epoch_seal = RefTxnId{*pse, *pss}; + if (!prev_epoch || !prev_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: prev_epoch_seal needs both !prev_epoch and !prev_seq"); + txn.prev_epoch_seal = RefTxnId{*prev_epoch, *prev_seq}; } if (!m.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: junk after meta line"); @@ -387,7 +387,7 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con } if (key != RefLogWire::op) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: record must start with \"op\""); - const RefOpKind kind = opKindFromWord(r.readString()); + const RefOpKind kind = refOpKindFromWireWord(r.readString()); txn.ops.push_back(readOpRecord(r, kind)); if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: junk after op record"); @@ -440,4 +440,41 @@ size_t removalFramingSize(const String & ns, const RefTxnId & txn_id, uint64_t o return out.size(); } +std::optional peekRefLogMeta(const String & sealed_bytes) +{ + try + { + const String text = openObject(FormatId::RefLog, sealed_bytes); + ReadBufferFromMemory in(text.data(), text.size()); + const uint64_t line_cap = traitsFor(FormatId::RefLog).line_cap; + readLine(in, line_cap, "cas_ref_log"); /// header line -- skipped, the version is not judged here + const String meta = readLine(in, line_cap, "cas_ref_log"); + ReadBufferFromMemory m(meta.data(), meta.size()); + JsonObjectReader r(m, KeyStrictness::Tolerant, "cas_ref_log"); + RefLogMetaPeek peek; + bool saw_ns = false; + bool saw_epoch = false; + bool saw_seq = false; + String key; + while (r.nextKey(key)) + { + if (key == RefLogWire::ns) { peek.ns = r.readString(); saw_ns = true; } + else if (key == RefLogWire::txn_epoch) { peek.writer_epoch = r.readU64String(); saw_epoch = true; } + else if (key == RefLogWire::txn_seq) { peek.ref_sequence = r.readU64String(); saw_seq = true; } + else r.skipUnknown(key); + } + if (!saw_ns || !saw_epoch || !saw_seq) + return std::nullopt; + return peek; + } + catch (...) + { + /// Deliberately total: a diagnostic that throws while explaining an anomaly replaces the + /// anomaly's report with its own. A seal-linked txn reaches here too -- its `!`-prefixed chain + /// keys make the tolerant reader refuse the line -- and answering `nullopt` is correct: this + /// peek identifies a writer, it does not certify an object. + return std::nullopt; + } +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h index c61e11275e98..e4949cf130b8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h @@ -13,7 +13,7 @@ namespace DB::Cas /// Text codec for `cas_ref_log`, the immutable object stored at `_log/`. Each object contains /// exactly one committed transaction: its namespace, transaction id, and the batch of `RefOp`s applied -/// by that commit. The body has a header, a meta line `{"ns","we","rs",["!pse","!pss"]}`, one JSON +/// by that commit. The body has a header, a meta line `{"namespace","txn_epoch","txn_seq",["!prev_epoch","!prev_seq"]}`, one JSON /// record per op, and a `{"n":count}` trailer. Records are emitted in the transaction's stored order /// and contain no codec-generated timestamps, so encoding the same value is byte-identical. This /// determinism is a property of the representation, not an adoption gate: ref commits use @@ -21,7 +21,7 @@ namespace DB::Cas /// returned text. /// /// `RefOpKind::EpochSeal` closes an epoch transition in-band (spec INV-2): a seal transaction contains -/// exactly that one op, and the meta line's optional `prev_epoch_seal` (wire fields `!pse`/`!pss`, +/// exactly that one op, and the meta line's optional `prev_epoch_seal` (wire fields `!prev_epoch`/`!prev_seq`, /// CRITICAL -- an unrecognized `!`-key fails closed with `UNKNOWN_FORMAT_VERSION` rather than being /// silently skipped, since dropping it would lose INV-2's chain evidence while still passing the /// structural grammar) chains to the transaction id of the seal that closed the PRECEDING epoch, and @@ -45,6 +45,9 @@ enum class RefOpKind : uint8_t /// `kind` is not represented by this format. std::string_view refOpKindToWireWord(RefOpKind kind); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +RefOpKind refOpKindFromWireWord(std::string_view w); + /// One operation inside a `RefLogTxn`. Only the fields documented next to `kind` are meaningful for /// that kind, and the codec never reads or writes the others. `OwnerTransition` optionally removes /// `old_binding` and/or installs `new_binding`; `SetPublishedAt` carries the expected manifest and the @@ -178,4 +181,22 @@ void validateEpochSealGrammarStructural(const RefLogTxn & txn); /// mint a sequence-1 transaction. Throws CORRUPTED_DATA on violation. void validateEpochSealGrammarContextual(const RefLogTxn & txn, uint64_t life_epoch); +/// The three identity fields of a `cas_ref_log` meta line, read WITHOUT trusting the object: this is +/// the anomaly diagnostic's view of an object found at a key it should not occupy, so the body is not +/// expected to match that key's identity. +struct RefLogMetaPeek +{ + String ns; + uint64_t writer_epoch = 0; + uint64_t ref_sequence = 0; +}; + +/// Best-effort identification of a sealed `cas_ref_log` object: opens it, skips the header line, and +/// reads the meta line's three identity fields. Never validates the header version, never reads past +/// the meta line, and answers `nullopt` for anything it cannot read -- truncation, garbage, a +/// different format, or a meta line missing one of the three. It lives HERE, beside the key +/// constants, because a caller that spelled those keys itself would silently stop matching the first +/// time they are renamed, and this reader has no output an ordinary test would miss. +std::optional peekRefLogMeta(const String & sealed_bytes); + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index 8ed19c9a1ac3..78caf9c5632c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -22,17 +22,16 @@ namespace namespace RefSnapWire { - constexpr WireKey ns{"ns"}; - constexpr WireKey snapshot_epoch{"we"}; - constexpr WireKey snapshot_seq{"rs"}; - constexpr WireKey lifecycle{"lc"}; - constexpr WireKey kind{"k"}; - constexpr WireKey ref{"rn"}; - constexpr WireKey published_ms{"ts"}; + constexpr WireKey ns{"namespace"}; + constexpr WireKey snapshot_epoch{"snapshot_epoch"}; + constexpr WireKey snapshot_seq{"snapshot_seq"}; + constexpr WireKey lifecycle{"lifecycle"}; + constexpr WireKey kind{"kind"}; + constexpr WireKey ref{"ref"}; + constexpr WireKey published_ms{"published_ms"}; } -constexpr std::string_view kCommittedTag = "c"; -constexpr std::string_view kPrecommitTag = "p"; +constexpr std::string_view kLiveLifecycleWord = "live"; void checkCommittedSorted(const std::vector & rows) { @@ -73,7 +72,7 @@ void writeCommittedRow(CasJsonWriter & out, const RefCommittedRow & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "committed"); bool first = true; - writeWordField(out, RefSnapWire::kind, kCommittedTag, first); + writeWordField(out, RefSnapWire::kind, refOwnerKindToWord(RefOwnerKind::Committed), first); writeStringField(out, RefSnapWire::ref, row.ref_name, first); writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); writeNumberField(out, RefSnapWire::published_ms, row.published_at_ms, first); @@ -90,15 +89,15 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "precommit"); bool first = true; - writeWordField(out, RefSnapWire::kind, kPrecommitTag, first); + writeWordField(out, RefSnapWire::kind, refOwnerKindToWord(RefOwnerKind::Precommit), first); writeStringField(out, RefSnapWire::ref, row.ref_name, first); writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); closeObject(out, first); writeChar('\n', out); } -/// The snapshot's header-object meta line (`ns`, `snapshot_id`, and the required generation-8 -/// `lc:"live"` constant). Shared by +/// The snapshot's header-object meta line (`namespace`, `snapshot_id`, and the required +/// `lifecycle:"live"` constant). Shared by /// `encodeRefTableSnapshot` and `snapshotFramingSize` so the two never disagree by a /// byte. Assumes the caller has already validated the snapshot (or is measuring framing only). void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) @@ -106,7 +105,10 @@ void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) bool first = true; writeStringField(out, RefSnapWire::ns, snapshot.ns, first); writeRefTxnIdFields(out, first, RefSnapWire::snapshot_epoch, RefSnapWire::snapshot_seq, snapshot.snapshot_id); - writeStringField(out, RefSnapWire::lifecycle, "live", first); + /// A snapshot object exists only for a live namespace -- `RefLifecycle::Removed` has no snapshot + /// representation -- so the wire carries exactly one lifecycle word. The reader keeps the + /// fail-closed half: any other word, or none, is rejected there. + writeStringField(out, RefSnapWire::lifecycle, kLiveLifecycleWord, first); closeObject(out, first); writeChar('\n', out); } @@ -149,30 +151,30 @@ RefTableSnapshot decodeRefTableSnapshot( ReadBufferFromMemory meta_buf(line.data(), line.size()); JsonObjectReader r(meta_buf, KeyStrictness::Tolerant, "cas_ref_snap"); bool saw_ns = false; - bool saw_we = false; - bool saw_rs = false; - bool saw_lc = false; + bool saw_snapshot_epoch = false; + bool saw_snapshot_seq = false; + RefLifecycle lifecycle = RefLifecycle::Removed; String key; while (r.nextKey(key)) { if (key == RefSnapWire::ns) { snapshot.ns = r.readString(); saw_ns = true; } - else if (key == RefSnapWire::snapshot_epoch) { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == RefSnapWire::snapshot_seq) { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } + else if (key == RefSnapWire::snapshot_epoch) { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_snapshot_epoch = true; } + else if (key == RefSnapWire::snapshot_seq) { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_snapshot_seq = true; } else if (key == RefSnapWire::lifecycle) { - const String lifecycle = r.readString(); - if (lifecycle != "live") + const String lifecycle_word = r.readString(); + if (lifecycle_word != kLiveLifecycleWord) throw Exception(ErrorCodes::CORRUPTED_DATA, - "RefTableSnapshot: lifecycle must be exactly 'live', got '{}'", lifecycle); - saw_lc = true; + "RefTableSnapshot: lifecycle must be exactly '{}', got '{}'", kLiveLifecycleWord, lifecycle_word); + lifecycle = RefLifecycle::Live; } else if (key == "rte" || key == "rts") throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta carries retired terminal field '{}'", key); else r.skipUnknown(key); } - if (!saw_ns || !saw_we || !saw_rs || !saw_lc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta line missing ns/we/rs/lc"); + if (!saw_ns || !saw_snapshot_epoch || !saw_snapshot_seq || lifecycle != RefLifecycle::Live) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta line missing namespace/snapshot_epoch/snapshot_seq/lifecycle"); if (!meta_buf.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after meta line"); } @@ -200,21 +202,21 @@ RefTableSnapshot decodeRefTableSnapshot( break; } if (key != RefSnapWire::kind) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"k\""); - const String k = r.readString(); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"kind\""); + const RefOwnerKind kind = refOwnerKindFromWord(r.readString(), "RefTableSnapshot row kind"); - std::optional rn; + std::optional ref; ManifestRefFields mf; - std::optional ts; + std::optional published_ms; while (r.nextKey(key)) { - if (key == RefSnapWire::ref) rn = r.readString(); + if (key == RefSnapWire::ref) ref = r.readString(); else if (matchManifestRefFields(key, r, kBareManifestRefKeys, mf)) { } - else if (key == RefSnapWire::published_ms) ts = r.readU64Number(); + else if (key == RefSnapWire::published_ms) published_ms = r.readU64Number(); else if (key == "pl") - /// `"pl"` (payload) was removed from the row wire in stage-1 T12. It is a KNOWN-removed + /// `"pl"` (payload) was removed from the row wire. It is a KNOWN-removed /// field, not a genuinely-unknown future one the tolerant reader may skip -- silently /// discarding a persisted payload would lose data -- so reject it explicitly. throw Exception(ErrorCodes::CORRUPTED_DATA, @@ -224,30 +226,36 @@ RefTableSnapshot decodeRefTableSnapshot( if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after record"); - if (k == kCommittedTag) + /// A `switch` rather than an if/else-if chain: the row kinds partition the enum, and a future + /// enumerator must not be able to arrive here, pass the word lookup, and then fall out of the + /// chain as a silently dropped row. With no default arm, adding one is a build error. + switch (kind) { - if (!rn || !ts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing rn/ts"); + case RefOwnerKind::Committed: + { + if (!ref || !published_ms) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing ref/published_ms"); RefCommittedRow row; - row.ref_name = *rn; + row.ref_name = *ref; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); row.manifest_ref = mf.buildRef("RefTableSnapshot", "committed"); - row.published_at_ms = *ts; + row.published_at_ms = *published_ms; snapshot.committed.push_back(std::move(row)); + break; } - else if (k == kPrecommitTag) + case RefOwnerKind::Precommit: { - if (!rn) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing rn"); + if (!ref) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing ref"); RefOwnerBinding row; - row.kind = RefOwnerKind::Precommit; - row.ref_name = *rn; + row.kind = kind; + row.ref_name = *ref; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); row.manifest_ref = mf.buildRef("RefTableSnapshot", "precommit"); snapshot.precommits.push_back(std::move(row)); + break; + } } - else - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: unknown row kind '{}'", k); } /// The object key is supplied separately from the body. Check the binding before accepting any diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h index 07972a0d47ab..79573cf4767a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h @@ -26,7 +26,7 @@ namespace DB::Cas /// published through the ordinary single-owner `putIfAbsentControlled` path, not a /// `putDeterministicArtifact` byte-adoption gate. -/// In-memory ref-table lifecycle. Only `Live` is serializable as a generation-8 snapshot; terminal +/// In-memory ref-table lifecycle. Only `Live` is serializable as a snapshot; terminal /// state lives in the removal log and fold evidence and has no snapshot DTO representation. enum class RefLifecycle : uint8_t { @@ -47,7 +47,7 @@ struct RefCommittedRow /// The complete state of one namespace's ref table in one canonical snapshot object. `precommits` /// reuses `RefOwnerBinding` from `CasRefWireVocab.h`; every entry's `kind` must be `Precommit`. -/// Generation 8 serializes only `Live` snapshots. Both row vectors must already be strictly sorted by +/// A snapshot serializes only `Live` namespaces. Both row vectors must already be strictly sorted by /// their documented keys, because the codec /// validates and emits the caller-provided order rather than sorting it. struct RefTableSnapshot diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h index e13fd116331e..0feacf7352d3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h @@ -35,9 +35,8 @@ struct RefOwnerBinding bool operator==(const RefOwnerBinding &) const = default; }; -/// Convert an owner-kind discriminator to its canonical text word. Throws `CORRUPTED_DATA` for a -/// value not represented by this format; accepting an unknown value would produce an ambiguous wire -/// record. +/// Convert an owner-kind discriminator to its canonical text word. Throws `LOGICAL_ERROR` for an +/// out-of-range enum value. std::string_view refOwnerKindToWord(RefOwnerKind k); /// Parse a canonical owner-kind word. `what` identifies the containing field in the @@ -48,7 +47,7 @@ RefOwnerKind refOwnerKindFromWord(std::string_view w, std::string_view what); /// in-progress JSON object, both as decimal STRINGS -- the representation is width-independent, so no /// consumer has to care how large a `ref_sequence` can get. `epoch_key`/`seq_key` name the two fields, /// letting each format distinguish its primary id from any secondary id it embeds (for example, -/// `cas_ref_log`'s `we`/`rs` versus its `prev_epoch_seal` pair) while sharing one writer so the +/// `cas_ref_log`'s `txn_epoch`/`txn_seq` versus its `prev_epoch_seal` pair) while sharing one writer so the /// formats can never disagree on the representation. void writeRefTxnIdFields(CasJsonWriter & out, bool & first, WireKey epoch_key, WireKey seq_key, const RefTxnId & id); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp index 68c26d47e0e6..a01a62ed30f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp @@ -16,26 +16,26 @@ namespace DB::Cas namespace OwnerWire { - constexpr WireKey server_uuid{"su"}; - constexpr WireKey retired_at_ms{"rt"}; + constexpr WireKey server_uuid{"server_uuid"}; + constexpr WireKey retired_at_ms{"retired_at_ms"}; } namespace ServerEpochWire { - constexpr WireKey next_writer_epoch{"nwe"}; + constexpr WireKey next_writer_epoch{"next_writer_epoch"}; } namespace MountLeaseWire { - constexpr WireKey server_uuid{"su"}; - constexpr WireKey writer_epoch{"we"}; - constexpr WireKey hostname{"hn"}; + constexpr WireKey server_uuid{"server_uuid"}; + constexpr WireKey writer_epoch{"writer_epoch"}; + constexpr WireKey hostname{"hostname"}; constexpr WireKey pid{"pid"}; - constexpr WireKey started_at_ms{"sat"}; + constexpr WireKey started_at_ms{"started_at_ms"}; constexpr WireKey seq{"seq"}; - constexpr WireKey expires_at_ms{"eat"}; - constexpr WireKey min_active{"ma"}; - constexpr WireKey gc_fenced{"fen"}; + constexpr WireKey expires_at_ms{"expires_at_ms"}; + constexpr WireKey min_active_build_sequence{"min_active_build_sequence"}; + constexpr WireKey gc_fenced{"gc_fenced"}; constexpr WireKey write_attempt_id{"write_attempt_id"}; } @@ -91,7 +91,7 @@ OwnerObject decodeOwner(std::string_view data) } o.retired_at_ms = rt; if (!saw) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: missing su"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: missing server_uuid"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: trailing bytes"); return o; @@ -130,7 +130,7 @@ ServerEpoch decodeServerEpoch(std::string_view data) r.skipUnknown(key); } if (!saw) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: missing nwe"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: missing next_writer_epoch"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: trailing bytes"); return e; @@ -148,7 +148,7 @@ String encodeMountLease(const MountLease & m) writeNumberField(out, MountLeaseWire::started_at_ms, m.started_at_ms, first); writeU64StringField(out, MountLeaseWire::seq, m.seq, first); writeNumberField(out, MountLeaseWire::expires_at_ms, m.expires_at_ms, first); - writeU64StringField(out, MountLeaseWire::min_active, m.min_active, first); + writeU64StringField(out, MountLeaseWire::min_active_build_sequence, m.min_active_build_sequence, first); writeBoolField(out, MountLeaseWire::gc_fenced, m.gc_fenced, first); writeHex128Field(out, MountLeaseWire::write_attempt_id, m.write_attempt_id, first); closeObject(out, first); @@ -191,8 +191,8 @@ MountLease decodeMountLease(std::string_view data) m.seq = r.readU64String(); else if (key == MountLeaseWire::expires_at_ms) m.expires_at_ms = r.readU64Number(); - else if (key == MountLeaseWire::min_active) - m.min_active = r.readU64String(); + else if (key == MountLeaseWire::min_active_build_sequence) + m.min_active_build_sequence = r.readU64String(); else if (key == MountLeaseWire::gc_fenced) m.gc_fenced = r.readBool(); else if (key == MountLeaseWire::write_attempt_id) @@ -203,8 +203,15 @@ MountLease decodeMountLease(std::string_view data) else r.skipUnknown(key); } - if (!saw_su || !saw_we || !saw_write_attempt_id || m.write_attempt_id == UInt128{}) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing or zero identity field"); + /// Named one by one rather than as a single condition: these are three separate identities, and a + /// shared message cannot tell an operator which of them the object is missing -- nor let a test + /// prove that each is actually required. + if (!saw_su) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing server_uuid"); + if (!saw_we) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing writer_epoch"); + if (!saw_write_attempt_id || m.write_attempt_id == UInt128{}) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing or zero write_attempt_id"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: trailing bytes"); return m; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h index 0c73e8c5a6e6..8e0bd8118a1e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h @@ -17,7 +17,7 @@ namespace DB::Cas /// The owner object permanently binds a configured `server_root_id` to one server UUID. The epoch /// object stores the next writer epoch and is CAS-bumped so an epoch is never reused after a /// restart or supersession. The mount object is the expirable liveness lease for one writer -/// incarnation; its `min_active` value carries the GC acknowledgement floor, while `gc_fenced` is a +/// incarnation; its `min_active_build_sequence` value carries the GC acknowledgement floor, while `gc_fenced` is a /// terminal fence-out marker for that incarnation. /// Permanent identity anchor for one configured server root. It is created with put-if-absent, never @@ -42,7 +42,7 @@ struct ServerEpoch /// The current liveness lease for one `(server_uuid, writer_epoch)` writer incarnation. The pool /// layer renews and replaces this object with CAS/overwrite operations, and GC may fence an expired -/// lease by setting `gc_fenced`; a fenced incarnation must not resume writing. `min_active` is the +/// lease by setting `gc_fenced`; a fenced incarnation must not resume writing. `min_active_build_sequence` is the /// merged GC acknowledgement floor, with `UINT64_MAX` marking a clean farewell (retired lease). struct MountLease { @@ -53,7 +53,7 @@ struct MountLease uint64_t started_at_ms = 0; uint64_t seq = 0; uint64_t expires_at_ms = 0; - uint64_t min_active = 0; /// UINT64_MAX = retired (farewell) + uint64_t min_active_build_sequence = 0; /// UINT64_MAX = retired (farewell) bool gc_fenced = false; /// GC fence-out of an expired lease; terminal /// One holder-originated body identity. Every physical retry of that one logical write reuses /// it; a GC fence copies the observed value while every successor holder body mints a new one. @@ -66,9 +66,9 @@ struct MountLease /// decisions belong to the caller that coordinates the server-root object. String encodeOwner(const OwnerObject & o); -/// Decode an owner anchor, requiring its `su` field, tolerating an absent optional `rt` retirement +/// Decode an owner anchor, requiring its `server_uuid` field, tolerating an absent optional `retired_at_ms` retirement /// timestamp, and rejecting bytes after the body line. Unknown JSON fields are skipped for -/// forward-compatible reads; malformed input, a missing `su`, and trailing data throw +/// forward-compatible reads; malformed input, a missing `server_uuid`, and trailing data throw /// `CORRUPTED_DATA`. OwnerObject decodeOwner(std::string_view data); @@ -77,13 +77,13 @@ OwnerObject decodeOwner(std::string_view data); /// codec. String encodeServerEpoch(const ServerEpoch & e); -/// Decode the epoch counter, requiring its `nwe` field and rejecting bytes after the body line. -/// Unknown JSON fields are skipped for forward-compatible reads; malformed input, a missing `nwe`, +/// Decode the epoch counter, requiring its `next_writer_epoch` field and rejecting bytes after the body line. +/// Unknown JSON fields are skipped for forward-compatible reads; malformed input, a missing `next_writer_epoch`, /// and trailing data throw `CORRUPTED_DATA`. ServerEpoch decodeServerEpoch(std::string_view data); /// Encode the complete mount-lease body as canonical text with the `cas_mount_lease` header and a -/// final newline. This preserves full-range `uint64_t` values such as `min_active` as decimal JSON +/// final newline. This preserves full-range `uint64_t` values such as `min_active_build_sequence` as decimal JSON /// strings and writes `gc_fenced` as a JSON boolean; lease renewal, fencing, and token checks remain /// in the caller. String encodeMountLease(const MountLease & m); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 8a53273956ee..8438ac786478 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -109,6 +109,20 @@ void CasJsonWriter::stringValue(std::string_view s) appendChar('"'); } +void CasJsonWriter::wordArray(std::span words) +{ + appendChar('['); + bool first = true; + for (const std::string_view word : words) + { + if (!first) + appendChar(','); + first = false; + stringValue(word); + } + appendChar(']'); +} + /// ---- read-side pull cursor ---- /// A canonical-text parse failure is CORRUPTED_DATA regardless of which ReadHelpers primitive @@ -187,6 +201,27 @@ String JsonObjectReader::readString() }); } +std::vector JsonObjectReader::readStringArray() +{ + return guarded([&] + { + std::vector words; + assertChar('[', in); + if (checkChar(']', in)) + return words; + + while (true) + { + String word; + readJSONString(word, in, jsonReadSettings()); + words.push_back(std::move(word)); + if (checkChar(']', in)) + return words; + assertChar(',', in); + } + }); +} + UInt128 JsonObjectReader::readHex128() { return guarded([&] diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 2ba6df5a1c5e..48f473c33d65 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -62,6 +63,9 @@ class CasJsonWriter /// Quoted JSON string with full escaping (bulk-run scan). Defined in CasTextFormat.cpp. void stringValue(std::string_view s); + /// JSON array of canonical word strings, emitted without intermediate storage. + void wordArray(std::span words); + void u64Number(uint64_t v) { char digits[24]; @@ -142,8 +146,9 @@ inline void writeIntText(uint64_t v, CasJsonWriter & out) { out.u64Number(v); } void writeHeaderLine(CasJsonWriter & out, FormatId id); void writeTrailerLine(CasJsonWriter & out, uint64_t n); -/// A wire-key carrier. The explicit constructor keeps raw string literals out of writer call -/// sites: a codec passes its named constant, and an inline `WireKey{"..."}` is deliberately loud. +/// A wire-key carrier for migrated writer call sites. Its explicit constructor makes a codec pass a +/// named constant, while an inline `WireKey{"..."}` is deliberately loud. `WireKey` borrows its +/// `string_view`; the referenced text must outlive the key, as with string literals and static constants. struct WireKey { std::string_view text; @@ -164,6 +169,12 @@ inline void writeWordField(CasJsonWriter & out, WireKey key, std::string_view wo writeStringValue(out, word); } +inline void writeWordArrayField(CasJsonWriter & out, WireKey key, std::span words, bool & first) +{ + writeKey(out, key, first); + out.wordArray(words); +} + inline void writeStringField(CasJsonWriter & out, WireKey key, std::string_view value, bool & first) { writeKey(out, key, first); @@ -220,6 +231,8 @@ class JsonObjectReader bool nextKey(String & key); /// Reads the value for the key returned by `nextKey` as a JSON string. String readString(); + /// Reads the value for the key returned by `nextKey` as an array of JSON strings. + std::vector readStringArray(); /// Reads a quoted 32-character lowercase hexadecimal string as a `UInt128`. UInt128 readHex128(); /// Reads a quoted decimal u64 string and rejects empty, trailing, or non-decimal text. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index e44bd062c33d..2b30d730aba5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -97,7 +97,7 @@ ManifestRef ManifestRefFields::buildRef(std::string_view what, std::string_view BlobRef BlobRefFields::build(std::string_view what) const { if (!algo_word || !digest_hex) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: blob ref missing ha/h", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: blob ref missing algo/digest", what); const BlobHashAlgo algo = blobHashAlgoFromWord(*algo_word, what); /// Validate the digest width before calling `fromHex`. A width mismatch otherwise produces /// `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed serialized input, @@ -114,4 +114,11 @@ BlobRef BlobRefFields::build(std::string_view what) const return BlobRef{algo, codecFor(algo).fromHex(*digest_hex)}; } +Token TokenFields::build(std::string_view what) const +{ + if (!type_word || !value) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: token missing token_type/token", what); + return Token{*value, tokenTypeFromWord(*type_word, what)}; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h index eea099e7711e..5ca327462192 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h @@ -29,8 +29,8 @@ inline constexpr EnumWireTable kObjectKindWords{{{ {ObjectKind::Blob, "blob"}, }}}; -/// Convert a token discriminator to its canonical wire word. Throws `CORRUPTED_DATA` if `t` is not -/// one of the token types understood by this build. +/// Convert a token discriminator to its canonical wire word. Throws `LOGICAL_ERROR` for an +/// out-of-range enum value. std::string_view tokenTypeToWord(TokenType t); /// Parse a canonical token-type word. `what` identifies the containing codec or field in the @@ -42,30 +42,30 @@ TokenType tokenTypeFromWord(std::string_view w, std::string_view what); /// `CORRUPTED_DATA` exception. BlobHashAlgo blobHashAlgoFromWord(std::string_view w, std::string_view what); -/// Convert an envelope object-kind discriminator to its canonical wire word. Throws -/// `CORRUPTED_DATA` if `k` is not represented by this format. +/// Convert an envelope object-kind discriminator to its canonical wire word. Throws `LOGICAL_ERROR` +/// for an out-of-range enum value. std::string_view objectKindToWord(ObjectKind k); /// Parse a canonical envelope object-kind word. `what` identifies the containing codec or field in /// the `CORRUPTED_DATA` exception; unknown words are rejected rather than treated as a default kind. ObjectKind objectKindFromWord(std::string_view w, std::string_view what); -/// Append the sibling fields `tt` and `tv` to an in-progress JSON object. The caller owns `first`, +/// Append the sibling fields `token_type` and `token` to an in-progress JSON object. The caller owns `first`, /// which must describe the fields already written to that object; the token value is JSON-escaped. void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t); -/// Append the sibling fields `ha` and `h` to an in-progress JSON object. The algorithm word and +/// Append the sibling fields `algo` and `digest` to an in-progress JSON object. The algorithm word and /// lowercase digest are canonical, and the digest is rendered at the width required by `r.algo`. void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r); -/// The `ha`/`h` and `tt`/`tv` key spellings, named once so `writeBlobRefFields`/`writeTokenFields` +/// The `algo`/`digest` and `token_type`/`token` key spellings, named once so `writeBlobRefFields`/`writeTokenFields` /// and the `match*Fields` collectors below can never drift apart on the literal. namespace SharedWire { - inline constexpr WireKey algo{"ha"}; - inline constexpr WireKey digest{"h"}; - inline constexpr WireKey token_type{"tt"}; - inline constexpr WireKey token{"tv"}; + inline constexpr WireKey algo{"algo"}; + inline constexpr WireKey digest{"digest"}; + inline constexpr WireKey token_type{"token_type"}; + inline constexpr WireKey token{"token"}; } /// One `ManifestRef`'s three flat key names. Every bundle spells the SAME wire representation @@ -79,13 +79,13 @@ struct ManifestRefWireKeys WireKey ord; }; -/// The unprefixed `me`/`mb`/`mo` spelling used by part manifests, snapshot rows, and the +/// The unprefixed `epoch`/`build`/`ord` spelling used by part manifests, snapshot rows, and the /// `set_published_at` ref-log op. -inline constexpr ManifestRefWireKeys kBareManifestRefKeys{WireKey{"me"}, WireKey{"mb"}, WireKey{"mo"}}; -/// The `ome`/`omb`/`omo` spelling for a ref-log owner_transition's OLD binding. -inline constexpr ManifestRefWireKeys kOldManifestRefKeys{WireKey{"ome"}, WireKey{"omb"}, WireKey{"omo"}}; -/// The `nme`/`nmb`/`nmo` spelling for a ref-log owner_transition's NEW binding. -inline constexpr ManifestRefWireKeys kNewManifestRefKeys{WireKey{"nme"}, WireKey{"nmb"}, WireKey{"nmo"}}; +inline constexpr ManifestRefWireKeys kBareManifestRefKeys{WireKey{"epoch"}, WireKey{"build"}, WireKey{"ord"}}; +/// The `old_epoch`/`old_build`/`old_ord` spelling for a ref-log owner_transition's OLD binding. +inline constexpr ManifestRefWireKeys kOldManifestRefKeys{WireKey{"old_epoch"}, WireKey{"old_build"}, WireKey{"old_ord"}}; +/// The `new_epoch`/`new_build`/`new_ord` spelling for a ref-log owner_transition's NEW binding. +inline constexpr ManifestRefWireKeys kNewManifestRefKeys{WireKey{"new_epoch"}, WireKey{"new_build"}, WireKey{"new_ord"}}; /// One owner binding's key names: the owner-kind word, the ref name, and its nested `ManifestRef` /// bundle. Only the ref-log owner_transition op uses this bundle (old/new binding sides). @@ -96,10 +96,10 @@ struct BindingWireKeys ManifestRefWireKeys manifest; }; -/// The `obk`/`orn`/`ome`/`omb`/`omo` spelling for the OLD binding side. -inline constexpr BindingWireKeys kOldBindingKeys{WireKey{"obk"}, WireKey{"orn"}, kOldManifestRefKeys}; -/// The `nbk`/`nrn`/`nme`/`nmb`/`nmo` spelling for the NEW binding side. -inline constexpr BindingWireKeys kNewBindingKeys{WireKey{"nbk"}, WireKey{"nrn"}, kNewManifestRefKeys}; +/// The `old_kind`/`old_ref`/`old_epoch`/`old_build`/`old_ord` spelling for the OLD binding side. +inline constexpr BindingWireKeys kOldBindingKeys{WireKey{"old_kind"}, WireKey{"old_ref"}, kOldManifestRefKeys}; +/// The `new_kind`/`new_ref`/`new_epoch`/`new_build`/`new_ord` spelling for the NEW binding side. +inline constexpr BindingWireKeys kNewBindingKeys{WireKey{"new_kind"}, WireKey{"new_ref"}, kNewManifestRefKeys}; /// Append the three flat `ManifestRef` fields named by `keys` to an in-progress JSON object. The /// two unbounded `uint64_t` values are decimal JSON strings; the bounded ordinal is a JSON number. @@ -132,7 +132,7 @@ struct ManifestRefFields ManifestRef buildRef(std::string_view what, std::string_view context) const; }; -/// Collector for one `BlobRef`'s two flat fields (`ha`/`h`), filled in by `matchBlobRefFields`. +/// Collector for one `BlobRef`'s two flat fields (`algo`/`digest`), filled in by `matchBlobRefFields`. struct BlobRefFields { std::optional algo_word; @@ -145,20 +145,22 @@ struct BlobRefFields BlobRef build(std::string_view what) const; }; -/// Collector for one `Token`'s two flat fields (`tt`/`tv`), filled in by `matchTokenFields`. Phase 1 -/// deliberately has no `build`: callers keep their own local requiredness checks until the unified -/// both-required build is introduced. +/// Collector for one `Token`'s two flat fields (`token_type`/`token`), filled in by `matchTokenFields`. struct TokenFields { std::optional type_word; std::optional value; + + /// Requires both fields and parses the token type word. `what` identifies the enclosing codec + /// in `CORRUPTED_DATA` exceptions. + Token build(std::string_view what) const; }; /// Each `match*Fields` helper tests `key` against the one or two field names it owns, consumes the /// value on a match via `r`, and reports whether it recognized the key. None of them loop over an /// object's keys or validate a completed group -- that is the caller's (tolerant-reader loop) and /// the collector's `build`/`buildRef` job respectively. Defined inline: a decoder's per-key dispatch -/// is a hot path and must not gain a function-call boundary here. +/// is a hot path, so the helpers are header-defined for the per-key dispatch to inline them. inline bool matchManifestRefFields(std::string_view key, JsonObjectReader & r, const ManifestRefWireKeys & keys, ManifestRefFields & fields) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index 7c4c99aa36fd..ea7a73ab41bf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -18,30 +18,52 @@ trailer, followed by a banner-framed raw payload zone for inline file bytes. | Key (under the pool prefix) | Object | Codec | Writer | |---|---|---|---| -| `_pool_meta` | pool identity + floors | `CasPoolMetaFormat` | pool create/admit | -| `cas/ns/stream//_log/…​.zst` | ref transaction log | `CasRefLogFormat` (`.zst`) | writer commit path | -| `cas/ns/stream//_snap/…​.zst` | complete ref table | `CasRefSnapshotFormat` (`.zst`) | writer/GC fold | -| `cas/ns/state//_ckpt` | mutable life checkpoint | `CasRefCkptFormat` | writer/GC fold | +| `_pool_meta` | pool identity + floors (`pool_id`, `blob_header_len`, `gc_shards`, `min_reader_generation`, `algos_used` array) | `CasPoolMetaFormat` | pool create/admit | +| `cas/ns/stream//_log/…​.zst` | ref transaction log (`namespace`, `txn_epoch`/`txn_seq`, optional critical `!prev_epoch`/`!prev_seq`; `set_published_at` uses `ref`/`published_ms`) | `CasRefLogFormat` (`.zst`) | writer commit path | +| `cas/ns/stream//_snap/…​.zst` | complete live ref table (`namespace`, `snapshot_epoch`/`snapshot_seq`, `lifecycle:"live"`; `kind` is `committed`/`precommit`, with `ref` and committed-only `published_ms`) | `CasRefSnapshotFormat` (`.zst`) | writer/GC fold | +| `cas/ns/state//_ckpt` | mutable life checkpoint (`life_epoch`, `committed_epoch`/`committed_seq`, `snapshot_epoch`/`snapshot_seq`, `seal_epoch`/`seal_seq`) | `CasRefCkptFormat` | writer/GC fold | | `cas/ns/state//_files/…​` | namespace-owned raw files | — | upper layers | -| `cas/manifests//-/.zst` | part manifest | `CasPartManifestFormat` | part build | -| blob keys (`CasLayout::blobKey`) | blob envelope + payload | `CasBlobEnvelopeFormat` | uploads | -| blob-meta keys (`CasLayout::blobMetaKey`) | freshness sidecar | `CasBlobMetaFormat` | dedup/GC | -| `gc/state`, `gc/hb` | GC state / leader heartbeat | `CasGcStateFormat` | GC | -| `gc/maintenance_state` | leak-only namespace-janitor cursor | `CasGcMaintenanceStateFormat` | future janitor | -| `gc/gen//attempt//outcomes/…​.zst` | outcome log | `CasGcOutcomesFormat` (`.zst`) | GC | -| `gc/gen//attempt//fold_seal` | fold seal (deterministic) | `CasFoldSealFormat` | GC | -| `gc/gen//…​/runs` | GC source-edge record-stream runs | `CasRecordStreamFormat` | GC | -| `gc/server-roots//{owner,epoch,mount}` | server-root singletons | `CasServerRootFormats` | mount | +| `cas/ref_catalog` | namespace lifecycle catalog (`kind:"entry"`, `ns`, `state`, `life`, `remove_round`, `creator`, `creator_epoch`, `creator_fence`) | `CasRefCatalogFormat` | namespace admission/removal | +| `cas/manifests//-/.zst` | part manifest (`root_namespace`, `payload_digest`; entry `path`, `place`, `size`) | `CasPartManifestFormat` | part build | +| blob keys (`CasLayout::blobKey`) | blob envelope (`type`, `v`, `tag`, `build`, `time_ms`, `creator`, `op`, `chver`, `ref`) + payload | `CasBlobEnvelopeFormat` | uploads | +| blob-meta keys (`CasLayout::blobMetaKey`) | freshness sidecar (`state`, `condemn_round`, `size`) | `CasBlobMetaFormat` | dedup/GC | +| `gc/state`, `gc/hb` | GC state (`round`, `gc_shards`, `snap_generation`, `snap_pruned_through`, `snap_attempt`, `manifest_sweep_cursor`, `lease_owner`, `lease_seq`) / heartbeat (`owner`, `hb_seq`) | `CasGcStateFormat` | GC | +| `gc/maintenance_state` | leak-only namespace-janitor cursor (`janitor_cursor`) | `CasGcMaintenanceStateFormat` | future janitor | +| `gc/gen//attempt//outcomes/…​.zst` | outcome log (`kind`, `outcome`) | `CasGcOutcomesFormat` (`.zst`) | GC | +| `gc/gen//attempt//fold_seal` | fold seal (deterministic; `generation`/`parent_generation`, `kind`: `ref_life`/`blob_run`/`condemned`) | `CasFoldSealFormat` | GC | +| `gc/gen//…​/runs` | GC source-edge record-stream runs (`ref`, `src`, `mark`; condemned: `pending`, `size`, `condemn_round`, `confirmed`) | `CasRecordStreamFormat` | GC | +| `gc/server-roots//{owner,epoch,mount}` | server-root singletons (`server_uuid`, optional `retired_at_ms`; `next_writer_epoch`; `server_uuid`, `writer_epoch`, `hostname`, `pid`, `started_at_ms`, `seq`, `expires_at_ms`, `min_active_build_sequence`, `gc_fenced`, `write_attempt_id`) | `CasServerRootFormats` | mount | | `roots/…` | raw passthrough (verbatim) | — (never interpreted) | upper layers | ## Codec table Authoritative per-format traits (type string, family, strictness, compression policy, caps) live -in `CasFormat.cpp` (`TRAITS`), asserted complete by `gtest_cas_text_format.cpp`. Key naming: keys -2–5 chars; fixed-width `UInt128` identities = 32-char lowercase hex strings; blob digests = -algo-width hex (two chars per digest byte), rendered with their algo name (`sha256:ab12…`) wherever -a bare hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/lengths/ms-timestamps -= numbers; units documented here per object as codecs land. +in `CasFormat.cpp` (`TRAITS`), asserted complete by `gtest_cas_text_format.cpp`. + +Key naming follows a deliberate split between metadata written once per object and fields repeated +once per record, not a flat character-count budget: + +- metadata written once per object (`namespace`, `writer_epoch`, `blob_header_len`, …) uses + descriptive names; +- fields repeated once per record (`ref`, `mark`, `op`, `class`, `place`, …) use short, semantic + words whose meaning is clear in the record rather than the C++ member name verbatim; +- the fixed `cas_blob` descriptor uses its own separately budgeted compact vocabulary (`tag`, + `build`, `chver`, …), because it must fit before the pool-wide fixed payload offset; +- common framing stays `type`, `v`, and `n`; +- `!` stays the must-understand prefix for critical fields; +- C++ member names obey an asymmetric rule: a member may be fuller than its wire key, never more + cryptic than it. + +Exact full C++ member names everywhere were deliberately rejected — see +`docs/superpowers/specs/2026-08-28-cas-semantic-wire-keys-design.md` ("Rejected alternatives"). +Fixed-width `UInt128` identities render as 32-char lowercase hex strings; blob digests render as +algo-width hex (two chars per digest byte), with their algo name (`sha256:ab12…`) wherever a bare +hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/lengths/ms-timestamps = +numbers; units documented here per object as codecs land. + +`CasWireVocab.{h,cpp}` owns repeated value fields: `BlobRef` uses `algo`/`digest`, `Token` uses +the jointly required `token_type`/`token`, `ManifestRef` uses `epoch`/`build`/`ord`, and owner-transition bindings use +the corresponding `old_*` and `new_*` key bundles. ## Evolution rules (one screen) @@ -62,18 +84,9 @@ a bare hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/l enforcement. In practice the mismatch never arises: `Always` objects are read via a constructed `.zst`-suffixed key, so a raw body is not GETtable at that key. -## Generation 10 mount-attempt identity {#generation-10-mount-attempt-identity} - -Generation 10 is a breaking, recreate-only change for the unreleased CAS format: - -- `MountLease` adds the required full-word key `write_attempt_id`, encoded as a nonzero 32-character - lowercase `UInt128` hex value. It identifies one holder-originated logical write; all physical - retries reuse the exact body and ID. A GC fence preserves the observed ID, while reclaim and - successor bodies mint a new one. The decoder rejects a missing or zero value. -- `FormatId::MountLease` has a breaking generation-10 change point because its canonical body gained - that required field. -- `FormatId::PoolMeta` has the matching generation-10 change point and reader floor. `decodePoolMeta` - rejects a generation-9 pool before any old mount body can be interpreted without attempt identity. +## Generation history {#generation-history} -There is no generation-9 decoder, compatibility alias, or migration. CAS is pre-release; recreate a -generation-9 pool with a generation-10 writer. +The format's generation history was reset to a flat `{1, 1}` baseline (`G_BUILD == 1`): CAS is +pre-release, carries no persisted data, and pays no compatibility cost for starting the count over. +Every class's `changePoints` begins at generation 1; a future breaking change appends a real entry to +that class's own array and bumps `G_BUILD`, the same way it always has. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index f9c9ca6f72a5..46db0402ed7a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -218,7 +218,7 @@ RefPlan buildRefWalkPlan(RoundInput && round_input) ++plan.dropped_holds; continue; } - it->second.fold_state.coverage.classification = 4; + it->second.fold_state.coverage.classification = CoverageClass::Clamped; it->second.fold_state.coverage.hold = hold; } for (const auto & [life_id, checkpoint] : ref_scan.checkpoint_observations) @@ -1194,8 +1194,8 @@ bool Gc::foldManifestEdges(const ManifestId & id, int sign, std::vector this namespace's frontier this round (normal end); /// absent, a listed id above it => impossible under contiguity, so the store is lying or a /// durable record was lost: HOLD the namespace at - /// classification 4 with its cursor unmoved. + /// classification `Clamped` with its cursor unmoved. /// /// Epochs are crossed only over a consumed `EpochSeal` (INV-2): the seal folds as an applied table /// no-op (probe B2 `produced=false`) and the next epoch's start is `{E', 1}`, reached through the @@ -2065,7 +2065,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & cursor_it != parent_ref_lives.end() ? cursor_it->second.coverage.hold : std::nullopt; RefCoverage cov; - cov.classification = 0; + cov.classification = CoverageClass::Absent; bool table_changed = false; /// THE FRONTIER PROOF for this namespace, and there is exactly one thing that establishes it: /// the walk read the expected-next position by exact key, found it ABSENT, and no witness put @@ -2577,7 +2577,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & : (same_position ? UINT32_MAX : 0); effective->next_retry_round = current_round + 1; cov.hold = effective; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; ++intake_tables_held; /// A held namespace is unproven BY DEFINITION -- the hold names a position the walk could /// not resolve, so everything at or above it is unaccounted. Stated here rather than left to @@ -2587,7 +2587,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & unproven_reason = FoldResult::FrontierUnproven::Held; } else - cov.classification = table_changed ? 2 : 1; + cov.classification = table_changed ? CoverageClass::Folded : CoverageClass::Unchanged; result.fold_seal.ref_lives.at(target.life_id).coverage = cov; ++result.frontier_namespaces; @@ -2646,7 +2646,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & for (const WalkTarget & target : walk_targets) { RefCoverage cov; - cov.classification = 1; + cov.classification = CoverageClass::Unchanged; if (const auto pit = parent_ref_lives.find(target.life_id); pit != parent_ref_lives.end()) { cov.last_folded_ref_id = pit->second.coverage.last_folded_ref_id; @@ -2656,7 +2656,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & if (pit->second.coverage.hold) { cov.hold = pit->second.coverage.hold; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; } } RefLifeFoldState & ref_life_state = result.fold_seal.ref_lives.at(target.life_id); @@ -4057,7 +4057,7 @@ RebuildReport Gc::rebuildBaseline(bool force) const RefTableState & st = recovered.state; RefCoverage cov; - cov.classification = 2; /// Folded (full coverage) unless a bodiless precommit clamps + cov.classification = CoverageClass::Folded; /// unless a bodiless precommit clamps it below cov.last_folded_ref_id = st.getGreatestApplied(); /// Whether the hold on this row was minted BY THIS REBUILD (and so still owes a retry round) /// rather than carried from the prior seal. Tracked explicitly instead of by looking for a @@ -4103,7 +4103,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// still missing -- and clears once the namespace makes durable progress. /// RESIDUAL, named rather than hidden: progress unrelated to this precommit also clears /// it, and the precommit's edges stay missing until another rebuild. - cov.classification = 4; /// Clamped + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::ManifestBodyMissing, .offending_position = RefTxnId{cov.last_folded_ref_id.writer_epoch, cov.last_folded_ref_id.ref_sequence + 1}, @@ -4126,7 +4126,7 @@ RebuildReport Gc::rebuildBaseline(bool force) const auto pit = prior_seal->ref_lives.find(life.incarnation); if (pit != prior_seal->ref_lives.end() && pit->second.coverage.hold) { - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = pit->second.coverage.hold; minted_here = false; /// a carried hold rides VERBATIM; its retry fields are not ours } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp index 17ba6337d1e4..1a4cdf2e551c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp @@ -40,7 +40,7 @@ void onGcEnumerationPage() /// namespace is rooted by `server_root_id`, but that id is a clean relative path and can contain slashes. /// Try namespace prefixes from longest to shortest and accept the first durable mount body. Without a /// mount there is no deletion authority, so the caller must leave the prefix untouched. The mount's -/// `writer_epoch` and `min_active` are the single durable epoch/floor pair used for eligibility, including +/// `writer_epoch` and `min_active_build_sequence` are the single durable epoch/floor pair used for eligibility, including /// across process replacement and the retired sentinel. std::optional floorForNamespace(Pool & store, const RootNamespace & ns) { @@ -420,21 +420,21 @@ bool manifestDeletionPremise(const NamespaceFoldView & view, const ManifestKey & /// UNCERTAINTY, hold arm. A hold names the exact position the fold could not resolve, and everything /// at or above it is unaccounted -- including, for all this predicate can tell, the record that - /// grants or removes this very manifest. `classification == 4` is tested separately from the hold - /// even though the seal's strict grammar pairs them: the thing standing between a clamped namespace - /// and an irreversible delete must not be a codec invariant enforced somewhere else. + /// grants or removes this very manifest. `classification == Clamped` is tested separately from the + /// hold even though the seal's strict grammar pairs them: the thing standing between a clamped + /// namespace and an irreversible delete must not be a codec invariant enforced somewhere else. if (cov.hold) return retain(SweepRetainClass::Hold, "namespace held at " + renderRefTxnId(cov.hold->offending_position) + " (" + String{holdReasonToWord(cov.hold->reason)} + ", retried " + std::to_string(cov.hold->retry_count) + " round(s)): every record at or above " "that position is unaccounted for"); - if (cov.classification == 4) + if (cov.classification == CoverageClass::Clamped) return retain(SweepRetainClass::Hold, - "namespace coverage is classified clamped (4) with no hold recorded: whatever " + "namespace coverage is classified clamped with no hold recorded: whatever " "stopped the fold was not carried, so nothing above its cursor is accounted for"); - if (cov.classification == 0) + if (cov.classification == CoverageClass::Absent) return retain(SweepRetainClass::NoCoverage, - "namespace coverage is classified absent (0): no round folded it, so its cursor " + "namespace coverage is classified absent: no round folded it, so its cursor " "is not the result of any walk"); /// RULE 1 (spec §6). Grants do not cross epochs, so every `+1` that could name an epoch-`E` build @@ -478,7 +478,7 @@ bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & /// Eligibility comes only from the durable mount-lease floor. A missing floor means NOT eligible; /// do not replace that authority check with a frozen-sequence or judged-dead guess. Compare /// `writer_epoch` first, then `build_sequence`, so old-epoch - /// debris drains after a process restart even when its build_sequence is above the current min_active. + /// debris drains after a process restart even when its build_sequence is above the current min_active_build_sequence. const auto floor = floorForNamespace(store, ns); if (!floor) return false; @@ -488,9 +488,9 @@ bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & return true; if (prefix.writer_epoch > w.writer_epoch) return false; - if (w.min_active == std::numeric_limits::max()) + if (w.min_active_build_sequence == std::numeric_limits::max()) return true; /// farewell/retired sentinel: every seq is retired - return w.min_active > prefix.build_sequence; + return w.min_active_build_sequence > prefix.build_sequence; } uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h index d2f44b205cb7..e5ae9e3176a4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h @@ -39,7 +39,7 @@ struct ManifestKey enum class SweepRetainClass : uint8_t { None = 0, /// the premise admitted the deletion; no retention happened - NoCoverage, /// no sealed coverage row for the namespace (a classification-0 row counts here) + NoCoverage, /// no sealed coverage row for the namespace (a classification-`Absent` row counts here) Hold, /// the namespace is held, or is classified clamped UnconsumedSeal, /// rule (1): the cursor has not consumed the build epoch's closing seal TailRemoval, /// rule (2): an unconsumed tail record names this manifest as a removal target @@ -153,7 +153,7 @@ struct ManifestSweepResult /// manifest bodies written before `PrecommitAdd` and never named by any live owner, scoped to ONE /// namespace + ONE build prefix. Rules: /// - eligibility from the durable watermark fact only: the retired sentinel -/// (`min_active == UINT64_MAX`), or `min_active > build_sequence`, or a replaced incarnation — +/// (`min_active_build_sequence == UINT64_MAX`), or `min_active_build_sequence > build_sequence`, or a replaced incarnation — /// NEVER a frozen-seq / judged-dead heuristic alone (a missing watermark => not eligible); /// - the active `ManifestId` set comes from the namespace's committed + live-precommit owner view; /// - delete only bodies whose `ManifestId` is ABSENT from the active set, by exact token; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp index 92c502cde3d0..d5424b4786a6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp @@ -1083,10 +1083,10 @@ void PartWriteTxn::abandon() /// `Uncertain` tolerance above, no-ops) -- it never corrupts. alive = false; - /// No longer in-flight: retire the seq so the per-server active-build floor (`min_active`) can advance + /// No longer in-flight: retire the seq so the per-server active-build floor (`min_active_build_sequence`) can advance /// (idempotent). This runs AFTER the precommit removal above (mirrors `PartWriteTxn::promote`, which retires /// after its commit) so the build stays active until its precommit binding's removal is durable: - /// retiring first would advance `min_active` past a build whose precommit binding is still live in the + /// retiring first would advance `min_active_build_sequence` past a build whose precommit binding is still live in the /// ref log, letting a freshness-window consumer judge the manifest build-dead while an un-removed /// precommit still names it. Ordering removal-before-retire keeps that happens-before clean. store->retireBuildSeq(build_seq); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 1ea5c2d567cb..ed6a957cbde6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -1458,7 +1459,7 @@ void Pool::enqueueWriterCleanupDuty( } catch (...) { - /// The build deliberately remains active. Advancing `min_active` after losing the only cleanup + /// The build deliberately remains active. Advancing `min_active_build_sequence` after losing the only cleanup /// duty would make an uncertain owner grant look dead; pinning the floor until process exit is /// the safe failure direction, and successor recovery handles the durable remnant. writer_cleanup_queue_failed.store(true, std::memory_order_release); @@ -1600,56 +1601,6 @@ BlobLocation Pool::locate(const ManifestEntry & entry) const return manifest_reader.locate(entry); } -namespace -{ -/// a tolerant, read-only peek at the -/// `cas_ref_log` TEXT object (codecs-v3 phase 3) WITHOUT `decodeRefLogTxn`'s expected-value cross-check -/// -- the whole point of this diagnostic is that the body is NOT expected to match this key's identity. -/// It `openObject`s the stored `.zst`, skips the header line, and reads `ns`/`we`/`rs` off the meta -/// line (`we`/`rs` are decimal u64 strings). Never validates the header `v`, never reads past the meta -/// line (the ops are irrelevant to identifying the writer), and swallows any truncation/garbage: this -/// is a background diagnostic only, never a decode anything else depends on. -struct ForeignRefLogHeaderPeek -{ - String ns; - uint64_t writer_epoch = 0; - uint64_t ref_sequence = 0; -}; - -std::optional peekForeignRefLogHeader(const String & bytes) -{ - try - { - const String text = openObject(FormatId::RefLog, bytes); - ReadBufferFromMemory in(text.data(), text.size()); - const uint64_t line_cap = traitsFor(FormatId::RefLog).line_cap; - readLine(in, line_cap, "cas_ref_log"); /// header line -- skip - const String meta = readLine(in, line_cap, "cas_ref_log"); - ReadBufferFromMemory m(meta.data(), meta.size()); - JsonObjectReader r(m, KeyStrictness::Tolerant, "cas_ref_log"); - ForeignRefLogHeaderPeek peek; - bool saw_ns = false; - bool saw_we = false; - bool saw_rs = false; - String key; - while (r.nextKey(key)) - { - if (key == "ns") { peek.ns = r.readString(); saw_ns = true; } - else if (key == "we") { peek.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { peek.ref_sequence = r.readU64String(); saw_rs = true; } - else r.skipUnknown(key); - } - if (!saw_ns || !saw_we || !saw_rs) - return std::nullopt; - return peek; - } - catch (...) - { - return std::nullopt; - } -} -} - void Pool::reportImpossibleInterference(const String & key, const String & reason, const std::optional & offending_ns) { @@ -1693,7 +1644,7 @@ void Pool::reportImpossibleInterference(const String & key, const String & reaso "time the background diagnostic GET ran", key); return; } - if (const auto peek = peekForeignRefLogHeader(got->bytes)) + if (const auto peek = peekRefLogMeta(got->bytes)) LOG_ERROR(getLogger("CasPool"), "CAS anomaly diagnostics: offending object at '{}' ({} bytes) decodes as a ref-log " "header: namespace='{}', writer_epoch={}, ref_sequence={}", diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 69eba5061fd3..a56588dcb305 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -427,7 +427,7 @@ class Pool : public std::enable_shared_from_this uint64_t minActive(); /// Test/assertion accessor for the next-to-allocate build_seq under the lock. uint64_t peekNextBuildSeq(); - /// Renew the merged heartbeat once (bump seq, refresh min_active from the live callback, stamp a + /// Renew the merged heartbeat once (bump seq, refresh min_active_build_sequence from the live callback, stamp a /// fresh expires_at_ms). The build-watermark floor rides this beat. In production this is driven by /// the background renewer (background_watermark). void renewWatermarkOnce(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp index 388d2a110521..34aaeaa37308 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp @@ -113,8 +113,8 @@ PoolMeta PoolMeta::createOrValidate( validatePoolBlobHeaderLen(blob_header_len, ErrorCodes::BAD_ARGUMENTS, "pool meta"); if (gc_shards == 0) throw Exception(ErrorCodes::BAD_ARGUMENTS, "CAS pool meta: gc_shards must be >= 1"); - /// Defense against a garbage `static_cast` past the caller's own boundary: `blobHashAlgoName` - /// throws BAD_ARGUMENTS for anything `BlobHashAlgo` does not actually admit. + /// `blobHashAlgoName` rejects an out-of-range `BlobHashAlgo` with `LOGICAL_ERROR`: a programming + /// error that aborts debug and sanitizer builds, not an input-validation fence. blobHashAlgoName(blob_hash_algo); const String key = layout.poolMetaKey(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 93b962c6980b..65bced4938e6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -755,7 +755,7 @@ uint64_t allocateWriterEpoch( const MountLease surviving = decodeMountLease(*mount_probe.body); /// Deliberately weaker than claimMount's reclaim gate (this file, ~:370-380), /// which never trusts a bare wall-clock comparison alone (only gc_fenced / - /// the clean-farewell min_active==UINT64_MAX marker / a caller-proven-dead + /// the clean-farewell min_active_build_sequence==UINT64_MAX marker / a caller-proven-dead /// token justify a reclaim there, because clock skew can misjudge liveness). /// This is still safe: (a) the mint below is DISTINCT from the survivor's /// epoch by construction, so no same-(uuid, epoch) pair is ever representable @@ -965,7 +965,7 @@ MountClaimResult claimMount( /// every renewal fails the token guard forever, so it can never write again) — there is no /// liveness left to wait for. This is what makes self-remount (and a fast restart after a /// fence-out) instant instead of an observation wait. - /// - the clean marker (`min_active == UINT64_MAX`) → the predecessor's OWN graceful farewell + /// - the clean marker (`min_active_build_sequence == UINT64_MAX`) → the predecessor's OWN graceful farewell /// (`MountLeaseKeeper::terminate`) — no observation needed either. /// - `proven_dead_token` matches the token we just read → the CALLER (`claimMountAwaitingExpiry`) /// already watched this exact token hold stable for the full observation threshold on its own @@ -974,7 +974,7 @@ MountClaimResult claimMount( /// Anything else → `LiveDoubleStart` (do NOT write): a same-uuid, different-epoch, not fenced, not /// clean-marked, not (yet) proven-dead lease may simply be a live twin, and `expires_at_ms` alone /// can never distinguish that from a dead predecessor across two different clocks. - const bool clean_marker = existing.min_active == std::numeric_limits::max(); + const bool clean_marker = existing.min_active_build_sequence == std::numeric_limits::max(); const bool proven_dead = proven_dead_token && *proven_dead_token == got->token; if (existing.gc_fenced || clean_marker || proven_dead) { @@ -1176,7 +1176,7 @@ HeartbeatFloor computeHeartbeatFloor(Backend & b, const Layout & l, uint64_t now obs.erase(srid); /// terminal — no further observation needed break; } - if (m.min_active == std::numeric_limits::max()) + if (m.min_active_build_sequence == std::numeric_limits::max()) { ++floor.terminated; obs.erase(srid); /// terminal — no further observation needed @@ -1283,7 +1283,7 @@ std::vector probeNonTerminalMountSlots(Backend & b, const continue; } - if (m.gc_fenced || m.min_active == std::numeric_limits::max()) + if (m.gc_fenced || m.min_active_build_sequence == std::numeric_limits::max()) continue; /// terminal: fenced out by GC, or the holder's own graceful farewell. slots.push_back(NonTerminalMountSlot{srid, fmt::format( @@ -1333,7 +1333,7 @@ std::vector listMounts(Backend & backend, const Layout & layout, uint } if (info.lease.gc_fenced) info.state = "fenced"; - else if (info.lease.min_active == std::numeric_limits::max()) + else if (info.lease.min_active_build_sequence == std::numeric_limits::max()) info.state = "terminated"; else if (now_ms <= info.lease.expires_at_ms + skew_margin_ms) info.state = "live"; @@ -1366,7 +1366,7 @@ FenceCertificate classifyFenceCertificate(const MountLease & lease, uint64_t fen { if (lease.gc_fenced) return FenceCertificate::GcFenced; - if (lease.min_active == std::numeric_limits::max()) + if (lease.min_active_build_sequence == std::numeric_limits::max()) return FenceCertificate::CleanFarewell; if (lease.writer_epoch != fence_writer_epoch) return FenceCertificate::SupersededEpoch; @@ -1419,7 +1419,7 @@ bool isCreatorFenceTerminal(Backend & backend, const Layout & layout, const Stri MountLeaseKeeper::MountLeaseKeeper( BackendPtr backend_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, - std::function min_active_fn_, + std::function min_active_build_sequence_fn_, CasEventSink event_sink_, std::chrono::milliseconds lease_safety_margin_, std::function boot_ms_fn_) @@ -1430,7 +1430,7 @@ MountLeaseKeeper::MountLeaseKeeper( , writer_epoch(writer_epoch_) , ttl(ttl_) , now_ms_fn(std::move(now_ms_fn_)) - , min_active_fn(std::move(min_active_fn_)) + , min_active_build_sequence_fn(std::move(min_active_build_sequence_fn_)) , event_sink(std::move(event_sink_)) , lease_safety_margin(lease_safety_margin_) , boot_ms_fn(boot_ms_fn_ ? std::move(boot_ms_fn_) : defaultBootMs) @@ -1438,7 +1438,7 @@ MountLeaseKeeper::MountLeaseKeeper( } String MountLeaseKeeper::encodeBody( - uint64_t seq_, uint64_t wall_ms, uint64_t min_active, UInt128 write_attempt_id) const + uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const { const uint64_t ttl_ms = static_cast(ttl.count()); const uint64_t expires_at_ms = wall_ms > std::numeric_limits::max() - ttl_ms @@ -1452,7 +1452,7 @@ String MountLeaseKeeper::encodeBody( .started_at_ms = wall_ms, .seq = seq_, .expires_at_ms = expires_at_ms, - .min_active = min_active, + .min_active_build_sequence = min_active_build_sequence, .write_attempt_id = write_attempt_id, }); } @@ -1544,7 +1544,7 @@ uint64_t MountLeaseKeeper::start() const uint64_t wall_ms = now_ms_fn(); const uint64_t attempt_start_boot_ms = boot_ms_fn(); - const String body = encodeBody(/*seq_=*/1, wall_ms, min_active_fn(), newMountWriteAttemptId()); + const String body = encodeBody(/*seq_=*/1, wall_ms, min_active_build_sequence_fn(), newMountWriteAttemptId()); const Token token = claim(body); seq = 1; @@ -1691,7 +1691,7 @@ MountRenewResult MountLeaseKeeper::renew( const uint64_t attempt_start_boot_ms = boot_clock(); const uint64_t next_seq = seq + 1; const UInt128 write_attempt_id = newMountWriteAttemptId(); - const String body = encodeBody(next_seq, wall_ms, min_active_fn(), write_attempt_id); + const String body = encodeBody(next_seq, wall_ms, min_active_build_sequence_fn(), write_attempt_id); const Token expected = last_token; const uint64_t safety_ms = static_cast(lease_safety_margin.count()); @@ -1829,7 +1829,7 @@ void MountLeaseKeeper::terminate() .started_at_ms = wall_ms, .seq = seq + 1, .expires_at_ms = wall_ms, - .min_active = std::numeric_limits::max(), + .min_active_build_sequence = std::numeric_limits::max(), .write_attempt_id = newMountWriteAttemptId(), }); const PutResult result = backend->putOverwrite(key, body, last_token); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index 5f19862e2348..6108141415dc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -181,7 +181,7 @@ uint64_t allocateWriterEpoch(Backend & b, const Layout & l, const String & srid, enum class MountPriorState { None, - Clean, /// the predecessor's own graceful farewell (`min_active == UINT64_MAX`) + Clean, /// the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) Fenced, /// the GC leader's own (already threshold-gated) fence-out (`gc_fenced`) UncleanObserved, /// OUR observation watched the write-token hold stable for the full threshold }; @@ -201,7 +201,7 @@ enum class MountPriorState /// `claimMountAwaitingExpiry` below for how a plain "looks expired" reading is turned into one): /// - `gc_fenced` (the GC leader already, itself, threshold-gated this incarnation dead; a fence /// costs an epoch, so its keeper can never renew again) → reclaim, `prior = Fenced`; -/// - the clean marker (`min_active == UINT64_MAX`, the predecessor's own graceful farewell) → +/// - the clean marker (`min_active_build_sequence == UINT64_MAX`, the predecessor's own graceful farewell) → /// reclaim, `prior = Clean`; /// - `proven_dead_token` matches the CURRENTLY OBSERVED token (the caller itself watched this /// exact token hold stable for the full observation threshold) → reclaim, `prior = @@ -325,7 +325,7 @@ using MountObservationMap = std::map; /// dead mounts (liveness only — graduation itself paces on GC rounds, not on heartbeat acks). /// Classification per body: /// - `gc_fenced` already set → excluded (`already_fenced`); a fenced mount is terminal, no PUT; -/// - terminated (`min_active == UINT64_MAX`, the farewell sentinel stamped by +/// - terminated (`min_active_build_sequence == UINT64_MAX`, the farewell sentinel stamped by /// `MountLeaseKeeper::terminate`) → excluded (`terminated`). `expires_at_ms` alone cannot /// distinguish a graceful farewell from an unclean stop, so the sentinel — not the timestamps — is the /// terminated marker; @@ -382,7 +382,7 @@ struct NonTerminalMountSlot /// still entitled to this prefix? A slot counts as terminal on exactly the two clock-free certificates /// the mount protocol already recognises (`computeHeartbeatFloor`'s own classification): `gc_fenced` /// (the GC leader fenced that incarnation out, and a fence costs an epoch, so its keeper can never -/// renew again) and `min_active == UINT64_MAX` (the holder's own graceful farewell). Everything else is +/// renew again) and `min_active_build_sequence == UINT64_MAX` (the holder's own graceful farewell). Everything else is /// reported, INCLUDING a body this build cannot decode -- an unreadable lease of some other format /// generation is precisely the case that must block, not the one to wave through. /// @@ -398,7 +398,7 @@ std::vector probeNonTerminalMountSlots(Backend & b, const /// A read-only snapshot of one server's mount slot, for introspection (`system.cas_mounts`). /// state: `live` (lease within TTL+skew), `expired` (lease ran out; the next GC round's heartbeat floor -/// will fence it), `terminated` (clean farewell: `min_active == UINT64_MAX`), `fenced` (`gc_fenced`), +/// will fence it), `terminated` (clean farewell: `min_active_build_sequence == UINT64_MAX`), `fenced` (`gc_fenced`), /// `corrupt` (body failed to decode — surfaced as a row, never an exception). struct MountInfo { @@ -436,7 +436,7 @@ std::vector listMounts(Backend & backend, const Layout & layout, uint /// identical question at pool-prefix and GC-heartbeat granularity — /// - `gc_fenced` (the GC leader already fenced this incarnation; a fence costs an epoch, so its /// keeper can never renew again), -/// - the clean-farewell sentinel `min_active == UINT64_MAX`, +/// - the clean-farewell sentinel `min_active_build_sequence == UINT64_MAX`, /// PLUS one more certificate available here that neither of those needs: a DIFFERENT `writer_epoch` /// currently live at that slot proves `writer_epoch`'s specific incarnation is superseded regardless of /// its OWN certificate — `allocateWriterEpoch`/`claimMount` are why an epoch, once superseded, is never @@ -487,7 +487,7 @@ class MountLeaseKeeper MountLeaseKeeper( BackendPtr backend_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, - std::function min_active_fn_, + std::function min_active_build_sequence_fn_, CasEventSink event_sink_ = {}, std::chrono::milliseconds lease_safety_margin_ = std::chrono::milliseconds(2000), /// boot-domain clock for the on_renew_ok anchor; empty = real CLOCK_BOOTTIME. Injectable for @@ -504,7 +504,7 @@ class MountLeaseKeeper uint64_t lastCommittedAttemptStartBootMs() const { return last_committed_attempt_start_boot_ms; } private: - String encodeBody(uint64_t seq_, uint64_t wall_ms, uint64_t min_active, UInt128 write_attempt_id) const; + String encodeBody(uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const; Token claim(const String & body); [[noreturn]] void throwRenewConflict(const CasOverwriteDiagnostics & diagnostics) const; MountRenewResult terminalResult( @@ -521,7 +521,7 @@ class MountLeaseKeeper uint64_t writer_epoch; std::chrono::milliseconds ttl; std::function now_ms_fn; - std::function min_active_fn; + std::function min_active_build_sequence_fn; CasEventSink event_sink; std::chrono::milliseconds lease_safety_margin; /// boot-domain clock for the on_renew_ok anchor; empty = real CLOCK_BOOTTIME. Injectable for diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h index 54360403b8d7..70f50f4e6682 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h @@ -57,8 +57,9 @@ std::string_view blobHashAlgoName(BlobHashAlgo algo); /// Returns the digest byte width for `algo`: 16 for `CityHash128` and `XXH3_128`, or 32 for /// `Sha256`. This is also the width used by `Cas::codecFor(algo)`'s `DigestCodec`; callers must -/// derive it from the algorithm rather than from pool state. Throws `BAD_ARGUMENTS` for an -/// out-of-range enum value, preserving the fail-closed contract of `blobHashAlgoName`. +/// derive it from the algorithm rather than from pool state. The functions over `BlobHashAlgo` +/// intentionally use different defensive codes: this one throws `BAD_ARGUMENTS` for an out-of-range +/// enum value, while `blobHashAlgoName` throws `LOGICAL_ERROR`. uint64_t blobHashLenFor(BlobHashAlgo algo); /// Parses the per-disk `blob_hash` CONFIG value: `"cityhash128"` | `"xxh3-128"` | `"sha256"`. Throws diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h index e59e13fcef6b..f2cacfa32f28 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h @@ -3,7 +3,8 @@ /// Compile-time coverage proof for EnumWireTable: SET EQUALITY with the enum's declared values. /// Size-plus-uniqueness is not enough (an invalid casted value satisfies both while an enumerator /// goes missing). This header pulls in magic_enum and therefore MUST be included only from .cpp -/// files and tests, never from another header. +/// files and tests, never from another header. The proof assumes every enumerator is in +/// magic_enum's reflectable range (by default -128..127), because `enum_values` sees only that range. #include diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index 416f5c327663..44a86a0c312b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -304,7 +304,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, } /// Graceful close stamps an already-expired lease and the watermark farewell - /// (`min_active = UINT64_MAX`), making the slot `terminated` before its mutable control objects + /// (`min_active_build_sequence = UINT64_MAX`), making the slot `terminated` before its mutable control objects /// are removed and its owner anchor is tombstoned. admin.reset(); @@ -334,7 +334,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const MountLease mount_value = decodeMountLease(farewell_mount->bytes); captures_match = epoch_value.next_writer_epoch != 0 && mount_value.writer_epoch == epoch_value.next_writer_epoch - 1 - && mount_value.min_active == std::numeric_limits::max() + && mount_value.min_active_build_sequence == std::numeric_limits::max() && !mount_value.gc_fenced; if (!captures_match) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index 4793eadd4e12..dda1c5921f2d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -168,7 +168,7 @@ String renderRefTableSnapshot(const RefTableSnapshot & s) .str(); } -/// The namespace's checkpoint (spec INV-4). Every field is optional and each absence means something +/// The namespace's checkpoint. Every field is optional and each absence means something /// different an operator needs to see: no `life_epoch` means no writer that knew this namespace's /// genesis epoch has written here yet, no `committed_through` means the life has no committed /// transaction, no `checkpoint_snapshot_id` means recovery has no snapshot base, and no @@ -258,7 +258,7 @@ String renderMountLease(const MountLease & m) .add("started_at_ms", jsonUInt(m.started_at_ms)) .add("seq", jsonUInt(m.seq)) .add("expires_at_ms", jsonUInt(m.expires_at_ms)) - .add("min_active", jsonUInt(m.min_active)) + .add("min_active_build_sequence", jsonUInt(m.min_active_build_sequence)) .add("gc_fenced", jsonBool(m.gc_fenced)) .add("write_attempt_id", jsonHex(m.write_attempt_id)) .str(); @@ -308,7 +308,7 @@ String renderRunRef(const RunRef & r) String renderRefCoverage(const RefCoverage & c) { return JsonObj() - .add("classification", jsonUInt(c.classification)) + .add("classification", jsonEscape(coverageClassToWord(c.classification))) .add("last_folded_ref_id", renderRefTxnIdObj(c.last_folded_ref_id)) .str(); } @@ -378,7 +378,7 @@ String renderEnvelopeHeader(const EnvelopeHeader & h) return JsonObj() .add("kind", jsonEscape(objectKindToWord(h.kind))) /// The blob identity is carried by the object key, so the envelope keeps only the provenance - /// fields needed for forensics (`ch` and `bld`) together with its compatibility version. + /// fields needed for forensics (`chver` and `build`) together with its compatibility version. .add("compatibility_version", jsonUInt(h.compatibility_version)) .add("incarnation_tag", jsonHex(h.incarnation_tag)) .add("build_id", jsonHex(h.build_id)) @@ -388,7 +388,7 @@ String renderEnvelopeHeader(const EnvelopeHeader & h) .str(); } -/// The word vocabulary a row's marker byte renders as, matching the `cas_run` NDJSON's own `"m"` field +/// The word vocabulary a row's marker byte renders as, matching the `cas_run` NDJSON's own `mark` field /// words (`runMarkerToWireWord`) so cas-inspect speaks the same vocabulary /// as the on-disk format rather than inventing a second one. String sourceEdgeRowKindName(RunMarker marker) diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 441ea58fc558..df987e1325ac 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -940,8 +940,8 @@ inline UInt128 catalogLifeIdForTest( /// real round. This is the durable fact the sweep's §6 deletion premise reads /// (`CasOrphanManifestSweep.cpp`): `cursor` is the namespace's `last_folded_ref_id`, and a manifest of /// an epoch-`E` build is deletable only once that cursor sits in an epoch STRICTLY above `E`. -/// `hold`, when set, makes the row classification 4 — the strict grammar `encodeFoldSeal` enforces in -/// both directions, so a hold and a non-4 classification cannot be seeded together. +/// `hold`, when set, makes the row classification `Clamped` — the strict grammar `encodeFoldSeal` +/// enforces in both directions, so a hold and a non-clamped classification cannot be seeded together. /// /// SHARP EDGE, HANDLED HERE SO NO CALLER HAS TO KNOW IT: a fold seal must carry a `condemned_summary` /// entry for EVERY shard in `0..gc_shards-1`. A later real round adopts this object as its PARENT and @@ -988,7 +988,7 @@ inline void seedFoldCursorForTest( seal.generation = generation; DB::Cas::RefCoverage cov; - cov.classification = hold ? 4 : 2; + cov.classification = hold ? DB::Cas::CoverageClass::Clamped : DB::Cas::CoverageClass::Folded; cov.last_folded_ref_id = cursor; cov.hold = hold; seal.ref_lives[life.incarnation].coverage = cov; @@ -1050,15 +1050,15 @@ inline uint64_t foldCursorOf( /// Set a server root's durable floor (so orphan-sweep eligibility can be driven). After the ack-floor /// merge the floor rides the mount lease body (`mountKey`), so this seeds a MountLease carrying -/// `{writer_epoch, min_active}` — exactly what `prefixEligible` reads. +/// `{writer_epoch, min_active_build_sequence}` — exactly what `prefixEligible` reads. inline void setWatermarkMinActive( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const String & server_root_id, - uint64_t writer_epoch, uint64_t min_active) + uint64_t writer_epoch, uint64_t min_active_build_sequence) { DB::Cas::MountLease m; m.server_uuid = DB::UInt128(0); m.writer_epoch = writer_epoch; - m.min_active = min_active; + m.min_active_build_sequence = min_active_build_sequence; m.seq = 1; m.write_attempt_id = DB::UInt128{1}; const String key = layout.mountKey(server_root_id); diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 495ed57f0325..25cd9577fe76 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -1,7 +1,11 @@ #include "cas_format_test_battery.h" #include +#include #include +#include + +#include #include using namespace DB::Cas; @@ -22,6 +26,47 @@ EnvelopeHeader sampleHeader(const String & ref) } constexpr uint32_t L = 256; +/// The `op` word with the most bytes on the wire, found by walking the enum through the REAL +/// public encoder-facing lookup (never by hardcoding "mutation") so a future longer word is +/// automatically picked up by the boundary tests below. +ProvenanceOp longestProvenanceOp() +{ + ProvenanceOp best = ProvenanceOp::Other; + size_t best_len = 0; + for (const auto op : magic_enum::enum_values()) + { + const size_t len = provenanceOpToWireWord(op).size(); + if (len > best_len) + { + best_len = len; + best = op; + } + } + return best; +} + +/// A header whose numeric provenance fields sit at their type maxima (`created_at_ms` at the +/// `uint64_t` max, `ch_version` at the `uint32_t` max, `op` at its longest wire word), so the +/// non-`ref` JSON this produces is the largest `encodeEnvelopeHeader` can emit for real field +/// values. `v` is not settable this way -- `encodeEnvelopeHeader` always stamps +/// `currentCompatibilityVersion()` -- so this is the worst case reachable through the real encoder +/// today, not the type-level bound `kMandatoryDescriptorWorstCase` proves for a hypothetical future +/// `v` at its own `uint32_t` maximum. +EnvelopeHeader maxReachableHeader(const String & ref) +{ + EnvelopeHeader h; + h.kind = ObjectKind::Blob; + h.incarnation_tag = hexToU128("0102030405060708090a0b0c0d0e0f10"); + h.build_id = hexToU128("1112131415161718191a1b1c1d1e1f20"); + h.provenance = Provenance{ + std::numeric_limits::max(), + hexToU128("2122232425262728292a2b2c2d2e2f30"), + std::numeric_limits::max(), + longestProvenanceOp()}; + h.intended_ref = ref; + return h; +} + /// The envelope has a fixed physical length. At generation 9 there is no unsupported one-digit /// version, so replacing `9` with `10` must consume one byte from the space pad rather than silently /// turning the 256-byte fixture into a different wire shape. @@ -53,8 +98,8 @@ TEST(CASBlobEnvelopeFormat, FixedLengthAndPadZone) EXPECT_EQ(head[L - 1], '\n'); /// terminator at byte 255 const String json = fmt::format(R"({{"type":"cas_blob","v":{},)", currentCompatibilityVersion()) + "\"tag\":\"0102030405060708090a0b0c0d0e0f10\"," - "\"bld\":\"1112131415161718191a1b1c1d1e1f20\",\"ts\":1752537600123," - "\"by\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"ch\":26006001," + "\"build\":\"1112131415161718191a1b1c1d1e1f20\",\"time_ms\":1752537600123," + "\"creator\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"chver\":26006001," "\"ref\":\"t-abc/all_1_2_0\"}"; ASSERT_LT(json.size(), L); EXPECT_EQ(head.substr(0, json.size()), json); /// '/' UNescaped (local escaper) @@ -93,6 +138,82 @@ TEST(CASBlobEnvelopeFormat, RefTruncatedToExactBudget) EXPECT_EQ(c, 'a'); } +TEST(CASBlobEnvelopeFormat, MandatoryWorstCaseBoundary) +{ + /// `kMandatoryDescriptorWorstCase` (239, proven at compile time against the 240 floor) assumes + /// `v` at its OWN type maximum (10 digits), because `currentCompatibilityVersion()` could grow + /// with a future generation. Nothing can make a running build emit that many digits today -- + /// `encodeEnvelopeHeader` always stamps the CURRENT `currentCompatibilityVersion()`, one digit at + /// this generation -- so the worst case reachable through the real encoder right now is 9 bytes + /// smaller: a 10-byte `ref` budget at the floor, not 1. That 9-byte gap is exactly + /// `kMaxU32DecimalLen - digit count of the current compatibility version`, so a generation that + /// reaches two digits narrows it and this expectation must be re-derived then -- the literal below + /// is deliberate, since deriving it from the version width here would restate the formula the + /// compile-time bound already owns and prove nothing about the encoder. + EnvelopeHeader h_floor = maxReachableHeader(""); + const String head_floor = encodeEnvelopeHeader(h_floor, static_cast(kMinBlobHeaderLen)); + ASSERT_EQ(head_floor.size(), kMinBlobHeaderLen); + EXPECT_EQ(head_floor[kMinBlobHeaderLen - 1], '\n'); + EXPECT_EQ(payloadOffset(decodeEnvelopeHeader(head_floor, head_floor.size(), ObjectKind::Blob)), kMinBlobHeaderLen); + const size_t json_len_floor = head_floor.find_last_not_of(' ', kMinBlobHeaderLen - 2) + 1; + const size_t budget_floor = (kMinBlobHeaderLen - 1) - json_len_floor; + EXPECT_EQ(budget_floor, 10u) << "ref budget reachable through the real encoder at the floor"; + + /// The default 256-byte header is exactly 16 bytes above the floor, so the SAME max-reachable + /// content leaves exactly 16 more bytes of `ref` budget. + EnvelopeHeader h_default = maxReachableHeader(""); + const String head_default = encodeEnvelopeHeader(h_default, L); + ASSERT_EQ(head_default.size(), L); + EXPECT_EQ(head_default[L - 1], '\n'); + EXPECT_EQ(payloadOffset(decodeEnvelopeHeader(head_default, head_default.size(), ObjectKind::Blob)), L); + const size_t json_len_default = head_default.find_last_not_of(' ', L - 2) + 1; + const size_t budget_default = (L - 1) - json_len_default; + EXPECT_EQ(budget_default, budget_floor + (L - kMinBlobHeaderLen)) + << "ref budget reachable through the real encoder at the default header length"; +} + +TEST(CASBlobEnvelopeFormat, CriticalKeyDescriptorStillFitsAtDefaultLength) +{ + /// The test-only `!x` critical key is written BEFORE `ref`; even at max-reachable field values + /// the descriptor still fits the default 256-byte header and fails closed as + /// UNKNOWN_FORMAT_VERSION, never CORRUPTED_DATA or a LOGICAL_ERROR from encode itself. + EnvelopeHeader h = maxReachableHeader("r"); + h.emit_unknown_critical_key = true; + const String head = encodeEnvelopeHeader(h, L); + ASSERT_EQ(head.size(), L); + cas_battery_detail::expectCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, + [&] { decodeEnvelopeHeader(head, head.size(), ObjectKind::Blob); }, + "critical-key blob envelope at max-reachable field values"); +} + +/// Closed-set pin: the six `ProvenanceOp` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASBlobEnvelopeFormat, ClosedSetPinsProvenanceOpWords) +{ + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Other), "other"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Insert), "insert"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Merge), "merge"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Mutation), "mutation"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Attach), "attach"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Repack), "repack"); + for (const auto op : magic_enum::enum_values()) + EXPECT_EQ(provenanceOpFromWireWord(provenanceOpToWireWord(op)), op); +} + +TEST(CASBlobEnvelopeFormat, UnknownOpWordFailsClosed) +{ + /// `op` is written as a plain (non-critical) key, so an unrecognized word is a decode-time + /// vocabulary violation, not a missing-extension one: CORRUPTED_DATA, not UNKNOWN_FORMAT_VERSION. + EnvelopeHeader h = sampleHeader("r"); + String head = encodeEnvelopeHeader(h, L); + const size_t op_at = head.find("\"op\":\"merge\""); + ASSERT_NE(op_at, String::npos); + head.replace(op_at, String("\"op\":\"merge\"").size(), "\"op\":\"bogus\""); + cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { decodeEnvelopeHeader(head, head.size(), ObjectKind::Blob); }, "unknown op word"); +} + TEST(CASBlobEnvelopeFormat, PadZoneSmugglingFailsClosed) { EnvelopeHeader h = sampleHeader("r"); @@ -163,8 +284,8 @@ TEST(CASFormatBattery, BlobEnvelope) /// the encoder to itself and pin nothing. const String json = fmt::format(R"({{"type":"cas_blob","v":{},)", currentCompatibilityVersion()) + "\"tag\":\"0102030405060708090a0b0c0d0e0f10\"," - "\"bld\":\"1112131415161718191a1b1c1d1e1f20\",\"ts\":1752537600123," - "\"by\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"ch\":26006001," + "\"build\":\"1112131415161718191a1b1c1d1e1f20\",\"time_ms\":1752537600123," + "\"creator\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"chver\":26006001," "\"ref\":\"t-abc/all_1_2_0\"}"; const String golden = json + String((L - 1) - json.size(), ' ') + '\n'; runFormatBattery(FormatBatteryCase{ diff --git a/src/Disks/tests/gtest_cas_blob_meta_format.cpp b/src/Disks/tests/gtest_cas_blob_meta_format.cpp index 59d1f7205dc4..581b56f1b070 100644 --- a/src/Disks/tests/gtest_cas_blob_meta_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta_format.cpp @@ -2,6 +2,8 @@ #include #include +#include + using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } @@ -39,8 +41,8 @@ TEST(CASFormatBattery, BlobMeta) .id = FormatId::BlobMeta, .encode = [&] { return sealObject(FormatId::BlobMeta, encodeBlobMeta(m)); }, .decode = [](std::string_view s) { decodeBlobMeta(std::string(openObject(FormatId::BlobMeta, s))); }, - .golden = "{\"type\":\"cas_blob_meta\",\"v\":10}\n" - "{\"st\":\"clean\",\"cr\":\"0\",\"sz\":\"12345\"}\n"}); + .golden = "{\"type\":\"cas_blob_meta\",\"v\":1}\n" + "{\"state\":\"clean\",\"condemn_round\":\"0\",\"size\":\"12345\"}\n"}); } TEST(CASBlobMetaFormat, CondemnedRoundTripAllFields) @@ -54,19 +56,30 @@ TEST(CASBlobMetaFormat, CondemnedRoundTripAllFields) EXPECT_EQ(back.condemn_round, 7u); EXPECT_EQ(back.size, 4096u); EXPECT_EQ(encodeBlobMeta(m), - "{\"type\":\"cas_blob_meta\",\"v\":10}\n{\"st\":\"condemned\",\"cr\":\"7\",\"sz\":\"4096\"}\n"); + "{\"type\":\"cas_blob_meta\",\"v\":1}\n{\"state\":\"condemned\",\"condemn_round\":\"7\",\"size\":\"4096\"}\n"); +} + +/// Closed-set pin: the two `MetaState` wire words, walked through `magic_enum::enum_values` so a +/// future state a `MetaState` construction can reach but no table entry names would fail this +/// exhaustive check rather than silently pass through unspecified. +TEST(CASBlobMetaFormat, ClosedSetPinsMetaStateWords) +{ + EXPECT_EQ(metaStateToWireWord(MetaState::Clean), "clean"); + EXPECT_EQ(metaStateToWireWord(MetaState::Condemned), "condemned"); + for (const auto state : magic_enum::enum_values()) + EXPECT_EQ(metaStateFromWireWord(metaStateToWireWord(state)), state); } TEST(CASBlobMetaFormat, FailsClosedOnUnknownStateAndTruncation) { /// Unknown state word -> CORRUPTED_DATA (mirrors the old `state > Condemned` reject). - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"st\":\"zombie\",\"cr\":\"0\",\"sz\":\"0\"}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":1}\n{\"state\":\"zombie\",\"condemn_round\":\"0\",\"size\":\"0\"}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(bad_state); }); /// Missing state key -> CORRUPTED_DATA. - const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"cr\":\"0\",\"sz\":\"0\"}\n"; + const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":1}\n{\"condemn_round\":\"0\",\"size\":\"0\"}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(no_state); }); /// Truncated (header only) -> CORRUPTED_DATA. - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":1}\n"); }); } diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 620dea710a3f..4f7d37e2b4fa 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -166,7 +166,7 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend if (armed && key == mount_key && result.outcome == PutOutcome::Done) { const MountLease mount = decodeMountLease(bytes); - if (mount.min_active == std::numeric_limits::max()) + if (mount.min_active_build_sequence == std::numeric_limits::max()) farewell_seen = true; } return result; @@ -201,7 +201,7 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend ++mount_value.seq; ++mount_value.started_at_ms; mount_value.expires_at_ms = mount_value.started_at_ms + 30'000; - mount_value.min_active = 0; + mount_value.min_active_build_sequence = 0; mount_value.gc_fenced = false; successor_mount_bytes = encodeMountLease(mount_value); const PutResult mount_put = InMemoryBackend::putOverwrite( @@ -277,7 +277,7 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend .started_at_ms = 1'000, .seq = 1, .expires_at_ms = 31'000, - .min_active = 0, + .min_active_build_sequence = 0, }); const PutResult mount_put = InMemoryBackend::putIfAbsent(mount_key, successor_mount_bytes, {}); if (mount_put.outcome != PutOutcome::Done) diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index 01f0b99a465a..45a075e50d3d 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -2,6 +2,10 @@ #include #include #include +#include +#include +#include +#include #include #include #include @@ -9,11 +13,40 @@ using namespace DB; using namespace DB::Cas; -/// These literals pin the CANONICAL BYTES of the CAS text encoders as of the commit that -/// introduced this file. The CasJsonWriter migration (2026-07-20 spec) must keep every one of -/// them green UNMODIFIED: canonical text is byte-compared on retries and deterministic adoption, -/// and the incremental ref budget counters assume these exact sizes. Never edit an expected -/// string here to make a test pass — that means the encoder's bytes drifted, which is the bug. +namespace +{ +String lineAt(const String & text, size_t index) +{ + size_t begin = 0; + for (size_t i = 0; i < index; ++i) + begin = text.find('\n', begin) + 1; + const size_t end = text.find('\n', begin); + return text.substr(begin, end - begin + 1); +} + +void expectDelta(const String & old_bytes, const String & new_bytes, size_t expected) +{ + EXPECT_EQ(new_bytes.size() - old_bytes.size(), expected) << "old: " << old_bytes << "new: " << new_bytes; +} + +CasFoldSeal oneFoldSeal() +{ + CasFoldSeal seal; + seal.generation = 5; + seal.parent_generation = 4; + return seal; +} +} + +/// The `CASEncodingPins` literals below pin the CANONICAL BYTES of the CAS text encoders: canonical +/// text is byte-compared on retries and deterministic adoption, and the incremental ref budget +/// counters assume these exact sizes. Never edit one of those expected strings to make a test pass — +/// that means the encoder's bytes drifted, which is the bug. +/// +/// The `CASWireCutDeltas` literals are the opposite kind: each is a HISTORICAL pre-cut row, kept so +/// the cost of the semantic-key rename stays measurable against what it replaced. They are +/// deliberately not the current bytes and must never be refreshed toward them — a delta measured +/// against today's encoder on both sides would always be zero. TEST(CASEncodingPins, RefLogTxnAllOpKinds) { @@ -49,13 +82,13 @@ TEST(CASEncodingPins, RefLogTxnAllOpKinds) txn.ops.push_back(removal); const String expected = fmt::format("{{\"type\":\"cas_ref_log\",\"v\":{}}}\n", currentCompatibilityVersion()) + - "{\"ns\":\"roots/pin\",\"we\":\"7\",\"rs\":\"9\"}\n" + "{\"namespace\":\"roots/pin\",\"txn_epoch\":\"7\",\"txn_seq\":\"9\"}\n" "{\"op\":\"namespace_birth\"}\n" - "{\"op\":\"owner_transition\",\"obk\":\"precommit\",\"orn\":\"20260101_0_1_1_1\"," - "\"ome\":\"1\",\"omb\":\"2\",\"omo\":3,\"nbk\":\"committed\",\"nrn\":\"20260101_0_1_1_1\"," - "\"nme\":\"1\",\"nmb\":\"2\",\"nmo\":3}\n" - "{\"op\":\"set_published_at\",\"rn\":\"20260101_0_1_1_1\\\"c\\nd\\u0001e\\u2028f\"," - "\"me\":\"1\",\"mb\":\"2\",\"mo\":3,\"ts\":1234}\n" + "{\"op\":\"owner_transition\",\"old_kind\":\"precommit\",\"old_ref\":\"20260101_0_1_1_1\"," + "\"old_epoch\":\"1\",\"old_build\":\"2\",\"old_ord\":3,\"new_kind\":\"committed\",\"new_ref\":\"20260101_0_1_1_1\"," + "\"new_epoch\":\"1\",\"new_build\":\"2\",\"new_ord\":3}\n" + "{\"op\":\"set_published_at\",\"ref\":\"20260101_0_1_1_1\\\"c\\nd\\u0001e\\u2028f\"," + "\"epoch\":\"1\",\"build\":\"2\",\"ord\":3,\"published_ms\":1234}\n" "{\"op\":\"remove_namespace\"}\n" "{\"n\":4}\n"; EXPECT_EQ(encodeRefLogTxn(txn), expected); @@ -76,9 +109,9 @@ TEST(CASEncodingPins, RefSnapshotLive) snap.precommits.push_back(RefOwnerBinding{RefOwnerKind::Precommit, "20260102_0_2_2_2", ManifestRef{4, 5, 6}}); const String expected = fmt::format("{{\"type\":\"cas_ref_snap\",\"v\":{}}}\n", currentCompatibilityVersion()) + - "{\"ns\":\"roots/pin\",\"we\":\"7\",\"rs\":\"9\",\"lc\":\"live\"}\n" - "{\"k\":\"c\",\"rn\":\"20260101_0_1_1_1\",\"me\":\"1\",\"mb\":\"2\",\"mo\":3,\"ts\":5}\n" - "{\"k\":\"p\",\"rn\":\"20260102_0_2_2_2\",\"me\":\"4\",\"mb\":\"5\",\"mo\":6}\n" + "{\"namespace\":\"roots/pin\",\"snapshot_epoch\":\"7\",\"snapshot_seq\":\"9\",\"lifecycle\":\"live\"}\n" + "{\"kind\":\"committed\",\"ref\":\"20260101_0_1_1_1\",\"epoch\":\"1\",\"build\":\"2\",\"ord\":3,\"published_ms\":5}\n" + "{\"kind\":\"precommit\",\"ref\":\"20260102_0_2_2_2\",\"epoch\":\"4\",\"build\":\"5\",\"ord\":6}\n" "{\"n\":2}\n"; EXPECT_EQ(encodeRefTableSnapshot(snap), expected); } @@ -108,16 +141,249 @@ TEST(CASEncodingPins, SourceEdgeRunLines) writer.finish(); out.finalize(); - /// The exact "b" rendering (algo byte + digest hex) is pinned as a whole line; the point is - /// that Task 8's line-scratch rewrite must reproduce it byte-for-byte. + /// The exact `ref` rendering (algo byte + digest hex) is pinned as a whole line; the point is + /// that the line-scratch rendering must reproduce it byte-for-byte. const String text = out.str(); const String header = fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()); const String expected_record = - "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; + "{\"ref\":\"0100000000000000000000000000000002\",\"src\":\"00000000000000000000000000000005\",\"mark\":\"edge\"}\n"; const String expected_condemned = - "{\"b\":\"0100000000000000000000000000000003\",\"s\":\"00000000000000000000000000000000\",\"m\":\"condemned\",\"pend\":true,\"tt\":\"etag\",\"tv\":\"token\",\"sz\":9,\"cr\":\"7\",\"mc\":true}\n"; + "{\"ref\":\"0100000000000000000000000000000003\",\"src\":\"00000000000000000000000000000000\",\"mark\":\"condemned\",\"pending\":true,\"token_type\":\"etag\",\"token\":\"token\",\"size\":9,\"condemn_round\":\"7\",\"confirmed\":true}\n"; const String trailer = "{\"n\":2}\n"; /// Both records must remain byte-identical to their canonical stored representation. const String expected_full = header + expected_record + expected_condemned + trailer; EXPECT_EQ(text, expected_full) << text; } + +TEST(CASWireCutDeltas, ActiveCasRunRow) +{ + SourceEdgeRecord record{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(2))}, .source_id = UInt128(5), .marker = RunMarker::Edge}; + WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(record); + writer.finish(); + out.finalize(); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; + expectDelta(old_bytes, lineAt(out.str(), 1), 7); +} + +TEST(CASWireCutDeltas, CondemnedCasRunRow) +{ + SourceEdgeRecord record{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(3))}, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = true, .token = Token{"token", TokenType::ETag}, .size = 9, .condemn_round = 7, .marker_confirmed = true}; + WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(record); + writer.finish(); + out.finalize(); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"b\":\"0100000000000000000000000000000003\",\"s\":\"00000000000000000000000000000000\",\"m\":\"condemned\",\"pend\":true,\"tt\":\"etag\",\"tv\":\"token\",\"sz\":9,\"cr\":\"7\",\"mc\":true}\n"; + expectDelta(old_bytes, lineAt(out.str(), 1), 41); +} + +TEST(CASWireCutDeltas, BlobPartManifestEntry) +{ + PartManifest manifest; + manifest.ref = ManifestRef{1, 2, 3}; + manifest.root_namespace_id = RootNamespace{"root"}; + manifest.entries = {ManifestEntry{"a", EntryPlacement::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, 9, {}}}; + const String text = encodePartManifest(manifest); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"p\":\"a\",\"pm\":\"blob\",\"ha\":\"ch128\",\"h\":\"00000000000000000000000000000004\",\"sz\":9}\n"; + expectDelta(old_bytes, lineAt(text, 2), 15); +} + +TEST(CASWireCutDeltas, InlinePartManifestEntry) +{ + PartManifest manifest; + manifest.ref = ManifestRef{1, 2, 3}; + manifest.root_namespace_id = RootNamespace{"root"}; + manifest.entries = {ManifestEntry{"a", EntryPlacement::Inline, {}, 0, "x"}}; + const String text = encodePartManifest(manifest); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"p\":\"a\",\"pm\":\"inline\",\"il\":1}\n"; + expectDelta(old_bytes, lineAt(text, 2), 8); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_banner = "==> \"a\" il=1 <==\n"; + expectDelta(old_banner, lineAt(text, 4), 2); +} + +TEST(CASWireCutDeltas, GcOutcomesRow) +{ + OutcomeLog log{{OutcomeEntry{ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, Token{"t", TokenType::ETag}, OutcomeKind::Deleted}}}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00000000000000000000000000000004\",\"tt\":\"etag\",\"tv\":\"t\",\"oc\":\"deleted\"}\n"; + expectDelta(old_bytes, lineAt(encodeOutcomeLog(log), 1), 26); +} + +/// The ref-log's own op rows: the highest-cardinality record of the format and, for +/// `owner_transition`, the largest single-row cost of the whole cut -- both old-side groups and both +/// new-side groups are renamed at once. +TEST(CASWireCutDeltas, OwnerTransitionRefLogOpRow) +{ + RefLogTxn txn; + txn.ns = "root"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = RefOpKind::OwnerTransition; + op.old_binding = RefOwnerBinding{RefOwnerKind::Precommit, "r", ManifestRef{3, 4, 5}}; + op.new_binding = RefOwnerBinding{RefOwnerKind::Committed, "r", ManifestRef{3, 4, 5}}; + txn.ops.push_back(op); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"op\":\"owner_transition\",\"obk\":\"precommit\",\"orn\":\"r\",\"ome\":\"3\",\"omb\":\"4\",\"omo\":5," + "\"nbk\":\"committed\",\"nrn\":\"r\",\"nme\":\"3\",\"nmb\":\"4\",\"nmo\":5}\n"; + expectDelta(old_bytes, lineAt(encodeRefLogTxn(txn), 2), 50); +} + +TEST(CASWireCutDeltas, SetPublishedAtRefLogOpRow) +{ + RefLogTxn txn; + txn.ns = "root"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = RefOpKind::SetPublishedAt; + op.ref_name = "r"; + op.expected_manifest_ref = ManifestRef{3, 4, 5}; + op.published_at_ms = 6; + txn.ops.push_back(op); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"op\":\"set_published_at\",\"rn\":\"r\",\"me\":\"3\",\"mb\":\"4\",\"mo\":5,\"ts\":6}\n"; + expectDelta(old_bytes, lineAt(encodeRefLogTxn(txn), 2), 18); +} + +/// The body-less ops are the cut's only free rows: the record is the `op` key alone. This measures +/// that the ROW costs nothing extra, not that the word itself is unchanged -- it builds its old side +/// from the current word, so it cannot see a word rename. The words are pinned literally by the +/// closed-set tests; what this adds is that no framing crept in around them. +TEST(CASWireCutDeltas, BodylessRefLogOpRowsAreUnchanged) +{ + for (const RefOpKind kind : {RefOpKind::NamespaceBirth, RefOpKind::EpochSeal}) + { + RefLogTxn txn; + txn.ns = "root"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = kind; + txn.ops.push_back(op); + const String old_bytes = fmt::format("{{\"op\":\"{}\"}}\n", refOpKindToWireWord(kind)); + expectDelta(old_bytes, lineAt(encodeRefLogTxn(txn), 2), 0); + } +} + +TEST(CASWireCutDeltas, CommittedRefSnapshotRow) +{ + RefTableSnapshot snapshot; + snapshot.ns = "root"; + snapshot.snapshot_id = RefTxnId{1, 2}; + snapshot.committed.push_back(RefCommittedRow{"r", ManifestRef{3, 4, 5}, 6}); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"c\",\"rn\":\"r\",\"me\":\"3\",\"mb\":\"4\",\"mo\":5,\"ts\":6}\n"; + expectDelta(old_bytes, lineAt(encodeRefTableSnapshot(snapshot), 2), 29); +} + +TEST(CASWireCutDeltas, PrecommitRefSnapshotRow) +{ + RefTableSnapshot snapshot; + snapshot.ns = "root"; + snapshot.snapshot_id = RefTxnId{1, 2}; + snapshot.precommits.push_back(RefOwnerBinding{RefOwnerKind::Precommit, "r", ManifestRef{3, 4, 5}}); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"p\",\"rn\":\"r\",\"me\":\"3\",\"mb\":\"4\",\"mo\":5}\n"; + expectDelta(old_bytes, lineAt(encodeRefTableSnapshot(snapshot), 2), 19); +} + +TEST(CASWireCutDeltas, BaseRefCatalogRow) +{ + RefCatalog catalog{{CatalogEntry{.ns = RootNamespace{"root"}, .state = NsState::Live, .incarnation = UInt128(7)}}}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"ent\",\"ns\":\"root\",\"st\":\"live\",\"inc\":\"00000000000000000000000000000007\"}\n"; + expectDelta(old_bytes, lineAt(encodeRefCatalog(catalog), 1), 9); +} + +/// The base row's delta is 22 bytes of keys and tags plus the `class` word, which costs one byte more +/// than its length (quotes, less the single numeric digit it replaces). Each word is measured +/// separately: a range over all four would accept a key rename hiding inside the spread, and a single +/// fixture would pin only one point of it. `clamped` cannot be a base row at all -- the grammar +/// requires a hold on exactly those rows -- so it is measured whole and its base part recovered by +/// subtracting the hold segment the next test pins. +TEST(CASWireCutDeltas, BaseRefLifeFoldSealRow) +{ + const auto base_delta = [](CoverageClass classification, uint8_t old_wire_value) + { + CasFoldSeal seal = oneFoldSeal(); + seal.ref_lives[UInt128(1)].coverage + = RefCoverage{.classification = classification, .last_folded_ref_id = RefTxnId{7, 11}}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = fmt::format( + "{{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":{},\"lfe\":\"7\",\"lfs\":\"11\"}}\n", + old_wire_value); + return lineAt(encodeFoldSeal(seal), 2).size() - old_bytes.size(); + }; + + /// The pre-cut wire numbered these 0/1/2, not the current enum's values. + EXPECT_EQ(base_delta(CoverageClass::Absent, 0), 29u); /// 22 + "absent" + EXPECT_EQ(base_delta(CoverageClass::Unchanged, 1), 32u); /// 22 + "unchanged" + EXPECT_EQ(base_delta(CoverageClass::Folded, 2), 29u); /// 22 + "folded" +} + +/// The ADDITIONS a hold contributes, isolated from the row it rides on. The with/without trick used +/// for cleanup evidence is unavailable here: the grammar requires a hold on exactly the clamped rows, +/// so a clamped row WITHOUT one cannot be encoded at all. Instead both sides are cut down to the hold +/// segment itself -- from its first key to the closing brace -- so the tag, the `class` word and the +/// fold pair are outside the comparison by construction rather than by cancellation. +TEST(CASWireCutDeltas, HoldBearingRefLifeAdditions) +{ + CasFoldSeal seal = oneFoldSeal(); + seal.ref_lives[UInt128(1)].coverage = RefCoverage{.classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{7, 11}, .hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{12, 13}, .retry_count = 14, .next_retry_round = 15}}; + + /// This literal is the pre-cut baseline this delta is measured against. + const String old_row = "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":4,\"lfe\":\"7\",\"lfs\":\"11\",\"hr\":\"gap_below_witness\",\"hpe\":\"12\",\"hps\":\"13\",\"hrc\":14,\"hnr\":\"15\"}\n"; + const String new_row = lineAt(encodeFoldSeal(seal), 2); + + const auto hold_segment = [](const String & row, std::string_view first_hold_key) + { + const size_t from = row.find(first_hold_key); + const size_t to = row.rfind('}'); + EXPECT_NE(from, String::npos) << "row does not carry " << first_hold_key << ": " << row; + EXPECT_NE(to, String::npos); + return to > from ? to - from : 0; + }; + + EXPECT_EQ(hold_segment(new_row, ",\"hold_reason\"") - hold_segment(old_row, ",\"hr\""), 33u); +} + +/// The cleanup-evidence pair isolated the same way: with and without, on both sides, so only the +/// two added keys remain in the difference. +TEST(CASWireCutDeltas, CleanupEvidenceRefLifeAdditions) +{ + CasFoldSeal without_evidence = oneFoldSeal(); + without_evidence.ref_lives[UInt128(1)] = RefLifeFoldState{.coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{7, 11}}}; + CasFoldSeal with_evidence = oneFoldSeal(); + with_evidence.ref_lives[UInt128(1)] = RefLifeFoldState{.coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{7, 11}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{12, 13}}}; + + /// These literals are the pre-cut baselines these deltas are measured against. + const String old_without = "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\"}\n"; + const String old_with = "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\",\"rte\":\"12\",\"rts\":\"13\"}\n"; + + const size_t base_delta = lineAt(encodeFoldSeal(without_evidence), 2).size() - old_without.size(); + const size_t whole_delta = lineAt(encodeFoldSeal(with_evidence), 2).size() - old_with.size(); + EXPECT_EQ(whole_delta - base_delta, 16u); +} + +TEST(CASWireCutDeltas, BlobRunFoldSealRow) +{ + CasFoldSeal seal = oneFoldSeal(); + seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(15), .shard = 0, .key_generation = 5}); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"btr\",\"key\":\"r0\",\"ck\":\"0000000000000000000000000000000f\",\"shard\":0,\"gen\":\"5\"}\n"; + expectDelta(old_bytes, lineAt(encodeFoldSeal(seal), 2), 25); +} + +TEST(CASWireCutDeltas, CondemnedFoldSealSummaryRow) +{ + CasFoldSeal seal = oneFoldSeal(); + seal.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 4}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"cnd\",\"shard\":0,\"ct\":3,\"pt\":1,\"ocr\":\"4\"}\n"; + expectDelta(old_bytes, lineAt(encodeFoldSeal(seal), 2), 30); +} diff --git a/src/Disks/tests/gtest_cas_enum_wire_table.cpp b/src/Disks/tests/gtest_cas_enum_wire_table.cpp index 90f59c866cb2..7940c57944eb 100644 --- a/src/Disks/tests/gtest_cas_enum_wire_table.cpp +++ b/src/Disks/tests/gtest_cas_enum_wire_table.cpp @@ -107,7 +107,7 @@ constexpr EnumWireTable invalid_value{{{ {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}, {static_cast(99), "plum"}}}}; static_assert(!casEnumTableCoversEnum()); -/// The two cases above fail the folded density check before the set-equality core runs, so the +/// `dup_value` and `invalid_value` fail the folded density check before the set-equality core runs, so the /// core needs its own failing witnesses — both dense and word-unique, so they reach it. /// Reaches the size comparison: one enumerator short. constexpr EnumWireTable missing_enumerator{{{ diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index 808830e46a9f..82920acb177c 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -554,7 +554,7 @@ String publishOneBlobPart(const PoolPtr & s, const String & ns, const String & r /// Whether the CURRENT retired list (any gc-shard) still holds an entry (ack-floor pipeline in flight). bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_fold_seal_codec.cpp b/src/Disks/tests/gtest_cas_fold_seal_codec.cpp index feee48a70069..dabde6cf3b82 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_codec.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_codec.cpp @@ -23,7 +23,7 @@ TEST(CASFoldSealCodec, RefLifeCoverageRoundTripsLastFoldedRefId) seal.generation = 3; seal.parent_generation = 2; RefCoverage cov; - cov.classification = 1; + cov.classification = CoverageClass::Unchanged; cov.last_folded_ref_id = RefTxnId{4, 11}; constexpr UInt128 life_id{1}; seal.ref_lives[life_id].coverage = cov; diff --git a/src/Disks/tests/gtest_cas_fold_seal_format.cpp b/src/Disks/tests/gtest_cas_fold_seal_format.cpp index d5e2144093eb..ba8c6f131be9 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_format.cpp @@ -4,6 +4,8 @@ #include #include +#include + using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; extern const int LOGICAL_ERROR; } @@ -15,8 +17,8 @@ CasFoldSeal sampleFoldSeal() CasFoldSeal seal; seal.generation = 7; seal.parent_generation = 6; - seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{3, 4}}; - seal.ref_lives[UInt128{2}].coverage = RefCoverage{.classification = 1}; + seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{3, 4}}; + seal.ref_lives[UInt128{2}].coverage = RefCoverage{.classification = CoverageClass::Unchanged}; seal.blob_target_runs.push_back(RunRef{.key = "gc/gen/7/blob_target/0/0", .checksum = UInt128(0xABCDEF)}); return seal; } @@ -36,7 +38,7 @@ TEST(CASFormatBattery, FoldSeal) CasFoldSeal seal; seal.generation = 5; seal.parent_generation = 4; - seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{7, 11}}; + seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{7, 11}}; seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(0x0f), .shard = 0, .key_generation = 5}); seal.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 4}; @@ -44,10 +46,10 @@ TEST(CASFormatBattery, FoldSeal) [&] { return sealObject(FormatId::FoldSeal, encodeFoldSeal(seal)); }, [](std::string_view s) { decodeFoldSeal(std::string(openObject(FormatId::FoldSeal, s))); }, currentFormatHeader("cas_fold_seal") + - "{\"g\":\"5\",\"pg\":\"4\"}\n" - "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\"}\n" - "{\"k\":\"btr\",\"key\":\"r0\",\"ck\":\"0000000000000000000000000000000f\",\"shard\":0,\"gen\":\"5\"}\n" - "{\"k\":\"cnd\",\"shard\":0,\"ct\":3,\"pt\":1,\"ocr\":\"4\"}\n" + "{\"generation\":\"5\",\"parent_generation\":\"4\"}\n" + "{\"kind\":\"ref_life\",\"life\":\"00000000000000000000000000000001\",\"class\":\"folded\",\"fold_epoch\":\"7\",\"fold_seq\":\"11\"}\n" + "{\"kind\":\"blob_run\",\"key\":\"r0\",\"checksum\":\"0000000000000000000000000000000f\",\"shard\":0,\"key_generation\":\"5\"}\n" + "{\"kind\":\"condemned\",\"shard\":0,\"condemned\":3,\"pending\":1,\"oldest_round\":\"4\"}\n" "{\"n\":3}\n"}); } @@ -59,7 +61,7 @@ TEST(CASFoldSealFormat, RoundTripsAllFields) EXPECT_EQ(out.generation, in.generation); EXPECT_EQ(out.parent_generation, in.parent_generation); ASSERT_EQ(out.ref_lives.size(), in.ref_lives.size()); - EXPECT_EQ(out.ref_lives.at(UInt128{1}).coverage.classification, 2); + EXPECT_EQ(out.ref_lives.at(UInt128{1}).coverage.classification, CoverageClass::Folded); EXPECT_EQ(out.ref_lives.at(UInt128{1}).coverage.last_folded_ref_id, (RefTxnId{3, 4})); ASSERT_EQ(out.blob_target_runs.size(), 1u); EXPECT_EQ(out.blob_target_runs[0].key, "gc/gen/7/blob_target/0/0"); @@ -125,11 +127,11 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel for (const std::string_view field : { R"(,"key":"p/gc/gen/7/attempt/1/blob_target/0/0")", - R"(,"ck":"00000000000000000000000000000001")", - R"(,"gen":"7")", - ",\"ct\":0", - ",\"pt\":0", - R"(,"ocr":"18446744073709551615")"}) + R"(,"checksum":"00000000000000000000000000000001")", + R"(,"key_generation":"7")", + ",\"condemned\":0", + ",\"pending\":0", + R"(,"oldest_round":"18446744073709551615")"}) { String malformed = valid; eraseRequiredField(malformed, field); @@ -138,19 +140,19 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel } /// `shard` occurs once on each row; remove each occurrence independently. - String missing_btr_shard = valid; - eraseRequiredField(missing_btr_shard, ",\"shard\":0"); + String missing_blob_run_shard = valid; + eraseRequiredField(missing_blob_run_shard, ",\"shard\":0"); cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(missing_btr_shard, layout, 1); }, "missing"); + [&] { decodeFoldSeal(missing_blob_run_shard, layout, 1); }, "missing"); - String missing_cnd_shard = valid; - const size_t first_shard = missing_cnd_shard.find(",\"shard\":0"); + String missing_condemned_shard = valid; + const size_t first_shard = missing_condemned_shard.find(",\"shard\":0"); ASSERT_NE(first_shard, String::npos); - const size_t second_shard = missing_cnd_shard.find(",\"shard\":0", first_shard + 1); + const size_t second_shard = missing_condemned_shard.find(",\"shard\":0", first_shard + 1); ASSERT_NE(second_shard, String::npos); - missing_cnd_shard.erase(second_shard, std::string_view(",\"shard\":0").size()); + missing_condemned_shard.erase(second_shard, std::string_view(",\"shard\":0").size()); cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(missing_cnd_shard, layout, 1); }, "missing"); + [&] { decodeFoldSeal(missing_condemned_shard, layout, 1); }, "missing"); } TEST(CASFoldSealFormat, AuthoritativeDecodeRejectsNoncanonicalRowsAndIncompleteSummaryDomain) @@ -241,7 +243,7 @@ TEST(CASFoldSeal, RejectsEmptyAndBadMagic) TEST(CASFoldSeal, CoverageRecordsEveryCatalogLife) { CasFoldSeal in = sampleFoldSeal(); - in.ref_lives[UInt128{3}].coverage = RefCoverage{.classification = 0}; + in.ref_lives[UInt128{3}].coverage = RefCoverage{.classification = CoverageClass::Absent}; const CasFoldSeal out = decodeFoldSeal(encodeFoldSeal(in)); EXPECT_TRUE(out.ref_lives.contains(UInt128{3})); EXPECT_EQ(out.ref_lives.size(), 3u); @@ -254,7 +256,7 @@ TEST(CASFoldSeal, FoldSealCondemnedSummaryRoundTrips) CasFoldSeal s; s.generation = 9; s.parent_generation = 8; - s.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2}; + s.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Folded}; s.blob_target_runs.push_back(RunRef{.key = "gc/gen/9/blob_target/0/0", .checksum = UInt128(0x77), .shard = 0, .key_generation = 9}); s.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, @@ -283,7 +285,7 @@ TEST(CASFoldSealFormat, UnifiedRefLifeRowRoundTripsCoverageHoldAndCleanupEvidenc const UInt128 life_id{0x1234}; seal.ref_lives.emplace(life_id, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{3, 4}, .hold = RefHold{ .reason = HoldReason::ManifestBodyMissing, @@ -293,36 +295,59 @@ TEST(CASFoldSealFormat, UnifiedRefLifeRowRoundTripsCoverageHoldAndCleanupEvidenc .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{9, 10}}}); const String expected = currentFormatHeader("cas_fold_seal") + - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000001234\",\"cls\":4," - "\"lfe\":\"3\",\"lfs\":\"4\",\"hr\":\"manifest_body_missing\",\"hpe\":\"5\"," - "\"hps\":\"6\",\"hrc\":7,\"hnr\":\"8\",\"rte\":\"9\",\"rts\":\"10\"}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"ref_life\",\"life\":\"00000000000000000000000000001234\",\"class\":\"clamped\"," + "\"fold_epoch\":\"3\",\"fold_seq\":\"4\",\"hold_reason\":\"manifest_body_missing\",\"hold_epoch\":\"5\"," + "\"hold_seq\":\"6\",\"retries\":7,\"retry_round\":\"8\",\"remove_epoch\":\"9\",\"remove_seq\":\"10\"}\n" "{\"n\":1}\n"; EXPECT_EQ(encodeFoldSeal(seal), expected); EXPECT_EQ(decodeFoldSeal(expected), seal); } -/// Mutation caught: accepting the generation-6 split coverage collection would leave a second -/// namespace-keyed source of lifecycle work in a generation-7 process. +/// Closed-set pin: `CoverageClass` and `HoldReason` +/// each walked through `magic_enum::enum_values`, which is what proves the renderer and the parser +/// consult the SAME table: a table entry missing altogether is already a build error at the +/// coverage assert, but two delegates drifting onto different tables is not. +TEST(CASFoldSealFormat, ClosedSetPinsCoverageClassAndHoldReasonWords) +{ + EXPECT_EQ(coverageClassToWord(CoverageClass::Absent), "absent"); + EXPECT_EQ(coverageClassToWord(CoverageClass::Unchanged), "unchanged"); + EXPECT_EQ(coverageClassToWord(CoverageClass::Folded), "folded"); + EXPECT_EQ(coverageClassToWord(CoverageClass::Clamped), "clamped"); + for (const auto c : magic_enum::enum_values()) + EXPECT_EQ(coverageClassFromWord(coverageClassToWord(c)), c); + + EXPECT_EQ(holdReasonToWord(HoldReason::GapBelowWitness), "gap_below_witness"); + EXPECT_EQ(holdReasonToWord(HoldReason::UnconsumedSealCrossing), "unconsumed_seal_crossing"); + EXPECT_EQ(holdReasonToWord(HoldReason::WitnessDisappeared), "witness_disappeared"); + EXPECT_EQ(holdReasonToWord(HoldReason::BodyUndecodable), "body_undecodable"); + EXPECT_EQ(holdReasonToWord(HoldReason::ManifestBodyMissing), "manifest_body_missing"); + EXPECT_EQ(holdReasonToWord(HoldReason::CheckpointUndecodable), "checkpoint_undecodable"); + for (const auto r : magic_enum::enum_values()) + EXPECT_EQ(holdReasonFromWord(holdReasonToWord(r)), r); +} + +/// Mutation caught: accepting the retired split coverage-collection kind would revive a second +/// namespace-keyed source of lifecycle work alongside the unified per-life row. TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyCoverageRecord) { const String old = - "{\"type\":\"cas_fold_seal\",\"v\":7}\n" - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"cov\",\"key\":\"name/0\",\"cls\":2,\"lfe\":\"3\",\"lfs\":\"4\"}\n" + "{\"type\":\"cas_fold_seal\",\"v\":1}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"cov\",\"key\":\"name/0\",\"class\":\"folded\",\"fold_epoch\":\"3\",\"fold_seq\":\"4\"}\n" "{\"n\":1}\n"; cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(old); }, "legacy coverage"); } -/// Mutation caught: accepting the generation-6 cleanup-item state would restore the independent -/// marker-driven `Pending`/`Completed` handshake. +/// Mutation caught: accepting the retired cleanup-item kind would restore the independent +/// marker-driven `Pending`/`Completed` handshake the unified row replaced. TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyNamespaceCleanupRecord) { const String old = - "{\"type\":\"cas_fold_seal\",\"v\":7}\n" - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"nsc\",\"ns\":\"name\",\"rte\":\"3\",\"rts\":\"4\",\"st\":\"completed\"}\n" + "{\"type\":\"cas_fold_seal\",\"v\":1}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"nsc\",\"ns\":\"name\",\"remove_epoch\":\"3\",\"remove_seq\":\"4\",\"st\":\"completed\"}\n" "{\"n\":1}\n"; cas_battery_detail::expectCode( DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(old); }, "legacy namespace cleanup"); diff --git a/src/Disks/tests/gtest_cas_forget.cpp b/src/Disks/tests/gtest_cas_forget.cpp index 2d8a72fc1850..b1c19993e06c 100644 --- a/src/Disks/tests/gtest_cas_forget.cpp +++ b/src/Disks/tests/gtest_cas_forget.cpp @@ -306,7 +306,7 @@ TEST(CASForget, ForgetOnIdentityLostPoolVanishesForgotten) } /// (a'') The clean-farewell is EARNED, never unconditional: on a drained pool FORGET stamps the mount lease -/// with the terminated sentinel (`min_active == UINT64_MAX`) so a same-server restart reclaims immediately, +/// with the terminated sentinel (`min_active_build_sequence == UINT64_MAX`) so a same-server restart reclaims immediately, /// but with an UNSETTLED (wedged) ref lane it must NOT — the lease is left to expire by observation. TEST(CASForget, ForgetCleanFarewellGatedOnDrain) { @@ -318,14 +318,14 @@ TEST(CASForget, ForgetCleanFarewellGatedOnDrain) auto backend = std::make_shared(); auto store = DB::Cas::tests::openPoolForTest(backend); const String mount_key = store->layout().mountKey(kSrid); - ASSERT_NE(decodeMountLease(backend->get(mount_key)->bytes).min_active, kTerminated); /// baseline + ASSERT_NE(decodeMountLease(backend->get(mount_key)->bytes).min_active_build_sequence, kTerminated); /// baseline store->forgetDisk([] {}, kForgetReason); ASSERT_EQ(store->lifecycle(), PoolLifecycle::VanishedForgotten); const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()); - EXPECT_EQ(decodeMountLease(got->bytes).min_active, kTerminated) + EXPECT_EQ(decodeMountLease(got->bytes).min_active_build_sequence, kTerminated) << "a drained FORGET earns the clean-release farewell"; } @@ -344,7 +344,7 @@ TEST(CASForget, ForgetCleanFarewellGatedOnDrain) const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()) << "the lease object must still be present (expiry by observation)"; - EXPECT_NE(decodeMountLease(got->bytes).min_active, kTerminated) + EXPECT_NE(decodeMountLease(got->bytes).min_active_build_sequence, kTerminated) << "an unearned clean farewell must NOT be written when the ref lanes did not drain"; } } diff --git a/src/Disks/tests/gtest_cas_format.cpp b/src/Disks/tests/gtest_cas_format.cpp index 9aa93e99f76f..6c5ad84cc0cc 100644 --- a/src/Disks/tests/gtest_cas_format.cpp +++ b/src/Disks/tests/gtest_cas_format.cpp @@ -1,7 +1,8 @@ #include #include -#include #include +#include +#include namespace DB::ErrorCodes { @@ -11,93 +12,52 @@ namespace DB::ErrorCodes using namespace DB::Cas; -TEST(CASFormat, ChangePointsExistForEveryClass) +/// Closed-set pin: the registry's complete set of object `type` strings. `allRegisteredFormatIds` +/// is the registry's own enumeration accessor, so this walks the SAME set the codecs and the object +/// header gate see -- a registered class with no test coverage here is a registered class this test +/// cannot see either, which is the point: a 17th, 18th, ... entry the spec's closed set does not +/// name would show up as a set-size mismatch instead of passing unnoticed. +TEST(CASFormat, RegistryTypeStringsArePinnedClosedSet) { - /// Every class that existed from the start has a non-empty, gen-1 baseline. - for (auto id : {FormatId::Blob, - FormatId::GcState, - FormatId::PoolMeta, FormatId::Roster, - FormatId::GcOutcomes, - FormatId::PartManifest, FormatId::RunFile, - FormatId::FoldSeal}) - { - auto cps = changePoints(id); - ASSERT_FALSE(cps.empty()); - EXPECT_EQ(cps.front().generation, 1u); - EXPECT_EQ(cps.front().min_reader, 1u); - } -} - -/// A class BORN after generation 1 begins its history at its birth generation, not at 1. `RefCkpt` -/// (spec INV-4) was introduced at generation 4: there is no such thing as a generation-1 `_ckpt`, and a -/// `{1, 1}` baseline would assert that a generation-1 reader could read one. Its history then gained a -/// three later breaking entries: generation 5 re-keyed it under `//`, generation 6 -/// moved it to opaque life-owned state, and generation 9 added the exact committed frontier. Neither -/// change touches the gen-1 baseline. Pinned because the decision is -/// invisible otherwise — nothing consults `changePoints` at decode time yet, so a wrong entry here -/// would sit unnoticed until the day a per-class reader floor is wired and starts admitting objects it -/// should refuse. -TEST(CASFormat, ChangePointsOfAClassBornAfterGenerationOneStartAtItsBirth) -{ - const auto cps = changePoints(FormatId::RefCkpt); - ASSERT_EQ(cps.size(), 4u); - EXPECT_EQ(cps.front().generation, kContiguousRefStreamsGeneration); - EXPECT_EQ(cps.front().min_reader, kContiguousRefStreamsGeneration); - EXPECT_GT(cps.front().generation, 1u) << "the point of this test is that it is NOT the gen-1 baseline"; - EXPECT_EQ(cps[1].generation, kNamespaceLifeKeyedGeneration); - EXPECT_EQ(cps[1].min_reader, kNamespaceLifeKeyedGeneration); - EXPECT_EQ(cps[2].generation, kOpaqueNamespaceLifeLayoutGeneration); - EXPECT_EQ(cps[2].min_reader, kOpaqueNamespaceLifeLayoutGeneration); - EXPECT_EQ(cps.back().generation, kCommittedRefFrontierGeneration); - EXPECT_EQ(cps.back().min_reader, kCommittedRefFrontierGeneration); -} - -TEST(CASFormat, PoolMetaTracksTheRecreateOnlyRecoveryFrontierGeneration) -{ - const auto cps = changePoints(FormatId::PoolMeta); - ASSERT_EQ(cps.size(), 4u); - EXPECT_EQ(cps.back().generation, kMountWriteAttemptIdGeneration); - EXPECT_EQ(cps.back().min_reader, kMountWriteAttemptIdGeneration); -} - -TEST(CASFormat, MountAttemptIdentityIsARecreateOnlyGenerationTenChange) -{ - EXPECT_EQ(G_BUILD, 10u); - EXPECT_EQ(kMountWriteAttemptIdGeneration, 10u); + const std::set expected{ + "cas_blob", "cas_blob_meta", "cas_pool_meta", "cas_ref_log", "cas_ref_snap", + "cas_ref_ckpt", "cas_ref_catalog", "cas_gc_maintenance_state", "cas_part_manifest", + "cas_run", "cas_fold_seal", "cas_gc_state", "cas_gc_hb", "cas_gc_outcomes", + "cas_owner", "cas_epoch", "cas_mount_lease"}; + ASSERT_EQ(expected.size(), 17u); - const auto mount_points = changePoints(FormatId::MountLease); - ASSERT_EQ(mount_points.back().generation, kMountWriteAttemptIdGeneration); - EXPECT_EQ(mount_points.back().min_reader, kMountWriteAttemptIdGeneration); + std::set actual; + for (const auto id : allRegisteredFormatIds()) + actual.insert(traitsFor(id).type); + EXPECT_EQ(actual, expected); - const auto pool_points = changePoints(FormatId::PoolMeta); - ASSERT_EQ(pool_points.back().generation, kMountWriteAttemptIdGeneration); - EXPECT_EQ(pool_points.back().min_reader, kMountWriteAttemptIdGeneration); + for (const auto & type : expected) + { + const FormatTraits * t = traitsForType(type); + ASSERT_NE(t, nullptr) << type; + EXPECT_EQ(t->type, type); + } } -TEST(CASPoolMeta, GenerationNinePoolIsRejectedAtReaderFloor) +/// The generation history is reset to a flat `{1, 1}` baseline for every class: CAS is pre-release and +/// carries no persisted data, so there is no compatibility cost to starting the count over. Pinned +/// because the decision is invisible otherwise — nothing consults `changePoints` at decode time yet, so +/// a wrong entry here would sit unnoticed until the day a per-class reader floor is wired and starts +/// admitting objects it should refuse. +TEST(CASFormat, EveryClassResetToTheBaselineGeneration) { - PoolMeta meta; - meta.pool_id = UInt128{1}; - meta.blob_header_len = 256; - meta.gc_shards = 1; - meta.min_reader_generation = 10; - meta.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - String encoded = encodePoolMeta(meta); - const String current = "\"v\":10"; - const size_t version = encoded.find(current); - ASSERT_NE(version, String::npos); - encoded.replace(version, current.size(), "\"v\":9"); - - try - { - decodePoolMeta(encoded); - FAIL() << "expected UNKNOWN_FORMAT_VERSION"; - } - catch (const DB::Exception & e) + for (auto id : allRegisteredFormatIds()) { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find("generation-10 mount-attempt-identity"), String::npos); + const auto cps = changePoints(id); + ASSERT_EQ(cps.size(), 1u) << "FormatId " << static_cast(id); + EXPECT_EQ(cps.front().generation, 1u); + EXPECT_EQ(cps.front().min_reader, 1u); } + + const auto roster_cps = changePoints(FormatId::Roster); + ASSERT_EQ(roster_cps.size(), 1u); + EXPECT_EQ(roster_cps.front().generation, 1u); + EXPECT_EQ(roster_cps.front().min_reader, 1u); } TEST(CASFormat, CurrentVersionsAreGBuild) diff --git a/src/Disks/tests/gtest_cas_format_battery.cpp b/src/Disks/tests/gtest_cas_format_battery.cpp index d6e9f01b3535..21251204d8a5 100644 --- a/src/Disks/tests/gtest_cas_format_battery.cpp +++ b/src/Disks/tests/gtest_cas_format_battery.cpp @@ -31,14 +31,28 @@ TEST(CASFormatBattery, PoolMeta) PoolMeta pm; pm.pool_id = hexToU128("00112233445566778899aabbccddeeff"); pm.blob_header_len = 256; - pm.min_reader_generation = 3; + pm.min_reader_generation = 1; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; runFormatBattery(FormatBatteryCase{ .id = FormatId::PoolMeta, .encode = [&] { return sealObject(FormatId::PoolMeta, encodePoolMeta(pm)); }, .decode = [](std::string_view s) { decodePoolMeta(std::string(openObject(FormatId::PoolMeta, s))); }, .golden = currentFormatHeader("cas_pool_meta") + - "{\"pid\":\"00112233445566778899aabbccddeeff\",\"hln\":256,\"gcs\":1,\"mrg\":3,\"alg\":\"ch128\"}\n"}); + "{\"pool_id\":\"00112233445566778899aabbccddeeff\",\"blob_header_len\":256,\"gc_shards\":1,\"min_reader_generation\":1,\"algos_used\":[\"ch128\"]}\n"}); +} + +TEST(CASPoolMeta, RejectsInvalidAlgoArrays) +{ + const auto decode = [](std::string_view algos_used) + { + return decodePoolMeta("{\"type\":\"cas_pool_meta\",\"v\":1}\n" + "{\"pool_id\":\"00112233445566778899aabbccddeeff\",\"blob_header_len\":256,\"gc_shards\":1,\"min_reader_generation\":1,\"algos_used\":" + String(algos_used) + "}\n"); + }; + + /// The first value is the field's PREVIOUS encoding -- a comma-joined string inside one JSON + /// value. It must fail closed rather than round-trip; the rest are malformed arrays. + for (const std::string_view bad : {"\"ch128,sha256\"", "[\"ch128\",1]", "[]", "[\"sha256\",\"ch128\"]", "[\"ch128\",\"ch128\"]", "[\"unknown\"]"}) + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decode(bad); }); } TEST(CASPoolMeta, ValidateAlgosUsedRejectsUnknownByte) diff --git a/src/Disks/tests/gtest_cas_fsck.cpp b/src/Disks/tests/gtest_cas_fsck.cpp index 71abb917eaa2..77c490656911 100644 --- a/src/Disks/tests/gtest_cas_fsck.cpp +++ b/src/Disks/tests/gtest_cas_fsck.cpp @@ -811,10 +811,10 @@ TEST(CASFsckAuthority, MissingBurnedEpochSealIsChainBroken) /// intermediate epoch is reported rather than treated as a sparse legal transition. String skipped_bytes = encodeRefLogTxn(RefLogTxn{ .ns = ns.string(), .txn_id = RefTxnId{7, 1}, .ops = {}, .prev_epoch_seal = RefTxnId{6, 1}}); - const String old_epoch_token = R"("!pse":"6")"; + const String old_epoch_token = R"("!prev_epoch":"6")"; const auto old_epoch = skipped_bytes.find(old_epoch_token); ASSERT_NE(old_epoch, String::npos); - skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!pse":"1")"); + skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!prev_epoch":"1")"); ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, skipped_bytes)).outcome, PutOutcome::Done); diff --git a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp index 870bcdc20295..ed4e76fac722 100644 --- a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp +++ b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp @@ -26,7 +26,7 @@ /// * absent at `expected`, no listed id above it => the namespace's frontier this round (normal end) /// * absent at `expected`, a listed id above it => IMPOSSIBLE under contiguity: the store is lying /// or a durable record was lost. Hold the namespace -/// (classification 4), cursor unmoved. +/// (classification `Clamped`), cursor unmoved. /// /// Epochs are crossed ONLY by consuming the `EpochSeal` that closes an epoch (INV-2). The seal folds as /// an applied no-op (probe B2: `produced=false`), and the next epoch's start is `{E', 1}` -- reached @@ -77,10 +77,10 @@ RefTxnId cursorOf(Backend & backend, const Layout & layout, const RootNamespace return cov ? cov->last_folded_ref_id : RefTxnId{}; } -uint8_t classificationOf(Backend & backend, const Layout & layout, const RootNamespace & ns) +CoverageClass classificationOf(Backend & backend, const Layout & layout, const RootNamespace & ns) { const auto cov = coverageOf(backend, layout, ns); - return cov ? cov->classification : 0; + return cov ? cov->classification : CoverageClass::Absent; } /// The `fold_ref_intake` phase metrics of the round `sched` runs -- the only place probe B1's two @@ -135,7 +135,7 @@ TEST(CASGCArithmeticIntake, HintOmittingMiddleRecordsFoldsThroughUnnoticed) ASSERT_GT(backend->holesServed(), 0u) << "the hint hole was never actually served"; EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 5})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2) << "a folded namespace is `changed`"; + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded) << "a folded namespace is `changed`"; for (uint64_t i = 1; i <= 5; ++i) EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(i)), 1) << "blob " << i << " lost its owner edge: its record was skipped because the hint omitted it"; @@ -166,13 +166,13 @@ TEST(CASGCArithmeticIntake, WalkEndsAtFrontierWithoutHold) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 3})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded); /// A second round over an unchanged namespace pays exactly one exact GET, finds the same frontier, /// and neither advances nor holds. ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 3})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 1) << "an unchanged namespace is `carried`"; + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Unchanged) << "an unchanged namespace is `carried`"; } /// ===================== EPOCHS ARE CROSSED ONLY BY CONSUMING A SEAL ===================== @@ -211,7 +211,7 @@ TEST(CASGCArithmeticIntake, SealCrossesEpochAndIsAppliedAsNoOp) ASSERT_GT(backend->holesServed(), 0u); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{2, 2})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded); for (uint64_t i = 1; i <= 4; ++i) EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(i)), 1) << "blob " << i; } @@ -291,19 +291,19 @@ TEST(CASGCArithmeticIntake, CursorRestingOnSealCrossesInALaterRound) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{2, 1})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(2)), 1); } /// ===================== IMPOSSIBLE SHAPES HOLD THE NAMESPACE ===================== /// /// `{1,3}` is genuinely absent while `{1,4}` is present AND listed. Contiguity says that cannot happen, -/// so whatever sits behind the gap may be an acked `+1`: the namespace is held at classification 4 with -/// its cursor UNMOVED, rather than sealing past the gap. +/// so whatever sits behind the gap may be an acked `+1`: the namespace is held at classification +/// `Clamped` with its cursor UNMOVED, rather than sealing past the gap. /// /// Listing-driven intake folded `{1,4}` and sealed the cursor at it -- permanently, since a record below /// the cursor is never re-read. -TEST(CASGCArithmeticIntake, GapBelowWitnessHoldsNamespaceAtClassificationFour) +TEST(CASGCArithmeticIntake, GapBelowWitnessHoldsNamespaceAtClampedClassification) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); @@ -326,7 +326,7 @@ TEST(CASGCArithmeticIntake, GapBelowWitnessHoldsNamespaceAtClassificationFour) EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 2})) << "the cursor must not advance past a gap"; - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(4)), 0) << "the record above the gap was not folded"; } @@ -357,7 +357,7 @@ TEST(CASGCArithmeticIntake, UnconsumedSealCrossingHoldsNamespace) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 1})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); const auto coverage = coverageOf(*backend, layout, ns); ASSERT_TRUE(coverage && coverage->hold.has_value()); EXPECT_EQ(coverage->hold->reason, HoldReason::UnconsumedSealCrossing); @@ -395,7 +395,7 @@ TEST(CASGCArithmeticIntake, CrossingFromANonSealRecordIsRefusedEvenWhenTheChainM EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 2})) << "epoch 1 was never sealed, so the cursor may not leave it"; - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(1)), 1) << "epoch 1's records still fold"; EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(2)), 1); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(3)), 0) @@ -462,7 +462,7 @@ TEST(CASGCArithmeticIntake, EpochStartThatAnswersOnlyEveryOtherReadHoldsInsteadO EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 2})) << "the cursor stops on the seal it consumed and never enters the unstable epoch"; - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(2)), 0); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(3)), 0) << "nothing above the unstable position may be folded either"; @@ -505,11 +505,11 @@ TEST(CASGCArithmeticIntake, CorruptBodyClampsOneNamespaceWhileAnotherFolds) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns_a), (RefTxnId{1, 1})); - EXPECT_EQ(classificationOf(*backend, layout, ns_a), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns_a), CoverageClass::Clamped); EXPECT_EQ(cursorOf(*backend, layout, ns_b), (RefTxnId{1, 3})) << "a sibling namespace's corrupt body must not stop this one"; - EXPECT_EQ(classificationOf(*backend, layout, ns_b), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns_b), CoverageClass::Folded); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(11)), 1); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(12)), 1); } @@ -582,7 +582,7 @@ TEST(CASGCArithmeticIntake, WhollyOmittedNamespaceFoldsThroughAuthoritativeCheck const auto hidden_cov = coverageOf(*backend, layout, ns); ASSERT_TRUE(hidden_cov.has_value()) << "the namespace is `Live` in the catalog, so it stays in the universe even fully hidden"; - EXPECT_EQ(hidden_cov->classification, 2) << "the checkpoint's frontier is folded by exact key"; + EXPECT_EQ(hidden_cov->classification, CoverageClass::Folded) << "the checkpoint's frontier is folded by exact key"; EXPECT_EQ(hidden_cov->last_folded_ref_id, (RefTxnId{1, 3})); /// The store stops lying: the already folded namespace reappears. diff --git a/src/Disks/tests/gtest_cas_gc_attempt.cpp b/src/Disks/tests/gtest_cas_gc_attempt.cpp index 49e63031c6df..0ae235777cbd 100644 --- a/src/Disks/tests/gtest_cas_gc_attempt.cpp +++ b/src/Disks/tests/gtest_cas_gc_attempt.cpp @@ -54,7 +54,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp index e9041dd7b223..384e9f29ea35 100644 --- a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp +++ b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp @@ -230,7 +230,7 @@ TEST(CASGCBoundedWalk, ARoundFoldsThroughItsRoundStartTailAndLeavesTheStragglers EXPECT_EQ(cov->last_folded_ref_id, (RefTxnId{1, planted})) << "the walk must fold through the round-start tail and no further -- it chased the writer"; EXPECT_FALSE(cov->hold.has_value()) << "reaching the committed frontier is not a hold"; - EXPECT_NE(cov->classification, 4) << "reaching the committed frontier is not a clamp"; + EXPECT_NE(cov->classification, CoverageClass::Clamped) << "reaching the committed frontier is not a clamp"; EXPECT_EQ(metric(intake, "tails_advanced"), 1u); EXPECT_EQ(metric(intake, "logs_applied"), planted) << "exactly the round-start backlog was folded"; @@ -505,7 +505,7 @@ TEST(CASGCBoundedWalk, AnAbsentManifestBodyStillHoldsWithoutAHead) ASSERT_TRUE(cov->hold.has_value()) << "an absent committed manifest body raises the fold barrier"; EXPECT_EQ(cov->hold->reason, HoldReason::ManifestBodyMissing); EXPECT_EQ(cov->hold->offending_position, (RefTxnId{1, 2})); - EXPECT_EQ(cov->classification, 4); + EXPECT_EQ(cov->classification, CoverageClass::Clamped); EXPECT_EQ(backend->headCount(layout.manifestKey(gone)), 0u) << "absence is decided by the GET, so the missing body costs no HEAD either"; } @@ -558,13 +558,13 @@ TEST(CASGCBoundedWalk, ANamespaceThatFoldedNothingKeepsItsSealedCursor) ASSERT_TRUE(after.has_value()) << "the coverage row was DROPPED -- the next round would re-fold this namespace from {0,0}"; /// The CURSOR and the HOLD are what the next round trusts, and both ride unchanged. - /// `classification` legitimately moves from 2 ("this round folded records") to 1 ("unchanged"), + /// `classification` legitimately moves from `Folded` ("this round folded records") to `Unchanged`, /// because that is what the round did — it is the one field that may differ, so it is the one field /// asserted loosely. EXPECT_EQ(after->last_folded_ref_id, before->last_folded_ref_id) << "a namespace that folded nothing must keep the cursor it had"; EXPECT_EQ(after->hold, before->hold); - EXPECT_NE(after->classification, 4) << "folding nothing is not a clamp"; + EXPECT_NE(after->classification, CoverageClass::Clamped) << "folding nothing is not a clamp"; EXPECT_EQ(metric(intake, "frontier_namespaces"), 2u) << "it stays in the round's universe, so its proof is still owed"; } diff --git a/src/Disks/tests/gtest_cas_gc_fold.cpp b/src/Disks/tests/gtest_cas_gc_fold.cpp index c3af2cbe4ef0..78435a6c2d78 100644 --- a/src/Disks/tests/gtest_cas_gc_fold.cpp +++ b/src/Disks/tests/gtest_cas_gc_fold.cpp @@ -464,7 +464,7 @@ TEST(CASGCFold, DeadPrecommitWithMissingBodyIsSkippedNotClampedForever) auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); /// The namespace's server-root prefix is "srv"; seed its watermark floor so build_sequence 5 is retired. const RootNamespace ns{"srv/tbl"}; - setWatermarkMinActive(*backend, store->layout(), "srv", /*writer_epoch*/1, /*min_active*/10); + setWatermarkMinActive(*backend, store->layout(), "srv", /*writer_epoch*/1, /*min_active_build_sequence*/10); /// A precommit naming a build (writer_epoch 1, build_sequence 5) whose body is never written. const ManifestRef dead = ManifestRef{.writer_epoch = 1, .build_sequence = 5, .manifest_ordinal = 1}; diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 13dc97ab826c..55f7247d87b3 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -321,7 +321,7 @@ CompletedRemovingFixture seedCompletedRemoving( CasFoldSeal parent; parent.generation = 1; parent.ref_lives.emplace(fixture.life_id, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); @@ -368,7 +368,7 @@ void seedCompletedRemovingBatch( parent.generation = 1; for (const CatalogEntry & entry : entries) parent.ref_lives.emplace(entry.incarnation, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); @@ -1480,7 +1480,7 @@ TEST(CASGCFrontierGate, TheOrphanManifestSweepAndItsCursorAreInertUnderSuppressi const ManifestRef r2{.writer_epoch = 5, .build_sequence = 0xCA02, .manifest_ordinal = 1}; writeManifestRaw(*backend, layout, ns, r1, {blobEntryFor("a", DB::UInt128(0xa1))}); writeManifestRaw(*backend, layout, ns, r2, {blobEntryFor("b", DB::UInt128(0xb2))}); - setWatermarkMinActive(*backend, layout, "test", r1.writer_epoch, /*min_active*/ 0xCA03); + setWatermarkMinActive(*backend, layout, "test", r1.writer_epoch, /*min_active_build_sequence*/ 0xCA03); /// The §6 deletion premise is a second precondition on the CONTROL arm below: a manifest of an /// epoch-`E` build is deletable only once the namespace's sealed fold cursor sits in an epoch /// strictly above `E`. Sealing that cursor here is what keeps this test about the GATE — without it @@ -2814,7 +2814,7 @@ TEST(CASGCFrontierGate, UnmatchedAdoptedParentLifeDoesNotSuppressAuthoritativeDe const UInt128 unmatched_life = hexToU128("fedcba98765432100123456789abcdef"); ASSERT_FALSE(parent.ref_lives.contains(unmatched_life)); parent.ref_lives.emplace(unmatched_life, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{9, 9}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{9, 9}}}); ASSERT_EQ( backend->putOverwrite(parent_seal_key, encodeFoldSeal(parent), parent_object->token).outcome, PutOutcome::Done); @@ -3137,7 +3137,7 @@ TEST(CASGCFrontierGate, DeferredRoundDrainsCompletedRemovingBeforeReturning) CasFoldSeal parent; parent.generation = 1; parent.ref_lives.emplace(life_id, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index 19a63bc6842a..9cc892165f14 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -24,9 +24,9 @@ /// DURABLE HOLDS (spec 2026-07-27 "ref chain complete cut" §5). /// /// A namespace whose ref-log walk meets an IMPOSSIBLE shape stops there, and that stop has to survive -/// the round. Before this task the stop was a single bit — `classification == 4` — and everything that -/// explained it (what went wrong, and exactly WHERE) lived in a log line and an in-memory anomaly, both -/// gone by the next round. That is not enough for three separate reasons: +/// the round. Before this task the stop was a single bit — `classification == CoverageClass::Clamped` — +/// and everything that explained it (what went wrong, and exactly WHERE) lived in a log line and an +/// in-memory anomaly, both gone by the next round. That is not enough for three separate reasons: /// /// * the next round could not RETRY the exact position, so a hold only survived while the round's /// hint happened to keep mentioning the namespace; @@ -36,10 +36,10 @@ /// baseline that looked proven when it was not. /// /// So the hold is now DURABLE and STRICTLY GRAMMARED: `{reason, offending_position, retry_count, -/// next_retry_round}` present if and only if `classification == 4`, rejected in both directions -/// otherwise. It rides the seal across rounds — including rounds whose hint omits the namespace -/// entirely — and across REBUILD, and it clears by exactly ONE event: the fold resolving the offending -/// position and that result being adopted in `gc/state`. +/// next_retry_round}` present if and only if `classification == CoverageClass::Clamped`, rejected in +/// both directions otherwise. It rides the seal across rounds — including rounds whose hint omits the +/// namespace entirely — and across REBUILD, and it clears by exactly ONE event: the fold resolving the +/// offending position and that result being adopted in `gc/state`. /// /// The carried hold is also a WITNESS, and a better one than the listing: it is durable proof that the /// walk once reached that position, so an absent below it is a gap rather than a frontier no matter @@ -177,8 +177,8 @@ RefHold holdOf(Backend & backend, const Layout & layout, const RootNamespace & n EXPECT_TRUE(cov.has_value()) << "no coverage row for " << ns.string(); if (!cov) return RefHold{}; - EXPECT_EQ(cov->classification, 4) << "a held namespace is classification 4"; - EXPECT_TRUE(cov->hold.has_value()) << "classification 4 without a hold is the forbidden shape"; + EXPECT_EQ(cov->classification, CoverageClass::Clamped) << "a held namespace is classification clamped"; + EXPECT_TRUE(cov->hold.has_value()) << "classification clamped without a hold is the forbidden shape"; return cov->hold ? *cov->hold : RefHold{}; } @@ -206,7 +206,7 @@ CasFoldSeal maximalHoldSeal(const String & map_key) seal.generation = std::numeric_limits::max(); seal.parent_generation = std::numeric_limits::max(); RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.last_folded_ref_id = RefTxnId{std::numeric_limits::max(), std::numeric_limits::max()}; cov.hold = RefHold{.reason = HoldReason::UnconsumedSealCrossing, /// the longest reason word @@ -233,7 +233,7 @@ CasFoldSeal cleanSeal(const String & map_key) seal.generation = 3; seal.parent_generation = 2; RefCoverage cov; - cov.classification = 2; + cov.classification = CoverageClass::Folded; cov.last_folded_ref_id = RefTxnId{4, 5}; fixtureCoverage(seal, map_key) = cov; return seal; @@ -245,7 +245,7 @@ CasFoldSeal heldSeal(const String & map_key) { CasFoldSeal seal = cleanSeal(map_key); RefCoverage & cov = fixtureCoverage(seal, map_key); - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{4, 6}, .retry_count = 7, .next_retry_round = 99}; return seal; @@ -271,20 +271,6 @@ String sealTextWith(const String & prototype, const std::vector & record return text + "{\"n\":" + std::to_string(records.size()) + "}\n"; } -/// Replace the coverage row's `cls` value with `raw`, VERBATIM. The point is to write integers no -/// `RefCoverage` can hold: the field is a byte in the struct, so a wide value exists only on the wire, -/// which is exactly where a reader has to catch it. `cls` is never the last field of a `cov` record, so -/// the value always ends at a comma. -String withRawClassification(const String & encoded, std::string_view raw) -{ - const size_t at = encoded.find("\"cls\":"); - EXPECT_NE(at, String::npos); - const size_t begin = at + strlen("\"cls\":"); - const size_t end = encoded.find(',', begin); - EXPECT_NE(end, String::npos); - return encoded.substr(0, begin) + String{raw} + encoded.substr(end); -} - /// Replace the FIRST occurrence of `field` with `replacement` (both are whole `"key":value` fragments), /// so a test states the exact wire shape it is feeding the decoder. String withField(const String & encoded, const String & field, const String & replacement) @@ -303,24 +289,27 @@ std::vector> illFormedSealsTheEncoderMustRe /// The pairing, both ways round. CasFoldSeal hold_on_folded = heldSeal("ns/0"); - fixtureCoverage(hold_on_folded, "ns/0").classification = 2; - out.emplace_back("a hold on a folded (2) row claims a stop that did not happen", hold_on_folded); + fixtureCoverage(hold_on_folded, "ns/0").classification = CoverageClass::Folded; + out.emplace_back("a hold on a folded row claims a stop that did not happen", hold_on_folded); CasFoldSeal clamped_without_hold = heldSeal("ns/0"); fixtureCoverage(clamped_without_hold, "ns/0").hold.reset(); - out.emplace_back("a clamped (4) row with no hold is indistinguishable from a clean cursor once " + out.emplace_back("a clamped row with no hold is indistinguishable from a clean cursor once " "durable", clamped_without_hold); - /// The closed set. 3 is the dangerous one: it passes the sweep's `== 4` and `== 0` refusals and - /// reaches the deletion premise, which is a refusal written in terms of the set. - CasFoldSeal classification_three = cleanSeal("ns/0"); - fixtureCoverage(classification_three, "ns/0").classification = 3; - out.emplace_back("classification 3 is not one of {0,1,2,4} and passes every refusal stated in terms " - "of them", classification_three); + /// The closed set is now the enum's declared values, so only an explicit cast reaches outside it. + /// 4 is the sharpest value to plant: it was the wire value for Clamped before this task's dense + /// renumbering, and under the new table it is simply out of range. + CasFoldSeal classification_retired_wire_value = cleanSeal("ns/0"); + fixtureCoverage(classification_retired_wire_value, "ns/0").classification + = static_cast(4); + out.emplace_back("classification 4 is outside the four values the wire table declares", + classification_retired_wire_value); CasFoldSeal classification_max = cleanSeal("ns/0"); - fixtureCoverage(classification_max, "ns/0").classification = 255; - out.emplace_back("classification 255 is not one of {0,1,2,4}", classification_max); + fixtureCoverage(classification_max, "ns/0").classification = static_cast(255); + out.emplace_back("classification 255 is outside the four values the wire table declares", + classification_max); /// The self-erasing hold, and its half-zero sibling. CasFoldSeal hold_at_zero = heldSeal("ns/0"); @@ -371,7 +360,7 @@ TEST(CASGCHoldGrammarBudget, SumsSaturateInsteadOfWrapping) EXPECT_FALSE(fitsObjectCap(kMax, 2, 256 * 1024 * 1024)); } -/// ===================== THE STRICT CLASSIFICATION-4 GRAMMAR ===================== +/// ===================== THE STRICT CLAMPED-CLASSIFICATION GRAMMAR ===================== TEST(CASGCHoldGrammar, EveryHoldReasonRoundTrips) { @@ -383,7 +372,7 @@ TEST(CASGCHoldGrammar, EveryHoldReasonRoundTrips) seal.generation = 3; seal.parent_generation = 2; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.last_folded_ref_id = RefTxnId{4, 5}; cov.hold = RefHold{.reason = reason, .offending_position = RefTxnId{4, 6}, .retry_count = 7, .next_retry_round = 99}; @@ -432,23 +421,20 @@ TEST(CASGCHoldGrammar, AHoldOnAnyOtherClassificationIsRefusedByTheDecoder) /// Bytes some other producer wrote. Built by demoting a legitimate held row's classification, so the /// hold fields are exactly the ones the encoder emits. - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, .retry_count = 0, .next_retry_round = 1}; fixtureCoverage(seal, "ns/0") = cov; - String text = encodeFoldSeal(seal); - const size_t at = text.find("\"cls\":4"); - ASSERT_NE(at, String::npos); - text[at + 6] = '2'; + const String text = withField(encodeFoldSeal(seal), R"("class":"clamped")", R"("class":"folded")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(text); }); } -TEST(CASGCHoldGrammar, ClassificationFourWithoutAHoldIsRefusedByTheDecoder) +TEST(CASGCHoldGrammar, ClampedWithoutAHoldIsRefusedByTheDecoder) { CasFoldSeal seal; seal.generation = 1; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.last_folded_ref_id = RefTxnId{1, 1}; /// Every single hold field is REQUIRED: dropping any one of them is corruption, not a default. @@ -456,8 +442,8 @@ TEST(CASGCHoldGrammar, ClassificationFourWithoutAHoldIsRefusedByTheDecoder) .retry_count = 3, .next_retry_round = 4}; fixtureCoverage(seal, "ns/0") = cov; const String whole = encodeFoldSeal(seal); - for (const String & field : {String(R"("hr":"body_undecodable")"), String(R"("hpe":"1")"), - String(R"("hps":"2")"), String(R"("hrc":3)"), String(R"("hnr":"4")")}) + for (const String & field : {String(R"("hold_reason":"body_undecodable")"), String(R"("hold_epoch":"1")"), + String(R"("hold_seq":"2")"), String(R"("retries":3)"), String(R"("retry_round":"4")")}) { SCOPED_TRACE("without " + field); const size_t at = whole.find(field); @@ -473,18 +459,18 @@ TEST(CASGCHoldGrammar, DuplicateHoldKeyIsCorruptedData) CasFoldSeal seal; seal.generation = 1; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, .retry_count = 0, .next_retry_round = 5}; fixtureCoverage(seal, "ns/0") = cov; const String whole = encodeFoldSeal(seal); - const String field = R"("hr":"gap_below_witness")"; + const String field = R"("hold_reason":"gap_below_witness")"; const size_t at = whole.find(field); ASSERT_NE(at, String::npos); /// The same key twice, with a DIFFERENT value: last-wins would silently rewrite the reason. String doubled = whole; - doubled.insert(at, R"("hr":"witness_disappeared",)"); + doubled.insert(at, R"("hold_reason":"witness_disappeared",)"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(doubled); }); } @@ -493,7 +479,7 @@ TEST(CASGCHoldGrammar, UnknownHoldReasonWordIsCorruptedData) CasFoldSeal seal; seal.generation = 1; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, .retry_count = 0, .next_retry_round = 5}; fixtureCoverage(seal, "ns/0") = cov; @@ -510,48 +496,46 @@ TEST(CASGCHoldGrammar, UnknownHoldReasonWordIsCorruptedData) /// The three shapes below are one finding, and it is about what a fold seal is FOR. The hold is the only /// durable record that a namespace stopped and where; everything downstream reads the seal and nothing /// re-derives the stop. So a seal that decodes into "no hold here" is not a lossy read, it is a licence -/// to delete: the sweep's §6 refusals are stated as `classification == 4` / `== 0` / `hold.has_value()`, -/// and a row that slips past all three reaches an irreversible delete of a manifest the fold never -/// accounted for. Each shape gets past a DIFFERENT one of the decoder's checks, which is why they are -/// pinned separately rather than as one "malformed seal" case. - -/// (1) The classification the reader never sees. `cls` is narrowed to a byte, so an integer on the wire -/// is truncated first and validated (if at all) afterwards: 258 becomes 2, "everything through the -/// cursor was folded". The value has to be judged WIDE, before the narrowing, or the wire can buy -/// coverage that no fold ever performed. +/// to delete: the sweep's §6 refusals are stated as `classification == Clamped` / `== Absent` / +/// `hold.has_value()`, and a row that slips past all three reaches an irreversible delete of a manifest +/// the fold never accounted for. Each shape gets past a DIFFERENT one of the decoder's checks, which is +/// why they are pinned separately rather than as one "malformed seal" case. + +/// (1) The classification is a WORD, closed the same way `hold_reason` already is: +/// `coverageClassFromWord` refuses anything outside the four named values as `CORRUPTED_DATA` before a +/// `CoverageClass` is ever constructed, so there is no wide-integer narrowing attack left to catch here — +/// the wire carries no integer at all. TEST(CASGCHoldGrammar, AClassificationOutsideTheGrammarIsCorruptedData) { const String clean = encodeFoldSeal(cleanSeal("ns/0")); - ASSERT_EQ(fixtureCoverage(decodeFoldSeal(clean), "ns/0").classification, 2) + ASSERT_EQ(fixtureCoverage(decodeFoldSeal(clean), "ns/0").classification, CoverageClass::Folded) << "the unmodified row is the one every case below deviates from"; - /// In-range bytes that are simply not classifications. 3 is the one the sweep's refusals miss. - for (const std::string_view raw : {"3", "5", "6", "255"}) + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - SCOPED_TRACE(String{"cls="} + String{raw}); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withRawClassification(clean, raw)); }); - } + decodeFoldSeal(withField(clean, R"("class":"folded")", R"("class":"foldedx")")); + }); - /// Wide integers whose LOW BYTE lands inside the grammar: 258 -> 2 (fully folded), 256 -> 0 - /// (absent), 260 -> 4 (clamped). Each would decode as a row the fold never wrote. - for (const std::string_view raw : {"256", "258", "260", "18446744073709551615"}) + /// The bare number `4` is the classification's pre-cut wire representation — the old byte-valued + /// form. A retired spelling is legal here because this is a marked negative fixture proving the + /// decoder still refuses it now that `class` takes a word; the byte-delta pins hold the other such + /// fixtures, and both kinds are exempt from the vocabulary sweeps for the same reason. + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - SCOPED_TRACE(String{"cls="} + String{raw}); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withRawClassification(clean, raw)); }); - } + decodeFoldSeal(withField(clean, R"("class":"folded")", R"("class":4)")); + }); } -/// And the field itself is required: an absent `cls` reads as 0, which is not "nothing was said about -/// this namespace" but the positive claim "no round folded it". +/// And the field itself is required: an absent `class` reads as `absent`, which is not "nothing was +/// said about this namespace" but the positive claim "no round folded it". TEST(CASGCHoldGrammar, ACoverageRowWithoutAClassificationIsCorruptedData) { const String clean = encodeFoldSeal(cleanSeal("ns/0")); - const size_t at = clean.find("\"cls\":2,"); + const String field = R"("class":"folded",)"; + const size_t at = clean.find(field); ASSERT_NE(at, String::npos); String without = clean; - without.erase(at, strlen("\"cls\":2,")); + without.erase(at, field.size()); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(without); }); } @@ -567,13 +551,13 @@ TEST(CASGCHoldGrammar, AHoldWhoseOffendingPositionHasAZeroComponentIsCorruptedDa expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - decodeFoldSeal(withField(withField(held, R"("hpe":"4")", R"("hpe":"0")"), - R"("hps":"6")", R"("hps":"0")")); + decodeFoldSeal(withField(withField(held, R"("hold_epoch":"4")", R"("hold_epoch":"0")"), + R"("hold_seq":"6")", R"("hold_seq":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(held, R"("hpe":"4")", R"("hpe":"0")")); }); + [&] { decodeFoldSeal(withField(held, R"("hold_epoch":"4")", R"("hold_epoch":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(held, R"("hps":"6")", R"("hps":"0")")); }); + [&] { decodeFoldSeal(withField(held, R"("hold_seq":"6")", R"("hold_seq":"0")")); }); } /// (3) The duplicate row. Two `cov` records for the same (namespace, shard) — held first, clean second — @@ -608,7 +592,7 @@ TEST(CASGCHoldGrammar, ASecondCoverageRowForTheSameKeyIsCorruptedData) EXPECT_EQ(seal.ref_lives.size(), 1u); } -/// The same one-record-per-key rule applies to `cnd`: a repeated row rewrites a shard's condemned +/// The same one-record-per-key rule applies to `condemned`: a repeated row rewrites a shard's condemned /// totals, which graduation paces on. TEST(CASGCHoldGrammar, ASecondCondemnedSummaryRecordIsCorruptedData) { @@ -617,7 +601,7 @@ TEST(CASGCHoldGrammar, ASecondCondemnedSummaryRecordIsCorruptedData) .oldest_nonpending_condemn_round = 3}; const String encoded = encodeFoldSeal(seal); - /// Lines 3..4 are `rfl`, `cnd` in the encoder's fixed order. + /// Lines 3..4 are `ref_life`, `condemned` in the encoder's fixed order. std::vector lines; for (size_t begin = headerAndMetaOf(encoded).size(); begin < encoded.size();) { @@ -626,14 +610,14 @@ TEST(CASGCHoldGrammar, ASecondCondemnedSummaryRecordIsCorruptedData) lines.push_back(encoded.substr(begin, end - begin)); begin = end + 1; } - ASSERT_EQ(lines.size(), 3u) << "rfl, cnd and the trailer"; + ASSERT_EQ(lines.size(), 3u) << "ref_life, condemned and the trailer"; const String ref_life_line = lines[0]; - const String cnd_line = lines[1]; + const String condemned_line = lines[1]; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(sealTextWith(encoded, {ref_life_line, cnd_line, cnd_line})); }); + [&] { decodeFoldSeal(sealTextWith(encoded, {ref_life_line, condemned_line, condemned_line})); }); /// The unduplicated assembly is the control. - const std::vector one_of_each{ref_life_line, cnd_line}; + const std::vector one_of_each{ref_life_line, condemned_line}; EXPECT_NO_THROW(decodeFoldSeal(sealTextWith(encoded, one_of_each))); } @@ -647,12 +631,12 @@ TEST(CASGCHoldGrammar, CleanupEvidenceWithAZeroRemovalIdIsCorruptedData) const String encoded = encodeFoldSeal(seal); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rte":"2")", R"("rte":"0")")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_epoch":"2")", R"("remove_epoch":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rts":"3")", R"("rts":"0")")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_seq":"3")", R"("remove_seq":"0")")); }); /// Omitted entirely is the same thing: the fields default to zero. expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rte":"2",)", "")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_epoch":"2",)", "")); }); } /// The OBJECT cap bounds the whole seal. Nothing on the fold-seal READ path enforces it (the seal @@ -1018,7 +1002,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointHoldsOnlyItsOwnNamespace) ASSERT_TRUE(good_cov.has_value()); EXPECT_FALSE(good_cov->hold.has_value()) << "the corrupt object belongs to the OTHER namespace"; EXPECT_EQ(good_cov->last_folded_ref_id, (RefTxnId{1, 2})); - EXPECT_EQ(good_cov->classification, 2); + EXPECT_EQ(good_cov->classification, CoverageClass::Folded); /// And nothing was destroyed for the held namespace: a hold shuts the round's destructive gate, so /// its ref objects — including the ones a cleanup range computed WITHOUT the unreadable checkpoint @@ -1091,7 +1075,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointWithNoWalkPositionRecordsAnAnomaly const auto cov = coverageOf(*backend, layout, phantom); ASSERT_TRUE(cov.has_value()); EXPECT_FALSE(cov->hold.has_value()) << "a hold here could only name a position no round ever read"; - EXPECT_EQ(cov->classification, 1) << "nothing was folded, so the row is `unchanged`"; + EXPECT_EQ(cov->classification, CoverageClass::Unchanged) << "nothing was folded, so the row is `unchanged`"; EXPECT_EQ(cov->last_folded_ref_id, (RefTxnId{})); /// Same isolation as the held arm: the pool keeps working. @@ -1199,7 +1183,7 @@ TEST(CASGCHoldGrammar, HoldClearsOnlyByFoldingThroughTheOffendingPosition) const auto cov = coverageOf(*backend, layout, ns); ASSERT_TRUE(cov.has_value()); EXPECT_FALSE(cov->hold.has_value()) << "folding through the offending position is what clears a hold"; - EXPECT_EQ(cov->classification, 2); + EXPECT_EQ(cov->classification, CoverageClass::Folded); EXPECT_EQ(cov->last_folded_ref_id, (RefTxnId{1, 4})) << "the walk resumed past the resolved gap"; EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(4)), 1) << "the record above the gap finally contributed its owner edge"; @@ -1261,10 +1245,10 @@ TEST(CASGCHoldGrammar, RebuildCarriesMatchingHoldAndDropsAbsentLife) mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); RefCoverage gone; - gone.classification = 4; + gone.classification = CoverageClass::Clamped; gone.last_folded_ref_id = RefTxnId{2, 2}; gone.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{2, 3}, .retry_count = 1, .next_retry_round = 2}; @@ -1278,7 +1262,7 @@ TEST(CASGCHoldGrammar, RebuildCarriesMatchingHoldAndDropsAbsentLife) ASSERT_TRUE(rebuilt.has_value()); const auto rediscovered = rebuilt->ref_lives.find(life_id); ASSERT_NE(rediscovered, rebuilt->ref_lives.end()); - EXPECT_EQ(rediscovered->second.coverage.classification, 4); + EXPECT_EQ(rediscovered->second.coverage.classification, CoverageClass::Clamped); ASSERT_TRUE(rediscovered->second.coverage.hold.has_value()); EXPECT_EQ(*rediscovered->second.coverage.hold, plantedHold()); EXPECT_FALSE(rebuilt->ref_lives.contains(absent_life_id)); @@ -1320,7 +1304,7 @@ TEST(CASGCHoldGrammar, RebuildStepsDownPastACrashedNewestGenerationToTheSealBelo mutateSealAt(*backend, layout, older_generation, older_attempt, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); }); @@ -1398,7 +1382,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWithAnUndecodablePriorSeal) const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":4}\nthis is not a seal body\n", + backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n", backend->head(seal_key).token); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc.rebuildBaseline(/*force=*/true); }); @@ -1433,7 +1417,7 @@ TEST(CASGCHoldGrammar, RebuildWithLostStateStillCarriesHoldsFromTheNewestSeal) mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); }); @@ -1470,7 +1454,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenTheNewestSealIsUnreadableAndTheStateIsL const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":4}\nthis is not a seal body\n", + backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n", backend->head(seal_key).token); const HeadResult sh = backend->head(layout.gcStateKey()); ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); @@ -1537,7 +1521,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); }); diff --git a/src/Disks/tests/gtest_cas_gc_leak.cpp b/src/Disks/tests/gtest_cas_gc_leak.cpp index 36a77237e9dc..06983bc52aab 100644 --- a/src/Disks/tests/gtest_cas_gc_leak.cpp +++ b/src/Disks/tests/gtest_cas_gc_leak.cpp @@ -47,7 +47,7 @@ PoolPtr openTestPool(std::shared_ptr & out_backend) /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 81a1dd46c2d3..47715169db12 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -33,7 +33,7 @@ TEST(CASFormatBattery, GcMaintenanceState) runFormatBattery({FormatId::GcMaintenanceState, [&] { return sealObject(FormatId::GcMaintenanceState, encodeGcMaintenanceState(state)); }, [](std::string_view s) { decodeGcMaintenanceState(std::string(openObject(FormatId::GcMaintenanceState, s))); }, - currentFormatHeader("cas_gc_maintenance_state") + "{\"cur\":\"cas/ns/a\"}\n"}); + currentFormatHeader("cas_gc_maintenance_state") + "{\"janitor_cursor\":\"cas/ns/a\"}\n"}); } TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) @@ -41,8 +41,8 @@ TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) EXPECT_EQ(static_cast(FormatId::GcMaintenanceState), 25); const auto points = changePoints(FormatId::GcMaintenanceState); ASSERT_EQ(points.size(), 1u); - EXPECT_EQ(points[0].generation, 7); - EXPECT_EQ(points[0].min_reader, 7); + EXPECT_EQ(points[0].generation, 1); + EXPECT_EQ(points[0].min_reader, 1); const FormatTraits & traits = traitsFor(FormatId::GcMaintenanceState); EXPECT_EQ(traits.type, "cas_gc_maintenance_state"); EXPECT_EQ(traits.family, TextFamily::Control); @@ -60,7 +60,7 @@ TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) const GcMaintenanceState empty; EXPECT_EQ(encodeGcMaintenanceState(empty), fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"cur\":\"\"}}\n", currentCompatibilityVersion())); + "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion())); const GcMaintenanceState state{.janitor_cursor = R"(cas/ns/a/"quoted"\\next)"}; EXPECT_EQ(decodeGcMaintenanceState(encodeGcMaintenanceState(state)), state); } @@ -69,28 +69,28 @@ TEST(CASGCMaintenanceStateFormat, RejectsMalformedAndBoundsCursor) { const auto bad = [](std::string_view body) { - return "{\"type\":\"cas_gc_maintenance_state\",\"v\":7}\n" + String(body); + return "{\"type\":\"cas_gc_maintenance_state\",\"v\":1}\n" + String(body); }; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeGcMaintenanceState(bad("{}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\",\"cur\":\"b\"}\n")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\",\"janitor_cursor\":\"b\"}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\",\"extra\":1}\n")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\",\"extra\":1}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\"}\nx")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\"}\nx")); }); const GcMaintenanceState at_limit{.janitor_cursor = String(kMaxGcMaintenanceCursorBytes, 'x')}; EXPECT_EQ(decodeGcMaintenanceState(encodeGcMaintenanceState(at_limit)), at_limit); const GcMaintenanceState over_limit{.janitor_cursor = String(kMaxGcMaintenanceCursorBytes + 1, 'x')}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LIMIT_EXCEEDED, [&] { (void)encodeGcMaintenanceState(over_limit); }); - const String raw = "{\"type\":\"cas_gc_maintenance_state\",\"v\":7}\n{\"cur\":\"" + over_limit.janitor_cursor + "\"}\n"; + const String raw = "{\"type\":\"cas_gc_maintenance_state\",\"v\":1}\n{\"janitor_cursor\":\"" + over_limit.janitor_cursor + "\"}\n"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeGcMaintenanceState(raw); }); - String oversized = R"({"type":"cas_gc_maintenance_state","v":7,"pad":")"; + String oversized = R"({"type":"cas_gc_maintenance_state","v":1,"pad":")"; oversized.append(448 * 1024, 'x'); - oversized += "\"}\n{\"cur\":\""; + oversized += "\"}\n{\"janitor_cursor\":\""; oversized.append(kMaxGcMaintenanceCursorBytes, 'y'); oversized += "\"}\n"; ASSERT_GT(oversized.size(), traitsFor(FormatId::GcMaintenanceState).object_cap); @@ -184,7 +184,7 @@ TEST(CASGCMaintenanceState, FutureVersionPropagatesInsteadOfResetting) const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); ASSERT_EQ(backend.putIfAbsent(key, fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"cur\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, + "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, PutOutcome::Done); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { (void)readGcMaintenanceState(backend, layout); }); diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index e03aa26dbaad..c9c20a7454cd 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -3,6 +3,8 @@ #include #include +#include + using namespace DB::Cas; namespace @@ -42,8 +44,8 @@ TEST(CASFormatBattery, GcOutcomes) [&] { return sealObject(FormatId::GcOutcomes, encodeOutcomeLog(log)); }, [](std::string_view d) { decodeOutcomeLog(std::string(openObject(FormatId::GcOutcomes, d))); }, currentFormatHeader("cas_gc_outcomes") + - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\"," - "\"tt\":\"etag\",\"tv\":\"e-1\",\"oc\":\"deleted\"}\n{\"n\":1}\n"}); + "{\"kind\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\"," + "\"token_type\":\"etag\",\"token\":\"e-1\",\"outcome\":\"deleted\"}\n{\"n\":1}\n"}); } TEST(CASGCOutcomesFormat, EmptyRoundTrips) @@ -77,7 +79,20 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) EXPECT_EQ(encodeOutcomeLog(d), text); } -TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) +/// Closed-set pin: the four `OutcomeKind` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASGCOutcomesFormat, ClosedSetPinsOutcomeKindWords) +{ + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Deleted), "deleted"); + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Absent), "absent"); + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Replaced), "replaced"); + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Spared), "spared"); + for (const auto o : magic_enum::enum_values()) + EXPECT_EQ(outcomeKindFromWireWord(outcomeKindToWireWord(o)), o); +} + +TEST(CASGCOutcomesFormat, RecordRequiresCompleteBlobRefAndTokenGroups) { OutcomeLog log; log.entries.push_back({ObjectKind::Blob, @@ -85,16 +100,12 @@ TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) Token{"e-1", TokenType::ETag}, OutcomeKind::Deleted}); const String bytes = encodeOutcomeLog(log); - const String token_value = R"(,"tv":"e-1")"; - const auto token_value_pos = bytes.find(token_value); - ASSERT_NE(token_value_pos, String::npos); - String missing_token_value = bytes; - missing_token_value.erase(token_value_pos, token_value.size()); - const OutcomeLog decoded = decodeOutcomeLog(missing_token_value); - ASSERT_EQ(decoded.entries.size(), 1u); - EXPECT_EQ(decoded.entries[0].token.value, ""); - - for (const String & field : {String(R"(,"ha":"ch128")"), String(R"(,"h":"00112233445566778899aabbccddeeff")"), String(R"(,"tt":"etag")")}) + for (const auto & [field, expected_message] : { + std::pair{String(R"(,"algo":"ch128")"), "CAS outcome log: blob ref missing algo/digest"}, + std::pair{String(R"(,"digest":"00112233445566778899aabbccddeeff")"), "CAS outcome log: blob ref missing algo/digest"}, + std::pair{String(R"(,"token_type":"etag")"), "CAS outcome log: token missing token_type/token"}, + std::pair{String(R"(,"token":"e-1")"), "CAS outcome log: token missing token_type/token"}, + }) { const auto pos = bytes.find(field); ASSERT_NE(pos, String::npos); @@ -108,32 +119,32 @@ TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) catch (const DB::Exception & e) { EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); - EXPECT_EQ(e.message(), "CAS outcome log: record missing ha/h/tt"); + EXPECT_EQ(e.message(), expected_message); } } } TEST(CASGCOutcomesFormat, GarbageAndUnknownWordsFailClosed) { - EXPECT_THROW(decodeOutcomeLog(String("")), DB::Exception); - EXPECT_THROW(decodeOutcomeLog(String("not a cas object\n")), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeOutcomeLog(String("")); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeOutcomeLog(String("not a cas object\n")); }); /// A record with an unknown outcome word fails closed. - const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n" - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\"," - "\"tt\":\"etag\",\"tv\":\"x\",\"oc\":\"bogus\"}\n{\"n\":1}\n"; - EXPECT_THROW(decodeOutcomeLog(bad), DB::Exception); + const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":1}\n" + "{\"kind\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\"," + "\"token_type\":\"etag\",\"token\":\"x\",\"outcome\":\"bogus\"}\n{\"n\":1}\n"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(bad); }); /// A trailer count mismatch fails closed. - const String miscount = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n{\"n\":5}\n"; - EXPECT_THROW(decodeOutcomeLog(miscount), DB::Exception); + const String miscount = "{\"type\":\"cas_gc_outcomes\",\"v\":1}\n{\"n\":5}\n"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(miscount); }); } TEST(CASGCOutcomesFormat, DigestWidthMismatchFailsClosedWithCorruptedData) { - /// `ch128` (CityHash128) digests are 16 bytes = 32 hex chars; here the "h" field is truncated + /// `ch128` (CityHash128) digests are 16 bytes = 32 hex chars; here the `digest` field is truncated /// to 30 hex chars. Must surface as CORRUPTED_DATA (malformed serialized input), not /// `fromHex`'s BAD_ARGUMENTS. - const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n" - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddee\"," - "\"tt\":\"etag\",\"tv\":\"x\",\"oc\":\"deleted\"}\n{\"n\":1}\n"; + const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":1}\n" + "{\"kind\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddee\"," + "\"token_type\":\"etag\",\"token\":\"x\",\"outcome\":\"deleted\"}\n{\"n\":1}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(bad); }); } diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index 1ad8f576809d..1b6344af63fb 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -500,7 +500,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) EXPECT_EQ(rep.committed_refs, blobs.size()); /// Multiple rebuild flushes still converge to one authoritative row domain: no more than one - /// canonical seq-0 `btr` per shard and exactly one `cnd` per shard. These are the cardinalities the + /// canonical seq-0 `blob_run` per shard and exactly one `condemned` per shard. These are the cardinalities the /// catalog admission reservation over-covers independently of catalog-entry count. const GcState rebuilt_state = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); const CasFoldSeal rebuilt_seal = decodeFoldSeal( @@ -535,7 +535,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) } /// Trimmed-but-live (design delta 2): the precommit's journal evidence is gone (trim), the build -/// is NOT provably dead (a live build holds min_active down) — the unowned-alive sweep must +/// is NOT provably dead (a live build holds min_active_build_sequence down) — the unowned-alive sweep must /// over-protect the manifest's edges. TEST(CASGCRebuild, UnownedAliveManifestOverProtected) { @@ -543,7 +543,7 @@ TEST(CASGCRebuild, UnownedAliveManifestOverProtected) auto store = openPoolForTest(backend); const RootNamespace ns{"00/aa@cas@"}; - /// A LIVE build pins min_active at its build_seq, so higher build sequences are not provably dead. + /// A LIVE build pins min_active_build_sequence at its build_seq, so higher build sequences are not provably dead. auto live_build = store->beginPartWrite({}); store->renewWatermarkOnce(); diff --git a/src/Disks/tests/gtest_cas_gc_resume.cpp b/src/Disks/tests/gtest_cas_gc_resume.cpp index 55ff116b7109..0a8384e644e0 100644 --- a/src/Disks/tests/gtest_cas_gc_resume.cpp +++ b/src/Disks/tests/gtest_cas_gc_resume.cpp @@ -28,7 +28,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// Whether the CURRENT retired list (any gc-shard) still holds an entry. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index 8ea415a9f024..b54b5e71d648 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -631,7 +631,7 @@ TEST(CASGCRound, PreviewReportsCondemnedRowsAndIsWriteFree) /// A fully idle fold pure-carries every shard's authoritative rows verbatim. The parent is first made /// non-vacuous with one live blob in each of two shards; the forced no-delta successor must preserve -/// both `btr` rows and the total `cnd` domain byte-for-byte. +/// both `blob_run` rows and the total `condemned` domain byte-for-byte. TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) { auto backend = std::make_shared(); @@ -675,9 +675,9 @@ TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) EXPECT_TRUE(seal1.condemned_summary.contains(0) && seal1.condemned_summary.contains(1)); EXPECT_TRUE(seal2.condemned_summary.contains(0) && seal2.condemned_summary.contains(1)); - /// Capacity reserves one widest `btr` row per shard. Pin the production pure-carry seal to the + /// Capacity reserves one widest `blob_run` row per shard. Pin the production pure-carry seal to the /// authoritative grammar that makes that bound sufficient: at most one in-range canonical seq-0 - /// run per shard, beside exactly one `cnd` row for every shard. + /// run per shard, beside exactly one `condemned` row for every shard. bool run_seen[2] = {false, false}; ASSERT_EQ(seal1.blob_target_runs.size(), 2u); ASSERT_EQ(seal2.blob_target_runs.size(), 2u); @@ -1808,7 +1808,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) const ManifestRef r2 = ref(5, 0xCA02); writeManifestRaw(*backend, store->layout(), ns, r1, {blobEntryFor("a", DB::UInt128(1))}); writeManifestRaw(*backend, store->layout(), ns, r2, {blobEntryFor("b", DB::UInt128(2))}); - setWatermarkMinActive(*backend, store->layout(), "test", r1.writer_epoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), "test", r1.writer_epoch, /*min_active_build_sequence*/6); /// The §6 deletion premise is a second precondition on every sweep deletion: a manifest of an /// epoch-`E` build is deletable only once the namespace's sealed fold cursor sits in an epoch @@ -1823,7 +1823,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) /// injected cursor would prove only that the premise reads a number, not that the number can be /// produced. /// - /// The live publications use build sequences ABOVE the watermark's `min_active`, so the only + /// The live publications use build sequences ABOVE the watermark's `min_active_build_sequence`, so the only /// sweep-ELIGIBLE manifests in the namespace remain the two debris bodies -- the premise, not the /// watermark, is what this test varies. publishAt(*backend, store->layout(), ns, RefTxnId{1, 1}, "tbl", /*build_sequence=*/7, @@ -1900,7 +1900,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) foreign_config.server_root_id = "test"; auto invalid_store = openTestPoolWithConfig(foreign_backend, std::move(foreign_config)); const String foreign_mount_key = invalid_store->layout().mountKey("test"); - setWatermarkMinActive(*foreign_backend, invalid_store->layout(), "test", r1.writer_epoch, /*min_active*/6); + setWatermarkMinActive(*foreign_backend, invalid_store->layout(), "test", r1.writer_epoch, /*min_active_build_sequence*/6); const auto occupant_before = foreign_backend->get(foreign_mount_key); ASSERT_TRUE(occupant_before.has_value()); const uint64_t violations_before diff --git a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp index c8bc92efffdf..96f486781f5e 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp @@ -563,7 +563,7 @@ TEST(CASGCShardRetireDrain, ReclaimsDroppableBlobOwnedByNonZeroShard) return backend->head(layout.manifestKey(id)).exists; }; /// Whether ANY gc-shard still holds an in-flight condemned entry (the ack-floor deletion pipeline is - /// in flight while this is true). Retired-in-snapshot (T4): reconstructed from the adopted fold seal's + /// in flight while this is true). Condemned state is reconstructed from the adopted fold seal's /// RunMarker::Condemned rows across all shards, not a separate retired list. auto anyRetiredPending = [&] { diff --git a/src/Disks/tests/gtest_cas_gc_state_format.cpp b/src/Disks/tests/gtest_cas_gc_state_format.cpp index 7e9452fdf8a7..360063e76bdd 100644 --- a/src/Disks/tests/gtest_cas_gc_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_state_format.cpp @@ -26,8 +26,8 @@ TEST(CASFormatBattery, GcState) [&] { return sealObject(FormatId::GcState, encodeGcState(s)); }, [](std::string_view d) { decodeGcState(std::string(openObject(FormatId::GcState, d))); }, currentFormatHeader("cas_gc_state") + - "{\"rnd\":\"4\",\"gcs\":1,\"sg\":\"9\",\"spt\":\"7\",\"sa\":\"3\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000001\",\"ls\":\"12\"}\n"}); + "{\"round\":\"4\",\"gc_shards\":1,\"snap_generation\":\"9\",\"snap_pruned_through\":\"7\",\"snap_attempt\":\"3\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000001\",\"lease_seq\":\"12\"}\n"}); } CAS_BATTERY_COVERS(GcHeartbeat); @@ -39,7 +39,7 @@ TEST(CASFormatBattery, GcHeartbeat) [&] { return sealObject(FormatId::GcHeartbeat, encodeGcHeartbeat(hb)); }, [](std::string_view d) { decodeGcHeartbeat(std::string(openObject(FormatId::GcHeartbeat, d))); }, currentFormatHeader("cas_gc_hb") + - "{\"by\":\"00000000000000000000000000000001\",\"seq\":\"1741\"}\n"}); + "{\"owner\":\"00000000000000000000000000000001\",\"hb_seq\":\"1741\"}\n"}); } /// ---------- field round-trips (migrated from gtest_cas_gc_formats.cpp, re-pointed at the text codec) ---------- @@ -87,11 +87,11 @@ TEST(CASGCStateFormat, DefaultsRoundTrip) TEST(CASGCStateFormat, RejectsZeroGcShards) { - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String bad = "{\"type\":\"cas_gc_state\",\"v\":3}\n" - "{\"rnd\":\"0\",\"gcs\":0,\"sg\":\"0\",\"spt\":\"0\",\"sa\":\"0\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000000\",\"ls\":\"0\"}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String bad = "{\"type\":\"cas_gc_state\",\"v\":1}\n" + "{\"round\":\"0\",\"gc_shards\":0,\"snap_generation\":\"0\",\"snap_pruned_through\":\"0\",\"snap_attempt\":\"0\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000000\",\"lease_seq\":\"0\"}\n"; EXPECT_THROW(decodeGcState(bad), DB::Exception); } @@ -127,13 +127,13 @@ TEST(CASGCStateFormatDeathTest, RejectsZeroGcShardsOnEncodeAborts) TEST(CASGCStateFormat, RejectsAbsentGcShards) { - /// An absent gcs key must fail closed (the writer always emits it) rather than silently defaulting + /// An absent gc_shards key must fail closed (the writer always emits it) rather than silently defaulting /// to the struct's gc_shards = 1 — a missing shard count means a corrupt object, not "use the floor". - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String bad = "{\"type\":\"cas_gc_state\",\"v\":3}\n" - "{\"rnd\":\"0\",\"sg\":\"0\",\"spt\":\"0\",\"sa\":\"0\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000000\",\"ls\":\"0\"}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String bad = "{\"type\":\"cas_gc_state\",\"v\":1}\n" + "{\"round\":\"0\",\"snap_generation\":\"0\",\"snap_pruned_through\":\"0\",\"snap_attempt\":\"0\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000000\",\"lease_seq\":\"0\"}\n"; EXPECT_THROW(decodeGcState(bad), DB::Exception); } @@ -161,9 +161,9 @@ TEST(CASGCHeartbeatFormat, RoundTripAndBoundaries) TEST(CASGCHeartbeatFormat, RejectsMissingIdentityFields) { - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String header = "{\"type\":\"cas_gc_hb\",\"v\":3}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String header = "{\"type\":\"cas_gc_hb\",\"v\":1}\n"; const auto expectCorrupted = [](const String & data) { @@ -178,6 +178,6 @@ TEST(CASGCHeartbeatFormat, RejectsMissingIdentityFields) } }; - expectCorrupted(header + "{\"seq\":\"1741\"}\n"); - expectCorrupted(header + "{\"by\":\"00000000000000000000000000000001\"}\n"); + expectCorrupted(header + "{\"hb_seq\":\"1741\"}\n"); + expectCorrupted(header + "{\"owner\":\"00000000000000000000000000000001\"}\n"); } diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index c32b03e23074..84d457b08158 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -26,7 +26,7 @@ using namespace DB::Cas; /// MountLeaseKeeper behavior: the per-server mount lease and the merged build-watermark floor ride the /// SAME slot, renewed by one beat. The keeper anchors durably before return, adopts a slot already /// written by `claimMount` (same uuid+epoch), re-reads the callback on each renew and bumps `seq`, -/// stamps the farewell sentinel (`min_active = UINT64_MAX`, `expires_at_ms <= now`) on `release`, and +/// stamps the farewell sentinel (`min_active_build_sequence = UINT64_MAX`, `expires_at_ms <= now`) on `release`, and /// returns typed terminal results on any foreign touch. namespace @@ -171,18 +171,18 @@ TEST(CASHeartbeat, AnchorCarriesFloor) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - uint64_t min_active_now = 5; + uint64_t min_active_build_sequence_now = 5; seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [&] { return min_active_now; }, {}, std::chrono::milliseconds(0)); + [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0)); keeper.start(); auto hr = backend->head(layout.mountKey(srid)); ASSERT_TRUE(hr.exists); auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); EXPECT_EQ(m.writer_epoch, 9u); - EXPECT_EQ(m.min_active, 5u); + EXPECT_EQ(m.min_active_build_sequence, 5u); EXPECT_EQ(m.seq, 1u); EXPECT_FALSE(m.gc_fenced); } @@ -194,20 +194,20 @@ TEST(CASHeartbeat, RenewRereadsCallbackAndBumpsSeq) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - uint64_t min_active_now = 5; + uint64_t min_active_build_sequence_now = 5; seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [&] { return min_active_now; }, {}, std::chrono::milliseconds(0)); + [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0)); keeper.start(); /// The dynamic field moves; the renewal re-reads it off the callback and bumps seq. now_ms = 1500; - min_active_now = 8; + min_active_build_sequence_now = 8; renewKeeperOrThrow(keeper); auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); - EXPECT_EQ(m.min_active, 8u); + EXPECT_EQ(m.min_active_build_sequence, 8u); EXPECT_EQ(m.seq, 2u); EXPECT_EQ(m.expires_at_ms, 1500u + 100u); } @@ -230,9 +230,9 @@ TEST(CASHeartbeat, StopStampsExpiredAndFarewellSentinel) auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); /// Terminal body stamps the lease already-expired (so a same-server reopen reclaims immediately) - /// AND folds the watermark farewell into it (min_active = UINT64_MAX). + /// AND folds the watermark farewell into it (min_active_build_sequence = UINT64_MAX). EXPECT_LE(m.expires_at_ms, now_ms); - EXPECT_EQ(m.min_active, std::numeric_limits::max()); + EXPECT_EQ(m.min_active_build_sequence, std::numeric_limits::max()); } /// Phase A (spec rev.4 2026-07-24): a confirmed renewal mismatch whose re-read shows OUR OWN diff --git a/src/Disks/tests/gtest_cas_inspect.cpp b/src/Disks/tests/gtest_cas_inspect.cpp index 2ed70c293259..c20466591ac4 100644 --- a/src/Disks/tests/gtest_cas_inspect.cpp +++ b/src/Disks/tests/gtest_cas_inspect.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include #include @@ -288,6 +289,32 @@ TEST(CASInspect, RendersRefCkptAbsencesAsExplicitNulls) EXPECT_NE(json.find(R"("last_epoch_seal":null)"), String::npos) << json; } +/// `CoverageClass` renders as its full wire word, not the enumerator's numeric value: `cas-inspect` is +/// exactly the tool an operator reaches for to read a fold seal directly, so a coverage row that still +/// printed a bare integer would send them back to this file's comment to decode it. +TEST(CASInspect, RendersCoverageClassificationWireWords) +{ + const Layout layout("p"); + CasFoldSeal seal; + seal.generation = 3; + seal.parent_generation = 2; + seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Absent}; + seal.ref_lives[UInt128{2}].coverage = RefCoverage{.classification = CoverageClass::Unchanged}; + seal.ref_lives[UInt128{3}].coverage + = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}; + seal.ref_lives[UInt128{4}].coverage = RefCoverage{ + .classification = CoverageClass::Clamped, + .hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, + .retry_count = 0, .next_retry_round = 1}}; + + const String key = layout.foldSealKey(/*generation*/3, /*attempt*/0); + const String json = caInspectToJson(layout, key, encodeFoldSeal(seal)); + EXPECT_NE(json.find(R"("classification":"absent")"), String::npos) << json; + EXPECT_NE(json.find(R"("classification":"unchanged")"), String::npos) << json; + EXPECT_NE(json.find(R"("classification":"folded")"), String::npos) << json; + EXPECT_NE(json.find(R"("classification":"clamped")"), String::npos) << json; +} + /// A listed physical id cannot supply a namespace. Inspect must receive the unique catalog join, and /// a different logical spelling at the same id is rejected by the decoded object's own namespace. TEST(CASInspect, RefObjectRequiresTheExactCatalogResolution) diff --git a/src/Disks/tests/gtest_cas_json_writer.cpp b/src/Disks/tests/gtest_cas_json_writer.cpp index 4eda28e717a3..ea12afa32bcb 100644 --- a/src/Disks/tests/gtest_cas_json_writer.cpp +++ b/src/Disks/tests/gtest_cas_json_writer.cpp @@ -15,17 +15,20 @@ TEST(CASJsonWriter, KeyValueSequenceMatchesCanonicalShape) { CasJsonWriter w; bool first = true; - w.key("we", first); + /// The names are shape labels, not format keys: this test is about the writer's primitives, and + /// borrowing a real wire spelling would put this file in every vocabulary sweep for no reason. + w.key("u64_string_field", first); w.u64StringValue(7); - w.key("mo", first); + w.key("number_field", first); w.u64Number(3); - w.key("ok", first); + w.key("bool_field", first); w.boolValue(true); - w.key("ome", first); + w.key("second_u64_string_field", first); w.u64StringValue(1); w.closeObject(first); w.newline(); - EXPECT_EQ(std::move(w).take(), "{\"we\":\"7\",\"mo\":3,\"ok\":true,\"ome\":\"1\"}\n"); + EXPECT_EQ(std::move(w).take(), + "{\"u64_string_field\":\"7\",\"number_field\":3,\"bool_field\":true,\"second_u64_string_field\":\"1\"}\n"); } TEST(CASJsonWriter, EmptyObjectAndClear) @@ -217,12 +220,12 @@ TEST(CASJsonWriter, WireKeyFieldHelpersMatchThePrimitivePairs) { CasJsonWriter w; bool first = true; - constexpr WireKey k_word{"st"}; - constexpr WireKey k_str{"hn"}; - constexpr WireKey k_u64s{"we"}; - constexpr WireKey k_num{"eat"}; - constexpr WireKey k_hex{"su"}; - constexpr WireKey k_bool{"fen"}; + constexpr WireKey k_word{"word_field"}; + constexpr WireKey k_str{"string_field"}; + constexpr WireKey k_u64s{"u64_string_field"}; + constexpr WireKey k_num{"number_field"}; + constexpr WireKey k_hex{"hex_field"}; + constexpr WireKey k_bool{"bool_field"}; writeWordField(w, k_word, "clean", first); writeStringField(w, k_str, "host-1", first); writeU64StringField(w, k_u64s, 7, first); @@ -232,11 +235,12 @@ TEST(CASJsonWriter, WireKeyFieldHelpersMatchThePrimitivePairs) w.closeObject(first); w.newline(); EXPECT_EQ(std::move(w).take(), - "{\"st\":\"clean\",\"hn\":\"host-1\",\"we\":\"7\",\"eat\":1752537630000," - "\"su\":\"00000000000000000000000000000001\",\"fen\":false}\n"); + "{\"word_field\":\"clean\",\"string_field\":\"host-1\",\"u64_string_field\":\"7\"," + "\"number_field\":1752537630000," + "\"hex_field\":\"00000000000000000000000000000001\",\"bool_field\":false}\n"); /// The reader-side comparison contract: a String key compares against the constant. - String key = "st"; + String key = "word_field"; EXPECT_TRUE(key == k_word); EXPECT_FALSE(key == k_str); } diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 55af475d2048..7a21b24985fd 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -1352,11 +1352,11 @@ TEST(CASMountLease, BodyCarriesFloorAndFence) m.started_at_ms = 1000; m.seq = 3; m.expires_at_ms = 2000; - m.min_active = 5; + m.min_active_build_sequence = 5; m.gc_fenced = true; m.write_attempt_id = UInt128{1}; const MountLease d = decodeMountLease(encodeMountLease(m)); - EXPECT_EQ(d.min_active, 5u); + EXPECT_EQ(d.min_active_build_sequence, 5u); EXPECT_TRUE(d.gc_fenced); EXPECT_EQ(d.writer_epoch, 7u); } @@ -1364,9 +1364,9 @@ TEST(CASMountLease, BodyCarriesFloorAndFence) TEST(CASMountLease, RetiredSentinelRoundTrips) { MountLease m; - m.min_active = std::numeric_limits::max(); + m.min_active_build_sequence = std::numeric_limits::max(); m.write_attempt_id = UInt128{1}; - EXPECT_EQ(decodeMountLease(encodeMountLease(m)).min_active, + EXPECT_EQ(decodeMountLease(encodeMountLease(m)).min_active_build_sequence, std::numeric_limits::max()); } @@ -1386,7 +1386,7 @@ constexpr uint64_t kStableThresholdMs = 10'000; /// `putIfAbsent`) — the same interface the keeper writes through. MountLease seedMount( Backend & b, const Layout & l, const String & srid, - uint64_t expires_at_ms, bool gc_fenced, uint64_t min_active, uint64_t seq = 1) + uint64_t expires_at_ms, bool gc_fenced, uint64_t min_active_build_sequence, uint64_t seq = 1) { MountLease m; m.server_uuid = UInt128(srid.back()); // distinct per srid; content is irrelevant to the gate @@ -1396,7 +1396,7 @@ MountLease seedMount( m.started_at_ms = kNowMs; m.seq = seq; m.expires_at_ms = expires_at_ms; - m.min_active = min_active; + m.min_active_build_sequence = min_active_build_sequence; m.gc_fenced = gc_fenced; m.write_attempt_id = UInt128{1}; b.putIfAbsent(l.mountKey(srid), encodeMountLease(m)); @@ -1425,7 +1425,7 @@ TEST(CASHeartbeatFloor, FirstSightNeverFencesEvenIfStampLooksExpired) /// A stamp that would have read as long-expired under the old skew-margin comparison — under /// rev.6 observation the stamp is never even consulted for the fence decision. - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; const HeartbeatFloor floor = computeHeartbeatFloor(*b, l, /*now_ms*/ kNowMs, /*mono_now_ms*/ 0, @@ -1441,7 +1441,7 @@ TEST(CASHeartbeatFloor, StableTokenPastThresholdIsFenced) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; const HeartbeatFloor floor_before = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); @@ -1465,7 +1465,7 @@ TEST(CASHeartbeatFloor, RenewalBetweenRoundsRestartsObservation) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); @@ -1494,8 +1494,8 @@ TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); - seedMount(*b, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(*b, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); @@ -1526,15 +1526,15 @@ TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) /// two live mounts — genuinely renewing between the two rounds below, so their observation never /// stabilizes. - seedMount(*b, l, "s1", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active*/ 0); - seedMount(*b, l, "s2", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(*b, l, "s2", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); /// dead — no renewal between the two rounds below — must be fenced-out by the second call. - seedMount(*b, l, "s3", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s3", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); /// already-fenced — excluded, body byte-identical after both calls (no PUT). - seedMount(*b, l, "s4", /*expires*/ kNowMs - 60'000, /*fenced*/ true, /*min_active*/ 0); - /// terminated (min_active == UINT64_MAX) with expired-looking timestamps — excluded, not fenced. + seedMount(*b, l, "s4", /*expires*/ kNowMs - 60'000, /*fenced*/ true, /*min_active_build_sequence*/ 0); + /// terminated (min_active_build_sequence == UINT64_MAX) with expired-looking timestamps — excluded, not fenced. seedMount(*b, l, "s5", /*expires*/ kNowMs - 60'000, /*fenced*/ false, - /*min_active*/ std::numeric_limits::max()); + /*min_active_build_sequence*/ std::numeric_limits::max()); MountObservationMap obs; @@ -1627,7 +1627,7 @@ TEST(CASHeartbeatFloor, FenceOutLosesTokenRaceReclassifiesLive) auto b = std::make_shared( l.mountKey("s1"), /*renewed_expires*/ kNowMs + 120'000); - seedMount(*b, l, "s1", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; /// Round 1: first sight, observation starts — never reaches the fence-out path (the race @@ -1912,7 +1912,7 @@ TEST(CASFenceTerminal, CleanFarewellIsTerminal) auto got = b.get(l.mountKey("r")); ASSERT_TRUE(got.has_value()); MountLease retired = decodeMountLease(got->bytes); - retired.min_active = std::numeric_limits::max(); + retired.min_active_build_sequence = std::numeric_limits::max(); ASSERT_EQ(b.putOverwrite(l.mountKey("r"), encodeMountLease(retired), got->token).outcome, PutOutcome::Done); EXPECT_TRUE(isCreatorFenceTerminal(b, l, "r", 7)); diff --git a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp index 901b223cb90f..ba1b0925a20f 100644 --- a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp @@ -1,7 +1,6 @@ #include #include -#include #include #include #include @@ -9,11 +8,6 @@ #include "cas_test_helpers.h" #include -namespace DB::ErrorCodes -{ - extern const int UNKNOWN_FORMAT_VERSION; -} - /// Namespace files are keyed by an opaque LIFE, not by its name: `cas/ns/state//_files/` /// (Stage B Task 4b, directive design change 2). This file pins the three properties that re-key exists /// to produce, and the one it must NOT produce. @@ -229,51 +223,3 @@ TEST(CASNsFileIncarnation, RebirthDoesNotWaitForFilesToBeEmpty) EXPECT_EQ(row_it->second.cleanup_evidence->remove_txn_id, (RefTxnId{1, 1})); EXPECT_TRUE(backend->head(debris_key).exists) << "cleanup evidence does not gate on physical deletion"; } - -/// An old-format pool carrying unqualified `roots//_files/x` keys is REFUSED AT OPEN. It is not -/// read, not migrated, and not silently re-keyed: the file layer rides Task 4's format bump B, and the -/// pool-open floor is what makes "there is nothing to migrate" true rather than merely intended. -/// -/// Asserted at OPEN rather than at the parser on purpose: `Layout` has no unqualified key constructor -/// at all (a compile-time concept check in `gtest_cas_namespace_life_id.cpp` pins that, and -/// `parseNamespaceFileKey`'s refusal of a legacy key is pinned there too), so the only reachable -/// question left is whether a pool that CONTAINS such keys can be opened. It cannot. -TEST(CASNsFileIncarnation, LegacyUnqualifiedFileKeyIsRefusedAtOpen) -{ - auto backend = std::make_shared(); - const Layout layout("p"); - - /// A generation-5 `_pool_meta`: the current encoder's output with its header generation moved back - /// one, so every other byte is exactly what that generation really wrote. - PoolMeta meta; - meta.pool_id = hexToU128("0123456789abcdef0123456789abcdef"); - meta.blob_header_len = 256; - meta.min_reader_generation = kNamespaceLifeKeyedGeneration - 1; - meta.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - String encoded = encodePoolMeta(meta); - const String current_v = "\"v\":" + std::to_string(G_BUILD); - const String legacy_v = "\"v\":" + std::to_string(kNamespaceLifeKeyedGeneration); - const size_t at = encoded.find(current_v); - /// Guard the substitution itself: a silent no-op here would leave a CURRENT-generation pool and the - /// test would pass by opening a pool it believes it downgraded. - ASSERT_NE(at, String::npos) << "pool-meta header no longer spells its generation as " << current_v; - encoded.replace(at, current_v.size(), legacy_v); - ASSERT_NE(encoded.find(legacy_v), String::npos); - backend->putIfAbsent(layout.poolMetaKey(), encoded); - - /// The legacy artifact this task removes: a namespace file keyed by NAME ONLY, with no incarnation - /// segment. Written as raw bytes because no code path in the tree can produce this key any more. - backend->putIfAbsent("p/roots/" + kNsString + "/_files/" + kFile, "1\n"); - - try - { - openPoolForTest(backend); - FAIL() << "an old-format pool must fail closed at open, naming recreation"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find("recreate"), String::npos) - << "the refusal must tell the operator what to do; got: " << e.message(); - } -} diff --git a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp index 3166230f5ec8..77ff85d2b72d 100644 --- a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp @@ -104,7 +104,7 @@ void deleteCatalogLife( CasFoldSeal parent; parent.ref_lives.emplace(life1.incarnation, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); if (CasRefCatalog::deleteCompletedRemoving( backend, layout, *it, parent, 1, diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index 45f82c1a9e33..0a19b2418a00 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -166,7 +166,7 @@ TEST(CASOrphanManifestSweep, EligibleAndUnownedIsDeleted) registerNamespaceRaw(*backend, store->layout(), ns); const ManifestRef r = ref(5, 0xAB); writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); // body, no owner - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active*/6); // 6 > 5 => eligible + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active_build_sequence*/6); // 6 > 5 => eligible seedConsumedSealCursor(*backend, store->layout(), ns); seedEmptyRecoveryAuthority(*backend, store->layout(), ns); @@ -214,7 +214,7 @@ TEST(CASOrphanManifestSweep, CheckpointSnapshotAtOlderEpochSealSkipsDeletion) const ManifestRef candidate = ref(5, 0xAC); const String candidate_key = layout.manifestKey(ManifestId{ns, candidate}); writeManifestRaw(*backend, layout, ns, candidate, {blobEntryFor("a", DB::UInt128(1))}); - setWatermarkMinActive(*backend, layout, kServerRoot, kWriterEpoch, /*min_active=*/6); + setWatermarkMinActive(*backend, layout, kServerRoot, kWriterEpoch, /*min_active_build_sequence=*/6); seedConsumedSealCursor(*backend, layout, ns); std::vector warnings; @@ -299,7 +299,7 @@ TEST(CASOrphanManifestSweep, CursorPageAdvancesAndWrapsWithListBudget) const ManifestRef r2 = ref(5, 0xE2); writeManifestRaw(*backend, store->layout(), ns, r1, {blobEntryFor("a", DB::UInt128(1))}); writeManifestRaw(*backend, store->layout(), ns, r2, {blobEntryFor("b", DB::UInt128(2))}); - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active_build_sequence*/6); const ManifestSweepResult first = sweepManifestCursorPageForTest(*store, "", /*list_budget*/1, /*delete_budget*/0); EXPECT_EQ(first.listed, 1u); @@ -553,11 +553,11 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip .ops = publishCommittedOps("phantom", phantom), .prev_epoch_seal = RefTxnId{6, 1}}; String malformed_later_link = encodeRefLogTxn(direct_later_link); - const String encoded_predecessor{R"("!pse":"6")"}; + const String encoded_predecessor{R"("!prev_epoch":"6")"}; const size_t predecessor_pos = malformed_later_link.find(encoded_predecessor); ASSERT_NE(predecessor_pos, String::npos); malformed_later_link.replace( - predecessor_pos, encoded_predecessor.size(), R"("!pse":"2")"); + predecessor_pos, encoded_predecessor.size(), R"("!prev_epoch":"2")"); ASSERT_EQ(backend->putIfAbsent( store->layout().refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, malformed_later_link)).outcome, PutOutcome::Done); diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index 43963c40e800..6285f171e3f3 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -150,7 +150,7 @@ ReadyFixture makeReadyFixture() .last_epoch_seal = RefTxnId{1, 2}, }); EXPECT_TRUE(runRegularRoundReclaiming(*f.gc).acquired_lease); - setWatermarkMinActive(*f.backend, f.store->layout(), "test", kCandidateEpoch, /*min_active=*/6); + setWatermarkMinActive(*f.backend, f.store->layout(), "test", kCandidateEpoch, /*min_active_build_sequence=*/6); std::vector entries; std::vector seeded_edges; diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index ac14519e04c5..e966b87a332a 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -5,6 +5,8 @@ #include #include +#include + using namespace DB::Cas; namespace @@ -67,11 +69,11 @@ TEST(CASFormatBattery, PartManifest) /// stays self-consistent with whatever sample() produces, now that decode verifies payload_digest. const String golden = currentFormatHeader("cas_part_manifest") + - "{\"me\":\"5\",\"mb\":\"15\",\"mo\":1,\"ns\":\"00/aa@cas@\",\"pd\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. - "{\"p\":\"a/b.bin\",\"pm\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\",\"sz\":4096}\n" - "{\"p\":\"c/small.txt\",\"pm\":\"inline\",\"il\":12}\n" + "{\"epoch\":\"5\",\"build\":\"15\",\"ord\":1,\"root_namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. + "{\"path\":\"a/b.bin\",\"place\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\",\"size\":4096}\n" + "{\"path\":\"c/small.txt\",\"place\":\"inline\",\"size\":12}\n" "{\"n\":2}\n" - "==> \"c/small.txt\" il=12 <==\n" + "==> \"c/small.txt\" size=12 <==\n" "hello world!\n"; runFormatBattery({FormatId::PartManifest, [&] { return sealObject(FormatId::PartManifest, encodePartManifest(m)); }, @@ -115,17 +117,88 @@ TEST(CASPartManifestFormat, EmptyEntriesRoundTrips) TEST(CASPartManifestFormat, PlacementWordsRenderAndRejectUnknown) { const String text = encodePartManifest(sample()); - EXPECT_NE(text.find("\"pm\":\"blob\""), String::npos); - EXPECT_NE(text.find("\"pm\":\"inline\""), String::npos); + EXPECT_NE(text.find("\"place\":\"blob\""), String::npos); + EXPECT_NE(text.find("\"place\":\"inline\""), String::npos); /// An unknown placement word fails closed. String bad = text; - const size_t pos = bad.find(R"("pm":"blob")"); + const size_t pos = bad.find(R"("place":"blob")"); ASSERT_NE(pos, String::npos); - bad.replace(pos, String(R"("pm":"blob")").size(), R"("pm":"bogus")"); + bad.replace(pos, String(R"("place":"blob")").size(), R"("place":"bogus")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } +/// Closed-set pin: the two `EntryPlacement` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASPartManifestFormat, ClosedSetPinsEntryPlacementWords) +{ + EXPECT_EQ(entryPlacementToWireWord(EntryPlacement::Inline), "inline"); + EXPECT_EQ(entryPlacementToWireWord(EntryPlacement::Blob), "blob"); + for (const auto p : magic_enum::enum_values()) + EXPECT_EQ(entryPlacementFromWireWord(entryPlacementToWireWord(p)), p); +} + +TEST(CASPartManifestFormat, SizeBeforePlaceIsAcceptedForBothPlacements) +{ + String blob_first = encodePartManifest(sample()); + const String blob_record = R"("place":"blob","algo":"ch128","digest":"00112233445566778899aabbccddeeff","size":4096)"; + const size_t blob_pos = blob_first.find(blob_record); + ASSERT_NE(blob_pos, String::npos); + blob_first.replace(blob_pos, blob_record.size(), + R"("size":4096,"place":"blob","algo":"ch128","digest":"00112233445566778899aabbccddeeff")"); + EXPECT_EQ(decodePartManifest(blob_first).entries[0].blob_size, 4096u); + + String inline_first = encodePartManifest(sample()); + const String inline_record = R"("place":"inline","size":12)"; + const size_t inline_pos = inline_first.find(inline_record); + ASSERT_NE(inline_pos, String::npos); + inline_first.replace(inline_pos, inline_record.size(), R"("size":12,"place":"inline")"); + EXPECT_EQ(decodePartManifest(inline_first).entries[1].inline_bytes, "hello world!"); +} + +TEST(CASPartManifestFormat, MissingSizeIsRejectedForBothPlacements) +{ + /// The MESSAGE is asserted, not just the code: a manifest whose entry lost its size also fails + /// the payload-digest check (blob) and the banner rebuild (inline), both of which raise the same + /// code, so a code-only assertion would still pass with the per-placement fences deleted. + const auto expect_message = [](const String & text, std::string_view expected) + { + try + { + static_cast(decodePartManifest(text)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), expected); + } + }; + + String blob_missing_size = encodePartManifest(sample()); + const size_t blob_pos = blob_missing_size.find(",\"size\":4096"); + ASSERT_NE(blob_pos, String::npos); + blob_missing_size.erase(blob_pos, String(",\"size\":4096").size()); + expect_message(blob_missing_size, "PartManifest: blob entry 'a/b.bin' missing size"); + + String inline_missing_size = encodePartManifest(sample()); + const size_t inline_pos = inline_missing_size.find(",\"size\":12"); + ASSERT_NE(inline_pos, String::npos); + inline_missing_size.erase(inline_pos, String(",\"size\":12").size()); + expect_message(inline_missing_size, "PartManifest: inline entry 'c/small.txt' missing size"); +} + +TEST(CASPartManifestFormat, DuplicateSizeIsRejected) +{ + String text = encodePartManifest(sample()); + const String size = R"("size":4096)"; + const size_t pos = text.find(size); + ASSERT_NE(pos, String::npos); + text.replace(pos, size.size(), R"("size":1,"size":4096)"); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(text); }); +} + /// Proves the payload zone, not JSON-string escaping: an Inline entry whose bytes contain an /// embedded '\n', a NUL byte, and a '"' character round-trip byte-faithfully. If this content were /// carried as a JSON string value it would need escaping (or would be flatly invalid for the NUL @@ -198,7 +271,7 @@ TEST(CASPartManifestFormat, InlineBannerCarriesTheEscapedPath) m.entries = {e}; m.payload_digest = computePayloadDigest(m); - EXPECT_NE(encodePartManifest(m).find("==> \"p\\nq.proj/c.txt\" il=1 <=="), String::npos); + EXPECT_NE(encodePartManifest(m).find("==> \"p\\nq.proj/c.txt\" size=1 <=="), String::npos); } TEST(CASPartManifestFormat, ByteDeterminism) @@ -322,8 +395,8 @@ TEST(CASPartManifestFormat, DecodeRejectsOutOfOrderEntries) m.payload_digest = computePayloadDigest(m); const String text = encodePartManifest(m); - const size_t pos_a = text.find(R"("p":"a/one.bin")"); - const size_t pos_b = text.find(R"("p":"b/two.bin")"); + const size_t pos_a = text.find(R"("path":"a/one.bin")"); + const size_t pos_b = text.find(R"("path":"b/two.bin")"); ASSERT_NE(pos_a, String::npos); ASSERT_NE(pos_b, String::npos); @@ -366,10 +439,10 @@ TEST(CASPartManifestFormat, DecodeRejectsNonAdjacentDuplicatePath) m.payload_digest = computePayloadDigest(m); String forged = encodePartManifest(m); - const String needle = R"("p":"ccc/three.bin")"; + const String needle = R"("path":"ccc/three.bin")"; const size_t pos = forged.find(needle); ASSERT_NE(pos, String::npos); - forged.replace(pos, needle.size(), R"("p":"aaa/one.bin")"); + forged.replace(pos, needle.size(), R"("path":"aaa/one.bin")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(forged); }); } @@ -377,10 +450,10 @@ TEST(CASPartManifestFormat, DecodeRejectsNonAdjacentDuplicatePath) TEST(CASPartManifestFormat, UnknownEntryAlgoFailsClosed) { String bad = encodePartManifest(sample()); - const String needle = R"("ha":"ch128")"; + const String needle = R"("algo":"ch128")"; const size_t pos = bad.find(needle); ASSERT_NE(pos, String::npos); - bad.replace(pos, needle.size(), R"("ha":"bogus")"); + bad.replace(pos, needle.size(), R"("algo":"bogus")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } @@ -390,7 +463,7 @@ TEST(CASPartManifestFormat, UnknownEntryAlgoFailsClosed) TEST(CASPartManifestFormat, DigestHexWidthMismatchFailsClosedNotBadArguments) { String bad = encodePartManifest(sample()); - const String key = R"("h":")"; + const String key = R"("digest":")"; const size_t key_pos = bad.find(key); ASSERT_NE(key_pos, String::npos); const size_t hex_start = key_pos + key.size(); @@ -429,18 +502,18 @@ TEST(CASPartManifestFormat, TrailingByteAfterPayloadZoneFailsClosed) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } -/// An Inline entry's record "il" disagrees with what the payload zone's banner+bytes actually -/// declare (the banner and bytes are left as originally written; only the record line's "il" is -/// edited). The record's declared `il` is what decode uses both to build the expected banner text +/// An Inline entry's record `size` disagrees with what the payload zone's banner+bytes actually +/// declare (the banner and bytes are left as originally written; only the record line's `size` is +/// edited). The record's declared `size` is what decode uses both to build the expected banner text /// and to know how many bytes to read from the zone, so this must fail closed rather than silently /// reading the wrong byte count. -TEST(CASPartManifestFormat, InlineRecordIlMismatchWithPayloadZoneBannerFailsClosed) +TEST(CASPartManifestFormat, InlineRecordSizeMismatchWithPayloadZoneBannerFailsClosed) { String bad = encodePartManifest(sample()); - const String needle = "\"il\":12"; + const String needle = "\"size\":12"; const size_t pos = bad.find(needle); ASSERT_NE(pos, String::npos); - bad.replace(pos, needle.size(), "\"il\":13"); + bad.replace(pos, needle.size(), "\"size\":13"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 7a426676cbc8..a92d8f9bc4d1 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -1260,7 +1260,7 @@ TEST(CASPartWriteTxn, AbandonRemovesStagedDebrisAndDisables) { /// Port of AbandonLeavesDebrisAndDisables to the new abandon semantics (CasPartWriteTxn.cpp abandon): /// abandon best-effort exact-token-DELETEs this build's STAGED manifest debris, leaves blob bodies - /// (full GC's job via min_active), and disables the build (further ops throw via requireAlive). + /// (full GC's job via min_active_build_sequence), and disables the build (further ops throw via requireAlive). auto b = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"srv1/tbl"}; diff --git a/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp b/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp index b1022505585d..1f86c6fe7d47 100644 --- a/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp +++ b/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp @@ -62,7 +62,7 @@ ManifestEntry blobEntry(const String & name, const String & payload) /// public PartWriteTxn/Pool/Gc API (no snap injection): /// /// PartWriteTxn A uploads blob P and publishes refA -> t1 -> { data.bin: P }. A is then RELEASED (dtor), -/// retiring its build_seq so the GC watermark `min_active` advances PAST A. P now carries A's +/// retiring its build_seq so the GC watermark `min_active_build_sequence` advances PAST A. P now carries A's /// `cas_owner` and is no longer protected by any in-flight build. /// /// PartWriteTxn B starts and ADOPTS the same blob P via tokenless evidence (adoptEvidence — the cross-node @@ -83,7 +83,7 @@ TEST(CASPartWriteTxnRootDangle, SharedBlobSurvivesSourceDropDuringBuild) const String P = "shared-blob-payload-P"; /// PartWriteTxn A: upload P, publish refA -> manifest -> { data.bin: P }, then release A so its build_seq - /// retires and min_active advances past it. + /// retires and min_active_build_sequence advances past it. { PartWriteInfo info; info.intended_ref = ns.string() + "/refA"; @@ -93,7 +93,7 @@ TEST(CASPartWriteTxnRootDangle, SharedBlobSurvivesSourceDropDuringBuild) a->putBlob(idOf(P), BlobSource::fromString(P)); a->promote(ns, "refA", a->buildId(), id); } - s->renewWatermarkOnce(); /// A is gone; min_active now advances past A's build_seq + s->renewWatermarkOnce(); /// A is gone; min_active_build_sequence now advances past A's build_seq /// PartWriteTxn B: adopt the SAME blob P (cross-node adopt — tokenless evidence via adoptEvidence), assemble /// its manifest, and precommitAdd it. The precommit pins P's closure (fold +1 edge) for the build. @@ -145,7 +145,7 @@ TEST(CASPartWriteTxnRootDangle, PrematureReclaimCommitFailsClosed) const RootNamespace ns{"test/tbl"}; const String P = "shared-blob-payload-P-reclaim"; - /// PartWriteTxn A: upload P, publish refA -> manifest, retire A so min_active advances past it. + /// PartWriteTxn A: upload P, publish refA -> manifest, retire A so min_active_build_sequence advances past it. { PartWriteInfo info; info.intended_ref = ns.string() + "/refA"; @@ -232,7 +232,7 @@ TEST(CASPartWriteTxnRoot, LivePrecommitNotReclaimed) const String Q = "live-build-blob-payload-Q"; /// PartWriteTxn B stays ALIVE: upload Q, assemble, precommitAdd — and we DO NOT retire its seq. So - /// `min_active <= build_seq` (B is in-flight) and the watermark keeps a live, advancing seq. + /// `min_active_build_sequence <= build_seq` (B is in-flight) and the watermark keeps a live, advancing seq. PartWriteInfo binfo; binfo.intended_ref = ns.string() + "/refLive"; auto b = s->beginPartWrite(binfo); @@ -240,7 +240,7 @@ TEST(CASPartWriteTxnRoot, LivePrecommitNotReclaimed) b->precommitAdd(ns, "refLive", t); b->putBlob(idOf(Q), BlobSource::fromString(Q)); s->renewWatermarkOnce(); - ASSERT_LE(s->minActive(), b->buildSeq()) << "precondition: B must be in-flight (min_active <= seq)"; + ASSERT_LE(s->minActive(), b->buildSeq()) << "precondition: B must be in-flight (min_active_build_sequence <= seq)"; /// GC to fixpoint while B is live. Gc gc(s, u128Of("gc-b8-live")); diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index c960c0547037..a74baf05fa61 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -680,22 +680,15 @@ TEST(CASPluggableHash, ForeignAlgoSegmentIsDebrisNotOurs) } /// ============================================================================================ -/// CAS reader-generation gate (`Core/Formats/CasFormat.h`'s `G_BUILD`) was raised to 4 for -/// per-namespace contiguous ref-log ids (INV-1) and has since moved again, to 5, for Stage B's -/// namespace-life-keyed ref layer ("format bump B", `kNamespaceLifeKeyedGeneration`) -- this test's -/// assertions read `G_BUILD` itself rather than a hardcoded generation number for exactly that reason, -/// so a THIRD bump does not silently make them false. `PoolMeta::createOrValidate`'s open-time -/// CAS-raise targets `G_BUILD`, and `decodePoolMeta` fail-closes BOTH on a FUTURE -/// `min_reader_generation` AND on a BACKWARD pool whose header `compatibility_version` is below -/// `kNamespaceLifeKeyedGeneration` (which, being the LATER of the two historical breaking-change -/// floors, subsumes `kContiguousRefStreamsGeneration` -- see `CasPoolMetaFormat.cpp`). +/// CAS reader-generation gate (`CasFormat.h`'s `G_BUILD`). This test's assertions read `G_BUILD` +/// itself rather than a hardcoded generation number, so a future bump does not silently make them +/// false. `PoolMeta::createOrValidate`'s open-time CAS-raise targets `G_BUILD`, and `decodePoolMeta` +/// fail-closes BOTH on a FUTURE `min_reader_generation` AND on a BACKWARD pool whose header +/// `compatibility_version` is below the format-generation baseline (see `CasPoolMetaFormat.cpp`). /// ============================================================================================ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { - EXPECT_GE(G_BUILD, kNamespaceLifeKeyedGeneration) - << "the reader-generation gate must be at least the namespace-life-keyed floor it enforces"; - /// A freshly opened/created pool records `min_reader_generation == G_BUILD` (the open-time /// CAS-raise, `PoolMeta::createOrValidate`, always targets this build's own floor). { @@ -709,8 +702,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) } /// FORWARD gate: a pool-meta carrying `min_reader_generation == G_BUILD + 1` (one generation past - /// THIS build's floor) still fails closed at open -- the startup gate (`decodePoolMeta`) rejects it - /// even though generation 4 is now understood. + /// THIS build's floor) fails closed at open -- the startup gate (`decodePoolMeta`) rejects it. { auto backend = std::make_shared(); const Layout layout("p"); @@ -722,12 +714,9 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); }); } - /// BACKWARD floor: a pool whose header `v` (compatibility_version) is BELOW `G_BUILD` was written - /// by an older build this reader can no longer trust -- today that is one generation short of - /// `kNamespaceLifeKeyedGeneration`, a pool whose ref-object keys carry no incarnation segment, - /// which this build's parsers refuse as corruption rather than read. Craft it at the text layer: - /// take a fresh pool-meta and rewrite its line-1 version gate down to `G_BUILD - 1` (an older - /// build would have stamped exactly that). + /// BACKWARD floor: a pool whose header `v` (compatibility_version) is below the format-generation + /// baseline predates every build this reader can trust. Craft it at the text layer: take a fresh + /// pool-meta and rewrite its line-1 version gate down to `G_BUILD - 1`. { auto backend = std::make_shared(); const Layout layout("p"); diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index b0ad1b340afd..8e07048439f8 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -511,6 +511,7 @@ TEST(CASPoolMeta, RejectsBadConstantsOnDecode) PoolMeta bad_pm; bad_pm.pool_id = hexToU128("00000000000000000000000000000001"); bad_pm.blob_header_len = 100; /// violates 8-alignment invariant + bad_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(bad_pm)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { PoolMeta::createOrValidate(*b, layout, 256); }); @@ -2009,7 +2010,7 @@ TEST(CASPoolShutdown, CleanStopDrainsAndWritesFarewell) const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()); const MountLease lease = decodeMountLease(got->bytes); - EXPECT_EQ(lease.min_active, std::numeric_limits::max()) + EXPECT_EQ(lease.min_active_build_sequence, std::numeric_limits::max()) << "a clean drain (no in-flight ref-log PUT) must write the farewell marker"; } @@ -2046,7 +2047,7 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()); const MountLease lease = decodeMountLease(got->bytes); - EXPECT_NE(lease.min_active, std::numeric_limits::max()) + EXPECT_NE(lease.min_active_build_sequence, std::numeric_limits::max()) << "an unresolved ref-log PUT must skip the clean-release farewell marker"; EXPECT_FALSE(lease.gc_fenced); @@ -2074,7 +2075,7 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) Layout l{"p"}; DB::Cas::tests::seedPoolMetaForRestart(*b); /// Predecessor: claim epoch 7, no farewell (simulate crash: just drop the keeper) -- a bare - /// `claimMount` plants the lease directly, with no clean-farewell `min_active` marker and no + /// `claimMount` plants the lease directly, with no clean-farewell `min_active_build_sequence` marker and no /// `gc_fenced`, so the successor below has no certificate of death until it observes one itself. ASSERT_EQ(claimMount(*b, l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, MountClaimResult::Claimed); @@ -2121,7 +2122,7 @@ TEST(CASMountOpenWaits, CleanOpenSkipsAllWaits) { auto b = std::make_shared(); /// Predecessor released cleanly (drain + farewell from Task 5): open, then reset() drives ~Pool(), - /// which -- with nothing in flight -- writes the farewell marker (min_active == UINT64_MAX). + /// which -- with nothing in flight -- writes the farewell marker (min_active_build_sequence == UINT64_MAX). auto predecessor = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test"}); predecessor.reset(); @@ -2677,7 +2678,7 @@ TEST(CASPoolRemount, TeardownJoinsBothWorkersBeforeRelease) runtime.stopBackgroundWorkers(); EXPECT_EQ(worker_exits.load(), 2u); runtime.finishTeardown(true); - EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active, + EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active_build_sequence, std::numeric_limits::max()); } @@ -3372,7 +3373,7 @@ TEST(CASPoolShutdown, PreSendCancellationAllowsFarewellButAmbiguityDoesNot) runtime.stopBackgroundWorkers(); } runtime.finishTeardown(true); - return decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active; + return decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active_build_sequence; }; EXPECT_EQ(run(false), std::numeric_limits::max()); diff --git a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp index ee1b2a4d4d99..07a9591e6fc6 100644 --- a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp +++ b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp @@ -431,7 +431,7 @@ TEST(CASRebuildCondemnNothing, CarriesHoldsVerbatimWhileCondemningNothing) const CasFoldSeal seal = decodeFoldSeal(backend->get(layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); const auto it = seal.ref_lives.find(catalogLifeIdForTest(*backend, layout, kNsA)); ASSERT_NE(it, seal.ref_lives.end()); - EXPECT_EQ(it->second.coverage.classification, 4); + EXPECT_EQ(it->second.coverage.classification, CoverageClass::Clamped); ASSERT_TRUE(it->second.coverage.hold.has_value()); EXPECT_EQ(*it->second.coverage.hold, planted) << "a rebuild retried nothing, so it rewrites nothing about the hold"; diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 2c2805f49310..355ec21a2126 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -7,6 +7,8 @@ #include #include +#include + using namespace DB; using namespace DB::Cas; @@ -76,7 +78,7 @@ TEST(CASFormatBattery, RunFile) [&] { return sealObject(FormatId::RunFile, encodeRun(records)); }, [](std::string_view s) { decodeRun(std::string(openObject(FormatId::RunFile, s))); }, fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()) + - "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n" + "{\"ref\":\"0100000000000000000000000000000002\",\"src\":\"00000000000000000000000000000005\",\"mark\":\"edge\"}\n" "{\"n\":1}\n"}); } @@ -126,6 +128,69 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) EXPECT_EQ(back[2].marker, RunMarker::Zero); } +/// Closed-set pin: the three `RunMarker` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASRecordStream, ClosedSetPinsRunMarkerWords) +{ + EXPECT_EQ(runMarkerToWireWord(RunMarker::Zero), "zero"); + EXPECT_EQ(runMarkerToWireWord(RunMarker::Edge), "edge"); + EXPECT_EQ(runMarkerToWireWord(RunMarker::Condemned), "condemned"); + for (const auto m : magic_enum::enum_values()) + EXPECT_EQ(runMarkerFromWireWord(runMarkerToWireWord(m)), m); +} + +/// The condemned row's six fields are all-or-nothing: a row that says `condemned` but drops one of +/// them would decode with a silently defaulted value (a zero size, an empty token, `pending` false), +/// which is a different retention decision than the writer recorded. +TEST(CASRecordStream, CondemnedRowMissingOneOfItsSixFieldsFailsClosed) +{ + const String good = encodeRun({condemned(chRef(2), Token{"e-1", TokenType::ETag}, 4242, 7, /*pend*/ true)}); + for (const std::string_view field : {R"(,"pending":true)", R"(,"token_type":"etag")", R"(,"token":"e-1")", + R"(,"size":4242)", R"(,"condemn_round":"7")", R"(,"confirmed":false)"}) + { + String bytes = good; + const size_t at = bytes.find(field); + ASSERT_NE(at, String::npos) << "fixture does not carry " << field; + bytes.erase(at, field.size()); + try + { + static_cast(decodeRun(bytes)); + FAIL() << "expected CORRUPTED_DATA after dropping " << field; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + const String expected_message = field == R"(,"token_type":"etag")" || field == R"(,"token":"e-1")" + ? "CAS cas_run: token missing token_type/token" + : "CAS cas_run: condemned record missing pending/size/condemn_round/confirmed"; + EXPECT_EQ(e.message(), expected_message); + } + } +} + +/// The mirror fence: an active row carrying any condemned field is a row whose two halves disagree +/// about what it is, and the reader must not pick one half. +TEST(CASRecordStream, ActiveRowCarryingACondemnedFieldFailsClosed) +{ + String bytes = encodeRun({edge(chRef(1), 10)}); + const String needle = R"(,"mark":"edge")"; + const size_t at = bytes.find(needle); + ASSERT_NE(at, String::npos); + bytes.insert(at + needle.size(), R"(,"size":4242)"); + + try + { + static_cast(decodeRun(bytes)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), "CAS cas_run: non-condemned record carries condemned fields"); + } +} + TEST(CASRecordStream, WriterIsByteDeterministic) { std::vector recs = { @@ -136,6 +201,24 @@ TEST(CASRecordStream, WriterIsByteDeterministic) EXPECT_EQ(encodeRun(recs), encodeRun(recs)); /// pure function of the sorted record set } +/// The run `ref` carries the algorithm as a raw leading BYTE, a second representation of the same +/// closed set the `algo` WORD spells elsewhere. The word side is proven exhaustive at compile time by +/// its wire table; the byte side is a hand-written switch, so nothing but this walk stops a new +/// algorithm from being written by `renderB` and rejected by the reader -- an asymmetry that would +/// appear as unreadable runs rather than as a failing build. +TEST(CASRecordStream, EveryBlobHashAlgoRoundTripsThroughTheRunRefByte) +{ + for (const BlobHashAlgo algo : magic_enum::enum_values()) + { + BlobDigest digest{}; + digest.bytes[0] = 0x10; + const BlobRef ref{algo, digest}; + const std::vector back = decodeRun(encodeRun({edge(ref, 1)})); + ASSERT_EQ(back.size(), 1u) << "algo " << magic_enum::enum_name(algo); + EXPECT_EQ(back[0].ref.algo, algo) << "the leading byte did not survive the round trip"; + } +} + TEST(CASRecordStream, SortOrderAcrossAlgosFollowsAlgoByte) { /// b = . The algo byte leads, so string-sorting b reproduces the @@ -173,9 +256,9 @@ TEST(CASRecordStream, SourceIdRendersAs32Hex) { const String bytes = encodeRun({edge(chRef(1), 10)}); /// The source id 10 is a 32-char lowercase hex string ending in 'a'. - EXPECT_NE(bytes.find("\"s\":\"0000000000000000000000000000000a\""), String::npos); - /// The record key `b` for a ch128 ref is the algo byte 01 + a 32-hex digest (34 chars total). - EXPECT_NE(bytes.find("\"b\":\"01"), String::npos); + EXPECT_NE(bytes.find("\"src\":\"0000000000000000000000000000000a\""), String::npos); + /// The record key `ref` for a ch128 ref is the algo byte 01 + a 32-hex digest (34 chars total). + EXPECT_NE(bytes.find("\"ref\":\"01"), String::npos); } TEST(CASRecordStream, SealChecksumMismatchFailsClosed) @@ -248,12 +331,13 @@ TEST(CASRecordStream, HeaderGates) { /// Wrong type. { - const String s = "{\"type\":\"cas_pool_meta\",\"v\":3,\"kind\":\"source_edge\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_pool_meta\",\"v\":1,\"kind\":\"source_edge\"}\n{\"n\":0}\n"; EXPECT_THROW(decodeRun(s), DB::Exception); } - /// Wrong kind. + /// Wrong kind. `v:1` is the baseline generation, so it always passes the header gate before the + /// kind check runs. { - const String s = "{\"type\":\"cas_run\",\"v\":3,\"kind\":\"blob_delta\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_run\",\"v\":1,\"kind\":\"blob_delta\"}\n{\"n\":0}\n"; EXPECT_THROW(decodeRun(s), DB::Exception); } /// Future version -> UNKNOWN_FORMAT_VERSION. diff --git a/src/Disks/tests/gtest_cas_recovery_grounding.cpp b/src/Disks/tests/gtest_cas_recovery_grounding.cpp index 5362813bdfa6..d908038a153f 100644 --- a/src/Disks/tests/gtest_cas_recovery_grounding.cpp +++ b/src/Disks/tests/gtest_cas_recovery_grounding.cpp @@ -269,9 +269,9 @@ TEST(CASRecoveryGrounding, RejectsLifeEpochAboveCommittedFrontierOnDecodeAndGrou { const RefCkpt invalid = ckpt(2, RefTxnId{1, 5}); String encoded = encodeRefCkpt(ckpt(1, RefTxnId{1, 5})); - const size_t life_epoch = encoded.find(R"("le":"1")"); + const size_t life_epoch = encoded.find(R"("life_epoch":"1")"); ASSERT_NE(life_epoch, String::npos); - encoded.replace(life_epoch, String{R"("le":"1")"}.size(), R"("le":"2")"); + encoded.replace(life_epoch, String{R"("life_epoch":"1")"}.size(), R"("life_epoch":"2")"); expectCode([&] { (void)decodeRefCkpt(encoded); }, DB::ErrorCodes::CORRUPTED_DATA); expectCode([&] { (void)chooseRecoveryGrounding(catalog(NsState::Live), invalid); }, @@ -568,9 +568,9 @@ TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = RefTxnId{1, 2}}); - const size_t frontier_sequence = malformed_ckpt.find(R"("cts":"2")"); + const size_t frontier_sequence = malformed_ckpt.find(R"("committed_seq":"2")"); ASSERT_NE(frontier_sequence, String::npos); - malformed_ckpt.replace(frontier_sequence, String{R"("cts":"2")"}.size(), R"("cts":"3")"); + malformed_ckpt.replace(frontier_sequence, String{R"("committed_seq":"2")"}.size(), R"("committed_seq":"3")"); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -635,9 +635,9 @@ TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); - const size_t frontier_epoch = malformed_ckpt.find(R"("cte":"1")"); + const size_t frontier_epoch = malformed_ckpt.find(R"("committed_epoch":"1")"); ASSERT_NE(frontier_epoch, String::npos); - malformed_ckpt.replace(frontier_epoch, String{R"("cte":"1")"}.size(), R"("cte":"2")"); + malformed_ckpt.replace(frontier_epoch, String{R"("committed_epoch":"1")"}.size(), R"("committed_epoch":"2")"); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index 41d88fa57487..a458bbb52b6b 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -16,6 +16,8 @@ #include #include +#include + using namespace DB::Cas; namespace ProfileEvents @@ -56,27 +58,27 @@ namespace DB::ErrorCodes namespace { -/// Hand-builds one raw "ent" line, bypassing `encodeRefCatalog` entirely -- used by the decode-side +/// Hand-builds one raw `entry` line, bypassing `encodeRefCatalog` entirely -- used by the decode-side /// rejection tests, which must exercise bytes the encoder itself would refuse to produce. -String rawEntLine(const String & ns, const String & state, const String & inc_hex, +String rawEntryLine(const String & ns, const String & state, const String & inc_hex, std::optional> creator = std::nullopt) { if (!creator) - return fmt::format(R"({{"k":"ent","ns":"{}","st":"{}","inc":"{}"}})", ns, state, inc_hex); + return fmt::format(R"({{"kind":"entry","ns":"{}","state":"{}","life":"{}"}})", ns, state, inc_hex); const auto & [srid, we, fg] = *creator; - return fmt::format(R"({{"k":"ent","ns":"{}","st":"{}","inc":"{}","csr":"{}","cwe":"{}","cfg":"{}"}})", + return fmt::format(R"({{"kind":"entry","ns":"{}","state":"{}","life":"{}","creator":"{}","creator_epoch":"{}","creator_fence":"{}"}})", ns, state, inc_hex, srid, we, fg); } -/// Wraps `ent_lines` in the header/trailer a real `cas_ref_catalog` object carries. `v:1` always +/// Wraps `entry_lines` in the header/trailer a real `cas_ref_catalog` object carries. `v:1` always /// passes the header gate (any version <= the build's `G_BUILD` does), matching the convention /// `gtest_cas_fold_seal_format.cpp`'s `RejectsOutOfRangeNsCleanupState` uses for the same reason. -String rawCatalog(const std::vector & ent_lines) +String rawCatalog(const std::vector & entry_lines) { String out = R"({"type":"cas_ref_catalog","v":1})" "\n"; - for (const String & l : ent_lines) + for (const String & l : entry_lines) out += l + "\n"; - out += fmt::format("{{\"n\":{}}}\n", ent_lines.size()); + out += fmt::format("{{\"n\":{}}}\n", entry_lines.size()); return out; } @@ -84,7 +86,7 @@ String withRemovalStartedRound(String line, uint64_t round) { const size_t close = line.rfind('}'); EXPECT_NE(close, String::npos); - line.insert(close, fmt::format(R"(,"rsr":"{}")", round)); + line.insert(close, fmt::format(R"(,"remove_round":"{}")", round)); return line; } @@ -223,12 +225,25 @@ TEST(CASFormatBattery, RefCatalog) [&] { return sealObject(FormatId::RefCatalog, encodeRefCatalog(c)); }, [](std::string_view s) { decodeRefCatalog(std::string(openObject(FormatId::RefCatalog, s))); }, currentFormatHeader("cas_ref_catalog") + - "{\"k\":\"ent\",\"ns\":\"a\",\"st\":\"creating\",\"inc\":\"00000000000000000000000000000001\"," - "\"csr\":\"srv1\",\"cwe\":\"5\",\"cfg\":\"2\"}\n" - "{\"k\":\"ent\",\"ns\":\"b\",\"st\":\"live\",\"inc\":\"00000000000000000000000000000002\"}\n" + "{\"kind\":\"entry\",\"ns\":\"a\",\"state\":\"creating\",\"life\":\"00000000000000000000000000000001\"," + "\"creator\":\"srv1\",\"creator_epoch\":\"5\",\"creator_fence\":\"2\"}\n" + "{\"kind\":\"entry\",\"ns\":\"b\",\"state\":\"live\",\"life\":\"00000000000000000000000000000002\"}\n" "{\"n\":2}\n"}); } +/// Closed-set pin: the three `NsState` words, walked through `magic_enum::enum_values`, which is what +/// proves the renderer and the parser consult the SAME table: a table entry missing altogether is +/// already a build error at the coverage assert, but two delegates drifting onto different tables is +/// not. +TEST(CASRefCatalogFormat, ClosedSetPinsNsStateWords) +{ + EXPECT_EQ(nsStateToWord(NsState::Creating), "creating"); + EXPECT_EQ(nsStateToWord(NsState::Live), "live"); + EXPECT_EQ(nsStateToWord(NsState::Removing), "removing"); + for (const auto s : magic_enum::enum_values()) + EXPECT_EQ(nsStateFromWord(nsStateToWord(s)), s); +} + /// ---------- codec round-trip ---------- TEST(CASRefCatalogFormat, RoundTripsAllThreeStates) @@ -262,16 +277,16 @@ TEST(CASRefCatalogFormat, RemovalStartedRoundIsRequiredExactlyForRemoving) .removal_started_round = 19}; const RefCatalog catalog{.entries = {removing}}; const String encoded = encodeRefCatalog(catalog); - EXPECT_NE(encoded.find("\"rsr\":\"19\""), String::npos); - EXPECT_NE(encoded.find("\"st\":\"removing\""), String::npos); + EXPECT_NE(encoded.find("\"remove_round\":\"19\""), String::npos); + EXPECT_NE(encoded.find("\"state\":\"removing\""), String::npos); EXPECT_EQ(decodeRefCatalog(encoded), catalog); const String inc = "00000000000000000000000000000009"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeRefCatalog(rawCatalog({rawEntLine("missing", "removing", inc)})); }); + [&] { (void)decodeRefCatalog(rawCatalog({rawEntryLine("missing", "removing", inc)})); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - (void)decodeRefCatalog(rawCatalog({withRemovalStartedRound(rawEntLine("forbidden", "live", inc), 21)})); + (void)decodeRefCatalog(rawCatalog({withRemovalStartedRound(rawEntryLine("forbidden", "live", inc), 21)})); }); } @@ -477,7 +492,7 @@ TEST(CASRefCatalogFormatDeathTest, EncodeRejectsLiveWithRemovalStartedRoundAbort #endif /// A namespace + creator server_root_id that both max out at their respective byte bounds (512 + -/// 255), escaped worst-case, land one "ent" line over the 4 KiB line cap (~4.7 KiB) -- reachable +/// 255), escaped worst-case, land one `entry` line over the 4 KiB line cap (~4.7 KiB) -- reachable /// because neither this codec nor `validateServerRootId` restricts the charset, only the length. /// The refusal must be `LIMIT_EXCEEDED` (a capacity refusal), not `LOGICAL_ERROR` (a bug report) -- /// `encodeFoldSeal`'s own `checkLineBytes` raises `LIMIT_EXCEEDED` for the identical shape of gate. @@ -496,60 +511,76 @@ TEST(CASRefCatalogFormat, EncodeLineOverCapRaisesLimitExceeded) TEST(CASRefCatalogFormat, DecodeRejectsDuplicateNamespace) { - const String bad = rawCatalog({rawEntLine("a", "live", u128ToHex(UInt128(1))), - rawEntLine("a", "live", u128ToHex(UInt128(2)))}); + const String bad = rawCatalog({rawEntryLine("a", "live", u128ToHex(UInt128(1))), + rawEntryLine("a", "live", u128ToHex(UInt128(2)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsNonCanonicalOrder) { - const String bad = rawCatalog({rawEntLine("b", "live", u128ToHex(UInt128(1))), - rawEntLine("a", "live", u128ToHex(UInt128(2)))}); + const String bad = rawCatalog({rawEntryLine("b", "live", u128ToHex(UInt128(1))), + rawEntryLine("a", "live", u128ToHex(UInt128(2)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsCreatorPresentOnLive) { - const String bad = rawCatalog({rawEntLine("a", "live", u128ToHex(UInt128(1)), + const String bad = rawCatalog({rawEntryLine("a", "live", u128ToHex(UInt128(1)), std::make_tuple(String("srv"), uint64_t(1), uint64_t(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsCreatorAbsentOnCreating) { - const String bad = rawCatalog({rawEntLine("a", "creating", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine("a", "creating", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsZeroIncarnation) { - const String bad = rawCatalog({rawEntLine("a", "live", u128ToHex(UInt128(0)))}); + const String bad = rawCatalog({rawEntryLine("a", "live", u128ToHex(UInt128(0)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsNameOverByteBound) { const String too_long_ns(kMaxNamespaceBytes + 1, 'a'); - const String bad = rawCatalog({rawEntLine(too_long_ns, "live", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine(too_long_ns, "live", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsUnknownState) { - const String bad = rawCatalog({rawEntLine("a", "bogus", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine("a", "bogus", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } +TEST(CASRefCatalogFormat, DecodeRejectsUnknownEntryKey) +{ + const String bad = rawCatalog( + {R"({"kind":"entry","ns":"a","state":"live","life":"00000000000000000000000000000001","unknown":"x"})"}); + try + { + (void)decodeRefCatalog(bad); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_NE(e.message().find("unknown entry key"), String::npos) << e.message(); + } +} + TEST(CASRefCatalogFormat, DecodeRejectsEmptyNamespace) { - const String bad = rawCatalog({rawEntLine("", "live", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine("", "live", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsMissingNamespaceKey) { /// No "ns" key at all -- must be refused exactly like an explicit empty one, not read as "". - const String bad = rawCatalog({R"({"k":"ent","st":"live","inc":")" + u128ToHex(UInt128(1)) + "\"}"}); + const String bad = rawCatalog({R"({"kind":"entry","state":"live","life":")" + u128ToHex(UInt128(1)) + "\"}"}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } @@ -596,7 +627,7 @@ TEST(CASRefCatalogFormat, RegistryRowIsControlStrictWithRawStorage) EXPECT_EQ(traits.compression, CompressionPolicy::Never); } -/// ---------- capacity admission: per-predicate boundary tests [codex r2/r3 finding 9] ---------- +/// ---------- capacity admission: per-predicate boundary tests ---------- TEST(CASRefCatalogAdmission, Predicate1AcceptsEqualityRefusesCapPlusOne) { @@ -688,7 +719,7 @@ TEST(CASRefCatalogAdmission, ReservationCoversActualWidestLegalRowsAcrossDecimal { seal.ref_lives.emplace(std::numeric_limits::max() - i, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{max, max}, .hold = RefHold{ .reason = HoldReason::UnconsumedSealCrossing, @@ -699,7 +730,7 @@ TEST(CASRefCatalogAdmission, ReservationCoversActualWidestLegalRowsAcrossDecimal } for (uint64_t shard = 0; shard < gc_shards; ++shard) { - /// Predicate 2 charges exactly `gc_shards` widest `btr` rows. This fixture is the maximum + /// Predicate 2 charges exactly `gc_shards` widest `blob_run` rows. This fixture is the maximum /// legal cardinality, not an optimistic producer convention: authoritative fold-seal /// grammar permits at most one run per shard and requires its canonical key to use seq 0. seal.blob_target_runs.push_back(RunRef{ @@ -1113,7 +1144,7 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CasFoldSeal held_parent; held_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{1, 2}, .hold = RefHold{.offending_position = RefTxnId{1, 3}}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); @@ -1124,7 +1155,7 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CasFoldSeal mismatched_parent; mismatched_parent.ref_lives.emplace(UInt128{8}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( backend, layout, removing, mismatched_parent, 5, @@ -1133,7 +1164,7 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); CatalogEntry live = removing; @@ -1196,7 +1227,7 @@ TEST(CASRefCatalogRemoval, ExactDeletionRefusesChangedEntryAndAdmissionCannotCar CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( backend, layout, removing, ready_parent, 5, @@ -1243,7 +1274,7 @@ TEST(CASRefCatalogRemoval, FenceLossRemainsControlOutcomeWhenWinnerRemovesOrRepl CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); std::optional replacement; if (replace) @@ -1290,7 +1321,7 @@ TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesBeforeEraseCas) PutOutcome::Done); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] @@ -1322,7 +1353,7 @@ TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesAfterEraseResolut PutOutcome::Done); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); size_t authority_checks = 0; @@ -1359,7 +1390,7 @@ TEST(CASRefCatalogRemoval, CasPutExceptionPropagatesAfterMandatoryResolution) PutOutcome::Done); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); backend.armCasPutThrow(layout.refCatalogKey()); @@ -1432,12 +1463,12 @@ TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuil RefScanSummary ordinary_scan; ordinary_scan.parent_ref_lives.emplace(UInt128{1}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}}); ordinary_scan.parent_ref_lives.emplace(UInt128{3}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{3, 3}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{3, 3}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{3, 3}}}); ordinary_scan.parent_ref_lives.emplace(UInt128{4}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{4, 4}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{4, 4}}}); ordinary_scan.listed_lives = {UInt128{1}, UInt128{2}, UInt128{4}}; ordinary_scan.holds.emplace(UInt128{1}, RefHold{.offending_position = RefTxnId{1, 2}}); ordinary_scan.holds.emplace(UInt128{2}, RefHold{.offending_position = RefTxnId{2, 2}}); @@ -1449,7 +1480,7 @@ TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuil RefScanSummary rebuild_scan; rebuild_scan.parent_ref_lives.emplace(UInt128{1}, ordinary_scan.parent_ref_lives.at(UInt128{1})); rebuild_scan.parent_ref_lives.emplace(UInt128{5}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{5, 5}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{5, 5}}}); rebuild_scan.listed_lives = {UInt128{1}, UInt128{3}, UInt128{5}}; rebuild_scan.holds.emplace(UInt128{1}, RefHold{.offending_position = RefTxnId{1, 3}}); rebuild_scan.holds.emplace(UInt128{3}, RefHold{.offending_position = RefTxnId{3, 4}}); @@ -1541,7 +1572,7 @@ TEST(CASGCStuckRemoval, BoundaryAndAbsentVersusUnreadableMessagesAreExact) EXPECT_NE(absent->find("terminal has not folded"), String::npos); EXPECT_EQ(absent->find("/_log/"), String::npos) << "an absent terminal has no exact id to name"; - row.fold_state.coverage.classification = 4; + row.fold_state.coverage.classification = CoverageClass::Clamped; row.fold_state.coverage.hold = RefHold{ .reason = HoldReason::BodyUndecodable, .offending_position = RefTxnId{5, 6}}; @@ -1601,7 +1632,7 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) seal.generation = 1; seal.ref_lives.emplace(life_id, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .hold = RefHold{ .reason = HoldReason::BodyUndecodable, .offending_position = RefTxnId{5, 6}, @@ -1659,9 +1690,9 @@ TEST(CASGCRefWalkPlan, UnmatchedAdoptedParentLifeIsObservedWithoutEnteringThePla .catalog = catalog, .token = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary scan; scan.parent_ref_lives.emplace(current_life, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{2, 3}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{2, 3}}}); scan.parent_ref_lives.emplace(unmatched_life, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{9, 9}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{9, 9}}}); const uint64_t events_before = ProfileEvents::global_counters[ProfileEvents::CASGCUnmatchedAdoptedParentLives].load(); @@ -1697,7 +1728,7 @@ TEST(CASGCRefPlan, RoundInputOwnsObservationsAndSuccessorStateCannotChangePlan) RefScanSummary observations; observations.max_log_by_life.emplace(UInt128{2}, RefTxnId{2, 7}); observations.parent_ref_lives.emplace(UInt128{2}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{2, 3}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{2, 3}}}); const RefPlan plan = tests::buildRefWalkPlanForTest(observations, cut); diff --git a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp index 1ea4b8096e4d..5954f9930f5a 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp @@ -341,13 +341,13 @@ TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsR const RootNamespace ns{"srv1/reconciled"}; /// A dead predecessor's `Creating` entry: its mount lease carries the clean-farewell sentinel - /// (`min_active == UINT64_MAX`), one of `isCreatorFenceTerminal`'s three certificates of death. + /// (`min_active_build_sequence == UINT64_MAX`), one of `isCreatorFenceTerminal`'s three certificates of death. const CreatorFence dead_creator{.server_root_id = "dead-server", .writer_epoch = 3, .fence_generation = 1}; const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(0xbeef), .creator = dead_creator}; CasRefCatalog::casAdmitEntry(*backend, layout, 1, entry); setWatermarkMinActive(*backend, layout, "dead-server", /*writer_epoch=*/3, - /*min_active=*/std::numeric_limits::max()); + /*min_active_build_sequence=*/std::numeric_limits::max()); /// The production path resumes creation itself: reconciles the stale entry onto THIS mount's own /// fence and completes it to `Live`, over the SAME incarnation the dead creator minted. diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index b864587eff7d..7582d9670355 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -276,8 +276,8 @@ TEST(CASRefCheckpoint, CommittedThroughHasCanonicalExactWireEncoding) .committed_through = RefTxnId{9, 11}, .checkpoint_snapshot_id = RefTxnId{9, 10}, .last_epoch_seal = RefTxnId{8, 12}}; - const String expected = R"({"type":"cas_ref_ckpt","v":10} -{"le":"7","cte":"9","cts":"11","cse":"9","css":"10","lse":"8","lss":"12"} + const String expected = R"({"type":"cas_ref_ckpt","v":1} +{"life_epoch":"7","committed_epoch":"9","committed_seq":"11","snapshot_epoch":"9","snapshot_seq":"10","seal_epoch":"8","seal_seq":"12"} )"; EXPECT_EQ(encodeRefCkpt(ckpt), expected); @@ -296,7 +296,7 @@ TEST(CASFormatBattery, RefCkpt) [&] { return sealObject(FormatId::RefCkpt, encodeRefCkpt(ckpt)); }, [](std::string_view s) { decodeRefCkpt(std::string(openObject(FormatId::RefCkpt, s))); }, currentFormatHeader("cas_ref_ckpt") + - "{\"le\":\"7\",\"cte\":\"9\",\"cts\":\"11\",\"cse\":\"9\",\"css\":\"10\",\"lse\":\"8\",\"lss\":\"12\"}\n"}); + "{\"life_epoch\":\"7\",\"committed_epoch\":\"9\",\"committed_seq\":\"11\",\"snapshot_epoch\":\"9\",\"snapshot_seq\":\"10\",\"seal_epoch\":\"8\",\"seal_seq\":\"12\"}\n"}); } /// `last_epoch_seal` is chain evidence, not an arbitrary lower bound. It either names the frontier @@ -323,9 +323,9 @@ TEST(CASRefCheckpoint, CodecRejectsIncoherentCommittedFrontierAndSealEpochs) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefCkpt(unsealed_non_genesis); }); String malformed = encodeRefCkpt(valid); - const size_t cte = malformed.find(R"("cte":"8")"); - ASSERT_NE(cte, String::npos); - malformed.replace(cte, String{R"("cte":"8")"}.size(), R"("cte":"10")"); + const size_t committed_epoch = malformed.find(R"("committed_epoch":"8")"); + ASSERT_NE(committed_epoch, String::npos); + malformed.replace(committed_epoch, String{R"("committed_epoch":"8")"}.size(), R"("committed_epoch":"10")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(malformed); }); } @@ -347,13 +347,30 @@ TEST(CASRefCheckpoint, RejectsAnUnknownKey) expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { decodeRefCkpt(with_critical); }); } +/// Replacing the abbreviated key is a format cut, not an alias. Treating it as an optional partial +/// pair would make an old writer's checkpoint appear to have no committed frontier. +TEST(CASRefCheckpoint, RejectsOldCommittedEpochKeyRatherThanAliasingIt) +{ + /// The values are chosen so ALIASING would be harmless: the spliced `"cte":"9"` re-assigns the + /// epoch the object already carries, leaving a valid checkpoint. A reader that honoured the old + /// spelling would therefore DECODE, and this test fails; only the strict unknown-key rejection + /// makes it throw. Values under which aliasing corrupts the object would let the invariant + /// checker throw the same code and hide the alias. + String with_old_key = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{9}, + .committed_through = RefTxnId{9, 1}, + .checkpoint_snapshot_id = RefTxnId{9, 1}, + .last_epoch_seal = std::nullopt}); + with_old_key.replace(with_old_key.rfind('}'), 1, R"(,"cte":"9"})"); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(with_old_key); }); +} + /// A duplicate key has no single meaning, so it can never be resolved by a reader's preference. TEST(CASRefCheckpoint, RejectsADuplicateKey) { const String good = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{7}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); String duplicated = good; - duplicated.replace(duplicated.rfind('}'), 1, R"(,"le":"9"})"); + duplicated.replace(duplicated.rfind('}'), 1, R"(,"life_epoch":"9"})"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(duplicated); }); } @@ -377,13 +394,13 @@ TEST(CASRefCheckpoint, RejectsTruncation) const String empty_body = good.substr(0, good.find('\n') + 1) + "{}\n"; EXPECT_EQ(decodeRefCkpt(empty_body), RefCkpt{}); - const String half_pair = good.substr(0, good.find('\n') + 1) + R"({"le":"7","cse":"1"})" + "\n"; + const String half_pair = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","snapshot_epoch":"1"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(half_pair); }); - const String other_half = good.substr(0, good.find('\n') + 1) + R"({"le":"7","lss":"2"})" + "\n"; + const String other_half = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","seal_seq":"2"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(other_half); }); - const String frontier_half = good.substr(0, good.find('\n') + 1) + R"({"le":"7","cte":"1"})" + "\n"; + const String frontier_half = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","committed_epoch":"1"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(frontier_half); }); } @@ -414,9 +431,9 @@ TEST(CASRefCheckpoint, RejectsInvalidFieldsOnEncodeAndOnDecode) const String header = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{7}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); const String prefix = header.substr(0, header.find('\n') + 1); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(prefix + R"({"le":"0"})" + "\n"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(prefix + R"({"life_epoch":"0"})" + "\n"); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeRefCkpt(prefix + R"({"le":"7","cse":"1","css":"0"})" + "\n"); }); + [&] { decodeRefCkpt(prefix + R"({"life_epoch":"7","snapshot_epoch":"1","snapshot_seq":"0"})" + "\n"); }); } /// The registry row is part of the contract: Control/Strict decides how the decoder treats unknown @@ -1227,26 +1244,21 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) /// The BODY is checked as exact length plus a 128-bit SipHash of it -- not literally byte for byte, /// but any change that survives both is a 128-bit collision at a fixed length, which is the trade for /// keeping the assertion readable. It is a function of `{format generation, ns, id, ops, - /// chain_link}` only -- no incarnation reaches it. Generation 10 changed the shared format header; - /// the plaintext discriminator below removes only that change and pins every remaining byte to the - /// generation-9 fixture before accepting the new deterministic compressed size and hash. + /// chain_link}` only -- no incarnation reaches it. const auto got = backend->get(key); ASSERT_TRUE(got.has_value()) << "the birth chunk must be durable at its canonical key"; - String as_generation_9 = openObject(FormatId::RefLog, got->bytes); - const String generation_10_header = R"({"type":"cas_ref_log","v":10})"; - ASSERT_TRUE(as_generation_9.starts_with(generation_10_header)); - as_generation_9.replace(0, generation_10_header.size(), R"({"type":"cas_ref_log","v":9})"); - EXPECT_EQ(as_generation_9, R"({"type":"cas_ref_log","v":9} -{"ns":"test/golden@cas@","we":"1","rs":"1"} + const String plaintext = openObject(FormatId::RefLog, got->bytes); + EXPECT_EQ(plaintext, R"({"type":"cas_ref_log","v":1} +{"namespace":"test/golden@cas@","txn_epoch":"1","txn_seq":"1"} {"op":"namespace_birth"} -{"op":"owner_transition","nbk":"precommit","nrn":"gold_ref","nme":"1","nmb":"7","nmo":1} -{"op":"owner_transition","obk":"precommit","orn":"gold_ref","ome":"1","omb":"7","omo":1,"nbk":"committed","nrn":"gold_ref","nme":"1","nmb":"7","nmo":1} +{"op":"owner_transition","new_kind":"precommit","new_ref":"gold_ref","new_epoch":"1","new_build":"7","new_ord":1} +{"op":"owner_transition","old_kind":"precommit","old_ref":"gold_ref","old_epoch":"1","old_build":"7","old_ord":1,"new_kind":"committed","new_ref":"gold_ref","new_epoch":"1","new_build":"7","new_ord":1} {"n":3} -)") << "generation 10 must change only the self-describing header of this ref-log fixture"; - EXPECT_EQ(got->bytes.size(), 179u) << "the sealed ref-log body changed size"; +)") << "the sealed ref-log plaintext changed"; + EXPECT_EQ(got->bytes.size(), 206u) << "the sealed ref-log body changed size"; SipHash body_hash; body_hash.update(got->bytes.data(), got->bytes.size()); - EXPECT_EQ(getHexUIntLowercase(body_hash.get128()), "ada75a83638e933c98d731183a46b7b7") + EXPECT_EQ(getHexUIntLowercase(body_hash.get128()), "21c275ad44a6b47a4d6c389c0d71bb34") << "the sealed ref-log body changed content -- preparation must seal the same bytes it sealed " "before the extraction"; } diff --git a/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp b/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp index e50603d9258d..0ae7eac58160 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp @@ -41,7 +41,7 @@ /// so the fence would not fire on the very change it exists to catch. Only a real producer /// populates a real field. /// - TRANSACTIONS and WRITER EPOCHS enter as the DECIMAL WIDTH of the two id pairs. That is not -/// equality: `{cse=1,css=1}` and `{cse=1,css=10000}` differ by four bytes. It is `O(1)` because +/// equality: `{snapshot_epoch=1,snapshot_seq=1}` and `{snapshot_epoch=1,snapshot_seq=10000}` differ by four bytes. It is `O(1)` because /// the fields are `uint64_t` and so the width is ceilinged at twenty digits, which is a bound a /// test asserts on a constructed worst case -- `EncodedCkptSizeHasAConstantCeiling...` below. @@ -73,9 +73,10 @@ constexpr uint64_t U64_MAX = std::numeric_limits::max(); /// Constraint 15's bound, as a number: the encoded size of the WIDEST `_ckpt` this build can produce /// (all three fields present, every integer component at `UINT64_MAX`). Pinned as a literal so that -/// adding a field, or widening one, fails a test rather than quietly moving the bound. Generation 10 -/// added one byte to the shared format-version header (`9` became `10`); the scalar body is unchanged. -constexpr size_t CKPT_WORST_CASE_ENCODED_BYTES = 235; +/// adding a field, or widening one, fails a test rather than quietly moving the bound. The shared +/// format-version header is the single-digit `v:1` baseline; a future generation bump that widens it +/// moves this constant too. +constexpr size_t CKPT_WORST_CASE_ENCODED_BYTES = 296; /// The high-cardinality side of the size fence, in ONE transaction. Bounded above by the append lane's /// 5000-operation cap on a normal-class item (`publishCommittedOps` emits two ops per ref), and kept at diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index bc911e4ad6db..13437baf17a8 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -289,114 +289,6 @@ TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) EXPECT_EQ(state.getGreatestApplied(), (RefTxnId{kEpoch + 1, 1})); } -/// The format floor. A pool written before contiguous ref streams holds ref logs whose ids this build -/// would read as a corrupt (holed) chain, so opening it must fail closed at the pool metadata, naming -/// recreation as the migration -- CAS is pre-release and has no in-place migration path. -TEST(CASRefContiguousAlloc, OldPoolFormatIsRefusedNamingRecreation) -{ - PoolMeta pm; - pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; - pm.min_reader_generation = G_BUILD; - pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - - const String current = encodePoolMeta(pm); - EXPECT_NO_THROW(decodePoolMeta(current)); - - /// Rewrite the header-line generation to the last pre-contiguous one, exactly as an older build - /// would have stamped it. - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(kContiguousRefStreamsGeneration - 1); - const size_t at = current.find(from); - ASSERT_NE(at, String::npos); - String old_format = current; - old_format.replace(at, from.size(), to); - - try - { - decodePoolMeta(old_format); - FAIL() << "a pre-contiguous pool must not open"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find(fmt::format("CAS pool format {} predates generation-10 mount-attempt-identity floor", - kContiguousRefStreamsGeneration - 1)), String::npos) - << "the message must name the migration: " << e.message(); - } -} - -/// Generation 6 is a recreate-only physical-layout cut. A generation-5 pool has contiguous, -/// incarnation-qualified streams but still repeats the logical namespace in every key; accepting it -/// would silently run the generation-6 parsers over a different grammar. -TEST(CASRefContiguousAlloc, GenerationFiveNamespaceBearingPoolIsRefusedNamingRecreation) -{ - PoolMeta pm; - pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; - pm.min_reader_generation = G_BUILD; - pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - - const String current = encodePoolMeta(pm); - EXPECT_NO_THROW(decodePoolMeta(current)); - - /// Rewrite the header to the immediately preceding generation, which used - /// `cas/refs///...`. - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(kNamespaceLifeKeyedGeneration); - const size_t at = current.find(from); - ASSERT_NE(at, String::npos); - String old_format = current; - old_format.replace(at, from.size(), to); - ASSERT_EQ(kNamespaceLifeKeyedGeneration + 1, kOpaqueNamespaceLifeLayoutGeneration) - << "this test pins the immediately preceding namespace-bearing generation"; - - try - { - decodePoolMeta(old_format); - FAIL() << "a generation-5 namespace-bearing pool must not open"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find(fmt::format("CAS pool format {} predates generation-10 mount-attempt-identity floor", - kNamespaceLifeKeyedGeneration)), String::npos) - << "the message must name the migration: " << e.message(); - } -} - -/// Mutation caught: leaving the pool floor at generation 6 would admit a seal whose independent -/// name-keyed coverage and cleanup collections this build no longer has. Generation 7 is a -/// recreate-only grammar cut, so the immediately preceding generation must fail at pool open. -TEST(CASRefContiguousAlloc, GenerationSixSplitFoldSealPoolIsRefusedNamingRecreation) -{ - PoolMeta pm; - pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; - pm.min_reader_generation = G_BUILD; - pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - - const String current = encodePoolMeta(pm); - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":6"; - const size_t at = current.find(from); - ASSERT_NE(at, String::npos); - String old_format = current; - old_format.replace(at, from.size(), to); - - try - { - decodePoolMeta(old_format); - FAIL() << "a generation-6 split ref-life fold seal pool must not open"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find("CAS pool format 6 predates generation-10 mount-attempt-identity floor"), String::npos) - << "the message must name the recreate-only grammar cut: " << e.message(); - } -} - TEST(CASPoolMeta, GcShardsIsPersistedAndOverridesMismatchedReopenConfig) { InMemoryBackend backend; diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index cd9ce6c74291..c265618996ee 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -224,11 +224,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"2")"; + const String needle = R"("txn_seq":"2")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"1","!pss":"1")"); + tampered.insert(pos + needle.size(), R"(,"!prev_epoch":"1","!prev_seq":"1")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -255,8 +255,8 @@ TEST(CASRefEpochSealFormat, EncodeRejectsPrevEpochSealWithZeroRefSequence) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefLogTxn(txn); }); } -/// Decode-side splice: `prev_epoch_seal` present as only one of its two wire fields ("!pse" without -/// "!pss") -- a shape only reachable via corrupted bytes, since the encoder always writes both +/// Decode-side splice: `prev_epoch_seal` present as only one of its two wire fields ("!prev_epoch" without +/// "!prev_seq") -- a shape only reachable via corrupted bytes, since the encoder always writes both /// together. Boundary-plus-one for the additive-field decode contract (Constraint 7). TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) { @@ -267,7 +267,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) txn.ops.push_back(epochSealOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"(,"!pss":"9")"; + const String needle = R"(,"!prev_seq":"9")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; @@ -326,11 +326,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealSkippingImmediateEpochSpli txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"1")"; + const String needle = R"("txn_seq":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"3","!pss":"1")"); + tampered.insert(pos + needle.size(), R"(,"!prev_epoch":"3","!prev_seq":"1")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -346,11 +346,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealPointingAtSameOrFutureEpoc txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); /// valid: sequence 1, no prev_epoch_seal - const String needle = R"("rs":"1")"; + const String needle = R"("txn_seq":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"5","!pss":"1")"); /// self-pointer + tampered.insert(pos + needle.size(), R"(,"!prev_epoch":"5","!prev_seq":"1")"); /// self-pointer expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -421,13 +421,13 @@ TEST(CASRefEpochSealFormat, ContextualPassesThroughNonSequenceOneAtOrBelowLifeEp /// Criticality of the prev_epoch_seal wire fields (review finding M4) /// =================================================================================== -/// `!pse`/`!pss` are `!`-prefixed CRITICAL keys: `prev_epoch_seal` is INV-2 chain evidence, and a +/// `!prev_epoch`/`!prev_seq` are `!`-prefixed CRITICAL keys: `prev_epoch_seal` is INV-2 chain evidence, and a /// build that silently dropped it would still pass the structural grammar (absent field => no check) /// while losing the chain link. Proven here by splicing in a DIFFERENT, genuinely-unrecognized -/// `!`-key (simulating a future critical field this build predates) rather than `!pse`/`!pss` +/// `!`-key (simulating a future critical field this build predates) rather than `!prev_epoch`/`!prev_seq` /// themselves, which this build DOES recognize: `JsonObjectReader::skipUnknown` rejects any /// unrecognized `!`-prefixed key with `UNKNOWN_FORMAT_VERSION` (never a silent skip), so this pins -/// the general mechanism the meta-line reader relies on to keep `!pse`/`!pss` safe against a decoder +/// the general mechanism the meta-line reader relies on to keep `!prev_epoch`/`!prev_seq` safe against a decoder /// that doesn't (yet, or anymore) understand them. TEST(CASRefEpochSealFormat, DecodeRejectsUnknownCriticalKeyInMetaLine) { @@ -437,7 +437,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsUnknownCriticalKeyInMetaLine) txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"1")"; + const String needle = R"("txn_seq":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; @@ -486,8 +486,8 @@ TEST(CASRefEpochSealFormat, FormatBatteryEpochSeal) runFormatBattery({FormatId::RefLog, [txn] { return sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); }, [ns, id](std::string_view s) { decodeRefLogTxn(openObject(FormatId::RefLog, s), ns, id); }, - "{\"type\":\"cas_ref_log\",\"v\":10}\n" - "{\"ns\":\"ns\",\"we\":\"3\",\"rs\":\"1\",\"!pse\":\"2\",\"!pss\":\"9\"}\n" + "{\"type\":\"cas_ref_log\",\"v\":1}\n" + "{\"namespace\":\"ns\",\"txn_epoch\":\"3\",\"txn_seq\":\"1\",\"!prev_epoch\":\"2\",\"!prev_seq\":\"9\"}\n" "{\"op\":\"epoch_seal\"}\n" "{\"n\":1}\n"}); } diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index ed6ed34ae599..cef428ab1d87 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -6,6 +6,8 @@ #include #include +#include + /// v3 text codec tests for `cas_ref_log` (codecs-v3 phase 3). Split out of the retired /// `gtest_cas_ref_codecs.cpp` and re-pointed at the TEXT codec: the encoder-side validation tests are /// format-agnostic (they only assert `encodeRefLogTxn` throws) and carry over verbatim; the old @@ -140,6 +142,20 @@ TEST(CASRefCodec, OrderMatchesLexicalOrderOfRender) /// RefLogTxn: round trip /// =================================================================================== +/// Closed-set pin: the five `RefOpKind` words, walked through `magic_enum::enum_values`, which is +/// what proves the renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASRefCodec, ClosedSetPinsRefOpKindWords) +{ + EXPECT_EQ(refOpKindToWireWord(RefOpKind::NamespaceBirth), "namespace_birth"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::OwnerTransition), "owner_transition"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::SetPublishedAt), "set_published_at"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::RemoveNamespace), "remove_namespace"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::EpochSeal), "epoch_seal"); + for (const auto k : magic_enum::enum_values()) + EXPECT_EQ(refOpKindFromWireWord(refOpKindToWireWord(k)), k); +} + TEST(CASRefCodec, RoundTripNamespaceBirth) { RefLogTxn txn; @@ -185,9 +201,9 @@ TEST(CASRefCodec, RoundTripSetPublishedAt) EXPECT_EQ(decoded, txn); } -/// No-tolerance decode pin (codex round-2, finding 3): the `"pl"` (payload) field was removed from the +/// No-tolerance decode pin: the `"pl"` (payload) field was removed from the /// ref-op wire in stage-1 T12. Although the retired `set_payload` op WORD is already rejected by -/// `opKindFromWord`, the generic op-record reader reads all field keys before switching on kind, so a +/// `refOpKindFromWireWord`, the generic op-record reader reads all field keys before switching on kind, so a /// `"pl"` field paired with a still-recognized op word would otherwise be `skipUnknown`'d. It is a /// removed field, not a genuinely-unknown one: decoding an op record that still carries `"pl"` must FAIL /// with `CORRUPTED_DATA` naming the removed field. @@ -204,8 +220,8 @@ TEST(CASRefCodec, DecodeRejectsRemovedPayloadFieldInOpRecord) txn.ops.push_back(op); const String bytes = encodeRefLogTxn(txn); - /// Splice the retired `"pl"` field back into the op record, just before its `"ts"` field. - const String needle = ",\"ts\":"; + /// Splice the retired `"pl"` field back into the op record, just before its `"published_ms"` field. + const String needle = ",\"published_ms\":"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); const String tampered = bytes.substr(0, pos) + R"(,"pl":"deadbeef")" + bytes.substr(pos); @@ -297,7 +313,7 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) txn.ops.push_back(op); const String bytes = encodeRefLogTxn(txn); - const String old_group = R"(,"obk":"precommit","orn":"old","ome":"1","omb":"1","omo":1)"; + const String old_group = R"(,"old_kind":"precommit","old_ref":"old","old_epoch":"1","old_build":"1","old_ord":1)"; const auto old_group_pos = bytes.find(old_group); ASSERT_NE(old_group_pos, String::npos); String old_absent = bytes; @@ -307,14 +323,14 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) EXPECT_FALSE(without_old.ops[0].old_binding.has_value()); EXPECT_TRUE(without_old.ops[0].new_binding.has_value()); - const String old_ref = R"(,"orn":"old")"; + const String old_ref = R"(,"old_ref":"old")"; const auto old_ref_pos = bytes.find(old_ref); ASSERT_NE(old_ref_pos, String::npos); String incomplete_old = bytes; incomplete_old.erase(old_ref_pos, old_ref.size()); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_old, txn.ns, txn.txn_id); }); - const String new_group = R"(,"nbk":"committed","nrn":"new","nme":"1","nmb":"1","nmo":1)"; + const String new_group = R"(,"new_kind":"committed","new_ref":"new","new_epoch":"1","new_build":"1","new_ord":1)"; const auto new_group_pos = bytes.find(new_group); ASSERT_NE(new_group_pos, String::npos); String new_absent = bytes; @@ -324,7 +340,7 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) EXPECT_TRUE(without_new.ops[0].old_binding.has_value()); EXPECT_FALSE(without_new.ops[0].new_binding.has_value()); - const String new_ref = R"(,"nrn":"new")"; + const String new_ref = R"(,"new_ref":"new")"; const auto new_ref_pos = bytes.find(new_ref); ASSERT_NE(new_ref_pos, String::npos); String incomplete_new = bytes; @@ -332,6 +348,35 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_new, txn.ns, txn.txn_id); }); } +/// The anomaly diagnostic identifies an object found at a key it should not occupy by reading the +/// meta line's three identity fields. It reads them through the codec's own key constants, so this +/// test is what proves the reader did not quietly stop matching when those keys were renamed: with a +/// stale spelling the tolerant reader skips every real key and the peek answers nullopt on a +/// perfectly good ref-log. +TEST(CASRefCodec, PeekReadsTheMetaIdentityOfASealedRefLog) +{ + RefLogTxn txn; + txn.ns = "srv1/db/table@cas@"; + txn.txn_id = RefTxnId{4, 9}; + RefOp birth; + birth.kind = RefOpKind::NamespaceBirth; + txn.ops.push_back(birth); + + const auto peek = peekRefLogMeta(sealObject(FormatId::RefLog, encodeRefLogTxn(txn))); + ASSERT_TRUE(peek.has_value()) << "a well-formed ref-log must identify its own writer"; + EXPECT_EQ(peek->ns, "srv1/db/table@cas@"); + EXPECT_EQ(peek->writer_epoch, 4u); + EXPECT_EQ(peek->ref_sequence, 9u); +} + +/// The other half of its contract: it identifies a writer, it never certifies an object, so anything +/// it cannot read is `nullopt` rather than an exception escaping into the anomaly report. +TEST(CASRefCodec, PeekAnswersNulloptForBytesThatAreNotARefLog) +{ + EXPECT_FALSE(peekRefLogMeta("not a sealed cas object at all").has_value()); + EXPECT_FALSE(peekRefLogMeta(sealObject(FormatId::RefLog, "{\"type\":\"cas_ref_log\",\"v\":1}\n")).has_value()); +} + TEST(CASRefCodec, RoundTripMultipleOpsInOneTransaction) { RefLogTxn txn; @@ -833,7 +878,7 @@ TEST(CASFormatBattery, RefLog) [txn] { return sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); }, [ns, id](std::string_view s) { decodeRefLogTxn(openObject(FormatId::RefLog, s), ns, id); }, currentFormatHeader("cas_ref_log") + - "{\"ns\":\"ns\",\"we\":\"1\",\"rs\":\"1\"}\n" - "{\"op\":\"set_published_at\",\"rn\":\"all_1_1_0\",\"me\":\"1\",\"mb\":\"1\",\"mo\":1,\"ts\":42}\n" + "{\"namespace\":\"ns\",\"txn_epoch\":\"1\",\"txn_seq\":\"1\"}\n" + "{\"op\":\"set_published_at\",\"ref\":\"all_1_1_0\",\"epoch\":\"1\",\"build\":\"1\",\"ord\":1,\"published_ms\":42}\n" "{\"n\":1}\n"}); } diff --git a/src/Disks/tests/gtest_cas_ref_read_contract.cpp b/src/Disks/tests/gtest_cas_ref_read_contract.cpp index a86d2530d4e8..7917d59231e7 100644 --- a/src/Disks/tests/gtest_cas_ref_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ref_read_contract.cpp @@ -77,7 +77,7 @@ void deleteCatalogLife(Backend & backend, const Layout & layout, const Namespace CasFoldSeal parent; parent.ref_lives.emplace(life.incarnation, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); if (CasRefCatalog::deleteCompletedRemoving( backend, layout, *it, parent, 1, diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 2fea715879e2..1c532c5add83 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -65,7 +65,7 @@ TEST(CASRefSnapshotCodec, DecodeRequiresLifecycleField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String field = R"(,"lc":"live")"; + const String field = R"(,"lifecycle":"live")"; const size_t at = bytes.find(field); ASSERT_NE(at, String::npos); bytes.erase(at, field.size()); @@ -78,10 +78,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsTerminalLifecycleWord) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); - bytes.replace(at, live.size(), R"("lc":"removed")"); + bytes.replace(at, live.size(), R"("lifecycle":"removed")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); @@ -91,7 +91,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnEpochField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); bytes.replace(at, live.size(), live + R"(,"rte":"7")"); @@ -104,7 +104,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnSequenceField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); bytes.replace(at, live.size(), live + R"(,"rts":"9")"); @@ -117,7 +117,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnFieldPair) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); bytes.replace(at, live.size(), live + R"(,"rte":"7","rts":"9")"); @@ -142,8 +142,8 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) s.committed.push_back(c); const String bytes = encodeRefTableSnapshot(s); - /// Splice the retired `"pl"` field back into the committed record, just before its `"ts"` field. - const String needle = ",\"ts\":"; + /// Splice the retired `"pl"` field back into the committed record, just before its `"published_ms"` field. + const String needle = ",\"published_ms\":"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); const String tampered = bytes.substr(0, pos) + R"(,"pl":"deadbeef")" + bytes.substr(pos); @@ -152,6 +152,30 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) [&] { decodeRefTableSnapshot(tampered, s.ns, s.snapshot_id); }); } +/// Row kinds are the owner-kind vocabulary, so an unknown kind word must fail closed at the word +/// table rather than being silently skipped as an unrecognized row -- a skipped row would lose a ref +/// from a snapshot the reader still reports as complete. +TEST(CASRefSnapshotCodec, DecodeRejectsUnknownRowKindWord) +{ + RefTableSnapshot s; + s.ns = "ns"; + s.snapshot_id = RefTxnId{1, 1}; + RefCommittedRow c; + c.ref_name = "all_1_1_0"; + c.manifest_ref = manifestRef(5, 10, 1); + c.published_at_ms = 1717000000000ULL; + s.committed.push_back(c); + + const String bytes = encodeRefTableSnapshot(s); + const String needle = "\"kind\":\"committed\""; + const auto pos = bytes.find(needle); + ASSERT_NE(pos, String::npos); + const String tampered = bytes.substr(0, pos) + "\"kind\":\"archived\"" + bytes.substr(pos + needle.size()); + + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { decodeRefTableSnapshot(tampered, s.ns, s.snapshot_id); }); +} + TEST(CASRefSnapshotCodec, RoundTripLiveEmpty) { RefTableSnapshot s; @@ -207,7 +231,7 @@ TEST(CASRefSnapshotFormat, MaximalRefSequenceRoundTripsAsADecimalString) const String text = encodeRefTableSnapshot(m); const RefTableSnapshot back = decodeRefTableSnapshot(text, m.ns, m.snapshot_id); EXPECT_EQ(back.snapshot_id.ref_sequence, std::numeric_limits::max()); - EXPECT_NE(text.find("\"rs\":\"18446744073709551615\""), String::npos); + EXPECT_NE(text.find("\"snapshot_seq\":\"18446744073709551615\""), String::npos); } /// =================================================================================== @@ -430,9 +454,9 @@ TEST(CASFormatBattery, RefSnapshot) [s] { return sealObject(FormatId::RefSnapshot, encodeRefTableSnapshot(s)); }, [ns, id](std::string_view d) { decodeRefTableSnapshot(openObject(FormatId::RefSnapshot, d), ns, id); }, currentFormatHeader("cas_ref_snap") + - "{\"ns\":\"srv1/db/table@cas@\",\"we\":\"5\",\"rs\":\"200\",\"lc\":\"live\"}\n" - "{\"k\":\"c\",\"rn\":\"all_1_1_0\",\"me\":\"5\",\"mb\":\"10\",\"mo\":1,\"ts\":1717000000000}\n" - "{\"k\":\"c\",\"rn\":\"all_2_2_0\",\"me\":\"5\",\"mb\":\"11\",\"mo\":1,\"ts\":1717000000001}\n" - "{\"k\":\"p\",\"rn\":\"all_3_3_0\",\"me\":\"5\",\"mb\":\"12\",\"mo\":1}\n" + "{\"namespace\":\"srv1/db/table@cas@\",\"snapshot_epoch\":\"5\",\"snapshot_seq\":\"200\",\"lifecycle\":\"live\"}\n" + "{\"kind\":\"committed\",\"ref\":\"all_1_1_0\",\"epoch\":\"5\",\"build\":\"10\",\"ord\":1,\"published_ms\":1717000000000}\n" + "{\"kind\":\"committed\",\"ref\":\"all_2_2_0\",\"epoch\":\"5\",\"build\":\"11\",\"ord\":1,\"published_ms\":1717000000001}\n" + "{\"kind\":\"precommit\",\"ref\":\"all_3_3_0\",\"epoch\":\"5\",\"build\":\"12\",\"ord\":1}\n" "{\"n\":3}\n"}); } diff --git a/src/Disks/tests/gtest_cas_server_root_format.cpp b/src/Disks/tests/gtest_cas_server_root_format.cpp index e302a4d81573..bf04753c9b9f 100644 --- a/src/Disks/tests/gtest_cas_server_root_format.cpp +++ b/src/Disks/tests/gtest_cas_server_root_format.cpp @@ -18,7 +18,7 @@ TEST(CASFormatBattery, Owner) OwnerObject o; o.server_uuid = hexToU128("0123456789abcdeffedcba9876543210"); const String golden = currentFormatHeader("cas_owner") + - "{\"su\":\"0123456789abcdeffedcba9876543210\"}\n"; + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\"}\n"; EXPECT_EQ(encodeOwner(o), golden); EXPECT_FALSE(decodeOwner(golden).retired_at_ms.has_value()); runFormatBattery({FormatId::Owner, @@ -34,7 +34,7 @@ TEST(CASOwnerFormat, RetiredAtRoundTrip) o.retired_at_ms = 1752537600000ULL; EXPECT_EQ(encodeOwner(o), currentFormatHeader("cas_owner") - + "{\"su\":\"0123456789abcdeffedcba9876543210\",\"rt\":1752537600000}\n"); + + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"retired_at_ms\":1752537600000}\n"); const OwnerObject back = decodeOwner(encodeOwner(o)); EXPECT_EQ(back.server_uuid, o.server_uuid); EXPECT_EQ(back.retired_at_ms, o.retired_at_ms); @@ -49,7 +49,7 @@ TEST(CASFormatBattery, ServerEpoch) runFormatBattery({FormatId::ServerEpoch, [&] { return sealObject(FormatId::ServerEpoch, encodeServerEpoch(e)); }, [](std::string_view s) { decodeServerEpoch(std::string(openObject(FormatId::ServerEpoch, s))); }, - currentFormatHeader("cas_epoch") + "{\"nwe\":\"7\"}\n"}); + currentFormatHeader("cas_epoch") + "{\"next_writer_epoch\":\"7\"}\n"}); } CAS_BATTERY_COVERS(MountLease); @@ -63,8 +63,8 @@ TEST(CASFormatBattery, MountLease) [&] { return sealObject(FormatId::MountLease, encodeMountLease(m)); }, [](std::string_view s) { decodeMountLease(std::string(openObject(FormatId::MountLease, s))); }, currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"host-1\",\"pid\":4242," - "\"sat\":1752537600000,\"seq\":\"5\",\"eat\":1752537630000,\"ma\":\"9\",\"fen\":false," + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"host-1\",\"pid\":4242," + "\"started_at_ms\":1752537600000,\"seq\":\"5\",\"expires_at_ms\":1752537630000,\"min_active_build_sequence\":\"9\",\"gc_fenced\":false," "\"write_attempt_id\":\"00112233445566778899aabbccddeeff\"}\n"}); } @@ -74,7 +74,7 @@ TEST(CASMountLeaseFormat, FarewellSentinelAndFencedSurvive) 1, 5, 2, std::numeric_limits::max(), true, hexToU128("00112233445566778899aabbccddeeff")}; const MountLease back = decodeMountLease(encodeMountLease(m)); - EXPECT_EQ(back.min_active, std::numeric_limits::max()); + EXPECT_EQ(back.min_active_build_sequence, std::numeric_limits::max()); EXPECT_TRUE(back.gc_fenced); EXPECT_EQ(back.hostname, "h"); EXPECT_EQ(back.writer_epoch, 7u); @@ -93,8 +93,8 @@ TEST(CASMountLeaseFormat, WriteAttemptIdIsRequiredAndCanonical) EXPECT_EQ(decodeMountLease(encoded).write_attempt_id, m.write_attempt_id); const String without_attempt_id = currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"\",\"pid\":0," - "\"sat\":0,\"seq\":\"0\",\"eat\":0,\"ma\":\"0\",\"fen\":false}\n"; + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"\",\"pid\":0," + "\"started_at_ms\":0,\"seq\":\"0\",\"expires_at_ms\":0,\"min_active_build_sequence\":\"0\",\"gc_fenced\":false}\n"; try { decodeMountLease(without_attempt_id); @@ -109,8 +109,8 @@ TEST(CASMountLeaseFormat, WriteAttemptIdIsRequiredAndCanonical) TEST(CASMountLeaseFormat, ZeroWriteAttemptIdIsRejected) { const String data = currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"\",\"pid\":0," - "\"sat\":0,\"seq\":\"0\",\"eat\":0,\"ma\":\"0\",\"fen\":false," + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"\",\"pid\":0," + "\"started_at_ms\":0,\"seq\":\"0\",\"expires_at_ms\":0,\"min_active_build_sequence\":\"0\",\"gc_fenced\":false," "\"write_attempt_id\":\"00000000000000000000000000000000\"}\n"; try { @@ -138,11 +138,18 @@ TEST(CASMountLeaseFormat, UnknownFieldsRemainTolerated) TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) { - const String header = "{\"type\":\"cas_mount_lease\",\"v\":3}\n"; - const String fields = "\"hn\":\"host-1\",\"pid\":4242,\"sat\":1752537600000," - "\"seq\":\"5\",\"eat\":1752537630000,\"ma\":\"9\",\"fen\":false}"; - - const auto expectCorrupted = [](const String & data) + /// Each arm drops exactly ONE identity and keeps the other two, and each asserts the message that + /// names the dropped one. A body missing two of them would satisfy whichever clause runs first, so + /// a shared fixture and a shared message together would let two of the three checks be deleted + /// with this test still green. + const String header = "{\"type\":\"cas_mount_lease\",\"v\":1}\n"; + const String uuid = R"("server_uuid":"0123456789abcdeffedcba9876543210",)"; + const String epoch = R"("writer_epoch":"7",)"; + const String attempt = R"("write_attempt_id":"00112233445566778899aabbccddeeff",)"; + const String rest = "\"hostname\":\"host-1\",\"pid\":4242,\"started_at_ms\":1752537600000," + "\"seq\":\"5\",\"expires_at_ms\":1752537630000,\"min_active_build_sequence\":\"9\",\"gc_fenced\":false}"; + + const auto expectMessage = [](const String & data, std::string_view expected) { try { @@ -152,9 +159,11 @@ TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) catch (const DB::Exception & e) { EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), expected); } }; - expectCorrupted(header + R"({"we":"7",)" + fields + "\n"); - expectCorrupted(header + R"({"su":"0123456789abcdeffedcba9876543210",)" + fields + "\n"); + expectMessage(header + "{" + epoch + attempt + rest + "\n", "CAS mount-lease: missing server_uuid"); + expectMessage(header + "{" + uuid + attempt + rest + "\n", "CAS mount-lease: missing writer_epoch"); + expectMessage(header + "{" + uuid + epoch + rest + "\n", "CAS mount-lease: missing or zero write_attempt_id"); } diff --git a/src/Disks/tests/gtest_cas_shutdown_context.cpp b/src/Disks/tests/gtest_cas_shutdown_context.cpp index 6ceb58b1188b..5884d8b47860 100644 --- a/src/Disks/tests/gtest_cas_shutdown_context.cpp +++ b/src/Disks/tests/gtest_cas_shutdown_context.cpp @@ -90,7 +90,7 @@ void emitTestEvent(DB::ContentAddressedMetadataStorage & storage) /// successor skip the observation window, so a phase-2 failure must leave it absent. const auto mount = backend->get(Layout(config.pool_prefix).mountKey(config.server_root_id)); const bool clean_release = mount - && decodeMountLease(mount->bytes).min_active == std::numeric_limits::max(); + && decodeMountLease(mount->bytes).min_active_build_sequence == std::numeric_limits::max(); const bool marker_must_be_absent = phase == 2; std::_Exit(marker_must_be_absent && clean_release ? 1 : 0); } diff --git a/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp b/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp index 4c0000b46236..71f936c7a3c7 100644 --- a/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp +++ b/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp @@ -59,9 +59,9 @@ struct OrphanFixture /// exercising the independent sweep-deletion premise. casAdmitRecoverableEntry(*backend, store->layout(), ns); writeManifestRaw(*backend, store->layout(), ns, orphan, {blobEntryFor("a", DB::UInt128(1))}); - /// min_active 6 > build_sequence 5: the durable watermark fact makes the prefix ELIGIBLE, which + /// min_active_build_sequence 6 > build_sequence 5: the durable watermark fact makes the prefix ELIGIBLE, which /// is the half the premise sits on top of. - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); } String orphanKey() const { return store->layout().manifestKey(ManifestId{ns, orphan}); } @@ -81,7 +81,7 @@ struct UndecodableOrphanFixture { store = openPoolForTest(backend); casAdmitRecoverableEntry(*backend, store->layout(), ns); - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); } String orphanKey() const { return store->layout().manifestKey(ManifestId{ns, orphan}); } @@ -179,7 +179,7 @@ TEST(CASSweepDeletionPremise, AnUnconsumedTailRemovalRetainsItsTarget) NamespaceFoldView view; RefCoverage cov; - cov.classification = 2; + cov.classification = CoverageClass::Folded; cov.last_folded_ref_id = RefTxnId{kBuildEpoch + 1, 1}; /// rule (1) satisfied view.coverage = cov; view.tail_removal_targets.insert(key); @@ -318,7 +318,7 @@ TEST(CASSweepDeletionPremise, DistinctRetainReasonsLandInDistinctCounters) .retry_count = 1, .next_retry_round = 4}; seedFoldCursorForTest(*backend, layout, ns_b, RefTxnId{kBuildEpoch + 1, 8}, hold); - setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget*/100, /*delete_budget*/10); @@ -403,9 +403,9 @@ TEST(CASSweepDeletionPremise, RecoveryWorkBudgetRetainsAndConvergesWithoutWedgin /// takes the fresh-`_ckpt` `putIfAbsent` path instead of `advanceRecoverableCkptForRawFixture`'s /// monotonic-advance-from-existing-value path (which throws on a null `committed_through`). casAdmitEntry(*backend, layout, ns); - setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active*/1000); + setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active_build_sequence*/1000); - /// Six orphan candidates, all eligible (build_sequence << min_active), none owned by any ref. + /// Six orphan candidates, all eligible (build_sequence << min_active_build_sequence), none owned by any ref. constexpr int kCandidates = 6; for (int i = 1; i <= kCandidates; ++i) writeManifestRaw(*backend, layout, ns, ref(i, 1), @@ -484,7 +484,7 @@ TEST(CASSweepDeletionPremise, NamespaceWorkBudgetCapsDistinctViewsPerPage) /// at all (`_ckpt.committed_through` unset), so absent the namespace cap BOTH would delete. seedFoldCursorForTest(*backend, layout, ns_a, RefTxnId{kBuildEpoch + 1, 1}); seedFoldCursorForTest(*backend, layout, ns_b, RefTxnId{kBuildEpoch + 1, 1}); - setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); GcRoundWorkBudget budget; budget.max_sweep_namespaces = 1; diff --git a/src/Disks/tests/gtest_cas_text_format.cpp b/src/Disks/tests/gtest_cas_text_format.cpp index 625271435b24..c300e522b739 100644 --- a/src/Disks/tests/gtest_cas_text_format.cpp +++ b/src/Disks/tests/gtest_cas_text_format.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include using namespace DB::Cas; @@ -140,6 +141,28 @@ TEST(CASJsonVocab, WriteAndReadBack) EXPECT_FALSE(r.nextKey(key)); } +TEST(CASJsonVocab, WordArrayFieldAndReaderRejectInvalidValues) +{ + CasJsonWriter out; + bool first = true; + const std::array words{"ch128", "sha256"}; + writeWordArrayField(out, WireKey{"algos_used"}, words, first); + closeObject(out, first); + EXPECT_EQ(std::move(out).take(), "{\"algos_used\":[\"ch128\",\"sha256\"]}"); + + const auto read = [](std::string_view text) + { + DB::ReadBufferFromMemory in(text.data(), text.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "test"); + String key; + EXPECT_TRUE(r.nextKey(key)); + return r.readStringArray(); + }; + EXPECT_EQ(read(R"({"algos_used":["ch128","sha256"]})"), (std::vector{"ch128", "sha256"})); + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { read(R"({"algos_used":"ch128"})"); }); + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { read(R"({"algos_used":["ch128",1]})"); }); +} + TEST(CASJsonVocab, FailClosedRules) { auto reader = [](std::string_view text, KeyStrictness s, auto && consume) @@ -180,12 +203,12 @@ TEST(CASJsonVocab, FailClosedRules) r.nextKey(k); }); }); /// bad hex width / junk in u64 string - expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"h":"0102"})", KeyStrictness::Tolerant, [](auto & r) + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"digest":"0102"})", KeyStrictness::Tolerant, [](auto & r) { String k; r.nextKey(k); r.readHex128(); }); }); - expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"s":"12x"})", KeyStrictness::Tolerant, [](auto & r) + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"u64_string_field":"12x"})", KeyStrictness::Tolerant, [](auto & r) { String k; r.nextKey(k); r.readU64String(); @@ -213,9 +236,9 @@ TEST(CASTextHeader, WriteExpectSniffGate) EXPECT_FALSE(sniffHeaderLine("PAR1 not a cas object").has_value()); /// wrong type -> CORRUPTED_DATA; future v -> UNKNOWN_FORMAT_VERSION - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String wrong = "{\"type\":\"cas_owner\",\"v\":3}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the type mismatch is + /// what has to fail here. + const String wrong = "{\"type\":\"cas_owner\",\"v\":1}\n"; DB::ReadBufferFromMemory in2(wrong.data(), wrong.size()); expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { expectHeaderLine(in2, FormatId::PoolMeta); }); const String future = fmt::format("{{\"type\":\"cas_pool_meta\",\"v\":{}}}\n", currentCompatibilityVersion() + 1); @@ -251,19 +274,16 @@ TEST(CASZstdArm, SealOpenPolicyAndCaps) { /// Always types compress regardless of size (no threshold — the .zst key must be /// constructible without knowing the body); a raw body is still readable (repair path). - /// `v:3` here is NOT the "any version <= G_BUILD passes" case the other negative bodies rely on: - /// `cas_ref_snap`'s own `changePoints` floor is generation 4, so a generation-3 ref snapshot is not - /// readable by this build in principle. It passes the header gate only because nothing consults - /// `changePoints` at decode time yet -- the gate is `v > G_BUILD` alone. Once a per-class floor is - /// wired in, this literal must move to `G_BUILD`; the test's subject is the truncated BODY, not the - /// version. - const String small = "{\"type\":\"cas_ref_snap\",\"v\":3}\n{}\n"; + /// `sealObject`/`openObject` are the storage-wrapper layer and never invoke the version gate + /// (that happens at decode, e.g. `decodeRefSnapshot`'s `expectHeaderLine`), so `v:1` here is just + /// the baseline header -- the test's subject is the compression arm, not the version. + const String small = "{\"type\":\"cas_ref_snap\",\"v\":1}\n{}\n"; const String sealed_small = sealObject(FormatId::RefSnapshot, small); ASSERT_TRUE(looksZstd(sealed_small)); EXPECT_EQ(openObject(FormatId::RefSnapshot, sealed_small), small); EXPECT_EQ(openObject(FormatId::RefSnapshot, small), small); - String big = "{\"type\":\"cas_ref_snap\",\"v\":3}\n{\"pad\":\""; + String big = "{\"type\":\"cas_ref_snap\",\"v\":1}\n{\"pad\":\""; big += String(8192, 'a'); big += "\"}\n"; const String sealed = sealObject(FormatId::RefSnapshot, big); diff --git a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp index 522b7a738e68..2f15a9d9df95 100644 --- a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp +++ b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp @@ -75,7 +75,7 @@ ManifestId publishPart2( /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index 207ad24a295f..93385f38177a 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -1,10 +1,13 @@ #include #include +#include #include #include #include #include +#include + using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } @@ -44,6 +47,23 @@ TEST(CASWireVocab, EnumTablesPinTheCurrentWords) EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::XXH3_128, "t"), "xxh3"); EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::Sha256, "t"), "sha256"); EXPECT_EQ(kObjectKindWords.toWord(ObjectKind::Blob, "t"), "blob"); + EXPECT_EQ(refOwnerKindToWord(RefOwnerKind::Committed), "committed"); + EXPECT_EQ(refOwnerKindToWord(RefOwnerKind::Precommit), "precommit"); +} + +/// Every enum wire table's closed set, walked through `magic_enum::enum_values` rather than a +/// hand-copied list -- a future enumerator the encoder can construct but no table entry covers +/// would otherwise round-trip silently through the untested value. +TEST(CASWireVocab, ClosedSetsRoundTripEveryEnumeratorExhaustively) +{ + for (const auto t : magic_enum::enum_values()) + EXPECT_EQ(tokenTypeFromWord(tokenTypeToWord(t), "t"), t); + for (const auto k : magic_enum::enum_values()) + EXPECT_EQ(objectKindFromWord(objectKindToWord(k), "k"), k); + for (const auto a : magic_enum::enum_values()) + EXPECT_EQ(blobHashAlgoFromWord(blobHashAlgoName(a), "a"), a); + for (const auto k : magic_enum::enum_values()) + EXPECT_EQ(refOwnerKindFromWord(refOwnerKindToWord(k), "k"), k); } TEST(CASWireVocab, EnumWordsRoundTrip) @@ -67,7 +87,7 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) closeObject(out, first); const String rendered = std::move(out).take(); EXPECT_EQ(rendered, - R"({"tt":"etag","tv":"etag-abc\"x","ha":"ch128","h":"00112233445566778899aabbccddeeff"})"); + R"({"token_type":"etag","token":"etag-abc\"x","algo":"ch128","digest":"00112233445566778899aabbccddeeff"})"); DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); @@ -78,10 +98,10 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) TokenType tt{}; while (r.nextKey(key)) { - if (key == "tt") tt = tokenTypeFromWord(r.readString(), "t"); - else if (key == "tv") tv = r.readString(); - else if (key == "ha") ha = r.readString(); - else if (key == "h") h = r.readString(); + if (key == "token_type") tt = tokenTypeFromWord(r.readString(), "t"); + else if (key == "token") tv = r.readString(); + else if (key == "algo") ha = r.readString(); + else if (key == "digest") h = r.readString(); else r.skipUnknown(key); } EXPECT_EQ(tt, TokenType::ETag); @@ -97,13 +117,13 @@ TEST(CASWireVocab, ManifestRefBundleWritesTheOldPrefixedKeys) bool first = true; writeManifestRefFields(w, first, kOldManifestRefKeys, ManifestRef{1, 2, 3}); w.closeObject(first); - EXPECT_EQ(std::move(w).take(), R"({"ome":"1","omb":"2","omo":3})"); + EXPECT_EQ(std::move(w).take(), R"({"old_epoch":"1","old_build":"2","old_ord":3})"); } TEST(CASWireVocab, MatchAndBuildRoundTripsABlobRef) { using namespace DB::Cas; - const String rendered = R"({"ha":"ch128","h":"00112233445566778899aabbccddeeff"})"; + const String rendered = R"({"algo":"ch128","digest":"00112233445566778899aabbccddeeff"})"; DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); BlobRefFields fields; @@ -143,10 +163,10 @@ TEST(CASWireVocab, BlobRefBuildFailsClosedOnRightWidthNonHexDigest) TEST(CASWireVocab, MatchManifestRefFieldsAndBuildRefRoundTripInAnyKeyOrder) { using namespace DB::Cas; - /// Fed out of writer order (mo, me, mb) to pin key-order independence. `me`/`mb` are quoted - /// decimal strings and `mo` is a bare number -- a swapped read primitive between the two shapes + /// Fed out of writer order (ord, epoch, build) to pin key-order independence. `epoch`/`build` are quoted + /// decimal strings and `ord` is a bare number -- a swapped read primitive between the two shapes /// would fail to parse this literal. - const String rendered = R"({"mo":3,"me":"7","mb":"9"})"; + const String rendered = R"({"ord":3,"epoch":"7","build":"9"})"; DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); ManifestRefFields fields; @@ -168,10 +188,10 @@ TEST(CASWireVocab, ManifestRefFieldsBuildRefFailsClosedOnHalfAGroup) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { fields.buildRef("t", "ctx"); }); } -TEST(CASWireVocab, MatchTokenFieldsConsumesTtTvAndLeavesUnrelatedKeyUnmatched) +TEST(CASWireVocab, MatchTokenFieldsConsumesSemanticKeysAndLeavesUnrelatedKeyUnmatched) { using namespace DB::Cas; - const String rendered = R"({"tt":"etag","tv":"abc","zz":1})"; + const String rendered = R"({"token_type":"etag","token":"abc","zz":1})"; DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); TokenFields fields; @@ -190,3 +210,49 @@ TEST(CASWireVocab, MatchTokenFieldsConsumesTtTvAndLeavesUnrelatedKeyUnmatched) EXPECT_EQ(*fields.value, "abc"); EXPECT_TRUE(saw_unmatched); } + +TEST(CASWireVocab, TokenFieldsBuildsInAnyKeyOrderAndRequiresBothFields) +{ + const String rendered = R"({"token":"abc","token_type":"etag"})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + TokenFields fields; + String key; + while (r.nextKey(key)) + { + if (matchTokenFields(key, r, fields)) + continue; + r.skipUnknown(key); + } + EXPECT_EQ(fields.build("t"), (Token{"abc", TokenType::ETag})); + + TokenFields only_type; + only_type.type_word = "etag"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { only_type.build("t"); }); +} + +TEST(CASWireVocab, OldManifestEpochKeyDoesNotAliasTheSemanticKey) +{ + const String rendered = R"({"me":"1","build":"2","ord":3})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + ManifestRefFields fields; + String key; + while (r.nextKey(key)) + { + if (matchManifestRefFields(key, r, kBareManifestRefKeys, fields)) + continue; + r.skipUnknown(key); + } + + try + { + fields.buildRef("RefTableSnapshot", "committed"); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), "CAS RefTableSnapshot: committed manifest_ref missing epoch/build/ord"); + } +} diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index 662c0daa165e..5539d58dda80 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -354,7 +354,7 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem const auto mount = backend->get(mount_key); ASSERT_TRUE(mount.has_value()); - EXPECT_NE(decodeMountLease(mount->bytes).min_active, std::numeric_limits::max()) + EXPECT_NE(decodeMountLease(mount->bytes).min_active_build_sequence, std::numeric_limits::max()) << "a live writer-cleanup duty forbids the clean-release certificate"; uint64_t fake_boot = 0; diff --git a/src/Storages/System/StorageSystemContentAddressedMounts.cpp b/src/Storages/System/StorageSystemContentAddressedMounts.cpp index 2a1b82a01ee7..e856acad2286 100644 --- a/src/Storages/System/StorageSystemContentAddressedMounts.cpp +++ b/src/Storages/System/StorageSystemContentAddressedMounts.cpp @@ -187,7 +187,7 @@ Pipe StorageSystemContentAddressedMounts::read( col_seq->insert(m.lease.seq); assert_cast(*col_started).insertValue(static_cast(m.lease.started_at_ms)); assert_cast(*col_expires).insertValue(static_cast(m.lease.expires_at_ms)); - col_min_active->insert(m.lease.min_active); + col_min_active->insert(m.lease.min_active_build_sequence); col_fenced->insert(static_cast(m.lease.gc_fenced)); col_state->insert(m.state); diff --git a/tests/integration/test_cas_gc_sharded/test.py b/tests/integration/test_cas_gc_sharded/test.py index a7c56edf2705..fca5d64eea51 100644 --- a/tests/integration/test_cas_gc_sharded/test.py +++ b/tests/integration/test_cas_gc_sharded/test.py @@ -122,13 +122,13 @@ def get_rustfs_object(key): # `gc/state`'s wire format is a plain JSON-like text object (CasGcStateFormat.cpp), not a binary -# blob: the two fields this test needs are literally spelled `"sg":""` (snap_generation) -# and `"sa":""` (snap_attempt) in the object bytes, so a direct regex read is exact without +# blob: the two fields this test needs are literally spelled `"snap_generation":""` +# and `"snap_attempt":""` in the object bytes, so a direct regex read is exact without # needing the C++ decoder. This mirrors the production reader that resolves "the adopted seal" # (Gc/CasOrphanManifestSweep.cpp): read gc/state, take (snap_generation, snap_attempt), then look up # that exact fold seal -- the only two-hop lookup that names one authoritative adopted pair. -_SNAP_GENERATION_RE = re.compile(r'"sg":"(\d+)"') -_SNAP_ATTEMPT_RE = re.compile(r'"sa":"(\d+)"') +_SNAP_GENERATION_RE = re.compile(r'"snap_generation":"(\d+)"') +_SNAP_ATTEMPT_RE = re.compile(r'"snap_attempt":"(\d+)"') def read_adopted_generation_and_attempt(): @@ -144,7 +144,13 @@ def read_adopted_generation_and_attempt(): sg_match = _SNAP_GENERATION_RE.search(text) sa_match = _SNAP_ATTEMPT_RE.search(text) if not sg_match or not sa_match: - return None + # A `gc/state` that exists but does not spell both fields is a wire-format change this + # reader has not followed, NOT "nothing adopted yet" -- say so instead of returning the + # absent-sentinel, which would poll to a timeout and blame the server. + raise AssertionError( + "gc/state exists but neither snap_generation nor snap_attempt could be read from it; " + "the object's key spelling changed and this regex reader is stale: " + text[:400] + ) generation = int(sg_match.group(1)) if generation == 0: return None diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 74294cf95fa5..7d63be5769db 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -784,7 +784,7 @@ def handle_control(path, method, query): return _no_such_key(meta_key) text = entry["body"].decode("utf-8", "strict") rewritten, replacements = re.subn( - r'"st":"clean","cr":"[0-9]+"', '"st":"condemned","cr":"1"', text, count=1 + r'"state":"clean","condemn_round":"[0-9]+"', '"state":"condemned","condemn_round":"1"', text, count=1 ) if replacements != 1: return _bad_request("blob metadata is not Clean: " + meta_key) diff --git a/tests/integration/test_cas_gcs/test.py b/tests/integration/test_cas_gcs/test.py index 285da574ac17..7df52e4dbbd3 100644 --- a/tests/integration/test_cas_gcs/test.py +++ b/tests/integration/test_cas_gcs/test.py @@ -469,7 +469,7 @@ def test_blob_publication_request_budget_and_default_mode(disk): for r in meta if r["method"] == "PUT" and r["headers"].get("x-goog-if-generation-match") == "0" - and '"st":"clean"' in r["request_body"] + and '"state":"clean"' in r["request_body"] ] assert len(creates) == 1, (key, meta) @@ -524,7 +524,7 @@ def test_blob_publication_request_budget_and_default_mode(disk): for r in _meta_requests(retry, target) if r["method"] == "PUT" and r["headers"].get("x-goog-if-generation-match", "0") != "0" - and '"st":"clean"' in r["request_body"] + and '"state":"clean"' in r["request_body"] ] assert len(clean_cas) == 1, clean_cas diff --git a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh index ef5e2377cc21..1cc521a210ca 100755 --- a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh +++ b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh @@ -4,17 +4,17 @@ # own unique local-object-storage pool and a per-run CAS disk name, so unlike 04290_cas_no_leftovers # it does not need no-parallel. -# FINDING #2 regression test: `DROP TABLE ... SYNC` on a content-addressed MergeTree used to leave the +# Regression test: `DROP TABLE ... SYNC` on a content-addressed MergeTree used to leave the # table's CAS ref-catalog row `live` forever whenever `DirShape::TableDir`'s `existsDirectory` observed # zero committed refs -- an empty table, or one whose last part was just removed. `dropAllData`'s own # `existsDirectory` precheck skipped `removeRecursive`/`dropNamespace` entirely in that shape, so the # SQL-level drop completed normally while the CAS catalog row leaked, one per create/drop cycle. # # The primary oracle is the pool's OWN plain-text `cas/ref_catalog` object, read directly off disk: the -# exact `st` (lifecycle) field recorded for the table's logical namespace. `SYSTEM CAS FSCK`'s +# exact `state` (lifecycle) field recorded for the table's logical namespace. `SYSTEM CAS FSCK`'s # unreachable/dangling counts are a secondary check only -- fsck correctly regards a `live` leak as # CONSISTENT (nothing is unreachable; the row simply never dies), so it cannot detect this defect on its -# own; `04290_cas_no_leftovers.sh`'s fsck-only oracle is exactly why FINDING #2 shipped unnoticed. +# own; `04290_cas_no_leftovers.sh`'s fsck-only oracle is exactly why that leak shipped unnoticed. CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # shellcheck source=../shell_config.sh @@ -34,7 +34,7 @@ catalog_line() { grep -F "\"ns\":\"$1\"" "${CATALOG_FILE}" 2>/dev/null || true } -# The `st` (lifecycle) word recorded for namespace $1: "live"/"creating"/"removing", or "absent" if the +# The `state` (lifecycle) word recorded for namespace $1: "live"/"creating"/"removing", or "absent" if the # namespace has no catalog row (matches `04290`'s field-by-name discipline: never assume a position). catalog_state() { local line @@ -43,7 +43,16 @@ catalog_state() { echo "absent" return fi - echo "${line}" | grep -o '"st":"[a-z]*"' | head -1 | sed -E 's/"st":"([a-z]*)"/\1/' + # The pipeline's exit status is `sed`'s, which is 0 even on empty input, so emptiness is the only + # usable signal that the row exists but its state field could not be read -- a stale key spelling + # here must be loud, never mistaken for "absent". + local state + state=$(echo "${line}" | grep -o '"state":"[a-z]*"' | head -1 | sed -E 's/"state":"([a-z]*)"/\1/') + if [ -z "${state}" ]; then + echo "catalog row for namespace $1 exists but has no readable state field" >&2 + return 1 + fi + echo "${state}" } # ClickHouse's own store// fanout with the CAS archive boundary marker, exactly as @@ -91,7 +100,7 @@ echo "empty_table_has_no_ref_stream_before_drop $([ "${STREAM_HITS_BEFORE}" -eq $CLICKHOUSE_CLIENT --query "DROP TABLE t_dropns_empty SYNC" -# The current branch fails here by leaving st:"live"; the fix must show "removing" (a terminal stream +# The current branch fails here by leaving state:"live"; the fix must show "removing" (a terminal stream # record now exists but the catalog row itself is not deleted until GC folds and reclaims it). echo "empty_table_state_after_sync_drop $(catalog_state "${EMPTY_NS}")" STREAM_HITS_AFTER=$(find "${POOL_DIR}/ca/cas/ns/stream" -type f 2>/dev/null | wc -l) @@ -145,7 +154,10 @@ for i in 1 2 3; do CYCLE_NS_LIST+=("${CYCLE_NS}") $CLICKHOUSE_CLIENT --query "DROP TABLE t_dropns_cycle SYNC" - if [ "$(catalog_state "${CYCLE_NS}")" = "live" ]; then + # Read the state into a variable first: `$(...)` inside a test swallows a non-zero return, so an + # unreadable row would count as "not live" and this counter would pass while the reader is broken. + CYCLE_STATE=$(catalog_state "${CYCLE_NS}") || exit 1 + if [ "${CYCLE_STATE}" = "live" ]; then CYCLE_LEAK_COUNT=$((CYCLE_LEAK_COUNT + 1)) fi done From 23e2d8bc8b52db09feede4cf701e7c0ac4f0071e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:03:25 +0200 Subject: [PATCH 5/8] =?UTF-8?q?cas:=20wire-keys=20phase=203=20=E2=80=94=20?= =?UTF-8?q?proof,=20review=20polish,=20and=20the=20ref-protocol=20benchmar?= =?UTF-8?q?k?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Small correctness and review follow-ups to the wire-key cut: the part manifest names its namespace field the way every other object does, the algorithm set is read from the proven `EnumWireTable` instead of two independent hand-kept lists, the GC lease and heartbeat keep their separate owner spellings (documented, not merged), and the wire-format word writer gets the contract it was always assumed to have. Extends the `benchmark_cas_ref_protocol` harness to cover every format and direction the wire-keys design measures, plus a review-round fix to that harness. Also fixes `c++expr`: the generated work function needs internal linkage, without which ClickHouse-mode compilation did not work. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../Formats/CasBlobEnvelopeFormat.cpp | 2 + .../Formats/CasBlobEnvelopeFormat.h | 9 + .../Formats/CasFoldSealFormat.cpp | 10 +- .../Formats/CasGcOutcomesFormat.cpp | 4 +- .../ContentAddressed/Formats/CasLayout.cpp | 15 +- .../Formats/CasPartManifestFormat.cpp | 4 +- .../Formats/CasPoolMetaFormat.h | 2 +- .../Formats/CasRecordStreamFormat.cpp | 17 +- .../Formats/CasRefCatalogFormat.h | 4 +- .../Formats/CasRefLogFormat.cpp | 2 +- .../Formats/CasRefSnapshotFormat.cpp | 2 +- .../Formats/CasTextFormat.cpp | 15 +- .../ContentAddressed/Formats/CasTextFormat.h | 12 +- .../ContentAddressed/Formats/README.md | 2 +- .../ContentAddressed/Gc/CasGc.cpp | 2 +- .../ContentAddressed/Pool/CasPool.cpp | 7 +- .../ContentAddressed/Pool/CasPoolMeta.cpp | 2 +- .../benchmarks/benchmark_cas_ref_protocol.cpp | 528 +++++++++++++++++- src/Disks/tests/cas_test_helpers.h | 9 +- src/Disks/tests/gtest_ca_wiring.cpp | 4 +- .../tests/gtest_cas_backend_generation.cpp | 2 +- src/Disks/tests/gtest_cas_blob_digest.cpp | 4 +- .../tests/gtest_cas_blob_envelope_format.cpp | 35 ++ src/Disks/tests/gtest_cas_encoding_pins.cpp | 2 +- .../tests/gtest_cas_fence_generation.cpp | 2 +- .../tests/gtest_cas_gc_frontier_gate.cpp | 9 +- src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 7 +- src/Disks/tests/gtest_cas_gc_round.cpp | 4 +- src/Disks/tests/gtest_cas_gc_round_defer.cpp | 8 +- ...est_cas_namespace_file_request_profile.cpp | 4 +- .../tests/gtest_cas_namespace_life_id.cpp | 15 +- src/Disks/tests/gtest_cas_parallel_commit.cpp | 2 +- .../tests/gtest_cas_part_manifest_format.cpp | 8 +- src/Disks/tests/gtest_cas_pluggable_hash.cpp | 16 +- src/Disks/tests/gtest_cas_ref_catalog.cpp | 9 +- .../tests/gtest_cas_ref_chunked_flush.cpp | 5 +- .../tests/gtest_cas_ref_contiguous_alloc.cpp | 7 +- .../tests/gtest_cas_ref_epoch_seal_format.cpp | 16 +- .../tests/gtest_cas_ref_install_safety.cpp | 6 +- src/Disks/tests/gtest_cas_ref_log_format.cpp | 2 +- .../tests/gtest_cas_ref_snapshot_format.cpp | 3 +- ...test_cas_ref_snapshot_publish_ordering.cpp | 2 +- .../gtest_cas_ref_wedge_every_attempt.cpp | 4 +- src/Disks/tests/gtest_cas_ref_writer.cpp | 5 +- utils/c++expr | 4 +- 45 files changed, 708 insertions(+), 125 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index 129714d6b9e9..ff9fda182b26 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -153,6 +153,8 @@ void writeEnvelopeRefField(String & json, size_t budget, std::string_view raw_re } +const size_t mandatory_descriptor_worst_case = kMandatoryDescriptorWorstCase; + std::string_view provenanceOpToWireWord(ProvenanceOp op) { return kProvenanceOpWords.toWord(op, "CAS blob envelope"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 0aa26e22e182..e1a9a228433d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -33,6 +34,14 @@ enum class ProvenanceOp : uint8_t }; /// Returns the persisted wire word for a validated provenance operation. +/// The largest descriptor `encodeEnvelopeHeader` can ever produce before the diagnostic `ref` gets any +/// budget: every mandatory field at its type maximum, the longest provenance word, the `ref` framing +/// with empty quotes, the closing brace and the trailing newline. A `static_assert` beside its +/// definition proves it fits under `kMinBlobHeaderLen`; this declaration exists so the boundary test +/// can confirm the SAME number against bytes the encoder actually produced, which is the half a +/// compile-time proof cannot do — an understated formula satisfies the assert quite happily. +extern const size_t mandatory_descriptor_worst_case; + std::string_view provenanceOpToWireWord(ProvenanceOp op); /// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index 129c4b5c3493..77c3d1dcece4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -104,7 +104,7 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { bool first = true; - writeStringField(out, FoldSealWire::kind, kind, first); + writeWordField(out, FoldSealWire::kind, kind, first); writeStringField(out, FoldSealWire::run_key, r.key, first); writeHex128Field(out, FoldSealWire::checksum, r.checksum, first); writeNumberField(out, FoldSealWire::shard, r.shard, first); @@ -311,14 +311,14 @@ String encodeFoldSeal(const CasFoldSeal & seal) life_state.cleanup_evidence->remove_txn_id.ref_sequence); bool first = true; - writeStringField(out, FoldSealWire::kind, kRefLifeTag, first); + writeWordField(out, FoldSealWire::kind, kRefLifeTag, first); writeHex128Field(out, FoldSealWire::life, life_id, first); - writeStringField(out, FoldSealWire::classification, classification_word, first); + writeWordField(out, FoldSealWire::classification, classification_word, first); writeU64StringField(out, FoldSealWire::fold_epoch, cov.last_folded_ref_id.writer_epoch, first); writeU64StringField(out, FoldSealWire::fold_seq, cov.last_folded_ref_id.ref_sequence, first); if (cov.hold) { - writeStringField(out, FoldSealWire::hold_reason, holdReasonToWord(cov.hold->reason), first); + writeWordField(out, FoldSealWire::hold_reason, holdReasonToWord(cov.hold->reason), first); writeU64StringField(out, FoldSealWire::hold_epoch, cov.hold->offending_position.writer_epoch, first); writeU64StringField(out, FoldSealWire::hold_seq, cov.hold->offending_position.ref_sequence, first); writeNumberField(out, FoldSealWire::retries, cov.hold->retry_count, first); @@ -349,7 +349,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const auto & [shard, s] : seal.condemned_summary) { bool first = true; - writeStringField(out, FoldSealWire::kind, kCondemnedTag, first); + writeWordField(out, FoldSealWire::kind, kCondemnedTag, first); writeNumberField(out, FoldSealWire::shard, shard, first); writeNumberField(out, FoldSealWire::condemned_total, s.condemned_total, first); writeNumberField(out, FoldSealWire::pending_total, s.pending_total, first); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index 1b19256d6d4b..9ae1ffd6282d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -53,10 +53,10 @@ String encodeOutcomeLog(const OutcomeLog & log) for (const OutcomeEntry & e : log.entries) { bool first = true; - writeStringField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); + writeWordField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); writeBlobRefFields(out, first, e.ref); /// algo + digest writeTokenFields(out, first, e.token); /// token_type + token - writeStringField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); + writeWordField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); closeObject(out, first); writeChar('\n', out); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp index 9466a8af735e..d3c92e767c12 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -67,14 +68,16 @@ std::optional Layout::parseBlobKey(std::string_view key) const if (shard.size() != 2 || hex.size() < 2 || shard != hex.substr(0, 2)) return std::nullopt; /// malformed shard/hex shape -- not ours - /// `` -> `BlobHashAlgo`: the small enum-value set makes a linear scan against - /// `blobHashAlgoName` (the ONE name authority) cheaper and safer than a second name table that - /// could drift from it. + /// `` -> `BlobHashAlgo` through the wire table itself, whose coverage is proven against + /// the enum at compile time. A hand-written candidate list here would be a second enumeration that + /// a new algorithm could silently outgrow: the parser would reject a segment the writer emits. + /// This path answers "is this key ours?", so an unknown segment is `nullopt` -- debris, not + /// corruption -- which is why it scans rather than calling the throwing `fromWord`. std::optional algo; - for (BlobHashAlgo candidate : {BlobHashAlgo::CityHash128, BlobHashAlgo::XXH3_128, BlobHashAlgo::Sha256}) - if (algo_name == blobHashAlgoName(candidate)) + for (const auto & entry : kBlobHashAlgoWords.entries) + if (algo_name == entry.word) { - algo = candidate; + algo = entry.value; break; } if (!algo) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index e8ff121dfee7..ac70c2c00b23 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -25,7 +25,7 @@ namespace namespace PartManifestWire { - constexpr WireKey ns{"root_namespace"}; + constexpr WireKey ns{"namespace"}; constexpr WireKey payload_digest{"payload_digest"}; constexpr WireKey path{"path"}; constexpr WireKey place{"place"}; @@ -161,7 +161,7 @@ PartManifest decodePartManifest(std::string_view data) else r.skipUnknown(key); } if (!ns) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing root_namespace"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing namespace"); if (!pd) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing payload_digest"); m.ref = fields.buildRef("PartManifest", "descriptor"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h index e25e767fb5e3..f17cb5b87fb1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h @@ -38,7 +38,7 @@ struct PoolMeta /// because changing it would move the blob payload offset for existing objects; a new hash algorithm /// is rejected unless `allow_new` is set, and concurrent admission is retried from fresh metadata. /// - /// `allow_mint` (spec §2 [C4][D2]) gates the create-if-absent path: minting a fresh `_pool_meta` is a + /// `allow_mint` gates the create-if-absent path: minting a fresh `_pool_meta` is a /// consequential write that establishes a brand-new pool identity, so it is permitted ONLY on the /// writable startup path that has just passed the zero-write residual proof (`Pool::open`). Every /// non-bootstrap caller — a read-only/observe open, `openForDecommission` — passes `false`; an absent diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index 857e428538f6..00d20988ac03 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -54,16 +54,17 @@ int hexNibble(char c) return -1; } +/// The run `ref` carries the algorithm as a raw leading byte, so this is the byte-side counterpart of +/// the word table -- and it walks that same table rather than listing the enumerators again. A second +/// list is how the writer and the reader come to disagree about which algorithms exist: `renderB` +/// writes whatever the enum holds, and a hand-written switch here would reject exactly what a new +/// enumerator adds. BlobHashAlgo algoFromByte(uint8_t b, std::string_view what) { - switch (b) - { - case static_cast(BlobHashAlgo::CityHash128): return BlobHashAlgo::CityHash128; - case static_cast(BlobHashAlgo::XXH3_128): return BlobHashAlgo::XXH3_128; - case static_cast(BlobHashAlgo::Sha256): return BlobHashAlgo::Sha256; - default: - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown algo byte {} in record key", what, b); - } + for (const auto & entry : kBlobHashAlgoWords.entries) + if (static_cast(entry.value) == b) + return entry.value; + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown algo byte {} in record key", what, b); } /// `ref` = the algo byte as two lowercase hex chars, then the digest hex at the algo's width. The diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h index 9515b00b425f..0e93845731f4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h @@ -103,8 +103,8 @@ struct RefCatalog /// instead) -- but deliberately does NOT enforce the whole-object cap itself: that predicate must /// name the namespace under admission, which only a caller of `checkCatalogAdmission` knows. /// -/// These bytes go to and come from the backend DIRECTLY, exactly like `cas_ref_ckpt`: the Pool-side -/// `CasRefCatalog::read`/`casUpdateImpl` (`Pool/CasRefCatalog.cpp`) bypass `sealObject`/`openObject`, +/// These bytes go to and come from the backend DIRECTLY: the catalog read and update paths bypass +/// `sealObject`/`openObject`, /// which are the identity under this class's `CompressionPolicy::Never` and would add nothing. A /// policy flip to `Always` therefore breaks this silently -- and is caught, because `storedSuffix` /// would stop being empty and the registry test asserting `storedSuffix(FormatId::RefCatalog) == ""` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index 4c6dc9fdf053..8d60260c8b53 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -126,7 +126,7 @@ struct BindingFields /// `!`-prefixed: `prev_epoch_seal` is INV-2 chain evidence, not cosmetic metadata -- a decoder that /// doesn't understand it must refuse the object rather than silently drop the chain link while /// otherwise passing the structural grammar (`JsonObjectReader::skipUnknown` rejects any unrecognized -/// `!`-key with `UNKNOWN_FORMAT_VERSION`, tolerant or not; see task-1 review finding M4). +/// `!`-key with `UNKNOWN_FORMAT_VERSION`, tolerant or not). void writeLogMeta(CasJsonWriter & out, const String & ns, const RefTxnId & txn_id, const std::optional & prev_epoch_seal) { bool first = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index 78caf9c5632c..04ecef15cd53 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -108,7 +108,7 @@ void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) /// A snapshot object exists only for a live namespace -- `RefLifecycle::Removed` has no snapshot /// representation -- so the wire carries exactly one lifecycle word. The reader keeps the /// fail-closed half: any other word, or none, is rejected there. - writeStringField(out, RefSnapWire::lifecycle, kLiveLifecycleWord, first); + writeWordField(out, RefSnapWire::lifecycle, kLiveLifecycleWord, first); closeObject(out, first); writeChar('\n', out); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 8438ac786478..d2f25aab615f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -109,6 +109,19 @@ void CasJsonWriter::stringValue(std::string_view s) appendChar('"'); } +void CasJsonWriter::wordValue(std::string_view word) +{ + /// The contract, checked where it is cheap to check: a vocabulary word carries no byte the JSON + /// escaper would rewrite. Violating it would emit malformed JSON rather than a mis-escaped + /// string, so this is a programming error and belongs in the debug build, not a runtime branch on + /// the encode hot path. + chassert(std::none_of(word.begin(), word.end(), + [](char c) { return isSpecialJsonByte(static_cast(c)); })); + appendChar('"'); + buf.append(word.data(), word.size()); + appendChar('"'); +} + void CasJsonWriter::wordArray(std::span words) { appendChar('['); @@ -118,7 +131,7 @@ void CasJsonWriter::wordArray(std::span words) if (!first) appendChar(','); first = false; - stringValue(word); + wordValue(word); } appendChar(']'); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 48f473c33d65..bd353f864d17 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -63,6 +63,13 @@ class CasJsonWriter /// Quoted JSON string with full escaping (bulk-run scan). Defined in CasTextFormat.cpp. void stringValue(std::string_view s); + /// A value from a wire VOCABULARY -- an enum table's word or a record tag. Those are drawn from + /// `[a-z0-9_]` by construction, so this writes the bytes as they are instead of running them + /// through the escaper's byte scan and state machine. Output is identical to `stringValue` for + /// every input the contract admits; a caller that passes an arbitrary string is the bug this + /// asserts against, and `writeWordField` is the only intended way in. + void wordValue(std::string_view word); + /// JSON array of canonical word strings, emitted without intermediate storage. void wordArray(std::span words); @@ -163,10 +170,13 @@ inline void writeKey(CasJsonWriter & out, WireKey key, bool & first) writeKey(out, key.text, first); } +/// For a value that comes from a wire vocabulary (an enum table's word, a record tag). It is NOT +/// interchangeable with `writeStringField`: this one promises its value needs no JSON escaping and +/// skips the escaper accordingly, which is why an open string must never be routed through it. inline void writeWordField(CasJsonWriter & out, WireKey key, std::string_view word, bool & first) { writeKey(out, key, first); - writeStringValue(out, word); + out.wordValue(word); } inline void writeWordArrayField(CasJsonWriter & out, WireKey key, std::span words, bool & first) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index ea7a73ab41bf..52cd954377fd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -24,7 +24,7 @@ trailer, followed by a banner-framed raw payload zone for inline file bytes. | `cas/ns/state//_ckpt` | mutable life checkpoint (`life_epoch`, `committed_epoch`/`committed_seq`, `snapshot_epoch`/`snapshot_seq`, `seal_epoch`/`seal_seq`) | `CasRefCkptFormat` | writer/GC fold | | `cas/ns/state//_files/…​` | namespace-owned raw files | — | upper layers | | `cas/ref_catalog` | namespace lifecycle catalog (`kind:"entry"`, `ns`, `state`, `life`, `remove_round`, `creator`, `creator_epoch`, `creator_fence`) | `CasRefCatalogFormat` | namespace admission/removal | -| `cas/manifests//-/.zst` | part manifest (`root_namespace`, `payload_digest`; entry `path`, `place`, `size`) | `CasPartManifestFormat` | part build | +| `cas/manifests//-/.zst` | part manifest (`namespace`, `payload_digest`; entry `path`, `place`, `size`) | `CasPartManifestFormat` | part build | | blob keys (`CasLayout::blobKey`) | blob envelope (`type`, `v`, `tag`, `build`, `time_ms`, `creator`, `op`, `chver`, `ref`) + payload | `CasBlobEnvelopeFormat` | uploads | | blob-meta keys (`CasLayout::blobMetaKey`) | freshness sidecar (`state`, `condemn_round`, `size`) | `CasBlobMetaFormat` | dedup/GC | | `gc/state`, `gc/hb` | GC state (`round`, `gc_shards`, `snap_generation`, `snap_pruned_through`, `snap_attempt`, `manifest_sweep_cursor`, `lease_owner`, `lease_seq`) / heartbeat (`owner`, `hb_seq`) | `CasGcStateFormat` | GC | diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 46db0402ed7a..baaf7b86fdc8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -2674,7 +2674,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// It counts the CUT ARITHMETICALLY, not by listed ids. Under arithmetic intake a listed-id count is /// not even the right question: a hint hole means a round legitimately applies records the listing /// never mentioned, so the old recomputation would report fewer logs than folded and fail every - /// healthy round on a lying store -- it would have made this task's own fix unshippable. + /// healthy round on a lying store. /// /// BE HONEST ABOUT WHAT IS LEFT. The old formula could disagree with reality because it was derived /// from a different source (the listing) than the counter. This one is derived from the runs the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index ed6a957cbde6..f5563abf4306 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -342,8 +342,9 @@ String Pool::lifecycleReasonDetail(PoolLifecycle lc) const void Pool::throwIfLifecycleTerminal() const { /// The typed error carries the sub-state in its message so a wrong diagnosis is impossible from the - /// first error line (spec §1 [D5]). `Live`/`TransientNotLive` proceed here — the transient class is - /// still gated only by the write fence in this task (the full six-class gate is Task 8). + /// first error line. `Live`/`TransientNotLive` proceed here — the transient class is + /// still gated only by the write fence; the destructive gate additionally requires the other + /// lifecycle proofs. const PoolLifecycle lc = mount_runtime.lifecycle(); if (lc == PoolLifecycle::Live || lc == PoolLifecycle::TransientNotLive) return; @@ -1627,7 +1628,7 @@ void Pool::reportImpossibleInterference(const String & key, const String & reaso /// requests -- never the caller's thread, and never blocking this call's own return. try { - /// The lease owns the pool reference for this task's lifetime; capturing one here as well would + /// The lease owns the pool reference until it releases it; capturing one here as well would /// put it outside the lease's release ordering. const bool dispatched = tryDispatchDetached([this, key](DetachedStopToken token) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp index 34aaeaa37308..8b1e4bd9ba02 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp @@ -132,7 +132,7 @@ PoolMeta PoolMeta::createOrValidate( /// build at all), so the reader-generation floor is stamped at THIS build's `G_BUILD` at /// creation, not left at 0. /// - /// BOOTSTRAP GATE (spec §2 [C4][D2]): minting is permitted ONLY on the verified bootstrap path. A + /// BOOTSTRAP GATE: minting is permitted ONLY on the verified bootstrap path. A /// non-bootstrap caller (a read-only/observe open, `openForDecommission`) passes `allow_mint=false` /// and fails closed here — never minting a fresh identity outside that path (an observe scan that /// minted would poison the next writable mount's residual check). The residual EMPTINESS proof itself diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp index f23f4dc06bae..41365299eaf4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp @@ -1,15 +1,38 @@ #include #include +#include +#include +#include #include +#include + #include +#include +#include #include #include +#include +#include +#include +#include +#include #include +#include +#include +#include +#include #include #include +namespace DB::ErrorCodes +{ + extern const int CORRUPTED_DATA; + extern const int LIMIT_EXCEEDED; + extern const int LOGICAL_ERROR; +} + /// Pure measurement, no pass/fail assertions -- see the cas-gc-rebuild BACKLOG.md entries /// "OPTIMIZATION OPPORTUNITY -- ref-ledger JSON encoding writes byte-by-byte" and the (now /// RESOLVED) "admits() re-encodes the WHOLE ref table once per state-growing op" entry for the @@ -38,6 +61,16 @@ /// ladder, rung 2 was NOT attempted either (it trades readability and needs a human decision); /// reported as DONE_WITH_CONCERNS. CasEncodingPins.* stayed byte-identical (green) throughout. /// +/// NOTE (2026-08, wire-key-rename campaign): the "Phase B baselines" table immediately below measures +/// a DIFFERENT investigation (the `RefTableState` encapsulation refactor) and predates the five-format, +/// both-directions wire-key-cut design entirely. It is NOT the "before" side for that campaign's +/// measurement, and a later reader must not diff against it for that purpose. The actual before side +/// is the pre-cut worktree pinned at commit `65ec8688cdb`; the recorded patch that builds this file +/// there lives under `docs/superpowers/cas/bench-wire-keys-phase3/`. The table is kept exactly as +/// written because it is real history for the investigation it belongs to, not because it answers this +/// one -- see the "Wire-key-cut instrument" section further down for the five new formats this +/// campaign added. +/// /// Phase B baselines, 2026-07-21, pre-encapsulation (this binary; `--benchmark_repetitions=3 /// --benchmark_report_aggregates_only=true`; medians reported). Recorded ahead of the /// `RefTableState` encapsulation refactor so later phases can re-run this exact suite unchanged and @@ -118,6 +151,15 @@ RefLogTxn makeSamplePromoteTxn() /// A synthetic snapshot of `n` committed rows plus one pending precommit ready to promote. /// Built as a RefTableSnapshot and materialized via the public `replay` entry point, so this /// helper keeps compiling unchanged when RefTableState's fields become private (Phase A). +/// +/// Committed-row field widths (load-bearing for the `cas_ref_snap` wire-key-cut benchmarks and byte +/// oracle, which measure the RELATIVE cost of a key rename against the encoded VALUE bytes as the +/// denominator): `published_at_ms` is a real 13-digit epoch-ms rather than the default `0`, and +/// `manifest_ref`'s `writer_epoch`/`build_sequence` are multi-digit (a pool old enough to have +/// restarted its writer decades of times, and a build counter past its 89811th commit -- the same +/// order of magnitude as the real ref-ledger key at the top of this file, `kSafeKeyLikeString`, and +/// `makeSamplePromoteTxn`'s ref name). A minimal `0`/`1`/`1` shrinks the value-byte denominator a key +/// rename is measured against and inflates the rename's apparent percentage cost. RefTableSnapshot makeSyntheticSnapshot(size_t n) { RefTableSnapshot snapshot; @@ -127,7 +169,8 @@ RefTableSnapshot makeSyntheticSnapshot(size_t n) { RefCommittedRow row; row.ref_name = "part_" + std::to_string(i) + "_20260719_0_1000_1"; - row.manifest_ref = ManifestRef{1, 1, static_cast(i + 1)}; + row.manifest_ref = ManifestRef{42, 89811 + static_cast(i), static_cast(i + 1)}; + row.published_at_ms = 1752900000000ULL + i; snapshot.committed.push_back(row); } std::sort(snapshot.committed.begin(), snapshot.committed.end(), @@ -550,4 +593,485 @@ static void BM_Materialize(benchmark::State & state) } BENCHMARK(BM_Materialize)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); -BENCHMARK_MAIN(); +/// ------------------------------------------------------------------------------------------------- +/// Wire-key-cut instrument (Task 7): encode AND decode for the five formats the campaign's wire-key +/// rename touched most (`cas_run`, `cas_ref_snap`, `cas_part_manifest`, `cas_fold_seal`, +/// `cas_ref_catalog`), plus a byte/cap oracle (below `reportFormatCaps`). This section only BUILDS the +/// instrument -- it does not take the before/after measurement itself, which is a later task run +/// against this same binary built on both sides of the cut. The "before" side is the pre-cut worktree +/// at `/home/mfilimonov/workspace/ClickHouse/cas-p2-before`, pinned at commit `65ec8688cdb`; the +/// recorded patch that adapts this file's one incompatible call site (`foldedClassification`/ +/// `clampedClassification` below) for that build lives under +/// `docs/superpowers/cas/bench-wire-keys-phase3/`. Every other line in this section is byte-identical +/// on both sides -- confirmed against the before-side headers, which differ from these only in +/// comment text (the retired terse wire spellings) and in `RefCoverage::classification`'s type. +/// ------------------------------------------------------------------------------------------------- + +namespace +{ + +/// `cas_run` fixture: `n` distinct blobs in strictly ascending digest order (`SourceEdgeRunWriter` +/// requires non-decreasing `(ref, source_id)` keys, and a monotonically increasing digest alone +/// satisfies that regardless of `source_id`). Marker mix models one healthy in-degree run: the +/// overwhelming majority of tracked blobs simply carry a live edge this generation (98% `Edge`); a +/// blob losing its LAST edge (`Zero`) or actually condemned for deletion (`Condemned`, carrying the +/// full retired-incarnation token) is comparatively rare at any one round -- 1% each here, not 0 and +/// not half. `source_id` is a synthetic per-record counter rather than a real backend id: the codec's +/// cost is driven by the DIGEST's hex width, not the id's numeric value. The condemned token mirrors a +/// real S3 ETag's width (a quoted 32-hex value) and `size` a realistic single-blob byte count (64 KiB, +/// a typical compressed column chunk). Record count ranges 100 to 100,000 (`RangeMultiplier(10)`), +/// matching every `Complexity()` benchmark already in this file. +std::vector makeSourceEdgeRecords(size_t n) +{ + std::vector records; + records.reserve(n); + for (size_t i = 0; i < n; ++i) + { + const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(i + 1))}; + SourceEdgeRecord rec; + rec.ref = ref; + if (i % 100 == 0) + { + rec.source_id = UInt128(0); + rec.marker = RunMarker::Condemned; + rec.delete_pending = (i % 200 == 0); + rec.token = Token{"\"e1b2c3d4e5f6071829300a0b0c0d0e0f\"", TokenType::ETag}; + rec.size = 64 * 1024; + rec.condemn_round = 7; + } + else if (i % 100 == 50) + { + rec.source_id = UInt128(0); + rec.marker = RunMarker::Zero; + } + else + { + rec.source_id = UInt128(i + 1); + rec.marker = RunMarker::Edge; + } + records.push_back(rec); + } + return records; +} + +/// Runs the real `SourceEdgeRunWriter` over `records`, exactly as `CASRecordStream`'s own +/// `encodeRun` helper does -- so decode below always consumes real encoder output, never a +/// hand-built string. +String encodeSourceEdgeRun(const std::vector & records) +{ + DB::WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + for (const auto & r : records) + writer.append(r); + writer.finish(); + /// `str()` returns a `std::string &`, so returning it plainly would copy-construct the whole + /// encoded run on every call (no NRVO is available for a reference) -- `std::move` here moves it + /// instead, matching the four other encoders, which all end with `std::move(out).take()` and copy + /// nothing. `str()` finalizes `out` itself, so no separate `finalize()` call is needed first. + return std::move(out.str()); +} + +/// The ONE call site whose TYPE differs across the wire-key-rename cut this benchmark spans: on this +/// (AFTER) side `RefCoverage::classification` is the closed `CoverageClass` enum; at the pre-cut +/// commit it is a raw `uint8_t` whose CLAMPED value is ALSO renumbered (4 there, 3 here -- see +/// `CasFoldSealFormat.h`'s own history comment on `CoverageClass`). A bare numeric literal at the call +/// site would therefore silently measure the WRONG row shape on the before-side build, so the Step-4 +/// patch touches only this pair of one-line functions; every benchmark body in this file stays +/// byte-identical on both sides. +CoverageClass foldedClassification() { return CoverageClass::Folded; } +CoverageClass clampedClassification() { return CoverageClass::Clamped; } + +/// Not the record axis under test (`n` below is `ref_lives` row count): fixed at a representative +/// multi-shard pool size. A single-shard fixture would fold `blob_target_runs`/`condemned_summary` to +/// one degenerate entry each, understating the per-shard fan-out a real multi-shard pool carries in +/// both sections. +constexpr uint64_t kFoldSealGcShards = 4; + +/// `cas_fold_seal` fixture: `n` `ref_lives` rows keyed by ascending life id, split base/hold-bearing/ +/// cleanup-evidence 90%/5%/5%. Per the spec's byte table, a hold-bearing row adds 33 bytes and a +/// cleanup-evidence row adds 16 bytes over a base row's 22-plus-class-word bytes; at this 90/5/5 mix +/// the recovered uplift over an all-base fixture is 0.05*33 + 0.05*16 = 2.45 bytes/row, about 8% over +/// a base row's own ~30 bytes (the full one-third the spec's deltas imply is the all-clamped extreme, +/// not this mix) -- still enough that omitting the two minority shapes entirely would misstate the +/// row-average cost in the wrong direction. The 90/5/5 split models a healthy pool: most namespaces +/// fold cleanly every round (base: `Folded`, no hold, no cleanup evidence); a minority sit behind a +/// transient barrier (hold-bearing: `Clamped`, `ManifestBodyMissing`); a minority are mid-teardown +/// (cleanup evidence: `Folded` plus a terminal `remove_namespace` fold). Neither minority shape is the +/// common case, but neither is negligible either -- both recur every round in a live pool. +/// `RefTxnId` epoch/sequence pairs and the hold's `retry_count`/`next_retry_round` are multi-digit +/// (a pool old enough to have restarted its writer dozens of times and folded past its 100,000th +/// ref-log transaction; a hold retried past its first round but nowhere near abandoned) rather than +/// the single-digit illustrative values the spec's byte table uses to name the three row SHAPES -- +/// matching the shapes, not the spec table's example digits, is what keeps the value-byte denominator +/// realistic (see `makeSyntheticSnapshot`'s doc comment for why that denominator matters). Record +/// count ranges 100 to 100,000, matching every `Complexity()` benchmark in this file. +CasFoldSeal makeFoldSeal(size_t n) +{ + CasFoldSeal seal; + seal.generation = 7; + seal.parent_generation = 6; + for (size_t i = 0; i < n; ++i) + { + RefLifeFoldState row; + if (i % 20 == 0) + { + row.coverage = RefCoverage{ + .classification = clampedClassification(), + .last_folded_ref_id = RefTxnId{42, 103482}, + .hold = RefHold{ + .reason = HoldReason::ManifestBodyMissing, + .offending_position = RefTxnId{42, 103500}, + .retry_count = 14, + .next_retry_round = 1042}}; + } + else if (i % 20 == 1) + { + row.coverage = RefCoverage{.classification = foldedClassification(), .last_folded_ref_id = RefTxnId{42, 118203}}; + row.cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{42, 118190}}; + } + else + { + row.coverage = RefCoverage{.classification = foldedClassification(), .last_folded_ref_id = RefTxnId{42, 100123}}; + } + seal.ref_lives.emplace(UInt128(i + 1), std::move(row)); + } + for (uint64_t shard = 0; shard < kFoldSealGcShards; ++shard) + { + seal.blob_target_runs.push_back(RunRef{ + .key = fmt::format("p/gc/gen/7/attempt/1/blob_target/{}/0", shard), + .checksum = UInt128(0x1000 + shard), .shard = shard, .key_generation = 7}); + seal.condemned_summary[shard] = CondemnedSummary{ + .condemned_total = 1000 + shard, .pending_total = 10 + shard, .oldest_nonpending_condemn_round = 4}; + } + return seal; +} + +/// `cas_part_manifest` fixture: `n` entries in path order, 90% `Blob` (the column/mark/index files +/// that dominate a real MergeTree part) and every 10th `Inline` (small metadata files like +/// `count.txt`/`checksums.txt` that get embedded rather than stored as a separate blob). Blob sizes +/// cycle 4-64 KiB across 16 steps to resemble the spread of real column-chunk sizes rather than one +/// repeated constant; inline bytes are a fixed 48-byte payload, resembling a small metadata file. +/// `ref`/`root_namespace_id` are fixed -- they do not scale with entry count in a real manifest +/// either. `encodePartManifest` sorts entries itself, so input order need not be canonical. Record +/// count ranges 100 to 100,000, matching every `Complexity()` benchmark in this file (a real part +/// rarely reaches the top of that range; it stress-tests a pathologically wide/many-column part). +PartManifest makePartManifest(size_t n) +{ + PartManifest m; + m.ref = ManifestRef{5, 15, 1}; + m.root_namespace_id = RootNamespace("00/aa@cas@"); + m.entries.reserve(n); + for (size_t i = 0; i < n; ++i) + { + ManifestEntry e; + if (i % 10 == 9) + { + e.path = fmt::format("{:06}_meta.txt", i); + e.placement = EntryPlacement::Inline; + e.inline_bytes = String(48, 'x'); + } + else + { + e.path = fmt::format("{:06}_data.bin", i); + e.placement = EntryPlacement::Blob; + e.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(i + 1))}; + e.blob_size = 4096 * (1 + (i % 16)); + } + m.entries.push_back(std::move(e)); + } + m.payload_digest = computePayloadDigest(m); + return m; +} + +/// `cas_ref_catalog` fixture: `n` entries in ascending namespace order (a 7-digit zero-padded ordinal +/// keeps ascending lexical order across the whole 100..100,000 range, well under `kMaxNamespaceBytes`). +/// The mix resembles one whole-pool catalog snapshot: most namespaces are simply `Live` (96%), with a +/// small steady trickle of admission (`Creating`, 2%) and teardown (`Removing`, 2%) in flight at any +/// moment -- neither churn state is the common case, but neither is negligible either. Record count +/// ranges 100 to 100,000, matching every `Complexity()` benchmark in this file. +RefCatalog makeRefCatalog(size_t n) +{ + RefCatalog catalog; + catalog.entries.reserve(n); + for (size_t i = 0; i < n; ++i) + { + CatalogEntry e; + e.ns = RootNamespace(fmt::format("roots/ca_tbl_{:07}", i)); + e.incarnation = UInt128(i + 1); + if (i % 50 == 0) + { + e.state = NsState::Creating; + e.creator = CreatorFence{"srv-bench", 1, 1}; + } + else if (i % 50 == 25) + { + e.state = NsState::Removing; + e.removal_started_round = 42; + } + else + { + e.state = NsState::Live; + } + catalog.entries.push_back(std::move(e)); + } + return catalog; +} + +} + +/// `cas_run` is streamed (`object_cap == 0`; see `CasRecordStreamFormat.h`) and never materialized +/// whole in production, but the benchmark still needs one complete encoded run to time and to decode: +/// `encodeSourceEdgeRun` drives the real `SourceEdgeRunWriter`/`SourceEdgeRunReader` pair over an +/// in-memory buffer, the same pair the streaming production path uses over its own `WriteBuffer`/ +/// `ReadBuffer`. +static void BM_CasRunEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const std::vector records = makeSourceEdgeRecords(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeSourceEdgeRun(records)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRunEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRunDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const String encoded = encodeSourceEdgeRun(makeSourceEdgeRecords(n)); + for (auto _ : state) + { + DB::ReadBufferFromMemory in(encoded.data(), encoded.size()); + SourceEdgeRunReader reader(in); + SourceEdgeRecord rec; + size_t count = 0; + while (reader.next(rec)) + ++count; + benchmark::DoNotOptimize(count); + } + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRunDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +/// `BM_SnapshotEncode` above already exists (for the E4 contiguous-scan investigation) and has no +/// decode counterpart. This pair is the one the wire-key-cut measurement uses: same fixture, but named +/// and shaped to match the other four formats' encode/decode pairs in this section. +static void BM_CasRefSnapEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefTableSnapshot snapshot = makeSyntheticSnapshot(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeRefTableSnapshot(snapshot)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefSnapEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRefSnapDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefTableSnapshot snapshot = makeSyntheticSnapshot(n); + const String encoded = encodeRefTableSnapshot(snapshot); + for (auto _ : state) + benchmark::DoNotOptimize(decodeRefTableSnapshot(encoded, snapshot.ns, snapshot.snapshot_id)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefSnapDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasPartManifestEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const PartManifest m = makePartManifest(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodePartManifest(m)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasPartManifestEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasPartManifestDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const PartManifest m = makePartManifest(n); + const String encoded = encodePartManifest(m); + for (auto _ : state) + benchmark::DoNotOptimize(decodePartManifest(encoded)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasPartManifestDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasFoldSealEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const CasFoldSeal seal = makeFoldSeal(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeFoldSeal(seal)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasFoldSealEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasFoldSealDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const CasFoldSeal seal = makeFoldSeal(n); + const String encoded = encodeFoldSeal(seal); + for (auto _ : state) + benchmark::DoNotOptimize(decodeFoldSeal(encoded)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasFoldSealDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRefCatalogEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefCatalog catalog = makeRefCatalog(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeRefCatalog(catalog)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefCatalogEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRefCatalogDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefCatalog catalog = makeRefCatalog(n); + const String encoded = encodeRefCatalog(catalog); + for (auto _ : state) + benchmark::DoNotOptimize(decodeRefCatalog(encoded)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefCatalogDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +namespace +{ + +/// Binary search on record count with the real encode -> `sealObject` -> `openObject` pipeline as the +/// oracle. `openObject` (`CasTextFormat.cpp`) enforces the registry's `object_cap` on BOTH the raw and +/// the zstd-frame-header path, so this one pipeline works whether or not the format compresses; a +/// format's OWN pre-put gate (e.g. fold-seal's `checkFoldSealObjectBytes`) may throw earlier, at the +/// encode step itself. Either `LIMIT_EXCEEDED` or `CORRUPTED_DATA` at this boundary means "does not +/// fit" and steers the search; any other exception is a fixture bug, not a capacity signal, and is +/// left to propagate rather than being misread as "found the cap". +/// +/// `known_fits_n`/`known_fits_bytes` seed the exponential search from a bytes-per-record estimate +/// measured at a small `n` -- NOT a hardcoded delta table -- purely to reduce how many large, +/// expensive encodes the search performs before bisecting. The estimate never becomes the answer: the +/// real encoder confirms every step of both the exponential growth and the final exact bisection. +template +uint64_t maxRecordCountUnderCap(FormatId id, uint64_t known_fits_n, uint64_t known_fits_bytes, Encode encode) +{ + auto fits = [&](uint64_t n) -> bool + { + try + { + const String stored = sealObject(id, encode(n)); + benchmark::DoNotOptimize(openObject(id, stored)); + return true; + } + catch (const DB::Exception & e) + { + if (e.code() == DB::ErrorCodes::CORRUPTED_DATA || e.code() == DB::ErrorCodes::LIMIT_EXCEEDED) + return false; + throw; + } + }; + + /// The bisection below is correct only if `fits(lo) == true`. The caller's `known_fits_n` comes + /// from an encode IT ran itself -- never through `openObject`, which is what actually enforces + /// `object_cap` (the raw-size check, or the zstd frame's declared decompressed size) -- so this + /// verifies the bound directly rather than trusting that claim. If `known_fits_n` itself is + /// already over the cap (e.g. a future, much larger report size), halve downward until a verified + /// fit is found; if even `n == 1` does not fit, that is a fixture/format bug, not a capacity + /// signal, and is raised loudly rather than silently reported as a wrong maximum. + uint64_t lo = known_fits_n; + while (lo > 1 && !fits(lo)) + lo /= 2; + if (!fits(lo)) + throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, + "maxRecordCountUnderCap: format {} does not fit its object cap even at n=1", static_cast(id)); + + const FormatTraits & traits = traitsFor(id); + const uint64_t per_record = std::max(1, known_fits_bytes / std::max(1, known_fits_n)); + uint64_t hi = std::max(lo * 2, traits.object_cap / per_record); + while (fits(hi)) + { + lo = hi; + hi *= 2; + } + while (hi - lo > 1) + { + const uint64_t mid = lo + (hi - lo) / 2; + (fits(mid) ? lo : hi) = mid; + } + return lo; +} + +/// One format's report line: decompressed bytes at `report_n`, stored bytes under the format's REAL +/// registered compression policy (or `n/a` for a policy that stores raw -- `Never`/`PinnedRaw` -- since +/// there is no separate compressed form to report, and a `0` there would read as a measurement rather +/// than "not applicable"), and the largest record count the real encoder admits under the format's +/// object cap. +template +void reportSealedFormat(std::string_view name, FormatId id, uint64_t report_n, Encode encode) +{ + const FormatTraits & traits = traitsFor(id); + const String decompressed = encode(report_n); + const String stored = sealObject(id, decompressed); + const bool stores_raw = traits.compression != CompressionPolicy::Always; + const uint64_t max_n = maxRecordCountUnderCap(id, report_n, decompressed.size(), encode); + + fmt::print("{:<18} decompressed={:>10} bytes (n={}) stored={} max_n_under_object_cap={}\n", + name, decompressed.size(), report_n, + stores_raw ? "n/a (stored raw, no compression)" : (std::to_string(stored.size()) + " bytes (zstd)"), + max_n); +} + +/// Step 3's byte and cap oracle: a small, main-less, flag-invoked harness (see `main` below) rather +/// than a benchmark or a gtest, so it never engages the timing loop and never needs a second `main` in +/// this binary. Reports, per format, at the stated `kReportN`: decompressed bytes, stored bytes under +/// the real compression policy (or `n/a`), and the maximum record count the real encoder admits under +/// the object cap (or `n/a` where none applies). +void reportFormatCaps() +{ + constexpr uint64_t kReportN = 1000; + fmt::print("=== cas format byte/cap report (n={}) ===\n", kReportN); + + /// `cas_run` is `object_cap == 0` (streamed, `RunFile` family): never materialized whole in + /// production, so there is no whole-object cap to search for and no compressed form to report. + { + const String encoded = encodeSourceEdgeRun(makeSourceEdgeRecords(kReportN)); + fmt::print("{:<18} decompressed={:>10} bytes (n={}) stored=n/a (PinnedRaw, never compressed) " + "max_n_under_object_cap=n/a (object_cap=0: streamed one line at a time, never materialized whole)\n", + "cas_run", encoded.size(), kReportN); + } + + reportSealedFormat("cas_ref_snap", FormatId::RefSnapshot, kReportN, + [](uint64_t n) { return encodeRefTableSnapshot(makeSyntheticSnapshot(n)); }); + reportSealedFormat("cas_part_manifest", FormatId::PartManifest, kReportN, + [](uint64_t n) { return encodePartManifest(makePartManifest(n)); }); + reportSealedFormat("cas_fold_seal", FormatId::FoldSeal, kReportN, + [](uint64_t n) { return encodeFoldSeal(makeFoldSeal(n)); }); + reportSealedFormat("cas_ref_catalog", FormatId::RefCatalog, kReportN, + [](uint64_t n) { return encodeRefCatalog(makeRefCatalog(n)); }); +} + +} + +/// Hand-written in place of `BENCHMARK_MAIN()` so `--report_format_caps` can dispatch to Step 3's +/// oracle BEFORE `benchmark::Initialize` ever sees argv -- keeping the byte/cap report in this same +/// binary without a second `main` or a separate gtest target, and without the report's args tripping +/// `ReportUnrecognizedArguments`. Absent that flag, behavior is exactly `BENCHMARK_MAIN()`'s. +int main(int argc, char ** argv) +{ + for (int i = 1; i < argc; ++i) + { + if (std::string_view(argv[i]) == "--report_format_caps") + { + reportFormatCaps(); + return 0; + } + } + + benchmark::Initialize(&argc, argv); + if (benchmark::ReportUnrecognizedArguments(argc, argv)) + return 1; + benchmark::RunSpecifiedBenchmarks(); + return 0; +} diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index df987e1325ac..0f1a610359aa 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -474,7 +474,7 @@ inline String encodeMinimalGcState(uint64_t round) } /// Inject condemned bookkeeping + gc/state directly (bypassing a real GC round) so a test can seed the -/// GC ledger's condemned state at an arbitrary round. Retired-in-snapshot: the condemned entries are +/// GC ledger's condemned state at an arbitrary round. The condemned entries are /// seeded the way a real round leaves them — as `RunMarker::Condemned` sentinel rows inside an adopted fold seal's /// shard run (there is no separate retired-list object). A synthetic +edge/-edge pair nets each blob to /// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `RunMarker::Condemned` row. @@ -542,7 +542,7 @@ inline void injectRetire( backend.putOverwrite(layout.gcStateKey(), state, head.token); } -/// Adopt a fold seal carrying a given per-gc-shard `condemned_summary` (retired-in-snapshot T4) and point +/// Adopt a fold seal carrying a given per-gc-shard `condemned_summary` and point /// gc/state at it (snap_generation / snap_attempt / gc_shards), bypassing a real GC round. If a seal /// already exists at (generation, attempt) it is overwritten with the new summary (its other fields are /// preserved); otherwise a fresh minimal seal is created. Read-modify-CAS on gc/state preserves the lease. @@ -620,7 +620,7 @@ inline bool runRoundsUntilAbsent( } /// The CURRENT condemned entries for `shard`, read from the adopted fold seal's `blob_target_runs` -/// (retired-in-snapshot T4): the round no longer writes a separate retired-list object — condemned +///: the round no longer writes a separate retired-list object — condemned /// entries RIDE the source-edge run as `RunMarker::Condemned` sentinel rows at the zero-sentinel key. This reads /// the seal at (snap_generation, snap_attempt), opens every run for `shard`, and reconstructs the /// `RetiredEntry` shape (hash from the run key, the rest from the decoded `CondemnedRow`). Empty when @@ -669,8 +669,7 @@ inline std::vector currentRetiredSet( } /// True iff ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row — the ack-floor deletion -/// pipeline is in flight while this is true (retired-in-snapshot T4 replacement for the old -/// "iterate gc/state.retired_refs" probe). `gc_shards` is read from gc/state when 0 is passed. +/// pipeline is in flight while this is true. `gc_shards` is read from gc/state when 0 is passed. inline bool anyCondemnedInSeal( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, uint64_t gc_shards = 0) { diff --git a/src/Disks/tests/gtest_ca_wiring.cpp b/src/Disks/tests/gtest_ca_wiring.cpp index 9d12f9ccb759..6d34e444d603 100644 --- a/src/Disks/tests/gtest_ca_wiring.cpp +++ b/src/Disks/tests/gtest_ca_wiring.cpp @@ -1903,7 +1903,7 @@ TEST(CASWiringExchange, AdoptIntoADetachedTargetPublishesADetachedRefAndNoLiveRe EXPECT_TRUE(storage->existsFile(detached_tmp_path + "/p.proj/data.bin")); EXPECT_TRUE(storage->existsFile(detached_tmp_path + "/uuid.txt")); - /// Finalization, unchanged by this task: `IMergeTreeDataPart::renameTo(detached/)` is a + /// Finalization: `IMergeTreeDataPart::renameTo(detached/)` is a /// moveDirectory of the staged dir to its final detached name, which on a content-addressed disk is /// a ref repoint WITHIN the same namespace -- the same shape the active path's /// `renameTempPartAndReplace` uses, and the reason the relinked detached part needs no new @@ -2825,7 +2825,7 @@ DB::Cas::PoolPtr openResurrectStore(std::shared_ptr & } /// Condemn (kind=Blob, hash, token) by seeding gc/state + a per-shard retired set (the durable GC ledger -/// shape — RetiredEntry, exact-token delete, unchanged by this task) AND condemning the per-hash freshness +/// shape — `RetiredEntry`, exact-token delete) AND condemning the per-hash freshness /// meta, which is what the writer's condemned decision ACTUALLY point-reads (spec §meta-protocols v3). /// Bumps the round so the retirement is a fresh one; leaves the object itself in place (condemn, NOT delete). void seedCondemnBlobToken(DB::Cas::Pool & store, const DB::UInt128 & hash, diff --git a/src/Disks/tests/gtest_cas_backend_generation.cpp b/src/Disks/tests/gtest_cas_backend_generation.cpp index b6f9502c45fa..04da7d8aba63 100644 --- a/src/Disks/tests/gtest_cas_backend_generation.cpp +++ b/src/Disks/tests/gtest_cas_backend_generation.cpp @@ -151,7 +151,7 @@ TEST(CASBackendGeneration, NativeHeadUsesNativeTokenMetadataApi) ASSERT_EQ(b->putIfAbsent("p/native-head/key", "v1").outcome, PutOutcome::Done); - /// putIfAbsent's own HEAD-fallback stamping path calls the ordinary API (untouched by this task); + /// `putIfAbsent`'s HEAD-fallback stamping path calls the ordinary API; /// reset the counters so only nativeHead's call, below, is observed. storage->ordinary_calls = 0; storage->native_calls = 0; diff --git a/src/Disks/tests/gtest_cas_blob_digest.cpp b/src/Disks/tests/gtest_cas_blob_digest.cpp index 9a87f08ccda2..b6f8c9f3f972 100644 --- a/src/Disks/tests/gtest_cas_blob_digest.cpp +++ b/src/Disks/tests/gtest_cas_blob_digest.cpp @@ -1,7 +1,7 @@ #include -/// CAS pluggable-blob-hash Phase 2, Task 1: `BlobDigest` (the pool-scoped variable-length content digest, ADDITIVE-ONLY -- no -/// existing `UInt128 blob_hash` field is migrated in this task) + the ONE `PoolMeta`-scoped +/// `BlobDigest` is the pool-scoped variable-length content digest. It is additive: existing +/// `UInt128 blob_hash` fields retain their representation. The `PoolMeta`-scoped /// `DigestCodec` all digest<->hex/bytes conversion must route through. /// /// THE KEY GATE (`ShardOfBitIdenticalToOldHighBitsOver200RandomValues` below): `DigestCodec`'s diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 25cd9577fe76..98bd9ef57ac3 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -172,6 +172,41 @@ TEST(CASBlobEnvelopeFormat, MandatoryWorstCaseBoundary) << "ref budget reachable through the real encoder at the default header length"; } +/// The half a `static_assert` cannot do. The compile-time bound proves the FORMULA fits under the +/// floor; it cannot notice a formula that understates the encoder — shrink any component and the +/// assert only grows happier. So this reconstructs the same number from bytes the real encoder +/// produced, and the only quantity it borrows is the version field's type width: +/// +/// what the encoder wrote at max-width values, with an empty ref +/// + the digits the version field did NOT use at this generation +/// == the mandatory worst case +/// +/// Every other field in the fixture is already at its type maximum, so nothing else is missing from +/// the measured side. An understated key cost, or a shrunken `kMaxU32DecimalLen`, moves the formula +/// without moving the encoder and lands here. +TEST(CASBlobEnvelopeFormat, WorstCaseFormulaMatchesTheEncoder) +{ + EnvelopeHeader h = maxReachableHeader(""); + const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen)); + /// The mandatory shape is everything up to and including the closing brace, plus the newline the + /// encoder reserves at the last byte; the padding between them is the ref budget this measures. + const size_t json_len = head.find_last_not_of(' ', kMinBlobHeaderLen - 2) + 1; + const size_t mandatory_at_current_version = json_len + 1; /// + the reserved '\n' + + size_t version_digits = 0; + for (uint32_t v = currentCompatibilityVersion(); ; v /= 10) + { + ++version_digits; + if (v < 10) + break; + } + const size_t unused_version_digits = std::numeric_limits::digits10 + 1 - version_digits; + + EXPECT_EQ(mandatory_at_current_version + unused_version_digits, mandatory_descriptor_worst_case) + << "the formula and the encoder disagree about the mandatory descriptor: encoder wrote " + << mandatory_at_current_version << " bytes at a " << version_digits << "-digit version"; +} + TEST(CASBlobEnvelopeFormat, CriticalKeyDescriptorStillFitsAtDefaultLength) { /// The test-only `!x` critical key is written BEFORE `ref`; even at max-reachable field values diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index 45a075e50d3d..7d7808d70ecc 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -71,7 +71,7 @@ TEST(CASEncodingPins, RefLogTxnAllOpKinds) /// not quote/newline/control bytes/U+2028, so `ref_name` -- the only free-form string `RefOp` /// still carries now that `payload` is gone -- exercises quote, newline, a bare control byte, /// and the three-byte U+2028 sequence. Backslash escaping is pinned separately, over an - /// unrestricted string, by `gtest_cas_json_writer.cpp`'s `CASJsonWriterEscaping` suite. + /// unrestricted string, by the JSON-writer escaping suite. set_published_at.ref_name = String("20260101_0_1_1_1\"c\nd") + "\x01" "e" + "\xE2\x80\xA8" "f"; set_published_at.expected_manifest_ref = ManifestRef{1, 2, 3}; set_published_at.published_at_ms = 1234; diff --git a/src/Disks/tests/gtest_cas_fence_generation.cpp b/src/Disks/tests/gtest_cas_fence_generation.cpp index be621ddd9dd0..dcc1f7b3c4b9 100644 --- a/src/Disks/tests/gtest_cas_fence_generation.cpp +++ b/src/Disks/tests/gtest_cas_fence_generation.cpp @@ -327,7 +327,7 @@ TEST(CASFenceGeneration, PlainObjectRemoveAbortsWhenFenceTripsBetweenAdmissionAn store->removeNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "victim"); }); - /// The durable delete never ran -- the object survives (reads are not fence-gated by this task). + /// The durable delete never ran, so the object survives; reads are not fence-gated. const auto still_there = store->getNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "victim"); ASSERT_TRUE(still_there.has_value()); EXPECT_EQ(*still_there, "still here"); diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 55f7247d87b3..8109fc851605 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -32,10 +32,9 @@ namespace DB::ErrorCodes /// /// Reachability is a property of the WHOLE POOL. A blob is unreferenced only if no namespace anywhere /// owns an edge to it, so a round that deletes one is asserting something about every namespace at -/// once -- including the ones it never looked at. Task 7 made the per-namespace half of that assertion -/// cheap and exact: one `GET` at the cursor's arithmetic successor, absent means end-of-stream. Task 8 -/// made a namespace that could NOT be walked say so durably. What neither can supply is the SET those -/// proofs have to cover, and that is what this task is about. +/// once -- including the ones it never looked at. A `GET` at the cursor's arithmetic successor makes +/// the per-namespace proof cheap and exact, and a namespace that cannot be walked reports that fact +/// durably. Neither supplies the SET those proofs have to cover. /// /// So the gate has three terms, and a round destroys only when all three are clear: /// @@ -1423,7 +1422,7 @@ TEST(CASGCFrontierGate, TheHandOffReclaimIsInertUnderSuppression) EXPECT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) << "the superseded generation's prefix survives a suppressed round intact"; - /// AND THE OPPORTUNITY IS CONSUMED, NOT DEFERRED -- the one place in this task where the gate + /// AND THE OPPORTUNITY IS CONSUMED, NOT DEFERRED -- the gate /// costs something permanent, so it is asserted here rather than left to be discovered later. /// /// The hand-off is a one-shot DIFFERENCE: it compares the PARENT seal's runs against the new diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index 9cc892165f14..ce6b09099487 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -24,8 +24,8 @@ /// DURABLE HOLDS (spec 2026-07-27 "ref chain complete cut" §5). /// /// A namespace whose ref-log walk meets an IMPOSSIBLE shape stops there, and that stop has to survive -/// the round. Before this task the stop was a single bit — `classification == CoverageClass::Clamped` — -/// and everything that explained it (what went wrong, and exactly WHERE) lived in a log line and an +/// the round. A classification alone cannot preserve the cause and position of the stop; without +/// durable hold evidence, that information lives only in a log line and an /// in-memory anomaly, both gone by the next round. That is not enough for three separate reasons: /// /// * the next round could not RETRY the exact position, so a hold only survived while the round's @@ -298,8 +298,7 @@ std::vector> illFormedSealsTheEncoderMustRe "durable", clamped_without_hold); /// The closed set is now the enum's declared values, so only an explicit cast reaches outside it. - /// 4 is the sharpest value to plant: it was the wire value for Clamped before this task's dense - /// renumbering, and under the new table it is simply out of range. + /// 4 is the sharpest value to plant: it is outside the closed wire vocabulary. CasFoldSeal classification_retired_wire_value = cleanSeal("ns/0"); fixtureCoverage(classification_retired_wire_value, "ns/0").classification = static_cast(4); diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index b54b5e71d648..aa0a8f914972 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -129,7 +129,7 @@ GcState readState(InMemoryBackend & b, const Pool & s) return decodeGcState(got->bytes); } -/// Whether ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row (retired-in-snapshot T4: the +/// Whether ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row (the /// retired state rides the snapshot run, not a separate retired-list object) — the ack-floor deletion /// pipeline is still in flight while this is true. bool anyRetiredPending(InMemoryBackend & b, const Pool & s) @@ -540,7 +540,7 @@ TEST(CASGCRound, PublishDropReclaimsBlobAndManifestToFixpoint) EXPECT_FALSE(blobExists(*backend, store->layout(), DB::UInt128(1))); } -/// retired-in-snapshot T4: after a round condemns one blob, the ADOPTED fold seal's per-shard +/// After a round condemns one blob, the ADOPTED fold seal's per-shard /// condemned_summary reflects it (condemned_total == 1, pending_total == 0) — distilled zero-I/O from the /// RunMarker::Condemned rows the fold sealed into the snapshot run. TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) diff --git a/src/Disks/tests/gtest_cas_gc_round_defer.cpp b/src/Disks/tests/gtest_cas_gc_round_defer.cpp index a33816a8bac2..adbccce614ec 100644 --- a/src/Disks/tests/gtest_cas_gc_round_defer.cpp +++ b/src/Disks/tests/gtest_cas_gc_round_defer.cpp @@ -41,7 +41,7 @@ TEST(CASGCRoundDefer, PredicateTruthTable) EXPECT_FALSE(shouldDeferRound(2, false, 8, 3, 8)); // bound reached => force fold } -/// graduationDue (retired-in-snapshot T4): read ZERO-I/O from the adopted seal's condemned_summary. An +/// `graduationDue` reads ZERO-I/O from the adopted seal's `condemned_summary`. An /// entry whose oldest non-pending condemn round crosses current_round forces it true; a delete_pending /// entry forces it true regardless of the round; otherwise false. TEST(CASGCRoundDefer, GraduationDueDetectsDuePendingAndRoundCrossing) @@ -584,9 +584,9 @@ TEST(CASGCRoundDefer, DueGraduationIsSoleFoldTriggerAtHighThreshold) writeBlobBody(*backend, layout, blob); - /// Seed the adopted fold seal's condemned_summary with B already `delete_pending` (pending_total = 1), - /// mirroring `CASGCRoundDefer.GraduationDueDetectsDuePendingAndRoundCrossing`. Retired-in-snapshot - /// (T4): graduationDue reads this summary ZERO-I/O off the adopted seal — a delete_pending entry forces + /// Seed the adopted fold seal's condemned_summary with B already `delete_pending` (pending_total = 1). + /// `graduationDue` + /// reads this summary ZERO-I/O from the adopted seal — a `delete_pending` entry forces /// it true regardless of the round. At `gc_fold_threshold = 1000` a real condemn -> graduate pipeline of /// `runRegularRound` calls is not usable to set this up: every round before graduation would ITSELF /// defer (nothing due yet, and changed_shards never nears 1000), so the due-pending summary is injected diff --git a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp index bfb62f817ecb..b8f803e514f8 100644 --- a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp +++ b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp @@ -500,8 +500,8 @@ TEST(CASNamespaceFileDiskProfile, TheLifeResolutionIsPaidOncePerTableOpen) } -/// THE REMOVAL PATHS MUST NOT CREATE A NAMESPACE — the case that regressed silently in this task's first -/// round, so it is pinned on the catalog rather than on the file outcome. +/// THE REMOVAL PATHS MUST NOT CREATE A NAMESPACE: the catalog, rather than the file outcome, proves +/// that invariant. /// /// Why the file outcome cannot pin it: `unlinkFile`/`removeRecursive` against a never-opened table /// answer "absent" both before and after the defect, because a freshly minted namespace has no files diff --git a/src/Disks/tests/gtest_cas_namespace_life_id.cpp b/src/Disks/tests/gtest_cas_namespace_life_id.cpp index 9c2156d52aea..65c116d89f07 100644 --- a/src/Disks/tests/gtest_cas_namespace_life_id.cpp +++ b/src/Disks/tests/gtest_cas_namespace_life_id.cpp @@ -205,8 +205,8 @@ TEST(CASNamespaceLifeIdDeathTest, ZeroIncarnationIsUnconstructibleAborts) } #endif -/// Generation-5 namespace-bearing keys are outside the generation-6 parser roots altogether. Pool -/// admission rejects their generation before any listed-key parser is involved. +/// Namespace-bearing keys outside the opaque-life layout are rejected before any listed-key parser is +/// involved. TEST(CASNamespaceLifeId, GenerationFiveNamespaceBearingKeysAreOutsideTheFinalGrammar) { Layout l("p"); @@ -312,8 +312,8 @@ TEST(CASNamespaceLifeId, NamespaceFileKeysCarryTheIncarnationSegment) EXPECT_FALSE(l.namespaceFileKey(second, "format_version.txt").starts_with(l.namespaceFilesPrefix(life))); } -/// Generation-5 namespace-bearing file keys are outside the final parser root. Malformed ids under the -/// final state root are corruption and name the offending key. +/// Namespace-bearing file keys outside the opaque-life layout are rejected. Malformed life ids under +/// the state root are corruption and name the offending key. TEST(CASNamespaceLifeId, NamespaceFileParserRefusesLegacyAndMalformedIncarnations) { Layout l("p"); @@ -373,8 +373,7 @@ TEST(CASNamespaceLifeId, PhysicalFileKeysIgnoreLogicalNamespaceSpelling) EXPECT_EQ(parsed->relative_name, nested_name); } -/// The "cannot compile" half of spec §9 r9-5 #3: after this task there is no way to reach a ref-layer -/// key from a namespace alone, so dropping the incarnation is a compile error rather than an aliasing +/// A ref-layer key cannot be reached from a namespace alone, so dropping the incarnation is a compile error rather than an aliasing /// bug. Each helper is asserted twice -- the namespace-only form absent, the incarnation form present. TEST(CASNamespaceLifeId, NamespaceOnlyKeyHelpersDoNotExist) { @@ -413,8 +412,8 @@ TEST(CASNamespaceLifeId, NamespaceLifeIdAndRootNamespaceDoNotInterconvert) SUCCEED(); } -/// The out-of-scope fences, and they are POSITIVE on purpose: Constraint 12 keeps loose mountpoint -/// objects and part manifests on the identity they have today, so this task must NOT have qualified +/// The out-of-scope fences are POSITIVE on purpose: loose mountpoint objects and part manifests keep +/// their namespace identity, so they must NOT be qualified /// them. If a negative here fails, someone added a life-scoped overload to a family the amendment /// explicitly excluded; if a positive fails, someone removed the un-scoped one those callers use. TEST(CASNamespaceLifeId, MountpointObjectsAndManifestsStayUnqualified) diff --git a/src/Disks/tests/gtest_cas_parallel_commit.cpp b/src/Disks/tests/gtest_cas_parallel_commit.cpp index 7ae5ab676712..8cdb007d6623 100644 --- a/src/Disks/tests/gtest_cas_parallel_commit.cpp +++ b/src/Disks/tests/gtest_cas_parallel_commit.cpp @@ -160,7 +160,7 @@ namespace /// Fixture for the `CasCommitRollback` suite: wraps a real `ContentAddressedMetadataStorage` and /// drives ordinary `ContentAddressedTransaction`s through disk paths, so the fault seams under test /// (`ContentAddressedMetadataStorage::armPromoteFailureForTest`/`setAfterPromoteHookForTest`, the -/// minimal test-only hooks this task adds) fire from the SAME `publishStaging` call path production +/// minimal test-only hooks) fire from the SAME `publishStaging` call path production /// `commit()` uses -- unlike `CaWiringFixture` above, which pokes the bare pool primitives directly. /// Every part in one fixture instance shares ONE fixed table uuid (and therefore one `RootNamespace`), /// matching every test's single `fx.ns()`. diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index e966b87a332a..dd6eab02c003 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -30,8 +30,7 @@ void expectThrowsCode(int expected_code, F && fn) } } -/// One Blob + one Inline entry, matching the plan's §text-shape illustration verbatim (codecs-v3 -/// phase 6): deliberately NOT path-sorted on input, so the round trip also exercises canonical +/// One Blob + one Inline entry, deliberately NOT path-sorted on input, so the round trip also exercises canonical /// path-order encoding. PartManifest sample() { @@ -69,7 +68,7 @@ TEST(CASFormatBattery, PartManifest) /// stays self-consistent with whatever sample() produces, now that decode verifies payload_digest. const String golden = currentFormatHeader("cas_part_manifest") + - "{\"epoch\":\"5\",\"build\":\"15\",\"ord\":1,\"root_namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. + "{\"epoch\":\"5\",\"build\":\"15\",\"ord\":1,\"namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. "{\"path\":\"a/b.bin\",\"place\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\",\"size\":4096}\n" "{\"path\":\"c/small.txt\",\"place\":\"inline\",\"size\":12}\n" "{\"n\":2}\n" @@ -517,8 +516,7 @@ TEST(CASPartManifestFormat, InlineRecordSizeMismatchWithPayloadZoneBannerFailsCl expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } -/// ==== migrated from gtest_cas_manifest_codec.cpp (deleted in the phase-6 binary->text cutover, -/// Task 3): these exercise refMatchesBody/manifestNamespaceMatches/findEntry/entryRange, pure +/// ==== Migrated manifest helpers: these exercise `refMatchesBody`, `manifestNamespaceMatches`, `findEntry`, and `entryRange`, pure /// functions carried over verbatim from the retired binary codec (untouched by the wire-shape /// migration) — reusing this file's own sample() fixture instead of reintroducing a second one. ==== diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index a74baf05fa61..6c7eb1d59667 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -439,15 +439,9 @@ TEST(CASPluggableHash, Sha256BlobSeenByCondemnSweepAndFsckNotSilentlySkipped) } /// ============================================================================================ -/// CAS pluggable-blob-hash Phase 2 Task 6 -- end-to-end sha256 WRITE path (in-memory; the real -/// wiring-level integration + soak is Task 7). -/// -/// Before this task, `PartWriteTxn`'s OWN write-path internals stayed a fixed 128-bit representation -/// downstream of the mint (`poolContentHash`/`PartWriteTxn::putBlob`'s `logical_hash`, the `deps` map key, the -/// event-log `object_hash` render, and `objectKey`) -- safe only because the disk-config factory guard -/// (`MetadataStorageFactory.cpp`) blocked any real sha256 pool from reaching `PartWriteTxn` at all (see the -/// Task 5 report and the "Task 6+" comments this task removes). Task 6 finishes those sites AND lifts -/// the guard in the SAME commit. This test drives a REAL `PartWriteTxn` (`putBlob` -> `stageManifest` -> +/// End-to-end SHA-256 write path. Every `PartWriteTxn` representation downstream of digest creation +/// must preserve the pool's variable-width digest; truncation would address a different blob. This +/// test drives a REAL `PartWriteTxn` (`putBlob` -> `stageManifest` -> /// `precommitAdd` -> `promote`) on a `Sha256` pool and asserts: /// 1. the blob lands under `blobs/sha256/<64-hex>` and the manifest entry's `blob_hash`, read back via /// `decodePartManifest`, is the FULL 32-byte digest (bytes beyond 16 are non-zero for a real sha256 @@ -510,7 +504,7 @@ TEST(CASPluggableHash, Sha256BuildWritesFullWidthDigestAndInlineEqualsBlob) /// THE CRUX (blob side): the blob body lands under the sha256-segmented path, addressed by the /// FULL 64-hex key -- `PartWriteTxn::putBlob`'s internal `logical_hash` must not have silently narrowed it - /// to a 32-hex (128-bit) key before this task. + /// to a 32-hex (128-bit) key. const String blob_key = store->layout().blobKey(id); EXPECT_NE(blob_key.find("/blobs/sha256/"), String::npos) << blob_key; ASSERT_TRUE(backend->head(blob_key).exists); @@ -545,7 +539,7 @@ TEST(CASPluggableHash, Sha256BuildWritesFullWidthDigestAndInlineEqualsBlob) /// validation at `foldManifestEdges` with refresh-on-miss. /// ============================================================================================ -/// spec §9.8 -- THE race regression this task exists to close. Each `Pool`'s `admitted_algos` cache +/// Each `Pool`'s `admitted_algos` cache /// is a MONOTONE snapshot seeded once at `Pool::open` and never re-read on its own; if node A admits /// a brand-new algo and publishes a manifest naming it, node B's stale cache must NOT fail the fold /// closed forever -- `foldManifestEdges` must refresh `_pool_meta` on the very first miss and accept diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index a458bbb52b6b..f2a3ec1d6570 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -71,8 +71,7 @@ String rawEntryLine(const String & ns, const String & state, const String & inc_ } /// Wraps `entry_lines` in the header/trailer a real `cas_ref_catalog` object carries. `v:1` always -/// passes the header gate (any version <= the build's `G_BUILD` does), matching the convention -/// `gtest_cas_fold_seal_format.cpp`'s `RejectsOutOfRangeNsCleanupState` uses for the same reason. +/// passes the header gate because any version <= the build's `G_BUILD` does. String rawCatalog(const std::vector & entry_lines) { String out = R"({"type":"cas_ref_catalog","v":1})" "\n"; @@ -618,11 +617,11 @@ TEST(CASRefCatalogFormat, RegistryRowIsControlStrictWithRawStorage) EXPECT_EQ(traits.object_cap, 256u * 1024u * 1024u); EXPECT_EQ(traits.line_cap, 4u * 1024u); EXPECT_EQ(traitsForType("cas_ref_catalog"), &traits); - /// Raw, so the key has no suffix: `Pool/CasRefCatalog.cpp` hands bytes to/from the backend - /// directly, bypassing `sealObject`/`openObject` because both are the identity under + /// Raw, so the key has no suffix: the catalog hands bytes directly to/from the backend, + /// bypassing `sealObject`/`openObject` because both are the identity under /// `CompressionPolicy::Never`. This line is the TRIPWIRE for that shortcut -- a policy flip to /// `Always` would silently write uncompressed bodies under a `.zst` key, which this assertion - /// catches first (see `CasRefCatalogFormat.h`'s comment on `encodeRefCatalog`). + /// catches first. EXPECT_EQ(storedSuffix(FormatId::RefCatalog), ""); EXPECT_EQ(traits.compression, CompressionPolicy::Never); } diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index 2cdc32ff1851..1afff867b905 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -51,9 +51,8 @@ extern const Event CASRefSnapshotPublishDispatched; /// single item whose own op count, or whose one op's encoded size, exceeds its cap fails ALONE; a /// neighbor co-batched into the same flush still commits. `ref_txn_max_ops` is checked exactly (the /// `build_ops` result's size), and the per-op cap is checked by encoding exactly one op at a time -- -/// no accumulation, matching the admission machinery this replaces. T9 (removal-class detection by -/// op inspection) and T10 (chunked flush across a whole-batch op-count overflow) extend this file; -/// this task adds only the per-item / per-op isolation tests and the canonical round-trip leg of +/// no accumulation, matching the admission machinery. The per-item / per-op isolation tests and +/// the canonical round-trip leg cover /// test 12 (the maximum legally-admissible normal-class transaction). /// /// The suite name is prefixed `RefWriter` so it is covered by the `RefWriter*` unit-test gate filter. diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index 13437baf17a8..a81339967eaa 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -242,9 +242,8 @@ TEST(CASRefContiguousAlloc, EpochChangeRestartsTheSequenceAtOne) } /// The read side is what makes INV-1 an invariant rather than a convention: a transaction whose id is -/// not the successor of `greatest_applied` is CORRUPTED_DATA, naming both ids. Before this task the -/// state machine checked strict increase only, so a stream with a hole applied cleanly and no reader -/// could tell a complete chain from a truncated one. +/// not the successor of `greatest_applied` is CORRUPTED_DATA, naming both ids. Strict increase alone +/// admits holes, so it cannot distinguish a complete chain from a truncated one. TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) { const String ns = "srv1/contig_density"; @@ -253,7 +252,7 @@ TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) RefTableState state = replay(DB::Cas::tests::minimalLiveSnapshot(ns, RefTxnId{kEpoch, 1}), {}); ASSERT_EQ(state.getGreatestApplied(), (RefTxnId{kEpoch, 1})); - /// Strictly greater, but skips {7,2}: admitted before this task, rejected now. + /// Strictly greater, but skips {7,2}: not the required successor. try { applyRefLogTxn(state, RefLogTxn{ns, RefTxnId{kEpoch, 3}, publishCommittedOps("r", ManifestRef{1, 1, 1}), std::nullopt}); diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index c265618996ee..3db61594d3ee 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -153,7 +153,7 @@ TEST(CASRefEpochSealFormat, EncodeRejectsSealTxnWithSecondNonSealOp) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefLogTxn(txn); }); } -/// Decode-side pin for the same op-count rule (review finding I1): `encodeRefLogTxn` can never +/// Decode-side pin for the same op-count rule: `encodeRefLogTxn` can never /// produce a 2-op seal body, so only a decode-only splice proves `decodeRefLogTxn` independently /// re-derives the rule rather than trusting whatever the encoder produced -- deleting the structural /// validator's call site inside `decodeRefLogTxn` would leave this the only failing test. @@ -213,7 +213,7 @@ TEST(CASRefEpochSealFormat, EncodeRejectsPrevEpochSealAtNonUnitSequence) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefLogTxn(txn); }); } -/// Decode-side pin for the sequence-1-only rule (review finding I1). `prev_epoch_seal`'s +/// Decode-side pin for the sequence-1-only rule. `prev_epoch_seal`'s /// writer_epoch (1) is strictly below the transaction's own (5), satisfying the I3 chain-direction /// rule, so this isolates the sequence-1 rule specifically rather than incidentally also tripping I3. TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) @@ -233,7 +233,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } -/// Well-formedness (review finding M2): a zero component inside `prev_epoch_seal` is rejected the +/// Well-formedness: a zero component inside `prev_epoch_seal` is rejected the /// same way a zero component in the primary `txn_id` is (`checkRefTxnIdNonzero`, shared code path). TEST(CASRefEpochSealFormat, EncodeRejectsPrevEpochSealWithZeroWriterEpoch) { @@ -276,7 +276,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } -/// Chain direction (review finding I3): a seal closing epoch E always has id `{E, T+1}`, and the +/// Chain direction: a seal closing epoch E always has id `{E, T+1}`, and the /// sequence-1 transaction in the next numeric epoch must name it. This remains context-free (a /// property of one transaction), so it belongs in the structural half; Tasks 2/6 walk this pointer /// backwards over untrusted decoded bodies and must not have to re-derive the rule themselves. @@ -335,7 +335,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealSkippingImmediateEpochSpli expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } -/// Decode-side pin for the chain-direction rule (review finding I3): the encoder's own check would +/// Decode-side pin for the chain-direction rule: the encoder's own check would /// refuse to produce this shape (the two Encode* tests above pin that direction), so a splice into an /// otherwise-valid sequence-1 body proves decode re-derives the rule independently. TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealPointingAtSameOrFutureEpochSpliced) @@ -382,7 +382,7 @@ TEST(CASRefEpochSealFormat, ContextualRejectsPrevEpochSealWhenForbidden) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { validateEpochSealGrammarContextual(txn, /*life_epoch=*/5); }); } -/// codex r2 finding 2: "genesis" is per-namespace. A namespace first born at global epoch 5 (not +/// "Genesis" is per-namespace. A namespace first born at global epoch 5 (not /// epoch 1) appends {5, 1} with NO prev_epoch_seal -- that IS its genesis, not a transition. TEST(CASRefEpochSealFormat, ContextualAllowsGenesisBirthAboveEpochOneWithoutPrevEpochSeal) { @@ -393,7 +393,7 @@ TEST(CASRefEpochSealFormat, ContextualAllowsGenesisBirthAboveEpochOneWithoutPrev EXPECT_NO_THROW(validateEpochSealGrammarContextual(txn, /*life_epoch=*/5)); } -/// Review finding I2: the `ref_sequence != 1` early return is load-bearing for Task 4's encode call +/// The `ref_sequence != 1` early return is load-bearing for the encode call /// site, which calls this on every txn it mints, including ordinary sequence->=2 transactions in a /// post-transition epoch that legitimately carry no `prev_epoch_seal`. Pinned on both sides of the /// life_epoch relation to prove the early return fires regardless of it. @@ -418,7 +418,7 @@ TEST(CASRefEpochSealFormat, ContextualPassesThroughNonSequenceOneAtOrBelowLifeEp } /// =================================================================================== -/// Criticality of the prev_epoch_seal wire fields (review finding M4) +/// Criticality of the `prev_epoch_seal` wire fields /// =================================================================================== /// `!prev_epoch`/`!prev_seq` are `!`-prefixed CRITICAL keys: `prev_epoch_seal` is INV-2 chain evidence, and a diff --git a/src/Disks/tests/gtest_cas_ref_install_safety.cpp b/src/Disks/tests/gtest_cas_ref_install_safety.cpp index 3c3c58b480d8..b44ce523465c 100644 --- a/src/Disks/tests/gtest_cas_ref_install_safety.cpp +++ b/src/Disks/tests/gtest_cas_ref_install_safety.cpp @@ -107,7 +107,7 @@ PoolPtr openPoolSingleAttempt(const BackendPtr & backend) /// With `openPoolFenceControlled`'s budget below that margin is 100 + 100 = 200 ms, so a 100 ms /// remaining lease sits BETWEEN them: the flush is admitted and then its very first pre-attempt gate /// refuses. That is -/// exactly the production shape this task is about (a lease too short to start a write, not a lost +/// exactly the production shape of a lease too short to start a write, not a lost /// one), and it needs no fault injection at all -- which is the point: nothing is sent. constexpr uint64_t FENCE_DEADLINE_HEALTHY_MS = 30000; constexpr uint64_t FENCE_DEADLINE_REFUSES_ATTEMPT_MS = 100; @@ -330,8 +330,8 @@ TEST(CASRefInstallSafety, PreAttemptRefusalDoesNotWedgeTheLane) << "the refused drop must not have taken effect"; /// The availability half of the claim: the lane is usable the moment the lease is healthy again -- - /// no remount, no wedge resolution, nothing to clear. Before this task the same sequence left a - /// wedge over a key that was never written, and this append would have failed forever. + /// no remount, no wedge resolution, nothing to clear. A refused pre-attempt never writes a key, + /// so it cannot leave a wedge to block this append. store->setMountDeadline(FENCE_DEADLINE_HEALTHY_MS); store->dropRef(ns, "part_a"); EXPECT_FALSE(store->resolveRef(ns, "part_a", /*allow_stale=*/false).has_value()) diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index cef428ab1d87..1ef57e5005d5 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -202,7 +202,7 @@ TEST(CASRefCodec, RoundTripSetPublishedAt) } /// No-tolerance decode pin: the `"pl"` (payload) field was removed from the -/// ref-op wire in stage-1 T12. Although the retired `set_payload` op WORD is already rejected by +/// ref-op wire. Although the retired `set_payload` op WORD is already rejected by /// `refOpKindFromWireWord`, the generic op-record reader reads all field keys before switching on kind, so a /// `"pl"` field paired with a still-recognized op word would otherwise be `skipUnknown`'d. It is a /// removed field, not a genuinely-unknown one: decoding an op record that still carries `"pl"` must FAIL diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 1c532c5add83..942fc170e139 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -126,8 +126,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnFieldPair) [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); } -/// No-tolerance decode pin (codex round-2, finding 3): the `"pl"` (payload) field was removed from the -/// committed-row wire in stage-1 T12. It is NOT a genuinely-unknown future field the tolerant reader may +/// No-tolerance decode pin: the `"pl"` (payload) field is not a genuinely-unknown future field the tolerant reader may /// skip -- silently discarding a persisted payload would lose data -- so decoding a committed row that /// still carries `"pl"` must FAIL with `CORRUPTED_DATA` naming the removed field, not `skipUnknown` it. TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp index ca34681f7b93..341484e8f84c 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp @@ -180,7 +180,7 @@ TEST(CASRefSnapshotPublishOrdering, AdoptionHappensLastAndOnlyAfterBothDurableEf /// 3. `NeedsRecovery` ("Poisoned") lane: recovery precedes any snapshot publication /// --------------------------------------------------------------------------------------------- -/// `Poisoned` is this task's plan's name for what the code spells `RefLaneState::NeedsRecovery` -- the +/// `RefLaneState::NeedsRecovery` is the state for a transaction known durable but not installable in the cache -- the /// state the header documents as "a transaction is known durable but cannot be installed in this cache /// ... a hard write and certification fence until replay completes". Recorded here as the vocabulary /// correction for later tasks: there is no state literally named `Poisoned` anywhere in `CasRefLedger`. diff --git a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp index ace02d24836e..c696323b3d80 100644 --- a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp +++ b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp @@ -802,8 +802,8 @@ TEST(CASRefWedgeEveryAttempt, SuccessorSealAtTheWedgedKeyRejectsConclusivelyAndS } /// The wire round trip of the same rule, driven from the OTHER producer of `last_epoch_seal`: -/// recovery's CAS-walk (Task 6), stood in for here by its test seam. The point is the encode call -/// site, which is this task's. +/// recovery's CAS-walk, represented here by its test seam. The point is the encode call +/// site. TEST(CASRefWedgeEveryAttempt, OrdinaryFirstAppendAfterASealedTransitionCarriesTheExactPrevEpochSeal) { auto backend = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index 48414871269b..c4e173fb96c5 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -1711,8 +1711,7 @@ TEST(CASRefWriterAppendLane, WedgedRefLaneCountTracksExactlyTheWedgedTableThroug /// The reaction is now the mount's, not the table's [review I5]: a foreign object at a key that /// mount-lease exclusivity says is exclusively ours contradicts the exclusivity itself, so the append /// site routes through `reportImpossibleInterference` exactly as the wedge-resolve site does -- fence -/// closed, remount scheduled. Before this task it failed closed and stayed closed, blocking the table -/// until somebody remounted by hand. So there are two separate scopes to keep straight, and this test +/// closed, remount scheduled. The fence is released only after remount, so there are two separate scopes to keep straight, and this test /// pins both: /// the FENCE is mount-wide -- while it is closed EVERY lane is refused, including untouched ones; /// the DAMAGE is per-namespace -- a real remount replaces both immutable runtimes, then recovery of @@ -3947,7 +3946,7 @@ TEST(CASRefWriterNamespaceRemoval, RemovalPublishesTerminalLogWithoutTerminalSna EXPECT_EQ(terminal_logs, 1u); } -/// Review fix (prerequisite to this task's dropNamespace rewiring): `flushRefBatch`'s per-item +/// `flushRefBatch`'s per-item /// validation previously previewed each op as its OWN single-op trial transaction, so a /// whole-transaction-shape rule ("remove_namespace must be the FINAL op") trivially passed on every /// singleton slice regardless of an item's REAL combined shape -- a malformed item would only have diff --git a/utils/c++expr b/utils/c++expr index 08ddff2e1363..54c44c8ecdbd 100755 --- a/utils/c++expr +++ b/utils/c++expr @@ -235,7 +235,9 @@ size_t max_tests = $BENCHMARK_TESTS; size_t max_steps = $BENCHMARK_STEPS; $GLOBAL -int work(int thread_id = 0) { +/// Internal linkage: the ClickHouse build enables -Werror,-Wmissing-prototypes, which rejects a +/// definition of an external function that has no preceding declaration. +static int work(int thread_id = 0) { (void)thread_id; try { EOF From b55e44595e652e03f820791fcaa2abdae846e986 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:03:45 +0200 Subject: [PATCH 6/8] cas: reuse the JSON object/row reader across a stream's rows; decode speedup Full before/after measurement of the wire-key cut found decode of four of the five formats barely slower, with `cas_fold_seal` the exception (its short strings make the longer keys dominate). Chasing that, the JSON object reader and the per-format row reader are now reused across a stream's rows instead of rebuilt for each one, cutting decode time 57-81% (53-79% net of the key-length cost). A separate copy-free string-read attempt was measured at a 6-7% regression on `cas_ref_catalog` and is not included here. Also lets the full stateless test suite run locally: `functional_tests.py` turns on verbose output for the dataset-attach step (so a `DNS_ERROR` that only fires outside CI doesn't get swallowed and misread as a Kafka failure downstream) and extends the "skip stateful tests when running locally" guard to a local run with no test selector at all. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- ci/jobs/functional_tests.py | 8 ++ ci/jobs/scripts/clickhouse_proc.py | 4 +- .../Formats/CasBlobEnvelopeFormat.cpp | 25 ++-- .../Formats/CasBlobEnvelopeFormat.h | 9 +- .../Formats/CasFoldSealFormat.cpp | 12 +- .../Formats/CasGcOutcomesFormat.cpp | 11 +- .../Formats/CasPartManifestFormat.cpp | 12 +- .../Formats/CasRecordStreamFormat.cpp | 25 ++-- .../Formats/CasRecordStreamFormat.h | 7 ++ .../Formats/CasRefCatalogFormat.cpp | 30 +++-- .../Formats/CasRefCkptFormat.cpp | 5 +- .../Formats/CasRefLogFormat.cpp | 11 +- .../Formats/CasRefSnapshotFormat.cpp | 12 +- .../Formats/CasTextFormat.cpp | 118 +++++++++++++----- .../ContentAddressed/Formats/CasTextFormat.h | 28 ++++- .../ContentAddressed/Formats/CasWireVocab.cpp | 21 ++-- src/Disks/tests/cas_format_test_battery.h | 15 ++- .../tests/gtest_cas_blob_envelope_format.cpp | 40 ++++-- src/Disks/tests/gtest_cas_format.cpp | 15 +++ 19 files changed, 287 insertions(+), 121 deletions(-) diff --git a/ci/jobs/functional_tests.py b/ci/jobs/functional_tests.py index b73f6f308211..227bc50bfc17 100644 --- a/ci/jobs/functional_tests.py +++ b/ci/jobs/functional_tests.py @@ -506,6 +506,14 @@ def main(): # for local run check if stateful tests are present to skip prepare_stateful_data and start faster if not has_stateful_tests = True + if info.is_local_run and not tests: + # A local run of the WHOLE suite cannot prepare the stateful datasets: `create.sql` attaches + # them from a web disk on `dockerhub-proxy.dockerhub-proxy-zone`, which resolves only inside + # CI, so the step dies with DNS_ERROR before a single test runs. Skipping it lets the + # stateless suite run locally; the tests that genuinely need `test.hits`/`test.visits` fail + # and are triaged as environment rather than taking the whole job with them. + print("Local full-suite run: skipping stateful data preparation (datasets are CI-hosted)") + has_stateful_tests = False if tests and info.is_local_run: from glob import glob diff --git a/ci/jobs/scripts/clickhouse_proc.py b/ci/jobs/scripts/clickhouse_proc.py index d4ce6313bf24..4a7740603c3e 100644 --- a/ci/jobs/scripts/clickhouse_proc.py +++ b/ci/jobs/scripts/clickhouse_proc.py @@ -839,7 +839,9 @@ def prepare_stateful_data(self, with_s3_storage, is_db_replicated): command = bootstrap_vars + command if with_s3_storage: command = "USE_S3_STORAGE_FOR_MERGE_TREE=1\n" + command - return Shell.check(command) + # verbose: this step loads the stateful datasets and it is the only place in the job + # that can fail without printing anything at all, which is exactly what happened. + return Shell.check(command, verbose=True) def insert_system_zookeeper_config(self): for _ in range(10): diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index ff9fda182b26..523912c519c4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -35,6 +35,8 @@ namespace EnvelopeWire constexpr WireKey op{"op"}; constexpr WireKey chver{"chver"}; constexpr WireKey ref{"ref"}; + /// Not a field this build understands: written only to exercise the reader's `!`-key policy. + constexpr WireKey unknown_critical{"!x"}; } constexpr EnumWireTable kProvenanceOpWords{{{ @@ -165,7 +167,8 @@ ProvenanceOp provenanceOpFromWireWord(std::string_view w) return kProvenanceOpWords.fromWord(w, "CAS blob envelope"); } -String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) +String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len, + std::optional version_override) { if (header.kind != ObjectKind::Blob) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -177,22 +180,20 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { CasJsonWriter buf(256); bool first = true; - writeKey(buf, EnvelopeWire::type, first); writeStringValue(buf, kBlobType); - writeKey(buf, EnvelopeWire::version, first); writeIntText(currentCompatibilityVersion(), buf); - writeKey(buf, EnvelopeWire::tag, first); writeHex128Value(buf, header.incarnation_tag); - writeKey(buf, EnvelopeWire::build, first); writeHex128Value(buf, header.build_id); + writeStringField(buf, EnvelopeWire::type, kBlobType, first); + writeNumberField(buf, EnvelopeWire::version, version_override.value_or(currentCompatibilityVersion()), first); + writeHex128Field(buf, EnvelopeWire::tag, header.incarnation_tag, first); + writeHex128Field(buf, EnvelopeWire::build, header.build_id, first); if (header.provenance) { - writeKey(buf, EnvelopeWire::time_ms, first); writeIntText(header.provenance->created_at_ms, buf); - writeKey(buf, EnvelopeWire::creator, first); writeHex128Value(buf, header.provenance->creator_server_id); - writeKey(buf, EnvelopeWire::op, first); writeStringValue(buf, provenanceOpToWireWord(header.provenance->op)); - writeKey(buf, EnvelopeWire::chver, first); writeIntText(header.provenance->ch_version, buf); + writeNumberField(buf, EnvelopeWire::time_ms, header.provenance->created_at_ms, first); + writeHex128Field(buf, EnvelopeWire::creator, header.provenance->creator_server_id, first); + writeStringField(buf, EnvelopeWire::op, provenanceOpToWireWord(header.provenance->op), first); + writeNumberField(buf, EnvelopeWire::chver, header.provenance->ch_version, first); } /// Test-only critical extension: an unknown `!`-key BEFORE `ref`. if (header.emit_unknown_critical_key) - { - writeKey(buf, "!x", first); writeStringValue(buf, "1"); - } + writeStringField(buf, EnvelopeWire::unknown_critical, "1", first); json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":1,...,"chver":26006001 (no ref, no closing brace) } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index e1a9a228433d..e968bbec5dbc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -94,7 +94,14 @@ struct EnvelopeHeader /// diagnostic `ref` is the only truncatable field and is shortened, never dropped, when necessary to /// preserve the fixed layout. The header is built without payload bytes, so an upload can stage the /// header before the payload is streamed. -String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len); +/// `version_override` exists for one caller: the boundary test that has to see what the descriptor +/// costs at the WIDEST version the budget reserves room for. The budget is sized for a ten-digit +/// version; production has only ever written a one-digit one, so a test that encodes at the current +/// version and adds the missing digits arithmetically never sends the boundary through the encoder +/// at all -- it re-derives the formula it is supposed to be checking. Production passes nothing and +/// gets `currentCompatibilityVersion()`. +String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len, + std::optional version_override = {}); /// Parses and validates the JSON descriptor, its expected `type`, and its compatibility version. /// Derives `header_len` from the terminating '\n' and requires every preceding byte in the pad zone to diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index 77c3d1dcece4..b2c0bf1f9756 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -391,11 +391,17 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect } uint64_t seen = 0; + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "fold seal"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Strict, "fold seal"); + readLineInto(in, row_line, line_cap, "fold seal"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Strict, "fold seal"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index 9ae1ffd6282d..2f9aa0e141d2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -71,11 +71,16 @@ OutcomeLog decodeOutcomeLog(std::string_view data) const uint64_t line_cap = traitsFor(FormatId::GcOutcomes).line_cap; OutcomeLog log; + /// One line scratch and one reader for the whole loop, as the other row decoders do: + /// rebuilding them per row costs an allocation per row for the seen-key store and the line. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "outcome log"); - ReadBufferFromMemory line_in(line.data(), line.size()); - JsonObjectReader r(line_in, KeyStrictness::Tolerant, "outcome log"); + readLineInto(in, row_line, line_cap, "outcome log"); + ReadBufferFromMemory line_in(row_line.data(), row_line.size()); + row_reader.reset(line_in, KeyStrictness::Tolerant, "outcome log"); + JsonObjectReader & r = row_reader; String key; /// The first key distinguishes a trailer (`n`) from a record (`kind`). diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index ac70c2c00b23..e0722add9bdc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -176,11 +176,17 @@ PartManifest decodePartManifest(std::string_view data) /// Index-aligned with `m.entries` (Blob entries push an unused 0 placeholder). std::vector inline_lens; String blob_ref_what; /// reused across Blob entries so the error context does not allocate per row + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "cas_part_manifest"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Tolerant, "cas_part_manifest"); + readLineInto(in, row_line, line_cap, "cas_part_manifest"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Tolerant, "cas_part_manifest"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index 00d20988ac03..78dedf30a4b2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -31,6 +31,13 @@ namespace RunWire constexpr WireKey confirmed{"confirmed"}; } +namespace RunHeaderWire +{ + constexpr WireKey type{"type"}; + constexpr WireKey version{"v"}; + constexpr WireKey kind{"kind"}; +} + constexpr EnumWireTable kRunMarkerWords{{{ {RunMarker::Zero, "zero"}, {RunMarker::Edge, "edge"}, @@ -119,12 +126,9 @@ void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) const FormatTraits & t = traitsFor(FormatId::RunFile); CasJsonWriter line(64); bool first = true; - writeKey(line, "type", first); - writeStringValue(line, t.type); - writeKey(line, "v", first); - writeIntText(currentCompatibilityVersion(), line); - writeKey(line, "kind", first); - writeStringValue(line, kind); + writeStringField(line, RunHeaderWire::type, t.type, first); + writeNumberField(line, RunHeaderWire::version, currentCompatibilityVersion(), first); + writeStringField(line, RunHeaderWire::kind, kind, first); closeObject(line, first); writeChar('\n', line); const std::string_view line_view = line.view(); @@ -242,9 +246,12 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) if (done) return false; - const String line = readLine(hashing, traitsFor(FormatId::RunFile).line_cap, "cas_run"); - ReadBufferFromMemory line_in(line.data(), line.size()); - JsonObjectReader r(line_in, KeyStrictness::Strict, "cas_run"); + readLineInto(hashing, scratch, traitsFor(FormatId::RunFile).line_cap, "cas_run"); + ReadBufferFromMemory line_in(scratch.data(), scratch.size()); + /// Re-point the reader rather than building one per row: a fresh reader re-allocates its + /// seen-key store and value scratch every row, and this loop runs once per record. + reader.reset(line_in, KeyStrictness::Strict, "cas_run"); + JsonObjectReader & r = reader; String key; if (!r.nextKey(key)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index fcc216b121ee..dbf9fd8fc9d8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -1,6 +1,7 @@ #pragma once #include #include +#include #include #include #include @@ -187,6 +188,12 @@ class SourceEdgeRunReader HashingReadBuffer hashing; uint64_t seen = 0; bool done = false; + /// Reused line scratch, mirroring the writer's: `readLineInto` clears it without releasing its + /// buffer, so a run of any length allocates only up to the longest line it has actually seen. + String scratch; + /// Reused object reader, for the same reason: its per-object buffers then cost one allocation + /// for the whole run rather than one per row. It starts unbound and every row re-points it. + JsonObjectReader reader; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index f2cea307342c..6cf44651508e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -147,19 +147,17 @@ String encodeRefCatalog(const RefCatalog & catalog) e.ns.string(), nsStateToWord(e.state), e.removal_started_round ? "carries" : "lacks"); bool first = true; - writeKey(out, RefCatalogWire::kind, first); writeStringValue(out, kEntryTag); - writeKey(out, RefCatalogWire::ns, first); writeStringValue(out, e.ns.string()); - writeKey(out, RefCatalogWire::state, first); writeStringValue(out, nsStateToWord(e.state)); - writeKey(out, RefCatalogWire::life, first); writeHex128Value(out, e.incarnation); + writeStringField(out, RefCatalogWire::kind, kEntryTag, first); + writeStringField(out, RefCatalogWire::ns, e.ns.string(), first); + writeStringField(out, RefCatalogWire::state, nsStateToWord(e.state), first); + writeHex128Field(out, RefCatalogWire::life, e.incarnation, first); if (e.removal_started_round) - { - writeKey(out, RefCatalogWire::remove_round, first); writeU64StringValue(out, *e.removal_started_round); - } + writeU64StringField(out, RefCatalogWire::remove_round, *e.removal_started_round, first); if (e.creator) { - writeKey(out, RefCatalogWire::creator, first); writeStringValue(out, e.creator->server_root_id); - writeKey(out, RefCatalogWire::creator_epoch, first); writeU64StringValue(out, e.creator->writer_epoch); - writeKey(out, RefCatalogWire::creator_fence, first); writeU64StringValue(out, e.creator->fence_generation); + writeStringField(out, RefCatalogWire::creator, e.creator->server_root_id, first); + writeU64StringField(out, RefCatalogWire::creator_epoch, e.creator->writer_epoch, first); + writeU64StringField(out, RefCatalogWire::creator_fence, e.creator->fence_generation, first); } closeObject(out, first); closeLine("entry"); @@ -180,11 +178,17 @@ RefCatalog decodeRefCatalog(std::string_view data) RefCatalog catalog; uint64_t seen = 0; + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; for (;;) { - const String line = readLine(in, line_cap, "ref catalog"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Strict, "ref catalog"); + readLineInto(in, row_line, line_cap, "ref catalog"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Strict, "ref catalog"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index dbc93315a99a..4034f5c244c1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -105,10 +105,7 @@ String encodeRefCkpt(const RefCkpt & ckpt) /// written by the one shared `RefTxnId` writer the `_log` and `_snap` formats also use, so the /// three ref formats cannot disagree on the encoding. if (ckpt.life_epoch) - { - writeKey(out, RefCkptWire::life_epoch, first); - writeU64StringValue(out, *ckpt.life_epoch); - } + writeU64StringField(out, RefCkptWire::life_epoch, *ckpt.life_epoch, first); if (ckpt.committed_through) writeRefTxnIdFields(out, first, RefCkptWire::committed_epoch, RefCkptWire::committed_seq, *ckpt.committed_through); if (ckpt.checkpoint_snapshot_id) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index 8d60260c8b53..8c9d010ee678 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -365,11 +365,16 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con expected_ns, expected_txn_id.writer_epoch, expected_txn_id.ref_sequence); /// op record lines, until the trailer + /// One line scratch and one reader for the whole loop, as the other row decoders do: + /// rebuilding them per row costs an allocation per row for the seen-key store and the line. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "cas_ref_log"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Tolerant, "cas_ref_log"); + readLineInto(in, row_line, line_cap, "cas_ref_log"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Tolerant, "cas_ref_log"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index 04ecef15cd53..8d8b9e6f8244 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -180,11 +180,17 @@ RefTableSnapshot decodeRefTableSnapshot( } /// record lines (committed then precommit), until the trailer + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "cas_ref_snap"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Tolerant, "cas_ref_snap"); + readLineInto(in, row_line, line_cap, "cas_ref_snap"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Tolerant, "cas_ref_snap"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index d2f25aab615f..aefa25c871c6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -166,13 +167,36 @@ auto JsonObjectReader::guarded(F && f) } JsonObjectReader::JsonObjectReader(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_) - : in(in_), strictness(strictness_), what(what_) + : in(&in_), strictness(strictness_), what(what_) { - guarded([&] { assertChar('{', in); }); + guarded([&] { assertChar('{', *in); }); +} + +void JsonObjectReader::reset(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_) +{ + in = &in_; + strictness = strictness_; + what = what_; + /// `clear` on both keeps their buffers: that is the whole point of reusing the reader. + seen_keys.clear(); + scratch.clear(); + first = true; + done = false; + guarded([&] { assertChar('{', *in); }); +} + +std::string_view JsonObjectReader::readStringIntoScratch() +{ + scratch.clear(); + readJSONStringInto(scratch, *in, jsonReadSettings()); + return scratch; } bool JsonObjectReader::nextKey(String & key) { + /// A default-constructed reader is unbound until `reset`; using one is a programming error, so + /// this belongs in the debug build rather than as a branch on the decode hot path. + chassert(in != nullptr); return guarded([&]() -> bool { if (done) @@ -180,7 +204,7 @@ bool JsonObjectReader::nextKey(String & key) if (first) { first = false; - if (checkChar('}', in)) + if (checkChar('}', *in)) { done = true; return false; @@ -188,15 +212,15 @@ bool JsonObjectReader::nextKey(String & key) } else { - if (checkChar('}', in)) + if (checkChar('}', *in)) { done = true; return false; } - assertChar(',', in); + assertChar(',', *in); } - readJSONString(key, in, jsonReadSettings()); - assertChar(':', in); + readJSONString(key, *in, jsonReadSettings()); + assertChar(':', *in); if (std::find(seen_keys.begin(), seen_keys.end(), key) != seen_keys.end()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: duplicate key '{}'", what, key); seen_keys.push_back(key); @@ -209,7 +233,7 @@ String JsonObjectReader::readString() return guarded([&] { String s; - readJSONString(s, in, jsonReadSettings()); + readJSONString(s, *in, jsonReadSettings()); return s; }); } @@ -219,18 +243,18 @@ std::vector JsonObjectReader::readStringArray() return guarded([&] { std::vector words; - assertChar('[', in); - if (checkChar(']', in)) + assertChar('[', *in); + if (checkChar(']', *in)) return words; while (true) { String word; - readJSONString(word, in, jsonReadSettings()); + readJSONString(word, *in, jsonReadSettings()); words.push_back(std::move(word)); - if (checkChar(']', in)) + if (checkChar(']', *in)) return words; - assertChar(',', in); + assertChar(',', *in); } }); } @@ -239,7 +263,7 @@ UInt128 JsonObjectReader::readHex128() { return guarded([&] { - const String hex = readString(); + const std::string_view hex = readStringIntoScratch(); if (hex.size() != 32 || std::any_of(hex.begin(), hex.end(), [](char c) { return !isLowercaseHexChar(c); })) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: expected 32 lowercase hex chars, got '{}'", what, hex); return unhexUInt(hex.data()); @@ -250,7 +274,7 @@ uint64_t JsonObjectReader::readU64String() { return guarded([&] { - const String s = readString(); + const std::string_view s = readStringIntoScratch(); ReadBufferFromMemory buf(s.data(), s.size()); uint64_t v = 0; readIntText(v, buf); @@ -265,7 +289,7 @@ uint64_t JsonObjectReader::readU64Number() return guarded([&] { uint64_t v = 0; - readIntText(v, in); + readIntText(v, *in); return v; }); } @@ -282,9 +306,9 @@ bool JsonObjectReader::readBool() { return guarded([&] { - if (checkString("true", in)) + if (checkString("true", *in)) return true; - assertString("false", in); + assertString("false", *in); return false; }); } @@ -298,20 +322,28 @@ void JsonObjectReader::skipUnknown(const String & key) "CAS {}: critical key '{}' is not understood by this build", what, key); if (strictness == KeyStrictness::Strict) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown key '{}' in a strict format", what, key); - skipJSONField(in, key, jsonReadSettings()); + skipJSONField(*in, key, jsonReadSettings()); }); } /// ---- header line / trailer line / raw line access ---- +namespace +{ +namespace ContainerWire +{ + constexpr WireKey type{"type"}; + constexpr WireKey version{"v"}; + constexpr WireKey count{"n"}; +} +} + void writeHeaderLine(CasJsonWriter & out, FormatId id) { const FormatTraits & t = traitsFor(id); bool first = true; - writeKey(out, "type", first); - writeStringValue(out, t.type); - writeKey(out, "v", first); - writeIntText(currentCompatibilityVersion(), out); + writeStringField(out, ContainerWire::type, t.type, first); + writeNumberField(out, ContainerWire::version, currentCompatibilityVersion(), first); closeObject(out, first); writeChar('\n', out); } @@ -319,29 +351,49 @@ void writeHeaderLine(CasJsonWriter & out, FormatId id) void writeTrailerLine(CasJsonWriter & out, uint64_t n) { bool first = true; - writeKey(out, "n", first); - writeIntText(n, out); + writeNumberField(out, ContainerWire::count, n, first); closeObject(out, first); writeChar('\n', out); } -String readLine(ReadBuffer & in, uint64_t line_cap, std::string_view what) +void readLineInto(ReadBuffer & in, String & line, uint64_t line_cap, std::string_view what) { - String line; + /// `clear` keeps the capacity, so a caller that reuses one scratch across a stream's rows stops + /// allocating after the longest line it has seen -- the same bound the writer's scratch has. + line.clear(); while (true) { if (in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: truncated object (line without terminator)", what); - const char c = *in.position(); - ++in.position(); - if (c == '\n') - return line; - line.push_back(c); - if (line.size() > line_cap) + + /// Take the whole run up to the terminator in one append rather than a byte at a time: a + /// per-character `push_back` also re-checks the cap on every character, and a stream row is + /// hundreds of characters long. + const char * const from = in.position(); + const char * const found = find_first_symbols<'\n'>(from, in.buffer().end()); + const size_t taken = static_cast(found - from); + if (line.size() + taken > line_cap) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: line exceeds the {}-byte cap", what, line_cap); + line.append(from, taken); + in.position() += taken; + + /// `find_first_symbols` stops either at the terminator or at the end of the buffered window. + /// Only the first case ends the line; the second needs the next window. + if (found != in.buffer().end()) + { + ++in.position(); + return; + } } } +String readLine(ReadBuffer & in, uint64_t line_cap, std::string_view what) +{ + String line; + readLineInto(in, line, line_cap, what); + return line; +} + namespace { TextHeader parseHeaderObject(std::string_view line, std::string_view what) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index bd353f864d17..8bbbf58a4efd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -235,6 +235,19 @@ class JsonObjectReader public: /// Consumes the opening `{`; throws `CORRUPTED_DATA` when the object does not start there. JsonObjectReader(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_); + + /// An unbound reader, for a decoder that wants one reader outside its row loop and re-points it + /// per row. `reset` must be called before any read; nothing else is valid on it. + JsonObjectReader() = default; + + /// Re-point an existing reader at another object, as the constructor would, but WITHOUT + /// releasing the buffers it has already grown. A stream decoder reads one object per row, and a + /// reader built fresh each time re-allocates its seen-key store and its value scratch on every + /// row; measured on the `cas_run` decode path, allocation accounting is about a fifth of all + /// instructions executed inside the decoder. Reusing one reader amortises that away. The + /// object-level state -- the key set and the position in the object -- is reset in full, so a + /// reused reader accepts and rejects exactly what a fresh one would. + void reset(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_); /// Advances to the next key; false when the closing '}' was consumed. The caller must /// consume the value (one read* / skipUnknown) before the next call. Duplicate keys are /// rejected with `CORRUPTED_DATA`. @@ -263,10 +276,16 @@ class JsonObjectReader template auto guarded(F && f); - ReadBuffer & in; - KeyStrictness strictness; + /// Reads one JSON string into `scratch` and returns a view of it, so a value that is parsed and + /// discarded -- a hex digest, a decimal counter -- costs no allocation once the scratch has + /// grown. The view is valid until the next read on this reader. + std::string_view readStringIntoScratch(); + + ReadBuffer * in = nullptr; + KeyStrictness strictness = KeyStrictness::Strict; String what; std::vector seen_keys; + String scratch; bool first = true; bool done = false; }; @@ -288,6 +307,11 @@ TextHeader expectHeaderLine(ReadBuffer & in, FormatId id); std::optional sniffHeaderLine(std::string_view bytes); /// Reads one line (excluding the '\n' terminator); CORRUPTED_DATA on missing terminator or a line /// longer than `line_cap`. +/// Read one terminator-delimited line into `line`, replacing its contents and KEEPING its capacity, +/// so a caller streaming many rows can reuse one scratch and stop allocating after the longest line. +void readLineInto(ReadBuffer & in, String & line, uint64_t line_cap, std::string_view what); + +/// Allocating form, for callers that read a single line. String readLine(ReadBuffer & in, uint64_t line_cap, std::string_view what); /// Position of the next byte `stringValue` treats specially (control byte, '"', '\\', or the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 2b30d730aba5..39191bad1a43 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -47,28 +47,21 @@ ObjectKind objectKindFromWord(std::string_view w, std::string_view what) void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t) { - writeKey(out, SharedWire::token_type, first); - writeStringValue(out, tokenTypeToWord(t.type)); - writeKey(out, SharedWire::token, first); - writeStringValue(out, t.value); + writeStringField(out, SharedWire::token_type, tokenTypeToWord(t.type), first); + writeStringField(out, SharedWire::token, t.value, first); } void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r) { - writeKey(out, SharedWire::algo, first); - writeStringValue(out, blobHashAlgoName(r.algo)); - writeKey(out, SharedWire::digest, first); - writeStringValue(out, codecFor(r.algo).toHex(r.digest)); + writeStringField(out, SharedWire::algo, blobHashAlgoName(r.algo), first); + writeStringField(out, SharedWire::digest, codecFor(r.algo).toHex(r.digest), first); } void writeManifestRefFields(CasJsonWriter & out, bool & first, const ManifestRefWireKeys & keys, const ManifestRef & r) { - writeKey(out, keys.epoch, first); - out.u64StringValue(r.writer_epoch); - writeKey(out, keys.build, first); - out.u64StringValue(r.build_sequence); - writeKey(out, keys.ord, first); - out.u64Number(r.manifest_ordinal); + writeU64StringField(out, keys.epoch, r.writer_epoch, first); + writeU64StringField(out, keys.build, r.build_sequence, first); + writeNumberField(out, keys.ord, r.manifest_ordinal, first); } ManifestRef manifestRefFromFields(uint64_t writer_epoch, uint64_t build_sequence, uint64_t manifest_ordinal, diff --git a/src/Disks/tests/cas_format_test_battery.h b/src/Disks/tests/cas_format_test_battery.h index 361947968f3c..efed9ce44a74 100644 --- a/src/Disks/tests/cas_format_test_battery.h +++ b/src/Disks/tests/cas_format_test_battery.h @@ -29,12 +29,19 @@ struct FormatBatteryCase std::function make_future_version = {}; }; -/// Canonical object headers track the current compatibility generation. The type remains an -/// explicit test literal at every call site, so a registry/type mismatch cannot be hidden by a -/// self-derived expectation. +/// The canonical object header, spelled literally. +/// +/// The version is the LITERAL 1, not `currentCompatibilityVersion()`. Deriving it from production +/// was the defect: encoder output and expected bytes would then move together across a generation +/// bump, and a golden that tracks the code it is meant to pin cannot fail. The type was already a +/// literal at every call site for the same reason; the version had been left behind. +/// +/// A future generation bump is therefore SUPPOSED to break every test that uses this. That is the +/// point: the new bytes get read, agreed to, and written down, rather than being adopted silently. +/// `HeaderVersionIsTheLiteralThisBatteryPins` below fails first and says so. inline String currentFormatHeader(std::string_view type) { - return fmt::format("{{\"type\":\"{}\",\"v\":{}}}\n", type, DB::Cas::currentCompatibilityVersion()); + return fmt::format("{{\"type\":\"{}\",\"v\":1}}\n", type); } namespace cas_battery_detail diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 98bd9ef57ac3..edfa45abf828 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -186,25 +186,39 @@ TEST(CASBlobEnvelopeFormat, MandatoryWorstCaseBoundary) /// without moving the encoder and lands here. TEST(CASBlobEnvelopeFormat, WorstCaseFormulaMatchesTheEncoder) { + /// Drive the encoder at the WIDEST version the budget reserves room for, rather than encoding at + /// today's one-digit version and adding the missing digits by arithmetic. Doing the arithmetic + /// here would re-derive the very formula this test exists to check, and would never send the + /// ten-digit boundary through the encoder's own number formatting. EnvelopeHeader h = maxReachableHeader(""); - const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen)); + const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen), + std::numeric_limits::max()); + /// The mandatory shape is everything up to and including the closing brace, plus the newline the /// encoder reserves at the last byte; the padding between them is the ref budget this measures. const size_t json_len = head.find_last_not_of(' ', kMinBlobHeaderLen - 2) + 1; - const size_t mandatory_at_current_version = json_len + 1; /// + the reserved '\n' + const size_t mandatory_at_max_version = json_len + 1; /// + the reserved '\n' - size_t version_digits = 0; - for (uint32_t v = currentCompatibilityVersion(); ; v /= 10) - { - ++version_digits; - if (v < 10) - break; - } - const size_t unused_version_digits = std::numeric_limits::digits10 + 1 - version_digits; + EXPECT_EQ(mandatory_at_max_version, mandatory_descriptor_worst_case) + << "the formula and the encoder disagree about the mandatory descriptor at the widest " + "version: encoder wrote " << mandatory_at_max_version << " bytes, formula says " + << mandatory_descriptor_worst_case; - EXPECT_EQ(mandatory_at_current_version + unused_version_digits, mandatory_descriptor_worst_case) - << "the formula and the encoder disagree about the mandatory descriptor: encoder wrote " - << mandatory_at_current_version << " bytes at a " << version_digits << "-digit version"; + /// And the whole point of the budget: even at that width one byte remains spare under the floor. + EXPECT_LE(mandatory_descriptor_worst_case, kMinBlobHeaderLen - 1); +} + +/// The version really is rendered at its full width by the encoder above, not merely accounted for. +/// Without this, an encoder that silently clamped or dropped the override would still satisfy the +/// equality it feeds. +TEST(CASBlobEnvelopeFormat, MaxWidthVersionIsActuallyRendered) +{ + EnvelopeHeader h = maxReachableHeader(""); + const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen), + std::numeric_limits::max()); + EXPECT_NE(head.find("\"v\":4294967295"), String::npos) + << "the max-width version was not rendered; the boundary above proves nothing. Header: " + << head; } TEST(CASBlobEnvelopeFormat, CriticalKeyDescriptorStillFitsAtDefaultLength) diff --git a/src/Disks/tests/gtest_cas_format.cpp b/src/Disks/tests/gtest_cas_format.cpp index 6c5ad84cc0cc..b3236797176c 100644 --- a/src/Disks/tests/gtest_cas_format.cpp +++ b/src/Disks/tests/gtest_cas_format.cpp @@ -84,3 +84,18 @@ TEST(CASFormat, CheckCompatibilityFailsClosedOnFuture) EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); } } + +/// The battery's goldens spell their header version as the literal 1 rather than asking production +/// for it, so that a generation bump cannot move the expectation and the encoder output together. +/// The cost of that is a golden set which goes stale silently if nobody notices the bump; this test +/// is what notices. It fails FIRST and says what to do, so the failure that greets a generation bump +/// is one explanatory test rather than every exact-encoding golden at once. +TEST(CASFormat, HeaderVersionIsTheLiteralThisBatteryPins) +{ + EXPECT_EQ(currentCompatibilityVersion(), 1u) + << "The compatibility version has moved away from the literal 1 that " + "`currentFormatHeader` in cas_format_test_battery.h writes into every golden header. " + "That is a deliberate wire change: read the new bytes, agree to them, and update the " + "literal and the goldens together. Do NOT make the helper derive the version from " + "production again -- a golden that tracks the code it pins cannot fail."; +} From acede29cd44e448b12f4b8840171123191581a02 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 31 Aug 2026 13:37:57 +0200 Subject: [PATCH 7/8] docs: recommend single-replica merges for `CAS` Signed-off-by: Mikhail Filimonov --- docs/en/antalya/cas/index.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/en/antalya/cas/index.md b/docs/en/antalya/cas/index.md index 2bc71046494b..cb1d563eebf0 100644 --- a/docs/en/antalya/cas/index.md +++ b/docs/en/antalya/cas/index.md @@ -64,6 +64,14 @@ Two consequences for planning: Each prefix is a fully independent pool (its own refs, leases, and `GC`), so rounds stay short regardless of the total fleet size. +:::tip +For replicated tables on `CAS`, enable +[`execute_merges_on_single_replica_time_threshold`](/operations/settings/merge-tree-settings#execute_merges_on_single_replica_time_threshold). +This lets one replica perform each merge while the others wait for and fetch the resulting part, +avoiding redundant merge work across replicas. Set the threshold higher than the usual merge +duration for your workload. +::: + ## Status {#status} `CAS` is **experimental**. It ships in Altinity Antalya builds. Experimental means the on-disk From f7c64c8cbb4b92e7e4cda7f0f8e903b395a97913 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 31 Aug 2026 19:56:11 +0200 Subject: [PATCH 8/8] Allow EXPORT PARTITION from a source on a CAS disk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ALTER TABLE ... EXPORT PARTITION` was refused with `SUPPORT_IS_DISABLED` on a content-addressed disk because it is absent from the partition-command allowlist in `MergeTreeData`. The rejection it fell into says the command "clones parts file-by-file with no transaction, which would corrupt the clone", and that reason does not describe exporting. `ExportPartTask` reads the source part through `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) under `readLockParts` and writes rows into the destination through a `SinkToStorage` on an ordinary query pipeline. Nothing is hard-linked or copied on the source disk; the command's own bookkeeping is in ZooKeeper. So the allowlist was rejecting it by omission rather than by an argument that applies to it, which the code around it already half concedes: `EXPORT_PARTITION` is listed among the commands permitted to target `PARTITION ALL` a few lines above. Verified end to end rather than by inspection, on a server built from this change: a `ReplicatedMergeTree` source on a CAS disk holding (1,2020), (2,2020), (3,2021), exported to an `IcebergLocal` destination. `EXPORT PARTITION ID '2020'` succeeds and the destination holds exactly (1,2020) and (2,2020) — the right partition, and the 2021 row correctly absent. Two limitations surfaced on the way and are NOT addressed here, because neither is about CAS. Export is implemented only for `ReplicatedMergeTree`: a plain `MergeTree` source now returns `Code: 48 NOT_IMPLEMENTED` instead of the CAS refusal, so the reproduction in the report — which uses a plain MergeTree — will still fail, just for its real reason. And the operation remains behind the server setting `allow_experimental_export_merge_tree_partition`. Closes: https://github.com/Altinity/ClickHouse/issues/2291 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KiHKrvEVy8u4nA1A8qYFUY Signed-off-by: Mikhail Filimonov --- src/Storages/MergeTree/MergeTreeData.cpp | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/Storages/MergeTree/MergeTreeData.cpp b/src/Storages/MergeTree/MergeTreeData.cpp index a507f0695d4c..e60e7fbd1a87 100644 --- a/src/Storages/MergeTree/MergeTreeData.cpp +++ b/src/Storages/MergeTree/MergeTreeData.cpp @@ -6848,6 +6848,14 @@ void MergeTreeData::checkAlterPartitionIsPossible( /// `FORGET PARTITION` is SUPPORTED on CA — it only manipulates ZooKeeper partition metadata /// (removes block-number nodes from ZooKeeper) and does not write, clone, or touch any part /// files on disk, so it is safe on a content-addressed disk. + /// `EXPORT PARTITION` is SUPPORTED because the reason this list exists does not apply + /// to it. The rejection below is about commands that clone parts file-by-file; + /// exporting does not clone at all. `ExportPartTask` reads the source part through + /// `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) and writes + /// rows into the destination through a `SinkToStorage` on an ordinary query + /// pipeline, so the source's part files are only READ, under `readLockParts`, and + /// nothing is hard-linked or copied on the content-addressed disk. The command's + /// own bookkeeping is ZooKeeper-side. /// NOTE: `MOVE_PARTITION` also admits cross-disk /// `MOVE ... TO DISK/VOLUME` (this check cannot distinguish the destination); that uses /// the byte-copy `clonePart` path (NOT the corrupting per-file hardlink), but only @@ -6864,6 +6872,7 @@ void MergeTreeData::checkAlterPartitionIsPossible( PartitionCommand::FREEZE_ALL_PARTITIONS, PartitionCommand::UNFREEZE_PARTITION, PartitionCommand::UNFREEZE_ALL_PARTITIONS, + PartitionCommand::EXPORT_PARTITION, }; if (!std::ranges::contains(supported_commands, command.type))