refactor: typed extension metadata, parsed and serialized at the wire

AGENTS.md

@@ -34,9 +34,10 @@ fsst    — io.github.dfa1.vortex.fsst: standalone FSST (Fast Static Symbol Tabl
           Zero dependency on core/reader/writer/FFM-wire-concerns beyond the JDK's own
           java.lang.foreign; writer/reader depend on it, not the other way around.
 core    — everything lives under `io.github.dfa1.vortex.core.*`:
-          core.model    DType, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype,
-                        EditionId, Edition, EditionFamily, Editions, MemorySize
-          core.io       IoBounds, PTypeIO, VortexFormat
+          core.model    DType, DTypes, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype,
+                        ExtensionMetadata, OpaqueMetadata, TimeUnitMetadata, TimestampMetadata,
+                        EditionId, Edition, EditionFamily, Editions
+          core.io       IoBounds, PTypeIO, VortexFormat, MemorySize
           core.error    VortexException
           core.compute  FastLanes, PrimitiveArrays, Utf8Order, Utf8Validation
           core.simd     SimdOperations (kernels shared by reader+writer, all eleven ptypes),
@@ -302,9 +303,16 @@ When stuck on encode/decode behavior, consult **in this order**:
 
 - **DType is pluggable only via `Extension`.** `DType` is a sealed interface; downstream code must
   not add variants. Use `new DType.Extension("ip.address", new DType.Primitive(PType.I32, false),
-  null, false)` and register an `ExtensionDecoder`/`ExtensionEncoder` on `ReadRegistry`/`WriteRegistry`.
-  Mirrors Rust's `ExtVTable` (`vortex.date`, `vortex.uuid`, …): an unregistered id fails the read
-  unless `allowUnknown()`, then reads as storage. No SPI for DType variants planned.
+  OpaqueMetadata.EMPTY, false)` and register an `ExtensionDecoder`/`ExtensionEncoder` on
+  `ReadRegistry`/`WriteRegistry`. Mirrors Rust's `ExtVTable` (`vortex.date`, `vortex.uuid`, …): an
+  unregistered id fails the read unless `allowUnknown()`, then reads as storage. No SPI for DType
+  variants planned.
+- **Extension metadata is typed, bytes exist only at the wire.** `DType.Extension#metadata()` is an
+  `ExtensionMetadata`, an open interface whose implementations must be records (checked by
+  `DType.Extension`), so dtypes compare by value. As Rust's `ExtVTable::{de,}serialize_metadata`,
+  the reader parses (`ExtensionMetadataParser` for the well-known ids, `ExtensionDecoder#parseMetadata`
+  for the rest, at open) and the writer serializes (`DTypeFbsSerializer`, `ExtensionEncoder#serializeMetadata`);
+  `core` holds only the types. Bytes nothing parses stay `OpaqueMetadata` (Rust's `ForeignExtMetadata`).
 - **Encoding and layout decode are both pluggable via builder-only registries** — no
   `ServiceLoader`. Encodings register on `ReadRegistry`/`WriteRegistry`
   (`ReadRegistry.builder().registerDefaults().register(custom).build()`); layouts register on

CHANGELOG.md

@@ -24,6 +24,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Breaking:** `ScanOptions.filter()` and `limit()` return `Optional<ScanFilter>` and `Optional<ScanLimit>`, as Rust's `ScanBuilder` holds them; `withLimit` takes a `ScanLimit`, and `NO_LIMIT`, `ScanOptions.limit(long)`, `rowFilter()`, `hasFilter()` and `hasLimit()` are gone. ([#598](https://github.com/dfa1/vortex-java/pull/598))
 - **Breaking:** CSV import stores an empty cell in a number or boolean column as null instead of `0` or `false`, so inferred number and boolean columns are now nullable (read through `MaskedArray`), and an empty cell or a missing column fails the import where the schema declares the column non-nullable. ([#592](https://github.com/dfa1/vortex-java/issues/592))
 - **Breaking:** the shared non-nullable dtypes move from `DType` to the new `DTypes` class: `DTypes.I64`, `DTypes.UTF8`, … ([#624](https://github.com/dfa1/vortex-java/pull/624))
+- **Breaking:** an extension dtype's metadata is a typed `ExtensionMetadata` record instead of a `MemorySegment`, parsed when a file opens and serialized when it is written, as Rust does: `TimeUnitMetadata` for date and time, `TimestampMetadata` (its zone kept as stored) for timestamp, `OpaqueMetadata` for bytes nothing parses, and a third-party record through `ExtensionDecoder#parseMetadata` and `ExtensionEncoder#serializeMetadata`. ([#625](https://github.com/dfa1/vortex-java/pull/625))
+- **Breaking:** `MemorySize` moves from `io.github.dfa1.vortex.core.model` to `io.github.dfa1.vortex.core.io`. ([#625](https://github.com/dfa1/vortex-java/pull/625))
 
 ### Fixed
 - Two threads first using `DType` at the same time no longer deadlock the JVM, as they could when, for example, two Parquet imports started together. ([#624](https://github.com/dfa1/vortex-java/pull/624))

calcite/src/main/java/io/github/dfa1/vortex/calcite/Temporal.java

@@ -4,12 +4,12 @@
 import io.github.dfa1.vortex.core.model.DType;
 import io.github.dfa1.vortex.core.model.ExtensionId;
 import io.github.dfa1.vortex.core.model.TimeUnit;
-import io.github.dfa1.vortex.core.model.TimestampDtype;
 import io.github.dfa1.vortex.reader.array.ByteArray;
 import io.github.dfa1.vortex.reader.array.IntArray;
 import io.github.dfa1.vortex.reader.array.LongArray;
 import io.github.dfa1.vortex.reader.array.MaskedArray;
 import io.github.dfa1.vortex.reader.array.ShortArray;
+import io.github.dfa1.vortex.reader.extension.ExtensionStorage;
 
 /// A `vortex.date`, `vortex.time` or `vortex.timestamp` column's conversion from its raw storage
 /// integer to Calcite's internal form: `DATE` as epoch days (`Integer`), `TIME` as milliseconds of
@@ -36,8 +36,8 @@ static Temporal of(DType type) {
                 && kind != ExtensionId.WellKnown.VORTEX_TIMESTAMP) {
             return null;
         }
-        // the unit byte leads every temporal extension's metadata
-        TimeUnit unit = TimestampDtype.readUnit(ext);
+        // date and time carry a TimeUnitMetadata, timestamp a TimestampMetadata: both have the unit
+        TimeUnit unit = ExtensionStorage.readUnit(ext);
         boolean valid = switch (kind) {
             case VORTEX_DATE -> unit == TimeUnit.Days || unit == TimeUnit.Milliseconds;
             default -> unit != TimeUnit.Days;

cli/src/test/java/io/github/dfa1/vortex/cli/SchemaCommandTest.java

@@ -3,6 +3,7 @@
 import io.github.dfa1.vortex.core.model.ColumnName;
 import io.github.dfa1.vortex.core.model.DType;
 import io.github.dfa1.vortex.core.model.DTypes;
+import io.github.dfa1.vortex.core.model.OpaqueMetadata;
 import io.github.dfa1.vortex.core.model.PType;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.io.TempDir;
@@ -90,8 +91,8 @@ static Stream<Arguments> dtypeCases() {
                 arguments(new DType.Map(DTypes.UTF8, i64, true, false), "map<utf8, I64, keys_sorted=true>"),
                 arguments(new DType.Map(DTypes.UTF8, new DType.Primitive(PType.I64, true), false, false),
                         "map<utf8, I64?, keys_sorted=false>"),
-                arguments(new DType.Extension("vortex.uuid", i64, null, false), "ext<vortex.uuid>"),
-                arguments(new DType.Extension("vortex.uuid", i64, null, true), "ext<vortex.uuid>?"),
+                arguments(new DType.Extension("vortex.uuid", i64, OpaqueMetadata.EMPTY, false), "ext<vortex.uuid>"),
+                arguments(new DType.Extension("vortex.uuid", i64, OpaqueMetadata.EMPTY, true), "ext<vortex.uuid>?"),
                 arguments(DTypes.VARIANT, "variant"),
                 arguments(new DType.Variant(true), "variant?"),
                 arguments(new DType.Struct(List.of(ColumnName.of("a"), ColumnName.of("b")), List.of(i64, DTypes.UTF8), false),

cli/src/test/java/io/github/dfa1/vortex/cli/VortexCliTest.java

@@ -161,7 +161,8 @@ private static Path writeCustomExtensionVortex(Path dir) throws IOException {
         Path file = dir.resolve("money.vortex");
         DType.Struct schema = new DType.Struct(List.of(ColumnName.of("price")),
                 List.of(new DType.Extension("acme.money", DTypes.I64,
-                        java.lang.foreign.MemorySegment.ofArray("EUR".getBytes(StandardCharsets.US_ASCII)), false)),
+                        new io.github.dfa1.vortex.core.model.OpaqueMetadata(
+                                java.lang.foreign.MemorySegment.ofArray("EUR".getBytes(StandardCharsets.US_ASCII))), false)),
                 false);
         try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
              VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.defaults())) {

cli/src/test/java/io/github/dfa1/vortex/cli/tui/GridRenderTest.java

@@ -206,7 +206,8 @@ void extensionDecodeFailureRendersDiagnostic() {
         try (Arena arena = Arena.ofConfined()) {
             // Given — vortex.date declared but storage is F64, which epochInteger rejects
             DType dateExt = new DType.Extension("vortex.date",
-                    DTypes.I32, null, false);
+                    DTypes.I32, new io.github.dfa1.vortex.core.model.TimeUnitMetadata(
+                            io.github.dfa1.vortex.core.model.TimeUnit.Days), false);
             Array badStorage = ArrayFixtures.doubles(arena, 1.5);
 
             // When — the decode throws and is caught
@@ -222,7 +223,7 @@ void unknownExtensionIdFallsThroughToStorageRendering() {
         try (Arena arena = Arena.ofConfined()) {
             // Given — an extension id this reader does not know: render the raw storage value
             DType unknownExt = new DType.Extension("custom.thing",
-                    DTypes.I32, null, false);
+                    DTypes.I32, io.github.dfa1.vortex.core.model.OpaqueMetadata.EMPTY, false);
             IntArray storage = ArrayFixtures.ints(arena, 42);
 
             // When / Then — no extension formatting, falls to the IntArray branch

cli/src/test/java/io/github/dfa1/vortex/cli/tui/InspectorRenderTest.java

@@ -147,7 +147,8 @@ void dateExtensionDecodeFailureFallsThroughToGeneric() {
             try (Arena arena = Arena.ofConfined()) {
                 // Given — vortex.date declared but storage is F64; the decode throws and is swallowed
                 DType dateExt = new DType.Extension("vortex.date",
-                        DTypes.I32, null, false);
+                        DTypes.I32, new io.github.dfa1.vortex.core.model.TimeUnitMetadata(
+                                io.github.dfa1.vortex.core.model.TimeUnit.Days), false);
                 Array badStorage = ArrayFixtures.doubles(arena, 1.5);
 
                 // When / Then — falls through to the generic DoubleArray rendering

core/src/main/java/io/github/dfa1/vortex/core/model/MemorySize.java → core/src/main/java/io/github/dfa1/vortex/core/io/MemorySize.java

@@ -1,4 +1,4 @@
-package io.github.dfa1.vortex.core.model;
+package io.github.dfa1.vortex.core.io;
 
 /// A validated, non-negative byte count.
 ///

core/src/main/java/io/github/dfa1/vortex/core/model/DType.java

@@ -2,7 +2,6 @@
 
 import io.github.dfa1.vortex.core.error.VortexException;
 
-import java.lang.foreign.MemorySegment;
 import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.Objects;
@@ -338,78 +337,25 @@ public Struct entriesDtype() {
     ///
     /// @param extensionId  unique string identifier for the extension type (e.g. `"vortex.timestamp"`)
     /// @param storageDType underlying storage dtype used for physical encoding
-    /// @param metadata     opaque extension-specific metadata bytes, or `null`
+    /// @param metadata     the extension's metadata, [OpaqueMetadata#EMPTY] for none
     /// @param nullable     whether null values are permitted
     record Extension(
             String extensionId,
             DType storageDType,
-            MemorySegment metadata,
+            ExtensionMetadata metadata,
             boolean nullable
     ) implements DType {
 
-        /// Hard cap on extension metadata size. Spec-defined extensions use 0–3 bytes
-        /// (e.g. timestamp = TimeUnit tag + tz length + tz UTF-8); 64 KiB is generous
-        /// for custom extensions while blocking attacker-supplied multi-megabyte blobs
-        /// that would inflate parser allocations.
-        public static final MemorySize MAX_METADATA_SIZE = MemorySize.ofKiB(64);
-
-        /// Bounds the metadata blob, which arrives from an untrusted file.
+        /// Requires metadata that is a record, so the dtype compares and hashes by value.
         ///
-        /// @throws VortexException if `metadata` carries more than
-        ///         [#MAX_METADATA_SIZE] readable bytes
+        /// @throws NullPointerException     if `metadata` is `null`
+        /// @throws IllegalArgumentException if `metadata` is not a record
         public Extension {
-            if (metadata != null && metadata.byteSize() > MAX_METADATA_SIZE.bytes()) {
-                throw new VortexException("extension metadata too large: "
-                        + metadata.byteSize() + " > " + MAX_METADATA_SIZE.bytes());
-            }
-            // A dtype outlives the reader that parsed it (callers keep reader.dtype()), but a slice
-            // of the file mapping fails once the reader closes: copy it onto the heap. Small, as
-            // the cap above bounds it.
-            if (metadata != null && metadata.isNative()) {
-                metadata = MemorySegment.ofArray(metadata.toArray(java.lang.foreign.ValueLayout.JAVA_BYTE))
-                        .asReadOnly();
-            }
-        }
-
-        /// Value equality with content-based metadata comparison.
-        ///
-        /// [MemorySegment] uses identity equality, so the default record `equals` would
-        /// treat a heap-built dtype and the same dtype read back from a mapped file as
-        /// unequal. Extension is a value type (compared in schema round-trips and as a
-        /// component of [Struct]/[List] dtypes), so metadata is compared by bytes —
-        /// matching the content semantics the previous `ByteBuffer` component had.
-        ///
-        /// @param o other object
-        /// @return whether `o` is an Extension with equal id, storage, nullability and metadata bytes
-        @Override
-        public boolean equals(Object o) {
-            if (this == o) {
-                return true;
-            }
-            if (!(o instanceof Extension(String id, DType dType, MemorySegment metadata1, boolean nullable1))) {
-                return false;
-            }
-            return nullable == nullable1
-                    && extensionId.equals(id)
-                    && storageDType.equals(dType)
-                    && metadataEquals(metadata, metadata1);
-        }
-
-        /// @return a hash consistent with [#equals(Object)] (metadata hashed by content)
-        @Override
-        public int hashCode() {
-            int h = java.util.Objects.hash(extensionId, storageDType, nullable);
-            if (metadata != null) {
-                h = 31 * h + java.util.Arrays.hashCode(metadata.toArray(java.lang.foreign.ValueLayout.JAVA_BYTE));
-            }
-            return h;
-        }
-
-        private static boolean metadataEquals(MemorySegment a, MemorySegment b) {
-            if (a == null || b == null) {
-                return a == b;
+            Objects.requireNonNull(metadata, "metadata");
+            if (!metadata.getClass().isRecord()) {
+                throw new IllegalArgumentException(
+                        "extension metadata must be a record, got " + metadata.getClass().getName());
             }
-            return a.byteSize() == b.byteSize() && a.mismatch(b) == -1;
         }
     }