latkit in Rust: one crate, the npm package on WebAssembly, and a CLI

.cargo/config.toml

@@ -0,0 +1,5 @@
+[alias]
+xtask = "run --quiet --package xtask --"
+
+[target.wasm32-unknown-unknown]
+rustflags = ["-C", "target-feature=+simd128"]

.changeset/config.json

@@ -7,13 +7,6 @@
   "access": "public",
   "baseBranch": "main",
   "updateInternalDependencies": "patch",
-  "ignore": [
-    "@latkit/colors-example",
-    "@latkit/linked-example",
-    "@latkit/monitor-example",
-    "@latkit/network-example",
-    "@latkit/video-example",
-    "@latkit/diagram-example",
-    "@latkit/benchmark"
-  ]
+  "privatePackages": { "version": false, "tag": false },
+  "ignore": []
 }

.changeset/data-contract.md

@@ -0,0 +1,10 @@
+---
+'latkit': minor
+---
+
+Patches are bytes written in TypeScript, datasets change in place, and types come from latkit's schema both ways.
+
+- Added: `Patch` without WebAssembly (`Patch.to(dataset)`, `fromBytes`, `stream`, `extend`), `Data.read` for latkit bytes and GeoJSON, `data.snapshot()`, queries' `ids`, `total`, and `contains`, `outputs.ts` (events, hits, stats, work, descriptions), and the documentation on Read the Docs.
+- Changed: `data.apply(patch)` changes the dataset in place; selections, row fits, `select` and `delete` events, and `neighbors` are row sets `{ data, table, ids }`; `Hit.mark` is the mark's name; picks, locations, and layouts reject with `timeout` past a deadline; `view.arrange()` answers patches that name their datasets; field kinds name geometries (`point`, `path`, `parts`, `polygons`); pictures and videos no longer make the pipelines only input needs.
+- Renamed: `put` to `replace`, a query's `where` to `filter`, vertices' `status` to `ring`, `rings` and (off plots) `trace` to `geometry`, `{ colormap: value, map }` to `{ colormap: map, value }`, `{ scale: value, range }` to `{ scale: [low, high], value }`, grid pitches to `{ lines: { pitch } }` and `{ dots: { pitch, snap } }`, the `'earth'` space to `'mercator'`, `forms.ts` to `inputs.ts`, and the error code `load` to `io`.
+- Removed: `geojson()` (`Data.read`), `Data.fromBytes`, patch handles and their `destroy()`, `removeTiles` (`removeData`), and bare numbers as row counts (`{ count }`).

.changeset/foundation.md

@@ -0,0 +1,10 @@
+---
+'latkit': minor
+---
+
+Histories fetched in spans, and a frame's work follows what changed.
+
+- Added: time tiles. `view.setTiles(name, url)` opens a `frames.json` manifest, as `latkit tile` writes it, and fetches the spans of each sampled field's history its lines draw, at the detail they draw; a span still loading draws the coarser one above it.
+- Added: `Patch.frames(table, field, { level, first }, at, values)`, a run of a sampled field's entries, as a time tile holds them.
+- Changed: errors reading JSON (scenes, cameras, fits, queries, rows, budgets, shots) name the path to what could not be read, such as `scene.marks[2].vertices.color`.
+- Changed: a patch evaluates only the pages it wrote; a transition evaluates once and blends each frame; rows drawn through tiles keep their places while the camera moves, so only arriving rows upload; the views of a `Gpu` share its budget, none planning past it together.

.changeset/opening.md

@@ -0,0 +1,24 @@
+---
+'latkit': minor
+---
+
+Views open in the time of their slowest pipeline with no frame held back and compile a third fewer pipelines, a patch costs the rows it changed, an edit of a diagram derives only what it reaches, text sets its fields on the GPU, geometry no longer answers the pointer unless asked, and diagrams draw what focus and drags mean.
+
+- Added: `gpu.step(now)` draws one frame now, and from the first no animation frames are asked for: a harness steps every frame itself; `gpu.done()` resolves once the GPU finished what was submitted, every readback landed, and every pipeline asked for compiled; `gpu.landed()` once work a view waits on lands after the call, or a view asks for a frame; `view.wantsFrame`.
+- Added: `Work.late` counts pipelines made without being asked for ahead; `Waiting` names `compiling` while the pipelines a view draws with compile.
+- Changed: a view asks for every pipeline its scene can use the first time it draws, and the browser compiles them off its GPU thread, the largest first: what it draws with all at once, the view drawing once that landed and keeping its last frame until then; what lights rows, probes, and answers picks a few at a time while frames draw, those waiting until it landed. No frame waits on a compile and nothing compiles where it is first used; `complete` waits for all of it.
+- Changed: the wasm is built without wasm-opt: as small over the wire, and views open sooner, the browser no longer compiling one giant function before their first frame.
+- Changed: one probe program per kind of mark answers the hover, picks, boxes, and `locate`, a step at a time: a page compiles about a third fewer pipelines and finishes opening sooner.
+- Changed: `pick` lists every row within reach, however many: a mark with more than 1,024 asks again with room for all of them.
+- Changed: text sets its distance fields on the GPU, every run set in a frame at once: the host only rasterizes, so labels fill in faster.
+- Changed: an edit of the values a diagram sizes and places rows by derives only the rows it wrote and what they reach: their boxes, the groups around them, and the wires they move. A text column written over codes only the rows written.
+- Changed: appends one after another to a table, as a stream of frames makes, write as one change.
+- Changed: unless `pick` says, rows that are things answer the pointer (vertices, edges, faces, a plot's traces) and geometry does not (areas, and lines along paths: coasts, borders, roads); `pick: true` makes a map's states or cells answer it.
+- Changed: a fit in a diagram frames its sheet, blocks and wires, at once without a readback; labels place once a fit has moved the camera; a mark with no rows no longer holds a fit back.
+- Changed: a patch of values in a table drawn in part writes only the rows whose values changed.
+- Changed: a `drag` event's `position` is where the dragged row goes: where it was grabbed, moved as the pointer moved, snapped to a diagram's dots when its grid snaps.
+- Changed: error paths run through the mark's kind, as its JSON does: `marks[2].vertices.color`.
+- Changed: the expression node types are named for what they are: `FieldNode`, `ColorNode`, … `ColormapNode`; a size in units is `Units`, your own colormap `CustomColormap`.
+- Fixed: lines seen up close no longer break up or vanish: a stroke reaching behind the eye is cut at the near plane instead of dropped, and its pixels are placed on screen however differently deep its ends lie.
+- Fixed: a hovered or selected group glows about its border instead of hiding what it holds; a lit block keeps its ports on top; wires stop at the rims of ports, so arrowheads show; rounded wires draw no stray junction dots; an unlabeled block makes room for its ports; great circles settle while their vertices animate.
+- Fixed: ports and group borders read against light themes.

.changeset/panel-text.md

@@ -0,0 +1,6 @@
+---
+'latkit': patch
+---
+
+Keep font-atlas uploads in command order when several views render before one GPU submission.
+Multi-panel pictures and videos now retain every panel's titles, axis labels, and legends.

.changeset/release-contract.md

@@ -0,0 +1,11 @@
+---
+'latkit': minor
+---
+
+Establish the LATK v1 format and tighten the npm, Rust, and CLI release contracts.
+
+- LATK now has a versioned header, 64-bit absolute sample indexes, explicit stored detail levels, and bounded decoding. Regenerate earlier unversioned files and time tiles from their source data.
+- Added `latkit/core` for DOM-free builders and CPU-only Rust expressions, queries, and data loading.
+- Fixed video cancellation and output cleanup, arbitrary query field names, URL resolution, and shared GPU device callback ownership.
+- Public declarations and API/CLI references now come from checked sources. Packages include consistent MIT licensing and third-party notices.
+- Releases verify the installed npm tarball and both Cargo artifacts, and resume missing publications safely before tagging.

.github/workflows/ci.yml

@@ -1,14 +1,70 @@
 name: CI
 
+# What a release depends on: the crates build, pass their tests, document, and package on every
+# platform they ship to; the npm tarball works in a consumer; the docs build. Frame budgets and the
+# benchmark need a GPU, which runners lack: run `cargo xtask check` and `cargo xtask bench` locally.
 on:
   pull_request:
   push:
-    branches:
-      - main
+    branches: [main]
+
+permissions:
+  contents: read
+
+# A new push to a pull request cancels its older run; every commit to main runs to the end, since
+# the release waits on it.
+concurrency:
+  group: ci-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_INCREMENTAL: 0
 
 jobs:
-  quality:
+  # What a pull request touched; a push to main checks everything.
+  changes:
     runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: read
+      pull-requests: read
+    outputs:
+      rust: ${{ github.event_name != 'pull_request' || steps.filter.outputs.rust == 'true' }}
+      npm: ${{ github.event_name != 'pull_request' || steps.filter.outputs.npm == 'true' }}
+      docs: ${{ github.event_name != 'pull_request' || steps.filter.outputs.docs == 'true' }}
+    steps:
+      - uses: dorny/paths-filter@v4
+        id: filter
+        if: github.event_name == 'pull_request'
+        with:
+          filters: |
+            rust: &rust
+              - 'crates/**'
+              - 'xtask/**'
+              - '.cargo/**'
+              - '*.toml'
+              - 'Cargo.lock'
+              - 'docs/reference/**'
+              - '.github/workflows/ci.yml'
+            npm:
+              - *rust
+              - 'npm/**'
+              - 'benchmark/**'
+              - 'package.json'
+              - 'pnpm-*.yaml'
+              - '*.config.*'
+              - 'tsconfig*.json'
+            docs:
+              - 'docs/**'
+              - '.readthedocs.yaml'
+              - 'npm/test/consumer/quickstart.ts'
+              - '.github/workflows/ci.yml'
+
+  # Prettier over every file: cheap, and any pull request can unformat one.
+  format:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
     steps:
       - uses: actions/checkout@v6
       - uses: pnpm/action-setup@v6
@@ -20,30 +76,68 @@ jobs:
           cache: pnpm
       - run: pnpm install --frozen-lockfile
       - run: pnpm format:check
-      - run: pnpm build
-      - run: pnpm lint
-      - run: pnpm build:examples
-      - run: pnpm typecheck
-      - run: pnpm test:run
-      - run: pnpm -r --filter "./packages/**" exec npm pack --dry-run
 
-  bench:
-    if: github.event_name == 'pull_request'
+  rust:
+    needs: changes
+    if: needs.changes.outputs.rust == 'true'
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
-      - uses: pnpm/action-setup@v6
+      - run: rustup toolchain install
+      - uses: Swatinem/rust-cache@v2
         with:
-          version: 10.30.0
-      - uses: actions/setup-node@v6
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - run: cargo fmt --all --check
+      - run: cargo clippy --locked --workspace --all-targets --all-features -- -D warnings
+      - run: cargo clippy --locked -p latkit --no-default-features --all-targets -- -D warnings
+      - run: cargo clippy --locked -p latkit-js --target wasm32-unknown-unknown -- -D warnings
+      # Lavapipe, Vulkan on the CPU, runs the GPU tests instead of skipping; text tests need fonts.
+      - name: Install a software Vulkan driver and fonts for the GPU tests
+        run: sudo apt-get update && sudo apt-get install -y mesa-vulkan-drivers fonts-dejavu-core
+      - run: cargo test --locked --workspace
+        env:
+          LATKIT_REQUIRE_GPU: 1
+      - run: cargo test --locked -p latkit --no-default-features
+        env:
+          LATKIT_REQUIRE_GPU: 1
+      - name: Document the library
+        run: cargo doc --locked -p latkit --no-deps --all-features
+        env:
+          RUSTDOCFLAGS: -D warnings
+      - name: Verify the native package
+        run: cargo package --locked -p latkit
+      - name: Verify the browser-target package
+        run: cargo package --locked -p latkit --target wasm32-unknown-unknown --features web
+      - name: Keep native font code out of browser builds
+        run: cargo check --locked -p latkit --target wasm32-unknown-unknown --features native,web
+
+  # The CLI's native code on the other platforms it installs on; Linux runs in `rust`.
+  native-cli:
+    needs: changes
+    if: needs.changes.outputs.rust == 'true'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [macos-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v6
+      - run: rustup toolchain install
+      - uses: Swatinem/rust-cache@v2
         with:
-          node-version: 24
-          cache: pnpm
-      - run: pnpm install --frozen-lockfile
-      - run: pnpm bench:gate
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - run: cargo test --locked --package latkit --features native --lib --test validation
+      - run: cargo test --locked --package latkit-cli
 
-  docs:
+  # The npm package: built from the crate, checked, tested, every pipeline compiled in Chrome on
+  # SwiftShader, and the packed tarball installed and drawn in a consumer.
+  npm:
+    needs: changes
+    if: needs.changes.outputs.npm == 'true'
     runs-on: ubuntu-latest
+    timeout-minutes: 40
     steps:
       - uses: actions/checkout@v6
       - uses: pnpm/action-setup@v6
@@ -53,9 +147,56 @@ jobs:
         with:
           node-version: 24
           cache: pnpm
+      - run: rustup toolchain install
+      - uses: Swatinem/rust-cache@v2
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - run: pnpm install --frozen-lockfile
+      - name: Read the WASM bindings generator's version
+        id: bindgen
+        run: |
+          echo "version=$(python3 -c 'import tomllib; print(next(p["version"] for p in tomllib.load(open("Cargo.lock", "rb"))["package"] if p["name"] == "wasm-bindgen"))')" >> "$GITHUB_OUTPUT"
+      - uses: actions/cache@v4
+        id: bindgen-cache
+        with:
+          path: ~/.cargo/bin/wasm-bindgen
+          key: wasm-bindgen-cli-${{ steps.bindgen.outputs.version }}
+      - name: Install the matching WASM bindings generator
+        if: steps.bindgen-cache.outputs.cache-hit != 'true'
+        run: cargo install wasm-bindgen-cli --version "${{ steps.bindgen.outputs.version }}" --locked
+      - run: pnpm --dir npm/examples exec playwright-core install --with-deps chromium
+      - run: cargo xtask wasm --dev
+      - run: pnpm lint
+      - run: node npm/docs.mjs --check
+      - run: pnpm typecheck
+      - run: pnpm test
+      - run: cargo xtask browser
+      - run: cargo xtask package
+      - run: cargo xtask package-test
+
+  # The docs as Read the Docs builds them: a warning fails.
+  docs:
+    needs: changes
+    if: needs.changes.outputs.docs == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v6
       - uses: actions/setup-python@v6
         with:
           python-version: '3.13'
-      - run: pnpm install --frozen-lockfile
-      - run: python -m pip install -r docs/requirements.txt
-      - run: pnpm docs:build
+          cache: pip
+          cache-dependency-path: docs/requirements.txt
+      - run: pip install -r docs/requirements.txt
+      - run: sphinx-build -W --keep-going -b html docs target/docs
+
+  # The one check branch protection requires: every job above passed, or was not needed.
+  passed:
+    name: CI passed
+    if: always()
+    needs: [changes, format, rust, native-cli, npm, docs]
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
+        run: exit 1