diff --git a/.changeset/api.md b/.changeset/api.md new file mode 100644 index 00000000..91a3f14d --- /dev/null +++ b/.changeset/api.md @@ -0,0 +1,15 @@ +--- +'latkit': minor +--- + +`View.create` opens a view in one call, a mark's kind key holds its table, and `Data` applies changes directly. + +- Added: `View.create(canvas, scene, { region, camera, sources, gpu, pixelRatio })` draws a view on a `` or `OffscreenCanvas`, over any views already there. With a `null` canvas, the view draws only into pictures. +- Added: views share the first `Gpu` created, and `View.create` creates one if there is none. `view.region` moves a view, and `view.gpu` is its GPU. +- Added: `data.table(name)` and `tiles.table(name)` return a `Table` for a mark to draw or `field(name).from(table)` to read. A view loads every dataset its scene's tables belong to, so `sources` lists only datasets named as `[data, table]`. +- Added: a `Data` method for every `Patch` change, applied immediately: `axis`, `replace`, `set`, `remove`, `append`, `forget`, `frames`. +- Added: `Shot.of(view, options)` captures what a view's canvas shows. `new Shot(size, { views })` draws views in their own regions, or in another region given as `{ view, region }`. +- Changed: a mark's kind key holds its table, next to its options: `{ vertices: grid.table('Bus'), radius: 4 }`, or `{ "vertices": ["grid", "Bus"], … }` in JSON. This replaces `{ vertices: { of: ['grid', 'Bus'], radius: 4 } }`. +- Changed: errors name the option's path, such as `marks[1].radius`. +- Changed: `replace` and `set` given only fields write rows `0` to `n - 1`. +- Removed: `gpu.view`, `gpu.canvas`, `Canvas` (with `canvas.show(views)` and its `panels`), and the `Panel` type. diff --git a/.changeset/camera.md b/.changeset/camera.md new file mode 100644 index 00000000..70787ba4 --- /dev/null +++ b/.changeset/camera.md @@ -0,0 +1,14 @@ +--- +'latkit': minor +--- + +A flat map shows one world, and every camera move stays within the camera's zoom and pan limits. + +- Changed: a flat map's ground ends at the antimeridian, as it ends at Web Mercator's latitude limit. +- Changed: on a flat map, the camera zooms out only until the whole world fits inside the insets, and pans only until the world's edges meet the view's edges. +- Changed: on a globe, the camera zooms out until the globe spans a third of the view's shorter side. +- Changed: setting the camera, flights, fits, orbits, and gestures keep the camera within its zoom and pan limits, and its pitch within 75°. After a fit, the limits narrow to the area the fit framed. +- Changed: a `camera` event fires when a resize, new insets, or a new space moves the camera back within its limits. +- Changed: on a flat map, a flight eases across the world instead of the short way over the antimeridian. On a globe, longitude still turns the short way. +- Fixed: panning a flat map no longer wraps its center across the antimeridian, which moved every row a world away. +- Fixed: a scene with another kind of space drops the old space's fits and any camera motion in progress. diff --git a/.changeset/labels.md b/.changeset/labels.md new file mode 100644 index 00000000..3c6d7db4 --- /dev/null +++ b/.changeset/labels.md @@ -0,0 +1,15 @@ +--- +'latkit': minor +--- + +Labels are placed from each frame alone, with no fades. Where labels would overlap, the row drawn on top keeps its label. + +- Added: a label's `text` takes a `when` that picks the text field or `null`. A row whose text is missing has no label, so `when(zoom().ge(6)).then(field('name')).otherwise(null)` labels rows from zoom 6 on. +- Changed: a row whose text is empty has no label, like a row whose text is missing. +- Changed: a label shows when its row is visible, its text is set, and the text fits, for example inside a block. +- Changed: labels beside or along their rows never overlap. Where they would, the row drawn on top wins, so a mark's `priority` decides. +- Changed: only labels that show need their texts, so a view of any size settles with as many texts as fit on screen. +- Changed: labels move with the camera in the frame it moves, keep showing while their mark gains texts, and never take another row's state when tiles change. +- Changed: when the text atlas is full of texts in view, it accepts no more until the view changes, instead of evicting texts it still shows. +- Changed: `'labels'` in `Waiting` means label texts are being read back or are not yet set. +- Removed: a label's `priority` and `max`, and label fades. A mark's `priority` orders its drawing, its thinning, and its labels. diff --git a/.changeset/sources.md b/.changeset/sources.md new file mode 100644 index 00000000..e32ea90f --- /dev/null +++ b/.changeset/sources.md @@ -0,0 +1,77 @@ +--- +'latkit': minor +--- + +Views draw from sources, every wait can be cancelled, colors and lengths are typed, and large tables draw in pages that stay on the GPU. + +- Added: `view.sources`. A source is a `Data`, drawn at its latest version from the frame after each patch. It can also be a `Snapshot`, one fixed version (`data.snapshot()`, `data.restore(snapshot)`), or `Tiles(name, url)`, time tiles or vector tiles fetched once per GPU. +- Added: `view.snapshot(name)` returns what a view currently draws of a source. +- Added: `load({ wasm, worker })` loads the WebAssembly. Using the library before it finishes throws `unloaded`. +- Added: every class and builder is exported from `latkit`. +- Added: every wait takes `{ signal }` and times out after 30 seconds by default: picks, `screenOf`, `arrange`, pictures, encodings, videos, `gpu.done()`, and `gpu.landed()`. A timeout rejects with `timeout`. An abort or a destroyed view rejects with `cancelled` and drops the work in progress. +- Added: `new Shot(size, { scale, transparent, capture, elapsed })`, where `capture: 'now'` takes views as they are, and `shot.video({ fps, output, bitrate, signal })`. +- Added: `px(n)` and `units(n)` set a length's unit. Lengths default to position units in diagrams and CSS pixels elsewhere. A stroke in pixels keeps its width as a diagram zooms. +- Added: theme groups `text`, `vertex`, `edge`, `line`, `area`, `focus`, `flow`, `diagram`, `earth`, and `plot`. +- Added: a comet's `flow.spacing` and `flow.length` take `{ px }` or `{ units }`. +- Added: `diagram.outline`, `corner`, `portSize`, and `bend` style blocks, groups, ports, and wire bends. +- Added: `shadow: true` on blocks and groups casts a soft shadow `diagram.shadow.lift` below them. +- Added: `Gpu.resident` and `Gpu.uploading`, the bytes on the GPU and the bytes waiting to upload, across all its views. +- Added: `Work.packed`, `Work.resolved`, `Work.cleared`, and `Work.uploads`. +- Added: `Handle`, the base class of every object that holds memory (`Data`, `Snapshot`, `Gpu`, `View`, `Image`), with `destroy()` and `using`. +- Added: `Dataset`, the reads that `Data` and `Snapshot` share. +- Added: `colormaps()` lists the catalog's colormaps with their names, labels, and kinds. +- Added: the package's types document every member, enumeration value, union form, and options field. +- Added: the CLI's `--camera` takes a camera as JSON or as a file, like `latkit query` takes a query. +- Changed: `view.scene`, `view.selection`, `view.screen(position)`, `view.position(point)`, `view.screenOf(row)`, `view.waiting`, and the `hovered` and `selection` events replace the methods and events removed below. +- Changed: every error the package throws is a `LatkitError` with a `code`. +- Changed: a failed fetch or an error status is `io`. A call on a destroyed object has the new code `destroyed`. A bad `Patch.stream` limit and a `Video` drawn out of turn are `invalid`. +- Changed: errors from the video encoder and worker jobs keep their codes. An abort during a fetch rejects as `cancelled` or `timeout`. +- Changed: latkit's bytes carry no version or flags. A frame's 12-byte header is the magic `LATK` and the frame's length. Files written by earlier versions no longer open. +- Changed: `theme.missing` is the color a colormap gives a missing value. An option given a missing value (`null`) still takes its theme value. +- Changed: `new Patch(dataset)` names its dataset. +- Changed: `Patch.read(source)` reads a patch's bytes or JSON, and `Data.read(source, { name, base })` reads a dataset's bytes, JSON, or GeoJSON. Both read from a `ByteSource` (bytes, a `Blob`, a `Response`, a byte stream or async iterable, or a URL) within the wait's `signal`. +- Changed: the signatures `patch.append(table, at, fields, { ids })`, `patch.frames(table, field, { level, first, at, values })`, and `data.envelope({ table, field, range, spans, ids })`. +- Changed: a video's size rounds down to whole 4×2 blocks, in the browser and the CLI. A smaller size is `invalid`. +- Changed: `view.screen` and `view.position` given the wrong number of coordinates throw `invalid` instead of returning `null`. +- Changed: the `'earth'` space takes `{ earth: { projection: 'mercator' | 'globe', graticule, daylight, atmosphere } }`. Its ground color is the theme's `earth.ground`. +- Changed: a table too large to draw at once draws in pages that keep their place on the GPU as the camera moves. +- Changed: pages no longer drawn stay on the GPU while the budget allows, so coming back to them is free. +- Changed: a write rebuilds only the pages it touched, and appending rows never thins what draws. +- Changed: lanes, net stars, joins, and fitted domains follow the data, not the camera. +- Changed: edges with far-apart ends draw within the budget. Their end rows load one by one, and only edges whose ends moved look them up again. +- Changed: a hit describes the frame it came from. +- Changed: all views of a GPU draw one frame together, and `Budget.uploadPerFrame` caps what they upload in it. +- Changed: editing a diagram recomputes only the moved blocks, the groups around them, and the wires near them. +- Changed: map tiles draw in the frame they finish loading. +- Changed: a plot measures the space its axis labels and titles need. +- Changed: a GPU picks its antialiasing once. +- Changed: zooming at a fit's limit keeps the center. +- Removed: `view.setData`, `view.setTiles`, `view.removeData`, and `view.data(name)`. Use `view.sources`. +- Removed: `view.show(scene)`, `view.select(rows)`, `view.point`, `view.ground`, `view.locate`, and the `hover` and `select` events. +- Removed: `Stats.waiting`, `Stats.uploading`, `Stats.resident`, and `Stats.budget`. +- Removed: `loadLatkit()` and its result object, and `latkit/core`. +- Removed: `createWorker`. Use `load({ worker })`. +- Removed: `{ deadline }` options. Use `{ signal }`. +- Removed: `Shot`'s builder methods, `shot.image(views)`, and `shot.video(gpu, fps, output)`. +- Removed: `Patch.to`, `Patch.fromBytes`, `Data.load`, and `Data.tables`. +- Removed: `data.ids()`, `data.fields()`, and `data.span`. Use `describe()`. +- Removed: the positional `envelope`. +- Removed: `expr.of(data, table)`. Use `from`. +- Removed: the `'mercator'` and `'globe'` spaces. Use `earth`. +- Removed: flat theme keys such as `vertexRadius`, `fontSize`, and `edgeSpacing`. +- Removed: reading a string as a color only when it begins with `#`. +- Removed: `Flow.units`. +- Removed: shadows on edges, lines, areas, and faces, and the theme's `shadow.light` and `shadow.softness`. +- Fixed: when a view's next version does not fit, the view releases what it draws so the new version can load. If nothing fits, it reports a `limit` error and stops. +- Fixed: on the earth, sizes in `units(n)` are degrees, like comet lengths, not Mercator radians. +- Fixed: zoomed in close on the globe, paged tables and vector tiles keep drawing their finer tiles. Past about zoom 12 they fell back to the coarsest. +- Fixed: a text the host cannot rasterize reports one error, not one every frame it shows. A text with nothing to draw is not an error. +- Fixed: a layout or shaping job that fails runs once until its inputs change, not every frame. +- Fixed: at the GPU's texture-layer limit, a new icon fails with a `limit` error instead of overwriting the last one. +- Fixed: `view.screenOf` returns `null` for a hidden stroke or face, and for a face beyond the globe's horizon. +- Fixed: edges and faces no longer compile an unused bounds pipeline before their first frame. +- Fixed: a `priority` that tests a field only with `contains()` now reads that field when ranking a paged table's rows. +- Fixed: a JSON patch's errors name the failing change the same way wherever they are found, such as `[0].append.fields.x`. +- Fixed: a tile index that is neither TileJSON nor PMTiles fails as `invalid`, and tile errors keep their code. +- Fixed: when the CLI cannot write an output, its error names that output, not the frames file. +- Fixed: a transition that starts in the frame a mark's tables grow no longer writes its rows over other data on the GPU, which could draw rows in the wrong colors or places. diff --git a/.changeset/spans.md b/.changeset/spans.md new file mode 100644 index 00000000..692f9f2a --- /dev/null +++ b/.changeset/spans.md @@ -0,0 +1,10 @@ +--- +'latkit': minor +--- + +The command line plays an axis span. + +- Added: `--span FROM,TO` and `--rate`. A video or image sequence without `--frames` plays the datasets' span. +- Added: with `--span`, each `--frames` line applies at its `at`, and the camera flies between lines that set it. +- Added: `view.fly(camera, seconds)`. +- Changed: setting or flying the camera replaces a pending fit. diff --git a/.changeset/tiles.md b/.changeset/tiles.md new file mode 100644 index 00000000..a519199e --- /dev/null +++ b/.changeset/tiles.md @@ -0,0 +1,14 @@ +--- +'latkit': minor +--- + +A vector tile costs only the rows it brings, and the tiles drawn largest split first. + +- Changed: an arriving vector tile appends its rows, so rows already drawn do not upload or evaluate again. +- Changed: a tile that stops drawing keeps its rows and draws them again if it comes back. Once rows that no tile draws outnumber the drawn rows by a page, they are all removed. Until then, `view.snapshot(name)` of a vector source includes them. +- Changed: tiles split by the area they cover on screen, so tiles near the globe's edge and a tilted map's horizon stay coarse. +- Changed: when the tile budget runs out, the tiles drawn largest split first, for vector tiles and paged tables alike. +- Changed: at most 128 vector tiles draw at once. +- Changed: a vector table draws each tile's rows in the order of the mark's `priority`, highest on top. +- Fixed: a flat map no longer fetches tiles past the antimeridian as if they were near. +- Fixed: two tiles arriving together with different kinds for the same field no longer fail every update. diff --git a/.changeset/values.md b/.changeset/values.md new file mode 100644 index 00000000..bb559d21 --- /dev/null +++ b/.changeset/values.md @@ -0,0 +1,17 @@ +--- +'latkit': minor +--- + +One way to write a value in every option and query: `'$name'` reads a field, any other string is a CSS color, and `null` is missing. + +- Changed: in a scene or a query, a string `'$name'` reads field `name` and any other string is a CSS color, whatever the option. `field(name)` writes `'$name'`. +- Changed: a color option that computes a number or a boolean is an error, and the error says how to map it to colors. +- Changed: an operation is an object with one key, its name: `{ add: [a, b] }`, `{ not: [a] }`, `{ is: ['$name', 'Austin'] }`. +- Changed: `null` is a missing value. +- Changed: one `when` chooses between values and colors alike. +- Changed: `.colormap()` returns `Mapped`, like `.scale()`. +- Changed: the layered layout's `flow` is renamed `direction`, so `flow` means only comets. +- Removed: `{ op, args }`, `{ color }`, and `{ missing: true }`. +- Removed: `Paint`, `PaintInput`, `Colormapped`, `Missing`, `color()`, and `missing()`. +- Removed: bare strings as field names. +- Removed: in Rust, converting a string or a `Name` into `Paint` or `Expr`. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 834fff28..a7c8ff32 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,8 +1,10 @@ name: CI # What a release depends on: the crates build, pass their tests, document, and package on every -# platform they ship to; the npm tarball works in a consumer; the docs build. Frame budgets and the -# benchmark need a GPU, which runners lack: run `cargo xtask check` and `cargo xtask bench` locally. +# platform they ship to; the npm tarball works in a consumer; the docs build. The jobs that build +# Rust run their steps as `cargo xtask ci `, and `cargo xtask check` runs every job's steps. +# Frame budgets and the benchmark need a GPU, which runners lack: run `cargo xtask check` and +# `cargo xtask bench` locally. on: pull_request: push: @@ -59,7 +61,6 @@ jobs: - 'docs/**' - '.readthedocs.yaml' - 'npm/test/consumer/quickstart.ts' - - 'npm/examples/src/examples/**' - '.github/workflows/ci.yml' # Prettier over every file: cheap, and any pull request can unformat one. @@ -89,29 +90,13 @@ jobs: - uses: Swatinem/rust-cache@v2 with: save-if: ${{ github.ref == 'refs/heads/main' }} - - run: cargo fmt --all --check - - run: cargo clippy --locked --workspace --all-targets --all-features -- -D warnings - - run: cargo clippy --locked -p latkit --no-default-features --all-targets -- -D warnings - - run: cargo clippy --locked -p latkit-js --target wasm32-unknown-unknown -- -D warnings # Lavapipe, Vulkan on the CPU, runs the GPU tests instead of skipping; text tests need fonts. - - name: Install a software Vulkan driver and fonts for the GPU tests - run: sudo apt-get update && sudo apt-get install -y mesa-vulkan-drivers fonts-dejavu-core - - run: cargo test --locked --workspace - env: - LATKIT_REQUIRE_GPU: 1 - - run: cargo test --locked -p latkit --no-default-features - env: - LATKIT_REQUIRE_GPU: 1 - - name: Document the library - run: cargo doc --locked -p latkit --no-deps --all-features - env: - RUSTDOCFLAGS: -D warnings - - name: Verify the native package - run: cargo package --locked -p latkit - - name: Verify the browser-target package - run: cargo package --locked -p latkit --target wasm32-unknown-unknown --features web - - name: Keep native font code out of browser builds - run: cargo check --locked -p latkit --target wasm32-unknown-unknown --features native,web + # Cached, since apt took up to 7 minutes. + - uses: awalsh128/cache-apt-pkgs-action@v1 + with: + packages: mesa-vulkan-drivers fonts-dejavu-core + version: 1 + - run: cargo xtask ci rust # The CLI's native code on the other platforms it installs on; Linux runs in `rust`. native-cli: @@ -129,11 +114,10 @@ jobs: - uses: Swatinem/rust-cache@v2 with: save-if: ${{ github.ref == 'refs/heads/main' }} - - run: cargo test --locked --package latkit --features native --lib --test validation - - run: cargo test --locked --package latkit-cli + - run: cargo xtask ci native-cli - # The npm package: built from the crate, checked, tested, every pipeline compiled in Chrome on - # SwiftShader, and the packed tarball installed and drawn in a consumer. + # The npm package: built once from the crate in release, checked, tested, every pipeline compiled + # in Chrome on SwiftShader, and that build packed, installed, and drawn in a consumer. npm: needs: changes if: needs.changes.outputs.npm == 'true' @@ -156,7 +140,8 @@ jobs: - name: Read the WASM bindings generator's version id: bindgen run: | - echo "version=$(python3 -c 'import tomllib; print(next(p["version"] for p in tomllib.load(open("Cargo.lock", "rb"))["package"] if p["name"] == "wasm-bindgen"))')" >> "$GITHUB_OUTPUT" + version=$(cargo xtask bindgen-version) + echo "version=$version" >> "$GITHUB_OUTPUT" - uses: actions/cache@v4 id: bindgen-cache with: @@ -165,15 +150,10 @@ jobs: - name: Install the matching WASM bindings generator if: steps.bindgen-cache.outputs.cache-hit != 'true' run: cargo install wasm-bindgen-cli --version "${{ steps.bindgen.outputs.version }}" --locked - - run: pnpm --dir npm/examples exec playwright-core install --with-deps chromium - - run: cargo xtask wasm --dev - - run: pnpm lint - - run: node npm/docs.mjs --check - - run: pnpm typecheck - - run: pnpm test - - run: cargo xtask browser - - run: cargo xtask package - - run: cargo xtask package-test + # The runner's Chrome drives the browser gate. + - run: cargo xtask ci npm + env: + LATKIT_CHROME: /usr/bin/google-chrome # The docs as Read the Docs builds them, from the locked environment: a warning fails. docs: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f3ec8441..4a013368 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -90,7 +90,8 @@ jobs: if: env.PUBLISH == 'true' id: bindgen run: | - echo "version=$(python3 -c 'import tomllib; print(next(p["version"] for p in tomllib.load(open("Cargo.lock", "rb"))["package"] if p["name"] == "wasm-bindgen"))')" >> "$GITHUB_OUTPUT" + version=$(cargo xtask bindgen-version) + echo "version=$version" >> "$GITHUB_OUTPUT" - uses: actions/cache@v4 if: env.PUBLISH == 'true' id: bindgen-cache diff --git a/.prettierignore b/.prettierignore index 25ee150d..a4d40692 100644 --- a/.prettierignore +++ b/.prettierignore @@ -10,9 +10,16 @@ npm/pkg npm/dist crates/latkit/colormaps npm/test/scenes.js -# Written by the CLI examples' scripts, and by the bench. +# Written by the CLI examples' scripts, the examples' history script, and the bench. crates/latkit-cli/examples/**/*.json +npm/examples/public/history benchmark/baselines # Written by the schema test: LATKIT_BLESS=1 cargo test -p latkit --test schema. npm/schema.json docs/_static/theme.json +# Written by LATKIT_BLESS=1 cargo test -p latkit-cli, then node npm/docs.mjs, in MyST's definition +# lists, which Prettier would break. +docs/reference/cli.json +docs/reference/api.md +docs/reference/types.md +docs/reference/cli.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0d59f203..70a64bf4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,42 +2,47 @@ ## Repository -| Part | What it is | -| --------------------------------- | -------------------------------------------------------------- | -| [`latkit`](crates/latkit) | The Rust library: data, expressions, scenes, views, and video | -| [npm `latkit`](npm) | The library in the browser, with types for every scene | -| [`latkit-cli`](crates/latkit-cli) | Pictures and video of scenes, from JSON or binary data | -| [Examples](npm/examples) | Every feature on a page, and the browser gate that drives them | -| [Benchmark](benchmark) | Every scenario's phases up its sizes, held to committed work | -| [Docs](docs) | The site on [Read the Docs](https://latkit.readthedocs.io) | -| [Gallery](docs/gallery) | The documentation's pictures, each drawn by latkit | +| Part | What it is | +| --------------------------------- | ----------------------------------------------------------------------- | +| [`latkit`](crates/latkit) | The Rust library: data, expressions, scenes, views, and video | +| [`latkit-js`](crates/latkit-js) | The WebAssembly bindings the npm package wraps | +| [npm `latkit`](npm) | The library in the browser, with types for every scene | +| [`latkit-cli`](crates/latkit-cli) | Pictures and video of scenes, from JSON or binary data | +| [Examples](npm/examples) | Small examples to copy, and the browser gate that tests them | +| [Benchmark](benchmark) | Every scenario at increasing sizes, checked against committed baselines | +| [Docs](docs) | The site on [Read the Docs](https://latkit.readthedocs.io) | +| [Gallery](docs/gallery) | The documentation's pictures, each drawn by latkit | +| [`xtask`](xtask) | Repository tasks, run as `cargo xtask ` | The `@latkit/*` TypeScript packages, a lighter alternative without WebAssembly, are maintained on the [`npm/js`](https://github.com/lukelowry/latkit/tree/npm/js) branch. ## Develop -Requires Rust (pinned in `rust-toolchain.toml`), Node.js 24, pnpm 10.30, and for the docs, uv. +You need Rust (pinned in `rust-toolchain.toml`), Node.js 24, pnpm 10.30, and uv for the docs. ```sh pnpm install cargo xtask examples --open # the example pages at http://127.0.0.1:5200 -cargo xtask check # format, lint, test, typecheck, and the browser gate +cargo xtask check # every CI job's steps: format, lint, test, browser gate, package cargo xtask bench --quick # every benchmark scenario once, at its smallest size ``` ## Documentation -The site builds with Python alone, from what the repository checks in; each generated part has the -command that writes it again: +The site builds with Python alone, from checked-in files. To rebuild the site and its generated +parts: ```sh uv run --project docs --group docs sphinx-build -W -b html docs target/docs # the site -node npm/docs.mjs # the JavaScript API and type references +LATKIT_BLESS=1 cargo test -p latkit-cli # the CLI's command tree, docs/reference/cli.json +node npm/docs.mjs # the API, type, and command references LATKIT_BLESS=1 cargo test -p latkit --test schema # the palette: latkit's default themes cargo build --locked -p latkit-cli --profile cli node docs/gallery/render.ts [name...] # the pictures, drawn by the latkit CLI node benchmark/docs.ts # the performance table, from a baseline ``` +Drawing the pictures also needs the npm package, built with `cargo xtask wasm`. + [Releasing](RELEASING.md) covers versions and publishing. diff --git a/README.md b/README.md index e3318241..b171f3ac 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- The ACTIVSg70k grid on the globe as one 765 kV line opens: buses raised by their frequency, branches that pick up its power glow amber, those it leaves glow cyan + The ACTIVSg70k grid on a globe as one 765 kV line opens. Buses rise with their frequency, branches that pick up the line's power glow amber, and branches that lose flow glow cyan

@@ -36,39 +36,32 @@ cargo add latkit # in Rust ## In the browser ```ts -import { loadLatkit, Patch, field, type Scene } from 'latkit'; +import { Data, View, field, load } from 'latkit'; -const { Data, Gpu } = await loadLatkit(); -const data = new Data('grid').apply( - new Patch().replace('Bus', [101, 102], { x: [0, 30], y: [0, 20], kv: [230, 115] }), -); -const scene: Scene = { +await load(); +const grid = new Data('grid').set('Bus', [101, 102], { x: [0, 30], y: [0, 20], kv: [230, 115] }); + +const view = await View.create(document.querySelector('canvas')!, { space: 'plane', marks: [ { - vertices: { - of: ['grid', 'Bus'], - position: ['x', 'y'], - radius: 6, - color: field('kv').colormap('viridis'), - }, + vertices: grid.table('Bus'), + position: ['$x', '$y'], + radius: 6, + color: field('kv').colormap('viridis'), }, ], -}; +}); -const gpu = await Gpu.create(); -const view = gpu.view(scene); -gpu.canvas(document.querySelector('canvas')!).show([view]); -view.setData(data); -view.fit(); +grid.set('Bus', [102], { kv: [345] }); // drawn the next frame ``` -Scenes, expressions, and patches are plain data, typed from latkit's schema: `latkit/core` builds -them without loading WebAssembly, in a worker or on a server. +A scene is plain data, with types from latkit's schema. `JSON.stringify(scene)` writes the JSON the +command line reads. ## On the command line -The same scene and data, as JSON, render to a picture, or with `--frames` to video: +Write the same scene and data as JSON to render a picture, or a video with `--frames`: ```sh latkit render scene.json --data grid.json --output grid.png @@ -85,12 +78,10 @@ latkit render scene.json --data grid.json --frames frames.jsonl --fps 60 --outpu "space": "plane", "marks": [ { - "vertices": { - "of": ["grid", "Bus"], - "position": ["x", "y"], - "radius": 6, - "color": { "colormap": "viridis", "value": { "field": "kv" } } - } + "vertices": ["grid", "Bus"], + "position": ["$x", "$y"], + "radius": 6, + "color": { "colormap": "viridis", "value": "$kv" } } ] } @@ -119,19 +110,19 @@ latkit render scene.json --data grid.json --frames frames.jsonl --fps 60 --outpu

Two thousand buses of a synthetic Texas grid: generators as amber discs sized by output, comets along its highest-voltage lines - The speed of 1,456 machines over ten seconds after a fault, near ones bright and swinging hardest, far ones dark - Routes between forty of the world's great cities, along great circles on a globe, comets flowing on them - A microgrid controller as blocks in nested groups, wired port to port with routed wires and a feedback loop + The speed of 1,456 machines over ten seconds after a fault. Machines near the fault are bright and swing hardest, and far ones are dark + Great-circle routes between forty of the world's great cities on a globe, with comets flowing along them + A microgrid controller as blocks in nested groups, with wires routed port to port and a feedback loop Sixty-four filaments in cyan, amber, and coral, twisting around a three-dimensional torus knot - A Clifford strange attractor of 260,000 translucent points, colored by the way each step turns + A Clifford strange attractor of 260,000 translucent points, colored by the direction of each step

-Every picture is drawn by latkit; [the examples](https://latkit.readthedocs.io/en/latest/examples/) hold them all with the code that draws them. +latkit drew every picture here. [The examples](https://latkit.readthedocs.io/en/latest/examples/) show the code for each one. ## Documentation -Guides, examples, and the reference are on [Read the Docs](https://latkit.readthedocs.io); the Rust -API is on [docs.rs](https://docs.rs/latkit). [Contributing](CONTRIBUTING.md) covers the repository and -how to build it. +Guides, examples, and the reference are on [Read the Docs](https://latkit.readthedocs.io), and the +Rust API is on [docs.rs](https://docs.rs/latkit). [Contributing](CONTRIBUTING.md) covers the +repository and how to build it. Created by [Luke Lowery](https://lukelowry.github.io/). [MIT License](LICENSE). diff --git a/RELEASING.md b/RELEASING.md index d160e635..b65b7220 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,30 +1,32 @@ # Releasing -One version is shared by npm `latkit`, Cargo `latkit`, and Cargo `latkit-cli` (binary `latkit`). -Changesets describes changes and opens the version PR; `cargo xtask version` updates Cargo's -workspace and dependency versions structurally. +npm `latkit`, Cargo `latkit`, and Cargo `latkit-cli` (binary `latkit`) share one version. +Changesets records changes and opens the version PR. `cargo xtask version` copies the npm version +into the Cargo workspace (its package version and its `latkit` dependency) and updates `Cargo.lock`. 1. Merge the version PR and let CI pass on that commit. -2. The release workflow builds `cargo xtask package` and runs `cargo xtask package-test` against - the exact npm tarball, including strict TypeScript consumers and browser rendering. -3. `cargo xtask release` verifies the clean commit, tarball hash, and test receipt, and packages - both Cargo crates. It publishes or verifies npm, the core crate, and the CLI in order. -4. Tags are created only after all registry artifacts match the build. A retry skips identical - artifacts and completes any missing publication. Different content at an existing version - stops the release; registry errors are never treated as absence. Once a version is tagged, +2. The release workflow builds the npm tarball with `cargo xtask package` and tests that exact + tarball with `cargo xtask package-test`, including strict TypeScript consumers and browser + rendering. +3. `cargo xtask release` verifies the clean commit, the tarball hash, and the test receipt, and + packages both Cargo crates. It then publishes npm, the core crate, and the CLI, in that order. A + package already published is verified instead. +4. Tags are created only after every registry artifact matches the build. A retry skips identical + artifacts and completes any missing publication. Different content at an existing version stops + the release, and a registry error never counts as a missing artifact. Once a version is tagged, later commits at that version publish nothing. -For recovery, rerun the failed workflow, or dispatch **Release** on `main`; manual recovery -requires a successful CI run for the exact commit. Do not bump versions just to recover a partial -release. Rebuild and retest after changing or committing source files. +To recover, rerun the failed workflow or dispatch **Release** on `main`. A manual dispatch needs a +successful CI run for the exact commit. Do not bump versions just to recover a partial release. +Rebuild and retest after changing or committing source files. -The workflow needs an npm trusted publisher for repository `lukelowry/latkit`, workflow -`release.yml`, and a `CARGO_REGISTRY_TOKEN` allowed to publish both crates. These are registry/account -settings, not files in this repository. First-publication ownership and authentication must be -configured before invoking release. See [npm trusted publishers](https://docs.npmjs.com/trusted-publishers/). +The workflow needs an npm trusted publisher for repository `lukelowry/latkit` and workflow +`release.yml`, and a `CARGO_REGISTRY_TOKEN` allowed to publish both crates. These are registry and +account settings, not files in this repository. Set up package ownership and authentication before +the first release. See [npm trusted publishers](https://docs.npmjs.com/trusted-publishers/). -`cargo install latkit-cli` is the normal user command. `--locked` optionally reproduces the -dependency versions tested for that release; CI and release builds use it for reproducibility. +Users install the CLI with `cargo install latkit-cli`. Adding `--locked` reproduces the dependency +versions a release was tested with. CI and release builds always use it. -LATK's format version is separate from package versions. V1 fixtures must remain stable across -compatible releases. A new layout requires a new format version and a migration policy. +latkit's binary format carries no version. A release that changes its layout says in its changeset +that files written before it no longer open, and updates `crates/latkit/tests/fixtures/patch.hex`. diff --git a/benchmark/README.md b/benchmark/README.md index 30e9801f..9db42cfc 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,12 +1,13 @@ # Benchmark -Each scenario's phases, up its sizes, in Chrome: time, GPU time, and the work the GPU did, held to -committed baselines. The page steps every frame itself, each waiting for the GPU rather than the -clock, so a run draws the same frames on any device; how frames pace a page is the gate's to judge. +The benchmark runs each scenario in Chrome at increasing sizes. It measures each phase's time, GPU +time, and GPU work, and checks them against committed baselines. The page steps every frame itself: +each frame starts when the GPU has finished the last one, not on the display's clock, so a run draws +the same frames on any device. The browser gate (`npm/examples/gate/gate.ts`) checks frame pacing. ```sh cargo xtask bench --quick # each scenario once at its smallest size: phases and work -cargo xtask bench # every size while the last fits, 3 repeats +cargo xtask bench # every size while the last fits at its fastest, 3 repeats cargo xtask bench --only grid,labels # some scenarios cargo xtask bench --software # SwiftShader on the CPU: work is judged, times only reported cargo xtask bench --update # lay this run's results over the baselines @@ -14,16 +15,21 @@ cargo xtask bench --update # lay this run's results over the baselines A run fails on: -- an error, a frame drawn while nothing changed, a pipeline compiled where it was first used - (`late`), or resident data over its budget; -- a phase growing with size faster than its order; -- more work than `baselines/work.json`: exactly for what the data sets (`uploaded`, `evaluated`, - `jobs`, `pipelines`, `late`, `resident`) in a phase that acts once, and within 2× for what moves - with the frame a job or a fetch lands in (`written`, `readbacks`, `texts`, every count of a phase - acting each frame, and all of a scenario fed by fetches); -- in a full sweep, on the machine `baselines/.json` was taken on, a phase over 1.25× slower, - and for a phase acting once, later by over a timer tick (16 ms). +- An error, a frame drawn when nothing changed, a pipeline compiled when first used (`late`), or + resident data over its budget. +- A phase whose time or uploaded bytes grow with size faster than its declared order. For loading, + resident bytes count too. Each upload (`uploads`) counts as at least one page of bytes. +- More work than `baselines/work.json`. In a phase that acts once, counts set by the data alone + (`uploaded`, `uploads`, `evaluated`, `packed`, `jobs`, `pipelines`, `late`, `resident`) may not + exceed the baseline. All other counts may reach 2× the baseline, because they depend on which + frame a job or fetch finishes in. These are `written`, `resolved`, `cleared`, `readbacks`, and + `texts`, every count in a phase that acts each frame, and every count in a scenario fed by + fetches. +- In a full sweep on the machine that recorded `baselines/.json`, a phase whose fastest + repeat takes more than 1.25× its baseline, plus one timer tick (16 ms) for a phase that acts + once. Only the fastest repeat counts, so a machine stall that slows one repeat does not fail the + run. A real regression slows every repeat. -On a machine with two GPUs, Chrome draws on the fast one. Reports and results land in -`target/bench/`. Scenarios are in `page/scenarios.ts`, the data they draw in `page/data.ts`, and how -a phase is measured in `page/stage.ts`. +On a machine with two GPUs, the runner makes Chrome draw on the faster one. Reports and results go +to `target/bench/`. Scenarios are in `page/scenarios.ts`, the data they draw in `page/data.ts`, and +how a phase is measured in `page/stage.ts`. diff --git a/benchmark/baselines/nvidia-turing-nvidia-geforce-rtx-2080-super-chrome-156.json b/benchmark/baselines/nvidia-turing-nvidia-geforce-rtx-2080-super-chrome-156.json index e1c98524..c319d3bc 100644 --- a/benchmark/baselines/nvidia-turing-nvidia-geforce-rtx-2080-super-chrome-156.json +++ b/benchmark/baselines/nvidia-turing-nvidia-geforce-rtx-2080-super-chrome-156.json @@ -4,1440 +4,1520 @@ "start": { "10000": { "init": { - "time": 280, + "time": 232.20000004768372, "gpu": 0 }, "data": { - "time": 20.799999952316284, + "time": 17.600000023841858, "gpu": 0 }, "load": { - "time": 353.30000001192093, - "gpu": 0.284448 + "time": 604.2000000476837, + "gpu": 0 } } }, "points": { "1000": { "init": { - "time": 142.4000000357628, + "time": 149.5, "gpu": 0 }, "data": { - "time": 14.100000023841858, + "time": 13.799999952316284, "gpu": 0 }, "load": { - "time": 350, - "gpu": 0.119456 + "time": 531.1999999284744, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.073792 + "gpu": 0.07088 }, "zoom": { "time": 0.099999994, - "gpu": 0.067584 + "gpu": 0.063488 + }, + "revisit": { + "time": 3.600000023841858, + "gpu": 0.0656 }, "restyle": { - "time": 5.5, - "gpu": 0.070016 + "time": 4.599999904632568, + "gpu": 0.065952 }, "one row": { "time": 0.099999994, - "gpu": 0.085568 + "gpu": 0.081824 }, "every row": { - "time": 3.900000035762787, - "gpu": 0.085664 + "time": 2.1999999284744263, + "gpu": 0.0816 }, "first pick": { - "time": 9, - "gpu": 0.085824 + "time": 8.5, + "gpu": 0.0816 }, "pick": { - "time": 8.300000011920929, - "gpu": 0.085728 + "time": 8, + "gpu": 0.08176 }, "hover": { - "time": 13.699999988079071, - "gpu": 0.085856 + "time": 13.5, + "gpu": 0.08176 }, "select": { - "time": 7.400000035762787, - "gpu": 0.085824 + "time": 5.799999952316284, + "gpu": 0.080896 + }, + "append": { + "time": 0.19999999, + "gpu": 0.09072 } }, "10000": { "init": { - "time": 177.70000004768372, + "time": 124.89999997615814, "gpu": 0 }, "data": { - "time": 17, + "time": 14.100000023841858, "gpu": 0 }, "load": { - "time": 354.0999999642372, - "gpu": 0.280032 + "time": 480.5, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.145408 + "gpu": 0.141728 }, "zoom": { "time": 0.099999994, - "gpu": 0.085152 + "gpu": 0.08192 + }, + "revisit": { + "time": 3.299999952316284, + "gpu": 0.102624 }, "restyle": { - "time": 9.400000035762787, - "gpu": 0.106496 + "time": 5.099999904632568, + "gpu": 0.110592 }, "one row": { - "time": 0.19999999, - "gpu": 0.168576 + "time": 0.099999994, + "gpu": 0.16368 }, "every row": { - "time": 4.800000011920929, - "gpu": 0.168448 + "time": 5, + "gpu": 0.163648 }, "first pick": { - "time": 13.199999988079071, - "gpu": 0.168448 + "time": 9.399999976158142, + "gpu": 0.163808 }, "pick": { - "time": 8.900000035762787, - "gpu": 0.167968 + "time": 6, + "gpu": 0.16384 }, "hover": { - "time": 15.599999964237213, - "gpu": 0.167904 + "time": 10.799999952316284, + "gpu": 0.16384 }, "select": { - "time": 10, - "gpu": 0.167904 + "time": 8.5, + "gpu": 0.16384 + }, + "append": { + "time": 0.19999999, + "gpu": 0.172032 } }, "100000": { "init": { - "time": 149.5, + "time": 107.89999997615814, "gpu": 0 }, "data": { - "time": 15.900000035762787, + "time": 16.199999928474426, "gpu": 0 }, "load": { - "time": 363.5, - "gpu": 2.904192 + "time": 459, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.859616 + "gpu": 0.868352 }, "zoom": { - "time": 0.099999994, - "gpu": 0.274432 + "time": 0, + "gpu": 0.2744 + }, + "revisit": { + "time": 3.600000023841858, + "gpu": 0.48128 }, "restyle": { - "time": 6.5, - "gpu": 0.479232 + "time": 4.400000095367432, + "gpu": 0.548288 }, "one row": { - "time": 0.19999999, - "gpu": 0.988192 + "time": 0.099999994, + "gpu": 0.973568 }, "every row": { - "time": 8.400000035762787, - "gpu": 0.98784 + "time": 6.899999976158142, + "gpu": 0.973856 }, "first pick": { - "time": 10, - "gpu": 0.98784 + "time": 10.300000071525574, + "gpu": 0.973856 }, "pick": { - "time": 8.800000011920929, - "gpu": 0.987296 + "time": 5.5, + "gpu": 0.973856 }, "hover": { - "time": 13.599999964237213, - "gpu": 0.987136 + "time": 13.100000023841858, + "gpu": 0.974304 }, "select": { - "time": 11.200000047683716, - "gpu": 0.987136 + "time": 7.399999976158142, + "gpu": 0.973664 + }, + "append": { + "time": 0.3, + "gpu": 0.984064 } }, "1000000": { "init": { - "time": 168.5, + "time": 137.60000002384186, "gpu": 0 }, "data": { - "time": 30, + "time": 40.10000002384186, "gpu": 0 }, "load": { - "time": 664, - "gpu": 1.245824 + "time": 614.2000000476837, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.183392 + "gpu": 0.17888 }, "zoom": { "time": 0.19999999, - "gpu": 0.497664 + "gpu": 0.479328 + }, + "revisit": { + "time": 4.100000023841858, + "gpu": 0.387008 }, "restyle": { - "time": 5.800000011920929, - "gpu": 0.415648 + "time": 5.199999928474426, + "gpu": 0.370976 }, "one row": { - "time": 0.19999999, - "gpu": 0.194976 + "time": 0.099999994, + "gpu": 0.192512 }, "every row": { - "time": 28.399999976158142, - "gpu": 0.195392 + "time": 28.600000023841858, + "gpu": 0.192512 }, "first pick": { - "time": 10.799999952316284, - "gpu": 0.195392 + "time": 8.199999928474426, + "gpu": 0.192512 }, "pick": { - "time": 7.800000011920929, - "gpu": 0.195776 + "time": 6.199999928474426, + "gpu": 0.192832 }, "hover": { - "time": 15.399999976158142, - "gpu": 0.196448 + "time": 11.899999976158142, + "gpu": 0.192832 }, "select": { - "time": 11.300000011920929, - "gpu": 0.196608 + "time": 9.600000023841858, + "gpu": 0.193376 + }, + "append": { + "time": 0.5, + "gpu": 0.21984 } }, "2000000": { "init": { - "time": 166.30000001192093, + "time": 122.60000002384186, "gpu": 0 }, "data": { - "time": 87.20000004768372, + "time": 49.89999997615814, "gpu": 0 }, "load": { - "time": 1000.8999999761581, - "gpu": 1.283104 + "time": 839.0999999046326, + "gpu": 1.265664 }, "pan": { "time": 0.099999994, - "gpu": 0.183488 + "gpu": 0.180224 }, "zoom": { "time": 0.19999999, - "gpu": 0.503168 + "gpu": 0.480352 + }, + "revisit": { + "time": 3.600000023841858, + "gpu": 0.391968 }, "restyle": { - "time": 7.5, - "gpu": 0.414432 + "time": 4.899999976158142, + "gpu": 0.370688 }, "one row": { "time": 0.099999994, - "gpu": 0.19504 + "gpu": 0.192512 }, "every row": { - "time": 50.30000001192093, - "gpu": 0.194848 + "time": 49.09999990463257, + "gpu": 0.192512 }, "first pick": { - "time": 12.900000035762787, - "gpu": 0.194944 + "time": 9.5, + "gpu": 0.192512 }, "pick": { - "time": 9.300000011920929, - "gpu": 0.195232 + "time": 6.700000047683716, + "gpu": 0.192512 }, "hover": { - "time": 14.5, - "gpu": 0.196 + "time": 13, + "gpu": 0.192512 }, "select": { - "time": 11.199999988079071, - "gpu": 0.196064 + "time": 8.299999952316284, + "gpu": 0.193248 + }, + "append": { + "time": 0.39999998, + "gpu": 0.219584 } } }, "grid": { "1000": { "init": { - "time": 142, + "time": 115.39999997615814, "gpu": 0 }, "data": { - "time": 14.699999988079071, + "time": 13.700000047683716, "gpu": 0 }, "load": { - "time": 526.4000000357628, - "gpu": 0.194464 + "time": 511.60000002384186, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.121408 + "gpu": 0.124928 }, "zoom": { "time": 0.099999994, - "gpu": 0.12304 + "gpu": 0.126944 + }, + "revisit": { + "time": 2.899999976158142, + "gpu": 0.126432 }, "one row": { "time": 0.099999994, - "gpu": 0.131072 + "gpu": 0.134112 }, "first pick": { - "time": 8.399999976158142, - "gpu": 0.131232 + "time": 7.900000095367432, + "gpu": 0.134048 }, "pick": { - "time": 7.100000023841858, - "gpu": 0.131264 + "time": 5.600000023841858, + "gpu": 0.133632 }, "hover": { - "time": 12, - "gpu": 0.131264 + "time": 10.899999976158142, + "gpu": 0.133632 + }, + "append": { + "time": 0.099999994, + "gpu": 0.165184 } }, "10000": { "init": { - "time": 163.69999998807907, + "time": 120, "gpu": 0 }, "data": { - "time": 16, + "time": 14, "gpu": 0 }, "load": { - "time": 586.6999999880791, - "gpu": 0.256 + "time": 537.5, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.17808 + "gpu": 0.182272 }, "zoom": { "time": 0.099999994, - "gpu": 0.173632 + "gpu": 0.178336 + }, + "revisit": { + "time": 3.200000047683716, + "gpu": 0.180224 }, "one row": { - "time": 0.19999999, - "gpu": 0.187584 + "time": 0.099999994, + "gpu": 0.190848 }, "first pick": { - "time": 10.5, - "gpu": 0.187712 + "time": 6, + "gpu": 0.190912 }, "pick": { - "time": 9.800000011920929, - "gpu": 0.187712 + "time": 6.5, + "gpu": 0.19088 }, "hover": { - "time": 12.700000047683716, - "gpu": 0.187584 + "time": 10.199999928474426, + "gpu": 0.190912 + }, + "append": { + "time": 0.19999999, + "gpu": 0.222464 } }, "100000": { "init": { - "time": 158.10000002384186, + "time": 116.79999995231628, "gpu": 0 }, "data": { - "time": 20.19999998807907, + "time": 19.100000023841858, "gpu": 0 }, "load": { - "time": 541.6000000238419, - "gpu": 0.94208 + "time": 530.5, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.6024 + "gpu": 0.609376 }, "zoom": { "time": 0.099999994, - "gpu": 0.524288 + "gpu": 0.53248 + }, + "revisit": { + "time": 3, + "gpu": 0.565856 }, "one row": { - "time": 0.19999999, - "gpu": 0.636288 + "time": 0.099999994, + "gpu": 0.621824 }, "first pick": { - "time": 14.300000011920929, - "gpu": 0.636224 + "time": 11.100000023841858, + "gpu": 0.62144 }, "pick": { - "time": 6.699999988079071, - "gpu": 0.636896 + "time": 5.299999952316284, + "gpu": 0.622176 }, "hover": { - "time": 17.69999998807907, - "gpu": 0.638496 + "time": 12.199999928474426, + "gpu": 0.621408 + }, + "append": { + "time": 0.3, + "gpu": 0.733184 } }, "1000000": { "init": { - "time": 161.30000001192093, + "time": 120.29999995231628, "gpu": 0 }, "data": { - "time": 63.799999952316284, + "time": 62.10000002384186, "gpu": 0 }, "load": { - "time": 1243.300000011921, - "gpu": 0.734304 + "time": 978.2000000476837, + "gpu": 0.36048 }, "pan": { - "time": 0.19999999, - "gpu": 0.581312 + "time": 0.099999994, + "gpu": 0.612352 }, "zoom": { - "time": 0.19999999, - "gpu": 1.359872 + "time": 0.3, + "gpu": 1.366016 + }, + "revisit": { + "time": 6, + "gpu": 1.552384 }, "one row": { "time": 0.3, - "gpu": 0.597408 + "gpu": 0.644448 }, "first pick": { - "time": 14, - "gpu": 0.612096 + "time": 8.899999976158142, + "gpu": 0.643648 }, "pick": { - "time": 7.099999964237213, - "gpu": 0.59712 + "time": 6.100000023841858, + "gpu": 0.643648 }, "hover": { - "time": 17.099999964237213, - "gpu": 0.614208 + "time": 11.100000023841858, + "gpu": 0.644672 + }, + "append": { + "time": 0.6, + "gpu": 0.709824 } }, "2000000": { "init": { - "time": 165, + "time": 126.5, "gpu": 0 }, "data": { - "time": 108.89999997615814, + "time": 118.29999995231628, "gpu": 0 }, "load": { - "time": 1460.3999999761581, - "gpu": 0.755744 + "time": 1597.8000000715256, + "gpu": 0.364544 }, "pan": { - "time": 0.099999994, - "gpu": 0.590656 + "time": 0.19999999, + "gpu": 0.627328 }, "zoom": { - "time": 0.19999999, - "gpu": 1.616288 + "time": 0.3, + "gpu": 1.323008 + }, + "revisit": { + "time": 5.799999952316284, + "gpu": 1.104 }, "one row": { - "time": 0.19999999, - "gpu": 0.599648 + "time": 0.3, + "gpu": 0.657408 }, "first pick": { - "time": 16.799999952316284, - "gpu": 0.604096 + "time": 10.399999976158142, + "gpu": 0.654816 }, "pick": { - "time": 6.699999988079071, - "gpu": 0.604096 + "time": 5.899999976158142, + "gpu": 0.654816 }, "hover": { - "time": 16.100000023841858, - "gpu": 0.610304 + "time": 11.799999952316284, + "gpu": 0.653632 + }, + "append": { + "time": 0.6, + "gpu": 0.72144 } } }, "labels": { "10000": { "init": { - "time": 176.0999999642372, + "time": 119.30000007152557, "gpu": 0 }, "data": { - "time": 22, + "time": 17.899999976158142, "gpu": 0 }, "load": { - "time": 568.1999999880791, - "gpu": 0.321536 + "time": 505.7999999523163, + "gpu": 0.181952 }, "pan": { - "time": 0.7, - "gpu": 0.32224 + "time": 0.5, + "gpu": 0.31712 }, "zoom": { - "time": 1.2, - "gpu": 0.286304 + "time": 0.9, + "gpu": 0.289728 } }, "100000": { "init": { - "time": 177.60000002384186, + "time": 131, "gpu": 0 }, "data": { - "time": 67.09999996423721, + "time": 65.09999990463257, "gpu": 0 }, "load": { - "time": 621.0999999642372, - "gpu": 1.501248 + "time": 546.5, + "gpu": 1.322784 }, "pan": { - "time": 0.79999995, - "gpu": 1.465056 + "time": 0.6, + "gpu": 1.36688 }, "zoom": { - "time": 1.5, - "gpu": 0.689888 + "time": 1.7, + "gpu": 0.644512 } }, "1000000": { "init": { - "time": 164.30000001192093, + "time": 118.79999995231628, "gpu": 0 }, "data": { - "time": 510.19999998807907, + "time": 489.59999990463257, "gpu": 0 }, "load": { - "time": 780.6000000238419, - "gpu": 0.681568 + "time": 687.5, + "gpu": 0.667648 }, "pan": { - "time": 1, - "gpu": 0.429344 + "time": 0.79999995, + "gpu": 0.415008 }, "zoom": { - "time": 1.7, - "gpu": 1.60848 + "time": 1.5, + "gpu": 0.886464 } } }, "lanes": { "100": { "init": { - "time": 161.10000002384186, + "time": 117.19999992847443, "gpu": 0 }, "data": { - "time": 14.600000023841858, + "time": 13.799999952316284, "gpu": 0 }, "load": { - "time": 535.3999999761581, - "gpu": 0.144448 + "time": 515.6000000238419, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.080704 + "gpu": 0.105344 }, "first pick": { - "time": 10.400000035762787, - "gpu": 0.080768 + "time": 8.899999976158142, + "gpu": 0.105408 }, "pick": { - "time": 5.699999988079071, - "gpu": 0.080928 + "time": 5.699999928474426, + "gpu": 0.105408 } }, "1000": { "init": { - "time": 157, + "time": 123.10000002384186, "gpu": 0 }, "data": { - "time": 13.799999952316284, + "time": 14.5, "gpu": 0 }, "load": { - "time": 532.8000000119209, - "gpu": 0.18368 + "time": 541.2000000476837, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.16784 + "gpu": 0.152032 }, "first pick": { - "time": 10.699999988079071, - "gpu": 0.166304 + "time": 9.899999976158142, + "gpu": 0.15344 }, "pick": { - "time": 6.5, - "gpu": 0.167648 + "time": 6, + "gpu": 0.15344 } }, "10000": { "init": { - "time": 150.4000000357628, + "time": 114.70000004768372, "gpu": 0 }, "data": { - "time": 14.799999952316284, + "time": 14.100000023841858, "gpu": 0 }, "load": { - "time": 534.7999999523163, - "gpu": 0.388384 + "time": 524.1999999284744, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 2.306144 + "gpu": 0.143648 }, "first pick": { - "time": 23.5, - "gpu": 2.270912 + "time": 7.700000047683716, + "gpu": 0.139264 }, "pick": { - "time": 15.5, - "gpu": 2.29008 + "time": 5.099999904632568, + "gpu": 0.137216 } }, "100000": { "init": { - "time": 190.30000001192093, + "time": 119.19999992847443, "gpu": 0 }, "data": { - "time": 20.19999998807907, + "time": 16.09999990463257, "gpu": 0 }, "load": { - "time": 816.3000000119209, - "gpu": 18.844288 + "time": 529.6000000238419, + "gpu": 0 }, "pan": { - "time": 0.19999999, - "gpu": 19.761152 + "time": 0.099999994, + "gpu": 0.176512 }, "first pick": { - "time": 76.10000002384186, - "gpu": 19.828447 + "time": 8.199999928474426, + "gpu": 0.175104 }, "pick": { - "time": 70.80000001192093, - "gpu": 19.762688 + "time": 6.199999928474426, + "gpu": 0.172768 } } }, "areas": { "100": { "init": { - "time": 187.4000000357628, + "time": 117.10000002384186, "gpu": 0 }, "data": { - "time": 16.100000023841858, + "time": 14.299999952316284, "gpu": 0 }, "load": { - "time": 493.9000000357628, - "gpu": 0.32768 + "time": 485.89999997615814, + "gpu": 0 }, "pan": { - "time": 0.19999999, - "gpu": 0.141408 + "time": 0.099999994, + "gpu": 0.137216 }, "zoom": { "time": 0.099999994, - "gpu": 0.139264 + "gpu": 0.134816 }, "one row": { - "time": 0.19999999, - "gpu": 0.296896 + "time": 0.099999994, + "gpu": 0.291328 }, "first pick": { - "time": 8.5, - "gpu": 0.296736 + "time": 7.299999952316284, + "gpu": 0.2912 }, "pick": { - "time": 8.800000011920929, - "gpu": 0.296736 + "time": 5.899999976158142, + "gpu": 0.290944 } }, "1000": { "init": { - "time": 148.0999999642372, + "time": 119.30000007152557, "gpu": 0 }, "data": { - "time": 18.19999998807907, + "time": 16.700000047683716, "gpu": 0 }, "load": { - "time": 396.0999999642372, - "gpu": 0.13952 + "time": 481.10000002384186, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.219424 + "gpu": 0.215104 }, "zoom": { "time": 0.099999994, - "gpu": 0.217728 + "gpu": 0.213824 }, "one row": { - "time": 0.19999999, - "gpu": 0.359136 + "time": 0.099999994, + "gpu": 0.353888 }, "first pick": { - "time": 8.300000011920929, - "gpu": 0.358944 + "time": 7.900000095367432, + "gpu": 0.353728 }, "pick": { - "time": 7.300000011920929, - "gpu": 0.35872 + "time": 6.100000023841858, + "gpu": 0.3536 } }, "10000": { "init": { - "time": 159.79999995231628, + "time": 121.89999997615814, "gpu": 0 }, "data": { - "time": 29.899999976158142, + "time": 27.5, "gpu": 0 }, "load": { - "time": 474, - "gpu": 0.1344 + "time": 512.8999999761581, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.604192 + "gpu": 0.600064 }, "zoom": { "time": 0.099999994, - "gpu": 0.602112 + "gpu": 0.598016 }, "one row": { "time": 0.099999994, - "gpu": 0.728768 + "gpu": 0.722912 }, "first pick": { - "time": 7.399999976158142, - "gpu": 0.72816 + "time": 6.899999976158142, + "gpu": 0.722816 }, "pick": { - "time": 6.899999976158142, - "gpu": 0.728128 + "time": 5.800000071525574, + "gpu": 0.722752 } }, "30000": { "init": { - "time": 150.19999998807907, + "time": 115.39999997615814, "gpu": 0 }, "data": { - "time": 59, + "time": 53.89999997615814, "gpu": 0 }, "load": { - "time": 779.6000000238419, - "gpu": 0.096448 + "time": 667.7999999523163, + "gpu": 0.087712 }, "pan": { "time": 0.099999994, - "gpu": 1.21536 + "gpu": 1.139136 }, "zoom": { "time": 0.099999994, - "gpu": 1.213888 + "gpu": 1.138016 }, "one row": { - "time": 0.099999994, - "gpu": 1.741728 + "time": 0.19999999, + "gpu": 1.651264 }, "first pick": { - "time": 7.100000023841858, - "gpu": 1.61152 + "time": 8.5, + "gpu": 1.651392 }, "pick": { - "time": 6.200000047683716, - "gpu": 1.6112 + "time": 6.5, + "gpu": 1.648 } } }, "tiles": { "16": { "init": { - "time": 143.39999997615814, + "time": 116, "gpu": 0 }, "data": { - "time": 11.100000023841858, + "time": 11, "gpu": 0 }, "load": { - "time": 459.5, - "gpu": 0.121216 + "time": 472.5, + "gpu": 0 }, "pan": { - "time": 0.19999999, - "gpu": 0.1232 + "time": 0.099999994, + "gpu": 0.118784 }, "zoom": { "time": 0.19999999, - "gpu": 0.118432 + "gpu": 0.114496 } }, "256": { "init": { - "time": 144.5, + "time": 122.60000002384186, "gpu": 0 }, "data": { - "time": 11.800000011920929, + "time": 11, "gpu": 0 }, "load": { - "time": 441.4000000357628, - "gpu": 0.1216 + "time": 479.5, + "gpu": 0 }, "pan": { - "time": 0.099999994, - "gpu": 0.13312 + "time": 0.19999999, + "gpu": 0.128544 }, "zoom": { - "time": 0.19999999, - "gpu": 0.12288 + "time": 0.099999994, + "gpu": 0.12064 } }, "4096": { "init": { - "time": 150.5999999642372, + "time": 120.79999995231628, "gpu": 0 }, "data": { - "time": 11.100000023841858, + "time": 11.800000071525574, "gpu": 0 }, "load": { - "time": 512.3000000119209, - "gpu": 0.122688 + "time": 504, + "gpu": 0 }, "pan": { - "time": 0.19999999, - "gpu": 0.171744 + "time": 0.099999994, + "gpu": 0.143072 }, "zoom": { - "time": 2.8, - "gpu": 0.155648 + "time": 0.099999994, + "gpu": 0.130624 } } }, "diagram": { "100": { "init": { - "time": 150.30000001192093, + "time": 118.89999997615814, "gpu": 0 }, "data": { - "time": 15.899999976158142, + "time": 14.700000047683716, "gpu": 0 }, "load": { - "time": 498.9000000357628, - "gpu": 1.456416 + "time": 498.39999997615814, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.083968 + "gpu": 0.079872 }, "edit": { - "time": 0.39999998, - "gpu": 0.122656 + "time": 0.3, + "gpu": 0.118784 }, "arrange": { - "time": 12, - "gpu": 0.12288 + "time": 11, + "gpu": 0.119744 } }, "1000": { "init": { - "time": 155.39999997615814, + "time": 119.10000002384186, "gpu": 0 }, "data": { - "time": 15.5, + "time": 14.599999904632568, "gpu": 0 }, "load": { - "time": 503.4000000357628, - "gpu": 1.454784 + "time": 514.1000000238419, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.096448 + "gpu": 0.09088 }, "edit": { - "time": 0.6, - "gpu": 0.142688 + "time": 0.39999998, + "gpu": 0.134272 }, "arrange": { - "time": 16.19999998807907, - "gpu": 0.143424 + "time": 15.199999928474426, + "gpu": 0.13488 } }, "10000": { "init": { - "time": 147.10000002384186, + "time": 126.19999992847443, "gpu": 0 }, "data": { - "time": 20.80000001192093, + "time": 22, "gpu": 0 }, "load": { - "time": 581.3999999761581, - "gpu": 1.478176 + "time": 600.4000000953674, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 0.216224 + "gpu": 0.188832 }, "edit": { - "time": 3.4, - "gpu": 0.297088 + "time": 0.7, + "gpu": 0.263232 }, "arrange": { - "time": 58.299999952316284, - "gpu": 0.297824 + "time": 53.60000002384186, + "gpu": 0.263008 } } }, "traces": { "10": { "init": { - "time": 148.5, + "time": 124.10000002384186, "gpu": 0 }, "data": { - "time": 17.30000001192093, + "time": 16.100000023841858, "gpu": 0 }, "load": { - "time": 432.2999999523163, - "gpu": 1.408352 + "time": 572.2000000476837, + "gpu": 0 }, "stream": { "time": 0.099999994, - "gpu": 0.097632 + "gpu": 0.095904 }, "scrub": { "time": 0.099999994, - "gpu": 0.077824 + "gpu": 0.075776 }, "narrow": { "time": 0.099999994, - "gpu": 0.090528 + "gpu": 0.087712 }, "hover": { - "time": 11.699999988079071, - "gpu": 0.08 + "time": 10.399999976158142, + "gpu": 0.077888 } }, "100": { "init": { - "time": 147.30000001192093, + "time": 122.39999997615814, "gpu": 0 }, "data": { - "time": 18.899999976158142, + "time": 19.399999976158142, "gpu": 0 }, "load": { - "time": 462.89999997615814, - "gpu": 1.466368 + "time": 564, + "gpu": 0 }, "stream": { "time": 0.099999994, - "gpu": 0.337184 + "gpu": 0.333248 }, "scrub": { "time": 0.099999994, - "gpu": 0.279744 + "gpu": 0.276416 }, "narrow": { "time": 0.099999994, - "gpu": 0.24784 + "gpu": 0.24512 }, "hover": { - "time": 12.5, - "gpu": 0.136704 + "time": 9.300000071525574, + "gpu": 0.131936 } }, "1000": { "init": { - "time": 177.39999997615814, + "time": 126.10000002384186, "gpu": 0 }, "data": { - "time": 37.69999998807907, + "time": 36.799999952316284, "gpu": 0 }, "load": { - "time": 460, - "gpu": 1.582336 + "time": 665, + "gpu": 0 }, "stream": { - "time": 0.099999994, - "gpu": 2.24256 + "time": 0.19999999, + "gpu": 2.235552 }, "scrub": { "time": 0.099999994, - "gpu": 1.92512 + "gpu": 1.912416 }, "narrow": { "time": 0.099999994, - "gpu": 1.506816 + "gpu": 1.492992 }, "hover": { - "time": 12.799999952316284, - "gpu": 0.538624 + "time": 13.200000047683716, + "gpu": 0.539168 } }, "10000": { "init": { - "time": 187.70000004768372, + "time": 121.20000004768372, "gpu": 0 }, "data": { - "time": 233.0999999642372, + "time": 221.19999992847443, "gpu": 0 }, "load": { - "time": 547.7999999523163, - "gpu": 3.033696 + "time": 589.8999999761581, + "gpu": 0 }, "stream": { "time": 0.3, - "gpu": 23.631264 + "gpu": 23.267328 }, "scrub": { "time": 0.19999999, - "gpu": 20.68048 + "gpu": 20.334593 }, "narrow": { "time": 0.19999999, - "gpu": 15.118592 + "gpu": 14.946752 }, "hover": { - "time": 19.600000023841858, - "gpu": 5.16896 + "time": 20.90000009536743, + "gpu": 4.993408 } } }, "history": { "1000": { "init": { - "time": 147.5999999642372, + "time": 120.30000007152557, "gpu": 0 }, "data": { - "time": 16.399999976158142, + "time": 16.699999928474426, "gpu": 0 }, "load": { - "time": 437, - "gpu": 1.41152 + "time": 536.2000000476837, + "gpu": 0 }, "scrub": { "time": 0.099999994, - "gpu": 0.058112 + "gpu": 0.057344 }, "narrow": { "time": 0.099999994, - "gpu": 0.074688 + "gpu": 0.074016 } }, "10000": { "init": { - "time": 156.19999998807907, + "time": 133.19999992847443, "gpu": 0 }, "data": { - "time": 22.899999976158142, + "time": 24.700000047683716, "gpu": 0 }, "load": { - "time": 435.7000000476837, - "gpu": 1.40288 + "time": 623, + "gpu": 0 }, "scrub": { "time": 0.099999994, - "gpu": 0.086016 + "gpu": 0.084864 }, "narrow": { "time": 0.19999999, - "gpu": 0.092768 + "gpu": 0.09312 } }, "100000": { "init": { - "time": 161.80000001192093, + "time": 130.79999995231628, "gpu": 0 }, "data": { - "time": 60.80000001192093, + "time": 61.10000002384186, "gpu": 0 }, "load": { - "time": 442.89999997615814, - "gpu": 1.410528 + "time": 565.6999999284744, + "gpu": 0 }, "scrub": { "time": 0.099999994, - "gpu": 0.286688 + "gpu": 0.282304 }, "narrow": { - "time": 0.19999999, - "gpu": 0.215744 + "time": 0.099999994, + "gpu": 0.213696 } }, "1000000": { "init": { - "time": 159.89999997615814, + "time": 133.79999995231628, "gpu": 0 }, "data": { - "time": 394.4000000357628, + "time": 413.2999999523163, "gpu": 0 }, "load": { - "time": 440.89999997615814, - "gpu": 1.403584 + "time": 598.5, + "gpu": 0 }, "scrub": { "time": 0.099999994, - "gpu": 0.405504 + "gpu": 0.39936 }, "narrow": { - "time": 0.19999999, - "gpu": 0.288896 + "time": 0.099999994, + "gpu": 0.28528 } } }, "transition": { "1000": { "init": { - "time": 149.29999995231628, + "time": 151.40000009536743, "gpu": 0 }, "data": { - "time": 14.800000011920929, + "time": 12.899999976158142, "gpu": 0 }, "load": { - "time": 313.89999997615814, - "gpu": 0.124672 + "time": 481.60000002384186, + "gpu": 0 }, "move": { "time": 0.099999994, - "gpu": 0.083392 + "gpu": 0.081056 } }, "10000": { "init": { - "time": 147.19999998807907, + "time": 122.80000007152557, "gpu": 0 }, "data": { - "time": 14.599999964237213, + "time": 13.299999952316284, "gpu": 0 }, "load": { - "time": 315, - "gpu": 0.279616 + "time": 432.10000002384186, + "gpu": 0 }, "move": { "time": 0.099999994, - "gpu": 0.172128 + "gpu": 0.169568 } }, "100000": { "init": { - "time": 161.29999995231628, + "time": 118.5, "gpu": 0 }, "data": { - "time": 16.69999998807907, + "time": 15.399999976158142, "gpu": 0 }, "load": { - "time": 334.89999997615814, - "gpu": 1.939008 + "time": 445.2000000476837, + "gpu": 0 }, "move": { "time": 0.099999994, - "gpu": 1.233024 + "gpu": 1.259776 } }, "1000000": { "init": { - "time": 148.4000000357628, + "time": 119.10000002384186, "gpu": 0 }, "data": { - "time": 31.19999998807907, + "time": 30.799999952316284, "gpu": 0 }, "load": { - "time": 816.6999999880791, - "gpu": 0.776192 + "time": 688.2000000476837, + "gpu": 0.789696 }, "move": { "time": 0.099999994, - "gpu": 0.223104 + "gpu": 0.223264 } } }, "views": { "1": { "init": { - "time": 159.5, + "time": 117.10000002384186, "gpu": 0 }, "data": { - "time": 23.600000023841858, + "time": 18.699999928474426, "gpu": 0 }, "load": { - "time": 544.7999999523163, - "gpu": 1.428128 + "time": 496.10000002384186, + "gpu": 0 }, "pan": { - "time": 0.099999994, - "gpu": 0.927744 + "time": 0, + "gpu": 0.925728 } }, "2": { "init": { - "time": 155.70000004768372, + "time": 113.79999995231628, "gpu": 0 }, "data": { - "time": 21.80000001192093, + "time": 19.199999928474426, "gpu": 0 }, "load": { - "time": 559.7999999523163, - "gpu": 2.794816 + "time": 528.5, + "gpu": 0 }, "pan": { "time": 0.099999994, - "gpu": 1.4248319999999999 + "gpu": 1.43424 } }, "4": { "init": { - "time": 157.5, + "time": 115.19999992847443, "gpu": 0 }, "data": { - "time": 19.899999976158142, + "time": 19.600000023841858, "gpu": 0 }, "load": { - "time": 562.7999999523163, - "gpu": 5.52096 + "time": 535.6999999284744, + "gpu": 0 }, "pan": { - "time": 0.299999982, - "gpu": 2.4928320000000004 + "time": 0.199999988, + "gpu": 2.3380159999999997 } }, "8": { "init": { - "time": 167, + "time": 114.39999997615814, "gpu": 0 }, "data": { - "time": 24.900000035762787, + "time": 19, "gpu": 0 }, "load": { - "time": 679.1000000238419, - "gpu": 7.603104 + "time": 549.3000000715256, + "gpu": 0 }, "pan": { - "time": 0.49999997199999996, - "gpu": 2.757472 + "time": 0.299999984, + "gpu": 2.7792320000000004 } } }, "budget": { "100000": { "init": { - "time": 162.69999998807907, + "time": 116.90000009536743, "gpu": 0 }, "data": { - "time": 22, + "time": 18.800000071525574, "gpu": 0 }, "load": { - "time": 779.6000000238419, - "gpu": 1.01232 + "time": 525.8999999761581, + "gpu": 0 }, "pan": { - "time": 0.19999999, - "gpu": 0.460448 + "time": 0.099999994, + "gpu": 0.233472 }, "zoom": { - "time": 0.19999999, - "gpu": 0.18672 + "time": 0.099999994, + "gpu": 0.128576 } }, "1000000": { "init": { - "time": 178.69999998807907, + "time": 122.89999997615814, "gpu": 0 }, "data": { - "time": 63.69999998807907, + "time": 64, "gpu": 0 }, "load": { - "time": 1287.199999988079, - "gpu": 1.517728 + "time": 941.6999999284744, + "gpu": 0.471168 }, "pan": { - "time": 0.19999999, - "gpu": 0.590176 + "time": 0.099999994, + "gpu": 0.24336 }, "zoom": { - "time": 0.19999999, - "gpu": 0.22128 + "time": 0.099999994, + "gpu": 0.122944 } }, "2000000": { "init": { - "time": 164.29999995231628, + "time": 116.70000004768372, "gpu": 0 }, "data": { - "time": 123.60000002384186, + "time": 111.69999992847443, "gpu": 0 }, "load": { - "time": 1516.3999999761581, - "gpu": 1.424768 + "time": 1492.5999999046326, + "gpu": 0.4744 }, "pan": { - "time": 0.19999999, - "gpu": 0.592096 + "time": 0.099999994, + "gpu": 0.244544 }, "zoom": { "time": 0.099999994, - "gpu": 0.221536 + "gpu": 0.123328 } } }, "shots": { "518400": { "init": { - "time": 137.79999995231628, + "time": 107.29999995231628, "gpu": 0 }, "data": { - "time": 21.30000001192093, + "time": 18.40000009536743, "gpu": 0 }, "load": { - "time": 567.3000000119209, - "gpu": 2.792736 + "time": 514.3999999761581, + "gpu": 0 }, "image": { - "time": 30.69999998807907, - "gpu": 2.314368 + "time": 25.899999976158142, + "gpu": 2.307104 }, "png": { - "time": 10.5, - "gpu": 2.314368 + "time": 10.199999928474426, + "gpu": 2.303264 }, "video": { - "time": 247.80000001192093, - "gpu": 2.094336 + "time": 221.19999992847443, + "gpu": 1.2333759999999998 }, "held": { - "time": 141.10000002384186, - "gpu": 2.127872 + "time": 140.29999995231628, + "gpu": 1.274592 } }, "2073600": { "init": { - "time": 135.39999997615814, + "time": 111.39999997615814, "gpu": 0 }, "data": { - "time": 20.399999976158142, + "time": 19.200000047683716, "gpu": 0 }, "load": { - "time": 545.8000000119209, - "gpu": 3.086304 + "time": 507.10000002384186, + "gpu": 0 }, "image": { - "time": 46.80000001192093, - "gpu": 2.3137600000000003 + "time": 36, + "gpu": 2.301984 }, "png": { - "time": 24.200000047683716, - "gpu": 2.313088 + "time": 23.899999976158142, + "gpu": 2.301984 }, "video": { - "time": 286.60000002384186, - "gpu": 2.1913600000000004 + "time": 278.8000000715256, + "gpu": 1.2743039999999999 }, "held": { - "time": 148.0999999642372, - "gpu": 2.199136 + "time": 135, + "gpu": 1.322976 } }, "8294400": { "init": { - "time": 139.5, + "time": 107.89999997615814, "gpu": 0 }, "data": { - "time": 20.399999976158142, + "time": 20.799999952316284, "gpu": 0 }, "load": { - "time": 543, - "gpu": 2.79392 + "time": 579, + "gpu": 0 }, "image": { - "time": 87, - "gpu": 2.385632 + "time": 72.60000002384186, + "gpu": 2.301088 }, "png": { - "time": 78.60000002384186, - "gpu": 2.385632 + "time": 80.60000002384186, + "gpu": 2.300768 }, "video": { - "time": 646.6999999880791, - "gpu": 2.6850240000000003 + "time": 606, + "gpu": 1.5112 }, "held": { - "time": 191, - "gpu": 2.7155199999999997 + "time": 180.69999992847443, + "gpu": 1.524576 } } } diff --git a/benchmark/baselines/work.json b/benchmark/baselines/work.json index d4a0b5d7..b33b26f5 100644 --- a/benchmark/baselines/work.json +++ b/benchmark/baselines/work.json @@ -3,8 +3,12 @@ "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -14,8 +18,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -25,14 +33,18 @@ }, "load": { "uploaded": 200000, - "written": 4004, + "uploads": 15, + "written": 3124, "evaluated": 10000, + "resolved": 0, + "cleared": 40960, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 327680 + "resident": 901120 } } }, @@ -40,8 +52,12 @@ "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -51,8 +67,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -62,120 +82,194 @@ }, "load": { "uploaded": 20000, - "written": 4004, + "uploads": 5, + "written": 3892, "evaluated": 1000, + "resolved": 0, + "cleared": 4096, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 20480 + "resident": 90112 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 20480 + "resident": 90112 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 20480 + "resident": 90112 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 832, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 90112 }, "restyle": { "uploaded": 0, - "written": 2788, + "uploads": 0, + "written": 2228, "evaluated": 1000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 20480 + "resident": 90112 }, "one row": { "uploaded": 160000, - "written": 64160, + "uploads": 40, + "written": 33600, "evaluated": 40000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 184320 + "resident": 253952 }, "every row": { "uploaded": 4000, - "written": 1604, + "uploads": 1, + "written": 840, "evaluated": 1000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 188416 + "resident": 258048 }, "first pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 188416 + "resident": 258048 }, "pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 188416 + "resident": 258048 }, "hover": { "uploaded": 0, - "written": 3752, + "uploads": 0, + "written": 2600, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 188416 + "resident": 258048 }, "select": { "uploaded": 0, - "written": 3284, + "uploads": 0, + "written": 2516, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 188416 + "resident": 258048 + }, + "append": { + "uploaded": 816400, + "uploads": 200, + "written": 53528, + "evaluated": 40820, + "resolved": 0, + "cleared": 8192, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 1409024 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -185,8 +279,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -196,120 +294,194 @@ }, "load": { "uploaded": 200000, - "written": 4004, + "uploads": 15, + "written": 3956, "evaluated": 10000, + "resolved": 0, + "cleared": 40960, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 327680 + "resident": 901120 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 327680 + "resident": 901120 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 327680 + "resident": 901120 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 832, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 901120 }, "restyle": { "uploaded": 0, - "written": 2788, + "uploads": 0, + "written": 2292, "evaluated": 10000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 327680 + "resident": 901120 }, "one row": { - "uploaded": 1600000, - "written": 64160, - "evaluated": 400000, + "uploaded": 655360, + "uploads": 40, + "written": 33600, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2949120 + "resident": 1556480 }, "every row": { "uploaded": 40000, - "written": 1604, + "uploads": 3, + "written": 856, "evaluated": 10000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3014656 + "resident": 1597440 }, "first pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3014656 + "resident": 1597440 }, "pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3014656 + "resident": 1597440 }, "hover": { "uploaded": 0, - "written": 3752, + "uploads": 0, + "written": 2600, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3014656 + "resident": 1597440 }, "select": { "uploaded": 0, - "written": 3284, + "uploads": 0, + "written": 2516, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3014656 + "resident": 1597440 + }, + "append": { + "uploaded": 1462800, + "uploads": 200, + "written": 54720, + "evaluated": 73140, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 3235840 } }, "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -319,8 +491,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -330,120 +506,194 @@ }, "load": { "uploaded": 2000000, - "written": 4032, + "uploads": 125, + "written": 4660, "evaluated": 100000, + "resolved": 0, + "cleared": 401408, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 2621440 + "resident": 8830976 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2621440 + "resident": 8830976 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2621440 + "resident": 8830976 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 832, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 8830976 }, "restyle": { "uploaded": 0, - "written": 2816, + "uploads": 0, + "written": 2996, "evaluated": 100000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2621440 + "resident": 8830976 }, "one row": { - "uploaded": 10485760, - "written": 64960, - "evaluated": 2621440, + "uploaded": 655360, + "uploads": 40, + "written": 33600, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13107200 + "resident": 9486336 }, "every row": { "uploaded": 400000, - "written": 1628, + "uploads": 25, + "written": 1032, "evaluated": 100000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13631488 + "resident": 9887744 }, "first pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13631488 + "resident": 9887744 }, "pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13631488 + "resident": 9887744 }, "hover": { "uploaded": 0, - "written": 3752, + "uploads": 0, + "written": 2600, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13631488 + "resident": 9887744 }, "select": { "uploaded": 0, - "written": 3284, + "uploads": 0, + "written": 2516, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13631488 + "resident": 9887744 + }, + "append": { + "uploaded": 1373200, + "uploads": 200, + "written": 52480, + "evaluated": 68660, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 11526144 } }, "1000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -453,8 +703,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -463,121 +717,195 @@ "resident": 0 }, "load": { - "uploaded": 1567520, - "written": 5608, - "evaluated": 78376, + "uploaded": 1638400, + "uploads": 100, + "written": 4868, + "evaluated": 81920, + "resolved": 0, + "cleared": 262144, + "packed": 409600, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 1638400 + "resident": 6094848 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33544, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1638400 + "resident": 2752512 }, "zoom": { - "uploaded": 13309320, - "written": 51652, - "evaluated": 665466, + "uploaded": 3932160, + "uploads": 240, + "written": 56424, + "evaluated": 196608, + "resolved": 0, + "cleared": 786432, + "packed": 983040, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15319040 + "resident": 20054016 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 1264, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 20054016 }, "restyle": { "uploaded": 0, - "written": 2788, - "evaluated": 15876, + "uploads": 0, + "written": 2340, + "evaluated": 16384, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15319040 + "resident": 20054016 }, "one row": { "uploaded": 0, - "written": 64000, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15319040 + "resident": 6684672 }, "every row": { - "uploaded": 63504, - "written": 1604, - "evaluated": 15876, + "uploaded": 65536, + "uploads": 4, + "written": 864, + "evaluated": 16384, + "resolved": 0, + "cleared": 0, + "packed": 16384, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15384576 + "resident": 6750208 }, "first pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15384576 + "resident": 6750208 }, "pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15384576 + "resident": 6750208 }, "hover": { "uploaded": 0, - "written": 3752, + "uploads": 0, + "written": 2600, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15384576 + "resident": 6750208 }, "select": { - "uploaded": 319480, - "written": 3672, - "evaluated": 15974, + "uploaded": 81920, + "uploads": 5, + "written": 3024, + "evaluated": 4096, + "resolved": 0, + "cleared": 16384, + "packed": 490, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15712256 + "resident": 7110656 + }, + "append": { + "uploaded": 3358720, + "uploads": 205, + "written": 54692, + "evaluated": 167936, + "resolved": 0, + "cleared": 16384, + "packed": 119785, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 10747904 } }, "2000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -587,8 +915,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -597,114 +929,184 @@ "resident": 0 }, "load": { - "uploaded": 1607860, - "written": 5608, - "evaluated": 80393, + "uploaded": 1638400, + "uploads": 100, + "written": 4868, + "evaluated": 81920, + "resolved": 0, + "cleared": 262144, + "packed": 409600, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 1638400 + "resident": 6094848 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33544, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1638400 + "resident": 2752512 }, "zoom": { - "uploaded": 13591040, - "written": 51652, - "evaluated": 679552, + "uploaded": 4014080, + "uploads": 245, + "written": 56640, + "evaluated": 200704, + "resolved": 0, + "cleared": 802816, + "packed": 1003520, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15564800 + "resident": 20414464 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 1272, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 20414464 }, "restyle": { "uploaded": 0, - "written": 2788, - "evaluated": 15876, + "uploads": 0, + "written": 2340, + "evaluated": 16384, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15564800 + "resident": 20414464 }, "one row": { "uploaded": 0, - "written": 64000, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15564800 + "resident": 6766592 }, "every row": { - "uploaded": 63504, - "written": 1604, - "evaluated": 15876, + "uploaded": 65536, + "uploads": 4, + "written": 864, + "evaluated": 16384, + "resolved": 0, + "cleared": 0, + "packed": 16384, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15630336 + "resident": 6832128 }, "first pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15630336 + "resident": 6832128 }, "pick": { "uploaded": 0, - "written": 2440, + "uploads": 0, + "written": 1672, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15630336 + "resident": 6832128 }, "hover": { "uploaded": 0, - "written": 3752, + "uploads": 0, + "written": 2600, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15630336 + "resident": 6832128 }, "select": { - "uploaded": 319500, - "written": 3672, - "evaluated": 15975, + "uploaded": 81920, + "uploads": 5, + "written": 3024, + "evaluated": 4096, + "resolved": 0, + "cleared": 16384, + "packed": 495, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 15958016 + "resident": 7192576 + }, + "append": { + "uploaded": 3276800, + "uploads": 200, + "written": 54200, + "evaluated": 163840, + "resolved": 0, + "cleared": 16384, + "packed": 234500, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 10747904 } } }, @@ -712,8 +1114,12 @@ "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -723,8 +1129,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -734,87 +1144,149 @@ }, "load": { "uploaded": 36352, - "written": 5080, + "uploads": 7, + "written": 12712, "evaluated": 3008, - "readbacks": 2, + "resolved": 1984, + "cleared": 12288, + "packed": 0, + "readbacks": 3, "jobs": 0, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 36864 + "resident": 180224 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 36864 + "resident": 180224 }, "zoom": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 36864 + "resident": 180224 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 1088, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 180224 }, "one row": { "uploaded": 163840, - "written": 74400, + "uploads": 40, + "written": 43840, "evaluated": 40960, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 200704 + "resident": 344064 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 200704 + "resident": 344064 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 200704 + "resident": 344064 }, "hover": { "uploaded": 0, - "written": 4536, + "uploads": 0, + "written": 3384, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 200704 + "resident": 344064 + }, + "append": { + "uploaded": 835600, + "uploads": 200, + "written": 50704, + "evaluated": 41780, + "resolved": 0, + "cleared": 8192, + "packed": 0, + "readbacks": 40, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 1986560 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -824,8 +1296,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -835,87 +1311,149 @@ }, "load": { "uploaded": 358400, - "written": 5080, + "uploads": 25, + "written": 11792, "evaluated": 29800, - "readbacks": 2, + "resolved": 19800, + "cleared": 122880, + "packed": 0, + "readbacks": 3, "jobs": 0, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 589824 + "resident": 1802240 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 589824 + "resident": 1802240 }, "zoom": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 589824 + "resident": 1802240 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 1088, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 1802240 }, "one row": { - "uploaded": 1600000, - "written": 74400, - "evaluated": 400000, + "uploaded": 655360, + "uploads": 40, + "written": 43840, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3211264 + "resident": 2457600 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3211264 + "resident": 2457600 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3211264 + "resident": 2457600 }, "hover": { "uploaded": 0, - "written": 4520, + "uploads": 0, + "written": 3368, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3211264 + "resident": 2457600 + }, + "append": { + "uploaded": 1462800, + "uploads": 200, + "written": 51968, + "evaluated": 73140, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 40, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 4096000 } }, "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -925,8 +1463,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -936,87 +1478,149 @@ }, "load": { "uploaded": 3589760, - "written": 5152, + "uploads": 223, + "written": 12552, "evaluated": 298936, - "readbacks": 2, + "resolved": 199080, + "cleared": 1204224, + "packed": 0, + "readbacks": 3, "jobs": 1, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 17661952 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 17661952 }, "zoom": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 17661952 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 1088, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 0, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 17661952 }, "one row": { - "uploaded": 10485760, - "written": 75200, - "evaluated": 2621440, + "uploaded": 655360, + "uploads": 40, + "written": 43840, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 14712832 + "resident": 18317312 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 14712832 + "resident": 18317312 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 14712832 + "resident": 18317312 }, "hover": { "uploaded": 0, - "written": 4520, + "uploads": 0, + "written": 3368, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 14712832 + "resident": 18317312 + }, + "append": { + "uploaded": 1258000, + "uploads": 200, + "written": 49728, + "evaluated": 62900, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 40, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 19955712 } }, "1000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1026,8 +1630,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1036,88 +1644,150 @@ "resident": 0 }, "load": { - "uploaded": 16751128, - "written": 37956, - "evaluated": 983822, - "readbacks": 6, + "uploaded": 4980736, + "uploads": 304, + "written": 20544, + "evaluated": 290816, + "resolved": 69632, + "cleared": 835584, + "packed": 1224809, + "readbacks": 5, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 18153472 + "resident": 17481728 }, "pan": { - "uploaded": 14622728, - "written": 77120, - "evaluated": 864442, - "readbacks": 6, + "uploaded": 0, + "uploads": 0, + "written": 48480, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 4, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 33619968 + "resident": 17481728 }, "zoom": { - "uploaded": 31139892, - "written": 63588, - "evaluated": 1990386, - "readbacks": 5, - "jobs": 4, + "uploaded": 12845056, + "uploads": 784, + "written": 212164, + "evaluated": 868352, + "resolved": 380928, + "cleared": 3440640, + "packed": 3195384, + "readbacks": 16, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 76759040 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 5088, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 1, + "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 65830912 + "resident": 76759040 }, "one row": { - "uploaded": 4775696, - "written": 75916, - "evaluated": 1193924, + "uploaded": 655360, + "uploads": 40, + "written": 43840, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 163840, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 70615040 + "resident": 77414400 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 70615040 + "resident": 77414400 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 70615040 + "resident": 77414400 }, "hover": { "uploaded": 0, - "written": 4520, + "uploads": 0, + "written": 3368, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 70615040 + "resident": 77414400 + }, + "append": { + "uploaded": 3276800, + "uploads": 200, + "written": 128232, + "evaluated": 163840, + "resolved": 0, + "cleared": 16384, + "packed": 119300, + "readbacks": 40, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 80969728 } }, "2000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1127,8 +1797,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1137,81 +1811,139 @@ "resident": 0 }, "load": { - "uploaded": 17197752, - "written": 37956, - "evaluated": 1008054, - "readbacks": 6, + "uploaded": 5144576, + "uploads": 314, + "written": 20904, + "evaluated": 299008, + "resolved": 69632, + "cleared": 851968, + "packed": 1245299, + "readbacks": 5, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 18481152 + "resident": 17924096 }, "pan": { - "uploaded": 14841448, - "written": 77120, - "evaluated": 875378, - "readbacks": 6, + "uploaded": 0, + "uploads": 0, + "written": 48512, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 4, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 35258368 + "resident": 17924096 }, "zoom": { - "uploaded": 39083136, - "written": 64504, - "evaluated": 2388039, - "readbacks": 5, - "jobs": 4, + "uploaded": 18939904, + "uploads": 1156, + "written": 272660, + "evaluated": 1175552, + "resolved": 385024, + "cleared": 4669440, + "packed": 4719371, + "readbacks": 16, + "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 76023040 + "resident": 104054784 + }, + "revisit": { + "uploaded": 0, + "uploads": 0, + "written": 5696, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 1, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 104054784 }, "one row": { - "uploaded": 2883584, - "written": 75884, - "evaluated": 720896, + "uploaded": 589824, + "uploads": 36, + "written": 43808, + "evaluated": 147456, + "resolved": 0, + "cleared": 0, + "packed": 147456, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 78906624 + "resident": 104644608 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 78906624 + "resident": 104644608 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 78906624 + "resident": 104644608 }, "hover": { "uploaded": 0, - "written": 4520, + "uploads": 0, + "written": 3368, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 78906624 + "resident": 104644608 + }, + "append": { + "uploaded": 3276800, + "uploads": 200, + "written": 181604, + "evaluated": 163840, + "resolved": 0, + "cleared": 16384, + "packed": 113700, + "readbacks": 40, + "jobs": 0, + "texts": 0, + "pipelines": 0, + "late": 0, + "resident": 108199936 } } }, @@ -1219,8 +1951,12 @@ "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1230,8 +1966,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1241,43 +1981,59 @@ }, "load": { "uploaded": 240000, - "written": 722428, + "uploads": 18, + "written": 712768, "evaluated": 10000, - "readbacks": 8, + "resolved": 0, + "cleared": 368640, + "packed": 0, + "readbacks": 7, "jobs": 0, - "texts": 207, - "pipelines": 20, + "texts": 187, + "pipelines": 33, "late": 0, - "resident": 393216 + "resident": 1269760 }, "pan": { "uploaded": 0, - "written": 2131648, + "uploads": 0, + "written": 2125904, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, - "texts": 586, + "texts": 526, "pipelines": 0, "late": 0, - "resident": 393216 + "resident": 1269760 }, "zoom": { "uploaded": 0, - "written": 4825740, + "uploads": 0, + "written": 5139848, "evaluated": 0, - "readbacks": 41, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 40, "jobs": 0, - "texts": 1297, + "texts": 1227, "pipelines": 0, "late": 0, - "resident": 393216 + "resident": 1269760 } }, "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1287,8 +2043,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1298,43 +2058,59 @@ }, "load": { "uploaded": 2400000, - "written": 1487188, + "uploads": 150, + "written": 1432848, "evaluated": 100000, - "readbacks": 8, + "resolved": 0, + "cleared": 3612672, + "packed": 0, + "readbacks": 7, "jobs": 0, - "texts": 216, - "pipelines": 20, + "texts": 187, + "pipelines": 33, "late": 0, - "resident": 3145728 + "resident": 12443648 }, "pan": { "uploaded": 0, - "written": 2831152, + "uploads": 0, + "written": 2350640, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, - "texts": 739, + "texts": 572, "pipelines": 0, "late": 0, - "resident": 3145728 + "resident": 12443648 }, "zoom": { "uploaded": 0, - "written": 6473928, + "uploads": 0, + "written": 6727060, "evaluated": 0, - "readbacks": 41, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 40, "jobs": 0, - "texts": 1641, + "texts": 1522, "pipelines": 0, "late": 0, - "resident": 3145728 + "resident": 12443648 } }, "1000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1344,8 +2120,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1354,37 +2134,49 @@ "resident": 0 }, "load": { - "uploaded": 1881024, - "written": 2977136, - "evaluated": 78376, - "readbacks": 18, + "uploaded": 1966080, + "uploads": 120, + "written": 2931332, + "evaluated": 81920, + "resolved": 0, + "cleared": 4653056, + "packed": 491520, + "readbacks": 16, "jobs": 1, - "texts": 392, - "pipelines": 20, + "texts": 338, + "pipelines": 33, "late": 0, - "resident": 1966080 + "resident": 3604480 }, "pan": { "uploaded": 0, - "written": 4540644, + "uploads": 0, + "written": 4017904, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, - "texts": 867, + "texts": 688, "pipelines": 0, "late": 0, - "resident": 1966080 + "resident": 3604480 }, "zoom": { - "uploaded": 15971184, - "written": 11933216, - "evaluated": 927610, - "readbacks": 45, + "uploaded": 4718592, + "uploads": 288, + "written": 11504400, + "evaluated": 196608, + "resolved": 0, + "cleared": 12582912, + "packed": 1179648, + "readbacks": 44, "jobs": 0, - "texts": 1659, + "texts": 1419, "pipelines": 0, "late": 0, - "resident": 18382848 + "resident": 27983872 } } }, @@ -1392,8 +2184,12 @@ "100": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1403,8 +2199,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1414,54 +2214,74 @@ }, "load": { "uploaded": 1456, - "written": 5496, + "uploads": 7, + "written": 10544, "evaluated": 116, - "readbacks": 3, + "resolved": 100, + "cleared": 768, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 2560 + "resident": 8448 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 45392, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2560 + "resident": 8448 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2560 + "resident": 8448 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2560 + "resident": 8448 } }, "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1471,8 +2291,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1482,54 +2306,74 @@ }, "load": { "uploaded": 12256, - "written": 5496, + "uploads": 7, + "written": 9456, "evaluated": 1016, - "readbacks": 3, + "resolved": 1000, + "cleared": 4352, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 13312 + "resident": 51456 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 45392, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13312 + "resident": 51456 }, "first pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13312 + "resident": 51456 }, "pick": { "uploaded": 0, - "written": 2960, + "uploads": 0, + "written": 2192, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13312 + "resident": 51456 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1539,8 +2383,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1550,54 +2398,74 @@ }, "load": { "uploaded": 120256, - "written": 5496, + "uploads": 13, + "written": 9560, "evaluated": 10016, - "readbacks": 3, + "resolved": 10000, + "cleared": 41216, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 197632 + "resident": 493824 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 45392, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 197632 + "resident": 493824 }, "first pick": { "uploaded": 0, - "written": 4448, + "uploads": 0, + "written": 2192, "evaluated": 0, - "readbacks": 2, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 197632 + "resident": 493824 }, "pick": { "uploaded": 0, - "written": 4448, + "uploads": 0, + "written": 2192, "evaluated": 0, - "readbacks": 2, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 197632 + "resident": 493824 } }, "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1607,8 +2475,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1618,47 +2490,63 @@ }, "load": { "uploaded": 1200256, - "written": 7348, + "uploads": 79, + "written": 10524, "evaluated": 100016, - "readbacks": 3, + "resolved": 100000, + "cleared": 401664, + "packed": 0, + "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 1573888 + "resident": 4819200 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 45392, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1573888 + "resident": 4819200 }, "first pick": { "uploaded": 0, - "written": 4448, + "uploads": 0, + "written": 2192, "evaluated": 0, - "readbacks": 2, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1573888 + "resident": 4819200 }, "pick": { "uploaded": 0, - "written": 4448, + "uploads": 0, + "written": 2192, "evaluated": 0, - "readbacks": 2, + "resolved": 0, + "cleared": 0, + "packed": 0, + "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1573888 + "resident": 4819200 } } }, @@ -1666,8 +2554,12 @@ "100": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1677,8 +2569,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1687,77 +2583,105 @@ "resident": 0 }, "load": { - "uploaded": 1340368, - "written": 3852, + "uploaded": 1340768, + "uploads": 52, + "written": 3968, "evaluated": 100, + "resolved": 0, + "cleared": 512, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 1344512 + "resident": 1641472 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1344512 + "resident": 1641472 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1344512 + "resident": 1641472 }, "one row": { "uploaded": 16000, - "written": 59680, + "uploads": 40, + "written": 35520, "evaluated": 4000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1364992 + "resident": 1661952 }, "first pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1364992 + "resident": 1661952 }, "pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1364992 + "resident": 1661952 } }, "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1767,8 +2691,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1777,77 +2705,105 @@ "resident": 0 }, "load": { - "uploaded": 4172168, - "written": 5348, + "uploaded": 4180168, + "uploads": 147, + "written": 7908, "evaluated": 1000, + "resolved": 0, + "cleared": 4096, + "packed": 0, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 4206592 + "resident": 4501504 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4206592 + "resident": 4501504 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4206592 + "resident": 4501504 }, "one row": { "uploaded": 160000, - "written": 61280, + "uploads": 40, + "written": 35520, "evaluated": 40000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4370432 + "resident": 4665344 }, "first pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4370432 + "resident": 4665344 }, "pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4370432 + "resident": 4665344 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1857,8 +2813,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1867,77 +2827,105 @@ "resident": 0 }, "load": { - "uploaded": 18104720, - "written": 5740, + "uploaded": 18151952, + "uploads": 645, + "written": 13328, "evaluated": 10000, + "resolved": 0, + "cleared": 40960, + "packed": 0, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 18284544 + "resident": 19697664 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 18284544 + "resident": 19697664 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 18284544 + "resident": 19697664 }, "one row": { - "uploaded": 1600000, - "written": 69760, - "evaluated": 400000, + "uploaded": 655360, + "uploads": 40, + "written": 35520, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 20905984 + "resident": 20353024 }, "first pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 20905984 + "resident": 20353024 }, "pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 20905984 + "resident": 20353024 } }, "30000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1947,8 +2935,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -1957,70 +2949,94 @@ "resident": 0 }, "load": { - "uploaded": 40370040, - "written": 60880, + "uploaded": 40413432, + "uploads": 1561, + "written": 31396, "evaluated": 30000, + "resolved": 0, + "cleared": 122880, + "packed": 0, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 40501248 + "resident": 43819008 }, "pan": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 40501248 + "resident": 43819008 }, "zoom": { "uploaded": 0, - "written": 48640, + "uploads": 0, + "written": 33280, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 40501248 + "resident": 43819008 }, "one row": { - "uploaded": 4800000, - "written": 83200, - "evaluated": 1200000, + "uploaded": 655360, + "uploads": 40, + "written": 35520, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 45744128 + "resident": 44474368 }, "first pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 45744128 + "resident": 44474368 }, "pick": { "uploaded": 0, - "written": 2432, + "uploads": 0, + "written": 1664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 45744128 + "resident": 44474368 } } }, @@ -2028,8 +3044,12 @@ "16": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2039,8 +3059,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2049,44 +3073,60 @@ "resident": 0 }, "load": { - "uploaded": 336000, - "written": 5260, + "uploaded": 277592, + "uploads": 12, + "written": 3764, "evaluated": 80, + "resolved": 0, + "cleared": 256, + "packed": 0, "readbacks": 0, "jobs": 4, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 344576 + "resident": 304640 }, "pan": { - "uploaded": 2160192, - "written": 50232, + "uploaded": 437472, + "uploads": 18, + "written": 35112, "evaluated": 512, + "resolved": 0, + "cleared": 512, + "packed": 0, "readbacks": 0, "jobs": 8, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2771968 + "resident": 781824 }, "zoom": { - "uploaded": 2119824, - "written": 51004, - "evaluated": 1088, + "uploaded": 792224, + "uploads": 42, + "written": 36072, + "evaluated": 1232, + "resolved": 0, + "cleared": 1024, + "packed": 0, "readbacks": 0, - "jobs": 23, + "jobs": 24, "texts": 0, "pipelines": 0, "late": 0, - "resident": 5546752 + "resident": 1672448 } }, "256": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2096,8 +3136,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2106,44 +3150,60 @@ "resident": 0 }, "load": { - "uploaded": 788640, - "written": 5264, + "uploaded": 642048, + "uploads": 32, + "written": 3864, "evaluated": 1280, + "resolved": 0, + "cleared": 5120, + "packed": 0, "readbacks": 0, "jobs": 4, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 922624 + "resident": 719872 }, "pan": { - "uploaded": 5062272, - "written": 50272, + "uploaded": 648192, + "uploads": 32, + "written": 35288, "evaluated": 8192, + "resolved": 0, + "cleared": 8192, + "packed": 0, "readbacks": 0, "jobs": 8, "texts": 0, "pipelines": 0, "late": 0, - "resident": 6333440 + "resident": 1449984 }, "zoom": { - "uploaded": 5673696, - "written": 51048, - "evaluated": 17152, + "uploaded": 1142528, + "uploads": 79, + "written": 36524, + "evaluated": 19456, + "resolved": 0, + "cleared": 16384, + "packed": 0, "readbacks": 0, - "jobs": 22, + "jobs": 23, "texts": 0, "pipelines": 0, "late": 0, - "resident": 13184000 + "resident": 3264512 } }, "4096": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2153,8 +3213,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2163,37 +3227,49 @@ "resident": 0 }, "load": { - "uploaded": 3344464, - "written": 6704, - "evaluated": 20480, + "uploaded": 2638176, + "uploads": 258, + "written": 6632, + "evaluated": 16384, + "resolved": 0, + "cleared": 65536, + "packed": 0, "readbacks": 0, - "jobs": 6, + "jobs": 4, "texts": 0, - "pipelines": 14, + "pipelines": 28, "late": 0, - "resident": 3424256 + "resident": 3194880 }, "pan": { - "uploaded": 21165184, - "written": 52928, - "evaluated": 172032, + "uploaded": 1286320, + "uploads": 128, + "written": 43052, + "evaluated": 57344, + "resolved": 0, + "cleared": 98304, + "packed": 0, "readbacks": 0, - "jobs": 12, + "jobs": 8, "texts": 0, "pipelines": 0, "late": 0, - "resident": 25509888 + "resident": 4759552 }, "zoom": { - "uploaded": 20321088, - "written": 55384, - "evaluated": 225280, + "uploaded": 2769536, + "uploads": 388, + "written": 51952, + "evaluated": 65536, + "resolved": 0, + "cleared": 131072, + "packed": 0, "readbacks": 0, - "jobs": 19, + "jobs": 10, "texts": 0, "pipelines": 0, "late": 0, - "resident": 46465024 + "resident": 8675328 } } }, @@ -2201,8 +3277,12 @@ "100": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2212,8 +3292,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2222,55 +3306,75 @@ "resident": 0 }, "load": { - "uploaded": 24240, - "written": 8108, + "uploaded": 25440, + "uploads": 22, + "written": 9880, "evaluated": 300, + "resolved": 0, + "cleared": 5120, + "packed": 0, "readbacks": 1, "jobs": 2, "texts": 0, - "pipelines": 23, + "pipelines": 28, "late": 0, - "resident": 33792 + "resident": 52736 }, "pan": { "uploaded": 0, - "written": 60352, + "uploads": 0, + "written": 44608, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 33792 + "resident": 52736 }, "edit": { - "uploaded": 371960, - "written": 105216, + "uploaded": 273560, + "uploads": 149, + "written": 49552, "evaluated": 4000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 537600 + "resident": 451072 }, "arrange": { "uploaded": 19000, - "written": 5836, + "uploads": 8, + "written": 3432, "evaluated": 100, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 2, "texts": 0, "pipelines": 0, "late": 0, - "resident": 564736 + "resident": 478208 } }, "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2280,8 +3384,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2290,55 +3398,75 @@ "resident": 0 }, "load": { - "uploaded": 242960, - "written": 15308, + "uploaded": 254960, + "uploads": 50, + "written": 18280, "evaluated": 3000, + "resolved": 0, + "cleared": 40960, + "packed": 0, "readbacks": 1, "jobs": 2, "texts": 0, - "pipelines": 23, + "pipelines": 28, "late": 0, - "resident": 335872 + "resident": 479232 }, "pan": { "uploaded": 0, - "written": 60352, + "uploads": 0, + "written": 44608, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 335872 + "resident": 479232 }, "edit": { - "uploaded": 3463800, - "written": 112384, + "uploaded": 770360, + "uploads": 149, + "written": 56720, "evaluated": 40000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 6070272 + "resident": 1372160 }, "arrange": { - "uploaded": 178144, - "written": 5836, + "uploaded": 104608, + "uploads": 19, + "written": 3472, "evaluated": 1000, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 2, "texts": 0, "pipelines": 0, "late": 0, - "resident": 6352896 + "resident": 1515520 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2348,8 +3476,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2358,48 +3490,64 @@ "resident": 0 }, "load": { - "uploaded": 2429800, - "written": 88800, + "uploaded": 2484264, + "uploads": 366, + "written": 89544, "evaluated": 30000, + "resolved": 0, + "cleared": 409600, + "packed": 0, "readbacks": 1, "jobs": 2, "texts": 0, - "pipelines": 23, + "pipelines": 28, "late": 0, - "resident": 3014656 + "resident": 4747264 }, "pan": { "uploaded": 0, - "written": 60352, + "uploads": 0, + "written": 44608, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 3014656 + "resident": 4747264 }, "edit": { - "uploaded": 34375160, - "written": 104512, - "evaluated": 400000, + "uploaded": 2479352, + "uploads": 149, + "written": 50928, + "evaluated": 163840, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 42336256 + "resident": 7335936 }, "arrange": { - "uploaded": 1710032, - "written": 5852, - "evaluated": 10000, + "uploaded": 333632, + "uploads": 50, + "written": 3696, + "evaluated": 4096, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 2, "texts": 0, "pipelines": 0, "late": 0, - "resident": 44236800 + "resident": 7794688 } } }, @@ -2407,8 +3555,12 @@ "10": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2418,8 +3570,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2429,65 +3585,89 @@ }, "load": { "uploaded": 49152, - "written": 44096, + "uploads": 2, + "written": 43948, "evaluated": 10, + "resolved": 0, + "cleared": 256, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 22, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 524288 + "resident": 524800 }, "stream": { "uploaded": 48, - "written": 194816, + "uploads": 2, + "written": 165952, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 524288 + "resident": 524800 }, "scrub": { "uploaded": 0, - "written": 179136, + "uploads": 0, + "written": 163776, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 524288 + "resident": 524800 }, "narrow": { "uploaded": 1872, - "written": 210040, + "uploads": 2, + "written": 181852, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 19, "pipelines": 0, "late": 0, - "resident": 524288 + "resident": 524800 }, "hover": { "uploaded": 0, - "written": 11816, + "uploads": 0, + "written": 10664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 524288 + "resident": 524800 } }, "100": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2497,8 +3677,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2508,65 +3692,89 @@ }, "load": { "uploaded": 417792, - "written": 44108, + "uploads": 3, + "written": 43960, "evaluated": 100, + "resolved": 0, + "cleared": 512, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 22, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 786432 + "resident": 788992 }, "stream": { "uploaded": 408, - "written": 195296, + "uploads": 2, + "written": 165952, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 786432 + "resident": 788992 }, "scrub": { "uploaded": 0, - "written": 179136, + "uploads": 0, + "written": 163776, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 786432 + "resident": 788992 }, "narrow": { "uploaded": 15912, - "written": 210364, + "uploads": 2, + "written": 182144, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 19, "pipelines": 0, "late": 0, - "resident": 786432 + "resident": 788992 }, "hover": { "uploaded": 0, - "written": 11816, + "uploads": 0, + "written": 10664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 786432 + "resident": 788992 } }, "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2576,8 +3784,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2587,65 +3799,89 @@ }, "load": { "uploaded": 4104192, - "written": 44468, + "uploads": 17, + "written": 44128, "evaluated": 1000, + "resolved": 0, + "cleared": 4096, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 22, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 4456448 + "resident": 4493312 }, "stream": { "uploaded": 4008, - "written": 202016, + "uploads": 2, + "written": 165952, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4456448 + "resident": 4493312 }, "scrub": { "uploaded": 0, - "written": 179136, + "uploads": 0, + "written": 163776, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4456448 + "resident": 4493312 }, "narrow": { "uploaded": 156312, - "written": 214324, + "uploads": 3, + "written": 187648, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 19, "pipelines": 0, "late": 0, - "resident": 4718592 + "resident": 4755456 }, "hover": { "uploaded": 0, - "written": 11816, + "uploads": 0, + "written": 10664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4718592 + "resident": 4755456 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2655,8 +3891,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2666,58 +3906,78 @@ }, "load": { "uploaded": 40968192, - "written": 51748, + "uploads": 158, + "written": 42456, "evaluated": 10000, + "resolved": 0, + "cleared": 40960, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 22, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 45088768 + "resident": 41787392 }, "stream": { "uploaded": 40008, - "written": 275048, + "uploads": 4, + "written": 172464, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 45088768 + "resident": 42311680 }, "scrub": { "uploaded": 0, - "written": 179136, + "uploads": 0, + "written": 163776, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 45088768 + "resident": 42311680 }, "narrow": { "uploaded": 1560312, - "written": 255868, + "uploads": 9, + "written": 235048, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 40, "jobs": 0, "texts": 19, "pipelines": 0, "late": 0, - "resident": 46923776 + "resident": 43622400 }, "hover": { "uploaded": 0, - "written": 11816, + "uploads": 0, + "written": 10664, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 2, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 46923776 + "resident": 43622400 } } }, @@ -2725,8 +3985,12 @@ "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2736,8 +4000,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2747,43 +4015,59 @@ }, "load": { "uploaded": 12000, - "written": 53080, + "uploads": 2, + "written": 52820, "evaluated": 1, + "resolved": 0, + "cleared": 256, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 26, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 524288 + "resident": 524800 }, "scrub": { "uploaded": 0, - "written": 179200, + "uploads": 0, + "written": 163840, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 524288 + "resident": 524800 }, "narrow": { "uploaded": 0, - "written": 211216, + "uploads": 0, + "written": 200228, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, - "texts": 19, + "texts": 27, "pipelines": 0, "late": 0, - "resident": 524288 + "resident": 524800 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2793,8 +4077,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2803,44 +4091,60 @@ "resident": 0 }, "load": { - "uploaded": 140000, - "written": 55252, + "uploaded": 20000, + "uploads": 3, + "written": 51248, "evaluated": 1, + "resolved": 0, + "cleared": 256, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 26, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 1310720 + "resident": 786944 }, "scrub": { "uploaded": 0, - "written": 179200, + "uploads": 0, + "written": 163840, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 1310720 + "resident": 786944 }, "narrow": { - "uploaded": 0, - "written": 209620, + "uploaded": 120000, + "uploads": 2, + "written": 198096, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, - "texts": 16, + "texts": 23, "pipelines": 0, "late": 0, - "resident": 1310720 + "resident": 1311232 } }, "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2850,8 +4154,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2860,44 +4168,60 @@ "resident": 0 }, "load": { - "uploaded": 749280, - "written": 57492, + "uploaded": 224992, + "uploads": 6, + "written": 53776, "evaluated": 1, + "resolved": 0, + "cleared": 256, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 26, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 2097152 + "resident": 1573376 }, "scrub": { "uploaded": 0, - "written": 179200, + "uploads": 0, + "written": 163840, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2097152 + "resident": 1573376 }, "narrow": { "uploaded": 0, - "written": 218196, + "uploads": 0, + "written": 208184, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, - "texts": 16, + "texts": 23, "pipelines": 0, "late": 0, - "resident": 2097152 + "resident": 1573376 } }, "1000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2907,8 +4231,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2917,37 +4245,49 @@ "resident": 0 }, "load": { - "uploaded": 805536, - "written": 59652, + "uploaded": 281248, + "uploads": 6, + "written": 56116, "evaluated": 1, + "resolved": 0, + "cleared": 256, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 26, - "pipelines": 15, + "pipelines": 29, "late": 0, - "resident": 2097152 + "resident": 1573376 }, "scrub": { "uploaded": 0, - "written": 179200, + "uploads": 0, + "written": 163840, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2097152 + "resident": 1573376 }, "narrow": { "uploaded": 0, - "written": 224116, + "uploads": 0, + "written": 217356, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 39, "jobs": 0, - "texts": 16, + "texts": 23, "pipelines": 0, "late": 0, - "resident": 2097152 + "resident": 1573376 } } }, @@ -2955,8 +4295,12 @@ "1000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2966,8 +4310,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -2977,32 +4325,44 @@ }, "load": { "uploaded": 20000, - "written": 4004, + "uploads": 5, + "written": 3892, "evaluated": 1000, + "resolved": 0, + "cleared": 4096, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 20480 + "resident": 90112 }, "move": { "uploaded": 20000, - "written": 74568, + "uploads": 5, + "written": 50808, "evaluated": 2000, + "resolved": 0, + "cleared": 65536, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 40960 + "resident": 241664 } }, "10000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3012,8 +4372,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3023,32 +4387,44 @@ }, "load": { "uploaded": 200000, - "written": 4004, + "uploads": 15, + "written": 3956, "evaluated": 10000, + "resolved": 0, + "cleared": 40960, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 327680 + "resident": 901120 }, "move": { "uploaded": 200000, - "written": 74568, + "uploads": 15, + "written": 50888, "evaluated": 20000, + "resolved": 0, + "cleared": 655360, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 655360 + "resident": 2416640 } }, "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3058,8 +4434,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3069,32 +4449,44 @@ }, "load": { "uploaded": 2000000, - "written": 4032, + "uploads": 125, + "written": 3828, "evaluated": 100000, + "resolved": 0, + "cleared": 401408, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 2621440 + "resident": 8830976 }, "move": { "uploaded": 2000000, - "written": 74596, + "uploads": 125, + "written": 51768, "evaluated": 200000, + "resolved": 0, + "cleared": 6422528, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 5242880 + "resident": 23683072 } }, "1000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3104,8 +4496,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3114,26 +4510,34 @@ "resident": 0 }, "load": { - "uploaded": 1567520, - "written": 388296, - "evaluated": 155802, + "uploaded": 1638400, + "uploads": 100, + "written": 629272, + "evaluated": 98304, + "resolved": 0, + "cleared": 5296640, + "packed": 409600, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 9, + "pipelines": 26, "late": 0, - "resident": 1638400 + "resident": 16904704 }, "move": { - "uploaded": 952560, - "written": 298688, - "evaluated": 71468, + "uploaded": 655360, + "uploads": 40, + "written": 245776, + "evaluated": 49152, + "resolved": 0, + "cleared": 557056, + "packed": 163840, "readbacks": 0, "jobs": 1, "texts": 0, "pipelines": 0, "late": 0, - "resident": 2621440 + "resident": 7110656 } } }, @@ -3141,8 +4545,12 @@ "1": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3152,8 +4560,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3163,32 +4575,44 @@ }, "load": { "uploaded": 3589760, - "written": 5104, + "uploads": 223, + "written": 11368, "evaluated": 298936, + "resolved": 199080, + "cleared": 1204224, + "packed": 0, "readbacks": 1, "jobs": 1, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 17661952 }, "pan": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 43520, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 17661952 } }, "2": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3198,8 +4622,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3209,32 +4637,44 @@ }, "load": { "uploaded": 3589760, - "written": 9184, + "uploads": 223, + "written": 21712, "evaluated": 597872, + "resolved": 398160, + "cleared": 2408448, + "packed": 0, "readbacks": 2, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "pan": { "uploaded": 0, - "written": 117760, + "uploads": 0, + "written": 87040, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 } }, "4": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3244,8 +4684,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3255,32 +4699,44 @@ }, "load": { "uploaded": 3589760, - "written": 17344, + "uploads": 223, + "written": 49936, "evaluated": 1195744, + "resolved": 796320, + "cleared": 4816896, + "packed": 0, "readbacks": 4, "jobs": 4, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 59809792 }, "pan": { "uploaded": 0, - "written": 235520, + "uploads": 0, + "written": 174080, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 59809792 } }, "8": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3290,8 +4746,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3301,25 +4761,33 @@ }, "load": { "uploaded": 3589760, - "written": 33664, + "uploads": 223, + "written": 93328, "evaluated": 2391488, + "resolved": 1592640, + "cleared": 9666560, + "packed": 0, "readbacks": 8, "jobs": 8, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 116006912 }, "pan": { "uploaded": 0, - "written": 471040, + "uploads": 0, + "written": 348160, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 116006912 } } }, @@ -3327,8 +4795,12 @@ "100000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3338,8 +4810,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3348,44 +4824,60 @@ "resident": 0 }, "load": { - "uploaded": 4764460, - "written": 12468, - "evaluated": 289253, + "uploaded": 2686976, + "uploads": 164, + "written": 32132, + "evaluated": 163840, + "resolved": 49152, + "cleared": 442368, + "packed": 658174, "readbacks": 1, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 5636096 + "resident": 9027584 }, "pan": { - "uploaded": 6457376, - "written": 72224, - "evaluated": 416008, + "uploaded": 0, + "uploads": 0, + "written": 47996, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 14417920 + "resident": 9027584 }, "zoom": { - "uploaded": 0, - "written": 58880, - "evaluated": 0, + "uploaded": 1376256, + "uploads": 84, + "written": 366848, + "evaluated": 98304, + "resolved": 114688, + "cleared": 229376, + "packed": 302439, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 14417920 + "resident": 10256384 } }, "1000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3395,8 +4887,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3405,44 +4901,60 @@ "resident": 0 }, "load": { - "uploaded": 9351656, - "written": 22148, - "evaluated": 557254, + "uploaded": 3588096, + "uploads": 219, + "written": 41200, + "evaluated": 208896, + "resolved": 49152, + "cleared": 589824, + "packed": 865039, "readbacks": 1, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 10092544 + "resident": 12435456 }, "pan": { - "uploaded": 8253296, - "written": 72320, - "evaluated": 505804, + "uploaded": 0, + "uploads": 0, + "written": 48232, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 16646144 + "resident": 12435456 }, "zoom": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 51136, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 16646144 + "resident": 12435456 } }, "2000000": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3452,8 +4964,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3462,37 +4978,49 @@ "resident": 0 }, "load": { - "uploaded": 9480488, - "written": 22148, - "evaluated": 564688, + "uploaded": 3588096, + "uploads": 219, + "written": 37944, + "evaluated": 208896, + "resolved": 49152, + "cleared": 589824, + "packed": 885564, "readbacks": 1, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 10420224 + "resident": 12435456 }, "pan": { - "uploaded": 8270536, - "written": 72320, - "evaluated": 506666, + "uploaded": 0, + "uploads": 0, + "written": 48232, + "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 16187392 + "resident": 12435456 }, "zoom": { "uploaded": 0, - "written": 58880, + "uploads": 0, + "written": 51136, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 16187392 + "resident": 12435456 } } }, @@ -3500,8 +5028,12 @@ "518400": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3511,8 +5043,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3522,65 +5058,89 @@ }, "load": { "uploaded": 3589760, - "written": 9184, + "uploads": 223, + "written": 23888, "evaluated": 597872, + "resolved": 398160, + "cleared": 2408448, + "packed": 0, "readbacks": 2, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "image": { "uploaded": 0, - "written": 8928, + "uploads": 0, + "written": 6624, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 6, + "pipelines": 1, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "png": { "uploaded": 0, - "written": 2944, + "uploads": 0, + "written": 2176, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "video": { "uploaded": 0, - "written": 97248, + "uploads": 0, + "written": 71904, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 30, "jobs": 0, "texts": 0, "pipelines": 1, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "held": { "uploaded": 0, - "written": 5984, + "uploads": 0, + "written": 4448, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 } }, "2073600": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3590,8 +5150,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3601,65 +5165,89 @@ }, "load": { "uploaded": 3589760, - "written": 9184, + "uploads": 223, + "written": 21712, "evaluated": 597872, + "resolved": 398160, + "cleared": 2408448, + "packed": 0, "readbacks": 2, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "image": { "uploaded": 0, - "written": 8928, + "uploads": 0, + "written": 6624, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 6, + "pipelines": 1, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "png": { "uploaded": 0, - "written": 2944, + "uploads": 0, + "written": 2176, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "video": { "uploaded": 0, - "written": 97248, + "uploads": 0, + "written": 71904, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 30, "jobs": 0, "texts": 0, "pipelines": 1, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "held": { "uploaded": 0, - "written": 5984, + "uploads": 0, + "written": 4448, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 } }, "8294400": { "init": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3669,8 +5257,12 @@ }, "data": { "uploaded": 0, + "uploads": 0, "written": 0, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, @@ -3680,58 +5272,78 @@ }, "load": { "uploaded": 3589760, - "written": 9184, + "uploads": 223, + "written": 19536, "evaluated": 597872, + "resolved": 398160, + "cleared": 2408448, + "packed": 0, "readbacks": 2, "jobs": 2, "texts": 0, - "pipelines": 18, + "pipelines": 28, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "image": { "uploaded": 0, - "written": 11872, + "uploads": 0, + "written": 6624, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, - "pipelines": 6, + "pipelines": 1, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "png": { "uploaded": 0, - "written": 2944, + "uploads": 0, + "written": 2176, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 0, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "video": { "uploaded": 0, - "written": 97248, + "uploads": 0, + "written": 71904, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 30, "jobs": 0, "texts": 0, "pipelines": 1, "late": 0, - "resident": 4227072 + "resident": 31711232 }, "held": { "uploaded": 0, - "written": 5984, + "uploads": 0, + "written": 4448, "evaluated": 0, + "resolved": 0, + "cleared": 0, + "packed": 0, "readbacks": 1, "jobs": 0, "texts": 0, "pipelines": 0, "late": 0, - "resident": 4227072 + "resident": 31711232 } } } diff --git a/benchmark/docs.ts b/benchmark/docs.ts index f6ed281c..97b4b6e8 100644 --- a/benchmark/docs.ts +++ b/benchmark/docs.ts @@ -1,22 +1,20 @@ /** - * The documentation's performance table, from a committed baseline: each scenario at the largest - * size it ran, what opening took, and what a frame took while it moved and was picked from. + * Writes the documentation's performance table, docs/_static/performance.csv, from a committed + * times baseline. Each row is a scenario at the largest size it ran: its `load` time, the slowest + * CPU and GPU frame times among its moving phases, and its slowest pick or hover. * * node benchmark/docs.ts [baselines/MACHINE.json] + * + * Without an argument it reads the last machine baseline in sorted file name order. */ import fs from 'node:fs'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; +import type { TimeBaseline } from './judge.ts'; import { scenarios } from './page/scenarios.ts'; -interface Measured { - time: number; - gpu: number; -} -interface Baseline { - machine: string; - times: Record>>; -} +/** One size's phases of a scenario, each with its times. */ +type Phases = TimeBaseline['times'][string][number]; const here = path.dirname(fileURLToPath(import.meta.url)); const baselines = path.join(here, 'baselines'); @@ -30,15 +28,15 @@ const file = .sort() .at(-1)!, ); -const baseline = JSON.parse(fs.readFileSync(file, 'utf8')) as Baseline; +const baseline = JSON.parse(fs.readFileSync(file, 'utf8')) as TimeBaseline; -/** Phases that run every frame, as the view moves or plays. */ +/** Phases that act every frame while the view moves or plays. */ const MOVING = ['pan', 'zoom', 'move', 'scrub', 'narrow', 'stream']; /** Phases that answer the pointer. */ const PICKING = ['pick', 'hover']; const ms = (v: number) => (v >= 100 ? Math.round(v).toLocaleString('en-US') : v.toFixed(1)); -const most = (phases: Record, names: string[], of: (m: Measured) => number) => { +const most = (phases: Phases, names: string[], of: (m: Phases[string]) => number) => { const found = names.filter((n) => phases[n]).map((n) => of(phases[n]!)); return found.length ? ms(Math.max(...found)) : ''; }; @@ -52,7 +50,7 @@ for (const [name, sizes] of Object.entries(baseline.times)) { .map(Number) .sort((a, b) => a - b) .at(-1)!; - const phases = sizes[String(largest)]!; + const phases = sizes[largest]!; rows.push([ scenario.about[0]!.toUpperCase() + scenario.about.slice(1), `${largest.toLocaleString('en-US')} ${scenario.unit}`, diff --git a/benchmark/judge.ts b/benchmark/judge.ts index b504c6ec..fbb44691 100644 --- a/benchmark/judge.ts +++ b/benchmark/judge.ts @@ -1,22 +1,27 @@ /** - * What the runs say. Rules that hold on any machine: no errors, nothing drawn idle, no pipeline - * compiled where it was first used, resident data within its budget, every phase growing no faster - * than its order, and no more work than the work baseline. Frames are stepped, each waiting for the - * GPU, so what the data and the frames drawn set (`EXACT`) is held exactly in a phase that acts once - * and settles, unless fetched data feeds it; the rest, and every count of a phase acting each frame, - * moves with the frame a job or a fetch lands in, and is held within `STEADY`. Then, on the machine - * the times baseline was taken on, no phase much slower. + * Judges the runs. These rules hold on any machine: no errors, no frames drawn while idle, no + * pipeline compiled at first use, resident data within its budget, no phase growing faster than its + * order, and no more work than the work baseline. + * + * Frames are stepped, each waiting for the GPU. So in a phase that acts once and settles, the + * `EXACT` counts, which only the data and the frames drawn set, may not exceed the baseline at all, + * unless the scenario fetches its data. Other counts, and every count of a steady phase, vary with + * the frame a job or fetch finishes in, so they may reach `STEADY` times the baseline. + * + * On the machine the times baseline was taken on, no phase's fastest repeat may be much slower than + * its baseline. A machine stall slows one repeat, but a regression slows every one. */ import type { Work } from 'latkit'; +import { quantile } from '../npm/examples/gate/quantile.ts'; import type { Listed, Run } from './page/main.ts'; -/** A phase at a size, across repeats: times their medians, work its least. */ +/** A phase at one size, summarized across repeats: its fastest times and its least work. */ export interface Summary { - /** A steady phase's CPU per frame, else its milliseconds. */ + /** Milliseconds: CPU time per frame for a steady phase, else the whole phase. */ time: number; gpu: number; steady: boolean; - /** Frames drawn, median. */ + /** Median frames drawn. */ frames: number; work: Work & { resident: number }; } @@ -32,38 +37,48 @@ export interface Verdict { failures: string[]; notes: string[]; summaries: By; - /** The exponent each measure grows by, by scenario and phase. */ + /** The fitted growth exponent of each measure, by scenario and phase. */ exponents: Record>; } -/** How far a fitted exponent may pass its order: noise, and a change of regime between sizes. */ +/** How far a fitted exponent may exceed its order, allowing for noise and a change of regime + * between sizes. */ const SLACK = 0.2; -/** Milliseconds a phase acting once may land later than its baseline by: a timer tick, which is - * 15.6 ms on Windows, so readbacks answer one tick or the next. */ +/** Milliseconds a phase that acts once may finish later than its baseline: one timer tick. A tick + * is 15.6 ms on Windows, so a readback comes back on one tick or the next. */ const TICK = 16; -/** Bytes a changed row uploads at least: its page, 65,536 rows of four bytes. */ -const PAGE = 65536 * 4; -/** Below these a cost counts as these: too small to fit an exponent to, or to call slower. A frame's - * CPU time is read on a page's clock, which steps by 0.1 ms, so it counts from five steps; a phase - * acting once lands on a tick. */ +/** Minimum bytes an upload counts as: one page, 4,096 rows of four bytes. */ +const PAGE = 4096 * 4; +/** Floors for costs too small to fit an exponent to or to call slower. Smaller costs count as + * these. A frame's CPU time comes from the page's clock, which steps by 0.1 ms, so its floor is + * five steps. A phase that acts once finishes on a timer tick. */ const FLOOR = { frame: 0.5, action: TICK, bytes: 64 * 1024 }; -/** How much slower than the baseline a median may be. */ +/** Factor by which the fastest repeat may be slower than the baseline. */ const SLOWER = 1.25; -/** Work the data and the frames drawn alone set, which repeats exactly once a phase settles. */ -const EXACT = new Set(['uploaded', 'evaluated', 'jobs', 'pipelines', 'late', 'resident']); -/** How much more of the rest a phase may do than the baseline: the frame a job or a fetch lands in - * moves label placements and tiles by up to about 1.7×, and a per-frame blowup by far more. */ +/** Work counts that only the data and the frames drawn set. They repeat exactly once a phase + * settles. */ +const EXACT = new Set([ + 'uploaded', + 'uploads', + 'evaluated', + 'packed', + 'jobs', + 'pipelines', + 'late', + 'resident', +]); +/** Factor by which other work counts may exceed the baseline. The frame a job or fetch finishes in + * changes the label texts set and the tiles by up to about 1.7×. A per-frame blowup changes far + * more. */ const STEADY = 2; -/** Below these, work counts as these when it is held within `STEADY`: bytes, and other counts. */ +/** Minimum baseline values for work held within `STEADY`: one for byte counts, one for the rest. */ const WORK_FLOOR = { bytes: 64 * 1024, count: 8 }; const BYTES = new Set(['uploaded', 'written', 'resident']); -const median = (xs: number[]) => - [...xs].sort((a, b) => a - b)[Math.floor((xs.length - 1) / 2)] ?? 0; const sum = (xs: number[]) => xs.reduce((a, b) => a + b, 0); const mib = (n: number) => `${(n / 2 ** 20).toFixed(1)} MiB`; -/** The least-squares slope of log cost against log size: the exponent the cost grows by. */ +/** The least-squares slope of log cost against log size, which is the cost's growth exponent. */ function slope(points: (readonly [number, number])[]): number { const xs = points.map(([n]) => Math.log(n)); const ys = points.map(([, c]) => Math.log(c)); @@ -78,12 +93,13 @@ function summarize(runs: Run[]): By { const { scenario, size, phases } = group[0]!; for (const [phase, { steady }] of Object.entries(phases)) { const costs = group.map((r) => r.phases[phase]!); + const frames = costs.map((c) => c.frames); const work = costs.map((c) => ({ ...c.work, resident: c.resident })); ((out[scenario] ??= {})[size] ??= {})[phase] = { steady, - time: median(costs.map((c) => (steady ? c.cpu : c.ms))), - gpu: median(costs.map((c) => c.gpu)), - frames: median(costs.map((c) => c.frames)), + time: Math.min(...costs.map((c) => (steady ? c.cpu : c.ms))), + gpu: Math.min(...costs.map((c) => c.gpu)), + frames: quantile(frames, 0.5), work: Object.fromEntries( Object.keys(work[0]!).map((k) => [ k, @@ -110,7 +126,7 @@ export function judge( for (const [phase, { work }] of Object.entries(r.phases)) if (work.late) failures.push( - `${at}, ${phase}: ${work.late} pipelines compiled where first used, not asked for ahead`, + `${at}, ${phase}: ${work.late} pipelines compiled at first use instead of ahead of time`, ); const resident = Math.max(...Object.values(r.phases).map((c) => c.resident)); if (resident > r.budget) @@ -136,8 +152,8 @@ export function judge( const fit: Growth = { time: grows((s) => s.time, steady ? FLOOR.frame : FLOOR.action), gpu: grows((s) => s.gpu, FLOOR.frame), - // Pages are what changes upload: below a page a frame drawn, bytes count as that. - uploaded: grows((s) => Math.max(s.work.uploaded, s.frames * PAGE), FLOOR.bytes), + // Changes upload whole pages, so each upload counts as at least a page of bytes. + uploaded: grows((s) => Math.max(s.work.uploaded, s.work.uploads * PAGE), FLOOR.bytes), resident: grows((s) => s.work.resident, FLOOR.bytes), }; (exponents[scenario.name] ??= {})[phase] = fit; @@ -149,8 +165,8 @@ export function judge( }; past('time', fit.time, order.time); past('uploaded bytes', fit.uploaded, order.bytes); - // What stays resident is the data's footprint, set by loading; later phases only add to it. - if (phase === 'load') past('resident bytes', fit.resident, order.bytes); + // Resident bytes are the data's footprint, which loading sets. Later phases only add to it. + if (phase === 'load') past('resident bytes', fit.resident, order.resident ?? order.bytes); for (const n of at) { const now = bySize[n]![phase]!; const where = `${scenario.name} at ${n}, ${phase}`; @@ -161,7 +177,7 @@ export function judge( const exact = !now.steady && !scenario.fetches && EXACT.has(k); const most = exact ? was : Math.max(was, floor) * STEADY; if (is > most) failures.push(`${where}: ${k} ${is}, was ${was}`); - else if (is < was) notes.push(`${where}: ${k} ${is}, was ${was}; --update keeps it`); + else if (is < was) notes.push(`${where}: ${k} ${is}, was ${was}; --update records it`); } const then = sameMachine ? base.times!.times[scenario.name]?.[n]?.[phase] : undefined; const floor = now.steady ? FLOOR.frame : TICK; @@ -169,7 +185,7 @@ export function judge( failures.push(`${where}: ${now.time.toFixed(2)} ms, was ${then.time.toFixed(2)}`); else if (then && now.time < then.time / SLOWER) notes.push( - `${where}: ${now.time.toFixed(2)} ms, was ${then.time.toFixed(2)}; --update keeps it`, + `${where}: ${now.time.toFixed(2)} ms, was ${then.time.toFixed(2)}; --update records it`, ); } } @@ -190,8 +206,8 @@ const map = (v: Verdict, read: (s: Summary) => T): By => ]), ); -/** `run` laid over `base`, scenario by size: each size this run measured replaces the base's, and - * the rest stays. */ +/** Merges `run` into `base`. Each scenario size this run measured replaces the base's entry, and + * the rest is kept. */ function over(run: By, base: By = {}): By { const out = structuredClone(base); for (const [scenario, bySize] of Object.entries(run)) @@ -199,13 +215,14 @@ function over(run: By, base: By = {}): By { return out; } -/** The work baseline: this run's laid over `base`. */ +/** The new work baseline: this run's work merged into `base`. */ export function work(v: Verdict, base?: WorkBaseline): WorkBaseline { const measured = map(v, (s) => s.work); return over(measured, base); } -/** The times baseline of `machine`: this run's laid over `base`, where `base` was taken on it. */ +/** The new times baseline for `machine`: this run's times merged into `base` if `base` was taken + * on the same machine, else this run's times alone. */ export function times(v: Verdict, machine: string, base?: TimeBaseline): TimeBaseline { const measured = map(v, (s) => ({ time: s.time, gpu: s.gpu })); return { machine, times: over(measured, base?.machine === machine ? base.times : undefined) }; diff --git a/benchmark/page/data.ts b/benchmark/page/data.ts index b0326817..895cefa9 100644 --- a/benchmark/page/data.ts +++ b/benchmark/page/data.ts @@ -1,22 +1,18 @@ /** - * What the scenarios draw, made before anything is measured: deterministic, so every run counts the - * same work, and linear in size. Each returns a `Write` that only hands latkit the arrays. + * The scenarios' data, made before anything is measured. It is deterministic, so every run counts + * the same work, and linear in size. Each function returns a `Write` that only hands latkit the + * arrays. */ import { reference, type Values } from 'latkit'; +import { congruential } from '../../npm/examples/src/random.ts'; import type { Write } from './stage.ts'; -/** Axis coordinates a second. */ +/** Samples per second on the time axis. */ const RATE = 60; -/** Numbers in [0, 1), the same every run. */ -function random(seed: number): () => number { - let s = seed >>> 0 || 1; - return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 2 ** 32; -} - /** `n` points in the unit square, each with a value, and a name when `named`. */ export function scatter(n: number, seed = 1, named = false): Write { - const next = random(seed); + const next = congruential(seed); const [x, y, v] = [new Float64Array(n), new Float64Array(n), new Float32Array(n)]; for (let i = 0; i < n; i++) [x[i], y[i], v[i]] = [next(), next(), next()]; const fields: Record = { x, y, v }; @@ -24,8 +20,8 @@ export function scatter(n: number, seed = 1, named = false): Write { return (p) => p.set('Point', { count: n }, fields); } -/** About `n` buses in a square lattice over most of the globe, each joined to its right and lower - * neighbours: a grid's shape at any size. */ +/** About `n` buses in a square lattice over most of the globe, each joined by a line to its + * neighbours in the next column and the next row. It has a grid's shape at any size. */ export function lattice(n: number): Write { const side = Math.max(2, Math.round(Math.sqrt(n))); const count = side * side; @@ -58,7 +54,8 @@ export function lattice(n: number): Write { .set('Line', { count: lines }, { from: reference('Bus', from), to: reference('Bus', to) }); } -/** `n` edges among 16 vertices on a ring, each joining a vertex to the next: lanes by the thousand. */ +/** `n` edges among 16 vertices on a ring, each joining a vertex to the next, so each pair has about + * `n / 16` parallel edges drawn as lanes. */ export function lanes(n: number): Write { const k = 16; const x = Float64Array.from({ length: k }, (_, i) => Math.cos((2 * Math.PI * i) / k)); @@ -71,7 +68,8 @@ export function lanes(n: number): Write { .set('Link', { count: n }, { from: reference('Point', from), to: reference('Point', to) }); } -/** `n` regular 32-gons in a grid over the mercator world, each with a value. */ +/** `n` regular polygons of `sides` sides (32 by default) in a grid over the Mercator world, each + * with a value. */ export function polygons(n: number, sides = 32): Write { const across = Math.ceil(Math.sqrt(2 * n)); const [w, h] = [340 / across, 140 / Math.ceil(n / across)]; @@ -90,7 +88,8 @@ export function polygons(n: number, sides = 32): Write { return (p) => p.set('Region', { count: n }, { rings, v }); } -/** `n` blocks in trees of 64: each block's output is a net, which its two children's inputs name. */ +/** `n` blocks in binary trees of 64. Each block's output is a net, and its two children's inputs + * reference that net. */ export function blocks(n: number): Write { const input = new Array(n); const name = new Array(n); @@ -111,7 +110,7 @@ export function blocks(n: number): Write { const wave = (r: number, f: number) => Math.sin(r * 0.37 + f * 0.05) + (r % 10) * 0.2; -/** `rows` traces of `frames` samples at 60 a second, each its own wave. */ +/** `rows` traces of `frames` samples at `RATE` samples per second, each a different wave. */ export function traces(rows: number, frames: number): Write { const all = new Float32Array(rows * frames); for (let f = 0; f < frames; f++) for (let r = 0; r < rows; r++) all[f * rows + r] = wave(r, f); @@ -119,16 +118,17 @@ export function traces(rows: number, frames: number): Write { return (p) => { p.axis('time', 's').set('Signal', { count: rows }, { k }); for (let f = 0; f < frames; f++) - p.append(f / RATE, 'Signal', { v: all.subarray(f * rows, (f + 1) * rows) }); + p.append('Signal', f / RATE, { v: all.subarray(f * rows, (f + 1) * rows) }); return p; }; } -/** `count` patches recording the frames from `from` on, each forgetting what falls `history` behind. */ +/** `count` patches, each appending the next sample from frame `from` on and forgetting samples more + * than `history` frames behind. */ export function feed(rows: number, from: number, count: number, history: number): Write[] { return Array.from({ length: count }, (_, i): Write => { const f = from + i; const v = Float32Array.from({ length: rows }, (_, r) => wave(r, f)); - return (p) => p.append(f / RATE, 'Signal', { v }).forget((f - history) / RATE); + return (p) => p.append('Signal', f / RATE, { v }).forget((f - history) / RATE); }); } diff --git a/benchmark/page/main.ts b/benchmark/page/main.ts index 85e3d2da..62d9416f 100644 --- a/benchmark/page/main.ts +++ b/benchmark/page/main.ts @@ -1,11 +1,14 @@ /** - * The bench page, one implementation of the contract the runner drives: `bench.ready()` resolves - * once a GPU device can be made; `bench.list()` names the scenarios and the orders their phases - * keep; `bench.run(name, size)` builds one at a size on a fresh GPU and measures it; `bench.lost()` - * says whether that GPU's device was lost. The runner opens a fresh page for every run, so nothing - * carries over but the browser's shader cache. + * The bench page. It implements the interface the runner drives: + * + * - `bench.ready()` resolves once a GPU device can be made. + * - `bench.list()` lists the scenarios and the growth orders of their phases. + * - `bench.run(name, size)` builds a scenario at a size on a fresh GPU and measures it. + * - `bench.lost()` returns whether that GPU's device was lost. + * + * The runner opens a fresh page for every run, so only the browser's shader cache carries over. */ -import { loadLatkit, type Gpu } from 'latkit'; +import { Gpu } from 'latkit'; import { LOAD, scenarios, type Order } from './scenarios.ts'; import { Stage, type Cost } from './stage.ts'; @@ -22,30 +25,34 @@ export interface Listed { export interface Run { scenario: string; size: number; - /** The adapter and browser: times compare only on the same machine. */ + /** The adapter and browser. Times compare only on the same machine. */ machine: string; /** Every phase in order: `init`, `data`, `load`, then the scenario's. */ phases: Record; - /** Frames drawn over frames stepped once everything settled: none, for views that sleep. */ + /** Frames drawn in the idle frames stepped after everything settled. Views that sleep draw + * none. */ idle: number; - /** Bytes the budget lets data take. */ + /** The data budget, in bytes. */ budget: number; - /** What the views told went wrong. */ + /** Errors the views reported. */ errors: string[]; } -/** What data may take unless a scenario sets it: fixed, so the work counted is every device's. */ +/** The GPU budget unless a scenario sets one. It is fixed, not the device's default, so the work + * counted is the same on every device. */ const BUDGET = { bytes: 256 * 2 ** 20, uploadPerFrame: 32 * 2 ** 20 }; -/** The GPU of the run under way, which `lost` asks after. */ +/** The GPU of the current run, which `lost` checks. */ let current: Gpu | undefined; -/** Resolves once a GPU device can be made: a wedged driver hangs here, so the runner asks first. */ +/** Resolves once a GPU device can be made. A wedged driver hangs here, so the runner calls this + * first. */ async function ready(): Promise { - (await (await loadLatkit()).Gpu.create()).destroy(); + (await Gpu.create()).destroy(); } -/** Whether the run's device was lost: the runner asks after a failure, and stops the sweep if so. */ +/** Whether the run's device was lost. The runner checks after a failure, and stops the sweep if + * so. */ function lost(): boolean { return current?.lost ?? false; } @@ -72,14 +79,13 @@ async function run(name: string, size: number): Promise { if (!scenario) throw new Error(`no scenario ${name}`); console.debug('phase init'); const begun = performance.now(); - const latkit = await loadLatkit(); - const gpu = (current = await latkit.Gpu.create()); + const gpu = (current = await Gpu.create()); const ms = performance.now() - begun; gpu.budget = scenario.budget ?? BUDGET; const built = scenario.build(size); - const stage = new Stage(latkit, gpu, await canvases(scenario.views?.(size) ?? 1), size); + const stage = new Stage(gpu, await canvases(scenario.views?.(size) ?? 1), size); const none = { frames: 0, cpu: 0, cpu95: 0, gpu: 0, resident: 0, steady: false }; - // Each phase says when it starts, so a page that stops answering is known by where it stopped. + // Each phase announces its start, so a page that stops responding shows where it stopped. const named = (phase: string, run: () => Promise) => { console.debug(`phase ${phase}`); return run().catch((error: unknown) => { @@ -105,7 +111,8 @@ async function run(name: string, size: number): Promise { }; } -/** `n` canvases filling the page in a grid, laid out before any view draws. */ +/** Adds `n` canvases filling the page in a grid, and waits a frame so they are laid out before any + * view draws. */ async function canvases(n: number): Promise { const main = document.querySelector('main')!; main.style.gridTemplateColumns = `repeat(${Math.ceil(Math.sqrt(n))}, minmax(0, 1fr))`; @@ -114,7 +121,7 @@ async function canvases(n: number): Promise { return made; } -/** The adapter, asked for as latkit asks for its own, and the browser. */ +/** Describes the adapter, requested with the options latkit uses, and the Chrome version. */ async function machine(): Promise { interface Adapter { info?: { vendor?: string; architecture?: string; description?: string }; diff --git a/benchmark/page/scenarios.ts b/benchmark/page/scenarios.ts index 8443d8e8..fb6653bb 100644 --- a/benchmark/page/scenarios.ts +++ b/benchmark/page/scenarios.ts @@ -1,52 +1,71 @@ /** - * Every scenario: what it sweeps, the scene and data at each size, and what it does. Each phase - * declares how its cost may grow with size, so the sweep checks the README's rules: CPU per frame - * stays flat in the rows, a one-row change costs the same at any size, restyling moves no data, - * views share pages, and loading grows no faster than the data. `init`, `data`, and `load` come - * first in every scenario, and idle frames are counted after the last. + * The benchmark scenarios: the sizes each sweeps, its scene and data at each size, and its phases. + * Each phase declares how its cost may grow with size. Together the phases check that: + * + * - CPU time per frame stays flat as rows grow; + * - a one-row change costs the same at any size; + * - restyling uploads no data; + * - views share pages; + * - returning the camera to where it was packs, uploads, and evaluates nothing; + * - appending never thins what draws; + * - loading grows no faster than the data. + * + * Every scenario starts with `init`, `data`, and `load`, and idle frames are counted after its last + * phase. */ -import { field, type Budget, type Camera, type ColormapName, type Scene, type Space } from 'latkit'; +import { + Shot, + field, + type Budget, + type Camera, + type ColormapName, + type Scene, + type Space, +} from 'latkit'; import * as make from './data.ts'; -import { deadline, type Cost, type Stage, type Write } from './stage.ts'; +import type { Cost, Stage, Write } from './stage.ts'; export interface Scenario { - /** What it stresses. */ + /** What the scenario stresses. */ about: string; - /** What a size counts, and the sizes swept, smallest first; `--quick` runs the smallest. */ + /** What a size counts, such as rows or buses. */ unit: string; + /** The sizes swept, smallest first. `--quick` runs only the smallest. */ sizes: readonly number[]; build(size: number): Built; phases: readonly Phase[]; - /** How `load` may grow; with the data, in time and bytes, unless set. */ + /** How `load` may grow. Defaults to `LOAD`: linear in time and bytes, like the data. */ load?: Order; - /** Views of one GPU drawing the scene; one unless set. */ + /** Number of views on one GPU drawing the scene. Defaults to one. */ views?(size: number): number; - /** The GPU's budget, where the scenario is about it. */ + /** The GPU budget, for a scenario that needs its own. */ budget?: Budget; - /** Run in a fresh browser without a shader cache: what a first visit costs. */ + /** Run in a fresh browser without a shader cache, to measure a first visit. */ cold?: boolean; - /** Its data is fetched as the camera asks, so what it uploads moves with the frame a fetch lands in. */ + /** Data is fetched as the camera needs it, so uploads vary with the frame a fetch arrives in. */ fetches?: boolean; } export interface Built { scene: Scene; - /** Each dataset, made beforehand, so `data` times latkit alone. */ + /** A writer per dataset, with arrays made beforehand so `data` times latkit alone. */ data: Record; - /** Vector tiles by dataset name: URL templates the bench server answers. */ + /** Vector tile URL templates by dataset name, served by the bench server. */ tiles?: Record; - /** Where the view looks; it fits everything unless set. */ + /** The starting camera. The view fits everything if it is not set. */ camera?: Camera; } /** - * How a phase's cost may grow with size: 0 not at all, 1 no faster than linearly. `time` is a steady - * phase's CPU per frame or an action's milliseconds; `bytes` is what it uploaded and what stays - * resident. A measure left out is held only to the baselines. + * How a phase's cost may grow with size: 0 means not at all, 1 means no faster than linearly. + * `time` is a steady phase's CPU time per frame, or a once phase's milliseconds. `bytes` limits the + * bytes uploaded, and for `load` also the bytes resident unless `resident` is set. A measure left + * out is checked only against the baselines. */ export interface Order { time?: 0 | 1; bytes?: 0 | 1; + resident?: 0 | 1; } export interface Phase { @@ -57,7 +76,7 @@ export interface Phase { export const LOAD: Order = { time: 1, bytes: 1 }; export const DECADES = [1e3, 1e4, 1e5, 1e6, 2e6]; -/** Frames a steady phase acts in: past its warm-up, enough for a steady median, and no more. */ +/** Frames a steady phase acts in: enough past its warm-up for a stable median, and no more. */ export const FRAMES = 40; export const HISTORY = 1024; type MapCamera = Extract; @@ -70,12 +89,10 @@ const dots = (space: Space, colormap: ColormapName = 'viridis', label?: string): ease: 0, marks: [ { - vertices: { - of: ['d', 'Point'], - position: ['x', 'y'], - color: field('v').colormap(colormap).domain(0, 1), - ...(label ? { label: field(label) } : {}), - }, + vertices: ['d', 'Point'], + position: ['$x', '$y'], + color: field('v').colormap(colormap).domain(0, 1), + ...(label ? { label: field(label) } : {}), }, ], }); @@ -85,13 +102,11 @@ const grid = (space: Space): Scene => ({ space, ease: 0, marks: [ - { edges: { of: ['g', 'Line'], ends: ['from', 'to'], width: 0.6 } }, + { edges: ['g', 'Line'], ends: ['from', 'to'], width: 0.6 }, { - vertices: { - of: ['g', 'Bus'], - position: ['lon', 'lat'], - color: field('load').colormap('viridis').domain(0, 1), - }, + vertices: ['g', 'Bus'], + position: ['$lon', '$lat'], + color: field('load').colormap('viridis').domain(0, 1), }, ], }); @@ -99,32 +114,31 @@ const grid = (space: Space): Scene => ({ const plot: Scene = { space: 'plot', ease: 0, - marks: [{ lines: { of: ['t', 'Signal'], trace: 'v' } }], + marks: [{ lines: ['t', 'Signal'], trace: 'v' }], }; const sheet: Scene = { space: 'diagram', ease: 0, - layout: { layered: { flow: 'right' } }, + layout: { layered: { direction: 'right' } }, marks: [ - { edges: { of: ['d', 'Net'], route: 'orthogonal' } }, + { edges: ['d', 'Net'], route: 'orthogonal' }, { - vertices: { - of: ['d', 'Block'], - marker: 'block', - label: { text: 'name' }, - ports: [ - ['in', 'left'], - ['out', { side: 'right', output: true }], - ], - }, + vertices: ['d', 'Block'], + marker: 'block', + label: { text: '$name' }, + ports: [ + ['in', 'left'], + ['out', { side: 'right', output: true }], + ], }, ], }; // Phases. -/** Pans and zooms gently about where the views looked, a frame at a time, back to where it began. */ +/** Pans and zooms gently around the starting camera, a step each frame, looping back to the + * start. */ const pan = (order: Order = { time: 0 }): Phase => ({ name: 'pan', order, @@ -143,7 +157,8 @@ const pan = (order: Order = { time: 0 }): Phase => ({ }, }); -/** Zooms two levels in and back: labels give way, tiles refine. */ +/** Zooms two levels in and back out, which asks for the texts of labels coming into view and + * refines tiles. */ const zoom = (): Phase => ({ name: 'zoom', order: { time: 0 }, @@ -156,14 +171,30 @@ const zoom = (): Phase => ({ }, }); -/** Recolors through `scene`'s colormap: the GPU evaluates again; no data moves. */ +/** Zooms two levels in, then back to the start. Only the way back is measured. It draws pages that + * are still resident, so it packs, uploads, and evaluates nothing. */ +const revisit = (): Phase => ({ + name: 'revisit', + order: { time: 0, bytes: 0 }, + run: async (s) => { + const start = s.view.camera as MapCamera; + await s.once(() => { + for (const v of s.views) v.camera = { ...start, zoom: start.zoom + 2 }; + }); + return s.once(() => { + for (const v of s.views) v.camera = start; + }); + }, +}); + +/** Recolors through `scene`'s colormap. The GPU evaluates again, but no data is uploaded. */ const restyle = (scene: Scene): Phase => ({ name: 'restyle', order: { bytes: 0 }, - run: (s) => s.once(() => s.show(scene)), + run: (s) => s.once(() => (s.scene = scene)), }); -/** Changes one row's value every frame: what one row costs at any size. */ +/** Changes one row's value every frame, to measure what one row costs at any size. */ const one = (data: string, table: string, name: string): Phase => ({ name: 'one row', order: { time: 0, bytes: 0 }, @@ -171,6 +202,31 @@ const one = (data: string, table: string, name: string): Phase => ({ s.steady(FRAMES, (i) => s.apply(data, (p) => p.set(table, [i], { [name]: [(i % 10) / 10] }))), }); +/** Appends a row to `table` every frame, like a feed of new rows. Fails if any view draws fewer + * rows than at the start. */ +const append = ( + data: string, + table: string, + row: (i: number) => Record, +): Phase => ({ + name: 'append', + order: { time: 0 }, + run: async (s) => { + const n = s.count(data, table); + const drawn = () => s.views.map((v) => v.stats.drawn); + const start = drawn(); + let thinned = ''; + const cost = await s.steady(FRAMES, (i) => { + const now = drawn(); + if (!thinned && now.some((d, v) => d < start[v]!)) + thinned = `${now.join(', ')} drawn, ${start.join(', ')} before`; + s.apply(data, (p) => p.set(table, [n + i], row(i))); + }); + if (thinned) throw new Error(`appending reduced the rows drawn: ${thinned}`); + return cost; + }, +}); + /** Changes every row's value at once. */ const every = (data: string, table: string, name: string): Phase => ({ name: 'every row', @@ -182,27 +238,30 @@ const every = (data: string, table: string, name: string): Phase => ({ }, }); -/** Asks what lies under the middle, as a click does: how long the answer takes. The first ask in a - * page makes the query's pipelines; later asks reuse them. */ +/** Picks what lies under the middle, as a click does, and measures how long the answer takes. The + * first pick in a page creates the query's pipelines, and later picks reuse them. */ const pick = (): Phase[] => [ { name: 'first pick', order: { time: 0, bytes: 0 }, - run: (s) => s.once(() => s.view.pick(s.middle, 6)), + run: (s) => s.once(() => s.view.pick(s.middle, { radius: 6 })), }, { name: 'pick', order: { time: 0, bytes: 0 }, - run: (s) => s.once(() => s.view.pick(s.middle, 6)), + run: (s) => s.once(() => s.view.pick(s.middle, { radius: 6 })), }, ]; -/** Moves the pointer onto a row near the middle and waits for the view to name it. */ +/** Moves the pointer onto a row near the middle and waits for the view's `hovered` event to report + * it. */ const hover = (): Phase => ({ name: 'hover', order: { time: 0, bytes: 0 }, run: async (s) => { - const [hit] = await deadline(s.until(s.view.pick(s.middle, 40)), 10_000, 'a row to hover'); + const [hit] = await s.until( + s.view.pick(s.middle, { radius: 40, signal: AbortSignal.timeout(10_000) }), + ); if (!hit) { const seen = { middle: s.middle, camera: s.view.camera, stats: s.view.stats }; throw new Error(`nothing near the middle to hover: ${JSON.stringify(seen)}`); @@ -212,7 +271,7 @@ const hover = (): Phase => ({ return await s.once( () => new Promise((resolve) => { - const stop = s.view.on('hover', ({ hit: h }) => { + const stop = s.view.on('hovered', ({ hit: h }) => { told.push(h?.row ?? null); if (!h) return; stop(); @@ -234,15 +293,18 @@ const hover = (): Phase => ({ }, }); -/** Selects a hundred rows: the glow, and what dims; a table drawn through tiles keeps them drawn. */ +/** Selects 100 rows, which glow while the rest dim. A paged table keeps the selected rows drawn. */ const select = (data: string, table: string): Phase => ({ name: 'select', order: { time: 0 }, run: (s) => - s.once(() => s.view.select([{ data, table, ids: Array.from({ length: 100 }, (_, id) => id) }])), + s.once( + () => (s.view.selection = [{ data, table, ids: Array.from({ length: 100 }, (_, id) => id) }]), + ), }); -/** Moves every row to a new place at once, easing: CPU per frame while they fly. */ +/** Moves every row to a new position at once, easing, and measures CPU time per frame while they + * move. */ const move = (): Phase => ({ name: 'move', order: { time: 0, bytes: 1 }, @@ -252,7 +314,8 @@ const move = (): Phase => ({ }, }); -/** Records a frame of every trace each animation frame, as a live feed does, forgetting the oldest. */ +/** Appends a sample to every trace each frame, like a live feed, and forgets samples older than + * `HISTORY` frames. */ const stream = (): Phase => ({ name: 'stream', order: { time: 1, bytes: 1 }, @@ -262,7 +325,8 @@ const stream = (): Phase => ({ }, }); -/** Moves the coordinate shown across what was recorded: the playhead, and colors read at it. */ +/** Moves the views' `at` coordinate across the recorded span, which moves the playhead and the + * colors read at it. */ const scrub = (): Phase => ({ name: 'scrub', order: { time: 0, bytes: 0 }, @@ -274,8 +338,8 @@ const scrub = (): Phase => ({ }, }); -/** Narrows the window shown from everything to its last tenth: levels of detail give way, each - * trace's finer one uploading as it first draws. */ +/** Narrows the visible window from the whole span to its last tenth. Each trace switches to finer + * levels of detail, uploading each level as it first draws. */ const narrow = (order: Order = { time: 0, bytes: 0 }): Phase => ({ name: 'narrow', order, @@ -287,7 +351,8 @@ const narrow = (order: Order = { time: 0, bytes: 0 }): Phase => ({ }, }); -/** Renames a block every frame, so it resizes and its tree routes again: an edit at any size. */ +/** Renames a block every frame, which resizes it and routes its tree again, to measure an edit at + * any size. */ const edit = (): Phase => ({ name: 'edit', order: { time: 0, bytes: 0 }, @@ -304,41 +369,43 @@ const arrange = (): Phase => ({ run: (s) => s.once(() => s.view.arrange()), }); -/** A shot of every view side by side, the scenario's size in pixels, at 16:9. */ +/** A shot of every view side by side, at 16:9, with about the scenario's size in pixels. */ function shot(s: Stage) { const width = Math.round(Math.sqrt((s.size * 16) / 9) / 4) * 4; const height = Math.round((width * 9) / 16 / 2) * 2; const n = s.views.length; - return new s.latkit.Shot([width, height]).panels(s.views.map((_, i) => [i / n, 0, 1 / n, 1])); + const views = s.views.map((view, i) => ({ view, region: [i / n, 0, 1 / n, 1] as const })); + return new Shot([width, height], { views }); } const image = (): Phase => ({ name: 'image', order: { time: 1 }, - run: (s) => s.once(() => shot(s).image(s.views)), + run: (s) => s.once(() => shot(s).image()), }); const png = (): Phase => ({ name: 'png', order: { time: 1 }, run: async (s) => { - const picture = await deadline(shot(s).image(s.views), 60_000, 'a picture'); + const picture = await shot(s).image({ signal: AbortSignal.timeout(60_000) }); return s.once(() => picture.encode('png')); }, }); -/** Thirty frames of video, turning; held, nothing moves, so frames after the first cost nothing. */ +/** Records 30 frames of video while the camera turns. Unless `moving`, the camera holds still, so + * frames after the first need no new drawing. */ const video = (moving: boolean): Phase => ({ name: moving ? 'video' : 'held', order: moving ? { time: 1 } : {}, run: (s) => { const start = s.view.camera as MapCamera; return s.once(async () => { - const video = shot(s).video(s.gpu, 30, new WritableStream()); + const video = shot(s).video({ fps: 30, output: new WritableStream() }); for (let i = 0; i < 30; i++) { if (moving) for (const v of s.views) v.camera = { ...start, bearing: start.bearing + 2 * i }; - await video.frame(s.views); + await video.frame(); } await video.finish(); }); @@ -364,12 +431,18 @@ export const scenarios: Record = { phases: [ pan(), zoom(), + revisit(), restyle(dots('plane', 'magma')), one('d', 'Point', 'v'), every('d', 'Point', 'v'), ...pick(), hover(), select('d', 'Point'), + append('d', 'Point', (i) => ({ + x: [((i * 37) % 100) / 100], + y: [((i * 53) % 100) / 100], + v: [0.5], + })), ], }, grid: { @@ -380,13 +453,25 @@ export const scenarios: Record = { scene: grid({ earth: { projection: 'globe' } }), data: { g: make.lattice(n) }, }), - phases: [pan(), zoom(), one('g', 'Bus', 'load'), ...pick(), hover()], + phases: [ + pan(), + zoom(), + revisit(), + one('g', 'Bus', 'load'), + ...pick(), + hover(), + append('g', 'Bus', (i) => ({ + lon: [-170 + (340 * ((i * 37) % 100)) / 100], + lat: [-60 + (120 * ((i * 53) % 100)) / 100], + load: [0.5], + })), + ], }, labels: { - about: 'every vertex labeled: placement, collisions, and the texts set', + about: 'every vertex labeled: the texts in view found, set, and drawn', unit: 'rows', - // From ten thousand rows on, more labels show than a frame sets texts for, so a frame's CPU - // holds at the text budget however many rows are labeled. + // From 10,000 rows on, more labels show than a frame sets texts for, so a frame's CPU time + // stays at the text budget however many rows are labeled. sizes: [1e4, 1e5, 1e6], build: (n) => ({ scene: dots('plane', 'viridis', 'name'), @@ -403,8 +488,8 @@ export const scenarios: Record = { space: 'plane', ease: 0, marks: [ - { edges: { of: ['d', 'Link'], ends: ['from', 'to'] } }, - { vertices: { of: ['d', 'Point'], position: ['x', 'y'] } }, + { edges: ['d', 'Link'], ends: ['from', 'to'] }, + { vertices: ['d', 'Point'], position: ['$x', '$y'] }, ], }, data: { d: make.lanes(n) }, @@ -417,15 +502,13 @@ export const scenarios: Record = { sizes: [1e2, 1e3, 1e4, 3e4], build: (n) => ({ scene: { - space: 'mercator', + space: 'earth', ease: 0, marks: [ { - areas: { - of: ['m', 'Region'], - geometry: 'rings', - color: field('v').colormap('viridis').domain(0, 1), - }, + areas: ['m', 'Region'], + geometry: 'rings', + color: field('v').colormap('viridis').domain(0, 1), }, ], }, @@ -434,15 +517,15 @@ export const scenarios: Record = { phases: [pan(), zoom(), one('m', 'Region', 'v'), ...pick()], }, tiles: { - about: 'vector tiles fetched as the camera asks for them, from a local server', + about: 'vector tiles fetched from a local server as the camera needs them', unit: 'squares a tile', sizes: [16, 256, 4096], fetches: true, build: (n) => ({ scene: { - space: 'mercator', + space: 'earth', ease: 0, - marks: [{ areas: { of: ['t', 'cells'], color: '#4a7fb5' } }], + marks: [{ areas: ['t', 'cells'], color: '#4a7fb5' }], }, data: {}, tiles: { t: `${location.origin}/tiles/${n}/{z}/{x}/{y}.mvt` }, @@ -462,7 +545,7 @@ export const scenarios: Record = { unit: 'traces', sizes: [10, 1e2, 1e3, 1e4], build: (n) => ({ scene: plot, data: { t: make.traces(n, HISTORY) } }), - // Every trace narrowed uploads its finer level. + // Narrowing uploads each trace's finer level, so its bytes grow with the traces. phases: [stream(), scrub(), narrow({ time: 0, bytes: 1 }), hover()], }, history: { @@ -484,15 +567,16 @@ export const scenarios: Record = { unit: 'views', sizes: [1, 2, 4, 8], views: (n) => n, - // Room for eight views' own rows, which the views' shares of a budget hold together, so the - // bytes that grow would be pages copied. + // A budget large enough that the views' shares of it fit the per-row data of eight views. + // Each view keeps its own per-row data, so resident bytes grow with the views. The pages are + // shared, so uploads do not. budget: { bytes: 1024 * 2 ** 20, uploadPerFrame: 32 * 2 ** 20 }, - load: { time: 1, bytes: 0 }, + load: { time: 1, bytes: 0, resident: 1 }, build: () => ({ scene: grid('plane'), data: { g: make.lattice(1e5) } }), phases: [pan({ time: 1, bytes: 0 })], }, budget: { - about: 'a 16 MiB budget: what draws fits, through tiles, and nothing fails', + about: 'a 16 MiB budget: paged tables fit, and nothing fails', unit: 'buses', sizes: [1e5, 1e6, 2e6], budget: { bytes: 16 * 2 ** 20, uploadPerFrame: 4 * 2 ** 20 }, diff --git a/benchmark/page/stage.ts b/benchmark/page/stage.ts index 6b9f264e..0f58996b 100644 --- a/benchmark/page/stage.ts +++ b/benchmark/page/stage.ts @@ -1,72 +1,57 @@ /** - * Where a run happens, and the one way anything in it is measured. The stage steps every frame - * itself, a sixtieth of a second apart on its own clock, and each frame waits for the GPU (its work - * done, its readbacks landed, the pipelines asked for compiled), never for time: a run draws the same - * frames however fast the device is, so what it counts repeats. A phase is an action, timed from when - * it starts until everything settled: no view wants a frame, every view is complete, and no tile is - * loading; work under way that wakes no frame is waited for as it lands. A steady phase acts before - * each of a number of frames, so each act shows in its own frame, and its CPU and GPU time are per - * frame. A lost device fails the phase at once, and every wait races a deadline. + * The stage a run happens on, and the single way anything in a run is measured. + * + * The stage steps every frame itself, 1/60 s apart on its own clock. Each frame waits for the GPU + * (its work done, its readbacks back, and the requested pipelines compiled), never for time. So a + * run draws the same frames however fast the device is, and its work counts repeat. + * + * A phase is an action, timed from its start until everything settles: no view wants a frame, + * every view is complete, and no tile is loading. Work in progress that wakes no frame is awaited + * until it finishes. A steady phase acts before each of a set number of frames, so each action + * shows in its own frame, and its CPU and GPU times are per frame. A lost device fails the phase at + * once, and every wait has a deadline that cancels the work it waits on. */ -import { - Patch, - type Camera, - type Data, - type Gpu, - type Latkit, - type Scene, - type View, - type Work, -} from 'latkit'; +import { Data, Patch, Tiles, View, type Camera, type Gpu, type Scene, type Work } from 'latkit'; +import { quantile } from '../../npm/examples/gate/quantile.ts'; /** Writes a dataset's rows into a patch, from arrays made beforehand. */ export type Write = (patch: Patch) => Patch; -/** What a phase cost. */ +/** The cost of a phase. */ export interface Cost { /** Milliseconds from the action until everything settled. */ ms: number; - /** Frames drawn meanwhile. */ + /** Frames drawn in that time. */ frames: number; - /** Milliseconds every view together took to record a frame on the CPU: median, 95th percentile. */ + /** CPU milliseconds all views together took to record a frame: the median here, and the 95th + * percentile in `cpu95`. */ cpu: number; cpu95: number; - /** Milliseconds a frame took on the GPU, median; 0 where the device cannot measure. */ + /** Median GPU milliseconds per frame, or 0 where the device cannot measure it. */ gpu: number; - /** What the GPU did until everything settled. */ + /** The GPU work done until everything settled. */ work: Work; - /** Bytes of data pages on the GPU afterwards. */ + /** Bytes resident on the GPU afterwards: data pages, and each view's per-row data. */ resident: number; - /** Whether it acted every frame, so `cpu` is its time; else `ms` is. */ + /** Whether the phase acted every frame. If so, `cpu` is its time; otherwise `ms` is. */ steady: boolean; } -/** Milliseconds between stepped frames: a display's sixtieth of a second. */ +/** Milliseconds between stepped frames: 1/60 s, as on a 60 Hz display. */ const FRAME = 1000 / 60; -/** The most frames a phase may take to settle: ten seconds of a display's frames. */ +/** Maximum frames a phase may take to settle: 10 s at 60 frames a second. */ const MOST = 600; -/** Frames stepped once everything settled, in which nothing should draw. */ +/** Frames stepped after everything settled, in which nothing should draw. */ const IDLE = 10; -/** Milliseconds a frame, or work landing, may take: past it, something hangs. */ +/** Maximum milliseconds a frame, or a wait for work to finish, may take. Longer means a hang. */ const DEADLINE = 30_000; -/** The longest a phase may take; the runner grows sizes within it. */ +/** Milliseconds a phase should take at most. The runner stops growing a scenario's size once a + * phase would exceed it. */ export const LIMIT = 10_000; -/** Frames a steady phase takes before it measures. */ +/** Frames a steady phase runs before it starts measuring. */ export const WARM = 10; -const quantile = (xs: number[], q: number) => - xs.length ? [...xs].sort((a, b) => a - b)[Math.floor(q * (xs.length - 1))]! : 0; - -/** `work`, or an error naming `what` once `ms` pass. */ -export function deadline(work: Promise, ms: number, what: string): Promise { - let timer: ReturnType | undefined; - const late = new Promise((_, reject) => { - timer = setTimeout(() => reject(new Error(`${what} took over ${ms} ms`)), ms); - }); - return Promise.race([work, late]).finally(() => clearTimeout(timer)); -} - -/** What the frames a phase stepped drew. */ +/** The frames a phase drew, and the CPU and GPU times of those measured. */ interface Drawn { frames: number; cpu: number[]; @@ -74,21 +59,22 @@ interface Drawn { } export class Stage { - readonly latkit: Latkit; readonly gpu: Gpu; readonly canvases: readonly HTMLCanvasElement[]; /** The scenario's size. */ readonly size: number; readonly views: View[] = []; - /** Errors the views told: a mark that does not fit its data fails the run. */ + /** Errors the views reported. Any error, such as a mark that does not fit its data, fails the + * run. */ readonly errors: string[] = []; - scene!: Scene; readonly #data = new Map(); - /** The stepped frames' clock, milliseconds. */ + /** The clock of the stepped frames, in milliseconds. */ #now = performance.now(); - constructor(latkit: Latkit, gpu: Gpu, canvases: readonly HTMLCanvasElement[], size: number) { - [this.latkit, this.gpu, this.canvases, this.size] = [latkit, gpu, canvases, size]; + constructor(gpu: Gpu, canvases: readonly HTMLCanvasElement[], size: number) { + [this.gpu, this.canvases, this.size] = [gpu, canvases, size]; + // Step from the start, so no animation frame draws while the views load either. + gpu.step(this.#now); } get view(): View { @@ -104,61 +90,62 @@ export class Stage { /** Units across `px` CSS pixels at the middle of the first view. */ across(px: number): number { const [x, y] = this.middle; - const [a, b] = [this.view.ground([x, y]), this.view.ground([x + px, y])]; + const [a, b] = [this.view.position([x, y]), this.view.position([x + px, y])]; return a && b ? Math.hypot(b[0] - a[0], b[1] - a[1]) : 1; } /** Rows of `table` in dataset `name`. */ count(name: string, table: string): number { - return this.#data.get(name)!.ids(table).length; + return this.#data + .get(name)! + .describe() + .tables.find((t) => t.name === table)!.rows; } - /** The axis coordinates dataset `name` recorded. */ + /** The range of axis coordinates dataset `name` recorded, or [0, 1] if none. */ span(name: string): [number, number] { - return this.#data.get(name)!.span ?? [0, 1]; + return this.#data.get(name)!.describe().span ?? [0, 1]; } - /** Takes each dataset in: latkit's work alone, the arrays made beforehand. */ + /** Applies each dataset to a new `Data`. The arrays are made beforehand, so this times latkit + * alone. */ ingest(data: Record): Promise { - const { Data } = this.latkit; return this.once(() => { for (const [name, write] of Object.entries(data)) this.#data.set(name, new Data(name).apply(write(new Patch()))); }); } - /** Opens a view on each canvas and draws `scene`: from nothing to everything drawn. */ + /** Opens a view on each canvas and draws `scene`, measuring from nothing until everything is + * drawn. */ load(scene: Scene, tiles: Record = {}, camera?: Camera): Promise { - this.scene = scene; - return this.once(() => { + const sources = [ + ...this.#data.values(), + ...Object.entries(tiles).map(([name, url]) => new Tiles(name, url)), + ]; + return this.once(async () => { for (const canvas of this.canvases) { - const view = this.gpu.view(scene); - this.gpu.canvas(canvas).show([view]); + const view = await View.create(canvas, scene, { gpu: this.gpu, sources, camera }); view.on('error', (error) => this.errors.push(`${error.path ?? ''} ${error.message}`.trim()), ); - for (const data of this.#data.values()) view.setData(data); - for (const [name, url] of Object.entries(tiles)) view.setTiles(name, url); - if (camera) view.camera = camera; - else view.fit('all'); this.views.push(view); } }); } - /** Shows `scene` in every view. */ - show(scene: Scene): void { - this.scene = scene; - for (const view of this.views) view.show(scene); + /** Draws `scene` in every view. */ + set scene(scene: Scene) { + for (const view of this.views) view.scene = scene; } - /** Writes a patch into dataset `name` and gives every view the next version. */ + /** Applies a patch to dataset `name`. Every view draws the change from the next frame. */ apply(name: string, write: Write): void { - const data = this.#data.get(name)!.apply(write(new Patch())); - for (const view of this.views) view.setData(data); + this.#data.get(name)!.apply(write(new Patch())); } - /** Sends a pointer event at `point`, CSS pixels in the first view, as a person's would arrive. */ + /** Dispatches a pointer event at `[x, y]`, in CSS pixels in the first view, as a real pointer + * would. */ pointer(type: string, [x, y]: [number, number], buttons = 0): void { const canvas = this.canvases[0]!; const box = canvas.getBoundingClientRect(); @@ -175,7 +162,7 @@ export class Stage { ); } - /** Acts once, and measures until everything settled. */ + /** Performs `act` once and measures until everything settles. */ once(act: () => unknown): Promise { return this.#phase(false, async (drawn) => { await this.until(Promise.resolve().then(act), drawn, true); @@ -183,8 +170,9 @@ export class Stage { }); } - /** `answer`, stepping frames until it comes: what waits on a frame, as a pick or a hover told - * does, gets one, and what waits on work landing is waited for. */ + /** Steps frames until `answer` settles, then returns it. A frame is stepped whenever a view wants + * one, as a pending pick or hover event does. Otherwise it waits for work to finish. Throws if + * the device is lost, or after `MOST` frames or `DEADLINE` ms. */ async until( answer: Promise, drawn: Drawn = { frames: 0, cpu: [], gpu: [] }, @@ -205,12 +193,21 @@ export class Stage { if (this.views.some((v) => v.wantsFrame)) { await this.#frame(drawn, measuring); frames++; - } else await Promise.race([coming, this.gpu.landed().catch(() => {})]); + } else { + // Waits for the answer or for work to land, then stops waiting for work, so no wait is left + // pending. + const answered = new AbortController(); + const timeout = AbortSignal.timeout(Math.max(0, late - performance.now())); + const landed = this.gpu.landed({ signal: AbortSignal.any([answered.signal, timeout]) }); + await Promise.race([coming, landed.catch(() => {})]); + answered.abort(); + } } return answer; } - /** Calls `act(i)` before each of `count` frames, and measures until everything settled. */ + /** Calls `act(i)` before each of `count` frames, and measures until everything settles. Frames + * after the first `WARM` are measured. */ steady(count: number, act: (i: number) => void): Promise { return this.#phase(true, async (drawn) => { for (let i = 0; i < count; i++) { @@ -221,7 +218,8 @@ export class Stage { }); } - /** Frames drawn over `IDLE` frames stepped once everything settled: none, for views that sleep. */ + /** Counts the frames drawn in `IDLE` frames stepped after everything settles. Views that sleep + * draw none. */ async idle(): Promise { const drawn: Drawn = { frames: 0, cpu: [], gpu: [] }; await this.#settle(drawn, false); @@ -247,18 +245,18 @@ export class Stage { cpu95: quantile(drawn.cpu, 0.95), gpu: quantile(drawn.gpu, 0.5), work, - resident: this.views[0]?.stats.resident ?? 0, + resident: this.gpu.resident, steady, }; } - /** Steps one frame, once the GPU is done with the last, and counts what the views drew in it: - * their CPU and GPU time, when `measuring`. */ + /** Steps one frame and waits for the GPU to finish it. Counts the frame if any view drew, and + * when `measuring` records the views' summed CPU and GPU times. */ async #frame(drawn: Drawn, measuring: boolean): Promise { if (this.gpu.lost) throw new Error('the GPU device was lost'); const before = this.views.map((v) => v.stats.frames); this.gpu.step((this.#now += FRAME)); - await deadline(this.gpu.done(), DEADLINE, 'a frame'); + await this.gpu.done({ signal: AbortSignal.timeout(DEADLINE) }); let [drew, cpu, gpu] = [false, 0, 0]; this.views.forEach((view, i) => { const stats = view.stats; @@ -273,11 +271,11 @@ export class Stage { } } - /** Steps frames until everything settled: no view wants a frame, every view is complete, and no - * tile is loading. Work under way that wakes no frame, as a layout or a fetch, is waited for as it - * lands. What a once phase draws is `measuring`. */ + /** Steps frames until everything settles: no view wants a frame, every view is complete, and no + * tile is loading. Work in progress that wakes no frame, such as a layout or a fetch, is awaited + * until it finishes. `measuring` is true for a once phase, whose every frame is measured. */ async #settle(drawn: Drawn, measuring: boolean): Promise { - await deadline(this.gpu.done(), DEADLINE, 'the GPU'); + await this.gpu.done({ signal: AbortSignal.timeout(DEADLINE) }); for (let frames = 0; frames < MOST;) { if (this.gpu.lost) throw new Error('the GPU device was lost'); if (this.views.some((v) => v.wantsFrame)) { @@ -285,17 +283,20 @@ export class Stage { frames++; continue; } - if (this.views.every((v) => !v.stats.waiting.length && !v.stats.tilesLoading)) return; - await deadline(this.gpu.landed(), DEADLINE, `${this.#waiting()} landing`); + if (this.views.every((v) => v.complete && !v.stats.tilesLoading)) return; + await this.gpu.landed({ signal: AbortSignal.timeout(DEADLINE) }).catch((error: unknown) => { + const message = error instanceof Error ? error.message : String(error); + throw new Error(`waiting on ${this.#waiting() || 'nothing'}: ${message}`); + }); } throw new Error( `still drawing after ${MOST} frames: waiting on ${this.#waiting() || 'nothing'}`, ); } - /** What the views wait on. */ + /** Lists what the views are waiting on, including tiles. */ #waiting(): string { const tiles = this.views.some((v) => v.stats.tilesLoading) ? ['tiles'] : []; - return [...new Set(this.views.flatMap((v) => v.stats.waiting)), ...tiles].join(', '); + return [...new Set(this.views.flatMap((v) => v.waiting)), ...tiles].join(', '); } } diff --git a/benchmark/report.ts b/benchmark/report.ts index f8fe7101..ee45fefc 100644 --- a/benchmark/report.ts +++ b/benchmark/report.ts @@ -1,6 +1,6 @@ /** - * The characterization: per scenario, a row per phase and measure, a column per size, and the - * exponent each grows by. + * Formats the results as a text table: per scenario, a row per phase and measure, a column per + * size, and each row's growth exponent. */ import type { Listed } from './page/main.ts'; import type { Summary, Verdict } from './judge.ts'; @@ -11,7 +11,7 @@ const bytes = (v: number) => const count = (n: number) => n >= 1e6 ? `${+(n / 1e6).toFixed(1)}M` : n >= 1e3 ? `${+(n / 1e3).toFixed(1)}k` : `${n}`; -/** Each scenario's phases at each size, and the exponent each grows by. */ +/** Formats each scenario's phases at each size, with each measure's growth exponent. */ export function report(list: Listed[], { summaries, exponents }: Verdict): string { const out: string[] = []; for (const scenario of list) { diff --git a/benchmark/run.ts b/benchmark/run.ts index 9dd7a05a..c38cfc0f 100644 --- a/benchmark/run.ts +++ b/benchmark/run.ts @@ -1,25 +1,35 @@ /** - * The benchmark: each scenario up its sizes, each run in a fresh page, judged and reported. + * The benchmark runner. It runs each scenario at increasing sizes, each run in a fresh page, then + * judges and reports the results. * * cargo xtask bench [--quick] [--only a,b] [--repeats n] [--update] [--software] * - * --quick runs each scenario once at its smallest size, a smoke test of every phase and the work - * gate in under a minute on a GPU; how phases grow, and their times, take the full sweep. --update lays this run's - * baselines over the committed ones, keeping what it did not run; --software draws with SwiftShader - * on the CPU, as Linux always does: its work counts the same, its times only report, and no shader - * can hang the GPU. + * `--quick` runs each scenario once at its smallest size. It checks every phase and the work + * baseline in under a minute on a GPU. Growth and times are judged only in a full sweep. + * `--repeats` sets the runs per size (3 by default). `--update` merges this run's results into the + * committed baselines, keeping entries it did not run. Only a full sweep on the GPU writes times. + * `--software` draws with SwiftShader on the CPU, as Linux always does. Work counts the same there, + * times are only reported, and no shader can hang the GPU. * - * A scenario climbs its sizes while the last one's phases, grown as their orders say, fit a phase's - * limit. A failure ends its scenario; a lost device, or a page that stopped answering, ends the - * sweep, since the next scenario would meet a resetting driver, and resets in a row are what wedge - * one. Every wait races a deadline. + * A scenario moves to its next size only while every phase's fastest repeat at the last size, + * scaled up by the phase's order, stays within `LIMIT`. The fastest repeat is used because a + * machine stall slows one repeat, while a scenario too large for the next size slows every one. A + * failure ends its scenario. A lost device or a page that stops responding ends the sweep, since + * the next scenario would meet a resetting driver, and repeated resets are what wedge a driver. + * Every wait races a deadline. */ import fs from 'node:fs'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import { parseArgs } from 'node:util'; -import { createServer } from 'vite'; -import { STEP, deadline, launch, type Chrome } from '../npm/examples/gate/browser.ts'; +import { + STEP, + deadline, + errorsOf, + launch, + serve, + type Chrome, +} from '../npm/examples/gate/browser.ts'; import { judge, times, work, type TimeBaseline, type WorkBaseline } from './judge.ts'; import { report } from './report.ts'; import type { Listed, Run } from './page/main.ts'; @@ -42,17 +52,18 @@ const software = flags.software || process.platform === 'linux'; const here = path.dirname(fileURLToPath(import.meta.url)); const baselines = path.join(here, 'baselines'); const results = path.resolve(here, '../target/bench'); -/** Timestamps, for what a frame costs the GPU. The page steps its frames, each as soon as the GPU - * finished the last, so the display's pace never enters a run. */ +/** Gives WebGPU timestamps, which measure each frame's GPU time, full precision. Chrome otherwise + * rounds them to 100 µs. The page steps each frame as soon as the GPU finishes the last, so the + * display's frame rate never affects a run. */ const FLAGS = ['--enable-webgpu-developer-features']; -/** The same page on every machine, so what it counts compares, and small enough that SwiftShader - * draws a frame quickly. */ +/** The page size. It is the same on every machine so work counts compare, and small enough that + * SwiftShader draws a frame quickly. */ const PAGE = { viewport: { width: 800, height: 500 }, deviceScaleFactor: 2 }; -/** Milliseconds a page may say nothing: each phase says when it starts, then acts and settles - * within its limits. */ +/** Maximum milliseconds between a page's phase announcements. Each phase announces its start, then + * acts and settles within its own limits. */ const QUIET = 2 * LIMIT + 20_000; -/** A failed run, and whether the sweep stops there. */ +/** A failed run. A `fatal` failure stops the sweep. */ class Failure extends Error { readonly fatal: boolean; constructor(message: string, fatal: boolean) { @@ -61,43 +72,43 @@ class Failure extends Error { } } -/** `work`, or an error once the page has said nothing for `QUIET` ms since `heard()`. */ +/** Resolves with `work`, or rejects once more than `QUIET` ms have passed since `heard()`, the time + * the page last announced a phase. */ function watched(work: Promise, heard: () => number, what: string): Promise { let timer: NodeJS.Timeout | undefined; const silent = new Promise((_, reject) => { timer = setInterval(() => { if (performance.now() - heard() > QUIET) - reject(new Error(`${what} said nothing for ${QUIET} ms`)); + reject(new Error(`${what} reported no phase for ${QUIET} ms`)); }, 1_000); }); return Promise.race([work, silent]).finally(() => clearInterval(timer)); } -/** The first phase of `last` that, grown to `size` as its order says (linearly where it says - * nothing), would pass a phase's limit. */ -function outgrown(last: Run, size: number, orders: Listed['orders']): string | undefined { - const grown = ([phase, cost]: [string, Run['phases'][string]]) => - cost.ms * (size / last.size) ** (orders[phase]?.time ?? 1); - return Object.entries(last.phases).find((entry) => grown(entry) > LIMIT)?.[0]; +/** Finds the first phase that would exceed `LIMIT` at `size`. Each phase's fastest time among the + * repeats `last`, all at one size, is scaled to `size` by its time order, linearly if it has none. + * Returns that phase and its fastest time. */ +function outgrown( + last: Run[], + size: number, + orders: Listed['orders'], +): [string, number] | undefined { + const from = last[0]!.size; + for (const phase of Object.keys(last[0]!.phases)) { + const ms = Math.min(...last.map((run) => run.phases[phase]?.ms ?? Infinity)); + if (ms * (size / from) ** (orders[phase]?.time ?? 1) > LIMIT) return [phase, ms]; + } + return undefined; } -const server = await createServer({ - root: here, - configFile: path.join(here, 'vite.config.ts'), - logLevel: 'error', - server: { port: 5230, strictPort: false }, -}); -await deadline(server.listen(), STEP, 'serve the bench'); -const address = server.resolvedUrls?.local[0]; -if (!address) throw new Error('the bench has no address'); -const url = address; +const { url, close } = await serve(here, 5230, 'the bench'); const runs: Run[] = []; const failures: string[] = []; const notes: string[] = []; let list: Listed[] = []; -/** One run of `scenario` at `size`: in a fresh page of `warm`, or, for a cold scenario, of a fresh - * Chrome without a shader cache. */ +/** Runs `scenario` once at `size` in a fresh page of `warm`. A cold scenario runs in a fresh Chrome + * without a shader cache instead. */ async function measure(warm: Chrome, scenario: Listed, size: number): Promise { if (!scenario.cold) return visit(warm, scenario, size); const cold = await launch([...FLAGS, '--disable-gpu-shader-disk-cache'], software); @@ -108,15 +119,13 @@ async function measure(warm: Chrome, scenario: Listed, size: number): Promise { const page = await deadline(chrome.browser.newPage(PAGE), STEP, 'open a page'); - const errors: string[] = []; + const errors = errorsOf(page); let [phase, heard] = ['opening', performance.now()]; - page.on('pageerror', (e) => errors.push(e.message)); page.on('console', (m) => { - if (m.type() === 'error') errors.push(m.text()); if (m.type() === 'debug' && m.text().startsWith('phase ')) [phase, heard] = [m.text().slice(6), performance.now()]; }); @@ -133,7 +142,7 @@ async function visit(chrome: Chrome, scenario: Listed, size: number): Promise window.bench.lost()), 5_000, - 'ask after the device', + 'check whether the device was lost', ).catch(() => true); throw new Failure( `${scenario.name} at ${size}, ${phase}: ${String(error).split('\n')[0]}`, @@ -148,11 +157,11 @@ const warm = await launch(FLAGS, software); try { const page = await deadline(warm.browser.newPage(PAGE), STEP, 'open a page'); await deadline(page.goto(url), STEP, 'open the bench page'); - // Making a device is where a wedged driver hangs, so the sweep starts only once one is made. + // A wedged driver hangs when a device is made, so the sweep starts only after one is made. await deadline( page.evaluate(() => window.bench.ready()), 15_000, - 'make a GPU device: a wedged driver hangs here, and needs a reset before benchmarking', + 'create a GPU device (a hung driver blocks here; reset it before benchmarking)', ); list = await deadline( page.evaluate(() => window.bench.list()), @@ -168,25 +177,28 @@ try { for (const scenario of list.filter((s) => !only || only.includes(s.name))) { const sizes = flags.quick ? scenario.sizes.slice(0, 1) : scenario.sizes; try { - // A first run fills the shader cache, so the measured ones load warm; `start` alone loads cold. + // An unmeasured first run fills the shader cache, so the measured runs load warm. Only cold + // scenarios (`start`) load cold. if (!scenario.cold && !flags.quick) await measure(warm, scenario, sizes[0]!); - let last: Run | undefined; + let last: Run[] = []; for (const size of sizes) { - const phase = last && outgrown(last, size, scenario.orders); - if (last && phase) { - const ms = last.phases[phase]!.ms.toFixed(0); + const outgrew = last.length ? outgrown(last, size, scenario.orders) : undefined; + if (outgrew) { + const [phase, ms] = outgrew; notes.push( - `${scenario.name} stops before ${size}: its ${phase} took ${ms} ms at ${last.size}`, + `${scenario.name} stops before ${size}: its ${phase} took ${ms.toFixed(0)} ms at ${last[0]!.size}`, ); break; } + last = []; for (let r = 0; r < repeats; r++) { const run = await measure(warm, scenario, size); console.log( ` ${scenario.name} at ${size}: loaded in ${run.phases.load?.ms.toFixed(0)} ms` + (run.errors.length ? `, ${run.errors.length} errors` : ''), ); - runs.push((last = run)); + runs.push(run); + last.push(run); } } } catch (error) { @@ -195,7 +207,7 @@ try { console.log(` ${failure.message.slice(0, 300)}`); if (failure.fatal) { failures.push( - 'the GPU lost its device or stopped answering, so the sweep stops: reset the driver before running again', + 'the GPU device was lost or the page stopped responding, so the sweep stopped. Reset the driver before running again', ); break; } @@ -203,7 +215,7 @@ try { } } finally { await warm.close(); - await server.close(); + await close(); } const read = (file: string): T | undefined => @@ -213,7 +225,7 @@ const write = (file: string, value: unknown) => const machines = [...new Set(runs.map((r) => r.machine))]; if (machines.length > 1) failures.push( - `runs drew on different GPUs, so their times cannot compare: ${machines.join('; ')}`, + `runs used different GPUs, so their times are not comparable: ${machines.join('; ')}`, ); if (!runs.length) failures.push('nothing ran'); const machine = machines[0] ?? 'unknown'; @@ -223,10 +235,12 @@ const slug = machine .replace(/^-|-$/g, ''); const workFile = path.join(baselines, 'work.json'); const timesFile = path.join(baselines, `${slug}.json`); -// A quick run, once each without a warm-up, holds no times to the baselines its warm repeats took. +// Times are held to a baseline, and written to one, only by a full sweep on the GPU. A quick run +// makes one run per scenario without a warm-up, and SwiftShader's times measure the CPU. +const timed = !flags.quick && !software; const verdict = judge(runs, list, { work: read(workFile), - times: flags.quick ? undefined : read(timesFile), + times: timed ? read(timesFile) : undefined, }); console.log(`\n${machine}`); console.log(report(list, verdict)); @@ -234,15 +248,15 @@ if (runs.length) { fs.mkdirSync(results, { recursive: true }); write(path.join(results, `${Date.now()}.json`), { machine, runs }); } -// Baselines come only from runs that all completed without errors; what they measured, slower or -// faster, is what --update accepts. +// Baselines are written only when every run completed without errors. --update accepts whatever +// they measured, slower or faster. if (flags.update) { if (failures.length || runs.some((r) => r.errors.length)) { failures.push('baselines not written: a run failed or reported errors'); } else { fs.mkdirSync(baselines, { recursive: true }); write(workFile, work(verdict, read(workFile))); - write(timesFile, times(verdict, machine, read(timesFile))); + if (timed) write(timesFile, times(verdict, machine, read(timesFile))); } } console.log(''); diff --git a/benchmark/tiles.ts b/benchmark/tiles.ts index eeb52baa..98b402bb 100644 --- a/benchmark/tiles.ts +++ b/benchmark/tiles.ts @@ -1,7 +1,7 @@ /** - * Vector tiles made on request, so the tiles scenario fetches over HTTP as a map does, the same - * bytes every run and nothing from the network: `/tiles////.mvt`, one layer - * `cells` of that many squares in a grid over the tile. + * A Vite plugin that serves vector tiles made on request, so the tiles scenario fetches over HTTP + * as a map does, with the same bytes every run and no network. `/tiles////.mvt` + * returns one layer, `cells`, of that many squares in a grid over the tile. */ import type { Plugin } from 'vite'; @@ -32,7 +32,8 @@ const bytes = (field: number, payload: number[]) => [ ...payload, ]; -/** A tile of `squares` squares, clockwise exterior rings in the tile's 4096 units. */ +/** Encodes a tile of `squares` squares as exterior rings, clockwise in the tile's 4096-unit extent + * (y down). */ function tile(squares: number): Uint8Array { const side = Math.ceil(Math.sqrt(squares)); const step = Math.floor(4096 / side); diff --git a/benchmark/vite.config.ts b/benchmark/vite.config.ts index b76f2314..02acda61 100644 --- a/benchmark/vite.config.ts +++ b/benchmark/vite.config.ts @@ -3,7 +3,7 @@ import { tiles } from './tiles.ts'; export default defineConfig({ plugins: [tiles()], - // The package loads its wasm next to itself; prebundling would move it. + // latkit loads its wasm from beside its own module, which prebundling would move. optimizeDeps: { exclude: ['latkit'] }, build: { target: 'es2022' }, }); diff --git a/crates/latkit-cli/README.md b/crates/latkit-cli/README.md index 7aca9443..1c39f14b 100644 --- a/crates/latkit-cli/README.md +++ b/crates/latkit-cli/README.md @@ -1,21 +1,21 @@ # latkit CLI -Draws latkit scenes of datasets into images, image sequences, raw frames, and videos, on the -machine's GPU, with the same renderer as the browser package. Inputs are latkit's JSON, so any -language can write them. The package is `latkit-cli`; the executable is `latkit`. +Renders latkit scenes to images, image sequences, raw frames, and video on the machine's GPU, with +the same renderer as the browser package. Inputs are latkit's JSON, so any language can write them. +The package is `latkit-cli` and the executable is `latkit`. ```sh cargo install latkit-cli latkit validate scene.json -d grid.json latkit render scene.json -d grid.json -o grid.png --size 2400x1350 --scale 2 latkit render scene.json -d grid.json --frames frames.jsonl --fps 60 -o film.mp4 -latkit query grid.bin '{"of":"Branch","filter":{"op":"gt","args":["loading",90]},"limit":20}' +latkit query grid.bin '{"of":"Branch","filter":{"gt":["$loading",90]},"limit":20}' latkit tile history.bin -o history/ # time tiles a view fetches in spans: --tiles grid=history/frames.json latkit schema scene > scene.schema.json ``` -Commands, inputs, frames, outputs, and diagnostics are in -[the documentation](https://latkit.readthedocs.io/en/latest/cli.html), and -[its examples](https://latkit.readthedocs.io/en/latest/examples/) are drawn by this CLI. -[The 70,000-bus fault](examples/bus-fault/README.md) and -[the WECC240 fault](examples/wecc240-fault/README.md) are made with Python and this CLI alone. +[The documentation](https://latkit.readthedocs.io/en/latest/cli.html) covers commands, inputs, +frames, outputs, and diagnostics. This CLI draws all of +[its examples](https://latkit.readthedocs.io/en/latest/examples/). The +[70,000-bus fault](examples/bus-fault/README.md) and +[WECC240 fault](examples/wecc240-fault/README.md) videos are made with only Python and this CLI. diff --git a/crates/latkit-cli/examples/branch-opening-orbit/README.md b/crates/latkit-cli/examples/branch-opening-orbit/README.md index fcbb95d1..117532e1 100644 --- a/crates/latkit-cli/examples/branch-opening-orbit/README.md +++ b/crates/latkit-cli/examples/branch-opening-orbit/README.md @@ -3,15 +3,15 @@ [Preview](preview.png) · The video is not in Git: [reproduce it](#reproduce) to write `animation.mp4` here. -A close 108° orbit around the network, starting farther toward the East Coast -and stopping before the far-western viewpoint. Magnification stays nearly -constant, with a higher viewing angle throughout and no late pullback or dip. -Centering and rotation change smoothly along the shorter arc. +A close orbit around the network. The bearing turns 108°, from −70° to +38°, while the center +drifts west from 84.5° W to 87° W at 37.5° N. Zoom stays nearly constant and pitch stays between +46° and 50°, so the view never pulls back or dips. Center and bearing change smoothly between +keyframes. The saved simulation and visual scales come from the [original example](../branch-opening/README.md): -70,000 buses, the Little Rock–Newark 765 kV line opening at 1 s, 0.4× playback, -reduced height, no plot, 1920 × 1080 at 60 fps. The video is 23.517 seconds (1,411 frames). -The original example and its video remain separate. +70,000 buses, the Little Rock–Newark 765 kV line opening at 1 s, 0.4× playback, reduced height, no +plot, 1920 × 1080 at 60 fps. The video is 23.517 seconds (1,411 frames). Rendering the orbit leaves +the original example and its video untouched. ## Reproduce @@ -22,27 +22,26 @@ python crates/latkit-cli/examples/branch-opening-orbit/render.py --cli target/cl python crates/latkit-cli/examples/branch-opening-orbit/render.py --cli target/cli/latkit.exe ``` -Python prepares the documents; Latkit draws and encodes the video. The shared -[data recipe](../branch-opening/render.py) and state boundaries are read from the -original example. Generated inputs go into `target/branch-opening-orbit/`. -No simulation rerun, duplicated frequency recording, or full image sequence is needed. - -Edit [camera.json](camera.json) to change the orbit. -`at` is simulation time in seconds, `center` is longitude/latitude, -`scale` is the plane's pixels per coordinate unit, and `pitch`/`bearing` are degrees. -The recipe writes native CLI `camera` values into every frame. Monotone cubic Hermite -interpolation gives continuous camera velocity without overshooting keyframes; -zoom interpolates in logarithmic space, with gentle starts and stops. - -`--preview --at 5.8` shows the matching camera at that simulation time. `--prepare-only` -writes the inputs and prints the complete CLI command. `--source`, `--work`, `--output`, -and `--limit` support alternate data, scratch directories, and short previews. -Each output's JSON records its camera path, source checksum, timing, and render command. - -Frequency and height retain the original example's interpretation: a filtered -voltage-angle frequency estimate and an exaggerated, nonlinear height encoding. -Close framing favors the network and response over peripheral state boundaries. - -The delivered video was decoded end to end (1,411 frames), with sample -brackets checked at every frame and framing inspected around the opening and -along the camera path. +Python writes the CLI inputs, and the latkit CLI draws and encodes the video. The +[data recipe](../branch-opening/render.py) and state boundaries come from the original example. +Generated inputs go to `target/branch-opening-orbit/`. The recipe does not rerun the simulation, +copy the frequency recording, or write an image sequence. + +Edit [camera.json](camera.json) to change the orbit. `at` is simulation time in seconds, `center` is +longitude/latitude, `scale` is the plane's pixels per coordinate unit, and `pitch`/`bearing` are +degrees. The recipe writes a `camera` value into every line of the CLI's frames file. Keyframes are +interpolated with monotone cubic Hermite splines, so camera velocity is continuous, no value +overshoots a keyframe, and the path starts and stops gently. Zoom is interpolated in log space. + +`--preview --at 5.8` renders the frame at that simulation time, with its camera. `--prepare-only` +writes the inputs and prints the complete CLI command. `--source`, `--work`, and `--output` choose +other data, scratch, and output directories. `--limit N` renders only the first N frames. Each +output's JSON records its camera path, source checksum, timing, and render command. + +Frequency and height mean the same as in the original example. Frequency is estimated from filtered +voltage angles, and height exaggerates it on a nonlinear scale. The close framing shows the +network's response, not the outer state boundaries. + +The finished video was decoded end to end (1,411 frames). At every frame, the samples on either +side of its time were checked. The framing was inspected around the opening and along the camera +path. diff --git a/crates/latkit-cli/examples/branch-opening/README.md b/crates/latkit-cli/examples/branch-opening/README.md index 45b260e5..ed029c81 100644 --- a/crates/latkit-cli/examples/branch-opening/README.md +++ b/crates/latkit-cli/examples/branch-opening/README.md @@ -1,18 +1,17 @@ -# ACTIVSg70k line opening — Python + Latkit CLI +# ACTIVSg70k line opening with Python and the latkit CLI [Preview](preview.png) · The video is not in Git: [render the saved study](#render-the-saved-study) to write `animation.mp4` here. -The synthetic 70,000-bus network responds to the Little Rock–Newark 765 kV line -(`branch_59757_59774_1`) opening at 1 s. It stays open. The video shows simulation -0.6–10 s at a constant 0.4× speed: **1,411 frames, 60 fps, 1920 × 1080, -23.517 seconds**. The map fills the frame, and a dashed gold line marks the disconnected -branch. A gentle 30° camera sweep reveals the response from changing angles. +The synthetic 70,000-bus network responds as the Little Rock–Newark 765 kV line +(`branch_59757_59774_1`) opens at 1 s and stays open. The video shows simulation time 0.6–10 s at a +constant 0.4× speed: **1,411 frames, 60 fps, 1920 × 1080, 23.517 seconds**. The map fills the +frame, and a dashed gold line marks the open branch. The camera's bearing sweeps 30° to show the +response from changing angles. -[`render.py`](render.py) uses Python's standard library and the Latkit CLI. Latkit -draws the network, state boundaries, labels, and frequency legend, and encodes H.264 -through its native Windows MP4 output. There is no Rust consumer, browser renderer, -Python imaging dependency, or external encoder in this recipe. +[`render.py`](render.py) uses only Python's standard library and the latkit CLI. latkit draws the +network, state boundaries, labels, and frequency legend, and encodes H.264 with its native Windows +MP4 output. The recipe needs no Rust code, browser, Python imaging library, or external encoder. ## Render the saved study @@ -24,14 +23,13 @@ python crates/latkit-cli/examples/branch-opening/render.py --cli target/cli/latk python crates/latkit-cli/examples/branch-opening/render.py --cli target/cli/latkit.exe ``` -An installed CLI works too: `python crates/latkit-cli/examples/branch-opening/render.py`. -Use `--preview --at 2.5` for a different still, `--limit 120 --output target/preview` -for a short movie, or `--prepare-only` to write the inputs and print the complete -CLI command. Scene and camera JSON are generated in `target/branch-opening/render/`; -edit those and run the printed CLI command directly to experiment with the view. -A recipe rerun regenerates its inputs. +An installed CLI works too: `python crates/latkit-cli/examples/branch-opening/render.py`. Use +`--preview --at 2.5` for a still at another time, `--limit 120 --output target/preview` for a short +movie, or `--prepare-only` to write the inputs and print the complete CLI command. The scene and +camera JSON are written to `target/branch-opening/render/`. To experiment with the view, edit them +and run the printed CLI command. Rerunning the recipe overwrites them. -The optional positional argument selects another saved-study directory. Its files are: +The optional positional argument selects another saved-study directory, which holds: ```text target/branch-opening/ @@ -42,43 +40,40 @@ target/branch-opening/ frequency.f32 ``` -The recording is local, ignored data; it is **not included in Git**. The small state -boundary asset is reused from the earlier bus-fault recording and included here. -`animation.json` and `preview.json` record the command, source checksum, scales, -dimensions, and render timing. Native MP4 output currently requires Windows. +The recording is local data, **not included in Git**. The small state-boundary file, +`states.geojson`, comes from the bus-fault recording and is included here. `animation.json` and +`preview.json` record the command, source checksum, scales, dimensions, and render timing. Native +MP4 output requires Windows. ## Data and display The study used `ghcr.io/lukelowry/gridkit:develop-perf`, image ID -`sha256:befba9e16edce341888e73c05b9d2c35aedf1357fa18b54c56114dbe25ccd565`, -and the original ACTIVSg70k case, SHA-256 -`8c3d1edd9deaf409436b1f38e55f41c963d08eae7709474bcfb2c1dffadbdb15`. -The full simulation manifest remains beside the recording. +`sha256:befba9e16edce341888e73c05b9d2c35aedf1357fa18b54c56114dbe25ccd565`, and the original +ACTIVSg70k case, SHA-256 `8c3d1edd9deaf409436b1f38e55f41c963d08eae7709474bcfb2c1dffadbdb15`. The +full simulation manifest is stored beside the recording. -Only bus voltage angles were monitored, at 60 Hz over 0–10 s, with both sides of -the event retained. Frequency is the causal 20 ms filtered derivative of unwrapped -voltage angle, expressed as Hz deviation from 60 Hz. The raw angles were streamed -and discarded. This is a bus voltage-frequency estimate, not generator rotor speed. -The switching angle jump produces a brief impulse (about −1.647 to +0.683 Hz). +Only bus voltage angles were monitored, at 60 Hz over 0–10 s, keeping samples on both sides of the +event. Frequency is the causal 20 ms filtered derivative of unwrapped voltage angle, expressed as +Hz deviation from 60 Hz. The raw angles were streamed and discarded. This is a bus +voltage-frequency estimate, not generator rotor speed. The switching angle jump produces a brief +impulse (about −1.647 to +0.683 Hz). Colors saturate at ±0.20 Hz. Signed height is exaggerated by -`height_scale * asinh(df / 0.02) / asinh(0.35 / 0.02)`; it is a visual encoding of -frequency, not physical displacement. The height gain is 0.45 of the earlier view, -reducing displacement by 55% while preserving its sign and nonlinear scale. The -camera keeps a fixed center, zoom, and 48° pitch, with bearing easing from −10° to -+20° over the full recording. This keeps the initial peak in view and makes the -geography easier to follow. There is no bottom plot or trace legend. - -`HEIGHT_GAIN` and `BEARINGS` near the top of `render.py` control the exaggeration -and camera sweep. The camera path uses simulation time, so an arbitrary preview -matches that moment in the movie. - -For slow motion, Latkit interpolates between the original samples and computes the -height expression on the GPU. The recipe appends only samples needed for the next -picture and forgets older ones. Its timestamp comparison exactly matches Latkit's: -the retained lower sample must be **at or before** the requested time. Both event -sides remain distinct, so post-opening data never leaks into pre-opening pictures. -No dense interpolated cache, height cache, or full image sequence is written. - -The CLI validates the generated documents before rendering. The delivered MP4 was -also decoded end to end and checked around the event and sample boundaries. +`height_scale * asinh(df / 0.02) / asinh(0.35 / 0.02)`. Height is a visual encoding of frequency, +not physical displacement. `height_scale` includes a gain of 0.45, which scales every height down by +55% while keeping its sign and nonlinear shape. The camera keeps a fixed center, zoom, and 48° +pitch, with bearing easing from −10° to +20° over the full recording. This keeps the initial peak +in view and makes the geography easier to follow. There is no bottom plot or trace legend. + +`HEIGHT_GAIN` and `BEARINGS` near the top of `render.py` control the exaggeration and camera sweep. +The camera path follows simulation time, so a preview at any time matches that moment in the movie. + +For slow motion, latkit interpolates between the recorded samples and computes the height +expression on the GPU. The recipe appends only the samples the next picture needs and forgets older +ones. It selects samples with the same comparison latkit uses: the lower sample must be **at or +before** the requested time. The samples on either side of the event stay distinct, so no picture +before the opening shows data from after it. The recipe writes no interpolated or height cache and +no image sequence. + +The CLI validates the generated documents before rendering. The finished MP4 was also decoded end +to end and checked around the event and sample boundaries. diff --git a/crates/latkit-cli/examples/branch-opening/render.py b/crates/latkit-cli/examples/branch-opening/render.py index 375fb729..223c97e8 100644 --- a/crates/latkit-cli/examples/branch-opening/render.py +++ b/crates/latkit-cli/examples/branch-opening/render.py @@ -37,7 +37,7 @@ def write(path, value): def op(name, *args): - return {"op": name, "args": list(args)} + return {name: list(args)} def sha256(path): @@ -124,32 +124,30 @@ def append(start, end): # asinh(df / 0.02), expressed with the CLI's ordinary GPU expression operators. # Use abs/sign to avoid cancellation in ln(x + sqrt(x*x+1)) for negative x. - x = op("div", op("abs", "df"), 0.02) + x = op("div", op("abs", "$df"), 0.02) magnitude = op("ln", op("add", x, op("sqrt", op("add", op("mul", x, x), 1)))) - signed = {"when": [[op("lt", "df", 0), op("neg", magnitude)]], "otherwise": magnitude} + signed = {"when": [[op("lt", "$df", 0), op("neg", magnitude)]], "otherwise": magnitude} height_scale = HEIGHT_GAIN * 1.7 * 0.15 * max(max(lon) - min(lon), max(lat) - min(lat)) elevation = op("mul", signed, height_scale / math.asinh(0.35 / 0.02)) - theme = {"background": BACKGROUND, "text": "#c9d4dc", "font": "Segoe UI, sans-serif", "fontSize": 15} + theme = {"background": BACKGROUND, "text": {"color": "#c9d4dc", "font": "Segoe UI, sans-serif", "size": 15}} scenes = { "map.json": { "space": {"plane": {"grid": "off"}}, "ease": 0, - "theme": {**theme, "edge": None, "laneBow": 0}, + "theme": {**theme, "edge": {"color": None, "bow": 0}}, "title": "ACTIVSg70k / Line opening", "subtitle": f"Little Rock - Newark, 765 kV / Opens at 1 s / Constant {SPEED:g}x speed", "marks": [ - {"lines": {"of": ["states", "polygons"], "color": "#5c7a8f6b", "width": 0.75, - "visible": op("not", op("or", {"is": ["name", "Alaska"]}, - {"is": ["name", "Hawaii"]}))}}, - {"edges": {"of": ["grid", "Branch"], "ends": ["from", "to"], "width": "width", "visible": "in_service"}}, - {"edges": {"of": ["grid", "Outage"], "ends": ["from", "to"], "width": 2.5, - "color": "#ffde86", "dash": "opened", - "label": {"text": "status", "size": 13, "priority": 100}}}, - {"vertices": {"of": ["grid", "Bus"], "position": ["lon", "lat"], "radius": 0.8, - "elevation": elevation, - "color": {"colormap": {"kind": "diverging", "colors": PALETTE}, - "value": "df", "domain": [-0.2, 0.2]}, - "label": {"text": "label", "size": 14, "priority": 101}, - "legend": "Bus frequency deviation / Hz"}}, + {"lines": ["states", "polygons"], "color": "#5c7a8f6b", "width": 0.75, + "visible": op("not", op("or", {"is": ["$name", "Alaska"]}, {"is": ["$name", "Hawaii"]}))}, + {"edges": ["grid", "Branch"], "ends": ["from", "to"], "width": "$width", "visible": "$in_service"}, + {"edges": ["grid", "Outage"], "ends": ["from", "to"], "width": 2.5, + "color": "#ffde86", "dash": "$opened", "label": {"text": "$status", "size": 13}}, + {"vertices": ["grid", "Bus"], "position": ["$lon", "$lat"], "radius": 0.8, + "elevation": elevation, + "color": {"colormap": {"kind": "diverging", "colors": PALETTE}, + "value": "$df", "domain": [-0.2, 0.2]}, + "label": {"text": "$label", "size": 14}, + "legend": "Bus frequency deviation / Hz"}, ], }, "map-camera.json": camera_at(frames[0]), diff --git a/crates/latkit-cli/examples/bus-fault/README.md b/crates/latkit-cli/examples/bus-fault/README.md index 844d6bf6..28a8dac5 100644 --- a/crates/latkit-cli/examples/bus-fault/README.md +++ b/crates/latkit-cli/examples/bus-fault/README.md @@ -1,16 +1,18 @@ -# 70,000-bus fault — Python + latkit CLI +# 70,000-bus fault with Python and the latkit CLI -[Preview](preview.png) · The animation is not in Git: [`render.py`](render.py), run as below, -writes `animation.mp4` here. +[Preview](preview.png) · The animation is not in Git. Run [`render.py`](render.py) as below to +write `animation.mp4` here. -The saved ACTIVSg70k recording: simulation 0.6–1.4 s at 0.1×, then 1.4–10 s at 1×, 997 frames at -60 fps and 1920 × 1080. The fault is on bus 56923 at 1 s and clears at 1.15 s. Every bus keeps its -frequency-dependent color and signed height; a plot below traces six buses with a playhead. +The saved ACTIVSg70k fault recording: simulation time 0.6–1.4 s at 0.1× speed, then 1.4–10 s at +1×, as 997 frames at 60 fps and 1920 × 1080. The fault is on bus 56923 at 1 s and clears at 1.15 s. +Each bus is colored and raised or lowered by its frequency deviation. A plot below traces six buses, +with a playhead. [`render.py`](render.py) uses only Python's standard library. It writes the grid and the traced -buses as latkit JSON, whose frame columns are slices of the saved float32 caches, and a frame per -output frame that appends them. latkit reads the states' GeoJSON, draws the map with its title and -legend over the plot, and encodes the MP4: 5.5 s for the 997 frames on an RTX 2080 SUPER. +buses as latkit JSON, with frame columns that are slices of the saved float32 caches. It also +writes one line per output frame that appends that frame's samples. latkit reads the states' +GeoJSON, draws the map with its title and legend above the plot, and encodes the MP4. All 997 +frames take 5.5 s on an RTX 2080 SUPER. ```sh python crates/latkit-cli/examples/bus-fault/render.py --cli target/cli/latkit --preview @@ -18,11 +20,12 @@ python crates/latkit-cli/examples/bus-fault/render.py --cli target/cli ``` `` is the recording's directory: `buses.json`, `topology.json`, `usa-states.geojson`, -`depth-metadata.json`, and the frame-major caches `depth-frequency.f32` and `depth-height.f32` in -the metadata's bus order. They are not in this repository. Frequency comes from a causal 20 ms -filtered derivative of bus voltage angle; height is `asinh(df / 0.02) / asinh(0.35 / 0.02)`; colors -clip to ±0.05 Hz. +`depth-metadata.json`, and the frame-major caches `depth-frequency.f32` and `depth-height.f32`, in +the metadata's bus order. These files are not in this repository. Frequency is a causal 20 ms +filtered derivative of bus voltage angle. Height is `asinh(df / 0.02) / asinh(0.35 / 0.02)`. Colors +saturate at ±0.05 Hz. -The recipe writes its JSON to `target/bus-fault/` and keeps existing scenes and cameras there, so -edits survive a rerun. `--prepare-only` prints the command without rendering; `--limit N` renders the -first N frames. `animation.json` and `preview.json` record the command and timings. +The recipe writes its JSON to `target/bus-fault/` and keeps any scenes and cameras already there, so +your edits survive a rerun. `--prepare-only` prints the command without rendering, and `--limit N` +renders only the first N frames. `animation.json` and `preview.json` record the command and +timings. diff --git a/crates/latkit-cli/examples/bus-fault/render.py b/crates/latkit-cli/examples/bus-fault/render.py index 0b5eaf91..1f345f8a 100644 --- a/crates/latkit-cli/examples/bus-fault/render.py +++ b/crates/latkit-cli/examples/bus-fault/render.py @@ -86,31 +86,31 @@ def frame(k): ]}) height_scale = 1.7 * 0.15 * max(max(lon) - min(lon), max(lat) - min(lat)) - theme = {"background": BACKGROUND, "text": "#c9d4dc", "font": "Segoe UI, sans-serif", "fontSize": 15} + theme = {"background": BACKGROUND, "text": {"color": "#c9d4dc", "font": "Segoe UI, sans-serif", "size": 15}} scenes = { "map.json": { "space": {"plane": {"grid": "off"}}, "ease": 0, - "theme": {**theme, "edge": None, "laneBow": 0}, + "theme": {**theme, "edge": {"color": None, "bow": 0}}, "title": "ACTIVSg70k", "subtitle": f"A fault on bus {FAULT} at 1 s, cleared at 1.15 s", "marks": [ - {"lines": {"of": ["states", "polygons"], "color": "#5c7a8f6b", "width": 0.75}}, - {"edges": {"of": ["grid", "Branch"], "ends": ["from", "to"], "width": "width"}}, - {"vertices": { - "of": ["grid", "Bus"], "position": ["lon", "lat"], "radius": 0.8, - "elevation": {"op": "mul", "args": ["height", height_scale]}, + {"lines": ["states", "polygons"], "color": "#5c7a8f6b", "width": 0.75}, + {"edges": ["grid", "Branch"], "ends": ["from", "to"], "width": "$width"}, + { + "vertices": ["grid", "Bus"], "position": ["$lon", "$lat"], "radius": 0.8, + "elevation": {"mul": ["$height", height_scale]}, "color": {"colormap": {"kind": "diverging", "colors": PALETTE}, - "value": "df", "domain": [-0.05, 0.05]}, + "value": "$df", "domain": [-0.05, 0.05]}, "legend": "Bus frequency deviation · Hz", - }}, + }, ], }, "plot.json": { "space": {"plot": {"x": {"label": "Simulation time · s"}, "y": {"label": "Δf · Hz", "precision": 2}}}, "ease": 0, - "theme": {**theme, "fontSize": 13, "playhead": "#eff5f8"}, - "marks": [{"lines": {"of": ["trace", "Bus"], "trace": "df", "width": 1.4, - "color": {"colormap": "tableau10", "value": "group"}}}], + "theme": {**theme, "text": {**theme["text"], "size": 13}, "plot": {"playhead": "#eff5f8"}}, + "marks": [{"lines": ["trace", "Bus"], "trace": "df", "width": 1.4, + "color": {"colormap": "tableau10", "value": "$group"}}], }, "map-camera.json": {"center": [-87.8, 35], "zoom": math.log2(40), "pitch": 55, "bearing": 8}, "plot-camera.json": {"x": [times[0], times[-1]], "y": [-0.05, 0.05]}, diff --git a/crates/latkit-cli/examples/wecc240-fault/README.md b/crates/latkit-cli/examples/wecc240-fault/README.md index bc4cbecd..50c475d7 100644 --- a/crates/latkit-cli/examples/wecc240-fault/README.md +++ b/crates/latkit-cli/examples/wecc240-fault/README.md @@ -1,17 +1,16 @@ -# WECC240 fault — GridKit + Latkit CLI +# WECC240 fault with GridKit and the latkit CLI [Preview](preview.png) · The video is not in Git: [reproduce it](#reproduce) to write `animation.mp4` here. -The reference WECC240 fault is applied at **John Day, bus 4005**, at 1.00 s and -cleared at 1.15 s, with fault impedance R = 0, X = 0.01 p.u. The model contains -243 buses, 447 branches, 103 synchronous machines, and 37 inverter groups. -GridKit simulates 0–10 s; Latkit renders **1920 × 1080, 60 fps, 1,561 frames** -into a 26.017-second H.264 MP4. Playback slows to 0.1× around the fault and runs -at 0.5× elsewhere. +The reference WECC240 fault is applied at **John Day, bus 4005**, at 1.00 s and cleared at 1.15 s, +with fault impedance R = 0, X = 0.01 p.u. The model has 243 buses, 447 branches, 103 synchronous +machines, and 37 inverter groups. GridKit simulates 0–10 s, and latkit renders +**1920 × 1080, 60 fps, 1,561 frames** into a 26.017-second H.264 MP4. Playback slows to 0.1× around +the fault and runs at 0.5× elsewhere. -The preview, case, compact recordings, and reproduction scripts are included in -Git. Intermediate CSV and generated CLI inputs stay in `target/`. +The preview, case, compact recordings, and reproduction scripts are in Git. Intermediate CSV files +and generated CLI inputs stay in `target/`. ## Reproduce @@ -24,14 +23,13 @@ python crates/latkit-cli/examples/wecc240-fault/render.py --preview python crates/latkit-cli/examples/wecc240-fault/render.py ``` -Skip the first command to render the included recording without Docker. -`render.py` finds `target/cli/latkit.exe` (or `target/cli/latkit`), then falls back -to an installed `latkit`. Use `--cli PATH` to select another executable, -`--preview --at 1.12` for a still, or `--limit 120 --output target/wecc-preview` -for a short movie. `--prepare-only` writes scene/data/camera/frame inputs under -`target/wecc240-fault/render/` and prints the CLI command. -Native MP4 encoding currently requires Windows; the Python recipes use only -the standard library. State boundaries reuse [`../branch-opening/states.geojson`](../branch-opening/states.geojson). +Skip the first command to render the included recording without Docker. `render.py` uses +`target/cli/latkit.exe` (or `target/cli/latkit`) if present, and otherwise an installed `latkit`. +Use `--cli PATH` to select another executable, `--preview --at 1.12` for a still, or +`--limit 120 --output target/wecc-preview` for a short movie. `--prepare-only` writes the scene, +data, camera, and frame inputs under `target/wecc240-fault/render/` and prints the CLI command. +Native MP4 encoding requires Windows. The Python recipes use only the standard library. The state +boundaries come from [`../branch-opening/states.geojson`](../branch-opening/states.geojson). ## Simulation and display @@ -41,35 +39,34 @@ The run used `ghcr.io/lukelowry/gridkit:develop-perf`, pinned for execution to: sha256:befba9e16edce341888e73c05b9d2c35aedf1357fa18b54c56114dbe25ccd565 ``` -`simulate.py` resolves the requested tag to its local image ID before running, -with two CPUs and networking disabled. To reproduce with this exact image, pass -that ID through `--image`. The saved [metadata](recording/metadata.json) includes -the image labels, case hashes, solver settings, sample times, and recording hashes. +`simulate.py` resolves the requested tag to its local image ID before running, and runs it with two +CPUs and networking disabled. To reproduce with this exact image, pass that ID with `--image`. The +saved [metadata](recording/metadata.json) includes the image labels, case hashes, solver settings, +sample times, and recording hashes. -Monitors sample at 240 Hz, retaining both sides of fault application and clearing: -2,403 samples total. The map shows bus frequency deviation, estimated with a -20 ms causal filtered derivative of unwrapped voltage angle; switching angle -jumps contribute impulses. Colors saturate at ±0.1 Hz. Height exaggerates this -same quantity with an `asinh` scale; it is a visual displacement. +Monitors sample at 240 Hz and keep both sides of the fault's application and clearing, 2,403 +samples in total. The map shows bus frequency deviation, estimated with a 20 ms causal filtered +derivative of unwrapped voltage angle. The angle jumps at switching cause brief impulses. Colors +saturate at ±0.1 Hz. Height exaggerates the same quantity on an `asinh` scale. It is a visual +encoding, not a physical displacement. -The upper plot shows actual GENROU rotor-speed deviation (`omega × 60 Hz`) for -4039_G, the machine with the largest absolute excursion. The lower plot shows -faulted-bus voltage. A white playhead follows simulation time. These are distinct -measurements: estimated bus frequency, rotor speed, and voltage. +The upper plot shows actual GENROU rotor-speed deviation (`omega × 60 Hz`) for 4039_G, the machine +with the largest absolute excursion. The lower plot shows faulted-bus voltage. A white playhead +follows simulation time. These are distinct measurements: estimated bus frequency, rotor speed, and +voltage. -Only monitor selection and monitor names change in the runtime case. -GridKit reports initialization adjustments to small Hygov time constants and -Repca reactive limits; the complete messages are retained in [solver.txt](solver.txt). -`animation.json` and `preview.json` record render settings, timing, and hashes. +The runtime case differs from the source case only in its monitors and their names. GridKit reports +initialization adjustments to small Hygov time constants and Repca reactive limits. The complete +messages are in [solver.txt](solver.txt). `animation.json` and `preview.json` record render +settings, timing, and hashes. ## Source -`source.case.json` comes from GridKit's -`cases/PhasorDynamics/WECC240/WECC240.case.json`; only line endings are normalized -to LF. Fault parameters match its -`examples/PhasorDynamics/Validation/WECC240/WECC240.solver.json`. -The upstream case README credits the National Laboratory of the Rockies test -case repository. The GridKit notice is retained in [SOURCE-LICENSE.txt](SOURCE-LICENSE.txt). +`source.case.json` comes from GridKit's `cases/PhasorDynamics/WECC240/WECC240.case.json`, with only +line endings normalized to LF. Fault parameters match its +`examples/PhasorDynamics/Validation/WECC240/WECC240.solver.json`. The upstream case README credits +the National Laboratory of the Rockies test case repository. The GridKit notice is retained in +[SOURCE-LICENSE.txt](SOURCE-LICENSE.txt). This product includes software produced by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the Department of Energy. diff --git a/crates/latkit-cli/examples/wecc240-fault/render.py b/crates/latkit-cli/examples/wecc240-fault/render.py index d2714685..b643650c 100644 --- a/crates/latkit-cli/examples/wecc240-fault/render.py +++ b/crates/latkit-cli/examples/wecc240-fault/render.py @@ -25,7 +25,7 @@ def write(path, value): def op(name, *args): - return {"op": name, "args": list(args)} + return {name: list(args)} def playback(): @@ -82,7 +82,6 @@ def column(name, rows): "lon": [b["extension"]["longitude"] for b in buses], "lat": [b["extension"]["latitude"] for b in buses], "label": [named.get(i, "") for i in ids], - "priority": [1000 if i == 4005 else 1 for i in ids], "is_fault": [i == 4005 for i in ids], "faulted": [i == 4005 and 1 <= frames[0] < 1.15 for i in ids], "voltage_trace": [i in voltage_ids for i in ids], @@ -106,26 +105,26 @@ def column(name, rows): write(work / "grid.json", {"name": "grid", "patch": patch}) # The original samples fit in memory for 243 buses. No interpolated cache is needed: # Latkit reads at each frame's time and preserves the duplicate event timestamps. - x = op("div", op("abs", "df"), 0.01) + x = op("div", op("abs", "$df"), 0.01) magnitude = op("ln", op("add", x, op("sqrt", op("add", op("mul", x, x), 1)))) - signed = {"when": [[op("lt", "df", 0), op("neg", magnitude)]], "otherwise": magnitude} + signed = {"when": [[op("lt", "$df", 0), op("neg", magnitude)]], "otherwise": magnitude} elevation = op("mul", signed, 1.4 / math.asinh(0.1 / 0.01)) - theme = {"background": "#0a111b", "text": "#d5e1ed", "font": "Segoe UI, sans-serif", "fontSize": 17} + theme = {"background": "#0a111b", "text": {"color": "#d5e1ed", "font": "Segoe UI, sans-serif", "size": 17}} map_scene = { "space": {"plane": {"grid": "off"}}, "ease": 0, - "theme": {**theme, "edge": None, "laneBow": 0}, + "theme": {**theme, "edge": {"color": None, "bow": 0}}, "title": "WECC 240 / FAULT RESPONSE", "subtitle": "JOHN DAY / Bus 4005 / Applied 1.00 s / Cleared 1.15 s", "marks": [ - {"lines": {"of": ["states", "polygons"], "color": "#58718b70", "width": 0.9}}, - {"edges": {"of": ["grid", "Branch"], "ends": ["from", "to"], "width": "width"}}, - {"vertices": { - "of": ["grid", "Bus"], "position": ["lon", "lat"], "elevation": elevation, - "radius": {"when": [["faulted", 8], ["is_fault", 5]], "otherwise": 2.8}, - "color": {"colormap": {"kind": "diverging", "colors": PALETTE}, "value": "df", "domain": [-0.1, 0.1]}, - "label": {"text": "label", "size": 15, "priority": "priority"}, + {"lines": ["states", "polygons"], "color": "#58718b70", "width": 0.9}, + {"edges": ["grid", "Branch"], "ends": ["from", "to"], "width": "$width"}, + { + "vertices": ["grid", "Bus"], "position": ["$lon", "$lat"], "elevation": elevation, + "radius": {"when": [["$faulted", 8], ["$is_fault", 5]], "otherwise": 2.8}, + "color": {"colormap": {"kind": "diverging", "colors": PALETTE}, "value": "$df", "domain": [-0.1, 0.1]}, + "label": {"text": "$label", "size": 15}, "legend": "Bus frequency deviation / Hz", - }}, + }, ], } @@ -133,18 +132,16 @@ def plot(title, table, field, visible, color): return { "space": {"plot": {"x": {"label": "Simulation time / s"}, "y": {"label": title, "precision": 3 if field == "df" else 2}}}, - "ease": 0, "theme": {**theme, "background": "#101b28", "fontSize": 14, "playhead": "#f3f7fc"}, - "marks": [{"lines": { - "of": ["grid", table], "trace": field, "width": 2.0, "visible": visible, - "color": color, - }}], + "ease": 0, "theme": {**theme, "background": "#101b28", "text": {**theme["text"], "size": 14}, + "plot": {"playhead": "#f3f7fc"}}, + "marks": [{"lines": ["grid", table], "trace": field, "width": 2.0, "visible": visible, "color": color}], } write(work / "map.json", map_scene) write(work / "rotor.json", plot(f"{rotor_names[0]} / Rotor speed deviation / Hz", - "Machine", "df", "traced", "#ffcb72")) + "Machine", "df", "$traced", "#ffcb72")) write(work / "voltage.json", plot("4005 / JOHN DAY / Voltage / p.u.", - "Bus", "vm", "voltage_trace", "#66d6ef")) + "Bus", "vm", "$voltage_trace", "#66d6ef")) write(work / "map-camera.json", camera(frames[0])) write(work / "rotor-camera.json", {"x": [0, 10], "y": [-0.05, 0.12]}) write(work / "voltage-camera.json", {"x": [0, 10], "y": [0.88, 1.12]}) diff --git a/crates/latkit-cli/src/args.rs b/crates/latkit-cli/src/args.rs index 908aebe1..6d88a516 100644 --- a/crates/latkit-cli/src/args.rs +++ b/crates/latkit-cli/src/args.rs @@ -1,5 +1,5 @@ -//! The command line: what `render`, `validate`, `inspect`, `query`, `tile`, and `schema` take, and the -//! checks that fail as usage errors before anything is read. +//! Command-line arguments of `render`, `validate`, `inspect`, `query`, `tile`, and `schema`, and +//! the checks that report usage errors before any input is read. use std::path::{Path, PathBuf}; use std::time::Duration; @@ -10,7 +10,7 @@ use latkit::{ImageFormat, Pixels, Shot}; #[derive(Parser)] #[command(name = "latkit", version, about = "Render scenes of JSON or binary data to images and videos")] pub(crate) struct Cli { - /// Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line + /// Format of errors and render progress on stderr #[arg(long, global = true, value_enum, default_value = "human")] pub(crate) diagnostics: Diagnostics, #[command(subcommand)] @@ -19,7 +19,9 @@ pub(crate) struct Cli { #[derive(Clone, Copy, PartialEq, Eq, ValueEnum)] pub(crate) enum Diagnostics { + /// Lines of text, for people Human, + /// One JSON object per line, for programs Json, } @@ -27,46 +29,46 @@ pub(crate) enum Diagnostics { pub(crate) enum Command { /// Draw scenes of datasets into an image, image sequence, raw frames, or a video Render(Job), - /// Check everything render reads, frames included, without a GPU + /// Check every input render reads, including frames, without a GPU Validate(Job), - /// Describe datasets as JSON: tables, rows, fields and their kinds, the axis and its span + /// Describe datasets as JSON: tables, row counts, fields with kinds, and the axis and its span Inspect { - /// Dataset files, each [NAME=]FILE; - reads stdin + /// Dataset files, each `[NAME=]FILE` (`-` reads stdin) #[arg(required = true, value_name = "[NAME=]FILE")] data: Vec, /// The longest a dataset may take to load, in seconds #[arg(long, default_value = "30", value_parser = deadline)] deadline: Duration, }, - /// Print the JSON Schema of what latkit reads: every input, or one of them + /// Print the JSON Schema of every input latkit reads, or of one input Schema { - /// One input's schema + /// Print only this input's schema #[arg(value_enum)] input: Option, }, - /// Read rows back as JSON Lines, an object a row: those a query keeps, sorted and paged + /// Print the rows a query selects, sorted and paged, as JSON Lines with one object per row Query { - /// The dataset, [NAME=]FILE; - reads stdin + /// The dataset, `[NAME=]FILE` (`-` reads stdin) #[arg(value_name = "[NAME=]FILE")] data: Source, - /// The query: JSON such as {"of":"Branch","filter":{"op":"gt","args":["loading",90]}}, or a - /// file of it + /// The query: JSON, such as `{"of":"Branch","filter":{"gt":["$loading",90]}}`, or a file of + /// it (`-` reads stdin) #[arg(value_name = "QUERY")] query: String, /// The longest the dataset may take to load, in seconds #[arg(long, default_value = "30", value_parser = deadline)] deadline: Duration, }, - /// Write a dataset as time tiles: frames.json, which --tiles opens, the dataset without its - /// sampled fields' histories, and their spans at every level of detail + /// Write a dataset as time tiles: `frames.json` (which `--tiles` opens), the dataset without + /// the histories of its sampled fields, and those histories in spans at every level of detail Tile { - /// The dataset, [NAME=]FILE; - reads stdin + /// The dataset, `[NAME=]FILE` (`-` reads stdin) #[arg(value_name = "[NAME=]FILE")] data: Source, - /// The directory the tiles go in + /// Directory to write the tiles to #[arg(short, long, value_name = "DIR")] output: PathBuf, - /// Entries a span holds: frames, or buckets of a level of detail; a multiple of 8 + /// Entries per span: frames at full detail, buckets at coarser levels. A multiple of 8 #[arg(long, default_value = "256")] chunk: u32, /// The longest the dataset may take to load, in seconds @@ -75,103 +77,114 @@ pub(crate) enum Command { }, } -/// What a render draws and where it goes. +/// The inputs and output of `render` and `validate`. #[derive(Args)] pub(crate) struct Job { - /// Scene JSON files, one per panel; - reads stdin + /// Scene JSON files, one per panel (`-` reads stdin) #[arg(required = true, value_name = "SCENE")] pub(crate) scenes: Vec, - /// A dataset: JSON document, latkit bytes, or GeoJSON (which needs a NAME); NAME= renames it + /// A dataset file: a JSON document, latkit bytes, or GeoJSON. `NAME=` sets its name, which + /// GeoJSON requires #[arg(short, long, value_name = "[NAME=]FILE")] pub(crate) data: Vec, - /// Tiles every view fetches as dataset NAME: time tiles' frames.json, or vector tiles' PMTiles - /// file, TileJSON, or {z}/{x}/{y} template + /// Tiles every view fetches as dataset `NAME`: a time tiles `frames.json`, or a vector tiles + /// PMTiles file, TileJSON, or `{z}/{x}/{y}` template #[arg(long, value_name = "NAME=FILE")] pub(crate) tiles: Vec, - /// Image file, %d pattern for an image sequence, or - for stdout + /// Output file, `%d` pattern for an image sequence, or `-` for stdout #[arg(short, long, value_name = "FILE")] pub(crate) output: Option, - /// What to write; otherwise the output's extension says, or PNG for stdout + /// Output format. Defaults to the format of the output's extension, or PNG #[arg(long, value_enum)] pub(crate) format: Option, - /// A JPEG's quality, from 1, smallest, to 100, sharpest + /// JPEG quality, from 1 (smallest) to 100 (sharpest) #[arg(long, default_value = "90", value_parser = clap::value_parser!(u8).range(1..=100))] pub(crate) quality: u8, - /// The picture's width and height in pixels, such as 1920x1080 + /// The picture's width and height in pixels, such as `1920x1080`. A video rounds its width down + /// to a multiple of 4 and its height to an even number #[arg(long, default_value = "1600x900", value_parser = size)] pub(crate) size: [u32; 2], - /// Pixels per CSS pixel: views lay out size / scale CSS pixels, so 2 draws twice as sharp - #[arg(long, default_value = "1", value_parser = positive)] + /// Pixels per CSS pixel. Views lay out size / scale CSS pixels, so 2 draws twice as sharp + #[arg(long, default_value = "1", value_parser = scale)] pub(crate) scale: f32, - /// A scene's panel of the picture, X,Y,WIDTH,HEIGHT as fractions; one per scene, in order + /// A scene's panel of the picture: its x, y, width, and height, as fractions. Give one per + /// scene, in order #[arg(long, value_name = "X,Y,W,H", value_parser = panel)] pub(crate) panel: Vec<[f32; 4]>, /// Keep alpha where the themes' backgrounds would be #[arg(long)] pub(crate) transparent: bool, - /// The longest a picture, or a video frame, may take, in seconds + /// The longest a picture or video frame may take, in seconds #[arg(long, default_value = "30", value_parser = deadline)] pub(crate) deadline: Duration, - /// TTF/OTF/TTC file; repeat for fallbacks; replaces system fonts + /// A TTF, OTF, or TTC font file. Repeat for fallbacks. Replaces the system fonts #[arg(long, value_name = "FILE")] pub(crate) font: Vec, - /// Camera JSON for a scene, in scene order; a view without one fits - #[arg(long, value_name = "FILE")] - pub(crate) camera: Vec, - /// What a view without a camera frames: "all", or a fit as JSON + /// A camera per scene, in scene order: JSON, or a file of it (`-` reads stdin). Views without + /// one use `--fit` + #[arg(long, value_name = "CAMERA")] + pub(crate) camera: Vec, + /// What views without a camera frame: `all`, or a fit as JSON #[arg(long, default_value = "all", value_name = "FIT")] pub(crate) fit: String, - /// CSS pixels kept clear at the top, right, bottom, and left, T,R,B,L + /// CSS pixels to keep clear at the top, right, bottom, and left #[arg(long, value_name = "T,R,B,L", value_parser = insets)] pub(crate) insets: Option<[f32; 4]>, - /// The axis coordinate shown; sampled fields read their last frame at or before it + /// The axis coordinate to show. Sampled fields show their last frame at or before it #[arg(long, value_parser = finite, allow_hyphen_values = true)] pub(crate) at: Option, - /// A still's effect time in seconds, where elapsed time and flowing comets read it + /// A still's fixed effect time in seconds, for elapsed-time expressions and flowing comets #[arg(long, value_parser = seconds, conflicts_with = "frames")] pub(crate) elapsed: Option, - /// Frames as JSON Lines, one per output frame, or - for stdin + /// Frames as JSON Lines (`-` reads stdin): one per output frame, or with `--span`, at their `at` #[arg(long, value_name = "FILE")] pub(crate) frames: Option, - /// Output frames a second + /// The axis interval to play. A video or image sequence without `--frames` plays the datasets' + /// span + #[arg(long, value_name = "FROM,TO", value_parser = span, allow_hyphen_values = true)] + pub(crate) span: Option<[f64; 2]>, + /// Axis units played per second. Defaults to 1 + #[arg(long, value_parser = positive)] + pub(crate) rate: Option, + /// Output frames per second #[arg(long, default_value = "60", value_parser = fps)] pub(crate) fps: f64, - /// MP4 bits a second; about 0.15 bits a pixel a frame otherwise + /// MP4 bitrate in bits per second. Defaults to about 0.15 bits per pixel per frame #[arg(long, value_parser = clap::value_parser!(u32).range(1..))] pub(crate) bitrate: Option, } -/// An input `schema` describes. +/// An input whose schema `schema` prints. #[derive(Clone, Copy, PartialEq, Eq, Debug, ValueEnum)] pub(crate) enum Input { /// A scene Scene, /// A dataset's JSON document Data, - /// A patch's JSON + /// A patch Patch, - /// A query of `latkit query` + /// A query for `latkit query` Query, - /// A camera of `--camera` + /// A camera for `--camera` Camera, - /// A fit of `--fit` + /// A fit for `--fit` Fit, - /// A line of `--frames` + /// One line of a `--frames` file Frames, } -/// What an output is. +/// An output format. #[derive(Clone, Copy, PartialEq, Eq, ValueEnum)] pub(crate) enum Format { - /// PNG, lossless with alpha: one picture, or one a frame where the output has a %d pattern + /// PNG, lossless with alpha. One picture, or one per frame with a `%d` output pattern Png, - /// JPEG at --quality, its alpha dropped: one picture, or one a frame + /// JPEG at `--quality`, without alpha. One picture, or one per frame Jpeg, - /// WebP, lossless with alpha: one picture, or one a frame + /// Lossless WebP with alpha. One picture, or one per frame Webp, - /// Raw RGBA, straight alpha, frames back to back + /// Raw RGBA with straight alpha, frames back to back Rgba, - /// YUV4MPEG2: 4:2:0 BT.709 limited range, for any encoder + /// YUV4MPEG2 (4:2:0, BT.709 limited range), which any encoder reads Y4m, /// H.264 in fragmented MP4, from the platform's encoder Mp4, @@ -192,14 +205,19 @@ impl std::str::FromStr for Source { }); Ok(match named { Some((name, path)) if !path.is_empty() => Source { name: Some(name.into()), path: path.into() }, - Some(_) => return Err("NAME= names a file".into()), + Some(_) => return Err("expected a file after NAME=".into()), None => Source { name: None, path: value.into() }, }) } } impl Format { - /// The bytes its pictures compose into. + /// Whether it is a video: MP4 or YUV4MPEG2. + pub(crate) fn video(self) -> bool { + matches!(self, Format::Mp4 | Format::Y4m) + } + + /// The pixel layout its pictures compose into. pub(crate) fn pixels(self) -> Pixels { match self { Format::Mp4 | Format::Y4m => Pixels::Nv12, @@ -207,7 +225,7 @@ impl Format { } } - /// The image file it writes, at `quality` for a JPEG; `None` for raw frames and video. + /// The image format it writes, with `quality` for JPEG. `None` for raw frames and video. pub(crate) fn image(self, quality: u8) -> Option { match self { Format::Png => Some(ImageFormat::Png), @@ -219,7 +237,7 @@ impl Format { } impl Job { - /// What the output is: `--format`, else its extension, else PNG. + /// The output format: `--format`, else the one the output's extension names, else PNG. pub(crate) fn format(&self) -> Format { self.format.unwrap_or_else(|| { let extension = self.output.as_deref().and_then(Path::extension).and_then(|e| e.to_str()); @@ -234,19 +252,26 @@ impl Job { }) } - /// The picture the job takes: its size, scale, panels, alpha, and deadline. + /// The job's shot: its size, scale, panels, transparency, and deadline. pub(crate) fn shot(&self) -> Shot { - let shot = Shot::new(self.size).scale(self.scale).deadline(self.deadline).panels(self.panel.clone()); + let shot = Shot::new(self.size).scale(self.scale).deadline(Some(self.deadline)).panels(self.panel.clone()); if self.transparent { shot.transparent() } else { shot } } - /// Whether the output names one file a frame: a `%d` or `%0Nd` pattern. + /// Whether the output is a `%d` or `%0Nd` pattern that names one file per frame. pub(crate) fn sequence(&self) -> bool { self.output.as_deref().is_some_and(|p| pattern(p).is_some()) } + + /// Whether the render plays a span: `--span`, or a video or image sequence without `--frames`, + /// which plays the datasets' span. + pub(crate) fn plays(&self) -> bool { + self.span.is_some() || (self.frames.is_none() && (self.format().video() || self.sequence())) + } } -/// The `%d` or `%0Nd` of a path's file name, and its width. +/// The byte range of the `%d` or `%0Nd` at the last `%` in `path`, and its zero-padded width (0 +/// for `%d`). pub(crate) fn pattern(path: &Path) -> Option<(std::ops::Range, usize)> { let text = path.to_str()?; let at = text.rfind('%')?; @@ -260,8 +285,8 @@ pub(crate) fn pattern(path: &Path) -> Option<(std::ops::Range, usize)> { } impl Cli { - /// The checks clap cannot make, failing as usage errors: one reader of stdin, an output a render - /// can write, and a panel for each scene. + /// Checks clap cannot express, reported as usage errors: at most one input reads stdin, the + /// output is one a render can write, and each scene has a panel. pub(crate) fn check(&self) -> clap::error::Result<()> { let invalid = |message: &str| Self::command().error(ErrorKind::InvalidValue, message); let job = match &self.command { @@ -280,21 +305,21 @@ impl Cli { return Ok(()); } }; - let stdin = job.scenes.iter().chain(job.data.iter().map(|s| &s.path)).chain(&job.frames).chain(&job.camera); - if stdin.filter(|path| is_stdio(path)).count() > 1 { + let files = job.scenes.iter().chain(job.data.iter().map(|s| &s.path)).chain(&job.frames).map(PathBuf::as_path); + if files.chain(job.camera.iter().map(Path::new)).filter(|path| is_stdio(path)).count() > 1 { return Err(Self::command().error(ErrorKind::ArgumentConflict, "only one input can read stdin")); } if job.tiles.iter().any(|t| t.name.is_none()) { return Err(invalid("--tiles takes NAME=FILE")); } if job.scenes.len() > 1 && job.panel.len() != job.scenes.len() { - return Err(invalid("a shot of several scenes takes a --panel for each")); + return Err(invalid("several scenes need one --panel each")); } if job.scenes.len() == 1 && job.panel.len() > 1 { - return Err(invalid("one scene fills one panel")); + return Err(invalid("one scene takes at most one --panel")); } if job.camera.len() > job.scenes.len() { - return Err(invalid("--camera is one per scene")); + return Err(invalid("at most one --camera per scene")); } let Some(output) = &job.output else { if matches!(self.command, Command::Render(_)) { @@ -307,15 +332,23 @@ impl Cli { let known = matches!(extension.as_deref(), Some("png" | "jpg" | "jpeg" | "webp" | "mp4" | "y4m" | "rgba")); if job.format.is_none() && !is_stdio(output) && !known { return Err(invalid( - "--output ends in .png, .jpg, .webp, .mp4, .y4m, or .rgba, or --format says what it is", + "--output must end in .png, .jpg, .webp, .mp4, .y4m, or .rgba, unless --format is set", )); } - let video = matches!(format, Format::Mp4 | Format::Y4m); - if (video || job.sequence()) && job.frames.is_none() { - return Err(invalid("a video or an image sequence takes --frames")); + if job.span.is_some() && !(format.video() || job.sequence() || format == Format::Rgba) { + return Err(invalid("--span plays a video, an image sequence, or raw frames")); + } + if job.plays() && (job.at.is_some() || job.elapsed.is_some()) { + return Err(Self::command().error( + ErrorKind::ArgumentConflict, + "a span sets each frame's axis coordinate and time, so it takes no --at or --elapsed", + )); + } + if job.rate.is_some() && !job.plays() { + return Err(invalid("--rate sets how fast a span plays: give --span, or no --frames")); } if job.sequence() && (format.image(job.quality).is_none() || is_stdio(output)) { - return Err(invalid("a %d pattern names an image file a frame")); + return Err(invalid("a %d pattern needs an image format: png, jpeg, or webp")); } job.shot().check(format.pixels()).map_err(|e| invalid(&e.to_string())) } @@ -329,9 +362,19 @@ fn finite(value: &str) -> Result { value.parse::().ok().filter(|n| n.is_finite()).ok_or_else(|| "expected a finite number".into()) } -fn positive(value: &str) -> Result { - let n = finite(value)? as f32; - if n.is_finite() && n > 0.0 { Ok(n) } else { Err("expected a positive number".into()) } +fn positive(value: &str) -> Result { + finite(value).and_then(|n| if n > 0.0 { Ok(n) } else { Err("expected a positive number".into()) }) +} + +fn scale(value: &str) -> Result { + let n = positive(value)? as f32; + if n.is_finite() { Ok(n) } else { Err("expected a positive number".into()) } +} + +fn span(value: &str) -> Result<[f64; 2], String> { + let (from, to) = value.split_once(',').ok_or("expected FROM,TO")?; + let [from, to] = [finite(from.trim())?, finite(to.trim())?]; + if to > from { Ok([from, to]) } else { Err("expected TO greater than FROM".into()) } } fn seconds(value: &str) -> Result { @@ -339,16 +382,16 @@ fn seconds(value: &str) -> Result { } fn deadline(value: &str) -> Result { - seconds(value).and_then(|d| if d.is_zero() { Err("a deadline is longer than zero".into()) } else { Ok(d) }) + seconds(value).and_then(|d| if d.is_zero() { Err("expected more than zero seconds".into()) } else { Ok(d) }) } fn fps(value: &str) -> Result { - finite(value).and_then(|f| if (1.0..=240.0).contains(&f) { Ok(f) } else { Err("1 to 240 frames a second".into()) }) + finite(value).and_then(|f| if (1.0..=240.0).contains(&f) { Ok(f) } else { Err("expected 1 to 240 fps".into()) }) } fn size(value: &str) -> Result<[u32; 2], String> { let (w, h) = value.split_once(['x', 'X']).ok_or("expected WIDTHxHEIGHT")?; - let side = |v: &str| v.trim().parse::().ok().filter(|n| *n > 0).ok_or("a side is a positive whole number"); + let side = |v: &str| v.trim().parse::().ok().filter(|n| *n > 0).ok_or("expected positive whole numbers"); Ok([side(w)?, side(h)?]) } @@ -363,40 +406,87 @@ fn panel(value: &str) -> Result<[f32; 4], String> { fn insets(value: &str) -> Result<[f32; 4], String> { numbers::<4>(value) - .and_then(|n| if n.iter().all(|v| *v >= 0.0) { Ok(n) } else { Err("insets are not negative".into()) }) + .and_then(|n| if n.iter().all(|v| *v >= 0.0) { Ok(n) } else { Err("expected nonnegative insets".into()) }) } #[cfg(test)] mod tests { + use clap::ArgAction; + use serde_json::{Value, json}; + use super::*; + /// The command tree as JSON, which `npm/docs.mjs` renders as the command reference: each + /// command's name, usage, and description, then its arguments and options. Clap's own `help` + /// command and its help and version flags are left out, and a command's copies of global + /// options too. + fn commands() -> Value { + let mut cli = Cli::command(); + cli.build(); + let command = |command: &clap::Command, name: String, top: bool| { + let shown = |arg: &&clap::Arg| { + let clap = matches!( + arg.get_action(), + ArgAction::Help | ArgAction::HelpShort | ArgAction::HelpLong | ArgAction::Version + ); + !clap && !arg.is_hide_set() && (top || !arg.is_global_set()) + }; + let usage = command.clone().bin_name(&name).render_usage().to_string(); + let usage = usage.strip_prefix("Usage: ").expect("clap's usage").to_owned(); + assert!(!usage.contains('\n'), "{name}: one line of usage"); + let text = |s: Option<&clap::builder::StyledStr>| s.map(ToString::to_string).unwrap_or_default(); + let args: Vec = command + .get_arguments() + .filter(shown) + .map(|arg| { + let takes = arg.get_action().takes_values(); + let values: Vec = if takes { arg.get_possible_values() } else { Vec::new() } + .iter() + .filter(|v| !v.is_hide_set()) + .map(|v| json!({ "name": v.get_name(), "help": text(v.get_help()) })) + .collect(); + json!({ + "id": arg.get_id().as_str(), + "long": arg.get_long(), + "short": arg.get_short().map(String::from), + "value": takes.then(|| { + let name = arg.get_value_names().and_then(|names| names.first()); + name.map_or_else(|| arg.get_id().as_str().to_uppercase(), ToString::to_string) + }), + "positional": arg.is_positional(), + "required": arg.is_required_set(), + "many": arg.get_num_args().is_some_and(|n| n.max_values() > 1), + "global": arg.is_global_set(), + "help": text(arg.get_long_help().or(arg.get_help())), + "defaults": if takes { arg.get_default_values().iter().map(|v| v.to_string_lossy()).collect() } else { Vec::new() }, + "values": values, + }) + }) + .collect(); + let about = text(command.get_long_about().or(command.get_about())); + json!({ "name": name, "usage": usage, "about": about, "args": args }) + }; + let mut all = vec![command(&cli, "latkit".into(), true)]; + for sub in cli.get_subcommands().filter(|c| c.get_name() != "help") { + all.push(command(sub, format!("latkit {}", sub.get_name()), false)); + } + Value::Array(all) + } + #[test] fn cli_reference_matches_help() { - use std::fmt::Write as _; - let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); // This checks repository documentation, which is not part of the published CLI crate. if !root.join(".readthedocs.yaml").exists() { return; } - let mut cli = Cli::command(); - let mut text = format!( - "# Command reference\n\nGenerated from Clap. See the [CLI guide](../cli.md) for examples.\n\n## latkit\n\n```text\n{}```\n", - cli.render_long_help() - ); - for command in cli.get_subcommands_mut() { - let name = command.get_name().to_owned(); - let command = command.clone().bin_name(format!("latkit {name}")); - let mut command = command; - write!(text, "\n## {name}\n\n```text\n{}```\n", command.render_long_help()).unwrap(); - } - let text = text.lines().map(str::trim_end).collect::>().join("\n") + "\n"; - let path = root.join("docs/reference/cli.md"); + let text = format!("{}\n", serde_json::to_string_pretty(&commands()).unwrap()); + let path = root.join("docs/reference/cli.json"); if std::env::var_os("LATKIT_BLESS").is_some() { - std::fs::create_dir_all(path.parent().unwrap()).unwrap(); std::fs::write(&path, &text).unwrap(); } - assert_eq!(std::fs::read_to_string(path).unwrap().replace("\r\n", "\n"), text); + let message = "run with LATKIT_BLESS=1, then node npm/docs.mjs"; + assert_eq!(std::fs::read_to_string(path).unwrap().replace("\r\n", "\n"), text, "{message}"); } #[test] diff --git a/crates/latkit-cli/src/inputs.rs b/crates/latkit-cli/src/inputs.rs index a18715ae..358cb6ef 100644 --- a/crates/latkit-cli/src/inputs.rs +++ b/crates/latkit-cli/src/inputs.rs @@ -1,5 +1,5 @@ -//! What a job reads: scenes, datasets, cameras, fits, and frames. Datasets and frames' patches are -//! the library's JSON documents, loaded through the host, which reads the files they name. +//! Reading a job's inputs: scenes, datasets, cameras, fits, and frames. Datasets and frame patches +//! load through the library and its host, which reads the files their documents reference. use std::fs::File; use std::io::{self, BufRead, BufReader, Read}; @@ -8,18 +8,17 @@ use std::sync::mpsc::{self, Receiver}; use std::time::Duration; use latkit::host::Native; -use latkit::{Camera, Data, Error, Fit, Name, Patch, Patches, Scene}; +use latkit::{Camera, Data, Error, Fit, Name, Patch, Patches, Reading, Scene}; use serde::Deserialize; use serde::de::DeserializeOwned; -use serde::de::IgnoredAny; use crate::args::{Source, is_stdio}; use crate::{Failure, block}; -/// Frames read and loaded ahead of the one drawing. +/// Number of frames read and loaded ahead of the one being drawn. const AHEAD: usize = 2; -/// The bytes at `path`, or stdin's for `-`. +/// Reads the bytes at `path`, or stdin for `-`. pub(crate) fn read(path: &Path) -> Result, Failure> { let mut bytes = Vec::new(); let result = if is_stdio(path) { @@ -31,84 +30,78 @@ pub(crate) fn read(path: &Path) -> Result, Failure> { Ok(bytes) } -/// What paths in the document at `path` are relative to: the document, or nothing for stdin. +/// The base for relative paths in the document at `path`: the document's path, or empty for stdin. fn base(path: &Path) -> String { if is_stdio(path) { String::new() } else { path.to_string_lossy().into_owned() } } -/// What the JSON document at `path` holds, read as a `T`. +/// Reads the JSON document at `path` as a `T`. pub(crate) fn document(path: &Path) -> Result { let bytes = read(path)?; let text = std::str::from_utf8(&bytes).map_err(|_| Failure::at(Error::invalid("not UTF-8 text"), path))?; latkit::from_json(text).map_err(|e| Failure::at(e, path)) } -/// A scene from its JSON. -pub(crate) fn scene(path: &Path) -> Result { - document(path) +/// Reads argument `arg` of the option or argument `name` as a `T`: JSON when it starts with `{`, +/// else the document in the file it names. +pub(crate) fn json(arg: &str, name: &str) -> Result { + if arg.trim_start().starts_with('{') { + latkit::from_json(arg).map_err(|e| Failure::from(e.at(name))) + } else { + document(Path::new(arg)) + } } -/// A camera from its JSON. -pub(crate) fn camera(path: &Path) -> Result { +/// Reads a scene from its JSON file. +pub(crate) fn scene(path: &Path) -> Result { document(path) } -/// A fit: `all`, or one as JSON. +/// Parses a fit: a bare name such as `all`, or JSON. pub(crate) fn fit(text: &str) -> Result { let json = if text.trim_start().starts_with(['{', '"', '[']) { text.to_owned() } else { format!("\"{text}\"") }; latkit::from_json(&json).map_err(|e| Failure::from(e.at("--fit"))) } -/// The dataset `source` names: a JSON document loaded through `host`, latkit bytes, or GeoJSON, -/// named as it says. +/// Loads the dataset `source` names: latkit bytes, GeoJSON, or a JSON document whose referenced +/// files `host` reads. The name `source` gives, if any, names the dataset. pub(crate) fn dataset(source: &Source, host: &Native, deadline: Duration) -> Result { let path = &source.path; let bytes = read(path)?; - let at = |error: Error| Failure::at(error, path); - let name = source.name.as_deref(); - if bytes.starts_with(b"LATK") || bytes.starts_with(&[0x1f, 0x8b]) || geojson(&bytes) { - return Data::read(&bytes, name).map_err(|e| at(e.at("NAME=FILE"))); - } - let text = std::str::from_utf8(&bytes).map_err(|_| at(Error::invalid("a dataset is JSON or latkit bytes")))?; - let data = block(deadline, Data::load(text, &base(path), host)).map_err(at)?; - Ok(match name { - Some(name) if name != data.name().as_str() => data.renamed(name), - _ => data, - }) -} - -/// Whether JSON `bytes` are GeoJSON: an object with a `type`. -fn geojson(bytes: &[u8]) -> bool { - #[derive(Deserialize)] - struct Probe { - #[serde(rename = "type")] - kind: Option, - } - serde_json::from_slice::(bytes).is_ok_and(|p| p.kind.is_some()) + let base = base(path); + let reading = Reading { name: source.name.as_deref(), base: &base, host: Some(host) }; + block(deadline, Data::read(&bytes, reading)).map_err(|e| Failure::at(e, path)) } -/// One output frame's changes, as a line of the frames file says. +/// One line of a frames file: an output frame's changes, or with `--span`, changes at `at`. #[derive(Deserialize, schemars::JsonSchema)] #[serde(deny_unknown_fields)] pub(crate) struct Frame { - /// The axis coordinate every view shows. + /// The axis coordinate every view shows. With `--span`, where the line applies (required). pub(crate) at: Option, - /// Patches by dataset name, in the order written. + /// Patches by dataset name, applied in the order written. #[serde(default)] patch: Patches, - /// Where views look at once: one camera for every view it fits, or one per view. + /// Cameras views switch to at once: one for every view whose space it fits, or one per view. pub(crate) camera: Option>, - /// Where views ease to. + /// Cameras views ease to, given like `camera`. pub(crate) fly: Option>, - /// What views frame. + /// What views fit: one fit for every view, or one per view. pub(crate) fit: Option>, /// CSS pixels every view keeps clear: top, right, bottom, left. pub(crate) insets: Option<[f32; 4]>, - /// Degrees a second every map or plane turns. + /// Degrees per second every map or plane turns. pub(crate) orbit: Option, } -/// One value for every view, or one for each in order, `null` leaving a view as it is. +impl Frame { + /// A frame that shows axis coordinate `at` and changes nothing else. + pub(crate) fn showing(at: f64) -> Frame { + Frame { at: Some(at), patch: Patches::default(), camera: None, fly: None, fit: None, insets: None, orbit: None } + } +} + +/// One value for every view, or one per view in order. `null` leaves a view unchanged. #[derive(Deserialize, schemars::JsonSchema)] #[serde(untagged)] #[schemars(inline)] @@ -127,10 +120,10 @@ impl Each { } } -/// A frame's index, what it changes, and its patches by dataset, loaded. +/// A frame's index, its changes, and its loaded patches by dataset name. pub(crate) type Next = (usize, Frame, Vec<(Name, Patch)>); -/// The lines of a frames file, each a frame, its patches loaded through the host. +/// Reads a frames file, one frame per line, loading each frame's patches through the host. pub(crate) struct Frames { path: PathBuf, lines: io::Lines>, @@ -149,7 +142,7 @@ impl Frames { Ok(Frames { path: path.to_path_buf(), lines: input.lines(), index: 0, host, deadline }) } - /// The next frame, its patches loaded; `None` at the end. Blank lines are skipped. + /// The next frame, with its patches loaded. `None` at the end. Skips blank lines. pub(crate) fn next(&mut self) -> Result, Failure> { loop { let Some(line) = self.lines.next() else { return Ok(None) }; @@ -159,20 +152,39 @@ impl Frames { } let index = self.index; self.index += 1; - let at = |error: Error| Failure::at(error.at(index).at("frames"), &self.path); + let at = |error: Error| blame(&self.path, index, error); let mut frame: Frame = latkit::from_json(&line).map_err(at)?; if frame.at.is_some_and(|a| !a.is_finite()) { - return Err(at(Error::invalid("at is finite").at("at"))); + return Err(at(Error::invalid("expected a finite number").at("at"))); } let base = base(&self.path); - let patches = std::mem::take(&mut frame.patch).load(&base, &self.host); + let reading = Reading { base: &base, host: Some(&self.host), name: None }; + let patches = std::mem::take(&mut frame.patch).read(reading); let patches = block(self.deadline, patches).map_err(|e| at(e.at("patch")))?; return Ok(Some((index, frame, patches))); } } - /// The frames read and loaded on a thread of their own, a few ahead of the one taken, so their - /// files are read while frames draw. + /// Every frame, with its patches loaded, for a span. Each needs `at`, in order. + pub(crate) fn ordered(mut self) -> Result, Failure> { + let mut frames: Vec = Vec::new(); + while let Some(next) = self.next()? { + let before = frames.last().and_then(|(_, frame, _)| frame.at).unwrap_or(f64::NEG_INFINITY); + let error = match next.1.at { + None => Error::invalid("expected `at`, where the span applies the frame"), + Some(at) if at < before => Error::invalid("expected `at` at or after the frame before"), + Some(_) => { + frames.push(next); + continue; + } + }; + return Err(blame(&self.path, next.0, error.at("at"))); + } + Ok(frames) + } + + /// Reads and loads frames on a separate thread, a few ahead of the one taken, so their files + /// load while frames draw. pub(crate) fn ahead(mut self) -> Ahead { let (send, frames) = mpsc::sync_channel(AHEAD); std::thread::spawn(move || { @@ -187,6 +199,11 @@ impl Frames { } } +/// `error` in frame `index` of the frames file at `path`. +pub(crate) fn blame(path: &Path, index: usize, error: Error) -> Failure { + Failure::at(error.at(index).at("frames"), path) +} + /// Frames loaded ahead of drawing. pub(crate) struct Ahead(Receiver>); @@ -217,25 +234,31 @@ mod tests { let host = Native::new(); let load = |text: &str| dataset(&text.parse().unwrap(), &host, deadline()); let path = dir.path().join("grid.json").to_string_lossy().into_owned(); - let data = load(&path).unwrap(); - assert_eq!(data.table("Bus").unwrap().get("a", "x"), Some(latkit::Value::Number(-2.5))); + let data = load(&path).unwrap().snapshot(); + assert_eq!(data.get("Bus", "a", "x", None), Some(latkit::Value::Number(-2.5))); assert_eq!(load(&format!("other={path}")).unwrap().name().as_str(), "other"); std::fs::write(dir.path().join("grid.bin"), data.to_bytes()).unwrap(); - assert_eq!(load(&dir.path().join("grid.bin").to_string_lossy()).unwrap().table("Bus").unwrap().len(), 2); + let rows = |data: &Data, table: &str| { + data.snapshot().describe().tables.into_iter().find(|t| t.name == table).map(|t| t.rows) + }; + let bin = load(&dir.path().join("grid.bin").to_string_lossy()).unwrap(); + assert_eq!(rows(&bin, "Bus"), Some(2)); let geo = r#"{"type": "Feature", "properties": {}, "geometry": {"type": "Point", "coordinates": [1, 2]}}"#; std::fs::write(dir.path().join("p.geojson"), geo).unwrap(); let geo_path = dir.path().join("p.geojson").to_string_lossy().into_owned(); - assert!(load(&geo_path).is_err(), "GeoJSON names its dataset"); - assert_eq!(load(&format!("places={geo_path}")).unwrap().table("points").unwrap().len(), 1); + assert!(load(&geo_path).is_err(), "GeoJSON needs a name"); + let places = load(&format!("places={geo_path}")).unwrap(); + assert_eq!(rows(&places, "points"), Some(1)); } #[test] - fn frames_keep_their_patches_order_and_apply_to_every_view_or_each() { + fn frames_keep_patch_order_and_take_one_value_or_one_per_view() { let frame: Frame = latkit::from_json(r#"{"patch": {"b": [], "a": []}}"#).unwrap(); - let patches = block(deadline(), frame.patch.load("", &Native::new())).unwrap(); + let host = Native::new(); + let patches = block(deadline(), frame.patch.read(Reading { host: Some(&host), ..Reading::default() })).unwrap(); let names: Vec<&str> = patches.iter().map(|(name, _)| name.as_str()).collect(); assert_eq!(names, ["b", "a"]); - assert!(latkit::from_json::(r#"{"patch": {"a": [], "a": []}}"#).is_err(), "a dataset patched twice"); + assert!(latkit::from_json::(r#"{"patch": {"a": [], "a": []}}"#).is_err(), "a dataset named twice fails"); let error = latkit::from_json::(r#"{"patch": {"a": [[]]}}"#).map(drop).unwrap_err(); assert!(error.to_string().starts_with("patch.a[0]"), "{error}"); let one: Each = latkit::from_json("2").unwrap(); diff --git a/crates/latkit-cli/src/main.rs b/crates/latkit-cli/src/main.rs index 2a20cefd..0d30f409 100644 --- a/crates/latkit-cli/src/main.rs +++ b/crates/latkit-cli/src/main.rs @@ -1,5 +1,5 @@ -//! The latkit command line: scenes and datasets in the library's JSON (or its bytes), drawn by the -//! library natively into images, image sequences, raw frames, and videos. +//! The latkit command line. It renders scenes and datasets, given as the library's JSON or bytes, +//! natively into images, image sequences, raw frames, and videos. mod args; mod inputs; @@ -8,7 +8,7 @@ mod report; mod sinks; use std::future::Future; -use std::io::{self, Write}; +use std::io; use std::path::Path; use std::process::ExitCode; use std::sync::Arc; @@ -21,6 +21,7 @@ use serde_json::{Value, json}; use args::{Cli, Command, Input, Source}; use report::Report; +use sinks::{Output, written}; fn main() -> ExitCode { let cli = Cli::parse(); @@ -45,7 +46,8 @@ fn main() -> ExitCode { } } -/// Why a command failed: the library's error, and the file it came from. +/// A command's failure: the library's error, and the input file at fault, if any. An output that +/// fails to be written is named in the error's message instead. #[derive(Debug)] pub(crate) struct Failure { pub(crate) error: Error, @@ -53,12 +55,12 @@ pub(crate) struct Failure { } impl Failure { - /// `error`, from the file at `path`. + /// `error`, caused by the input file at `path`. pub(crate) fn at(error: Error, path: &Path) -> Failure { Failure { error, source: Some(path.to_string_lossy().into_owned()) } } - /// An I/O `error` with the file at `path`. + /// An I/O `error` reading the input file at `path`. pub(crate) fn io(error: &io::Error, path: &Path) -> Failure { Failure::at(Error::Io(error.to_string().into()), path) } @@ -70,30 +72,35 @@ impl From for Failure { } } -/// Every dataset's description as JSON on stdout: its name, axis and span, and each table's rows and -/// fields. +/// Prints `value` to stdout as indented JSON. +fn print_json(value: &impl serde::Serialize) -> Result<(), Failure> { + let mut out = Output::create(Path::new("-"))?; + out.json(value, true)?; + Ok(out.commit()?) +} + +/// Prints every dataset's description as JSON to stdout: its name, axis and span, and each table's +/// rows and fields. fn inspect(sources: &[Source], deadline: Duration) -> Result<(), Failure> { let host = latkit::host::Native::new(); let mut all = Vec::with_capacity(sources.len()); for source in sources { - all.push(inputs::dataset(source, &host, deadline)?.describe()); + all.push(inputs::dataset(source, &host, deadline)?.snapshot().describe()); } - let mut out = io::stdout().lock(); - serde_json::to_writer_pretty(&mut out, &all).map_err(|e| Failure::from(Error::Io(e.to_string().into())))?; - writeln!(out).map_err(|e| Failure::io(&e, Path::new("-"))) + print_json(&all) } -/// The JSON Schema of every input on stdout, or of `input`: its definition, with every definition -/// it is built of. +/// Prints the JSON Schema of every input to stdout. With `input`, the root `$ref` points to that +/// input's definition. fn schema(input: Option) -> Result<(), Failure> { let mut schemas = latkit::schemas(); let frames = latkit::schema::(); - let defs = schemas["$defs"].as_object_mut().expect("definitions"); + let defs = schemas["$defs"].as_object_mut().expect("the schemas have $defs"); for (name, def) in frames["$defs"].as_object().into_iter().flatten() { defs.entry(name.clone()).or_insert_with(|| def.clone()); } let mut frame = frames.clone(); - frame.as_object_mut().expect("a schema").retain(|key, _| key != "$defs" && key != "$schema"); + frame.as_object_mut().expect("a schema is an object").retain(|key, _| key != "$defs" && key != "$schema"); defs.insert("Frame".into(), frame); if let Some(input) = input { let name = match input { @@ -107,22 +114,17 @@ fn schema(input: Option) -> Result<(), Failure> { }; schemas["$ref"] = json!(format!("#/$defs/{name}")); } - let mut out = io::stdout().lock(); - serde_json::to_writer_pretty(&mut out, &schemas).map_err(|e| Failure::from(Error::Io(e.to_string().into())))?; - writeln!(out).map_err(|e| Failure::io(&e, Path::new("-"))) + print_json(&schemas) } -/// The rows `query` reads from the dataset as JSON Lines on stdout: `id`, then each field, a -/// reference as its target's id and a missing value as `null`. +/// Prints the rows `query` selects from the dataset as JSON Lines to stdout. Each line has `id`, +/// then each field. A reference prints as its target's id, and a missing value as `null`. fn query(source: &Source, query: &str, deadline: Duration) -> Result<(), Failure> { - let text = - if query.trim_start().starts_with('{') { query.as_bytes().to_vec() } else { inputs::read(Path::new(query))? }; - let text = std::str::from_utf8(&text).map_err(|_| Failure::from(Error::invalid("not UTF-8 text").at("query")))?; - let query: Query = latkit::from_json(text).map_err(|e| Failure::from(e.at("query")))?; - let (rows, _) = inputs::dataset(source, &latkit::host::Native::new(), deadline)?.query(&query)?; + let query: Query = inputs::json(query, "query")?; + let rows = inputs::dataset(source, &latkit::host::Native::new(), deadline)?.snapshot().query(&query)?.table; let columns: Vec<(String, Values)> = - rows.fields().map(|(name, _)| (name.to_string(), rows.values(name).expect("a field the table has"))).collect(); - let mut out = io::BufWriter::new(io::stdout().lock()); + rows.fields().map(|(name, _)| (name.to_string(), rows.values(name).expect("the field exists"))).collect(); + let mut out = Output::create(Path::new("-"))?; for (row, id) in rows.ids().enumerate() { let mut line = serde_json::Map::with_capacity(columns.len() + 1); line.insert("id".into(), json!(id)); @@ -133,39 +135,30 @@ fn query(source: &Source, query: &str, deadline: Duration) -> Result<(), Failure }; line.insert(name.clone(), value); } - serde_json::to_writer(&mut out, &line).map_err(|e| Failure::from(Error::Io(e.to_string().into())))?; - writeln!(out).map_err(|e| Failure::io(&e, Path::new("-")))?; + out.json(&line, false)?; } - out.flush().map_err(|e| Failure::io(&e, Path::new("-"))) + Ok(out.commit()?) } -/// The dataset as time tiles in directory `output`: its manifest, the dataset without its sampled -/// fields' histories, and their spans of `chunk` entries. +/// Writes the dataset as time tiles in directory `output`: the manifest, the dataset without the +/// histories of its sampled fields, and those histories in spans of `chunk` entries. fn tile(source: &Source, output: &Path, chunk: u32, deadline: Duration) -> Result<(), Failure> { - let data = inputs::dataset(source, &latkit::host::Native::new(), deadline)?; - let mut failure = None; - let written = data.to_tiles(chunk, |path, bytes| { + let data = inputs::dataset(source, &latkit::host::Native::new(), deadline)?.snapshot(); + let tiled = data.to_tiles(chunk, |path, bytes| { let file = output.join(path); let wrote = file.parent().map_or(Ok(()), std::fs::create_dir_all).and_then(|()| std::fs::write(&file, bytes)); - wrote.map_err(|e| { - failure = Some(Failure::io(&e, &file)); - Error::Io(e.to_string().into()) - }) + wrote.map_err(|e| written(&e, &file)) }); - match (written, failure) { - (Ok(()), _) => Ok(()), - (Err(_), Some(failure)) => Err(failure), - (Err(error), None) => Err(Failure::from(error)), - } + Ok(tiled?) } -/// `future`'s result, driven on this thread and given up at `deadline`. +/// Drives `future` on this thread and returns its result, or [`Error::Timeout`] at `deadline`. pub(crate) fn block(deadline: Duration, future: impl Future>) -> latkit::Result { until(deadline, future)? } -/// A synchronous command needs only one parked thread to drive its future, with a deadline. The -/// future must yield; like other async timeouts this cannot interrupt a synchronous OS or driver call. +/// Drives `future` on this thread, parking between polls, until it finishes or `deadline` passes. +/// The deadline is checked only between polls, so it cannot interrupt a blocking OS or driver call. fn until(deadline: Duration, future: impl Future) -> latkit::Result { struct Thread(std::thread::Thread); impl Wake for Thread { diff --git a/crates/latkit-cli/src/render.rs b/crates/latkit-cli/src/render.rs index 1821b41c..dc083cae 100644 --- a/crates/latkit-cli/src/render.rs +++ b/crates/latkit-cli/src/render.rs @@ -1,24 +1,25 @@ -//! Rendering a job, and validating one: views of the scenes with the datasets, then a still picture, -//! or a video of the frames handed to a sink. Validating reads, loads, and checks everything a render -//! does, frames included, without a GPU; both change frames with [`apply`]. +//! Rendering and validating a job. Rendering creates a view of each scene over the datasets, then +//! takes a still picture, or draws a video into a sink. Validating reads, loads, and checks every +//! input a render does, including frames, without a GPU. Both read what plays with [`playback`] and +//! apply frames with [`apply`]. use std::path::{Path, PathBuf}; +use std::time::Duration; -use latkit::Tiles; use latkit::host::Native; -use latkit::{Camera, Data, Error, Fit, Gpu, Mp4, Name, Patch, Scene, Sink, View}; +use latkit::{Camera, Data, Error, Fit, Gpu, Mp4, Name, Patch, Scene, Sink, Tiles, View}; use crate::args::{Format, Job, Source}; -use crate::inputs::{self, Ahead, Each, Frame, Frames}; +use crate::inputs::{self, Each, Frame, Frames, Next, blame}; use crate::report::Report; use crate::sinks::{Commit, Images, Output, Raw, Y4m}; use crate::{Failure, block}; -/// What a job reads before it draws, and the host that read it. +/// A job's inputs, read before drawing, and the host that read them. struct Inputs { scenes: Vec<(PathBuf, Scene)>, data: Vec, - /// What every view fetches as it draws. + /// Tiles every view fetches as it draws. tiles: Vec, cameras: Vec, fit: Fit, @@ -39,32 +40,34 @@ impl Inputs { Tiles::new(name.as_deref().unwrap_or_default(), path.to_string_lossy().as_ref()) }) .collect(); - for (path, scene) in &scenes { - scene.validate(&data, &tiles).map_err(|e| Failure::at(e, path))?; + let cameras: Vec = job.camera.iter().map(|c| inputs::json(c, "--camera")).collect::>()?; + if let Some(i) = cameras.iter().zip(&scenes).position(|(camera, (_, scene))| !camera.fits(scene.space())) { + return Err(Error::invalid("the camera does not match the space of its scene").at(i).at("--camera").into()); } - let cameras = job.camera.iter().map(|p| inputs::camera(p)).collect::, _>>()?; - for (i, camera) in cameras.iter().enumerate() { - if !camera.fits(scenes[i].1.space()) { - return Err(Failure::at( - Error::invalid("the camera does not look at its scene's space"), - &job.camera[i], - )); - } + let inputs = Inputs { scenes, data, tiles, cameras, fit: inputs::fit(&job.fit)?, host }; + for (path, scene) in &inputs.scenes { + scene.validate(&inputs.sources()).map_err(|e| Failure::at(e, path))?; } - Ok(Inputs { scenes, data, tiles, cameras, fit: inputs::fit(&job.fit)?, host }) + Ok(inputs) + } + + /// The sources every view draws: each dataset, and the tiles. + fn sources(&self) -> Vec { + self.data.iter().map(latkit::Source::from).chain(self.tiles.iter().map(latkit::Source::from)).collect() + } + + /// The datasets' span: from the first axis coordinate any of them holds to the last. + fn span(&self) -> Option<[f64; 2]> { + self.data.iter().filter_map(|d| d.snapshot().describe().span).reduce(|a, b| [a[0].min(b[0]), a[1].max(b[1])]) } - /// A view of each scene, given every dataset and the tiles, looking where the job says. + /// Creates a view of each scene over every dataset and the tiles, with the job's insets, axis + /// coordinate, and camera or fit. fn views(&self, gpu: &Gpu, job: &Job) -> Result, Failure> { let mut views = Vec::with_capacity(self.scenes.len()); for (i, (path, scene)) in self.scenes.iter().enumerate() { let mut view = View::new(gpu, scene.clone()).map_err(|e| Failure::at(e, path))?; - for data in &self.data { - view.set_data(data.clone()); - } - for tiles in &self.tiles { - view.set_tiles(tiles.clone()); - } + view.set_sources(self.sources()).map_err(|e| Failure::at(e, path))?; if let Some(insets) = job.insets { view.set_insets(insets); } @@ -82,12 +85,12 @@ impl Inputs { } pub(crate) fn render(job: &Job, report: &mut Report) -> Result<(), Failure> { - let mut inputs = Inputs::read(job)?; + let inputs = Inputs::read(job)?; let gpu = block(job.deadline, Gpu::new())?; gpu.set_host(inputs.host.clone()); let mut views = inputs.views(&gpu, job)?; - let output = job.output.as_deref().expect("render takes an output"); - let Some(path) = &job.frames else { + let output = job.output.as_deref().expect("render requires --output"); + let Some(playback) = playback(job, &inputs)? else { let mut shot = job.shot(); if let Some(elapsed) = job.elapsed { shot = shot.elapsed(elapsed); @@ -95,16 +98,15 @@ pub(crate) fn render(job: &Job, report: &mut Report) -> Result<(), Failure> { let image = block(job.deadline, shot.image(&mut views.iter_mut().collect::>()))?; let mut out = Output::create(output)?; match job.format().image(job.quality) { - Some(format) => image.write(format, &mut out).map_err(|e| Failure::at(e, output))?, - None => std::io::Write::write_all(&mut out, &image.rgba).map_err(|e| Failure::io(&e, output))?, + Some(format) => image.write(format, &mut out)?, + None => out.put(&image.rgba)?, } out.commit()?; report.done(1, &views, &gpu); return Ok(()); }; - let frames = Frames::open(path, inputs.host.clone(), job.deadline)?.ahead(); let host = inputs.host.clone(); - let mut sequence = Sequence { job, path, gpu: &gpu, views: &mut views, inputs: &mut inputs, report }; + let sequence = Sequence { job, playback, gpu: &gpu, views: &mut views, inputs: &inputs, report }; let format = job.format(); if let Some(image) = format.image(job.quality) { let images = if job.sequence() { @@ -112,71 +114,186 @@ pub(crate) fn render(job: &Job, report: &mut Report) -> Result<(), Failure> { } else { Images::one(Output::create(output)?, image, host) }; - return sequence.run(frames, images); + return sequence.run(images); } match format { - Format::Png | Format::Jpeg | Format::Webp => unreachable!("images write above"), - Format::Rgba => sequence.run(frames, Raw::new(Output::create(output)?)), - Format::Y4m => sequence.run(frames, Y4m::new(Output::create(output)?)), + Format::Png | Format::Jpeg | Format::Webp => unreachable!("image formats are handled above"), + Format::Rgba => sequence.run(Raw::new(Output::create(output)?)), + Format::Y4m => sequence.run(Y4m::new(Output::create(output)?)), Format::Mp4 => { let mut mp4 = Mp4::new(Output::create(output)?); if let Some(bitrate) = job.bitrate { mp4 = mp4.bitrate(bitrate); } - sequence.run(frames, mp4) + sequence.run(mp4) + } + } +} + +/// What a render plays: the frames file's lines, in order, and with a span, how its output frames +/// step through the axis. +struct Playback { + lines: Box>>, + span: Option, + /// What errors in the lines name: the frames file, or `--span` without one. + file: PathBuf, +} + +/// An axis interval played at `rate` units per second, and the flights between its lines. +struct Span { + from: f64, + to: f64, + rate: f64, + flights: Flights, +} + +/// What `job` plays, or `None` for a still. +fn playback(job: &Job, inputs: &Inputs) -> Result, Failure> { + let file = job.frames.clone().unwrap_or_else(|| PathBuf::from("--span")); + let frames = job.frames.as_deref().map(|path| Frames::open(path, inputs.host.clone(), job.deadline)).transpose()?; + if !job.plays() { + let Some(frames) = frames else { return Ok(None) }; + let mut lines = frames.ahead().peekable(); + if lines.peek().is_none() { + return Err(Failure::at(Error::invalid("the frames file has no frames"), &file)); } + return Ok(Some(Playback { lines: Box::new(lines), span: None, file })); } + let lines = frames.map(Frames::ordered).transpose()?.unwrap_or_default(); + let [from, to] = job.span.or_else(|| inputs.span()).ok_or_else(|| { + Failure::from(Error::invalid("nothing to play: the datasets record no axis span, so give --span or --frames")) + })?; + let rate = job.rate.unwrap_or(1.0); + let flights = Flights::plan(&lines, inputs.scenes.len(), rate); + let span = Span { from, to, rate, flights }; + Ok(Some(Playback { lines: Box::new(lines.into_iter().map(Ok)), span: Some(span), file })) +} + +/// For each line of a span, the camera each view it sets flies on to, and the axis coordinate of +/// the next line that sets that view's camera. A line that also flies or fits the view starts none. +#[derive(Default)] +struct Flights { + onward: Vec>>, + rate: f64, } -/// A video of the frames file at `path`, drawn as each line says. +impl Flights { + fn plan(lines: &[Next], views: usize, rate: f64) -> Flights { + // From the end back: the next camera each view is set to. + let mut later: Vec> = vec![None; views]; + let mut onward = vec![Vec::new(); lines.len()]; + for (j, (_, frame, _)) in lines.iter().enumerate().rev() { + let set = |i: usize| frame.camera.as_ref().and_then(|c| c.of(i)); + let steered = |i: usize| { + frame.fly.as_ref().and_then(|c| c.of(i)).is_some() || frame.fit.as_ref().and_then(|f| f.of(i)).is_some() + }; + onward[j] = (0..views).map(|i| later[i].filter(|_| set(i).is_some() && !steered(i))).collect(); + for (i, later) in later.iter_mut().enumerate() { + if let (Some(camera), Some(at)) = (set(i), frame.at) { + *later = Some((at, camera)); + } + } + } + Flights { onward, rate } + } + + /// Flies each view that line `j` set on to its next camera, from axis coordinate `at`. + fn start(&self, views: &mut [View], j: usize, at: f64) { + let Some(onward) = self.onward.get(j) else { return }; + for (view, onward) in views.iter_mut().zip(onward) { + if let Some((arrive, camera)) = *onward { + view.fly(camera, Some(Duration::from_secs_f64(((arrive - at) / self.rate).max(0.0)))); + } + } + } +} + +/// One output frame: the lines it applies, in order, and in a span, the axis coordinate it shows. +struct Shown { + lines: Vec, + at: Option, +} + +/// The output frames of `lines`: one per line, or with `span`, one per step through it, each +/// applying the lines it reached. +fn shown<'a>( + lines: Box>>, + span: Option<&'a Span>, + fps: f64, +) -> Box> + 'a> { + let Some(span) = span else { + return Box::new(lines.map(|line| line.map(|line| Shown { lines: vec![line], at: None }))); + }; + // A little slack keeps the last frame when the span divides evenly into frames. + let count = ((span.to - span.from) * fps / span.rate + 1e-9).floor() as usize + 1; + let mut lines = lines.peekable(); + Box::new((0..count).map(move |n| { + let at = span.from + n as f64 * span.rate / fps; + let mut due = Vec::new(); + while let Some(line) = + lines.next_if(|line| line.as_ref().is_ok_and(|(_, f, _)| f.at.is_some_and(|key| key <= at))) + { + due.push(line?); + } + Ok(Shown { lines: due, at: Some(at) }) + })) +} + +/// A video of what plays, drawn into a sink. struct Sequence<'a> { job: &'a Job, - path: &'a Path, + playback: Playback, gpu: &'a Gpu, views: &'a mut Vec, - inputs: &'a mut Inputs, + inputs: &'a Inputs, report: &'a mut Report, } impl Sequence<'_> { - fn run(&mut self, frames: Ahead, sink: S) -> Result<(), Failure> { - let (job, path) = (self.job, self.path); - let output = job.output.as_deref().unwrap_or(path); - let mut video = job.shot().video(self.gpu, job.fps, sink).map_err(|e| Failure::at(e, output))?; - let mut any = false; - for next in frames { - let (index, frame, patches) = next?; - any = true; - let at = |error: Error| Failure::at(error.at(index).at("frames"), path); - let changed = apply(&mut self.inputs.data, &self.inputs.scenes, &frame, patches).map_err(at)?; - for i in changed { - for view in self.views.iter_mut() { - view.set_data(self.inputs.data[i].clone()); + fn run(self, sink: S) -> Result<(), Failure> { + let Sequence { job, playback: Playback { lines, span, file }, gpu, views, inputs, report } = self; + // An output frame's own errors name the frames file, or `--span` in a span. + let source = if span.is_some() { Path::new("--span") } else { file.as_path() }; + let mut video = job.shot().video(gpu, job.fps, sink)?; + for next in shown(lines, span.as_ref(), job.fps) { + let Shown { lines, at } = next?; + for (index, mut frame, patches) in lines { + apply(&inputs.data, &inputs.scenes, &frame, patches).map_err(|e| blame(&file, index, e))?; + // A span shows its own axis coordinate, and flies on from each line that sets a camera. + if at.is_some() { + frame.at = None; + } + steer(views, &frame); + if let (Some(at), Some(span)) = (at, &span) { + span.flights.start(views, index, at); } } - steer(self.views, &frame); - block(job.deadline, video.frame(&mut self.views.iter_mut().collect::>())).map_err(at)?; - self.report.progress(video.frames(), video.time()); - } - if !any { - return Err(Failure::at(Error::invalid("the frames are empty"), path)); + if let Some(at) = at { + steer(views, &Frame::showing(at)); + } + // Only invalid input is the frame's fault. An output that fails names itself. + let index = usize::try_from(video.frames()).unwrap_or(usize::MAX); + let drawn = block(job.deadline, video.frame(&mut views.iter_mut().collect::>())); + drawn.map_err(|e| if let Error::Invalid { .. } = e { blame(source, index, e) } else { e.into() })?; + report.progress(video.frames(), video.time()); } let frames = video.frames(); - block(job.deadline, video.finish()).map_err(|e| Failure::at(e, output))?.commit()?; - self.report.done(frames, self.views, self.gpu); + block(job.deadline, video.finish())?.commit()?; + report.done(frames, views, gpu); Ok(()) } } -/// Applies `frame`'s patches to the datasets they name, in order, and checks the cameras it gives -/// each view against that view's scene: the datasets it changed, by index. +/// Applies `frame`'s patches, in order, to the datasets they name. Every view draws the changes +/// from its next frame. Also checks each per-view camera against its view's scene. Returns whether +/// any dataset changed. fn apply( - data: &mut [Data], + data: &[Data], scenes: &[(PathBuf, Scene)], frame: &Frame, patches: Vec<(Name, Patch)>, -) -> latkit::Result> { - // One camera for every view looks through those it fits; one per view must fit its own. +) -> latkit::Result { + // A single camera applies to the views it fits. A per-view camera must fit its own view. for (option, each) in [("camera", &frame.camera), ("fly", &frame.fly)] { let Some(Each::Each(cameras)) = each else { continue }; for (i, camera) in cameras.iter().enumerate() { @@ -186,25 +303,22 @@ fn apply( (Some(_), None) => false, }; if !fits { - return Err(Error::invalid("the camera does not look at its view's space").at(i).at(option)); + return Err(Error::invalid("the camera does not match the space of its view").at(i).at(option)); } } } - let mut changed = Vec::new(); + let changed = !patches.is_empty(); for (name, patch) in patches { - let i = data + let dataset = data .iter() - .position(|d| *d.name() == name) + .find(|d| *d.name() == name) .ok_or_else(|| Error::invalid(format!("no dataset is named {name}")).at(name.as_str()).at("patch"))?; - data[i].apply(patch).map_err(|e| e.at(name.as_str()).at("patch"))?; - if !changed.contains(&i) { - changed.push(i); - } + dataset.apply(patch).map_err(|e| e.at(name.as_str()).at("patch"))?; } Ok(changed) } -/// Points every view where `frame` says. +/// Applies `frame`'s axis coordinate, insets, orbit, cameras, and fits to every view. fn steer(views: &mut [View], frame: &Frame) { for (i, view) in views.iter_mut().enumerate() { if let Some(at) = frame.at { @@ -220,7 +334,7 @@ fn steer(views: &mut [View], frame: &Frame) { view.set_camera(camera); } if let Some(camera) = frame.fly.as_ref().and_then(|c| c.of(i)) { - view.fly(camera); + view.fly(camera, None); } if let Some(fit) = frame.fit.as_ref().and_then(|f| f.of(i)) { view.fit(fit); @@ -228,21 +342,19 @@ fn steer(views: &mut [View], frame: &Frame) { } } -/// Reads, loads, and checks everything `render` would, without a GPU: scenes against the datasets, -/// cameras, fonts, and every frame applied in turn. +/// Reads, loads, and checks every input `render` would, without a GPU: scenes against the +/// datasets, cameras, fonts, and every frame applied in turn. pub(crate) fn validate(job: &Job) -> Result<(), Failure> { - let mut inputs = Inputs::read(job)?; - let Some(path) = &job.frames else { return Ok(()) }; - let mut frames = Frames::open(path, inputs.host.clone(), job.deadline)?; - let mut any = false; - while let Some((index, frame, patches)) = frames.next()? { - any = true; - let at = |error: Error| Failure::at(error.at(index).at("frames"), path); - if !apply(&mut inputs.data, &inputs.scenes, &frame, patches).map_err(at)?.is_empty() { + let inputs = Inputs::read(job)?; + let Some(Playback { lines, file, .. }) = playback(job, &inputs)? else { return Ok(()) }; + for next in lines { + let (index, frame, patches) = next?; + let at = |error: Error| blame(&file, index, error); + if apply(&inputs.data, &inputs.scenes, &frame, patches).map_err(at)? { for (_, scene) in &inputs.scenes { - scene.validate(&inputs.data, &inputs.tiles).map_err(at)?; + scene.validate(&inputs.sources()).map_err(at)?; } } } - if any { Ok(()) } else { Err(Failure::at(Error::invalid("the frames are empty"), path)) } + Ok(()) } diff --git a/crates/latkit-cli/src/report.rs b/crates/latkit-cli/src/report.rs index 0f8beed5..a843d614 100644 --- a/crates/latkit-cli/src/report.rs +++ b/crates/latkit-cli/src/report.rs @@ -1,7 +1,7 @@ -//! What a command tells on stderr. As JSON, one object a line, `{ "type", "detail" }` as a view's -//! events cross to JavaScript: `progress` after each frame, `stats` once done, and `error` with the -//! library's `{ code, message, path }` and the file it came from. For people, an error a line, and on -//! a terminal a line of progress that rewrites itself. +//! Diagnostics on stderr. As JSON, one `{ "type", "detail" }` object per line, the shape of a +//! view's events in JavaScript: `progress` after each frame, `stats` when done, and `error` with +//! the library's `{ code, message, path }` and the source file. For people, one line per error, +//! and on a terminal a progress line that rewrites itself. use std::io::{self, IsTerminal, Write}; use std::time::Instant; @@ -14,10 +14,10 @@ use crate::args::Diagnostics; pub(crate) struct Report { json: bool, - /// Whether people read stderr on a terminal: then progress shows. + /// Whether stderr is a terminal, where progress is shown. terminal: bool, started: Instant, - /// Whether a progress line is showing. + /// Whether a progress line is on screen. showing: bool, } @@ -31,7 +31,7 @@ impl Report { } } - /// `frames` drawn, the next showing at `time` seconds. + /// Reports `frames` drawn, with the next at `time` seconds of video. pub(crate) fn progress(&mut self, frames: u64, time: f64) { if self.json { Report::line(&json!({"type": "progress", "detail": {"frames": frames, "time": time}})); @@ -42,7 +42,7 @@ impl Report { } } - /// What was drawn, and what it took. + /// Reports the frames drawn and the time, stats, and GPU work they took. pub(crate) fn done(&mut self, frames: u64, views: &[View], gpu: &Gpu) { let seconds = self.started.elapsed().as_secs_f64(); if self.json { @@ -59,7 +59,7 @@ impl Report { } } - /// Why the command failed. + /// Reports why the command failed. pub(crate) fn error(&mut self, failure: &Failure) { if self.json { let mut detail = serde_json::to_value(&failure.error).unwrap_or(Value::Null); diff --git a/crates/latkit-cli/src/sinks.rs b/crates/latkit-cli/src/sinks.rs index e790e030..3b185ce6 100644 --- a/crates/latkit-cli/src/sinks.rs +++ b/crates/latkit-cli/src/sinks.rs @@ -1,5 +1,6 @@ -//! Where pictures go: one output written whole or not at all, and the sinks a video's frames go to -//! besides the library's MP4: image files, raw RGBA, and YUV4MPEG2. +//! Output destinations: an output written all at once or not at all, and the video sinks other than +//! the library's MP4: image files, raw RGBA, and YUV4MPEG2. An output's errors name the output in +//! their message, whoever writes it, since they reach the command through the library's video. use std::collections::VecDeque; use std::fs::File; @@ -10,17 +11,23 @@ use std::path::{Path, PathBuf}; use latkit::host::{Host, Job, Native}; use latkit::{Encoding, Error, Image, ImageFormat, Mp4, Pending, Pixels, Sink}; -use crate::Failure; use crate::args::{is_stdio, pattern}; -/// Image files encoding at once, on the host's workers. +/// The most image files encoding at once on the host's workers. const ENCODING: usize = 4; -fn written(error: &io::Error, path: &Path) -> Error { - Error::Io(format!("{}: {error}", path.display()).into()) +/// `error`, its message naming the output `path`: a file, or stdout for `-`. +fn named(error: &io::Error, path: &Path) -> io::Error { + let name = if is_stdio(path) { "stdout".to_owned() } else { path.display().to_string() }; + io::Error::new(error.kind(), format!("{name}: {error}")) } -/// Bytes to stdout, or to a file that replaces `path` only once every byte is written. +/// The library's I/O error for failing to write the output `path`. +pub(crate) fn written(error: &io::Error, path: &Path) -> Error { + Error::Io(named(error, path).to_string().into()) +} + +/// Bytes to stdout, or to a temporary file that replaces `path` only once every byte is written. pub(crate) enum Output { Stdout(BufWriter), File { path: PathBuf, file: BufWriter, temporary: tempfile::TempPath }, @@ -28,25 +35,24 @@ pub(crate) enum Output { impl Output { /// An output to `path`, or stdout for `-`. - pub(crate) fn create(path: &Path) -> Result { + pub(crate) fn create(path: &Path) -> latkit::Result { if is_stdio(path) { return Ok(Output::Stdout(BufWriter::new(io::stdout()))); } let parent = path.parent().filter(|p| !p.as_os_str().is_empty()).unwrap_or(Path::new(".")); let suffix = path.extension().map(|e| format!(".{}", e.to_string_lossy())).unwrap_or_default(); - let temporary = - tempfile::Builder::new().suffix(&suffix).tempfile_in(parent).map_err(|e| Failure::io(&e, path))?; + let temporary = tempfile::Builder::new().suffix(&suffix).tempfile_in(parent).map_err(|e| written(&e, path))?; let (file, temporary) = temporary.into_parts(); Ok(Output::File { path: path.to_path_buf(), file: BufWriter::new(file), temporary }) } - /// Writes what is left and, for a file, replaces its destination. - pub(crate) fn commit(self) -> Result<(), Failure> { + /// Flushes the remaining bytes and, for a file, moves it into place at its destination. + pub(crate) fn commit(self) -> latkit::Result<()> { match self { - Output::Stdout(mut out) => out.flush().map_err(|e| Failure::io(&e, Path::new("-"))), + Output::Stdout(mut out) => out.flush().map_err(|e| written(&e, Path::new("-"))), Output::File { path, file, temporary } => { - drop(file.into_inner().map_err(|e| Failure::io(e.error(), &path))?); - temporary.persist(&path).map_err(|e| Failure::io(&e.error, &path)) + drop(file.into_inner().map_err(|e| written(e.error(), &path))?); + temporary.persist(&path).map_err(|e| written(&e.error, &path)) } } } @@ -58,66 +64,85 @@ impl Output { } } - /// Writes `bytes`, naming the destination in its error. - fn put(&mut self, bytes: &[u8]) -> latkit::Result<()> { - self.write_all(bytes).map_err(|e| written(&e, self.path())) + /// Writes `bytes`. + pub(crate) fn put(&mut self, bytes: &[u8]) -> latkit::Result<()> { + self.write_all(bytes).map_err(|e| Error::Io(e.to_string().into())) + } + + /// Writes `value` as JSON, indented if `pretty`, and a newline. + pub(crate) fn json(&mut self, value: &impl serde::Serialize, pretty: bool) -> latkit::Result<()> { + let wrote = if pretty { + serde_json::to_writer_pretty(&mut *self, value) + } else { + serde_json::to_writer(&mut *self, value) + }; + wrote.map_err(|e| Error::Io(e.to_string().into()))?; + self.put(b"\n") + } + + /// Flushes the bytes written so far. + fn flushed(&mut self) -> latkit::Result<()> { + self.flush().map_err(|e| Error::Io(e.to_string().into())) } } +/// Writes whose errors name the output. impl Write for Output { fn write(&mut self, bytes: &[u8]) -> io::Result { - match self { + let wrote = match self { Output::Stdout(out) => out.write(bytes), Output::File { file, .. } => file.write(bytes), - } + }; + wrote.map_err(|e| named(&e, self.path())) } fn flush(&mut self) -> io::Result<()> { - match self { + let flushed = match self { Output::Stdout(out) => out.flush(), Output::File { file, .. } => file.flush(), - } + }; + flushed.map_err(|e| named(&e, self.path())) } } -/// What a sink leaves once its video is finished: its output, written and in place. +/// Finalizes a sink's output once its video is finished, leaving it written and in place. pub(crate) trait Commit { - fn commit(self) -> Result<(), Failure>; + fn commit(self) -> latkit::Result<()>; } impl Commit for Mp4 { - fn commit(self) -> Result<(), Failure> { + fn commit(self) -> latkit::Result<()> { self.into_inner().commit() } } -/// Pictures as image files, encoded on the host's workers a few at once: one a frame, numbered by -/// the pattern of the output's name, each file written whole and in order, a held frame's file the -/// frame before's; or one file, of frames that are one. +/// Pictures as image files, encoded a few at a time on the host's workers. Either one file per +/// frame, numbered by the output's pattern and each written complete and in order, or one file for +/// a single frame. A held frame (where nothing moved) repeats the previous frame's file. pub(crate) struct Images { to: To, format: ImageFormat, host: Native, size: [u32; 2], - /// Frames under way, oldest first: each frame's index, and a drawn frame's file. + /// Frames in progress, oldest first: each frame's index and, if drawn, its file being encoded. writing: VecDeque<(u64, Option>>)>, /// The last drawn frame's file. last: Vec, } -/// Where image files go. +/// Where image files are written. enum To { Numbered(PathBuf), One(Output), } impl Images { - /// A file a frame, at the paths `pattern` numbers. + /// One file per frame, at the paths `pattern` numbers. pub(crate) fn numbered(pattern: PathBuf, format: ImageFormat, host: Native) -> Images { Images::new(To::Numbered(pattern), format, host) } - /// One file, to `out`. + /// A single file, written to `out`. pub(crate) fn one(out: Output, format: ImageFormat, host: Native) -> Images { Images::new(To::One(out), format, host) } @@ -126,10 +151,10 @@ impl Images { Images { to, format, host, size: [0, 0], writing: VecDeque::new(), last: Vec::new() } } - /// The path of frame `index` where `names` numbers the frames. + /// The path of frame `index`, from the numbered pattern `names`. fn path(names: &Path, index: u64) -> PathBuf { let text = names.to_string_lossy(); - let (range, width) = pattern(names).expect("a pattern"); + let (range, width) = pattern(names).expect("the output is a %d pattern"); format!("{}{index:0width$}{}", &text[..range.start], &text[range.end..]).into() } @@ -139,15 +164,14 @@ impl Images { if let Some(file) = file { self.last = file.await?; if self.last.is_empty() { - return Err(Error::invalid(format!("the picture could not be written as {}", self.format))); + return Err(Error::invalid(format!("the picture could not be encoded as {}", self.format))); } } match &mut self.to { To::Numbered(names) => { - let path = Images::path(names, index); - let mut out = Output::create(&path).map_err(|f| f.error)?; + let mut out = Output::create(&Images::path(names, index))?; out.put(&self.last)?; - out.commit().map_err(|f| f.error) + out.commit() } To::One(out) => out.put(&self.last), } @@ -164,7 +188,7 @@ impl Sink for Images { async fn frame(&mut self, index: u64, pixels: Option>) -> latkit::Result<()> { if index > 0 && matches!(self.to, To::One(_)) { - return Err(Error::invalid("an image file is one frame; a %d pattern in the output writes one a frame")); + return Err(Error::invalid("an image file holds one frame; use a %d pattern to write one per frame")); } let file = match pixels { Some(rgba) => Some(self.host.spawn(Job::encode(&Image::new(self.size, rgba)?, self.format))), @@ -179,7 +203,7 @@ impl Sink for Images { async fn finish(&mut self, end: u64) -> latkit::Result<()> { if end == 0 { - return Err(Error::invalid("the frames are empty")); + return Err(Error::invalid("no frames to write")); } while !self.writing.is_empty() { self.write_oldest().await?; @@ -189,7 +213,7 @@ impl Sink for Images { } impl Commit for Images { - fn commit(self) -> Result<(), Failure> { + fn commit(self) -> latkit::Result<()> { match self.to { To::Numbered(_) => Ok(()), To::One(out) => out.commit(), @@ -197,7 +221,7 @@ impl Commit for Images { } } -/// Raw RGBA, straight alpha, frames back to back; a held frame is the frame before again. +/// Raw RGBA with straight alpha, frames back to back. A held frame repeats the previous frame. pub(crate) struct Raw { out: Output, last: Vec, @@ -220,22 +244,22 @@ impl Sink for Raw { } fn finish(&mut self, _: u64) -> impl Future> { - ready(self.out.flush().map_err(|e| written(&e, self.out.path()))) + ready(self.out.flushed()) } } impl Commit for Raw { - fn commit(self) -> Result<(), Failure> { + fn commit(self) -> latkit::Result<()> { self.out.commit() } } -/// YUV4MPEG2: a header, then each frame's planes, 4:2:0 from the GPU's NV12. A held frame is the -/// frame before again. +/// YUV4MPEG2: a header, then each frame's 4:2:0 planes, converted from the GPU's NV12. A held frame +/// repeats the previous frame. pub(crate) struct Y4m { out: Output, size: [u32; 2], - /// The last frame written, `FRAME` and planes. + /// The last frame written: its `FRAME` marker and planes. last: Vec, } @@ -251,7 +275,7 @@ impl Sink for Y4m { fn begin(&mut self, encoding: &Encoding, _: Option<&dyn Host>) -> latkit::Result<()> { self.size = encoding.size; let ([width, height], [num, den]) = (encoding.size, encoding.rate()); - // The GPU averages each 2×2 block's chroma: centered, as JPEG sites it. + // The GPU averages each 2×2 block's chroma, so samples are centered, as `C420jpeg` says. let header = format!("YUV4MPEG2 W{width} H{height} F{num}:{den} Ip A1:1 C420jpeg XCOLORRANGE=LIMITED\n"); self.out.put(header.as_bytes()) } @@ -270,12 +294,12 @@ impl Sink for Y4m { } fn finish(&mut self, _: u64) -> impl Future> { - ready(self.out.flush().map_err(|e| written(&e, self.out.path()))) + ready(self.out.flushed()) } } impl Commit for Y4m { - fn commit(self) -> Result<(), Failure> { + fn commit(self) -> latkit::Result<()> { self.out.commit() } } diff --git a/crates/latkit-cli/tests/cli.rs b/crates/latkit-cli/tests/cli.rs index ca6c84dc..6b9f8102 100644 --- a/crates/latkit-cli/tests/cli.rs +++ b/crates/latkit-cli/tests/cli.rs @@ -1,5 +1,5 @@ -//! The executable's contracts: files, pipes, exit codes, diagnostics, and every output a render -//! writes, from a still to a video. +//! Tests of the executable's behavior: files, pipes, exit codes, diagnostics, and every output a +//! render writes, from a still to a video. use std::io::Write; use std::path::Path; @@ -8,7 +8,7 @@ use std::process::{Command, Output, Stdio}; use latkit::Image; const SCENE: &str = - r#"{"space":"plane","ease":0,"marks":[{"vertices":{"of":["grid","Bus"],"position":["x","y"],"radius":6}}]}"#; + r#"{"space":"plane","ease":0,"marks":[{"vertices":["grid","Bus"],"position":["$x","$y"],"radius":6}]}"#; const DATA: &str = r#"{"name":"grid","patch":[{"replace":{"table":"Bus","ids":[10,20],"fields":{"x":[-10,10],"y":[0,0],"name":["Alpha","Beta"]}}}]}"#; const CAMERA: &str = r#"{"center":[0,0],"zoom":2,"pitch":0,"bearing":0}"#; @@ -36,7 +36,7 @@ fn fixture() -> tempfile::TempDir { dir } -/// Whether the machine has no GPU, which only `LATKIT_REQUIRE_GPU` makes a failure. +/// Whether the run failed for lack of a GPU. With `LATKIT_REQUIRE_GPU` set, that fails the test. fn no_gpu(output: &Output) -> bool { let stderr = String::from_utf8_lossy(&output.stderr); let missing = !output.status.success() && (stderr.contains("no GPU adapter") || stderr.contains("no GPU device")); @@ -48,7 +48,7 @@ fn ok(output: &Output) { assert!(output.status.success(), "{}", String::from_utf8_lossy(&output.stderr)); } -/// Writes frames: `x` positions of the two buses, as patches of slices of one binary file. +/// Writes a frames file whose lines set the two buses' `x` from slices of one binary file. fn frames(dir: &Path, xs: &[[f32; 2]]) { let bytes: Vec = xs.iter().flatten().flat_map(|x| x.to_le_bytes()).collect(); std::fs::create_dir_all(dir.join("sequence")).unwrap(); @@ -90,12 +90,16 @@ fn usage_errors_exit_2_before_reading_or_drawing() { vec!["render", "missing.json", "-o", "out.png", "--elapsed", "1e100"], vec!["render", "missing.json", "-o", "out.png", "--deadline", "0"], vec!["render", "missing.json", "-o", "out.gif"], - vec!["render", "missing.json", "-o", "out.mp4"], - vec!["render", "missing.json", "--frames", "f.jsonl", "-o", "out.mp4", "--size", "62x64"], + vec!["render", "missing.json", "-o", "out.mp4", "--span", "2,1"], + vec!["render", "missing.json", "-o", "out.png", "--span", "0,1"], + vec!["render", "missing.json", "-o", "out.mp4", "--at", "1"], + vec!["render", "missing.json", "-o", "out.mp4", "--span", "0,1", "--elapsed", "1"], + vec!["render", "missing.json", "--frames", "f.jsonl", "-o", "out.mp4", "--rate", "2"], + vec!["render", "missing.json", "-o", "out.mp4", "--rate", "0"], + vec!["render", "missing.json", "--frames", "f.jsonl", "-o", "out.mp4", "--size", "3x64"], vec!["render", "missing.json", "--frames", "f.jsonl", "-o", "out.mp4", "--transparent"], vec!["render", "missing.json", "--frames", "f.jsonl", "-o", "out.mp4", "--fps", "0"], vec!["render", "missing.json", "--frames", "f.jsonl", "-o", "out/%04d.y4m"], - vec!["render", "missing.json", "-o", "out/%04d.png"], vec!["render", "a.json", "b.json", "-o", "out.png"], vec!["render", "a.json", "-o", "out.png", "--tiles", "base.pmtiles"], vec!["validate", "-", "-d", "-"], @@ -118,7 +122,7 @@ fn validation_reads_everything_without_a_gpu_and_is_silent_on_success() { ok(&output); assert!(output.stdout.is_empty() && output.stderr.is_empty()); } - // A frame naming no dataset fails at its line, as JSON when asked. + // A frame that patches a missing dataset fails at its line, reported as JSON when requested. let bad = r#"{"at":0,"patch":{"absent":[{"forget":{"before":0}}]}}"#; std::fs::write(dir.path().join("sequence/bad.jsonl"), format!("{{\"at\":0}}\n{bad}")).unwrap(); let args = ["--diagnostics", "json", "validate", "scene.json", "-d", "grid.json", "--frames", "sequence/bad.jsonl"]; @@ -128,37 +132,43 @@ fn validation_reads_everything_without_a_gpu_and_is_silent_on_success() { assert_eq!(told["type"], "error"); assert_eq!(told["detail"]["code"], "invalid"); assert_eq!(told["detail"]["path"], "frames[1].patch.absent"); - // A camera for each view looks at its own view's space: a plot's cannot look at a plane. + // A per-view camera must fit its own view's space: a plot camera cannot look at a plane. std::fs::write(dir.path().join("sequence/bad.jsonl"), r#"{"camera": [{"x": [0, 1]}]}"#).unwrap(); let output = run(dir.path(), &args, None); let told: serde_json::Value = serde_json::from_slice(&output.stderr).unwrap(); assert_eq!(told["detail"]["path"], "frames[0].camera[0]"); assert!(told["detail"]["source"].as_str().unwrap().ends_with("bad.jsonl")); + // `--camera` takes JSON or a file of it, as a query does. + ok(&run(dir.path(), &["validate", "scene.json", "-d", "grid.json", "--camera", CAMERA], None)); + let plot = r#"{"x": [0, 1], "y": [0, 1]}"#; + let args = ["--diagnostics", "json", "validate", "scene.json", "-d", "grid.json", "--camera", plot]; + let told: serde_json::Value = serde_json::from_slice(&run(dir.path(), &args, None).stderr).unwrap(); + assert_eq!(told["detail"]["path"], "--camera[0]"); } #[test] -fn failures_name_the_part_and_the_file_and_keep_the_output() { +fn failures_report_path_and_source_and_leave_existing_output_intact() { let dir = fixture(); let output = run(dir.path(), &["--diagnostics", "json", "validate", "scene.json"], None); assert_eq!(output.status.code(), Some(1)); let told: serde_json::Value = serde_json::from_slice(&output.stderr).unwrap(); - assert_eq!(told["detail"]["path"], "marks[0].vertices.of"); + assert_eq!(told["detail"]["path"], "marks[0].vertices"); assert_eq!(told["detail"]["source"], "scene.json"); let twice = run(dir.path(), &["validate", "scene.json", "-d", "grid.json", "-d", "grid.json"], None); - assert!(String::from_utf8_lossy(&twice.stderr).contains("duplicate")); + assert!(String::from_utf8_lossy(&twice.stderr).contains("two sources named 'grid'")); std::fs::write(dir.path().join("out.png"), b"keep this").unwrap(); assert_eq!(run(dir.path(), &["render", "scene.json", "-o", "out.png"], None).status.code(), Some(1)); assert_eq!(std::fs::read(dir.path().join("out.png")).unwrap(), b"keep this"); - // A legend shows a color mapped through a colormap with a domain or named categories. - let legend = r#"{"space":"plane","marks":[{"vertices":{"of":["grid","Bus"],"legend":"Buses"}}]}"#; + // A legend needs a color mapped through a colormap with a domain or named categories. + let legend = r#"{"space":"plane","marks":[{"vertices":["grid","Bus"],"legend":"Buses"}]}"#; let output = run(dir.path(), &["--diagnostics", "json", "validate", "-", "-d", "grid.json"], Some(legend.as_bytes())); let told: serde_json::Value = serde_json::from_slice(&output.stderr).unwrap(); - assert_eq!(told["detail"]["path"], "marks[0].vertices.legend"); + assert_eq!(told["detail"]["path"], "marks[0].legend"); } #[test] -fn inspect_describes_datasets_and_geojson_takes_a_name() { +fn inspect_describes_datasets_and_geojson_needs_a_name() { let dir = fixture(); let geo = r#"{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"n":1},"geometry":{"type":"Point","coordinates":[1,2]}}]}"#; std::fs::write(dir.path().join("places.geojson"), geo).unwrap(); @@ -189,10 +199,9 @@ fn schema_describes_every_input_and_frames() { } #[test] -fn query_writes_a_json_line_a_row() { +fn query_writes_one_json_line_per_row() { let dir = fixture(); - let query = - r#"{"of":"Bus","sort":[["x","descending"]],"select":{"name":"name","twice":{"op":"mul","args":["x",2]}}}"#; + let query = r#"{"of":"Bus","sort":[["$x","descending"]],"select":{"name":"$name","twice":{"mul":["$x",2]}}}"#; let output = run(dir.path(), &["query", "grid.json", query], None); ok(&output); let lines: Vec = @@ -204,7 +213,7 @@ fn query_writes_a_json_line_a_row() { serde_json::json!({ "id": 10, "name": "Alpha", "twice": -20.0 }), ] ); - std::fs::write(dir.path().join("query.json"), r#"{"of":"Bus","filter":{"op":"lt","args":["x",0]}}"#).unwrap(); + std::fs::write(dir.path().join("query.json"), r#"{"of":"Bus","filter":{"lt":["$x",0]}}"#).unwrap(); let output = run(dir.path(), &["query", "-", "query.json"], Some(DATA.as_bytes())); ok(&output); assert_eq!(String::from_utf8(output.stdout).unwrap().lines().count(), 1); @@ -214,7 +223,7 @@ fn query_writes_a_json_line_a_row() { } #[test] -fn a_history_tiles_and_draws_from_its_spans_as_from_itself() { +fn a_tiled_history_draws_the_same_as_the_whole_dataset() { let dir = tempfile::tempdir().unwrap(); let at: Vec = (0..2000).map(|t| t.to_string()).collect(); let v: Vec = (0..2000).map(|t| format!("{:.4}", 0.5 + 0.4 * (f64::from(t) / 40.0).sin())).collect(); @@ -224,7 +233,7 @@ fn a_history_tiles_and_draws_from_its_spans_as_from_itself() { v.join(",") ); std::fs::write(dir.path().join("signals.json"), data).unwrap(); - let scene = r##"{"space":"plot","ease":0,"marks":[{"lines":{"of":["signals","Signal"],"trace":"v","color":"#ff0000","width":2}}]}"##; + let scene = r##"{"space":"plot","ease":0,"marks":[{"lines":["signals","Signal"],"trace":"v","color":"#ff0000","width":2}]}"##; std::fs::write(dir.path().join("plot.json"), scene).unwrap(); std::fs::write(dir.path().join("camera.json"), r#"{"x":[100,300],"y":[0,1]}"#).unwrap(); ok(&run(dir.path(), &["tile", "signals.json", "-o", "tiles", "--chunk", "64"], None)); @@ -246,13 +255,13 @@ fn a_history_tiles_and_draws_from_its_spans_as_from_itself() { let pixels = |o: &Output| o.stdout.as_chunks::<4>().0.to_vec(); let differing = pixels(&whole).iter().zip(pixels(&tiled)).filter(|(a, b)| a[0].abs_diff(b[0]) > 8).count(); assert!(differing < 32, "{differing} pixels differ"); - assert!(pixels(&tiled).iter().any(|p| p[0] > 200 && p[1] < 60), "the trace draws"); + assert!(pixels(&tiled).iter().any(|p| p[0] > 200 && p[1] < 60), "the trace is drawn"); } #[test] fn a_labeled_still_renders_to_a_file_and_to_stdout() { let dir = fixture(); - let scene = r#"{"space":"diagram","layout":{"layered":{}},"title":"Grid","marks":[{"vertices":{"of":["grid","Bus"],"label":"name"}}]}"#; + let scene = r#"{"space":"diagram","layout":{"layered":{}},"title":"Grid","marks":[{"vertices":["grid","Bus"],"label":"$name"}]}"#; std::fs::write(dir.path().join("scene.json"), scene).unwrap(); std::fs::write(dir.path().join("out.png"), b"replace me").unwrap(); let args = ["render", "scene.json", "-d", "grid.json", "-o", "out.png", "--size", "127x129", "--transparent"]; @@ -275,7 +284,7 @@ fn a_labeled_still_renders_to_a_file_and_to_stdout() { for (file, starts) in [("out.jpg", &[0xff, 0xd8, 0xff][..]), ("out.webp", &b"RIFF"[..])] { args[5] = file; ok(&run(dir.path(), &args, None)); - assert!(std::fs::read(dir.path().join(file)).unwrap().starts_with(starts), "{file} is what it says"); + assert!(std::fs::read(dir.path().join(file)).unwrap().starts_with(starts), "{file} has its format's signature"); } } @@ -310,9 +319,9 @@ fn panels_draw_a_scene_each() { } #[test] -fn a_frame_stream_writes_every_output_and_holds_still_frames() { +fn a_frame_stream_writes_every_output_and_repeats_unchanged_frames() { let dir = fixture(); - // The third frame changes nothing: it is held, and a raw stream repeats the second. + // The third frame changes nothing, so it is held and the raw stream repeats the second. frames(dir.path(), &[[-4.0, 4.0], [0.0, 8.0], [0.0, 8.0]]); let mut args = COMMON.to_vec(); args.extend(["--format", "rgba", "-o", "-"]); @@ -327,7 +336,7 @@ fn a_frame_stream_writes_every_output_and_holds_still_frames() { assert_ne!(first, second); assert_eq!(second, third); - // A sequence never truncates itself into one PNG, nor destroys an old file. + // Several frames to a single PNG fail, and the existing file stays intact. std::fs::write(dir.path().join("out.png"), b"keep").unwrap(); let mut args = COMMON.to_vec(); args.extend(["-o", "out.png"]); @@ -347,6 +356,13 @@ fn a_frame_stream_writes_every_output_and_holds_still_frames() { |n: u32| Image::from_png(&std::fs::read(dir.path().join(format!("pngs/frame-{n:03}.png"))).unwrap()).unwrap(); assert_eq!(&png(0).rgba[..], first); assert_eq!(png(1), png(2)); + // A file that cannot be written is named in the error, which is not the frames file's fault. + let mut args = [&["--diagnostics", "json"][..], &COMMON].concat(); + args.extend(["-o", "absent/frame-%03d.png"]); + let stderr = String::from_utf8(run(dir.path(), &args, None).stderr).unwrap(); + let told: serde_json::Value = serde_json::from_str(stderr.lines().last().unwrap()).unwrap(); + assert_eq!((&told["detail"]["code"], &told["detail"]["source"]), (&"io".into(), &serde_json::Value::Null)); + assert!(told["detail"]["message"].as_str().unwrap().contains("absent/frame-000.png"), "{told}"); let mut args = COMMON.to_vec(); args.extend(["-o", "out.y4m"]); @@ -355,6 +371,11 @@ fn a_frame_stream_writes_every_output_and_holds_still_frames() { let header = b"YUV4MPEG2 W64 H64 F60:1 Ip A1:1 C420jpeg XCOLORRANGE=LIMITED\n"; assert!(y4m.starts_with(header)); assert_eq!(y4m.len(), header.len() + 3 * (6 + 64 * 64 * 3 / 2)); + // A video rounds its size down to whole 4×2 blocks. + args[8] = "--size=67x65"; + args[10] = "odd.y4m"; + ok(&run(dir.path(), &args, None)); + assert!(std::fs::read(dir.path().join("odd.y4m")).unwrap().starts_with(header)); #[cfg(windows)] { @@ -367,6 +388,80 @@ fn a_frame_stream_writes_every_output_and_holds_still_frames() { let absent = "{\"at\":0}\n{\"at\":0,\"patch\":{\"absent\":[]}}"; std::fs::write(dir.path().join("sequence/frames.jsonl"), absent).unwrap(); assert_eq!(run(dir.path(), &args, None).status.code(), Some(1)); - assert_eq!(std::fs::read(dir.path().join("out.mp4")).unwrap(), mp4, "a failed video keeps the file before"); + assert_eq!(std::fs::read(dir.path().join("out.mp4")).unwrap(), mp4, "a failed video leaves the file intact"); + } +} + +#[test] +fn a_span_plays_frames_at_their_at_and_flies_between_their_cameras() { + let dir = fixture(); + let camera = |x: f64| format!(r#"{{"center":[{x},0],"zoom":2,"pitch":0,"bearing":0}}"#); + let lines = + [format!(r#"{{"at":0,"camera":{}}}"#, camera(0.0)), format!(r#"{{"at":0.05,"camera":{}}}"#, camera(8.0))]; + std::fs::write(dir.path().join("span.jsonl"), lines.join("\n")).unwrap(); + let common = ["scene.json", "-d", "grid.json", "--size=64x64", "--format", "rgba", "-o", "-"]; + let args = [&["render", "--frames", "span.jsonl", "--span", "0,0.05"][..], &common].concat(); + let output = run(dir.path(), &args, None); + if no_gpu(&output) { + return; + } + ok(&output); + // 0.05 axis units at the default rate of 1 and 60 fps: frames at 0, 1/60, 2/60, and 3/60. + let size = 64 * 64 * 4; + let frames: Vec<&[u8]> = output.stdout.chunks(size).collect(); + assert_eq!(frames.len(), 4); + assert!(frames[0] != frames[1] && frames[1] != frames[2], "the camera flies between the lines"); + // The last frame reaches the last line, so it shows that line's camera, as a still with it does. + let last = camera(8.0); + let still = run(dir.path(), &[&["render", "--camera", last.as_str()][..], &common].concat(), None); + ok(&still); + assert_eq!(frames[3], &still.stdout[..], "arrives on the last line"); + + // Each line needs `at`, in order, and its patches name datasets that exist. + for (lines, path) in [ + ([r#"{"at":0}"#, r#"{"orbit":1}"#], "frames[1].at"), + ([r#"{"at":0.02}"#, r#"{"at":0.01}"#], "frames[1].at"), + ([r#"{"at":0}"#, r#"{"at":0.01,"patch":{"absent":[]}}"#], "frames[1].patch"), + ] { + std::fs::write(dir.path().join("bad.jsonl"), lines.join("\n")).unwrap(); + let args = + ["validate", "scene.json", "-d", "grid.json", "--frames", "bad.jsonl", "--span", "0,1", "-o", "x.y4m"]; + let output = run(dir.path(), &args, None); + assert_eq!(output.status.code(), Some(1)); + assert!( + String::from_utf8_lossy(&output.stderr).contains(path), + "{path}: {}", + String::from_utf8_lossy(&output.stderr) + ); } } + +#[test] +fn a_video_without_frames_plays_the_datasets_span_at_its_rate() { + let dir = fixture(); + let sampled = r#"{"name":"grid","patch":[ + {"replace":{"table":"Bus","ids":[10,20],"fields":{"x":[-10,10],"y":[0,0]}}}, + {"axis":{"name":"time","unit":"s"}}, + {"append":{"table":"Bus","at":[0,0.05],"fields":{"v":[1,2,3,4]}}}]}"#; + std::fs::write(dir.path().join("sampled.json"), sampled).unwrap(); + let header = b"YUV4MPEG2 W64 H64 F60:1 Ip A1:1 C420jpeg XCOLORRANGE=LIMITED\n"; + let frames = |extra: &[&str]| { + let args = + [&["render", "scene.json", "-d", "sampled.json", "--size=64x64", "-o", "out.y4m"][..], extra].concat(); + let output = run(dir.path(), &args, None); + (!no_gpu(&output)).then(|| { + ok(&output); + let y4m = std::fs::read(dir.path().join("out.y4m")).unwrap(); + (y4m.len() - header.len()) / (6 + 64 * 64 * 3 / 2) + }) + }; + // The span is 0 to 0.05: four frames at a rate of 1, and seven at half that. + if let Some(count) = frames(&[]) { + assert_eq!(count, 4); + assert_eq!(frames(&["--rate", "0.5"]), Some(7)); + } + // Without a span, a video has nothing to play. + let output = run(dir.path(), &["render", "scene.json", "-d", "grid.json", "--size=64x64", "-o", "none.y4m"], None); + assert_eq!(output.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&output.stderr).contains("nothing to play")); +} diff --git a/crates/latkit-js/src/canvas.rs b/crates/latkit-js/src/canvas.rs index a3810012..973afea8 100644 --- a/crates/latkit-js/src/canvas.rs +++ b/crates/latkit-js/src/canvas.rs @@ -1,27 +1,41 @@ -//! Canvases: a canvas shows views in panels. +//! Canvases, which show views in the panels of their layout. -use latkit::web::PageView; +use latkit::web::{PageView, Panel}; use wasm_bindgen::prelude::*; use crate::Error; -/// A view, for a canvas to show or a shot to draw: what `view.handle()` gives. +/// A view handle from `view.handle()`, for a canvas to show or a shot to draw. #[wasm_bindgen] pub struct Handle(pub(crate) PageView); -/// The views `handles` name. -pub(crate) fn views_of(handles: Vec) -> Vec { - handles.into_iter().map(|h| h.0).collect() +/// Pairs each of `views` with its panel. Canvases and shots both encode their layout this way. `at` +/// holds four fractions per view, `[x, y, width, height]`, in order. Fails unless `at` has four +/// values per view. +pub(crate) fn layout(views: Vec, at: &[f32]) -> Result, Error> { + if at.len() != views.len() * 4 { + return Err(latkit::Error::invalid("expected four fractions per view").at("layout").into()); + } + let (ats, _) = at.as_chunks::<4>(); + Ok(views.into_iter().zip(ats).map(|(view, at)| Panel { view: view.0, at: *at }).collect()) } -/// Views shown in the panels of a `` or an `OffscreenCanvas`; freeing it stops drawing them. +/// Views shown in a `` or an `OffscreenCanvas`, in the panels of its layout. Freeing it +/// stops drawing them. #[wasm_bindgen] pub struct Canvas(pub(crate) latkit::web::Canvas); #[wasm_bindgen] impl Canvas { - /// Shows `views`, one per panel, in place of those it showed. - pub fn show(&self, views: Vec) -> Result<(), Error> { - Ok(self.0.show(&views_of(views))?) + /// Replaces the shown views with `views`, each at its four fractions of `at`. + #[wasm_bindgen(js_name = setLayout)] + pub fn set_layout(&self, views: Vec, at: &[f32]) -> Result<(), Error> { + Ok(self.0.set_layout(layout(views, at)?)?) + } + + /// The canvas's size in CSS pixels, `[width, height]`. + #[wasm_bindgen(getter, js_name = cssSize)] + pub fn css_size(&self) -> Vec { + self.0.css_size().to_vec() } } diff --git a/crates/latkit-js/src/convert.rs b/crates/latkit-js/src/convert.rs index 40c03513..b4dff35f 100644 --- a/crates/latkit-js/src/convert.rs +++ b/crates/latkit-js/src/convert.rs @@ -1,15 +1,12 @@ -//! latkit's rows and ids to JavaScript, and ids back: a query's columns cross as typed arrays and -//! arrays, built here once; everything else crosses as bytes or JSON. +//! Conversion of latkit rows and ids to JavaScript, and of ids back. A query's columns cross as +//! typed arrays and arrays, built here once. Everything else crosses as bytes or JSON. use js_sys::{Array, Float64Array, Object, Reflect}; use latkit::Error; use wasm_bindgen::{JsCast, JsValue}; -/// The largest whole number a JavaScript number holds exactly: 2^53 - 1. -const SAFE: f64 = 9_007_199_254_740_991.0; - -/// A value for JavaScript: a number, boolean, or text; a geometry as nested `[x, y]` arrays; a row a -/// reference names as its id. +/// `value` as a JavaScript value. Numbers, booleans, and text convert directly. A geometry becomes +/// nested `[x, y]` arrays, and a referenced row becomes its id. fn value(value: &latkit::Value) -> JsValue { let xy = |p: &[f64; 2]| -> JsValue { Array::of2(&p[0].into(), &p[1].into()).into() }; let path = |p: &Vec<[f64; 2]>| -> JsValue { p.iter().map(xy).collect::().into() }; @@ -29,9 +26,11 @@ fn value(value: &latkit::Value) -> JsValue { } } -/// `table` as a query answers it: `{ table, ids, fields, total }`, numbers as a `Float64Array` -/// (missing NaN), other values as arrays (missing `null`), a reference as `{ reference, ids }`. -pub(crate) fn table(table: &latkit::Table, total: usize) -> JsValue { +/// A query result as `{ table, ids, fields, total }`. A number column becomes a `Float64Array` +/// (missing values are NaN), a reference column `{ reference, ids }`, and any other column an array +/// (missing values are `null`). +pub(crate) fn query_result(read: &latkit::QueryResult) -> JsValue { + let table = &read.table; let fields = Object::create(JsValue::NULL.unchecked_ref::()); for field in table.describe().fields { let values = table.values(&field.name).unwrap_or_else(|| latkit::Values::from(Vec::::new())); @@ -49,16 +48,16 @@ pub(crate) fn table(table: &latkit::Table, total: usize) -> JsValue { ("table", JsValue::from_str(table.name())), ("ids", ids.into()), ("fields", fields.into()), - ("total", (total as f64).into()), + ("total", (read.total as f64).into()), ]) } -/// Ids held as values, missing `null`. +/// A column of ids as an array, with `null` for missing ids. fn ids_value(values: &latkit::Values) -> JsValue { (0..values.len()).map(|i| values.id(i).map_or(JsValue::NULL, |id| id_value(&id))).collect::().into() } -/// An object of `entries`. +/// A JavaScript object with `entries` as its properties. pub(crate) fn object(entries: &[(&str, JsValue)]) -> JsValue { let object = Object::new(); for (key, value) in entries { @@ -67,23 +66,21 @@ pub(crate) fn object(entries: &[(&str, JsValue)]) -> JsValue { object.into() } -/// An id as a number or a text: a number too large for JavaScript's as its decimal text, which -/// names the same row. +/// An id as JSON writes it: a number, or a string for a text id or a number above 2^53 − 1. pub(crate) fn id_value(id: &latkit::Id) -> JsValue { - match id { - latkit::Id::Number(n) if *n as f64 <= SAFE => (*n as f64).into(), - latkit::Id::Number(n) => JsValue::from_str(&n.to_string()), - latkit::Id::Text(t) => JsValue::from_str(t), - } + id.exact_number().map_or_else(|| JsValue::from_str(&id.to_string()), JsValue::from) } -/// The id JavaScript names: a whole number from 0 to 2^53 - 1, or a text. +/// Parses a JavaScript id: a whole number from 0 to 2^53 − 1, or a string. pub(crate) fn id(value: &JsValue) -> Result { - if let Some(text) = value.as_string() { - return Ok(latkit::Id::from(text)); - } - match value.as_f64() { - Some(n) if n.fract() == 0.0 && (0.0..=SAFE).contains(&n) => Ok(latkit::Id::Number(n as u64)), - _ => Err(Error::invalid("an id is a whole number from 0 to 2^53 - 1, or a text").at("id")), - } + value + .as_string() + .map(latkit::Id::from) + .or_else(|| value.as_f64().and_then(latkit::Id::from_number)) + .ok_or_else(|| Error::invalid("expected a whole number from 0 to 2^53 - 1, or a string").at("id")) +} + +/// `values` as exactly `N` numbers, or an invalid error at `what`. +pub(crate) fn fixed(values: &[T], what: &str) -> Result<[T; N], Error> { + values.try_into().map_err(|_| Error::invalid(format!("expected {N} numbers")).at(what)) } diff --git a/crates/latkit-js/src/data.rs b/crates/latkit-js/src/data.rs new file mode 100644 index 00000000..6c8ddc09 --- /dev/null +++ b/crates/latkit-js/src/data.rs @@ -0,0 +1,164 @@ +//! Datasets: a `Data` changed in place, a `Snapshot` of one version, and the sources views draw. +//! `Data` and `Snapshot` share the same read methods: description, values, queries, envelopes, and +//! bytes. + +use latkit::host::Host as _; +use wasm_bindgen::prelude::*; + +use crate::pending::{Task, task}; +use crate::{Error, convert, json, to_json}; + +/// A dataset changed in place. Clones share it. +#[wasm_bindgen] +pub struct Data(pub(crate) latkit::Data); + +#[wasm_bindgen] +impl Data { + #[wasm_bindgen(constructor)] + pub fn new(name: &str) -> Data { + Data(latkit::Data::new(name)) + } + + /// Reads the dataset in `bytes`, named `name` if given. `bytes` hold latkit bytes, GeoJSON + /// (parsed in a worker, and only with a `name`), or a JSON document whose file slices are + /// fetched relative to `base`. + pub fn read(bytes: Vec, name: Option, base: String) -> Task { + task(async move { + if let Some(name) = &name + && let Some(job) = latkit::host::Job::geojson(name, &bytes) + { + let read = latkit::host::Browser.spawn(job).await?; + if read.is_empty() { + return Err(latkit::Error::invalid(format!("{name}: the GeoJSON could not be read"))); + } + let reading = latkit::Reading { name: Some(name), ..latkit::Reading::default() }; + return Ok(Data(latkit::Data::read(&read, reading).await?)); + } + let reading = latkit::Reading { name: name.as_deref(), base: &base, host: Some(&latkit::host::Browser) }; + Ok(Data(latkit::Data::read(&bytes, reading).await?)) + }) + } + + #[wasm_bindgen(getter)] + pub fn name(&self) -> String { + self.0.name().to_string() + } + + /// Applies the patch in `bytes`: every change, or none if any is invalid. + pub fn apply(&self, bytes: &[u8]) -> Result<(), Error> { + Ok(self.0.apply(latkit::Patch::from_bytes(bytes)?)?) + } + + /// The current version, taken in O(1) and sharing every page. + #[must_use] + pub fn snapshot(&self) -> Snapshot { + Snapshot(self.0.snapshot()) + } + + /// Makes `snapshot`, a version of this dataset, the current version. + pub fn restore(&self, snapshot: &Snapshot) -> Result<(), Error> { + Ok(self.0.restore(&snapshot.0)?) + } + + /// A source for views that follows the dataset's changes. + pub fn source(&self) -> Source { + Source(latkit::Source::from(&self.0)) + } +} + +impl Data { + fn version(&self) -> latkit::Snapshot { + self.0.snapshot() + } +} + +/// One version of a dataset. +#[wasm_bindgen] +pub struct Snapshot(pub(crate) latkit::Snapshot); + +#[wasm_bindgen] +impl Snapshot { + #[wasm_bindgen(getter)] + pub fn name(&self) -> String { + self.0.name().to_string() + } + + /// A source for views that draws this version unchanged. + pub fn source(&self) -> Source { + Source(latkit::Source::from(&self.0)) + } +} + +impl Snapshot { + fn version(&self) -> latkit::Snapshot { + self.0.clone() + } +} + +/// Adds the read methods to each class that holds a dataset version. +macro_rules! reads { + ($class:ident) => { + #[wasm_bindgen] + impl $class { + /// The dataset's description, as JSON. + pub fn describe(&self) -> String { + to_json(&self.version().describe()) + } + + /// The value of `field` for the row `id` in `table`, at axis coordinate `at` if given, + /// as JSON. `null` if missing. + pub fn get(&self, table: &str, id: &JsValue, field: &str, at: Option) -> Result { + Ok(to_json(&self.version().get(table, convert::id(id)?, field, at))) + } + + /// The rows `query` (JSON) selects, as `{ table, ids, fields, total }`. Fields are in + /// the forms a patch writes, and `total` counts the selected rows before paging. + pub fn query(&self, query: &str) -> Result { + Ok(convert::query_result(&self.version().query(&json("query", query)?)?)) + } + + /// Each row's least and greatest value of a sampled field over equal spans of the axis, + /// per `request` (JSON), as `{ at, least, greatest }`. + pub fn envelope(&self, request: &str) -> Result { + let envelope = self.version().envelope(&json("request", request)?)?; + Ok(convert::object(&[ + ("at", js_sys::Float64Array::from(&envelope.at[..]).into()), + ("least", js_sys::Float32Array::from(&envelope.least[..]).into()), + ("greatest", js_sys::Float32Array::from(&envelope.greatest[..]).into()), + ])) + } + + /// The dataset as latkit bytes: the patch that recreates it. + #[wasm_bindgen(js_name = toBytes)] + pub fn to_bytes(&self) -> Vec { + self.version().to_bytes() + } + } + }; +} + +reads!(Data); +reads!(Snapshot); + +/// A source a view draws a dataset from. +#[wasm_bindgen] +pub struct Source(pub(crate) latkit::Source); + +#[wasm_bindgen] +impl Source { + /// Tiles at `url`, drawn as dataset `name`. + pub fn tiles(name: &str, url: &str) -> Source { + Source(latkit::Source::from(latkit::Tiles::new(name, url))) + } +} + +/// Reads the patch in `bytes`, as latkit bytes or JSON, and returns it as latkit bytes. File slices +/// are fetched relative to `base`. +#[wasm_bindgen(js_name = readPatch)] +pub fn read_patch(bytes: Vec, base: String) -> Task { + task(async move { + let reading = latkit::Reading { base: &base, host: Some(&latkit::host::Browser), name: None }; + let patch = latkit::Patch::read(&bytes, reading).await?; + Ok(js_sys::Uint8Array::from(&patch.to_bytes()[..])) + }) +} diff --git a/crates/latkit-js/src/error.rs b/crates/latkit-js/src/error.rs index f9eb236b..d090800f 100644 --- a/crates/latkit-js/src/error.rs +++ b/crates/latkit-js/src/error.rs @@ -1,6 +1,6 @@ -//! Errors as JavaScript throws them: an `Error` with latkit's message, its `code` (`invalid`, -//! `unsupported`, `limit`, `io`, `device-lost`, `cancelled`, or `timeout`), and for invalid input the -//! `path` to the part. +//! latkit errors thrown as JavaScript `Error`s, with latkit's message, a `code` (`invalid`, +//! `unsupported`, `limit`, `io`, `device-lost`, `cancelled`, or `timeout`), and, for invalid input, +//! the `path` to the invalid part. use js_sys::Reflect; use wasm_bindgen::JsValue; diff --git a/crates/latkit-js/src/gpu.rs b/crates/latkit-js/src/gpu.rs new file mode 100644 index 00000000..5986f46d --- /dev/null +++ b/crates/latkit-js/src/gpu.rs @@ -0,0 +1,127 @@ +//! The GPU, which creates views and canvases and reports its frames, budget, and work. + +use wasm_bindgen::JsCast; +use wasm_bindgen::prelude::*; + +use crate::canvas::{Canvas, Handle, layout}; +use crate::pending::{Task, task}; +use crate::view::View; +use crate::{Error, json, to_json}; + +/// The GPU device views draw with. +#[wasm_bindgen] +pub struct Gpu(latkit::Gpu); + +#[wasm_bindgen] +impl Gpu { + /// Requests a WebGPU device. + pub async fn create() -> Result { + Ok(Gpu(latkit::Gpu::new().await?)) + } + + /// Creates a view of `scene` (JSON) that sends its events to `tell` from the start, each batch + /// as a JSON array of `{ type, detail }`. A canvas shows it. A shot draws it with or without + /// a canvas. + pub fn view(&self, scene: &str, tell: js_sys::Function) -> Result { + let view = latkit::web::PageView::new(latkit::View::new(&self.0, json("scene", scene)?)?); + let telling = view.on(move |events| { + let _ = tell.call1(&JsValue::NULL, &JsValue::from_str(&to_json(&events))); + }); + Ok(View { view, _telling: telling }) + } + + /// Creates a canvas showing `views` in `target`, each at its four fractions of `at`. `target` + /// is a `` element, or an `OffscreenCanvas` at `pixel_ratio` pixels per CSS pixel + /// (default 1). + pub fn canvas( + &self, + target: &JsValue, + views: Vec, + at: &[f32], + pixel_ratio: Option, + ) -> Result { + let layout = layout(views, at)?; + let canvas = if let Some(element) = target.dyn_ref::() { + latkit::web::Canvas::new(&self.0, element, layout)? + } else if let Some(offscreen) = target.dyn_ref::() { + latkit::web::Canvas::offscreen(&self.0, offscreen, layout, pixel_ratio.unwrap_or(1.0))? + } else { + return Err(latkit::Error::invalid("the target must be a or an OffscreenCanvas").into()); + }; + Ok(Canvas(canvas)) + } + + /// Calls `listener` with the frame's timestamp every animation frame, before the views draw. + #[wasm_bindgen(js_name = onFrame)] + pub fn on_frame(&self, listener: js_sys::Function) -> Listener { + Listener(latkit::web::on_frame(&self.0, move |now| { + let _ = listener.call1(&JsValue::NULL, &now.into()); + })) + } + + /// Whether the device was lost. + #[wasm_bindgen(getter)] + pub fn lost(&self) -> bool { + self.0.lost() + } + + /// The GPU memory budget for data, as JSON `{ bytes, uploadPerFrame }`. + #[wasm_bindgen(getter)] + pub fn budget(&self) -> String { + to_json(&self.0.budget()) + } + + /// Sets the GPU memory budget for data from JSON `{ bytes, uploadPerFrame }`. + #[wasm_bindgen(js_name = setBudget)] + pub fn set_budget(&self, budget: &str) -> Result<(), Error> { + self.0.set_budget(json("budget", budget)?); + Ok(()) + } + + /// Exact counts of the GPU's work since it was created, as JSON. + #[wasm_bindgen(getter)] + pub fn work(&self) -> String { + to_json(&self.0.work()) + } + + /// Bytes resident on the GPU that its views draw from. + #[wasm_bindgen(getter)] + pub fn resident(&self) -> f64 { + self.0.resident() as f64 + } + + /// Bytes waiting to upload, across all its views. + #[wasm_bindgen(getter)] + pub fn uploading(&self) -> f64 { + self.0.uploading() as f64 + } + + /// Draws one frame immediately, with `now` as its timestamp in milliseconds. After the first + /// call, no animation frames are requested. + pub fn step(&self, now: f64) { + latkit::web::step(&self.0, now); + } + + /// Resolves once the GPU has finished its submitted work, every readback has arrived, and every + /// requested pipeline has compiled. + pub fn done(&self) -> Task { + let gpu = self.0.clone(); + task(async move { + latkit::web::done(&gpu).await; + Ok(JsValue::UNDEFINED) + }) + } + + /// Resolves when work a view depends on arrives after this call, or a view requests a frame. + pub fn landed(&self) -> Task { + let landed = latkit::web::landed(&self.0); + task(async move { + landed.await; + Ok(JsValue::UNDEFINED) + }) + } +} + +/// A frame listener. Freeing it stops it. +#[wasm_bindgen] +pub struct Listener(#[allow(dead_code, reason = "kept until freed")] latkit::web::Listener); diff --git a/crates/latkit-js/src/lib.rs b/crates/latkit-js/src/lib.rs index 7fb47678..47b5f6cb 100644 --- a/crates/latkit-js/src/lib.rs +++ b/crates/latkit-js/src/lib.rs @@ -1,9 +1,11 @@ -//! The latkit npm package: data and views for the browser, over WebGPU. +//! The WebAssembly binding behind the latkit npm package: data and views in the browser, over +//! WebGPU. //! -//! What crosses is bytes or JSON, in the forms `latkit` reads and writes. Patches cross as their byte -//! frames, which the package's TypeScript writes itself; scenes, cameras, fits, queries, and rows go -//! in as JSON, and events, hits, stats, and work come out as JSON, a view's events a batch at a time. -//! A query's columns come out as typed arrays, copied once. +//! Values cross as bytes or JSON, in the forms `latkit` reads and writes. Patches cross as byte +//! frames, which the package's TypeScript writes. Scenes, cameras, fits, queries, and rows go in as +//! JSON. Events, hits, stats, and work come out as JSON, with a view's events in batches. A query's +//! columns come out as typed arrays, copied once. Asynchronous results cross as a cancellable +//! `Task`. #![cfg(target_arch = "wasm32")] #![allow(unsafe_code, reason = "wasm-bindgen's generated glue")] @@ -11,478 +13,66 @@ mod canvas; mod convert; +mod data; mod error; +mod gpu; +mod pending; mod shot; +mod view; -use wasm_bindgen::JsCast; use wasm_bindgen::prelude::*; pub use canvas::{Canvas, Handle}; +pub use data::{Data, Snapshot, Source, read_patch}; pub use error::Error; -pub use shot::{Image, Video, image, set_encoder}; +pub use gpu::{Gpu, Listener}; +pub use pending::Task; +pub use shot::{Image, Video, image, set_encoder, video}; +pub use view::View; #[wasm_bindgen(start)] fn start() { console_error_panic_hook::set_once(); } -/// What `text` holds, in the JSON form latkit reads; or why not, at the path from `what`. +/// Deserializes `text` from latkit's JSON form. Error paths are prefixed with `what`. fn json(what: &str, text: &str) -> Result { latkit::from_json(text).map_err(|e| e.at(what).into()) } -/// `value` in the JSON form the npm package's types describe. +/// Serializes `value` to the JSON form the npm package's types describe. fn to_json(value: &impl serde::Serialize) -> String { - serde_json::to_string(value).expect("view state serializes") + serde_json::to_string(value).expect("latkit values serialize to JSON") } - -/// A promise of what `future` answers, its error thrown as latkit's. -fn promise>(future: impl Future> + 'static) -> js_sys::Promise { - wasm_bindgen_futures::future_to_promise(async move { - future.await.map(Into::into).map_err(|e| JsValue::from(Error::from(e))) - }) -} - -/// The GPU device views draw with. -#[wasm_bindgen] -pub struct Gpu(latkit::Gpu); - -#[wasm_bindgen] -impl Gpu { - /// Requests a WebGPU device. - pub async fn create() -> Result { - Ok(Gpu(latkit::Gpu::new().await?)) - } - - /// A view of `scene` (JSON), telling `tell` its events from the start: each batch a JSON array of - /// `{ type, detail }`. A canvas shows it; a shot draws it with or without one. - pub fn view(&self, scene: &str, tell: js_sys::Function) -> Result { - let view = latkit::web::PageView::new(latkit::View::new(&self.0, json("scene", scene)?)?); - let telling = view.on(move |events| { - let _ = tell.call1(&JsValue::NULL, &JsValue::from_str(&to_json(&events))); - }); - Ok(View { view, _telling: telling }) - } - - /// A canvas showing views in `panels` (JSON) of `target`: a `` element, or an - /// `OffscreenCanvas` at `pixel_ratio` pixels a CSS pixel. - pub fn canvas(&self, target: &JsValue, panels: &str, pixel_ratio: Option) -> Result { - let panels: latkit::Panels = json("panels", panels)?; - let canvas = if let Some(element) = target.dyn_ref::() { - latkit::web::Canvas::new(&self.0, element, panels)? - } else if let Some(offscreen) = target.dyn_ref::() { - latkit::web::Canvas::offscreen(&self.0, offscreen, panels, pixel_ratio.unwrap_or(1.0))? - } else { - return Err(latkit::Error::invalid("a canvas draws into a or an OffscreenCanvas").into()); - }; - Ok(Canvas(canvas)) - } - - /// Calls `listener` with the frame's timestamp every animation frame, before the views draw. - #[wasm_bindgen(js_name = onFrame)] - pub fn on_frame(&self, listener: js_sys::Function) -> Listener { - Listener(latkit::web::on_frame(&self.0, move |now| { - let _ = listener.call1(&JsValue::NULL, &now.into()); - })) - } - - /// Whether the device was lost. - #[wasm_bindgen(getter)] - pub fn lost(&self) -> bool { - self.0.lost() - } - - /// How much GPU memory data takes, as JSON `{ bytes, uploadPerFrame }`. - #[wasm_bindgen(getter)] - pub fn budget(&self) -> String { - to_json(&self.0.budget()) - } - - /// Sets how much GPU memory data takes, from JSON `{ bytes, uploadPerFrame }`. - #[wasm_bindgen(js_name = setBudget)] - pub fn set_budget(&self, budget: &str) -> Result<(), Error> { - self.0.set_budget(json("budget", budget)?); - Ok(()) - } - - /// What the GPU did since it was made, counted exactly, as JSON. - #[wasm_bindgen(getter)] - pub fn work(&self) -> String { - to_json(&self.0.work()) - } - - /// Draws one frame now at `now`, milliseconds as an animation frame's timestamp; from the first, - /// no animation frames are asked for. - pub fn step(&self, now: f64) { - latkit::web::step(&self.0, now); - } - - /// Resolves once the GPU finished what was submitted, every readback landed, and every pipeline - /// asked for compiled. - pub fn done(&self) -> js_sys::Promise { - let gpu = self.0.clone(); - promise(async move { - latkit::web::done(&gpu).await; - Ok(JsValue::UNDEFINED) - }) - } - - /// Resolves when work a view waits on lands after this call, or a view asks for a frame. - pub fn landed(&self) -> js_sys::Promise { - let landed = latkit::web::landed(&self.0); - promise(async move { - landed.await; - Ok(JsValue::UNDEFINED) - }) - } -} - -#[wasm_bindgen] -impl Gpu { - /// A video of `shot`'s (JSON) pictures at `fps` frames a second, encoded by WebCodecs at - /// `bitrate` bits a second, or about 0.15 bits a pixel a frame. - pub fn video(&self, shot: &str, fps: f64, bitrate: Option) -> Result { - shot::video(&self.0, shot, fps, bitrate) - } -} - -/// A listener; freeing it stops it. -#[wasm_bindgen] -pub struct Listener(#[allow(dead_code, reason = "kept until freed")] latkit::web::Listener); - -/// A named collection of tables, changed in place by patches; a snapshot keeps a version. -#[wasm_bindgen] -pub struct Data(latkit::Data); - -#[wasm_bindgen] -impl Data { - #[wasm_bindgen(constructor)] - pub fn new(name: &str) -> Data { - Data(latkit::Data::new(name)) - } - - /// The dataset latkit bytes hold, renamed `name` when given. - #[allow(clippy::needless_pass_by_value, reason = "wasm-bindgen hands optional texts over owned")] - pub fn read(bytes: &[u8], name: Option) -> Result { - if !bytes.starts_with(b"LATK") { - return Err(latkit::Error::invalid("not latkit bytes").at("bytes").into()); - } - Ok(Data(latkit::Data::read(bytes, name.as_deref())?)) - } - - /// A promise of the dataset its JSON document builds, its slices fetched relative to `base`. - pub fn load(json: String, base: String) -> js_sys::Promise { - promise(async move { Ok(Data(latkit::Data::load(&json, &base, &latkit::host::Browser).await?)) }) - } - - #[wasm_bindgen(getter)] - pub fn name(&self) -> String { - self.0.name().to_string() - } - - /// The first and last axis coordinates the sampled fields span. - #[wasm_bindgen(getter)] - pub fn span(&self) -> Option> { - self.0.span().map(Vec::from) - } - - /// Applies the patch `bytes` hold: every change, or none when one is invalid. - pub fn apply(&mut self, bytes: &[u8]) -> Result<(), Error> { - Ok(self.0.apply(latkit::Patch::from_bytes(bytes)?)?) - } - - /// This version, kept as `apply` changes this one: O(1), sharing every page. - #[must_use] - pub fn snapshot(&self) -> Data { - Data(self.0.clone()) - } - - #[wasm_bindgen(js_name = toBytes)] - pub fn to_bytes(&self) -> Vec { - self.0.to_bytes() - } - - /// The value of `field` for the row with `id` in `table`, at axis coordinate `at` when given, as - /// JSON; `null` when missing. - pub fn get(&self, table: &str, id: &JsValue, field: &str, at: Option) -> Result { - let row = latkit::Row::new(self.0.name(), table, convert::id(id)?); - let value = match at { - Some(at) => self.0.get_at(&row, field, at), - None => self.0.get(&row, field), - }; - Ok(to_json(&value)) - } - - /// The names of the tables. - pub fn tables(&self) -> Vec { - self.0.tables().map(|t| t.name().to_string()).collect() - } - - /// Every row's id of `table`, in row order. - pub fn ids(&self, table: &str) -> js_sys::Array { - self.0.table(table).map(|t| t.ids().map(|id| convert::id_value(&id)).collect()).unwrap_or_default() - } - - /// The names of `table`'s fields. - pub fn fields(&self, table: &str) -> Vec { - self.0.table(table).map(|t| t.fields().map(|(name, _)| name.to_string()).collect()).unwrap_or_default() - } - - /// What the dataset holds, as JSON. - pub fn describe(&self) -> String { - to_json(&self.0.describe()) - } - - /// The rows `query` (JSON) keeps, as `{ table, ids, fields, total }`: fields in the forms a patch - /// writes, `total` the rows kept before paging. - pub fn query(&self, query: &str) -> Result { - let (table, total) = self.0.query(&json("query", query)?)?; - Ok(convert::table(&table, total)) - } - - /// Each row's least and greatest value of sampled `field` of `table` over `spans` spans of - /// `range`: of every row, or of rows `ids` (JSON), as `{ at, least, greatest }` typed arrays. - pub fn envelope( - &self, - table: &str, - field: &str, - range: &[f64], - spans: u32, - ids: Option, - ) -> Result { - let table = self.0.table(table).ok_or_else(|| latkit::Error::invalid("no such table").at(table))?; - let [from, to] = range else { return Err(latkit::Error::invalid("a range is [from, to]").into()) }; - let ids: Option = ids.map(|ids| json("ids", &ids)).transpose()?; - let envelope = table.envelope(field, [*from, *to], spans as usize, ids.as_ref())?; - Ok(convert::object(&[ - ("at", js_sys::Float64Array::from(&envelope.at[..]).into()), - ("least", js_sys::Float32Array::from(&envelope.least[..]).into()), - ("greatest", js_sys::Float32Array::from(&envelope.greatest[..]).into()), - ])) - } -} - -/// A promise of the bytes of the patch its JSON makes, its slices fetched relative to `base`. -#[wasm_bindgen(js_name = loadPatch)] -pub fn load_patch(json: String, base: String) -> js_sys::Promise { - promise(async move { - let patch = latkit::Patch::load(&json, &base, &latkit::host::Browser).await?; - Ok(js_sys::Uint8Array::from(&patch.to_bytes()[..])) - }) -} - -/// A view of a scene, telling its events from the start. -#[wasm_bindgen] -pub struct View { - view: latkit::web::PageView, - _telling: latkit::web::Listener, -} - -#[wasm_bindgen] -impl View { - /// Draws `scene` (JSON) instead; an invalid scene changes nothing. - pub fn show(&self, scene: &str) -> Result<(), Error> { - let scene = json("scene", scene)?; - Ok(self.view.with(|v| v.show(scene))?) - } - - #[wasm_bindgen(js_name = setData)] - pub fn set_data(&self, data: &Data) { - self.view.with(|v| v.set_data(data.0.clone())); - } - - /// Fetches the vector tiles at `url` as the camera asks for them, drawn as dataset `name`. - #[wasm_bindgen(js_name = setTiles)] - pub fn set_tiles(&self, name: &str, url: &str) { - self.view.with(|v| v.set_tiles(latkit::Tiles::new(name, url))); - } - - #[wasm_bindgen(js_name = removeData)] - pub fn remove_data(&self, name: &str) { - self.view.with(|v| v.remove_data(name)); - } - - /// The dataset named `name`: one given, or the tiles the view fetched. - pub fn data(&self, name: &str) -> Option { - self.view.read(|v| v.data(name).cloned()).map(Data) - } - - #[wasm_bindgen(getter)] - pub fn at(&self) -> f64 { - self.view.read(latkit::View::at) - } - - #[wasm_bindgen(setter)] - pub fn set_at(&self, at: f64) { - self.view.with(|v| v.set_at(at)); - } - - /// The camera as JSON. - #[wasm_bindgen(getter)] - pub fn camera(&self) -> String { - self.view.read(|v| to_json(v.camera())) - } - - /// Looks through the camera (JSON) at once. - #[wasm_bindgen(js_name = setCamera)] - pub fn set_camera(&self, camera: &str) -> Result<(), Error> { - let camera = json("camera", camera)?; - self.view.with(|v| v.set_camera(camera)); - Ok(()) - } - - /// Eases to the camera (JSON). - pub fn fly(&self, camera: &str) -> Result<(), Error> { - let camera = json("camera", camera)?; - self.view.with(|v| v.fly(camera)); - Ok(()) - } - - /// Frames what `fit` (JSON) names. - pub fn fit(&self, fit: &str) -> Result<(), Error> { - let fit = json("fit", fit)?; - self.view.with(|v| v.fit(fit)); - Ok(()) - } - - /// Lays every vertex row without a position out again by the scene's layout: a promise of the - /// bytes of a patch to each dataset, writing the places into the data. - pub fn arrange(&self) -> js_sys::Promise { - let pending = self.view.with(latkit::View::arrange); - promise(async move { - let patches = pending.await?; - Ok(patches.iter().map(|patch| js_sys::Uint8Array::from(&patch.to_bytes()[..])).collect::()) - }) - } - - /// Where `position` (x, y, elevation) lands on screen, CSS pixels from the top left. - pub fn point(&self, position: &[f64]) -> Option> { - let position: [f64; 3] = position.try_into().ok()?; - self.view.read(|v| v.point(position)).map(Vec::from) - } - - /// The position on the ground under screen point `point`. - pub fn ground(&self, point: &[f32]) -> Option> { - let point: [f32; 2] = point.try_into().ok()?; - self.view.read(|v| v.ground(point)).map(Vec::from) - } - - #[wasm_bindgen(getter)] - pub fn insets(&self) -> Vec { - self.view.read(|v| v.insets().to_vec()) - } - - #[wasm_bindgen(setter)] - pub fn set_insets(&self, insets: &[f32]) -> Result<(), Error> { - let insets: [f32; 4] = insets.try_into().map_err(|_| latkit::Error::invalid("four insets").at("insets"))?; - self.view.with(|v| v.set_insets(insets)); - Ok(()) - } - - #[wasm_bindgen(getter)] - pub fn orbit(&self) -> f64 { - self.view.read(latkit::View::orbit) - } - - #[wasm_bindgen(setter)] - pub fn set_orbit(&self, degrees_per_second: f64) { - self.view.with(|v| v.set_orbit(degrees_per_second)); - } - - /// The rows selected, as JSON sets of rows. - #[wasm_bindgen(getter)] - pub fn selection(&self) -> String { - self.view.read(|v| to_json(&v.selection())) - } - - /// Selects the rows (JSON sets of rows), and no others. - pub fn select(&self, rows: &str) -> Result<(), Error> { - let rows: Vec = json("rows", rows)?; - self.view.with(|v| v.select(rows)); - Ok(()) - } - - /// What the user points at, as a JSON hit or `null`. - #[wasm_bindgen(getter)] - pub fn hovered(&self) -> String { - self.view.read(|v| to_json(&v.hovered())) - } - - /// Every mark that draws nothing because its data no longer fits it, as JSON failures. - #[wasm_bindgen(getter)] - pub fn failures(&self) -> String { - self.view.read(|v| to_json(&v.failures().collect::>())) - } - - /// What the view does each frame, as JSON. - #[wasm_bindgen(getter)] - pub fn stats(&self) -> String { - to_json(&self.view.read(latkit::View::stats)) - } - - /// Whether the view wants a frame: it changed, or work it waited on landed. - #[wasm_bindgen(getter, js_name = wantsFrame)] - pub fn wants_frame(&self) -> bool { - self.view.wants_frame() - } - - /// Every row within `radius` CSS pixels of `point`, nearest first, as JSON hits. - pub fn pick(&self, point: &[f32], radius: f32) -> Result { - let point: [f32; 2] = point.try_into().map_err(|_| latkit::Error::invalid("a point is [x, y]").at("point"))?; - let pending = self.view.with(|v| v.pick(point, radius)); - Ok(promise(async move { Ok(JsValue::from_str(&to_json(&pending.await?))) })) - } - - /// Where the row (JSON) lies on screen, `[x, y]`, or `null`. - pub fn locate(&self, row: &str) -> Result { - let row: latkit::Row = json("row", row)?; - let pending = self.view.with(|v| v.locate(&row)); - Ok(promise(async move { - let point = pending.await?; - Ok(point.map_or(JsValue::NULL, |p| js_sys::Array::of2(&p[0].into(), &p[1].into()).into())) - })) - } - - /// The rows joined to the row (JSON), as JSON sets of rows: a vertex's neighbors across edges and - /// nets, or an edge's or a net's vertices. - pub fn neighbors(&self, row: &str) -> Result { - let row: latkit::Row = json("row", row)?; - Ok(self.view.read(|v| to_json(&v.neighbors(&row)))) - } - - /// The view, for a canvas to show or a shot to draw. - pub fn handle(&self) -> Handle { - Handle(self.view.clone()) - } -} - -/// Has views' jobs done by `run`: `(bytes: Uint8Array) => Promise`, the page's workers. +/// Runs views' jobs with `run`, the page's workers: `(bytes: Uint8Array) => Promise`. #[wasm_bindgen(js_name = setJobs)] pub fn set_jobs(run: js_sys::Function) { latkit::host::Browser::set_jobs(run); } -/// Does a job a view handed the page, in a worker: its answer's bytes. +/// Runs a job a view handed to the page, in a worker, and returns the result's bytes. #[wasm_bindgen] pub fn job(bytes: &[u8]) -> Vec { latkit::host::Job::from_bytes(bytes.to_vec()).run() } -/// The job reading GeoJSON `bytes`, gzip-compressed or not, as dataset `name`, for a worker to do: it -/// answers the bytes of the patch building the dataset, or nothing when the bytes are no GeoJSON. -#[wasm_bindgen(js_name = geojsonJob)] -pub fn geojson_job(name: &str, bytes: &[u8]) -> Vec { - latkit::host::Job::geojson(name, bytes).into_bytes() -} - /// The compiled module, for workers to instantiate. #[wasm_bindgen] pub fn module() -> JsValue { wasm_bindgen::module() } -/// A CSS gradient through `map` (JSON), a catalog name or colors of your own, toward `direction`: the -/// colors the GPU draws, a band each for a categorical map's categories. +/// The catalog of named colormaps, in order, as JSON `[{ name, label, kind }]`. +#[wasm_bindgen] +pub fn colormaps() -> String { + let catalog: Vec = latkit::Colormap::catalog() + .map(|map| serde_json::json!({ "name": map.name(), "label": map.label(), "kind": map.kind() })) + .collect(); + to_json(&catalog) +} + +/// A CSS `linear-gradient` toward `direction` through `map`, given as JSON (a catalog name or +/// custom colors), in the colors the GPU draws. A categorical map gets one band per category. #[wasm_bindgen(js_name = colormapCss)] pub fn colormap_css(map: &str, direction: &str) -> Result { let map: latkit::Colormap = json("colormap", map)?; diff --git a/crates/latkit-js/src/pending.rs b/crates/latkit-js/src/pending.rs new file mode 100644 index 00000000..9adcea3d --- /dev/null +++ b/crates/latkit-js/src/pending.rs @@ -0,0 +1,39 @@ +//! Cancellable promises of latkit results. Cancelling drops the work's future, so pending work that +//! nothing else needs is skipped: a view's query leaves its queue, and a host job that has not +//! started never runs. + +use futures_util::future::{AbortHandle, Abortable}; +use wasm_bindgen::prelude::*; + +use crate::Error; + +/// A promise of a latkit result, and a way to cancel it. +#[wasm_bindgen] +pub struct Task { + promise: js_sys::Promise, + abort: AbortHandle, +} + +#[wasm_bindgen] +impl Task { + /// The result. Rejects with a `cancelled` error once cancelled. + #[wasm_bindgen(getter)] + pub fn promise(&self) -> js_sys::Promise { + self.promise.clone() + } + + /// Cancels the work, dropping what it waits on. + pub fn cancel(&self) { + self.abort.abort(); + } +} + +/// Wraps `future` in a task whose promise rejects with its errors as latkit errors. +pub(crate) fn task>(future: impl Future> + 'static) -> Task { + let (abort, registration) = AbortHandle::new_pair(); + let promise = wasm_bindgen_futures::future_to_promise(async move { + let result = Abortable::new(future, registration).await.unwrap_or(Err(latkit::Error::Cancelled)); + result.map(Into::into).map_err(|e| JsValue::from(Error::from(e))) + }); + Task { promise, abort } +} diff --git a/crates/latkit-js/src/shot.rs b/crates/latkit-js/src/shot.rs index 46801ea5..7e706f4b 100644 --- a/crates/latkit-js/src/shot.rs +++ b/crates/latkit-js/src/shot.rs @@ -1,17 +1,20 @@ -//! Pictures and videos of views: a shot crosses as JSON, -//! `{ size, scale, panels, transparent, deadline, elapsed }` with times in seconds, and draws the views -//! given, whose canvases hold their last frames meanwhile. An image's pixels cross once and its files -//! are written in workers; a video hands back the bytes each frame made, for the page to write. +//! Pictures and videos of views. A shot crosses as JSON, `{ size, scale, transparent, capture, +//! elapsed }` with times in seconds. Its layout crosses as a canvas's does: views, and four +//! fractions per view. Canvases showing those views keep their last frames meanwhile. The page +//! decides how long to wait. An image's pixels cross once, and its files are encoded in workers. A +//! video returns the bytes each frame produced, for the page to write. use std::cell::RefCell; use std::io::Write; use std::rc::Rc; -use futures_util::future::{AbortHandle, Abortable}; +use latkit::host::Host as _; +use latkit::web::PageView; use wasm_bindgen::prelude::*; use crate::Error; -use crate::canvas::{Handle, views_of}; +use crate::canvas::{Handle, layout}; +use crate::pending::{Task, task}; /// RGBA pixels, straight alpha, rows top to bottom. #[wasm_bindgen] @@ -35,35 +38,46 @@ impl Image { self.0.rgba.to_vec() } - /// The job writing the image as a `png`, `jpeg` at `quality` from 1 to 100, or `webp` file, for a - /// worker to do. - #[wasm_bindgen(js_name = encodeJob)] - pub fn encode_job(&self, format: &str, quality: Option) -> Result, Error> { + /// Encodes the image in a worker as a `png`, `jpeg` (at `quality` 1 to 100, default 90), or + /// `webp` file. + pub fn encode(&self, format: &str, quality: Option) -> Result { let quality = match quality { None => 90, Some(q) if (1.0..=100.0).contains(&q) => q.round() as u8, - Some(_) => return Err(latkit::Error::invalid("a quality runs from 1 to 100").at("quality").into()), + Some(_) => return Err(latkit::Error::invalid("expected 1 to 100").at("quality").into()), }; let format = match format { "png" => latkit::ImageFormat::Png, "jpeg" => latkit::ImageFormat::Jpeg { quality }, "webp" => latkit::ImageFormat::WebP, - other => return Err(latkit::Error::invalid(format!("no image format {other:?}")).at("format").into()), + other => return Err(latkit::Error::invalid(format!("unknown image format {other:?}")).at("format").into()), }; - Ok(latkit::host::Job::encode(&self.0, format).into_bytes()) + let job = latkit::host::Job::encode(&self.0, format); + Ok(task(async move { + let file = latkit::host::Browser.spawn(job).await?; + if file.is_empty() { + return Err(latkit::Error::invalid(format!("the image could not be encoded as {format}"))); + } + Ok(js_sys::Uint8Array::from(&file[..])) + })) } } -/// The picture `shot` (JSON) takes of the views of `views`, one per panel, once they are complete -/// and still. +/// Takes a picture of `views` as `shot` (JSON) describes, each view at its four fractions of `at`, +/// once the views are complete as the shot's `capture` says. #[wasm_bindgen] -pub fn image(shot: &str, views: Vec) -> Result { - let shot: latkit::Shot = crate::json("shot", shot)?; - let views = views_of(views); - Ok(wasm_bindgen_futures::future_to_promise(async move { - let image = latkit::web::image(&shot, &views).await.map_err(|e| JsValue::from(Error::from(e)))?; - Ok(Image(image).into()) - })) +pub fn image(shot: &str, views: Vec, at: &[f32]) -> Result { + let (shot, views) = shot_of(shot, views, at)?; + Ok(task(async move { Ok(Image(latkit::web::image(&shot, &views).await?)) })) +} + +/// The shot `json` describes, with its panels from `views` and `at` and no deadline (the page +/// decides how long to wait), and its views. +fn shot_of(json: &str, views: Vec, at: &[f32]) -> Result<(latkit::Shot, Vec), Error> { + let panels = layout(views, at)?; + let shot = crate::json::("shot", json)?.deadline(None); + let shot = shot.panels(panels.iter().map(|panel| panel.at).collect::>()); + Ok((shot, panels.into_iter().map(|panel| panel.view).collect())) } /// Bytes a video wrote and the page has not yet taken. @@ -81,111 +95,83 @@ impl Write for Written { } } -/// A video of a shot's pictures, encoded by WebCodecs as frames are drawn. +/// A video of a shot's pictures of its views, encoded with WebCodecs as frames are drawn. One +/// operation runs at a time. Cancelling its task drops the video with it. #[wasm_bindgen] pub struct Video { - /// The video, or nothing while a frame draws it, and once it is finished. + /// The video. `None` while a frame is drawing it, and after it is finished or cancelled. inner: Rc>>>>, + views: Vec, written: Written, - fps: f64, frames: Rc>, - active: Rc>>, } #[wasm_bindgen] impl Video { - /// Seconds into the video the next frame shows at. - #[wasm_bindgen(getter)] - pub fn time(&self) -> f64 { - *self.frames.borrow() as f64 / self.fps - } - /// Frames drawn or held so far. #[wasm_bindgen(getter)] pub fn frames(&self) -> f64 { *self.frames.borrow() as f64 } - /// Draws the next frame of the views of `views`, one per panel; resolves to the bytes of the file - /// ready to write, once there is room for another frame. - pub fn frame(&self, views: Vec) -> js_sys::Promise { + /// Draws the next frame of its views. Resolves to the file bytes ready to write, once the + /// encoder can take another frame. + pub fn frame(&self) -> Task { let (inner, written, frames) = (self.inner.clone(), self.written.clone(), self.frames.clone()); - let views = views_of(views); - self.cancellable(async move { + let views = self.views.clone(); + task(async move { let taken = inner.borrow_mut().take(); - let error = || latkit::Error::invalid("a video draws one frame at a time, until it is finished"); - let mut video = taken.ok_or_else(|| JsValue::from(Error::from(error())))?; + let mut video = taken.ok_or_else(|| latkit::Error::invalid("the video is finished or cancelled"))?; let drawn = latkit::web::frame(&mut video, &views).await; *frames.borrow_mut() = video.frames(); *inner.borrow_mut() = Some(video); - drawn.map_err(|e| JsValue::from(Error::from(e)))?; - Ok(js_sys::Uint8Array::from(&std::mem::take(&mut *written.0.borrow_mut())[..]).into()) + drawn?; + Ok(written.take()) }) } - /// Cancels an in-flight operation, releasing its views and encoder when the future is dropped. + /// Ends the video and releases its views and encoder. A task in progress is cancelled by the + /// page. pub fn cancel(&self) { - if let Some(active) = self.active.borrow_mut().take() { - active.abort(); - } self.inner.borrow_mut().take(); } - /// Encodes every frame under way and ends the video; resolves to the file's last bytes. - pub fn finish(&self) -> js_sys::Promise { + /// Encodes the remaining frames and ends the video. Resolves to the file's last bytes. + pub fn finish(&self) -> Task { let (inner, written) = (self.inner.clone(), self.written.clone()); - self.cancellable(async move { - let taken = inner.borrow_mut().take(); - let video = taken.ok_or_else(|| JsValue::from(Error::from(latkit::Error::Cancelled)))?; - video.finish().await.map_err(|e| JsValue::from(Error::from(e)))?; - Ok(js_sys::Uint8Array::from(&std::mem::take(&mut *written.0.borrow_mut())[..]).into()) + task(async move { + let video = inner.borrow_mut().take().ok_or(latkit::Error::Cancelled)?; + video.finish().await?; + Ok(written.take()) }) } } -impl Video { - fn cancellable( - &self, - future: impl std::future::Future> + 'static, - ) -> js_sys::Promise { - let mut active = self.active.borrow_mut(); - if active.is_some() { - return js_sys::Promise::reject(&JsValue::from(Error::from(latkit::Error::invalid( - "a video runs one operation at a time", - )))); - } - let (abort, registration) = AbortHandle::new_pair(); - *active = Some(abort); - let active = self.active.clone(); - wasm_bindgen_futures::future_to_promise(async move { - let result = Abortable::new(future, registration).await; - active.borrow_mut().take(); - result.unwrap_or_else(|_| Err(JsValue::from(Error::from(latkit::Error::Cancelled)))) - }) - } -} - -impl Drop for Video { - fn drop(&mut self) { - self.cancel(); +impl Written { + /// The bytes written since the last call. + fn take(&self) -> js_sys::Uint8Array { + js_sys::Uint8Array::from(&std::mem::take(&mut *self.0.borrow_mut())[..]) } } -/// A video of `shot`'s (JSON) pictures at `fps` frames a second, as MP4 at `bitrate` bits a second -/// or about 0.15 bits a pixel a frame. -pub(crate) fn video(gpu: &latkit::Gpu, shot: &str, fps: f64, bitrate: Option) -> Result { - let shot: latkit::Shot = crate::json("shot", shot)?; +/// A video of pictures of `views` as `shot` (JSON) describes, each view at its four fractions of +/// `at`, at `fps` frames per second. It encodes MP4 at `bitrate` bits per second. Defaults to about +/// 0.15 bits per pixel per frame. +#[wasm_bindgen] +pub fn video(shot: &str, views: Vec, at: &[f32], fps: f64, bitrate: Option) -> Result { + let (shot, views) = shot_of(shot, views, at)?; + let gpu = views.first().ok_or_else(|| latkit::Error::invalid("expected at least one view").at("layout"))?.gpu(); let written = Written::default(); let mut mp4 = latkit::Mp4::new(written.clone()); if let Some(bitrate) = bitrate { mp4 = mp4.bitrate(bitrate as u32); } - let video = shot.video(gpu, fps, mp4)?; - Ok(Video { inner: Rc::new(RefCell::new(Some(video))), written, fps, frames: Rc::default(), active: Rc::default() }) + let video = shot.video(&gpu, fps, mp4)?; + Ok(Video { inner: Rc::new(RefCell::new(Some(video))), views, written, frames: Rc::default() }) } -/// Has videos encoded by what `create` makes of `(width, height, fps, bitrate)`: the npm package's -/// WebCodecs encoder. +/// Sets the factory for video encoders: `create(width, height, fps, bitrate)` returns one. The npm +/// package passes its WebCodecs encoder. #[wasm_bindgen(js_name = setEncoder)] pub fn set_encoder(create: js_sys::Function) { latkit::host::Browser::set_encoder(create); diff --git a/crates/latkit-js/src/view.rs b/crates/latkit-js/src/view.rs new file mode 100644 index 00000000..92f99b98 --- /dev/null +++ b/crates/latkit-js/src/view.rs @@ -0,0 +1,207 @@ +//! Views: a scene drawn from its sources, sending its events from the start. Its state is read and +//! set between frames, and its queries are answered by the frames that draw it. + +use wasm_bindgen::prelude::*; + +use crate::canvas::Handle; +use crate::convert::fixed; +use crate::data::{Snapshot, Source}; +use crate::pending::{Task, task}; +use crate::{Error, json, to_json}; + +/// A view of a scene, sending its events from the start. +#[wasm_bindgen] +pub struct View { + pub(crate) view: latkit::web::PageView, + pub(crate) _telling: latkit::web::Listener, +} + +#[wasm_bindgen] +impl View { + /// Replaces the scene with `scene` (JSON). An invalid scene changes nothing. + #[wasm_bindgen(js_name = setScene)] + pub fn set_scene(&self, scene: &str) -> Result<(), Error> { + let scene = json("scene", scene)?; + Ok(self.view.with(|v| v.set_scene(scene))?) + } + + /// Replaces the view's sources with `sources`. + #[wasm_bindgen(js_name = setSources)] + pub fn set_sources(&self, sources: Vec) -> Result<(), Error> { + Ok(self.view.with(|v| v.set_sources(sources.into_iter().map(|s| s.0)))?) + } + + /// The version of dataset `name` the view draws. + pub fn snapshot(&self, name: &str) -> Option { + self.view.read(|v| v.snapshot(name).cloned()).map(Snapshot) + } + + #[wasm_bindgen(getter)] + pub fn at(&self) -> f64 { + self.view.read(latkit::View::at) + } + + #[wasm_bindgen(setter)] + pub fn set_at(&self, at: f64) { + self.view.with(|v| v.set_at(at)); + } + + /// The camera as JSON. + #[wasm_bindgen(getter)] + pub fn camera(&self) -> String { + self.view.read(|v| to_json(v.camera())) + } + + /// Switches to `camera` (JSON) immediately. + #[wasm_bindgen(js_name = setCamera)] + pub fn set_camera(&self, camera: &str) -> Result<(), Error> { + let camera = json("camera", camera)?; + self.view.with(|v| v.set_camera(camera)); + Ok(()) + } + + /// Eases to `camera` (JSON) over `seconds`, or over twice the scene's ease without it. + pub fn fly(&self, camera: &str, seconds: Option) -> Result<(), Error> { + let camera = json("camera", camera)?; + let seconds = seconds + .map(|s| { + std::time::Duration::try_from_secs_f64(s) + .map_err(|_| latkit::Error::invalid("expected nonnegative seconds").at("seconds")) + }) + .transpose()?; + self.view.with(|v| v.fly(camera, seconds)); + Ok(()) + } + + /// Frames what `fit` (JSON) names. + pub fn fit(&self, fit: &str) -> Result<(), Error> { + let fit = json("fit", fit)?; + self.view.with(|v| v.fit(fit)); + Ok(()) + } + + /// Runs the scene's layout again on every vertex row without a position. Resolves to one patch + /// per dataset, as bytes, that writes the computed positions into the data. + pub fn arrange(&self) -> Task { + let pending = self.view.with(latkit::View::arrange); + task(async move { + let patches = pending.await?; + Ok(patches.iter().map(|patch| js_sys::Uint8Array::from(&patch.to_bytes()[..])).collect::()) + }) + } + + /// The screen point of `position` (x, y, elevation), in CSS pixels from the top left. + pub fn screen(&self, position: &[f64]) -> Result>, Error> { + let position = fixed::<_, 3>(position, "position")?; + Ok(self.view.read(|v| v.screen(position)).map(Vec::from)) + } + + /// The position on the ground under screen point `point`. + pub fn position(&self, point: &[f32]) -> Result>, Error> { + let point = fixed::<_, 2>(point, "point")?; + Ok(self.view.read(|v| v.position(point)).map(Vec::from)) + } + + #[wasm_bindgen(getter)] + pub fn insets(&self) -> Vec { + self.view.read(|v| v.insets().to_vec()) + } + + #[wasm_bindgen(setter)] + pub fn set_insets(&self, insets: &[f32]) -> Result<(), Error> { + let insets = fixed::<_, 4>(insets, "insets")?; + self.view.with(|v| v.set_insets(insets)); + Ok(()) + } + + #[wasm_bindgen(getter)] + pub fn orbit(&self) -> f64 { + self.view.read(latkit::View::orbit) + } + + #[wasm_bindgen(setter)] + pub fn set_orbit(&self, degrees_per_second: f64) { + self.view.with(|v| v.set_orbit(degrees_per_second)); + } + + /// The selected rows, as JSON sets of rows. + #[wasm_bindgen(getter)] + pub fn selection(&self) -> String { + self.view.read(|v| to_json(&v.selection())) + } + + /// Selects exactly `rows` (JSON sets of rows). + #[wasm_bindgen(js_name = setSelection)] + pub fn set_selection(&self, rows: &str) -> Result<(), Error> { + let rows: Vec = json("rows", rows)?; + self.view.with(|v| v.set_selection(rows)); + Ok(()) + } + + /// The row the user points at, as a JSON hit or `null`. + #[wasm_bindgen(getter)] + pub fn hovered(&self) -> String { + self.view.read(|v| to_json(&v.hovered())) + } + + /// The pending work the last frame drew placeholders for, as JSON. + #[wasm_bindgen(getter)] + pub fn waiting(&self) -> String { + to_json(&self.view.read(latkit::View::waiting)) + } + + /// Whether the last frame was complete, with nothing in `waiting` still pending. + #[wasm_bindgen(getter)] + pub fn complete(&self) -> bool { + self.view.read(latkit::View::complete) + } + + /// Every mark that draws nothing because its data no longer fits it, as JSON failures. + #[wasm_bindgen(getter)] + pub fn failures(&self) -> String { + self.view.read(|v| to_json(&v.failures().collect::>())) + } + + /// The view's per-frame statistics, as JSON. + #[wasm_bindgen(getter)] + pub fn stats(&self) -> String { + to_json(&self.view.read(latkit::View::stats)) + } + + /// Whether the view needs a frame: it changed, or work it waited on arrived. + #[wasm_bindgen(getter, js_name = wantsFrame)] + pub fn wants_frame(&self) -> bool { + self.view.wants_frame() + } + + /// Every row within `radius` CSS pixels of `point`, nearest first, as JSON hits. + pub fn pick(&self, point: &[f32], radius: f32) -> Result { + let point = fixed::<_, 2>(point, "point")?; + let pending = self.view.with(|v| v.pick(point, radius)); + Ok(task(async move { Ok(JsValue::from_str(&to_json(&pending.await?))) })) + } + + /// The screen point of `row` (JSON), as `[x, y]` in CSS pixels, or `null` if no mark draws it + /// in view. + #[wasm_bindgen(js_name = screenOf)] + pub fn screen_of(&self, row: &str) -> Result { + let row: latkit::Row = json("row", row)?; + let pending = self.view.with(|v| v.screen_of(&row)); + Ok(task(async move { + let point = pending.await?; + Ok(point.map_or(JsValue::NULL, |p| js_sys::Array::of2(&p[0].into(), &p[1].into()).into())) + })) + } + + /// The rows connected to `row` (JSON), as JSON sets of rows: a vertex's neighbors across edges + /// and nets, or an edge's or net's vertices. + pub fn neighbors(&self, row: &str) -> Result { + let row: latkit::Row = json("row", row)?; + Ok(self.view.read(|v| to_json(&v.neighbors(&row)))) + } + + /// A handle to the view, for a canvas to show or a shot to draw. + pub fn handle(&self) -> Handle { + Handle(self.view.clone()) + } +} diff --git a/crates/latkit/Cargo.toml b/crates/latkit/Cargo.toml index 0a4858c9..5c9b7e65 100644 --- a/crates/latkit/Cargo.toml +++ b/crates/latkit/Cargo.toml @@ -70,6 +70,7 @@ web-sys = { workspace = true, optional = true, features = [ "KeyboardEvent", "MediaQueryList", "MouseEvent", + "Navigator", "OffscreenCanvas", "OffscreenCanvasRenderingContext2d", "PointerEvent", @@ -81,6 +82,7 @@ web-sys = { workspace = true, optional = true, features = [ "WheelEvent", "Window", "WorkerGlobalScope", + "WorkerNavigator", ] } [build-dependencies] diff --git a/crates/latkit/README.md b/crates/latkit/README.md index b6bc63a4..2d1723cb 100644 --- a/crates/latkit/README.md +++ b/crates/latkit/README.md @@ -11,16 +11,16 @@ cargo add latkit use latkit::{Data, Ids, Patch, Query, Table, field}; fn main() -> Result<(), latkit::Error> { - let mut data = Data::new("grid"); - data.apply(Patch::new().replace(Table::new("Bus", Ids::count(2)).field("kv", vec![115.0, 230.0])))?; - let (high_voltage, total) = data.query(&Query::of("Bus").filter(field("kv").ge(200.0)))?; - assert_eq!((high_voltage.len(), total), (1, 1)); + let grid = Data::new("grid"); + grid.apply(Patch::new().replace(Table::new("Bus", Ids::count(2)).field("kv", vec![115.0, 230.0])))?; + let high_voltage = grid.snapshot().query(&Query::of("Bus").filter(field("kv").ge(200.0)))?; + assert_eq!((high_voltage.table.len(), high_voltage.total), (1, 1)); Ok(()) } ``` -- `default-features = false`: data, patches, LATK, JSON loading, queries, expressions, colormaps, - GeoJSON, and images; no GPU dependency. +- `default-features = false`: data, patches, LATK, JSON reading, queries, expressions, colormaps, + GeoJSON, and images, with no GPU dependency. - `gpu` (default): scenes, views, picking, layouts, and rendering into wgpu targets. - `native`: system fonts, file loading, jobs, and Windows Media Foundation video. - `web`: browser integration on `wasm32-unknown-unknown`. @@ -28,7 +28,7 @@ fn main() -> Result<(), latkit::Error> { `cargo install latkit-cli` installs the `latkit` executable. `npm install latkit` installs the browser runtime and plain-data builders. -LATK v1 uses 64-bit absolute frame indexes. Sampled values use `f32`; coordinates use `f64`. -See the [format specification](https://latkit.readthedocs.io/en/latest/formats.html). +The binary format uses 64-bit absolute frame indexes. Sampled values are `f32` and coordinates are +`f64`. See the [format specification](https://latkit.readthedocs.io/en/latest/guide.html#bytes). -MIT licensed. Notices are included in `THIRD_PARTY.md` and `colormaps/NOTICES.md`. +MIT licensed. Third-party notices are in `THIRD_PARTY.md` and `colormaps/NOTICES.md`. diff --git a/crates/latkit/build.rs b/crates/latkit/build.rs index 43f7e303..9d21de83 100644 --- a/crates/latkit/build.rs +++ b/crates/latkit/build.rs @@ -1,4 +1,5 @@ -//! Generates the Rust side of the GPU contract from the WESL shaders, and bakes the colormap catalog. +//! Build script: generates Rust mirrors of the WESL shaders' types and bind layouts, and +//! precomputes the colormap catalog. fn main() { println!("cargo::rerun-if-changed=build.rs"); diff --git a/crates/latkit/build/colormaps.rs b/crates/latkit/build/colormaps.rs index 0bb9462b..0d0ea5a6 100644 --- a/crates/latkit/build/colormaps.rs +++ b/crates/latkit/build/colormaps.rs @@ -1,5 +1,5 @@ -//! Bakes `colormaps/catalog.txt` into `OUT_DIR/colormaps.rs`: per map its name, label, kind, color -//! count, and 256 sRGB texels, made by the crate's own `ramp`. +//! Compiles `colormaps/catalog.txt` into `OUT_DIR/colormaps.rs`. Each map gets its name, label, +//! kind, color count, and 256 sRGB texels computed by the crate's own `ramp`. use crate::ramp::{Spread, texels}; use std::env; diff --git a/crates/latkit/build/shaders.rs b/crates/latkit/build/shaders.rs index 48d74a5b..4d304401 100644 --- a/crates/latkit/build/shaders.rs +++ b/crates/latkit/build/shaders.rs @@ -1,17 +1,17 @@ -//! Links every WESL entry, validates it with naga, and writes `OUT_DIR/shaders.rs`: +//! Links every WESL entry, validates it with naga, and writes `OUT_DIR/shaders.rs`, containing: //! -//! - every constant and struct of `lib/types.wesl`, as Rust constants and `#[repr(C)]` mirrors whose -//! size is checked against WGSL; -//! - the shared bind group layouts of `lib/bind.wesl` (group 0 per view, group 1 per mark); -//! - per shader and variant: its declarations, entry points, the layout of its pass group (group 2), -//! and whether it reads the shared groups at all. +//! - Every constant and struct of `lib/types.wesl`, as Rust constants and `#[repr(C)]` mirrors +//! whose sizes are checked against WGSL. +//! - The shared bind group layouts of `lib/bind.wesl` (group 0 per view, group 1 per mark). +//! - Per shader variant: its declarations, pipeline constant names, pass group layout (group 2), +//! and whether it uses the shared groups. //! -//! Variants share most of what they link, so each distinct top-level declaration is stored once, in -//! `OUT_DIR/shaders.wgsl`, and a variant is the list of its declarations, which join back into exactly -//! the module it linked. +//! Variants share most of their linked code, so each distinct top-level declaration is stored once +//! in `OUT_DIR/shaders.wgsl`. A variant is a list of declaration indices that join back into +//! exactly the module it linked. //! -//! It also writes each variant's whole module and `OUT_DIR/shaders.json`, every variant's file, entry -//! points, and the pipeline constants it is made with, from which the browser gate times each +//! It also writes each variant's full module, and `OUT_DIR/shaders.json` listing every variant's +//! file, entry points, and pipeline constant values. The browser gate reads it to time each //! pipeline in Chrome. use std::collections::{BTreeMap, HashMap}; @@ -23,7 +23,7 @@ use std::path::{Path, PathBuf}; use naga::valid::{Capabilities, ModuleInfo, ValidationFlags, Validator}; use naga::{AddressSpace, ArraySize, ImageClass, ImageDimension, ScalarKind, StorageAccess, TypeInner, VectorSize}; -/// A shader module and the variants it is linked in: each variant turns on its WESL features. +/// A shader module and its variants. Each variant enables its listed WESL features. struct Entry { module: &'static str, variants: &'static [(&'static str, &'static [&'static str])], @@ -31,9 +31,9 @@ struct Entry { const ENTRIES: &[Entry] = &[ Entry { module: "compute::evaluate", variants: &[("main", &["writes"])] }, - Entry { module: "compute::freeze", variants: &[("main", &[])] }, + Entry { module: "compute::freeze", variants: &[("main", &["writes"])] }, Entry { module: "compute::bounds", variants: &[("main", &[])] }, - Entry { module: "compute::normals", variants: &[("main", &[])] }, + Entry { module: "compute::normals", variants: &[("main", &["atomic"])] }, Entry { module: "compute::probe", variants: &[("main", &[]), ("multisampled", &["multisampled"])] }, Entry { module: "compute::lists", variants: &[("main", &["lists"])] }, Entry { module: "draw::backdrop", variants: &[("main", &[])] }, @@ -41,37 +41,33 @@ const ENTRIES: &[Entry] = &[ module: "draw::points", variants: &[ ("main", &[]), - ("cast", &["casts"]), ("custom", &["custom"]), - ("custom_cast", &["custom", "casts"]), ("custom_focus", &["custom", "focused"]), ("ports", &["ports"]), ("focus", &["focused"]), ], }, - Entry { module: "draw::strokes", variants: &[("main", &[]), ("cast", &["casts"]), ("focus", &["focused"])] }, - Entry { module: "draw::triangles", variants: &[("main", &[]), ("cast", &["casts"])] }, + Entry { module: "draw::strokes", variants: &[("main", &[]), ("focus", &["focused"])] }, + Entry { module: "draw::triangles", variants: &[("main", &[])] }, Entry { module: "draw::areas", variants: &[("main", &[]), ("focus", &["focused"])] }, - Entry { module: "draw::shade", variants: &[("main", &[])] }, Entry { module: "draw::glyphs", variants: &[("main", &[]), ("ports", &["ports"])] }, - Entry { module: "compute::candidates", variants: &[("main", &[])] }, - Entry { module: "compute::labels", variants: &[("main", &[])] }, + Entry { module: "compute::texts", variants: &[("main", &[])] }, Entry { module: "draw::overlay", variants: &[("main", &[])] }, Entry { module: "compute::compose", variants: &[("main", &[])] }, Entry { module: "compute::fields", variants: &[("main", &[])] }, ]; -const FEATURES: &[&str] = &["writes", "casts", "custom", "multisampled", "lists", "ports", "focused"]; +const FEATURES: &[&str] = &["writes", "atomic", "custom", "multisampled", "lists", "ports", "focused"]; -/// What a custom marker supplies, for validating its variant. +/// A placeholder custom marker function, appended to validate the custom variants. const CUSTOM_MARKER: &str = "fn marker(f: MarkerFragment) -> MarkerColor { return MarkerColor(f.color, 0.0); }"; -/// Uniforms bound at a dynamic offset: each mark's, each pass's eye, and each probe dispatch's step. -const DYNAMIC: &[&str] = &["mark", "eye", "step"]; +/// Uniforms bound at a dynamic offset: the per-mark uniform and each probe dispatch's step. +const DYNAMIC: &[&str] = &["mark", "step"]; -/// The values each pipeline constant is made with, as constants of `lib/types.wesl` or booleans: the -/// primitive a probe or a label's anchor reads, the path a stroke reads, and whether channels read -/// joins, samples, and the view. +/// The values pipelines use for each pipeline constant, as `lib/types.wesl` constants or booleans. +/// `PRIMITIVE` is the primitive a probe or label anchor reads, and `PATH` the path a stroke reads. +/// The booleans say whether channels read joins, samples, and the view. const OVERRIDES: &[(&str, &[&str])] = &[ ("PRIMITIVE", &["POINTS", "STROKES", "TRIANGLES"]), ("PATH", &["PATH_ENDS", "PATH_TRACE", "PATH_GEOMETRY", "PATH_POLES"]), @@ -104,7 +100,7 @@ pub(crate) fn generate() { for entry in ENTRIES { for &(variant, features) in entry.variants { let source = link(&root, entry.module, features, false); - // A custom marker's variant lacks the marker an application supplies; a stand-in validates it. + // Custom variants lack the application's marker, so validation appends a placeholder. let custom = features.contains(&"custom"); let checked = if custom { format!("{source}\n{CUSTOM_MARKER}") } else { source.clone() }; let (module, info) = validate(&format!("{}.{variant}", entry.module), &checked); @@ -130,7 +126,7 @@ pub(crate) fn generate() { fs::write(out.join("shaders.wgsl"), &pieces.text).unwrap(); writeln!( rust, - "/// Where each declaration lies in `shaders.wgsl`.\npub(crate) static PIECES: &[(u32, u32)] = &{:?};\n", + "/// Each declaration's byte range in `shaders.wgsl`.\npub(crate) static PIECES: &[(u32, u32)] = &{:?};\n", pieces.ranges ) .unwrap(); @@ -140,7 +136,7 @@ pub(crate) fn generate() { fs::write(out.join("shaders.json"), format!("[\n{}\n]\n", manifest.join(",\n"))).unwrap(); } -/// Top-level declarations, each stored once: their text back to back, and where each lies. +/// Deduplicated top-level declarations: their concatenated text and each one's byte range. #[derive(Default)] struct Pieces { text: String, @@ -149,8 +145,8 @@ struct Pieces { } impl Pieces { - /// The declarations of `source`, as indices, storing those not stored yet; they join back into - /// `source` exactly. + /// Splits `source` into declarations and returns their indices, storing new ones. The + /// declarations join back into exactly `source`. fn add(&mut self, source: &str) -> Vec { let refs: Vec = source .split(SEPARATOR) @@ -172,11 +168,12 @@ impl Pieces { } } -/// What separates top-level declarations in a linked module, and joins them back. +/// The separator between top-level declarations in a linked module, used to split and rejoin them. const SEPARATOR: &str = "\n\n"; -/// A variant as JSON: its file, whether it draws, its compute entry points, every combination of the -/// pipeline constants it declares, and the WGSL appended to make it whole (a custom marker's stand-in). +/// A variant as JSON: its file, whether it draws, its compute entry points, every combination of +/// its pipeline constants that pipelines use, and the WGSL to append to complete it (the +/// placeholder custom marker). fn manifest_entry(shader: &str, file: &str, module: &naga::Module, types: &naga::Module, custom: bool) -> String { let compute: Vec = module .entry_points @@ -218,9 +215,9 @@ fn manifest_entry(shader: &str, file: &str, module: &naga::Module, types: &naga: ) } -/// Whether a pipeline of `module` is ever made with `constants`: only channels go without joins or -/// samples; faces read their ends and areas their geometry; a pipeline made for points reads no path, -/// so it keeps the default one, and one made for triangles reads a face's ends or an area's geometry. +/// Whether any pipeline of `module` uses `constants`. Only channel evaluation (`compute::evaluate`) +/// runs without joins or samples. Faces read their ends, and areas their geometry. Points read no +/// path, so they keep the default `PATH_ENDS`. Triangles read a face's ends or an area's geometry. fn made(module: &str, constants: &[(&str, u32)], types: &naga::Module) -> bool { let value = |name: &str| constants.iter().find(|(n, _)| *n == name).map(|&(_, v)| v); if module != "compute::evaluate" && (value("JOINED") == Some(0) || value("TIMED") == Some(0)) { @@ -284,7 +281,7 @@ fn validate(name: &str, source: &str) -> (naga::Module, ModuleInfo) { (module, info) } -/// Every `const` of a scalar type, as a Rust constant. +/// Writes every `u32`, `i32`, and `f32` constant as a Rust constant. fn mirror_consts(module: &naga::Module, rust: &mut String) { for (_, constant) in module.constants.iter() { let Some(name) = &constant.name else { continue }; @@ -299,7 +296,7 @@ fn mirror_consts(module: &naga::Module, rust: &mut String) { rust.push('\n'); } -/// Every struct as a `#[repr(C)]` mirror with explicit padding. +/// Writes every struct as a `#[repr(C)]` mirror with explicit padding and a size assertion. fn mirror_structs(module: &naga::Module, rust: &mut String) { for (_, ty) in module.types.iter() { let (Some(name), TypeInner::Struct { members, span }) = (&ty.name, &ty.inner) else { continue }; @@ -355,7 +352,8 @@ fn rust_type(module: &naga::Module, ty: naga::Handle) -> (String, u3 } } -/// The bindings of `group` as `Binding` literals; with `info`, visibility is where entry points use them. +/// The bindings of `group` as `Binding` literals. With `info`, each binding is visible to the +/// stages whose entry points use it. Without `info`, every stage sees it. fn bindings(module: &naga::Module, info: Option<&ModuleInfo>, group: u32) -> Vec { let mut found = BTreeMap::new(); for (handle, global) in module.global_variables.iter() { @@ -403,12 +401,12 @@ fn bindings(module: &naga::Module, info: Option<&ModuleInfo>, group: u32) -> Vec let dynamic = global.name.as_deref().is_some_and(|name| DYNAMIC.contains(&name)); let entry = format!("Binding {{ slot: {}, kind: {kind}, stages: {stages}, dynamic: {dynamic} }}", binding.binding); - assert!(found.insert(binding.binding, entry).is_none(), "binding {group}.{} twice", binding.binding); + assert!(found.insert(binding.binding, entry).is_none(), "binding {group}.{} declared twice", binding.binding); } found.into_values().collect() } -/// Every group 0 and 1 binding a shader declares must match the shared declaration. +/// Asserts that every group 0 and 1 binding a shader declares matches the shared declaration. fn check_shared(module: &naga::Module, shared: &naga::Module, name: &str) { for (_, global) in module.global_variables.iter() { let Some(binding) = &global.binding else { continue }; diff --git a/crates/latkit/shaders/compute/bounds.wesl b/crates/latkit/shaders/compute/bounds.wesl index afe01bbf..51a62494 100644 --- a/crates/latkit/shaders/compute/bounds.wesl +++ b/crates/latkit/shaders/compute/bounds.wesl @@ -1,16 +1,18 @@ -// What marks draw, measured for fitting: the least and greatest x, y, and z of each mark's shown -// instances in position units, as ordered words, through the same geometry the draws use: a vertex -// as its box, a stroke as its points, an area as its triangles' corners. While a fit measures chosen -// rows, only rows marked FIT count. +// Extents for fitting: the least and greatest x, y, and z of each mark's shown instances, in +// position units, as order-preserving words. It uses the same geometry as the draws: a vertex's +// box, a stroke's points, and an area's triangle corners. While a fit measures chosen rows, only +// rows marked `FIT` count. import package::lib::types::{NONE, FIT, PATH_ENDS, PATH_GEOMETRY, EXTENT_WORDS, VERTEX_POSITION, HALF_UNITS, item}; -import package::lib::bind::{frame, eye, mark, state}; +import package::lib::bind::{frame, mark}; import package::lib::focus::focus_of; +import package::lib::mark::px_per_length; import package::lib::reduce::ordered; import package::lib::state::{row_at, vertex_at}; import package::lib::points::{placed, vertex_half}; import package::lib::path::{PATH, segment, segment_ends, stroke_style, stroke_instances}; -import package::lib::triangles::{triangle_corner, triangle_row, triangle_instances}; +import package::lib::triangles::{triangle_corner, triangle_instances}; +import package::lib::drawn::held_row; @group(2) @binding(2) var extents: array>; @@ -31,18 +33,22 @@ fn counted(row: u32) -> bool { @compute @workgroup_size(256) fn points(@builtin(global_invocation_id) id: vec3u) { let i = item(id); - if i >= mark.rows || !counted(i) { + if i >= mark.rows { return; } - let v = vertex_at(row_at(i)); + let row = held_row(i); + if !counted(row) { + return; + } + let v = vertex_at(row_at(row)); if (v.flags & 1u) == 0u || !placed(v) { return; } - // A marker sized in units is part of the box. One sized in pixels is that size at any zoom, so - // it counts as its position, and the fit's padding holds its pixels: measured at the zoom shown, - // its pixels would stand for any number of units. + // A marker sized in units is part of the box. A marker sized in pixels has that size at any + // zoom, so only its position counts, and the fit's padding covers its pixels. Measured at the + // current zoom, its pixels could correspond to any number of units. let units = (v.flags & HALF_UNITS) != 0u; - let half = select(vec2f(0.0), vertex_half(v) / eye.px_per_unit.x, units); + let half = select(vec2f(0.0), vertex_half(v) / px_per_length(), units); let p = v.hi + v.lo; extend(vec3f(p - half, v.elevation)); extend(vec3f(p + half, v.elevation)); @@ -55,7 +61,7 @@ fn strokes(@builtin(global_invocation_id) id: vec3u) { return; } let s = segment(i); - if s.row >= mark.rows || !counted(s.row) || !stroke_style(s.row).shown { + if s.item >= mark.rows || !counted(s.row) || !stroke_style(s.row).shown { return; } let ends = segment_ends(s); @@ -72,8 +78,8 @@ fn areas(@builtin(global_invocation_id) id: vec3u) { if PATH != PATH_GEOMETRY || i >= triangle_instances() { return; } - let row = triangle_row(i); - if row >= mark.rows || !counted(row) { + let s = segment(i); + if s.item >= mark.rows || !counted(s.row) { return; } for (var k = 0u; k < 3u; k++) { diff --git a/crates/latkit/shaders/compute/candidates.wesl b/crates/latkit/shaders/compute/candidates.wesl deleted file mode 100644 index 6ddc0ce1..00000000 --- a/crates/latkit/shaders/compute/candidates.wesl +++ /dev/null @@ -1,71 +0,0 @@ -// The first step of label placement: each labeled row's label measured where its anchor puts it, by -// the size of its text, or an estimate of it while the text is not set, and keyed by its priority, -// then whether it showed before, then its id: a label that arrives later still displaces lesser -// ones, and among equals those shown keep their place. `compute/labels.wesl` places them. Made for -// one primitive, it holds only that one's anchors. A diagram's labels scale with its sheet, so they -// need no placement: only this step runs there, to ask for the texts of labels in view. - -import package::lib::types::{ - VISIBLE, LABEL_CODE, LABEL_PRIORITY, CANDIDATE_BOX, CANDIDATE_KEY, CANDIDATE_STATE, WANTS, PLACED_LABEL, - STICKY, ESTIMATED, TEXT_ESTIMATE, SPACE_DIAGRAM, item, -}; -import package::lib::bind::{frame, eye, mark, state}; -import package::lib::state::extra_at; -import package::lib::read::missing_bits; -import package::lib::label::{anchor, length_px, legibility, text_at, fits, in_view}; -import package::lib::placement::{word, put, candidate, want, seen}; - -@compute @workgroup_size(256) -fn candidates(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows { - return; - } - let at = candidate(row); - let before = word(at + CANDIDATE_STATE); - let sticky = (before & PLACED_LABEL) != 0u; - let code_bits = state[extra_at(row, mark.label + LABEL_CODE)]; - var next = select(0u, STICKY, sticky); - if missing_bits(code_bits) || (state[mark.state + row * mark.stride + mark.stride - 1u] & VISIBLE) == 0u { - put(at + CANDIDATE_STATE, next); - return; - } - let code = u32(bitcast(code_bits)); - let uv0 = word(mark.runs + 2u * code); - let uv1 = word(mark.runs + 2u * code + 1u); - let a = anchor(row); - let size = length_px(mark.label_size); - let diagram = frame.space == SPACE_DIAGRAM; - let estimated = (uv0 & TEXT_ESTIMATE) != 0u; - if (uv0 == 0u && uv1 == 0u) || (estimated && diagram) { - // The atlas lacks this text, and the label cannot wait for placement to ask for it: ask now, - // if it could show, legible and near the view. - let near = 0.25 * eye.size; - if a.shown && legibility(size) > 0.0 && in_view(a.at - near, a.at + near) { - want(code); - } - put(at + CANDIDATE_STATE, next); - return; - } - if estimated { - next |= ESTIMATED; - } - let t = text_at(uv0 & ~TEXT_ESTIMATE, uv1, size, a.at, a.align); - if a.shown && in_view(t.lo, t.hi) { - // A diagram shows every label in view that fits; elsewhere `ask` notes those placed. - if diagram { - seen(code); - } - if fits(t.hi - t.lo, a.room) { - next |= WANTS; - } - } - put(at + CANDIDATE_BOX, bitcast(t.lo.x)); - put(at + CANDIDATE_BOX + 1u, bitcast(t.lo.y)); - put(at + CANDIDATE_BOX + 2u, bitcast(t.hi.x)); - put(at + CANDIDATE_BOX + 3u, bitcast(t.hi.y)); - let level = u32(clamp(bitcast(state[extra_at(row, mark.label + LABEL_PRIORITY)]), 0.0, 255.0)); - let id_bits = (mark.label_id + row + 1u) & 0x7fffffu; - put(at + CANDIDATE_KEY, ((255u - level) << 24u) | select(0x800000u, 0u, sticky) | id_bits); - put(at + CANDIDATE_STATE, next); -} diff --git a/crates/latkit/shaders/compute/compose.wesl b/crates/latkit/shaders/compute/compose.wesl index 25690aa3..70d43c37 100644 --- a/crates/latkit/shaders/compute/compose.wesl +++ b/crates/latkit/shaders/compute/compose.wesl @@ -1,6 +1,6 @@ -// A shot's panels composed into the bytes a consumer takes, so no pixel is touched on the CPU: RGBA -// rows with straight alpha for an image, or NV12 planes (BT.709, limited range) for an encoder. One -// dispatch per panel writes its panel of the picture in whole words; a fill writes the background +// Composes a shot's panels into output bytes on the GPU, so the CPU touches no pixels: RGBA rows +// with straight alpha for an image, or NV12 planes (BT.709, limited range) for a video encoder. One +// dispatch per panel writes its part of the picture in whole words. A fill writes the background // where no panel draws. import package::lib::types::Compose; @@ -14,8 +14,8 @@ const LUMA: vec3f = vec3f(0.2126, 0.7152, 0.0722); const BLUE: f32 = 1.8556; const RED: f32 = 1.5748; -// A panel's premultiplied color as the picture keeps it: straight with its alpha in a transparent -// image, else over black. +// A panel's premultiplied color as the picture stores it: straight alpha in a transparent image, +// else composited over black. fn kept(c: vec4f) -> vec4f { if compose.alpha == 0u { return vec4f(c.rgb, 1.0); @@ -26,7 +26,7 @@ fn kept(c: vec4f) -> vec4f { return vec4f(c.rgb / c.a, c.a); } -// Writes the 4×2 block of NV12 at `at` from its eight colors, rows of four. +// Writes the 4×2 NV12 block at `at` from its eight colors, in two rows of four. fn write_block(at: vec2u, rgb: array) { var chroma = vec4f(0.0); for (var row = 0u; row < 2u; row++) { @@ -35,7 +35,7 @@ fn write_block(at: vec2u, rgb: array) { let c = rgb[row * 4u + column]; let y = dot(c, LUMA); luma[column] = y; - // Cb and Cr of each 2×2, averaged. + // Cb and Cr, averaged over each 2×2. let pair = column / 2u * 2u; chroma[pair] += (c.b - y) / BLUE * 0.25; chroma[pair + 1u] += (c.r - y) / RED * 0.25; diff --git a/crates/latkit/shaders/compute/evaluate.wesl b/crates/latkit/shaders/compute/evaluate.wesl index 56726384..3cafc7ad 100644 --- a/crates/latkit/shaders/compute/evaluate.wesl +++ b/crates/latkit/shaders/compute/evaluate.wesl @@ -1,53 +1,64 @@ -// The expression machine: every row of a mark runs the mark's channels, which store into what the -// view binds as its state. All rows run the same instructions, so control flow stays uniform across a -// workgroup. A dispatch covers the row blocks the mark lists, only those whose rows changed when only -// those did. +// The expression machine, a stack interpreter. Each row of a mark runs the mark's channels, which +// store into the row's state words. All rows run the same instructions, so control flow stays +// uniform across a workgroup. A dispatch covers the row pages the mark lists, which are only the +// changed pages when only some rows changed. // -// Evaluating knows nothing of transitions: while a mark eases, the view binds its settled mirror as -// the state, and each frame `ease` writes what every row shows from its frozen and settled state by -// the mark's progress: numbers and colors mix, positions move by their offset, a row whose visibility -// flipped fades, rows past `mark.rows` left and fade out, and the rest show as evaluated. No channel -// runs to ease. +// Evaluation ignores transitions. During a transition, channels store into the settled words, and +// each frame `ease` computes what every row shows from its frozen and settled words by the mark's +// progress. Numbers and colors mix, positions move by their offset, rows whose visibility flipped +// fade, rows past the live ones (which left) fade out, and other words show as evaluated. Easing +// runs no channels. Every row here is a held row, whose words lie in pages listed by its mark's +// page tables. Resolving an edge's or face's vertices to held rows is a separate step +// (`lib/ends.wesl`), not a channel. import package::lib::types::{ - NONE, MISSING, MISSING_COLOR, STACK, VISIBLE, PLACED, FADE_IN, FADE_OUT, ENTERING, BLOCK_ROWS, item, - BLEND_KEEP, BLEND_NUMBER, BLEND_COLOR, BLEND_POSITION, STEP_REDUCE, + NONE, MISSING, MISSING_COLOR, STACK, VISIBLE, PLACED, FADE_IN, FADE_OUT, ENTERING, PAGE_ROWS, FRESH, item, + BLEND_KEEP, BLEND_NUMBER, BLEND_COLOR, BLEND_POSITION, STEP_REDUCE, STEP_ENDS, VERTEX_POSITION, VERTEX_ELEVATION, VERTEX_FLAGS, VERTEX_HALF, VERTEX_RANGE, HALF_UNITS, OWN_HEIGHT, SIZED, MARK_VERTICES, OP_CONST, OP_MISSING, OP_FIELD, OP_SELECTED, OP_HOVERED, OP_POINTER, OP_ELAPSED, OP_THEME, OP_ZOOM, OP_NEG, OP_NOT, OP_ABS, OP_SQRT, OP_SIN, OP_COS, OP_FLOOR, OP_LOG, OP_EXP, OP_ADD, OP_SUB, OP_MUL, OP_DIV, OP_MIN, OP_MAX, OP_MOD, OP_POW, OP_LT, OP_LE, OP_GT, OP_GE, OP_EQ, OP_NE, OP_AND, OP_OR, OP_SELECT, OP_SCALE, OP_COLORMAP, OP_STORE, OP_STORE_COLOR, OP_STORE_FLAG, OP_STORE_POSITION, - OP_POSITION, OP_STORE_END, OP_STORE_CODE, OP_REDUCE, + OP_POSITION, OP_STORE_CODE, OP_REDUCE, }; -import package::lib::bind::{frame, program, mark, state}; -import package::lib::state::{travel_words, pack_range}; -import package::lib::read::{column_value, bits_value, missing, missing_bits}; +import package::lib::bind::{frame, program, mark, pages, theme_word}; +import package::lib::state::{travel_words, pack_range, record_at}; +import package::lib::read::{column_value, missing, missing_bits}; import package::lib::focus::{is_selected, is_hovered}; import package::lib::color::{scale, colormap}; import package::lib::project::{project, to_screen}; import package::lib::reduce::{ordered, unordered}; +import package::lib::ends::resolve; -// Each row's state as drawn when the mark's transition began. -@group(2) @binding(1) var frozen: array; -// Per fitted domain: the order-preserving bits of its least and greatest value. +// Two words per fitted domain: the complement of its least value's order-preserving bits, then its +// greatest value's bits (see `lib/reduce.wesl`). @group(2) @binding(4) var domains: array>; -// Each row's state as its channels evaluated it, where it settles once the mark's transition ends: -// the state's mirror, which `ease` reads. -@group(2) @binding(14) var settled: array; -// Each mark's row blocks a dispatch covers: at `mark.block`, `mark.blocks` of them. +// Each mark's row blocks a dispatch covers. At `mark.block`: the blocks its channels evaluate, with +// `FRESH` on those whose state words start over. At `mark.end_block`: the blocks whose ends are +// resolved. @group(2) @binding(16) var blocks: array; -// What a dispatch of `evaluate` runs: the mark's channels over its listed blocks, or over every row -// the reductions fitting its domains. +// The step a dispatch of `evaluate` runs: the mark's channels over its listed blocks, the +// reductions that fit its domains over every row, or end resolution over the blocks listed for it. @group(2) @binding(17) var step: u32; -// Whether the pipeline's marks read the view: selection, hover, the pointer, the zoom. Only those -// made for marks that do compile it. +// Whether the pipeline's marks read view state: selection, hover, the pointer, or the zoom. Only +// pipelines made for marks that do compile those reads. override VIEWED: bool = true; -// The row invocation `id` evaluates: row `id.x` of the mark's `id.y`th listed block. +// The row invocation `id` evaluates: row `id.x` of the mark's `id.y`th listed page. fn row_of(id: vec3u) -> u32 { - return blocks[mark.block + id.y] * BLOCK_ROWS + id.x; + return (blocks[mark.block + id.y] & ~FRESH) * PAGE_ROWS + id.x; +} + +// Whether the state words of the row invocation `id` evaluates start over. +fn fresh(id: vec3u) -> bool { + return (blocks[mark.block + id.y] & FRESH) != 0u; +} + +// The mark's live row count. Rows past it left and are fading out. +fn live() -> u32 { + return mark.held - mark.exits; } fn constant(i: u32) -> u32 { @@ -58,7 +69,8 @@ fn constant_float(i: u32) -> f32 { return bitcast(constant(i)); } -// A domain record in constants: fitted flag, least, greatest, reduction index. +// The domain whose record starts at constant `at`: a fitted flag, least, greatest, and reduction +// index. A fitted domain reads its reduction, and is (0, 1) while no row has a value. fn domain(at: u32) -> vec2f { if constant(at) == 0u { return vec2f(constant_float(at + 1u), constant_float(at + 2u)); @@ -71,39 +83,46 @@ fn domain(at: u32) -> vec2f { return vec2f(unordered(~atomicLoad(&domains[d])), unordered(high)); } -fn theme_word(slot: u32) -> u32 { - return frame.theme[slot / 4u][slot % 4u]; -} - -// Where word `word` of `row` lives: a kind word, or past the kind words an extra word. -fn word_at(row: u32, word: u32) -> u32 { - if word < mark.stride { - return mark.state + row * mark.stride + word; - } - return mark.extra + row * mark.extras + (word - mark.stride); +// Where word `w` of `row` lies, given page table `table`. Words from `mark.stride` on are extra +// words. +fn word_in(table: u32, row: u32, w: u32) -> u32 { + return record_at(table, mark.stride + mark.extras, row) + w; } fn easing() -> bool { return mark.frozen != NONE; } -fn load(row: u32, word: u32) -> u32 { - return state[word_at(row, word)]; +// Where channels store word `w` of `row`: in the settled words during a transition (which `ease` +// reads), else in the state words. +fn word_at(row: u32, w: u32) -> u32 { + return word_in(select(mark.state, mark.settled, easing()), row, w); } -fn store(row: u32, word: u32, value: u32) { - state[word_at(row, word)] = value; +fn load(row: u32, w: u32) -> u32 { + return pages[word_at(row, w)]; } -// Word `word` of `row` as it was drawn when the transition began. -fn was(row: u32, word: u32) -> u32 { - if word < mark.stride { - return frozen[mark.frozen + row * mark.stride + word]; - } - return frozen[mark.frozen_extra + row * mark.extras + (word - mark.stride)]; +fn store(row: u32, w: u32, value: u32) { + pages[word_at(row, w)] = value; } -// Whether `row` has nothing to ease from: it entered with this change. +// Word `w` of `row` as drawn when the transition began. +fn was(row: u32, w: u32) -> u32 { + return pages[word_in(mark.frozen, row, w)]; +} + +// Word `w` of `row` as its channels evaluated it: the value the transition settles on. +fn settled(row: u32, w: u32) -> u32 { + return pages[word_in(mark.settled, row, w)]; +} + +// Where the shown value of word `w` of `row` lies. +fn shown_at(row: u32, w: u32) -> u32 { + return word_in(mark.state, row, w); +} + +// Whether `row` entered with this change, leaving nothing to ease from. fn entering(row: u32, flags: u32) -> bool { return (was(row, flags) & ENTERING) != 0u; } @@ -112,25 +131,25 @@ fn number(x: u32) -> f32 { return bitcast(x); } -fn word(x: f32) -> u32 { +fn bits(x: f32) -> u32 { return bitcast(x); } fn truth(x: bool) -> u32 { - return word(f32(x)); + return bits(f32(x)); } // CSS pixels from a vertex to the pointer. fn pointer_distance(row: u32) -> f32 { let at = word_at(row, VERTEX_POSITION); - let hi = vec2f(bitcast(state[at]), bitcast(state[at + 1u])); - let lo = vec2f(bitcast(state[at + 2u]), bitcast(state[at + 3u])); - let z = bitcast(state[word_at(row, VERTEX_ELEVATION)]); + let hi = vec2f(bitcast(pages[at]), bitcast(pages[at + 1u])); + let lo = vec2f(bitcast(pages[at + 2u]), bitcast(pages[at + 3u])); + let z = bitcast(pages[word_at(row, VERTEX_ELEVATION)]); return distance(to_screen(project(hi, lo, z)), frame.pointer); } -// What the view knows of `row` and of itself, as op `code` reads it: whether it is selected or -// hovered, how far it is from the pointer, the zoom. +// The view state op `code` reads for `row`: whether it is selected or hovered, its distance to the +// pointer, or the zoom. fn viewed(code: u32, row: u32) -> u32 { if !VIEWED { return MISSING; @@ -138,8 +157,8 @@ fn viewed(code: u32, row: u32) -> u32 { switch code { case OP_SELECTED: { return truth(is_selected(row)); } case OP_HOVERED: { return truth(is_hovered(row)); } - case OP_POINTER: { return word(pointer_distance(row)); } - default: { return word(frame.zoom); } + case OP_POINTER: { return bits(pointer_distance(row)); } + default: { return bits(frame.zoom); } } } @@ -185,7 +204,7 @@ fn fallback(x: u32, slot: u32) -> u32 { return x; } -// A number eased from what `row` showed; a row that entered shows its own at once, and appears as +// A number eased from the value `row` showed. A row that entered shows its new value at once, and // its draw grows it in. fn ease_number(row: u32, w: u32, now: u32, flags: u32) -> u32 { if !easing() || missing_bits(now) || entering(row, flags) { @@ -195,7 +214,7 @@ fn ease_number(row: u32, w: u32, now: u32, flags: u32) -> u32 { if missing_bits(start) { return now; } - return word(mix(number(start), number(now), mark.motion)); + return bits(mix(number(start), number(now), mark.motion)); } fn ease_color(row: u32, w: u32, now: u32, flags: u32) -> u32 { @@ -205,21 +224,22 @@ fn ease_color(row: u32, w: u32, now: u32, flags: u32) -> u32 { return pack4x8unorm(mix(unpack4x8unorm(was(row, w)), unpack4x8unorm(now), mark.motion)); } -// A position's high word `w` (x or y) eased by its offset from what the row showed, so the -// destination keeps its precision; its low word, two after it, shows as evaluated. +// A position's high word `w` (x or y), moved by the remaining offset from the old position, so the +// destination keeps its precision. Its low word, two words later, shows as evaluated. fn ease_position(row: u32, w: u32, now: u32, flags: u32) -> u32 { if !easing() || entering(row, flags) || (was(row, flags) & PLACED) == 0u { return now; } - let now_lo = number(settled[word_at(row, w + 2u)]); + let now_lo = number(settled(row, w + 2u)); let offset = (number(was(row, w)) - number(now)) + (number(was(row, w + 2u)) - now_lo); - return word(number(now) + offset * (1.0 - mark.motion)); + return bits(number(now) + offset * (1.0 - mark.motion)); } -// The flags `row` shows while its mark eases: as evaluated, but a row whose visibility flipped stays -// drawn and fades, in where it became visible or entered, out where it stopped being. +// The flags `row` shows during a transition: as evaluated, except that a row whose visibility +// flipped stays drawn and fades. It fades in if it became visible or entered, and out if it stopped +// being visible. fn faded(row: u32, flags: u32) -> u32 { - let now = settled[word_at(row, flags)] & ~(FADE_IN | FADE_OUT); + let now = settled(row, flags) & ~(FADE_IN | FADE_OUT); let fresh = entering(row, flags); let before = fresh || (was(row, flags) & VISIBLE) != 0u; let on = (now & VISIBLE) != 0u; @@ -232,14 +252,15 @@ fn faded(row: u32, flags: u32) -> u32 { return now; } -// Sets or clears `bit` of the flags at word `w`; whether the row fades is the ease's to show. +// Sets or clears `bit` of the flags at word `w`. It also clears `ENTERING` and the fade bits, which +// only transitions set. fn store_flag(row: u32, w: u32, bit: u32, on: bool) { let stored = load(row, w); store(row, w, select(stored & ~bit, stored | bit, on) & ~(ENTERING | FADE_IN | FADE_OUT)); } -// A position, from data or in a diagram from its sheet, as high and low parts of x and y; a row -// without one is not placed. +// Stores a position (from data, or in a diagram from its sheet) as the high and low parts of x and +// y, and sets `PLACED`. A missing x or y clears `PLACED` instead. fn position(row: u32, w: u32, x: vec2u, y: vec2u) { let flags = w + VERTEX_FLAGS - VERTEX_POSITION; if missing_bits(x.x) || missing_bits(y.x) { @@ -267,15 +288,15 @@ fn run(row: u32, start: u32, end: u32) { case OP_MISSING: { stack[sp] = MISSING; sp++; } case OP_FIELD: { stack[sp] = column_value(a, row).x; sp++; } case OP_SELECTED, OP_HOVERED, OP_POINTER, OP_ZOOM: { stack[sp] = viewed(code, row); sp++; } - case OP_ELAPSED: { stack[sp] = word(frame.elapsed); sp++; } + case OP_ELAPSED: { stack[sp] = bits(frame.elapsed); sp++; } case OP_THEME: { stack[sp] = theme_word(a); sp++; } case OP_NEG, OP_NOT, OP_ABS, OP_SQRT, OP_SIN, OP_COS, OP_FLOOR, OP_LOG, OP_EXP: { - stack[sp - 1u] = word(unary(code, number(stack[sp - 1u]))); + stack[sp - 1u] = bits(unary(code, number(stack[sp - 1u]))); } case OP_ADD, OP_SUB, OP_MUL, OP_DIV, OP_MIN, OP_MAX, OP_MOD, OP_POW, OP_LT, OP_LE, OP_GT, OP_GE, OP_EQ, OP_NE, OP_AND, OP_OR: { sp--; - stack[sp - 1u] = word(binary(code, number(stack[sp - 1u]), number(stack[sp]))); + stack[sp - 1u] = bits(binary(code, number(stack[sp - 1u]), number(stack[sp]))); } case OP_SELECT: { sp -= 2u; @@ -283,7 +304,7 @@ fn run(row: u32, start: u32, end: u32) { } case OP_SCALE: { let range = vec2f(constant_float(b), constant_float(b + 1u)); - stack[sp - 1u] = word(scale(number(stack[sp - 1u]), domain(a), range)); + stack[sp - 1u] = bits(scale(number(stack[sp - 1u]), domain(a), range)); } case OP_COLORMAP: { stack[sp - 1u] = colormap(a & 0xffffu, a >> 16u, number(stack[sp - 1u]), domain(b)); @@ -311,12 +332,6 @@ fn run(row: u32, start: u32, end: u32) { case OP_POSITION: { position(row, a, column_value(b, row), column_value(c, row)); } - case OP_STORE_END: { - // Without a column, the end is the row itself, as a pole's. - let end_row = select(bits_value(b, row), row, b == NONE); - let first = program[mark.ends + c]; - store(row, a, select(first + end_row, NONE, end_row == NONE || first == NONE)); - } case OP_REDUCE: { sp--; let x = number(stack[sp]); @@ -331,32 +346,42 @@ fn run(row: u32, start: u32, end: u32) { } } -// A row that left fades out with its mark's transition; its frozen state is what it showed, but for -// its comets, which travel on. +// A row that left fades out during the transition. It shows its frozen words, except its comet +// travel, which keeps advancing. fn fade_exit(row: u32, flags: u32) { let travel = travel_words().x; for (var w = 0u; w < mark.stride; w++) { if w != travel { - state[word_at(row, w)] = was(row, w); + pages[shown_at(row, w)] = was(row, w); } } - state[word_at(row, flags)] = (was(row, flags) & ~FADE_IN) | FADE_OUT; + pages[shown_at(row, flags)] = (was(row, flags) & ~FADE_IN) | FADE_OUT; } @compute @workgroup_size(256) fn evaluate(@builtin(global_invocation_id) id: vec3u) { - // Reductions run over every row; channels over the listed blocks, those the mask names. One - // loop runs either, since a backend compiles each call of the machine whole. - let reducing = step == STEP_REDUCE; - var row: u32; - if reducing { - row = item(id); - } else { - row = row_of(id); + // Ends are resolved over the blocks listed for them. A row that left keeps the ends it had. + if step == STEP_ENDS { + let row = blocks[mark.end_block + id.y] * PAGE_ROWS + id.x; + if row < live() { + resolve(row); + } + return; } - if row >= mark.rows { + // Reductions run over every live block, which a domain change lists. Channels run over the + // listed blocks, and only the channels the mask selects. One loop runs either, because a + // backend compiles each call of the machine in full. + let reducing = step == STEP_REDUCE; + let row = row_of(id); + if row >= live() { return; } + // A row whose words start over is zeroed first, so a word no channel writes reads as zero. + if !reducing && fresh(id) { + for (var w = 0u; w < mark.stride + mark.extras; w++) { + pages[shown_at(row, w)] = 0u; + } + } let first = select(0u, mark.channels, reducing); let end = select(mark.channels, mark.channels + mark.reductions, reducing); for (var k = first; k < end; k++) { @@ -375,15 +400,15 @@ fn evaluate(@builtin(global_invocation_id) id: vec3u) { } } -// Each frame a mark eases toward its settled state: every word a channel stores shows the frozen -// state moved toward the settled one by the mark's progress, as the program says each eases, and the -// flags show rows fading; rows that left fade out. No channel runs. +// Writes what every row shows, each frame of a transition. Each word a channel stores moves from +// its frozen value toward its settled value by the mark's progress, as its blend says, and the +// flags show fading rows. Rows that left fade out. No channel runs. @compute @workgroup_size(256) fn ease(@builtin(global_invocation_id) id: vec3u) { let row = item(id); let flags = mark.stride - 1u; - if row >= mark.rows { - if row < mark.rows + mark.exits { + if row >= live() { + if row < mark.held { fade_exit(row, flags); } return; @@ -393,7 +418,7 @@ fn ease(@builtin(global_invocation_id) id: vec3u) { if blend == BLEND_KEEP { continue; } - let now = settled[word_at(row, w)]; + let now = settled(row, w); var shown = now; switch blend { case BLEND_NUMBER: { shown = ease_number(row, w, now, flags); } @@ -401,38 +426,38 @@ fn ease(@builtin(global_invocation_id) id: vec3u) { case BLEND_POSITION: { shown = ease_position(row, w, now, flags); } default: {} } - state[word_at(row, w)] = shown; + pages[shown_at(row, w)] = shown; } - state[word_at(row, flags)] = faded(row, flags); + pages[shown_at(row, flags)] = faded(row, flags); } -// Comets travel on by each stroke row's flow over the frame's step, so a flow that changes carries -// on from where its comets are. Wrapped at their spacing, travel keeps its precision however long -// they run. +// Advances each stroke row's comets by its flow times the frame's time step, so a changed flow +// continues from where the comets are. Travel wraps at the comet spacing, so it keeps its precision +// however long comets run. @compute @workgroup_size(256) fn travel(@builtin(global_invocation_id) id: vec3u) { let row = item(id); let words = travel_words(); - if row >= mark.rows + mark.exits || words.x == NONE { + if row >= mark.held || words.x == NONE { return; } - let at = word_at(row, words.x); - let flow = number(state[word_at(row, words.y)]); + let at = shown_at(row, words.x); + let flow = number(pages[shown_at(row, words.y)]); let spacing = max(frame.flow_spacing, 1e-6); - var moved = number(state[at]) + select(flow, 0.0, missing(flow)) * frame.flow_step; + var moved = number(pages[at]) + select(flow, 0.0, missing(flow)) * frame.flow_step; moved -= spacing * floor(moved / spacing); - // Words a row never held read as no travel. - state[at] = word(select(0.0, moved, moved >= 0.0 && moved <= spacing)); + // A result outside [0, spacing], such as NaN from a word never written, reads as no travel. + pages[at] = bits(select(0.0, moved, moved >= 0.0 && moved <= spacing)); } -// Where a transition moved rows, their comets carry on from where they were: each row takes the -// travel its frozen state holds. +// When a transition moves rows, their comets continue from where they were: each row takes the +// travel from its frozen words. @compute @workgroup_size(256) fn carry(@builtin(global_invocation_id) id: vec3u) { let row = item(id); let travel = travel_words().x; - if row >= mark.rows + mark.exits || travel == NONE || !easing() { + if row >= mark.held || travel == NONE || !easing() { return; } - state[word_at(row, travel)] = was(row, travel); + pages[shown_at(row, travel)] = was(row, travel); } diff --git a/crates/latkit/shaders/compute/fields.wesl b/crates/latkit/shaders/compute/fields.wesl index 2cbb8df8..a1596fdf 100644 --- a/crates/latkit/shaders/compute/fields.wesl +++ b/crates/latkit/shaders/compute/fields.wesl @@ -1,25 +1,27 @@ -// Distance fields of text: the coverage a host set for a run, turned into each texel's signed distance -// to the text's edge, inside positive, as a byte of `0.5 + distance / (2 TEXT_SPREAD)`, over the run -// widened by `TEXT_SPREAD` texels on every side. Partial coverage places the edge within its texel, so -// antialiasing becomes subtexel distance. +// Text distance fields. For each run, converts the coverage the host rasterized into each texel's +// signed distance to the text's edge (positive inside), stored as the byte +// `0.5 + distance / (2 TEXT_SPREAD)`. The field covers the run plus `TEXT_SPREAD` texels on each +// side. Partial coverage places the edge within its texel, so antialiased edges keep subtexel +// precision. // -// A texel's squared distance to the text is the least, over the field's texels, of each one's own (none -// inside the text, a partial texel's subtexel distance squared, far outside it) plus the squared -// distance to it; likewise to what lies outside the text. The least splits into one down each column, -// then one along each row, and only texels within the spread bring a texel within it, so each looks -// that far and no farther: exact wherever the field is not clamped. After the columns, one of a -// texel's two distances is its own, none, so one signed word holds both: to the outside above zero, to -// the text below. +// A texel's squared distance to the text is the minimum, over the field's texels, of each texel's +// own squared distance plus the squared distance to that texel. A texel's own squared distance is 0 +// inside the text, its subtexel distance squared where coverage is partial, and effectively +// infinite outside. The distance to the outside works the same way. The minimum splits into a pass +// down each column, then a pass along each row. Only texels within the spread can bring a texel +// within the spread, so each pass looks only that far, which is exact wherever the field is not +// clamped. After the column pass, one of a texel's two distances is always 0, so one signed word +// holds both: the distance to the outside above zero, and to the text below. import package::lib::types::{ TEXT_SPREAD, FIELD_COVERAGE, FIELD_WIDTH, FIELD_HEIGHT, FIELD_COLUMNS, FIELD_AT, FIELD_ROW, FIELD_WORDS, }; -// The runs, `FIELD_WORDS` each, then their coverage, four bytes a word, their columns' least -// distances, and their fields, four texels a word. +// The runs, `FIELD_WORDS` words each, then their coverage (four bytes per word), their column +// minima, and their fields (four texels per word). @group(2) @binding(0) var fields: array; -// Texels the field reaches past the text, and the column minima a word of four texels reads. +// `SPREAD`: texels the field reaches past the text. `NEAR`: column minima read per 4-texel word. const SPREAD: i32 = i32(TEXT_SPREAD); const NEAR: i32 = 4 + 2 * SPREAD; // Farther than any texel within the spread. @@ -31,7 +33,7 @@ struct Run { size: vec2u, columns: u32, at: u32, - // Words between the starts of the field's rows. + // Words from the start of one field row to the next. row: u32, } @@ -46,7 +48,7 @@ fn run(i: u32) -> Run { ); } -// Texels across and down its field: its coverage's, and the spread on every side. +// The field's width and height in texels: the coverage plus the spread on every side. fn field_size(r: Run) -> vec2u { return r.size + vec2u(2u * u32(SPREAD)); } @@ -70,7 +72,8 @@ fn own(r: Run, p: vec2i) -> vec2f { return select(vec2f(0.0, d * d), vec2f(d * d, 0.0), d > 0.0); } -// Down each column: the least of each texel's own within the spread above and below. +// Column pass: for each texel, the least over the texels within the spread above and below it of +// their own squared distance plus the squared vertical distance. @compute @workgroup_size(16, 16) fn columns(@builtin(global_invocation_id) id: vec3u) { let r = run(id.z); @@ -88,8 +91,8 @@ fn columns(@builtin(global_invocation_id) id: vec3u) { fields[r.columns + id.y * size.x + id.x] = bitcast(least.y - least.x); } -// Along each row, four texels a word: the least of the columns' within the spread to either side, -// then each texel's byte. +// Row pass, four texels per word: for each texel, the least over the column minima within the +// spread to either side of each plus the squared horizontal distance, then the texel's byte. @compute @workgroup_size(16, 16) fn rows(@builtin(global_invocation_id) id: vec3u) { let r = run(id.z); @@ -97,7 +100,7 @@ fn rows(@builtin(global_invocation_id) id: vec3u) { if id.x * 4u >= size.x || id.y >= size.y { return; } - // Past the field's edge lies nothing. + // Nothing lies past the field's edge. var near: array; let first = i32(id.x * 4u) - SPREAD; for (var k = 0; k < NEAR; k++) { diff --git a/crates/latkit/shaders/compute/freeze.wesl b/crates/latkit/shaders/compute/freeze.wesl index a5438e0c..ef7be83a 100644 --- a/crates/latkit/shaders/compute/freeze.wesl +++ b/crates/latkit/shaders/compute/freeze.wesl @@ -1,45 +1,55 @@ -// Freezing: what each row of a mark showed when its transition began, gathered into the frozen -// buffer at the row's new place. The remap words list where each new row was (NONE for a row that -// entered), then the rows that left, which fade out past `mark.rows`. Slot words of a mark whose -// ends point at vertices follow them to their new slots. +// Freezing: copies what each row showed when its transition began into its frozen words, at the +// row's new held row. The remap words at `mark.remap` hold a copy of the page table of the words +// shown before the change, how many rows showed, and each new row's old row (NONE for a row that +// entered), followed by the rows that left, which fade out past the live ones. Without that list, +// every row keeps its held row. For each kind word naming a vertex, the remap words also say where +// that vertex mark's rows moved, so the ends of rows easing out follow them. -import package::lib::types::{NONE, ENTERING, item}; -import package::lib::bind::{mark, state}; +import package::lib::types::{NONE, ENTERING, PAGE_BITS, PAGE_MASK, item}; +import package::lib::bind::{mark, pages}; +import package::lib::state::record_at; -@group(2) @binding(1) var frozen: array; -// At `mark.remap`: the old kind and extra offsets, a mask of slot words, where the slot table -// starts or NONE, then a source row per new row and exit. @group(2) @binding(12) var remap: array; +// Remap header words: where the page table of the shown words starts, how many rows showed, where +// each new row's old row is listed (NONE when no row moved), and where each kind word's vertex +// moves start (NONE when no vertex moved). +const REMAP_TABLE: u32 = 0u; +const REMAP_SHOWN: u32 = 1u; +const REMAP_SOURCES: u32 = 2u; +const REMAP_ENDS: u32 = 3u; + @compute @workgroup_size(256) fn gather(@builtin(global_invocation_id) id: vec3u) { let i = item(id); - if i >= mark.rows + mark.exits { + if i >= mark.held { return; } - let old_state = remap[mark.remap]; - let old_extra = remap[mark.remap + 1u]; - let slots = remap[mark.remap + 2u]; - let table = remap[mark.remap + 3u]; - let source = remap[mark.remap + 4u + i]; + let header = mark.remap; + let sources = remap[header + REMAP_SOURCES]; + var source = select(NONE, i, i < remap[header + REMAP_SHOWN]); + if sources != NONE { + source = remap[sources + i]; + } + let words = mark.stride + mark.extras; + let to = record_at(mark.frozen, words, i); let flags = mark.stride - 1u; - for (var w = 0u; w < mark.stride; w++) { - var word = 0u; - if source == NONE { - word = select(0u, ENTERING, w == flags); - } else { - word = state[old_state + source * mark.stride + w]; - if ((slots >> w) & 1u) != 0u && word != NONE && table != NONE { - word = remap[table + word]; - } + if source == NONE { + for (var w = 0u; w < words; w++) { + pages[to + w] = select(0u, ENTERING, w == flags); } - frozen[mark.frozen + i * mark.stride + w] = word; + return; } - for (var w = 0u; w < mark.extras; w++) { - var word = 0u; - if source != NONE { - word = state[old_extra + source * mark.extras + w]; + let shown = remap[remap[header + REMAP_TABLE] + (source >> PAGE_BITS)] + (source & PAGE_MASK) * words; + let ends = remap[header + REMAP_ENDS]; + for (var w = 0u; w < words; w++) { + var word = pages[shown + w]; + // A vertex mark's moves: its old held row count, then each old row's new held row, NONE for + // a row that left. + let moves = select(NONE, remap[ends + min(w, mark.stride - 1u)], ends != NONE && w < mark.stride); + if moves != NONE && word != NONE { + word = select(NONE, remap[moves + 1u + word], word < remap[moves]); } - frozen[mark.frozen_extra + i * mark.extras + w] = word; + pages[to + w] = word; } } diff --git a/crates/latkit/shaders/compute/labels.wesl b/crates/latkit/shaders/compute/labels.wesl deleted file mode 100644 index b3979adb..00000000 --- a/crates/latkit/shaders/compute/labels.wesl +++ /dev/null @@ -1,242 +0,0 @@ -// Label placement, once `compute/candidates.wesl` measured each label: rounds of `claim` (each -// label wanting room takes the least key into every cell it covers, or drops if one is held), -// `decide` (a label holding all its cells is placed), and `occupy` (placed labels hold their cells) -// settle, with cells not held cleared between rounds. That places labels as greedily by key as a -// sequential pass would. `ask` then asks for the texts of placed labels the atlas lacks and notes -// those it holds, so it keeps them: texts are set only for labels that show, however many rows are in -// view. `fade` eases each label in or out. Placement only decides which labels show; draws place them -// from their anchors every frame. -// -// A mark that shows at most `label_max` labels keeps, of those wanting room, the ones of least key, -// before any claims: keys are unique, so a radix select over three digits of 11, 11, and 10 bits finds -// the greatest key kept exactly, each digit a pass over the candidates and one over its buckets. - -import package::lib::types::{ - NONE, LABEL_CODE, CANDIDATE_BOX, CANDIDATE_KEY, CANDIDATE_STATE, CANDIDATE_ALPHA, WANTS, PLACED_LABEL, DROPPED, - ESTIMATED, OCCUPIED, CELL, item, -}; -import package::lib::bind::{frame, mark, state}; -import package::lib::state::extra_at; -import package::lib::read::missing_bits; -import package::lib::placement::{labels, word, put, candidate, want, seen}; - -// Buckets of a digit of the select capping labels. -const SELECT_BUCKETS: u32 = 2048u; - -// Word `k` of the select's state, past its buckets: the key's digits found, how many of the least -// keys are still to keep, and the digit next counted (4 once fewer want room than the cap). -fn select_word(k: u32) -> u32 { - return mark.label_select + SELECT_BUCKETS + k; -} - -// Where digit `d` of a key starts, and how many bits it has. -fn digit(d: u32) -> vec2u { - switch d { - case 0u: { return vec2u(21u, 11u); } - case 1u: { return vec2u(10u, 11u); } - default: { return vec2u(0u, 10u); } - } -} - -fn wanting(row: u32) -> bool { - return (word(candidate(row) + CANDIDATE_STATE) & WANTS) != 0u; -} - -@compute @workgroup_size(256) -fn select_start(@builtin(global_invocation_id) id: vec3u) { - let i = item(id); - if mark.label_max == NONE { - return; - } - if i < SELECT_BUCKETS { - put(mark.label_select + i, 0u); - } - if i == 0u { - put(select_word(0u), 0u); - put(select_word(1u), mark.label_max); - put(select_word(2u), 0u); - } -} - -@compute @workgroup_size(256) -fn select_count(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - let d = word(select_word(2u)); - if row >= mark.rows || mark.label_max == NONE || d >= 3u || !wanting(row) { - return; - } - let key = word(candidate(row) + CANDIDATE_KEY); - let at = digit(d); - // Only keys whose higher digits are those found count. - let above = at.x + at.y; - if above < 32u && (key >> above) != word(select_word(0u)) { - return; - } - atomicAdd(&labels[mark.label_select + ((key >> at.x) & (SELECT_BUCKETS - 1u))], 1u); -} - -// The bucket holding the least key still to keep, clearing the buckets for the next digit. -@compute @workgroup_size(1) -fn select_find() { - let d = word(select_word(2u)); - // A cap of none keeps no label: the cut drops them all. - if mark.label_max == NONE || mark.label_max == 0u || d >= 3u { - return; - } - var left = word(select_word(1u)); - var found = SELECT_BUCKETS; - for (var b = 0u; b < SELECT_BUCKETS; b++) { - let n = word(mark.label_select + b); - put(mark.label_select + b, 0u); - if found == SELECT_BUCKETS { - if n >= left { - found = b; - } else { - left -= n; - } - } - } - if found == SELECT_BUCKETS { - // Fewer want room than the cap: each keeps its chance. - put(select_word(2u), 4u); - return; - } - put(select_word(0u), (word(select_word(0u)) << digit(d).y) | found); - put(select_word(1u), left); - put(select_word(2u), d + 1u); -} - -@compute @workgroup_size(256) -fn select_cut(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows || mark.label_max == NONE || !wanting(row) || word(select_word(2u)) == 4u { - return; - } - let at = candidate(row) + CANDIDATE_STATE; - if mark.label_max == 0u || word(candidate(row) + CANDIDATE_KEY) > word(select_word(0u)) { - put(at, word(at) & ~WANTS); - } -} - -// The cells row `row`'s label covers: least and greatest column and row. -fn cells_of(row: u32) -> vec4u { - let at = candidate(row); - let lo = vec2f(bitcast(word(at + CANDIDATE_BOX)), bitcast(word(at + CANDIDATE_BOX + 1u))); - let hi = vec2f(bitcast(word(at + CANDIDATE_BOX + 2u)), bitcast(word(at + CANDIDATE_BOX + 3u))); - let top = vec2f(f32(frame.cell_columns), f32(frame.cell_rows)) - 1.0; - let a = vec2u(clamp(floor(lo / CELL), vec2f(0.0), top)); - let b = vec2u(clamp(floor(hi / CELL), vec2f(0.0), top)); - return vec4u(a, b); -} - -fn cell(x: u32, y: u32) -> u32 { - return frame.cells + y * frame.cell_columns + x; -} - -// Whether row `row`'s label still wants room this round. -fn undecided(row: u32) -> bool { - let s = word(candidate(row) + CANDIDATE_STATE); - return (s & WANTS) != 0u && (s & (PLACED_LABEL | DROPPED)) == 0u; -} - -@compute @workgroup_size(256) -fn clear_cells(@builtin(global_invocation_id) id: vec3u) { - let i = item(id); - if i < frame.cell_columns * frame.cell_rows { - put(frame.cells + i, NONE); - } -} - -@compute @workgroup_size(256) -fn reset_cells(@builtin(global_invocation_id) id: vec3u) { - let i = item(id); - if i < frame.cell_columns * frame.cell_rows && word(frame.cells + i) != OCCUPIED { - put(frame.cells + i, NONE); - } -} - -@compute @workgroup_size(256) -fn claim(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows || !undecided(row) { - return; - } - let at = candidate(row); - let key = word(at + CANDIDATE_KEY); - let c = cells_of(row); - for (var y = c.y; y <= c.w; y++) { - for (var x = c.x; x <= c.z; x++) { - if word(cell(x, y)) == OCCUPIED { - put(at + CANDIDATE_STATE, word(at + CANDIDATE_STATE) | DROPPED); - return; - } - atomicMin(&labels[cell(x, y)], key); - } - } -} - -@compute @workgroup_size(256) -fn decide(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows || !undecided(row) { - return; - } - let at = candidate(row); - let key = word(at + CANDIDATE_KEY); - let c = cells_of(row); - for (var y = c.y; y <= c.w; y++) { - for (var x = c.x; x <= c.z; x++) { - if word(cell(x, y)) != key { - return; - } - } - } - put(at + CANDIDATE_STATE, word(at + CANDIDATE_STATE) | PLACED_LABEL); -} - -@compute @workgroup_size(256) -fn occupy(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows || (word(candidate(row) + CANDIDATE_STATE) & PLACED_LABEL) == 0u { - return; - } - let c = cells_of(row); - for (var y = c.y; y <= c.w; y++) { - for (var x = c.x; x <= c.z; x++) { - put(cell(x, y), OCCUPIED); - } - } -} - -@compute @workgroup_size(256) -fn ask(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows { - return; - } - let s = word(candidate(row) + CANDIDATE_STATE); - if (s & PLACED_LABEL) == 0u { - return; - } - let code = u32(bitcast(state[extra_at(row, mark.label + LABEL_CODE)])); - if (s & ESTIMATED) != 0u { - want(code); - } else { - seen(code); - } -} - -@compute @workgroup_size(256) -fn fade(@builtin(global_invocation_id) id: vec3u) { - let row = item(id); - if row >= mark.rows { - return; - } - let at = candidate(row); - // A label placed by an estimate waits for its text to fade in. - let s = word(at + CANDIDATE_STATE); - let placed = (s & PLACED_LABEL) != 0u && (s & ESTIMATED) == 0u; - let alpha = bitcast(word(at + CANDIDATE_ALPHA)); - let next = clamp(select(alpha - frame.label_fade, alpha + frame.label_fade, placed), 0.0, 1.0); - put(at + CANDIDATE_ALPHA, bitcast(select(next, 0.0, missing_bits(bitcast(next))))); -} diff --git a/crates/latkit/shaders/compute/lists.wesl b/crates/latkit/shaders/compute/lists.wesl index 92f857f7..332d7b15 100644 --- a/crates/latkit/shaders/compute/lists.wesl +++ b/crates/latkit/shaders/compute/lists.wesl @@ -1,21 +1,24 @@ -// The scan of a list: each block of 256 counts scans in its workgroup and leaves its total, one -// workgroup scans the totals, and every count adds its block's offset. Counts become where each -// item's entries start, the total follows them, and the header's indirect arguments take the -// entries the list keeps. Each count is held to its share of a u32, so the total never wraps. -// Reduce-then-scan needs no forward progress between workgroups, so it is safe on every device. +// The scan of a list. Each block of 256 counts is scanned in its workgroup, which writes the +// block's total. One workgroup then scans the totals, and every count adds its block's offset. The +// counts become each item's first entry, followed by the total, and the header's indirect arguments +// get the number of entries the list keeps. Each count is capped at its share of a u32, so the +// total never wraps. Reduce-then-scan needs no forward-progress guarantee between workgroups, so it +// is safe on every device. -import package::lib::bind::mark; +import package::lib::bind::{mark, program}; import package::lib::types::{ LIST_ITEMS, LIST_COUNTS, LIST_SUMS, LIST_ENTRIES, LIST_DRAW, LIST_DISPATCH, LIST_KEPT, PATH_TRACE, TRIANGLES, - BLOCK_ROWS, item, + DISPATCH_ROWS, RUN_WORDS, FADE_IN, FADE_OUT, DRAWN_ENTERING, DRAWN_LEAVING, item, }; import package::lib::lists::{lists, list_start}; +import package::lib::read::last_at_most; import package::lib::path::{row_segments, path_point, path_length, PATH}; +import package::lib::drawn::held_row; var tile: array; var shared_items: u32; -// The list this dispatch scans: the mark's. +// The list this dispatch scans: the current mark's. fn header() -> u32 { return mark.list; } @@ -35,10 +38,11 @@ fn scan_blocks(@builtin(local_invocation_id) l: vec3u, @builtin(workgroup_id) g: let h = header(); let n = lists[h + LIST_ITEMS]; let counts = lists[h + LIST_COUNTS]; - // Past one dimension's workgroups, blocks of BLOCK_ROWS items step along the second. - let block = g.y * (BLOCK_ROWS / 256u) + g.x; + // Beyond one dimension's workgroups, each step along the second covers `DISPATCH_ROWS` items. + let block = g.y * (DISPATCH_ROWS / 256u) + g.x; let i = block * 256u + l.x; - // Whatever the counts, the starts climb, so every item's writes end. + // Capping each count at its share keeps the starts increasing for any counts, so every item's + // write loop ends. let share = 0xffffffffu / max(n, 1u); let own = select(0u, min(lists[counts + min(i, max(n, 1u) - 1u)], share), i < n); tile[l.x] = own; @@ -92,14 +96,30 @@ fn add_sums(@builtin(global_invocation_id) id: vec3u) { } } -// A mark's segments, numbered by its list: each row counts the segments of its path, then writes -// its row at each of their places, with how far along its path each starts, so a segment's instance -// finds its row, its index in it, and where dashes and comets stand at its start. +// Writes each drawn row's word from the program's run list: its held row, found in the last run +// starting at or before it, and whether that run's tile fades in or out. Every pass that draws or +// probes the mark reads this one word. It runs when the runs change. +@compute @workgroup_size(256) +fn expand_runs(@builtin(global_invocation_id) id: vec3u) { + let i = item(id); + if i >= mark.rows + mark.exits { + return; + } + let run = last_at_most(mark.row_runs + 1u, max(program[mark.row_runs], 1u), RUN_WORDS, 0u, i); + let fade = program[run + 2u]; + let entering = select(0u, DRAWN_ENTERING, (fade & FADE_IN) != 0u); + let leaving = select(0u, DRAWN_LEAVING, (fade & FADE_OUT) != 0u); + lists[mark.drawn + i] = (program[run + 1u] + (i - program[run])) | entering | leaving; +} + +// A mark's segment list. `count_segments` counts each drawn row's segments. After the scan, +// `write_segments` writes each segment's drawn row and its distance along the path, so a segment +// instance finds its row, its index in the row, and where dashes and comets are at its start. @compute @workgroup_size(256) fn count_segments(@builtin(global_invocation_id) id: vec3u) { let i = item(id); if i < lists[mark.list + LIST_ITEMS] { - lists[lists[mark.list + LIST_COUNTS] + i] = row_segments(i); + lists[lists[mark.list + LIST_COUNTS] + i] = row_segments(held_row(i)); } } @@ -112,15 +132,16 @@ fn write_segments(@builtin(global_invocation_id) id: vec3u) { let at = list_start(mark.list, i); let end = list_start(mark.list, i + 1u); let entries = lists[mark.list + LIST_ENTRIES]; - // An area's triangles have no distance along. + // An area's triangles and a trace's segments have no distance along. let measured = mark.primitive != TRIANGLES && PATH != PATH_TRACE; var along = 0.0; - var last = path_point(i, 0u); + let row = held_row(i); + var last = path_point(row, end - at, 0u); for (var k = at; k < end; k++) { lists[entries + 2u * k] = i; lists[entries + 2u * k + 1u] = bitcast(along); if measured { - let next = path_point(i, k - at + 1u); + let next = path_point(row, end - at, k - at + 1u); along += select(0.0, path_length(last, next), last.shown && next.shown); last = next; } diff --git a/crates/latkit/shaders/compute/normals.wesl b/crates/latkit/shaders/compute/normals.wesl index 3754e758..0f40593e 100644 --- a/crates/latkit/shaders/compute/normals.wesl +++ b/crates/latkit/shaders/compute/normals.wesl @@ -1,11 +1,11 @@ -// Vertex normals for faces: each face adds its unit normal to its three corners in fixed point, so the -// sums are exact in any order. +// Vertex normals for faces. Each face adds its unit normal, in fixed point, to the normal words of +// its three corner vertices, in pages of the vertex mark they belong to. Fixed point makes the sums +// exact in any order. import package::lib::types::{NONE, item}; -import package::lib::bind::mark; +import package::lib::bind::{mark, pages}; import package::lib::triangles::triangle_corner; - -@group(2) @binding(3) var normals: array>; +import package::lib::state::end_normal; @compute @workgroup_size(256) fn accumulate(@builtin(global_invocation_id) id: vec3u) { @@ -16,7 +16,7 @@ fn accumulate(@builtin(global_invocation_id) id: vec3u) { let a = triangle_corner(i, 0u); let b = triangle_corner(i, 1u); let c = triangle_corner(i, 2u); - if !(a.shown && b.shown && c.shown) || a.slot == NONE { + if !(a.shown && b.shown && c.shown) || a.vertex == NONE { return; } let n = cross(b.world - a.world, c.world - a.world); @@ -24,11 +24,15 @@ fn accumulate(@builtin(global_invocation_id) id: vec3u) { if !(length2 > 1e-30) { return; } - let q = vec3i(round(n * inverseSqrt(length2) * 65536.0)); + let q = bitcast(vec3i(round(n * inverseSqrt(length2) * 65536.0))); + let vertices = vec3u(a.vertex, b.vertex, c.vertex); for (var k = 0u; k < 3u; k++) { - let at = select(select(c.slot, b.slot, k == 1u), a.slot, k == 0u) * 3u; - atomicAdd(&normals[at], q.x); - atomicAdd(&normals[at + 1u], q.y); - atomicAdd(&normals[at + 2u], q.z); + let at = end_normal(k, vertices[k]); + if at == NONE { + continue; + } + atomicAdd(&pages[at], q.x); + atomicAdd(&pages[at + 1u], q.y); + atomicAdd(&pages[at + 2u], q.z); } } diff --git a/crates/latkit/shaders/compute/probe.wesl b/crates/latkit/shaders/compute/probe.wesl index cd978946..da44a2ed 100644 --- a/crates/latkit/shaders/compute/probe.wesl +++ b/crates/latkit/shaders/compute/probe.wesl @@ -1,31 +1,35 @@ -// Probes and queries: what a frame drew, asked about through the same geometry the draws use. One -// program, made for one primitive's geometry, answers every question a step at a time, the step a -// dispatch's uniform names; each step that reads instances puts each on screen once (`seen`) and asks -// of it what the step needs. +// Probes and queries against a drawn frame, using the same geometry as the draws. One program, +// compiled per primitive, answers every question in steps, and a dispatch's uniform selects the +// step. Each step that reads instances projects each one to the screen once (`seen`) and tests what +// the step needs. // -// The hover takes three steps over every mark it reaches: `nearest` keeps the least key over every -// instance within reach of the probe (the distance in sixty-fourths of a pixel, then the mark drawn -// last), `topmost` keeps the last instance of that mark at that key, and `describe` writes where that -// hit lies. Rows behind what the frame drew are not reached. `under` writes the nearest depth drawn -// around the probe, which gestures turn and zoom about. +// The hover takes three steps over every mark it reaches. `nearest` keeps the least key over all +// instances within the probe's radius: the distance in 1/64 pixel, then the mark drawn last. +// `topmost` keeps the last instance of that mark at that key, and `describe` writes where that hit +// lies. Rows hidden behind other drawn marks are not hit. `under` writes the nearest depth drawn +// around the probe, which gestures rotate and zoom about. // -// Queries write past the probe's words, each mark from the word its `query` names: `hits` appends every -// instance within reach of the query's point while the mark has room; `boxed` sets a bit for every row -// with an instance in the query's box, hidden or not. `locate` writes where one row lies on screen. +// Queries write after the probe's words, each mark from the word its `query` names. `hits` appends +// every instance within the query's radius of its point while the mark has space for hits. `boxed` +// sets a bit for every row with an instance in the query's box, whether or not something drawn +// covers it. `locate` writes where one drawn row lies on screen. Every row an answer names is a +// held row, which the view maps to a table row using its record of that frame. import package::lib::types::{ - NONE, FADE_OUT, POINTS, STROKES, TRIANGLES, MARK_EDGES, PATH_GEOMETRY, PROBE_KEY, PROBE_INSTANCE, PROBE_DEPTH, - PROBE_X, PROBE_Y, PROBE_UNDER, PROBE_ROW, PROBE_SEGMENT, PROBE_LOCATE_X, PROBE_LOCATE_Y, PROBE_LOCATED, - STEP_NEAREST, STEP_TOPMOST, STEP_DESCRIBE, STEP_HITS, STEP_BOXED, STEP_LOCATE, STEP_UNDER, HIT_WORDS, item, + NONE, FADE_OUT, POINTS, STROKES, MARK_EDGES, PROBE_KEY, PROBE_INSTANCE, PROBE_DEPTH, PROBE_X, PROBE_Y, + PROBE_UNDER, PROBE_ROW, PROBE_SEGMENT, PROBE_LOCATE_X, PROBE_LOCATE_Y, PROBE_LOCATED, STEP_NEAREST, STEP_TOPMOST, + STEP_DESCRIBE, STEP_HITS, STEP_BOXED, STEP_LOCATE, STEP_UNDER, HIT_WORDS, item, }; -import package::lib::bind::{frame, eye, mark, state}; +import package::lib::bind::{frame, eye, mark, word}; import package::lib::state::row_at; import package::lib::project::{to_screen, clip_of}; import package::lib::points::{point, port_at}; -import package::lib::path::{PATH, Segment, segment, segment_screen, stroke_style, stroke_instances, path_count}; -import package::lib::triangles::{triangle_corner, triangle_row, triangle_instances, area_anchor}; +import package::lib::path::{segment, segment_screen, stroke_style, stroke_instances, path_count}; +import package::lib::triangles::{triangle_corner, triangle_item, triangle_instances}; import package::lib::marker::outline_distance; import package::lib::mark::PRIMITIVE; +import package::lib::drawn::drawn_row; +import package::lib::center::{row_center, Center}; // Which step a dispatch takes. struct Step { @@ -51,15 +55,18 @@ fn unreached() -> Reach { return Reach(1e30, 0.0, vec2f(0.0), NONE, 0u); } -// An instance as the frame drew it, on screen, depths as z: a point's center and, for a port, the -// port's center as `b`; a segment's ends; a triangle's corners, with whether each is shown, ahead of -// the eye, and in front of it. `part` is a port's number from 1, or a segment's index along its path. +// An instance as the frame drew it, on screen, with depth as z: a point's center (and for a port, +// the port's center as `b`), a segment's ends, or a triangle's corners with whether each is shown, +// ahead of the globe's horizon, and in front of the eye. `item` is its drawn row and `row` that +// row's held row. `part` is a port's number from 1, or a segment's index along its path. struct Seen { + item: u32, row: u32, part: u32, - // The row is the mark's and not fading out. + // The row is a drawn row, not an exit, and neither it nor its tile is fading out. kept: bool, - // A point is shown; a stroke's style is shown and its segment on screen. + // A point is shown. A stroke's style is shown and its segment is on screen. A triangle's + // corners are all shown and in front of the eye, and one is ahead of the horizon. shown: bool, a: vec3f, b: vec3f, @@ -83,19 +90,21 @@ fn nearest_on(p: vec2f, a: vec3f, b: vec3f) -> vec3f { return mix(a, b, t); } -// Whether a row is fading out: it no longer answers the pointer. -fn leaving(row: u32) -> bool { - return (state[row_at(row) + mark.stride - 1u] & FADE_OUT) != 0u; +// Whether drawn row `item` (held row `row`) responds to the pointer: it is not an exit, its tile is +// not `leaving`, and the row is not fading out. +fn answers(item: u32, row: u32, leaving: bool) -> bool { + return item < mark.rows && !leaving && (word(row_at(row) + mark.stride - 1u) & FADE_OUT) == 0u; } -// A point's marker, or past the rows one of their ports. +// A point's marker, or for an instance past the rows, one of their ports. fn seen_point(instance: u32) -> Seen { var s: Seen; let port = select(0u, instance - mark.rows, instance >= mark.rows); - let row = select(instance, port / max(mark.ports, 1u), instance >= mark.rows); - let pt = point(row); - s.row = row; - s.kept = row < mark.rows && !leaving(row); + s.item = select(instance, port / max(mark.ports, 1u), instance >= mark.rows); + let d = drawn_row(s.item); + s.row = d.row; + let pt = point(s.row); + s.kept = answers(s.item, s.row, d.leaving); s.shown = pt.shown; s.a = screen(pt.clip); s.front = vec3(pt.clip.w > 0.0); @@ -110,19 +119,15 @@ fn seen_point(instance: u32) -> Seen { return s; } -// A segment of a stroke: the instance's, or, locating, the middle one of row `instance`'s path. -fn seen_segment(instance: u32, locating: bool) -> Seen { +// A stroke segment. +fn seen_segment(instance: u32) -> Seen { var s: Seen; - var g = Segment(instance, 0u, 0.0); - if locating { - g.index = path_count(instance) / 2u; - } else { - g = segment(instance); - } + let g = segment(instance); + s.item = g.item; s.row = g.row; s.part = g.index; - s.kept = g.row < mark.rows && !leaving(g.row); - if !s.kept && !locating { + s.kept = answers(g.item, g.row, g.leaving); + if !s.kept { return s; } let style = stroke_style(g.row); @@ -136,12 +141,13 @@ fn seen_segment(instance: u32, locating: bool) -> Seen { } // A triangle: a face, or one of an area's. -fn seen_triangle(instance: u32, locating: bool) -> Seen { +fn seen_triangle(instance: u32) -> Seen { var s: Seen; - let row = select(triangle_row(instance), instance, locating); - s.row = row; - s.kept = row < mark.rows && !leaving(row); - if !s.kept && !locating { + s.item = triangle_item(instance); + let d = drawn_row(s.item); + s.row = d.row; + s.kept = answers(s.item, s.row, d.leaving); + if !s.kept { return s; } let ka = triangle_corner(instance, 0u); @@ -160,16 +166,17 @@ fn seen_triangle(instance: u32, locating: bool) -> Seen { return s; } -// Instance `instance` of the mark's primitive on screen; locating, row `instance`'s place. -fn seen(instance: u32, locating: bool) -> Seen { +// Instance `instance` of the mark's primitive on screen. +fn seen(instance: u32) -> Seen { switch PRIMITIVE { case POINTS: { return seen_point(instance); } - case STROKES: { return seen_segment(instance, locating); } - default: { return seen_triangle(instance, locating); } + case STROKES: { return seen_segment(instance); } + default: { return seen_triangle(instance); } } } -// How far `p` lies from what the instance drew, and the nearest of it on screen with its depth. +// How far `p` lies from the instance's drawn shape, and the shape's nearest point on screen, with +// its depth. fn reach(s: Seen, p: vec2f) -> Reach { if !s.kept || !s.shown { return unreached(); @@ -184,10 +191,10 @@ fn reach(s: Seen, p: vec2f) -> Reach { } case STROKES: { let q = nearest_on(p, s.a, s.b); - // The nearer end names the sample a trace's hit lies at; for an edge, which of its ends. + // The nearer end gives the sample a trace's hit is at. For an edge, it gives which end. var end = select(s.part, s.part + 1u, length(q.xy - s.b.xy) < length(q.xy - s.a.xy)); if mark.kind == MARK_EDGES { - end = u32(2u * end >= path_count(s.row)); + end = u32(2u * end >= path_count(s.item)); } return Reach(max(length(p - q.xy) - 0.5 * s.half.x, 0.0), q.z, q.xy, s.row, end); } @@ -231,7 +238,7 @@ fn boxed(s: Seen) -> bool { if !s.shown { return false; } - // The segment crosses the box when its ends do, or it passes through it. + // Tests nine evenly spaced points along the segment, its ends included. for (var k = 0u; k <= 8u; k++) { if in_box(mix(s.a.xy, s.b.xy, f32(k) / 8.0)) { return true; @@ -246,23 +253,6 @@ fn boxed(s: Seen) -> bool { } } -// Where an area's row `row` lies on screen, and 1 as z where it shows: where its label hangs. -fn anchored(row: u32) -> vec3f { - let anchor = area_anchor(row); - let clip = clip_of(anchor.world); - return vec3f(to_screen(clip), f32(anchor.shown && anchor.ahead && clip.w > 0.0)); -} - -// Where the row seen locating lies on screen, and 1 as z where it shows: a point's center, a path's -// middle segment's middle, a face's middle. -fn located(s: Seen) -> vec3f { - switch PRIMITIVE { - case POINTS: { return vec3f(s.a.xy, f32(s.shown && s.front.x)); } - case STROKES: { return vec3f(0.5 * (s.a.xy + s.b.xy), f32(s.front.x)); } - default: { return vec3f((s.a.xy + s.b.xy + s.c.xy) / 3.0, f32(all(s.front))); } - } -} - fn drawn_depth(texel: vec2i) -> f32 { return textureLoad(depth, texel, 0); } @@ -298,8 +288,8 @@ fn instances() -> u32 { return triangle_instances(); } -// Appends hit `r` to the mark's, while it has room: they follow how many it found and how many it -// keeps, `HIT_WORDS` each. +// Appends hit `r` to the mark's hits while there is room. At `mark.query` lie the number found and +// the number kept, then the hits, `HIT_WORDS` each. fn append(r: Reach) { let at = atomicAdd(&probe[mark.query], 1u); if at >= atomicLoad(&probe[mark.query + 1u]) { @@ -314,17 +304,21 @@ fn append(r: Reach) { atomicStore(&probe[entry + 5u], bitcast(r.depth)); } -// Writes where the row located lies. -fn locate(at: vec3f) { - atomicStore(&probe[PROBE_LOCATE_X], bitcast(at.x)); - atomicStore(&probe[PROBE_LOCATE_Y], bitcast(at.y)); - atomicStore(&probe[PROBE_LOCATED], u32(at.z)); +// Writes where drawn row `item` lies on screen, and whether it shows there. +fn locate(item: u32) { + var center = Center(vec2f(0.0), false); + if item < mark.rows { + center = row_center(item); + } + atomicStore(&probe[PROBE_LOCATE_X], bitcast(center.at.x)); + atomicStore(&probe[PROBE_LOCATE_Y], bitcast(center.at.y)); + atomicStore(&probe[PROBE_LOCATED], u32(center.shown)); } @compute @workgroup_size(256) fn answer(@builtin(global_invocation_id) id: vec3u) { let s = step.step; - // Steps that read one thing take one invocation. + // Steps that read a single item use one invocation. if s == STEP_UNDER || s == STEP_LOCATE || s == STEP_DESCRIBE { if any(id != vec3u(0u)) { return; @@ -334,22 +328,15 @@ fn answer(@builtin(global_invocation_id) id: vec3u) { atomicStore(&probe[PROBE_UNDER], bitcast(under())); return; } - // Locating reads the row the query names; describing, the topmost instance the hover found, when - // it is this mark's. Every step that reads an instance sees it through the one call below. - let locating = s == STEP_LOCATE; + // Locating reads the drawn row the query names. + if s == STEP_LOCATE { + locate(bitcast(frame.query.x)); + return; + } + // Describing reads the topmost instance the hover found, if it belongs to this mark. Every step + // that reads an instance goes through the single `seen` call below. var i = item(id); - if locating { - i = bitcast(frame.query.x); - if i >= mark.rows { - locate(vec3f(0.0)); - return; - } - // An area lies where its label hangs. - if PRIMITIVE == TRIANGLES && PATH == PATH_GEOMETRY { - locate(anchored(i)); - return; - } - } else if s == STEP_DESCRIBE { + if s == STEP_DESCRIBE { let key = atomicLoad(&probe[PROBE_KEY]); let instance = atomicLoad(&probe[PROBE_INSTANCE]); if key == NONE || instance == 0u || 255u - (key & 255u) != min(mark.index, 255u) { @@ -359,11 +346,7 @@ fn answer(@builtin(global_invocation_id) id: vec3u) { } else if i >= instances() { return; } - let drawn = seen(i, locating); - if locating { - locate(located(drawn)); - return; - } + let drawn = seen(i); if s == STEP_BOXED { if boxed(drawn) { atomicOr(&probe[mark.query + drawn.row / 32u], 1u << (drawn.row % 32u)); diff --git a/crates/latkit/shaders/compute/texts.wesl b/crates/latkit/shaders/compute/texts.wesl new file mode 100644 index 00000000..b88e6511 --- /dev/null +++ b/crates/latkit/shaders/compute/texts.wesl @@ -0,0 +1,128 @@ +// The labels that show, and the texts they need. `claim` writes each labeled row's key into the +// grid cells its label covers. Each cell keeps the largest key. `texts` then asks the host to set +// the text of each label that is not crowded out, when the atlas lacks the text and the label's +// anchor is near the view (and, in a diagram, its text is large enough to read). It also marks a +// set text in view as seen, so the atlas keeps it. Rows whose tile is leaving claim nothing. +// `draw/glyphs.wesl` decides from the same claims whether each label shows. Each pipeline is made +// for one primitive and compiles only that primitive's anchors. + +import package::lib::types::{NONE, VISIBLE, LABEL_CODE, WANTED, item}; +import package::lib::bind::{eye, mark, word}; +import package::lib::state::{row_at, extra_at}; +import package::lib::read::missing_bits; +import package::lib::label::{ + Anchor, anchor, length_px, legibility, in_view, claims_room, claim_key, claimed, cells_of, cell, Cells, +}; +import package::lib::drawn::drawn_row; + +// Each labeled mark's words: its run table, read here, and the codes it asks for and has seen, +// written atomically by every row. +@group(2) @binding(13) var labels: array>; +// The claim grid, cleared before `claim` runs. +@group(2) @binding(14) var cells: array>; + +// A labeled row as the texts passes see it: its text code (NONE without a label to show), its +// anchor, and its run table entry. The entry's words are both 0 until the text is set, and NONE +// when it draws nothing. +struct Labeled { + code: u32, + a: Anchor, + uv0: u32, + uv1: u32, +} + +fn labeled(i: u32) -> Labeled { + var l = Labeled(NONE, anchor(i), 0u, 0u); + let drawn = drawn_row(i); + let code_bits = word(extra_at(drawn.row, mark.label + LABEL_CODE)); + let flags = word(row_at(drawn.row) + mark.stride - 1u); + if missing_bits(code_bits) || (flags & VISIBLE) == 0u || drawn.leaving || !l.a.shown { + return l; + } + let code = u32(bitcast(code_bits)); + l.uv0 = atomicLoad(&labels[mark.runs + 2u * code]); + l.uv1 = atomicLoad(&labels[mark.runs + 2u * code + 1u]); + // A text that draws nothing, such as an empty one, gives no label. + if l.uv0 != NONE { + l.code = code; + } + return l; +} + +// Whether claim `key` kept every cell of `c`. +fn keeps(c: Cells, key: u32) -> bool { + for (var y = c.first.y; y <= c.last.y; y++) { + for (var x = c.first.x; x <= c.last.x; x++) { + if atomicLoad(&cells[cell(c, x, y)]) != key { + return false; + } + } + } + return true; +} + +// Asks the host to set the text of `code`, once however many rows show it. A text is either set or +// not for the whole pass, so the seen bit of a text not set marks it asked for. +fn want(code: u32) { + let bit = 1u << (code % 32u); + if (atomicOr(&labels[mark.seen + code / 32u], bit) & bit) != 0u { + return; + } + let asked = atomicAdd(&labels[mark.wanted], 1u); + if asked < WANTED { + atomicStore(&labels[mark.wanted + 1u + asked], code); + } +} + +// Marks the text of `code` as shown, so the atlas keeps it. +fn seen(code: u32) { + atomicOr(&labels[mark.seen + code / 32u], 1u << (code % 32u)); +} + +@compute @workgroup_size(256) +fn claim(@builtin(global_invocation_id) id: vec3u) { + let i = item(id); + if i >= mark.rows { + return; + } + let l = labeled(i); + if l.code == NONE || !claims_room(l.a) { + return; + } + let c = cells_of(claimed(l.a, l.uv0, l.uv1, length_px(mark.label_size))); + let key = claim_key(i); + for (var y = c.first.y; y <= c.last.y; y++) { + for (var x = c.first.x; x <= c.last.x; x++) { + atomicMax(&cells[cell(c, x, y)], key); + } + } +} + +@compute @workgroup_size(256) +fn texts(@builtin(global_invocation_id) id: vec3u) { + let i = item(id); + if i >= mark.rows { + return; + } + let l = labeled(i); + if l.code == NONE { + return; + } + let size = length_px(mark.label_size); + let t = claimed(l.a, l.uv0, l.uv1, size); + if claims_room(l.a) && !keeps(cells_of(t), claim_key(i)) { + return; + } + if l.uv0 == 0u && l.uv1 == 0u { + // The atlas lacks this text. Ask for it if the label could show soon: legible and near the + // view. + let near = 0.25 * eye.size; + if legibility(size) > 0.0 && in_view(l.a.at - near, l.a.at + near) { + want(l.code); + } + return; + } + if in_view(t.lo, t.hi) { + seen(l.code); + } +} diff --git a/crates/latkit/shaders/draw/areas.wesl b/crates/latkit/shaders/draw/areas.wesl index 5975ecd4..9f56d917 100644 --- a/crates/latkit/shaders/draw/areas.wesl +++ b/crates/latkit/shaders/draw/areas.wesl @@ -1,21 +1,21 @@ -// Areas: one instanced triangle per triangle of every area, numbered by the mark's list, flat on the -// ground in its row's color and lit by the earth's daylight. Areas lie under what draws after them: -// they test depth but write none. The focus variant draws the triangles of the rows the mark's focus -// lights again, over everything, tinted toward the theme's focus colors. +// Areas: one instanced triangle per area triangle, numbered by the mark's list, flat on the ground +// in its row's color and lit by the earth's daylight. Areas lie under anything drawn after them: +// they test depth but do not write it. The focus variant redraws the lit rows' triangles over +// everything, tinted toward the theme's focus colors. import package::lib::types::{SELECTED, HOVERED}; import package::lib::bind::frame; import package::lib::project::culled; import package::lib::triangles::{triangle_corner, triangle_shown, corner_clip}; import package::lib::focus::focus_of; -import package::lib::shade::daylight; +import package::lib::daylight::daylight; import package::lib::color::output; struct Varyings { @builtin(position) clip: vec4f, @location(0) color: vec4f, @location(1) @interpolate(flat) row: u32, - // The row's focus bits, which the focus draw tints by. + // The row's focus bits, which choose the focus draw's tint. @location(2) @interpolate(flat) focus: u32, } @@ -46,7 +46,7 @@ fn fragment(v: Varyings) -> @location(0) vec4f { var rgb = v.color.rgb; let focus = v.focus; if (focus & SELECTED) != 0u { - // A selected color with a negative red asks for each row's own, brightened. + // A selected color with a negative red means each row's own color, brightened. let tint = select(frame.selected.rgb, min(rgb * 1.6 + 0.08, vec3f(1.0)), frame.selected.r < 0.0); rgb = mix(rgb, tint, 0.5); } else if (focus & HOVERED) != 0u { diff --git a/crates/latkit/shaders/draw/backdrop.wesl b/crates/latkit/shaders/draw/backdrop.wesl index baf54787..fe58973e 100644 --- a/crates/latkit/shaders/draw/backdrop.wesl +++ b/crates/latkit/shaders/draw/backdrop.wesl @@ -1,28 +1,26 @@ -// The backdrop under every mark: a plane's or a diagram's grid, or the earth's surface and graticule, -// darkened where casters shadow the ground, lit by daylight, and on the globe ringed by its air. Lines -// are a pixel wide and antialiased per axis. Their spacing steps by tens, and the next step fades in as -// the lines crowd, so no seam shows. They fade where the view grazes the ground, and on a plane away -// from what the camera looks at. The round globe writes its depth a hair under its surface, so what -// lies behind it hides and what lies on it draws. +// The backdrop under every mark: a plane's or diagram's grid, or the earth's ground and graticule, +// lit by daylight, with the atmosphere around the globe. Lines are one pixel wide and antialiased +// per axis. Their spacing steps by powers of ten, and the next coarser spacing fades in as lines +// get dense, so no seam shows. Lines fade where the view grazes the ground, and on a plane, with +// distance from where the camera looks. The round globe writes its depth slightly below the ground, +// so marks behind it are hidden and marks on it still draw. import package::lib::types::{SPACE_EARTH, SPACE_PLANE, SPACE_DIAGRAM, GRID_OFF, GRID_DOTS}; import package::lib::bind::{frame, eye}; import package::lib::color::output; import package::lib::project::{PI, NEAR_Z, FAR_Z, globe_center}; -import package::lib::shade::{lit, daylight}; - -@group(2) @binding(10) var shadows: texture_2d; +import package::lib::daylight::daylight; struct Varyings { @builtin(position) clip: vec4f, @location(0) ndc: vec2f, } -// How far under its surface the globe's depth lies, in its radii: deeper than a chord of a few -// degrees sags, so paths and areas on the ground draw over it. -const SURFACE_BIAS: f32 = 1e-3; +// How far below the ground the globe's depth is written, in globe radii. It exceeds the sag of a +// chord a few degrees long, so paths and areas on the ground draw over the globe. +const GROUND_BIAS: f32 = 1e-3; -// What a pixel of the backdrop draws: its color, and the depth of the ground there. +// A backdrop pixel: its color, and the ground's depth there. struct Ground { color: vec4f, depth: f32, @@ -40,8 +38,8 @@ fn vertex(@builtin(vertex_index) i: u32) -> Varyings { return Varyings(vec4f(ndc, 0.0, 1.0), ndc); } -// Coverage of lines `pitch` apart through `p`, which moves `w` per pixel: a pixel wide, fading out as -// their cells shrink to a few pixels. +// Coverage of grid lines `pitch` apart at `p`, where `p` changes by `w` per pixel. Lines are a +// pixel wide and fade out as their cells shrink to a few pixels. fn lines(p: vec2f, w: vec2f, pitch: f32) -> f32 { let cells = max(w / pitch, vec2f(1e-12)); let px = abs(fract(p / pitch + 0.5) - 0.5) / cells; @@ -49,16 +47,17 @@ fn lines(p: vec2f, w: vec2f, pitch: f32) -> f32 { return max(c.x, c.y); } -// Dots `pitch` apart through `p`, which moves `w` per pixel: a CSS pixel across on any screen, fading -// out as their cells shrink to a few pixels. +// Coverage of grid dots `pitch` apart at `p`, where `p` changes by `w` per pixel. Dots are a CSS +// pixel across on any screen and fade out as their cells shrink to a few pixels. fn dots(p: vec2f, w: vec2f, pitch: f32) -> f32 { let cells = max(w / pitch, vec2f(1e-12)); let px = abs(fract(p / pitch + 0.5) - 0.5) / cells; return saturate(eye.pixel_ratio + 0.5 - length(px)) * (1.0 - smoothstep(0.08, 0.25, max(cells.x, cells.y))); } -// Lines spaced `pitch` times a power of ten, from the `lowest` power to the `highest`: the spacing -// nearest twenty pixels draws, and the next coarser one fades in as it crowds. +// Grid lines spaced `pitch` times a power of ten, with the power between `lowest` and `highest`. +// Lines draw at the spacing just under 20 pixels, and the next coarser spacing fades in as that one +// gets denser. fn grid(p: vec2f, w: vec2f, pitch: f32, lowest: f32, highest: f32, dotted: bool) -> f32 { let level = clamp(log2(max(max(w.x, w.y) * 20.0 / pitch, 1e-30)) * 0.30103, lowest, highest); let k = floor(level); @@ -74,18 +73,12 @@ fn blend(color: vec4f, over: vec4f, coverage: f32) -> vec4f { return vec4f(mix(color.rgb, over.rgb, coverage * over.a), color.a); } -// `color` darkened where casters shadow ground point `g`. -fn shaded(color: vec4f, g: vec3f, ahead: bool) -> vec4f { - let shadow = select(0.0, 1.0 - lit(shadows, g, true), ahead); - return vec4f(mix(color.rgb, frame.shade.color.rgb, shadow), color.a); -} - -// The plane: its grid where the ray meets elevation 0 at `g`, faded where the ray grazes and away -// from the center. +// The plane's grid at `g`, where the view ray meets elevation 0, faded where the ray grazes the +// plane and away from the center. fn plane(g: vec3f, d: vec3f, ahead: bool) -> vec4f { let p = g.xy + frame.center_hi + frame.center_lo; let w = fwidth(g.xy); - let ground = shaded(frame.background, g, ahead); + let ground = frame.background; if frame.grid_kind == GRID_OFF { return ground; } @@ -110,12 +103,18 @@ fn globe_degrees(w: vec3f, radius: f32) -> vec2f { return vec2f(lon, degrees(asin(clamp(v.y, -1.0, 1.0)))); } -// The earth: the mercator band or the globe's disc, eased between as the projection morphs, lit by -// daylight, the graticule on it, and around the globe its air. +// The earth: the flat map's square or the globe's disc, blended as the projection morphs, lit by +// daylight, with the graticule on it and the atmosphere around the globe. fn earth(o: vec3f, d: vec3f, g: vec3f, ahead: bool) -> Ground { - // The flat map ends where web mercator does, at y = ±π. + // The flat map is one world. It ends where Web Mercator does, at y = ±π, and at the + // antimeridian, x = ±π from the prime meridian. The distance to the antimeridian starts from + // the center's longitude part by part, so the edge holds still at any zoom. let band = PI - abs(g.y + frame.mercator_y); - let flat = saturate(band / max(fwidth(band), 1e-9) + 0.5) * f32(ahead); + let east = radians(180.0 - frame.center_hi.x - frame.center_lo.x) - g.x; + let west = radians(180.0 + frame.center_hi.x + frame.center_lo.x) + g.x; + let side = min(east, west); + let across = saturate(side / max(fwidth(side), 1e-9) + 0.5); + let flat = saturate(band / max(fwidth(band), 1e-9) + 0.5) * across * f32(ahead); let center = globe_center(); let radius = -center.z; let to_center = center - o; @@ -127,26 +126,26 @@ fn earth(o: vec3f, d: vec3f, g: vec3f, ahead: bool) -> Ground { let hit = o + d * (along - sqrt(max(radius * radius - off * off, 0.0))); let globe = frame.globe >= 0.5; var p: vec2f; - var ground = frame.surface; + var ground = frame.ground; if globe { p = globe_degrees(hit, radius); - ground = vec4f(ground.rgb * daylight(p), ground.a); } else { p = map_degrees(g); - ground = shaded(vec4f(ground.rgb * daylight(p), ground.a), g, ahead); } + ground = vec4f(ground.rgb * daylight(p), ground.a); var color = blend(frame.background, ground, covered); let facing = dot(normalize(hit - center), -d); if frame.graticule != 0u { - // Longitude jumps where it wraps; its other wrapping steadies the derivative there. + // Longitude jumps where it wraps. A second, shifted wrapping keeps the derivative steady + // there. let wrapped = fract(p.x / 360.0) * 360.0; let w = vec2f(min(fwidth(p.x), fwidth(wrapped)), fwidth(p.y)); let fade = select(smoothstep(0.02, 0.15, abs(d.z)), smoothstep(0.0, 0.3, facing), globe); color = blend(color, frame.grid_color, grid(p, w, 1.0, -2.0, 1.0, false) * fade * covered); } if frame.atmosphere.a > 0.0 && frame.globe > 0.0 { - // A thin rim of air over the globe's edge and a halo just past it, as thick as the air is - // beside the earth, lit where the sun is up over the ground the ray passes nearest. + // A thin rim of atmosphere over the globe's edge, and a halo out to 1.045 globe radii. It + // is lit where the sun is up over the ground nearest the view ray. let rim = pow(1.0 - saturate(facing), 6.0) * round; let halo = pow(1.0 - smoothstep(radius, radius * 1.045, off), 2.0) * (1.0 - round) * f32(along > 0.0); let below = daylight(globe_degrees(o + d * along, radius)); @@ -155,7 +154,7 @@ fn earth(o: vec3f, d: vec3f, g: vec3f, ahead: bool) -> Ground { } var depth = FAR_Z; if frame.globe >= 1.0 && round > 0.5 { - let under = eye.clip_from_world * vec4f(hit - normalize(hit - center) * radius * SURFACE_BIAS, 1.0); + let under = eye.clip_from_world * vec4f(hit - normalize(hit - center) * radius * GROUND_BIAS, 1.0); depth = clamp(under.z / under.w, FAR_Z, NEAR_Z); } return Ground(color, depth); diff --git a/crates/latkit/shaders/draw/glyphs.wesl b/crates/latkit/shaders/draw/glyphs.wesl index 704748e2..6429801d 100644 --- a/crates/latkit/shaders/draw/glyphs.wesl +++ b/crates/latkit/shaders/draw/glyphs.wesl @@ -1,23 +1,42 @@ -// Glyphs: every labeled row's text, placed from its anchor each frame and shown as placement decided, -// over a halo of the background so it reads over whatever lies under it. The ports variant draws -// each port's label beside it, inside its marker. Text is a distance field: edges stay crisp at any -// size, and the halo is a wider threshold of the same field. +// Glyphs: every labeled row's text, placed from its anchor each frame. A label shows when its text +// is set, its row shows, and it fits its anchor's `room`. It fades with its row's tile and, in a +// diagram, once too small to read. A label beside its row also shows only if it kept its claim on +// every cell it covers this frame (`compute/texts.wesl`), so labels never overlap. Nothing depends +// on an earlier frame. A halo of the background color keeps text readable over anything below it. +// The ports variant draws each port's label beside it, inside its marker. Text is a distance field, +// so edges stay crisp at any size, and the halo is a wider threshold of the same field. import package::lib::types::{ - CANDIDATE_WORDS, CANDIDATE_ALPHA, LABEL_CODE, LABEL_COLOR, PORT_WORDS, PORT_CODE, PORT_UV, SIDE_LEFT, - SIDE_RIGHT, SIDE_TOP, TEXT_SPREAD, TEXT_ESTIMATE, SPACE_DIAGRAM, THEME_TEXT, + NONE, LABEL_CODE, LABEL_COLOR, PORT_WORDS, PORT_CODE, PORT_UV, SIDE_LEFT, SIDE_RIGHT, SIDE_TOP, TEXT_SPREAD, + SPACE_DIAGRAM, THEME_TEXT, }; -import package::lib::bind::{frame, eye, mark, state, program, ramp}; +import package::lib::bind::{frame, eye, mark, word, program, ramp}; import package::lib::state::extra_at; import package::lib::read::missing_bits; -import package::lib::points::{point, port_at, port_inward}; +import package::lib::points::{shown_point, port_at, port_inward}; import package::lib::marker::theme_port_size; -import package::lib::label::{anchor, length_px, legibility, text_at, fits, Text}; -import package::lib::project::culled; +import package::lib::label::{ + anchor, length_px, legibility, text_at, fits, Text, claims_room, claim_key, cells_of, cell, Cells, +}; +import package::lib::project::{culled, to_clip}; import package::lib::color::output; +import package::lib::drawn::drawn_row; @group(2) @binding(8) var atlas: texture_2d; @group(2) @binding(13) var labels: array; +@group(2) @binding(14) var cells: array; + +// Whether claim `key` kept every cell of `c`. +fn keeps(c: Cells, key: u32) -> bool { + for (var y = c.first.y; y <= c.last.y; y++) { + for (var x = c.first.x; x <= c.last.x; x++) { + if cells[cell(c, x, y)] != key { + return false; + } + } + } + return true; +} struct Varyings { @builtin(position) clip: vec4f, @@ -31,13 +50,12 @@ struct Varyings { fn quad(t: Text, corner: u32, color: vec4f) -> Varyings { var out: Varyings; let q = vec2f(f32(corner & 1u), f32(corner >> 1u)); - // Snapped to whole target pixels while it is set at its own size, so text stays crisp. + // Outside a diagram, text has a fixed size. Snapping to whole target pixels keeps it crisp. var p = mix(t.quad_lo, t.quad_hi, q); if frame.space != SPACE_DIAGRAM { p = round(p * eye.pixel_ratio) / eye.pixel_ratio; } - let ndc = p / eye.size * 2.0 - 1.0; - out.clip = vec4f(ndc.x, -ndc.y, 0.0, 1.0); + out.clip = to_clip(p, vec4f(0.0, 0.0, 0.0, 1.0)); out.uv = mix(t.uv_lo, t.uv_hi, q); out.color = color; out.scale = t.scale; @@ -52,45 +70,46 @@ fn unshown() -> Varyings { @if(!ports) @vertex -fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) row: u32) -> Varyings { - if row >= mark.rows { +fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: u32) -> Varyings { + if instance >= mark.rows { return unshown(); } - let code_bits = state[extra_at(row, mark.label + LABEL_CODE)]; + let drawn = drawn_row(instance); + let row = drawn.row; + let code_bits = word(extra_at(row, mark.label + LABEL_CODE)); if missing_bits(code_bits) { return unshown(); } let code = u32(bitcast(code_bits)); let uv0 = labels[mark.runs + 2u * code]; let uv1 = labels[mark.runs + 2u * code + 1u]; - // Nothing to draw until the text is set. - if (uv0 == 0u && uv1 == 0u) || (uv0 & TEXT_ESTIMATE) != 0u { + // Nothing to draw until the text is set, or when it draws nothing. + if (uv0 == 0u && uv1 == 0u) || uv0 == NONE { return unshown(); } - let a = anchor(row); + let a = anchor(instance); let size = length_px(mark.label_size); let t = text_at(uv0, uv1, size, a.at, a.align); - // A diagram's labels show while they fit and read; elsewhere as placement decided. - var alpha = bitcast(labels[mark.candidates + row * CANDIDATE_WORDS + CANDIDATE_ALPHA]); - if frame.space == SPACE_DIAGRAM { - alpha = f32(a.shown && fits(t.hi - t.lo, a.room)) * legibility(size); - } + // A label shows while its row shows, it fits, and it kept its claimed cells. Its opacity + // follows its legibility at this size and its row's tile fade. + let room = fits(t.hi - t.lo, a.room) && (!claims_room(a) || keeps(cells_of(t), claim_key(instance))); + let alpha = f32(a.shown && room) * legibility(size) * drawn.presence; if !(alpha > 0.0) { return unshown(); } - let color = unpack4x8unorm(state[extra_at(row, mark.label + LABEL_COLOR)]); + let color = unpack4x8unorm(word(extra_at(row, mark.label + LABEL_COLOR))); return quad(t, corner, vec4f(color.rgb, color.a * alpha)); } -// A port's label: beside the port, inside its marker, aligned away from the side it sits on. It -// shows while it reads and keeps clear of the ports across from it. +// A port's label: beside the port, inside its marker, aligned away from the port's side. It shows +// while legible and while it fits its side's share of the marker, clear of the opposite ports. @if(ports) @vertex fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: u32) -> Varyings { let n = max(mark.ports, 1u); - let row = instance / n; + let item = instance / n; let k = instance % n; - if row >= mark.rows { + if item >= mark.rows { return unshown(); } let info = mark.port_info + k * PORT_WORDS; @@ -99,7 +118,7 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: if uv0 == 0u && uv1 == 0u { return unshown(); } - let p = point(row); + let p = shown_point(item); let port = port_at(p, k); let code = program[info + PORT_CODE]; let inward = port_inward(code); @@ -135,7 +154,7 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: return quad(t, corner, vec4f(text.rgb, text.a * alpha)); } -// The distance field at `uv`: texels past the text's edge, inside positive. +// The distance field at `uv`, in texels from the text's edge, positive inside. fn inside(uv: vec2f) -> f32 { let d = textureSampleLevel(atlas, ramp, uv / vec2f(textureDimensions(atlas)), 0.0).r; return (d - 0.5) * 2.0 * TEXT_SPREAD; @@ -143,7 +162,7 @@ fn inside(uv: vec2f) -> f32 { @fragment fn fragment(v: Varyings) -> @location(0) vec4f { - // Target pixels past the edge. + // Distance from the edge in target pixels. let d = inside(v.uv) * v.scale * eye.pixel_ratio; let a_text = clamp(0.5 + d, 0.0, 1.0) * v.color.a; let back = frame.background; diff --git a/crates/latkit/shaders/draw/overlay.wesl b/crates/latkit/shaders/draw/overlay.wesl index c7473e2f..0142f112 100644 --- a/crates/latkit/shaders/draw/overlay.wesl +++ b/crates/latkit/shaders/draw/overlay.wesl @@ -1,10 +1,11 @@ -// The overlay: lines, boxes, filled rectangles, and runs of text in CSS pixels over every mark, such -// as a plot's axes, a scene's title and legends, a selection box, and a connection being drawn. Text -// is the atlas's distance field, its quad reaching over the field's spread. +// The overlay: lines, boxes, filled rectangles, and runs of text in CSS pixels, drawn over every +// mark. It draws a plot's axes, a scene's title and legends, a selection box, and a connection +// being drawn. Text samples the atlas's distance field, and its quad covers the field's spread. import package::lib::types::{Sprite, SPRITE_TEXT, SPRITE_BOX, SPRITE_FILL, TEXT_SPREAD}; import package::lib::bind::{eye, ramp}; import package::lib::mark::coverage; +import package::lib::project::to_clip; import package::lib::color::output; @group(2) @binding(7) var sprites: array; @@ -31,8 +32,7 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) sprite: u let along = mix(s.a - dir * reach, s.b + dir * reach, quad.x); p = along + vec2f(-dir.y, dir.x) * (quad.y * 2.0 - 1.0) * reach; } - let ndc = p / eye.size * 2.0 - 1.0; - return Varyings(vec4f(ndc.x, -ndc.y, 0.0, 1.0), p, sprite); + return Varyings(to_clip(p, vec4f(0.0, 0.0, 0.0, 1.0)), p, sprite); } @fragment diff --git a/crates/latkit/shaders/draw/points.wesl b/crates/latkit/shaders/draw/points.wesl index 760966c2..606ac96a 100644 --- a/crates/latkit/shaders/draw/points.wesl +++ b/crates/latkit/shaders/draw/points.wesl @@ -1,21 +1,19 @@ -// Points: one instanced quad per vertex row, shaded by its marker; a group's rows draw as -// containers. A quad covers its marker and a pixel of edge, no more, and a small one a later one -// covers in a crowded cell is left out. The focus variant draws the rows the mark's focus lights -// again, over everything, reaching out to their glow; a group only glows about its border, so what -// it holds still shows. The cast variant draws the same quads through the light's eye into the -// shadow map; the ports variant draws one quad per port of every row, on its side of the row's -// marker. - -import package::lib::types::{STATUS, MARKER_ICON, MARKER_SHAPE, MARKER_BLOCK, GROUP, CASTS, SPACE_DIAGRAM}; -import package::lib::bind::{frame, eye, mark, state, ramp, program, focused}; +// Points: one instanced quad per vertex row, shaded by its marker. A group's rows draw as +// containers. A quad covers its marker and any status ring or shadow, plus one pixel of antialiased +// edge. The focus variant redraws the lit rows over everything, with quads enlarged for their glow. +// A group glows only around its border, so its contents stay visible. The ports variant draws one +// quad per port of every row, on its side of the row's marker. + +import package::lib::types::{STATUS, MARKER_ICON, MARKER_SHAPE, MARKER_BLOCK, GROUP, SHADOW, SPACE_DIAGRAM}; +import package::lib::bind::{frame, eye, mark, word, ramp, program, focused}; import package::lib::state::extra_at; import package::lib::project::{to_screen, to_clip, nearer, culled}; -import package::lib::points::{Point, point, reach, port_at}; +import package::lib::points::{Point, shown_point, reach, port_at}; import package::lib::marker::{ MarkerColor, MarkerFragment, builtin_marker, container, border, dropped, outline_distance, box_distance, shape_distance, filled, over, status_ring, glow, premultiplied, theme_text, theme_corner, theme_lift, }; -import package::lib::shade::{caster, daylight}; +import package::lib::daylight::daylight; import package::lib::mark::fringe; import package::lib::color::output; @@ -28,15 +26,14 @@ struct Varyings { @location(2) color: vec4f, @location(3) @interpolate(flat) half: vec2f, @location(4) @interpolate(flat) status: vec4f, - @location(5) @interpolate(flat) height: f32, - @location(6) @interpolate(flat) shape: u32, - // A group's depth; for a port, 1 when its reference names a net. - @location(7) @interpolate(flat) extra: f32, - // The row's focus bits, which the focus draw glows by. - @location(8) @interpolate(flat) focus: u32, + @location(5) @interpolate(flat) shape: u32, + // A group's depth. For a port, 1 when its reference names a net. + @location(6) @interpolate(flat) extra: f32, + // The row's focus bits, which choose the focus draw's glow. + @location(7) @interpolate(flat) focus: u32, } -// What an instance draws: a row's marker, or with `ports` one of a row's ports. +// One drawn instance: a row's marker, or in the ports variant, one of a row's ports. struct Drawn { p: Point, center: vec2f, @@ -48,20 +45,20 @@ struct Drawn { @if(!ports && !focused) fn drawn(instance: u32) -> Drawn { - let p = point(instance); + let p = shown_point(instance); return Drawn(p, to_screen(p.clip), p.half, reach(p, false), mark.shape, p.depth); } @if(focused) fn drawn(instance: u32) -> Drawn { - let p = point(focused[mark.focused + instance]); + let p = shown_point(focused[mark.focused + instance]); return Drawn(p, to_screen(p.clip), p.half, reach(p, true), mark.shape, p.depth); } @if(ports) fn drawn(instance: u32) -> Drawn { let n = max(mark.ports, 1u); - let p = point(instance / n); + let p = shown_point(instance / n); let port = port_at(p, instance % n); let half = vec2f(port.half); return Drawn(p, port.center, half, half + 1.5 * frame.unit + fringe(), port.shape, f32(port.wired)); @@ -78,7 +75,6 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: out.color = vec4f(d.p.color.rgb * daylight(d.p.position), d.p.color.a); out.half = max(d.half, vec2f(0.5)); out.status = d.p.status; - out.height = d.p.world.z; out.shape = d.shape; out.extra = d.extra; out.focus = d.p.focus; @@ -91,13 +87,13 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: fn inputs(row: u32) -> array { var input = array(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0); for (var i = 0u; i < min(mark.inputs, 8u); i++) { - input[i] = bitcast(state[extra_at(row, mark.input + i)]); + input[i] = bitcast(word(extra_at(row, mark.input + i))); } return input; } -// An icon: image `index` of the mark's images, fitted in the marker's box, faded with its row. The -// palette holds how many images there are, then each one's layer of the icon atlas. +// An icon: image `f.input[0]` of the mark's images, fitted to the marker's box and faded with its +// row. The palette holds the image count, then each image's layer in the icon atlas. fn icon(f: MarkerFragment) -> MarkerColor { let uv = vec2f(f.p.x, -f.p.y) / (2.0 * f.half) + 0.5; let count = max(program[mark.palette], 1u); @@ -116,9 +112,10 @@ fn marker(f: MarkerFragment) -> MarkerColor { return builtin_marker(f); } -// A port: its shape in its row's color mixed toward the theme's text, so it reads on the marker in -// either theme, ringed by the background so it reads apart from the marker; hollow while no net plugs -// into it. Its ring and outline are part of the drawing, as the port is. +// A port: its shape in the row's color mixed toward the theme's text color, so it stays readable on +// the marker in light and dark themes. A ring of the background color sets it apart from the +// marker. It draws hollow while no net connects to it. Its ring and outline are sized in length +// units, like the port itself. @if(ports) fn shaded(v: Varyings) -> MarkerColor { let u = frame.unit; @@ -133,8 +130,8 @@ fn shaded(v: Varyings) -> MarkerColor { return over(hollow, ring); } -// Whether the mark draws a diagram's blocks: shapes and blocks on a diagram's sheet, which the theme -// borders, and shadows when they cast them. +// Whether the mark draws a diagram's blocks: shape or block markers on a diagram's sheet, which get +// the theme's border. fn sheet_blocks() -> bool { return frame.space == SPACE_DIAGRAM && (mark.marker == MARKER_SHAPE || mark.marker == MARKER_BLOCK); } @@ -152,8 +149,8 @@ fn shaded(v: Varyings) -> MarkerColor { if (mark.flags & STATUS) != 0u && v.status.a > 0.0 { m = status_ring(m, vec4f(v.status.rgb, v.status.a * v.color.a)); } - if (mark.flags & CASTS) != 0u && frame.space == SPACE_DIAGRAM { - // y is up in a marker: the shadow falls below, measured from its outline lowered. + if (mark.flags & SHADOW) != 0u { + // y is up in a marker. The shadow falls below, measured from the outline shifted down. let below = v.p + vec2f(0.0, theme_lift()); let distance = select( outline_distance(below, v.half), @@ -165,7 +162,7 @@ fn shaded(v: Varyings) -> MarkerColor { return m; } -@if(!casts && !focused) +@if(!focused) @fragment fn fragment(v: Varyings) -> @location(0) vec4f { return output(premultiplied(shaded(v))); @@ -176,18 +173,9 @@ fn fragment(v: Varyings) -> @location(0) vec4f { fn fragment(v: Varyings) -> @location(0) vec4f { let m = shaded(v); if (mark.flags & GROUP) != 0u { - // Drawn over what it holds, a group's body would hide it: it glows on both sides of its border. + // A group is drawn over its contents, so its body would hide them. It glows on both sides + // of its border instead. return output(glow(MarkerColor(vec4f(0.0), abs(m.distance)), v.focus)); } return output(glow(m, v.focus)); } - -@if(casts) -@fragment -fn fragment(v: Varyings) -> @location(0) vec4f { - let m = shaded(v); - if m.color.a <= 0.01 { - discard; - } - return caster(m.color.a, v.height); -} diff --git a/crates/latkit/shaders/draw/shade.wesl b/crates/latkit/shaders/draw/shade.wesl deleted file mode 100644 index a86b41b2..00000000 --- a/crates/latkit/shaders/draw/shade.wesl +++ /dev/null @@ -1,21 +0,0 @@ -// One level of the shadow map from the level below: coverage averaged, so it softens; the lowest -// caster kept, and the highest. - -@group(2) @binding(11) var below: texture_2d; - -@vertex -fn vertex(@builtin(vertex_index) i: u32) -> @builtin(position) vec4f { - let uv = vec2f(f32((i << 1u) & 2u), f32(i & 2u)); - return vec4f(uv * 2.0 - 1.0, 0.0, 1.0); -} - -@fragment -fn fragment(@builtin(position) p: vec4f) -> @location(0) vec4f { - let size = vec2i(textureDimensions(below)) - 1; - let at = vec2i(p.xy) * 2; - let a = textureLoad(below, min(at, size), 0); - let b = textureLoad(below, min(at + vec2i(1, 0), size), 0); - let c = textureLoad(below, min(at + vec2i(0, 1), size), 0); - let d = textureLoad(below, min(at + vec2i(1, 1), size), 0); - return vec4f((a.r + b.r + c.r + d.r) * 0.25, min(min(a.g, b.g), min(c.g, d.g)), min(min(a.b, b.b), min(c.b, d.b)), 0.0); -} diff --git a/crates/latkit/shaders/draw/strokes.wesl b/crates/latkit/shaders/draw/strokes.wesl index a49d1253..0776e57b 100644 --- a/crates/latkit/shaders/draw/strokes.wesl +++ b/crates/latkit/shaders/draw/strokes.wesl @@ -1,43 +1,42 @@ -// Strokes: one instanced quad per segment of every row's path, for edges and lines alike, covering -// the stroke, or its comets where they are wider, and a pixel of edge; a small segment a later one covers in a crowded cell is left out. A -// segment that ends a part of its path carries the arrowhead; in a diagram, a segment of no length is -// a junction, drawn as a dot. The focus variant draws the pieces of the rows the mark's focus lights -// again, over everything, reaching out to their glow. The cast variant draws the same quads through -// the light's eye into the shadow map. +// Strokes: one instanced quad per segment of every row's path, for edges and lines alike. A quad +// covers the stroke, or its comets where they are wider, any arrowhead, and one pixel of +// antialiased edge. The last segment of each part of a path draws the arrowhead. In a diagram, a +// zero-length segment is a junction, drawn as a dot. The focus variant redraws the lit rows' +// segments over everything, with quads enlarged for their glow. import package::lib::types::{ARROWS, PATH_TRACE, SPACE_DIAGRAM}; import package::lib::path::PATH; import package::lib::bind::{frame, mark, eye}; import package::lib::project::{to_clip, nearer, culled}; import package::lib::path::{ - segment, segment_screen, stroke_style, ends_colored, ends_part, path_count, path_length, px_per_length, + segment, segment_screen, stroke_style, ends_colored, ends_part, path_count, path_length, }; import package::lib::marker::{MarkerColor, glow}; import package::lib::focus::focus_of; -import package::lib::mark::fringe; -import package::lib::stroke::{Stroke, covered, stroke, stroke_distance}; -import package::lib::shade::{caster, daylight}; +import package::lib::mark::{fringe, px_per_length}; +import package::lib::stroke::{Stroke, stroke, stroke_distance}; +import package::lib::daylight::daylight; import package::lib::color::output; struct Varyings { - // The corner in clip space; in the fragment, the target pixel, which `pixel` reads. + // The corner in clip space. In the fragment, the target pixel, which `pixel` reads. @builtin(position) clip: vec4f, @location(0) @interpolate(flat) a: vec2f, @location(1) @interpolate(flat) b: vec2f, @location(2) @interpolate(flat) row: u32, @location(3) color: vec4f, - // Width, distance along at `a` and at `b`, flow. + // Width, distance along at `a` and at `b` in CSS pixels, and flow. @location(4) @interpolate(flat) style: vec4f, - @location(5) height: f32, - // Whether the segment ends in an arrowhead, then whether it dashes. - @location(6) @interpolate(flat) arrow: u32, - // The row's focus bits, which the focus draw glows by. - @location(7) @interpolate(flat) focus: u32, - // Distance along at `a` and at `b` in length units, and how far the row's comets traveled. - @location(8) @interpolate(flat) comet: vec3f, + // Bit 0: the segment ends in an arrowhead. Bit 1: it dashes. + @location(5) @interpolate(flat) arrow: u32, + // The row's focus bits, which choose the focus draw's glow. + @location(6) @interpolate(flat) focus: u32, + // Distance along at `a` and at `b` in position units, and how far the row's comets have + // traveled. + @location(7) @interpolate(flat) comet: vec3f, } -// Whether this variant draws lit rows, its quads reaching out to their glow. +// Whether this variant draws lit rows, with quads enlarged for their glow. @if(focused) const LIT_DRAW: bool = true; @if(!focused) @@ -48,11 +47,14 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: var out: Varyings; let s = segment(instance); out.row = s.row; - if LIT_DRAW && (s.row >= mark.rows || s.index >= path_count(s.row)) { + if LIT_DRAW && (s.item >= mark.rows || s.index >= path_count(s.item)) { out.clip = culled(); return out; } - let style = stroke_style(s.row); + // A stroke fades and scales with its row's tile. + var style = stroke_style(s.row); + style.color.a *= s.presence; + style.width *= s.growth; let on = segment_screen(s); let ends = on.ends; if !style.shown || !on.shown { @@ -77,7 +79,7 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: out.clip = nearer(to_clip(screen, select(ca, cb, t > 0.5))); out.a = a; out.b = b; - // How far along the whole segment this corner lies, of the part drawn. + // This corner's fraction along the whole segment, at the start or end of the drawn part. let f = mix(on.kept.x, on.kept.y, t); var color = style.color; if ends_colored() { @@ -85,9 +87,9 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: } let light = daylight(mix(ends[0].position.xy, ends[1].position.xy, f)); out.color = vec4f(color.rgb * light, color.a); - // In length units, a path runs from its start, a listed one's segment from where its list says it - // starts, its ends stopping short where the stroke does. On screen, a trace runs along x, a - // listed path as it does in length units, and else a segment from where its stroke starts. + // In position units, a segment's distance along starts where its list says (0 without a list), + // and its ends stop short where the stroke does. On screen, a trace measures along x, a listed + // path converts its distance along to pixels, and any other segment starts at 0. let along = s.along + path_length(ends[0], ends[1]) * on.kept; let pixels = length(b - a); var phase = vec2f(0.0, pixels); @@ -98,15 +100,15 @@ fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: } out.style = vec4f(width, phase, style.flow); out.comet = vec3f(along, style.travel); - out.height = mix(ends[0].world.z, ends[1].world.z, f); out.arrow = u32(arrow) | (u32(style.dash && !junction) << 1u); out.focus = focus_of(s.row); return out; } -// The fragment's pixel in CSS pixels, a screen position as `a` and `b` are. Read from the fragment's -// own position, never interpolated: a quad whose ends lie at very different depths skews what is -// interpolated, and a renderer clipping the quad at the screen's edge, as lavapipe does, moves it. +// The fragment's position in CSS pixels, in the same screen space as `a` and `b`. It comes from the +// fragment's own position, never an interpolated value. Interpolation skews across a quad whose +// ends lie at very different depths, and a renderer that clips the quad at the screen's edge (as +// lavapipe does) shifts it. fn pixel(v: Varyings) -> vec2f { return v.clip.xy / eye.pixel_ratio; } @@ -116,7 +118,7 @@ fn segment_of(v: Varyings) -> Stroke { return Stroke(v.a, v.b, v.style.yz, v.comet.xy, v.style.x, v.color, v.style.w, v.comet.z, dash, (v.arrow & 1u) != 0u); } -@if(!casts && !focused) +@if(!focused) @fragment fn fragment(v: Varyings) -> @location(0) vec4f { let c = stroke(pixel(v), segment_of(v)); @@ -130,14 +132,3 @@ fn fragment(v: Varyings) -> @location(0) vec4f { let p = pixel(v); return output(glow(MarkerColor(stroke(p, s), stroke_distance(p, s).x), v.focus)); } - -@if(casts) -@fragment -fn fragment(v: Varyings) -> @location(0) vec4f { - let s = segment_of(v); - let alpha = s.color.a * covered(stroke_distance(pixel(v), s), s); - if alpha <= 0.01 { - discard; - } - return caster(alpha, v.height); -} diff --git a/crates/latkit/shaders/draw/triangles.wesl b/crates/latkit/shaders/draw/triangles.wesl index ce734c92..25462a2f 100644 --- a/crates/latkit/shaders/draw/triangles.wesl +++ b/crates/latkit/shaders/draw/triangles.wesl @@ -1,49 +1,45 @@ -// Triangles: one instanced triangle per face row, lit from the camera so a flat mesh seen from above -// shows its exact colors; turned away, a face darkens by the theme's shading. Faces receive the -// ground's shadow map. The cast variant draws them through the light's eye into the map. +// Triangles: one instanced triangle per face row, lit from the camera so a flat mesh seen from +// above shows its exact colors. A face turned away darkens by the theme's shading. -import package::lib::bind::{frame, eye}; +import package::lib::bind::{frame, eye, word}; import package::lib::project::culled; import package::lib::triangles::{triangle_corner, triangle_shown, corner_clip}; +import package::lib::state::end_normal; import package::lib::marker::{MarkerColor, glow}; import package::lib::focus::focus_of; -import package::lib::shade::{caster, lit}; import package::lib::color::output; -// Each vertex's summed face normals, in fixed point. -@group(2) @binding(3) var normals: array; -@if(!casts) -@group(2) @binding(10) var shadows: texture_2d; - struct Varyings { @builtin(position) clip: vec4f, @location(0) world: vec3f, @location(1) normal: vec3f, @location(2) color: vec4f, @location(3) @interpolate(flat) row: u32, - // The row's focus bits: a lit face tints by them. + // The row's focus bits, which tint a lit face. @location(4) @interpolate(flat) focus: u32, } @vertex -fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) row: u32) -> Varyings { +fn vertex(@builtin(vertex_index) corner: u32, @builtin(instance_index) instance: u32) -> Varyings { var out: Varyings; - out.row = row; - let c = triangle_corner(row, corner); - if !triangle_shown(row) { + let c = triangle_corner(instance, corner); + out.row = c.row; + if !triangle_shown(instance) { out.clip = culled(); return out; } out.clip = corner_clip(c); out.world = c.world; - let s = c.slot * 3u; - out.normal = vec3f(f32(normals[s]), f32(normals[s + 1u]), f32(normals[s + 2u])); + // The face normals summed at this corner's vertex, in fixed point. + let n = end_normal(corner, c.vertex); + if n != NONE { + out.normal = vec3f(f32(bitcast(word(n))), f32(bitcast(word(n + 1u))), f32(bitcast(word(n + 2u)))); + } out.color = c.color; - out.focus = focus_of(row); + out.focus = focus_of(c.row); return out; } -@if(!casts) @fragment fn fragment(v: Varyings) -> @location(0) vec4f { let to_eye = normalize(eye.position.xyz - v.world); @@ -55,12 +51,5 @@ fn fragment(v: Varyings) -> @location(0) vec4f { let shine = pow(max(dot(n, normalize(frame.light.xyz + to_eye)), 0.0), 48.0) * 0.3 * frame.shading; let rim = pow(1.0 - facing, 4.0) * 0.25 * frame.shading; let rgb = min(v.color.rgb * diffuse + shine + rim, vec3f(1.0)); - let shadowed = mix(rgb, frame.shade.color.rgb, 1.0 - lit(shadows, v.world, false)); - return output(glow(MarkerColor(vec4f(shadowed, v.color.a), 0.0), v.focus)); -} - -@if(casts) -@fragment -fn fragment(v: Varyings) -> @location(0) vec4f { - return caster(v.color.a, v.world.z); + return output(glow(MarkerColor(vec4f(rgb, v.color.a), 0.0), v.focus)); } diff --git a/crates/latkit/shaders/lib/bind.wesl b/crates/latkit/shaders/lib/bind.wesl index 0fee192f..5d1b98b6 100644 --- a/crates/latkit/shaders/lib/bind.wesl +++ b/crates/latkit/shaders/lib/bind.wesl @@ -3,23 +3,38 @@ import package::lib::types::{Frame, Eye, Mark}; @group(0) @binding(0) var frame: Frame; -// Data pages: column values as f32 or u32 words, in blocks the arena hands out. -@group(0) @binding(1) var pages: array; -// Compiled marks: column descriptors, page tables, code, constants, ends. -@group(0) @binding(2) var program: array; -// Selection and hover bits, one word per row. -@group(0) @binding(3) var focus: array; -@group(0) @binding(4) var colormaps: texture_2d_array; -@group(0) @binding(5) var ramp: sampler; -// What the pass draws through, at a dynamic offset. -@group(0) @binding(6) var eye: Eye; -// The rows each mark's focus draw lights: every mark's, at its `focused`. -@group(0) @binding(7) var focused: array; +// Every mark's compiled program: column descriptors, page tables, code, constants, ends. +@group(0) @binding(1) var program: array; +@group(0) @binding(2) var colormaps: texture_2d_array; +@group(0) @binding(3) var ramp: sampler; +// The camera the pass draws through. +@group(0) @binding(4) var eye: Eye; +// The drawn rows each mark's focus draw lights, starting at the mark's `focused`. +@group(0) @binding(5) var focused: array; @group(1) @binding(0) var mark: Mark; -// Evaluated rows of every mark: compute passes write them, render passes read them. -@if(writes) -@group(2) @binding(0) var state: array; +// The theme's word in slot `slot`, one of the `THEME_*` slots. +fn theme_word(slot: u32) -> u32 { + return frame.theme[slot / 4u][slot % 4u]; +} + +// All pages, in blocks allocated by the arena. Data pages hold column values as f32 or u32 words. +// Each mark's pages hold its per-row words, which compute passes write and render passes read. A +// pass that sums into them binds them as atomics. +@if(atomic) +@group(2) @binding(0) var pages: array>; +@elif(writes) +@group(2) @binding(0) var pages: array; +@else +@group(2) @binding(0) var pages: array; + +// Word `at` of the pages. +@if(atomic) +fn word(at: u32) -> u32 { + return atomicLoad(&pages[at]); +} @else -@group(2) @binding(0) var state: array; +fn word(at: u32) -> u32 { + return pages[at]; +} diff --git a/crates/latkit/shaders/lib/center.wesl b/crates/latkit/shaders/lib/center.wesl new file mode 100644 index 00000000..04312d8b --- /dev/null +++ b/crates/latkit/shaders/lib/center.wesl @@ -0,0 +1,59 @@ +// Where a drawn row lies on screen: the point a screen query locates a row at, and the point its +// label anchors to. + +import package::lib::types::{POINTS, STROKES, PATH_TRACE, PATH_GEOMETRY}; +import package::lib::mark::PRIMITIVE; +import package::lib::project::{to_screen, clip_of}; +import package::lib::points::{point, Point}; +import package::lib::path::{PATH, path_segment, segment_screen, stroke_style, path_count}; +import package::lib::triangles::{triangle_corner, area_anchor, Corner}; +import package::lib::drawn::held_row; + +// A row's center on screen in CSS pixels, and whether the row shows there. +struct Center { + at: vec2f, + shown: bool, +} + +// Point `p`'s center, shown when its row is visible and placed, in front of the eye. +fn point_center(p: Point) -> Center { + return Center(to_screen(p.clip), p.shown); +} + +// Corner `c` on screen, shown when it has a position, in front of the eye and ahead of the globe's +// horizon. +fn corner_center(c: Corner) -> Center { + let clip = clip_of(c.world); + return Center(to_screen(clip), c.shown && c.ahead && clip.w > 0.0); +} + +// Where drawn row `item` lies on screen: a point's center, the middle of a path's middle segment, a +// trace's last sample, an area's label anchor (inside its largest polygon), or a face's centroid. +// The row shows there when it is visible and that point is on screen. +fn row_center(item: u32) -> Center { + switch PRIMITIVE { + case POINTS: { + return point_center(point(held_row(item))); + } + case STROKES: { + let count = path_count(item); + let last = PATH == PATH_TRACE; + let s = path_segment(item, select(count / 2u, max(count, 1u) - 1u, last), 0.0); + let on = segment_screen(s); + let shown = count > 0u && on.shown && stroke_style(s.row).shown; + return Center(select(0.5 * (on.a + on.b), on.b, last), shown); + } + default: { + if PATH == PATH_GEOMETRY { + return corner_center(area_anchor(held_row(item))); + } + var center = Center(vec2f(0.0), true); + for (var k = 0u; k < 3u; k++) { + let c = corner_center(triangle_corner(item, k)); + center.at += c.at / 3.0; + center.shown = center.shown && c.shown; + } + return center; + } + } +} diff --git a/crates/latkit/shaders/lib/color.wesl b/crates/latkit/shaders/lib/color.wesl index 0a068ae9..27a2587f 100644 --- a/crates/latkit/shaders/lib/color.wesl +++ b/crates/latkit/shaders/lib/color.wesl @@ -1,7 +1,7 @@ // Colors are sRGB-encoded, as in CSS, and packed RGBA8 in state. -import package::lib::types::{MISSING_COLOR, COLORMAP_CYCLIC, COLORMAP_CATEGORICAL}; -import package::lib::bind::{frame, colormaps, ramp}; +import package::lib::types::{THEME_MISSING, COLORMAP_CYCLIC, COLORMAP_CATEGORICAL}; +import package::lib::bind::{frame, colormaps, ramp, theme_word}; import package::lib::read::missing; // `x` mapped linearly from `domain` onto `range`, clamped. @@ -14,10 +14,11 @@ fn scale(x: f32, domain: vec2f, range: vec2f) -> f32 { return mix(range.x, range.y, clamp(t, 0.0, 1.0)); } -// `x` through colormap `layer` over `domain`, packed; missing for a missing `x`. +// `x` through colormap `layer` over `domain`, packed RGBA8. The theme's missing color if `x` is +// missing. fn colormap(layer: u32, kind: u32, x: f32, domain: vec2f) -> u32 { if missing(x) { - return MISSING_COLOR; + return theme_word(THEME_MISSING); } if kind == COLORMAP_CATEGORICAL { let bin = u32(abs(floor(x))) % 256u; @@ -32,7 +33,8 @@ fn to_linear(c: vec3f) -> vec3f { return select(pow((c + 0.055) / 1.055, vec3f(2.4)), c / 12.92, c <= vec3f(0.04045)); } -// A premultiplied fragment color for the target: sRGB values convert when it stores linear color. +// A premultiplied fragment color prepared for the target. sRGB values convert to linear when the +// target stores linear color. fn output(premultiplied: vec4f) -> vec4f { if frame.linear == 0u || premultiplied.a <= 0.0 { return premultiplied; diff --git a/crates/latkit/shaders/lib/daylight.wesl b/crates/latkit/shaders/lib/daylight.wesl new file mode 100644 index 00000000..1b9e01d7 --- /dev/null +++ b/crates/latkit/shaders/lib/daylight.wesl @@ -0,0 +1,19 @@ +// The earth's daylight: how brightly the sun lights a place, down to the theme's night brightness. + +import package::lib::types::SPACE_EARTH; +import package::lib::bind::frame; + +// Night has fully fallen once the cosine of the sun's zenith angle drops to `-TWILIGHT`. +const TWILIGHT: f32 = 0.1; + +// The daylight factor at longitude and latitude `p`, in degrees: 1 by day, easing across the +// terminator down to the night brightness. 1 everywhere when daylight is off or outside the earth. +fn daylight(p: vec2f) -> f32 { + let sun = frame.sun; + if frame.space != SPACE_EARTH || dot(sun.xyz, sun.xyz) == 0.0 { + return 1.0; + } + let r = radians(p); + let n = vec3f(cos(r.y) * cos(r.x), cos(r.y) * sin(r.x), sin(r.y)); + return mix(sun.w, 1.0, smoothstep(-TWILIGHT, TWILIGHT * 0.5, dot(n, sun.xyz))); +} diff --git a/crates/latkit/shaders/lib/drawn.wesl b/crates/latkit/shaders/lib/drawn.wesl new file mode 100644 index 00000000..ff569ee3 --- /dev/null +++ b/crates/latkit/shaders/lib/drawn.wesl @@ -0,0 +1,37 @@ +// Drawn rows: the held row behind each row a mark draws, and its fade while tiles enter and leave. +// A table drawn whole draws held row `i` as drawn row `i`. For a paged table, the program lists the +// runs the mark draws, and `expand_runs` (`compute/lists.wesl`) writes each drawn row's word to the +// lists at `mark.drawn`: its held row, and whether its tile fades in or out. + +import package::lib::types::{NONE, DRAWN_ROW, DRAWN_ENTERING, DRAWN_LEAVING}; +import package::lib::bind::mark; +import package::lib::lists::lists; + +struct DrawnRow { + row: u32, + // Opacity and scale factors while its tile fades in or out. 1 otherwise. + presence: f32, + growth: f32, + // Whether its tile is leaving. A leaving row ignores the pointer. + leaving: bool, +} + +// Drawn row `i` of the current mark. +fn drawn_row(i: u32) -> DrawnRow { + if mark.drawn == NONE { + return DrawnRow(i, 1.0, 1.0, false); + } + let word = lists[mark.drawn + i]; + let entering = (word & DRAWN_ENTERING) != 0u; + let leaving = (word & DRAWN_LEAVING) != 0u; + let presence = select(select(1.0, mark.reveal, entering), 1.0 - mark.reveal, leaving); + return DrawnRow(word & DRAWN_ROW, presence, select(1.0, mark.reveal, entering), leaving); +} + +// The held row of the current mark's drawn row `i`. +fn held_row(i: u32) -> u32 { + if mark.drawn == NONE { + return i; + } + return lists[mark.drawn + i] & DRAWN_ROW; +} diff --git a/crates/latkit/shaders/lib/ends.wesl b/crates/latkit/shaders/lib/ends.wesl new file mode 100644 index 00000000..45bcf9fa --- /dev/null +++ b/crates/latkit/shaders/lib/ends.wesl @@ -0,0 +1,78 @@ +// Resolves each end of an edge or face row to a held row of the vertex mark that draws the end's +// table. A row's ends name places in that table's order, and places stay fixed as the camera moves. +// Their held rows change as the camera moves, because the vertex mark makes other pages and kept +// rows resident in its slots. So no channel stores an end's held row. The expression machine's +// end step resolves it for the rows whose ends changed, and for every row once the vertices moved. +// It writes end `k` into the row's kind word `k`, which draws, probes, and normals read. + +import package::lib::types::{NONE, PAGE_BITS, PAGE_MASK, MARK_FACES, END_WORDS, END_STATE, END_SLOTS, END_COLUMN}; +import package::lib::bind::{program, mark, pages}; +import package::lib::state::record_at; +import package::lib::read::bits_value; + +// The held row of place `place`, from the end table at `slots`. The end table holds the page +// count, then each page's slot (NONE when not resident), then the kept-row hash table: the bucket +// count, then each bucket's place and held row (NONE where empty). A place sits in the first empty +// bucket at or after its hash, as `view/kept.rs` lays them out. Returns NONE when the place is not +// resident. Without an end table (a table drawn whole), each place is its own held row. +fn held_of(slots: u32, place: u32) -> u32 { + if slots == NONE || place == NONE { + return place; + } + let pages = program[slots]; + let page = place >> PAGE_BITS; + if page < pages { + let slot = program[slots + 1u + page]; + if slot != NONE { + return (slot << PAGE_BITS) | (place & PAGE_MASK); + } + } + let kept = slots + 1u + pages; + let buckets = program[kept]; + if buckets == 0u { + return NONE; + } + var b = home(place, buckets); + // At most four fifths of the buckets are full, so a probe always reaches an empty bucket. + for (var probe = 0u; probe < buckets; probe++) { + let at = kept + 1u + 2u * b; + let held = program[at]; + if held == place { + return program[at + 1u]; + } + if held == NONE { + return NONE; + } + b += 1u; + if b == buckets { + b = 0u; + } + } + return NONE; +} + +// The bucket where a probe for place `d` starts: a hash of `d`, matching `view/kept.rs`. +fn home(d: u32, buckets: u32) -> u32 { + var h = d ^ (d >> 16u); + h *= 0x7feb352du; + h ^= h >> 15u; + h *= 0x846ca68bu; + return (h ^ (h >> 16u)) % buckets; +} + +// Writes the held row of each end of the current mark's live row `row`: an edge's two ends, or a +// face's three corners. +fn resolve(row: u32) { + let at = record_at(mark.state, mark.stride + mark.extras, row); + let ends = select(2u, 3u, mark.kind == MARK_FACES); + for (var k = 0u; k < ends; k++) { + let e = mark.ends + END_WORDS * k; + let column = program[e + END_COLUMN]; + // A pole's end is the row itself, at the same held row. + var held = row; + if column != NONE { + held = held_of(program[e + END_SLOTS], bits_value(column, row)); + } + pages[at + k] = select(held, NONE, program[e + END_STATE] == NONE); + } +} diff --git a/crates/latkit/shaders/lib/focus.wesl b/crates/latkit/shaders/lib/focus.wesl index a01b61ae..d38c8941 100644 --- a/crates/latkit/shaders/lib/focus.wesl +++ b/crates/latkit/shaders/lib/focus.wesl @@ -1,13 +1,16 @@ -// Selection and hover: one word of bits per row; and how present a row is while its mark eases. +// Selection and hover: one word of focus bits per held row, in pages listed by the page table at +// `mark.focus`. Also how much of a row shows during a transition. import package::lib::types::{SELECTED, HOVERED, FADE_IN, FADE_OUT}; -import package::lib::bind::{frame, focus, mark}; +import package::lib::bind::{frame, mark, word}; +import package::lib::state::record_at; +// The focus bits of held row `row`. None for a row that left. fn focus_of(row: u32) -> u32 { - if row >= mark.rows { + if row >= mark.held - mark.exits { return 0u; } - return focus[mark.focus + row]; + return word(record_at(mark.focus, 1u, row)); } fn is_selected(row: u32) -> bool { @@ -18,7 +21,7 @@ fn is_hovered(row: u32) -> bool { return (focus_of(row) & HOVERED) != 0u; } -// How much of a row shows while its mark eases: rows appearing fade in, rows leaving fade out. +// A row's opacity factor during a transition: appearing rows fade in, leaving rows fade out. fn presence(flags: u32) -> f32 { if (flags & FADE_IN) != 0u { return mark.motion; @@ -29,8 +32,8 @@ fn presence(flags: u32) -> f32 { return 1.0; } -// How large a row shows while its mark eases: rows appearing grow from nothing. Draws grow them, so -// a mark whose rows only appear evaluates once. +// A row's scale factor during a transition: appearing rows grow from nothing. Draws apply it, so a +// mark whose rows only appear evaluates once. fn growth(flags: u32) -> f32 { return select(1.0, mark.motion, (flags & FADE_IN) != 0u); } diff --git a/crates/latkit/shaders/lib/label.wesl b/crates/latkit/shaders/lib/label.wesl index 5ef4edcc..54703a2a 100644 --- a/crates/latkit/shaders/lib/label.wesl +++ b/crates/latkit/shaders/lib/label.wesl @@ -1,22 +1,22 @@ -// Labels: where a labeled row's text hangs, how large it draws, and whether it can show. Draws place -// every label from its anchor each frame, so labels follow the camera and the rows they name exactly; -// placement measures the same box only to decide which labels give way. Text is a distance field set -// at `TEXT_BASE`, drawn at any size: CSS pixels, or units in a diagram, where text scales with the -// sheet and fades out once too small to read. +// Labels: where a labeled row's text is anchored, its size, the space it claims, and whether it can +// show. Draws place every label from its anchor each frame, so labels track the camera and their +// rows exactly. The label passes measure the same boxes to claim space and find the visible texts. +// Text is a distance field set at `TEXT_BASE` and drawn at any size: in CSS pixels, or in a diagram +// in length units. In a diagram, text scales with the sheet and fades out once too small to read. import package::lib::types::{ - POINTS, STROKES, MARKER_BLOCK, SIZED, GROUP, PATH_TRACE, PATH_GEOMETRY, TEXT_BASE, TEXT_SPREAD, HEADER_BAND, - SPACE_DIAGRAM, + POINTS, MARKER_BLOCK, SIZED, GROUP, PATH_TRACE, TEXT_BASE, TEXT_SPREAD, HEADER_BAND, SPACE_DIAGRAM, + LABEL_CELL, LABEL_MARGIN, LABEL_GUESS, }; import package::lib::bind::{frame, eye, mark}; import package::lib::mark::PRIMITIVE; -import package::lib::project::{to_screen, clip_of}; import package::lib::points::point; -import package::lib::path::{PATH, Segment, segment_screen, path_count}; -import package::lib::triangles::{triangle_corner, area_anchor}; +import package::lib::path::PATH; +import package::lib::center::{row_center, point_center}; +import package::lib::drawn::held_row; -// Where row `row`'s label hangs, CSS pixels, how it aligns to that point (0 left or top, 0.5 -// centered), the room it must fit in (none when zero), and whether its row shows. +// A label's anchor point in CSS pixels, its alignment there (0 left or top, 0.5 centered), the room +// it must fit in (no limit when zero), and whether its row shows. struct Anchor { at: vec2f, align: vec2f, @@ -24,75 +24,49 @@ struct Anchor { shown: bool, } -// CSS pixels a size in length units spans this frame. +// A size in length units, in CSS pixels at this frame's zoom. fn length_px(size: f32) -> f32 { return size * frame.unit; } -fn anchor(row: u32) -> Anchor { - var a: Anchor; - switch PRIMITIVE { - case POINTS: { - let p = point(row); - a.shown = p.shown; - let center = to_screen(p.clip); - let text = length_px(mark.label_size); - if (mark.flags & GROUP) != 0u { - // A group's title sits in its header band, inset from the left. - let header = text * HEADER_BAND; - a.at = center + vec2f(-p.half.x + frame.group_padding, -p.half.y + 0.5 * header); - a.align = vec2f(0.0, 0.5); - a.room = vec2f(2.0 * (p.half.x - frame.group_padding), header); - } else if mark.marker == MARKER_BLOCK || (mark.flags & SIZED) != 0u { - // A block, or any marker given a width and height, holds its label while it fits. - a.at = center; - a.align = vec2f(0.5); - a.room = vec2f(2.0 * p.half.x - text, 2.0 * p.half.y); - } else { - a.at = center + vec2f(p.half.x + 0.4 * text, 0.0); - a.align = vec2f(0.0, 0.5); - } - } - case STROKES: { - let count = path_count(row); - // A trace labels its last sample; other paths their middle. - let last = PATH == PATH_TRACE; - let on = segment_screen(Segment(row, select(count / 2u, max(count, 1u) - 1u, last), 0.0)); - a.shown = count > 0u && on.shown; - if last { - a.at = on.b + vec2f(6.0, 0.0); - a.align = vec2f(0.0, 0.5); - } else { - a.at = 0.5 * (on.a + on.b); - a.align = vec2f(0.5); - } - } - default: { - a.align = vec2f(0.5); - if PATH == PATH_GEOMETRY { - // An area's label hangs inside its largest polygon, farthest from its edges. - let c = area_anchor(row); - let clip = clip_of(c.world); - a.shown = c.shown && c.ahead && clip.w > 0.0; - a.at = to_screen(clip); - return a; - } - var sum = vec2f(0.0); - a.shown = true; - for (var k = 0u; k < 3u; k++) { - let c = triangle_corner(row, k); - let clip = clip_of(c.world); - a.shown = a.shown && c.shown && c.ahead && clip.w > 0.0; - sum += to_screen(clip); - } - a.at = sum / 3.0; +// The label anchor of drawn row `item`: centered on the row's center, except beside a point's +// marker, inside a block or a sized marker, in a group's header band, and beside a trace's last +// sample. +fn anchor(item: u32) -> Anchor { + var a = Anchor(vec2f(0.0), vec2f(0.5), vec2f(0.0), false); + if PRIMITIVE != POINTS { + let c = row_center(item); + a.at = c.at; + a.shown = c.shown; + if PATH == PATH_TRACE { + a.at.x += 6.0; + a.align = vec2f(0.0, 0.5); } + return a; + } + let p = point(held_row(item)); + let c = point_center(p); + a.shown = c.shown; + let text = length_px(mark.label_size); + if (mark.flags & GROUP) != 0u { + // A group's title sits in its header band, inset from the left. + let header = text * HEADER_BAND; + a.at = c.at + vec2f(-p.half.x + frame.group_padding, -p.half.y + 0.5 * header); + a.align = vec2f(0.0, 0.5); + a.room = vec2f(2.0 * (p.half.x - frame.group_padding), header); + } else if mark.marker == MARKER_BLOCK || (mark.flags & SIZED) != 0u { + // A block, or a marker with a width and height, shows its label inside if it fits. + a.at = c.at; + a.room = vec2f(2.0 * p.half.x - text, 2.0 * p.half.y); + } else { + a.at = c.at + vec2f(p.half.x + 0.4 * text, 0.0); + a.align = vec2f(0.0, 0.5); } return a; } -// How much text `px` CSS pixels tall shows: in a diagram it fades out as the view zooms away from -// it; elsewhere it is set at a size that reads. +// Opacity of text `px` CSS pixels tall. In a diagram, text fades out as the view zooms out. +// Elsewhere text has a fixed, readable size, so it always shows. fn legibility(px: f32) -> f32 { if frame.space != SPACE_DIAGRAM { return 1.0; @@ -100,9 +74,8 @@ fn legibility(px: f32) -> f32 { return smoothstep(3.5, 6.0, px); } -// A run of text placed on screen: the text's box and the quad around it, which reaches over the -// distance field's spread, in CSS pixels; its corners in the atlas, texels; and CSS pixels per -// texel. +// A run of text placed on screen: the text's box and its quad, which extends over the distance +// field's spread (both in CSS pixels), its atlas corners in texels, and CSS pixels per texel. struct Text { lo: vec2f, hi: vec2f, @@ -113,12 +86,13 @@ struct Text { scale: f32, } -// A rectangle's corner in the atlas: 16 bits of x, then 16 of y. +// An atlas corner packed in one word: x in the low 16 bits, y in the high 16. fn corner(word: u32) -> vec2f { return vec2f(f32(word & 0xffffu), f32(word >> 16u)); } -// The run at atlas corners `uv0` and `uv1`, set `size` CSS pixels tall, hung at `at` by `align`. +// The run at atlas corners `uv0` and `uv1`, drawn at size `size` in CSS pixels, aligned by `align` +// at point `at`. fn text_at(uv0: u32, uv1: u32, size: f32, at: vec2f, align: vec2f) -> Text { var t: Text; t.uv_lo = corner(uv0); @@ -133,12 +107,61 @@ fn text_at(uv0: u32, uv1: u32, size: f32, at: vec2f, align: vec2f) -> Text { return t; } -// Whether text of extent `extent` fits `room`; no room holds any. +// Whether text of extent `extent` fits `room`. A room of zero width fits any text. fn fits(extent: vec2f, room: vec2f) -> bool { return room.x <= 0.0 || all(extent <= room); } -// Whether a box reaches into the view. +// Whether a box overlaps the view. fn in_view(lo: vec2f, hi: vec2f) -> bool { return all(hi > vec2f(0.0)) && all(lo < eye.size); } + +// Labels beside their rows compete for space. Each label writes its claim key into the cells its +// box covers, and each cell keeps the largest key. A label shows only if it keeps every cell it +// covers, so where labels would overlap, the label of the row drawn on top shows. The claims depend +// on this frame alone. A label inside its row (a block's label, or a group's title) or in a diagram +// does not compete. + +// Whether the label anchored at `a` competes for space with other labels. +fn claims_room(a: Anchor) -> bool { + return a.room.x <= 0.0 && frame.space != SPACE_DIAGRAM; +} + +// The claim key of drawn row `i`. Later marks, then later rows, draw on top, so their keys are +// larger. Never 0, which marks a cell no label claimed. +fn claim_key(i: u32) -> u32 { + return (min(mark.index, 255u) << 24u) | min(i + 1u, 0xffffffu); +} + +// The box a label claims: its text's box once set, else space for a short text at its anchor. +fn claimed(a: Anchor, uv0: u32, uv1: u32, size: f32) -> Text { + if uv0 != 0u || uv1 != 0u { + return text_at(uv0, uv1, size, a.at, a.align); + } + var t: Text; + t.lo = a.at - vec2f(LABEL_GUESS * size, size) * a.align; + t.hi = t.lo + vec2f(LABEL_GUESS * size, size); + return t; +} + +// The claim grid's cells a box covers: its first and last column and row (none when it lies off the +// grid), and the grid's width in cells. +struct Cells { + first: vec2i, + last: vec2i, + width: i32, +} + +fn cells_of(t: Text) -> Cells { + let margin = LABEL_MARGIN * eye.size; + let size = vec2i(ceil((eye.size + 2.0 * margin) / LABEL_CELL)); + let first = max(vec2i(floor((t.lo + margin) / LABEL_CELL)), vec2i(0)); + let last = min(vec2i(floor((t.hi + margin) / LABEL_CELL)), size - 1); + return Cells(first, last, size.x); +} + +// Cell `x`, `y`'s index in the grid of `c`. +fn cell(c: Cells, x: i32, y: i32) -> u32 { + return u32(y * c.width + x); +} diff --git a/crates/latkit/shaders/lib/lists.wesl b/crates/latkit/shaders/lib/lists.wesl index 4766eeb7..acaa4cf8 100644 --- a/crates/latkit/shaders/lib/lists.wesl +++ b/crates/latkit/shaders/lib/lists.wesl @@ -1,4 +1,5 @@ -// Lists: per-item counts, scanned in place to where each item's entries start, then the entries. +// Lists: per-item counts, then entries. A scan turns the counts in place into each item's first +// entry. import package::lib::types::{LIST_ITEMS, LIST_COUNTS, LIST_ENTRIES, LIST_DRAW, LIST_KEPT}; @@ -17,9 +18,9 @@ fn list_entry(header: u32, i: u32) -> u32 { return lists[header + LIST_ENTRIES] + i; } -// Where item `i`'s entries start among those the list at `header` keeps: where it counted them, or, -// when the items counted more than the list keeps, as far in proportion, so every item keeps a share -// of its entries instead of the last items losing all of theirs. +// Where item `i`'s entries start among those the list at `header` keeps. When the items counted +// more entries than the list keeps, the starts scale down in proportion, so every item keeps a +// share of its entries instead of the last items losing all of theirs. fn list_start(header: u32, i: u32) -> u32 { let counts = lists[header + LIST_COUNTS]; let total = lists[counts + lists[header + LIST_ITEMS]]; diff --git a/crates/latkit/shaders/lib/mark.wesl b/crates/latkit/shaders/lib/mark.wesl index 8e671a36..b873ed50 100644 --- a/crates/latkit/shaders/lib/mark.wesl +++ b/crates/latkit/shaders/lib/mark.wesl @@ -1,19 +1,25 @@ -// What every mark's shaders share. +// Helpers shared by every mark's shaders. -import package::lib::types::POINTS; -import package::lib::bind::{eye, mark}; +import package::lib::types::{POINTS, SPACE_EARTH}; +import package::lib::bind::{frame, eye, mark}; -// The primitive a pipeline's marks draw, `POINTS`, `STROKES`, or `TRIANGLES`, for the shaders made for -// one: each pipeline holds only that one's geometry, and compiles quickly. +// The primitive a pipeline's marks draw: `POINTS`, `STROKES`, or `TRIANGLES`. Each pipeline +// compiles only that primitive's geometry, so it compiles quickly. override PRIMITIVE: u32 = POINTS; -// A size in CSS pixels: state word `word` holds position units when the mark measures it in units. +// CSS pixels per position unit at the camera's center. On the earth a position unit is a degree, +// and the eye's scale is per mercator radian. +fn px_per_length() -> f32 { + return eye.px_per_unit.x * select(1.0, radians(1.0), frame.space == SPACE_EARTH); +} + +// `size` in CSS pixels. State word `word` holds position units when the mark measures it in units. fn px(size: f32, word: u32) -> f32 { - return select(size, size * eye.px_per_unit.x, (mark.units & (1u << word)) != 0u); + return select(size, size * px_per_length(), (mark.units & (1u << word)) != 0u); } -// A marker's half, CSS pixels, scaled to keep its larger side within `range`: what a size in units -// keeps of its looks at any zoom. +// A marker's half size in CSS pixels, scaled so its larger side stays within `range`. Both sides +// scale together, so a size in units keeps its shape at any zoom. fn within(half: vec2f, range: vec2f) -> vec2f { let side = max(half.x, half.y); if side <= 0.0 { @@ -27,7 +33,7 @@ fn coverage(d: f32) -> f32 { return clamp(0.5 - d * eye.pixel_ratio, 0.0, 1.0); } -// CSS pixels a quad reaches past the shape it draws: one target pixel, over which its edge fades. +// How far a quad extends past its shape, in CSS pixels: one target pixel, for the antialiased edge. fn fringe() -> f32 { return 1.0 / eye.pixel_ratio; } diff --git a/crates/latkit/shaders/lib/marker.wesl b/crates/latkit/shaders/lib/marker.wesl index b2434781..73624f41 100644 --- a/crates/latkit/shaders/lib/marker.wesl +++ b/crates/latkit/shaders/lib/marker.wesl @@ -1,10 +1,10 @@ -// Markers are signed distance fields in CSS pixels; a custom marker supplies `fn marker`. +// Markers, drawn as signed distance fields in CSS pixels. A custom marker supplies `fn marker`. import package::lib::types::{ SELECTED, HOVERED, SHAPE_RECTANGLE, SHAPE_ROUNDED, SHAPE_DIAMOND, MARKER_GAUGE, MARKER_PIE, MARKER_BLOCK, THEME_TEXT, THEME_OUTLINE, THEME_CORNER, THEME_PORT_SIZE, THEME_LIFT, }; -import package::lib::bind::{frame, eye, mark, program}; +import package::lib::bind::{frame, eye, mark, program, theme_word}; import package::lib::mark::coverage; import package::lib::project::TAU; @@ -36,21 +36,17 @@ fn shape_distance(shape: u32, p: vec2f, half: vec2f) -> f32 { } } -fn theme_word(slot: u32) -> u32 { - return frame.theme[slot / 4u][slot % 4u]; -} - -// The theme's text color: what ports mix toward, so they read on light and dark grounds. +// The theme's text color. Ports mix toward it so they stay readable on light and dark backgrounds. fn theme_text() -> vec3f { return unpack4x8unorm(theme_word(THEME_TEXT)).rgb; } -// The border of a diagram's blocks and groups; clear for none. +// The border color of a diagram's blocks and groups. Clear for none. fn theme_outline() -> vec4f { return unpack4x8unorm(theme_word(THEME_OUTLINE)); } -// A diagram's corner radius, a port's width, and how far a shadow falls: CSS pixels at this zoom. +// A diagram's corner radius, port width, and shadow offset, in CSS pixels at the current zoom. fn theme_corner() -> f32 { return bitcast(theme_word(THEME_CORNER)) * frame.unit; } @@ -63,8 +59,8 @@ fn theme_lift() -> f32 { return bitcast(theme_word(THEME_LIFT)) * frame.unit; } -// The outline a marker's distance is measured from: blocks are rounded boxes, their corners the -// theme's unless the marker sets its own, others their shape. +// Signed distance from a marker's outline. Blocks are rounded boxes with the theme's corner radius +// unless the marker sets its own. Other markers use their shape. fn outline_distance(p: vec2f, half: vec2f) -> f32 { if mark.marker == MARKER_BLOCK { let corner = select(mark.corner * frame.unit, theme_corner(), mark.corner < 0.0); @@ -78,7 +74,7 @@ fn filled(color: vec4f, distance: f32) -> MarkerColor { return MarkerColor(vec4f(color.rgb, color.a * coverage(distance)), distance); } -// `top` over `bottom`. +// `top` composited over `bottom`. fn over(top: MarkerColor, bottom: MarkerColor) -> MarkerColor { let a = top.color.a + bottom.color.a * (1.0 - top.color.a); let rgb = (top.color.rgb * top.color.a + bottom.color.rgb * bottom.color.a * (1.0 - top.color.a)) / max(a, 1e-6); @@ -124,15 +120,16 @@ fn builtin_marker(f: MarkerFragment) -> MarkerColor { } } -// A group's container: a box with the theme's corners, filled in its color, tinted a step toward the -// text color for each group holding it, and bordered; its title sits in the room atop it. +// A group's container: a bordered box with the theme's corner radius, filled with its color. Each +// enclosing group tints the fill a step toward the text color. The title sits in its header band. fn container(f: MarkerFragment, depth: f32) -> MarkerColor { let fill = mix(f.color.rgb, theme_text(), min(0.06 * depth, 0.3)); let d = box_distance(f.p, f.half, min(theme_corner(), 0.5 * min(f.half.x, f.half.y))); return border(filled(vec4f(fill, f.color.a), d), d, f.color.a); } -// `m`, its outline `d` bordered inside in the theme's outline a unit wide, never under a pixel. +// `m` with a border just inside its outline `d`, in the theme's outline color, one length unit wide +// but at least one target pixel. fn border(m: MarkerColor, d: f32, alpha: f32) -> MarkerColor { let outline = theme_outline(); if outline.a <= 0.0 { @@ -142,16 +139,16 @@ fn border(m: MarkerColor, d: f32, alpha: f32) -> MarkerColor { return MarkerColor(over(filled(vec4f(outline.rgb, outline.a * alpha), abs(d + half) - half), m).color, m.distance); } -// `m` over its shadow on the sheet: its outline `distance` lowered by the theme's lift, as soft -// again, in the theme's shadow color. +// `m` over its shadow on the sheet, in the theme's shadow color. `distance` is measured from the +// outline lowered by the theme's lift, and the shadow's edge blurs over that lift on either side. fn dropped(m: MarkerColor, distance: f32, alpha: f32) -> MarkerColor { let lift = max(theme_lift(), 0.5); - let shade = frame.shade.color; - let a = shade.a * alpha * (1.0 - smoothstep(-lift, lift, distance)); - return MarkerColor(over(m, MarkerColor(vec4f(shade.rgb, a), distance)).color, m.distance); + let a = frame.shadow.a * alpha * (1.0 - smoothstep(-lift, lift, distance)); + return MarkerColor(over(m, MarkerColor(vec4f(frame.shadow.rgb, a), distance)).color, m.distance); } -// A ring of the status color just outside the marker's outline, a part of the drawing in a diagram. +// A ring of the status color just outside the marker's outline. Its size is in length units, so in +// a diagram it scales with the drawing. fn status_ring(m: MarkerColor, status: vec4f) -> MarkerColor { let u = frame.unit; return MarkerColor(over(filled(status, abs(m.distance - 1.5 * u) - 1.25 * u), m).color, m.distance); @@ -162,8 +159,8 @@ fn premultiplied(m: MarkerColor) -> vec4f { return vec4f(m.color.rgb * m.color.a, m.color.a); } -// A premultiplied color with the glow of focus bits `focus` around its shape: the selection's, the -// hover's, or for the end of a lit edge the hover's, fainter. +// A premultiplied color with a glow around its shape for focus bits `focus`: the selection color, +// the hover color, or for the end of a lit edge, a fainter hover color. fn glow(m: MarkerColor, focus: u32) -> vec4f { let base = premultiplied(m); if focus == 0u || frame.glow <= 0.0 { diff --git a/crates/latkit/shaders/lib/path.wesl b/crates/latkit/shaders/lib/path.wesl index e9d59f00..b40b635f 100644 --- a/crates/latkit/shaders/lib/path.wesl +++ b/crates/latkit/shaders/lib/path.wesl @@ -1,39 +1,54 @@ -// Paths: the segments a stroke draws, whatever mark it belongs to. A row's path runs between an -// edge's ends (straight, through the bends of a geometry column, or along the great circle on the -// earth, bowed into its lane), through a line's samples across the axis, or through a geometry -// column's points: a route a diagram's sheet worked out, or a path field. Draws, casts, probes, bounds, and labels all read a -// segment through `segment_ends` and `segment_screen`. Which path a pipeline reads is its `PATH`, so -// each compiles one kind of path. +// Paths: the segments a stroke draws, for any mark kind. A row's path runs between an edge's ends +// (straight, through the bends of a geometry column, or along a great circle on the earth, bowed +// into its lane), through a line's samples across the axis, or through a geometry column's points +// (a route computed for a diagram's sheet, or a path field). Draws, probes, bounds, and labels all +// read segments through `segment_ends` and `segment_screen`. A pipeline's `PATH` selects one kind +// of path, so each pipeline compiles only that kind. import package::lib::types::{ NONE, VISIBLE, PLACED, DASH, ENDS_COLOR, GEODESIC, STEP_BEFORE, STEP_AFTER, MARK_EDGES, PATH_ENDS, PATH_TRACE, - PATH_GEOMETRY, PATH_POLES, EDGE_WIDTH, LINE_WIDTH, GEODESIC_STEP, OWN_HEIGHT, TRIANGLES, MAX_PIECES, BOW_PIECES, SPACE_EARTH, + PATH_GEOMETRY, PATH_POLES, EDGE_WIDTH, LINE_WIDTH, GEODESIC_STEP, OWN_HEIGHT, TRIANGLES, MAX_PIECES, BOW_PIECES, + MISSING, FAMILY_LANES, }; import package::lib::bind::{frame, eye, mark, focused}; -import package::lib::state::{row_at, edge_at, line_at, vertex_slot, Vertex}; -import package::lib::project::{PI, world, clip_of, shows, to_screen}; +import package::lib::state::{row_at, edge_at, line_at, end_vertex, same_vertex, Vertex}; +import package::lib::project::{world, clip_of, shows, to_screen}; import package::lib::read::{missing_bits, column_value, points_of, point_bits}; import package::lib::trace::{trace_run, trace_level, trace_axis, trace_entry}; import package::lib::focus::{faded, growth}; -import package::lib::mark::px; +import package::lib::mark::{px, px_per_length}; import package::lib::points::vertex_half; import package::lib::lists::{lists, list_entry, list_start, list_total}; +import package::lib::drawn::drawn_row; -// The path a pipeline's strokes read: one of the `PATH_` codes, fixed when the pipeline is made. +// The path kind a pipeline's strokes read, one of the `PATH_` codes, fixed when the pipeline is +// created. override PATH: u32 = PATH_ENDS; -// One segment of a row's path, and how far along the path it starts, in length units: position -// units, degrees on the earth. A list keeps each segment's distance along; a row of one segment -// starts at 0. +// One segment of a row's path. `along` is the distance along the path where it starts, in position +// units (degrees on the earth). The segment list stores it. Without a list it is 0. `row` is the +// held row, whose state and columns it reads. `item` is the drawn row, which the mark's list +// numbers. While its tile fades in or out, the segment shows with opacity `presence` and scale +// `growth`, and a leaving segment ignores the pointer. struct Segment { row: u32, + item: u32, index: u32, along: f32, + presence: f32, + growth: f32, + leaving: bool, } -// A point a path runs through: where it stands (position units as high and low parts, elevation), -// in world units and in position units, its color, and whether it is shown. Sources set where it -// stands; `path_point` places it in the world once. +// Segment `index` of drawn row `item`'s path, starting at distance `along`. +fn path_segment(item: u32, index: u32, along: f32) -> Segment { + let d = drawn_row(item); + return Segment(d.row, item, index, along, d.presence, d.growth, d.leaving); +} + +// A point on a path. Its source sets its position as high and low parts plus elevation, and +// `placed_point` sets the same point in world and position units. It also has a color and whether +// it shows. struct PathPoint { hi: vec2f, lo: vec2f, @@ -41,15 +56,16 @@ struct PathPoint { world: vec3f, position: vec3f, color: vec4f, - // A vertex's half size in CSS pixels, a box when it has its own height; zero elsewhere. + // For a vertex, its half size in CSS pixels and whether it is a box (has its own height). Zero + // elsewhere. half: vec2f, boxed: bool, - // Whether it has a place to stand, and whether it lies in front of the globe's horizon. + // Whether it has a position, and whether it lies in front of the globe's horizon. shown: bool, ahead: bool, } -// How a row's path is stroked: its comets' flow, and how far they traveled. +// How a row's path is stroked, including its comets' flow and how far they have traveled. struct Style { width: f32, color: vec4f, @@ -59,9 +75,9 @@ struct Style { shown: bool, } -// A segment on screen: its ends as points, in CSS pixels, and in clip space, whether it shows, and -// the part of it drawn, as fractions of it: cut at the near plane, and stopping short of the vertices -// its ends meet. +// A segment on screen: its end points, its ends in CSS pixels and in clip space, whether it shows, +// and the drawn part as fractions along it. The drawn part is cut at the near plane and stops short +// of the vertices at its ends. struct OnScreen { ends: array, a: vec2f, @@ -72,44 +88,39 @@ struct OnScreen { kept: vec2f, } -// The segment drawn as instance `instance`: a fixed number per row, or numbered by the mark's list, -// whose entries are each segment's row and distance along. +// The segment drawn as instance `instance`. Rows have a fixed number of segments, or the mark's +// list numbers them, with each entry holding a segment's drawn row and distance along. @if(!focused) fn segment(instance: u32) -> Segment { if mark.per_row != NONE { let per = max(mark.per_row, 1u); - return Segment(instance / per, instance % per, 0.0); + return path_segment(instance / per, instance % per, 0.0); } let entry = list_entry(mark.list, 2u * instance); - let row = lists[entry]; - return Segment(row, instance - list_start(mark.list, row), bitcast(lists[entry + 1u])); + let item = lists[entry]; + return path_segment(item, instance - list_start(mark.list, item), bitcast(lists[entry + 1u])); } -// In the focus draw, piece `instance % mark.pieces` of the lit row `instance / mark.pieces`, as far -// along as its list keeps it; pieces past the row's own end draw nothing. +// In the focus draw: segment `instance % mark.pieces` of lit row `instance / mark.pieces`, with its +// distance along from the list. Segments past the row's last draw nothing. @if(focused) fn segment(instance: u32) -> Segment { let pieces = max(mark.pieces, 1u); - let row = focused[mark.focused + instance / pieces]; + let item = focused[mark.focused + instance / pieces]; let index = instance % pieces; var along = 0.0; - if mark.per_row == NONE && row < mark.rows && index < path_count(row) { - along = bitcast(lists[list_entry(mark.list, 2u * (list_start(mark.list, row) + index)) + 1u]); + if mark.per_row == NONE && item < mark.rows && index < path_count(item) { + along = bitcast(lists[list_entry(mark.list, 2u * (list_start(mark.list, item) + index)) + 1u]); } - return Segment(row, index, along); + return path_segment(item, index, along); } -// Length units between two points of a path: what a segment adds to the distance along it. +// Distance between two path points in position units, which a segment adds to the distance along. fn path_length(a: PathPoint, b: PathPoint) -> f32 { return length(b.position.xy - a.position.xy); } -// CSS pixels a length unit spans where the camera looks: a degree's on the earth. -fn px_per_length() -> f32 { - return eye.px_per_unit.x * select(1.0, PI / 180.0, frame.space == SPACE_EARTH); -} - -// Segments of every live row's path. +// Segments across all drawn rows' paths. fn stroke_instances() -> u32 { if mark.per_row != NONE { return mark.rows * mark.per_row; @@ -117,15 +128,15 @@ fn stroke_instances() -> u32 { return list_total(mark.list); } -// Segments in row `row`'s path, as drawn. -fn path_count(row: u32) -> u32 { +// Segments in drawn row `item`'s path. +fn path_count(item: u32) -> u32 { if mark.per_row != NONE { return mark.per_row; } - return list_start(mark.list, row + 1u) - list_start(mark.list, row); + return list_start(mark.list, item + 1u) - list_start(mark.list, item); } -// Segments a geodesic from `a` to `b` takes: one per few degrees of arc. +// Segments a geodesic from `a` to `b` takes: one per `GEODESIC_STEP` degrees of arc. fn arcs(a: vec2f, b: vec2f) -> u32 { let p = radians(a); let q = radians(b); @@ -133,17 +144,60 @@ fn arcs(a: vec2f, b: vec2f) -> u32 { return clamp(u32(ceil(degrees(d) / GEODESIC_STEP)), 1u, MAX_PIECES); } -// Points the geometry column gives row `row`: an edge's bends, or a whole path. +// The geometry column's point count for held row `row`: an edge's bends, or a whole path. Rows that +// left have none. fn geometry_count(row: u32) -> u32 { - if row >= mark.rows { + if row >= mark.held - mark.exits { return 0u; } return points_of(mark.geometry, row).count; } -// Segments row `row`'s path has, or for an area its triangles: the count a list over the mark's rows -// counts. A path between two ends without bends runs in one piece, or bowed into a lane in enough -// for the curve, and along a great circle in one per few degrees of its arc. +// Whether lane `l` of edge row `row` draws. A family draws only the lanes the screen can show +// apart, so a family of thousands of lanes costs only as many lanes as the screen resolves, in +// draws and probes alike. A lane whose curve lies off screen draws nothing. Where neighboring lanes +// lie less than a pixel apart, only every `step`-th lane draws, `step` a power of two, so drawn +// lanes are at least a pixel apart. +fn lane_drawn(row: u32, l: Lane) -> bool { + let e = edge_at(row_at(row)); + let a = end_vertex(0u, e.a); + let b = end_vertex(1u, e.b); + let ca = clip_of(world(a.hi, a.lo, a.elevation)); + let cb = clip_of(world(b.hi, b.lo, b.elevation)); + if ca.w <= 0.0 || cb.w <= 0.0 { + return true; + } + let sa = to_screen(ca); + let sb = to_screen(cb); + if (mark.flags & GEODESIC) == 0u { + // The curve stays within the triangle of its ends and its control point, which lies twice + // as far out as the curve's middle. + let c = between(a, b, 0.5, 2.0 * l.offset * frame.bow); + let cc = clip_of(world(c.hi, c.lo, c.z)); + if cc.w > 0.0 { + let sc = to_screen(cc); + // Room for the stroke's width past the curve. + let margin = vec2f(64.0); + let lo = min(min(sa, sb), sc) - margin; + let hi = max(max(sa, sb), sc) + margin; + if any(hi < vec2f(0.0)) || any(lo > eye.size) { + return false; + } + } + } + // Pixels between neighboring lanes at their middle. + let apart = distance(sa, sb) * abs(frame.bow) * l.step; + if !(apart < 1.0) { + return true; + } + let step = u32(exp2(ceil(log2(1.0 / max(apart, 1e-6))))); + return l.index % step == 0u; +} + +// The segment count of held row `row`'s path, or an area's triangle count, as a segment list counts +// them. A path between two ends without bends takes one segment, or `BOW_PIECES` when bowed into a +// lane, and none when its lane is not drawn (`lane_drawn`). On a great circle it takes one segment +// per `GEODESIC_STEP` degrees of arc instead, when that is more (up to `MAX_PIECES`). fn row_segments(row: u32) -> u32 { switch PATH { case PATH_ENDS: { @@ -151,7 +205,11 @@ fn row_segments(row: u32) -> u32 { if middle > 0u { return middle + 1u; } - let pieces = select(1u, BOW_PIECES, lane(row) != 0.0); + let l = lane_of(row); + if l.offset != 0.0 && !lane_drawn(row, l) { + return 0u; + } + let pieces = select(1u, BOW_PIECES, l.offset != 0.0); if (mark.flags & GEODESIC) == 0u { return pieces; } @@ -159,14 +217,14 @@ fn row_segments(row: u32) -> u32 { if e.a == NONE || e.b == NONE { return 0u; } - let a = vertex_slot(e.a); - let b = vertex_slot(e.b); + let a = end_vertex(0u, e.a); + let b = end_vertex(1u, e.b); return max(pieces, arcs(a.hi + a.lo, b.hi + b.lo)); } case PATH_GEOMETRY: { let count = geometry_count(row); if mark.primitive == TRIANGLES { - // An area's triangles follow the point its label hangs from, three points each. + // An area's points start with its label anchor, then three per triangle. return select(0u, (count - 1u) / 3u, count > 0u); } return max(count, 1u) - 1u; @@ -177,9 +235,9 @@ fn row_segments(row: u32) -> u32 { } } -// The vertex in slot `slot` as a point of a path, placed. -fn vertex_point(slot: u32) -> PathPoint { - let v = vertex_slot(slot); +// Vertex `held` of end `end` as a placed path point. +fn vertex_point(end: u32, held: u32) -> PathPoint { + let v = end_vertex(end, held); var p: PathPoint; p.hi = v.hi; p.lo = v.lo; @@ -191,7 +249,7 @@ fn vertex_point(slot: u32) -> PathPoint { return placed_point(p); } -// CSS pixels from a vertex's middle to its outline along `dir`: a box's or a circle's. +// CSS pixels from a vertex's center to its outline along `dir`, for a box or a circle. fn outline(p: PathPoint, dir: vec2f) -> f32 { if p.boxed { return min(p.half.x / max(abs(dir.x), 1e-6), p.half.y / max(abs(dir.y), 1e-6)); @@ -215,7 +273,7 @@ fn position_point(xy: vec2f, z: f32, color: vec4f) -> PathPoint { return p; } -// `p` placed in the world. +// `p` with `world`, `position`, and `ahead` computed. fn placed_point(p: PathPoint) -> PathPoint { var out = p; out.world = world(p.hi, p.lo, p.z); @@ -224,14 +282,14 @@ fn placed_point(p: PathPoint) -> PathPoint { return out; } -// The point a fraction `t` of the way along the great circle from `a` to `b`, in degrees, set off to -// the left by `bow` of the arc's length. +// The point a fraction `t` along the great circle from `a` to `b` (in degrees), offset to the left +// by `bow` times the arc's length. fn along_circle(a: vec2f, b: vec2f, t: f32, bow: f32) -> vec2f { let p = radians(a); let q = radians(b); let u = vec3f(cos(p.y) * cos(p.x), cos(p.y) * sin(p.x), sin(p.y)); let v = vec3f(cos(q.y) * cos(q.x), cos(q.y) * sin(q.x), sin(q.y)); - // Half the chord's angle, which stays precise for the shortest arcs. + // The arc's angle from its chord, which stays precise for the shortest arcs. let angle = 2.0 * asin(min(1.0, 0.5 * length(v - u))); var w = mix(u, v, t); if angle > 1e-5 { @@ -244,9 +302,9 @@ fn along_circle(a: vec2f, b: vec2f, t: f32, bow: f32) -> vec2f { return vec2f(lon, degrees(asin(clamp(w.z, -1.0, 1.0)))); } -// The point a fraction `t` of the way from vertex `a` to `b`, set off to the left by `bow` of the -// length between them: along the great circle on the earth, else straight from `a`'s high and low -// parts, so it keeps their precision. +// The point a fraction `t` from vertex `a` to `b`, offset to the left by `bow` times their +// distance. A `GEODESIC` mark follows the great circle. Otherwise the point is offset from `a`'s +// high and low parts along a straight line, which keeps their precision. fn between(a: Vertex, b: Vertex, t: f32, bow: f32) -> PathPoint { var p: PathPoint; p.z = mix(a.elevation, b.elevation, t); @@ -260,26 +318,26 @@ fn between(a: Vertex, b: Vertex, t: f32, bow: f32) -> PathPoint { return p; } -// Point `i` of an edge's path, placed: its first end, its bends, then its second end. Without bends, -// it runs between its ends in as many pieces as its list kept for it, each lane bowing in world units -// so the camera never moves it. An end reads only its own vertex. Each kind of point is made in one -// place, so shaders compile small. -fn end_point(row: u32, i: u32) -> PathPoint { +// Point `i` of an edge's path, placed: its first end, its bends, then its second end. Without +// bends, it runs between its ends in `count` segments, and a lane bows in world space, so its curve +// stays fixed as the camera moves. An end reads only its own vertex. Each kind of point is built in +// one place, which keeps compiled shaders small. +fn end_point(row: u32, count: u32, i: u32) -> PathPoint { let e = edge_at(row_at(row)); - if e.a == NONE || e.b == NONE || e.a == e.b { + if e.a == NONE || e.b == NONE || same_vertex(e.a, e.b) { return hidden(); } let bends = points_of(mark.geometry, row); - let last = select(bends.count + 1u, path_count(row), bends.count == 0u); + let last = select(bends.count + 1u, count, bends.count == 0u); if i == 0u || i >= last { - return vertex_point(select(e.b, e.a, i == 0u)); + return vertex_point(select(1u, 0u, i == 0u), select(e.b, e.a, i == 0u)); } - let a = vertex_slot(e.a); - let b = vertex_slot(e.b); + let a = end_vertex(0u, e.a); + let b = end_vertex(1u, e.b); let t = f32(i) / f32(last); var p: PathPoint; if bends.count == 0u { - p = between(a, b, t, lane(row) * frame.lane_bow * 4.0 * t * (1.0 - t)); + p = between(a, b, t, lane(row) * frame.bow * 4.0 * t * (1.0 - t)); } else { let w = point_bits(bends, i - 1u); if missing_bits(w.x) || missing_bits(w.y) { @@ -292,9 +350,9 @@ fn end_point(row: u32, i: u32) -> PathPoint { return placed_point(p); } -// Point `k` of a trace: in a run of buckets, each bucket's least and greatest value at its middle; in -// a run of frames, each frame's value, or with steps the corners between them. A point past its run's -// own breaks the line before the next run. +// Point `k` of a trace. In a run of buckets, each bucket gives its least and greatest value at its +// middle. In a run of frames, each frame gives its value, plus the corners between frames when the +// mark steps. A point past its run's last breaks the line before the next run. fn trace_point(row: u32, k: u32) -> PathPoint { let run = trace_run(k); let local = k - run.point; @@ -337,9 +395,9 @@ fn trace_point(row: u32, k: u32) -> PathPoint { return p; } -// Point `i` of a geometry column's path: polylines whose parts are broken by a missing point. While -// the mark eases, each point moves from the path it ran through when the transition began: point -// for point when both have as many, else from the old point at the same fraction along. +// Point `i` of a geometry column's path: polylines whose parts are separated by missing points. +// During a transition, each point moves from the path at the transition's start: from the same +// point when both paths have the same count, else from the old point at the same fraction along. fn geometry_point(row: u32, i: u32) -> PathPoint { let now = points_of(mark.geometry, row); if i >= now.count { @@ -364,13 +422,13 @@ fn geometry_point(row: u32, i: u32) -> PathPoint { return position_point(xy, 0.0, vec4f(1.0)); } -// Point `i` of a pole: the row's own vertex where it stands, then the ground beneath it. +// Point `i` of a pole: the row's own vertex, then the ground below it. fn pole_point(row: u32, i: u32) -> PathPoint { let e = edge_at(row_at(row)); if e.a == NONE { return hidden(); } - var p = vertex_point(e.a); + var p = vertex_point(0u, e.a); if i > 0u { p.z = 0.0; p.half = vec2f(0.0); @@ -379,10 +437,10 @@ fn pole_point(row: u32, i: u32) -> PathPoint { return p; } -// Point `i` of row `row`'s path, whatever it runs through, placed in the world. -fn path_point(row: u32, i: u32) -> PathPoint { +// Point `i` of held row `row`'s path of `count` segments, for any path kind, placed in the world. +fn path_point(row: u32, count: u32, i: u32) -> PathPoint { switch PATH { - case PATH_ENDS: { return end_point(row, i); } + case PATH_ENDS: { return end_point(row, count, i); } case PATH_TRACE: { return placed_point(trace_point(row, i)); } case PATH_GEOMETRY: { return placed_point(geometry_point(row, i)); } case PATH_POLES: { return pole_point(row, i); } @@ -390,39 +448,61 @@ fn path_point(row: u32, i: u32) -> PathPoint { } } -// The two points segment `s` runs between: points `index` and `index + 1` of its row's path. One loop -// makes both, so the path is compiled once. +// Segment `s`'s two end points: points `index` and `index + 1` of its row's path. One loop computes +// both, so the path code is compiled once. fn segment_ends(s: Segment) -> array { var ends: array; for (var e = 0u; e < 2u; e++) { - ends[e] = path_point(s.row, s.index + e); + ends[e] = path_point(s.row, path_count(s.item), s.index + e); } return ends; } -// Whether segment `s` ends a part of its row's path: the path's last, or one before a break. +// Whether segment `s` ends a part of its row's path: the path's last segment, or one before a +// break. fn ends_part(s: Segment) -> bool { if PATH == PATH_GEOMETRY { let points = points_of(mark.geometry, s.row); let next = s.index + 2u; return next >= points.count || missing_bits(point_bits(points, next).x); } - return s.index + 1u == path_count(s.row); + return s.index + 1u == path_count(s.item); } -// Lanes from the middle that row `row` runs in among edges joining the same two vertices. -fn lane(row: u32) -> f32 { - if mark.lane == NONE || row >= mark.rows { - return 0.0; +// A row's lane in its family (the edges joining the same two vertices): its offset from the middle +// in lanes of the theme's bow, its index in the family counted from one side, and the step between +// lanes. A family spreads at most `FAMILY_LANES` lanes wide, so a larger family packs its lanes +// closer, `step` apart. +struct Lane { + offset: f32, + index: u32, + step: f32, +} + +fn lane_of(row: u32) -> Lane { + if mark.lane == NONE || row >= mark.held - mark.exits { + return Lane(0.0, 0u, 1.0); + } + // The high 16 bits hold twice the lane's position from the middle, plus 32768. The low 16 bits + // hold the family's size. + let word = column_value(mark.lane, row).x; + let size = word & 0xffffu; + if word == NONE || word == MISSING || size < 2u { + return Lane(0.0, 0u, 1.0); } - let bits = column_value(mark.lane, row).x; - return select(bitcast(bits), 0.0, missing_bits(bits)); + let twice = i32(word >> 16u) - 32768; + let step = min(1.0, FAMILY_LANES / f32(size)); + return Lane(0.5 * f32(twice) * step, u32((twice + i32(size) - 1) / 2), step); +} + +fn lane(row: u32) -> f32 { + return lane_of(row).offset; } // The part of the segment from clip position `ca` to `cb` in front of the near plane, as fractions -// along it. Depth is reversed, so a position lies in front where z <= w, and w - z runs linearly along -// the segment: one reaching behind the eye is cut where it crosses, not dropped. Empty where both ends -// lie behind. +// along it. Depth is reversed, so a position lies in front where z <= w, and w - z is linear along +// the segment. A segment reaching behind the eye is cut where it crosses, not dropped. The range is +// empty when both ends lie behind. fn in_front(ca: vec4f, cb: vec4f) -> vec2f { let da = ca.w - ca.z; let db = cb.w - cb.z; @@ -430,7 +510,7 @@ fn in_front(ca: vec4f, cb: vec4f) -> vec2f { return vec2f(select(0.0, t, da < 0.0), select(1.0, t, db < 0.0)); } -// Segment `s` on screen: the part in front of the near plane, `kept` saying which part of it shows. +// Segment `s` on screen: the part in front of the near plane, with `kept` giving the drawn part. fn segment_screen(s: Segment) -> OnScreen { let ends = segment_ends(s); var out: OnScreen; @@ -438,7 +518,7 @@ fn segment_screen(s: Segment) -> OnScreen { let ca = clip_of(ends[0].world); let cb = clip_of(ends[1].world); let cut = in_front(ca, cb); - // A segment reaching past the globe's edge draws, and the globe's depth cuts it there. + // A segment reaching past the globe's edge still draws. The globe's depth clips it there. let ahead = ends[0].ahead || ends[1].ahead; out.shown = ends[0].shown && ends[1].shown && ahead && cut.x < cut.y; out.ca = mix(ca, cb, cut.x); @@ -446,8 +526,8 @@ fn segment_screen(s: Segment) -> OnScreen { out.a = to_screen(out.ca); out.b = to_screen(out.cb); out.kept = cut; - // Strokes stop at the outlines of the vertices they end at, without passing each other; an end - // cut at the near plane has no vertex there. + // Strokes stop at the outlines of their end vertices, and each end gives up at most half the + // length. An end cut at the near plane has no vertex to stop at. let along = out.b - out.a; let length_along = length(along); if length_along > 1e-3 { @@ -462,14 +542,14 @@ fn segment_screen(s: Segment) -> OnScreen { return out; } -// How row `row`'s path is stroked: never thinner than a hairline. Its comets' flow and travel are in -// the lengths the theme's flow measures them in. +// How row `row`'s path is stroked, never thinner than 0.75 CSS pixels. Comet flow and travel are in +// the theme's flow unit. fn stroke_style(row: u32) -> Style { var s: Style; if mark.kind == MARK_EDGES { let e = edge_at(row_at(row)); s.width = clamp(px(e.width, EDGE_WIDTH), mark.size_min, mark.size_max) * growth(e.flags); - // An edge colored by its ends takes their colors where it draws; this only fades it. + // An edge colored by its ends gets their colors in the draw. Here it only fades. s.color = select(unpack4x8unorm(e.color), vec4f(1.0), ends_colored()); s.color.a *= faded(row, e.flags); s.flow = e.flow; diff --git a/crates/latkit/shaders/lib/placement.wesl b/crates/latkit/shaders/lib/placement.wesl deleted file mode 100644 index f279ed31..00000000 --- a/crates/latkit/shaders/lib/placement.wesl +++ /dev/null @@ -1,32 +0,0 @@ -// The buffer label placement works in: each labeled mark's candidates, the texts it asks the host to -// set and those its labels show, and the cells of the screen labels claim. - -import package::lib::types::{CANDIDATE_WORDS, WANTED}; -import package::lib::bind::mark; - -@group(2) @binding(13) var labels: array>; - -fn word(at: u32) -> u32 { - return atomicLoad(&labels[at]); -} - -fn put(at: u32, value: u32) { - atomicStore(&labels[at], value); -} - -fn candidate(row: u32) -> u32 { - return mark.candidates + row * CANDIDATE_WORDS; -} - -// Asks the host to set the text of `code`. -fn want(code: u32) { - let asked = atomicAdd(&labels[mark.wanted], 1u); - if asked < WANTED { - put(mark.wanted + 1u + asked, code); - } -} - -// Notes that a label shows the text of `code`, so the atlas keeps it. -fn seen(code: u32) { - atomicOr(&labels[mark.seen + code / 32u], 1u << (code % 32u)); -} diff --git a/crates/latkit/shaders/lib/points.wesl b/crates/latkit/shaders/lib/points.wesl index e051ec03..a4bee567 100644 --- a/crates/latkit/shaders/lib/points.wesl +++ b/crates/latkit/shaders/lib/points.wesl @@ -1,22 +1,23 @@ -// Points: where a vertex row's marker stands, how large, and in what color. Draws, casts, probes, -// bounds, and labels all read a point through `point`. +// Points: a vertex row's marker position, size, and color. Draws, probes, bounds, and labels all +// read a point through `point`. import package::lib::types::{ - NONE, VISIBLE, PLACED, SIZED, STATUS, CASTS, HOVERED, VERTEX_HALF, VERTEX_WORDS, SIDE_LEFT, SIDE_RIGHT, - SIDE_TOP, PORT_WORDS, PORT_CODE, PORT_SHAPE, PORT_COLUMN, HALF_UNITS, OWN_HEIGHT, SPACE_DIAGRAM, + NONE, VISIBLE, PLACED, SIZED, STATUS, SHADOW, HOVERED, VERTEX_HALF, VERTEX_WORDS, SIDE_LEFT, SIDE_RIGHT, + SIDE_TOP, PORT_WORDS, PORT_CODE, PORT_SHAPE, PORT_COLUMN, HALF_UNITS, OWN_HEIGHT, }; -import package::lib::bind::{frame, eye, mark, program}; -import package::lib::state::{vertex_slot, Vertex}; +import package::lib::bind::{frame, mark, program}; +import package::lib::state::{row_at, vertex_at, Vertex}; import package::lib::project::{world, clip_of, shows, to_screen}; import package::lib::focus::{focus_of, faded, growth}; import package::lib::read::bits_value; -import package::lib::mark::{px, fringe, within}; +import package::lib::mark::{px, px_per_length, fringe, within}; import package::lib::marker::{theme_port_size, theme_lift}; +import package::lib::drawn::drawn_row; struct Point { clip: vec4f, world: vec3f, - // Where it stands, in position units. + // Its position, in position units. position: vec2f, // Half the marker's width and height, CSS pixels. half: vec2f, @@ -33,10 +34,10 @@ fn placed(v: Vertex) -> bool { return (v.flags & PLACED) != 0u; } -// The marker of row `row` of the current vertex mark. A hovered marker sized in pixels grows; one -// sized in units is part of the drawing, and only glows. +// The marker of the current vertex mark's held row `row`. A hovered marker sized in pixels grows. +// One sized in units is part of the drawing, so it only glows. fn point(row: u32) -> Point { - let v = vertex_slot(mark.state / VERTEX_WORDS + row); + let v = vertex_at(row_at(row)); var half = vec2f(px(v.half.x, VERTEX_HALF)); if (mark.flags & SIZED) != 0u { half.y = px(v.half.y, VERTEX_HALF + 1u); @@ -44,7 +45,7 @@ fn point(row: u32) -> Point { half = within(half, vec2f(mark.size_min, mark.size_max)); let focus = focus_of(row); let grows = (mark.units & (1u << VERTEX_HALF)) == 0u && (focus & HOVERED) != 0u; - half *= select(1.0, frame.hover_scale, grows) * growth(v.flags); + half *= select(1.0, frame.grow, grows) * growth(v.flags); let color = unpack4x8unorm(v.color); var p: Point; p.world = world(v.hi, v.lo, v.elevation); @@ -60,18 +61,28 @@ fn point(row: u32) -> Point { return p; } -// Half the marker of the vertex in any slot, CSS pixels: its flags say whether its size is in -// position units and whether it has its own height. +// The marker of drawn row `item`, faded and scaled while its tile fades in or out. +fn shown_point(item: u32) -> Point { + let d = drawn_row(item); + var p = point(d.row); + p.color.a *= d.presence; + p.half *= d.growth; + return p; +} + +// Half the size of a vertex's marker in CSS pixels, for a vertex of any mark. Its flags say whether +// its size is in position units and whether it has its own height. fn vertex_half(v: Vertex) -> vec2f { var half = vec2f(v.half.x, select(v.half.x, v.half.y, (v.flags & OWN_HEIGHT) != 0u)); if (v.flags & HALF_UNITS) != 0u { - half *= eye.px_per_unit.x; + half *= px_per_length(); } return within(half, v.range); } -// Where a port sits on a marker of half size `half`, CSS pixels from its center, y down: its side in -// the code's low two bits, then its place among the ports on that side and how many share it. +// A port's offset from the center of a marker of half size `half`, in CSS pixels with y down. The +// code holds the side in its low two bits, then the port's index on that side and the number of +// ports there. fn port_place(half: vec2f, code: u32) -> vec2f { let along = (f32((code >> 2u) & 255u) + 1.0) / (f32((code >> 10u) & 255u) + 1.0) * 2.0 - 1.0; switch code & 3u { @@ -82,7 +93,7 @@ fn port_place(half: vec2f, code: u32) -> vec2f { } } -// The way into a marker from a port on its side, y down. +// The unit direction from a port into its marker, y down. fn port_inward(code: u32) -> vec2f { switch code & 3u { case SIDE_LEFT: { return vec2f(1.0, 0.0); } @@ -94,12 +105,12 @@ fn port_inward(code: u32) -> vec2f { struct Port { center: vec2f, - // Half its size, CSS pixels: never past a quarter of its marker's shorter side, so a small + // Half its size in CSS pixels, at most a quarter of the marker's shorter side, so a small // marker still shows under its ports. half: f32, shape: u32, code: u32, - // Whether its reference names a net: a free port draws hollow. + // Whether its reference names a net. An unconnected port draws hollow. wired: bool, } @@ -114,11 +125,11 @@ fn port_at(p: Point, k: u32) -> Port { return Port(center, half, program[info + PORT_SHAPE], code, wired); } -// CSS pixels a point's quad reaches past its center: the marker, its status ring when it has one, -// its shadow on a diagram's sheet when it casts one, a target pixel of edge, and when `lit`, as the -// focus draw draws it, its glow. +// How far a point's quad extends from its center, in CSS pixels. It covers the marker, its status +// ring or its shadow on a diagram's sheet, one target pixel of antialiased edge, and when `lit` (in +// the focus draw), its glow. fn reach(p: Point, lit: bool) -> vec2f { let ring = select(0.0, 3.0 * frame.unit, (mark.flags & STATUS) != 0u && p.status.a > 0.0); - let shadow = select(0.0, 2.0 * theme_lift(), (mark.flags & CASTS) != 0u && frame.space == SPACE_DIAGRAM); + let shadow = select(0.0, 2.0 * theme_lift(), (mark.flags & SHADOW) != 0u); return p.half + max(ring, shadow) + select(0.0, frame.glow, lit) + fringe(); } diff --git a/crates/latkit/shaders/lib/project.wesl b/crates/latkit/shaders/lib/project.wesl index fc7b3ed3..9275a1b4 100644 --- a/crates/latkit/shaders/lib/project.wesl +++ b/crates/latkit/shaders/lib/project.wesl @@ -1,6 +1,6 @@ -// From positions to the screen. Positions are (high, low) pairs: their difference from the camera's -// center is taken in parts, so f64 data keeps its precision at any zoom. Every pass projects through -// its eye: the view, the light that casts shadows, or an exported image. +// Projection from positions to the screen. Positions are (high, low) f32 pairs, and their offset +// from the camera's center is computed part by part, so f64 data keeps its precision at any zoom. +// Every pass projects through the view's `eye`. import package::lib::types::{SPACE_EARTH}; import package::lib::bind::{frame, eye}; @@ -8,11 +8,11 @@ import package::lib::bind::{frame, eye}; const PI: f32 = 3.14159265; const TAU: f32 = 6.28318531; const MAX_LATITUDE: f32 = 85.05113; -// Depth is reversed: the near plane is at 1 and the far plane at 0, which keeps precision far away. +// Depth is reversed: the near plane is at 1 and the far plane at 0, for precision far away. const NEAR_Z: f32 = 1.0; const FAR_Z: f32 = 0.0; -// The offset of `p` from the camera's center, in position units. +// The offset of position (`hi`, `lo`) from the camera's center, in position units. fn relative(hi: vec2f, lo: vec2f) -> vec2f { return (hi - frame.center_hi) + (lo - frame.center_lo); } @@ -22,7 +22,8 @@ fn mercator_y(latitude: f32) -> f32 { return log(tan(PI / 4.0 + lat / 2.0)); } -// Web mercator radians from the center; near the center, a local expansion keeps precision. +// Web Mercator coordinates relative to the center, in radians. Near the center, a local expansion +// keeps precision. fn mercator(hi: vec2f, lo: vec2f) -> vec2f { let d = relative(hi, lo); let lat = clamp(hi.y + lo.y, -MAX_LATITUDE, MAX_LATITUDE); @@ -31,7 +32,7 @@ fn mercator(hi: vec2f, lo: vec2f) -> vec2f { return vec2f(radians(d.x), select(far, near, abs(d.y) < 0.25)); } -// `p` on the unit sphere, turned so the camera's center faces +z. +// The position on the unit sphere, rotated so the camera's center faces +z. fn on_sphere(hi: vec2f, lo: vec2f) -> vec3f { let lon = radians(relative(hi, lo).x); let lat = radians(clamp(hi.y + lo.y, -90.0, 90.0)); @@ -40,8 +41,8 @@ fn on_sphere(hi: vec2f, lo: vec2f) -> vec3f { return vec3f(v.x, v.y * cos(lat0) - v.z * sin(lat0), v.y * sin(lat0) + v.z * cos(lat0)); } -// World coordinates: plane or plot units, or mercator radians morphing onto the globe. Only the -// morph works out both. +// World coordinates: plane or plot units relative to the center, or Mercator radians morphing onto +// the globe. Only a morph in progress computes both projections. fn world(hi: vec2f, lo: vec2f, z: f32) -> vec3f { if frame.space != SPACE_EARTH { return vec3f(relative(hi, lo), z); @@ -66,15 +67,15 @@ fn project(hi: vec2f, lo: vec2f, z: f32) -> vec4f { return clip_of(world(hi, lo, z)); } -// The globe's center in world units: a sphere of radius `-center.z` whose top touches the ground at -// the camera's center. +// The globe's center in world units. The globe is a sphere of radius `-center.z` whose top touches +// the ground at the camera's center. fn globe_center() -> vec3f { return vec3f(0.0, 0.0, -1.0 / cos(radians(frame.center_hi.y + frame.center_lo.y))); } -// Whether world point `w` shows past the globe: on the round globe, whether it lies in front of the -// plane the eye sees the globe's edge on, exactly, at any pitch. On the map, or while the globe -// morphs, everything shows, and what the ground hides is left to its depth. +// Whether world point `w` is on the visible side of the globe. On the round globe, it must lie in +// front of the plane of the globe's visible edge, a test that is exact at any pitch. On the map, or +// while the globe morphs, everything shows, and the ground's depth hides what lies behind it. fn shows(w: vec3f) -> bool { if frame.space != SPACE_EARTH || frame.globe < 1.0 { return true; @@ -83,7 +84,7 @@ fn shows(w: vec3f) -> bool { return dot(w - c, eye.position.xyz - w) > 0.0; } -// CSS pixels from the top left. +// A clip position in CSS pixels from the top left. fn to_screen(clip: vec4f) -> vec2f { let ndc = clip.xy / clip.w; return (vec2f(ndc.x, -ndc.y) * 0.5 + 0.5) * eye.size; @@ -95,7 +96,7 @@ fn to_clip(screen: vec2f, at: vec4f) -> vec4f { return vec4f(vec2f(ndc.x, -ndc.y) * at.w, at.z, at.w); } -// `clip` a little toward the camera, so a mark lying on a face draws over it. +// `clip` moved slightly toward the camera, so a mark lying on a face draws over it. fn nearer(clip: vec4f) -> vec4f { return vec4f(clip.xy, min(clip.z * 1.003, clip.w), clip.w); } diff --git a/crates/latkit/shaders/lib/read.wesl b/crates/latkit/shaders/lib/read.wesl index 3216ca6f..25b8f9cd 100644 --- a/crates/latkit/shaders/lib/read.wesl +++ b/crates/latkit/shaders/lib/read.wesl @@ -1,19 +1,21 @@ -// Reading columns: page tables in `program` point at blocks of `pages`. Values stay `u32` bits until -// arithmetic needs them, so a missing value is never an `f32` constant (WGSL cannot represent NaN -// constants). +// Reading columns. Page tables in `program` point at blocks of `pages`, each page holding +// `PAGE_ROWS` rows. A column's pages are its data's pages, or for a paged table the mark's resident +// pages, one per slot. Values stay `u32` bits until arithmetic needs them, because WGSL cannot +// express a missing value as an `f32` constant (NaN). import package::lib::types::{ NONE, MISSING, COLUMN_WORDS, COLUMN_FLAGS, COLUMN_ROWS, COLUMN_TABLE, COLUMN_LOW, COLUMN_JOIN, COLUMN_T, - SAMPLED, STEP, PAGE_BITS, PAGE_MASK, POINT_BITS, POINT_MASK, + SAMPLED, STEP, PAGE_BITS, PAGE_MASK, POINT_BITS, POINT_MASK, PATH_PAGE_WORDS, }; -import package::lib::bind::{pages, program, mark}; +import package::lib::bind::{program, mark, word}; -// Whether a pipeline's marks read fields of other tables by row id, and fields sampled over an axis -// at the shown coordinate: set when it is made, so a scene doing neither compiles neither. +// Whether a pipeline's marks read fields of other tables through a join, and fields sampled over +// the axis. Both are set when the pipeline is created, so a scene that uses neither compiles +// neither. override JOINED: bool = true; override TIMED: bool = true; -// Whether `x` is NaN or infinite: missing. +// Whether `x` is missing: NaN or infinite. fn missing(x: f32) -> bool { return missing_bits(bitcast(x)); } @@ -26,17 +28,39 @@ fn column_word(column: u32, word: u32) -> u32 { return program[mark.columns + column * COLUMN_WORDS + word]; } +// Binary search over `count` items of `stride` words at `at` in `program`. Returns where the last +// item whose word `key` is at most `value` starts, or the first item when none is. +fn last_at_most(at: u32, count: u32, stride: u32, key: u32, value: u32) -> u32 { + var lo = 0u; + var hi = count; + while hi - lo > 1u { + let mid = (lo + hi) / 2u; + if program[at + mid * stride + key] <= value { + lo = mid; + } else { + hi = mid; + } + } + return at + lo * stride; +} + fn page_word(block: u32, offset: u32, fallback: u32) -> u32 { if block == NONE { return fallback; } - return pages[block + offset]; + return word(block + offset); } -// A static column's value as (high, low) bits: the low part carries what f32 drops of an f64. +// The page row `row` lies in, and where in it. +fn page_of(row: u32) -> vec2u { + return vec2u(row >> PAGE_BITS, row & PAGE_MASK); +} + +// A static column's value as (high, low) bits. The low part holds what f32 loses of an f64. fn static_value(column: u32, row: u32) -> vec2u { - let page = row >> PAGE_BITS; - let local = row & PAGE_MASK; + let at = page_of(row); + let page = at.x; + let local = at.y; let table = column_word(column, COLUMN_TABLE); let hi = page_word(program[table + page], local, MISSING); let low = column_word(column, COLUMN_LOW); @@ -46,11 +70,13 @@ fn static_value(column: u32, row: u32) -> vec2u { return vec2u(hi, page_word(program[low + page], local, 0u)); } -// A sampled column's value at the shown coordinate, as bits. Each row block has a cursor of six -// words: block and word of this frame, block and word of the next frame, and both frames' row counts. +// A sampled column's value at the shown coordinate, as bits. Each page of rows has a six-word +// cursor: the block and word of the frame at the shown coordinate, the same for the next frame, and +// both frames' row counts. fn sample_value(column: u32, row: u32) -> u32 { - let cursor = column_word(column, COLUMN_TABLE) + (row >> PAGE_BITS) * 6u; - let local = row & PAGE_MASK; + let at = page_of(row); + let cursor = column_word(column, COLUMN_TABLE) + at.x * 6u; + let local = at.y; if local >= program[cursor + 4u] { return MISSING; } @@ -67,13 +93,14 @@ fn sample_value(column: u32, row: u32) -> u32 { return bitcast(mix(bitcast(a), bitcast(b), t)); } -// A bits column's word: row indices of a reference, NONE where unwired. +// A bits column's word, such as a reference's row index. NONE where unwired. fn bits_value(column: u32, row: u32) -> u32 { if row >= column_word(column, COLUMN_ROWS) { return NONE; } let table = column_word(column, COLUMN_TABLE); - return page_word(program[table + (row >> PAGE_BITS)], row & PAGE_MASK, NONE); + let at = page_of(row); + return page_word(program[table + at.x], at.y, NONE); } // The row a column reads for `row`: through its join when it reads another table. @@ -97,40 +124,36 @@ fn column_value(column: u32, row: u32) -> vec2u { return static_value(column, r); } -// One row's points in a geometry column: where its page's blocks of points are listed in `program`, -// its first point among the page's, and how many. Parts are broken by a missing point. +// One row's points in a geometry column: where its first piece is listed in `program`, its first +// point in that piece, and its point count. A missing point separates parts. struct Points { blocks: u32, first: u32, count: u32, } -// Row `row`'s points of geometry column `column`. The column's table holds, per page, the block of -// each row's first point, where the page's blocks of points are listed in the table, and how many -// points the page has; a block holds `1 << POINT_BITS` points as x and y bits. +// Row `row`'s points in geometry column `column`, from its page's row table. A piece holds up to +// `1 << POINT_BITS` points as x and y bits. A row larger than a piece continues into the pieces +// after its first. fn points_of(column: u32, row: u32) -> Points { if column == NONE || row >= column_word(column, COLUMN_ROWS) { return Points(0u, 0u, 0u); } - let rows = column_word(column, COLUMN_ROWS); let table = column_word(column, COLUMN_TABLE); - let page = table + 3u * (row >> PAGE_BITS); - let firsts = program[page]; - if firsts == NONE { + let at = page_of(row); + let page = table + PATH_PAGE_WORDS * at.x; + let rows = program[page]; + if rows == NONE || at.y >= program[page + 3u] { return Points(0u, 0u, 0u); } - let local = row & PAGE_MASK; - let first = pages[firsts + local]; - var end = program[page + 2u]; - if local < PAGE_MASK && row + 1u < rows { - end = pages[firsts + local + 1u]; - } - return Points(table + program[page + 1u], first, end - first); + let entry = word(rows + 2u * at.y); + let listed = table + program[page + 1u] + (entry >> POINT_BITS); + return Points(listed, entry & POINT_MASK, word(rows + 2u * at.y + 1u)); } // Point `k` of a row's points, as x and y bits. fn point_bits(points: Points, k: u32) -> vec2u { let i = points.first + k; let at = program[points.blocks + (i >> POINT_BITS)] + 2u * (i & POINT_MASK); - return vec2u(pages[at], pages[at + 1u]); + return vec2u(word(at), word(at + 1u)); } diff --git a/crates/latkit/shaders/lib/reduce.wesl b/crates/latkit/shaders/lib/reduce.wesl index a7d2d2c5..af5afa17 100644 --- a/crates/latkit/shaders/lib/reduce.wesl +++ b/crates/latkit/shaders/lib/reduce.wesl @@ -1,5 +1,6 @@ -// Reductions with atomics: an `f32` as `u32` bits that sort in the same order, so `atomicMax` of -// `ordered(x)` keeps the greatest value and `atomicMax` of `~ordered(x)` the least. +// Atomic reductions over `f32`. `ordered` maps an `f32` to `u32` bits that sort in the same +// order, so `atomicMax` of `ordered(x)` keeps the greatest value, and `atomicMax` of `~ordered(x)` +// keeps the least. fn ordered(x: f32) -> u32 { let b = bitcast(x); diff --git a/crates/latkit/shaders/lib/shade.wesl b/crates/latkit/shaders/lib/shade.wesl deleted file mode 100644 index a6c73007..00000000 --- a/crates/latkit/shaders/lib/shade.wesl +++ /dev/null @@ -1,65 +0,0 @@ -// Light: the ground's shadow map, and the earth's daylight. Casters draw into the map through the -// light's eye, blended by MIN: the most covered (1 − coverage), the lowest caster above the ground, -// and the highest (negated). The ground darkens at the level its lowest caster sets, sharp at contact -// and soft from high up; faces compare their height with the highest caster over them. - -import package::lib::types::SPACE_EARTH; -import package::lib::bind::{frame, eye, ramp}; - -// How far the sun stands below the horizon, in the cosine of its angle, where night has fallen. -const TWILIGHT: f32 = 0.1; - -// How lit the earth is at longitude and latitude `p`, in degrees: 1 by day, down to the night's -// brightness, eased across the terminator; 1 everywhere without daylight. -fn daylight(p: vec2f) -> f32 { - let sun = frame.sun; - if frame.space != SPACE_EARTH || dot(sun.xyz, sun.xyz) == 0.0 { - return 1.0; - } - let r = radians(p); - let n = vec3f(cos(r.y) * cos(r.x), cos(r.y) * sin(r.x), sin(r.y)); - return mix(sun.w, 1.0, smoothstep(-TWILIGHT, TWILIGHT * 0.5, dot(n, sun.xyz))); -} - -// What a caster writes, blended by MIN. -fn caster(coverage: f32, height: f32) -> vec4f { - let h = height + eye.lift; - return vec4f(1.0 - coverage, h, -h, 0.0); -} - -// Where world point `p` falls on the map, cast away from the light onto the ground, and its height. -fn on_map(p: vec3f) -> vec3f { - let l = frame.shade.light.xyz; - let ground = p.xy - l.xy / max(l.z, 1e-3) * p.z; - let r = frame.shade.rect; - return vec3f((ground - r.xy) / (r.zw - r.xy), p.z); -} - -// How much light reaches world point `p`: 1 lit, down to 1 − the shadow's alpha. -fn lit(shadows: texture_2d, p: vec3f, ground: bool) -> f32 { - if frame.shade.on == 0u { - return 1.0; - } - let m = on_map(p); - if any(m.xy < vec2f(0.0)) || any(m.xy > vec2f(1.0)) { - return 1.0; - } - let uv = vec2f(m.x, 1.0 - m.y); - let around = textureSampleLevel(shadows, ramp, uv, 3.0); - let spread = frame.shade.light.w / max(frame.shade.texel, 1e-30); - var shadow = 0.0; - if ground { - // The lowest caster around sets how far the shadow spreads. - let gap = max(around.g, 0.0); - let level = clamp(log2(max(gap * spread, 1.0)), 0.0, frame.shade.levels - 1.0); - shadow = 1.0 - textureSampleLevel(shadows, ramp, uv, level).r; - } else { - // A face is shadowed where something stands higher over it on its light ray. - let gap = max(-around.b - m.z, 0.0); - let level = clamp(log2(max(gap * spread, 1.0)), 0.0, frame.shade.levels - 1.0); - let top = -textureSampleLevel(shadows, ramp, uv, level).b; - let bias = 2.0 * frame.shade.texel; - shadow = smoothstep(bias, bias * 4.0, top - m.z); - } - return 1.0 - frame.shade.color.a * shadow; -} diff --git a/crates/latkit/shaders/lib/state.wesl b/crates/latkit/shaders/lib/state.wesl index 65e772fa..a128f2ba 100644 --- a/crates/latkit/shaders/lib/state.wesl +++ b/crates/latkit/shaders/lib/state.wesl @@ -1,14 +1,16 @@ -// Evaluated rows: words of the `state` buffer. A row's kind words sit at `mark.state`, `mark.stride` -// apart, so vertex slots count every vertex mark's rows; its extra words (labels, marker inputs) sit -// at `mark.extra`, `mark.extras` apart. +// Reading evaluated rows. Each mark keeps its per-row words in pages of `PAGE_ROWS` held rows, +// listed by the page table at `mark.state`. A row has `mark.stride` kind words, then `mark.extras` +// extra words (labels, marker inputs). An edge or face mark reads its vertices through the page +// tables of the vertex marks that draw them, which its ends list. import package::lib::types::{ - NONE, VERTEX_WORDS, VERTEX_POSITION, VERTEX_COLOR, VERTEX_STATUS, VERTEX_HALF, VERTEX_ELEVATION, VERTEX_DEPTH, - VERTEX_RANGE, VERTEX_FLAGS, EDGE_FROM, EDGE_TO, EDGE_COLOR, EDGE_WIDTH, EDGE_FLOW, EDGE_TRAVEL, EDGE_FLAGS, FACE_A, FACE_B, - FACE_C, FACE_COLOR, FACE_FLAGS, LINE_COLOR, LINE_WIDTH, LINE_FLOW, LINE_TRAVEL, LINE_FLAGS, AREA_COLOR, - AREA_FLAGS, MARK_EDGES, MARK_LINES, + NONE, PAGE_BITS, PAGE_MASK, END_WORDS, END_STATE, END_STRIDE, END_NORMALS, VERTEX_POSITION, VERTEX_COLOR, + VERTEX_STATUS, VERTEX_HALF, VERTEX_ELEVATION, VERTEX_DEPTH, VERTEX_RANGE, VERTEX_FLAGS, EDGE_FROM, EDGE_TO, + EDGE_COLOR, EDGE_WIDTH, EDGE_FLOW, EDGE_TRAVEL, EDGE_FLAGS, FACE_A, FACE_B, FACE_C, FACE_COLOR, FACE_FLAGS, + LINE_COLOR, LINE_WIDTH, LINE_FLOW, LINE_TRAVEL, LINE_FLAGS, AREA_COLOR, AREA_FLAGS, MARK_EDGES, MARK_LINES, + NORMAL_WORDS, }; -import package::lib::bind::{state, mark}; +import package::lib::bind::{program, mark, word}; struct Vertex { hi: vec2f, @@ -18,7 +20,7 @@ struct Vertex { half: vec2f, elevation: f32, depth: f32, - // The CSS pixels its marker's half keeps within. + // The range of CSS pixels its marker's half size is clamped to. range: vec2f, flags: u32, } @@ -52,8 +54,8 @@ struct Area { flags: u32, } -// A range of CSS pixels in one word: each end in sixteenths of a pixel, up to 4095, the top value -// meaning no end. +// Packs a range of CSS pixels into one word: each end in 16 bits, in 1/16 pixel, up to 4095.9375 +// pixels. An upper end of 65535 means no upper bound. fn pack_range(range: vec2f) -> u32 { let ends = vec2u(round(clamp(range, vec2f(0.0), vec2f(4095.9375)) * 16.0)); let top = select(ends.y, 65535u, range.y > 4095.9375); @@ -62,22 +64,27 @@ fn pack_range(range: vec2f) -> u32 { fn unpack_range(word: u32) -> vec2f { let ends = vec2f(f32(word & 65535u), f32(word >> 16u)) / 16.0; - // The top of what a word holds is no end at all. + // The top value means no upper bound. return vec2f(ends.x, select(ends.y, 3.4e38, (word >> 16u) == 65535u)); } fn float_at(at: u32) -> f32 { - return bitcast(state[at]); + return bitcast(word(at)); } -// The first kind word of `row` of the current mark. +// Where `row`'s words start in the pages, given page table `table` and `stride` words per row. +fn record_at(table: u32, stride: u32, row: u32) -> u32 { + return program[table + (row >> PAGE_BITS)] + (row & PAGE_MASK) * stride; +} + +// Where the first kind word of the current mark's `row` lies in the pages. fn row_at(row: u32) -> u32 { - return mark.state + row * mark.stride; + return record_at(mark.state, mark.stride + mark.extras, row); } -// Extra word `word` of `row` of the current mark. -fn extra_at(row: u32, word: u32) -> u32 { - return mark.extra + row * mark.extras + word; +// Where extra word `w` of the current mark's `row` lies in the pages. +fn extra_at(row: u32, w: u32) -> u32 { + return row_at(row) + mark.stride + w; } fn vertex_at(at: u32) -> Vertex { @@ -85,50 +92,66 @@ fn vertex_at(at: u32) -> Vertex { return Vertex( vec2f(float_at(p), float_at(p + 1u)), vec2f(float_at(p + 2u), float_at(p + 3u)), - state[at + VERTEX_COLOR], - state[at + VERTEX_STATUS], + word(at + VERTEX_COLOR), + word(at + VERTEX_STATUS), vec2f(float_at(at + VERTEX_HALF), float_at(at + VERTEX_HALF + 1u)), float_at(at + VERTEX_ELEVATION), float_at(at + VERTEX_DEPTH), - unpack_range(state[at + VERTEX_RANGE]), - state[at + VERTEX_FLAGS], + unpack_range(word(at + VERTEX_RANGE)), + word(at + VERTEX_FLAGS), ); } -// The vertex in graph slot `slot`, from any vertex mark. -fn vertex_slot(slot: u32) -> Vertex { - return vertex_at(slot * VERTEX_WORDS); +// Vertex `held` of end `end`: a held row of the vertex mark that draws the end's table. +fn end_vertex(end: u32, held: u32) -> Vertex { + let e = mark.ends + END_WORDS * end; + return vertex_at(record_at(program[e + END_STATE], program[e + END_STRIDE], held)); +} + +// Whether an edge's ends `a` and `b`, held rows of the vertex marks its two ends attach to, are the +// same vertex. +fn same_vertex(a: u32, b: u32) -> bool { + return a == b && program[mark.ends + END_STATE] == program[mark.ends + END_WORDS + END_STATE]; +} + +// Where the face normal sum for vertex `held` of end `end` lies, or NONE. +fn end_normal(end: u32, held: u32) -> u32 { + let table = program[mark.ends + END_WORDS * end + END_NORMALS]; + if table == NONE { + return NONE; + } + return record_at(table, NORMAL_WORDS, held); } fn edge_at(at: u32) -> Edge { return Edge( - state[at + EDGE_FROM], - state[at + EDGE_TO], - state[at + EDGE_COLOR], + word(at + EDGE_FROM), + word(at + EDGE_TO), + word(at + EDGE_COLOR), float_at(at + EDGE_WIDTH), float_at(at + EDGE_FLOW), float_at(at + EDGE_TRAVEL), - state[at + EDGE_FLAGS], + word(at + EDGE_FLAGS), ); } fn face_at(at: u32) -> Face { - let corners = vec3u(state[at + FACE_A], state[at + FACE_B], state[at + FACE_C]); - return Face(corners, state[at + FACE_COLOR], state[at + FACE_FLAGS]); + let corners = vec3u(word(at + FACE_A), word(at + FACE_B), word(at + FACE_C)); + return Face(corners, word(at + FACE_COLOR), word(at + FACE_FLAGS)); } fn line_at(at: u32) -> Line { return Line( - state[at + LINE_COLOR], + word(at + LINE_COLOR), float_at(at + LINE_WIDTH), float_at(at + LINE_FLOW), float_at(at + LINE_TRAVEL), - state[at + LINE_FLAGS], + word(at + LINE_FLAGS), ); } -// The current mark's words for how far a stroke row's comets traveled and for its flow; NONE for a -// mark that draws no strokes. +// The current mark's word indices for a stroke row's comet travel and its flow. NONE for a mark +// that draws no strokes. fn travel_words() -> vec2u { switch mark.kind { case MARK_EDGES: { return vec2u(EDGE_TRAVEL, EDGE_FLOW); } @@ -138,5 +161,5 @@ fn travel_words() -> vec2u { } fn area_at(at: u32) -> Area { - return Area(state[at + AREA_COLOR], state[at + AREA_FLAGS]); + return Area(word(at + AREA_COLOR), word(at + AREA_FLAGS)); } diff --git a/crates/latkit/shaders/lib/stroke.wesl b/crates/latkit/shaders/lib/stroke.wesl index e201a36b..56bc726c 100644 --- a/crates/latkit/shaders/lib/stroke.wesl +++ b/crates/latkit/shaders/lib/stroke.wesl @@ -2,19 +2,20 @@ import package::lib::bind::frame; import package::lib::mark::coverage; +import package::lib::types::{FLOW_PX, FLOW_UNITS}; struct Stroke { a: vec2f, b: vec2f, - // Distance along the whole line at `a` and at `b`, CSS pixels: dashes and comets run on across - // segments. + // Distance along the whole line at `a` and at `b`, in CSS pixels, so dashes and comets continue + // across segments. phase: vec2f, - // The same in length units, as far along as the line's own points: comets drawn in position - // units run along it. + // The same distances in position units, measured along the line's own points. Comets spaced in + // position units use them. along: vec2f, width: f32, color: vec4f, - // Comet speed, negative backward, 0 for none; and how far the row's comets traveled. + // Comet speed (negative runs backward, 0 for none), and how far the row's comets have traveled. flow: f32, travel: f32, dash: bool, @@ -22,7 +23,7 @@ struct Stroke { arrow: bool, } -// The arrowhead at `b` pointing along `dir`: a triangle `frame.arrow` long, as a signed distance. +// Signed distance to the arrowhead at `b`, a triangle `frame.arrow + width` long along `dir`. fn head_distance(p: vec2f, b: vec2f, dir: vec2f, width: f32) -> f32 { let length = frame.arrow + width; let q = p - b; @@ -35,8 +36,8 @@ fn head_distance(p: vec2f, b: vec2f, dir: vec2f, width: f32) -> f32 { return max(max(flank, along), -along - length); } -// (distance outside the stroke's edge, distance along the line, fraction of the way from `a` to `b`) -// for pixel `p`. +// For pixel `p`: (distance outside the stroke's edge, distance along the line, fraction of the way +// from `a` to `b`). fn stroke_distance(p: vec2f, s: Stroke) -> vec3f { let ab = s.b - s.a; let len = length(ab); @@ -53,7 +54,7 @@ fn stroke_distance(p: vec2f, s: Stroke) -> vec3f { return vec3f(d, mix(s.phase.x, s.phase.y, along), along); } -// How much of the pixel at `d`, from `stroke_distance`, the line of stroke `s` covers, dashed. +// Coverage of the pixel at `d` (from `stroke_distance`) by stroke `s`'s line, with its dashes. fn covered(d: vec3f, s: Stroke) -> f32 { var alpha = coverage(d.x); if s.dash { @@ -62,23 +63,27 @@ fn covered(d: vec3f, s: Stroke) -> f32 { return alpha; } -// How much of the pixel at `d` a comet covers: comets `frame.flow_spacing` apart, moved on by stroke -// `s`'s travel, each a round head `frame.flow_width` wide, never narrower than the stroke, brightest -// at its middle, trailing a tail `frame.flow_length` long that thins and fades. +// Coverage of the pixel at `d` by comets. Comets are `frame.flow_spacing` apart, offset by stroke +// `s`'s travel. Each has a round head `frame.flow_width` wide (at least the stroke's width), +// brightest at its middle, and a tail `frame.flow_length` long that thins and fades. fn comet(d: vec3f, s: Stroke) -> f32 { if s.flow == 0.0 || frame.flow.a <= 0.0 { return 0.0; } - // Where the pixel lies along the line in the comets' lengths, and CSS pixels per length: position - // units where comets are part of the drawing, else the theme's. + // The pixel's distance along the line in the comet unit, and CSS pixels per comet unit: + // position units (`FLOW_UNITS`), CSS pixels (`FLOW_PX`), or else the space's length unit. var along = d.y / frame.unit; var px = frame.unit; - if frame.flow_units != 0u { + if frame.flow_unit == FLOW_UNITS { along = mix(s.along.x, s.along.y, d.z); px = length(s.b - s.a) / max(abs(s.along.y - s.along.x), 1e-12); + } else if frame.flow_unit == FLOW_PX { + along = d.y; + px = 1.0; } let spacing = max(frame.flow_spacing, 1e-6); - // How far the pixel leads the head behind it, the way comets run, and trails the head ahead. + // How far the pixel is ahead of the comet head behind it, in the direction comets run, and how + // far it is behind the head in front of it. let lead = fract(sign(s.flow) * (along - s.travel) / spacing) * spacing; let trail = spacing - lead; let across = d.x + 0.5 * s.width; @@ -89,7 +94,7 @@ fn comet(d: vec3f, s: Stroke) -> f32 { return fade * fade * (1.0 - smoothstep(0.35 * reach, reach, distance)); } -// Stroke `s` at pixel `p`, comets over its line; a comet fades with its row. +// Stroke `s`'s color at pixel `p`, with comets over its line. Comets fade with the row's opacity. fn stroke(p: vec2f, s: Stroke) -> vec4f { let d = stroke_distance(p, s); let line = s.color.a * covered(d, s); diff --git a/crates/latkit/shaders/lib/trace.wesl b/crates/latkit/shaders/lib/trace.wesl index 90b2de8b..2ed94ce3 100644 --- a/crates/latkit/shaders/lib/trace.wesl +++ b/crates/latkit/shaders/lib/trace.wesl @@ -1,42 +1,33 @@ -// Traces: what a line draws of a sampled field across the axis: runs of entries left to right, each -// of one level, frames or buckets. The traced column's table holds a header (how many runs, row -// blocks, and levels); per run its level's place among the levels, its first entry, how many, and the -// point it starts at; per level the level, how many axis pages it has, where they are listed, and -// where its row blocks' lists start; per level and row block how many pages of frames or least values -// it has and where they are listed, then of greatest values; then those lists: axis pages as (offset, -// first entry), pages of values as (offset, first entry, words per entry). Offsets within the table -// are relative to it. A line runs through each frame, and through each bucket's least and greatest -// value where it draws; between runs that do not meet, a point is left out, breaking the line. +// Traces: a line drawn through a sampled field across the axis, as runs of entries from left to +// right. Each run reads one level: frames, or buckets of frames. The line passes through each +// frame, and through each bucket's least and greatest values. Between runs that do not meet, a +// point is left out, which breaks the line. +// +// The traced column's table, with positions in it relative to its start: +// - Header (`HEADER` words): the run count, the row page count, the level count, and the total +// point count. +// - Per run (`TRACE_RUN_WORDS`): its level's index, its first entry, its entry count, and its first +// point. +// - Per level (`LEVEL_WORDS`): the level, its axis page count, where its axis pages are listed, and +// where its row pages' lists start. +// - Per level and row page, four words: the count and list position of its pages of frames (or of +// least values), then the same for its pages of greatest values. +// - The lists: axis pages as (offset in `pages`, first entry), and value pages as (offset in +// `pages`, first entry, words per entry). -import package::lib::types::{NONE, MISSING, COLUMN_TABLE, PAGE_BITS, PAGE_MASK}; -import package::lib::bind::{pages, program, mark}; -import package::lib::read::column_word; +import package::lib::types::{NONE, MISSING, COLUMN_TABLE}; +import package::lib::bind::{program, mark, word}; +import package::lib::read::{column_word, page_of, last_at_most}; const HEADER: u32 = 4u; -const RUN_WORDS: u32 = 4u; +const TRACE_RUN_WORDS: u32 = 4u; const LEVEL_WORDS: u32 = 4u; fn trace_table() -> u32 { return column_word(mark.trace, COLUMN_TABLE); } -// The last of `count` items of `stride` words from `at` whose word `key` is at most `value`. -fn last_at_most(at: u32, count: u32, stride: u32, key: u32, value: u32) -> u32 { - var lo = 0u; - var hi = count; - while hi - lo > 1u { - let mid = (lo + hi) / 2u; - if program[at + mid * stride + key] <= value { - lo = mid; - } else { - hi = mid; - } - } - return at + lo * stride; -} - -// A run of the trace: its level's place among the levels, its first entry, how many, and the point -// it starts at. +// A run of the trace: its level's index, its first entry, its entry count, and its first point. struct TraceRun { slot: u32, first: u32, @@ -47,17 +38,17 @@ struct TraceRun { // The run point `k` lies in: the last starting at or before it. fn trace_run(k: u32) -> TraceRun { let t = trace_table(); - let r = last_at_most(t + HEADER, max(program[t], 1u), RUN_WORDS, 3u, k); + let r = last_at_most(t + HEADER, max(program[t], 1u), TRACE_RUN_WORDS, 3u, k); return TraceRun(program[r], program[r + 1u], program[r + 2u], program[r + 3u]); } // Where level `slot`'s words start. fn trace_level_at(slot: u32) -> u32 { let t = trace_table(); - return t + HEADER + RUN_WORDS * program[t] + LEVEL_WORDS * slot; + return t + HEADER + TRACE_RUN_WORDS * program[t] + LEVEL_WORDS * slot; } -// The level a run of `slot` draws. +// The level at level index `slot`. fn trace_level(slot: u32) -> u32 { return program[trace_level_at(slot)]; } @@ -68,14 +59,15 @@ fn trace_axis(slot: u32, entry: u32) -> vec2f { let l = trace_level_at(slot); let e = last_at_most(trace_table() + program[l + 2u], max(program[l + 1u], 1u), 2u, 1u, entry); let w = program[e] + (entry - program[e + 1u]) * 2u; - return vec2f(bitcast(pages[w]), bitcast(pages[w + 1u])); + return vec2f(bitcast(word(w)), bitcast(word(w + 1u))); } // The bits of `row`'s value of entry `entry` of level `slot`: a frame's, or a bucket's least // (`side` 0) or greatest (1). fn trace_entry(row: u32, slot: u32, entry: u32, side: u32) -> u32 { let t = trace_table(); - let block = row >> PAGE_BITS; + let at = page_of(row); + let block = at.x; if block >= program[t + 1u] { return MISSING; } @@ -85,10 +77,10 @@ fn trace_entry(row: u32, slot: u32, entry: u32, side: u32) -> u32 { return MISSING; } let e = last_at_most(t + program[lists + 1u], count, 3u, 1u, entry); - let local = row & PAGE_MASK; + let local = at.y; let per = program[e + 2u]; if program[e] == NONE || local >= per || entry < program[e + 1u] { return MISSING; } - return pages[program[e] + (entry - program[e + 1u]) * per + local]; + return word(program[e] + (entry - program[e + 1u]) * per + local); } diff --git a/crates/latkit/shaders/lib/triangles.wesl b/crates/latkit/shaders/lib/triangles.wesl index 5d6c474c..ef7a8e99 100644 --- a/crates/latkit/shaders/lib/triangles.wesl +++ b/crates/latkit/shaders/lib/triangles.wesl @@ -1,30 +1,32 @@ -// Triangles: a face row's corners, at the vertices its references name; or an area's triangles, three -// points at a time from its geometry column after the point its label hangs from, numbered by the -// mark's list. Draws, probes, bounds, and labels read a triangle through `triangle_corner`; which kind -// a pipeline reads is its `PATH`: ends for faces, geometry for areas. +// Triangles: a face row's corners at the vertices its references name, or an area's triangles. An +// area's geometry column holds its label anchor, then three points per triangle, and the mark's +// list numbers the triangles. Draws, probes, bounds, and labels read triangles through +// `triangle_corner`. A pipeline's `PATH` selects the kind: `PATH_ENDS` for faces, `PATH_GEOMETRY` +// for areas. import package::lib::types::{NONE, VISIBLE, PLACED, ENDS_COLOR, PATH_GEOMETRY}; import package::lib::bind::mark; -import package::lib::state::{row_at, face_at, area_at, vertex_slot}; +import package::lib::state::{row_at, face_at, area_at, end_vertex}; import package::lib::project::{world, clip_of, shows}; import package::lib::focus::faded; import package::lib::read::{points_of, point_bits, missing_bits}; import package::lib::path::{PATH, Segment, segment, stroke_instances}; +import package::lib::drawn::drawn_row; struct Corner { world: vec3f, - // Where it stands, in position units. + // Its position, in position units. position: vec3f, - // A face corner's vertex slot; `NONE` for an area's. - slot: u32, + // A face corner's vertex: a held row of the vertex mark its end attaches to. NONE for an area. + vertex: u32, color: vec4f, row: u32, - // Whether it has a place to stand, and whether it lies in front of the globe's horizon. + // Whether it has a position, and whether it lies in front of the globe's horizon. shown: bool, ahead: bool, } -// Triangles every live row draws: a face per row, or each of every area's, as its list numbers them. +// Triangles across all drawn rows: one per face, or every area triangle the list numbers. fn triangle_instances() -> u32 { if PATH == PATH_GEOMETRY { return stroke_instances(); @@ -32,16 +34,22 @@ fn triangle_instances() -> u32 { return mark.rows; } -// Corner `corner` of triangle `instance`. +// Corner `corner` of triangle `instance`, faded while its tile fades in or out. fn triangle_corner(instance: u32, corner: u32) -> Corner { if PATH == PATH_GEOMETRY { - return area_point(segment(instance), 1u + corner); + let s = segment(instance); + var c = area_point(s, 1u + corner); + c.color.a *= s.presence; + return c; } - return face_corner(instance, corner); + let d = drawn_row(instance); + var c = face_corner(d.row, corner); + c.color.a *= d.presence; + return c; } -// Whether triangle `instance` draws: every corner placed, and one in front of the globe's horizon; -// the globe's depth cuts a triangle reaching past its edge. +// Whether triangle `instance` draws: every corner has a position, and at least one lies in front of +// the globe's horizon. The globe's depth clips a triangle reaching past its edge. fn triangle_shown(instance: u32) -> bool { var ahead = false; for (var k = 0u; k < 3u; k++) { @@ -54,10 +62,10 @@ fn triangle_shown(instance: u32) -> bool { return ahead; } -// The row triangle `instance` belongs to. -fn triangle_row(instance: u32) -> u32 { +// The drawn row triangle `instance` belongs to. +fn triangle_item(instance: u32) -> u32 { if PATH == PATH_GEOMETRY { - return segment(instance).row; + return segment(instance).item; } return instance; } @@ -70,8 +78,8 @@ fn face_corner(row: u32, corner: u32) -> Corner { c.shown = false; return c; } - c.slot = f.corners[corner]; - let v = vertex_slot(c.slot); + c.vertex = f.corners[corner]; + let v = end_vertex(corner, c.vertex); c.world = world(v.hi, v.lo, v.elevation); c.position = vec3f(v.hi + v.lo, v.elevation); let color = select(unpack4x8unorm(f.color), unpack4x8unorm(v.color), (mark.flags & ENDS_COLOR) != 0u); @@ -81,12 +89,12 @@ fn face_corner(row: u32, corner: u32) -> Corner { return c; } -// Point `offset` of the points of triangle `t.index` of area row `t.row`: 0 is where its label hangs, -// 1 to 3 its corners. +// Point `offset` of triangle `t.index` of area row `t.row`. Offset 0 is the area's label anchor. +// Offsets 1 to 3 are the triangle's corners. fn area_point(t: Segment, offset: u32) -> Corner { var c: Corner; c.row = t.row; - c.slot = NONE; + c.vertex = NONE; let a = area_at(row_at(t.row)); let points = points_of(mark.geometry, t.row); let k = select(3u * t.index + offset, 0u, offset == 0u); @@ -109,9 +117,9 @@ fn area_point(t: Segment, offset: u32) -> Corner { return c; } -// Where area row `row`'s label hangs: inside its largest polygon, farthest from its edges. +// Area row `row`'s label anchor: inside its largest polygon, at the point farthest from its edges. fn area_anchor(row: u32) -> Corner { - return area_point(Segment(row, 0u, 0.0), 0u); + return area_point(Segment(row, NONE, 0u, 0.0, 1.0, 1.0, false), 0u); } fn corner_clip(c: Corner) -> vec4f { diff --git a/crates/latkit/shaders/lib/types.wesl b/crates/latkit/shaders/lib/types.wesl index ad8ea89f..96693f68 100644 --- a/crates/latkit/shaders/lib/types.wesl +++ b/crates/latkit/shaders/lib/types.wesl @@ -1,13 +1,14 @@ -// The GPU contract shared with Rust: build.rs mirrors every struct and constant here. +// Constants and structs shared with Rust. The build script mirrors each one in Rust. -// No index, slot, or page. +// Sentinel for no index, slot, or page. const NONE: u32 = 0xffffffffu; -// A missing number: NaN bits. Any NaN or infinity reads as missing. +// A missing number, as NaN bits. Any NaN or infinity reads as missing. const MISSING: u32 = 0x7fc00000u; -// A missing color: transparent magenta, replaced by the theme's color on store. +// A missing color (transparent magenta), as an option given `missing()` holds. A color store +// replaces it with the option's theme color. const MISSING_COLOR: u32 = 0x00ff00ffu; -// Opcodes of the expression machine; an op is four words: code, a, b, c. +// Expression machine opcodes. Each instruction is four words: opcode, a, b, c. const OP_CONST: u32 = 0u; const OP_MISSING: u32 = 1u; const OP_FIELD: u32 = 2u; @@ -44,7 +45,7 @@ const OP_STORE_COLOR: u32 = 32u; const OP_STORE_FLAG: u32 = 33u; const OP_STORE_POSITION: u32 = 34u; const OP_POSITION: u32 = 35u; -const OP_STORE_END: u32 = 36u; +const OP_STORE_CODE: u32 = 36u; const OP_REDUCE: u32 = 37u; const OP_FLOOR: u32 = 38u; const OP_MOD: u32 = 39u; @@ -52,65 +53,116 @@ const OP_POW: u32 = 40u; const OP_LOG: u32 = 41u; const OP_EXP: u32 = 42u; const OP_ZOOM: u32 = 43u; -const OP_STORE_CODE: u32 = 44u; -// Values the expression stack holds at once. +// Depth of the expression machine's value stack. const STACK: u32 = 16u; -// Column descriptor words, eight per column in `meta`. -const COLUMN_WORDS: u32 = 8u; +// Column descriptors, `COLUMN_WORDS` words per column, start at `mark.columns` in the program. +const COLUMN_WORDS: u32 = 6u; const COLUMN_FLAGS: u32 = 0u; const COLUMN_ROWS: u32 = 1u; +// Where the column's page table starts in the program. A sampled column has cursors here instead. const COLUMN_TABLE: u32 = 2u; +// Where the page table of an f64 column's low words starts. NONE for an f32 column. const COLUMN_LOW: u32 = 3u; +// The bits column mapping each row to a row of another table. NONE when it reads its own table. const COLUMN_JOIN: u32 = 4u; +// A sampled column's f32 interpolation fraction from the frame at the shown coordinate to the next. const COLUMN_T: u32 = 5u; -// Column flags. +// Column flags. SAMPLED: values are samples over the axis. STEP: a sampled column holds each +// frame's value instead of interpolating. BITS: values are raw u32 words, such as row indices. const SAMPLED: u32 = 1u; const STEP: u32 = 2u; const BITS: u32 = 4u; -// Rows per data page, as a shift. -const PAGE_BITS: u32 = 16u; -const PAGE_MASK: u32 = 65535u; -// Rows a block holds: a page of every column. A dispatch over more items than one dimension holds -// steps a block at a time in its second, 256 workgroups of 256 wide. -const BLOCK_ROWS: u32 = 65536u; - -// The item invocation `id` of a dispatch over items covers: blocks of BLOCK_ROWS along the second -// dimension, when the first holds too few. +// Rows per page, also as a shift and a mask. Data columns, a paged table's slots, and each mark's +// rows are stored in pages, and marks evaluate their rows page by page. +const PAGE_BITS: u32 = 12u; +const PAGE_MASK: u32 = 4095u; +const PAGE_ROWS: u32 = 4096u; +// Items per step of a dispatch's second dimension: 256 workgroups of 256. Dispatches too large for +// one dimension use it (see `item`). +const DISPATCH_ROWS: u32 = 65536u; +// Flag on a listed row block whose state words start over. Evaluation zeroes those rows' words +// before any channel runs. +const FRESH: u32 = 0x80000000u; + +// Words per run in a mark's run list at `mark.row_runs`, after the run count: the run's first drawn +// row, the held row it starts at, and its tile's fade flags (FADE_IN, FADE_OUT). +const RUN_WORDS: u32 = 3u; +// A drawn row's word, written by `expand_runs`: its held row in the low 30 bits, and flags for a +// tile fading in or out. +const DRAWN_ROW: u32 = 0x3fffffffu; +const DRAWN_ENTERING: u32 = 0x40000000u; +const DRAWN_LEAVING: u32 = 0x80000000u; + +// The item index of invocation `id`. A dispatch too large for one dimension adds `DISPATCH_ROWS` +// items per step along the second. fn item(id: vec3u) -> u32 { - return id.y * BLOCK_ROWS + id.x; + return id.y * DISPATCH_ROWS + id.x; } -// How a word of a row eases while its mark does: kept as it is (no channel stores it), shown as -// evaluated at once, a number or a color mixed, or a position's high word moved by its offset. +// How each word of a row eases during a transition. KEEP: no channel stores it, so it is left +// unchanged. SNAP: shows the evaluated value at once. NUMBER, COLOR: mixed from the old value. +// POSITION: a position's high word, moved by its offset from the old position. const BLEND_KEEP: u32 = 0u; const BLEND_SNAP: u32 = 1u; const BLEND_NUMBER: u32 = 2u; const BLEND_COLOR: u32 = 3u; const BLEND_POSITION: u32 = 4u; -// Points per block of a geometry's points, as a shift: a block of x and y words is a chunk. -const POINT_BITS: u32 = 15u; -const POINT_MASK: u32 = 32767u; - -// Vertex state words. +// Points per piece of a geometry column, as a shift and a mask. Each point is an x and a y word. A +// piece holds the points of a group of a page's rows, so rewriting a row rewrites only its group's +// piece. A row with more points than a piece spans consecutive pieces. +const POINT_BITS: u32 = 12u; +const POINT_MASK: u32 = 4095u; +// Words per page in a geometry column's table: +// 0: the block of the page's row table, two words per row. The first word is +// `piece << POINT_BITS | first`, where `piece` indexes the page's pieces and `first` is the +// row's first point in that piece. The second word is the row's point count. +// 1: where the page's piece list starts, relative to the column's table. +// 2: the page's point count. +// 3: the page's row count. +const PATH_PAGE_WORDS: u32 = 4u; + +// Words per end of an edge or face mark, at `mark.ends` in the program. An end's vertices are rows +// of the vertex mark that draws the end's table. +// STATE, STRIDE: that vertex mark's state page table (NONE when no vertex mark draws the table) and +// its words per row. +// SLOTS: the end table, which maps a place to a held row (see `held_of`). NONE unless the table is +// paged. +// NORMALS: the page table of the face normals summed at its rows. NONE when no face sums them. +// COLUMN: the program column of the reference field naming the end's rows; NONE when the end is the +// row's own vertex, as for a pole. +// The end pass writes the held row of end `k` into the row's kind word `k`. +const END_WORDS: u32 = 5u; +const END_STATE: u32 = 0u; +const END_STRIDE: u32 = 1u; +const END_SLOTS: u32 = 2u; +const END_NORMALS: u32 = 3u; +const END_COLUMN: u32 = 4u; + +// Words of the face normal sum at a vertex row: x, y, and z in fixed point. +const NORMAL_WORDS: u32 = 3u; + +// Vertex state words. POSITION is four words: the high parts of x and y, then their low parts. HALF +// is two words: the marker's half width and half height. const VERTEX_WORDS: u32 = 12u; const VERTEX_POSITION: u32 = 0u; const VERTEX_COLOR: u32 = 4u; const VERTEX_STATUS: u32 = 5u; const VERTEX_HALF: u32 = 6u; const VERTEX_ELEVATION: u32 = 8u; -// How many groups hold a group row: containers tint by it. +// How many groups contain a group row. Containers tint by it. const VERTEX_DEPTH: u32 = 9u; -// The CSS pixels its marker's half keeps within at any zoom, as two halves: the mark's, so every -// mark reading the vertex trims at the outline it draws. +// The range of CSS pixels the marker's half size is clamped to, packed by `pack_range`. It is the +// vertex mark's range, so edges ending at the vertex stop at the outline it draws. const VERTEX_RANGE: u32 = 10u; const VERTEX_FLAGS: u32 = 11u; -// Edge state words. Every kind's flags are its last word. A stroke's travel is how far its comets -// moved, which no channel writes. +// Edge state words. Every kind's flags word is its last word. FROM and TO are the held rows of the +// edge's two vertices, written by the end pass. TRAVEL is how far its comets have moved. No channel +// writes FROM, TO, or TRAVEL. const EDGE_WORDS: u32 = 7u; const EDGE_FROM: u32 = 0u; const EDGE_TO: u32 = 1u; @@ -120,9 +172,10 @@ const EDGE_FLOW: u32 = 4u; const EDGE_TRAVEL: u32 = 5u; const EDGE_FLAGS: u32 = 6u; -// A vertex mark's ports, beside its program: each port's code (its side in two bits, its place -// among the ports on that side and how many share it in eight each, and whether it is an output), -// its shape, the program column of its reference field, and its label's rectangle in the atlas. +// Words per port of a vertex mark, at `mark.port_info` in the program: its code, its shape, the +// program column of its reference field, and its label's atlas rectangle as two corner words. A +// code holds the side in bits 0-1, the port's index on that side in bits 2-9, the number of ports +// on that side in bits 10-17, and the `PORT_OUTPUT` bit. const PORT_WORDS: u32 = 5u; const PORT_CODE: u32 = 0u; const PORT_SHAPE: u32 = 1u; @@ -133,16 +186,19 @@ const SIDE_LEFT: u32 = 0u; const SIDE_RIGHT: u32 = 1u; const SIDE_TOP: u32 = 2u; const SIDE_BOTTOM: u32 = 3u; -// A group's header, in label sizes: its title sits in it. +// Height of a group's header band, in label sizes. The group's title sits in it. const HEADER_BAND: f32 = 1.8; -// Angle a geodesic segment spans at most, in degrees. +// Largest arc a geodesic segment spans, in degrees. const GEODESIC_STEP: f32 = 2.0; -// Pieces a path between two ends takes at most: along a great circle, or bowed into a lane. +// Most segments a path between two ends uses, along a great circle or bowed into a lane. const MAX_PIECES: u32 = 64u; -// Pieces a path bowed into a lane takes at least. +// Fewest segments a path bowed into a lane uses. const BOW_PIECES: u32 = 16u; +// Widest spread, in lanes, of a family of edges joining the same two vertices. Larger families pack +// their lanes closer. +const FAMILY_LANES: f32 = 8.0; -// Face state words. +// Face state words. A, B, and C are the held rows of the corner vertices, written by the end pass. const FACE_WORDS: u32 = 5u; const FACE_A: u32 = 0u; const FACE_B: u32 = 1u; @@ -163,35 +219,18 @@ const AREA_WORDS: u32 = 2u; const AREA_COLOR: u32 = 0u; const AREA_FLAGS: u32 = 1u; -// Label words, among a labeled mark's extra words: the text's code, its priority as a level from 0 -// to 255, and its color. -const LABEL_WORDS: u32 = 3u; +// Label words among a labeled mark's extra words: text code (missing for no label) and color. +const LABEL_WORDS: u32 = 2u; const LABEL_CODE: u32 = 0u; -const LABEL_PRIORITY: u32 = 1u; -const LABEL_COLOR: u32 = 2u; - -// A label's words in the labels buffer, per labeled row: the box placement measured it in, CSS -// pixels; its key; its state; and how present it is. Draws place a label from its anchor every -// frame; placement only decides whether it shows. -const CANDIDATE_WORDS: u32 = 7u; -const CANDIDATE_BOX: u32 = 0u; -const CANDIDATE_KEY: u32 = 4u; -const CANDIDATE_STATE: u32 = 5u; -const CANDIDATE_ALPHA: u32 = 6u; -// Text is set once per run at this size, in atlas texels, as a distance field reaching this far -// past the text's edge. +const LABEL_COLOR: u32 = 1u; + +// Each text run is set once, at a size of `TEXT_BASE` atlas texels, as a distance field reaching +// `TEXT_SPREAD` texels past the text's edge. const TEXT_BASE: f32 = 32.0; const TEXT_SPREAD: f32 = 8.0; -// Candidate states: in view and wanting room, placed, dropped, placed in the frame before, and -// measured by an estimate while its text is not set. -const WANTS: u32 = 1u; -const PLACED_LABEL: u32 = 2u; -const DROPPED: u32 = 4u; -const STICKY: u32 = 8u; -const ESTIMATED: u32 = 16u; -// A text run whose distance field the GPU makes, as words at the start of the fields buffer: where its -// coverage starts, its coverage's width and height, where its columns' least distances start, and -// where its field starts and the words between the starts of its rows. +// Words per text run at the start of the fields buffer, for runs whose distance field the GPU +// computes: where its coverage starts, the coverage's width and height, where its column minima +// start, where its field starts, and the field's row pitch in words. const FIELD_COVERAGE: u32 = 0u; const FIELD_WIDTH: u32 = 1u; const FIELD_HEIGHT: u32 = 2u; @@ -200,28 +239,27 @@ const FIELD_AT: u32 = 4u; const FIELD_ROW: u32 = 5u; const FIELD_WORDS: u32 = 6u; -// A run table entry whose text is not set yet holds this flag as its first corner and the run's -// estimated size as its second, so its label can be placed before it is set. -const TEXT_ESTIMATE: u32 = 0x80000000u; -// A cell a placed label holds. -const OCCUPIED: u32 = 0u; -// CSS pixels a cell of the label grid covers. -const CELL: f32 = 8.0; -// Codes a mark asks the host to set at most per frame. +// Most text codes a mark asks the host to set per texts pass. const WANTED: u32 = 64u; - -// State flags. +// Labels claim the screen cells their text covers, `LABEL_CELL` CSS pixels square, over the view +// and `LABEL_MARGIN` of its size beyond each side. Until its text is set, a label claims space for +// text `LABEL_GUESS` times its size wide. +const LABEL_CELL: f32 = 8.0; +const LABEL_MARGIN: f32 = 0.25; +const LABEL_GUESS: f32 = 4.0; + +// Row state flags, in the last word of each row's kind words. const VISIBLE: u32 = 1u; -// The row has a position: from its data, or in a diagram from its sheet. +// The row has a position, from its data or, in a diagram, from its sheet. const PLACED: u32 = 2u; const DASH: u32 = 4u; -// While a mark eases, a row that appears fades in and one that disappears or left fades out. +// During a transition, a row that appears fades in, and one that disappears or left fades out. const FADE_IN: u32 = 8u; const FADE_OUT: u32 = 16u; -// A row that entered with this change: it has no state to ease from. +// The row entered with this change, so it has no old state to ease from. const ENTERING: u32 = 32u; -// A vertex whose size is in position units, and one with its own height: what any reader of its -// slot needs to know its size in pixels. +// The vertex's size is in position units, or it has its own height. Any mark reading the vertex +// needs these to compute its size in pixels. const HALF_UNITS: u32 = 64u; const OWN_HEIGHT: u32 = 128u; @@ -237,13 +275,13 @@ const POINTS: u32 = 0u; const STROKES: u32 = 1u; const TRIANGLES: u32 = 2u; -// What a stroke's segments run between: two vertices (straight, along a great circle, or through -// bends), samples across the axis, or a geometry column's points. A triangle's corners come from -// three vertices, or from a geometry column three points at a time. +// Path kinds, the source of a stroke's points. ENDS: two vertices, joined straight, along a great +// circle, or through bends. TRACE: samples across the axis. GEOMETRY: a geometry column's points. +// Faces use ENDS for their three vertices. Areas use GEOMETRY, three points per triangle. const PATH_ENDS: u32 = 0u; const PATH_TRACE: u32 = 1u; const PATH_GEOMETRY: u32 = 2u; -// A pole: from each row's own vertex down to the ground under it. +// A pole, from each row's own vertex down to the ground below it. const PATH_POLES: u32 = 3u; // Mark flags. @@ -255,24 +293,26 @@ const STATUS: u32 = 16u; const LABELED: u32 = 32u; const STEP_BEFORE: u32 = 64u; const STEP_AFTER: u32 = 128u; -const CASTS: u32 = 256u; -// A vertex mark whose rows hold rows: drawn as containers. +// A vertex mark whose blocks and groups drop shadows onto a diagram's sheet. +const SHADOW: u32 = 256u; +// A vertex mark whose rows contain other rows. Its rows draw as containers. const GROUP: u32 = 512u; -// Focus bits per row: selected, hovered, and an end of a selected or hovered edge, which the focus -// draw lights over the rest; FIT marks the rows a fit measures. +// Focus bits per row. SELECTED, HOVERED, and END (an end of a selected or hovered edge) make a row +// lit, and the focus draw redraws lit rows over everything. FIT marks the rows a fit measures. const SELECTED: u32 = 1u; const HOVERED: u32 = 2u; const FIT: u32 = 4u; const END: u32 = 8u; const LIT: u32 = 11u; -// Ordered words per mark in the extents buffer: least x, y, z as complements, then greatest. +// Words per mark in the extents buffer, as order-preserving bits (`ordered`): the complements of +// the least x, y, and z, then the greatest x, y, and z. const EXTENT_WORDS: u32 = 6u; -// Probe words: the least key (distance, then the mark drawn last), the instance plus one, where the -// hit lies (its depth and screen point), the nearest depth drawn under the probe, and the instance's -// row and segment. +// Probe buffer words. KEY: the least hit key (distance, then the mark drawn last). INSTANCE: the +// hit instance plus one. DEPTH, X, Y: where the hit lies. UNDER: the nearest depth drawn around the +// probe. ROW, SEGMENT: the hit's held row and part. const PROBE_KEY: u32 = 0u; const PROBE_INSTANCE: u32 = 1u; const PROBE_DEPTH: u32 = 2u; @@ -281,14 +321,15 @@ const PROBE_Y: u32 = 4u; const PROBE_UNDER: u32 = 5u; const PROBE_ROW: u32 = 6u; const PROBE_SEGMENT: u32 = 7u; -// Where a located row lies on screen, and whether it shows. What queries find follows the probe's words. +// Where a located row lies on screen, and whether it shows. Query results follow the probe's +// `PROBE_WORDS` words. const PROBE_LOCATE_X: u32 = 8u; const PROBE_LOCATE_Y: u32 = 9u; const PROBE_LOCATED: u32 = 10u; const PROBE_WORDS: u32 = 12u; -// The steps of the probe program, as its step uniform names them: the hover's three, then the -// queries', then the depth under the probe. +// Probe program steps, chosen by its step uniform: the hover's three, the queries' three, then the +// depth under the probe. const STEP_NEAREST: u32 = 0u; const STEP_TOPMOST: u32 = 1u; const STEP_DESCRIBE: u32 = 2u; @@ -298,12 +339,12 @@ const STEP_LOCATE: u32 = 5u; const STEP_UNDER: u32 = 6u; const STEPS: u32 = 7u; -// What a dispatch of the expression machine's `evaluate` runs, as its step uniform names it: a -// mark's channels, or the reductions fitting its domains. +// Steps of the expression machine's `evaluate`, chosen by its step uniform: run a mark's channels, +// run the reductions that fit its domains, or resolve its rows' ends to held rows. const STEP_CHANNELS: u32 = 0u; const STEP_REDUCE: u32 = 1u; -// Words of a hit a query finds: its row, distance in sixty-fourths of a pixel, part, screen x and y, -// and depth. +const STEP_ENDS: u32 = 2u; +// Words per query hit: held row, distance in 1/64 CSS pixel, part, screen x and y, and depth. const HIT_WORDS: u32 = 6u; // Overlay sprites. @@ -345,66 +386,58 @@ const THEME_LINE: u32 = 4u; const THEME_LINE_WIDTH: u32 = 5u; const THEME_AREA: u32 = 6u; const THEME_TEXT: u32 = 7u; +// The color a colormap gives a missing value. const THEME_MISSING: u32 = 8u; -// Nothing: a fully transparent color, what a missing status draws. +// A fully transparent color, which a missing status draws. const THEME_CLEAR: u32 = 9u; -// What a group's container is filled with. -const THEME_SURFACE: u32 = 10u; -// A diagram's look, in units: the border of its blocks and groups (clear for none), their corners' -// radius, its ports' width, and how far a block's shadow falls below it. +// A group container's fill color. +const THEME_GROUP: u32 = 10u; +// Diagram style. OUTLINE: the border color of blocks and groups, clear for none. CORNER, PORT_SIZE, +// LIFT: the corner radius, port width, and how far a block's shadow falls, in length units. const THEME_OUTLINE: u32 = 11u; const THEME_CORNER: u32 = 12u; const THEME_PORT_SIZE: u32 = 13u; const THEME_LIFT: u32 = 14u; const THEME_SLOTS: u32 = 16u; +// The unit of comet spacing and length: the space's length unit, position units, or CSS pixels. +const FLOW_LENGTHS: u32 = 0u; +const FLOW_UNITS: u32 = 1u; +const FLOW_PX: u32 = 2u; + // Backdrop grids. const GRID_OFF: u32 = 0u; const GRID_LINES: u32 = 1u; const GRID_DOTS: u32 = 2u; -// A list's header: its item count; where its counts (scanned in place to where each item's entries -// start, then the total), block sums, and entries start; the indirect arguments a draw and a -// dispatch over its entries read; and how many entries it keeps. +// List header words. ITEMS: the item count. COUNTS, SUMS, ENTRIES: where the per-item counts, the +// block sums, and the entries start. The scan turns the counts in place into each item's first +// entry, followed by the total. const LIST_ITEMS: u32 = 0u; const LIST_COUNTS: u32 = 1u; const LIST_SUMS: u32 = 2u; const LIST_ENTRIES: u32 = 3u; -// Draw arguments: vertices, instances (the total), first vertex, first instance. +// Indirect draw arguments: vertices, instances (the kept entries), first vertex, first instance. const LIST_DRAW: u32 = 4u; -// Dispatch arguments: workgroups over the total, 1, 1. +// Indirect dispatch arguments: workgroups over the kept entries, 1, 1. const LIST_DISPATCH: u32 = 8u; -// Entries the list keeps: past them, entries are counted but not written. +// How many entries the list keeps. Entries past it are counted but not written. const LIST_KEPT: u32 = 11u; const LIST_HEADER: u32 = 12u; -// Counts a workgroup of the scan takes. +// Counts per workgroup of the scan. const SCAN_WORKGROUP: u32 = 256u; -// How shadows fall: the light, how they darken and soften, and the ground the map covers. -struct Shade { - color: vec4f, - // The direction toward the light, in world units, and how much shadows spread per unit of height. - light: vec4f, - // The ground the map covers: least x and y, greatest x and y, in world units. - rect: vec4f, - // World units flat marks cast from, world units per texel of the map, its levels, and whether - // anything casts. - lift: f32, - texel: f32, - levels: f32, - on: u32, -} - -// What every draw and dispatch of a frame reads. +// Per-frame values every draw and dispatch reads. struct Frame { center_hi: vec2f, center_lo: vec2f, pointer: vec2f, - // Where a probe looks, CSS pixels, and the box a query covers. + // The probe's point in CSS pixels. `query` holds the running query's box (`boxed`), point and + // radius (`hits`), or drawn row (`locate`). probe: vec2f, query: vec4f, - // The direction toward the key light faces are lit by, which turns with the camera. + // Direction toward the key light that shades faces. It turns with the camera. light: vec4f, space: u32, globe: f32, @@ -414,39 +447,33 @@ struct Frame { selecting: u32, // 1 while a fit measures only the rows marked FIT. fitting: u32, - // 1 when the target stores linear color (an sRGB format): outputs convert from sRGB. + // 1 when the target stores linear color (an sRGB format), so outputs convert from sRGB. linear: u32, glow: f32, dim: f32, dash: f32, - // Comets: the distance between them and how far a tail trails its head, in the theme's lengths - // or with `flow_units` in position units; how wide a head is, in CSS pixels; and the seconds + // Comets: spacing and tail length in `flow_unit`, head width in CSS pixels, and the seconds // they travel this frame. flow_spacing: f32, flow_length: f32, flow_width: f32, - flow_units: u32, + flow_unit: u32, flow_step: f32, - // How far a lane bows, in its edge's lengths at its middle. - lane_bow: f32, - hover_scale: f32, + // How far a lane bows at its middle, as a fraction of its edge's length. + bow: f32, + // Scale factor for a hovered vertex sized in CSS pixels. + grow: f32, shading: f32, arrow: f32, halo: f32, - // CSS pixels between a group's members and its outline: its title's inset. + // CSS pixels between a group's members and its outline. It is also the title's inset. group_padding: f32, - // CSS pixels per length unit: per position unit in a diagram, where every length is a unit; 1 - // elsewhere. Lengths in the frame above are already in pixels. + // CSS pixels per length unit: per position unit in a diagram, 1 elsewhere. The theme lengths + // above are already in CSS pixels, except comet spacing and length. unit: f32, zoom: f32, - // The label grid: where its cells start in the labels buffer, and its columns and rows; and how - // far labels fade per frame. - cells: u32, - cell_columns: u32, - cell_rows: u32, - label_fade: f32, - // A plane's grid: its kind, pitch in position units, and the world distance from the center it - // fades out by, 0 for none; and whether the earth draws its graticule. + // A plane's grid: kind, pitch in position units, and the world distance from the center at + // which it has faded out (0 for no fade). `graticule` is 1 if the earth draws a graticule. grid_kind: u32, grid_pitch: f32, grid_fade: f32, @@ -455,30 +482,30 @@ struct Frame { selected: vec4f, flow: vec4f, background: vec4f, - surface: vec4f, + // The earth's ground color. + ground: vec4f, grid_color: vec4f, - // The direction toward the sun in the earth's frame (x toward 0° longitude on the equator, z - // toward the north pole), and how bright night stays; no direction where daylight is off. + // Direction toward the sun in earth-fixed coordinates (x toward 0° longitude on the equator, z + // toward the north pole), with night's brightness in w. All zero when daylight is off. sun: vec4f, - // The air glowing around the globe; clear where the earth has none. + // Color of the atmosphere around the globe. Clear when the earth has none. atmosphere: vec4f, - shade: Shade, + // Shadow color of a diagram's blocks and groups. + shadow: vec4f, theme: array, } -// What one pass draws through: the view, or the light casting shadows. +// The view's camera, which every pass projects through. struct Eye { clip_from_world: mat4x4f, world_from_clip: mat4x4f, // The camera's position in world units. position: vec4f, - // The target's size in CSS pixels; the light's map counts CSS pixels of the view it serves. + // The target's size in CSS pixels. size: vec2f, px_per_unit: vec2f, // Target pixels per CSS pixel. pixel_ratio: f32, - // World units casters are raised by: the shadow's lift in the light's eye, 0 elsewhere. - lift: f32, } // One mark's parameters, at a 256-byte aligned dynamic offset. @@ -486,24 +513,38 @@ struct Mark { index: u32, kind: u32, primitive: u32, - // Rows drawn and evaluated, and rows past them that left and fade out. + // Drawn rows, and the rows after them that left and are fading out. rows: u32, exits: u32, - // Each row's kind words: where row 0's start and how many; then its extra words. + // Held rows: the rows the mark keeps evaluated on the GPU, leaving rows included. For a table + // drawn whole, these are its rows. For a paged table, they are the rows of its slots. Every + // per-row word belongs to a held row, in pages of `PAGE_ROWS` held rows listed by page tables + // in the program. + held: u32, + // Where the program lists the drawn rows as runs of held rows, and where the lists buffer holds + // each drawn row's word expanded from them. `drawn` is NONE when drawn row `i` is held row `i`, + // as for a table drawn whole. + row_runs: u32, + drawn: u32, + // Fade progress of tiles entering and leaving, from 0 to 1. + reveal: f32, + // The page table of each row's words: `stride` kind words, then `extras` extra words. state: u32, stride: u32, - extra: u32, extras: u32, - // While the mark eases, where row 0's frozen kind and extra words start; NONE otherwise. + // During a transition, the page tables of each row's words as shown when it began (`frozen`) + // and as its channels now evaluate them (`settled`). NONE otherwise. `motion` is its eased + // progress, from 0 to 1. frozen: u32, - frozen_extra: u32, + settled: u32, motion: f32, - // Where the program holds how each word of a row eases, a word each. + // Where the program holds each row word's blend (`BLEND_*`), one word per row word. blends: u32, - // The row blocks a dispatch over the mark's rows covers: at `block` in the blocks buffer, - // `blocks` of them. + // Where the blocks buffer lists the row blocks a dispatch covers: at `block` those its channels + // evaluate, and at `end_block` those whose ends the end pass resolves. block: u32, - blocks: u32, + end_block: u32, + // The page table of each row's focus word. focus: u32, domains: u32, columns: u32, @@ -520,57 +561,58 @@ struct Mark { shape: u32, corner: f32, hole: f32, - // Marker inputs: how many, the extra word of the first, and where their colors start in the constants. + // Marker inputs: how many, the extra word of the first, and where the marker's palette starts + // in the program. inputs: u32, input: u32, palette: u32, // The program column a line traces, or NONE. trace: u32, - // Segments of each row when every row has as many; NONE when the mark's list numbers them. + // Segments per row when every row has the same count. NONE when the segment list numbers them. per_row: u32, - // Where the header of the mark's list of segments starts in the lists buffer, and where what the - // running query finds of it starts among the probe's words. + // Where the mark's segment list header starts in the lists buffer, and where the running + // query's results for the mark start in the probe buffer. list: u32, query: u32, - // Where the mark's remap words start, while its transition is being frozen. + // Where the mark's remap words start, while `gather` freezes its transition. remap: u32, - // The rows its focus draw lights, at `focused` in the focused buffer, how many, and the - // instances each takes: a point one, a path as many as its longest piece count. + // The rows the focus draw lights: where they start in the focused buffer, how many, and the + // instances per row (one for a point, the most segments of any lit path). focused: u32, focused_rows: u32, pieces: u32, - // Program columns: the geometry a stroke's path runs through (a route or bends), the geometry - // it ran through when its transition began, and each row's lane among edges joining the same - // two vertices; NONE without. + // Program columns, NONE when absent: the geometry a stroke's path follows (a route or + // bends), that geometry at the start of the transition, and each edge's lane among edges + // joining the same two vertices. geometry: u32, frozen_geometry: u32, lane: u32, - // Each port of a vertex mark's rows, and how many each row has. + // Where the program lists the mark's ports (`PORT_WORDS` each), and how many each row has. port_info: u32, ports: u32, - // The extra word of the label's code, or NONE; the labels buffer's offsets of its candidates, its - // run table (two words per code), its wanted codes (a count, then codes), and its seen codes (a - // bit per code shown in view); the id of its first candidate, unique across marks; the size - // labels and port labels are set at, in length units. + // `label`: the extra word holding the label's code, or NONE. The next three are offsets in the + // labels buffer. `runs` is the run table: two corner words per code, both 0 until the text is + // set. `wanted` is a count followed by the codes to set, each once. `seen` holds a bit per code + // the texts pass met: a set text shown in view, or an unset text it asked for. The two sizes + // are in length units. label: u32, - candidates: u32, runs: u32, wanted: u32, seen: u32, - label_id: u32, label_size: f32, port_label_size: f32, - // The most labels shown, or NONE; and where the labels buffer holds the select that caps them. - label_max: u32, - label_select: u32, - // The CSS pixels the mark's size keeps within at any zoom: a vertex's half, a stroke's width. + // The range of CSS pixels the mark's size is clamped to at any zoom: a vertex's half size, or a + // stroke's width. size_min: f32, size_max: f32, + // A vertex mark's page table of the face normals summed at its rows. NONE when no face reads + // them. + normals: u32, } -// A line or a run of text drawn over the marks, in CSS pixels. +// An overlay sprite in CSS pixels: a line, box, fill, or run of text, drawn over the marks. struct Sprite { - // A line's ends, or a text's top left and bottom right. + // A line's ends, or opposite corners of a rectangle (top left and bottom right for text). a: vec2f, b: vec2f, // A text's rectangle in the atlas, texels. @@ -580,18 +622,18 @@ struct Sprite { kind: u32, } -// One panel of a shot composed into the bytes a consumer takes: RGBA rows for an image, or NV12 -// planes for an encoder. Pixels are the shot's. +// One panel of a shot, composed into output bytes: RGBA rows for an image, or NV12 planes for a +// video encoder. Sizes are in the shot's pixels. struct Compose { - // Where the panel's top left lands in the picture, and its size. + // The panel's top-left position in the picture, and its size. origin: vec2u, size: vec2u, // The picture's size. picture: vec2u, // Where NV12's interleaved chroma plane starts, in words. chroma: u32, - // Whether pixels keep their alpha: an image of a transparent shot. + // Whether pixels keep their alpha, for an image of a transparent shot. alpha: u32, - // What shows where no panel draws: straight sRGB. + // The color where no panel draws, as straight-alpha sRGB. background: vec4f, } diff --git a/crates/latkit/src/data/bits.rs b/crates/latkit/src/data/bits.rs index e75ce1e3..4c95bcfa 100644 --- a/crates/latkit/src/data/bits.rs +++ b/crates/latkit/src/data/bits.rs @@ -2,7 +2,7 @@ use std::ops::Range; use super::buffer::Buffer; -/// A shared, immutable bit vector, used for booleans and validity. +/// A shared, immutable bit vector for booleans and validity. Cloning and slicing share the words. #[derive(Clone, Default)] pub(crate) struct Bits { words: Buffer, @@ -29,7 +29,7 @@ impl Bits { Bits { words: self.words.slice(first..last.max(first)), offset: bit & 63, len: range.end - range.start } } - /// `len` bits, bit `i` being `bit(i)`, packed a word at a time. + /// `len` bits where bit `i` is `bit(i)`, packed 64 to a word. pub(crate) fn from_fn(len: usize, bit: impl Fn(usize) -> bool) -> Bits { let words = (0..len.div_ceil(64)) .map(|w| { @@ -40,14 +40,6 @@ impl Bits { Bits { words: words.into(), offset: 0, len } } - /// How many bits are set. - pub(crate) fn count(&self) -> usize { - if self.offset != 0 { - return (0..self.len).filter(|&i| self.get(i)).count(); - } - self.to_words().iter().map(|w| w.count_ones() as usize).sum() - } - pub(crate) fn all(&self) -> bool { if self.offset != 0 { return (0..self.len).all(|i| self.get(i)); @@ -57,7 +49,8 @@ impl Bits { self.words[..full].iter().all(|&w| w == u64::MAX) && (rest == 0 || self.words[full] & mask == mask) } - /// The bits as little-endian words starting at bit 0, for byte encoding. + /// The bits as words starting at bit 0, least significant bit first, with the unused high bits + /// of the last word cleared. This is the byte encoding's layout. pub(crate) fn to_words(&self) -> Vec { if self.offset == 0 { let mut words = self.words[..self.len.div_ceil(64)].to_vec(); @@ -98,7 +91,7 @@ impl std::fmt::Debug for Bits { } } -/// Builds [`Bits`] one bit at a time, keeping the word it fills apart. +/// Builds [`Bits`] one bit at a time. The partly filled word is kept separately until it is full. pub(crate) struct BitsBuilder { words: Vec, word: u64, diff --git a/crates/latkit/src/data/buffer.rs b/crates/latkit/src/data/buffer.rs index 9ef9a4cd..1421c1f2 100644 --- a/crates/latkit/src/data/buffer.rs +++ b/crates/latkit/src/data/buffer.rs @@ -1,7 +1,7 @@ use std::ops::{Deref, Range}; use std::sync::Arc; -/// A shared, immutable slice: making one from a vector, cloning it, and slicing it never copy. +/// A shared, immutable slice. Creating one from a `Vec`, cloning, and slicing never copy elements. pub(crate) struct Buffer { data: Arc>, start: usize, diff --git a/crates/latkit/src/data/bytes.rs b/crates/latkit/src/data/bytes.rs index ca00e975..26783c4a 100644 --- a/crates/latkit/src/data/bytes.rs +++ b/crates/latkit/src/data/bytes.rs @@ -1,31 +1,38 @@ -//! One byte encoding for patches and datasets: a frame of a magic, its length, the name of the -//! dataset its patch changes (empty when it names none), and the patch's operations. Little endian -//! throughout. A frame says its length up front, so frames back to back split without decoding, and a -//! stream knows at once whether it holds a whole one. A dataset's bytes are the patch to it that -//! builds it. +//! The binary encoding shared by patches and datasets. //! -//! | Part | Layout | -//! | -------- | ------------------------------------------------------------------------------------- | -//! | frame | `b"LATK"`, `u16` version (1), `u16` flags (0), `u64` length of the whole frame, name, `u32` op count, ops | -//! | op | `u8` tag and its payload: axis, replace, set, remove, append, forget, or frames | -//! | axis | name, unit | -//! | replace | table, as `set` | -//! | remove | table name, ids | -//! | append | table name, `u8` 1 and ids or `u8` 0 for every row, `u32` frames, `f64` coordinates, | -//! | | `u32` field count; per field: name, values frame-major | -//! | forget | `f64` before | -//! | frames | table name, field name, `u32` rows, run | -//! | table | name, ids, `u32` field count; per field: name, `u8` tag, payload | -//! | field | values; a reference's table and its target ids as values; or samples | -//! | ids | `u8` 0 and `u32` count, or `u8` 1 and values: `u64`, `u32` read as `u64`, or texts | -//! | values | `u8` storage, `u32` len, data, `u8` has-validity, validity words (`u64`) | -//! | samples | `u32` rows, `u64` first frame read, `u64` frames read, `u8` highest level, `u32` run count, runs; buckets | -//! | | whose eight entries a level below are held are left out, as they follow from those | -//! | run | `u8` level, `u64` first entry, `u32` entries, `f64` coordinates (a frame's own; a | -//! | | bucket's first and last frame's), `f32` values entry-major: a frame's per row, or a | -//! | | bucket's least per row and, after every bucket's, its greatest (NaN missing) | -//! | `str` | `u32` byte length, UTF-8 | - +//! A patch encodes as one frame. The frame holds a header (magic and total length), the name of the +//! dataset the patch changes (empty if it names none), and the patch's operations. Everything is +//! little endian. Because the header states the frame's length, back-to-back frames split without +//! decoding, and a reader knows as soon as the header arrives whether it has a whole frame. A +//! dataset encodes as the patch that builds it from an empty dataset. +//! +//! | Part | Layout | +//! | ------------ | ---------------------------------------------------------------------------- | +//! | frame | `b"LATK"`, `u64` length of the whole frame, name, `u32` op count, ops | +//! | op | `u8` tag (0 axis, 1 replace, 2 set, 3 remove, 4 append, 5 forget, 6 frames), | +//! | | then its payload | +//! | axis | name, unit | +//! | replace, set | table | +//! | remove | table name, ids | +//! | append | table name, `u8` 1 and ids (or `u8` 0 for every row), `u32` frame count, | +//! | | `f64` coordinates, `u32` field count; per field: name, values frame-major | +//! | forget | `f64` before | +//! | frames | table name, field name, `u32` rows, run | +//! | table | name, ids, `u32` field count; per field: name, `u8` tag, payload | +//! | field | values; a reference's target table name and target ids as values; or samples | +//! | ids | `u8` 0 and `u32` count, or `u8` 1 and values: `u64`, `u32` (read as `u64`), | +//! | | or text | +//! | values | `u8` storage, `u32` length, data, `u8` has-validity, validity words (`u64`) | +//! | samples | `u32` rows, `u64` first frame, `u64` frame count, `u8` highest level, `u32` | +//! | | run count, runs. Buckets whose eight entries one level below are all present | +//! | | are omitted, since decoding derives them | +//! | run | `u8` level, `u64` first entry, `u32` entry count, `f64` coordinates (one per | +//! | | frame; two per bucket, its first and last frame's), then `f32` values | +//! | | entry-major: for frames, one per row; for buckets, every bucket's least | +//! | | value per row, then every bucket's greatest. NaN is missing | +//! | `str` | `u32` byte length, UTF-8 | + +use std::io::Read; use std::sync::Arc; use super::bits::Bits; @@ -38,20 +45,25 @@ use super::samples::{self, MAX_LEVEL, Run, Samples}; use super::table::{Field, Reference, Table}; use super::text::Text; use super::values::{Array, Values}; +use crate::wire::{In, Out}; use crate::{Error, Result}; const MAGIC: &[u8; 4] = b"LATK"; -/// Magic, version, flags, and total frame length. -const HEADER: usize = 16; -const VERSION: u16 = 1; - -/// Limits for one binary frame. These bound encoded bytes and conservative decoded storage -/// accounting; applying a patch and retaining datasets have separate memory costs. +/// Header size in bytes: the magic and the frame's total length. +const HEADER: usize = 12; + +/// Size limits for decoding one binary frame. Exceeding either is an +/// [`Error::Limit`](crate::Error::Limit). +/// +/// The limits bound the encoded bytes and a conservative estimate of decoded storage. Memory used +/// afterwards, to apply the patch or keep the dataset, is not counted. #[derive(Clone, Copy, Debug)] pub struct DecodeLimits { - /// Maximum encoded frame size; checked as soon as the header arrives. + /// The largest encoded frame, in bytes. Checked as soon as the header arrives. Defaults to + /// 256 MiB. pub frame_bytes: usize, - /// Budget for decoded vectors, names, operations, pages, and derived sample buckets. + /// The budget, in bytes, for decoded vectors, names, operations, pages, and derived sample + /// buckets. Defaults to 512 MiB. pub decoded_bytes: usize, } @@ -83,104 +95,152 @@ const STORE_TEXT: u8 = 6; const STORE_GEOMETRY: u8 = 7; pub(crate) fn encode(patch: &Patch) -> Vec { - let mut w = Writer(Vec::new()); - w.0.extend_from_slice(MAGIC); - w.0.extend_from_slice(&VERSION.to_le_bytes()); - w.0.extend_from_slice(&0_u16.to_le_bytes()); - w.0.extend_from_slice(&[0; 8]); - w.str(patch.data.as_deref().unwrap_or("")); - w.u32(patch.ops.len()); + let mut w = Writer(Out::default()); + w.0.0.extend_from_slice(MAGIC); + w.0.u64(0); + w.0.str(patch.data.as_deref().unwrap_or("")); + w.0.len(patch.ops.len()); for op in &patch.ops { match op { Op::Axis { name, unit } => { - w.u8(OP_AXIS); - w.str(name); - w.str(unit); + w.0.u8(OP_AXIS); + w.0.str(name); + w.0.str(unit); } Op::Replace(table) => { - w.u8(OP_REPLACE); + w.0.u8(OP_REPLACE); w.table(table); } Op::Set(table) => { - w.u8(OP_SET); + w.0.u8(OP_SET); w.table(table); } Op::Remove { table, ids } => { - w.u8(OP_REMOVE); - w.str(table); + w.0.u8(OP_REMOVE); + w.0.str(table); w.ids(ids); } Op::Append { table, at, ids, fields } => { - w.u8(OP_APPEND); - w.str(table); + w.0.u8(OP_APPEND); + w.0.str(table); match ids { Some(ids) => { - w.u8(1); + w.0.u8(1); w.ids(ids); } - None => w.u8(0), + None => w.0.u8(0), } - w.u32(at.len()); - for &t in at { - w.f64(t); - } - w.u32(fields.len()); + w.0.len(at.len()); + w.0.pod(at); + w.0.len(fields.len()); for (name, values) in fields { - w.str(name); + w.0.str(name); w.values(values); } } Op::Forget { before } => { - w.u8(OP_FORGET); - w.f64(*before); + w.0.u8(OP_FORGET); + w.0.f64(*before); } Op::Frames { table, field, run, at, values } => { - w.u8(OP_FRAMES); - w.str(table); - w.str(field); + w.0.u8(OP_FRAMES); + w.0.str(table); + w.0.str(field); let per_row = run.len() * samples::sides(run.level); - w.u32(values.len().checked_div(per_row).unwrap_or(0)); + w.0.len(values.len().checked_div(per_row).unwrap_or(0)); w.run(run, at, values); } } } - let length = w.0.len() as u64; - w.0[8..HEADER].copy_from_slice(&length.to_le_bytes()); - w.0 + let mut bytes = w.0.0; + let length = bytes.len() as u64; + bytes[MAGIC.len()..HEADER].copy_from_slice(&length.to_le_bytes()); + bytes +} + +/// Encodes `values` alone, in the format's `values` layout. Jobs use it to pass columns to and +/// from the host. +#[cfg(feature = "gpu")] +pub(crate) fn values_bytes(values: &Values) -> Vec { + let mut w = Writer(Out::default()); + w.values(values); + w.0.0 +} + +/// Decodes values written by [`values_bytes`]. Errors if `bytes` are malformed or extend past +/// the values. +#[cfg(feature = "gpu")] +pub(crate) fn read_values(bytes: &[u8]) -> Result { + let mut r = Reader { input: In::new(bytes), remaining: DecodeLimits::default().decoded_bytes }; + let values = r.values()?; + if !r.input.is_empty() { + return Err(malformed("trailing bytes")); + } + Ok(values) } -/// The patch `bytes` hold, every byte of them. +/// Decodes one frame that fills `bytes` exactly, with default limits. Errors if `bytes` are +/// malformed, truncated, or extend past the frame. pub(crate) fn decode(bytes: &[u8]) -> Result { - match decode_with_limits(bytes, DecodeLimits::default())? { + match frame(bytes, DecodeLimits::default())? { Some((patch, used)) if used == bytes.len() => Ok(patch), Some(_) => Err(malformed("trailing bytes")), None => Err(malformed("truncated")), } } -/// The patch at the start of `bytes` and how many bytes it takes; `None` while `bytes` hold only -/// its start. A frame its length says runs past `limit` bytes is an error at once. -pub(crate) fn decode_prefix(bytes: &[u8], limit: usize) -> Result> { - decode_with_limits(bytes, DecodeLimits { frame_bytes: limit, ..DecodeLimits::default() }) +/// Decodes back-to-back frames from `read`, yielding each patch once its frame has fully arrived. +/// Each header is checked against `limits` as it arrives and gives the number of bytes to read. +/// The iterator ends when the input ends between frames, or after the first error. +pub(crate) fn stream(mut read: R, limits: DecodeLimits) -> impl Iterator> { + let mut ended = false; + std::iter::from_fn(move || { + if ended { + return None; + } + let patch = next(&mut read, limits).transpose(); + ended = !matches!(patch, Some(Ok(_))); + patch + }) +} + +/// Reads the next frame from `read`. Returns `None` if the input ends between frames. An end +/// inside a frame is a truncation error. +fn next(read: &mut impl Read, limits: DecodeLimits) -> Result> { + let failed = |error: std::io::Error| match error.kind() { + std::io::ErrorKind::UnexpectedEof => malformed("truncated"), + _ => Error::Io(error.to_string().into()), + }; + let mut bytes = vec![0; HEADER]; + let mut held = 0; + // Check the header as its bytes arrive, so input that is not latkit's fails immediately. + let length = loop { + if let Some(length) = length(&bytes[..held], limits.frame_bytes)? { + break length; + } + match read.read(&mut bytes[held..]) { + Ok(0) if held == 0 => return Ok(None), + Ok(0) => return Err(malformed("truncated")), + Ok(n) => held += n, + Err(error) if error.kind() == std::io::ErrorKind::Interrupted => {} + Err(error) => return Err(failed(error)), + } + }; + bytes.resize(length, 0); + read.read_exact(&mut bytes[HEADER..]).map_err(failed)?; + let (patch, _) = frame(&bytes, limits)?.expect("a whole frame"); + Ok(Some(patch)) } -pub(crate) fn decode_with_limits(bytes: &[u8], limits: DecodeLimits) -> Result> { - let limit = limits.frame_bytes; +/// The total length of the frame starting at `bytes`, read from its header. Validates the part +/// of the magic that has arrived, so foreign input fails early. Returns `None` until the whole +/// header is present. Errors if the length exceeds `limit` or is shorter than the header. +fn length(bytes: &[u8], limit: usize) -> Result> { let held = bytes.len().min(MAGIC.len()); if bytes[..held] != MAGIC[..held] { return Err(malformed("not latkit bytes")); } - if let Some(version) = bytes.get(4..6) - && version != VERSION.to_le_bytes() - { - return Err(malformed("unsupported LATK version")); - } - if let Some(flags) = bytes.get(6..8) - && flags != [0, 0] - { - return Err(malformed("unsupported LATK flags")); - } - let Some(length) = bytes.get(8..HEADER) else { return Ok(None) }; + let Some(length) = bytes.get(MAGIC.len()..HEADER) else { return Ok(None) }; let length = u64::from_le_bytes(length.try_into().expect("eight bytes")); let length = usize::try_from(length).map_err(|_| too_long(limit))?; if length > limit { @@ -189,10 +249,17 @@ pub(crate) fn decode_with_limits(bytes: &[u8], limits: DecodeLimits) -> Result Result> { + let Some(length) = length(bytes, limits.frame_bytes)? else { return Ok(None) }; + let Some(frame) = bytes.get(HEADER..length) else { return Ok(None) }; + let mut r = Reader { input: In::new(frame), remaining: limits.decoded_bytes }; let patch = r.patch()?; - if r.at != frame.len() { + if !r.input.is_empty() { return Err(malformed("a frame longer than its patch")); } Ok(Some((patch, length))) @@ -216,7 +283,7 @@ impl Reader<'_> { let ids = match r.u8()? { 0 => None, 1 => Some(r.ids()?), - _ => return Err(malformed("an append names its rows or none")), + _ => return Err(malformed("an append's ids flag is neither 0 nor 1")), }; let frames = r.len(8)?; let at = r.vec(frames)?; @@ -248,70 +315,43 @@ fn too_long(limit: usize) -> Error { Error::Limit(format!("a patch longer than {limit} bytes").into()) } -struct Writer(Vec); +/// Writes the format's parts. +struct Writer(Out); impl Writer { - fn u8(&mut self, v: u8) { - self.0.push(v); - } - - fn u32(&mut self, v: usize) { - self.0.extend_from_slice(&u32::try_from(v).expect("counts fit in u32").to_le_bytes()); - } - - fn u64(&mut self, v: u64) { - self.0.extend_from_slice(&v.to_le_bytes()); - } - - fn f64(&mut self, v: f64) { - self.0.extend_from_slice(&v.to_le_bytes()); - } - - fn str(&mut self, s: &str) { - self.u32(s.len()); - self.0.extend_from_slice(s.as_bytes()); - } - - fn slice(&mut self, values: &[T]) { - #[cfg(target_endian = "little")] - self.0.extend_from_slice(bytemuck::cast_slice(values)); - #[cfg(target_endian = "big")] - compile_error!("latkit's byte format is written on little-endian targets"); - } - fn table(&mut self, table: &Table) { - self.str(&table.name); + self.0.str(&table.name); match &table.ids { RowIds::Count(n) => { - self.u8(0); - self.u32(*n as usize); + self.0.u8(0); + self.0.u32(*n); } RowIds::Listed(column) => { - self.u8(1); + self.0.u8(1); self.values(&column.values()); } } - self.u32(table.fields.len()); + self.0.len(table.fields.len()); for (name, field) in &table.fields { - self.str(name); + self.0.str(name); match field { Field::Values(column, _) => { - self.u8(FIELD_VALUES); + self.0.u8(FIELD_VALUES); self.column(column); } Field::Reference(reference) => { - self.u8(FIELD_REFERENCE); - self.str(&reference.to); + self.0.u8(FIELD_REFERENCE); + self.0.str(&reference.to); self.column(&reference.ids); } Field::Sampled(samples, _) => { - self.u8(FIELD_SAMPLED); - self.u32(table.len()); - self.u64(samples.start()); - self.u64(samples.end() - samples.start()); - self.u8(samples.levels() as u8); + self.0.u8(FIELD_SAMPLED); + self.0.len(table.len()); + self.0.u64(samples.start()); + self.0.u64(samples.end() - samples.start()); + self.0.u8(samples.levels() as u8); let runs = samples.stored_runs(); - self.u32(runs.len()); + self.0.len(runs.len()); let rows: Vec = (0..table.len()).collect(); for run in &runs { let (at, values) = samples.read(run, &rows); @@ -322,23 +362,24 @@ impl Writer { } } - /// `run`: its level, first entry, and entries, its coordinates `at`, and its `values`. + /// Writes `run`: its level, first entry, and entry count, then its coordinates `at` and its + /// `values`. fn run(&mut self, run: &Run, at: &[f64], values: &[f32]) { - self.u8(run.level as u8); - self.u64(run.entries.start); - self.u32(run.len()); - self.slice(at); - self.slice(values); + self.0.u8(run.level as u8); + self.0.u64(run.entries.start); + self.0.len(run.len()); + self.0.pod(at); + self.0.pod(values); } fn ids(&mut self, ids: &Ids) { match &ids.0 { IdList::Count(n) => { - self.u8(0); - self.u32(*n as usize); + self.0.u8(0); + self.0.u32(*n); } IdList::Listed(values) => { - self.u8(1); + self.0.u8(1); self.values(values); } } @@ -357,53 +398,56 @@ impl Writer { Array::U32(v) => self.numbers(STORE_U32, v), Array::U64(v) => self.numbers(STORE_U64, v), Array::Bool(bits) => { - self.u8(STORE_BOOL); - self.u32(len); - self.slice(&bits.to_words()); + self.0.u8(STORE_BOOL); + self.0.len(len); + self.0.pod(&bits.to_words()); } Array::Text(text) => { - self.u8(STORE_TEXT); - self.u32(len); + self.0.u8(STORE_TEXT); + self.0.len(len); let (offsets, bytes) = text.compact(); - self.slice(&offsets); - self.str(bytes); + self.0.pod(&offsets); + self.0.str(bytes); } Array::Geometry(geometry) => { - self.u8(STORE_GEOMETRY); - self.u32(len); - self.u8(geometry.depth()); + self.0.u8(STORE_GEOMETRY); + self.0.len(len); + self.0.u8(geometry.depth()); let (levels, coords) = geometry.compact(); for level in &levels { - self.u32(level.len()); - self.slice(level); + self.0.len(level.len()); + self.0.pod(level); } - self.u32(coords.len()); - self.slice(bytemuck::cast_slice::<[f64; 2], f64>(&coords)); + self.0.len(coords.len()); + self.0.pod(&coords); } } match &values.valid { Some(bits) => { - self.u8(1); - self.slice(&bits.to_words()); + self.0.u8(1); + self.0.pod(&bits.to_words()); } - None => self.u8(0), + None => self.0.u8(0), } } fn numbers(&mut self, store: u8, values: &[T]) { - self.u8(store); - self.u32(values.len()); - self.slice(values); + self.0.u8(store); + self.0.len(values.len()); + self.0.pod(values); } } -/// Reads one frame, whose bytes are all there. +/// Decodes one frame whose bytes are all present. `remaining` is the decoded-storage budget left. struct Reader<'a> { - bytes: &'a [u8], - at: usize, + input: In<'a>, remaining: usize, } +fn truncated() -> Error { + malformed("truncated") +} + impl Reader<'_> { fn charge(&mut self, count: usize, size: usize) -> Result<()> { let bytes = count.checked_mul(size).ok_or_else(|| malformed("decoded size overflow"))?; @@ -414,43 +458,34 @@ impl Reader<'_> { Ok(()) } - /// Whether `n` more bytes are in the frame. - fn reach(&self, n: usize) -> Result<()> { - if self.at.saturating_add(n) > self.bytes.len() { - return Err(malformed("truncated")); - } - Ok(()) - } - fn take(&mut self, n: usize) -> Result<&[u8]> { - self.reach(n)?; - let end = self.at + n; - let slice = &self.bytes[self.at..end]; - self.at = end; - Ok(slice) + self.input.bytes(n).ok_or_else(truncated) } fn u8(&mut self) -> Result { - Ok(self.take(1)?[0]) + self.input.u8().ok_or_else(truncated) } fn u32(&mut self) -> Result { - Ok(u32::from_le_bytes(self.take(4)?.try_into().expect("four bytes"))) + self.input.u32().ok_or_else(truncated) } - /// A count of items at least `size` bytes each, checked against the bytes left. + /// Reads a `u32` count of items of at least `size` bytes each. Errors if the remaining bytes + /// cannot hold that many, so a corrupt count cannot cause a huge allocation. fn len(&mut self, size: usize) -> Result { let n = self.u32()? as usize; - self.reach(n.saturating_mul(size))?; + if !self.input.has(n.saturating_mul(size)) { + return Err(truncated()); + } Ok(n) } fn u64(&mut self) -> Result { - Ok(u64::from_le_bytes(self.take(8)?.try_into().expect("eight bytes"))) + self.input.u64().ok_or_else(truncated) } fn f64(&mut self) -> Result { - Ok(f64::from_le_bytes(self.take(8)?.try_into().expect("eight bytes"))) + self.input.f64().ok_or_else(truncated) } fn name(&mut self) -> Result { @@ -461,18 +496,22 @@ impl Reader<'_> { std::str::from_utf8(bytes).map(Name::from).map_err(|_| malformed("text is not UTF-8")) } + /// Reads `n` plain values by copying their bytes. fn vec(&mut self, n: usize) -> Result> { - // Includes conversion into column pages and the sampled levels derived from runs. + // Charge four times the raw size, to cover conversion into column pages and the sample + // levels derived from runs. self.charge(n, 4 * size_of::())?; - let bytes = self.take(n.checked_mul(size_of::()).ok_or_else(|| malformed("too large"))?)?; - Ok(bytes.chunks_exact(size_of::()).map(bytemuck::pod_read_unaligned).collect()) + if n.checked_mul(size_of::()).is_none() { + return Err(malformed("too large")); + } + self.input.pod(n).ok_or_else(truncated) } fn ids(&mut self) -> Result { match self.u8()? { 0 => Ok(Ids(IdList::Count(self.u32()?))), 1 => Ok(Ids(IdList::Listed(self.id_values()?))), - tag => Err(malformed(format!("unknown ids {tag}"))), + tag => Err(malformed(format!("unknown ids encoding {tag}"))), } } @@ -480,8 +519,8 @@ impl Reader<'_> { Ok(Column::new(&self.values()?)) } - /// Ids as values: `u32` ids, as writers without 64-bit integers send them, read as the `u64` every - /// numbered id is. + /// Reads ids as values. `u32` ids, sent by writers without 64-bit integers, are widened to + /// `u64`, which stores every numeric id. fn id_values(&mut self) -> Result { let values = self.values()?; Ok(match values.array { @@ -511,15 +550,15 @@ impl Reader<'_> { Field::Reference(Reference::new(to, Column::new(&self.id_values()?))) } FIELD_SAMPLED => Field::Sampled(self.samples().map_err(|e| e.at(&name))?, Version::next()), - tag => return Err(malformed(format!("unknown field {tag}"))), + tag => return Err(malformed(format!("unknown field kind {tag}"))), }; table.insert(name, field); } Ok(table) } - /// A sampled field: its runs put in level by level, each level's buckets that follow from the - /// level below written before its own runs. + /// Reads a sampled field. Runs are inserted level by level. Before each level's own runs, the + /// buckets derivable from the level below are rebuilt. fn samples(&mut self) -> Result { let rows = self.u32()? as usize; let first = self.u64()?; @@ -548,7 +587,7 @@ impl Reader<'_> { Ok(samples) } - /// A run of entries of `rows` rows: what it holds, its coordinates, and its values. + /// Reads a run of entries for `rows` rows: the run, its coordinates, and its values. fn run(&mut self, rows: usize) -> Result<(Run, Vec, Vec)> { let level = u32::from(self.u8()?); if level > MAX_LEVEL { @@ -584,7 +623,7 @@ impl Reader<'_> { let n = self.len(1)?; self.charge(n, 2)?; let bytes = String::from_utf8(self.take(n)?.to_vec()).map_err(|_| malformed("text is not UTF-8"))?; - Array::Text(Text::from_parts(offsets, bytes).ok_or_else(|| malformed("text offsets"))?) + Array::Text(Text::from_parts(offsets, bytes).ok_or_else(|| malformed("invalid text offsets"))?) } STORE_GEOMETRY => { let depth = self.u8()?; @@ -595,12 +634,11 @@ impl Reader<'_> { }) .collect::>>()?; let n = self.len(16)?; - let flat: Vec = self.vec(n * 2)?; - let coords = flat.as_chunks::<2>().0.to_vec(); + let coords: Vec<[f64; 2]> = self.vec(n)?; let geometry = - Geometry::from_parts(depth, levels, coords).ok_or_else(|| malformed("geometry offsets"))?; + Geometry::from_parts(depth, levels, coords).ok_or_else(|| malformed("invalid geometry offsets"))?; if geometry.len() != len { - return Err(malformed("geometry length")); + return Err(malformed("geometry count does not match the value count")); } Array::Geometry(geometry) } @@ -609,7 +647,7 @@ impl Reader<'_> { let valid = match self.u8()? { 0 => None, 1 => Some(Bits::from_words(self.vec(len.div_ceil(64))?, len)), - _ => return Err(malformed("validity")), + _ => return Err(malformed("the validity flag is neither 0 nor 1")), }; Ok(Values { array, valid }) } @@ -618,21 +656,22 @@ impl Reader<'_> { #[cfg(test)] mod tests { use super::*; - use crate::{Data, Row, Value}; + use crate::{Data, Row, Snapshot, Value}; #[test] - fn v1_fixture_and_large_indexes_are_stable() { - let hex = include_str!("../../tests/fixtures/latk-v1.hex").trim(); + fn the_patch_fixture_and_large_indexes_are_stable() { + let hex = include_str!("../../tests/fixtures/patch.hex").trim(); let bytes: Vec = (0..hex.len()).step_by(2).map(|i| u8::from_str_radix(&hex[i..i + 2], 16).unwrap()).collect(); let patch = Patch::new().frames("T", "v", 0, (1_u64 << 32) + 3, [0.0, 1.0], [1.0, 2.0]); assert_eq!(patch.to_bytes(), bytes); assert_eq!(Patch::from_bytes(&bytes).unwrap().to_bytes(), bytes); - let mut data = Data::new("test"); + let data = Data::new("test"); data.apply(Patch::new().replace(Table::new("T", Ids::count(1))).extend(patch)).unwrap(); - data.apply(Patch::new().append([2.0], "T", [("v", vec![3.0_f32])])).unwrap(); + data.apply(Patch::new().append("T", [2.0], [("v", vec![3.0_f32])])).unwrap(); + let data = data.snapshot(); assert_eq!(data.span(), Some([0.0, 2.0])); - assert_eq!(Data::from_bytes(&data.to_bytes()).unwrap().to_bytes(), data.to_bytes()); + assert_eq!(Snapshot::from_bytes(&data.to_bytes()).unwrap().to_bytes(), data.to_bytes()); let mut files = std::collections::HashMap::new(); data.to_tiles(8, |path, bytes| { files.insert(path.to_owned(), bytes.to_vec()); @@ -640,21 +679,13 @@ mod tests { }) .unwrap(); assert!(files.keys().any(|p| p.contains("536870912"))); - let mut version = bytes.clone(); - version[4] = 2; - assert!(Patch::from_bytes(&version).is_err()); - let mut flags = bytes.clone(); - flags[6] = 1; - assert!(Patch::from_bytes(&flags).is_err()); - assert!(matches!( - Patch::decode_with_limits(&bytes, DecodeLimits { frame_bytes: 1024, decoded_bytes: 1 }), - Err(Error::Limit(_)) - )); + let tight = DecodeLimits { frame_bytes: 1024, decoded_bytes: 1 }; + assert!(matches!(Patch::stream(&bytes[..], tight).next(), Some(Err(Error::Limit(_))))); } #[test] fn frame_overflow_is_atomic_and_high_levels_round_trip() { - let mut data = Data::new("test"); + let data = Data::new("test"); data.apply(Patch::new().replace(Table::new("T", Ids::count(1))).frames( "T", "v", @@ -664,10 +695,10 @@ mod tests { [1.0], )) .unwrap(); - let before = data.to_bytes(); - assert!(data.apply(Patch::new().append([1.0], "T", [("v", vec![2.0_f32])])).is_err()); - assert_eq!(data.to_bytes(), before); - let mut data = Data::new("test"); + let before = data.snapshot().to_bytes(); + assert!(data.apply(Patch::new().append("T", [1.0], [("v", vec![2.0_f32])])).is_err()); + assert_eq!(data.snapshot().to_bytes(), before); + let data = Data::new("test"); data.apply(Patch::new().replace(Table::new("T", Ids::count(1))).frames( "T", "v", @@ -677,11 +708,12 @@ mod tests { [1.0, 2.0], )) .unwrap(); - assert_eq!(Data::from_bytes(&data.to_bytes()).unwrap().to_bytes(), data.to_bytes()); + let data = data.snapshot(); + assert_eq!(Snapshot::from_bytes(&data.to_bytes()).unwrap().to_bytes(), data.to_bytes()); } fn grid() -> Data { - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply( Patch::new() .axis("time", "s") @@ -699,99 +731,112 @@ mod tests { )), ) .expect("valid"); - data.apply(Patch::new().append([1.0], "Bus", [("vm", vec![1.0, 1.01, 0.99])])).expect("frame"); - data.apply(Patch::new().append([2.0], "Bus", [("vm", vec![1.02, f64::NAN, 0.98])])).expect("frame"); + data.apply(Patch::new().append("Bus", [1.0], [("vm", vec![1.0, 1.01, 0.99])])).expect("frame"); + data.apply(Patch::new().append("Bus", [2.0], [("vm", vec![1.02, f64::NAN, 0.98])])).expect("frame"); data } #[test] fn datasets_round_trip() { - let data = grid(); + let data = grid().snapshot(); let bytes = data.to_bytes(); - let back = Data::from_bytes(&bytes).expect("decodes"); + let back = Snapshot::from_bytes(&bytes).expect("decodes"); assert_eq!(back.name(), "grid"); assert_eq!(back.axis(), Some(("time", "s"))); - let bus = |id: &str| Row::new("grid", "Bus", id); for field in ["kv", "name", "point", "open", "vm"] { for id in ["a", "b", "c"] { - assert_eq!(back.get(&bus(id), field), data.get(&bus(id), field), "{field} of {id}"); + assert_eq!(back.get("Bus", id, field, None), data.get("Bus", id, field, None), "{field} of {id}"); } } - assert_eq!(back.get_at(&bus("a"), "vm", 1.5), Some(Value::Number(f64::from(1.0_f32)))); - assert_eq!(back.get(&Row::new("grid", "Branch", 1_u64), "to"), Some(Value::Row(bus("c")))); + assert_eq!(back.get("Bus", "a", "vm", 1.5), Some(Value::Number(f64::from(1.0_f32)))); + assert_eq!(back.get("Branch", 1_u64, "to", None), Some(Value::Row(Row::new("grid", "Bus", "c")))); assert_eq!(back.to_bytes(), bytes, "encoding is deterministic"); } #[test] fn patches_round_trip() { - let patch = Patch::new().remove("Bus", vec!["a"]).append([3.0], "Bus", [("vm", vec![1.0, 1.0, 1.0])]); + let patch = Patch::new().remove("Bus", vec!["a"]).append("Bus", [3.0], [("vm", vec![1.0, 1.0, 1.0])]); let bytes = patch.to_bytes(); assert_eq!(Patch::from_bytes(&bytes).expect("decodes").to_bytes(), bytes); - assert!(Data::from_bytes(&bytes).is_err()); + assert!(Snapshot::from_bytes(&bytes).is_err()); } - #[test] - fn patches_back_to_back_decode_one_by_one() { - let patches = [grid().to_patch(), Patch::new().forget(1.5), Patch::new().remove("Bus", vec!["a"])]; - let stream: Vec = patches.iter().flat_map(Patch::to_bytes).collect(); - let (mut at, mut read) = (0, Vec::new()); - // Arriving a few bytes at a time, as from a socket. - for end in (0..=stream.len()).step_by(5).chain([stream.len()]) { - while let Some((patch, used)) = Patch::decode(&stream[at..end], usize::MAX).expect("valid so far") { - read.push(patch.to_bytes()); - at += used; - } + /// A reader that returns at most `n` bytes per read, like a socket. + struct Trickle<'a>(&'a [u8], usize); + + impl Read for Trickle<'_> { + fn read(&mut self, into: &mut [u8]) -> std::io::Result { + let n = self.1.min(into.len()).min(self.0.len()); + into[..n].copy_from_slice(&self.0[..n]); + self.0 = &self.0[n..]; + Ok(n) } + } + + #[test] + fn back_to_back_patches_stream_individually() { + let patches = [grid().snapshot().to_patch(), Patch::new().forget(1.5), Patch::new().remove("Bus", vec!["a"])]; + let bytes: Vec = patches.iter().flat_map(Patch::to_bytes).collect(); + let read: Vec> = Patch::stream(Trickle(&bytes, 5), DecodeLimits::default()) + .map(|patch| patch.expect("valid").to_bytes()) + .collect(); assert_eq!(read, patches.iter().map(Patch::to_bytes).collect::>()); - assert!(Patch::decode(b"LATX....", 64).is_err(), "not latkit bytes, however many follow"); - // A length past the limit fails at once rather than waiting for bytes that may never come. + let first = |bytes: &[u8], limits: DecodeLimits| Patch::stream(bytes, limits).next(); + assert!(first(&[], DecodeLimits::default()).is_none(), "empty input yields no patch"); + let foreign = first(b"LATX", DecodeLimits::default()); + assert!(foreign.is_some_and(|e| e.is_err_and(|e| e.to_string().contains("not latkit bytes"))), "immediately"); + // A length over the limit fails as soon as the header arrives, without waiting for bytes + // that may never come. let long = Patch::new().set(Table::new("Bus", Ids::count(1)).field("name", vec!["x".repeat(100)])).to_bytes(); - assert!(matches!(Patch::decode(&long[..70], 64), Err(Error::Limit(_)))); - assert!(Patch::decode(&long[..70], 1 << 20).expect("short").is_none()); + let short = DecodeLimits { frame_bytes: 64, ..DecodeLimits::default() }; + assert!(matches!(first(&long[..HEADER], short), Some(Err(Error::Limit(_))))); + assert!(matches!(first(&long[..70], DecodeLimits::default()), Some(Err(Error::Invalid { .. }))), "truncated"); + let mut twice = Patch::stream(&long[..70], DecodeLimits::default()); + assert!(twice.next().is_some_and(|patch| patch.is_err()) && twice.next().is_none(), "ends after an error"); } #[test] fn frames_keep_their_numbers_through_bytes_and_gathers() { - let mut data = grid(); + let data = grid(); data.apply(Patch::new().forget(1.5)).expect("forget"); - let first = |data: &Data| match data.table("Bus").and_then(|t| t.field_named("vm")) { + let first = |data: &Snapshot| match data.table("Bus").and_then(|t| t.field_named("vm")) { Some(Field::Sampled(samples, _)) => samples.start(), _ => panic!("sampled"), }; - assert_eq!(first(&data), 1); - assert_eq!(first(&Data::from_bytes(&data.to_bytes()).expect("decodes")), 1); + assert_eq!(first(&data.snapshot()), 1); + assert_eq!(first(&Snapshot::from_bytes(&data.snapshot().to_bytes()).expect("decodes")), 1); data.apply(Patch::new().remove("Bus", vec!["b"])).expect("remove"); - assert_eq!(first(&data), 1, "rows removed keep the frames' numbers"); + assert_eq!(first(&data.snapshot()), 1, "removing rows keeps frame numbers"); } #[test] - fn time_tiles_round_trip_with_their_holes() { - let mut data = grid(); - // The bucket of frames 64..72, and frames 200 and 201, past a hole. + fn time_tiles_round_trip_with_gaps() { + let data = grid(); + // The level-1 bucket of frames 64..72, then frames 200 and 201 after a gap. let bucket = Patch::new().frames("Bus", "vm", 1, 8, [64.0, 71.0], [0.9, 0.95, 0.97, 1.1, 1.05, 1.0]); let tile = bucket.frames("Bus", "vm", 0, 200, [200.0, 201.0], [1.0, 1.0, 1.0, 0.5, 0.5, 0.5]); let bytes = tile.to_bytes(); assert_eq!(Patch::from_bytes(&bytes).expect("decodes").to_bytes(), bytes); data.apply(tile.clone()).expect("tiles"); - assert!(data.apply(tile).is_err(), "entries held already"); - let back = Data::from_bytes(&data.to_bytes()).expect("decodes"); + assert!(data.apply(tile).is_err(), "entries already present are refused"); + let data = data.snapshot(); + let back = Snapshot::from_bytes(&data.to_bytes()).expect("decodes"); assert_eq!(back.to_bytes(), data.to_bytes()); - let bus = |id: &str| Row::new("grid", "Bus", id); - assert_eq!(back.get_at(&bus("c"), "vm", 201.5), Some(Value::Number(0.5))); - assert_eq!(back.get_at(&bus("a"), "vm", 100.0), None, "a hole holds no frame"); + assert_eq!(back.get("Bus", "c", "vm", 201.5), Some(Value::Number(0.5))); + assert_eq!(back.get("Bus", "a", "vm", 100.0), None, "a gap reads as missing"); assert_eq!(back.describe().span, Some([1.0, 201.0])); } #[test] fn malformed_bytes_are_invalid_not_panics() { - let bytes = grid().to_bytes(); + let bytes = grid().snapshot().to_bytes(); for end in 0..bytes.len() { - assert!(Data::from_bytes(&bytes[..end]).is_err()); + assert!(Snapshot::from_bytes(&bytes[..end]).is_err()); } let mut flipped = bytes.clone(); for i in (0..flipped.len()).step_by(7) { flipped[i] ^= 0x5a; - let _ = Data::from_bytes(&flipped); + let _ = Snapshot::from_bytes(&flipped); } } } diff --git a/crates/latkit/src/data/column.rs b/crates/latkit/src/data/column.rs index d8aefba9..819b581f 100644 --- a/crates/latkit/src/data/column.rs +++ b/crates/latkit/src/data/column.rs @@ -1,28 +1,34 @@ -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; use super::name::PageId; use super::values::{Kind, Picks, Value, Values}; use crate::Result; -/// Rows per page: pages are the unit of sharing between versions and of upload to the GPU. -pub(crate) const PAGE_BITS: u32 = 16; +/// Log2 of the rows per page (4,096). A page is the unit of sharing between versions, of GPU +/// upload, of residency when a paged table is drawn, and of a mark's per-row evaluation. +pub(crate) const PAGE_BITS: u32 = 12; pub(crate) const PAGE_ROWS: usize = 1 << PAGE_BITS; pub(crate) const PAGE_MASK: usize = PAGE_ROWS - 1; -/// Writes a page remembers back: past them, it reads as written whole. +/// How many writes back a page records which rows changed. Beyond that, the page counts as +/// entirely rewritten. const REMEMBERED: u8 = 8; -/// One page of a column: up to [`PAGE_ROWS`] values, the id of exactly these contents, and, when it -/// was written over another page, which page and which rows. +/// One page of a column. It holds up to [`PAGE_ROWS`] values and an id unique to these contents. A +/// page made by writing over another records that page and the rows written (`written`). It also +/// caches its geometry's point count. #[derive(Clone, Debug)] pub(crate) struct Page { pub(crate) id: PageId, pub(crate) values: Values, pub(crate) written: Option>, + /// Computed on first use and cached, since pages are immutable. + points: OnceLock, } -/// What a page was written from: the page it copied, the rows written over it there, offsets into -/// the page, and how that page was written in turn. +/// How a page was written. It records the page it copied (`from`), the rows written or added as +/// offsets into the page, and how `from` was written in turn (`before`), up to [`REMEMBERED`] +/// writes back. #[derive(Debug)] pub(crate) struct Written { pub(crate) from: PageId, @@ -33,19 +39,25 @@ pub(crate) struct Written { impl Page { pub(crate) fn new(values: Values) -> Page { - Page { id: PageId::next(), values, written: None } + Page { id: PageId::next(), values, written: None, points: OnceLock::new() } } - /// `values`: page `old` with `rows` written over it, offsets into the page. + /// A new page holding `values`, made by writing `rows` (offsets into the page) over `old`. fn over(values: Values, old: &Page, rows: Vec) -> Page { let before = old.written.clone().filter(|w| w.depth < REMEMBERED); let depth = before.as_ref().map_or(1, |w| w.depth + 1); let written = Written { from: old.id, rows: rows.into(), before, depth }; - Page { id: PageId::next(), values, written: Some(Arc::new(written)) } + Page { id: PageId::next(), values, written: Some(Arc::new(written)), points: OnceLock::new() } } - /// The rows written since page `before`, offsets into the page, in order: `None` unless this page - /// was written over it, a few writes back at most. + /// The page's geometry point count as shaders read it. See [`Values::points`]. + pub(crate) fn points(&self) -> usize { + *self.points.get_or_init(|| self.values.points()) + } + + /// The rows written since page `before`, as sorted offsets into the page. Empty if this page is + /// `before`. `None` unless this page was written from `before` and still remembers it (see + /// [`REMEMBERED`]). pub(crate) fn rows_since(&self, before: PageId) -> Option> { if self.id == before { return Some(Vec::new()); @@ -65,12 +77,13 @@ impl Page { } } -/// A field's values in pages. Writes replace only the pages they touch; others keep their ids. +/// A field's values in pages of [`PAGE_ROWS`] rows. Writes replace only the pages they touch. The +/// others keep their ids. #[derive(Clone, Debug)] pub(crate) struct Column { len: usize, pages: Arc<[Page]>, - /// An empty slice of the column's storage, so an empty column still knows what it holds. + /// An empty slice with the column's storage type, so an empty column still knows its kind. like: Values, } @@ -83,20 +96,21 @@ impl Column { Column { len, pages, like: values.slice(0..0) } } - /// `len` missing values stored like `like`. + /// A column of `len` missing values with the storage of `like`. pub(crate) fn missing(like: &Values, len: usize) -> Column { let empty = Column { len: 0, pages: Arc::new([]), like: like.slice(0..0) }; empty.write(len, &[], &empty).expect("same storage") } - /// A column of whole pages; every page but the last holds [`PAGE_ROWS`] values. + /// A column of `pages`, each given a new id. Every page but the last must hold [`PAGE_ROWS`] + /// values. pub(crate) fn from_pages(pages: Vec, like: &Values) -> Column { debug_assert!(pages.iter().rev().skip(1).all(|p| p.len() == PAGE_ROWS)); let len = pages.iter().map(Values::len).sum(); Column { len, pages: pages.into_iter().map(Page::new).collect(), like: like.slice(0..0) } } - /// A column of `pages` as they are, their ids kept; every page but the last holds + /// A column of existing `pages`, keeping their ids. Every page but the last must hold /// [`PAGE_ROWS`] values. pub(crate) fn of_pages(pages: Vec, like: &Values) -> Column { debug_assert!(pages.iter().rev().skip(1).all(|p| p.values.len() == PAGE_ROWS)); @@ -116,6 +130,12 @@ impl Column { &self.pages } + /// Whether `other` has the same length and page ids. That means the values are equal, without + /// comparing any. + pub(crate) fn shares(&self, other: &Column) -> bool { + self.len == other.len && self.pages.iter().zip(other.pages.iter()).all(|(a, b)| a.id == b.id) + } + pub(crate) fn like(&self) -> &Values { &self.like } @@ -124,11 +144,28 @@ impl Column { self.pages.get(row >> PAGE_BITS)?.values.get(row & PAGE_MASK) } - /// The column with `len` rows (at least the current count) and `writes` applied. + /// The values of the page holding row `row`, and the row's index in them. `None` past the + /// column's end. + pub(crate) fn at(&self, row: usize) -> Option<(&Values, usize)> { + let values = &self.pages.get(row >> PAGE_BITS)?.values; + let i = row & PAGE_MASK; + (i < values.len()).then_some((values, i)) + } + + /// The values of rows `rows`, in that order, for GPU upload. Rows past the column's end come + /// back missing, because a placeholder derived table, shown until its job finishes, can be + /// shorter than the rows requested. + pub(crate) fn pick(&self, rows: &[u32]) -> Values { + let from: Vec> = rows.iter().map(|&r| Some(r as usize).filter(|&r| r < self.len)).collect(); + let sources: Vec<&Values> = std::iter::once(&self.like).chain(self.pages.iter().map(|p| &p.values)).collect(); + Values::assemble(&sources, Picks::Writes { from: &from, kept: 0 }).expect("one storage") + } + + /// A copy with `len` rows (at least the current count) and `writes` applied. /// - /// `writes` are `(row, row of source)` sorted by row with no row twice; a [`HOLE`](super::values::HOLE) - /// for the row of source writes a missing value. Pages that no write touches and whose length does - /// not change keep their ids; new rows not written are missing. + /// `writes` are `(row, source row)` pairs sorted by row, each row at most once. Pages that no + /// write touches and that keep their length keep their ids. New rows not written are missing. + /// Returns an error if `source` holds another kind of value. pub(crate) fn write(&self, len: usize, writes: &[(usize, usize)], source: &Column) -> Result { debug_assert!(len >= self.len); debug_assert!(writes.windows(2).all(|w| w[0].0 < w[1].0)); @@ -162,7 +199,7 @@ impl Column { let kept = sources[0].len(); let values = Values::assemble(&sources, Picks::Writes { from: &from, kept })?; pages.push(match old { - // The page remembers what it was written over: the rows written, and those past its end. + // Record the rows written, plus the rows added past the old page's end. Some(old) => { let mut rows: Vec = page_writes.iter().map(|&(row, _)| (row - start) as u32).collect(); rows.extend(kept as u32..page_len as u32); @@ -174,9 +211,9 @@ impl Column { Ok(Column { len, pages: pages.into(), like: self.like.clone() }) } - /// The rows written since `before`, which this column was written from a few writes back at most, - /// in order: `None` where a page of it was not, or the number of rows changed. A row written the - /// value it held counts. + /// The rows written since `before`, in order, where this column was written from `before` a + /// few writes back. `None` if the row count changed or a changed page no longer remembers + /// `before`. A row rewritten with its old value still counts. pub(crate) fn rows_since(&self, before: &Column) -> Option> { if self.len != before.len { return None; @@ -190,8 +227,40 @@ impl Column { Some(rows) } - /// Every value in row order, as one run: a single page's without copying, else the pages copied - /// one after another. + /// Whether the first `rows` rows are unchanged since `before`, which this column was written + /// from a few writes back. That holds when every write since is at row `rows` or later, as with + /// appends. Writes that a page no longer remembers count as changes. + pub(crate) fn kept_below(&self, before: &Column, rows: usize) -> bool { + (0..rows.div_ceil(PAGE_ROWS)).all(|p| { + let (Some(now), Some(then)) = (self.pages.get(p), before.pages.get(p)) else { return false }; + let start = p * PAGE_ROWS; + now.id == then.id + || now.rows_since(then.id).is_some_and(|written| written.iter().all(|&r| start + r as usize >= rows)) + }) + } + + /// The rows written or added since `before`, in order. For each page that differs from + /// `before`'s, these are the rows it remembers writing since, or all its rows if it does not + /// remember `before`'s page. A page that `before` lacks counts in full. Rows of `before` past + /// this column's end are not included. + pub(crate) fn written_since(&self, before: &Column) -> Vec { + let mut rows = Vec::new(); + for (p, now) in self.pages.iter().enumerate() { + let start = p * PAGE_ROWS; + match before.pages.get(p) { + Some(then) if then.id == now.id => {} + Some(then) => match now.rows_since(then.id) { + Some(written) => rows.extend(written.into_iter().map(|r| start + r as usize)), + None => rows.extend(start..start + now.values.len()), + }, + None => rows.extend(start..start + now.values.len()), + } + } + rows + } + + /// All values in row order as one [`Values`]. A single-page column returns its page without + /// copying. Otherwise the pages are concatenated. pub(crate) fn values(&self) -> Values { match &self.pages[..] { [] => self.like.clone(), @@ -200,9 +269,9 @@ impl Column { } } - /// The rows whose values differ from `before`'s, in order: of the rows written since, while the - /// pages remember them, those that now hold another value; else every row of the pages that - /// changed, compared. + /// The rows whose values differ from `before`'s, in order. When the pages remember the rows + /// written since, only those rows are compared. Otherwise every row of each changed page is + /// compared (see [`differing`](Column::differing)). pub(crate) fn changed_since(&self, before: &Column) -> Vec { let Some(written) = self.rows_since(before) else { return self.differing(before) }; written @@ -211,13 +280,13 @@ impl Column { .collect() } - /// Whether row `row` holds what row `at` of `values` does. + /// Whether row `row` equals row `at` of `values`, as stored (see [`Values::same_row`]). pub(crate) fn holds(&self, row: usize, values: &Values, at: usize) -> bool { self.pages[row / PAGE_ROWS].values.same_row(row % PAGE_ROWS, values, at) } - /// The rows whose values differ from `before`'s, in order. A page both share holds the same values, - /// so only the pages that changed are compared, row by row; rows past the shorter column differ. + /// The rows whose values differ from `before`'s, in order. Pages with the same id are skipped, + /// and the rest are compared row by row. Rows past the shorter column's end count as different. pub(crate) fn differing(&self, before: &Column) -> Vec { let mut rows = Vec::new(); for p in 0..self.pages.len().max(before.pages.len()) { @@ -234,8 +303,8 @@ impl Column { rows } - /// The column of rows `rows`, in that order, missing for a [`HOLE`](super::values::HOLE). Leading - /// pages whose rows are unchanged keep their ids. + /// A column of rows `rows` of this one, in that order. Leading whole pages whose rows are + /// unchanged (`rows[i] == i`) keep their ids. pub(crate) fn gather(&self, rows: &[usize]) -> Column { let kept = rows.iter().enumerate().take_while(|&(i, &r)| i == r).count(); let kept_pages = (kept / PAGE_ROWS).min(rows.len() / PAGE_ROWS); @@ -269,19 +338,6 @@ mod tests { assert_eq!(written.get(PAGE_ROWS + 4), Some(Value::Number((PAGE_ROWS + 4) as f64))); } - #[test] - fn holes_write_and_gather_missing() { - let column = Column::new(&numbers(4)); - let source = Column::new(&Values::from(vec![7.0_f64])); - let written = column.write(5, &[(1, super::super::values::HOLE), (4, 0)], &source).expect("numbers"); - assert_eq!( - (written.get(0), written.get(1), written.get(4)), - (Some(Value::Number(0.0)), None, Some(Value::Number(7.0))) - ); - let gathered = column.gather(&[3, super::super::values::HOLE, 0]); - assert_eq!((gathered.get(0), gathered.get(1)), (Some(Value::Number(3.0)), None)); - } - #[test] fn growth_pads_with_missing() { let column = Column::new(&numbers(3)); @@ -296,13 +352,13 @@ mod tests { fn differing_rows_are_found_in_the_pages_that_changed() { let column = Column::new(&numbers(PAGE_ROWS * 2 + 10)); let source = Column::new(&Values::from(vec![-1.0_f64, (PAGE_ROWS + 4) as f64])); - // Row 3 of the second page changes; its row 4 is written the value it already held. + // Row 3 of the second page changes. Its row 4 is rewritten with the value it already had. let written = column.write(column.len(), &[(PAGE_ROWS + 3, 0), (PAGE_ROWS + 4, 1)], &source); let written = written.expect("numbers"); assert_eq!(written.differing(&column), [PAGE_ROWS + 3]); - assert_eq!(column.differing(&column), Vec::::new(), "a column holds what it holds"); + assert_eq!(column.differing(&column), Vec::::new(), "a column does not differ from itself"); let longer = column.write(column.len() + 2, &[], &source).expect("numbers"); - assert_eq!(longer.differing(&column), [PAGE_ROWS * 2 + 10, PAGE_ROWS * 2 + 11], "rows past it differ"); + assert_eq!(longer.differing(&column), [PAGE_ROWS * 2 + 10, PAGE_ROWS * 2 + 11], "added rows differ"); } #[test] @@ -314,11 +370,11 @@ mod tests { assert_eq!(twice.rows_since(&column), Some(vec![3, 5, PAGE_ROWS + 4]), "every write since, in order"); assert_eq!(twice.rows_since(&once), Some(vec![5])); assert_eq!(twice.rows_since(&twice), Some(Vec::new())); - assert_eq!(column.rows_since(&twice), None, "nothing is written backward"); + assert_eq!(column.rows_since(&twice), None, "an older column is not written from a newer one"); assert_eq!(Column::new(&numbers(PAGE_ROWS + 10)).rows_since(&column), None, "nor from another column"); let grown = column.write(column.len() + 2, &[], &source).expect("numbers"); - assert_eq!(grown.rows_since(&column), None, "rows added change the rows themselves"); - // A page remembers a few writes back; past them, it reads as written whole. + assert_eq!(grown.rows_since(&column), None, "a changed row count lists no rows"); + // A page remembers a few writes back. Beyond that it counts as entirely rewritten. let mut near = once.clone(); for _ in 1..REMEMBERED { near = near.write(near.len(), &[(7, 0)], &source).expect("numbers"); @@ -330,7 +386,34 @@ mod tests { } #[test] - fn rows_written_the_value_they_held_have_not_changed() { + fn rows_written_or_added_since_are_found_page_by_page() { + let column = Column::new(&numbers(PAGE_ROWS + 10)); + let source = Column::new(&Values::from(vec![-1.0_f64])); + let written = column.write(PAGE_ROWS + 12, &[(3, 0)], &source).expect("numbers"); + assert_eq!( + written.written_since(&column), + [3, PAGE_ROWS + 10, PAGE_ROWS + 11], + "the row written, and the rows added past the end" + ); + let fresh = Column::new(&numbers(PAGE_ROWS + 10)); + assert_eq!(fresh.written_since(&column).len(), PAGE_ROWS + 10, "an unrelated page counts every row"); + assert_eq!(column.written_since(&column), Vec::::new(), "nothing written since itself"); + } + + #[test] + fn appends_leave_earlier_rows_unchanged() { + let column = Column::new(&numbers(PAGE_ROWS + 10)); + let source = Column::new(&Values::from(vec![-1.0_f64])); + let appended = column.write(PAGE_ROWS + 12, &[(PAGE_ROWS + 11, 0)], &source).expect("numbers"); + assert!(appended.kept_below(&column, PAGE_ROWS + 10), "rows appended after them leave them unchanged"); + let written = appended.write(appended.len(), &[(PAGE_ROWS + 9, 0)], &source).expect("numbers"); + assert!(written.kept_below(&appended, PAGE_ROWS + 9), "a row written after the rows checked"); + assert!(!written.kept_below(&appended, PAGE_ROWS + 10), "a row written within them"); + assert!(!Column::new(&numbers(PAGE_ROWS + 10)).kept_below(&column, 1), "an unrelated column keeps no rows"); + } + + #[test] + fn rows_rewritten_with_the_same_value_are_unchanged() { let column = Column::new(&numbers(PAGE_ROWS + 10)); // Row 4 holds 4. let source = Column::new(&Values::from(vec![-1.0_f64, 4.0])); @@ -338,7 +421,21 @@ mod tests { assert_eq!(written.rows_since(&column), Some(vec![3, 4, PAGE_ROWS + 4]), "every row written"); assert_eq!(written.changed_since(&column), vec![3, PAGE_ROWS + 4], "the rows whose values changed"); let far = (0..REMEMBERED).fold(written, |c, _| c.write(c.len(), &[(3, 0)], &source).expect("numbers")); - assert_eq!(far.changed_since(&column), vec![3, PAGE_ROWS + 4], "compared where the pages forgot"); + assert_eq!(far.changed_since(&column), vec![3, PAGE_ROWS + 4], "compared once pages forget their writes"); + } + + #[test] + fn a_page_counts_its_points_once_and_a_rewritten_page_recounts() { + let path = |n: usize| (0..n).map(|i| [i as f64, 0.0]).collect::>(); + let column = Column::new(&Values::from(vec![path(3), path(5)])); + let page = &column.pages()[0]; + assert_eq!(page.points.get(), None, "nothing counted until asked"); + assert_eq!(page.points(), 8); + assert_eq!(page.points.get(), Some(&8), "counted once, then cached"); + assert_eq!(column.clone().pages()[0].points.get(), Some(&8), "a copy of the column shares the count"); + let source = Column::new(&Values::from(vec![path(10)])); + let written = column.write(column.len(), &[(0, 0)], &source).expect("paths"); + assert_eq!(written.pages()[0].points(), 15, "the rewritten page counts its own points"); } #[test] diff --git a/crates/latkit/src/data/describe.rs b/crates/latkit/src/data/describe.rs index 922c7916..73809696 100644 --- a/crates/latkit/src/data/describe.rs +++ b/crates/latkit/src/data/describe.rs @@ -1,59 +1,82 @@ -//! Reading data back without moving it: what a dataset holds, and a sampled field's extremes over -//! spans of its axis. +//! Summaries of a dataset. A [`Description`] lists its tables and fields. An [`Envelope`] holds +//! each row's least and greatest value of a sampled field over spans of the axis. use super::column::{PAGE_BITS, PAGE_MASK}; -use super::dataset::Data; use super::ids::Ids; use super::name::Name; +use super::snapshot::Snapshot; use super::table::{Field, Table}; use super::values::Kind; use crate::{Error, Result}; -/// The most values an envelope holds, least and greatest each: rows times spans. +/// The largest envelope allowed, as rows times spans. `least` and `greatest` each hold this many +/// values at most. const ENVELOPE_VALUES: usize = 1 << 26; -/// What a dataset holds: its tables, their rows and fields, and the axis its sampled fields are -/// recorded over and span. +/// A summary of a dataset: its tables with their rows and fields, the axis sampled fields are +/// recorded over, and the part of the axis they cover. #[derive(Clone, PartialEq, Debug, serde::Serialize, schemars::JsonSchema)] pub struct Description { /// The dataset's name. pub name: Name, - /// The axis's name and unit. + /// The axis name and unit, if set. pub axis: Option<[Name; 2]>, - /// The first and last coordinates its sampled fields hold. + /// The first and last axis coordinates held by any sampled field, or `None` if no field holds + /// any. pub span: Option<[f64; 2]>, - /// Its tables, by name. + /// Its tables, sorted by name. pub tables: Vec, } -/// A table described: its rows, and its fields by name. +/// A summary of one table: its row count and its fields, sorted by name. #[derive(Clone, PartialEq, Debug, serde::Serialize, schemars::JsonSchema)] pub struct TableDescription { /// The table's name. pub name: Name, - /// How many rows it has. + /// The number of rows. pub rows: usize, - /// Its fields, by name. + /// Its fields, sorted by name. pub fields: Vec, } -/// A field described: what it holds, the table a reference names, the coordinates samples span. +/// A summary of one field: its kind, plus its target table or axis span where those apply. #[derive(Clone, PartialEq, Debug, serde::Serialize, schemars::JsonSchema)] pub struct FieldDescription { /// The field's name. pub name: Name, - /// What it holds. + /// The kind of values it holds. pub kind: Kind, - /// The table a reference names. + /// For a reference, the target table. #[serde(skip_serializing_if = "Option::is_none")] pub to: Option, - /// The first and last coordinates a sampled field holds. + /// For a sampled field, the first and last axis coordinates it holds. #[serde(skip_serializing_if = "Option::is_none")] pub span: Option<[f64; 2]>, } -impl Data { - /// What the dataset holds: tables, rows, fields and their kinds, the axis and its span. +/// A request for each row's least and greatest value of a sampled field over equal spans of the +/// axis. It reads field `field` of table `table` over `spans` equal spans from `range[0]` to +/// `range[1]`, for every row or for the rows in `ids`, in that order. Like a query, it is plain +/// data that deserializes from JSON. +#[derive(Clone, PartialEq, Debug, serde::Deserialize, schemars::JsonSchema)] +#[serde(deny_unknown_fields)] +pub struct EnvelopeRequest { + /// The table to read. + pub table: Name, + /// The sampled field to read. + pub field: Name, + /// The axis range the spans cover, `[first, last]`. Both are finite, with `first <= last`. + pub range: [f64; 2], + /// The number of equal spans to divide the range into. At least 1. + pub spans: usize, + /// The rows to read, in this order. Defaults to every row. + #[serde(default)] + pub ids: Option, +} + +impl Snapshot { + /// A summary of the dataset: its tables, rows, fields and their kinds, and the axis and its + /// span. pub fn describe(&self) -> Description { Description { name: self.name().clone(), @@ -62,10 +85,37 @@ impl Data { tables: self.tables().map(Table::describe).collect(), } } + + /// Each row's least and greatest value of a sampled field over equal spans of the axis, as + /// `request` describes. Values are laid out row by row, with NaN where a span holds no frame. + /// + /// Each span reads the coarsest stored levels of detail that fit inside it, so it costs a few + /// reads however many frames it covers. Where only coarser levels are present, such as while + /// time tiles load, those are used and may include frames just outside the span. + /// + /// Returns an error if the table, the field, or a requested row does not exist, the field is + /// not sampled, the range is not finite and ordered, `spans` is 0, or rows times spans exceeds + /// 2^26. + /// + /// ``` + /// # use latkit::{Data, EnvelopeRequest, Ids, Patch, Table}; + /// let grid = Data::new("grid"); + /// grid.apply(Table::new("Bus", Ids::count(1)))?; + /// grid.apply(Patch::new().append("Bus", [0.0, 1.0, 2.0, 3.0], [("vm", vec![1.0, 0.9, 1.1, 1.0])]))?; + /// let request = EnvelopeRequest { table: "Bus".into(), field: "vm".into(), range: [0.0, 4.0], spans: 2, ids: None }; + /// let envelope = grid.snapshot().envelope(&request)?; + /// assert_eq!(envelope.least, [0.9, 1.0]); + /// # Ok::<(), latkit::Error>(()) + /// ``` + pub fn envelope(&self, request: &EnvelopeRequest) -> Result { + let EnvelopeRequest { table, field, range, spans, ids } = request; + let rows = self.table(table).ok_or_else(|| Error::invalid("no such table").at(table.as_str()))?; + rows.envelope(field, *range, *spans, ids.as_ref()).map_err(|e| e.at(table.as_str())) + } } impl Table { - /// What the table holds: its rows, and its fields' kinds, targets, and spans. + /// A summary of the table: its row count and each field's kind, reference target, and span. pub fn describe(&self) -> TableDescription { let fields = self.fields.iter().map(|(name, field)| FieldDescription { name: name.clone(), @@ -83,16 +133,13 @@ impl Table { } /// Each row's least and greatest value of sampled field `field` over `spans` equal spans of the - /// axis from `range[0]` to `range[1]`: of every row, or of the rows `ids` names, in its order. A - /// span reads the coarsest levels of detail the field holds that fit inside it, so it costs a few - /// reads, however many frames it covers; where the field holds only coarser ones, as time tiles - /// still loading leave it, those stand in. - pub fn envelope(&self, field: &str, range: [f64; 2], spans: usize, ids: Option<&Ids>) -> Result { + /// axis from `range[0]` to `range[1]`. See [`Snapshot::envelope`]. + pub(crate) fn envelope(&self, field: &str, range: [f64; 2], spans: usize, ids: Option<&Ids>) -> Result { let Some(Field::Sampled(samples, _)) = self.field_named(field) else { return Err(Error::invalid("not a sampled field").at(field)); }; if !(range[0].is_finite() && range[1].is_finite() && range[0] <= range[1]) || spans == 0 { - return Err(Error::invalid("an envelope spans a finite range, in one span or more")); + return Err(Error::invalid("an envelope needs a finite, ordered range and at least one span")); } let rows: Vec = match ids { None => (0..self.len()).collect(), @@ -102,18 +149,19 @@ impl Table { }; if rows.len().saturating_mul(spans) > ENVELOPE_VALUES { return Err(Error::Limit( - format!("an envelope holds at most {ENVELOPE_VALUES} values: rows times spans").into(), + format!("an envelope holds at most {ENVELOPE_VALUES} values (rows times spans)").into(), )); } let width = (range[1] - range[0]) / spans as f64; let at: Vec = (0..=spans).map(|s| if s == spans { range[1] } else { range[0] + width * s as f64 }).collect(); - // A span holds the frames from its start up to the next span's; the last holds its end too. + // Each span covers frames from its start up to the next span's start. The last span also + // includes its end. let bounds: Vec = at.iter().enumerate().map(|(s, &t)| samples.bound(t, s == spans)).collect(); let pieces: Vec> = bounds.windows(2).map(|b| samples.pieces(b[0]..b[1])).collect(); let mut least = vec![f32::NAN; rows.len() * spans]; let mut greatest = least.clone(); - // Rows by block, so each piece's page is read once for all of a block's rows. + // Group rows by row block, so each piece's page is read once for all of a block's rows. let mut by_block: Vec<(usize, usize)> = rows.iter().enumerate().map(|(i, &row)| (row >> PAGE_BITS, i)).collect(); by_block.sort_unstable(); @@ -136,13 +184,17 @@ impl Table { } } -/// Rows' least and greatest values of a sampled field over spans of its axis. +/// Each row's least and greatest value of a sampled field over spans of the axis, as +/// [`Snapshot::envelope`] returns them. #[derive(Clone, PartialEq, Debug, serde::Serialize)] pub struct Envelope { - /// Where each span starts, then where the last ends: one more coordinate than spans. + /// Each span's start coordinate, then the last span's end. It has one more value than there + /// are spans. pub at: Vec, - /// Each row's least value in each span, row after row: NaN where it has none. + /// The least value of each requested row in each span, row-major (`row * spans + span`). NaN + /// where the span has no value. pub least: Vec, - /// Each row's greatest value in each span, row after row: NaN where it has none. + /// The greatest value of each requested row in each span, row-major (`row * spans + span`). + /// NaN where the span has no value. pub greatest: Vec, } diff --git a/crates/latkit/src/data/forms.rs b/crates/latkit/src/data/forms.rs index a62d1ef9..f39f1c28 100644 --- a/crates/latkit/src/data/forms.rs +++ b/crates/latkit/src/data/forms.rs @@ -1,6 +1,6 @@ -//! What JSON writes for data, decoded once: a patch's changes, a table's fields, values, and ids. A -//! slice of a file is the one form read later, by [`Patch::load`] through a host; everything else is -//! decoded as it is read. JavaScript writes the byte frames instead. +//! Serde forms for data written as JSON: a patch's changes, a table's fields, values, and ids. +//! Everything decodes in one pass except slices of binary files, which [`Patch::read`] fetches +//! through a host afterwards. JavaScript sends binary frames, not JSON. use std::fmt; use std::sync::Arc; @@ -12,18 +12,16 @@ use serde::Deserialize; use serde::de::{self, DeserializeSeed, Deserializer, IgnoredAny, MapAccess, SeqAccess, Visitor}; use super::ids::Ids; -use super::name::Id; +use super::name::{Id, Name}; +use super::ordered::Ordered; use super::patch::{Op, Patch}; use super::table::Table; use super::values::{Value, Values}; use crate::{Error, Result}; -/// The largest whole number a number names an id by when written with a fraction: 2^53 − 1. -const SAFE: f64 = 9_007_199_254_740_991.0; +const NUMERIC_ID: &str = "a numeric id must be a whole number from 0 to 2^53 - 1"; -const NUMERIC_ID: &str = "a numeric id is a whole number from 0 to 2^53 - 1"; - -/// A dataset: its name and the patch that builds it. +/// A dataset's JSON document: its name and the patch that builds it. #[derive(Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] pub(crate) struct DataForm { @@ -32,23 +30,23 @@ pub(crate) struct DataForm { pub(crate) patch: Vec, } -/// One change of a patch, named by the method that makes it. +/// One change in a patch, keyed by the name of the `Patch` method that makes it. #[derive(Deserialize, JsonSchema)] #[serde(rename_all = "lowercase", deny_unknown_fields)] #[schemars(rename = "Change")] pub(crate) enum OpForm { - /// Names the axis sampled fields are recorded over, such as `time` in `s`. + /// Sets the axis that sampled fields are recorded over, such as `time` in `s`. Axis { name: String, unit: String }, /// Replaces a table with these rows and fields, or adds it. Replace(TableForm), - /// Writes fields for these rows: rows with known ids change, others are added, and fields left - /// out keep their values. + /// Writes fields for these rows. Rows with existing ids are updated, new ids add rows, and + /// fields left out keep their values. A missing table is added. Set(TableForm), - /// Removes the rows with these ids. + /// Removes the rows with these ids. Ids that match no row are ignored. Remove { table: String, ids: IdsForm }, - /// Records a frame at each axis coordinate of `at`: for each field, a value per row at every - /// coordinate, frame-major; of every row, or of the rows `ids` names, the rest holding their last - /// values. + /// Records a frame at each axis coordinate in `at`. Each field gives one value per row per + /// coordinate, frame-major. Covers every row, or only the rows in `ids`. Other rows keep their + /// last values. Append { at: At, table: String, @@ -58,10 +56,11 @@ pub(crate) enum OpForm { }, /// Forgets every sampled field's frames before axis coordinate `before`. Forget { before: f64 }, - /// Holds a run of sampled field `field`'s entries of level `level`, numbered from `first`: at - /// level 0 frames, each a coordinate in `at` and a value per row in `values`; at level `k` buckets - /// of `8^k` frames, each its first and last frame's coordinates, and each row's least value over - /// each bucket, then each row's greatest. Null is missing. + /// Inserts a run of entries of sampled field `field` at level `level`, numbered from `first`, + /// in the form time tiles store them. At level 0 each entry is a frame, with one coordinate in + /// `at` and one value per row in `values`. At level `k` each entry is a bucket of `8^k` frames. + /// `at` holds the coordinates of each bucket's first and last frame. `values` holds every + /// bucket's least value per row, followed by every bucket's greatest. Null is missing. Frames { table: String, field: String, @@ -93,7 +92,7 @@ pub(crate) enum At { } /// A dataset's JSON document. -impl JsonSchema for super::dataset::Data { +impl JsonSchema for super::source::Data { fn schema_name() -> Cow<'static, str> { "Data".into() } @@ -103,7 +102,7 @@ impl JsonSchema for super::dataset::Data { } } -/// A patch's JSON: its changes in order. +/// A patch's JSON form: its changes, in order. impl JsonSchema for Patch { fn schema_name() -> Cow<'static, str> { "Patch".into() @@ -132,12 +131,12 @@ impl JsonSchema for IdsForm { fn json_schema(generator: &mut SchemaGenerator) -> Schema { let id = generator.subschema_for::>(); json_schema!({ - "description": "Row ids: numbers or texts (null leaves a reference unwired), or `{count}` naming rows 0..count.", + "description": "Row ids: a list of numbers or texts, or `{count}` for rows 0 to count - 1. In a reference, null names no row.", "anyOf": [ { "type": "array", "items": id }, { "type": "object", - "properties": { "count": { "type": "integer", "minimum": 0, "maximum": u32::MAX } }, + "properties": { "count": { "description": "The number of rows, whose ids are 0 to count - 1.", "type": "integer", "minimum": 0, "maximum": u32::MAX } }, "required": ["count"], "additionalProperties": false, }, @@ -208,7 +207,7 @@ impl JsonSchema for Geometry { fn json_schema(generator: &mut SchemaGenerator) -> Schema { let geometry = generator.subschema_for::(); json_schema!({ - "description": "A point `[x, y]` (a third coordinate is read past), or a list of geometries one level shallower: a path, its parts, or polygons.", + "description": "A point `[x, y]` (a third coordinate is ignored), or a list of geometries one level down: a path is a list of points, parts a list of paths, and polygons a list of parts.", "anyOf": [ { "type": "array", "prefixItems": [{ "type": "number" }, { "type": "number" }], "items": { "type": "number" }, "minItems": 2, "maxItems": 3 }, { "type": "array", "items": geometry }, @@ -217,22 +216,22 @@ impl JsonSchema for Geometry { } } -/// A patch deserializes from its JSON: every change but a slice of a file, which [`Patch::load`] -/// reads. +/// Deserializes a patch from its JSON form. A field given as a slice of a binary file is an error +/// here, since only [`Patch::read`] fetches slices. impl<'de> Deserialize<'de> for Patch { fn deserialize>(deserializer: D) -> std::result::Result { patch(Vec::::deserialize(deserializer)?).map_err(de::Error::custom) } } -/// Ids deserialize from a list of numbers and texts, or `{count}`. +/// Deserializes ids from a list of numbers or texts, or from `{count}`. impl<'de> Deserialize<'de> for Ids { fn deserialize>(deserializer: D) -> std::result::Result { IdsForm::deserialize(deserializer).map(|form| form.0) } } -/// An id is a whole number from 0, or a text. +/// An id's schema: a non-negative whole number, or a text. impl schemars::JsonSchema for Id { fn schema_name() -> std::borrow::Cow<'static, str> { "Id".into() @@ -246,48 +245,51 @@ impl schemars::JsonSchema for Id { } } -/// An id deserializes from a whole number from 0 to 2^53 − 1, or a text. +/// Deserializes an id from a non-negative whole number or a text. A number parsed as a float must +/// be at most 2^53 − 1. Null is an error. impl<'de> Deserialize<'de> for Id { fn deserialize>(deserializer: D) -> std::result::Result { - Key::deserialize(deserializer)?.0.ok_or_else(|| de::Error::custom("an id is a whole number or a text")) + Key::deserialize(deserializer)?.0.ok_or_else(|| de::Error::custom("an id must be a whole number or a text")) } } impl OpForm { - /// The columns its fields are written as. - pub(crate) fn columns(&mut self) -> &mut [(String, Column)] { + /// The field columns of a replace, set, or append, or empty for other changes. + pub(crate) fn columns(&mut self) -> &mut [(Name, Column)] { match self { OpForm::Replace(TableForm { fields, .. }) | OpForm::Set(TableForm { fields, .. }) - | OpForm::Append { fields, .. } => &mut fields.0, + | OpForm::Append { fields, .. } => fields.columns(), _ => &mut [], } } } -/// The patch `ops` make, every slice of a file read already. +/// Builds the patch for `ops`. Every slice of a file must already be replaced by its values. Errors +/// name the part like JSON reading, such as `[2].set.fields.x`. pub(crate) fn patch(ops: Vec) -> Result { let mut patch = Patch::new(); for (i, op) in ops.into_iter().enumerate() { let op = match op { OpForm::Axis { name, unit } => Op::Axis { name: name.into(), unit: unit.into() }, - OpForm::Replace(form) => Op::Replace(form.table().map_err(|e| e.at(i))?), - OpForm::Set(form) => Op::Set(form.table().map_err(|e| e.at(i))?), + OpForm::Replace(form) => Op::Replace(form.table().map_err(|e| e.at("replace").at(i))?), + OpForm::Set(form) => Op::Set(form.table().map_err(|e| e.at("set").at(i))?), OpForm::Remove { table, ids } => Op::Remove { table: table.into(), ids: ids.0 }, OpForm::Append { at, table, ids, fields } => { let at = match at { At::One(at) => vec![at], At::Many(at) => at, }; - let mut columns = Vec::with_capacity(fields.0.len()); - for (name, column) in fields.0 { + let fields = fields.into_columns(); + let mut columns = Vec::with_capacity(fields.len()); + for (name, column) in fields { let values = match column { Column::Reference { .. } => Err(Error::invalid("append records values, not references")), column => column.values(), }; columns.push(( name.as_str().into(), - values.map_err(|e| e.at(name.as_str()).at(table.as_str()).at(i))?, + values.map_err(|e| e.at(name.as_str()).at("fields").at("append").at(i))?, )); } Op::Append { table: table.into(), at, ids: ids.map(|ids| ids.0), fields: columns } @@ -307,11 +309,11 @@ impl TableForm { fn table(self) -> Result { let name = self.table; let mut table = Table::new(name.as_str(), self.ids.0); - for (field, column) in self.fields.0 { + for (field, column) in self.fields.into_columns() { table = match column { Column::Reference { to, ids } => table.reference(field, to, ids), column => { - let values = column.values().map_err(|e| e.at(field.as_str()).at(name.as_str()))?; + let values = column.values().map_err(|e| e.at(field.as_str()).at("fields"))?; table.field(field, values) } }; @@ -320,7 +322,7 @@ impl TableForm { } } -/// A field's column as written: values, a reference, or a slice of a file. +/// A field's column as written in JSON: values, a reference, or a slice of a binary file. pub(crate) enum Column { Values(Values), Reference { to: String, ids: Ids }, @@ -336,25 +338,26 @@ pub(crate) struct Slice { } impl Slice { - /// The byte after the slice. + /// The offset just past the slice's last byte. Errors on overflow. pub(crate) fn end(&self) -> Result { let length = self.count.checked_mul(self.dtype.width()).ok_or_else(|| Error::invalid("the slice is too long"))?; - self.offset.checked_add(length).ok_or_else(|| Error::invalid("the slice ends past every file")) + self.offset.checked_add(length).ok_or_else(|| Error::invalid("the slice's end offset overflows")) } - /// The values its fetched `bytes` hold. + /// Decodes the slice's fetched `bytes`. Errors if their length is not the slice's length. pub(crate) fn read(&self, bytes: &[u8]) -> Result { let length = self.end()? - self.offset; if bytes.len() as u64 != length { let file = &self.file; - return Err(Error::Io(format!("{file}: {} bytes of the slice's {length}", bytes.len()).into())); + return Err(Error::Io(format!("{file}: read {} of the slice's {length} bytes", bytes.len()).into())); } Ok(self.dtype.decode(bytes)) } } -/// How a slice's bytes hold values: little endian; `bool` a byte each, nonzero true. +/// The value type of a slice's bytes. Numbers are little endian. `bool` is one byte each, nonzero +/// meaning true. #[derive(Clone, Copy, Deserialize, JsonSchema)] enum Dtype { #[serde(rename = "f32le")] @@ -382,7 +385,7 @@ impl Dtype { } } - /// The values little-endian `bytes` hold, whole values only. + /// Decodes little-endian `bytes`, ignoring a trailing partial value. fn decode(self, bytes: &[u8]) -> Values { macro_rules! decode { ($t:ty, $n:literal) => { @@ -402,62 +405,46 @@ impl Dtype { } impl Column { - /// The values, read already: a slice of a file is read by [`Patch::load`]. + /// The column's values. Errors for a slice of a file, which only [`Patch::read`] can fetch. fn values(self) -> Result { match self { Column::Values(values) => Ok(values), - Column::Reference { .. } => unreachable!("references are fields of their own"), + Column::Reference { .. } => unreachable!("callers handle reference columns first"), Column::Slice(slice) => { - Err(Error::invalid(format!("{}: a slice of a file is read by Patch::load", slice.file))) + Err(Error::invalid(format!("{}: a slice of a file must be read with Patch::read", slice.file))) } } } } -/// A table's fields in the order written. -#[derive(Default)] -pub(crate) struct Fields(pub(crate) Vec<(String, Column)>); +/// A table's fields in the order written. Each name appears once. +#[derive(Default, Deserialize)] +#[serde(transparent)] +pub(crate) struct Fields(Ordered); -impl<'de> Deserialize<'de> for Fields { - fn deserialize>(deserializer: D) -> std::result::Result { - struct Map; - impl<'de> Visitor<'de> for Map { - type Value = Fields; - fn expecting(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str("an object of fields") - } - fn visit_map>(self, mut map: A) -> std::result::Result { - let mut fields = Vec::new(); - while let Some(name) = map.next_key::()? { - if fields.iter().any(|(n, _)| *n == name) { - return Err(de::Error::custom(format!("field {name} is written twice"))); - } - let column = map.next_value_seed(ColumnSeed(&name))?; - fields.push((name, column)); - } - Ok(Fields(fields)) - } - } - deserializer.deserialize_map(Map) +impl Fields { + pub(crate) fn columns(&mut self) -> &mut [(Name, Column)] { + &mut self.0.0 } -} -/// A column, named for its errors. -struct ColumnSeed<'a>(&'a str); + fn into_columns(self) -> Vec<(Name, Column)> { + self.0.0 + } +} -impl<'de> DeserializeSeed<'de> for ColumnSeed<'_> { - type Value = Column; - fn deserialize>(self, deserializer: D) -> std::result::Result { - deserializer.deserialize_any(ColumnVisitor(self.0)) +/// Reads a column. Errors say only what is wrong. The caller adds the field to the error's path. +impl<'de> Deserialize<'de> for Column { + fn deserialize>(deserializer: D) -> std::result::Result { + deserializer.deserialize_any(ColumnVisitor) } } -struct ColumnVisitor<'a>(&'a str); +struct ColumnVisitor; -impl<'de> Visitor<'de> for ColumnVisitor<'_> { +impl<'de> Visitor<'de> for ColumnVisitor { type Value = Column; fn expecting(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "field {}: an array of values, a reference, or a slice of a file", self.0) + f.write_str("an array of values, a reference, or a slice of a file") } fn visit_seq>(self, mut seq: A) -> std::result::Result { @@ -467,7 +454,7 @@ impl<'de> Visitor<'de> for ColumnVisitor<'_> { } Values::try_from(values).map(Column::Values).map_err(|e| { let reason = if let Error::Invalid { reason, .. } = e { reason } else { e.to_string().into() }; - de::Error::custom(format!("field {}: {reason}", self.0)) + de::Error::custom(reason) }) } @@ -483,28 +470,28 @@ impl<'de> Visitor<'de> for ColumnVisitor<'_> { "count" => count = Some(map.next_value::()?), other => { map.next_value::()?; - return Err(de::Error::custom(format!("field {}: unknown key {other}", self.0))); + return Err(de::Error::custom(format!("unknown key {other}"))); } } } - let missing = |what: &str| de::Error::custom(format!("field {}: {what}", self.0)); + let missing = |what: &str| de::Error::custom(what); match (reference, file) { (Some(to), None) if dtype.is_none() && offset.is_none() && count.is_none() => { - Ok(Column::Reference { to, ids: ids.ok_or_else(|| missing("a reference names its ids"))? }) + Ok(Column::Reference { to, ids: ids.ok_or_else(|| missing("a reference needs ids"))? }) } (None, Some(file)) if ids.is_none() => Ok(Column::Slice(Slice { file, - dtype: dtype.ok_or_else(|| missing("a slice names its dtype"))?, + dtype: dtype.ok_or_else(|| missing("a slice needs a dtype"))?, offset: offset.unwrap_or(0), - count: count.ok_or_else(|| missing("a slice names its count"))?, + count: count.ok_or_else(|| missing("a slice needs a count"))?, })), - _ => Err(missing("an object is {reference, ids} or {file, dtype, offset, count}")), + _ => Err(missing("an object must be {reference, ids} or {file, dtype, offset, count}")), } } } -/// One value of an array column: a number, boolean, text, or `[x, y]` geometry; `null`, or an empty -/// geometry, missing. +/// One value of an array column: a number, boolean, text, or geometry of `[x, y]` points. `null` +/// and an empty geometry are missing. struct Element(Option); impl<'de> Deserialize<'de> for Element { @@ -551,7 +538,8 @@ enum Geometry { } impl Geometry { - /// How deep it nests below its points: 0 a point, 1 a path, 2 parts, 3 polygons. + /// Its nesting depth, following the first item at each level: 0 a point, 1 a path, 2 parts, + /// 3 polygons. fn depth(&self) -> usize { match self { Geometry::Point(_) => 0, @@ -573,7 +561,8 @@ impl Geometry { } } - /// The value it writes: `None` when it is empty. + /// The geometry as a [`Value`], or `None` when it is empty. Errors when nested deeper than + /// polygons. fn value(self) -> std::result::Result, &'static str> { if matches!(&self, Geometry::List(items) if items.is_empty()) { return Ok(None); @@ -583,7 +572,7 @@ impl Geometry { 1 => Value::Path(self.list(Geometry::point)), 2 => Value::Parts(self.list(|path| path.list(Geometry::point))), 3 => Value::Polygons(self.list(|parts| parts.list(|path| path.list(Geometry::point)))), - _ => return Err("geometries nest at most polygons of rings of [x, y] points"), + _ => return Err("geometry nests deeper than polygons of rings of [x, y] points"), })) } } @@ -600,7 +589,7 @@ impl<'de> Visitor<'de> for GeometryVisitor { match first { Part::Number(x) => { let y: f64 = seq.next_element()?.ok_or_else(|| de::Error::custom("a point is [x, y]"))?; - // A third coordinate, elevation in GeoJSON, is read past. + // Skip a third coordinate (GeoJSON elevation) and any after it. while seq.next_element::()?.is_some() {} Ok(Geometry::Point([x, y])) } @@ -655,7 +644,8 @@ impl<'de> Deserialize<'de> for Part { } } -/// Ids as written: numbers or texts with `null` for none, as [`Ids`] takes them, or `{count}`. +/// Ids as written in JSON: a list of numbers or texts with `null` for none, as [`Ids`] takes them, +/// or `{count}`. pub(crate) struct IdsForm(pub(crate) Ids); impl<'de> Deserialize<'de> for IdsForm { @@ -674,7 +664,7 @@ impl<'de> Deserialize<'de> for IdsForm { other => return Err(de::Error::custom(format!("ids: unknown key {other}"))), } } - count.map(|n| IdsForm(Ids::count(n))).ok_or_else(|| de::Error::custom("ids as a count are {count}")) + count.map(|n| IdsForm(Ids::count(n))).ok_or_else(|| de::Error::custom("an ids object must be {count}")) } fn visit_seq>(self, mut seq: A) -> std::result::Result { let mut ids = Vec::with_capacity(seq.size_hint().unwrap_or(0)); @@ -688,7 +678,7 @@ impl<'de> Deserialize<'de> for IdsForm { } } -/// One id as written. +/// One id as written in JSON, with `null` as `None`. struct Key(Option); impl<'de> Deserialize<'de> for Key { @@ -713,11 +703,7 @@ impl<'de> Deserialize<'de> for Key { Ok(Key(Some(Id::Number(n)))) } fn visit_f64(self, v: f64) -> std::result::Result { - if v.fract() == 0.0 && (0.0..=SAFE).contains(&v) { - Ok(Key(Some(Id::Number(v as u64)))) - } else { - Err(E::custom(NUMERIC_ID)) - } + Id::from_number(v).map(|id| Key(Some(id))).ok_or_else(|| E::custom(NUMERIC_ID)) } fn visit_str(self, v: &str) -> std::result::Result { Ok(Key(Some(Id::from(v)))) @@ -732,7 +718,7 @@ mod tests { use crate::{Data, Id, Ids, Patch, Value}; #[test] - fn patches_decode_as_written() { + fn patches_decode_from_json() { let ops = serde_json::json!([ {"replace": {"table": "Bus", "ids": [7, 9], "fields": { "kv": [1.5, 2.5], @@ -742,8 +728,9 @@ mod tests { {"remove": {"table": "Bus", "ids": [9]}}, ]); let patch: Patch = serde_json::from_value(ops).expect("a patch"); - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply(patch).expect("valid"); + let data = data.snapshot(); let bus = data.table("Bus").expect("Bus"); assert_eq!(bus.ids().collect::>(), [Id::Number(7)]); assert_eq!(bus.get(7_u64, "kv"), Some(Value::Number(1.5))); @@ -752,12 +739,12 @@ mod tests { } #[test] - fn forms_refuse_what_they_cannot_read_now() { + fn forms_refuse_file_slices_bare_counts_and_invalid_ids() { let slice = serde_json::json!([{"replace": {"table": "Bus", "ids": {"count": 1}, "fields": { "kv": {"file": "kv.f64", "dtype": "f64le", "count": 1}}}}]); - let error = serde_json::from_value::(slice).expect_err("a slice needs Patch::load").to_string(); - assert!(error.contains("Patch::load"), "{error}"); - assert!(serde_json::from_value::(serde_json::json!(3)).is_err(), "a bare number is no count"); + let error = serde_json::from_value::(slice).expect_err("a slice needs Patch::read").to_string(); + assert!(error.contains("Patch::read"), "{error}"); + assert!(serde_json::from_value::(serde_json::json!(3)).is_err(), "a bare number is not a count"); assert_eq!(serde_json::from_value::(serde_json::json!({"count": 3})).expect("a count").len(), 3); assert_eq!(serde_json::from_value::(serde_json::json!(3)).expect("an id"), Id::Number(3)); assert!(serde_json::from_value::(serde_json::json!(1.5)).is_err()); diff --git a/crates/latkit/src/data/frames.rs b/crates/latkit/src/data/frames.rs index c16a323f..4b3e7da1 100644 --- a/crates/latkit/src/data/frames.rs +++ b/crates/latkit/src/data/frames.rs @@ -1,33 +1,36 @@ -//! Time tiles: a dataset written as the files a view fetches its sampled fields' histories from, in -//! spans, as it asks for them. A span of level `k` holds `chunk` entries of the level: frames at level -//! 0, else buckets of `8^k` frames. Eight spans make the one a level up, as four map tiles make the -//! tile a zoom out, so a span still loading draws the nearest span above it that has landed. +//! Time tiles: a dataset written as files, from which a view fetches the history of sampled fields +//! in spans, on demand. A span at level `k` holds `chunk` entries of that level. The entries are +//! frames at level 0 and buckets of `8^k` frames above it. One span covers the same frames as eight +//! spans of the level below, as one map tile covers four tiles one zoom level in. While a span +//! loads, the view draws the nearest coarser span that has arrived. //! -//! | File | What it holds | -//! | ------------- | ------------------------------------------------------------------------------- | -//! | `frames.json` | the manifest: `data`, `chunk`, `url`, and `frames`, each sampled field's table, | -//! | | name, and the frames it reads, `[first, end]` | -//! | `base.latk` | the dataset with every sampled field reading its frames but holding none | -//! | spans | at `url`, `{field}` the field's place in `frames`: a patch holding the span's | -//! | | entries, one `frames` change for each run of them | +//! | File | Contents | +//! | ------------- | ---------------------------------------------------------------------------- | +//! | `frames.json` | the manifest: `data`, `chunk`, `url`, and `frames`, which lists each sampled | +//! | | field's `table`, `field`, `levels` (highest level stored), and `frames` | +//! | | (`[first, end]`) | +//! | `base.latk` | the dataset with each sampled field's frame range but none of its entries | +//! | spans | at `url`, with `{field}` the field's index in `frames`: a patch with one | +//! | | `frames` change per run of the span's entries | use std::ops::Range; -use super::dataset::Data; use super::name::Name; use super::patch::{Op, Patch}; use super::samples::Run; +use super::snapshot::Snapshot; use super::table::Field; use crate::{Error, Result}; -/// The manifest's name. +/// The manifest's file name. pub(crate) const MANIFEST: &str = "frames.json"; -/// The dataset without its sampled fields' entries. +/// The file name of the dataset without its sampled fields' entries. const BASE: &str = "base.latk"; -/// Where each span lies, beside the manifest. +/// The path template of each span, relative to the manifest. const URL: &str = "{field}/{level}/{index}.latk"; -/// A sampled field a manifest lists: its table and name, and the frames it reads, `[first, end]`. +/// A sampled field listed in a manifest: its table and name, its highest stored level, and the +/// frames it covers, written `[first, end]`. #[derive(Clone, PartialEq, Debug, serde::Serialize, serde::Deserialize)] pub(crate) struct Listed { pub(crate) table: Name, @@ -37,8 +40,8 @@ pub(crate) struct Listed { pub(crate) frames: Range, } -/// What a manifest says: where the dataset without its sampled fields' entries lies, the entries a -/// span holds, where each span lies, and the fields listed. +/// The contents of `frames.json`: the path of the base dataset (`data`), the entries per span +/// (`chunk`), the span path template (`url`), and the listed fields. #[derive(Clone, PartialEq, Debug, serde::Serialize, serde::Deserialize)] pub(crate) struct Manifest { pub(crate) data: String, @@ -48,7 +51,7 @@ pub(crate) struct Manifest { pub(crate) fields: Vec, } -/// Frames `first..end`, written `[first, end]`. +/// Serializes frames `first..end` as `[first, end]`. mod range { use std::ops::Range; @@ -61,29 +64,30 @@ mod range { pub(super) fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result, D::Error> { match <[u64; 2]>::deserialize(d)? { [first, end] if first <= end => Ok(first..end), - _ => Err(de::Error::custom("frames are [first, end], the first at most the end")), + _ => Err(de::Error::custom("frames must be [first, end] with first at most end")), } } } impl Manifest { - /// The manifest `bytes` hold; `None` where they hold none, being no JSON object listing `frames`. + /// Parses a manifest. Returns `None` when `bytes` are not a JSON object with a `frames` list, + /// so other tile formats can be tried. Returns an error when they are such an object but it is + /// invalid. pub(crate) fn from_bytes(bytes: &[u8]) -> Option> { - /// What any manifest holds, whatever else it does. + /// The key every manifest has, used to recognize one. #[derive(serde::Deserialize)] struct Listing { #[serde(rename = "frames")] _frames: Vec, } - let text = std::str::from_utf8(bytes).ok()?; - crate::from_json::(text).ok()?; - let manifest = crate::from_json::(text).and_then(|m| { + crate::json::from_bytes::(bytes).ok()?; + let manifest = crate::json::from_bytes::(bytes).and_then(|m| { if !chunks(m.chunk) { - return Err(Error::invalid("a span holds a multiple of 8 entries, 8 to 2^20").at("chunk")); + return Err(Error::invalid("a span must hold a multiple of 8 entries, from 8 to 2^20").at("chunk")); } for (i, field) in m.fields.iter().enumerate() { if field.levels > super::samples::MAX_LEVEL { - return Err(Error::invalid("sample levels run from 0 to 21").at("levels").at(i).at("frames")); + return Err(Error::invalid("sample levels must be from 0 to 21").at("levels").at(i).at("frames")); } } Ok(m) @@ -91,7 +95,8 @@ impl Manifest { Some(manifest.map_err(|e| e.at(MANIFEST))) } - /// Where span `index` of `level` of the field listed `field`th lies, `url` its template. + /// The path of span `index` at `level` of the field at index `field` in the manifest, from + /// template `url`. pub(crate) fn span(url: &str, field: usize, level: u32, index: u64) -> String { url.replace("{field}", &field.to_string()) .replace("{level}", &level.to_string()) @@ -99,24 +104,30 @@ impl Manifest { } } -/// Whether a span may hold `chunk` entries: a multiple of 8, so a bucket a level up never straddles two. +/// Whether `chunk` is a valid number of entries per span. It must be from 8 to 2^20 and a multiple +/// of 8, so a bucket one level up never straddles two spans. fn chunks(chunk: u32) -> bool { (8..=1 << 20).contains(&chunk) && chunk.is_multiple_of(8) } -impl Data { - /// The dataset as time tiles, each file's path and bytes handed to `write`: `frames.json`, the - /// manifest a view's tiles open; `base.latk`, the dataset with its sampled fields reading their - /// frames but holding none; and a patch for each span of `chunk` entries of each level of each - /// sampled field that holds any, at `{field}/{level}/{index}.latk`. A view given the manifest as - /// [`Tiles`](crate::Tiles) fetches the spans its lines draw, at the level of detail its plot - /// window asks, and those about the coordinate it shows; until a span lands, the span above it - /// draws. +impl Snapshot { + /// Writes the dataset as time tiles, passing each file's path and bytes to `write`: /// - /// The manifest is written last, so a reader finding it finds every file it names. + /// - `frames.json`, the manifest. + /// - `base.latk`, the dataset with each sampled field's frame range but none of its entries. + /// - `{field}/{level}/{index}.latk`, a patch for each span of `chunk` entries, for every level + /// of every sampled field that has entries there. + /// + /// A view given the manifest as [`Tiles`](crate::Tiles) fetches the spans its lines draw, at + /// the level of detail the plot window needs. It also fetches the frames around the coordinate + /// it shows. Until a span arrives, the view draws the coarser span above it. + /// + /// The manifest is written last, so a reader that finds it finds every file it names. Returns + /// an error if + /// `chunk` is not a multiple of 8 from 8 to 2^20, or if `write` fails. pub fn to_tiles(&self, chunk: u32, mut write: impl FnMut(&str, &[u8]) -> Result<()>) -> Result<()> { if !chunks(chunk) { - return Err(Error::invalid("a span holds a multiple of 8 entries, 8 to 2^20").at("chunk")); + return Err(Error::invalid("a span must hold a multiple of 8 entries, from 8 to 2^20").at("chunk")); } let entries_per_span = u64::from(chunk); let mut base = self.clone(); @@ -132,7 +143,8 @@ impl Data { levels: samples.levels(), frames: samples.start()..samples.end(), }); - // Runs come level by level, in order, so each span is whole once the next begins. + // Runs come level by level in entry order, so a span is complete once the next one + // begins. let mut span: Option<((u32, u64), Patch)> = None; for run in samples.runs() { for index in run.entries.start / entries_per_span..run.entries.end.div_ceil(entries_per_span) { @@ -174,18 +186,19 @@ mod tests { use std::collections::BTreeMap; use super::*; - use crate::{Ids, Table}; + use crate::{Data, Ids, Table}; #[test] - fn a_history_writes_as_spans_a_manifest_names() { - let mut data = Data::new("grid"); - data.apply( + fn a_history_writes_as_spans_listed_in_a_manifest() { + let grid = Data::new("grid"); + grid.apply( Patch::new().axis("time", "s").replace(Table::new("Bus", Ids::count(2)).field("kv", vec![1.0, 2.0])), ) .expect("rows"); let at: Vec = (0..100).map(f64::from).collect(); let values: Vec = (0..200).map(|v| f64::from(v % 7)).collect(); - data.apply(Patch::new().append(&at, "Bus", [("vm", values)])).expect("frames"); + grid.apply(Patch::new().append("Bus", &at, [("vm", values)])).expect("frames"); + let data = grid.snapshot(); let mut files = BTreeMap::new(); data.to_tiles(16, |path, bytes| { files.insert(path.to_owned(), bytes.to_vec()); @@ -194,20 +207,21 @@ mod tests { .expect("tiles"); let manifest = Manifest::from_bytes(&files[MANIFEST]).expect("a manifest").expect("valid"); assert_eq!(manifest.fields, [Listed { table: "Bus".into(), field: "vm".into(), levels: 2, frames: 0..100 }]); - // 100 frames: 7 spans of 16 at level 0; 12 buckets of 8 in 1 span at level 1; 1 of 64 at level 2. + // 100 frames: 7 spans of 16 at level 0, 12 buckets of 8 in 1 span at level 1, and 1 bucket + // of 64 at level 2. let spans: Vec<&str> = files .keys() .map(String::as_str) .filter(|p| std::path::Path::new(p).extension().is_some_and(|e| e == "latk") && *p != BASE) .collect(); assert_eq!(spans.len(), 7 + 1 + 1); - let mut back = Data::from_bytes(&files[BASE]).expect("the base"); - assert_eq!(back.describe().span, None, "the base holds no entries"); + let back = Data::from(Snapshot::from_bytes(&files[BASE]).expect("the base")); + assert_eq!(back.snapshot().describe().span, None, "the base holds no entries"); for path in &spans { - back.apply(Patch::from_bytes(&files[*path]).expect("a span")).expect("held once"); + back.apply(Patch::from_bytes(&files[*path]).expect("a span")).expect("each span applies once"); } - assert_eq!(back.to_bytes(), data.to_bytes(), "every span put back makes the history again"); - assert!(Manifest::from_bytes(br#"{"tiles": []}"#).is_none(), "TileJSON is no manifest"); + assert_eq!(back.snapshot().to_bytes(), data.to_bytes(), "applying every span rebuilds the history"); + assert!(Manifest::from_bytes(br#"{"tiles": []}"#).is_none(), "TileJSON is not a manifest"); let backwards = br#"{"data": "b", "chunk": 16, "url": "u", "frames": [{"table": "t", "field": "f", "levels": 0, "frames": [3, 1]}]}"#; let error = Manifest::from_bytes(backwards).expect("a manifest").unwrap_err(); diff --git a/crates/latkit/src/data/geometry.rs b/crates/latkit/src/data/geometry.rs index 25c4705f..b0dc2c68 100644 --- a/crates/latkit/src/data/geometry.rs +++ b/crates/latkit/src/data/geometry.rs @@ -5,9 +5,10 @@ use super::values::Value; /// Shared, immutable geometries of one depth: nested offsets over `[x, y]` coordinates. /// -/// Depth 0 is a point per row, 1 a path, 2 parts (rings of a polygon, or the paths of a multi-path), -/// and 3 polygons. `levels[0]` is indexed by row; each level's offsets index the next level, and the -/// last level indexes `coords`. Offsets are absolute, so slicing rows slices only `levels[0]`. +/// Depth 0 is a point per row, 1 a path, 2 parts (rings of a polygon, or the paths of a +/// multi-path), and 3 polygons. `levels[0]` is indexed by row. Each level's offsets index the next +/// level, and the last level indexes `coords`. Offsets are absolute, so slicing rows slices only +/// `levels[0]`. #[derive(Clone)] pub(crate) struct Geometry { depth: u8, @@ -42,7 +43,8 @@ impl Geometry { Geometry { depth: self.depth, levels, coords } } - /// The span `item` of `level` covers in the next level, or in `coords` from the last level. + /// The range of the next level (or of `coords`, from the last level) that `item` of `level` + /// covers. pub(crate) fn span(&self, level: usize, item: usize) -> Range { let offsets = &self.levels[level]; offsets[item] as usize..offsets[item + 1] as usize @@ -52,8 +54,18 @@ impl Geometry { &self.coords } - /// Whether row `row` holds exactly what `other`'s row `other_row` does: the same parts of the same - /// coordinates, compared by their bits. + /// The number of coordinates its rows cover. For a slice, only its own rows count. + pub(crate) fn points(&self) -> usize { + let Some(rows) = self.levels.first() else { return self.coords.len() }; + let mut span = rows[0] as usize..rows[rows.len() - 1] as usize; + for offsets in &self.levels[1..] { + span = offsets[span.start] as usize..offsets[span.end] as usize; + } + span.len() + } + + /// Whether row `row` equals `other`'s row `other_row`: the same part structure and + /// bit-identical coordinates. pub(crate) fn same_row(&self, row: usize, other: &Geometry, other_row: usize) -> bool { fn same(a: &[[f64; 2]], b: &[[f64; 2]]) -> bool { a.len() == b.len() && a.iter().zip(b).all(|(p, q)| p.map(f64::to_bits) == q.map(f64::to_bits)) @@ -77,7 +89,7 @@ impl Geometry { walk(self, other, 0, row, other_row) } - /// Row `row`'s coordinates, every part's in order. + /// Row `row`'s coordinates, all parts in order. pub(crate) fn row_coords(&self, row: usize) -> &[[f64; 2]] { if self.depth == 0 { return &self.coords[row..=row]; @@ -89,7 +101,7 @@ impl Geometry { &self.coords[span] } - /// A digest of row `row`: its coordinates and how they break into parts. + /// A hash of row `row`: its coordinates and how they split into parts. pub(crate) fn row_hash(&self, row: usize) -> u64 { use std::hash::Hasher; let mut hasher = rustc_hash::FxHasher::default(); @@ -109,25 +121,13 @@ impl Geometry { hasher.finish() } - /// Coordinates the rows span, every part's. - pub(crate) fn coords_spanned(&self) -> usize { - if self.depth == 0 { - return self.len(); - } - let mut span = 0..self.len(); - for offsets in &self.levels { - span = offsets[span.start] as usize..offsets[span.end] as usize; - } - span.len() - } - - /// How many points row `row`'s path takes as shaders read it: every coordinate, and a break - /// between its parts or rings. + /// The number of points shaders read for row `row`: every coordinate, plus one break between + /// consecutive parts or rings. pub(crate) fn row_points(&self, row: usize) -> usize { if self.depth == 0 { return 1; } - // The row's parts or rings: the items of the level above its coordinates. + // Find the row's parts or rings: its items at the level just above the coordinates. let mut span = row..row + 1; for offsets in &self.levels[..self.levels.len() - 1] { span = offsets[span.start] as usize..offsets[span.end] as usize; @@ -149,7 +149,8 @@ impl Geometry { } } - /// Offsets relative to each level's first entry and the coordinates the rows span. + /// Each level's offsets rebased to start at 0, and the coordinates the rows cover, for encoding + /// a slice. pub(crate) fn compact(&self) -> (Vec>, Vec<[f64; 2]>) { let mut builder = GeometryBuilder::new(self.depth); for row in 0..self.len() { @@ -204,6 +205,25 @@ impl GeometryBuilder { } } + /// Appends rows `rows` of `source`, copying each level's offsets and the coordinates in bulk. + pub(crate) fn push_rows(&mut self, source: &Geometry, rows: Range) { + debug_assert_eq!(source.depth, self.depth); + let mut span = rows; + for level in 0..self.levels.len() { + let offsets = &source.levels[level]; + let (first, last) = (offsets[span.start], offsets[span.end]); + // Items already pushed at the next level. This run's offsets continue from there. + let next = match self.levels.get(level + 1) { + Some(below) => below.len() - 1, + None => self.coords.len(), + }; + let next = u32::try_from(next).expect("a geometry page holds at most 2^32 items"); + self.levels[level].extend(offsets[span.start + 1..=span.end].iter().map(|&o| o - first + next)); + span = first as usize..last as usize; + } + self.coords.extend_from_slice(&source.coords[span]); + } + fn copy(&mut self, source: &Geometry, level: usize, item: usize) { let span = source.span(level, item); if level + 1 == source.levels.len() { @@ -216,7 +236,7 @@ impl GeometryBuilder { self.close(level); } - /// An empty row: no coordinates, or a missing point. + /// Appends an empty row: no coordinates, or a NaN point at depth 0. pub(crate) fn push_empty(&mut self) { if self.depth == 0 { self.coords.push([f64::NAN; 2]); @@ -299,6 +319,26 @@ mod tests { assert_eq!(tail.get(1), geometry.get(2)); } + #[test] + fn copying_a_run_matches_copying_each_row() { + let geometry = polygons(); + let (mut one, mut runs) = (GeometryBuilder::new(3), GeometryBuilder::new(3)); + for row in [0, 1, 2, 1, 2] { + one.push_from(&geometry, row); + } + runs.push_rows(&geometry, 0..3); + runs.push_rows(&geometry, 1..3); + let (one, runs) = (one.finish(), runs.finish()); + assert_eq!( + (0..5).map(|row| runs.get(row)).collect::>(), + (0..5).map(|row| one.get(row)).collect::>() + ); + let tail = geometry.slice(1..3); + let mut sliced = GeometryBuilder::new(3); + sliced.push_rows(&tail, 0..2); + assert_eq!(sliced.finish().get(1), geometry.get(2), "a slice's offsets need not start at 0"); + } + #[test] fn compact_parts_validate() { let geometry = polygons().slice(1..3); diff --git a/crates/latkit/src/data/ids.rs b/crates/latkit/src/data/ids.rs index b1796042..d23bde52 100644 --- a/crates/latkit/src/data/ids.rs +++ b/crates/latkit/src/data/ids.rs @@ -1,4 +1,5 @@ use std::hash::BuildHasher; +use std::ops::Range; use hashbrown::HashTable; use hashbrown::hash_table::Entry; @@ -9,11 +10,12 @@ use super::name::{Id, IdRef}; use super::values::{Array, Values}; use crate::{Error, Result}; -/// Row ids, as given to [`Table::new`](crate::Table::new) or [`Patch::remove`](crate::Patch::remove), -/// and the target ids of a reference. +/// Row ids, as given to [`Table::new`](crate::Table::new) or +/// [`Patch::remove`](crate::Patch::remove), or the target ids of a reference. /// /// [`Ids::count`] names rows `0..n`. Lists convert from vectors, arrays, and slices of numbers or -/// texts; for a reference, `None` leaves a row unwired. A bare number is never a count. +/// texts. For a reference, `None` leaves a row unwired. A bare number is never a count. Use +/// [`Ids::count`], or `{"count": n}` in JSON. /// /// ``` /// # use latkit::{Ids, Table}; @@ -29,7 +31,7 @@ pub(crate) enum IdList { Listed(Values), } -/// Ids are equal when they name the same rows in the same order. +/// Ids are equal when they list the same ids in the same order, whether counted or listed. impl PartialEq for Ids { fn eq(&self, other: &Ids) -> bool { self.len() == other.len() && (0..self.len()).all(|i| self.key(i) == other.key(i)) @@ -170,10 +172,11 @@ impl RowIds { } } + /// The id of row `row`, or `None` past the last row. pub(crate) fn key(&self, row: usize) -> Option> { match self { - RowIds::Count(_) => Some(IdRef::Number(row as u64)), - RowIds::Listed(c) => key(&c.pages()[row >> PAGE_BITS].values, row & PAGE_MASK), + RowIds::Count(n) => (row < *n as usize).then_some(IdRef::Number(row as u64)), + RowIds::Listed(c) => c.at(row).and_then(|(values, i)| key(values, i)), } } @@ -181,8 +184,8 @@ impl RowIds { self.key(row).map(IdRef::to_id) } - /// The ids with `added` appended; a count stays a count while the new ids continue it. Ids of - /// another kind than the table's (texts for a table of numbers) are an error. + /// The ids with `added` appended. A count stays a count when the new ids continue it. Errors + /// if `added` holds another kind of id than the table's, such as texts for a table of numbers. pub(crate) fn extend(&self, added: &[Id]) -> Result { if added.is_empty() { return Ok(self.clone()); @@ -195,31 +198,13 @@ impl RowIds { let column = self.listed(); let IdList::Listed(source) = Ids::from(added.to_vec()).0 else { unreachable!("a list of ids") }; if source.kind() != column.kind() { - return Err(Error::invalid("ids of another kind than the table's: numbers and texts do not mix")); + return Err(Error::invalid("ids must be the same kind as the table's ids (numbers or texts)")); } let len = column.len() + added.len(); let writes: Vec<(usize, usize)> = (0..added.len()).map(|k| (column.len() + k, k)).collect(); Ok(RowIds::Listed(column.write(len, &writes, &Column::new(&source))?)) } - /// The ids with `len` rows (at least the current count) and `writes` applied, `(row, row of - /// source)` by row, a hole where the row of source is a [`HOLE`](super::values::HOLE). Pages no - /// write touches keep their ids. - pub(crate) fn write(&self, len: usize, writes: &[(usize, usize)], source: &RowIds) -> Result { - // The ids written, gathered first, so a count's ids are made only for those. - let from: Vec = writes.iter().map(|&(_, s)| s).collect(); - let written = match source { - RowIds::Count(_) => { - let ids: Vec> = - from.iter().map(|&s| (s != super::values::HOLE).then_some(s as u64)).collect(); - Column::new(&Values::from(ids)) - } - RowIds::Listed(column) => column.gather(&from), - }; - let at: Vec<(usize, usize)> = writes.iter().enumerate().map(|(k, &(row, _))| (row, k)).collect(); - Ok(RowIds::Listed(self.listed().write(len, &at, &written)?)) - } - /// The ids of `rows`, in that order. pub(crate) fn gather(&self, rows: &[usize]) -> RowIds { if rows.iter().enumerate().all(|(i, &r)| i == r) { @@ -238,8 +223,8 @@ impl RowIds { } } - /// Every page's numbered ids, when every id is a number: what lookups read without matching each - /// id's kind. + /// Each page's `u64` ids, when every page stores numbers with none missing. Lookups use them to + /// read ids without matching each one's kind. pub(crate) fn numbers(&self) -> Option> { let RowIds::Listed(column) = self else { return None }; column @@ -252,8 +237,7 @@ impl RowIds { .collect() } - /// The first row without an id: a hole, which only the rows a view draws of a table drawn through - /// tiles have. + /// The first row with a missing id. Tables must not have one. pub(crate) fn hole(&self) -> Option { let RowIds::Listed(column) = self else { return None }; let mut start = 0; @@ -268,20 +252,32 @@ impl RowIds { None } - /// How many rows are holes, without an id. - pub(crate) fn holes(&self) -> usize { - let RowIds::Listed(column) = self else { return 0 }; - column.pages().iter().filter_map(|p| p.values.valid.as_ref().map(|v| p.values.len() - v.count())).sum() - } - - /// The id → row index, past holes; an error names a repeated id. + /// Builds the id-to-row index, skipping rows without ids. Errors on a repeated id, naming its + /// row. pub(crate) fn index(&self) -> Result { let RowIds::Listed(_) = self else { return Ok(Index::Identity(self.len() as u32)) }; + let mut table = HashTable::with_capacity(self.len()); + self.insert(&mut table, 0..self.len())?; + Ok(Index::Table(table)) + } + + /// Adds rows `rows` to `table`, skipping rows without ids. Errors on a repeated id. When every + /// id is a number, ids are read page by page without matching each one's kind. + fn insert(&self, table: &mut HashTable, rows: Range) -> Result<()> { if let Some(pages) = self.numbers() { - return numbered(&pages, self.len()); + let id = |r: u32| pages[r as usize >> PAGE_BITS][r as usize & PAGE_MASK]; + for row in rows { + let n = id(row as u32); + match table.entry(hash(IdRef::Number(n)), |&r| id(r) == n, |&r| hash(IdRef::Number(id(r)))) { + Entry::Occupied(_) => return Err(Error::invalid(format!("id {n} appears twice")).at(row)), + Entry::Vacant(slot) => { + slot.insert(row as u32); + } + } + } + return Ok(()); } - let mut table = HashTable::with_capacity(self.len()); - for row in 0..self.len() { + for row in rows { let Some(key) = self.key(row) else { continue }; let eq = |&r: &u32| self.key(r as usize) == Some(key); let rehash = |&r: &u32| hash(self.key(r as usize).expect("indexed rows have ids")); @@ -292,29 +288,12 @@ impl RowIds { } } } - Ok(Index::Table(table)) - } -} - -/// The index of numbered ids, read page by page. -fn numbered(pages: &[&[u64]], len: usize) -> Result { - let id = |r: u32| pages[r as usize >> PAGE_BITS][r as usize & PAGE_MASK]; - let mut table = HashTable::with_capacity(len); - for (p, page) in pages.iter().enumerate() { - for (i, &n) in page.iter().enumerate() { - let row = p << PAGE_BITS | i; - match table.entry(hash(IdRef::Number(n)), |&r| id(r) == n, |&r| hash(IdRef::Number(id(r)))) { - Entry::Occupied(_) => return Err(Error::invalid(format!("id {n} appears twice")).at(row)), - Entry::Vacant(slot) => { - slot.insert(row as u32); - } - } - } + Ok(()) } - Ok(Index::Table(table)) } -/// Finds a row by id without allocating. +/// Finds a row by id without allocating. Counted ids are their own row numbers. Other ids use a +/// hash table of row numbers. #[derive(Debug)] pub(crate) enum Index { Identity(u32), @@ -326,8 +305,43 @@ pub(crate) fn hash(key: IdRef<'_>) -> u64 { } impl Index { - /// The rows numbered ids `ids` name, `unwired` where one names none or is missing, and whether a - /// present id named no row: the loop chosen once for the ids' and the rows' kinds. + /// The index of `ids`, which are this index's ids followed by new ones from row `from` on. + /// Copies this index and adds only the new rows, without rehashing the existing ones. Rebuilds + /// it when counted ids became a list. + pub(crate) fn extended(&self, ids: &RowIds, from: usize) -> Result { + match (self, ids) { + (Index::Identity(_), RowIds::Count(n)) => Ok(Index::Identity(*n)), + (Index::Table(table), RowIds::Listed(_)) => { + let mut table = table.clone(); + ids.insert(&mut table, from..ids.len())?; + Ok(Index::Table(table)) + } + _ => ids.index(), + } + } + + /// The index of `ids`, which are this index's ids without the rows flagged in `doomed`. Drops + /// those rows and renumbers the rest to close the gaps, without rehashing. Rebuilds it for + /// counted ids. + pub(crate) fn without(&self, doomed: &[bool], ids: &RowIds) -> Result { + let (Index::Table(table), RowIds::Listed(_)) = (self, ids) else { return ids.index() }; + let mut moved = Vec::with_capacity(doomed.len()); + let mut next = 0; + for &gone in doomed { + moved.push(next); + next += u32::from(!gone); + } + let mut table = table.clone(); + table.retain(|r| !doomed[*r as usize]); + for r in &mut table { + *r = moved[*r as usize]; + } + Ok(Index::Table(table)) + } + + /// The row number each id in `ids` names, or `unwired` where the id is missing or names no row, + /// plus whether any present id named no row. Picks one loop for the storage of the ids and rows + /// up front, so it does not match storage per id. pub(crate) fn resolve(&self, rows: &RowIds, ids: &Values, unwired: u32) -> (Vec, bool) { let valid = |i: usize| ids.valid.as_ref().is_none_or(|v| v.get(i)); let mut dangling = false; @@ -360,8 +374,8 @@ impl Index { (resolved, dangling) } - /// The row with id `key`. A numbered row is also named by its id in decimal text, as ids too - /// large for a JSON or JavaScript number are written. + /// The row with id `key`. A numeric id can also be given as its decimal text, which is how ids + /// too large for JSON or JavaScript numbers are written. pub(crate) fn get(&self, ids: &RowIds, key: IdRef<'_>) -> Option { let number = |text: &str| text.parse::().ok().and_then(|n| self.get(ids, IdRef::Number(n))); match (self, key) { @@ -389,7 +403,25 @@ mod tests { let index = ids.index().expect("unique"); assert_eq!(index.get(&ids, IdRef::Number(9)), Some(5)); assert_eq!(index.get(&ids, IdRef::Number(2)), Some(2)); - assert!(ids.extend(&[Id::from("a")]).is_err(), "texts do not join numbers"); + assert!(ids.extend(&[Id::from("a")]).is_err(), "text ids cannot extend numeric ids"); + } + + #[test] + fn indexes_follow_rows_added_and_removed() { + let ids = RowIds::new(&Ids::from(vec![10_u64, 20, 30])); + let index = ids.index().expect("unique"); + let more = ids.extend(&[Id::Number(40), Id::Number(50)]).expect("numbers"); + let extended = index.extended(&more, 3).expect("unique"); + assert_eq!(extended.get(&more, IdRef::Number(50)), Some(4)); + assert_eq!(extended.get(&more, IdRef::Number(20)), Some(1)); + assert!(index.extended(&more.extend(&[Id::Number(10)]).expect("numbers"), 5).is_err(), "repeated id"); + // Remove rows 1 and 3. The others move up to close the gaps. + let doomed = [false, true, false, true, false]; + let kept = more.gather(&[0, 2, 4]); + let fewer = extended.without(&doomed, &kept).expect("unique"); + assert_eq!(fewer.get(&kept, IdRef::Number(50)), Some(2)); + assert_eq!(fewer.get(&kept, IdRef::Number(30)), Some(1)); + assert_eq!(fewer.get(&kept, IdRef::Number(20)), None); } #[test] @@ -404,14 +436,4 @@ mod tests { assert_eq!(ids.id(0), Some(Id::from("c"))); assert_eq!(RowIds::Count(4).gather(&[0, 2]).id(1), Some(Id::Number(2))); } - - #[test] - fn holes_have_no_id_and_index_past() { - let ids = RowIds::new(&Ids::from(vec!["a", "b", "c"])).gather(&[2, super::super::values::HOLE, 0]); - assert_eq!((ids.id(1), ids.hole()), (None, Some(1))); - let index = ids.index().expect("unique past the hole"); - assert_eq!(index.get(&ids, IdRef::Text("a")), Some(2)); - let numbered = RowIds::Count(3).gather(&[1, super::super::values::HOLE]); - assert_eq!(numbered.index().expect("unique").get(&numbered, IdRef::Number(1)), Some(0)); - } } diff --git a/crates/latkit/src/data/json.rs b/crates/latkit/src/data/json.rs index 700e6501..846a3973 100644 --- a/crates/latkit/src/data/json.rs +++ b/crates/latkit/src/data/json.rs @@ -1,72 +1,151 @@ -//! Datasets and patches as JSON: the format is documented on [`Patch::load`], its forms in -//! `forms.rs`; slices of binary files are fetched through a host. +//! Reads datasets and patches from latkit's binary format, GeoJSON, or JSON. The format is detected +//! from the first bytes. [`Patch::read`] documents the JSON format, and `forms.rs` holds its serde +//! forms. A host fetches slices of binary files. use serde::Deserialize; -use super::dataset::Data; use super::forms::{Column, DataForm, OpForm, patch}; use super::name::Name; use super::ordered::Ordered; use super::patch::Patch; +use super::snapshot::Snapshot; +use super::source::Data; use crate::resource::{Fetch, Request}; use crate::{Error, Result}; +/// Options for [`Data::read`] and [`Patch::read`]. They give the dataset's name, the document's +/// location, and the host that fetches the file slices it references. +#[derive(Clone, Copy, Default)] +pub struct Reading<'a> { + /// The dataset's name. Required for GeoJSON. latkit's bytes and JSON documents carry their own + /// name, which this overrides when set. + pub name: Option<&'a str>, + /// The document's own path or URL. Slice file paths resolve in its directory, unless it is + /// empty or the file path is absolute. + pub base: &'a str, + /// Fetches the file slices a document references. Required when it references any. + pub host: Option<&'a dyn Fetch>, +} + impl Patch { - /// A patch from its JSON: what any language writes without latkit. The patch is a list of the - /// changes [`Patch`]'s methods make, each named by its method with its parameters by name. + /// Reads a patch from `bytes`. The bytes are either one frame of latkit's binary format, as + /// [`Patch::to_bytes`] writes it, or JSON, which any language can write without latkit. The + /// JSON is a list of changes. Each change is an object keyed by the [`Patch`] method that makes + /// it, with that method's parameters by name. /// /// ```json /// [ /// {"replace": {"table": "Bus", "ids": [101, 205], "fields": {"x": [0, 10], "name": ["North", "South"]}}}, /// {"replace": {"table": "Branch", "ids": {"count": 1}, "fields": {"from": {"reference": "Bus", "ids": [101]}}}}, /// {"axis": {"name": "time", "unit": "s"}}, - /// {"append": {"at": [0, 0.5], "table": "Bus", "fields": {"vm": {"file": "vm.f32", "dtype": "f32le", "count": 4}}}} + /// {"append": {"table": "Bus", "at": [0, 0.5], "fields": {"vm": {"file": "vm.f32", "dtype": "f32le", "count": 4}}}} /// ] /// ``` /// - /// - `replace` and `set` take `{table, ids, fields}`, `remove` `{table, ids}`, `append` `{at, table, - /// ids?, fields}`, `forget` `{before}`, `axis` `{name, unit}`, and `frames` `{table, field, - /// level?, first, at, values}`, as [`Patch::frames`] takes them. - /// - `ids` is a count naming rows `0..n`, or numbers or texts; `null` leaves a reference unwired. - /// - A field is an array of one kind of value, as [`Values`](crate::Values) takes them; `{"reference": table, - /// "ids": ids}`; or a slice of a binary file, `{"file", "dtype", "offset", "count"}`: `count` - /// values from byte `offset` (0 unless given) of dtype `f32le`, `f64le`, `i32le`, `u32le`, - /// `u64le`, `u8` (numbers), or `bool` (a byte each, nonzero true). - /// - `append`'s `at` is one axis coordinate or several, as [`Patch::append`] takes them. - /// - An error names the path to what could not be read, such as `[2].set.fields.x`. + /// - `replace` and `set` take `{table, ids, fields}`. `remove` takes `{table, ids}`. `append` + /// takes `{table, at, fields, ids?}`. `forget` takes `{before}`. `axis` takes + /// `{name, unit}`. `frames` takes `{table, field, level?, first, at, values}`, like + /// [`Patch::frames`]. + /// - `ids` is `{"count": n}` for rows `0..n`, or a list of numbers or texts. `null` leaves a + /// reference unwired. + /// - A field is an array of one kind of value, as [`Values`](crate::Values) accepts, a + /// reference `{"reference": table, "ids": ids}`, or a slice of a binary file + /// `{"file", "dtype", "offset", "count"}`. A slice holds `count` values starting at byte + /// `offset` (default 0). Its dtype is `f32le`, `f64le`, `i32le`, `u32le`, `u64le`, or `u8`, + /// all read as numbers, or `bool`, one byte each with nonzero meaning true. + /// - `append`'s `at` is one axis coordinate or a list, like [`Patch::append`]. + /// - Errors name the path to the part that could not be read, such as `[2].set.fields.x`. /// - /// Slices are fetched through `host`, all at once: a file's path or URL relative to `base`, the - /// document's own, or as it is when `base` is empty or the file's is absolute. + /// `reading.host` fetches the slices, with all requests issued together and paths relative to + /// `reading.base`. Slices without a host are an + /// [`Error::Unsupported`](crate::Error::Unsupported). /// /// ```no_run /// # async fn example(host: &impl latkit::resource::Fetch) -> latkit::Result<()> { - /// let json = r#"[{"append": {"at": 0.5, "table": "Bus", "fields": {"vm": {"file": "vm.f32", "dtype": "f32le", "count": 3}}}}]"#; - /// let patch = latkit::Patch::load(json, "data/grid.json", host).await?; // reads data/vm.f32 + /// use latkit::{Patch, Reading}; + /// let json = br#"[{"append": {"table": "Bus", "at": 0.5, "fields": {"vm": {"file": "vm.f32", "dtype": "f32le", "count": 3}}}}]"#; + /// let patch = Patch::read(json, Reading { base: "data/grid.json", host: Some(host), ..Reading::default() }).await?; // reads data/vm.f32 /// # Ok(()) } /// ``` - pub async fn load(json: &str, base: &str, host: &(impl Fetch + ?Sized)) -> Result { - let ops: Vec = crate::from_json(json)?; - build(ops, base, host).await + pub async fn read(bytes: &[u8], reading: Reading<'_>) -> Result { + if bytes.starts_with(b"LATK") { + return Patch::from_bytes(bytes); + } + let ops: Vec = crate::from_json(text(bytes, "a patch must be latkit's bytes or JSON")?)?; + build(ops, reading).await } } impl Data { - /// A dataset from its JSON, `{"name": ..., "patch": [...]}`: its name and the patch that builds it, - /// as its bytes are. The patch loads as [`Patch::load`] loads one. - pub async fn load(json: &str, base: &str, host: &(impl Fetch + ?Sized)) -> Result { - let form: DataForm = crate::from_json(json)?; - if form.name.is_empty() { - return Err(Error::invalid("a dataset is named").at("name")); - } - let patch = build(form.patch, base, host).await.map_err(|e| e.at("patch"))?; - let mut data = Data::new(form.name); - data.apply(patch)?; - Ok(data) + /// Reads a dataset from `bytes`, detecting the format from the first bytes: + /// + /// - latkit's binary format, as [`Snapshot::to_bytes`] writes it. + /// - GeoJSON, gzip-compressed or not. Features become rows of tables `points`, `lines`, and + /// `polygons` by geometry type, with the geometry in field `geometry` and each property as a + /// field. + /// - A JSON document `{"name": ..., "patch": [...]}` with the dataset's name and the patch that + /// builds it, read like [`Patch::read`]. + /// + /// `reading.name` names the dataset and is required for GeoJSON. The other formats carry their + /// own name, which `reading.name` overrides when set. + /// + /// ``` + /// # use latkit::{Data, Reading}; + /// let city = pollster::block_on(Data::read( + /// br#"{"type": "Feature", "properties": {"name": "Hall"}, "geometry": {"type": "Point", "coordinates": [-97.74, 30.27]}}"#, + /// Reading { name: Some("city"), ..Reading::default() }, + /// ))?; + /// let points = city.snapshot().describe().tables.into_iter().find(|t| t.name == "points"); + /// assert_eq!(points.map(|t| t.rows), Some(1)); + /// # Ok::<(), latkit::Error>(()) + /// ``` + pub async fn read(bytes: &[u8], reading: Reading<'_>) -> Result { + let read = if bytes.starts_with(b"LATK") { + Snapshot::from_bytes(bytes)? + } else if geojson(bytes) { + let name = reading.name.ok_or_else(|| Error::invalid("GeoJSON needs a dataset name").at("name"))?; + return crate::geo::geojson(name, bytes).map(Data::from); + } else { + let form: DataForm = + crate::from_json(text(bytes, "a dataset must be latkit's bytes, GeoJSON, or a JSON document")?)?; + if form.name.is_empty() { + return Err(Error::invalid("a dataset needs a name").at("name")); + } + let patch = build(form.patch, reading).await.map_err(|e| e.at("patch"))?; + Snapshot::new(form.name).applied(patch)? + }; + Ok(Data::from(match reading.name { + Some(name) if name != read.name().as_str() => read.renamed(name), + _ => read, + })) } } -/// Patches by dataset name, in the order written: what one step of a stream of changes carries, as -/// JSON `{"grid": [...], "weather": [...]}`, each patch in [`Patch::load`]'s form. +/// Whether `bytes` look like GeoJSON, not latkit's binary format or a dataset document. That is, +/// they are gzip-compressed, or a JSON object whose first key is neither `name` nor `patch`. +pub(crate) fn geojson(bytes: &[u8]) -> bool { + if bytes.starts_with(&[0x1f, 0x8b]) { + return true; + } + let Some(object) = bytes.trim_ascii_start().strip_prefix(b"{") else { return false }; + let Some(key) = object.trim_ascii_start().strip_prefix(b"\"") else { return false }; + let key = key.split(|&b| b == b'"').next().unwrap_or_default(); + key != b"name" && key != b"patch" +} + +/// `bytes` as JSON text, which must be UTF-8 that opens an object or an array. Anything else, such +/// as an HTML page served for a missing file, is an error that names the `expected` format. +fn text<'a>(bytes: &'a [u8], expected: &'static str) -> Result<&'a str> { + let text = std::str::from_utf8(bytes).map_err(|_| Error::invalid(expected))?; + match text.trim_start().as_bytes().first() { + Some(b'{' | b'[') => Ok(text), + _ => Err(Error::invalid(expected)), + } +} + +/// Patches keyed by dataset name, in the order written, such as one message in a stream of +/// changes. The JSON is `{"grid": [...], "weather": [...]}`, with each patch in the JSON form that +/// [`Patch::read`] reads. #[derive(Default, Deserialize)] pub struct Patches(Ordered>); @@ -83,32 +162,38 @@ impl schemars::JsonSchema for Patches { } impl Patches { - /// Each dataset's patch, in order, their slices fetched through `host` relative to `base`. - pub async fn load(self, base: &str, host: &(impl Fetch + ?Sized)) -> Result> { + /// Builds each dataset's patch, in order, fetching file slices through `reading.host` relative + /// to `reading.base`. + pub async fn read(self, reading: Reading<'_>) -> Result> { let mut patches = Vec::with_capacity(self.0.0.len()); for (name, ops) in self.0.0 { - let patch = build(ops, base, host).await.map_err(|e| e.at(name.as_str()))?; + let patch = build(ops, reading).await.map_err(|e| e.at(name.as_str()))?; patches.push((name, patch)); } Ok(patches) } } -/// The patch `ops` make: their slices of files fetched through `host` at once, relative to `base`, -/// then each change built in order. -async fn build(mut ops: Vec, base: &str, host: &(impl Fetch + ?Sized)) -> Result { +/// Builds the patch for `ops`. Issues every file slice request through `reading.host`, relative to +/// `reading.base`, before awaiting any. Then builds each change in order. +async fn build(mut ops: Vec, reading: Reading<'_>) -> Result { let mut slices: Vec<&mut Column> = ops .iter_mut() .flat_map(|op| op.columns().iter_mut().map(|(_, column)| column)) .filter(|column| matches!(column, Column::Slice(_))) .collect(); - let fetching = slices - .iter() - .map(|column| { - let Column::Slice(slice) = &**column else { unreachable!("slices only") }; - Ok(host.fetch(&Request::range(crate::url::resolve(base, &slice.file), slice.offset..slice.end()?))) - }) - .collect::>>()?; + let fetching = match reading.host { + Some(host) => slices + .iter() + .map(|column| { + let Column::Slice(slice) = &**column else { unreachable!("slices only") }; + let range = slice.offset..slice.end()?; + Ok(host.fetch(&Request::range(crate::url::resolve(reading.base, &slice.file), range))) + }) + .collect::>>()?, + None if slices.is_empty() => Vec::new(), + None => return Err(Error::Unsupported("fetching a document's file slices needs a host".into())), + }; for (column, bytes) in slices.iter_mut().zip(fetching) { let Column::Slice(slice) = &**column else { unreachable!("slices only") }; let values = slice.read(&bytes.await?)?; @@ -125,7 +210,7 @@ mod tests { use super::*; use crate::{Kind, Pending, Row, Value}; - /// A host fetching from files in memory, and counting what it fetched. + /// A host serving files from memory and recording each request. #[derive(Default)] struct Files(HashMap>, std::sync::Mutex>); @@ -140,8 +225,11 @@ mod tests { } } - fn load_data(json: &str, host: &Files) -> Result { - pollster::block_on(Data::load(json, "docs/grid.json", host)) + fn read_data(json: &str, host: &Files) -> Result { + pollster::block_on(Data::read( + json.as_bytes(), + Reading { base: "docs/grid.json", host: Some(host), name: None }, + )) } #[test] @@ -152,51 +240,75 @@ mod tests { "at": [[0, 1], [2, 3, 9]], "path": [[[0, 0], [1, 1]], []]}}}, {"replace": {"table": "Branch", "ids": {"count": 1}, "fields": {"from": {"reference": "Bus", "ids": ["a"]}}}} ]}"#; - let data = load_data(json, &Files::default()).unwrap(); - let bus = data.table("Bus").unwrap(); - assert_eq!(bus.get("b", "x"), Some(Value::Number(1.5))); - assert_eq!(bus.get("a", "x"), None); - assert_eq!(bus.get("a", "on"), Some(Value::Bool(false))); - assert_eq!(bus.get("b", "name"), Some(Value::Text("B".into()))); - assert_eq!(bus.get("a", "at"), Some(Value::Point([2.0, 3.0]))); - assert_eq!(bus.kind("path"), Some(Kind::Path)); - assert_eq!(bus.get("b", "path"), Some(Value::Path(vec![[0.0, 0.0], [1.0, 1.0]]))); - let row = Row { data: "grid".into(), table: "Branch".into(), id: 0_u64.into() }; + let data = read_data(json, &Files::default()).unwrap().snapshot(); + let bus = |id: &str, field: &str| data.get("Bus", id, field, None); + assert_eq!(bus("b", "x"), Some(Value::Number(1.5))); + assert_eq!(bus("a", "x"), None); + assert_eq!(bus("a", "on"), Some(Value::Bool(false))); + assert_eq!(bus("b", "name"), Some(Value::Text("B".into()))); + assert_eq!(bus("a", "at"), Some(Value::Point([2.0, 3.0]))); + assert_eq!(bus("b", "path"), Some(Value::Path(vec![[0.0, 0.0], [1.0, 1.0]]))); + let kinds = data.describe().tables.into_iter().flat_map(|t| t.fields).find(|f| f.name == "path"); + assert_eq!(kinds.map(|f| f.kind), Some(Kind::Path)); let target = Row { data: "grid".into(), table: "Bus".into(), id: "a".into() }; - assert_eq!(data.get(&row, "from"), Some(Value::Row(target))); + assert_eq!(data.get("Branch", 0_u64, "from", None), Some(Value::Row(target))); } #[test] - fn slices_fetch_at_once_relative_to_the_document_and_append_frames() { + fn slices_fetch_together_relative_to_the_document_and_append_frames() { let floats: Vec = [9_f32, 1.0, 2.0, 3.0, 4.0].into_iter().flat_map(f32::to_le_bytes).collect(); let host = Files([("docs/v.f32".to_owned(), floats)].into(), std::sync::Mutex::default()); let json = r#"{"name": "grid", "patch": [ {"replace": {"table": "Bus", "ids": {"count": 2}, "fields": {"kv": {"file": "v.f32", "dtype": "f32le", "offset": 4, "count": 2}}}}, {"axis": {"name": "time", "unit": "s"}}, - {"append": {"at": [0, 1], "table": "Bus", "fields": {"vm": {"file": "v.f32", "dtype": "f32le", "offset": 4, "count": 4}}}}, + {"append": {"table": "Bus", "at": [0, 1], "fields": {"vm": {"file": "v.f32", "dtype": "f32le", "offset": 4, "count": 4}}}}, {"forget": {"before": -1}} ]}"#; - let data = load_data(json, &host).unwrap(); + let data = read_data(json, &host).unwrap().snapshot(); let asked: Vec = host.1.lock().unwrap().clone(); assert_eq!(asked, [Request::range("docs/v.f32", 4..12), Request::range("docs/v.f32", 4..20)]); - assert_eq!(data.span(), Some([0.0, 1.0])); - let row = |id: u64| Row { data: "grid".into(), table: "Bus".into(), id: id.into() }; - assert_eq!(data.get(&row(1), "kv"), Some(Value::Number(2.0))); - assert_eq!(data.get_at(&row(1), "vm", 1.0), Some(Value::Number(4.0))); + assert_eq!(data.describe().span, Some([0.0, 1.0])); + assert_eq!(data.get("Bus", 1_u64, "kv", None), Some(Value::Number(2.0))); + assert_eq!(data.get("Bus", 1_u64, "vm", 1.0), Some(Value::Number(4.0))); + } + + #[test] + fn the_format_is_detected_from_the_first_bytes() { + let bytes = br#"{"type": "Feature", "properties": {}, "geometry": {"type": "Point", "coordinates": [1, 2]}}"#; + assert!(geojson(bytes) && geojson(b" {\"features\": []}") && geojson(&[0x1f, 0x8b, 8])); + assert!(!geojson(br#"{"name": "grid", "patch": []}"#) && !geojson(b"[]") && !geojson(b"LATK")); + let named = |name| Reading { name, ..Reading::default() }; + let error = pollster::block_on(Data::read(bytes, named(None))).unwrap_err(); + assert_eq!(error.to_string(), "name: GeoJSON needs a dataset name"); + let read = pollster::block_on(Data::read(bytes, named(Some("city")))).unwrap(); + assert_eq!(read.name().as_str(), "city"); + let renamed = pollster::block_on(Data::read(&read.snapshot().to_bytes(), named(Some("town")))).unwrap(); + assert_eq!(renamed.snapshot().table("points").map(crate::Table::len), Some(1), "renaming keeps the tables"); + assert_eq!(renamed.name().as_str(), "town"); + // An HTML page served for a missing file is none of these, and the error names the format. + let page = b""; + let error = pollster::block_on(Data::read(page, named(None))).unwrap_err().to_string(); + assert_eq!(error, "input: a dataset must be latkit's bytes, GeoJSON, or a JSON document"); + let error = pollster::block_on(Patch::read(page, Reading::default())).map(|_| ()).unwrap_err().to_string(); + assert_eq!(error, "input: a patch must be latkit's bytes or JSON"); } #[test] fn errors_name_the_change_table_and_field() { + let host = Files::default(); let error = |json: &str| { - pollster::block_on(Patch::load(json, "", &Files::default())).map(|_| ()).unwrap_err().to_string() + let reading = Reading { host: Some(&host), ..Reading::default() }; + pollster::block_on(Patch::read(json.as_bytes(), reading)).map(|_| ()).unwrap_err().to_string() }; - assert!( - error(r#"[{"replace": {"table": "Bus", "ids": {"count": 2}, "fields": {"x": [1, "a"]}}}]"#) - .contains("field x") - ); + let mixed = error(r#"[{"replace": {"table": "Bus", "ids": {"count": 2}, "fields": {"x": [1, "a"]}}}]"#); + assert!(mixed.starts_with("[0].replace.fields.x: "), "{mixed}"); assert!(error(r#"[{"bend": {}}]"#).contains("unknown variant")); let named = error(r#"[{"axis": {"name": "t", "unit": "s"}}, {"remove": {"table": 3, "ids": []}}]"#); assert!(named.starts_with("[1].remove.table"), "{named}"); + // Errors found after reading name the part the same way. + let reference = + error(r#"[{"append": {"table": "Bus", "at": 0, "fields": {"x": {"reference": "Other", "ids": [1]}}}}]"#); + assert!(reference.starts_with("[0].append.fields.x: append records values"), "{reference}"); assert!( error( r#"[{"replace": {"table": "Bus", "ids": {"count": 1}, "fields": {"x": {"file": "f", "count": 1}}}}]"# @@ -204,8 +316,10 @@ mod tests { .contains("dtype") ); let missing = r#"[{"replace": {"table": "Bus", "ids": {"count": 1}, "fields": {"x": {"file": "f.bin", "dtype": "f64le", "count": 1}}}}]"#; - assert_eq!(error(missing), "io: f.bin: 0 bytes of the slice's 8"); - let invalid = load_data( + assert_eq!(error(missing), "io: f.bin: read 0 of the slice's 8 bytes"); + let hostless = pollster::block_on(Patch::read(missing.as_bytes(), Reading::default())); + assert!(matches!(hostless, Err(Error::Unsupported(_))), "a slice needs a host to fetch it"); + let invalid = read_data( r#"{"name": "grid", "patch": [{"replace": {"table": "Bus", "ids": [1, 1]}}]}"#, &Files::default(), ); diff --git a/crates/latkit/src/data/mod.rs b/crates/latkit/src/data/mod.rs index 42c2cee0..04563751 100644 --- a/crates/latkit/src/data/mod.rs +++ b/crates/latkit/src/data/mod.rs @@ -1,10 +1,10 @@ -//! Datasets: named tables of paged, immutable columns, changed only by patches. +//! Datasets: named tables of immutable, paged columns. Only patches change a dataset, and each +//! version is a snapshot. mod bits; mod buffer; mod bytes; mod column; -mod dataset; mod describe; mod forms; mod frames; @@ -16,39 +16,43 @@ mod ordered; mod patch; mod query; mod samples; +mod snapshot; +mod source; mod table; mod text; mod values; pub use bytes::DecodeLimits; -pub use dataset::Data; -pub use describe::{Description, Envelope, FieldDescription, TableDescription}; +pub use describe::{Description, Envelope, EnvelopeRequest, FieldDescription, TableDescription}; pub use ids::Ids; +pub use json::{Patches, Reading}; pub use name::{Id, Name, Row, Rows}; #[cfg(feature = "gpu")] pub(crate) use name::{Keyed, Named}; pub use patch::Patch; +pub use query::{Order, Query, QueryResult}; +pub use snapshot::Snapshot; +pub use source::Data; +#[cfg(feature = "gpu")] +pub use source::Source; pub use table::Table; pub use values::{Kind, Value, Values}; -pub use { - json::Patches, - query::{Order, Query}, -}; pub(crate) use {geometry::Geometry, table::Field}; -// What the GPU side reads of the data. +// Data internals the GPU side uses. #[cfg(feature = "gpu")] pub(crate) use { + bytes::{read_values, values_bytes}, column::{Column, PAGE_ROWS, Page}, frames::{Listed, Manifest}, geometry::GeometryBuilder, ids::RowIds, + json::geojson, name::{PageId, Version}, patch::Op, samples::{AxisPage, FRAME_PAGE_WORDS, FramePage, MAX_LEVEL, Run, Samples, bucket_frames, detail, sides}, table::{Inverse, Reference, UNWIRED}, - values::HOLE, }; #[cfg(test)] diff --git a/crates/latkit/src/data/name.rs b/crates/latkit/src/data/name.rs index 961d1a13..fe84d196 100644 --- a/crates/latkit/src/data/name.rs +++ b/crates/latkit/src/data/name.rs @@ -7,12 +7,12 @@ use std::sync::atomic::{AtomicU64, Ordering}; use rustc_hash::FxHashMap; -/// The name of a dataset, table, or field: cheap to clone, compared and ordered by its text. +/// The name of a dataset, table, or field. Cheap to clone, and compared and ordered by its text. #[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize)] #[serde(transparent)] pub struct Name(Arc); -/// A name is a text. +/// A name's schema is a string. impl schemars::JsonSchema for Name { fn inline_schema() -> bool { true @@ -108,8 +108,9 @@ impl fmt::Debug for Name { } } -/// A row's identity within its table: a number or a text. A number above 2^53 − 1 serializes as -/// decimal text, which names the same row, since JSON readers such as JavaScript's round it. +/// A row's id within its table, either a number or a text. A number above 2^53 − 1 serializes as +/// decimal text, since JSON readers such as JavaScript's would round it. The text names the same +/// row. #[derive(Clone, PartialEq, Eq, PartialOrd, Ord)] pub enum Id { /// A numeric id. @@ -125,9 +126,24 @@ impl Id { Id::Text(t) => IdRef::Text(t), } } + + /// The id as a number that every JSON reader represents exactly, as it serializes. `None` for a + /// text id or a number above 2^53 − 1. + pub fn exact_number(&self) -> Option { + match self { + Id::Number(n) if *n <= SAFE => Some(*n as f64), + _ => None, + } + } + + /// The id that a JSON or JavaScript number names, if it is a whole number from 0 to 2^53 − 1. + /// `None` for any other number, which may already be rounded. + pub fn from_number(number: f64) -> Option { + (number.fract() == 0.0 && (0.0..=SAFE as f64).contains(&number)).then_some(Id::Number(number as u64)) + } } -/// The largest whole number every JSON reader holds exactly. +/// The largest whole number every JSON reader represents exactly. const SAFE: u64 = (1 << 53) - 1; impl serde::Serialize for Id { @@ -222,7 +238,8 @@ impl IdRef<'_> { } } -/// A row of a table in a dataset. It survives any change that keeps the names and the id. +/// One row, identified by dataset name, table name, and id. It stays valid across any change that +/// keeps those names and the id. #[derive( Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Debug, serde::Serialize, serde::Deserialize, schemars::JsonSchema, )] @@ -242,7 +259,8 @@ impl Row { } } -/// Rows of one table in a dataset, by id: what a view selects, fits, and tells as sets of rows. +/// Rows of one table in a dataset, by id. Views use it for selections, fits, and the rows events +/// report. #[derive(Clone, PartialEq, Eq, Debug, serde::Serialize, serde::Deserialize, schemars::JsonSchema)] pub struct Rows { /// The dataset's name. @@ -264,7 +282,7 @@ impl Rows { self.ids.iter().map(|id| Row { data: self.data.clone(), table: self.table.clone(), id: id.clone() }) } - /// `rows` as sets of rows, a set per table in the order each first appears. + /// Groups `rows` into one set per dataset and table, in order of first appearance. pub(crate) fn group(rows: impl IntoIterator) -> Vec { let mut sets: Vec = Vec::new(); for Row { data, table, id } in rows { @@ -306,13 +324,21 @@ macro_rules! counter { counter!( /// Identifies one state of something that changes: a dataset, a table's rows, or a field. Version, - /// Identifies the contents of one page; a written page gets a new id. + /// Identifies the contents of one page. A written page gets a new id. PageId ); -/// Values by key: one hash table of keys for every kind of value, so a map by name compiles once -/// whatever it holds, and the values in a list, in the order inserted; a removal moves the last into -/// its place. +impl PageId { + /// A page id made from `digest`, a hash of the page's words, so pages with equal contents + /// share one id. Its top bit is set, so it never equals a counter-assigned id. + pub(crate) fn of_content(digest: u64) -> PageId { + PageId(digest | 1 << 63) + } +} + +/// A map that keeps its entries in a `Vec` in insertion order, with a hash index from key to +/// position. The index type does not depend on `V`, so the hash map code is compiled once per key +/// type. Removing an entry moves the last entry into its position. #[derive(Clone, Debug)] pub(crate) struct Keyed { index: FxHashMap, @@ -387,12 +413,12 @@ impl Keyed { Some(value) } - /// Where `key`'s value is, or goes. + /// The entry for `key`, present or not. pub(crate) fn entry(&mut self, key: K) -> Entry<'_, K, V> { Entry { keyed: self, key } } - /// Keeps the entries `keep` holds to, in their order. + /// Keeps the entries for which `keep` returns true, in their order. pub(crate) fn retain(&mut self, mut keep: impl FnMut(&K, &mut V) -> bool) { let before = self.entries.len(); self.entries.retain_mut(|(key, value)| keep(key, value)); @@ -411,6 +437,10 @@ impl Keyed { self.entries.iter().map(|(key, value)| (key, value)) } + pub(crate) fn iter_mut(&mut self) -> impl Iterator { + self.entries.iter_mut().map(|(key, value)| (&*key, value)) + } + pub(crate) fn keys(&self) -> impl Iterator { self.entries.iter().map(|(key, _)| key) } @@ -454,14 +484,14 @@ impl<'a, K, V> IntoIterator for &'a Keyed { } } -/// Where a key's value is in a [`Keyed`], or goes. +/// The entry of a [`Keyed`] for one key, present or not. pub(crate) struct Entry<'a, K, V> { keyed: &'a mut Keyed, key: K, } impl<'a, K: Hash + Eq + Clone, V> Entry<'a, K, V> { - /// The key's value, made by `make` where it has none. + /// The key's value, inserting `make()` if it has none. pub(crate) fn or_insert_with(self, make: impl FnOnce() -> V) -> &'a mut V { let Entry { keyed, key } = self; let i = keyed.at(&key).unwrap_or_else(|| { @@ -488,6 +518,6 @@ impl, Q: Hash + Eq + ?Sized, V> std::ops::Index type Output = V; fn index(&self, key: &Q) -> &V { - self.get(key).expect("a key the table holds") + self.get(key).expect("the key is present") } } diff --git a/crates/latkit/src/data/ordered.rs b/crates/latkit/src/data/ordered.rs index 4a87a916..89b527b9 100644 --- a/crates/latkit/src/data/ordered.rs +++ b/crates/latkit/src/data/ordered.rs @@ -1,5 +1,5 @@ -//! Named entries in the order written: what a JSON object means where order matters, such as a -//! patch's fields or a query's selected columns. +//! JSON objects read as named entries in the order written. Used where key order matters, such as +//! a query's selected fields or a message's patches by dataset. use std::fmt; use std::marker::PhantomData; @@ -8,7 +8,7 @@ use serde::de::{self, Deserialize, Deserializer, MapAccess, Visitor}; use super::name::Name; -/// Entries in the order written, each name once. +/// Entries in the order written. Each name appears once. #[derive(Clone, PartialEq, Debug)] pub(crate) struct Ordered(pub(crate) Vec<(Name, V)>); @@ -18,7 +18,7 @@ impl Default for Ordered { } } -/// Named entries as an object of them. +/// The schema is an object of `V` values by name. impl schemars::JsonSchema for Ordered { fn inline_schema() -> bool { true @@ -40,14 +40,14 @@ impl<'de, V: Deserialize<'de>> Deserialize<'de> for Ordered { type Value = Ordered; fn expecting(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str("an object, each name once") + f.write_str("an object with unique names") } fn visit_map>(self, mut map: A) -> Result, A::Error> { let mut entries: Vec<(Name, V)> = Vec::new(); while let Some(name) = map.next_key::()? { if entries.iter().any(|(n, _)| n.as_str() == name) { - return Err(de::Error::custom(format!("{name} is named twice"))); + return Err(de::Error::custom(format!("{name} appears twice"))); } entries.push((name.into(), map.next_value()?)); } diff --git a/crates/latkit/src/data/patch.rs b/crates/latkit/src/data/patch.rs index a46742b7..e8e1073b 100644 --- a/crates/latkit/src/data/patch.rs +++ b/crates/latkit/src/data/patch.rs @@ -1,19 +1,21 @@ -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; +use super::bytes::DecodeLimits; use super::column::Column; -use super::dataset::Inner; use super::ids::Ids; use super::name::{Name, Version}; use super::samples::{Run, Samples}; +use super::snapshot::Inner; use super::table::{Field, Reference, Table}; use super::values::Values; use crate::{Error, Result}; -/// Changes to a dataset, applied in order and all at once by [`Data::apply`](crate::Data::apply). +/// A list of changes to a dataset. [`Data::apply`](crate::Data::apply) applies them in order, all +/// at once or not at all. /// -/// Code builds patches and sockets carry them as bytes; every change reaches a dataset the same way. -/// A patch made by [`Patch::to`] names the dataset it changes, as a stream of several datasets' patches -/// does: applying it to another dataset is an error. +/// Patches are built in code or decoded from bytes or JSON. They are the only way to change a +/// dataset. A patch made by [`Patch::to`] names its dataset, as patches in a stream of several +/// datasets do. Applying it to another dataset is an error. #[derive(Clone, Default, Debug)] pub struct Patch { pub(crate) data: Option, @@ -22,30 +24,31 @@ pub struct Patch { #[derive(Clone, Debug)] pub(crate) enum Op { - /// Names the axis sampled fields are recorded over. + /// Sets the axis sampled fields are recorded over. Axis { name: Name, unit: Name }, /// Replaces a whole table, sampled fields included. Replace(Table), - /// Writes the given fields of the given rows; unknown ids add rows, unknown tables are added. + /// Writes the given fields of the given rows. Unknown ids add rows, and an unknown table is + /// added. Set(Table), - /// Removes rows by id; ids that name no row are ignored. + /// Removes rows by id. Ids that name no row are ignored. Remove { table: Name, ids: Ids }, - /// Records frames of sampled fields, frame-major: of every row, or of the rows `ids` names, the - /// rest holding their last values. + /// Records frames of sampled fields, with values frame-major. Covers every row, or only the + /// rows in `ids`. Other rows keep their last values. Append { table: Name, at: Vec, ids: Option, fields: Vec<(Name, Values)> }, /// Forgets every sampled field's frames before a coordinate. Forget { before: f64 }, - /// Holds a run of a sampled field's entries: their coordinates, and their values entry-major. + /// Inserts a run of a sampled field's entries: their coordinates, and their values entry-major. Frames { table: Name, field: Name, run: Run, at: Vec, values: Vec }, } impl Patch { - /// An empty patch, for whichever dataset it is applied to. + /// An empty patch that applies to any dataset. pub fn new() -> Patch { Patch::default() } - /// An empty patch to dataset `data`. + /// An empty patch for dataset `data` only. Applying it to another dataset is an error. pub fn to(data: impl Into) -> Patch { Patch { data: Some(data.into()), ops: Vec::new() } } @@ -55,45 +58,47 @@ impl Patch { self.data.as_ref() } - /// Names the axis sampled fields are recorded over, such as `("time", "s")`. + /// Sets the axis that sampled fields are recorded over, such as `("time", "s")`. #[must_use] pub fn axis(mut self, name: impl Into, unit: impl Into) -> Patch { self.ops.push(Op::Axis { name: name.into(), unit: unit.into() }); self } - /// Replaces table `table.name()` with `table`, or adds it. + /// Replaces table `table.name()` with `table`, sampled fields included, or adds it. #[must_use] pub fn replace(mut self, table: Table) -> Patch { self.ops.push(Op::Replace(table)); self } - /// Writes `table`'s fields for its rows: rows with known ids change, others are added, and fields - /// it leaves out keep their values. Adds the table when there is none of its name. + /// Writes the fields of `table` for its rows. Rows with existing ids are updated and new ids + /// add rows. Fields that `table` leaves out keep their values, or are missing for added rows. + /// Adds the table if no table has its name. #[must_use] pub fn set(mut self, table: Table) -> Patch { self.ops.push(Op::Set(table)); self } - /// Removes the rows of `table` with `ids`. + /// Removes the rows of `table` with `ids`. Ids that match no row are ignored. A missing table + /// is an error. #[must_use] pub fn remove(mut self, table: impl Into, ids: impl Into) -> Patch { self.ops.push(Op::Remove { table: table.into(), ids: ids.into() }); self } - /// Records a frame at each axis coordinate of `at`: for each field, a value per row of `table` - /// at every coordinate, frame-major, every row's at the first coordinate, then at the next. + /// Records a frame of `table` at each axis coordinate in `at`. Each field gives a value per row + /// per coordinate, frame-major: every row's value at the first coordinate, then at the next. /// - /// A field's first frame makes it sampled. Coordinates never decrease; at equal coordinates the - /// last frame reads. + /// A field's first frame makes it sampled. Coordinates must be finite and must not decrease. At + /// equal coordinates, reads return the last frame. #[must_use] pub fn append( mut self, - at: impl AsRef<[f64]>, table: impl Into, + at: impl AsRef<[f64]>, fields: impl IntoIterator, ) -> Patch where @@ -105,16 +110,16 @@ impl Patch { self } - /// Records a frame at each axis coordinate of `at` for the rows of `table` that `ids` names, as - /// [`append`](Patch::append) does for every row: a value per row named at every coordinate, - /// frame-major. Rows left out hold their last value. + /// Like [`append`](Patch::append), but records values only for the rows in `ids`, one per row + /// per coordinate, frame-major. Other rows keep their last value. An id that names no row is an + /// error. #[must_use] pub fn append_rows( mut self, - at: impl AsRef<[f64]>, table: impl Into, - ids: impl Into, + at: impl AsRef<[f64]>, fields: impl IntoIterator, + ids: impl Into, ) -> Patch where F: Into, @@ -125,30 +130,33 @@ impl Patch { self } - /// Forgets every sampled field's frames before axis coordinate `before`. Whole frame pages drop, - /// so it costs O(pages); older versions of the dataset keep what they read. + /// Forgets every sampled field's frames before axis coordinate `before`. Whole pages are + /// dropped, so the cost is O(pages). Later frames keep their numbers, and older snapshots still + /// read the forgotten frames. #[must_use] pub fn forget(mut self, before: f64) -> Patch { self.ops.push(Op::Forget { before }); self } - /// Holds a run of sampled field `field`'s entries of level `level`, numbered from `first`: what a - /// time tile holds, as `latkit tile` writes them. At level 0 the entries are frames, each a - /// coordinate in `at` and a value per row of `table` in `values`. At level `k` they are buckets of - /// `8^k` frames, each its first and last frame's coordinates in `at`, and in `values` each row's - /// least value over each bucket, then each row's greatest. NaN is missing. + /// Inserts a run of entries of sampled field `field` at level `level`, numbered from `first`, + /// in the form time tiles store them (such as the tiles `latkit tile` writes). At level 0 the + /// entries are frames, each with one coordinate in `at` and one value per row of `table` in + /// `values`. At level `k` they are buckets of `8^k` frames. `at` holds the coordinates of each + /// bucket's first and last frame. `values` holds every bucket's least value per row, followed + /// by every bucket's greatest. NaN is missing. /// - /// No entry may be held already; frames past the field's widen it, and those it skips stay missing. + /// None of the entries may already be present. Entries outside the field's frame range extend + /// it, and frames in any gap stay missing. /// /// ``` /// # use latkit::{Data, Ids, Patch, Table}; - /// let mut grid = Data::new("grid"); + /// let grid = Data::new("grid"); /// grid.apply(Patch::new().replace(Table::new("Bus", Ids::count(2))))?; /// // Frames 100 and 101, then the bucket of frames 0..8. /// grid.apply(Patch::new().frames("Bus", "vm", 0, 100, [100.0, 101.0], [1.0, 0.9, 1.02, 0.98]))?; /// grid.apply(Patch::new().frames("Bus", "vm", 1, 0, [0.0, 7.0], [0.95, 0.9, 1.05, 1.0]))?; - /// assert_eq!(grid.describe().span, Some([0.0, 101.0])); + /// assert_eq!(grid.snapshot().describe().span, Some([0.0, 101.0])); /// # Ok::<(), latkit::Error>(()) /// ``` #[must_use] @@ -165,7 +173,7 @@ impl Patch { self } - /// This patch followed by `next`'s changes, to this patch's dataset. + /// This patch followed by `next`'s changes. The result keeps this patch's dataset. #[must_use] pub fn extend(mut self, next: Patch) -> Patch { self.ops.extend(next.ops); @@ -177,42 +185,37 @@ impl Patch { self.ops.is_empty() } - /// The patch as bytes: one frame, naming its dataset when it has one. + /// Encodes the patch as one binary frame, including its dataset's name when it has one. pub fn to_bytes(&self) -> Vec { super::bytes::encode(self) } - /// A patch from [`Patch::to_bytes`] or [`Data::to_bytes`](crate::Data::to_bytes). + /// Decodes one whole frame written by [`Patch::to_bytes`] or + /// [`Snapshot::to_bytes`](crate::Snapshot::to_bytes), with default [`DecodeLimits`]. Returns an + /// error if `bytes` are malformed, truncated, or extend past the frame. pub fn from_bytes(bytes: &[u8]) -> Result { super::bytes::decode(bytes) } - /// The patch at the start of `bytes` and how many bytes it takes, or `None` while `bytes` hold - /// only its start: frames back to back, as files, pipes, and sockets carry them, decode one by - /// one, and each says its length up front, so a partial one costs nothing to wait on. A frame - /// longer than `limit` bytes is [`Error::Limit`](crate::Error::Limit) at once, so a stream of - /// untrusted bytes never waits on more than `limit`. + /// Decodes back-to-back frames from `read`, such as a file, pipe, or socket. Yields each patch + /// once its whole frame has arrived. Each frame states its length up front, so a frame longer + /// than `limits` allows fails with [`Error::Limit`](crate::Error::Limit) before its body is + /// read. An untrusted stream buffers at most one frame at a time. The iterator ends when the + /// input ends between frames, or after the first error. /// /// ``` - /// # use latkit::{Ids, Patch, Table}; - /// let mut stream = Patch::to("grid").set(Table::new("Bus", Ids::count(1))).to_bytes(); - /// stream.extend(Patch::to("grid").remove("Bus", [0_u64]).to_bytes()); - /// let limit = 1 << 20; - /// let (first, used) = Patch::decode(&stream, limit)?.expect("a whole patch"); - /// let (second, rest) = Patch::decode(&stream[used..], limit)?.expect("a whole patch"); - /// assert_eq!(used + rest, stream.len()); - /// assert_eq!(second.data().map(|d| d.as_str()), Some("grid")); - /// assert!(Patch::decode(&stream[..used - 1], limit)?.is_none(), "more bytes to come"); - /// assert!(Patch::decode(&stream, 8).is_err(), "longer than the limit"); + /// # use latkit::{DecodeLimits, Ids, Patch, Table}; + /// let mut bytes = Patch::to("grid").set(Table::new("Bus", Ids::count(1))).to_bytes(); + /// bytes.extend(Patch::to("grid").remove("Bus", [0_u64]).to_bytes()); + /// let patches = Patch::stream(&bytes[..], DecodeLimits::default()).collect::>>()?; + /// assert_eq!(patches.len(), 2); + /// assert_eq!(patches[1].data().map(|d| d.as_str()), Some("grid")); + /// let small = DecodeLimits { frame_bytes: 16, ..DecodeLimits::default() }; + /// assert!(Patch::stream(&bytes[..], small).all(|patch| patch.is_err()), "longer than the limit"); /// # Ok::<(), latkit::Error>(()) /// ``` - pub fn decode_with_limits(bytes: &[u8], limits: super::bytes::DecodeLimits) -> Result> { - super::bytes::decode_with_limits(bytes, limits) - } - - /// Decodes one frame with an encoded byte limit and the default decoded storage budget. - pub fn decode(bytes: &[u8], limit: usize) -> Result> { - super::bytes::decode_prefix(bytes, limit) + pub fn stream(read: R, limits: DecodeLimits) -> impl Iterator> { + super::bytes::stream(read, limits) } } @@ -222,8 +225,8 @@ impl From
for Patch { } } -/// Appends to the same fields of every row of one table, one after another: recorded as one, so a -/// field takes its frames in one write however many changes carried them. +/// Consecutive appends to the same fields of every row of one table, merged into one append. Each +/// field is then written once, however many changes carried its frames. pub(crate) struct Appends { table: Name, at: Vec, @@ -235,9 +238,10 @@ impl Appends { Appends { table, at, fields: fields.into_iter().map(|(name, values)| (name, vec![values])).collect() } } - /// Whether an append of `fields` at `at` to `table`, which has `rows` rows, carries the run on: the - /// same fields of the same kinds, a value for every row at each coordinate, coordinates that never - /// decrease. One that does not records on its own, so its error names it. + /// Whether an append of `fields` at `at` to `table` (with `rows` rows) can join this run: the + /// same table, the same fields of the same kinds, a value for every row at each coordinate, and + /// finite coordinates that do not decrease. An append that cannot join is applied alone, so its + /// error names it. pub(crate) fn continues(&self, table: &Name, at: &[f64], fields: &[(Name, Values)], rows: Option) -> bool { let Some(rows) = rows else { return false }; let last = self.at.last().copied().unwrap_or(f64::NEG_INFINITY); @@ -277,7 +281,8 @@ impl Appends { } impl Op { - /// Holds entries of `level` from `first`, as many as `at` holds coordinates of. + /// An [`Op::Frames`] of entries at `level`, numbered from `first`. The number of coordinates in + /// `at` sets the entry count. pub(crate) fn frames(table: Name, field: Name, level: u32, first: u64, at: Vec, values: Vec) -> Op { let entries = (at.len() / super::samples::coordinates(level)) as u64; let run = Run { level, entries: first..first.saturating_add(entries) }; @@ -303,7 +308,7 @@ impl Op { let name = table.name.clone(); table.check().map_err(|e| e.at(&name))?; if let Some((field, _)) = table.fields.iter().find(|(_, f)| matches!(f, Field::Sampled(..))) { - return Err(Error::invalid("set writes values and references; append records samples") + return Err(Error::invalid("sampled fields are written with append, not with set") .at(field) .at(&name)); } @@ -357,7 +362,7 @@ impl Op { let per_row = run.len() * super::samples::sides(run.level); let rows = values.len().checked_div(per_row).unwrap_or(0); if rows > target.len() { - return Err(Error::invalid(format!("values for {rows} rows of {}", target.len())) + return Err(Error::invalid(format!("values for {rows} rows, but the table has {}", target.len())) .at(&field) .at(&table)); } @@ -374,8 +379,8 @@ impl Op { } } -/// The row of `table` each of `ids` names; an id naming none is an error, as an append records -/// frames of rows the table holds. +/// The row of `table` for each id in `ids`. An id that names no row is an error, since an append +/// records frames only for existing rows. fn rows_named(table: &Table, ids: &Ids) -> Result> { let index = table.index()?; (0..ids.len()) @@ -404,7 +409,8 @@ fn set(existing: &Table, input: Table) -> Result
{ let mut table = existing.clone(); if !added.is_empty() { table.ids = existing.ids.extend(&added).map_err(|e| e.at("ids"))?; - table.index = Arc::default(); + let index = index.extended(&table.ids, existing.len()).map_err(|e| e.at("ids"))?; + table.index = Arc::new(OnceLock::from(index)); table.structure = Version::next(); } for (name, field) in input.fields { @@ -465,5 +471,7 @@ fn remove(existing: &Table, ids: &Ids) -> Result
{ return Ok(existing.clone()); } let keep: Vec = (0..existing.len()).filter(|&r| !doomed[r]).collect(); - Ok(existing.rows(&keep)) + let mut table = existing.rows(&keep); + table.index = Arc::new(OnceLock::from(index.without(&doomed, &table.ids)?)); + Ok(table) } diff --git a/crates/latkit/src/data/query.rs b/crates/latkit/src/data/query.rs index 44a7e13e..00ad1e6f 100644 --- a/crates/latkit/src/data/query.rs +++ b/crates/latkit/src/data/query.rs @@ -1,7 +1,6 @@ -//! Reading data back: the rows of a table a query keeps, in the order it sorts them, a page of them, -//! with their fields or what expressions make of them. Queries run on the CPU through the same -//! expressions marks draw with, a plain field read page by page, and return a table, so what comes -//! back has the shape of what goes in. +//! Queries read a page of a table's rows, filtered and sorted, with their fields or computed +//! values. They evaluate on the CPU with the same expressions marks draw with, reading a plain +//! field a page at a time. The result is a table, so it has the same shape as the input. use std::cmp::Ordering; @@ -10,67 +9,70 @@ use super::name::Name; use super::ordered::Ordered; use super::table::{Field, Table}; use super::values::Values; -use super::{Data, column::Column}; +use super::{Snapshot, column::Column}; use crate::expr::Expr; use crate::expr::cpu::Rows; use crate::{Error, Result}; -/// The rows of a table to read back: of the rows `ids` names (every row unless set), those `filter` -/// keeps, sorted by `sort`, a page of them from `offset` of at most `limit`, with the fields `select` -/// names or computes; sampled fields read at axis coordinate `at`, their last frame by default. Plain -/// data, like a scene. +/// A request to read rows of a table. It starts from the rows in `ids` (default: every row), keeps +/// the rows `filter` accepts, sorts them by `sort`, and returns up to `limit` of them starting at +/// `offset`. Each row comes with the fields `select` names or computes (default: every field). +/// Sampled fields are read at axis coordinate `at` (default: their last frame). Like a scene, it is +/// plain data that deserializes from JSON. /// /// ``` /// use latkit::{Data, Ids, Order, Query, Table, field}; -/// let mut data = Data::new("grid"); +/// let data = Data::new("grid"); /// data.apply(Table::new("Branch", Ids::count(3)).field("loading", vec![40.0, 95.0, 120.0]).field("name", vec!["a", "b", "c"]))?; -/// let (hot, kept) = data.query(&Query::of("Branch").filter(field("loading").gt(90.0)).sort(field("loading"), Order::Descending).limit(1))?; -/// assert_eq!(hot.get(2_u64, "name"), Some(latkit::Value::Text("c".into()))); -/// assert_eq!((hot.len(), kept), (1, 2)); +/// let hot = data.snapshot().query(&Query::of("Branch").filter(field("loading").gt(90.0)).sort(field("loading"), Order::Descending).limit(1))?; +/// assert_eq!(hot.table.get(2_u64, "name"), Some(latkit::Value::Text("c".into()))); +/// assert_eq!((hot.table.len(), hot.total), (1, 2)); /// # Ok::<(), latkit::Error>(()) /// ``` #[derive(Clone, PartialEq, Debug, serde::Deserialize, schemars::JsonSchema)] #[serde(deny_unknown_fields)] #[schemars(description = "The rows of a table to read back, run through the expressions marks draw with.")] pub struct Query { - /// The table read. + /// The table to read. of: Name, - /// The rows read, in this order unless sorted; every row unless set. Ids that name no row are - /// passed over. + /// The rows to read, in this order unless sorted. Defaults to every row. Ids that name no row + /// are skipped. #[serde(default)] ids: Option, - /// Keeps the rows it is true for. + /// Keeps only the rows for which this is true. #[serde(default)] filter: Option, - /// Keys to sort by, each after those before it; rows a key reads nothing for come last, and rows - /// all keys order alike keep their order. + /// Keys to sort by. Each key breaks ties left by the keys before it. Rows where a key has no + /// value sort last, and rows equal on every key keep their order. #[serde(default)] sort: Vec<(Expr, Order)>, - /// The kept rows skipped first. + /// How many of the kept rows to skip. #[serde(default)] offset: usize, - /// The most rows read; a limit picks them in linear time before sorting only those. + /// The most rows to return. When sorting with a limit, the first `offset + limit` rows are + /// selected in linear time and only those are sorted. limit: Option, - /// Fields to read by name: a field as it is, anything else as numbers. Every field unless set. + /// Fields to return, by output name. A plain field returns that field as stored, and any other + /// expression returns numbers. Defaults to every field. An empty object returns the ids alone. #[serde(default)] - select: Ordered, - /// The axis coordinate sampled fields are read at; their last frame unless set. + select: Option>, + /// The axis coordinate at which sampled fields are read. Defaults to their last frame. at: Option, } -/// Which way a sort runs; rows a key reads nothing for come last either way. +/// Sort direction. Rows where the key has no value sort last in either direction. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default, serde::Deserialize, schemars::JsonSchema)] #[serde(rename_all = "lowercase")] pub enum Order { - /// Least first. + /// Smallest first. #[default] Ascending, - /// Greatest first. + /// Largest first. Descending, } impl Query { - /// Every row of table `table`, every field. + /// A query for every row and field of table `table`. pub fn of(table: impl Into) -> Query { Query { of: table.into(), @@ -79,51 +81,60 @@ impl Query { sort: Vec::new(), offset: 0, limit: None, - select: Ordered::default(), + select: None, at: None, } } - /// Reads only the rows `ids` names, in that order unless sorted. + /// Reads only the rows `ids` names, in that order unless sorted. Ids that name no row are + /// skipped. #[must_use] pub fn ids(mut self, ids: impl Into) -> Query { self.ids = Some(ids.into()); self } - /// Keeps the rows `keep` is true for. + /// Keeps only the rows for which `keep` is true. #[must_use] pub fn filter(mut self, keep: impl Into) -> Query { self.filter = Some(keep.into()); self } - /// Sorts by `key`, after the keys given before; rows it orders alike keep their order. + /// Sorts by `key`, breaking ties left by earlier keys. Rows equal on every key keep their + /// order, and rows where `key` has no value sort last. #[must_use] pub fn sort(mut self, key: impl Into, order: Order) -> Query { self.sort.push((key.into(), order)); self } - /// Skips the first `offset` rows kept. + /// Skips the first `offset` kept rows, after sorting. #[must_use] pub fn offset(mut self, offset: usize) -> Query { self.offset = offset; self } - /// Reads at most `limit` rows. + /// Returns at most `limit` rows. #[must_use] pub fn limit(mut self, limit: usize) -> Query { self.limit = Some(limit); self } - /// Reads field `name` as `value` makes it: a field as it is, anything else as numbers. With none - /// selected, every field is read. + /// Adds output field `name`, computed by `value`. A plain field returns that field as stored, + /// and any other expression returns numbers. Without any `select`, every field is returned. #[must_use] pub fn select(mut self, name: impl Into, value: impl Into) -> Query { - self.select.0.push((name.into(), value.into())); + self.select.get_or_insert_with(Ordered::default).0.push((name.into(), value.into())); + self + } + + /// Returns only the rows' ids, with no fields. + #[must_use] + pub fn ids_alone(mut self) -> Query { + self.select = Some(Ordered::default()); self } @@ -135,11 +146,23 @@ impl Query { } } -impl Data { - /// The rows of a table `query` asks for, as a table of their own: their ids, and their fields or - /// what the query computes, sampled fields read at its coordinate; and how many rows its filter - /// kept, before the page. A limit picks its rows in linear time before sorting only them. - pub fn query(&self, query: &Query) -> Result<(Table, usize)> { +/// The result of [`Snapshot::query`]: the requested page of rows as a table, and how many rows +/// passed the filter. +#[derive(Clone, Debug)] +pub struct QueryResult { + /// The page of rows, as a table with the source table's name. It holds their ids and their + /// selected fields or computed values. Sampled fields come back as plain values read at the + /// query's coordinate. + pub table: Table, + /// The number of rows that passed the filter, before `offset` and `limit`. + pub total: usize, +} + +impl Snapshot { + /// Returns the page of rows `query` describes, as a table, and the number of rows that passed + /// the filter. When sorting with a limit, the rows are selected in linear time and + /// only those are sorted. Errors if the table or a selected field does not exist. + pub fn query(&self, query: &Query) -> Result { let table = self.table(&query.of).ok_or_else(|| Error::invalid("no such table").at(query.of.as_str()))?; let at = query.at.unwrap_or(f64::INFINITY); let all = std::slice::from_ref(self); @@ -183,13 +206,13 @@ impl Data { structure: version, order: version, }; - if query.select.0.is_empty() { + let Some(select) = &query.select else { for (name, field) in &table.fields { out.insert(name.clone(), plain(field, at, page)); } - return Ok((out, total)); - } - for (name, value) in &query.select.0 { + return Ok(QueryResult { table: out, total }); + }; + for (name, value) in &select.0 { let field = if let Some(f) = value.field_name() { let Some(field) = table.field_named(f) else { return Err(Error::invalid(format!("no field {f}")).at(name.as_str()).at("select")); @@ -202,11 +225,11 @@ impl Data { }; out.insert(name.clone(), field); } - Ok((out, total)) + Ok(QueryResult { table: out, total }) } } -/// How two sort keys order rows: missing last, whichever way the sort runs. +/// Orders two sort-key values. Non-finite (missing) values sort last in either direction. fn compare(a: f64, b: f64, order: Order) -> Ordering { match (a.is_finite(), b.is_finite()) { (true, true) if order == Order::Descending => b.total_cmp(&a), @@ -217,7 +240,8 @@ fn compare(a: f64, b: f64, order: Order) -> Ordering { } } -/// `field` of rows `rows` as plain values: a sampled field read at `at`, one frame, not its history. +/// `field` for rows `rows` as plain values. A sampled field becomes its values in the frame at +/// `at`, without its history. fn plain(field: &Field, at: f64, rows: &[usize]) -> Field { match field { Field::Sampled(samples, _) => { diff --git a/crates/latkit/src/data/samples.rs b/crates/latkit/src/data/samples.rs index 4a108930..4dbe1e15 100644 --- a/crates/latkit/src/data/samples.rs +++ b/crates/latkit/src/data/samples.rs @@ -1,14 +1,17 @@ -//! Samples: a field's values over the data's axis. Level 0 holds frames, each row's value at one -//! coordinate; each level `k` above it holds buckets of `8^k` frames, each row's least and greatest over -//! them, so a long history draws from as few entries as there are pixels. Every level keeps its -//! entries in pages, with each entry's coordinates (a frame's own; a bucket's first and last frame's), -//! so a level draws without the levels below it. Frames are numbered from the first ever written, and -//! a version reads `start..end`. +//! Samples: a field's values over the dataset's axis, stored as levels of detail. //! -//! Any entry may be missing. Frames appended grow the newest pages in place and seal each bucket whose -//! entries below are all held; runs of entries read from bytes or fetched as time tiles fill any span, -//! whatever lies around them. An entry is held when its level's coordinates hold it: every write writes -//! an entry's coordinates and every row block's values together. +//! Level 0 holds frames, each row's value at one axis coordinate. Each level `k` above holds +//! buckets of `8^k` frames, with each row's least and greatest value over the bucket. A long +//! history can then be drawn from about as many entries as there are pixels. Each level stores its +//! entries in pages together with their coordinates (a frame's own, or a bucket's first and last +//! frame's), so a level can be drawn without the levels below it. Frames are numbered from the +//! first ever written, and a version reads frames `start..end`. +//! +//! Any entry may be missing. Appended frames grow the newest pages in place and write each bucket +//! whose eight entries one level below are all present. Runs of entries decoded from bytes or +//! fetched as time tiles can fill any range, whatever surrounds it. An entry is present when its +//! level's coordinate pages contain it, because every write stores an entry's coordinates and every +//! row block's values together. use std::ops::Range; use std::sync::{Arc, PoisonError, RwLock}; @@ -18,44 +21,46 @@ use super::name::PageId; use super::values::Values; use crate::{Error, Result}; -/// Words in a frame page: entries of one row block, entry-major. An axis page holds as many words on -/// the GPU, two for each entry. +/// Words in a frame page, which holds entries of one row block, entry-major. An axis page takes at +/// most as many words on the GPU, two per entry. pub(crate) const FRAME_PAGE_WORDS: usize = 1 << 16; -/// Levels of detail kept above frames: level `k` holds each row's least and greatest value over buckets -/// of `8^k` frames. +/// Levels of detail that appends write above frames. Level `k` holds each row's least and greatest +/// value over buckets of `8^k` frames. Inserted runs can go higher, up to [`MAX_LEVEL`]. pub(crate) const LEVELS: u32 = 6; -/// Highest format level: a bucket spans 2^63 frames and still fits a u64. +/// The highest level the format allows: a bucket of `8^21 = 2^63` frames still fits a `u64`. pub(crate) const MAX_LEVEL: u32 = 21; -/// Frames an entry of level `level` spans: a frame, or a bucket's frames. +/// The number of frames an entry of `level` covers: `8^level`. pub(crate) fn bucket_frames(level: u32) -> u64 { 1 << (3 * level) } -/// The level of detail `frames` frames draw at across `columns` pixel columns: frames while there are -/// at most two a column, else the finest level whose buckets are no fewer. +/// The level of detail for drawing `frames` frames across `columns` pixel columns. Uses frames +/// (level 0) while there are at most two per column, else the finest level with at most two +/// buckets per column. pub(crate) fn detail(frames: u64, columns: u32) -> u32 { let needed = frames.div_ceil(2 * u64::from(columns.max(1))); if needed <= 1 { 0 } else { (1..=MAX_LEVEL).find(|&k| bucket_frames(k) >= needed).unwrap_or(MAX_LEVEL) } } -/// Coordinates an entry of `level` holds: a frame its own, a bucket its first and last frame's. +/// Coordinates stored per entry of `level`: 1 for a frame, 2 for a bucket (its first and last +/// frame's). pub(crate) fn coordinates(level: u32) -> usize { if level == 0 { 1 } else { 2 } } -/// Values a row holds of an entry of `level`: a frame's, or a bucket's least and greatest. +/// Values stored per row per entry of `level`: 1 for a frame, 2 for a bucket (least and greatest). pub(crate) fn sides(level: u32) -> usize { if level == 0 { 1 } else { 2 } } -/// The log of a row block holding `level`'s values of `side`: its frames, or a level's least (0) or -/// greatest (1). +/// The index, among a row block's logs, of the log holding `level`'s `side` values: 0 for frames, +/// then each level's least (`side` 0) and greatest (`side` 1). fn slot(level: u32, side: usize) -> usize { if level == 0 { 0 } else { 2 * level as usize - 1 + side } } -/// Entries `entries` of level `level`. +/// A range of entries of one level. #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub(crate) struct Run { pub(crate) level: u32, @@ -63,25 +68,26 @@ pub(crate) struct Run { } impl Run { - /// Number of entries; runs in memory always fit the address space. + /// The number of entries. A run held in memory always fits in `usize`. pub(crate) fn len(&self) -> usize { usize::try_from(self.entries.end - self.entries.start).expect("run fits in memory") } - /// The frames its entries span. + /// The frames its entries cover. pub(crate) fn frames(&self) -> Range { let span = bucket_frames(self.level); self.entries.start.saturating_mul(span)..self.entries.end.saturating_mul(span) } - /// How many coordinates it holds, and how many values for `rows` rows. + /// Its coordinate count and its value count for `rows` rows, or `None` on overflow. pub(crate) fn sizes(&self, rows: usize) -> Option<(usize, usize)> { let n = usize::try_from(self.entries.end.checked_sub(self.entries.start)?).ok()?; Some((n.checked_mul(coordinates(self.level))?, n.checked_mul(rows)?.checked_mul(sides(self.level))?)) } } -/// Appends `run` to `runs`, joining it to the last where it follows it at the same level. +/// Appends `run` to `runs`, merging it into the last run when it continues that run at the same +/// level. fn join(runs: &mut Vec, run: Run) { match runs.last_mut() { Some(last) if last.level == run.level && last.entries.end == run.entries.start => { @@ -92,10 +98,10 @@ fn join(runs: &mut Vec, run: Run) { } /// A list shared between versions. Indices are absolute: a version reads items `start..end`, so -/// forgetting the oldest never renumbers the rest. +/// forgetting the oldest items never renumbers the rest. /// -/// A version that appends while it is the newest grows the shared list in place; any other change, an -/// older version appending, an item put between others or taken out, forks a copy of what it reads. +/// The newest version appends to the shared list in place. Any other change (an older version +/// appending, or an item inserted or removed) first copies the items the version reads. pub(crate) struct Log { items: Arc>>, /// The absolute index of `items[0]`. @@ -129,7 +135,7 @@ impl Log { self.edit(|items| items.push(item)); } - /// Changes the items this version reads in a copy of its own. + /// Applies `f` to a private copy of the items this version reads. fn edit(&mut self, f: impl FnOnce(&mut Vec)) { let mut items = self.read(<[T]>::to_vec); f(&mut items); @@ -138,7 +144,7 @@ impl Log { self.base = self.start; } - /// Reads items `start..end`. + /// Calls `f` with items `start..end`. pub(crate) fn read(&self, f: impl FnOnce(&[T]) -> R) -> R { let items = self.items.read().unwrap_or_else(PoisonError::into_inner); f(&items[self.start - self.base..self.end - self.base]) @@ -148,8 +154,9 @@ impl Log { self.read(|items| items.last().cloned()) } - /// Stops reading items before absolute index `start`. Storage shared with other versions stays - /// theirs; this version copies what it keeps once the forgotten part outgrows a quarter of it. + /// Stops reading items before absolute index `start`. Shared storage is left to the other + /// versions. Once the forgotten prefix exceeds a quarter of the kept items (plus 16), this + /// version copies the items it keeps. pub(crate) fn forget(&mut self, start: usize) { self.start = start.clamp(self.start, self.end); if self.start - self.base > (self.end - self.start) / 4 + 16 { @@ -160,16 +167,17 @@ impl Log { } } -/// What a page stores, shared by every version reading it, and whether appends grow it. +/// A page's storage, shared by every version that reads the page, and whether appends may grow it. struct Store { grows: bool, items: RwLock>, } -/// Consecutive entries of a level, `per` values each, entry-major: a row block's frames or a level's -/// least or greatest values, `per` its rows; or a level's coordinates, `per` an entry's. A page an -/// append opened grows in place while it is the newest version's last; a version reads at most `cap` -/// entries of it, fewer than it stores where a newer version grew it. +/// Consecutive entries of one level, `per` items each, entry-major. In a frame page the items are +/// a row block's frames or one side (least or greatest) of a level, and `per` is the block's row +/// count. In an axis page the items are a level's coordinates, and `per` is the coordinates per +/// entry. A page opened by an append grows in place while it is the newest version's last page. A +/// version reads at most `cap` entries, fewer than are stored when a newer version grew the page. #[derive(Clone)] pub(crate) struct Page { pub(crate) id: PageId, @@ -179,9 +187,11 @@ pub(crate) struct Page { store: Arc>, } -/// A page of a row block's frames, or of a level's least or greatest values: `per` rows an entry. +/// A page of a row block's frames, or of a level's least or greatest values, with `per` rows per +/// entry. pub(crate) type FramePage = Page; -/// A page of a level's coordinates: `per` an entry, a frame's own or a bucket's first and last. +/// A page of a level's coordinates, `per` per entry. A frame has its own coordinate, and a bucket +/// has its first and last frame's. pub(crate) type AxisPage = Page; impl Page { @@ -195,17 +205,18 @@ impl Page { } } - /// Reads the page's stored items. + /// Calls `f` with every stored item, which may include items past `cap`. pub(crate) fn read(&self, f: impl FnOnce(&[T]) -> R) -> R { f(&self.store.items.read().unwrap_or_else(PoisonError::into_inner)) } - /// Whether appends grow it: it takes room to grow into on the GPU. + /// Whether appends may grow it. The GPU allocates such pages at full size so they can grow in + /// place. pub(crate) fn grows(&self) -> bool { self.store.grows } - /// Entries this version reads of it. + /// The number of entries this version reads. pub(crate) fn len(&self) -> u32 { ((self.read(<[T]>::len) / self.per.max(1) as usize) as u32).min(self.cap) } @@ -214,8 +225,9 @@ impl Page { self.first + u64::from(self.len()) } - /// Appends `entry`'s `items` where this version reads the whole page, appends grow it, and the - /// entry follows its last and fits within `limit` items. + /// Appends `entry`'s `items` in place, returning whether it did. Succeeds only if the page + /// grows, this version reads all of it and nothing newer is stored past it, `entry` directly + /// follows the last entry, and the page stays within `limit` items. fn extend(&self, entry: u64, items: &[T], limit: usize) -> bool { if entry != self.end() { return false; @@ -230,14 +242,15 @@ impl Page { true } - /// The page as a version that reads only its first `len` entries sees it. + /// A handle to the same page that reads only its first `len` entries. fn capped(&self, len: u32) -> Page { Page { cap: len, ..self.clone() } } } impl AxisPage { - /// Coordinate `which` of entry `index`: a frame's own, a bucket's first (0) or last (1) frame's. + /// Coordinate `which` of entry `index`: a frame's own, or a bucket's first (0) or last (1) + /// frame's. fn coordinate(&self, index: u64, which: u32) -> Option { if index < self.first || index >= self.end() { return None; @@ -246,17 +259,17 @@ impl AxisPage { self.read(|c| c.get(i).copied()) } - /// Where entry `index` draws: a frame at its coordinate, a bucket at its middle. + /// The coordinate at which entry `index` is drawn: a frame's own, or a bucket's midpoint. fn drawn(&self, index: u64) -> Option { Some(f64::midpoint(self.coordinate(index, 0)?, self.coordinate(index, 1)?)) } - /// Words the page takes on the GPU: where each entry draws, as a high and a low `f32`. + /// The page's size in GPU words: each entry's drawn coordinate as a high and a low `f32`. pub(crate) fn words_len(&self) -> usize { 2 * (self.read(<[f64]>::len) / self.per as usize) } - /// The page's GPU words from word `from` on: a growing page uploads only its new tail. + /// The page's GPU words from word `from` on, so a growing page uploads only its new tail. pub(crate) fn words(&self, from: usize) -> Vec { let per = self.per as usize; self.read(|c| { @@ -271,7 +284,7 @@ fn split(x: f64) -> [u32; 2] { [hi.to_bits(), ((x - f64::from(hi)) as f32).to_bits()] } -/// The page of `log` holding entry `index`, where one does. +/// The page of `log` containing entry `index`, if any. fn holding(log: &Log>, index: u64) -> Option> { log.read(|pages| { let i = pages.partition_point(|p| p.first <= index).checked_sub(1)?; @@ -279,8 +292,9 @@ fn holding(log: &Log>, index: u64) -> Option> { }) } -/// The pages of `log` holding any of `entries`, in order, each with those it holds. A page that stores -/// entries past the next page's first, where a version forked, holds them no longer. +/// The pages of `log` containing any of `entries`, in order, each with the range of `entries` it +/// contains. A page that stores entries past the next page's first entry (left by a fork) is cut +/// off at that entry. fn held(log: &Log>, entries: Range) -> Vec<(Page, Range)> { if entries.is_empty() { return Vec::new(); @@ -302,7 +316,7 @@ fn held(log: &Log>, entries: Range) -> Vec<(Page, Range }) } -/// The first of `entries` a page of `log` holds. +/// The first of `entries` that a page of `log` contains. fn first_held(log: &Log>, entries: Range) -> Option { log.read(|pages| { let from = pages.partition_point(|p| p.first <= entries.start).saturating_sub(1); @@ -320,7 +334,7 @@ fn first_held(log: &Log>, entries: Range) -> Option { }) } -/// Puts `page` in its place in `log`, among pages holding none of its entries. +/// Inserts `page` into `log` in entry order. No page of `log` may contain its entries. fn place(log: &mut Log>, page: Page) { let (at, tip) = log.read(|pages| { let at = pages.partition_point(|p| p.first < page.first); @@ -329,8 +343,8 @@ fn place(log: &mut Log>, page: Page) { if tip { log.push(page) } else { log.edit(|pages| pages.insert(at, page)) } } -/// Makes the last page of `log` hold no entry from `end` on, as a version reading entries before `end` -/// sees it, before the version writes past `end`. +/// Caps the last page of `log` at entry `end`, so this version no longer reads entries a newer +/// version appended from `end` on. Called before this version writes past `end`. fn cap(log: &mut Log>, end: u64) { if log.last().is_some_and(|page| page.end() > end) { log.edit(|pages| { @@ -340,7 +354,7 @@ fn cap(log: &mut Log>, end: u64) { } } -/// Takes the pages of `log` holding only entries of `entries` out of it. +/// Removes the pages of `log` whose entries all lie within `entries`. fn take_out(log: &mut Log>, entries: &Range) { let (from, to) = log.read(|pages| { let from = pages.partition_point(|p| p.first < entries.start); @@ -358,16 +372,16 @@ fn forget_before(log: &mut Log>, index: u64) { log.forget(start + dropped); } -/// Writes entry `entry`, `items` of it, after the last of `log`: into its last page where that grows -/// and has room within `limit` items, else into a new page that grows. +/// Writes `entry`'s `items` after the last entry of `log`. They go into the last page if it can +/// grow and has room within `limit` items, else into a new growable page. fn grow(log: &mut Log>, entry: u64, items: &[T], limit: usize) { if !log.last().is_some_and(|page| page.extend(entry, items, limit)) { log.push(Page::new(entry, items.len(), items.to_vec(), true)); } } -/// Writes `entries` of a level, `items` of `per` each, as pages placed in order, as many entries to a -/// page as `limit` items allow. +/// Writes `entries` of a level, `per` items each, as new fixed-size pages inserted in order, each +/// holding as many entries as fit in `limit` items. fn put(log: &mut Log>, entries: &Range, items: &[T], per: usize, limit: usize) { let step = (limit / per.max(1)).max(1); for (k, chunk) in items.chunks(step * per.max(1)).enumerate() { @@ -376,30 +390,31 @@ fn put(log: &mut Log>, entries: &Range, items: &[T], per: } } -/// The logs a write writes into: every level's coordinates, and each row block's logs. +/// Writable copies of the logs: every level's coordinates, and each row block's value logs. struct Logs { axes: Vec>, blocks: Vec>>, } -/// A field's samples over the data's axis: frames, and levels of detail above them, with holes. +/// A sampled field's values over the axis, as frames and the levels of detail above them. Any level +/// may have gaps. Values are stored per row block of [`PAGE_ROWS`] rows. #[derive(Clone, Default)] pub(crate) struct Samples { - /// Frames this version reads, numbered from the first ever written. + /// The frames this version reads, `start..end`, numbered from the first ever written. start: u64, end: u64, - /// Frames before this one were forgotten: pages may still store some, which this version must not - /// read again. + /// Frames before this were forgotten. Shared pages may still store some, and this version must + /// not read them again. forgotten: u64, - /// Per level from 0: its entries' coordinates; empty until anything is written. + /// Per level from 0, its entries' coordinates. Empty until something is written. axes: Arc<[Log]>, - /// Per row block: its frames, then each level's least and greatest values. + /// Per row block, its frames, then each level's least and greatest values. blocks: Arc<[Vec>]>, } impl Samples { - /// No frames yet, the first to be appended numbered `first`: what samples read back from bytes or - /// gathered from others start from, so their frames keep their numbers. + /// Empty samples whose first frame will be numbered `first`. Samples decoded from bytes or + /// gathered from others start here, so their frames keep their numbers. pub(crate) fn from_frame(first: u64) -> Samples { Samples { start: first, end: first, ..Samples::default() } } @@ -414,31 +429,35 @@ impl Samples { self.end } - /// Reads frames `frames`: what bytes say, after the runs they hold are put in. + /// Sets the frames this version reads to `frames`. Decoding calls it after inserting the runs, + /// to restore the encoded frame range. pub(crate) fn reads(&mut self, frames: Range) { (self.start, self.end) = (frames.start, frames.end); } - /// The same frames, none of them held: what a dataset written as time tiles starts from. + /// Samples with the same frame range but no entries: the base of a dataset written as time + /// tiles. pub(crate) fn emptied(&self) -> Samples { let mut empty = Samples::from_frame(self.start); empty.reads(self.start..self.end); empty } - /// The entries of `level` this version reads: those whose frames lie before its end, from the one - /// holding its first frame. + /// The entries of `level` this version reads: from the entry containing frame `start` through + /// the last entry that ends by frame `end`. fn visible(&self, level: u32) -> Range { let k = 3 * level; self.start >> k..self.end >> k } - /// Copies of the logs to write into, with every level's and at least `blocks` row blocks'. + /// Writable copies of the logs, covering at least levels 0 through [`LEVELS`] and `blocks` row + /// blocks. fn logs(&self, blocks: usize) -> Logs { self.logs_at(blocks, LEVELS) } - /// Highest level stored; the binary format records it independently of the append policy. + /// The highest level with stored pages. The binary format records it rather than assuming + /// [`LEVELS`], which only bounds the levels appends write. pub(crate) fn levels(&self) -> u32 { self.axes.iter().rposition(|log| log.read(|pages| !pages.is_empty())).unwrap_or(0) as u32 } @@ -461,12 +480,13 @@ impl Samples { self.blocks = logs.blocks.into(); } - /// The log of row block `block` holding `level`'s values of `side`. + /// The log of row block `block` holding `level`'s `side` values. fn log(&self, block: usize, level: u32, side: usize) -> Option<&Log> { self.blocks.get(block)?.get(slot(level, side)) } - /// The page of `level`'s coordinates holding entry `index`, where this version holds it. + /// The coordinate page of `level` containing entry `index`, if this version reads that entry + /// and it is present. fn axis_page(&self, level: u32, index: u64) -> Option { if !self.visible(level).contains(&index) { return None; @@ -474,32 +494,34 @@ impl Samples { holding(self.axes.get(level as usize)?, index) } - /// Whether no entry is held, of any level. + /// Whether no entry of any level is present. pub(crate) fn is_empty(&self) -> bool { self.axes.iter().enumerate().all(|(level, log)| first_held(log, self.visible(level as u32)).is_none()) } - /// Whether entry `index` of `level` is held. + /// Whether entry `index` of `level` is present. pub(crate) fn held(&self, level: u32, index: u64) -> bool { self.axis_page(level, index).is_some() } - /// Coordinate `which` of entry `index` of `level`: a frame's own, a bucket's first (0) or last (1). + /// Coordinate `which` of entry `index` of `level`: a frame's own, or a bucket's first (0) or + /// last (1) frame's. pub(crate) fn coordinate(&self, level: u32, index: u64, which: u32) -> Option { self.axis_page(level, index)?.coordinate(index, which) } - /// Where entry `index` of `level` draws: a frame at its coordinate, a bucket at its middle. + /// The coordinate at which entry `index` of `level` is drawn: a frame's own, or a bucket's + /// midpoint. pub(crate) fn drawn_at(&self, level: u32, index: u64) -> Option { self.axis_page(level, index)?.drawn(index) } - /// The coordinate of `frame`, where it is held. + /// The coordinate of `frame`, if present. pub(crate) fn at(&self, frame: u64) -> Option { self.coordinate(0, frame, 0) } - /// The runs of entries held, level by level, each as long as held entries follow one another. + /// The runs of present entries, level by level, each run as long as its entries are contiguous. pub(crate) fn runs(&self) -> Vec { let mut runs = Vec::new(); for (level, log) in self.axes.iter().enumerate() { @@ -511,8 +533,8 @@ impl Samples { runs } - /// The runs bytes hold: every run held but the buckets whose eight entries a level below are all - /// held, since those follow from them. + /// The runs the binary format stores: every present run, minus the buckets whose eight entries + /// one level below are all present, since decoding derives those. pub(crate) fn stored_runs(&self) -> Vec { let runs = self.runs(); let mut stored = Vec::new(); @@ -521,7 +543,7 @@ impl Samples { stored.push(run.clone()); continue; } - // Buckets that follow from the runs below, in order: what this run leaves out. + // Buckets derivable from the runs one level below, in order. This run omits them. let derived = runs .iter() .filter(|r| r.level + 1 == run.level) @@ -545,7 +567,7 @@ impl Samples { stored } - /// The first and last coordinates of what this version holds, at any level. + /// The first and last coordinates of the entries this version reads, across all levels. pub(crate) fn span(&self) -> Option<[f64; 2]> { let mut span: Option<[f64; 2]> = None; for (level, log) in self.axes.iter().enumerate() { @@ -557,12 +579,14 @@ impl Samples { span } - /// The coordinate of the last frame, or of the last entry of any level. + /// The coordinate of the last frame, or if that frame is missing, the end of the span across + /// all levels. fn last_coordinate(&self) -> Option { self.end.checked_sub(1).and_then(|f| self.at(f)).or_else(|| self.span().map(|s| s[1])) } - /// Appends frames: `values` holds `at.len()` frames of `rows` values each, frame-major. + /// Appends frames: `values` holds `at.len()` frames of `rows` values each, frame-major. Errors + /// on a length mismatch, a non-finite coordinate, or a coordinate below the last one. pub(crate) fn append(&mut self, at: &[f64], values: &Values, rows: usize) -> Result<()> { if at.len().checked_mul(rows) != Some(values.len()) { return Err(Error::invalid(format!("{} values for {} frames of {rows} rows", values.len(), at.len()))); @@ -599,8 +623,9 @@ impl Samples { Ok(()) } - /// Appends frames of some rows: `values` holds `at.len()` frames of a value per row of `rows`, - /// frame-major. The rest of the `total` rows hold their last frame's values, missing before any. + /// Appends frames for some rows: `values` holds `at.len()` frames of one value per row in + /// `rows`, frame-major. The other rows of `total` repeat their value from the last frame, or + /// are missing if there is none. pub(crate) fn append_rows(&mut self, at: &[f64], values: &Values, rows: &[usize], total: usize) -> Result<()> { let given = rows.len(); if at.len().checked_mul(given) != Some(values.len()) { @@ -622,36 +647,40 @@ impl Samples { self.append(at, &Values::from(dense), total) } - /// Puts `run`, held nowhere yet, in: its `coordinates` ([`coordinates`] an entry), and its values - /// for `rows` rows, entry-major, at level 0 one a row, above each row's least then, after every - /// entry's, each row's greatest. What it spans past this version's frames widens them. + /// Inserts `run`, none of whose entries may be present yet. `coordinates` holds + /// [`coordinates`] per entry, and `values` holds its values for `rows` rows, entry-major. At + /// level 0 that is one value per row. Above it, every entry's least values come first, then + /// every entry's greatest. Frames the run covers outside this version's range extend the range. + /// + /// Errors if the level is above [`MAX_LEVEL`], the sizes do not match, the coordinates are not + /// finite and non-decreasing, the run starts before forgotten frames, or an entry is present. pub(crate) fn insert(&mut self, run: &Run, coordinates: &[f64], values: &[f32], rows: usize) -> Result<()> { let level = run.level; if level > MAX_LEVEL { - return Err(Error::invalid(format!("no level {level}: levels run to {MAX_LEVEL}"))); + return Err(Error::invalid(format!("level {level} is above the highest level, {MAX_LEVEL}"))); } let (per, sides) = (self::coordinates(level), sides(level)); if run.entries.is_empty() || run.sizes(rows) != Some((coordinates.len(), values.len())) { - return Err(Error::invalid(format!("a run of {} entries of level {level} for {rows} rows", run.len()))); + return Err(Error::invalid(format!("wrong sizes for {} level-{level} entries and {rows} rows", run.len()))); } if coordinates.iter().any(|c| !c.is_finite()) || coordinates.windows(2).any(|w| w[1] < w[0]) { return Err(Error::invalid("a run's coordinates must be finite and must not decrease")); } let frames = run.frames(); if run.entries.end.checked_mul(bucket_frames(level)).is_none() { - return Err(Error::invalid("a run past the last frame a field numbers")); + return Err(Error::invalid("the run ends past the largest frame number")); } if frames.start < self.forgotten { - return Err(Error::invalid("a run before frames the field forgot")); + return Err(Error::invalid("the run starts at a forgotten frame")); } let visible = self.visible(level); let within = run.entries.start.max(visible.start)..run.entries.end.min(visible.end); if self.axes.get(level as usize).is_some_and(|log| first_held(log, within).is_some()) { - return Err(Error::invalid(format!("entries {:?} of level {level} are held already", run.entries))); + return Err(Error::invalid(format!("entries {:?} of level {level} are already present", run.entries))); } let blocks = rows.div_ceil(PAGE_ROWS); let mut logs = self.logs_at(blocks, level); - // What a newer version grew past this one's frames stays the newer version's. + // Entries a newer version appended past this version's frames stay the newer version's. cap(&mut logs.axes[level as usize], visible.end); put(&mut logs.axes[level as usize], &run.entries, coordinates, per, FRAME_PAGE_WORDS / 2); let n = run.len(); @@ -678,7 +707,8 @@ impl Samples { Ok(()) } - /// Takes the pages holding only entries of `run` out: a span let go of. + /// Removes the pages whose entries all lie within `run`, as when a time tile span is no longer + /// needed. pub(crate) fn remove(&mut self, run: &Run) { let mut logs = self.logs(0); take_out(&mut logs.axes[run.level as usize], &run.entries); @@ -690,8 +720,8 @@ impl Samples { self.keep(logs); } - /// Writes each bucket of `level` not held whose eight entries a level below are all held: what - /// bytes leave out, as they follow from those. + /// Writes each missing bucket of `level` whose eight entries one level below are all present. + /// Decoding uses it to rebuild the buckets the binary format omits. pub(crate) fn derive(&mut self, level: u32) { let below: Vec = self.runs().into_iter().filter(|r| r.level + 1 == level).collect(); for run in below { @@ -699,12 +729,12 @@ impl Samples { for missing in self.missing(level, buckets) { let (coordinates, values, rows) = self.bucket_values(level, &missing); let run = Run { level, entries: missing }; - self.insert(&run, &coordinates, &values, rows).expect("buckets held nowhere"); + self.insert(&run, &coordinates, &values, rows).expect("only missing buckets are inserted"); } } } - /// The parts of `entries` of `level` not held. + /// The ranges of `entries` of `level` that are not present. fn missing(&self, level: u32, entries: Range) -> Vec> { let visible = self.visible(level); let held_runs = self @@ -725,8 +755,8 @@ impl Samples { out } - /// Buckets `buckets` of `level`, each from its eight entries a level below: their coordinates, and - /// their values as [`insert`](Samples::insert) takes them, with how many rows. + /// Computes buckets `buckets` of `level` from their eight entries one level below: their + /// coordinates, their values in [`insert`](Samples::insert)'s layout, and the row count. fn bucket_values(&self, level: u32, buckets: &Range) -> (Vec, Vec, usize) { let children = buckets.start * 8..buckets.end * 8; let mut coordinates = Vec::with_capacity(2 * (buckets.end - buckets.start) as usize); @@ -759,7 +789,7 @@ impl Samples { (coordinates, values, rows) } - /// Rows the row blocks' pages hold values of. + /// The number of rows the row blocks' pages store values for. fn rows(&self) -> usize { let block_rows = |b: usize| self.blocks[b].iter().filter_map(|log| log.last().map(|p| p.per as usize)).max(); (0..self.blocks.len()) @@ -768,8 +798,8 @@ impl Samples { .unwrap_or(0) } - /// `run`'s coordinates, and its values for rows `rows`, in that order, as - /// [`insert`](Samples::insert) takes them. + /// `run`'s coordinates, and its values for rows `rows` in that order, in + /// [`insert`](Samples::insert)'s layout. Missing entries read as NaN. pub(crate) fn read(&self, run: &Run, rows: &[usize]) -> (Vec, Vec) { let per = coordinates(run.level); let n = run.len(); @@ -781,7 +811,7 @@ impl Samples { at.resize(n * per, f64::NAN); let sides = sides(run.level); let mut values = vec![f32::NAN; sides * n * rows.len()]; - // Rows by block, so each page is read once for all of a block's rows. + // Group rows by row block, so each page is read once for all of a block's rows. let mut by_block: Vec<(usize, usize)> = rows.iter().enumerate().map(|(i, &row)| (row >> PAGE_BITS, i)).collect(); by_block.sort_unstable(); @@ -806,8 +836,8 @@ impl Samples { (at, values) } - /// Forgets frames before coordinate `before`; pages that hold only forgotten entries drop. True - /// when anything was forgotten. + /// Forgets frames before coordinate `before`, dropping pages that contain only forgotten + /// entries. Returns whether anything was forgotten. pub(crate) fn forget(&mut self, before: f64) -> bool { let first = self.bound(before, false).min(self.end); if first <= self.start { @@ -827,9 +857,10 @@ impl Samples { true } - /// How many entries of `level` lie before `at` (with `past`, at it too), each by its coordinate - /// `which`: by its last (1), those wholly before; by its first (0), those begun before. Where the - /// entries held of the level settle it; else where the hole holding `at` starts. + /// Counts the entries of `level` before `at`, or at or before it with `past`. Compares each + /// entry's coordinate `which`. The last coordinate (1) counts entries wholly before `at`, and + /// the first (0) counts entries begun before it. Returns `Ok(count)` when the present entries + /// determine it, or `Err(start)` with the first entry of the gap containing `at`. fn locate(&self, level: u32, at: f64, past: bool, which: u32) -> std::result::Result { let before = |t: f64| if past { t <= at } else { t < at }; let visible = self.visible(level); @@ -859,9 +890,10 @@ impl Samples { if n > 0 || range.start == gap { Ok(range.start + n) } else { Err(gap) } } - /// The frames before `at`, or with `past` at or before it: frames where held, else the whole - /// buckets before it of the finest level held there. Where no level settles it, where the hole - /// holding it starts. + /// The number of frames before `at` (at or before it, with `past`), as an absolute frame index + /// clamped to `start..=end`. Counts frames where present, and elsewhere whole buckets of the + /// finest level present. If no level determines it, returns where the gap containing `at` + /// starts. pub(crate) fn bound(&self, at: f64, past: bool) -> u64 { let mut hole = None; for level in 0..self.axes.len() as u32 { @@ -876,9 +908,10 @@ impl Samples { hole.unwrap_or(self.start).clamp(self.start, self.end) } - /// The frames at or before `at`, as finely as the finest level settling it tells: where frames - /// are held, exactly; else the whole buckets at or before it and as far into the next as `at` lies - /// between its first and last frame's coordinates. + /// The fractional number of frames at or before `at`, as precise as the finest level that + /// determines it. Exact where frames are present. Elsewhere it is the whole buckets at or + /// before `at`, plus a fraction of the next bucket set by where `at` lies between that bucket's + /// first and last frame coordinates. fn position(&self, at: f64) -> Option { (0..self.axes.len() as u32).find_map(|level| { let n = self.locate(level, at, true, 1).ok()?; @@ -890,8 +923,9 @@ impl Samples { }) } - /// The frames a trace across `across` draws: from the last frame at or before its start through - /// the first after its end, as finely as what is held tells. + /// The frames a trace over coordinates `across` draws. They run from the last frame at or + /// before its start through the first frame after its end, as precisely as the present levels + /// allow. Returns all frames if either bound cannot be determined. pub(crate) fn across(&self, [x0, x1]: [f64; 2]) -> Range { let (Some(a), Some(b)) = (self.position(x0), self.position(x1)) else { return self.start..self.end }; let first = (a.floor() as u64).saturating_sub(1).max(self.start); @@ -899,7 +933,7 @@ impl Samples { first..last.max(first) } - /// The first frame held from `frame` on, of any level: where a hole ends. + /// The first frame from `frame` on that any level has present, which is where a gap ends. fn next_held(&self, frame: u64) -> Option { let mut next: Option = None; for (level, log) in self.axes.iter().enumerate() { @@ -912,9 +946,10 @@ impl Samples { next } - /// What covers frames `frames`, left to right: `(level, index)`, the coarsest held bucket that fits - /// at each step, or a held frame as level 0; where neither does, the finest held bucket holding the - /// frame stands in, reaching past the frames. Frames nothing holds are skipped. + /// The entries covering frames `frames`, left to right, as `(level, index)`. Each step takes + /// the coarsest present bucket that fits in the remaining frames, or a present frame (level 0). + /// If neither exists, it takes the finest present bucket containing the frame, which can extend + /// past `frames`. Frames no level has are skipped. pub(crate) fn pieces(&self, frames: Range) -> Vec<(u32, u64)> { let mut pieces = Vec::new(); let (mut f, end) = (frames.start.max(self.start), frames.end.min(self.end)); @@ -941,9 +976,10 @@ impl Samples { pieces } - /// What draws frames `frames` at `level`, left to right: runs of that level's entries where held; - /// else the nearest coarser level's entry holding them, standing in; else finer levels' entries. - /// Frames nothing holds draw nothing. + /// The runs that draw frames `frames` at `level`, left to right. Uses that level's entries + /// where present. Elsewhere it uses, as a placeholder, the entry of the nearest coarser level + /// that contains them, and failing that, entries of finer levels. Frames no level has are not + /// drawn. pub(crate) fn cover(&self, frames: Range, level: u32) -> Vec { let mut runs = Vec::new(); let frames = frames.start.max(self.start)..frames.end.min(self.end); @@ -983,8 +1019,8 @@ impl Samples { } } - /// Folds piece `(level, index)` of row block `block` into `extent`, a least and a greatest value per - /// row of the block, NaN until one is read. + /// Folds piece `(level, index)` of row block `block` into `extent`, which holds a least and a + /// greatest value per row of the block (NaN until a value is read). pub(crate) fn fold(&self, block: usize, (level, index): (u32, u64), extent: &mut [[f32; 2]]) { if !self.held(level, index) { return; @@ -1005,13 +1041,15 @@ impl Samples { } } - /// The last frame held at or before `at`. + /// The last frame at or before `at`. `None` if `at` lies before the first frame or in a gap in + /// the frames. pub(crate) fn frame_at(&self, at: f64) -> Option { let frame = self.locate(0, at, true, 0).ok()?.checked_sub(1)?; self.held(0, frame).then_some(frame) } - /// How far `at` lies from `frame` toward the next frame, in `0..=1`. + /// How far `at` lies from `frame` toward the next frame, in `0..=1`. 0 if the next frame is + /// missing. pub(crate) fn t(&self, frame: u64, at: f64) -> f32 { match (self.at(frame), self.at(frame + 1)) { (Some(a), Some(b)) if b > a => ((at - a) / (b - a)).clamp(0.0, 1.0) as f32, @@ -1019,7 +1057,8 @@ impl Samples { } } - /// The page holding `frame` of `block` and the word where the frame starts in it. + /// The frame page of row block `block` containing `frame`, and the word offset where that + /// frame starts in it. pub(crate) fn cursor(&self, block: usize, frame: u64) -> Option<(FramePage, u32)> { if !self.held(0, frame) { return None; @@ -1029,8 +1068,9 @@ impl Samples { Some((page, base as u32)) } - /// The pages of `block` holding any of `level`'s entries `entries`, each with those it holds: - /// frames, or a bucket's least (`side` 0) or greatest (1) values. + /// The pages of row block `block` containing any of `level`'s `entries` that this version + /// reads, each with the range it contains: frames, or buckets' least (`side` 0) or greatest (1) + /// values. pub(crate) fn pages( &self, block: usize, @@ -1043,7 +1083,8 @@ impl Samples { self.log(block, level, side).map_or_else(Vec::new, |log| held(log, entries)) } - /// The coordinate pages of `level` holding any of entries `entries`, each with those it holds. + /// The coordinate pages of `level` containing any of `entries` that this version reads, each + /// with the range it contains. pub(crate) fn axis_pages(&self, level: u32, entries: Range) -> Vec<(AxisPage, Range)> { let visible = self.visible(level); let entries = entries.start.max(visible.start)..entries.end.min(visible.end); @@ -1055,8 +1096,8 @@ impl Samples { self.entry(row, 0, 0, frame) } - /// The value of `row` at entry `index` of `level`: a frame's, or a bucket's least (`side` 0) or - /// greatest (1); `None` when missing. + /// The value of `row` at entry `index` of `level`. That is a frame's value, or a bucket's least + /// (`side` 0) or greatest (1). `None` when missing. pub(crate) fn entry(&self, row: usize, level: u32, side: usize, index: u64) -> Option { if !self.held(level, index) { return None; @@ -1068,7 +1109,8 @@ impl Samples { (!value.is_nan()).then_some(f64::from(value)) } - /// `frame`'s values for the first `rows` rows into `out`; rows the frame lacks are NaN. + /// Writes `frame`'s values for the first `out.len()` rows into `out`. Rows without a value are + /// NaN. fn read_frame(&self, frame: u64, out: &mut [f32]) { out.fill(f32::NAN); let rows = out.len(); @@ -1085,36 +1127,22 @@ impl Samples { self.gather_runs(rows, &self.runs(), self.start..self.end) } - /// The samples of rows `rows`, in that order, over the frames of `frames` held: what a table drawn - /// in part shows of them about one coordinate. - pub(crate) fn gather_frames(&self, rows: &[usize], frames: Range) -> Samples { - let start = frames.start.max(self.start); - let frames = start..frames.end.min(self.end).max(start); - let runs: Vec = self - .runs() - .into_iter() - .filter(|r| r.level == 0) - .map(|r| Run { level: 0, entries: r.entries.start.max(frames.start)..r.entries.end.min(frames.end) }) - .filter(|r| !r.entries.is_empty()) - .collect(); - self.gather_runs(rows, &runs, frames) - } - /// The samples of rows `rows` over `runs`, reading frames `frames`. fn gather_runs(&self, rows: &[usize], runs: &[Run], frames: Range) -> Samples { let mut gathered = Samples::from_frame(frames.start); for run in runs { let (at, values) = self.read(run, rows); - gathered.insert(run, &at, &values, rows.len()).expect("runs held apart"); + gathered.insert(run, &at, &values, rows.len()).expect("runs do not overlap"); } gathered.reads(frames); gathered } } -/// After `frame` is appended to `logs`, writes each bucket it completes whose eight entries a level -/// below are all held, level by level: its first and last frame's coordinates, and per row block the -/// least of its entries' least values and the greatest of their greatest. +/// After `frame` is appended to `logs`, writes each bucket it completes, level by level, while the +/// bucket's eight entries one level below are all present. A bucket stores its first and last +/// frame's coordinates and, per row block, the least of its entries' least values and the greatest +/// of their greatest values. fn seal(logs: &mut Logs, frame: u64) { let next = frame + 1; for level in 1..=LEVELS { @@ -1165,7 +1193,7 @@ mod tests { let v = if t == 70 { -5.0 } else { f64::from(t) }; samples.append(&[f64::from(t)], &Values::from(vec![v, 2.0 * v]), 2).expect("frame"); } - assert!(samples.held(1, 15) && !samples.held(1, 16), "130 frames seal 16 buckets of 8"); + assert!(samples.held(1, 15) && !samples.held(1, 16), "130 frames complete 16 buckets of 8"); assert!(samples.held(2, 1) && !samples.held(2, 2), "and 2 of 64"); // Bucket 8 spans frames 64..72, where frame 70 dips to -5. assert_eq!(samples.entry(0, 1, 0, 8), Some(-5.0)); @@ -1208,12 +1236,13 @@ mod tests { } #[test] - fn an_older_version_writing_past_its_end_keeps_the_newer_ones_frames_out() { + fn an_older_version_writing_past_its_end_ignores_newer_frames() { let mut newest = Samples::default(); newest.append(&[0.0, 1.0], &Values::from(vec![1.0_f64, 2.0]), 1).expect("frames"); let mut older = newest.clone(); newest.append(&[2.0, 3.0], &Values::from(vec![3.0_f64, 4.0]), 1).expect("frames"); - // The older version holds frames 0 and 1; frames 4 and 5 widen it past a hole at 2 and 3. + // The older version has frames 0 and 1. Inserting frames 4 and 5 extends it, leaving a gap + // at 2 and 3. older.insert(&Run { level: 0, entries: 4..6 }, &[4.0, 5.0], &[5.0, 6.0], 1).expect("a run"); assert_eq!(older.value(0, 2), None, "the newer version's frame 2 is not the older one's"); assert_eq!(older.value(0, 4), Some(5.0)); @@ -1240,7 +1269,7 @@ mod tests { samples.append(&[100.0], &Values::from(vec![1.0_f32; rows]), rows).expect("frame"); assert_eq!(samples.end(), 101); let run = Run { level: 0, entries: 10..11 }; - assert!(samples.insert(&run, &[10.0], &vec![0.0; rows], rows).is_err(), "frames forgotten stay so"); + assert!(samples.insert(&run, &[10.0], &vec![0.0; rows], rows).is_err(), "forgotten frames stay forgotten"); } #[test] @@ -1250,8 +1279,6 @@ mod tests { let gathered = samples.gather(&[1]); assert_eq!(gathered.value(0, 0), Some(2.0)); assert_eq!(gathered.value(0, 1), Some(4.0)); - let one = samples.gather_frames(&[0], 1..2); - assert_eq!((one.start(), one.end(), one.value(0, 1), one.value(0, 0)), (1, 2, Some(3.0), None)); } /// Level 1 buckets 0..4 (frames 0..32) and the frames of bucket 1, of two rows. @@ -1270,22 +1297,22 @@ mod tests { } #[test] - fn runs_fill_any_span_and_cover_draws_the_finest_held() { + fn runs_fill_any_span_and_cover_draws_the_finest_stored() { let (mut samples, buckets, frames) = holed(); assert_eq!((samples.start(), samples.end()), (0, 32)); let (at, values) = samples.read(&frames, &[0, 1]); - assert!(samples.insert(&frames, &at, &values, 2).is_err(), "held already"); + assert!(samples.insert(&frames, &at, &values, 2).is_err(), "already present"); assert_eq!( samples.cover(0..32, 0), vec![Run { level: 1, entries: 0..1 }, Run { level: 0, entries: 8..16 }, Run { level: 1, entries: 2..4 },], - "frames where held, else the buckets standing in" + "frames where stored, buckets elsewhere" ); assert_eq!(samples.cover(0..32, 1), vec![buckets.clone()]); assert_eq!(samples.entry(1, 1, 1, 2), Some(8.0)); assert_eq!(samples.value(0, 9), Some(9.0)); assert_eq!(samples.frame_at(12.5), Some(12)); - assert_eq!(samples.frame_at(20.0), None, "frame 20 is not held"); - assert_eq!(samples.bound(20.0, false), 16, "past the frames, whole buckets count"); + assert_eq!(samples.frame_at(20.0), None, "frame 20 is not stored"); + assert_eq!(samples.bound(20.0, false), 16, "past the stored frames, bounds fall on whole buckets"); assert_eq!(samples.pieces(0..32), vec![(1, 0), (1, 1), (1, 2), (1, 3)]); let read = samples.read(&buckets, &[0, 1]); let mut copy = Samples::default(); @@ -1295,7 +1322,7 @@ mod tests { let stored = vec![frames.clone(), Run { level: 1, entries: 0..1 }, Run { level: 1, entries: 2..4 }]; assert_eq!(samples.stored_runs(), stored); samples.remove(&frames); - assert!(!samples.held(0, 9), "a span let go of"); + assert!(!samples.held(0, 9), "a removed span is no longer stored"); assert_eq!(samples.cover(0..32, 0), vec![buckets]); } @@ -1323,7 +1350,7 @@ mod tests { } #[test] - fn appends_after_a_run_seal_only_whole_buckets() { + fn appends_after_a_run_seal_only_complete_buckets() { let mut samples = Samples::default(); let run = Run { level: 0, entries: 4..8 }; samples.insert(&run, &[4.0, 5.0, 6.0, 7.0], &[1.0, 2.0, 3.0, 4.0], 1).expect("run"); diff --git a/crates/latkit/src/data/dataset.rs b/crates/latkit/src/data/snapshot.rs similarity index 50% rename from crates/latkit/src/data/dataset.rs rename to crates/latkit/src/data/snapshot.rs index 11cda757..a7226aa4 100644 --- a/crates/latkit/src/data/dataset.rs +++ b/crates/latkit/src/data/snapshot.rs @@ -3,32 +3,32 @@ use std::sync::Arc; use rustc_hash::FxHasher; -use super::name::{Name, Row, Version}; +use super::name::{Id, Name, Row, Version}; use super::patch::{Appends, Op, Patch}; use super::samples::Samples; use super::table::{Field, Table}; use super::values::Value; use crate::{Error, Result}; -/// A named collection of tables, changed in place by patches. +/// One immutable version of a dataset, taken with [`Data::snapshot`](crate::Data::snapshot). +/// Taking one is O(1), and it shares unchanged pages with the versions before and after it. A view +/// given a snapshot draws that version. /// -/// Cloning is O(1) and a clone is a snapshot: [`apply`](Data::apply) changes only `self`, and every -/// page it does not write stays shared with earlier snapshots. +/// Later changes to the dataset never affect a snapshot. Its description, values, queries, and +/// envelopes stay the same. /// /// ``` -/// # use latkit::{Data, Ids, Patch, Table}; -/// let mut grid = Data::new("grid"); -/// grid.apply( -/// Patch::new() -/// .axis("time", "s") -/// .replace(Table::new("Bus", vec![1_u64, 2]).field("kv", vec![230.0, 115.0])) -/// .replace(Table::new("Branch", Ids::count(1)).reference("from", "Bus", vec![1_u64]).reference("to", "Bus", vec![2_u64])), -/// )?; -/// grid.apply(Patch::new().append([0.0], "Bus", [("vm", vec![1.01, 0.99])]))?; +/// # use latkit::{Data, Patch, Table, Value}; +/// let grid = Data::new("grid"); +/// grid.apply(Table::new("Bus", vec![1_u64, 2]).field("kv", vec![230.0, 115.0]))?; +/// let before = grid.snapshot(); +/// grid.apply(Patch::new().remove("Bus", [2_u64]))?; +/// assert_eq!(before.get("Bus", 2_u64, "kv", None), Some(Value::Number(115.0))); +/// assert_eq!(grid.snapshot().get("Bus", 2_u64, "kv", None), None); /// # Ok::<(), latkit::Error>(()) /// ``` #[derive(Clone)] -pub struct Data(pub(crate) Arc); +pub struct Snapshot(pub(crate) Arc); #[derive(Clone)] pub(crate) struct Inner { @@ -37,18 +37,27 @@ pub(crate) struct Inner { /// Sorted by name. pub(crate) tables: Vec
, pub(crate) version: Version, - /// The version the dataset began at, which every version applied from it shares: a view eases - /// rows from one of them to the next, and replaces the rows of another dataset of the same name. + /// The version the dataset started from. Every version that patches derive from it shares it. + /// A view matches rows across versions with the same origin to animate them. For a same-named + /// dataset with a different origin, every old row leaves and every new row enters. pub(crate) origin: Version, - /// Changes when a table, field, or kind is added or changed; marks recompile only then. + /// A hash of the table names and the fields' names, kinds, and targets. Marks recompile only + /// when it changes. pub(crate) schema: u64, } -impl Data { +impl Snapshot { /// An empty dataset named `name`. - pub fn new(name: impl Into) -> Data { + pub(crate) fn new(name: impl Into) -> Snapshot { let version = Version::next(); - Data(Arc::new(Inner { name: name.into(), axis: None, tables: Vec::new(), version, origin: version, schema: 0 })) + Snapshot(Arc::new(Inner { + name: name.into(), + axis: None, + tables: Vec::new(), + version, + origin: version, + schema: 0, + })) } /// The dataset's name. @@ -57,22 +66,22 @@ impl Data { } /// The name and unit of the axis sampled fields are recorded over. - pub fn axis(&self) -> Option<(&str, &str)> { + pub(crate) fn axis(&self) -> Option<(&str, &str)> { self.0.axis.as_ref().map(|(name, unit)| (name.as_str(), unit.as_str())) } - /// Every table, by name. - pub fn tables(&self) -> impl Iterator + '_ { + /// Every table, sorted by name. + pub(crate) fn tables(&self) -> impl Iterator + '_ { self.0.tables.iter() } /// The table named `name`. - pub fn table(&self, name: &str) -> Option<&Table> { + pub(crate) fn table(&self, name: &str) -> Option<&Table> { self.0.table(name) } - /// The axis coordinates the sampled fields span. - pub fn span(&self) -> Option<[f64; 2]> { + /// The first and last axis coordinates held by any sampled field. + pub(crate) fn span(&self) -> Option<[f64; 2]> { let spans = self.0.tables.iter().flat_map(|t| &t.fields).filter_map(|(_, field)| match field { Field::Sampled(samples, _) => samples.span(), _ => None, @@ -80,48 +89,39 @@ impl Data { spans.reduce(|a, b| [a[0].min(b[0]), a[1].max(b[1])]) } - /// The value of `field` for `row`; a sampled field reads its last frame. - pub fn get(&self, row: &Row, field: &str) -> Option { - self.get_at(row, field, f64::INFINITY) - } - - /// The value of `field` for `row` at axis coordinate `at`: the last frame at or before it. - pub fn get_at(&self, row: &Row, field: &str, at: f64) -> Option { - if row.data != self.0.name { - return None; - } - let table = self.table(&row.table)?; - let index = table.row(row.id.as_ref())?; - match table.field_named(field)? { + /// The value of `field` for the row of `table` with `id`, or `None` if the table, row, field, + /// or value is missing. A sampled field reads its last frame at or before axis coordinate `at`, + /// or its last frame when `at` is `None`. A reference returns the [`Row`] it names. + pub fn get(&self, table: &str, id: impl Into, field: &str, at: impl Into>) -> Option { + let rows = self.table(table)?; + let index = rows.row(id.into().as_ref())?; + match rows.field_named(field)? { Field::Reference(reference) => { let Value::Number(target) = reference.rows.get(index)? else { return None }; - let target_table = self.table(&reference.to)?; - let id = target_table.ids.id(target as usize)?; + let id = self.table(&reference.to)?.ids.id(target as usize)?; Some(Value::Row(Row { data: self.0.name.clone(), table: reference.to.clone(), id })) } - _ => table.value(index, field, at), + _ => rows.value(index, field, at.into().unwrap_or(f64::INFINITY)), } } - /// Applies every change in `patch`, or none of them when one is invalid. A patch to another - /// dataset is an error. - /// - /// The error's path names the change and part, such as `patch[1].Bus.kv`. - pub fn apply(&mut self, patch: impl Into) -> Result<()> { - let patch = patch.into(); + /// Returns the version `patch` makes from this one. Applies all changes or none. If a change is + /// invalid, the error's path names the change and part, such as `patch[1].Bus.kv`. A patch for + /// another dataset is an error. + pub(crate) fn applied(&self, patch: Patch) -> Result { if let Some(data) = &patch.data && *data != self.0.name { return Err(Error::invalid(format!("a patch to '{data}' applied to '{}'", self.0.name))); } if patch.ops.is_empty() { - return Ok(()); + return Ok(self.clone()); } let mut next = (*self.0).clone(); let mut ops = patch.ops.into_iter().enumerate().peekable(); while let Some((i, op)) = ops.next() { let op = match op { - // A run of appends to the same fields of every row of a table records as one. + // Merge consecutive appends to the same fields of every row of a table into one. Op::Append { table, at, ids: None, fields } => { let rows = next.table(&table).map(Table::len); let mut run = Appends::new(table, at, fields); @@ -140,11 +140,10 @@ impl Data { next.resolve()?; next.schema = next.signature(); next.version = Version::next(); - self.0 = Arc::new(next); - Ok(()) + Ok(Snapshot(Arc::new(next))) } - /// The patch to this dataset that builds it from an empty one. + /// The patch that builds this dataset from an empty one. It names this dataset. pub fn to_patch(&self) -> Patch { let mut patch = Patch::to(self.0.name.clone()); if let Some((name, unit)) = &self.0.axis { @@ -153,65 +152,46 @@ impl Data { self.0.tables.iter().fold(patch, |patch, table| patch.replace(table.clone())) } - /// The dataset as bytes: the patch to it that builds it. + /// Encodes the dataset's [`to_patch`](Snapshot::to_patch) as bytes. pub fn to_bytes(&self) -> Vec { self.to_patch().to_bytes() } - /// A dataset from [`to_bytes`](Data::to_bytes), or any bytes of a patch to a dataset. - pub fn from_bytes(bytes: &[u8]) -> Result { + /// Decodes a dataset from bytes written by [`to_bytes`](Snapshot::to_bytes). Returns an error + /// if the bytes do not hold a patch that names a dataset. + pub(crate) fn from_bytes(bytes: &[u8]) -> Result { let patch = Patch::from_bytes(bytes)?; let Some(name) = patch.data.clone() else { - return Err(Error::invalid("the bytes hold a patch to no dataset").at("bytes")); + return Err(Error::invalid("the bytes hold a patch that names no dataset").at("bytes")); }; - let mut data = Data::new(name); - data.apply(patch)?; - Ok(data) + Snapshot::new(name).applied(patch) } - /// A dataset from bytes: latkit's own, as [`to_bytes`](Data::to_bytes) writes them, or GeoJSON, - /// gzip-compressed or not. GeoJSON's features become rows of tables `points`, `lines`, and - /// `polygons` by their geometry, each with its geometry as field `geometry` and its properties as - /// fields. `name` names the dataset, as GeoJSON needs; latkit's bytes name their own unless given - /// one. - /// - /// ``` - /// let city = latkit::Data::read(br#"{"type": "Feature", "properties": {"name": "Hall"}, - /// "geometry": {"type": "Point", "coordinates": [-97.74, 30.27]}}"#, Some("city"))?; - /// assert_eq!(city.table("points").map(latkit::Table::len), Some(1)); - /// # Ok::<(), latkit::Error>(()) - /// ``` - pub fn read(bytes: &[u8], name: Option<&str>) -> Result { - if bytes.starts_with(b"LATK") { - let data = Data::from_bytes(bytes)?; - return Ok(match name { - Some(name) if name != data.name().as_str() => data.renamed(name), - _ => data, - }); - } - let name = name.ok_or_else(|| Error::invalid("GeoJSON names its dataset").at("name"))?; - crate::geo::geojson(name, bytes) + /// The same tables as a new dataset named `name`, with a new origin, so views treat it as a + /// separate dataset. + pub(crate) fn renamed(&self, name: impl Into) -> Snapshot { + let version = Version::next(); + Snapshot(Arc::new(Inner { name: name.into(), version, origin: version, ..(*self.0).clone() })) } - /// The same tables as dataset `name`: another dataset, which views draw apart from this one. - #[must_use] - pub fn renamed(&self, name: impl Into) -> Data { - let version = Version::next(); - Data(Arc::new(Inner { name: name.into(), version, origin: version, ..(*self.0).clone() })) + /// This snapshot's version. Snapshots with the same version are equal. + pub(crate) fn version(&self) -> Version { + self.0.version } pub(crate) fn schema(&self) -> u64 { self.0.schema } - /// The version the dataset began at; see [`Inner::origin`]. + /// The version the dataset started from; see [`Inner::origin`]. pub(crate) fn origin(&self) -> Version { self.0.origin } - /// Changes sampled field `field` of table `table` with `change`, starting it empty where the table - /// has no field of its name: what time tiles put in and let go of. Whether anything changed: not - /// where `change` says so, or where there is no such table or the field holds no samples. + /// Updates sampled field `field` of table `table` with `change`, starting from empty samples if + /// the field does not exist. Time tiles use it to insert and remove spans. Returns whether + /// anything changed: false if `change` returns false, the table does not exist, or the field is + /// not sampled. pub(crate) fn sampled(&mut self, table: &str, field: &str, change: impl FnOnce(&mut Samples) -> bool) -> bool { let Ok(i) = self.0.position(table) else { return false }; let mut samples = match self.0.tables[i].field_named(field) { @@ -229,31 +209,17 @@ impl Data { self.0 = Arc::new(next); true } - - /// A version of `self` holding `tables` instead, its name, axis, schema, and origin kept: what a - /// view draws of a dataset it draws in part, eased from one version to the next like `self`'s. - pub(crate) fn drawn(&self, mut tables: Vec
) -> Data { - crate::order::by_name(&mut tables, |t| t.name.as_str()); - Data(Arc::new(Inner { - name: self.0.name.clone(), - axis: self.0.axis.clone(), - tables, - version: Version::next(), - origin: self.0.origin, - schema: self.0.schema, - })) - } } -impl PartialEq for Data { - fn eq(&self, other: &Data) -> bool { +impl PartialEq for Snapshot { + fn eq(&self, other: &Snapshot) -> bool { self.0.version == other.0.version } } -impl std::fmt::Debug for Data { +impl std::fmt::Debug for Snapshot { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("Data").field("name", &self.0.name).field("tables", &self.0.tables).finish() + f.debug_struct("Snapshot").field("name", &self.0.name).field("tables", &self.0.tables).finish() } } @@ -285,7 +251,7 @@ impl Inner { Ok(()) } - /// A deterministic digest of every table's name and every field's name, kind, and target. + /// A deterministic hash of every table's name and every field's name, kind, and target. fn signature(&self) -> u64 { let mut hasher = FxHasher::default(); for table in &self.tables { diff --git a/crates/latkit/src/data/source.rs b/crates/latkit/src/data/source.rs new file mode 100644 index 00000000..e23f7731 --- /dev/null +++ b/crates/latkit/src/data/source.rs @@ -0,0 +1,176 @@ +//! Dataset sources for views: a [`Data`] that changes in place and that views follow, a fixed +//! [`Snapshot`], or [`Tiles`](crate::Tiles) fetched in pieces. + +use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; + +use super::name::Name; +use super::patch::Patch; +use super::snapshot::Snapshot; +use crate::pending::Watchers; +use crate::{Error, Result}; + +/// A dataset that changes in place. It holds named tables of columns, and only patches change it. +/// Clones share the same dataset. A view drawing it shows the latest version from the frame after a +/// patch applies. A [`Snapshot`] keeps one version unchanged and costs only the pages that later +/// versions rewrite. +/// +/// ``` +/// # use latkit::{Data, Ids, Patch, Table}; +/// let grid = Data::new("grid"); +/// grid.apply( +/// Patch::new() +/// .axis("time", "s") +/// .replace(Table::new("Bus", vec![1_u64, 2]).field("kv", vec![230.0, 115.0])) +/// .replace(Table::new("Branch", Ids::count(1)).reference("from", "Bus", vec![1_u64]).reference("to", "Bus", vec![2_u64])), +/// )?; +/// let before = grid.snapshot(); +/// grid.apply(Patch::new().append("Bus", [0.0], [("vm", vec![1.01, 0.99])]))?; +/// grid.restore(&before)?; +/// # Ok::<(), latkit::Error>(()) +/// ``` +#[derive(Clone)] +pub struct Data(Arc); + +struct Shared { + name: Name, + current: Mutex, + /// A waker per view drawing the dataset, by view id. Applying a patch wakes each one to request + /// a frame. + watchers: Watchers, +} + +impl Data { + /// An empty dataset named `name`. + pub fn new(name: impl Into) -> Data { + Data::from(Snapshot::new(name)) + } + + /// The dataset's name. + pub fn name(&self) -> &Name { + &self.0.name + } + + /// Applies every change in `patch`, or none if any is invalid. A patch for another dataset is + /// an error. Views drawing the dataset show the new version from their next frame. + /// + /// The error's path names the change and part, such as `patch[1].Bus.kv`. + pub fn apply(&self, patch: impl Into) -> Result<()> { + let mut current = self.current(); + *current = current.applied(patch.into())?; + drop(current); + self.0.watchers.wake_all(); + Ok(()) + } + + /// The current version, which later patches do not change. O(1). + pub fn snapshot(&self) -> Snapshot { + self.current().clone() + } + + /// Makes `snapshot`, a version of this dataset, the current version, like an undo. Views + /// drawing the dataset show it from their next frame. A snapshot of another dataset (by name) + /// is an error. + pub fn restore(&self, snapshot: &Snapshot) -> Result<()> { + if snapshot.name() != self.name() { + return Err(Error::invalid(format!("a snapshot of '{}' restored to '{}'", snapshot.name(), self.name()))); + } + *self.current() = snapshot.clone(); + self.0.watchers.wake_all(); + Ok(()) + } + + /// The views drawing the dataset, which applying a patch wakes to request a frame. + #[cfg(feature = "gpu")] + pub(crate) fn watchers(&self) -> &Watchers { + &self.0.watchers + } + + fn current(&self) -> MutexGuard<'_, Snapshot> { + self.0.current.lock().unwrap_or_else(PoisonError::into_inner) + } +} + +impl From for Data { + /// A dataset whose current version is `snapshot`. Patches apply from there. + fn from(snapshot: Snapshot) -> Data { + Data(Arc::new(Shared { + name: snapshot.name().clone(), + current: Mutex::new(snapshot), + watchers: Watchers::default(), + })) + } +} + +impl std::fmt::Debug for Data { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_tuple("Data").field(&*self.current()).finish() + } +} + +/// Where a view gets a dataset, matched by the dataset's name. A [`Data`] is drawn as it changes, a +/// [`Snapshot`] is drawn unchanged, and [`Tiles`](crate::Tiles) are fetched in pieces as the view +/// needs them. +#[cfg(feature = "gpu")] +#[derive(Clone, Debug)] +pub enum Source { + /// A dataset drawn as it changes. + Data(Data), + /// A fixed version, drawn unchanged. + Snapshot(Snapshot), + /// Pieces fetched as the view needs them. + Tiles(crate::Tiles), +} + +#[cfg(feature = "gpu")] +impl Source { + /// The name of the dataset it provides. + pub fn name(&self) -> &Name { + match self { + Source::Data(data) => data.name(), + Source::Snapshot(snapshot) => snapshot.name(), + Source::Tiles(tiles) => tiles.name(), + } + } +} + +#[cfg(feature = "gpu")] +impl From for Source { + fn from(data: Data) -> Source { + Source::Data(data) + } +} + +#[cfg(feature = "gpu")] +impl From<&Data> for Source { + fn from(data: &Data) -> Source { + Source::Data(data.clone()) + } +} + +#[cfg(feature = "gpu")] +impl From for Source { + fn from(snapshot: Snapshot) -> Source { + Source::Snapshot(snapshot) + } +} + +#[cfg(feature = "gpu")] +impl From<&Snapshot> for Source { + fn from(snapshot: &Snapshot) -> Source { + Source::Snapshot(snapshot.clone()) + } +} + +#[cfg(feature = "gpu")] +impl From for Source { + fn from(tiles: crate::Tiles) -> Source { + Source::Tiles(tiles) + } +} + +#[cfg(feature = "gpu")] +impl From<&crate::Tiles> for Source { + fn from(tiles: &crate::Tiles) -> Source { + Source::Tiles(tiles.clone()) + } +} diff --git a/crates/latkit/src/data/table.rs b/crates/latkit/src/data/table.rs index 6a67f000..a5f8d8a8 100644 --- a/crates/latkit/src/data/table.rs +++ b/crates/latkit/src/data/table.rs @@ -1,14 +1,15 @@ use std::sync::{Arc, OnceLock}; use super::bits::Bits; -use super::column::{Column, Page}; +use super::column::Column; use super::ids::{Ids, Index, RowIds}; use super::name::{Id, IdRef, Name, PageId, Version}; use super::samples::Samples; use super::values::{Array, Kind, Value, Values}; use crate::{Error, Result}; -/// A named set of rows with fields: the rows of one type in a dataset, or rows to write into one. +/// A named set of rows with fields. It holds the rows of one type in a dataset, or rows to write +/// into one. /// /// ``` /// # use latkit::{Ids, Table}; @@ -37,27 +38,28 @@ pub(crate) enum Field { Sampled(Samples, Version), } -/// A field naming rows of another table by id. Row indices derive from the ids and the target's rows. +/// A field referencing rows of another table by id. Target row numbers are resolved from the ids +/// against the target table. #[derive(Clone)] pub(crate) struct Reference { pub(crate) to: Name, pub(crate) ids: Column, - /// Target row of each row as `u32`; missing where unwired or the id names no row. Its pages align - /// with the pages of `ids`. + /// Each row's target row as a `u32`, missing where the row is unwired or its id names no row. + /// Its pages line up with the pages of `ids`. pub(crate) rows: Column, - /// What each page of `rows` was resolved from. + /// The inputs each page of `rows` was resolved from, used to detect stale pages. pub(crate) resolved: Arc<[Resolved]>, pub(crate) version: Version, - /// The rows naming each target row, built on first use from `rows`, and shared by every snapshot - /// holding them. + /// The rows that reference each target row. Built from `rows` on first use and shared by every + /// snapshot with the same `rows`. pub(crate) inverse: Arc>, } -/// A reference row's target where it names no row. +/// The target row of a reference row that names no row. pub(crate) const UNWIRED: u32 = u32::MAX; -/// The rows naming each target row of a reference: target `t`'s are `rows[starts[t]..starts[t + 1]]`, -/// in row order. +/// The rows that reference each target row. Target `t` has `rows[starts[t]..starts[t + 1]]`, in +/// row order. #[derive(Default)] pub(crate) struct Inverse { starts: Vec, @@ -65,7 +67,7 @@ pub(crate) struct Inverse { } impl Inverse { - /// The rows naming target row `target`. + /// The rows that reference target row `target`. pub(crate) fn of(&self, target: usize) -> &[u32] { match (self.starts.get(target), self.starts.get(target + 1)) { (Some(&start), Some(&end)) => &self.rows[start as usize..end as usize], @@ -101,8 +103,8 @@ impl Field { } } - /// The field of rows `rows`, in that order: a reference names the same targets by id, resolved - /// again once applied; samples keep every frame. + /// The field for rows `rows`, in that order. A reference keeps its target ids, and its target + /// rows are resolved again when the result is applied. Samples keep every frame. pub(crate) fn gather(&self, rows: &[usize]) -> Field { match self { Field::Values(column, _) => Field::Values(column.gather(rows), Version::next()), @@ -123,7 +125,7 @@ impl Reference { Reference { to, ids, rows, resolved: Arc::new([]), version: Version::next(), inverse: Arc::default() } } - /// The target row row `row` names. + /// The target row that row `row` references, if it is wired. pub(crate) fn target(&self, row: usize) -> Option { match self.rows.get(row)? { Value::Number(n) => Some(n as u32), @@ -131,7 +133,25 @@ impl Reference { } } - /// Every row's target row, [`UNWIRED`] where it names none: read page by page, not value by value. + /// The target row that row `row` references, or [`UNWIRED`] if it has none or `row` is past the + /// end. Reads the page's `u32` storage directly, like [`targets`](Reference::targets). + pub(crate) fn target_row(&self, row: usize) -> u32 { + let Some((values, i)) = self.rows.at(row) else { return UNWIRED }; + match (&values.array, &values.valid) { + (Array::U32(rows), None) => rows[i], + (Array::U32(rows), Some(valid)) => { + if valid.get(i) { + rows[i] + } else { + UNWIRED + } + } + _ => self.target(row).unwrap_or(UNWIRED), + } + } + + /// Every row's target row, or [`UNWIRED`] where it names none. Reads a page at a time, not a + /// value at a time. pub(crate) fn targets(&self) -> Vec { let mut out = Vec::with_capacity(self.rows.len()); for page in self.rows.pages() { @@ -150,8 +170,8 @@ impl Reference { out } - /// The rows naming each target row: counted, the counts summed, then each row written at its - /// target's place. + /// The rows that reference each target row. Built once by counting the rows per target, + /// prefix-summing the counts, then writing each row into its target's range. pub(crate) fn inverse(&self) -> &Inverse { self.inverse.get_or_init(|| { let targets = self.targets(); @@ -175,48 +195,6 @@ impl Reference { }) } - /// This reference for ids `ids`, naming rows `targets` of a target table of `order` and - /// `structure`, [`UNWIRED`] where none: resolved already, as a table drawn in part holds it. - /// `targets` holds a row for each id. A page of `kept`, as it was resolved before, whose ids and - /// targets are those of the same page now, keeps its id, so it does not upload again. - pub(crate) fn resolved_to( - &self, - ids: Column, - targets: &[u32], - order: Version, - structure: Version, - kept: Option<&Reference>, - ) -> Reference { - let mut pages = Vec::with_capacity(ids.pages().len()); - let mut resolved = Vec::with_capacity(ids.pages().len()); - let mut start = 0; - for (p, page) in ids.pages().iter().enumerate() { - let rows = &targets[start..start + page.values.len()]; - start += rows.len(); - let dangling = (0..rows.len()).any(|i| rows[i] == UNWIRED && page.values.is_valid(i)); - let same = - kept.and_then(|k| Some((k.rows.pages().get(p)?, k.ids.pages().get(p)?))).filter(|(old, old_ids)| { - old_ids.id == page.id && matches!(&old.values.array, Array::U32(old) if old[..] == *rows) - }); - if let Some((old, _)) = same { - pages.push(old.clone()); - } else { - let valid = Bits::from_fn(rows.len(), |i| rows[i] != UNWIRED); - let values = Values { array: Array::U32(rows.to_vec().into()), valid: (!valid.all()).then_some(valid) }; - pages.push(Page::new(values)); - } - resolved.push(Resolved { ids: page.id, order, structure, dangling }); - } - Reference { - to: self.to.clone(), - ids, - rows: Column::of_pages(pages, &Values::from(Vec::::new())), - resolved: resolved.into(), - version: Version::next(), - inverse: Arc::default(), - } - } - /// Whether `rows` must be resolved again against a target with these versions. pub(crate) fn stale(&self, order: Version, structure: Version) -> bool { self.resolved.len() != self.ids.pages().len() @@ -228,7 +206,7 @@ impl Reference { .any(|(page, r)| r.ids != page.id || r.order != order || (r.dangling && r.structure != structure)) } - /// Resolves the pages whose inputs changed against `target`. + /// Resolves against `target` the pages whose inputs changed, reusing the rest. pub(crate) fn resolve(&self, target: &Table) -> Result { let index = target.index()?; let (order, structure) = (target.order, target.structure); @@ -262,7 +240,8 @@ impl Reference { } impl Table { - /// A table named `name` with rows `ids`: a count for rows `0..n`, or a list of numbers or texts. + /// A table named `name` with rows `ids`. `ids` is a count for rows `0..n`, or a list of numbers + /// or texts. pub fn new(name: impl Into, ids: impl Into) -> Table { let version = Version::next(); Table { @@ -275,7 +254,8 @@ impl Table { } } - /// Adds or replaces field `name`: one value per row. + /// Adds or replaces field `name`, with one value per row. A length mismatch is an error when + /// the table is applied. #[must_use] pub fn field(mut self, name: impl Into, values: impl Into) -> Table { let column = Column::new(&values.into()); @@ -283,7 +263,9 @@ impl Table { self } - /// Adds or replaces field `name` naming rows of table `to` by id; `None` leaves a row unwired. + /// Adds or replaces field `name`, referencing rows of table `to` by id, one id per row. `None` + /// leaves a row unwired. A missing table `to` is an error when the table is applied. An id that + /// matches no row stays unwired until such a row exists. #[must_use] pub fn reference(mut self, name: impl Into, to: impl Into, ids: impl Into) -> Table { let ids = match RowIds::new(&ids.into()) { @@ -314,25 +296,25 @@ impl Table { (0..self.len()).filter_map(|row| self.ids.id(row)) } - /// Every field's name and kind, by name. + /// Every field's name and kind, sorted by name. pub fn fields(&self) -> impl Iterator + '_ { self.fields.iter().map(|(name, field)| (name, field.kind())) } - /// What field `name` holds. + /// The kind of field `name`, if it exists. pub fn kind(&self, name: &str) -> Option { self.field_named(name).map(Field::kind) } - /// The value of `field` for the row with `id`. A sampled field reads its last frame; a reference - /// reads the target's id. + /// The value of `field` for the row with `id`, or `None` if missing. A sampled field reads its + /// last frame. A reference returns the target's id. pub fn get(&self, id: impl Into, field: &str) -> Option { let row = self.row(id.into().as_ref())?; self.value(row, field, f64::INFINITY) } - /// Every row's value of `field`, in row order: a reference's as the target ids, a sampled - /// field's at its last frame. + /// Every row's value of `field`, in row order. A reference gives target ids and a sampled field + /// gives its last frame. pub fn values(&self, field: &str) -> Option { Some(match self.field_named(field)? { Field::Values(column, _) => column.values(), @@ -352,8 +334,8 @@ impl Table { } } - /// The table of rows `rows` of this one, in that order, with every field: what removing rows - /// keeps, and what a query reads back. + /// A table of the given `rows` of this one, in that order, with every field. Used to remove + /// rows. pub(crate) fn rows(&self, rows: &[usize]) -> Table { let version = Version::next(); Table { @@ -391,7 +373,8 @@ impl Table { } } - /// Checks that every field has one value per row and the ids are present and unique. + /// Checks that the table has a name, every row has a unique id, and every field except a + /// sampled one has one value per row. pub(crate) fn check(&self) -> Result<()> { if self.name.is_empty() { return Err(Error::invalid("a table needs a name")); diff --git a/crates/latkit/src/data/tests.rs b/crates/latkit/src/data/tests.rs index ce89e2d8..680f2110 100644 --- a/crates/latkit/src/data/tests.rs +++ b/crates/latkit/src/data/tests.rs @@ -1,8 +1,9 @@ -//! Patches applied to a dataset agree with a plain model of rows, whatever the sequence. +//! Dataset tests, including a property test that any sequence of patches leaves a dataset matching +//! a plain model of its rows. use proptest::prelude::*; -use crate::{Data, Error, Id, Ids, Patch, Row, Table, Value}; +use crate::{Data, EnvelopeRequest, Error, Id, Ids, Patch, Row, Snapshot, Table, Value}; /// One row of the model: id, number field `x`, and reference `r` to another row's id. type ModelRow = (u64, Option, Option); @@ -67,21 +68,20 @@ impl Model { (0..self.rows.len()).map(|i| values.first().copied().flatten().map(|v| v + i as f64)).collect(); let at = self.frames.len() as f64; self.frames.push(self.rows.iter().zip(&values).map(|(m, v)| (m.0, *v)).collect()); - Patch::new().append([at], "T", [("s", values)]) + Patch::new().append("T", [at], [("s", values)]) } } } - fn check(&self, data: &Data) { + fn check(&self, data: &Snapshot) { let table = data.table("T").expect("the table exists"); let ids: Vec = table.ids().collect(); assert_eq!(ids, self.rows.iter().map(|m| Id::Number(m.0)).collect::>()); for &(id, x, r) in &self.rows { - let row = Row::new("grid", "T", id); let has_x = table.kind("x").is_some(); - assert_eq!(data.get(&row, "x"), if has_x { x.map(Value::Number) } else { None }, "x of {id}"); + assert_eq!(data.get("T", id, "x", None), if has_x { x.map(Value::Number) } else { None }, "x of {id}"); let wired = r.filter(|r| self.rows.iter().any(|m| m.0 == *r)); - assert_eq!(data.get(&row, "r"), wired.map(|r| Value::Row(Row::new("grid", "T", r))), "r of {id}"); + assert_eq!(data.get("T", id, "r", None), wired.map(|r| Value::Row(Row::new("grid", "T", r))), "r of {id}"); } } } @@ -91,23 +91,24 @@ proptest! { #[test] fn patches_agree_with_the_model(changes in proptest::collection::vec(change(), 1..12)) { - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply(Patch::new().set(Table::new("T", Vec::::new()))).expect("an empty table"); let mut model = Model { rows: Vec::new(), frames: Vec::new() }; for change in &changes { - let before = data.clone(); + let before = data.snapshot(); let before_bytes = before.to_bytes(); let patch = model.apply(change); data.apply(patch).expect("valid patches apply"); - model.check(&data); + model.check(&data.snapshot()); prop_assert_eq!(before.to_bytes(), before_bytes, "an earlier snapshot never changes"); - let back = Data::from_bytes(&data.to_bytes()).expect("round trip"); + let back = Snapshot::from_bytes(&data.snapshot().to_bytes()).expect("round trip"); model.check(&back); } + let data = data.snapshot(); for (frame, rows) in model.frames.iter().enumerate() { for &(id, value) in rows { if let Some(at) = model.rows.iter().position(|m| m.0 == id) { - let read = data.get_at(&Row::new("grid", "T", id), "s", frame as f64); + let read = data.get("T", id, "s", frame as f64); let want = value.map(|v| Value::Number(f64::from(v as f32))); prop_assert_eq!(read, want, "frame {} of {} (row {})", frame, id, at); } @@ -118,9 +119,9 @@ proptest! { #[test] fn a_failed_patch_changes_nothing() { - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply(Table::new("Bus", vec![1_u64, 2]).field("kv", vec![230.0, 115.0])).expect("valid"); - let before = data.to_bytes(); + let before = data.snapshot().to_bytes(); let error = data .apply( Patch::new() @@ -130,86 +131,86 @@ fn a_failed_patch_changes_nothing() { .expect_err("text into a number field"); assert!(matches!(&error, Error::Invalid { .. }), "{error}"); assert_eq!(error.to_string(), "patch[1].Bus.kv: Text values for a Number field"); - assert_eq!(data.to_bytes(), before); + assert_eq!(data.snapshot().to_bytes(), before); } #[test] -fn appends_one_after_another_record_as_one_and_name_their_own_errors() { +fn consecutive_appends_merge_and_report_their_own_errors() { let table = || Table::new("Bus", vec![1_u64, 2]).field("kv", vec![230.0, 115.0]); - let mut one = Data::new("grid"); + let one = Data::new("grid"); one.apply(Patch::new().replace(table()).append( - [0.0, 1.0, 2.0], "Bus", + [0.0, 1.0, 2.0], [("vm", vec![1.0_f32, 2.0, 3.0, 4.0, 5.0, 6.0])], )) .expect("valid"); - let mut each = Data::new("grid"); + let each = Data::new("grid"); let mut patch = Patch::new().replace(table()); for f in 0..3 { let v = 2.0 * f as f32; - patch = patch.append([f64::from(f)], "Bus", [("vm", vec![v + 1.0, v + 2.0])]); + patch = patch.append("Bus", [f64::from(f)], [("vm", vec![v + 1.0, v + 2.0])]); } each.apply(patch).expect("valid"); - assert_eq!(each.to_bytes(), one.to_bytes(), "a frame a change records as the frames in one"); + assert_eq!(each.snapshot().to_bytes(), one.snapshot().to_bytes(), "appends of one frame each merge into one"); let error = each .apply( Patch::new() - .append([3.0], "Bus", [("vm", vec![1.0_f32, 1.0])]) - .append([4.0], "Bus", [("vm", vec![1.0_f32])]) - .append([5.0], "Bus", [("vm", vec![1.0_f32, 1.0])]), + .append("Bus", [3.0], [("vm", vec![1.0_f32, 1.0])]) + .append("Bus", [4.0], [("vm", vec![1.0_f32])]) + .append("Bus", [5.0], [("vm", vec![1.0_f32, 1.0])]), ) - .expect_err("a frame short of a row"); + .expect_err("a frame missing a row's value"); assert!(error.to_string().starts_with("patch[1].Bus.vm"), "{error}"); } #[test] fn field_lengths_must_match_rows() { - let mut data = Data::new("grid"); + let data = Data::new("grid"); let error = data.apply(Table::new("Bus", Ids::count(3)).field("kv", vec![1.0, 2.0])).expect_err("short field"); assert_eq!(error.to_string(), "patch[0].Bus.kv: 2 values for 3 rows"); } #[test] fn repeated_ids_are_refused() { - let mut data = Data::new("grid"); + let data = Data::new("grid"); let error = data.apply(Table::new("Bus", vec!["a", "a"])).expect_err("repeated id"); assert!(error.to_string().starts_with("patch[0].Bus.ids[1]"), "{error}"); } #[test] fn references_to_unknown_tables_are_refused() { - let mut data = Data::new("grid"); + let data = Data::new("grid"); let error = data.apply(Table::new("Branch", Ids::count(1)).reference("from", "Bus", vec![1_u64])).expect_err("no Bus"); assert_eq!(error.to_string(), "Branch.from: no table Bus"); } #[test] -fn dangling_references_wire_up_when_the_row_arrives() { - let mut data = Data::new("grid"); +fn dangling_references_resolve_once_the_row_exists() { + let data = Data::new("grid"); data.apply(Patch::new().set(Table::new("Bus", vec![1_u64])).set(Table::new("Branch", Ids::count(1)).reference( "to", "Bus", vec![2_u64], ))) .expect("valid"); - let branch = Row::new("grid", "Branch", 0_u64); - assert_eq!(data.get(&branch, "to"), None); + let to = || data.snapshot().get("Branch", 0_u64, "to", None); + assert_eq!(to(), None); data.apply(Table::new("Bus", vec![2_u64])).expect("valid"); - assert_eq!(data.get(&branch, "to"), Some(Value::Row(Row::new("grid", "Bus", 2_u64)))); + assert_eq!(to(), Some(Value::Row(Row::new("grid", "Bus", 2_u64)))); data.apply(Patch::new().remove("Bus", vec![1_u64])).expect("valid"); - assert_eq!(data.get(&branch, "to"), Some(Value::Row(Row::new("grid", "Bus", 2_u64)))); + assert_eq!(to(), Some(Value::Row(Row::new("grid", "Bus", 2_u64)))); } #[test] fn inverses_follow_every_change_and_snapshots_keep_theirs() { - let inverse = |data: &Data, target: usize| { + let inverse = |data: &Snapshot, target: usize| { let Some(super::table::Field::Reference(r)) = data.table("Branch").and_then(|t| t.field_named("from")) else { panic!("a reference"); }; r.inverse().of(target).to_vec() }; - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply( Patch::new().set(Table::new("Bus", vec![1_u64, 2, 3])).set(Table::new("Branch", Ids::count(4)).reference( "from", @@ -218,20 +219,22 @@ fn inverses_follow_every_change_and_snapshots_keep_theirs() { )), ) .expect("valid"); - assert_eq!(inverse(&data, 0), vec![0, 2]); - assert_eq!(inverse(&data, 2), vec![3]); - let before = data.clone(); + assert_eq!(inverse(&data.snapshot(), 0), vec![0, 2]); + assert_eq!(inverse(&data.snapshot(), 2), vec![3]); + let before = data.snapshot(); data.apply(Table::new("Branch", vec![1_u64]).reference("from", "Bus", vec![3_u64])).expect("valid"); - assert_eq!((inverse(&data, 0), inverse(&data, 2)), (vec![0, 2], vec![1, 3])); + let now = data.snapshot(); + assert_eq!((inverse(&now, 0), inverse(&now, 2)), (vec![0, 2], vec![1, 3])); data.apply(Patch::new().remove("Bus", vec![1_u64])).expect("valid"); - // Bus 1 left: what named it names nothing, and the rest moved up a row. - assert_eq!((inverse(&data, 0), inverse(&data, 1)), (vec![], vec![1, 3])); + // Bus 1 is gone: the rows that referenced it are unwired, and the later buses moved up a row. + let now = data.snapshot(); + assert_eq!((inverse(&now, 0), inverse(&now, 1)), (vec![], vec![1, 3])); assert_eq!(inverse(&before, 0), vec![0, 2]); } -/// A dataset of `rows` rows of `Line` with a `flow` sampled over `frames` frames, one a second. -fn sampled(rows: usize, frames: usize) -> Data { - let mut data = Data::new("grid"); +/// A dataset of `rows` rows of `Line` with a `flow` sampled over `frames` frames, one per second. +fn sampled(rows: usize, frames: usize) -> Snapshot { + let data = Data::new("grid"); let at: Vec = (0..frames).map(|f| f as f64).collect(); let flow: Vec = (0..frames * rows).map(|i| ((i * 7919) % 1000) as f64 - 500.0).collect(); data.apply( @@ -239,10 +242,10 @@ fn sampled(rows: usize, frames: usize) -> Data { .axis("time", "s") .replace(Table::new("Line", Ids::count(rows as u32)).reference("from", "Bus", vec![0_u32; rows])) .replace(Table::new("Bus", Ids::count(1)).field("name", vec!["a"])) - .append(at, "Line", [("flow", flow)]), + .append("Line", at, [("flow", flow)]), ) .expect("valid"); - data + data.snapshot() } #[test] @@ -263,13 +266,17 @@ fn describe_names_tables_fields_and_spans() { } #[test] -fn envelopes_read_levels_as_frames_would() { +fn envelopes_match_a_scan_of_the_frames() { let (rows, frames) = (3, 700); let data = sampled(rows, frames); - let line = data.table("Line").expect("Line"); + let envelope = |field: &str, range: [f64; 2], spans: usize, ids: Option>| { + let request = + EnvelopeRequest { table: "Line".into(), field: field.into(), range, spans, ids: ids.map(Ids::from) }; + data.envelope(&request) + }; let value = |row: usize, frame: usize| (((frame * rows + row) * 7919) % 1000) as f32 - 500.0; for (range, spans) in [([0.0, 699.0], 1), ([0.0, 699.0], 7), ([13.5, 600.0], 5), ([650.0, 2000.0], 3)] { - let envelope = line.envelope("flow", range, spans, None).expect("an envelope"); + let envelope = envelope("flow", range, spans, None).expect("an envelope"); assert_eq!(envelope.at.len(), spans + 1); for row in 0..rows { for s in 0..spans { @@ -288,45 +295,56 @@ fn envelopes_read_levels_as_frames_would() { } } } - let picked = line.envelope("flow", [0.0, 699.0], 2, Some(&vec![2_u32, 0].into())).expect("an envelope"); - let all = line.envelope("flow", [0.0, 699.0], 2, None).expect("an envelope"); + let picked = envelope("flow", [0.0, 699.0], 2, Some(vec![2, 0])).expect("an envelope"); + let all = envelope("flow", [0.0, 699.0], 2, None).expect("an envelope"); assert_eq!(picked.least, [&all.least[4..6], &all.least[0..2]].concat()); - assert!(line.envelope("from", [0.0, 1.0], 1, None).is_err()); - assert!(line.envelope("flow", [0.0, 1.0], 1, Some(&vec![9_u32].into())).is_err()); + assert!(envelope("from", [0.0, 1.0], 1, None).is_err()); + assert!(envelope("flow", [0.0, 1.0], 1, Some(vec![9])).is_err()); + let json: EnvelopeRequest = serde_json::from_value(serde_json::json!({ + "table": "Line", "field": "flow", "range": [0, 699], "spans": 2, "ids": [2, 0], + })) + .expect("a request"); + assert_eq!(data.envelope(&json).expect("an envelope"), picked); + let error = data.envelope(&EnvelopeRequest { table: "Nope".into(), ..json }).expect_err("no table"); + assert_eq!(error.to_string(), "Nope: no such table"); } #[cfg(feature = "gpu")] #[test] fn queries_filter_sort_page_and_select() { - use crate::{Order, Query, field}; - let mut data = Data::new("grid"); + use crate::{Order, Query, QueryResult, field}; + let data = Data::new("grid"); let loading = vec![40.0, 95.0, 120.0, 95.0, 10.0]; data.apply( Table::new("Branch", Ids::count(5)).field("loading", loading).field("name", vec!["a", "b", "c", "d", "e"]), ) .expect("valid"); - let names = |table: &Table| table.ids().map(|id| table.get(id, "name")).collect::>(); + let data = data.snapshot(); + let names = |read: &QueryResult| read.table.ids().map(|id| read.table.get(id, "name")).collect::>(); let text = |s: &str| Some(Value::Text(s.into())); let hot = Query::of("Branch").filter(field("loading").gt(90.0)).sort(field("loading"), Order::Descending); - let (sorted, kept) = data.query(&hot).expect("a query"); - assert_eq!((names(&sorted), kept), (vec![text("c"), text("b"), text("d")], 3), "ties keep their order"); - let (page, kept) = data.query(&hot.clone().offset(1).limit(1)).expect("a query"); - assert_eq!((names(&page), kept), (vec![text("b")], 3), "the filter kept three, the page one"); - let (least, _) = - data.query(&Query::of("Branch").sort(field("loading"), Order::Ascending).limit(2)).expect("a query"); + let sorted = data.query(&hot).expect("a query"); + assert_eq!((names(&sorted), sorted.total), (vec![text("c"), text("b"), text("d")], 3), "ties keep their order"); + let page = data.query(&hot.clone().offset(1).limit(1)).expect("a query"); + assert_eq!((names(&page), page.total), (vec![text("b")], 3), "the filter kept three, the page one"); + let least = data.query(&Query::of("Branch").sort(field("loading"), Order::Ascending).limit(2)).expect("a query"); assert_eq!(names(&least), [text("e"), text("a")]); - let (selected, _) = - data.query(&Query::of("Branch").limit(1).select("double", field("loading") * 2.0)).expect("a query"); + let selected = + data.query(&Query::of("Branch").limit(1).select("double", field("loading") * 2.0)).expect("a query").table; assert_eq!(selected.fields().map(|(name, _)| name.as_str()).collect::>(), ["double"]); assert_eq!(selected.get(0_u64, "double"), Some(Value::Number(80.0))); + let ids = data.query(&Query::of("Branch").limit(2).ids_alone()).expect("a query").table; + assert_eq!((ids.len(), ids.fields().count()), (2, 0), "ids alone"); let json: Query = serde_json::from_value(serde_json::json!({ - "of": "Branch", "filter": { "op": "lt", "args": ["loading", 100] }, "sort": [["loading", "descending"]], "limit": 2, + "of": "Branch", "filter": { "lt": ["$loading", 100] }, "sort": [["$loading", "descending"]], "limit": 2, + "select": { "name": "$name" }, })) .expect("a query"); - assert_eq!(names(&data.query(&json).expect("a query").0), [text("b"), text("d")]); - let (named, _) = data.query(&Query::of("Branch").ids([3_u64, 1, 9])).expect("by ids"); - assert_eq!(names(&named), [text("d"), text("b")], "in the order named, unknown ids passed over"); - let (lake, _) = data.query(&Query::of("Branch").filter(field("name").contains("C"))).expect("contains"); + let read = data.query(&json).expect("a query"); + assert_eq!(names(&read), [text("b"), text("d")], "a selected field comes back as stored"); + let named = data.query(&Query::of("Branch").ids([3_u64, 1, 9])).expect("by ids"); + assert_eq!(names(&named), [text("d"), text("b")], "in the order given, unknown ids skipped"); + let lake = data.query(&Query::of("Branch").filter(field("name").contains("C"))).expect("contains"); assert_eq!(names(&lake), [text("c")], "ignoring case"); assert!(data.query(&Query::of("Nope")).is_err()); } @@ -336,65 +354,65 @@ fn queries_filter_sort_page_and_select() { fn queries_read_sampled_fields_at_a_coordinate() { use crate::{Query, field}; let data = sampled(2, 4); - let (at_two, _) = data.query(&Query::of("Line").at(2.0).select("flow", field("flow"))).expect("a query"); + let at_two = data.query(&Query::of("Line").at(2.0).select("flow", field("flow"))).expect("a query").table; let flow = |row: usize, frame: usize| Value::Number(((((frame * 2 + row) * 7919) % 1000) as f64) - 500.0); assert_eq!(at_two.get(1_u64, "flow"), Some(flow(1, 2))); - let (last, _) = data.query(&Query::of("Line")).expect("a query"); + let last = data.query(&Query::of("Line")).expect("a query").table; assert_eq!(last.kind("flow"), Some(crate::Kind::Number)); assert_eq!(last.get(0_u64, "flow"), Some(flow(0, 3))); } #[test] -fn ids_too_large_for_json_numbers_cross_as_text() { +fn ids_too_large_for_json_numbers_are_written_as_text() { let big = (1_u64 << 53) + 1; - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply(Table::new("Bus", vec![big, 7]).field("kv", vec![1.0, 2.0])).expect("valid"); assert_eq!(serde_json::to_string(&Id::Number(big)).expect("an id"), format!("\"{big}\"")); assert_eq!(serde_json::to_string(&Id::Number(7)).expect("an id"), "7"); - let bus = data.table("Bus").expect("Bus"); - assert_eq!(bus.get(Id::from(big.to_string()), "kv"), Some(Value::Number(1.0)), "the text names the row"); - assert_eq!(bus.get(big, "kv"), Some(Value::Number(1.0))); - assert_eq!(bus.values("kv").and_then(|v| v.numbers()), Some(vec![1.0, 2.0])); + let now = data.snapshot(); + let text = Id::from(big.to_string()); + assert_eq!(now.get("Bus", text, "kv", None), Some(Value::Number(1.0)), "the decimal text finds the row"); + assert_eq!(now.get("Bus", big, "kv", None), Some(Value::Number(1.0))); let counted = Table::new("Count", Ids::count(3)).field("x", vec![0.0, 1.0, 2.0]); data.apply(counted).expect("valid"); - assert_eq!(data.table("Count").expect("Count").get(Id::from("2"), "x"), Some(Value::Number(2.0))); + assert_eq!(data.snapshot().get("Count", Id::from("2"), "x", None), Some(Value::Number(2.0))); } #[test] fn schema_changes_only_with_fields() { - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply(Table::new("Bus", Ids::count(2)).field("kv", vec![1.0, 2.0])).expect("valid"); - let schema = data.schema(); + let schema = data.snapshot().schema(); data.apply(Table::new("Bus", Ids::count(3)).field("kv", vec![1.0, 2.0, 3.0])).expect("valid"); - assert_eq!(data.schema(), schema); + assert_eq!(data.snapshot().schema(), schema); data.apply(Table::new("Bus", Ids::count(1)).field("name", vec!["a"])).expect("valid"); - assert_ne!(data.schema(), schema); + assert_ne!(data.snapshot().schema(), schema); } #[test] -fn rows_appended_by_id_leave_the_rest_holding() { - let mut data = Data::new("grid"); +fn appending_by_id_keeps_other_rows_last_values() { + let data = Data::new("grid"); data.apply(Patch::new().set(Table::new("Bus", [10_u64, 20, 30])).append( - [0.0], "Bus", + [0.0], [("vm", vec![1.0, 1.1, 1.2])], )) .expect("valid"); - data.apply(Patch::new().append_rows([1.0], "Bus", [30_u64], [("vm", vec![0.9])])).expect("by id"); - let vm = |id: u64| data.get_at(&Row::new("grid", "Bus", id), "vm", 1.0); - assert_eq!(vm(10), Some(Value::Number(1.0)), "held"); + data.apply(Patch::new().append_rows("Bus", [1.0], [("vm", vec![0.9])], [30_u64])).expect("by id"); + let vm = |id: u64| data.snapshot().get("Bus", id, "vm", 1.0); + assert_eq!(vm(10), Some(Value::Number(1.0)), "other rows keep their last value"); assert_eq!(vm(30), Some(Value::Number(f64::from(0.9_f32))), "recorded"); - let error = data.apply(Patch::new().append_rows([2.0], "Bus", [40_u64], [("vm", vec![1.0])])); - assert!(error.is_err_and(|e| e.to_string().contains("no row")), "rows appended to are rows the table holds"); + let error = data.apply(Patch::new().append_rows("Bus", [2.0], [("vm", vec![1.0])], [40_u64])); + assert!(error.is_err_and(|e| e.to_string().contains("no row")), "appending to an unknown id fails"); } #[test] -fn a_patch_to_a_dataset_changes_it_alone() { - let mut grid = Data::new("grid"); - grid.apply(Patch::to("grid").set(Table::new("Bus", Ids::count(1)))).expect("its own"); +fn a_patch_applies_only_to_its_named_dataset() { + let grid = Data::new("grid"); + grid.apply(Patch::to("grid").set(Table::new("Bus", Ids::count(1)))).expect("its own dataset"); let error = grid.apply(Patch::to("weather").set(Table::new("Cell", Ids::count(1)))); assert!(error.is_err_and(|e| e.to_string().contains("weather"))); - let bytes = grid.to_bytes(); + let bytes = grid.snapshot().to_bytes(); assert_eq!(Patch::from_bytes(&bytes).expect("a patch").data().map(crate::Name::as_str), Some("grid")); - assert!(Data::from_bytes(&Patch::new().to_bytes()).is_err(), "a dataset is a patch to one"); + assert!(Snapshot::from_bytes(&Patch::new().to_bytes()).is_err(), "dataset bytes must name the dataset"); } diff --git a/crates/latkit/src/data/text.rs b/crates/latkit/src/data/text.rs index b78958e5..59c24117 100644 --- a/crates/latkit/src/data/text.rs +++ b/crates/latkit/src/data/text.rs @@ -27,7 +27,7 @@ impl Text { (0..self.len()).map(|i| self.get(i)) } - /// The UTF-8 bytes the slice spans and offsets relative to them. + /// The offsets rebased to start at 0, and the UTF-8 text they index, for encoding a slice. pub(crate) fn compact(&self) -> (Vec, &str) { let base = self.offsets.first().copied().unwrap_or(0); let end = self.offsets.last().copied().unwrap_or(0); diff --git a/crates/latkit/src/data/values.rs b/crates/latkit/src/data/values.rs index f7b8b737..c30dde23 100644 --- a/crates/latkit/src/data/values.rs +++ b/crates/latkit/src/data/values.rs @@ -31,7 +31,7 @@ pub(crate) enum Array { Geometry(Geometry), } -/// What a field holds. +/// The kind of values a field holds. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug, serde::Serialize, serde::Deserialize, schemars::JsonSchema)] #[serde(rename_all = "lowercase")] pub enum Kind { @@ -51,12 +51,13 @@ pub enum Kind { Polygons, /// Rows of another table, by id. Reference, - /// Numbers sampled over the data's axis. + /// Numbers sampled over the dataset's axis. Sampled, } impl Kind { - /// The geometry of `depth` levels below its points: 0 a point, 1 a path, 2 parts, 3 polygons. + /// The geometry kind of nesting depth `depth`: 0 a point, 1 a path, 2 parts, 3 or more + /// polygons. pub(crate) fn geometry(depth: u8) -> Kind { match depth { 0 => Kind::Point, @@ -66,7 +67,7 @@ impl Kind { } } - /// How many levels a geometry nests below its points; `None` for any other kind. + /// A geometry kind's nesting depth, or `None` for other kinds. pub(crate) fn depth(self) -> Option { match self { Kind::Point => Some(0), @@ -99,8 +100,8 @@ pub enum Value { Row(Row), } -/// A value writes as JSON: a number (`null` when not finite), boolean, or text; a geometry as nested -/// `[x, y]` points; a reference as the id of the row it names. +/// Serializes as JSON. Numbers, booleans, and texts are written as is, with non-finite numbers as +/// `null`. A geometry is nested `[x, y]` points, and a reference is its target row's id. impl serde::Serialize for Value { fn serialize(&self, serializer: S) -> Result { match self { @@ -117,7 +118,7 @@ impl serde::Serialize for Value { } } -/// A value as JSON writes it. +/// The schema of a value's JSON form. impl schemars::JsonSchema for Value { fn schema_name() -> std::borrow::Cow<'static, str> { "Value".into() @@ -135,22 +136,22 @@ impl schemars::JsonSchema for Value { } } -/// A row picked that names none: the value made of it is missing, as a hole among the rows drawn -/// of a table drawn through tiles is. -pub(crate) const HOLE: usize = usize::MAX; - -/// Where each value an assembly makes comes from, among its sources: the first, then pages of -/// [`PAGE_ROWS`](super::column::PAGE_ROWS) rows each. A concrete type rather than a closure, so -/// [`Values::assemble`] is compiled once. +/// Where each output value of [`Values::assemble`] comes from. Source 0 is the first source. The +/// sources after it are pages of [`PAGE_ROWS`](super::column::PAGE_ROWS) rows, addressed by row. +/// It is a concrete type, not a closure, so [`Values::assemble`] is compiled only once. #[derive(Clone, Copy)] pub(crate) enum Picks<'a> { - /// Value `i` is row `rows[i]` of the pages, missing for a [`HOLE`]. + /// Value `i` is row `rows[i]` of the pages. Rows(&'a [usize]), - /// Value `i` is row `from[i]` of the pages where it is written, missing for a [`HOLE`], else row - /// `i` of the first source while `i < kept`, else missing. + /// Value `i` is row `from[i]` of the pages where it is written, else row `i` of the first + /// source while `i < kept`, else missing. Writes { from: &'a [Option], kept: usize }, } +/// A run of values to copy, and its length. The run is a source and its first row, with +/// consecutive rows following, or `None` for missing values. +type Run = (Option<(usize, usize)>, usize); + impl Picks<'_> { fn len(self) -> usize { match self { @@ -162,7 +163,7 @@ impl Picks<'_> { /// The source and row of value `i`, or `None` where it is missing. #[inline] fn get(self, i: usize) -> Option<(usize, usize)> { - let paged = |row: usize| (row != HOLE).then_some((1 + (row >> PAGE_BITS), row & PAGE_MASK)); + let paged = |row: usize| Some((1 + (row >> PAGE_BITS), row & PAGE_MASK)); match self { Picks::Rows(rows) => paged(rows[i]), Picks::Writes { from, kept } => match from[i] { @@ -171,6 +172,20 @@ impl Picks<'_> { }, } } + + /// The picks grouped into runs, in order. + fn runs(self) -> Vec { + let mut runs: Vec = Vec::new(); + for i in 0..self.len() { + let pick = self.get(i); + match runs.last_mut() { + Some((Some((s, r)), n)) if pick == Some((*s, *r + *n)) => *n += 1, + Some((None, n)) if pick.is_none() => *n += 1, + _ => runs.push((pick, 1)), + } + } + runs + } } impl Values { @@ -193,7 +208,7 @@ impl Values { self.len() == 0 } - /// What the values are. + /// The kind of the values. pub fn kind(&self) -> Kind { match &self.array { Array::F64(_) | Array::F32(_) | Array::I32(_) | Array::U32(_) | Array::U64(_) => Kind::Number, @@ -203,7 +218,7 @@ impl Values { } } - /// Whether value `i` is present. + /// Whether value `i` exists and is not missing. pub fn is_valid(&self, i: usize) -> bool { i < self.len() && self.valid.as_ref().is_none_or(|v| v.get(i)) } @@ -221,17 +236,18 @@ impl Values { }) } - /// Value `i` as a row id, exactly: `None` when missing, or neither a whole number nor a text. + /// Value `i` as a row id. `None` when missing, or when the values are not stored as `u64` + /// numbers or texts. pub fn id(&self, i: usize) -> Option { super::ids::key(self, i).map(super::name::IdRef::to_id) } - /// Every value as a number, missing as NaN: `None` unless the values are numbers. + /// Every value as a number, with NaN for missing values. `None` if the values are not numbers. pub fn numbers(&self) -> Option> { (self.kind() == Kind::Number).then(|| self.map_numbers(|x| x.unwrap_or(f64::NAN))) } - /// A zero-copy slice. + /// A slice of the values, without copying. Panics if `range` is out of bounds. #[must_use] pub fn slice(&self, range: Range) -> Values { let array = match &self.array { @@ -247,35 +263,22 @@ impl Values { Values { array, valid: self.valid.as_ref().map(|v| v.slice(range)) } } - /// Every value as `number` reads it, `None` where missing, mapped by `f`: a loop per storage, so - /// the storage is matched once rather than per value. - pub(crate) fn map_numbers(&self, mut f: impl FnMut(Option) -> T) -> Vec { - macro_rules! each { - ($v:expr, $number:expr) => { - match &self.valid { - None => $v.iter().map(|&x| f(Some($number(x)))).collect(), - Some(valid) => $v.iter().enumerate().map(|(i, &x)| f(valid.get(i).then(|| $number(x)))).collect(), - } - }; - } - match &self.array { - Array::F64(v) => each!(v, |x: f64| x), - Array::F32(v) => each!(v, f64::from), - Array::I32(v) => each!(v, f64::from), - Array::U32(v) => each!(v, f64::from), - Array::U64(v) => each!(v, |x: u64| x as f64), - _ => (0..self.len()).map(|i| f(self.is_valid(i).then(|| self.number(i)))).collect(), - } + /// Maps every value through `f`. Values are read like `number`, with `None` where missing. + pub(crate) fn map_numbers(&self, f: impl FnMut(Option) -> T) -> Vec { + let mut out = Vec::with_capacity(self.len()); + self.extend_numbers(&mut out, f); + out } - /// Calls `f` with every value as `number` reads it, `None` where missing: a loop per storage, - /// allocating nothing. - pub(crate) fn each_number(&self, mut f: impl FnMut(Option)) { + /// Appends every value to `out`, mapped through `f`. Values are read like `number`, with `None` + /// where missing. Each storage type has its own loop, so the storage is matched once, not per + /// value. + pub(crate) fn extend_numbers(&self, out: &mut Vec, mut f: impl FnMut(Option) -> T) { macro_rules! each { ($v:expr, $number:expr) => { match &self.valid { - None => $v.iter().for_each(|&x| f(Some($number(x)))), - Some(valid) => $v.iter().enumerate().for_each(|(i, &x)| f(valid.get(i).then(|| $number(x)))), + None => out.extend($v.iter().map(|&x| f(Some($number(x))))), + Some(valid) => out.extend($v.iter().enumerate().map(|(i, &x)| f(valid.get(i).then(|| $number(x))))), } }; } @@ -285,11 +288,11 @@ impl Values { Array::I32(v) => each!(v, f64::from), Array::U32(v) => each!(v, f64::from), Array::U64(v) => each!(v, |x: u64| x as f64), - _ => (0..self.len()).for_each(|i| f(self.is_valid(i).then(|| self.number(i)))), + _ => out.extend((0..self.len()).map(|i| f(self.is_valid(i).then(|| self.number(i))))), } } - /// Value `i` as a number; booleans are 0 or 1, others NaN. + /// Value `i` as a number. Booleans are 0 or 1, and texts and geometries are NaN. pub(crate) fn number(&self, i: usize) -> f64 { match &self.array { Array::F64(v) => v[i], @@ -302,22 +305,6 @@ impl Values { } } - /// The values as `u32`s, when they are stored so and none is missing. - pub(crate) fn u32s(&self) -> Option<&[u32]> { - match (&self.array, &self.valid) { - (Array::U32(v), None) => Some(v), - _ => None, - } - } - - /// The values as `u64`s, when they are stored so and none is missing. - pub(crate) fn u64s(&self) -> Option<&[u64]> { - match (&self.array, &self.valid) { - (Array::U64(v), None) => Some(v), - _ => None, - } - } - pub(crate) fn text(&self) -> Option<&Text> { match &self.array { Array::Text(t) => Some(t), @@ -332,22 +319,80 @@ impl Values { } } - /// Geometries as values, every one present. + /// The number of points shaders read for the present geometries. That is every coordinate, + /// plus one break between a row's parts or rings. 0 for values that are not geometries. + pub(crate) fn points(&self) -> usize { + let Some(geometry) = self.geometry() else { return 0 }; + (0..self.len()).filter(|&row| self.is_valid(row)).map(|row| geometry.row_points(row)).sum() + } + + /// Values holding `geometry`, none missing. pub(crate) fn of_geometry(geometry: Geometry) -> Values { Values { array: Array::Geometry(geometry), valid: None } } + /// Geometry of `depth` that copies each of `rows`. Each is a row of a geometry, or `None` for + /// an empty row, which draws nothing. + pub(crate) fn of_rows<'g>(depth: u8, rows: impl IntoIterator>) -> Values { + let mut builder = GeometryBuilder::new(depth); + for row in rows { + match row { + Some((geometry, i)) => builder.push_from(geometry, i), + None => builder.push_empty(), + } + } + Values::of_geometry(builder.finish()) + } + + /// A field of `values`, typed by the values present. It is numbers if every present value is a + /// number, booleans if every one is a boolean, and texts otherwise, with other values written + /// as text. GeoJSON and vector tile properties are typed this way. + pub(crate) fn infer(values: &[Option]) -> Values { + let present = || values.iter().flatten(); + if present().all(|v| matches!(v, Value::Number(_))) { + let numbers = values.iter().map(|v| if let Some(Value::Number(n)) = v { Some(*n) } else { None }); + return numbers.collect::>().into(); + } + if present().all(|v| matches!(v, Value::Bool(_))) { + let booleans = values.iter().map(|v| if let Some(Value::Bool(b)) = v { Some(*b) } else { None }); + return booleans.collect::>().into(); + } + let text = |value: &Value| match value { + Value::Text(text) => text.to_string(), + Value::Number(n) => n.to_string(), + Value::Bool(b) => b.to_string(), + other => serde_json::to_string(other).unwrap_or_default(), + }; + values.iter().map(|v| v.as_ref().map(text)).collect::>>().into() + } + /// The values `picks` names among `sources`, missing where it names none. /// - /// The result keeps the first source's storage; numbers from other sources convert to it. A - /// source of another kind is an error. + /// The result keeps the first source's storage, and numbers from other sources convert to it. + /// A source of another kind is an error. pub(crate) fn assemble(sources: &[&Values], picks: Picks<'_>) -> Result { - let (len, pick) = (picks.len(), |i: usize| picks.get(i)); - let like = sources.first().expect("a source"); - let kind = like.kind(); + let kind = sources.first().expect("a source").kind(); if let Some(other) = sources.iter().find(|s| s.kind() != kind) { return Err(Error::invalid(format!("{:?} values for a {kind:?} field", other.kind()))); } + // Consecutive rows from one source copy as a run. + Ok(Values::of_runs(sources, &picks.runs())) + } + + /// `parts` concatenated in the first part's storage, converting numbers from other storages. + /// Joins a column's pages by copying each part whole. Every part must have the same kind as the + /// first. + pub(crate) fn concat(parts: &[&Values]) -> Values { + debug_assert!(parts.iter().all(|p| p.kind() == parts[0].kind())); + let runs: Vec = parts.iter().enumerate().map(|(s, part)| (Some((s, 0)), part.len())).collect(); + Values::of_runs(parts, &runs) + } + + /// The values `runs` copy from `sources`, in the first source's storage. Numbers from other + /// sources convert to it. + fn of_runs(sources: &[&Values], runs: &[Run]) -> Values { + let len = runs.iter().map(|&(_, n)| n).sum(); + let like = sources.first().expect("a source"); macro_rules! numbers { ($variant:ident, $t:ty, $store:expr) => {{ let cast: Vec = sources.iter().map(|s| s.cast(|x| x as $t, $store)).collect(); @@ -358,7 +403,14 @@ impl Values { _ => unreachable!("cast to the first source's storage"), }) .collect(); - Array::$variant((0..len).map(|i| pick(i).map_or(<$t>::default(), |(s, r)| typed[s][r])).collect()) + let mut out: Vec<$t> = Vec::with_capacity(len); + for &(run, n) in runs { + match run { + Some((s, r)) => out.extend_from_slice(&typed[s][r..r + n]), + None => out.resize(out.len() + n, <$t>::default()), + } + } + Array::$variant(out.into()) }}; } let array = match &like.array { @@ -368,40 +420,52 @@ impl Values { Array::U32(_) => numbers!(U32, u32, Store::U32), Array::U64(_) => numbers!(U64, u64, Store::U64), Array::Bool(_) => Array::Bool( - (0..len) - .map(|i| pick(i).is_some_and(|(s, r)| matches!(&sources[s].array, Array::Bool(v) if v.get(r)))) + runs.iter() + .flat_map(|&(run, n)| (0..n).map(move |k| run.map(|(s, r)| (s, r + k)))) + .map(|pick| pick.is_some_and(|(s, r)| matches!(&sources[s].array, Array::Bool(v) if v.get(r)))) .collect(), ), Array::Text(_) => { let mut builder = TextBuilder::with_capacity(len); - for i in 0..len { - builder.push(pick(i).and_then(|(s, r)| sources[s].text().map(|t| t.get(r))).unwrap_or("")); + for &(run, n) in runs { + match run.and_then(|(s, r)| Some((sources[s].text()?, r))) { + Some((text, r)) => (r..r + n).for_each(|i| builder.push(text.get(i))), + None => (0..n).for_each(|_| builder.push("")), + } } Array::Text(builder.finish()) } Array::Geometry(g) => { let mut builder = GeometryBuilder::new(g.depth()); - for i in 0..len { - match pick(i).and_then(|(s, r)| sources[s].geometry().map(|g| (g, r))) { - Some((geometry, r)) => builder.push_from(geometry, r), - None => builder.push_empty(), + for &(run, n) in runs { + match run.and_then(|(s, r)| Some((sources[s].geometry()?, r))) { + Some((geometry, r)) => builder.push_rows(geometry, r..r + n), + None => (0..n).for_each(|_| builder.push_empty()), } } Array::Geometry(builder.finish()) } }; + // No validity bits are needed when no run is missing and no source it reads has missing + // values. + let gaps = runs.iter().any(|(run, _)| run.is_none_or(|(s, _)| sources[s].valid.is_some())); + if !gaps { + return Values { array, valid: None }; + } let mut bits = BitsBuilder::with_capacity(len); let mut all = true; - for i in 0..len { - let valid = pick(i).is_some_and(|(s, r)| sources[s].is_valid(r)); - all &= valid; - bits.push(valid); + for &(run, n) in runs { + for k in 0..n { + let valid = run.is_some_and(|(s, r)| sources[s].is_valid(r + k)); + all &= valid; + bits.push(valid); + } } - Ok(Values { array, valid: (!all).then(|| bits.finish()) }) + Values { array, valid: (!all).then(|| bits.finish()) } } - /// Whether `other` holds exactly these values in the same storage, compared where they are stored: - /// floats by their bits, so a NaN equals itself, and no value is made to compare it. + /// Whether `other` holds exactly these values in the same storage. Compares the stored form + /// without building any [`Value`]. Floats compare by their bits, so NaN equals itself. pub(crate) fn same(&self, other: &Values) -> bool { let len = self.len(); if other.len() != len || (0..len).any(|i| self.is_valid(i) != other.is_valid(i)) { @@ -421,9 +485,8 @@ impl Values { } } - /// Whether value `i` and `other`'s value `j` are the same, compared where they are stored: floats by - /// their bits, so a NaN equals itself, and missing values alike; values in different storages - /// differ. + /// Whether value `i` equals value `j` of `other` as stored. Floats compare by their bits, so + /// NaN equals itself. Two missing values are equal, and values in different storages differ. pub(crate) fn same_row(&self, i: usize, other: &Values, j: usize) -> bool { let valid = self.is_valid(i); if valid != other.is_valid(j) { @@ -445,65 +508,7 @@ impl Values { } } - /// `parts` one after another, in the first part's storage: what a column's pages make, a copy per - /// part with nothing picked. Every part holds the first's kind. - pub(crate) fn concat(parts: &[&Values]) -> Values { - let first = parts.first().expect("a part"); - debug_assert!(parts.iter().all(|p| p.kind() == first.kind())); - let len = parts.iter().map(|p| p.len()).sum(); - macro_rules! numbers { - ($variant:ident, $t:ty, $store:expr) => {{ - let mut out: Vec<$t> = Vec::with_capacity(len); - for part in parts { - match &part.cast(|x| x as $t, $store).array { - Array::$variant(v) => out.extend_from_slice(v), - _ => unreachable!("cast to the first part's storage"), - } - } - Array::$variant(Buffer::from(out)) - }}; - } - let array = match &first.array { - Array::F64(_) => numbers!(F64, f64, Store::F64), - Array::F32(_) => numbers!(F32, f32, Store::F32), - Array::I32(_) => numbers!(I32, i32, Store::I32), - Array::U32(_) => numbers!(U32, u32, Store::U32), - Array::U64(_) => numbers!(U64, u64, Store::U64), - Array::Bool(_) => { - let mut bits = BitsBuilder::with_capacity(len); - for part in parts { - if let Array::Bool(v) = &part.array { - (0..v.len()).for_each(|i| bits.push(v.get(i))); - } - } - Array::Bool(bits.finish()) - } - Array::Text(_) => { - let mut builder = TextBuilder::with_capacity(len); - for text in parts.iter().filter_map(|p| p.text()) { - (0..text.len()).for_each(|i| builder.push(text.get(i))); - } - Array::Text(builder.finish()) - } - Array::Geometry(g) => { - let mut builder = GeometryBuilder::new(g.depth()); - for geometry in parts.iter().filter_map(|p| p.geometry()) { - (0..geometry.len()).for_each(|i| builder.push_from(geometry, i)); - } - Array::Geometry(builder.finish()) - } - }; - let valid = parts.iter().any(|p| p.valid.is_some()).then(|| { - let mut bits = BitsBuilder::with_capacity(len); - for part in parts { - (0..part.len()).for_each(|i| bits.push(part.is_valid(i))); - } - bits.finish() - }); - Values { array, valid } - } - - /// Numbers converted to `store`; other kinds unchanged. + /// Numbers converted to `store`. Other kinds are unchanged. fn cast(&self, convert: impl Fn(f64) -> T, store: Store) -> Values where Buffer: Into, @@ -683,9 +688,10 @@ impl From>>>> for Values { } } -/// A column of values of one kind, `None` missing, as JSON and JavaScript arrays give them: the first -/// present value decides what the rest are, numbers, booleans, texts, or geometries nesting alike. -/// A column of nothing present is numbers; a missing point is NaN, a missing path or polygon empty. +/// Builds a column from values of one kind, with `None` for missing, as JSON and JavaScript arrays +/// provide them. The first present value sets the kind: numbers, booleans, texts, or geometries of +/// equal nesting. A column with no present values is numbers. A missing point becomes NaN, and a +/// missing path, parts, or polygons value becomes empty. Mixed kinds and [`Value::Row`] are errors. impl TryFrom>> for Values { type Error = Error; @@ -693,7 +699,7 @@ impl TryFrom>> for Values { fn all(values: Vec>, of: impl Fn(Value) -> Option) -> Result>> { let mixed = || { Error::invalid( - "a column holds one kind of value: numbers, booleans, texts, or geometries nesting alike", + "a column must hold one kind of value: numbers, booleans, texts, or geometries of one depth", ) }; values.into_iter().map(|v| v.map(|v| of(v).ok_or_else(mixed)).transpose()).collect() @@ -722,7 +728,7 @@ impl TryFrom>> for Values { let polygons = all(values, |v| if let Value::Polygons(p) = v { Some(p) } else { None })?; polygons.into_iter().map(Option::unwrap_or_default).collect::>().into() } - Value::Row(_) => return Err(Error::invalid("a column names rows as a reference: {reference, ids}")), + Value::Row(_) => return Err(Error::invalid("rows must be written as a reference: {reference, ids}")), }) } } @@ -763,7 +769,7 @@ mod tests { } #[test] - fn same_compares_what_is_stored() { + fn same_compares_the_stored_representation() { let a = Values::from(vec![Some(1.0_f64), None, Some(f64::NAN)]); assert!(a.same(&a.slice(0..3)), "a NaN equals itself"); assert!(!a.same(&Values::from(vec![Some(1.0_f64), Some(0.0), Some(f64::NAN)])), "missing differs"); diff --git a/crates/latkit/src/derive/graph.rs b/crates/latkit/src/derive/graph.rs index 99091d04..873013a7 100644 --- a/crates/latkit/src/derive/graph.rs +++ b/crates/latkit/src/derive/graph.rs @@ -1,30 +1,30 @@ -//! The graph the marks draw: one vertex mark and one edge mark per table, and where edges meet -//! vertices, by their two ends or by the ports naming a net. Layouts link vertices by it, a diagram's -//! wires run along it, and nets' stars join it. +//! The graph the scene's marks draw: one vertex mark and one edge mark per table, and how edges +//! meet vertices, either by two endpoint references or through ports that reference a net. +//! Layouts, a diagram's wires, and the stars of nets all read it. -use crate::data::{Data, Field, Inverse, Name, Reference, Row, Table, Value}; +use crate::data::{Field, Inverse, Name, Reference, Row, Snapshot, Table, Value}; use crate::expr::cpu::Rows; use crate::host::Key; use crate::scene::mark::Style; use crate::{Edges, Mark, Port, Vertices}; -/// The vertex mark a table's rows take their places from, and the vertex index of its first row. +/// The vertex mark that places a table's rows, with the vertex index of the table's first row. pub(crate) struct VertexMark<'a> { pub(crate) mark: &'a Vertices, - pub(crate) data: &'a Data, + pub(crate) data: &'a Snapshot, pub(crate) table: &'a Table, pub(crate) first: usize, } -/// The edge mark a table's rows take their paths from. +/// The edge mark that draws a table's rows as paths. pub(crate) struct EdgeMark<'a> { pub(crate) mark: &'a Edges, - pub(crate) data: &'a Data, + pub(crate) data: &'a Snapshot, pub(crate) table: &'a Table, } -/// The vertex marks and edge marks of the scene, one per table: its unnamed mark of the kind, else -/// its first; each vertex mark with the vertex index of its first row. +/// The scene's vertex and edge marks, one of each kind per table: the table's unnamed mark of that +/// kind, else its first. Vertex indices number the rows of every vertex mark in turn. pub(crate) fn marks<'a>(input: &super::Input<'a>) -> (Vec>, Vec>) { let (mut vertices, mut edges): (Vec>, Vec>) = (Vec::new(), Vec::new()); for mark in &input.scene.marks { @@ -59,23 +59,25 @@ pub(crate) fn marks<'a>(input: &super::Input<'a>) -> (Vec>, Vec(mut styles: impl Iterator, style: &Style) -> Option { styles.position(|s| s.data == style.data && s.table == style.table) } -/// Whether the mark of `style` takes its table from the one of `before`: an unnamed mark from a -/// named one. +/// Whether the mark of `style` replaces the mark of `before` for their table: an unnamed mark +/// replaces a named one. fn replaces(before: &Style, style: &Style) -> bool { before.name.is_some() && style.name.is_none() } -/// The vertex mark drawing `table` of dataset `data`, which ends and members attach to. +/// The index of the vertex mark drawing `table` of dataset `data`, which edge endpoints and group +/// members attach to. pub(crate) fn drawing(vertex_marks: &[VertexMark<'_>], data: &Name, table: &Name) -> Option { vertex_marks.iter().position(|vm| vm.data.name() == data && vm.table.name() == table) } -/// The position row `row`'s data gives it, if any: its position, else its point geometry. +/// Row `row`'s position from its data: the mark's position fields, else its point geometry (finite +/// points only). `None` where the data gives none. pub(crate) fn position(mark: &Vertices, rows: &Rows<'_>, row: usize) -> Option<[f64; 2]> { match &mark.position { Some([x, y]) => Some([rows.number(x, row)?, rows.number(y, row)?]), @@ -91,7 +93,8 @@ pub(crate) fn key_of(row: &Row) -> Key { (row.table.clone(), row.id.clone()) } -/// What a layout links: each edge's two ends, and each net from its outputs to its inputs, by vertex. +/// The links a layout reads, by vertex index: each edge's two endpoints (self-loops dropped), and +/// each net's links from its outputs to its inputs (see `net_links`). pub(crate) fn links(marks: &[VertexMark<'_>], edge_marks: &[EdgeMark<'_>]) -> Vec<[u32; 2]> { let codes: Vec> = marks.iter().map(|vm| port_codes(vm.mark)).collect(); let mut links = Vec::new(); @@ -112,15 +115,16 @@ pub(crate) fn links(marks: &[VertexMark<'_>], edge_marks: &[EdgeMark<'_>]) -> Ve links } -/// Where an edge mark's rows meet vertices: the references naming each row's two ends, with the -/// first vertex of the table each names; or, for nets, every port reference naming its rows. +/// How an edge mark's rows meet vertices. For an edge table, the two endpoint references, each with +/// the first vertex index of the table it references. For a net table, every port reference to its +/// rows. pub(crate) struct Joins<'a> { ends: Option<[(&'a Reference, usize); 2]>, ports: Vec>, } -/// A port reference naming a net table's rows: the vertex mark, the port's code and whether it is an -/// output, and the rows naming each net. +/// A port reference to a net table's rows: the vertex mark, the port's code, whether the port is an +/// output, and the inverse index from each net to the rows that reference it. struct Naming<'a> { mark: usize, code: u32, @@ -153,14 +157,15 @@ impl<'a> Joins<'a> { Joins { ends: None, ports } } - /// Row `row`'s two ends, by vertex. + /// Row `row`'s two endpoints, as vertex indices. `None` for a net table, or where either + /// reference is unset. pub(crate) fn ends(&self, row: usize) -> Option<[usize; 2]> { let [(a, first_a), (b, first_b)] = self.ends?; Some([first_a + a.target(row)? as usize, first_b + b.target(row)? as usize]) } - /// Net row `row`'s terminals: each vertex naming it at a port, the port's code, and whether it is - /// an output. + /// Net row `row`'s terminals: each vertex whose port references it, with the port's code and + /// whether the port is an output. pub(crate) fn terminals(&self, marks: &[VertexMark<'_>], row: usize) -> Vec<(usize, u32, bool)> { let mut terminals = Vec::new(); for naming in &self.ports { @@ -171,14 +176,15 @@ impl<'a> Joins<'a> { } } -/// Each port's code on a vertex mark's rows. +/// The code of each port of a vertex mark. pub(crate) fn port_codes(mark: &Vertices) -> Vec { Port::codes(&mark.ports.iter().map(|(_, port)| port).collect::>()) } -/// The links of a net whose terminals are `ends`, each a vertex and whether it drives the net: every -/// driver to its first reader, and its first driver to every reader, so a wide net adds as many links -/// as it has terminals. A net without both drivers and readers runs from its first terminal. +/// The links of a net whose terminals are `ends`, each a vertex and whether it drives the net. +/// Every driver links to the first reader, and the first driver to every other reader, so a net +/// adds about one link per terminal. A net lacking drivers or readers treats its first terminal as +/// the only driver. Self-links are dropped. fn net_links(ends: &[(u32, bool)]) -> Vec<[u32; 2]> { let (mut drivers, mut readers): (Vec, Vec) = (Vec::new(), Vec::new()); for &(vertex, drives) in ends { @@ -200,6 +206,6 @@ mod tests { #[test] fn nets_link_their_drivers_to_their_readers() { assert_eq!(net_links(&[(0, false), (1, true), (2, false)]), vec![[1, 0], [1, 2]]); - assert_eq!(net_links(&[(4, false), (5, false)]), vec![[4, 5]], "without a driver, the first drives"); + assert_eq!(net_links(&[(4, false), (5, false)]), vec![[4, 5]], "without a driver, the first terminal drives"); } } diff --git a/crates/latkit/src/derive/lanes.rs b/crates/latkit/src/derive/lanes.rs index c7452936..cc97026c 100644 --- a/crates/latkit/src/derive/lanes.rs +++ b/crates/latkit/src/derive/lanes.rs @@ -1,27 +1,36 @@ -//! Lanes: outside a diagram, each edge row's lane among the edges joining the same two vertices, so -//! they bow apart instead of drawing over each other. A lane is data, made from the table's ends -//! alone: the stroke bows in world space by its lane, so it never moves as the camera does. Derived -//! again only when the table's rows or ends change: on the frame while that fits its budget, else in -//! a job; a table whose edges share no ends derives nothing. +//! Lanes of parallel edges, outside diagrams. +//! +//! Edges joining the same two vertices form a family. Each edge row gets a lane within its family, +//! so the strokes bow apart instead of drawing over each other. Lanes depend only on the table's +//! endpoints, and strokes bow in world space, so lanes never change as the camera moves. +//! +//! A row's lane and its family size share one 32-bit word: twice the lane plus `HALF` in the high +//! 16 bits, the family size in the low 16. Lanes are recomputed only when the table's rows or +//! endpoints change: on the frame when that fits the budget, else in a job. A table whose edges +//! share no endpoints gets no lane field. use rustc_hash::{FxBuildHasher, FxHashMap, FxHashSet}; use super::{Derive, Derived, Input, Jobs, Tables, beside, kept, stamp_of}; use crate::Mark; use crate::Pending; -use crate::data::{Field, Ids, Keyed, Name, Patch, Reference, Table, UNWIRED, Values}; +use crate::data::{Field, Keyed, Name, Reference, Table, UNWIRED, Values}; use crate::host::Job; -/// Microseconds a row takes to find its lane, by estimate. +/// Estimated microseconds to compute one row's lane. const LANE_COST: f64 = 0.06; +/// The offset added to twice the lane, so the high half of a lane word is never negative. +const HALF: u32 = 1 << 15; +/// The largest family size a lane word stores. Larger families store this. +const MOST: u32 = (1 << 16) - 1; -/// What a table's lanes were made from, and a job under way. +/// The stamp a table's lanes were made from, and any pending job. type Made = Derived<(), Pending>>; #[derive(Default)] pub(crate) struct Lanes { tables: Tables, - /// Each table's lanes, made or making. + /// Each table's lane state: made, or pending in a job. made: Keyed<(Name, Name), Made>, } @@ -54,7 +63,7 @@ impl Derive for Lanes { let rows = table.len(); if jobs.fits((rows as f64 * LANE_COST) as u64) { made.make(stamp, ()); - changed |= keep(&mut self.tables, key, table, lanes(&a.targets(), &b.targets())); + changed |= keep(&mut self.tables, key, table, words(&a.targets(), &b.targets())); } else { made.start(stamp, jobs.spawn(Job::lanes(&a.targets(), &b.targets()))); } @@ -79,12 +88,14 @@ impl Derive for Lanes { } } -/// Keeps `lanes` beside `table` as its field `lane`, or drops its lanes when there are none; whether -/// anything changed. -fn keep(tables: &mut Tables, key: (Name, Name), table: &Table, lanes: Option>) -> bool { +/// Writes `lanes`, one lane word per row, as field `lane` of the table beside `table`. Drops that +/// table when there are no lanes or their count differs from the rows. Returns whether it wrote or +/// dropped a table. +fn keep(tables: &mut Tables, key: (Name, Name), table: &Table, lanes: Option>) -> bool { match lanes.filter(|l| l.len() == table.len()) { Some(lanes) => { - let lane = kept("lane", &Values::from(lanes), table, tables.get(&key)); + let words: Vec = lanes.into_iter().map(f64::from).collect(); + let lane = kept("lane", &Values::from(words), table, tables.get(&key)); tables.insert(key, beside(table, [("lane".into(), lane)])); true } @@ -100,24 +111,35 @@ fn reference<'t>(table: &'t Table, name: &str) -> Option<&'t Reference> { } } -/// The lanes a lanes job answered: its table's field `lane`, or none. -fn answered(bytes: &[u8]) -> Option> { - let patch = Patch::from_bytes(bytes).ok()?; - patch.ops.into_iter().find_map(|op| match op { - crate::data::Op::Set(table) => { - let Some(Field::Values(column, _)) = table.field_named("lane") else { return None }; - let pages = column.pages().iter(); - Some(pages.flat_map(|page| (0..page.values.len()).map(|i| page.values.number(i) as f32)).collect()) - } - _ => None, - }) +/// The lane words in a lanes job's answer, each a little-endian `u32`. `None` for an empty answer, +/// which means no rows share endpoints. +fn answered(bytes: &[u8]) -> Option> { + (!bytes.is_empty()).then(|| crate::wire::words(bytes)) } -/// Each row's lane among the rows joining the same two vertices, from the rows' two ends -/// ([`UNWIRED`] where a reference names no row): offsets from the middle, signed so rows joining them -/// either way bow apart. `None` when no two rows share their ends. Counts each pair once, then hands -/// out lanes only to the pairs shared, so it allocates nothing per pair. -pub(crate) fn lanes(from: &[u32], to: &[u32]) -> Option> { +/// Each row's lane word, as shaders read it, from the rows' endpoints: twice the lane plus `HALF` +/// in the high 16 bits, the family size (at most `MOST`) in the low 16. `None` when no two rows +/// share endpoints. +pub(crate) fn words(from: &[u32], to: &[u32]) -> Option> { + let lanes = lanes(from, to)?; + Some( + lanes + .into_iter() + .map(|(lane, size)| { + let twice = (2.0 * lane).clamp(-f32::from(HALF as u16 - 1), f32::from(HALF as u16 - 1)) as i32; + ((twice + HALF as i32) as u32) << 16 | size.min(MOST) + }) + .collect(), + ) +} + +/// Each row's lane among the rows joining the same two vertices, and the size of that family, from +/// the rows' endpoints ([`UNWIRED`] where a reference is unset). A lane is an offset from the +/// family's middle, negated for rows running the other way, so every row of a family bows apart. +/// A row with no parallel rows (including unset rows and self-loops) gets lane 0 in a family of 1. +/// `None` when no two rows share endpoints. It counts each pair in one map, then assigns lanes only +/// to shared pairs, so it allocates nothing per pair. +pub(crate) fn lanes(from: &[u32], to: &[u32]) -> Option> { let key = |r: usize| { let (a, b) = (from[r], to[r]); (a != UNWIRED && b != UNWIRED && a != b).then(|| u64::from(a.min(b)) << 32 | u64::from(a.max(b))) @@ -140,43 +162,43 @@ pub(crate) fn lanes(from: &[u32], to: &[u32]) -> Option> { Some( (0..rows) .map(|r| { - let Some((k, &n)) = key(r).and_then(|k| Some((k, counts.get(&k)?))) else { return 0.0 }; + let Some((k, &n)) = key(r).and_then(|k| Some((k, counts.get(&k)?))) else { return (0.0, 1) }; let next = taken.entry(k).or_insert(0); let lane = *next as f32 - (n - 1) as f32 / 2.0; *next += 1; - if from[r] > to[r] { -lane } else { lane } + (if from[r] > to[r] { -lane } else { lane }, n) }) .collect(), ) } -/// Lanes as a job's answer: a table of every row's lane, or one without rows when there are none. -pub(crate) fn answer(from: &[u32], to: &[u32]) -> Patch { - let lanes = lanes(from, to).unwrap_or_default(); - Patch::new().set(Table::new("lanes", Ids::count(lanes.len() as u32)).field("lane", lanes)) -} - -/// Each of `rows` rows' lane among the rows joining the same two ends, which `ends` names: the -/// sheet's straight wires spread by it. `None` when no two share their ends. +/// The lane of each of `rows` rows among the rows joining the same two vertices. `ends` gives each +/// row's endpoints. The sheet spreads straight wires by lane. `None` if no rows share endpoints. pub(crate) fn spread(rows: usize, ends: impl Fn(usize) -> Option<[usize; 2]>) -> Option> { let (from, to): (Vec, Vec) = (0..rows).map(|r| ends(r).map_or((UNWIRED, UNWIRED), |[a, b]| (a as u32, b as u32))).unzip(); - lanes(&from, &to).map(|lanes| lanes.into_iter().map(f64::from).collect()) + lanes(&from, &to).map(|lanes| lanes.into_iter().map(|(lane, _)| f64::from(lane)).collect()) } #[cfg(test)] mod tests { - use super::{UNWIRED, lanes}; + use super::{HALF, UNWIRED, lanes, words}; #[test] - fn rows_joining_the_same_ends_bow_apart_either_way() { + fn edges_joining_the_same_vertices_bow_apart_in_either_direction() { // Rows 0, 1, and 3 join vertices 1 and 2, row 3 the other way; row 2 is alone. - let lanes = lanes(&[1, 1, 4, 2, UNWIRED], &[2, 2, 5, 1, 0]).expect("shared ends"); - assert_eq!(lanes, vec![-1.0, 0.0, 0.0, -1.0, 0.0]); + let lanes = lanes(&[1, 1, 4, 2, UNWIRED], &[2, 2, 5, 1, 0]).expect("rows share endpoints"); + assert_eq!(lanes, vec![(-1.0, 3), (0.0, 3), (0.0, 1), (-1.0, 3), (0.0, 1)]); + } + + #[test] + fn a_lane_word_holds_twice_the_lane_and_the_family_size() { + let words = words(&[1, 1], &[2, 2]).expect("rows share endpoints"); + assert_eq!(words, vec![(HALF - 1) << 16 | 2, (HALF + 1) << 16 | 2], "lanes -0.5 and 0.5 in a family of two"); } #[test] - fn tables_sharing_no_ends_have_no_lanes() { + fn rows_sharing_no_endpoints_get_no_lanes() { assert_eq!(lanes(&[0, 1, 2], &[1, 2, 0]), None); } } diff --git a/crates/latkit/src/derive/matches.rs b/crates/latkit/src/derive/matches.rs new file mode 100644 index 00000000..e4753780 --- /dev/null +++ b/crates/latkit/src/derive/matches.rs @@ -0,0 +1,127 @@ +//! Text matches. For each `field.contains(text)` a mark uses, computes whether each row's text +//! field contains `text`, ignoring case. The results are boolean fields of a table beside the +//! mark's own table, which the GPU reads. Texts stay on the CPU, so matches are computed here, and +//! again only when the field or the rows change. A page whose matches are unchanged keeps its +//! column page, so only changed pages upload. + +use std::sync::Arc; + +use super::{Derive, Input, Jobs, Tables, beside, kept}; +use crate::data::{Field, Keyed, Name, Table, Values, Version}; + +/// The name of the derived field holding whether each row's text field `field` contains `text`, +/// ignoring case. +pub(crate) fn contains_of(field: &str, text: &str) -> Name { + format!("{field}.contains({text})").into() +} + +/// What a table's matches were made from: its structure version, and each field and text tested, +/// with the field's version. +type Made = (Version, Vec<(Name, Arc, Option)>); + +/// For each table, the distinct field and text pairs its marks test with `contains`, in mark order. +type Asked = Keyed<(Name, Name), Vec<(Name, Arc)>>; + +#[derive(Default)] +pub(crate) struct Matches { + tables: Tables, + made: Keyed<(Name, Name), Made>, +} + +impl Derive for Matches { + fn update(&mut self, input: &Input<'_>, _: &mut Jobs<'_>) -> bool { + let mut asked = Asked::default(); + for mark in &input.scene.marks { + let style = mark.style(); + let key = (style.data.clone(), style.table.clone()); + mark.for_each_expr(&mut |expr| { + expr.for_each_contains(&mut |field, text| { + let fields = asked.entry(key.clone()).or_default(); + if !fields.iter().any(|(f, t)| *f == field.name && **t == *text) { + fields.push((field.name.clone(), text.into())); + } + }); + }); + } + let mut changed = false; + for (key, fields) in asked.iter() { + let Some(table) = input.data(&key.0).and_then(|d| d.table(&key.1)) else { continue }; + let version = |f: &Name| table.field_named(f).map(Field::version); + let made: Made = + (table.structure, fields.iter().map(|(f, t)| (f.clone(), t.clone(), version(f))).collect()); + if self.made.get(key) == Some(&made) { + continue; + } + let before = self.tables.get(key); + let columns: Vec<(Name, Field)> = fields + .iter() + .map(|(field, text)| { + let name = contains_of(field, text); + let column = kept(&name, &holds(table, field, text), table, before); + (name, column) + }) + .collect(); + self.tables.insert(key.clone(), beside(table, columns)); + self.made.insert(key.clone(), made); + changed = true; + } + let before = self.tables.len(); + self.tables.retain(|key, _| asked.contains_key(key)); + self.made.retain(|key, _| asked.contains_key(key)); + changed || self.tables.len() != before + } + + fn busy(&self) -> bool { + false + } + + fn tables(&self) -> &Tables { + &self.tables + } + + fn clear(&mut self) { + self.tables.clear(); + self.made.clear(); + } +} + +/// Whether each row of `table`'s text field `field` contains `text`, ignoring case. Missing +/// values, and fields that are missing or not text, match nothing. Reads page by page without +/// copying texts. +fn holds(table: &Table, field: &str, text: &str) -> Values { + let needle = text.to_lowercase(); + let mut out = Vec::with_capacity(table.len()); + match table.field_named(field) { + Some(Field::Values(column, _)) => { + for page in column.pages() { + let values = &page.values; + match values.text() { + Some(texts) => out.extend( + texts.iter().enumerate().map(|(i, t)| values.is_valid(i) && crate::expr::holds(t, &needle)), + ), + None => out.extend(std::iter::repeat_n(false, values.len())), + } + } + } + _ => out.resize(table.len(), false), + } + out.resize(table.len(), false); + Values::from(out) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{Ids, Value}; + + #[test] + fn rows_contain_a_text_ignoring_case() { + let names = vec![Some("Lake Travis"), None, Some("LAKE"), Some("Café")]; + let table = Table::new("Lake", Ids::count(4)).field("name", names); + let rows = + |values: Values| (0..values.len()).map(|i| values.get(i) == Some(Value::Bool(true))).collect::>(); + assert_eq!(rows(holds(&table, "name", "lake")), [true, false, true, false]); + assert_eq!(rows(holds(&table, "name", "CAFÉ")), [false, false, false, true], "beyond ASCII too"); + assert_eq!(rows(holds(&table, "absent", "x")), [false; 4]); + } +} diff --git a/crates/latkit/src/derive/mod.rs b/crates/latkit/src/derive/mod.rs index 9d5a19ef..e35dcd84 100644 --- a/crates/latkit/src/derive/mod.rs +++ b/crates/latkit/src/derive/mod.rs @@ -1,28 +1,30 @@ -//! Derived data: what the view works out rather than is given. Every derivation shares one lifecycle: -//! a stamp of what it reads, so it works again only when that changes; work small enough for the -//! frame done on it, within one budget all derivations share; larger work a host job answering a -//! patch, kept only while what it was asked from still holds; and tables it writes beside the tables -//! marks draw, row for row, which marks read like data. What the GPU works out (the shadow map, -//! normals, segment counts, label placement) follows the same stamps, its answers read back. +//! Derived data: tables the view computes from the scene and data. //! -//! The derivations: a diagram's sheet (places, sizes, boxes, wires); elsewhere the lanes of edges -//! sharing their ends, the stars of nets, and where the scene has a layout the places of vertex rows -//! without positions; -//! the triangles of every area; and the tiles the camera asks for, which patch datasets the view -//! owns. +//! Every derivation works the same way. It stamps its inputs and recomputes only when the stamp +//! changes. Small work runs on the frame, within one budget ([`FRAME_WORK`]) that all derivations +//! share. Larger work runs as a host job, and its result is kept only while its inputs are +//! unchanged. Results are tables that stand beside the drawn tables, row for row, and marks read +//! them like data. Values the GPU computes (normals, segment counts, the texts labels need) use the +//! same stamps and are read back. +//! +//! In a diagram, the sheet derives places, sizes, group boxes, and wires. Outside diagrams, the +//! derivations are lanes of edges that share both endpoints, stars of nets, and, when the scene has +//! a layout, places of vertex rows without positions. In every space they also derive shapes of +//! geometry fields (every area's triangles), text matches for `contains`, the pieces of given tile +//! sets the camera needs, and what each paged table draws. pub(crate) mod graph; mod lanes; +pub(crate) mod matches; +pub(crate) mod order; pub(crate) mod places; pub(crate) mod pyramid; -mod shapes; +pub(crate) mod shapes; pub(crate) mod sight; mod stars; pub(crate) mod tiled; -pub(crate) mod tiles; -mod tiling; -pub(crate) use lanes::{answer as lanes_answer, spread}; +pub(crate) use lanes::{spread, words as lane_words}; use std::hash::{Hash, Hasher}; use std::sync::{Arc, Mutex}; @@ -30,25 +32,30 @@ use std::task::Waker; use rustc_hash::FxHasher; -use crate::data::{Column, Data, Field, Keyed, Name, Op, Page, Patch, Table, Value, Values, Version}; +use crate::data::{Column, Field, Keyed, Name, Op, Page, Patch, Snapshot, Table, Value, Values, Version}; use crate::gpu::readback::Readback; use crate::gpu::work::{Count, Counters}; use crate::host::{Host, Job, Key, Request}; use crate::sheet::Sheet; +use crate::tiles::TileSets; use crate::{Pending, Result, Scene, Space}; use places::Places; -use tiles::TileSets; -/// A stamp of `inputs`: never 0, which no value is made from. +/// Hashes `inputs` into a stamp. Never 0, which means no value has been made yet. pub(crate) fn stamp_of(inputs: impl Hash) -> u64 { + stamp_with(|hasher| inputs.hash(hasher)) +} + +/// A stamp of what `hash` writes, for inputs that hash themselves into a hasher. +pub(crate) fn stamp_with(hash: impl FnOnce(&mut FxHasher)) -> u64 { let mut hasher = FxHasher::default(); - inputs.hash(&mut hasher); + hash(&mut hasher); hasher.finish().max(1) } -/// Work that lands later: bytes the GPU reads back, or a job or fetch the host does. +/// Work whose bytes arrive later: a GPU readback, or a host job or fetch. pub(crate) trait Answer { - /// The bytes once they landed, taking them; `None` while they have not. + /// Takes the bytes once they arrive. `None` until then. fn take(&self) -> Option>>; } @@ -64,96 +71,149 @@ impl Answer for Pending> { } } -/// A derived product: its value, the stamp it was made from, and the work under way toward the value -/// of a newer stamp. +/// Work under way: its answer, the stamp of the inputs it was started for, and the context its +/// answer is read with. Work that lands counts as done for its stamp, whether it succeeded or +/// failed, so no work for that stamp starts again. A job runs once for each set of inputs. Stamp 0 +/// is for work started on request, not for inputs, and is not tracked. +pub(crate) struct Working>> { + pending: Option<(u64, A, C)>, + /// The stamp of the work that landed last, or 0 for none. + done: u64, +} + +impl Default for Working { + fn default() -> Self { + Working { pending: None, done: 0 } + } +} + +impl Working { + /// Whether work is pending. + pub(crate) fn busy(&self) -> bool { + self.pending.is_some() + } + + /// Whether work for `stamp` may start: none is pending, and none for `stamp` landed. + pub(crate) fn may_start(&self, stamp: u64) -> bool { + self.pending.is_none() && (stamp == 0 || stamp != self.done) + } + + /// The stamp of the pending work. + pub(crate) fn stamp(&self) -> Option { + self.pending.as_ref().map(|(stamp, ..)| *stamp) + } + + /// The context of the pending work. + pub(crate) fn context(&self) -> Option<&C> { + self.pending.as_ref().map(|(_, _, context)| context) + } + + /// Records `answer` as pending for `stamp`, read with `context`, replacing any pending work. + pub(crate) fn start(&mut self, stamp: u64, answer: A, context: C) { + self.pending = Some((stamp, answer, context)); + } + + /// Drops the pending work. Its answer is ignored. + pub(crate) fn cancel(&mut self) { + self.pending = None; + } + + /// Takes finished work: its stamp, context, and bytes or error. + pub(crate) fn landed(&mut self) -> Option<(u64, C, Result>)> { + let answer = self.pending.as_ref()?.1.take()?; + let (stamp, _, context) = self.pending.take().expect("pending work"); + self.done = stamp; + Some((stamp, context, answer)) + } +} + +/// A derived value, the stamp it was made from, and any pending work toward a newer stamp. pub(crate) struct Derived { pub(crate) value: T, stamp: u64, - working: Option<(u64, A)>, + working: Working<(), A>, } impl Default for Derived { fn default() -> Self { - Derived { value: T::default(), stamp: 0, working: None } + Derived { value: T::default(), stamp: 0, working: Working::default() } } } impl Derived { - /// The stamp the value was made from; 0 before it was. + /// The stamp the value was made from, or 0 before the first value. pub(crate) fn stamp(&self) -> u64 { self.stamp } - /// Whether `stamp` is neither what the value was made from nor what work under way is for. + /// Whether `stamp` matches neither the value nor the pending work, so new work is needed. pub(crate) fn due(&self, stamp: u64) -> bool { - stamp != self.stamp && self.working.as_ref().is_none_or(|(s, _)| *s != stamp) + stamp != self.stamp && self.working.stamp() != Some(stamp) } - /// Takes `value` as made from `stamp`, dropping any work under way. + /// Stores `value` as made from `stamp`, dropping any pending work. pub(crate) fn make(&mut self, stamp: u64, value: T) { self.value = value; - self.stamp = stamp; - self.working = None; + self.mark(stamp); } - /// Takes the value as made from `stamp` as it stands, dropping any work under way. + /// Marks the current value, unchanged, as made from `stamp`, dropping any pending work. pub(crate) fn mark(&mut self, stamp: u64) { self.stamp = stamp; - self.working = None; + self.working.cancel(); } - /// Starts `work` toward the value of `stamp`, replacing any under way. + /// Records `work` as pending toward `stamp`, replacing any pending work. pub(crate) fn start(&mut self, stamp: u64, work: A) { - self.working = Some((stamp, work)); + self.working.start(stamp, work, ()); } - /// The bytes of work that landed, and the stamp it was for. Work that failed lands as made, - /// keeping the value, so it is not tried again until what it reads changes. + /// Takes the bytes of finished work and the stamp it was for. Failed work marks its stamp as + /// made and keeps the old value, so it is not retried until the inputs change. pub(crate) fn landed(&mut self) -> Option<(u64, Vec)> { - let (stamp, work) = self.working.as_ref()?; - let stamp = *stamp; - let landed = work.take()?; - self.working = None; - let Ok(bytes) = landed else { + let (stamp, (), answer) = self.working.landed()?; + let Ok(bytes) = answer else { self.stamp = stamp; return None; }; Some((stamp, bytes)) } - /// Whether work is under way. + /// Whether work is pending. pub(crate) fn busy(&self) -> bool { - self.working.is_some() + self.working.busy() } } -/// Microseconds of work, by estimate, the derivations do on one frame between them; what does not fit -/// goes to the host as jobs. -const FRAME_WORK: u32 = 4000; +/// Estimated microseconds of derivation work all of a GPU's views may do on one frame together. +/// Work that does not fit goes to the host as jobs. +pub(crate) const FRAME_WORK: u32 = 4000; -/// The frame's budget of inline work and the host's pool, which every derivation shares. Each job -/// and fetch wakes the view when it lands. +/// The frame's inline work budget and the host's job pool, shared by every derivation. Each job and +/// fetch wakes the view when it finishes. pub(crate) struct Jobs<'a> { errors: &'a Arc>>, host: Option<&'a dyn Host>, waker: &'a Waker, - /// What counts the jobs handed out. + /// Counts the jobs spawned. counters: &'a Counters, - /// Microseconds of inline work left this frame, by estimate. + /// Estimated microseconds of inline work left this frame. left: u32, } impl<'a> Jobs<'a> { + /// A frame's jobs, with `left` microseconds of inline work still to spend. pub(crate) fn new( host: Option<&'a dyn Host>, waker: &'a Waker, counters: &'a Counters, errors: &'a Arc>>, + left: u32, ) -> Jobs<'a> { - Jobs { host, waker, counters, errors, left: FRAME_WORK } + Jobs { errors, host, waker, counters, left } } - /// Whether work of `cost` microseconds fits what is left of the frame's budget, spending it if so. + /// Whether `cost` microseconds fit the rest of the frame's budget. Spends them if so. pub(crate) fn fits(&mut self, cost: u64) -> bool { let fits = cost <= u64::from(self.left); if fits { @@ -162,17 +222,29 @@ impl<'a> Jobs<'a> { fits } - /// Microseconds of inline work left this frame, by estimate. + /// Estimated microseconds of inline work left this frame. pub(crate) fn left(&self) -> u64 { u64::from(self.left) } - /// Spends `cost` microseconds of the budget whether it fits or not: work the frame cannot defer. + /// The waker its jobs and fetches wake when they finish. + pub(crate) fn waker(&self) -> &Waker { + self.waker + } + + /// The same host's jobs and fetches, but waking `waker` instead of this frame's view, and with + /// no inline budget. Used for work shared by every view that draws its result. + pub(crate) fn waking<'b>(&'b self, waker: &'b Waker) -> Jobs<'b> { + Jobs { errors: self.errors, host: self.host, waker, counters: self.counters, left: 0 } + } + + /// Spends `cost` microseconds of the budget even when it does not fit, for work the frame + /// cannot defer. The budget stops at zero. pub(crate) fn spend(&mut self, cost: u64) { self.left = self.left.saturating_sub(cost.min(u64::from(u32::MAX)) as u32); } - /// Has the host do `job` off the frame; without a host, at once. + /// Runs `job` on the host, off the frame. Without a host, runs it immediately. pub(crate) fn spawn(&self, job: Job) -> Pending> { self.counters.add(Count::Jobs, 1); let pending = match self.host { @@ -184,7 +256,7 @@ impl<'a> Jobs<'a> { pending } - /// Has the host fetch `request`; without a host, nothing can be fetched. + /// Fetches `request` through the host. Without a host, the fetch fails as unsupported. pub(crate) fn fetch(&self, request: &Request) -> Pending> { let pending = match self.host { Some(host) => host.fetch(request), @@ -195,27 +267,39 @@ impl<'a> Jobs<'a> { } } -/// What every derivation reads: the scene and its data, the host that measures text, and whether -/// rows are being dragged. Jobs go through the frame's [`Jobs`]. +/// The inputs every derivation reads: the scene, its data, the host (which measures text), whether +/// rows are being dragged, and what each paged table draws. Jobs go through the frame's [`Jobs`]. pub(crate) struct Input<'a> { pub(crate) scene: &'a Scene, /// Bumps whenever the scene changes. pub(crate) scene_version: u64, - pub(crate) data: &'a [Data], + pub(crate) data: &'a [Snapshot], pub(crate) host: Option<&'a dyn Host>, - /// Whether rows are being dragged: what they move derives on the frame, however much. + /// Whether rows are being dragged. While true, work the drag causes runs on the frame + /// regardless of budget. pub(crate) dragging: bool, + /// Each paged table, by dataset and table. Row-local derivations work only on the pages and + /// kept rows listed here. + pub(crate) paged: Option<&'a Keyed<(Name, Name), Paged>>, +} + +/// What a paged table draws this frame: its order, the pages drawn, and the places of kept rows +/// outside those pages. Both lists ascend. +pub(crate) struct Paged { + pub(crate) order: order::Order, + pub(crate) pages: Vec, + pub(crate) kept: Vec, } impl Input<'_> { /// The dataset named `name`. - pub(crate) fn data(&self, name: &str) -> Option<&Data> { + pub(crate) fn data(&self, name: &str) -> Option<&Snapshot> { self.data.iter().find(|d| d.name() == name) } - /// Hashes what the structure of the scene's marks reads: of each table drawn, its rows and the - /// fields its places, sizes, groups, ports, wires, and nets read. A field only a look reads, such - /// as a value streaming in, changes nothing it hashes. + /// Hashes the inputs to the structure of the scene's marks: each drawn table's rows, and the + /// fields that set its places, sizes, groups, ports, wires, and nets. Fields read only for + /// appearance, such as a streaming value, do not affect the hash. pub(crate) fn structure(&self, hasher: &mut impl Hasher) { self.reads(&mut |read| match read { Read::Rows { data, table, rows } => (data, table, rows.structure, rows.order).hash(hasher), @@ -223,8 +307,8 @@ impl Input<'_> { }); } - /// Hands `read` what the structure of the scene's marks reads, in order: each table drawn, then - /// each field its places, sizes, groups, ports, wires, and nets read. + /// Calls `read` with each input to the structure of the scene's marks, in order: each drawn + /// table, then each field that sets its places, sizes, groups, ports, wires, and nets. pub(crate) fn reads<'a>(&'a self, read: &mut impl FnMut(Read<'a>)) { for mark in &self.scene.marks { let style = mark.style(); @@ -266,8 +350,9 @@ impl Input<'_> { } } - /// Hands `read` every field `expr` of a mark drawing `table` reads: a field of the table itself is - /// one it sizes or places each row by, `own`, unless a domain fits it to every row's. + /// Calls `read` with every field that `expr` reads, for a mark drawing `table`. When `own` is + /// set, a field of `table` itself is reported as `own`, since it sizes or places each row by + /// that row's value. The exception is when a domain fits `expr` to every row's values. fn read_expr<'a>( &'a self, table: (&'a Name, &'a Name), @@ -277,44 +362,43 @@ impl Input<'_> { ) { let own = own && !expr.fits(); expr.for_each_field(&mut |f| { - let (data, of) = f.of.as_ref().map_or(table, |(d, t)| (d, t)); - let field = self.data(data).and_then(|d| d.table(of)).and_then(|t| t.field_named(&f.name)); - read(Read::Field { data, table: of, name: f.name.as_str(), field, own: own && f.of.is_none() }); + let (data, read_from) = f.from.as_ref().map_or(table, |(d, t)| (d, t)); + let field = self.data(data).and_then(|d| d.table(read_from)).and_then(|t| t.field_named(&f.name)); + read(Read::Field { data, table: read_from, name: f.name.as_str(), field, own: own && f.from.is_none() }); }); } - /// Hands `read` field `name` of `table`. + /// Calls `read` with field `name` of `table`. fn read_field<'a>(&'a self, (data, table): (&'a Name, &'a Name), name: &'a str, read: &mut impl FnMut(Read<'a>)) { let field = self.data(data).and_then(|d| d.table(table)).and_then(|t| t.field_named(name)); read(Read::Field { data, table, name, field, own: false }); } } -/// What the structure of the scene's marks reads. +/// One input to the structure of the scene's marks. pub(crate) enum Read<'a> { - /// A table a mark draws: its rows. + /// A table a mark draws. Rows { data: &'a Name, table: &'a Name, rows: &'a Table }, - /// A field: whether it exists, and as what; and whether it is `own`, a value of a vertex mark's - /// own table that sizes or places each row by that row's value alone. + /// A field, `None` where it is missing. `own` marks a field of a vertex mark's own table that + /// sizes or places each row by that row's value alone. Field { data: &'a Name, table: &'a Name, name: &'a str, field: Option<&'a Field>, own: bool }, } /// Derived tables, by the dataset and table each stands beside. pub(crate) type Tables = Keyed<(Name, Name), Table>; -/// One derivation: what it reads, stamped; what it works out, on the frame or as jobs; and the tables -/// it writes. +/// One derivation: it stamps its inputs, computes on the frame or in jobs, and writes tables. pub(crate) trait Derive { - /// Brings what it derives up to `input`, handing `jobs` what the frame cannot afford; whether its - /// tables changed. + /// Updates the derivation to `input`, sending work the frame cannot afford to `jobs`. Returns + /// whether its tables changed. fn update(&mut self, input: &Input<'_>, jobs: &mut Jobs<'_>) -> bool; - /// Whether work it asked for is under way. + /// Whether work it started is pending. fn busy(&self) -> bool; - /// Whether what it derived is final: nothing it writes stands in for work under way. Rows - /// standing where they were until a layout lands, elbows until routes land, earlier triangles - /// until shapes land, and straight edges until lanes land all stand in. + /// Whether its tables are final, with no placeholder for pending work. Placeholders include + /// rows at their old places until a layout arrives, elbows until routes arrive, rows without + /// triangles until shapes arrive, and straight edges until lanes arrive. fn complete(&self) -> bool { !self.busy() } @@ -322,39 +406,42 @@ pub(crate) trait Derive { /// The tables it derives. fn tables(&self) -> &Tables; - /// Forgets what it derived: its space no longer asks for it. + /// Drops its derived tables, once the scene's space no longer needs them. fn clear(&mut self); - /// Failures to tell since last asked, such as texts its host could not measure. + /// Takes the errors since the last call, such as texts the host could not measure. fn take_errors(&mut self) -> Vec { Vec::new() } } -/// Every derivation of a view, run in order each frame with one budget. +/// Every derivation of a view, updated in order each frame within one budget. #[derive(Default)] pub(crate) struct Derivations { /// A diagram's sheet. pub(crate) sheet: Sheet, - /// Elsewhere, the lanes of edges sharing their ends. + /// Outside diagrams, the lanes of edges that share both endpoints. lanes: lanes::Lanes, - /// Elsewhere, where the scene has a layout, the places of vertex rows without positions. + /// Outside diagrams, when the scene has a layout, the places of vertex rows without positions. places: Places, - /// Elsewhere, the stars of nets joining vertex rows. + /// Outside diagrams, the stars of nets joining vertex rows. stars: stars::Stars, shapes: shapes::Shapes, - /// The tiles the camera asks for, as datasets the view owns. + /// Whether each row's text contains a string a mark asks about. + matches: matches::Matches, + /// The pieces of each given tile set the camera needs, as datasets. pub(crate) tiles: TileSets, - /// What the view draws of each dataset it draws in part. + /// The pyramids of each dataset with paged tables, and what those tables draw. pub(crate) tiled: tiled::Tiled, } -/// The derived tables beside one drawn table: the sheet's or the lanes', then the shapes'. -pub(crate) type Beside<'a> = [Option<&'a Table>; 2]; +/// The derived tables beside one drawn table: the first found of the sheet's, places', stars', or +/// lanes' tables, then the shapes' table, then the matches' table. +pub(crate) type Beside<'a> = [Option<&'a Table>; 3]; impl Derivations { - /// Brings every derivation up to `input`, spending what the tiles left of the frame's `jobs`; - /// whether any table changed. + /// Updates every derivation to `input`, spending what remains of `jobs`' frame budget after + /// the tiles. Returns whether any table changed. pub(crate) fn update(&mut self, input: &Input<'_>, jobs: &mut Jobs<'_>) -> bool { let mut changed = if matches!(input.scene.space, Space::Diagram(_)) { self.lanes.clear(); @@ -366,24 +453,25 @@ impl Derivations { let placed = self.places.update(input, jobs); self.lanes.update(input, jobs) | placed | self.stars.update(input, self.places.tables()) }; - changed |= self.shapes.update(input, jobs); + changed |= self.shapes.update(input, &self.tiles, jobs); + changed |= self.matches.update(input, jobs); changed } - /// Lays every vertex row without a position out again: a diagram's sheet, or elsewhere the - /// places. + /// Lays out every vertex row without a position again: through the sheet in a diagram, else + /// through the places. pub(crate) fn arrange(&mut self, diagram: bool) { if diagram { self.sheet.arrange() } else { self.places.arrange() } } - /// What the last arranging laid out, once it landed: each row placed, and where, in the - /// space's coordinates. + /// Takes the result of the last `arrange` once it arrives: each row laid out and its position, + /// in the space's coordinates. pub(crate) fn take_arranged(&mut self) -> Option> { self.sheet.take_arranged().or_else(|| self.places.take_arranged()) } - /// Whether everything derived is final: no layout, route, lane, or shape job stands in, no tile - /// draws its ancestor, and no table drawn in part draws its first sample. + /// Whether everything derived is final: no layout, route, lane, or shape job is pending, no + /// tile draws its ancestor as a placeholder, and no paged table draws its first sample. pub(crate) fn complete(&self) -> bool { self.sheet.complete() && self.lanes.complete() @@ -393,12 +481,18 @@ impl Derivations { && self.tiled.complete() } - /// Whether shapes are being triangulated or bent: until they land, marks draw the ones before. + /// Whether a layout job is pending: a diagram's sheet, or the places of vertex rows without a + /// position. Rows stand at placeholder places until it lands, so a fit waits for it. + pub(crate) fn placing(&self) -> bool { + self.sheet.placing() || self.places.busy() + } + + /// Whether a triangulation job is pending. The rows it shapes draw nothing until it finishes. pub(crate) fn shaping(&self) -> bool { self.shapes.busy() } - /// Failures every derivation has to tell since last asked. + /// Takes the errors every derivation reported since the last call. pub(crate) fn take_errors(&mut self) -> Vec { let mut errors = self.sheet.take_errors(); errors.extend(self.places.take_errors()); @@ -412,11 +506,18 @@ impl Derivations { let first = [self.sheet.tables(), self.places.tables(), self.stars.tables(), self.lanes.tables()] .into_iter() .find_map(|tables| tables.get(&key)); - [first, self.shapes.tables().get(&key)] + [first, self.shapes.tables().get(&key), self.matches.tables().get(&key)] + } + + /// The shapes of paged table `table` of dataset `data`, made per page of its order, keyed by + /// the derived field each is written as. + pub(crate) fn paged(&self, data: &Name, table: &Name) -> Vec<(Name, &shapes::PagedShapes)> { + self.shapes.paged(data, table) } } -/// A table beside `source`, row for row, holding `fields`. +/// A table holding `fields` that stands beside `source`, row for row: it shares `source`'s ids, +/// index, structure, and order. pub(crate) fn beside(source: &Table, fields: impl IntoIterator) -> Table { let mut table = Table { name: source.name.clone(), @@ -432,9 +533,10 @@ pub(crate) fn beside(source: &Table, fields: impl IntoIterator) -> Field { let fresh = Column::new(values); let old = before.filter(|b| b.structure == source.structure).and_then(|b| match b.field_named(name) { @@ -454,7 +556,8 @@ pub(crate) fn kept(name: &str, values: &Values, source: &Table, before: Option<& Field::Values(Column::of_pages(pages, column.like()), Version::next()) } -/// The values `fields` an answer's patch writes for each row it names, by its key. +/// Reads a job's answer patch: for each row it sets, the row's key and its values of `fields`. An +/// unreadable patch yields nothing. pub(crate) fn answers(bytes: &[u8], fields: &[&str]) -> Vec<(Key, Vec>)> { let Ok(patch) = Patch::from_bytes(bytes) else { return Vec::new() }; let mut out = Vec::new(); @@ -469,13 +572,44 @@ pub(crate) fn answers(bytes: &[u8], fields: &[&str]) -> Vec<(Key, Vec usize { + self.0.load(Ordering::Relaxed) + } + } + + impl Host for Failing { + fn rasterize(&self, _: &TextRun) -> crate::Result { + Err(crate::Error::Unsupported("this host has no text".into())) + } + + fn spawn(&self, _: Job) -> Pending> { + self.0.fetch_add(1, Ordering::Relaxed); + Pending::ready(Err(crate::Error::Io("the worker stopped".into()))) + } + } +} + #[cfg(test)] mod tests { use super::*; use crate::data::PAGE_ROWS; #[test] - fn kept_keeps_the_pages_that_hold_the_same() { + fn kept_reuses_pages_whose_values_are_unchanged() { let rows = PAGE_ROWS + 10; let source = Table::new("T", crate::Ids::count(rows as u32)); let numbers = |changed: f64| { @@ -483,10 +617,10 @@ mod tests { }; let first = beside(&source, [(Name::from("x"), kept("x", &numbers(1.0), &source, None))]); let same = kept("x", &numbers(1.0), &source, Some(&first)); - assert_eq!(same.version(), first.field_named("x").expect("x").version(), "nothing changed"); + assert_eq!(same.version(), first.field_named("x").expect("x").version(), "unchanged values keep the version"); let Field::Values(before, _) = first.field_named("x").expect("x") else { panic!("values") }; let Field::Values(after, _) = kept("x", &numbers(2.0), &source, Some(&first)) else { panic!("values") }; - assert_eq!(after.pages()[0].id, before.pages()[0].id, "the first page held the same"); + assert_eq!(after.pages()[0].id, before.pages()[0].id, "the unchanged first page is reused"); assert_ne!(after.pages()[1].id, before.pages()[1].id, "the last page changed"); } } diff --git a/crates/latkit/src/derive/order.rs b/crates/latkit/src/derive/order.rs new file mode 100644 index 00000000..05c83dcc --- /dev/null +++ b/crates/latkit/src/derive/order.rs @@ -0,0 +1,267 @@ +//! Draw orders of paged tables. +//! +//! A mark lays out every per-row input and every per-row GPU word of a paged table in pages of its +//! order. Choosing tiles then chooses ranges of pages, so nothing is gathered as the camera moves. +//! +//! Until its pyramid arrives, a table draws an even sample of its rows first. The endpoint rows of +//! sampled edges come before the sample, as a pyramid would show them with those edges. A pyramid +//! is used while the rows it was made from are unchanged. Rows appended since draw after them, in +//! table order. +//! +//! A table of vector tiles draws in table order, with each tile's rows appended as they arrive. So +//! its tiles are already ranges of its order, and appending moves no row. + +use std::sync::Arc; + +use super::pyramid::Pyramid; +use super::stamp_of; +use crate::data::Version; +use crate::gpu::shaders::PAGE_ROWS; + +/// A paged table's draw order: which row draws at each place. +#[derive(Clone, Debug)] +pub(crate) enum Order { + /// The rows in `lifted` first. Then every `step`th row of the table's `rows` rows (those at + /// multiples of `step`) that is not lifted. Then the remaining rows. Each group is in table + /// order. + Sampled { rows: u32, step: u32, lifted: Lifted }, + /// A pyramid's order of the table's first rows, then the rows appended since, in table order, + /// up to `rows` rows. + Pyramid { pyramid: Arc, rows: u32 }, + /// The table's `rows` rows in table order, while its row order version (`Table::order`) is + /// `order`. Rows appended draw after the rest. Removing rows changes the version, and so the + /// order. + Table { rows: u32, order: Version }, +} + +/// The rows a sample draws before its own: the endpoint rows of the sampled edges. A pyramid +/// likewise shows a vertex at a level at least as coarse as the coarsest edge ending at it. Holds +/// the rows ascending, each row's place in the plain sample (without lifted rows) ascending, and a +/// stamp of the rows. +#[derive(Clone, Debug, Default)] +pub(crate) struct Lifted { + rows: Arc<[u32]>, + places: Arc<[u32]>, + stamp: u64, +} + +impl Lifted { + /// Lifts `rows` of a table of `len` rows into a sample of `sample` rows. The lifted rows count + /// toward the sample. Returns them with the step of the even sample that fills the rest. + pub(crate) fn within(rows: Vec, len: u32, sample: u32) -> (Lifted, u32) { + let rows = ascending(rows, len); + let step = len.div_ceil(sample.saturating_sub(rows.len() as u32).max(1)).max(1); + (Lifted::new(rows, len, step), step) + } + + /// Lifts `rows` of a table of `len` rows into a sample taking every `step`th row. + fn new(rows: Vec, len: u32, step: u32) -> Lifted { + let rows = ascending(rows, len); + let mut places: Vec = rows.iter().map(|&r| sample_rank(r, len, step)).collect(); + places.sort_unstable(); + let stamp = stamp_of(&rows); + Lifted { rows: rows.into(), places: places.into(), stamp } + } + + fn len(&self) -> u32 { + self.rows.len() as u32 + } +} + +/// The `rows` below `len`, sorted ascending without duplicates. +fn ascending(mut rows: Vec, len: u32) -> Vec { + rows.retain(|&r| r < len); + rows.sort_unstable(); + rows.dedup(); + rows +} + +/// The place of row `r` in an even sample of `rows` rows taking every `step`th: the sampled rows +/// first, then the rest, each in table order. +fn sample_rank(r: u32, rows: u32, step: u32) -> u32 { + if r.is_multiple_of(step) { r / step } else { rows.div_ceil(step) + r - r / step - 1 } +} + +/// The row at place `d` in an even sample of `rows` rows taking every `step`th. The inverse of +/// `sample_rank`. +fn sample_source(d: u32, rows: u32, step: u32) -> u32 { + let first = rows.div_ceil(step); + if d < first || step <= 1 { + return d * step; + } + // Past the sample: each step's rows except its first, in table order. + let k = d - first; + (k / (step - 1)) * step + k % (step - 1) + 1 +} + +/// The `e`th place (from 0) missing from `places`, which ascends. A binary search. +fn unlisted(places: &[u32], e: u32) -> u32 { + let (mut lo, mut hi) = (e, e + places.len() as u32); + while lo < hi { + let mid = lo + (hi - lo) / 2; + // How many places up to `mid` are missing from `places`. + let free = mid + 1 - places.partition_point(|&p| p <= mid) as u32; + if free > e { + hi = mid; + } else { + lo = mid + 1; + } + } + lo +} + +impl Order { + /// The number of rows it orders: every row of its table. + pub(crate) fn len(&self) -> u32 { + match self { + Order::Sampled { rows, .. } | Order::Pyramid { rows, .. } | Order::Table { rows, .. } => *rows, + } + } + + /// The number of pages it spans. + pub(crate) fn pages(&self) -> u32 { + self.len().div_ceil(PAGE_ROWS) + } + + /// How many rows a sample draws first: the lifted rows plus the sample's own. 0 for the other + /// orders. + pub(crate) fn sampled(&self) -> u32 { + match self { + Order::Sampled { rows, step, lifted } => { + let first = rows.div_ceil(*step); + lifted.len() + first - lifted.places.partition_point(|&p| p < first) as u32 + } + Order::Pyramid { .. } | Order::Table { .. } => 0, + } + } + + /// The row drawn at place `d`. + pub(crate) fn source(&self, d: u32) -> u32 { + match self { + Order::Sampled { rows, step, lifted } => match lifted.rows.get(d as usize) { + Some(&r) => r, + None => sample_source(unlisted(&lifted.places, d - lifted.len()), *rows, *step), + }, + Order::Pyramid { pyramid, .. } => pyramid.order.get(d as usize).copied().unwrap_or(d), + Order::Table { .. } => d, + } + } + + /// The place where row `r` draws. The inverse of `source`. + pub(crate) fn rank(&self, r: u32) -> u32 { + match self { + Order::Sampled { rows, step, lifted } => { + if let Ok(i) = lifted.rows.binary_search(&r) { + return i as u32; + } + let place = sample_rank(r, *rows, *step); + lifted.len() + place - lifted.places.partition_point(|&p| p < place) as u32 + } + Order::Pyramid { pyramid, .. } => pyramid.rank.get(r as usize).copied().unwrap_or(r), + Order::Table { .. } => r, + } + } + + /// The rows on page `page`, in order. + pub(crate) fn page(&self, page: u32) -> Vec { + let (start, end) = (page * PAGE_ROWS, ((page + 1) * PAGE_ROWS).min(self.len())); + let Order::Sampled { rows, step, lifted } = self else { return (start..end).map(|d| self.source(d)).collect() }; + // The page's lifted rows, then the sample's rows in turn, skipping lifted places. + let m = lifted.len(); + let mut out: Vec = + lifted.rows.get(start.min(m) as usize..end.min(m) as usize).unwrap_or_default().to_vec(); + if end > m { + let places = &lifted.places; + let mut place = unlisted(places, start.max(m) - m); + let mut next = places.partition_point(|&p| p <= place); + for _ in start.max(m)..end { + out.push(sample_source(place, *rows, *step)); + place += 1; + while places.get(next) == Some(&place) { + (place, next) = (place + 1, next + 1); + } + } + } + out + } + + /// The key that identifies pages packed in this order. The ids of a pyramid and of table order + /// ignore rows appended since. A sample's id depends on its row count, step, and lifted rows. + pub(crate) fn id(&self) -> u64 { + match self { + Order::Sampled { rows, step, lifted } => stamp_of(("sampled", rows, step, lifted.stamp)), + Order::Pyramid { pyramid, .. } => stamp_of(pyramid.made), + Order::Table { order, .. } => stamp_of(("table", order)), + } + } +} + +impl PartialEq for Order { + fn eq(&self, other: &Order) -> bool { + self.id() == other.id() && self.len() == other.len() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn sample(rows: u32, step: u32) -> Order { + Order::Sampled { rows, step, lifted: Lifted::default() } + } + + #[test] + fn samples_draw_every_step_first_then_the_rest() { + let order = sample(10, 3); + let drawn: Vec = (0..10).map(|d| order.source(d)).collect(); + assert_eq!(drawn, [0, 3, 6, 9, 1, 2, 4, 5, 7, 8]); + for (d, &r) in drawn.iter().enumerate() { + assert_eq!(order.rank(r), d as u32, "row {r}'s rank is its place"); + } + assert_eq!(order.sampled(), 4); + assert_eq!(sample(4, 1).page(0), [0, 1, 2, 3], "a step of one keeps table order"); + } + + #[test] + fn lifted_rows_draw_before_the_sample_and_only_once() { + // Rows 4 and 6 lifted: 6 was a sample's row, 4 was not. + let order = Order::Sampled { rows: 10, step: 3, lifted: Lifted::new(vec![6, 4, 6, 12], 10, 3) }; + let drawn: Vec = (0..10).map(|d| order.source(d)).collect(); + assert_eq!(drawn, [4, 6, 0, 3, 9, 1, 2, 5, 7, 8]); + for (d, &r) in drawn.iter().enumerate() { + assert_eq!(order.rank(r), d as u32, "row {r}'s rank is its place"); + } + assert_eq!(order.sampled(), 5, "2 lifted rows plus 3 sampled rows"); + assert_ne!(order.id(), sample(10, 3).id(), "lifted rows change the order's id"); + // Each page lists the same rows as `source` gives place by place. + let rows = 3 * PAGE_ROWS + 17; + let lifted = Lifted::new((0..rows).filter(|r| r % 5 == 1 || r % 7 == 0).collect(), rows, 4); + let order = Order::Sampled { rows, step: 4, lifted }; + for page in 0..order.pages() { + let one: Vec = + (page * PAGE_ROWS..((page + 1) * PAGE_ROWS).min(rows)).map(|d| order.source(d)).collect(); + assert_eq!(order.page(page), one, "page {page}"); + } + } + + #[test] + fn rows_appended_after_a_pyramid_draw_after_it_in_table_order() { + let pyramid = Arc::new(Pyramid::new(vec![2, 0, 1], Vec::new())); + let order = Order::Pyramid { pyramid, rows: 5 }; + assert_eq!(order.page(0), [2, 0, 1, 3, 4]); + assert_eq!((order.rank(2), order.rank(4)), (0, 4)); + } + + #[test] + fn table_order_keeps_its_id_while_rows_are_only_appended() { + let version = Version::next(); + let order = Order::Table { rows: 3, order: version }; + assert_eq!(order.page(0), [0, 1, 2]); + assert_eq!((order.source(2), order.rank(1), order.sampled()), (2, 1, 0)); + let grown = Order::Table { rows: PAGE_ROWS + 5, order: version }; + assert_eq!(grown.id(), order.id(), "pages packed before rows were appended stay valid"); + assert_eq!(grown.page(1), (PAGE_ROWS..PAGE_ROWS + 5).collect::>()); + let removed = Order::Table { rows: 2, order: Version::next() }; + assert_ne!(removed.id(), order.id(), "removing rows makes another order"); + } +} diff --git a/crates/latkit/src/derive/places.rs b/crates/latkit/src/derive/places.rs index 156e0688..a9db01d5 100644 --- a/crates/latkit/src/derive/places.rs +++ b/crates/latkit/src/derive/places.rs @@ -1,41 +1,44 @@ -//! Places outside diagrams: where a vertex row without a position stands on a plane or the earth, -//! by the scene's layout. Rows whose data positions them pin the layout, and links are the edges -//! between them, so a row without coordinates stands among its neighbors; gaps are in the space's -//! units. As on a diagram's sheet, a place persists by row, layouts place only rows without one -//! until arranging lays every row without a position out again, and the layout is a host job. The -//! places are a table beside each vertex table: its rows' positions, given or laid out, which the -//! vertex marks read. +//! Places of vertex rows without positions, outside diagrams. +//! +//! On a plane or the earth, the scene's layout places the vertex rows that have no position. Rows +//! with positions are pinned, and edges are the layout's links, so an unpositioned row ends up +//! among its neighbors. Gaps are in the space's units. As on a diagram's sheet, places persist by +//! row. The layout (a host job) places only rows without a place, until `arrange` lays out every +//! unpositioned row again. The result is a table beside each vertex table that holds its rows' +//! positions, given or laid out. The vertex marks read it. + +use std::hash::Hash; use rustc_hash::FxHashMap; use super::graph::{key_of, links, marks, position}; -use super::{Derive, Input, Jobs, Tables, answers, beside, kept}; +use super::{Derive, Input, Jobs, Tables, Working, answers, beside, kept, stamp_of, stamp_with}; +use crate::Layout; use crate::data::{Name, Row, Value, Values}; use crate::expr::cpu::Rows; use crate::host::{Job, Key, Placement}; -use crate::{Layout, Pending}; #[derive(Default)] pub(crate) struct Places { tables: Tables, - /// Where layouts placed rows without positions, kept by row. + /// The places layouts gave rows without positions, by row. laid: FxHashMap, - /// What the tables were last made from. + /// The stamp the tables were last made from. stamp: u64, - /// Bumps when places start over, so layouts asked before are dropped. + /// Bumps when places start over, so answers to earlier layouts are dropped. generation: u64, /// Whether every row without a position is to be laid out again. arranging: bool, - /// What the last arranging laid out, until the view takes it. + /// The result of the last arrange, until the view takes it. arranged: Option>, - placing: Option, - /// Bumps whenever an answer lands. + /// The layout job, for the stamp of the inputs it lays out. + placing: Working, + /// Bumps whenever an answer arrives. landed: u64, } -/// A layout under way: the generation it was asked in, and the row each key it answers names. +/// A layout job's context: the generation it was started in, and the row each answer key names. struct Placing { - pending: Pending>, generation: u64, rows: FxHashMap, } @@ -52,33 +55,35 @@ impl Derive for Places { return had; } let mut changed = false; - // A job its host does at once is taken before the frame draws. + // Repeat so a job the host finishes at once is taken before the frame draws. for _ in 0..4 { self.settle(); - let mut hasher = rustc_hash::FxHasher::default(); - std::hash::Hash::hash(&(input.scene_version, self.landed, self.generation), &mut hasher); - input.structure(&mut hasher); - let stamp = std::hash::Hasher::finish(&hasher).max(1); + let inputs = stamp_with(|hasher| { + (input.scene_version, self.generation).hash(hasher); + input.structure(hasher); + }); + // An answer changes the tables, but not what a layout would be asked to place. + let stamp = stamp_of((inputs, self.landed)); if stamp == self.stamp { break; } self.stamp = stamp; - self.build(input, jobs); + self.build(input, inputs, jobs); changed = true; } changed } fn busy(&self) -> bool { - self.placing.is_some() + self.placing.busy() } fn tables(&self) -> &Tables { &self.tables } - /// Forgets the tables but keeps the places: a scene with a layout again finds its rows where they - /// stood. + /// Drops the tables but keeps the places, so when the scene has a layout again, rows return to + /// where they were. fn clear(&mut self) { self.tables.clear(); self.stamp = 0; @@ -86,29 +91,28 @@ impl Derive for Places { } impl Places { - /// Lays every row without a position out again. + /// Lays out every row without a position again, dropping any pending layout. pub(crate) fn arrange(&mut self) { self.arranging = true; - self.placing = None; + self.placing.cancel(); self.generation += 1; self.stamp = 0; } - /// What the last arranging laid out, once it landed: each row placed, and where. + /// Takes the result of the last `arrange` once it arrives: each row laid out and its position. pub(crate) fn take_arranged(&mut self) -> Option> { self.arranged.take() } fn settle(&mut self) { - let Some(answer) = self.placing.as_ref().and_then(|p| p.pending.take()) else { return }; - let placing = self.placing.take().expect("a layout under way"); + let Some((_, placing, answer)) = self.placing.landed() else { return }; if placing.generation == self.generation { let mut placed = Vec::new(); for (key, values) in answer.map(|bytes| answers(&bytes, &["x", "y"])).unwrap_or_default() { if let (Some(row), [Some(Value::Number(x)), Some(Value::Number(y))]) = (placing.rows.get(&key), &values[..]) { - // The plane and the earth run y up, as layouts do. + // The plane and the earth are y up, like layouts. self.laid.insert(row.clone(), [*x, *y]); placed.push((row.clone(), [*x, *y])); } @@ -120,8 +124,9 @@ impl Places { self.landed += 1; } - /// Writes each vertex table's positions, asking the layout for the rows without one. - fn build(&mut self, input: &Input<'_>, jobs: &Jobs<'_>) { + /// Writes each vertex table's positions, starting a layout job for rows without one unless a + /// job for the same `inputs` already ran. + fn build(&mut self, input: &Input<'_>, inputs: u64, jobs: &Jobs<'_>) { let (vertex_marks, edge_marks) = marks(input); let mut keys = Vec::new(); let mut pins = Vec::new(); @@ -135,7 +140,7 @@ impl Places { } let arranging = self.arranging; let wanted = |i: usize| pins[i].is_none() && (arranging || !self.laid.contains_key(&keys[i])); - if self.placing.is_none() && (0..keys.len()).any(wanted) { + if self.placing.may_start(inputs) && (0..keys.len()).any(wanted) { let pinned: Vec> = (0..keys.len()) .map(|i| pins[i].or_else(|| (!arranging).then(|| self.laid.get(&keys[i]).copied()).flatten())) .collect(); @@ -158,9 +163,9 @@ impl Places { keys: &job_keys, }; let pending = jobs.spawn(Job::layout(&input.scene.layout, &placement)); - self.placing = Some(Placing { pending, generation: self.generation, rows }); - } else if self.arranging && self.placing.is_none() { - // Nothing to lay out: the arranging is done as asked. + self.placing.start(inputs, pending, Placing { generation: self.generation, rows }); + } else if self.arranging && !self.placing.busy() { + // Nothing to lay out: the arrange finishes with no rows. self.arranging = false; self.arranged = Some(Vec::new()); } @@ -180,3 +185,43 @@ impl Places { self.tables = tables; } } + +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + use std::task::Waker; + + use super::*; + use crate::derive::testing::Failing; + use crate::gpu::work::Counters; + use crate::{Data, Ids, Plane, Scene, Stress, Table, Vertices}; + + #[test] + fn a_failed_layout_runs_once_until_its_inputs_change() { + let data = Data::new("g"); + data.apply(Table::new("V", Ids::count(3))).expect("valid"); + let data = data.snapshot(); + let scene = Scene::new(Plane::default()).layout(Stress::default()).draw(Vertices::of("g", "V")); + let (host, counters, errors) = (Failing::default(), Counters::default(), Arc::new(Mutex::new(Vec::new()))); + let mut places = Places::default(); + let frame = |places: &mut Places| { + let input = Input { + scene: &scene, + scene_version: 1, + data: std::slice::from_ref(&data), + host: Some(&host), + dragging: false, + paged: None, + }; + places.update(&input, &mut Jobs::new(Some(&host), Waker::noop(), &counters, &errors, u32::MAX)); + }; + for _ in 0..5 { + frame(&mut places); + } + assert_eq!(host.jobs(), 1, "one job for these inputs, not one per frame"); + assert_eq!(errors.lock().expect("unpoisoned").len(), 1, "its failure is reported once"); + places.arrange(); + frame(&mut places); + assert_eq!(host.jobs(), 2, "laying out again asks again"); + } +} diff --git a/crates/latkit/src/derive/pyramid.rs b/crates/latkit/src/derive/pyramid.rs index 34b20b7c..2ee001bc 100644 --- a/crates/latkit/src/derive/pyramid.rs +++ b/crates/latkit/src/derive/pyramid.rs @@ -1,42 +1,57 @@ -//! Pyramids: a table's rows grouped into tiles by the coarsest level each shows at, worked out in a -//! job. Each level splits the tiles above it four ways, and each tile holds a grid of cells, a row to -//! a cell, so no tile holds more rows than its grid has cells. Rows land by priority: each at the -//! coarsest level where its cell is free, so what matters more shows from farther out, and rows -//! crowding one cell wait for a finer level, where they spread over cells of their own. A row never -//! lands finer than its size allows, so its tile grown by half its own size holds it whole. An edge -//! lands by its own priority, in the cell of its middle, and edges joining the same two vertices land -//! together; what draws an edge draws its ends with it. A table drawn whole by priority is one tile, -//! its rows in priority order. +//! Pyramids: a table's rows grouped into quadtree tiles by the coarsest level at which each shows. +//! +//! A dataset's pyramids are computed together, in one job unless cheap enough for the frame. +//! Each level splits the tiles above it four ways, and each tile has a grid of cells, one row per +//! cell. Rows are assigned by priority, each to the coarsest level where its cell is still free, +//! so more important rows show from farther out. Rows crowding one cell wait for a finer level, +//! where they spread over separate cells. A row is never assigned finer than its size allows: its +//! tile, grown by half a tile on each side, contains it. +//! +//! An edge is assigned by its own priority, in the cell of its midpoint, and edges joining the +//! same two vertices are assigned together. A vertex is assigned no finer than the coarsest edge +//! ending at it, so the endpoints of every drawn edge show from as far out as the edge. A table +//! drawn whole by priority is one tile, its rows in priority order. +//! +//! A pyramid is an order: rows by tile, coarsest level first, each tile's rows contiguous, plus +//! each row's place in it. The camera chooses tiles, each a range of the order. use std::ops::Range; use std::sync::Arc; use rustc_hash::{FxHashMap, FxHashSet}; -use super::sight::{Domain, Region, Sight, TILE_PX, meets}; -use crate::data::{Field, Ids, Op, Patch, Table}; +use super::sight::{Domain, Region, SPLIT_PX, Sight, meets}; +use crate::data::{UNWIRED, Version}; +use crate::gpu::shaders::PAGE_ROWS; use crate::tiles::TileKey; -/// Cells across a tile's grid, as a shift: a tile holds at most `4^GRID_BITS` rows of its level. +/// Log2 of the cells across a tile's grid. A tile holds at most `4^GRID_BITS` rows of its level, +/// except rows forced to that level, such as the endpoints of its edges. pub(crate) const GRID_BITS: u32 = 7; /// The finest level. pub(crate) const MAX_LEVEL: u8 = 24; -/// A level no row has landed at yet. +/// Marks a row not yet assigned a level. const NEVER: u8 = u8::MAX; -/// Tiles a camera may ask a pyramid for at most: past it, tiles stop splitting. +/// The most tiles a camera request splits into per level. Past it, the tiles drawn largest split +/// and the rest stop splitting. const MAX_TILES: usize = 1024; +/// Log2 of the sub-cells across a large tile. Each sub-cell is a range of the order that the camera +/// can request separately, so only the visible part of a large tile draws. +const SUB_BITS: u32 = 3; +/// Tiles with more rows than this are split into sub-cells. +const SUB_ROWS: usize = 256; -/// What a pyramid job places: tables of rows in a square of their space. +/// A pyramid job's input: tables of rows in one square of their space. #[derive(Clone, Debug)] pub(crate) struct Leveling { pub(crate) domain: Domain, - /// The tables, every table an edge table's ends name before it. + /// The tables. A table that an edge table's endpoints reference comes before that edge table. pub(crate) tables: Vec, } -/// Where a table's rows lie, in position units: none where nothing places them, a point each, or the -/// box a geometry spans (west, south, east, north). NaN places nothing. Single precision places a -/// row within a centimeter on the earth, finer than any tile's cell. +/// Where a table's rows lie, in position units: nowhere, a point per row, or per row the box its +/// geometry spans (west, south, east, north). NaN marks an unplaced row. `f32` keeps positions on +/// the earth to within about a meter at worst (near ±180°), coarser than the finest levels' cells. #[derive(Clone, Debug, Default)] pub(crate) enum Places { #[default] @@ -49,19 +64,23 @@ pub(crate) enum Places { #[derive(Clone, Debug, Default)] pub(crate) struct Placing { pub(crate) rows: u32, - /// Whether its rows land in tiles by level; else every row is one tile, in priority order. + /// Whether its rows are assigned to tiles by level. If not, they form one tile in priority + /// order. pub(crate) leveled: bool, - /// Whether the answer holds its pyramid: a table placed only for the edges ending at it is not. + /// Whether the answer includes its pyramid. A table placed only to anchor the edges ending at + /// it is not answered. pub(crate) answered: bool, pub(crate) places: Places, - /// Each row's priority, higher first; none where rows are alike. + /// Each row's priority, higher first. Empty when all rows are equal. pub(crate) priority: Arc<[f32]>, - /// For a table placed by its ends, the table they name and each end's rows there. + /// For an edge table placed by its endpoints: the index of the table they reference, and each + /// endpoint reference's target rows. pub(crate) ends: Option<(u32, Vec>)>, } -/// Where each row of `places` lies across `domain`'s square, fixed point (a turn of `u32` spans it, -/// from the west and from the north), and half its size alike; nothing where it lies nowhere. +/// Each row's anchor in `domain`'s square in `u32` fixed point (the full `u32` range spans the +/// square, from the west and from the north), and for boxes, half the box's larger side in the same +/// units. Unplaced rows anchor at the origin with no extent. `Places::None` gives nothing. fn anchored(places: &Places, domain: &Domain) -> (Vec<[u32; 2]>, Vec) { let fixed = |u: f64| (u.clamp(0.0, 1.0) * 4_294_967_295.0) as u32; match places { @@ -89,18 +108,26 @@ fn anchored(places: &Places, domain: &Domain) -> (Vec<[u32; 2]>, Vec) { } } -/// The pyramid of each table a job answered for, by its place among the tables asked, with where -/// each row lies across the square and how far it reaches. -pub(crate) fn answer(leveling: &Leveling) -> Patch { +/// One table's rows in the square: anchors, extents (half sizes), the coarsest level each may be +/// assigned (0, or `NEVER` to wait for the finest), and for an edge table each row's endpoint pair. +struct Anchored { + anchors: Vec<[u32; 2]>, + extents: Vec, + floor: Vec, + pairs: Option>, +} + +/// Computes the pyramid of each answered table, with the table's index in `leveling`. +pub(crate) fn answer(leveling: &Leveling) -> Vec<(u32, Pyramid)> { let wraps = leveling.domain == Domain::Earth; - let mut anchors: Vec> = Vec::with_capacity(leveling.tables.len()); - let mut patch = Patch::new(); - for (i, table) in leveling.tables.iter().enumerate() { + let tables = &leveling.tables; + let mut placed: Vec = Vec::with_capacity(tables.len()); + for table in tables { let n = table.rows as usize; - let (own_anchors, extents, floor, pairs) = if let Some((target, ends)) = &table.ends { - let Some(at) = anchors.get(*target as usize) else { return Patch::new() }; - let (own, extents) = by_ends(ends, at, wraps, n); - // A row with an end naming no row draws nothing: it lands only at the finest level. + let anchoring = if let Some((target, ends)) = &table.ends { + let Some(at) = placed.get(*target as usize).map(|a| &a.anchors) else { return Vec::new() }; + let (anchors, extents) = by_ends(ends, at, wraps, n); + // A row with an unset endpoint draws nothing, so it goes only to the finest level. let floor = (0..n) .map(|r| { let wired = ends.iter().all(|end| end.get(r).is_some_and(|&t| (t as usize) < at.len())); @@ -108,49 +135,69 @@ pub(crate) fn answer(leveling: &Leveling) -> Patch { }) .collect(); let pairs = (ends.len() == 2).then(|| pairs(&ends[0], &ends[1])); - (own, extents, floor, pairs) + Anchored { anchors, extents, floor, pairs } } else { - let (own, extents) = anchored(&table.places, &leveling.domain); - (own, extents, vec![0; n], None) + let (anchors, extents) = anchored(&table.places, &leveling.domain); + Anchored { anchors, extents, floor: vec![0; n], pairs: None } }; - let leveled = table.leveled && own_anchors.len() == n; - let sorted = if leveled { along(&own_anchors) } else { Vec::new() }; - let ranks = ranks(&table.priority, n); - let level = if leveled { place(&sorted, &extents, &ranks, &floor, pairs.as_deref()) } else { vec![0; n] }; - if table.answered { - let (order, tiles) = if leveled { - tiled(&sorted, &own_anchors, &extents, &ranks, &level) - } else { - let mut order: Vec = (0..table.rows).collect(); - crate::order::by_key(&mut order, |&r| ranks[r as usize]); - (order, vec![(pack(TileKey { z: 0, x: 0, y: 0 }), table.rows, u32::MAX)]) - }; - let (keys, (ends, reaches)): (Vec, (Vec, Vec)) = - tiles.into_iter().map(|(k, e, r)| (k, (e, r))).unzip(); - patch = patch - .set( - Table::new(format!("tiles{i}"), Ids::count(keys.len() as u32)) - .field("key", keys) - .field("end", ends) - .field("reach", reaches), - ) - .set(Table::new(format!("order{i}"), Ids::count(order.len() as u32)).field("row", order)); - if leveled { - let (u, v): (Vec, Vec) = own_anchors.iter().map(|a| (a[0], a[1])).unzip(); - let mut placed = Table::new(format!("anchors{i}"), Ids::count(n as u32)).field("u", u).field("v", v); - if !extents.is_empty() { - placed = placed.field("extent", extents.clone()); + placed.push(anchoring); + } + let leveled = |i: usize| tables[i].leveled && placed[i].anchors.len() == tables[i].rows as usize; + let mut sorted: Vec> = Vec::with_capacity(tables.len()); + let mut ranked: Vec> = Vec::with_capacity(tables.len()); + for (i, table) in tables.iter().enumerate() { + sorted.push(if leveled(i) { along(&placed[i].anchors) } else { Vec::new() }); + ranked.push(ranks(&table.priority, table.rows as usize)); + } + let level_of = |i: usize, ceiling: &[u8]| -> Vec { + if !leveled(i) { + return vec![0; tables[i].rows as usize]; + } + let a = &placed[i]; + place(&sorted[i], &a.extents, &ranked[i], &a.floor, a.pairs.as_deref(), ceiling) + }; + // Level edges first: a vertex is assigned no finer than the coarsest edge ending at it. + let mut levels: Vec>> = vec![None; tables.len()]; + let mut ceilings: Vec> = vec![Vec::new(); tables.len()]; + for (i, table) in tables.iter().enumerate() { + let Some((target, ends)) = &table.ends else { continue }; + let level = level_of(i, &[]); + let target = *target as usize; + if leveled(i) && leveled(target) { + let ceiling = &mut ceilings[target]; + ceiling.resize(tables[target].rows as usize, MAX_LEVEL); + for end in ends { + for (r, &t) in end.iter().enumerate() { + if let Some(c) = ceiling.get_mut(t as usize).filter(|_| t != UNWIRED) { + *c = (*c).min(level[r]); + } } - patch = patch.set(placed); } } - anchors.push(own_anchors); + levels[i] = Some(level); } - patch + let mut out = Vec::new(); + for (i, table) in tables.iter().enumerate() { + let level = levels[i].take().unwrap_or_else(|| level_of(i, &ceilings[i])); + if !table.answered { + continue; + } + let (order, tiles) = if leveled(i) { + tiled(&sorted[i], &placed[i].anchors, &placed[i].extents, &ranked[i], &level) + } else { + let mut order: Vec = (0..table.rows).collect(); + crate::order::by_key(&mut order, |&r| ranked[i][r as usize]); + let whole = Tile { key: TileKey { z: 0, x: 0, y: 0 }, rows: 0..table.rows, reach: 1.0, cells: Vec::new() }; + (order, vec![whole]) + }; + out.push((i as u32, Pyramid::new(order, tiles))); + } + out } -/// Where each of `n` rows placed by `ends` lies, among rows anchored at `at`: the middle of the box -/// its ends span, and half that box's size. An end naming no row anchors nothing. +/// Anchors and extents for `n` rows placed by `ends`, among target rows anchored at `at`: the +/// middle of the box the endpoints span, and half its larger side. A row with an unset endpoint +/// gets the origin and no extent. fn by_ends(ends: &[Arc<[u32]>], at: &[[u32; 2]], wraps: bool, n: usize) -> (Vec<[u32; 2]>, Vec) { let mut anchors = Vec::with_capacity(n); let mut extents = Vec::with_capacity(n); @@ -161,8 +208,8 @@ fn by_ends(ends: &[Arc<[u32]>], at: &[[u32; 2]], wraps: bool, n: usize) -> (Vec< extents.push(0); continue; } - // The box of offsets from the end of the lowest row, the short way round where the square - // wraps: the same whichever way the row joins its ends. + // The box of offsets from the endpoint with the lowest row index, going the short way + // around where the square wraps, so the result does not depend on the edge's direction. let first = at[ends.iter().map(|end| end[r]).min().unwrap_or(0) as usize]; let (mut lo, mut hi) = ([0_i64; 2], [0_i64; 2]); for end in ends { @@ -189,8 +236,9 @@ fn by_ends(ends: &[Arc<[u32]>], at: &[[u32; 2]], wraps: bool, n: usize) -> (Vec< (anchors, extents) } -/// For each row, the next row joining the same two vertices, round to itself: a cycle per pair. Such -/// rows share their anchor, so they lie side by side along the curve. +/// For each row, the next row joining the same two vertices, so each pair forms one cycle. A row +/// with no parallel rows maps to itself. Parallel rows share an anchor, so they are adjacent in +/// curve order. fn mates(sorted: &[(u64, u32)], pairs: &[u64]) -> Vec { let mut mates: Vec = (0..sorted.len() as u32).collect(); let mut i = 0; @@ -217,7 +265,7 @@ fn mates(sorted: &[(u64, u32)], pairs: &[u64]) -> Vec { mates } -/// Each row's pair of ends, the same whichever way it joins them. +/// Each row's endpoint pair as one key, independent of the edge's direction. fn pairs(from: &[u32], to: &[u32]) -> Vec { from.iter().zip(to).map(|(&a, &b)| u64::from(a.min(b)) << 32 | u64::from(a.max(b))).collect() } @@ -227,34 +275,31 @@ fn tie(row: u32) -> u32 { (u64::from(row).wrapping_mul(0x9E37_79B9_7F4A_7C15) >> 32) as u32 } -/// Each of `n` rows' rank, higher first: its priority, missing lowest, then the hash of its place, -/// so equals spread evenly. +/// The rank of each of `n` rows, higher first: its priority (missing or NaN lowest), then a hash +/// of its row index, so equal priorities spread evenly. fn ranks(priority: &[f32], n: usize) -> Vec { (0..n) .map(|r| { - let p = priority.get(r).copied().filter(|p| !p.is_nan()).map_or(0, |p| { - let bits = p.to_bits(); - if bits >> 31 == 1 { !bits } else { bits | 0x8000_0000 } - }); - u64::from(p) << 32 | u64::from(tie(r as u32)) + let p = priority.get(r).copied().filter(|p| !p.is_nan()).map_or(0, crate::order::number32); + crate::order::pair(p, tie(r as u32)) }) .collect() } -/// Rows along the curve, by their place on it. +/// The rows sorted along the Morton curve, each with its curve key. fn along(anchors: &[[u32; 2]]) -> Vec<(u64, u32)> { let mut sorted: Vec<(u64, u32)> = anchors.iter().enumerate().map(|(r, &a)| (morton(a), r as u32)).collect(); sorted.sort_unstable(); sorted } -/// The finest level whose tile, grown by half its size, holds a row reaching `extent` from its -/// anchor. +/// The finest level whose tiles, grown by half a tile on each side, contain a row reaching `extent` +/// from its anchor. fn fit(extent: u32) -> u8 { (0..=MAX_LEVEL).rev().find(|&z| u64::from(extent) <= 1_u64 << (31 - u32::from(z))).unwrap_or(0) } -/// `x`'s bits spread to every other bit. +/// `x`'s bits spread to the even bit positions. fn spread_bits(x: u32) -> u64 { let mut x = u64::from(x); x = (x | (x << 16)) & 0x0000_FFFF_0000_FFFF; @@ -264,34 +309,20 @@ fn spread_bits(x: u32) -> u64 { (x | (x << 1)) & 0x5555_5555_5555_5555 } -/// The point's place along the curve that visits every cell of a level before the next cell of the -/// level above: cells at every level are runs of it. +/// The point's position on the Morton (Z-order) curve. The curve visits all of a cell's sub-cells +/// before leaving it, so the cells of every level are contiguous runs of it. fn morton([u, v]: [u32; 2]) -> u64 { spread_bits(u) | spread_bits(v) << 1 } -/// The tile at level `z` holding a point anchored at `anchor`. +/// The tile at level `z` containing anchor `anchor`. fn tile_of(anchor: [u32; 2], z: u8) -> TileKey { let shift = 32 - u32::from(z); let at = |c: u32| if shift >= 32 { 0 } else { c >> shift }; TileKey { z, x: at(anchor[0]), y: at(anchor[1]) } } -/// `cover` grown to hold tile `key`: a box from the west and the north, fixed point. -fn covering(cover: [u64; 4], key: TileKey) -> [u64; 4] { - let shift = 32 - u32::from(key.z); - let (x, y) = (u64::from(key.x) << shift, u64::from(key.y) << shift); - let size = 1_u64 << shift; - [cover[0].min(x), cover[1].min(y), cover[2].max(x + size), cover[3].max(y + size)] -} - -/// Whether a row anchored at `anchor`, reaching `extent` from it, reaches into `cover`. -pub(crate) fn reaches(cover: &[u64; 4], anchor: [u32; 2], extent: u32) -> bool { - let (u, v, e) = (u64::from(anchor[0]), u64::from(anchor[1]), u64::from(extent)); - u + e >= cover[0] && u.saturating_sub(e) < cover[2] && v + e >= cover[1] && v.saturating_sub(e) < cover[3] -} - -/// A tile as a job's answer numbers it: in 53 bits, so a number holds it exactly. +/// A tile key packed into 53 bits as a job's answer stores it, so an `f64` holds it exactly. pub(crate) fn pack(key: TileKey) -> u64 { u64::from(key.z) << 48 | u64::from(key.x) << 24 | u64::from(key.y) } @@ -300,24 +331,40 @@ fn unpack(key: u64) -> TileKey { TileKey { z: (key >> 48) as u8, x: ((key >> 24) & 0xFF_FFFF) as u32, y: (key & 0xFF_FFFF) as u32 } } -/// The level each row lands at: rows in each cell of each level, coarsest first, give way by rank; a -/// row lands at the first level where it outranks every row left in its cell, no coarser than -/// `floor`, at the finest its size allows when it has not landed by then. +/// The level each row is assigned. At each level, coarsest first, each cell takes the +/// highest-ranked row left in it, so a row is assigned at the first level where it outranks every +/// other row left in its cell. Rows with a nonzero `floor` are never chosen by rank. A row is never +/// assigned finer than its size or `ceiling` allows. If no cell took it by then, it is forced in at +/// that level. Rows `pairs` marks as parallel are assigned together. Rows left over go to the +/// finest level. /// -/// Rows along the curve are taken in runs, each the rows left in one cell. While a run stays in one -/// cell level after level, those levels take its highest rows in rank order at once; where it -/// spreads over several cells, each cell's rows go on as a run of their own. So a crowd sharing a -/// cell for many levels costs its size once, not once a level. -fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pairs: Option<&[u64]>) -> Vec { +/// Rows are processed in runs along the curve. Each run is the rows left in one cell. While a run +/// stays within one cell for several levels, those levels take its top rows in rank order in one +/// pass, using a heap. Where a run spreads over several cells, each cell's rows continue as a run +/// of their own. So a crowd in one cell for many levels costs its size once, not once per level. +fn place( + sorted: &[(u64, u32)], + extents: &[u32], + ranks: &[u64], + floor: &[u8], + pairs: Option<&[u64]>, + ceiling: &[u8], +) -> Vec { let n = sorted.len(); - let fits: Vec = (0..n).map(|r| extents.get(r).map_or(MAX_LEVEL, |&e| fit(e))).collect(); + let fits: Vec = (0..n) + .map(|r| { + let own = extents.get(r).map_or(MAX_LEVEL, |&e| fit(e)); + own.min(ceiling.get(r).copied().unwrap_or(MAX_LEVEL)) + }) + .collect(); + let forcing = !extents.is_empty() || !ceiling.is_empty(); let mates = pairs.map(|pairs| mates(sorted, pairs)).unwrap_or_default(); let mut level = vec![NEVER; n]; let mut work: Vec<(u64, u32)> = sorted.to_vec(); let mut runs: Vec<(usize, usize, u8)> = vec![(0, n, 0)]; let mut scratch: Vec<(u64, u32)> = Vec::new(); let mut forced: Vec = Vec::new(); - // Lands row `r` at level `l`, with the rows joining the same two vertices. + // Assigns row `r` level `l`, along with the rows joining the same two vertices. let land = |r: u32, l: u8, level: &mut [u8]| { level[r as usize] = l; let mut m = mates.get(r as usize).copied().unwrap_or(r); @@ -332,10 +379,10 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa while a < b && z <= MAX_LEVEL { let (first, last) = (work[a].0, work[b - 1].0); let common = if first == last { 64 } else { (first ^ last).leading_zeros() }; - // The finest level whose cells hold the whole run. + // The finest level at which one cell holds the whole run. let single = i64::from(common / 2) - i64::from(GRID_BITS); if single < i64::from(z) { - // The run spreads over several cells of this level: each goes on alone. + // The run spans several cells at this level: split it into one run per cell. let shift = 2 * (32 - u32::from(z) - GRID_BITS); let mut i = a; while i < b { @@ -348,7 +395,7 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa break; } let deepest = single.min(i64::from(MAX_LEVEL)) as u8; - // One row a level, in rank order: for one level the highest, for more a heap of them. + // One row per level, in rank order: the top row for a single level, else a heap. if deepest == z { let best = work[a..b] .iter() @@ -365,7 +412,7 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa ); let mut heap = std::collections::BinaryHeap::from(std::mem::take(&mut scratch)); for l in z..=deepest { - // A row too large for this level lands where its size says, below. + // Skip rows too large for this level. The forcing step below assigns them. while let Some((_, r)) = heap.pop() { if level[r as usize] == NEVER && fits[r as usize] >= l { land(r, l, &mut level); @@ -375,8 +422,8 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa } scratch = heap.into_vec(); } - // Rows no finer level holds land where their size says. - if !extents.is_empty() { + // Force rows that no finer level may take to the level their size or edges allow. + if forcing { forced.clear(); forced.extend(work[a..b].iter().map(|&(_, r)| r).filter(|&r| fits[r as usize] <= deepest)); for &r in &forced { @@ -385,7 +432,7 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa } } } - // What is left goes on, along the curve, a level finer. + // The remaining rows continue, in curve order, one level finer. let mut kept = a; for i in a..b { if level[work[i].1 as usize] == NEVER { @@ -396,7 +443,7 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa b = kept; z = deepest + 1; } - // Rows no level held, as rows naming no vertex, land at the finest. + // Rows no level took, such as edges with an unset endpoint, go to the finest level. for &(_, r) in &work[a..b] { if level[r as usize] == NEVER { level[r as usize] = MAX_LEVEL; @@ -406,18 +453,18 @@ fn place(sorted: &[(u64, u32)], extents: &[u32], ranks: &[u64], floor: &[u8], pa level } -/// Rows by tile, coarsest level first, each tile's rows in priority order, highest last; and each -/// tile packed, where its rows end, and how far they reach past it, fixed point. `sorted` holds the -/// rows along the curve. +/// Orders rows by tile, coarsest level first, and returns the order with the tiles. Each tile's +/// rows are in priority order, highest last. A large tile's rows are grouped by sub-cell, each +/// sub-cell in priority order. `sorted` holds the rows along the curve. fn tiled( sorted: &[(u64, u32)], anchors: &[[u32; 2]], extents: &[u32], ranks: &[u64], level: &[u8], -) -> (Vec, Vec<(u64, u32, u32)>) { +) -> (Vec, Vec) { let n = anchors.len(); - // Rows by level, along the curve within each, so each tile's rows are a run. + // Bucket rows by level, in curve order within each, so each tile's rows are contiguous. let mut starts = vec![0_usize; usize::from(MAX_LEVEL) + 2]; for &l in level { starts[usize::from(l) + 1] += 1; @@ -446,103 +493,117 @@ fn tiled( j += 1; } let start = order.len(); - order.extend_from_slice(&rows[i..j]); - // By priority alone: rows alike draw in either order. - crate::order::by_key(&mut order[start..], |&r| ranks[r as usize] >> 32); - tiles.push((pack(key), order.len() as u32, reach)); + let members = &rows[i..j]; + // Sort by priority only, without the tie hash. Equal rows keep their curve order. + let by_priority = |order: &mut Vec, from: usize| { + crate::order::by_key(&mut order[from..], |&r| ranks[r as usize] >> 32); + }; + let mut cells = Vec::new(); + if members.len() > SUB_ROWS { + // Along the curve, each sub-cell's rows are contiguous. + let shift = 2 * (32 - u32::from(z) - SUB_BITS); + let mask = (1_u64 << (2 * SUB_BITS)) - 1; + let mut k = 0; + for cell in 0..=mask { + let from = order.len(); + while k < members.len() && (morton(anchors[members[k] as usize]) >> shift) & mask == cell { + order.push(members[k]); + k += 1; + } + by_priority(&mut order, from); + cells.push(order.len() as u32); + } + } else { + order.extend_from_slice(members); + by_priority(&mut order, start); + } + let reach = f64::from(reach) / 4_294_967_296.0; + tiles.push(Tile { key, rows: start as u32..order.len() as u32, reach, cells }); i = j; } } (order, tiles) } -/// One tile of a pyramid: where its rows lie in the pyramid's order, and how far they reach past it, -/// across the square. +/// One tile of a pyramid: its rows' range in the pyramid's order, how far its rows reach past it +/// (as a fraction of the square), and for a large tile, where each sub-cell's rows end, with +/// sub-cells in curve order. #[derive(Clone, Debug, PartialEq)] pub(crate) struct Tile { pub(crate) key: TileKey, pub(crate) rows: Range, pub(crate) reach: f64, + pub(crate) cells: Vec, +} + +/// How many pages of the order `ranges` touch beyond page `last`, and the last page they reach. +/// Levels lie in the order coarsest first, so a level's pages follow those of the levels above, +/// except for one page they may share. +fn pages_past(ranges: &[(u8, Range)], last: Option) -> (u32, Option) { + let mut spans: Vec<(u32, u32)> = ranges + .iter() + .filter(|(_, r)| r.start < r.end) + .map(|(_, r)| (r.start / PAGE_ROWS, (r.end - 1) / PAGE_ROWS)) + .collect(); + spans.sort_unstable(); + let (mut pages, mut reached) = (0, last); + for (first, end) in spans { + let from = reached.map_or(first, |l| first.max(l + 1)); + pages += (end + 1).saturating_sub(from); + reached = Some(reached.map_or(end, |l| l.max(end))); + } + (pages, reached) } -/// The tiles the camera asks of a pyramid, each with whether it is one of the finest asked, and the -/// part of the square, from the west and the north, the finest cover: a coarser tile draws only its -/// rows reaching into it. -#[derive(Clone, Debug, Default, PartialEq, Hash)] -pub(crate) struct Wanted { - pub(crate) tiles: Vec<(u32, bool)>, - pub(crate) cover: Option<[u64; 4]>, +/// The first `rows` rows of `ranges`, in order: the rows drawn of a level whose pages alone exceed +/// the budget. +fn first(ranges: Vec<(u8, Range)>, mut rows: u32) -> Vec<(u8, Range)> { + let mut out = Vec::new(); + for (z, range) in ranges { + if rows == 0 { + break; + } + let n = range.len().min(rows as usize) as u32; + out.push((z, range.start..range.start + n)); + rows -= n; + } + out +} + +/// The tile key of sub-cell `cell` of tile `key`, with sub-cells numbered along the curve. +fn sub_key(key: TileKey, cell: usize) -> TileKey { + let (mut x, mut y) = (0, 0); + for b in 0..SUB_BITS { + x |= ((cell >> (2 * b)) as u32 & 1) << b; + y |= ((cell >> (2 * b + 1)) as u32 & 1) << b; + } + TileKey { z: key.z + SUB_BITS as u8, x: key.x << SUB_BITS | x, y: key.y << SUB_BITS | y } } -/// A table's rows grouped into tiles: a job's answer, read. -#[derive(Debug, Default)] +/// A table's rows grouped into tiles: the order (rows by tile, coarsest level first, each tile's +/// rows in priority order, highest last), each row's place in it, and the tiles. +#[derive(Debug)] pub(crate) struct Pyramid { - /// Rows by tile, coarsest level first, each tile's rows in priority order, highest last. pub(crate) order: Vec, + pub(crate) rank: Vec, pub(crate) tiles: Vec, - /// Where each row lies across the square, fixed point, and half its size alike: none for a - /// table in one tile, and no sizes for points. - pub(crate) anchors: Vec<[u32; 2]>, - pub(crate) extents: Vec, + /// Identifies this pyramid among all others. Pages packed in its order are keyed by it. + pub(crate) made: Version, + /// Each tile's index in `tiles`, by key. index: FxHashMap, - /// Every tile with tiles below it. + /// Every key that has tiles below it. above: FxHashSet, } impl Pyramid { - /// The pyramids a job answered, by the place of the table among those asked. - pub(crate) fn read(bytes: &[u8]) -> FxHashMap { - Patch::from_bytes(bytes).map(Pyramid::answered).unwrap_or_default() - } - - /// The pyramids `patch` answers, by the place of the table among those asked. - pub(crate) fn answered(patch: Patch) -> FxHashMap { - let mut tiles: FxHashMap> = FxHashMap::default(); - let mut orders: FxHashMap> = FxHashMap::default(); - let mut placed: FxHashMap, Vec)> = FxHashMap::default(); - for op in patch.ops { - let Op::Set(table) = op else { continue }; - let number = |prefix: &str| table.name.strip_prefix(prefix).and_then(|i| i.parse::().ok()); - let column = |name: &str| match table.field_named(name) { - Some(Field::Values(column, _)) => Some(column.clone()), - _ => None, - }; - // Words read as they are stored, page by page. - let words = |name: &str| -> Vec { - column(name).map_or_else(Vec::new, |c| { - c.pages().iter().flat_map(|p| p.values.u32s().unwrap_or(&[]).iter().copied()).collect() - }) - }; - if let Some(i) = number("tiles") { - let keys: Vec = column("key").map_or_else(Vec::new, |c| { - c.pages().iter().flat_map(|p| p.values.u64s().unwrap_or(&[]).iter().copied()).collect() - }); - let (ends, reaches) = (words("end"), words("reach")); - let mut start = 0; - let read = keys.iter().zip(&ends).zip(&reaches).map(|((&key, &end), &reach)| { - let rows = start..end; - start = end; - Tile { key: unpack(key), rows, reach: f64::from(reach) / 4_294_967_296.0 } - }); - tiles.insert(i, read.collect()); - } else if let Some(i) = number("order") { - orders.insert(i, words("row")); - } else if let Some(i) = number("anchors") { - let anchors = words("u").into_iter().zip(words("v")).map(|(u, v)| [u, v]).collect(); - placed.insert(i, (anchors, words("extent"))); + /// A pyramid of `order` and `tiles`, computing each row's place in the order. + pub(crate) fn new(order: Vec, tiles: Vec) -> Pyramid { + let mut rank = vec![UNWIRED; order.len()]; + for (d, &r) in order.iter().enumerate() { + if let Some(slot) = rank.get_mut(r as usize) { + *slot = d as u32; } } - tiles - .into_iter() - .filter_map(|(i, tiles)| { - let mut pyramid = Pyramid::new(orders.remove(&i)?, tiles); - (pyramid.anchors, pyramid.extents) = placed.remove(&i).unwrap_or_default(); - Some((i, pyramid)) - }) - .collect() - } - - pub(crate) fn new(order: Vec, tiles: Vec) -> Pyramid { let index = tiles.iter().enumerate().map(|(i, t)| (t.key, i as u32)).collect(); let mut above = FxHashSet::default(); for tile in &tiles { @@ -552,106 +613,179 @@ impl Pyramid { } } } - Pyramid { order, tiles, anchors: Vec::new(), extents: Vec::new(), index, above } + Pyramid { order, rank, tiles, made: Version::next(), index, above } } - /// The tiles the camera asks for, coarsest first: at each level, every tile its rows reach into - /// what the camera shows from, splitting while a tile draws larger than its size. A level whose - /// rows would pass `rows` goes, coarser ones drawing whole in its place. - pub(crate) fn wanted(&self, sight: &Sight<'_>, domain: &Domain, rows: u32) -> Wanted { - if self.tiles.len() == 1 && self.tiles[0].key.z == 0 { - return Wanted { tiles: vec![(0, true)], cover: None }; + /// The number of rows it orders. + pub(crate) fn rows(&self) -> u32 { + self.order.len() as u32 + } + + /// The ranges of the order the camera needs, coarsest level first, each with its level, and + /// whether the `rows` budget dropped any of them. + /// + /// At each level it takes every tile whose rows reach into the visible region (for a large + /// tile, only the sub-cells that do). It splits a tile into its children while the tile draws + /// larger than `SPLIT_PX` near the camera's focus. When more tiles would split than a level + /// allows, those drawn largest split first, and the next level lists their children in that + /// order. A drawing holds whole pages of the order. So once a level's pages would exceed `rows` + /// rows, that level and all finer ones are dropped and the coarser levels draw. If the first + /// visible level exceeds the budget alone (for example, edges spanning the square, which cannot + /// go finer), as many of its first rows as fit are drawn. + pub(crate) fn wanted(&self, sight: &Sight<'_>, domain: &Domain, rows: u32) -> (Vec<(u8, Range)>, bool) { + let fit = rows.div_ceil(PAGE_ROWS); + if let [tile] = &self.tiles[..] + && tile.key.z == 0 + && tile.rows.len() as u32 <= fit * PAGE_ROWS + { + return (vec![(0, tile.rows.clone())], false); } let region: Region = sight.region(); let focus = sight.focus(); - let limit = TILE_PX * 1.25; - let mut out: Vec<(u32, bool)> = Vec::new(); - let mut cover = [u64::MAX, u64::MAX, 0, 0]; - let mut total = 0_u32; + let mut out: Vec<(u8, Range)> = Vec::new(); + let (mut held, mut last) = (0_u32, None); let mut level = vec![TileKey { z: 0, x: 0, y: 0 }]; - // The tiles of the level above that split, and where among those found each is. - let mut split: Vec<(TileKey, Option)> = Vec::new(); while !level.is_empty() { let mut found = Vec::new(); - let mut leaves = Vec::new(); - let mut splitting = Vec::new(); - let mut added = 0_u32; - let mut next = Vec::new(); + // The level's tiles with tiles below them that draw larger than `SPLIT_PX`, with their + // sizes. + let mut large: Vec<(TileKey, f64)> = Vec::new(); for key in level { let grow = 0.5 / f64::from(1_u32 << key.z); if !meets(®ion, domain.bounds(key, grow)) { continue; } - let splits = self.above.contains(&key) - && next.len() < MAX_TILES - && sight.measure(&focus, domain.bounds(key, 0.0)).0 > limit; - let mut at = None; if let Some(&t) = self.index.get(&key) { let tile = &self.tiles[t as usize]; if meets(®ion, domain.bounds(key, tile.reach)) { - at = Some(out.len() + found.len()); - found.push((t, !splits)); - added = added.saturating_add(tile.rows.len() as u32); + let first = found.len(); + if tile.cells.is_empty() { + found.push((key.z, tile.rows.clone())); + } + // A sub-cell's rows lie within it and reach no farther than the tile's. + let grow = tile.reach + 0.5 / f64::from(1_u32 << (u32::from(key.z) + SUB_BITS)); + let mut start = tile.rows.start; + for (cell, &end) in tile.cells.iter().enumerate() { + if end > start && meets(®ion, domain.bounds(sub_key(key, cell), grow)) { + let ours = found.len() > first; + match found.last_mut() { + Some((_, range)) if ours && range.end == start => range.end = end, + _ => found.push((key.z, start..end)), + } + } + start = end; + } } } - if splits { - next.extend(key.children()); - splitting.push((key, at)); - } else { - leaves.push(key); + if self.above.contains(&key) { + let (size, _) = sight.measure(&focus, domain, key); + if size > SPLIT_PX { + large.push((key, size)); + } } } - if total.saturating_add(added) > rows && !out.is_empty() { - // The tiles that split draw whole instead. - for (key, at) in split { - cover = covering(cover, key); - if let Some(at) = at { - out[at].1 = true; - } + // The tiles drawn largest split first, a quarter of `MAX_TILES` at most. + crate::order::by_key(&mut large, |&(_, size)| !crate::order::number(size)); + let next: Vec = large.iter().take(MAX_TILES / 4).flat_map(|(key, _)| key.children()).collect(); + // If this level's pages exceed the budget, drop it, and the coarser levels draw. + let (pages, reached) = pages_past(&found, last); + if held + pages > fit { + if out.is_empty() { + out = first(found, fit * PAGE_ROWS); } - break; + return (out, true); } - total = total.saturating_add(added); + (held, last) = (held + pages, reached); out.extend(found); - for key in leaves { - cover = covering(cover, key); - } - split = splitting; level = next; } - Wanted { tiles: out, cover: (cover[0] <= cover[2]).then_some(cover) } + (out, false) + } + + /// Decodes the pyramids a job answered, as [`encode`] wrote them, each with its table's index + /// among those asked. Stops at the first incomplete record. + pub(crate) fn read(bytes: &[u8]) -> Vec<(u32, Pyramid)> { + let words = crate::wire::words(bytes); + let mut words = words.as_slice(); + let mut take = |n: usize| -> Option<&[u32]> { + let (head, rest) = words.split_at_checked(n)?; + words = rest; + Some(head) + }; + let mut out = Vec::new(); + 'tables: while let Some(&[table, n]) = take(2) { + let n = n as usize; + let (Some(order), Some(&[count])) = (take(n).map(<[u32]>::to_vec), take(1)) else { break }; + let mut tiles = Vec::with_capacity(count as usize); + let mut start = 0; + for _ in 0..count { + let Some(&[lo, hi, end, reach, cells]) = take(5) else { break 'tables }; + let Some(cells) = take(cells as usize) else { break 'tables }; + let key = unpack(u64::from(lo) | u64::from(hi) << 32); + let reach = f64::from(reach) / 4_294_967_296.0; + tiles.push(Tile { key, rows: start..end, reach, cells: cells.to_vec() }); + start = end; + } + out.push((table, Pyramid::new(order, tiles))); + } + out } } +/// Encodes `pyramids` as a job's answer, in little-endian `u32` words. Per pyramid: its table's +/// index among those asked, its row count, its order, and its tile count. Then per tile: its +/// packed key (two words), the end of its rows, its reach, its sub-cell count, and where each +/// sub-cell's rows end. +pub(crate) fn encode(pyramids: &[(u32, Pyramid)]) -> Vec { + let mut words: Vec = Vec::new(); + for (table, pyramid) in pyramids { + words.extend([*table, pyramid.rows()]); + words.extend_from_slice(&pyramid.order); + words.push(pyramid.tiles.len() as u32); + for tile in &pyramid.tiles { + let key = pack(tile.key); + let reach = (tile.reach * 4_294_967_296.0).min(f64::from(u32::MAX)) as u32; + words.extend([key as u32, (key >> 32) as u32, tile.rows.end, reach, tile.cells.len() as u32]); + words.extend_from_slice(&tile.cells); + } + } + words.iter().flat_map(|w| w.to_le_bytes()).collect() +} + #[cfg(test)] mod tests { use super::*; - use crate::data::UNWIRED; - /// Fixed point across the square. + /// The anchor `u`, `v` across the square, in fixed point. fn at(u: f64, v: f64) -> [u32; 2] { [(u * 4_294_967_296.0) as u32, (v * 4_294_967_296.0) as u32] } - /// A unit square of a plane, its rows placed `u` from the west and `v` from the north. + /// A unit square of a plane, with rows placed `u` from the west and `v` from the north. fn square(points: &[[f64; 2]]) -> (Domain, Places) { let domain = Domain::Plane { min: [0.0, 0.0], size: 1.0 }; (domain, Places::Points(points.iter().map(|&[u, v]| [u as f32, (1.0 - v) as f32]).collect())) } + /// The level row `row` of `pyramid` is assigned. + fn level_of(pyramid: &Pyramid, row: u32) -> u8 { + let at = pyramid.rank[row as usize]; + pyramid.tiles.iter().find(|t| t.rows.contains(&at)).expect("a tile").key.z + } + #[test] - fn crowded_rows_wait_for_finer_levels_by_priority() { - // Two rows in one cell: the higher lands at the root, the lower a level finer. + fn crowded_rows_go_to_finer_levels_by_priority() { + // Two rows in one cell: the higher goes to the root, the lower one level finer. let anchors = [at(0.3, 0.3), at(0.3, 0.3), at(0.8, 0.8)]; - let level = place(&along(&anchors), &[], &ranks(&[1.0, 2.0, 0.0], 3), &[0, 0, 0], None); - assert_eq!(level[1], 0, "the higher priority lands at the root"); - assert_eq!(level[2], 0, "a row alone in its cell lands at the root"); - assert_eq!(level[0], 1, "the lower lands once the higher has left its cell"); + let level = place(&along(&anchors), &[], &ranks(&[1.0, 2.0, 0.0], 3), &[0, 0, 0], None, &[]); + assert_eq!(level[1], 0, "the higher-priority row goes to the root"); + assert_eq!(level[2], 0, "a row alone in its cell goes to the root"); + assert_eq!(level[0], 1, "the lower-priority row goes one level finer"); let alike = [at(0.3, 0.3); 4]; - let level = place(&along(&alike), &[], &ranks(&[], 4), &[0; 4], None); + let level = place(&along(&alike), &[], &ranks(&[], 4), &[0; 4], None, &[]); let mut sorted = level.clone(); sorted.sort_unstable(); - assert_eq!(sorted, vec![0, 1, 2, 3], "a row a level, equals in an order of their own"); + assert_eq!(sorted, vec![0, 1, 2, 3], "one row per level, ties broken by a hash of the row"); } #[test] @@ -661,25 +795,34 @@ mod tests { (0..n).map(|i| at(f64::from(i % 250) / 250.0, f64::from(i / 250) / 200.0)).collect(); let sorted = along(&anchors); let ranks = ranks(&[], n as usize); - let level = place(&sorted, &[], &ranks, &vec![0; n as usize], None); + let level = place(&sorted, &[], &ranks, &vec![0; n as usize], None, &[]); let (order, tiles) = tiled(&sorted, &anchors, &[], &ranks, &level); assert_eq!(order.len(), n as usize); - let mut start = 0; - for &(_, end, _) in &tiles { - assert!(end - start <= 1 << (2 * GRID_BITS), "a tile holds at most its grid's cells"); - start = end; + for tile in &tiles { + assert!(tile.rows.len() <= 1 << (2 * GRID_BITS), "a tile has at most as many rows as grid cells"); } assert!(tiles.len() > 1); + // A large tile's rows are grouped by sub-cell, and each group lies within its sub-cell. + let large = tiles.iter().find(|t| !t.cells.is_empty()).expect("a large tile"); + let mut start = large.rows.start; + for (cell, &end) in large.cells.iter().enumerate() { + let sub = sub_key(large.key, cell); + for &r in &order[start as usize..end as usize] { + assert_eq!(tile_of(anchors[r as usize], sub.z), sub, "row {r} lies in sub-cell {cell}"); + } + start = end; + } + assert_eq!(start, large.rows.end); } #[test] - fn large_rows_stay_coarse_and_edges_land_by_their_own_priority() { + fn large_rows_stay_coarse_and_edges_get_levels_by_their_own_priority() { let anchors = [at(0.1, 0.1), at(0.9, 0.9)]; let extents = [1 << 30, 0]; - let level = place(&along(&anchors), &extents, &ranks(&[0.0, 0.0], 2), &[0, 0], None); + let level = place(&along(&anchors), &extents, &ranks(&[0.0, 0.0], 2), &[0, 0], None, &[]); assert!(level[0] <= 1, "a row reaching a quarter of the square stays coarse"); - // Two vertices crowd one cell; the edges between them land by their own priority, parallel - // ones together, and one naming no row only at the finest level. + // Two vertices crowd one cell. The edges between them are assigned by their own priority, + // parallel ones together, and one with an unset endpoint only at the finest level. let (domain, places) = square(&[[0.3, 0.3], [0.3, 0.3], [0.31, 0.3]]); let leveling = Leveling { domain, @@ -701,22 +844,62 @@ mod tests { }, ], }; - let pyramids = Pyramid::read(&answer(&leveling).to_bytes()); + let pyramids: FxHashMap = answer(&leveling).into_iter().collect(); let edges = &pyramids[&1]; - let level_of = |p: &Pyramid, row: u32| { - let at = p.order.iter().position(|&r| r == row).unwrap() as u32; - p.tiles.iter().find(|t| t.rows.contains(&at)).unwrap().key.z + assert_eq!(level_of(edges, 0), 0, "an edge alone in its cell shows from the root"); + assert_eq!(level_of(edges, 1), 0, "a parallel edge takes the same level"); + assert_eq!(level_of(edges, 2), MAX_LEVEL, "an edge with an unset endpoint goes to the finest level"); + // Its endpoints show with it: vertices 1 and 2 go to the root though they share a cell. + let vertices = &pyramids[&0]; + assert_eq!((level_of(vertices, 1), level_of(vertices, 2)), (0, 0), "endpoints take their edge's level"); + } + + #[test] + fn answers_read_back_as_written() { + let (domain, places) = square(&[[0.1, 0.1], [0.9, 0.9], [0.5, 0.5]]); + let leveling = Leveling { + domain, + tables: vec![Placing { rows: 3, leveled: true, answered: true, places, ..Placing::default() }], }; - assert_eq!(level_of(edges, 0), 0, "an edge alone in its cell shows from the root, its ends carried"); - assert_eq!(level_of(edges, 1), 0, "with the edge joining the same vertices"); - assert_eq!(level_of(edges, 2), MAX_LEVEL, "an edge naming no row waits for the finest level"); + let answered = answer(&leveling); + let read = Pyramid::read(&encode(&answered)); + assert_eq!(read.len(), 1); + let (table, pyramid) = &read[0]; + assert_eq!(*table, 0); + assert_eq!(pyramid.order, answered[0].1.order); + assert_eq!(pyramid.tiles, answered[0].1.tiles); + for (d, &r) in pyramid.order.iter().enumerate() { + assert_eq!(pyramid.rank[r as usize], d as u32, "each row's rank is its place in the order"); + } + } + + #[test] + fn levels_count_only_the_pages_past_the_level_above() { + // Two sub-cells in one page, one in the next, and a level below reaching the page where the + // one above ended. + let above = [(0, 10..20), (0, 50..60), (0, PAGE_ROWS + 3..PAGE_ROWS + 9)]; + let (pages, last) = pages_past(&above, None); + assert_eq!((pages, last), (2, Some(1))); + let below = [(1, PAGE_ROWS + 20..3 * PAGE_ROWS + 1)]; + assert_eq!(pages_past(&below, last), (2, Some(3)), "pages 2 and 3; page 1 was counted above"); + } + + #[test] + fn a_level_over_the_budget_draws_its_first_rows() { + let ranges = vec![(0, 0..3 * PAGE_ROWS), (0, 5 * PAGE_ROWS..6 * PAGE_ROWS)]; + assert_eq!(first(ranges.clone(), 2 * PAGE_ROWS), [(0, 0..2 * PAGE_ROWS)], "its first two pages"); + assert_eq!( + first(ranges, 4 * PAGE_ROWS), + [(0, 0..3 * PAGE_ROWS), (0, 5 * PAGE_ROWS..6 * PAGE_ROWS)], + "every range when all fit" + ); } #[test] - fn tiles_number_exactly() { + fn tile_keys_round_trip_through_an_f64() { let key = TileKey { z: MAX_LEVEL, x: (1 << 24) - 1, y: 12_345 }; assert_eq!(unpack(pack(key)), key); - assert_eq!(unpack(pack(key) as f64 as u64), key, "a number holds it exactly"); + assert_eq!(unpack(pack(key) as f64 as u64), key, "an f64 holds a packed key exactly"); } #[test] @@ -725,8 +908,8 @@ mod tests { domain: Domain::Earth, tables: vec![Placing { rows: 3, answered: true, priority: [5.0, 1.0, 3.0].into(), ..Placing::default() }], }; - let pyramids = Pyramid::read(&answer(&leveling).to_bytes()); - assert_eq!(pyramids[&0].order, vec![1, 2, 0], "highest last, drawn on top"); - assert_eq!(pyramids[&0].tiles.len(), 1); + let pyramids = answer(&leveling); + assert_eq!(pyramids[0].1.order, vec![1, 2, 0], "highest last, drawn on top"); + assert_eq!(pyramids[0].1.tiles.len(), 1); } } diff --git a/crates/latkit/src/derive/shapes.rs b/crates/latkit/src/derive/shapes.rs index 657e7c99..6326f788 100644 --- a/crates/latkit/src/derive/shapes.rs +++ b/crates/latkit/src/derive/shapes.rs @@ -1,29 +1,138 @@ -//! Shapes: what geometry fields become to draw, written beside the tables marks draw them from. Every -//! area's triangles, the point its label hangs from first; and on the earth every line's path, -//! unwrapped across the antimeridian and bent with the globe. Shapes are made page by page from a -//! field's pages and kept while a page holds; a page written again keeps the shapes of its rows -//! whose geometry did not change, so a tile landing shapes only its own rows. Triangles that do not -//! fit the frame's budget are made in a job, their rows drawing nothing until it lands; bending a -//! path is cheap, and done at once. +//! Shapes: the drawable forms of geometry fields. +//! +//! Each area becomes triangles, preceded by its label point. On the earth, each line's path is +//! unwrapped across the antimeridian and bent to follow the globe. +//! +//! A table drawn whole is shaped one page of its field at a time, into a field beside it. A page +//! keeps the shapes made from it before. A page written over another keeps the shapes of the rows +//! not written, so a write reshapes only the rows it wrote. A paged table is shaped one page of its +//! order at a time, the first time each page draws. A row whose geometry did not change keeps its +//! shape as its pages come and go. +//! +//! Triangulation that does not fit the frame's budget runs in a job. A page where every row needs +//! shaping goes to the job whole. Rows waiting on a job draw nothing until it finishes. Bending a +//! path is cheap and done at once. Vector tiles bring their own shapes, made in the job that +//! decodes them, as a field beside their rows. Their tables are paged and read that field one page +//! of their order at a time, like any other, so nothing shapes those rows again. -use rustc_hash::FxHashMap; +use rustc_hash::{FxHashMap, FxHashSet}; -use super::{Derive, Input, Jobs, Tables, answers, beside, stamp_of}; +use super::order::Order; +use super::{Input, Jobs, Paged, Tables, Working, beside, stamp_of}; use crate::data::{ - Column, Field, GeometryBuilder, Id, Keyed, Name, PAGE_ROWS, Page, PageId, Table, Value, Values, Version, + Column, Field, Geometry, GeometryBuilder, Id, Keyed, Name, Page, PageId, RowIds, Table, Values, Version, }; -use crate::host::{Job, Shaping}; -use crate::scene::areas::triangles_of; -use crate::scene::lines::path_of; -use crate::{Mark, Pending, Space}; +use crate::gpu::shaders::PAGE_ROWS; +use crate::host::{Job, Shaping, shapes_answer}; +use crate::tiles::TileSets; +use crate::{Mark, Scene, Space}; -/// Microseconds a point takes to triangulate, by estimate. +/// Estimated microseconds to triangulate one point. const TRIANGULATE: f64 = 0.5; -/// Rows by id, each with the digest of the geometry it was shaped from. -type Digests = FxHashMap; +/// Rows to shape individually: each row index, its id, and its geometry digest. +type Rows = Vec<(usize, Id, u64)>; -/// A field to shape: its dataset, table, and field, and how. +/// Work left to shape: whole pages of the field where every row needs shaping (by page id, with +/// their values), and individual rows. +#[derive(Default)] +struct Missing { + pages: Vec<(PageId, Values)>, + rows: Rows, +} + +impl Missing { + fn is_empty(&self) -> bool { + self.pages.is_empty() && self.rows.is_empty() + } + + /// The number of points to shape. + fn points(&self, column: &Column) -> usize { + let pages: usize = self.pages.iter().map(|(_, values)| values.geometry().map_or(0, Geometry::points)).sum(); + pages + self.rows.iter().map(|&(row, ..)| points_of(column, row)).sum::() + } + + /// Splits the work into a job's parts, each a geometry column: each whole page, then one part + /// holding all the rows. Also returns what each part shapes. + fn parts(self, column: &Column) -> (Vec, Vec) { + let (mut parts, mut asked) = (Vec::new(), Vec::new()); + for (id, values) in self.pages { + asked.push(Asked::Page(id, values.len())); + parts.push(values); + } + if !self.rows.is_empty() { + let rows: Vec = self.rows.iter().map(|&(row, ..)| row).collect(); + parts.push(geometry_of(column, &rows)); + asked.push(Asked::Rows(self.rows.into_iter().map(|(_, id, digest)| (id, digest)).collect())); + } + (parts, asked) + } +} + +/// What one part of a job shapes: a whole page of the field (by page id, with its row count), or +/// rows (by id, each with its geometry digest). +#[derive(Hash)] +enum Asked { + Page(PageId, usize), + Rows(Vec<(Id, u64)>), +} + +impl Asked { + /// Empty shapes of depth `depth` for every row this part asks for. + fn empty(&self, depth: u8) -> Values { + empty_shapes( + depth, + match self { + Asked::Page(_, rows) => *rows, + Asked::Rows(rows) => rows.len(), + }, + ) + } +} + +/// `rows` empty shapes of depth `depth`, which draw nothing. +fn empty_shapes(depth: u8, rows: usize) -> Values { + Values::of_rows(depth, std::iter::repeat_n(None, rows)) +} + +/// Shapes made since the last complete set. Whole pages are keyed by the id of the field page each +/// was made from. Rows are keyed by id, each with the geometry digest it was made from and its part +/// and index in `parts`. +#[derive(Default)] +struct Landed { + pages: FxHashMap, + rows: FxHashMap, + parts: Vec, +} + +impl Landed { + /// Stores `shapes`, the shapes of each part `asked` names. + fn land(&mut self, asked: Vec, shapes: Vec) { + for (asked, shapes) in asked.into_iter().zip(shapes) { + match asked { + Asked::Page(id, _) => { + self.pages.insert(id, Page::new(shapes)); + } + Asked::Rows(rows) => { + let part = self.parts.len() as u32; + for (i, (id, digest)) in rows.into_iter().enumerate() { + self.rows.insert(id, (digest, part, i as u32)); + } + self.parts.push(shapes); + } + } + } + } + + /// Where the shape of the row with `id` lies, if made from geometry with digest `digest`. + fn row(&self, id: &Id, digest: u64) -> Option<(&Geometry, usize)> { + let &(made, part, at) = self.rows.get(id)?; + (made == digest).then_some(())?; + Some((self.parts[part as usize].geometry()?, at as usize)) + } +} + +/// A field to shape: its dataset, table, field, and shaping. #[derive(Clone, PartialEq, Eq, Hash, Debug)] struct Want { data: Name, @@ -35,27 +144,77 @@ struct Want { impl Want { /// The derived field the shapes are written as. fn derived(&self) -> Name { - match self.shaping { - Shaping::Triangles { .. } => triangles_of(&self.field), - Shaping::Path => path_of(&self.field), + self.shaping.field_of(&self.field) + } +} + +/// The shapes of a paged table, per page of its order: each page holds its rows' shapes in order, +/// with a page id of its own. Kept rows whose pages are not drawn are shaped individually, by their +/// place in the order. +pub(crate) struct PagedShapes { + /// The id of the order these pages belong to. + pub(crate) order: u64, + /// Bumps whenever a page or kept row is shaped. + pub(crate) version: Version, + pub(crate) pages: FxHashMap, + pub(crate) kept: FxHashMap, + /// The nesting depth of the shapes' geometry. + pub(crate) depth: u8, +} + +/// One shaped page of an order. +pub(crate) struct ShapedPage { + pub(crate) id: PageId, + pub(crate) shapes: Values, + /// Whether every row has its shape, with none waiting for a job. + whole: bool, +} + +/// What a paged table's shapes were made from: the order, the source table and field column, and +/// the shaped pages. +struct Paging { + order: Order, + source: Table, + column: Column, + shapes: PagedShapes, +} + +impl Paging { + /// Where a complete page holds the shape of the row with `id`, if the row's geometry digest is + /// still `digest`: the page's geometry and the row's index in it. + fn held(&self, id: &Id, digest: u64) -> Option<(&Geometry, usize)> { + let row = self.source.row(id.as_ref())?; + let (values, i) = self.column.at(row).filter(|&(values, i)| values.is_valid(i))?; + if row >= self.order.len() as usize || values.geometry()?.row_hash(i) != digest { + return None; } + let d = self.order.rank(row as u32); + let page = self.shapes.pages.get(&(d / PAGE_ROWS)).filter(|page| page.whole)?; + Some((page.shapes.geometry()?, (d % PAGE_ROWS) as usize)) } } -/// One field's shapes: the last whole set made, the derived field written, shapes made but not yet -/// written, and a job under way. +/// One field's shapes and any pending job. For a table drawn whole, it holds the last complete set, +/// the derived field written, and shapes made but not yet written. For a paged table, it holds its +/// pages and the previous order's pages. #[derive(Default)] struct Shaped { - /// The source table, its field, and the derived column the shapes were last wholly made from. + /// The source table, its field column, and the derived column from the last time every row + /// was shaped. made: Option<(Table, Column, Column)>, - /// The derived field written last, and the stamp of what it was written from. + /// The derived field written last. field: Option, + /// The stamp of the inputs `field` was written from. stamp: u64, - /// Shapes made since, by row id: the digest of the geometry each was made from, and the shape. - landed: FxHashMap, - /// A job under way, with the digest of each row it shapes. - working: Option<(Pending>, Digests)>, - /// Bumps whenever a job's answer lands. + /// Shapes made since the last complete set. + landed: Landed, + /// Paged: its shaped pages. + paging: Option, + /// The pages of the previous order. New pages copy the shapes of unchanged rows from it. + before: Option, + /// A shaping job, for the stamp of what it asks, with what each of its parts shapes. + working: Working>, + /// Bumps whenever a job's answer arrives. landings: u64, } @@ -65,17 +224,28 @@ pub(crate) struct Shapes { shaped: FxHashMap, } -impl Derive for Shapes { - fn update(&mut self, input: &Input<'_>, jobs: &mut Jobs<'_>) -> bool { +impl Shapes { + /// Updates the shapes to `input`, using as is any shapes the vector tiles in `tiles` bring, and + /// sending work the frame cannot afford to `jobs`. Returns whether its tables changed. + pub(crate) fn update(&mut self, input: &Input<'_>, tiles: &TileSets, jobs: &mut Jobs<'_>) -> bool { let wants = wants(input); self.shaped.retain(|want, _| wants.contains(want)); let mut changed = false; for want in &wants { let Some(source) = input.data(&want.data).and_then(|d| d.table(&want.table)) else { continue }; let shaped = self.shaped.entry(want.clone()).or_default(); - changed |= shaped.update(want, source, jobs); + // Vector tiles bring their own shapes, row for row. + let brought = tiles.shaped(&want.data).and_then(|d| d.table(&want.table)?.field_named(&want.derived())); + if let Some(field @ Field::Values(column, _)) = brought + && column.len() == source.len() + { + changed |= shaped.take(field); + continue; + } + let paged = input.paged.and_then(|paged| paged.get(&(want.data.clone(), want.table.clone()))); + changed |= shaped.update(want, source, paged, jobs); } - // Each drawn table's derived table holds every field shaped from it. + // Each table drawn whole has a derived table holding every field shaped from it. let mut fields: Keyed<(Name, Name), Vec<(Name, Field)>> = Keyed::default(); for want in &wants { if let Some(field) = self.shaped.get(want).and_then(|s| s.field.clone()) { @@ -94,18 +264,50 @@ impl Derive for Shapes { changed } - fn busy(&self) -> bool { - self.shaped.values().any(|s| s.working.is_some()) + /// Whether a shape job is pending. The rows it shapes draw nothing until it finishes. + pub(crate) fn busy(&self) -> bool { + self.shaped.values().any(|s| s.working.busy()) } - fn tables(&self) -> &Tables { + /// Whether every drawn row has its shape. + pub(crate) fn complete(&self) -> bool { + !self.busy() + } + + /// The derived tables beside tables drawn whole, each holding the fields shaped from it. + pub(crate) fn tables(&self) -> &Tables { &self.tables } - fn clear(&mut self) { - self.tables.clear(); - self.shaped.clear(); + /// The shapes of paged table `table` of dataset `data`, each with its derived field's name. + pub(crate) fn paged(&self, data: &Name, table: &Name) -> Vec<(Name, &PagedShapes)> { + self.shaped + .iter() + .filter(|(want, _)| want.data == *data && want.table == *table) + .filter_map(|(want, shaped)| Some((want.derived(), &shaped.paging.as_ref()?.shapes))) + .collect() + } +} + +/// The field mark `mark` shapes, and how: an area's polygons, or on the earth a line's paths. +fn shaping_of(mark: &Mark, earth: bool) -> Option<(Name, Shaping)> { + match mark { + Mark::Areas(areas) => Some((areas.field(), Shaping::Triangles { earth })), + Mark::Lines(lines) if earth => Some((lines.traced(), Shaping::Path)), + _ => None, + } +} + +/// The fields of table `table` of dataset `data` that the scene's marks shape, and how, each once. +pub(crate) fn shapings(scene: &Scene, data: &Name, table: &Name) -> Vec<(Name, Shaping)> { + let earth = matches!(scene.space, Space::Earth(_)); + let mut shapings = Vec::new(); + for mark in scene.marks.iter().filter(|m| m.style().data == *data && m.style().table == *table) { + if let Some(shaping) = shaping_of(mark, earth).filter(|s| !shapings.contains(s)) { + shapings.push(shaping); + } } + shapings } /// The fields the scene's marks shape: every area's polygons, and on the earth every line's paths. @@ -113,11 +315,7 @@ fn wants(input: &Input<'_>) -> Vec { let earth = matches!(input.scene.space, Space::Earth(_)); let mut wants: Vec = Vec::new(); for mark in &input.scene.marks { - let (field, shaping) = match mark { - Mark::Areas(areas) => (areas.field(), Shaping::Triangles { earth }), - Mark::Lines(lines) if earth => (lines.traced(), Shaping::Path), - _ => continue, - }; + let Some((field, shaping)) = shaping_of(mark, earth) else { continue }; let style = mark.style(); let want = Want { data: style.data.clone(), table: style.table.clone(), field, shaping }; if !wants.contains(&want) { @@ -128,136 +326,389 @@ fn wants(input: &Input<'_>) -> Vec { } impl Shaped { - /// Brings the shapes up to `source`; whether the derived field changed. - fn update(&mut self, want: &Want, source: &Table, jobs: &mut Jobs<'_>) -> bool { - if let Some((pending, asked)) = &self.working - && let Some(answer) = pending.take() - { - for ((_, id), values) in answer.map(|bytes| answers(&bytes, &["shape"])).unwrap_or_default() { - if let (Some(&digest), Some(Some(shape))) = (asked.get(&id), values.into_iter().next()) { - self.landed.insert(id, (digest, shape)); - } - } - self.working = None; - self.landings += 1; - } - let Some(Field::Values(column, version)) = source.field_named(&want.field) else { + /// Updates the shapes to `source`: as a paged table when `paged` is given, else drawn whole. + /// Returns whether they changed. + fn update(&mut self, want: &Want, source: &Table, paged: Option<&Paged>, jobs: &mut Jobs<'_>) -> bool { + let Some(Field::Values(column, version)) = source.field_named(&want.field) else { return self.forget() }; + let Some(depth) = column.kind().depth().filter(|&depth| want.shaping.takes(depth)) else { + return self.forget(); + }; + self.settle(want.shaping.depth(depth)); + if let Some(paged) = paged { + // If it draws whole again later, its field is rewritten from scratch. let had = self.field.take().is_some(); self.stamp = 0; - return had; - }; - let depth = column.kind().depth(); - let ((Some(depth @ (2 | 3)), Shaping::Triangles { .. }) | (Some(depth @ 1..), Shaping::Path)) = - (depth, want.shaping) - else { - return self.field.take().is_some(); - }; - let stamp = stamp_of((source.structure, *version, self.landings, want.shaping)); + return self.update_paged(want, source, column, depth, paged, jobs) || had; + } + // Drawn whole again: unchanged rows take their shapes from the paged shapes. + let had = self.paging.is_some(); + if had { + self.before = self.paging.take(); + } + self.update_whole(want, source, column, *version, depth, jobs) || had + } + + /// Takes a finished job's shapes, of depth `depth`. A job that failed, or that answered other + /// parts than it was asked, gives its rows empty shapes: they draw nothing until their geometry + /// changes. + fn settle(&mut self, depth: u8) { + let Some((_, asked, answer)) = self.working.landed() else { return }; + let shapes = answer.ok().and_then(|bytes| shapes_answer(&bytes)).filter(|shapes| shapes.len() == asked.len()); + let shapes = shapes.unwrap_or_else(|| asked.iter().map(|part| part.empty(depth)).collect()); + self.landed.land(asked, shapes); + self.landings += 1; + } + + /// Uses `field`, made elsewhere (by vector tiles), as its shapes. Returns whether they changed. + fn take(&mut self, field: &Field) -> bool { + let changed = self.field.as_ref().is_none_or(|f| f.version() != field.version()); + *self = Shaped { field: Some(field.clone()), ..Shaped::default() }; + changed + } + + /// Drops its shapes and any pending job once its field no longer needs shaping. Returns whether + /// it had any shapes. + fn forget(&mut self) -> bool { + let had = self.field.is_some() || self.paging.is_some(); + (self.field, self.paging, self.before, self.stamp) = (None, None, None, 0); + self.working.cancel(); + had + } + + /// Updates the shapes of a table drawn whole to `source`. Returns whether its field changed. + fn update_whole( + &mut self, + want: &Want, + source: &Table, + column: &Column, + version: Version, + depth: u8, + jobs: &mut Jobs<'_>, + ) -> bool { + let stamp = stamp_of((source.structure, version, self.landings, want.shaping)); if stamp == self.stamp { return false; } self.stamp = stamp; - // Rows shaped on the frame land at once; a second pass writes them. - for _ in 0..2 { - let (derived, missing) = self.build(want, source, column, depth); - if missing.is_empty() { - self.field = Some(Field::Values(derived.clone(), Version::next())); - self.made = Some((source.clone(), column.clone(), derived)); - self.landed.clear(); - return true; + let (mut derived, mut missing) = self.build(want, source, column, depth); + // Shapes made on the frame apply at once. Rebuild the column with them. + if !missing.is_empty() { + match self.shape_now(want, column, missing, jobs) { + Ok(()) => (derived, missing) = self.build(want, source, column, depth), + Err(left) => missing = left, } - let points: usize = missing.iter().map(|&(row, ..)| points_of(column, row)).sum(); - if jobs.fits((points as f64 * TRIANGULATE).ceil() as u64) { - for (row, id, digest) in missing { - let Some(value) = column.get(row) else { continue }; - let shape = want.shaping.shape(&value).unwrap_or(value); - self.landed.insert(id, (digest, shape)); - } - continue; + } + if missing.is_empty() { + self.field = Some(Field::Values(derived.clone(), Version::next())); + self.made = Some((source.clone(), column.clone(), derived)); + self.landed = Landed::default(); + self.before = None; + return true; + } + self.ask(want, column, missing, jobs); + // Until the job finishes, its rows draw nothing and the rest draw unchanged. + self.field = Some(Field::Values(derived, Version::next())); + true + } + + /// Updates the shapes of a paged table to `source`. Shapes each page `paged` draws the first + /// time it draws, and again once the rows it waited on arrive. Returns whether a page was made. + fn update_paged( + &mut self, + want: &Want, + source: &Table, + column: &Column, + depth: u8, + paged: &Paged, + jobs: &mut Jobs<'_>, + ) -> bool { + let order = &paged.order; + // A new order: its pages are made from scratch, copying unchanged rows' shapes from the + // previous order's pages. + if self.paging.as_ref().is_none_or(|p| p.shapes.order != order.id()) { + let shapes = PagedShapes { + order: order.id(), + version: Version::next(), + pages: FxHashMap::default(), + kept: FxHashMap::default(), + depth: want.shaping.depth(depth), + }; + let fresh = Paging { order: order.clone(), source: source.clone(), column: column.clone(), shapes }; + self.before = self.paging.replace(fresh).or(self.before.take()); + } + let paging = self.paging.as_mut().expect("paged"); + paging.order = order.clone(); + // Remake the pages holding rows written since, and reshape those rows where kept. + if !paging.column.shares(column) { + let written = column.written_since(&paging.column); + for &row in written.iter().filter(|&&row| row < order.len() as usize) { + paging.shapes.pages.remove(&(order.rank(row as u32) / PAGE_ROWS)); } - if self.working.is_none() { - let rows: Vec = missing.iter().map(|&(row, ..)| row).collect(); - let job = Job::shapes(want.shaping, rows_of(source, column, &rows)); - let asked = missing.into_iter().map(|(_, id, digest)| (id, digest)).collect(); - self.working = Some((jobs.spawn(job), asked)); + if !paging.shapes.kept.is_empty() { + let written: FxHashSet = written.into_iter().map(|row| row as u32).collect(); + paging.shapes.kept.retain(|&d, _| !written.contains(&order.source(d))); } - // Until the job lands, its rows draw nothing and the rest draw as they were. - self.field = Some(Field::Values(derived, Version::next())); - return true; + (paging.source, paging.column) = (source.clone(), column.clone()); + } + paging.shapes.kept.retain(|d, _| paged.kept.binary_search(d).is_ok()); + // Make drawn pages that are not made yet, or that wait on rows while no job runs. While no + // job runs, also shape kept rows not shaped yet. + let running = self.working.busy(); + let made = &paging.shapes.pages; + let make: Vec = + paged.pages.iter().copied().filter(|p| made.get(p).is_none_or(|page| !page.whole && !running)).collect(); + let kept_made = &paging.shapes.kept; + let keep: Vec = if running { + Vec::new() + } else { + paged.kept.iter().copied().filter(|d| !kept_made.contains_key(d)).collect() + }; + if make.is_empty() && keep.is_empty() { + return false; + } + let (mut made, mut kept, mut missing) = self.make_paged(want, order, &make, &keep, source, column, depth); + // Shapes made on the frame apply at once. Remake the pages with them. + if !missing.rows.is_empty() { + match self.shape_now(want, column, missing, jobs) { + Ok(()) => (made, kept, missing) = self.make_paged(want, order, &make, &keep, source, column, depth), + Err(left) => missing = left, + } + } + if !missing.is_empty() { + self.ask(want, column, missing, jobs); + } + let paging = self.paging.as_mut().expect("paged"); + for (page, shapes, whole) in made { + paging.shapes.pages.insert(page, ShapedPage { id: PageId::next(), shapes, whole }); + } + paging.shapes.kept.extend(kept); + paging.shapes.version = Version::next(); + // Every page complete and no job pending: the pages hold every shape that arrived, and the + // previous order is no longer needed. + if !self.working.busy() && paging.shapes.pages.values().all(|page| page.whole) { + self.landed = Landed::default(); + self.before = None; + } + true + } + + /// Shapes pages `make` of `order`, each with whether it is complete, and kept rows `keep` (by + /// place in the order). Also returns the rows left to shape. + #[allow(clippy::too_many_arguments, clippy::type_complexity)] + fn make_paged( + &self, + want: &Want, + order: &Order, + make: &[u32], + keep: &[u32], + source: &Table, + column: &Column, + depth: u8, + ) -> (Vec<(u32, Values, bool)>, Vec<(u32, Values)>, Missing) { + let mut missing = Missing::default(); + let mut made = Vec::with_capacity(make.len()); + for &page in make { + let (shapes, rows) = self.make_rows(want, &order.page(page), source, column, depth); + made.push((page, shapes, rows.is_empty())); + missing.rows.extend(rows); + } + let mut kept = Vec::with_capacity(keep.len()); + for &d in keep { + let (shape, rows) = self.make_rows(want, &[order.source(d)], source, column, depth); + if rows.is_empty() { + kept.push((d, shape)); + } + missing.rows.extend(rows); + } + (made, kept, missing) + } + + /// Shapes `missing` on the frame if it fits the rest of the budget. Otherwise returns it. + fn shape_now( + &mut self, + want: &Want, + column: &Column, + missing: Missing, + jobs: &mut Jobs<'_>, + ) -> Result<(), Missing> { + if !jobs.fits((missing.points(column) as f64 * TRIANGULATE).ceil() as u64) { + return Err(missing); } - false + let (parts, asked) = missing.parts(column); + let shapes = parts.iter().map(|part| want.shaping.shapes_of(part)).collect(); + self.landed.land(asked, shapes); + Ok(()) } - /// The derived column for `column` of `source`, page by page: a page whose source page is the - /// one shaped last keeps its shapes; others are assembled row by row from shapes landed, or kept - /// from the last set where a row's geometry did not change. Rows with neither are missing: each - /// row, its id, and the digest of its geometry. - fn build(&self, want: &Want, source: &Table, column: &Column, depth: u8) -> (Column, Vec<(usize, Id, u64)>) { + /// Starts a job to shape `missing`, unless the same job already ran. While a job is pending it + /// does nothing, and a later update asks again. + fn ask(&mut self, want: &Want, column: &Column, missing: Missing, jobs: &mut Jobs<'_>) { + if self.working.busy() { + return; + } + let (parts, asked) = missing.parts(column); + let stamp = stamp_of(&asked); + if self.working.may_start(stamp) { + self.working.start(stamp, jobs.spawn(Job::shapes(want.shaping, &parts)), asked); + } + } + + /// Builds the derived column for `column` of `source`, page by page, and returns what is left + /// to shape. A page reuses the shapes made from it before, or that arrived for it. A page + /// written over a shaped page keeps the shapes of its rows not written. A page at the same + /// place in the last complete set keeps the shapes of its unchanged rows. A page where no row + /// can reuse a shape is shaped whole. Otherwise each row reuses a shape made from the same + /// geometry, or is left to shape. + fn build(&self, want: &Want, source: &Table, column: &Column, depth: u8) -> (Column, Missing) { let made_pages: FxHashMap = self .made .iter() .flat_map(|(_, field, derived)| field.pages().iter().map(|p| p.id).zip(derived.pages())) .collect(); let depth = want.shaping.depth(depth); + // Rows below `aligned` are at the same places as in the last complete set: since then they + // were only written over, or rows were appended after them. + let aligned = self.made.as_ref().map_or(0, |(table, ..)| aligned(source, table)); + // Whether row `row` may find a shape made before: it is aligned, or rows moved since the + // last complete set (`aligned` is 0), so a lookup by id may find one. + let found = |row: usize| row < aligned || self.made.is_some() && aligned == 0; + let fresh = self.before.is_none() && self.landed.rows.is_empty() && want.shaping != Shaping::Path; let mut pages: Vec = Vec::with_capacity(column.pages().len()); - let mut missing = Vec::new(); + let mut missing = Missing::default(); for (p, page) in column.pages().iter().enumerate() { if let Some(&kept) = made_pages.get(&page.id) { pages.push(kept.clone()); continue; } + if let Some(landed) = self.landed.pages.get(&page.id) { + pages.push(landed.clone()); + continue; + } let Some(geometry) = page.values.geometry() else { - pages.push(Page::new(Values::of_geometry(GeometryBuilder::new(depth).finish()))); + pages.push(Page::new(empty_shapes(depth, 0))); continue; }; - // A path page that needs no bending is its own shape: shared, nothing uploads again. - if want.shaping == Shaping::Path && !bends(geometry.coords()) { + // A path page needing no bending is its own shape. Sharing it means no new upload. + if want.shaping == Shaping::Path && !bends(geometry) { pages.push(page.clone()); continue; } + let start = p * PAGE_ROWS as usize; + let len = page.values.len(); + let kept = self.over(page, &made_pages).or_else(|| { + let (_, field, derived) = self.made.as_ref().filter(|_| start < aligned)?; + let was = &field.pages().get(p)?.values; + let same = (0..len).map(|i| start + i < aligned && page.values.same_row(i, was, i)).collect(); + Some((&derived.pages().get(p)?.values, same)) + }); + if fresh && kept.is_none() && !found(start) { + // Every row needs shaping: shape the page whole. It draws nothing until then. + pages.push(Page::new(empty_shapes(depth, len))); + missing.pages.push((page.id, page.values.clone())); + continue; + } let mut shapes = GeometryBuilder::new(depth); - for local in 0..page.values.len() { - let row = p * PAGE_ROWS + local; - let Some(id) = source.ids.id(row) else { - shapes.push_empty(); - continue; - }; - if !page.values.is_valid(local) { - shapes.push_empty(); + for local in 0..len { + if let Some((was, keep)) = &kept + && keep.get(local).copied().unwrap_or(false) + && let Some(was) = was.geometry() + { + shapes.push_from(was, local); continue; } - let digest = geometry.row_hash(local); - if let Some((d, shape)) = self.landed.get(&id) - && *d == digest - { - shapes.push_value(shape); - } else if let Some((page, at)) = self.kept(&id, digest) { - shapes.push_from(page, at); - } else if want.shaping == Shaping::Path { - // Bending is cheap, so done here; rows that need none keep their geometry. - match page.values.get(local).and_then(|v| want.shaping.shape(&v)) { - Some(shape) => { - shapes.push_value(&shape); - } - None => shapes.push_from(geometry, local), - } - } else { - missing.push((row, id, digest)); - shapes.push_empty(); + let row = start + local; + match source.ids.id(row).filter(|_| page.values.is_valid(local)) { + Some(id) => self.shape(want, &mut shapes, (&page.values, local), row, id, &mut missing.rows), + None => shapes.push_empty(), } } pages.push(Page::new(Values::of_geometry(shapes.finish()))); } - let like = Values::of_geometry(GeometryBuilder::new(depth).finish()); - (Column::of_pages(pages, &like), missing) + (Column::of_pages(pages, &empty_shapes(depth, 0)), missing) + } + + /// The shapes of the page that `page` was written over, searching back through its recorded + /// writes (a few at most), with whether each row keeps its shape (rows not written since do). + fn over<'a>(&'a self, page: &Page, made: &FxHashMap) -> Option<(&'a Values, Vec)> { + let mut written: Vec = Vec::new(); + let mut at = page.written.as_deref(); + while let Some(w) = at { + written.extend_from_slice(&w.rows); + if let Some(shapes) = made.get(&w.from).copied().or_else(|| self.landed.pages.get(&w.from)) { + let mut keep = vec![true; page.values.len()]; + for &row in &written { + if let Some(k) = keep.get_mut(row as usize) { + *k = false; + } + } + keep.truncate(shapes.values.len()); + return Some((&shapes.values, keep)); + } + at = w.before.as_deref(); + } + None } - /// Where the last whole set holds the shape of the row with `id`, when its geometry's digest was - /// `digest`: the derived page's geometry and the row's place in it. - fn kept(&self, id: &Id, digest: u64) -> Option<(&crate::data::Geometry, usize)> { + /// Shapes rows `rows` of `source` in that order, as [`shape`](Shaped::shape) does, and returns + /// the rows left to shape. + fn make_rows(&self, want: &Want, rows: &[u32], source: &Table, column: &Column, depth: u8) -> (Values, Rows) { + let mut shapes = GeometryBuilder::new(want.shaping.depth(depth)); + let mut missing = Vec::new(); + for row in rows.iter().map(|&r| r as usize) { + let found = column.at(row).filter(|&(values, i)| values.is_valid(i)); + match (found, source.ids.id(row)) { + (Some(at), Some(id)) => self.shape(want, &mut shapes, at, row, id, &mut missing), + _ => shapes.push_empty(), + } + } + (Values::of_geometry(shapes.finish()), missing) + } + + /// Writes the shape of row `row` (with `id`), whose geometry is value `i` of `values`. Reuses a + /// shape that arrived or was made before from the same geometry. Bends a path at once. + /// Otherwise writes an empty shape and adds the row to `missing`. + fn shape( + &self, + want: &Want, + shapes: &mut GeometryBuilder, + (values, i): (&Values, usize), + row: usize, + id: Id, + missing: &mut Rows, + ) { + let Some(geometry) = values.geometry() else { + shapes.push_empty(); + return; + }; + let digest = geometry.row_hash(i); + if let Some((landed, at)) = self.landed.row(&id, digest).or_else(|| self.kept(&id, digest)) { + shapes.push_from(landed, at); + } else if want.shaping == Shaping::Path { + // Bending is cheap, so it runs here. Rows needing none keep their geometry. + match values.get(i).and_then(|v| want.shaping.shape(&v)) { + Some(shape) => { + shapes.push_value(&shape); + } + None => shapes.push_from(geometry, i), + } + } else { + missing.push((row, id, digest)); + shapes.push_empty(); + } + } + + /// Where an earlier shape of the row with `id` lies, if made from geometry with digest + /// `digest`: in the last complete set, or in a complete page of the previous paged order. + fn kept(&self, id: &Id, digest: u64) -> Option<(&Geometry, usize)> { + self.whole(id, digest).or_else(|| self.before.as_ref()?.held(id, digest)) + } + + /// Where the last complete set holds the shape of the row with `id`, if made from geometry with + /// digest `digest`: the derived page's geometry and the row's index in it. + fn whole(&self, id: &Id, digest: u64) -> Option<(&Geometry, usize)> { let (table, field, derived) = self.made.as_ref()?; let row = table.row(id.as_ref())?; - let (page, at) = (row / PAGE_ROWS, row % PAGE_ROWS); + let (page, at) = (row / PAGE_ROWS as usize, row % PAGE_ROWS as usize); let was = field.pages().get(page)?.values.geometry()?; if !field.pages()[page].values.is_valid(at) || was.row_hash(at) != digest { return None; @@ -266,31 +717,201 @@ impl Shaped { } } -/// Whether any two coordinates in a row of `coords` lie farther apart than an earth path's step: a -/// page with none needs no bending. -fn bends(coords: &[[f64; 2]]) -> bool { +/// How many leading rows of `now` are `then`'s rows at the same places: all of them if no rows came +/// or went, `then`'s rows if rows were only appended after them, else 0. +fn aligned(now: &Table, then: &Table) -> usize { + if now.structure == then.structure { + return now.len(); + } + match (&now.ids, &then.ids) { + (RowIds::Count(_), RowIds::Count(_)) => now.len().min(then.len()), + (RowIds::Listed(ids), RowIds::Listed(was)) if ids.kept_below(was, was.len()) => was.len(), + _ => 0, + } +} + +/// Whether any two consecutive coordinates of `geometry` lie farther apart than an earth path's +/// step. A page with none needs no bending. +fn bends(geometry: &Geometry) -> bool { let step = crate::geo::shapes::EARTH_STEP; + let coords = geometry.coords(); coords.windows(2).any(|w| (w[1][0] - w[0][0]).abs() > step || (w[1][1] - w[0][1]).abs() > step) } /// How many points row `row` of geometry column `column` holds. fn points_of(column: &Column, row: usize) -> usize { - let Some(page) = column.pages().get(row / PAGE_ROWS) else { return 0 }; - page.values.geometry().map_or(0, |g| g.row_coords(row % PAGE_ROWS).len()) + column.at(row).and_then(|(values, i)| values.geometry().map(|g| g.row_coords(i).len())).unwrap_or(0) } -/// Rows `rows` of `source`, by id, with their geometry from `column` as field `geometry`: what a -/// shapes job reads. -fn rows_of(source: &Table, column: &Column, rows: &[usize]) -> Table { - let ids: Vec = rows.iter().filter_map(|&row| source.ids.id(row)).collect(); +/// The geometry of rows `rows` of geometry column `column`, in that order. +fn geometry_of(column: &Column, rows: &[usize]) -> Values { let depth = column.kind().depth().expect("a geometry column"); - let mut geometry = GeometryBuilder::new(depth); - for &row in rows { - let page = &column.pages()[row / PAGE_ROWS]; - match page.values.geometry() { - Some(g) if page.values.is_valid(row % PAGE_ROWS) => geometry.push_from(g, row % PAGE_ROWS), - _ => geometry.push_empty(), + Values::of_rows( + depth, + rows.iter().map(|&row| { + let (values, i) = column.at(row).filter(|&(values, i)| values.is_valid(i))?; + Some((values.geometry()?, i)) + }), + ) +} + +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + use std::task::Waker; + + use super::*; + use crate::derive::pyramid::Pyramid; + use crate::derive::testing::Failing; + use crate::gpu::work::Counters; + use crate::{Areas, Data, Ids, Plane, Scene, Snapshot}; + + type Ring = Vec<[f64; 2]>; + + /// A unit square at `x`. + fn square(x: f64) -> Vec { + vec![vec![[x, 0.0], [x + 1.0, 0.0], [x + 1.0, 1.0], [x, 1.0], [x, 0.0]]] + } + + /// The shaped pages of the one paged table in `shapes`, with their ids, by page. + fn made(shapes: &Shapes) -> Vec<(u32, PageId)> { + let paged = shapes.paged(&"map".into(), &"Zone".into()); + let mut pages: Vec<(u32, PageId)> = paged[0].1.pages.iter().map(|(&page, made)| (page, made.id)).collect(); + pages.sort_unstable_by_key(|&(page, _)| page); + pages + } + + /// The derived column of the one table drawn whole in `shapes`. + fn derived(shapes: &Shapes) -> Column { + match shapes.tables().get(&("map".into(), "Zone".into())).and_then(|t| t.field_named("geometry.triangles")) { + Some(Field::Values(column, _)) => column.clone(), + _ => panic!("triangles"), + } + } + + #[test] + fn a_paged_table_shapes_only_the_pages_it_draws() { + let n = 3 * PAGE_ROWS as usize; + let data = Data::new("map"); + let squares: Vec> = (0..n).map(|i| square(i as f64)).collect(); + data.apply(Table::new("Zone", Ids::count(n as u32)).field("geometry", squares)).expect("valid"); + let scene = Scene::new(Plane::default()).draw(Areas::of("map", "Zone")); + // The order draws the rows backward. + let pyramid = Arc::new(Pyramid::new((0..n as u32).rev().collect(), Vec::new())); + let order = Order::Pyramid { pyramid, rows: n as u32 }; + let (counters, errors) = (Counters::default(), Arc::new(Mutex::new(Vec::new()))); + let mut shapes = Shapes::default(); + let update = |shapes: &mut Shapes, data: &Snapshot, pages: Vec| { + let mut paged = Keyed::default(); + paged.insert(("map".into(), "Zone".into()), Paged { order: order.clone(), pages, kept: Vec::new() }); + let input = Input { + scene: &scene, + scene_version: 1, + data: std::slice::from_ref(data), + host: None, + dragging: false, + paged: Some(&paged), + }; + let mut jobs = Jobs::new(None, Waker::noop(), &counters, &errors, u32::MAX); + shapes.update(&input, &TileSets::default(), &mut jobs); + }; + update(&mut shapes, &data.snapshot(), vec![1]); + let first = made(&shapes); + assert_eq!(first.iter().map(|&(page, _)| page).collect::>(), [1], "only the page drawn is shaped"); + let paged = shapes.paged(&"map".into(), &"Zone".into()); + let page = &paged[0].1.pages[&1]; + assert_eq!(page.shapes.len(), PAGE_ROWS as usize); + let triangles = page.shapes.geometry().expect("triangles").row_coords(0).len(); + assert!(triangles >= 3, "its rows' triangles, in draw order: {triangles} points"); + // Drawing another page shapes it alone; the first keeps its shapes. + update(&mut shapes, &data.snapshot(), vec![1, 2]); + let second = made(&shapes); + assert_eq!((second.len(), second[0]), (2, first[0])); + // A write to a row on page 2 reshapes only that page. + let row = u64::from(order.source(2 * PAGE_ROWS + 5)); + data.apply(Table::new("Zone", vec![row]).field("geometry", vec![square(-9.0)])).expect("valid"); + update(&mut shapes, &data.snapshot(), vec![1, 2]); + let third = made(&shapes); + assert_eq!(third[0], second[0], "page 1 has no written rows"); + assert_ne!(third[1].1, second[1].1, "page 2 is shaped again"); + } + + #[test] + fn a_whole_table_shapes_in_a_job_and_reshapes_only_written_rows() { + let n = 2 * PAGE_ROWS as usize + 10; + let data = Data::new("map"); + let squares: Vec> = (0..n).map(|i| square(i as f64)).collect(); + data.apply(Table::new("Zone", Ids::count(n as u32)).field("geometry", squares)).expect("valid"); + let scene = Scene::new(Plane::default()).draw(Areas::of("map", "Zone")); + let (counters, errors) = (Counters::default(), Arc::new(Mutex::new(Vec::new()))); + let mut shapes = Shapes::default(); + // No frame budget: every shape is made in a job, which finishes by the next update. + let update = |shapes: &mut Shapes, data: &Data| { + let data = data.snapshot(); + let input = Input { + scene: &scene, + scene_version: 1, + data: std::slice::from_ref(&data), + host: None, + dragging: false, + paged: None, + }; + let mut jobs = Jobs::new(None, Waker::noop(), &counters, &errors, 0); + shapes.update(&input, &TileSets::default(), &mut jobs) + }; + let points = |column: &Column, row: usize| { + column.at(row).map_or(0, |(values, i)| values.geometry().expect("triangles").row_coords(i).len()) + }; + assert!(update(&mut shapes, &data)); + assert!(shapes.busy(), "a job shapes them"); + assert_eq!(points(&derived(&shapes), 3), 0, "until the job finishes, rows draw nothing"); + assert!(update(&mut shapes, &data)); + let whole = derived(&shapes); + assert!(!shapes.busy()); + assert_eq!(whole.pages().len(), 3); + assert!((0..n).all(|row| points(&whole, row) >= 4), "every row holds its label point and triangles"); + // A write to one row reshapes only it: other pages keep their shapes, and its page keeps + // those of the rows not written. + let row = PAGE_ROWS as usize + 7; + data.apply(Table::new("Zone", vec![row as u64]).field("geometry", vec![square(-9.0)])).expect("valid"); + assert!(update(&mut shapes, &data)); + let written = derived(&shapes); + assert_eq!(written.pages()[0].id, whole.pages()[0].id); + assert_eq!(written.pages()[2].id, whole.pages()[2].id); + assert_eq!(points(&written, row), 0, "the written row draws nothing until its job finishes"); + assert!(points(&written, row + 1) >= 4, "the rest of its page keeps their shapes"); + assert!(update(&mut shapes, &data)); + assert!(points(&derived(&shapes), row) >= 4, "then its shapes arrive"); + } + + #[test] + fn a_failed_shaping_job_draws_its_rows_empty_and_runs_once() { + let data = Data::new("map"); + data.apply(Table::new("Zone", Ids::count(3)).field("geometry", vec![square(0.0), square(2.0), square(4.0)])) + .expect("valid"); + let data = data.snapshot(); + let scene = Scene::new(Plane::default()).draw(Areas::of("map", "Zone")); + let (host, counters, errors) = (Failing::default(), Counters::default(), Arc::new(Mutex::new(Vec::new()))); + let mut shapes = Shapes::default(); + for _ in 0..5 { + let input = Input { + scene: &scene, + scene_version: 1, + data: std::slice::from_ref(&data), + host: Some(&host), + dragging: false, + paged: None, + }; + // No frame budget: the rows are shaped in a job. + shapes.update( + &input, + &TileSets::default(), + &mut Jobs::new(Some(&host), Waker::noop(), &counters, &errors, 0), + ); } + assert_eq!(host.jobs(), 1, "one job, not one per frame"); + assert!(!shapes.busy(), "the failure finished the work"); + let column = derived(&shapes); + assert!((0..3).all(|row| column.at(row).is_some()), "every row has a shape, drawing nothing"); } - Table::new(source.name.clone(), ids).field("geometry", Values::of_geometry(geometry.finish())) } diff --git a/crates/latkit/src/derive/sight.rs b/crates/latkit/src/derive/sight.rs index 8d199c3a..be1f1908 100644 --- a/crates/latkit/src/derive/sight.rs +++ b/crates/latkit/src/derive/sight.rs @@ -1,19 +1,25 @@ -//! What the camera sees, as tiles: the one place derivations read the camera. Tiles split a -//! square of the space four ways a level, the earth's web mercator square or a square of a plane; a -//! tile is wanted where it meets what the camera shows, and splits while it draws larger than its -//! size where it lies nearest the camera's focus. +//! The camera's view as tiles. This is the only place derivations read the camera. +//! +//! Tiles split a square of the space four ways per level: the earth's Web Mercator square, or a +//! square of a plane. On the globe, the earth's square wraps at the antimeridian. On a flat map it +//! ends there, so the map is one world. A tile is needed where it meets the visible region. It +//! splits while it draws larger than its nominal size (`TILE_PX`) at its point nearest the camera's +//! focus. A tile's drawn size is the square root of the screen area it covers there, so a tile +//! foreshortened toward the globe's limb or a tilted horizon stays coarse. When a budget limits how +//! many tiles split, the tiles drawn largest split first. use glam::DVec3; +use crate::geo::mercator; use crate::tiles::TileKey; use crate::view::Projected; -/// CSS pixels a tile spans drawn at its own level. +/// CSS pixels a tile spans when drawn at its own level. pub(crate) const TILE_PX: f64 = 512.0; -/// The latitude web mercator reaches, north and south. -const MAX_LATITUDE: f64 = 85.051_128_779_806_59; +/// CSS pixels past which a tile splits: its nominal size with 25% slack. +pub(crate) const SPLIT_PX: f64 = TILE_PX * 1.25; -/// Where a space's tiles lie: the earth's web mercator square, or a square of a plane. +/// The square a space's tiles divide: the earth's Web Mercator square, or a square of a plane. #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) enum Domain { Earth, @@ -25,8 +31,25 @@ pub(crate) enum Domain { } impl Domain { - /// The square holding positions within `bounds` (west, south, east, north) on a plane, with a - /// little room so nothing lies on its edge. + /// The words a job writes the square as: zeros for the earth, else 1, its west and south edges, + /// and its size. + pub(crate) fn words(self) -> [f64; 4] { + match self { + Domain::Earth => [0.0; 4], + Domain::Plane { min, size } => [1.0, min[0], min[1], size], + } + } + + /// The square written as `words`. + pub(crate) fn from_words(words: [f64; 4]) -> Domain { + match words { + [0.0, ..] => Domain::Earth, + [_, x, y, size] => Domain::Plane { min: [x, y], size }, + } + } + + /// The plane square containing `bounds` (west, south, east, north), centered on them and 2% + /// larger so nothing lies on its edge. Inverted bounds give the unit square. pub(crate) fn plane(bounds: [f64; 4]) -> Domain { let [w, s, e, n] = if bounds[0] <= bounds[2] { bounds } else { [0.0, 0.0, 1.0, 1.0] }; let size = (e - w).max(n - s).max(1e-9) * 1.02; @@ -34,31 +57,25 @@ impl Domain { Domain::Plane { min: [center[0] - size / 2.0, center[1] - size / 2.0], size } } - /// Where position `p` lies across the square, 0 to 1 from the west and from the north. + /// Where position `p` lies in the square, 0 to 1 from the west and from the north. On the + /// earth, latitude is clamped to Web Mercator's range and longitude wraps. pub(crate) fn unit(&self, p: [f64; 2]) -> [f64; 2] { match *self { - Domain::Earth => { - let lat = p[1].clamp(-MAX_LATITUDE, MAX_LATITUDE).to_radians(); - let y = 0.5 - (std::f64::consts::FRAC_PI_4 + lat / 2.0).tan().ln() / std::f64::consts::TAU; - [((p[0] + 180.0) / 360.0).rem_euclid(1.0), y] - } + Domain::Earth => [((p[0] + 180.0) / 360.0).rem_euclid(1.0), mercator::down(p[1])], Domain::Plane { min, size } => [(p[0] - min[0]) / size, 1.0 - (p[1] - min[1]) / size], } } - /// The position at `u` across the square. + /// The position at `u` in the square. The inverse of `unit`. pub(crate) fn position(&self, u: [f64; 2]) -> [f64; 2] { match *self { - Domain::Earth => { - let lat = (std::f64::consts::PI * (1.0 - 2.0 * u[1])).sinh().atan().to_degrees(); - [u[0] * 360.0 - 180.0, lat] - } + Domain::Earth => [u[0] * 360.0 - 180.0, mercator::latitude_down(u[1])], Domain::Plane { min, size } => [min[0] + u[0] * size, min[1] + (1.0 - u[1]) * size], } } - /// West, south, east, and north of tile `key`, grown on every side by `grow` across the square, - /// within the square. + /// West, south, east, and north of tile `key` in position units, grown on every side by `grow` + /// (a fraction of the square) and clamped to the square. pub(crate) fn bounds(&self, key: TileKey, grow: f64) -> [f64; 4] { let n = f64::from(1_u32 << key.z); let (x, y) = (f64::from(key.x), f64::from(key.y)); @@ -72,20 +89,20 @@ impl Domain { } } -/// What the camera sees: where tiles are wanted and how large they draw. +/// The camera's view, for choosing tiles and measuring how large they draw. pub(crate) struct Sight<'a> { pub(crate) projected: &'a Projected, - /// The view's size, CSS pixels. + /// The view's size, in CSS pixels. pub(crate) size: [f64; 2], pub(crate) earth: bool, } -/// West, south, east, north boxes, in position units: what the camera shows, in one box, or on the -/// earth two where it wraps the antimeridian. +/// The visible region as west, south, east, north boxes in position units: one box, or on the +/// globe two boxes where the region crosses the antimeridian. pub(crate) type Region = Vec<[f64; 4]>; impl Sight<'_> { - /// Where position `p` lands on screen, if it shows. + /// Where position `p` appears on screen. `None` when it does not show. fn project(&self, p: [f64; 2]) -> Option<[f64; 2]> { let w = self.projected.world([p[0], p[1], 0.0]); if !self.facing(w) { @@ -96,12 +113,12 @@ impl Sight<'_> { /// The position on the ground under screen point `p`. fn ground(&self, p: [f64; 2]) -> Option<[f64; 2]> { - let w = self.projected.surface(p)?; + let w = self.projected.ground(p)?; let [x, y, _] = self.projected.position(w); Some([x, y]) } - /// The center of the globe the earth is, and its radius, in world units. + /// The globe's center and radius, in world units. fn sphere(&self) -> (DVec3, f64) { let radius = 1.0 / self.projected.center[1].to_radians().cos(); (DVec3::new(0.0, 0.0, -radius), radius) @@ -111,7 +128,8 @@ impl Sight<'_> { self.earth && self.projected.globe >= 0.5 } - /// Whether world point `w` faces the camera: always on a map, on the globe its near side. + /// Whether world point `w` faces the camera: always on a flat map, and only on the near side on + /// the globe. fn facing(&self, w: DVec3) -> bool { if !self.globe() { return true; @@ -120,14 +138,26 @@ impl Sight<'_> { (w - center).dot(self.projected.eye - w) > 0.0 } - /// Where tiles draw largest: where the camera looks, and tilted, the nearest ground it shows. + /// The part of `domain`'s square that one CSS pixel spans at the camera's scale: the earth's + /// square is a turn of longitude, and a plane's is `size` position units. + fn pixel(&self, domain: &Domain) -> f64 { + let square = match *domain { + Domain::Earth => std::f64::consts::TAU, + Domain::Plane { size, .. } => size, + }; + 1.0 / (square * self.projected.px_per_unit[0]).max(f64::MIN_POSITIVE) + } + + /// Where tiles draw largest: the camera's center, and the ground at the bottom middle of the + /// view, which is the nearest ground when the camera is tilted. pub(crate) fn focus(&self) -> Vec<[f64; 2]> { let center = [self.projected.center[0], self.projected.center[1]]; let near = self.ground([self.size[0] / 2.0, self.size[1]]); std::iter::once(center).chain(near).collect() } - /// The ground the camera shows, sampled on a grid of points across the view. + /// The visible ground, sampled on a grid of points across the view. On a flat map it ends at + /// the world's edges. On the globe it wraps the antimeridian. pub(crate) fn region(&self) -> Region { const SAMPLES: usize = 7; let globe = self.globe(); @@ -142,7 +172,7 @@ impl Sight<'_> { missed = true; continue; }; - // Longitudes kept beside the center's, so a view across the antimeridian is one span. + // Keep longitudes near the center's, so a view across the antimeridian is one span. let x = if globe { center[0] + turn(center[0], x) } else { x }; (w, e, s, n) = (w.min(x), e.max(x), s.min(y), n.max(y)); } @@ -152,7 +182,8 @@ impl Sight<'_> { return Vec::new(); } if missed { - // A tilted plane past its horizon: everything toward it. + // Some samples miss a tilted plane, which shows its horizon: grow the region by + // four times its size on each side to cover the ground toward it. let size = (e - w).max(n - s) * 4.0; return vec![[w - size, s - size, e + size, n + size]]; } @@ -176,19 +207,22 @@ impl Sight<'_> { if w > e { return vec![[-180.0, -90.0, 180.0, 90.0]]; } - if missed { - // A tilted map past its horizon: everything toward it. + // Some samples miss a tilted map, which shows its horizon: grow the region by four times + // its size on each side to cover the ground toward it. Otherwise pad it by 2%. + let (grow_x, grow_y) = if missed { let size = (e - w).max(n - s) * 4.0; - return boxes(w - size, e + size, s - size, n + size); - } - let (pad_x, pad_y) = ((e - w) * 0.02, (n - s) * 0.02); + (size, size) + } else { + ((e - w) * 0.02, (n - s) * 0.02) + }; if globe { - return boxes(w - pad_x, e + pad_x, s - pad_y, n + pad_y); + return boxes(w - grow_x, e + grow_x, s - grow_y, n + grow_y); } - vec![[(w - pad_x).max(-180.0), (s - pad_y).max(-90.0), (e + pad_x).min(180.0), (n + pad_y).min(90.0)]] + // A flat map is one world: the region ends at its edges rather than wrapping. + vec![[(w - grow_x).max(-180.0), (s - grow_y).max(-90.0), (e + grow_x).min(180.0), (n + grow_y).min(90.0)]] } - /// The cap of the globe the camera can see past its horizon. + /// The cap of the globe the camera can see, out to its horizon. fn cap(&self) -> Region { let (center, radius) = self.sphere(); let out = self.projected.eye - center; @@ -205,30 +239,50 @@ impl Sight<'_> { boxes(lon - half, lon + half, south, north) } - /// How large a tile of bounds `b` draws where it lies nearest each focus, and how far from the - /// view's middle that lands. - pub(crate) fn measure(&self, focus: &[[f64; 2]], b: [f64; 4]) -> (f64, f64) { - let [w, s, e, n] = b; - let (dx, dy) = ((e - w) / 64.0, (n - s) / 64.0); + /// How large tile `key` of `domain`'s square draws, in CSS pixels, and how far from the view's + /// center the measured point is. The size is measured at the tile's point nearest each focus, + /// and the largest is kept. It is the square root of the screen area the tile would cover at + /// the scale there: the area spanned by steps of at most a pixel along its two edges, scaled up + /// to the whole tile. So a tile foreshortened toward the globe's limb or a tilted horizon + /// measures less than its longer side. Steps are short because a longer step from a large tile + /// could pass the globe's horizon, where nothing projects, once the camera is close to the + /// ground. On a flat map seen from above, every tile of a level measures the same wherever it + /// lies. + pub(crate) fn measure(&self, focus: &[[f64; 2]], domain: &Domain, key: TileKey) -> (f64, f64) { + let side = 1.0 / f64::from(1_u32 << key.z); + let (west, north) = (f64::from(key.x) * side, f64::from(key.y) * side); + let step = (side / 64.0).min(self.pixel(domain)); let (mut size, mut distance) = (0.0_f64, f64::INFINITY); - for &[fx, fy] in focus { - let near = if self.earth { fx + 360.0 * ((w.midpoint(e) - fx) / 360.0).round() } else { fx }; - let (x, y) = (near.clamp(w, e - dx), fy.clamp(s, n - dy)); - let Some(p) = self.project([x, y]) else { continue }; - let reach = |q: Option<[f64; 2]>| q.map_or(0.0, |q| 64.0 * (q[0] - p[0]).hypot(q[1] - p[1])); - size = size.max(reach(self.project([x + dx, y]))).max(reach(self.project([x, y + dy]))); + for &f in focus { + let [fx, fy] = domain.unit(f); + let fx = match domain { + // The globe wraps: shift the focus a turn either way, whichever is nearer. + Domain::Earth if self.globe() => fx + (west + side / 2.0 - fx).round(), + // A flat map is one world: a focus past its edge stays there, far from the tiles + // across the antimeridian. + Domain::Earth => (f[0] + 180.0) / 360.0, + Domain::Plane { .. } => fx, + }; + let at = [fx.clamp(west, west + side - step), fy.clamp(north, north + side - step)]; + let Some(p) = self.project(domain.position(at)) else { continue }; + // Where a step along each edge moves on screen. A step past the globe's horizon counts + // as no move. + let moved = |q: [f64; 2]| self.project(domain.position(q)).map_or([0.0; 2], |q| [q[0] - p[0], q[1] - p[1]]); + let ([ax, ay], [bx, by]) = (moved([at[0] + step, at[1]]), moved([at[0], at[1] + step])); + size = size.max(side / step * (ax * by - ay * bx).abs().sqrt()); distance = distance.min((p[0] - self.size[0] / 2.0).hypot(p[1] - self.size[1] / 2.0)); } (size, distance) } } -/// The shortest turn from longitude `from` to `to`, in degrees. +/// The shortest signed turn from longitude `from` to `to`, in degrees. fn turn(from: f64, to: f64) -> f64 { (to - from + 540.0).rem_euclid(360.0) - 180.0 } -/// The boxes of a longitude range that may wrap, and a latitude range. +/// The boxes covering a longitude range that may wrap the antimeridian, and a latitude range +/// clamped to ±90°. fn boxes(west: f64, east: f64, south: f64, north: f64) -> Region { let (s, n) = (south.max(-90.0), north.min(90.0)); if east - west >= 360.0 { @@ -239,7 +293,7 @@ fn boxes(west: f64, east: f64, south: f64, north: f64) -> Region { if e <= 180.0 { vec![[w, s, e, n]] } else { vec![[w, s, 180.0, n], [-180.0, s, e - 360.0, n]] } } -/// Whether box `b` meets what the camera shows. +/// Whether box `b` meets the visible region. pub(crate) fn meets(region: &Region, b: [f64; 4]) -> bool { region.iter().any(|r| b[0] <= r[2] && b[2] >= r[0] && b[1] <= r[3] && b[3] >= r[1]) } @@ -249,7 +303,7 @@ mod tests { use super::*; #[test] - fn domains_place_positions_across_their_square() { + fn domains_map_positions_into_their_unit_square() { let earth = Domain::Earth; let u = earth.unit([0.0, 0.0]); assert!((u[0] - 0.5).abs() < 1e-12 && (u[1] - 0.5).abs() < 1e-12); @@ -257,8 +311,77 @@ mod tests { assert!((back[0] + 97.5).abs() < 1e-9 && (back[1] - 30.25).abs() < 1e-9); let plane = Domain::plane([0.0, 0.0, 10.0, 5.0]); let u = plane.unit([5.0, 2.5]); - assert!((u[0] - 0.5).abs() < 1e-12 && (u[1] - 0.5).abs() < 1e-12, "the middle of the data"); + assert!((u[0] - 0.5).abs() < 1e-12 && (u[1] - 0.5).abs() < 1e-12, "the middle of the data maps to 0.5"); let b = plane.bounds(TileKey { z: 1, x: 0, y: 0 }, 0.0); assert!(b[0] < 0.0 && b[3] > 5.0, "the northwest quarter: {b:?}"); } + + #[test] + fn tiles_under_the_focus_measure_their_size_on_the_globe_at_any_zoom() { + let globe = crate::Space::Earth(crate::Earth::default().projection(crate::Projection::Globe)); + let size = [800.0, 600.0]; + let center = [-74.0, 40.7]; + // Past zoom 12 the camera's horizon is nearer than a sixty-fourth of the root tile. + for zoom in [3.0, 8.0, 13.0, 18.0, 22.0] { + let camera = crate::Camera::Map { center: [center[0], center[1], 0.0], zoom, pitch: 0.0, bearing: 0.0 }; + let projected = Projected::new(&globe, &camera, size, [0.0, 0.0, size[0], size[1]], 1.0); + let sight = Sight { projected: &projected, size, earth: true }; + let [u, v] = Domain::Earth.unit(center); + for z in [0_u8, 6, 12] { + let n = f64::from(1_u32 << z); + let key = TileKey { z, x: (u * n) as u32, y: (v * n) as u32 }; + let (px, _) = sight.measure(&sight.focus(), &Domain::Earth, key); + // Under the center the globe is seen square on, so a tile measures its size there. + // A large tile measures where it is largest, which may lie south of the center, + // where a mercator step covers more of the globe. + let nominal = 256.0 * 2_f64.powf(zoom) / n; + assert!((0.95..1.3).contains(&(px / nominal)), "zoom {zoom}, level {z}: {px} px, not {nominal}"); + } + } + } + + #[test] + fn tiles_foreshortened_by_a_tilted_camera_measure_the_area_they_cover() { + let map = crate::Space::Earth(crate::Earth::default()); + let size = [800.0, 600.0]; + let center = [-74.0, 40.7]; + let [u, v] = Domain::Earth.unit(center); + let key = TileKey { z: 12, x: (u * 4096.0) as u32, y: (v * 4096.0) as u32 }; + let measured = |pitch: f64| { + let camera = crate::Camera::Map { center: [center[0], center[1], 0.0], zoom: 8.0, pitch, bearing: 0.0 }; + let projected = Projected::new(&map, &camera, size, [0.0, 0.0, size[0], size[1]], 0.0); + let sight = Sight { projected: &projected, size, earth: true }; + sight.measure(&sight.focus(), &Domain::Earth, key).0 + }; + // Tilted 60°, a tile under the center keeps its width on screen and draws half as tall, so + // its area as a side is about √½ of what it measures seen from above. + let ratio = measured(60.0) / measured(0.0); + assert!((0.65..0.8).contains(&ratio), "{ratio}"); + } + + #[test] + fn a_flat_map_ends_at_the_antimeridian() { + let map = crate::Space::Earth(crate::Earth::default()); + let size = [800.0, 600.0]; + let tile = |lon: f64| { + let [u, v] = Domain::Earth.unit([lon, 0.0]); + TileKey { z: 10, x: (u * 1024.0) as u32, y: (v * 1024.0) as u32 } + }; + let (near, across) = (tile(179.0), tile(-179.0)); + // Seen from above, and tilted to show the horizon, which grows the region several times + // over: on a globe that region would wrap past 180° onto the tiles across it. + for pitch in [0.0, 75.0] { + let camera = crate::Camera::Map { center: [179.0, 0.0, 0.0], zoom: 10.0, pitch, bearing: 0.0 }; + let projected = Projected::new(&map, &camera, size, [0.0, 0.0, size[0], size[1]], 0.0); + let sight = Sight { projected: &projected, size, earth: true }; + let region = sight.region(); + assert!(region.iter().all(|r| r[0] >= -180.0 && r[2] <= 180.0), "pitch {pitch}: {region:?}"); + assert!(meets(®ion, near.bounds()), "pitch {pitch}: {region:?}"); + assert!(!meets(®ion, across.bounds()), "pitch {pitch}: {region:?}"); + let focus = sight.focus(); + let ((_, beside), (_, beyond)) = + (sight.measure(&focus, &Domain::Earth, near), sight.measure(&focus, &Domain::Earth, across)); + assert!(beside < 1.0 && beyond > size[0], "pitch {pitch}: {beside} px and {beyond} px from the center"); + } + } } diff --git a/crates/latkit/src/derive/stars.rs b/crates/latkit/src/derive/stars.rs index 3a1c6830..c1945350 100644 --- a/crates/latkit/src/derive/stars.rs +++ b/crates/latkit/src/derive/stars.rs @@ -1,10 +1,15 @@ -//! Stars outside diagrams: a net's edges. Each row of an edge mark that `joins` a table is a net -//! joining the vertex rows whose reference field names it, drawn from their middle to each, as the -//! vertex mark drawing them places them. The star of each net row is a path field beside the net -//! table, which the mark draws as any geometry; it works again only when the data or scene change. +//! Stars of nets, outside diagrams. +//! +//! Each row of an edge mark with `joins` is a net. Its members are the vertex rows whose reference +//! field names it. The net draws as a star: a segment from the members' centroid to each member, at +//! the places their vertex mark gives them. Stars are a `route` path field beside the net table, +//! and the mark draws them like any geometry. They are recomputed only when the data, the scene, or +//! the layout's places change. + +use std::hash::Hash; use super::graph::{VertexMark, marks, position}; -use super::{Input, Tables, beside, kept}; +use super::{Input, Tables, beside, kept, stamp_with}; use crate::data::{Field, Name, UNWIRED, Values}; use crate::expr::cpu::Rows; use crate::{Mark, Space}; @@ -16,8 +21,8 @@ pub(crate) struct Stars { } impl Stars { - /// Brings the stars up to `input`, the members placed by their data or by `places`, the layout's - /// places beside their tables; whether a table changed. + /// Updates the stars to `input`. A member is at its data's position, else at its place in the + /// layout's `places` beside its table. Returns whether a table changed. pub(crate) fn update(&mut self, input: &Input<'_>, places: &Tables) -> bool { let joining = input.scene.marks.iter().any(|m| matches!(m, Mark::Edges(e) if e.joins.is_some())); if matches!(input.scene.space, Space::Diagram(_)) || !joining { @@ -27,10 +32,10 @@ impl Stars { } let field = |t: &crate::Table, name: &str| t.field_named(name).map(Field::version); let placed: Vec<_> = places.iter().map(|(key, t)| (key, field(t, "x"), field(t, "y"))).collect(); - let mut hasher = rustc_hash::FxHasher::default(); - std::hash::Hash::hash(&(input.scene_version, placed), &mut hasher); - input.structure(&mut hasher); - let stamp = std::hash::Hasher::finish(&hasher).max(1); + let stamp = stamp_with(|hasher| { + (input.scene_version, placed).hash(hasher); + input.structure(hasher); + }); if stamp == self.stamp { return false; } @@ -82,7 +87,7 @@ impl Stars { } } -/// Where member row `row` stands: as its data positions it, else where the layout placed it. +/// The position of member row `row`: its data's position, else the layout's place for it. fn place(vm: &VertexMark<'_>, rows: &Rows<'_>, row: usize, placed: Option<&crate::Table>) -> Option<[f64; 2]> { position(vm.mark, rows, row).or_else(|| { let table = placed?; @@ -93,7 +98,8 @@ fn place(vm: &VertexMark<'_>, rows: &Rows<'_>, row: usize, placed: Option<&crate }) } -/// A net's star: a path from the middle of `members` to each; none for fewer than two. +/// A net's star: a path from the centroid of `members` to each member. Empty for fewer than two +/// members. fn star(members: &[[f64; 2]]) -> Vec> { if members.len() < 2 { return Vec::new(); diff --git a/crates/latkit/src/derive/tiled.rs b/crates/latkit/src/derive/tiled.rs index e2071077..b25292b1 100644 --- a/crates/latkit/src/derive/tiled.rs +++ b/crates/latkit/src/derive/tiled.rs @@ -1,35 +1,38 @@ -//! Tables drawn in part. A table with a priority draws its rows in priority order, the highest on -//! top; one too large to draw whole draws the rows of the tiles the camera asks for, or where nothing -//! places its rows, those that matter most. What a view draws of such a dataset is a dataset of its -//! own, of the same name, ids, and origin: each such table's rows drawn in drawing order, with the -//! rows the edges drawn end at carried along and the rows pinned (selected, or framed by a fit) -//! kept; other tables as they are, their references naming the rows drawn. Marks draw it like any -//! other dataset, so picks, selection, hover, transitions, lanes, and labels follow it. +//! Tables drawn in a computed order: by priority, or by the visible tiles. //! -//! Each table's pyramid is worked out from where its rows lie and their priority: on the frame while -//! that fits its budget, else in a job of its own, so a dataset's tables are placed side by side and -//! each draws once its own lands; until then a table draws an even sample of its rows, as many as a -//! first look needs. A pyramid serves while the rows it was placed from hold. The rows drawn are laid -//! out again when the tiles asked for, the rows pinned, or the rows a table holds change; when only -//! values change, every row stays where it is drawn and each field writes only the rows whose values -//! changed, so a patch of one row costs one row. Only the fields marks read are gathered. +//! A table with a priority draws its rows in priority order, highest on top. A paged table (too +//! large to draw whole) draws the rows of the tiles the camera needs. Where nothing places its +//! rows, it draws the rows that matter most. Each draws in an [`Order`]: its pyramid's order, +//! computed from where its rows lie and their priority. The pyramid is computed on the frame when +//! that fits the budget, else in one job per dataset. Until it arrives, the table draws an even +//! sample of its rows first, enough for a first look, after the endpoint rows of sampled edges. +//! +//! A table and the table its edges end at both draw from their pyramids, or both from their +//! samples. A pyramid is used while the rows it was made from are unchanged. Rows appended since +//! draw after them, so appending never thins the drawn rows. Once the appended rows reach an eighth +//! of the pyramid's rows (at least a page), the pyramid is rebuilt. The camera needs ranges of a +//! table's order, so nothing is gathered as it moves. +//! +//! A table of vector tiles needs no pyramid. Its tiles are the vector tiles themselves. Its tile +//! set appends their rows as they arrive and chooses what it draws (`TileSets::choose`). +use std::ops::Range; use std::sync::Arc; -use rustc_hash::FxHashSet; - -use super::pyramid::{Leveling, Places, Placing, Pyramid, Wanted, answer, reaches}; +use super::order::{Lifted, Order}; +use super::pyramid::{Leveling, Places, Placing, Pyramid, answer}; use super::sight::{Domain, Sight}; -use super::{Jobs, stamp_of}; -use crate::data::{Data, Field, Keyed, Name, Named, RowIds, Table, UNWIRED, Version}; +use super::{Jobs, Working, stamp_of}; +use crate::data::{Column, Field, Keyed, Name, Named, Reference, Snapshot, Table, UNWIRED, Version}; +use crate::expr::Expr; use crate::expr::cpu::Rows; -use crate::expr::{Expr, Node}; +use crate::gpu::shaders::PAGE_ROWS; use crate::host::Job; -use crate::{Pending, Row}; +use crate::tiles::TileSets; -/// Microseconds placing a row takes, by estimate. +/// Estimated microseconds to place one row in a pyramid. const PLACE_COST: f64 = 1.0; -/// Rows a table drawn in part draws at most until its pyramid lands: a first look at all of it. +/// The most rows a table samples before its pyramid arrives: enough for a first look at all of it. const SAMPLE: u32 = 1 << 16; /// How a table draws. @@ -37,7 +40,7 @@ const SAMPLE: u32 = 1 << 16; pub(crate) enum Draw { /// Every row. Whole, - /// The rows of the tiles the camera asks for, as many as `rows` at most. + /// The rows of the tiles the camera needs, at most `rows`. Tiles { rows: u32 }, /// The `rows` rows that matter most. First { rows: u32 }, @@ -46,185 +49,182 @@ pub(crate) enum Draw { /// Where a table's rows lie, as the mark drawing them places them. #[derive(Clone, Debug, PartialEq)] pub(crate) enum Place { - /// Nowhere a camera's tiles reach: off a map or a plane. + /// Not in a space tiles cover: neither a map nor a plane. Nowhere, - /// At a point each: x and y. + /// At one point per row: x and y. Points([Expr; 2]), /// Along a geometry field. Geometry(Name), - /// Between the rows its reference fields name. + /// Between the rows its endpoint reference fields name. Ends(Vec), } -/// How one table draws, and what that reads. +/// How one table draws, with the inputs its order reads: priority and placement. #[derive(Clone, Debug, PartialEq)] pub(crate) struct Planned { pub(crate) draw: Draw, - /// Each row's priority, as the first mark giving one has it. + /// Each row's priority, from the first mark that gives one. pub(crate) priority: Option, pub(crate) place: Place, - /// The fields marks read of it, which its rows drawn hold with every reference field; `None` - /// where they hold every field. - pub(crate) reads: Option>, - /// The sampled fields marks trace across the axis, which its rows drawn hold whole: every frame, - /// at every level, where other sampled fields hold only the frames about the coordinate shown. - pub(crate) traced: FxHashSet, } impl Planned { - /// Whether the table draws as it is: whole, in its own order. + /// Whether the table draws whole in table order, needing no pyramid. pub(crate) fn as_is(&self) -> bool { self.draw == Draw::Whole && self.priority.is_none() } } -/// How each table drawn draws, by dataset and table. +/// How each drawn table draws, by dataset and table. pub(crate) type Plans = Keyed<(Name, Name), Planned>; -/// What drawing in part reads this frame. +/// This frame's inputs for tables drawn in a computed order. pub(crate) struct Ask<'a> { - pub(crate) sources: &'a [Data], + pub(crate) data: &'a [Snapshot], pub(crate) plans: &'a Plans, - /// What the camera sees, on a map or a plane. + /// The tile sets: datasets of vector tiles have no pyramids. + pub(crate) tiles: &'a TileSets, + /// The camera's view, on a map or a plane. pub(crate) sight: Option<&'a Sight<'a>>, pub(crate) earth: bool, - /// Rows drawn whatever the tiles: those selected, and those a fit frames. - pub(crate) pinned: &'a [Row], - /// The axis coordinate shown, which sampled fields drawn in part are gathered about. - pub(crate) at: f64, } -/// What each dataset drawn in part draws. +/// What a view draws of a paged table: its order, and the ranges of it to draw. Ranges are listed +/// in draw order, so later ranges draw on top. A pyramid's tiles draw finest first, so more +/// important rows draw on top. Vector tiles draw coarsest first, so finer tiles draw over the +/// ancestors standing in for them. `capped` is whether the plan's row budget dropped some of what +/// the camera needs. Only then does a smaller budget draw less of the table. +#[derive(Clone, Debug, PartialEq)] +pub(crate) struct Chosen { + pub(crate) order: Order, + pub(crate) ranges: Vec>, + pub(crate) capped: bool, +} + +/// What each paged table draws, by dataset and table. +pub(crate) type Choices = Keyed<(Name, Name), Chosen>; + +/// The pyramids of each dataset with paged tables. #[derive(Default)] pub(crate) struct Tiled { sets: Vec, } impl Tiled { - /// Brings what each dataset drawn in part draws up to `ask`: the datasets drawn that changed, and - /// the names of those no longer drawn in part, which draw as they are. - pub(crate) fn update(&mut self, ask: &Ask<'_>, jobs: &mut Jobs<'_>) -> (Vec, Vec) { + /// Updates each dataset's pyramids to `ask`, and chooses what each paged table draws, except + /// the tables of vector tiles. + pub(crate) fn update(&mut self, ask: &Ask<'_>, jobs: &mut Jobs<'_>) -> Choices { let mut wanted: Vec<&Name> = Vec::new(); for ((data, _), planned) in ask.plans { - if !planned.as_is() && !wanted.contains(&data) { + if !planned.as_is() && !wanted.contains(&data) && !ask.tiles.vector(data) { wanted.push(data); } } - let mut gone = Vec::new(); - self.sets.retain(|set| { - let keep = wanted.contains(&&set.name) && ask.sources.iter().any(|d| *d.name() == set.name); - if !keep { - gone.push(set.name.clone()); - } - keep - }); - let mut changed = Vec::new(); + self.sets.retain(|set| wanted.contains(&&set.name) && ask.data.iter().any(|d| *d.name() == set.name)); + let mut choices = Choices::default(); for name in wanted { - let Some(source) = ask.sources.iter().find(|d| d.name() == name) else { continue }; + let Some(source) = ask.data.iter().find(|d| d.name() == name) else { continue }; let i = self.sets.iter().position(|s| s.name == *name).unwrap_or_else(|| { self.sets.push(Set::new(name.clone())); self.sets.len() - 1 }); - if let Some(drawn) = self.sets[i].update(source, ask, jobs) { - changed.push(drawn); - } + self.sets[i].update(source, ask, jobs, &mut choices); } - (changed, gone) + choices } - /// Whether every table drawn in part draws from its pyramid, its rows in drawing order: none draws - /// its first sample while its pyramid is worked out, nor waits to be laid out again. + /// Whether every paged table draws from its pyramid: no pyramid job is pending and no table + /// draws its sample. pub(crate) fn complete(&self) -> bool { - self.sets.iter().all(|s| s.working.is_empty() && !s.deferred && !s.sampling && s.settled()) - } - - /// Whether a dataset waits for a frame: to ask for its pyramids, or to lay its rows out again in - /// drawing order once the camera rests. - pub(crate) fn due(&self) -> bool { - self.sets.iter().any(|s| s.deferred || !s.settled()) + self.sets.iter().all(|s| !s.working.busy() && !s.sampling) } - /// Whether dataset `name` draws in part. - pub(crate) fn draws(&self, name: &str) -> bool { - self.sets.iter().any(|s| *s.name == *name) - } - - /// The bounds of every row of table `table` of dataset `data` drawn through tiles, west, south, - /// east, north in position units: what a fit of everything frames, whatever tiles are drawn. + /// The bounds of every row of paged table `table` of dataset `data` (west, south, east, north, + /// in position units), regardless of the tiles drawn. A fit to everything frames them. pub(crate) fn bounds(&self, data: &Name, table: &Name) -> Option<[f64; 4]> { let set = self.sets.iter().find(|s| s.name == *data)?; set.placed.get(table).and_then(|placed| placed.bounds) } } -/// A table's pyramid: the rows it was placed from (the table's structure and order), the square it -/// tiles, the bounds of its rows in position units, and the version that tells it from any other. +/// A table's pyramid, with the table's order version it was made from, the square it tiles, and +/// the bounds of its rows in position units. struct Placed { - made: Version, - rows: (Version, Version), + order: Version, domain: Domain, - pyramid: Pyramid, + pyramid: Arc, bounds: Option<[f64; 4]>, } -/// A pyramid job under way: the table it places, where the answer holds its pyramid, and what the -/// pyramid is kept with once it lands. -struct Working { - table: Name, - pending: Pending>, - answered: u32, - placed: Placed, +impl Placed { + /// Whether it still serves `table`: the rows it was made from are unchanged, apart from rows + /// appended since. + fn serves(&self, table: &Table) -> bool { + self.order == table.order && table.len() >= self.pyramid.rows() as usize + } } -/// One table's rows drawn last time and where each lies, the versions of the table drawn while they -/// hold, and its fields as gathered with what each was gathered from. -#[derive(Default)] -struct Gathered { - /// Each row drawn: its source row, or a [`HOLE`]. - rows: Option>, - slots: Slots, - /// The rows written since the ids and fields kept were gathered, `(row, source row or HOLE)` by - /// row; `None` where every row was laid out afresh, and they gather again. - writes: Option>, - versions: Option<(Version, Version)>, - ids: Option, - fields: Named, - /// Each reference field's ids as gathered and resolved, as of a version of it, and of that version - /// and the versions the rows it names hold where they are drawn; and every row's target by it, as - /// of a version of it. - references: Named<(u64, u64, Field)>, - targets: Named<(Version, Arc<[u32]>)>, +/// What a dataset's pyramid job answers: the tables (each with its index among those asked, its +/// order version when asked, and its bounds) and the square the pyramids tile. +struct Pyramids { + tables: Vec<(u32, Name, Version, Option<[f64; 4]>)>, + domain: Domain, } -/// A field as gathered: a stamp of what it was gathered from, that field of the table, and the field -/// drawn. -struct Kept { - stamp: u64, - source: Field, - drawn: Field, +/// The inputs a table's pyramid was requested from, as last seen: its row count and order version, +/// a stamp of how it is placed, and the columns placing it. Also whether any of those rows was +/// written since. +struct Read { + rows: usize, + order: Version, + inputs: u64, + columns: Vec<(Name, Column)>, + written: bool, } -/// One dataset drawn in part. +impl Read { + fn new(table: &Table, planned: &Planned, earth: bool) -> Read { + let columns = fields_read(planned) + .into_iter() + .filter_map(|name| column_of(table, &name).map(|column| (name.clone(), column.clone()))) + .collect(); + Read { rows: table.len(), order: table.order, inputs: inputs_of(planned, earth), columns, written: false } + } + + /// Checks `table` again, setting `written` if a row it had was written since, or its rows were + /// reordered or removed. Once set, it stays set. + fn see(&mut self, table: &Table) { + if self.written { + return; + } + let rows = self.rows; + self.written = table.order != self.order + || table.len() < rows + || self.columns.iter_mut().any(|(name, then)| match column_of(table, name) { + Some(now) if now.kept_below(then, rows) => { + *then = now.clone(); + false + } + _ => true, + }); + } +} + +/// The pyramids of one dataset with paged tables. struct Set { name: Name, - /// The stamp of everything its pyramids are placed from: every table placed, and the square. + /// The stamp of everything its pyramids are placed from, as last seen. stamp: u64, - /// Each table's pyramid, as last placed. - placed: Named>, - working: Vec, - /// Whether asking for its pyramids waits for a frame that gathers nothing. - deferred: bool, - /// What decides which rows draw and where, and what they hold, as last gathered. - laid: u64, - held: u64, - /// Updates since its rows were laid out. - quiet: u32, - tables: Named, - /// Each table drawn in part as last laid out: its rows drawn, and the versions it holds while they - /// do. - rows: Named>, - versions: Named<(Version, Version)>, - /// Whether a table draws an even sample of its rows until its pyramid lands. + /// Each table's latest pyramid. + placed: Named, + /// The pyramid job, for the stamp it was asked at. + working: Working, + /// The inputs of the last pyramid request, by table. + read: Named, + /// Per table: the stamp of its inputs, the endpoint rows of the sampled edges ending at it + /// (lifted into its sample), and the step of its own sample. + lifted: Named<(u64, Lifted, u32)>, + /// Whether any table draws its sample because no pyramid serves it yet. sampling: bool, } @@ -234,677 +234,261 @@ impl Set { name, stamp: 0, placed: Named::default(), - working: Vec::new(), - deferred: false, - laid: 0, - held: 0, - quiet: 0, - tables: Named::default(), - rows: Named::default(), - versions: Named::default(), + working: Working::default(), + read: Named::default(), + lifted: Named::default(), sampling: false, } } - /// Whether every table's rows lie in drawing order: none waits to be laid out again. - fn settled(&self) -> bool { - self.tables.values().all(|g| g.slots.ordered || g.slots.rows.is_empty()) - } - - /// Brings the pyramids and the rows drawn up to `ask`; the dataset drawn, when it changed. - fn update(&mut self, source: &Data, ask: &Ask<'_>, jobs: &mut Jobs<'_>) -> Option { + /// Updates the pyramids to `ask`, and chooses what each paged table draws. + fn update(&mut self, source: &Snapshot, ask: &Ask<'_>, jobs: &mut Jobs<'_>, choices: &mut Choices) { + let landed = self.land(); let placing = placed(source, ask); - // Pyramids that landed serve from now on. - let mut i = 0; - while i < self.working.len() { - let Some(landed) = self.working[i].pending.take() else { - i += 1; - continue; - }; - let working = self.working.swap_remove(i); - if let Ok(bytes) = landed - && let Some(pyramid) = Pyramid::read(&bytes).remove(&working.answered) - { - self.placed - .insert(working.table, Arc::new(Placed { pyramid, made: Version::next(), ..working.placed })); - } - } let stamp = stamp(source, ask, &placing); - let due = !placing.is_empty() && stamp != self.stamp; - // Pyramids small enough are placed at once, before the rows drawn are gathered from them; - // asking jobs for larger ones waits for a frame that gathers nothing, so no frame does both. - let cost: f64 = placing - .iter() - .map(|(t, p)| { - t.len() as f64 * if matches!(p.draw, Draw::Tiles { .. }) { PLACE_COST } else { PLACE_COST / 20.0 } - }) - .sum(); - let at_once = cost <= jobs.left() as f64; - if due && at_once { - self.level(&placing, ask, stamp, jobs); - } - let drawn = self.gather(source, ask); - self.deferred = due && !at_once && drawn.is_some(); - if due && !at_once && drawn.is_none() { - self.level(&placing, ask, stamp, jobs); - } - drawn - } - - /// Asks for the pyramids of `placing`, from what `stamp` stamps: at once where one fits the - /// frame, else in a job of its own. - fn level(&mut self, placing: &[(&Table, &Planned)], ask: &Ask<'_>, stamp: u64, jobs: &mut Jobs<'_>) { - { + if !placing.is_empty() && (stamp != self.stamp || landed) { self.stamp = stamp; - // What was asked before no longer serves once it lands. - self.working.clear(); - let (leveling, bounds) = leveling(ask, placing); - let domain = leveling.domain; - for (i, table) in leveling.tables.iter().enumerate().filter(|(_, t)| t.answered) { - let name = placing[i].0.name.clone(); - // A table placed by its ends goes with the table they name, which only anchors it. - let mut alone = Leveling { domain, tables: Vec::with_capacity(2) }; - let mut table = table.clone(); - if let Some((target, _)) = table.ends.as_mut() { - let at = &leveling.tables[*target as usize]; - alone.tables.push(Placing { rows: at.rows, places: at.places.clone(), ..Placing::default() }); - *target = 0; + for (table, _) in &placing { + if let Some(read) = self.read.get_mut(&table.name) { + read.see(table); } - let answered = alone.tables.len() as u32; - let rows = (placing[i].0.structure, placing[i].0.order); - alone.tables.push(table); - // Placing a row takes about a microsecond; ordering one by priority alone, far less. - let per_row = if alone.tables[answered as usize].leveled { PLACE_COST } else { PLACE_COST / 20.0 }; - let cost = per_row * alone.tables.iter().map(|t| f64::from(t.rows)).sum::(); - let made = Version::next(); - let placed = - Placed { made, rows, domain, pyramid: Pyramid::default(), bounds: bounds.get(&name).copied() }; - if jobs.fits(cost as u64) { - let pyramid = Pyramid::answered(answer(&alone)).remove(&answered).unwrap_or_default(); - self.placed.insert(name, Arc::new(Placed { pyramid, ..placed })); - } else { - let job = Job::pyramid(&alone); - let pending = jobs.spawn(job); - self.working.push(Working { table: name, pending, answered, placed }); - } - } - } - } - - /// Table `name`'s pyramid while it serves: placed from the rows the table holds. - fn pyramid(&self, source: &Data, name: &Name) -> Option> { - let table = source.table(name)?; - self.placed.get(name).filter(|p| p.rows == (table.structure, table.order)).cloned() - } - - /// The dataset drawn, gathered again when what it is gathered from changed. - fn gather(&mut self, source: &Data, ask: &Ask<'_>) -> Option { - // Each table's tiles drawn, or how many of its rows, by what it draws. - let mut chosen: Vec<(Name, Choice)> = Vec::new(); - let mut serving: Vec<(Name, Arc)> = Vec::new(); - for table in source.tables() { - let Some(planned) = ask.plans.get(&(source.name().clone(), table.name.clone())) else { continue }; - if planned.as_is() { - continue; } - let placed = self.pyramid(source, &table.name); - if let Some(placed) = &placed { - serving.push((table.name.clone(), placed.clone())); + if !self.holds(&placing, ask.earth) { + self.level(&placing, ask, jobs); } - let pyramid = placed.as_deref().map(|placed| (&placed.pyramid, &placed.domain)); - let choice = match (pyramid, planned.draw) { - (None, Draw::Whole) => Choice::AsIs, - (None, Draw::Tiles { rows } | Draw::First { rows }) => { - Choice::Sample((table.len() as u32).div_ceil(rows.clamp(1, SAMPLE)).max(1)) - } - (Some(_), Draw::Whole) => Choice::Tiles(Wanted { tiles: vec![(0, true)], cover: None }), - (Some((pyramid, _)), Draw::First { rows }) => Choice::Last(rows.min(pyramid.order.len() as u32)), - (Some((pyramid, domain)), Draw::Tiles { rows }) => { - Choice::Tiles(ask.sight.map(|sight| pyramid.wanted(sight, domain, rows)).unwrap_or_default()) - } - }; - chosen.push((table.name.clone(), choice)); - } - self.sampling = chosen.iter().any(|(_, choice)| matches!(choice, Choice::Sample(_))); - let pinned: Vec<&Row> = ask.pinned.iter().filter(|r| r.data == self.name).collect(); - let frames: Vec> = sampled_frames(source, ask); - let pyramids: Vec = serving.iter().map(|(_, p)| p.made).collect(); - // Where rows draw, apart from what they hold: a patch writing values lays nothing out again. - let laid = stamp_of((pyramids, &chosen, &pinned, rows_of(source))); - let held = stamp_of((source.0.version, frames, reads_of(source, ask))); - if laid == self.laid { - // Once the camera rests, rows laid out of drawing order are laid out again in it. - self.quiet = self.quiet.saturating_add(1); - if held == self.held && (self.settled() || self.quiet < SETTLE) { - return None; - } - } else { - self.quiet = 0; - } - let afresh = self.quiet >= SETTLE && !self.settled(); - let relaid = laid != self.laid || afresh; - (self.laid, self.held) = (laid, held); - if !relaid { - return Some(self.rewrite(source, ask)); } - let serving: Named> = serving.into_iter().collect(); - Some(self.build(source, ask, &serving, &chosen, &pinned, afresh)) - } - - /// The dataset drawn when only what its rows hold changed: every row stays where it is drawn, and - /// each field writes only the rows whose values changed. - fn rewrite(&mut self, source: &Data, ask: &Ask<'_>) -> Data { - let (rows, versions) = (std::mem::take(&mut self.rows), std::mem::take(&mut self.versions)); - for name in rows.keys() { - if let Some(gathered) = self.tables.get_mut(name) { - gathered.writes = Some(Vec::new()); - } - } - let tables: Vec
= - source.tables().map(|table| self.drawn(source, table, ask, &rows, &versions)).collect(); - (self.rows, self.versions) = (rows, versions); - source.drawn(tables) - } - - /// The dataset drawn: every table's rows drawn, pinned rows kept, the ends of edges drawn carried - /// along, each table's rows laid out, afresh where `afresh` says so, and gathered. - fn build( - &mut self, - source: &Data, - ask: &Ask<'_>, - serving: &Named>, - chosen: &[(Name, Choice)], - pinned: &[&Row], - afresh: bool, - ) -> Data { - let mut wanted: Named> = chosen - .iter() - .filter_map(|(name, choice)| { - let len = source.table(name).map_or(0, Table::len) as u32; - Some((name.clone(), choice.rows(serving.get(name).map(Arc::as_ref), len)?)) + self.sampling = false; + let drawn: Vec<(&Table, &Planned)> = source + .tables() + .filter_map(|table| { + let planned = ask.plans.get(&(source.name().clone(), table.name.clone()))?; + (!planned.as_is()).then_some((table, planned)) }) .collect(); - // Rows pinned, and the rows the edges drawn end at, draw whatever their tiles. - let mut drawn: Named = wanted + // A table draws from its pyramid only while that pyramid serves it and so do the pyramids + // of the table its edges end at and of the edge tables ending at it, since edges and their + // endpoints are assigned levels together. + let ends: Vec> = drawn .iter() - .map(|(name, list)| (name.clone(), Bitset::of(list, source.table(name).map_or(0, Table::len)))) + .map(|(table, planned)| { + let to = ends_of(table, planned).filter(|to| **to != table.name)?; + drawn.iter().position(|(t, _)| t.name == *to) + }) .collect(); - for row in pinned { - let (Some(list), Some(table)) = (wanted.get_mut(&row.table), source.table(&row.table)) else { continue }; - if let Some(r) = table.row(row.id.as_ref()) - && drawn.get_mut(&row.table).is_some_and(|set| set.insert(r as u32)) - { - list.push(r as u32); - } - } - let mut carried: Vec<(Name, Vec)> = Vec::new(); - for table in source.tables() { - // Only the ends of edges and faces drawn are carried. - let Some(Place::Ends(ends)) = ask.plans.get(&(source.name().clone(), table.name.clone())).map(|p| &p.place) - else { - continue; - }; - for (name, field) in &table.fields { - let Field::Reference(reference) = field else { continue }; - if !wanted.contains_key(&reference.to) || !ends.contains(name) { - continue; - } - let targets = self.targets(&table.name, name, field); - let ends: Vec = match wanted.get(&table.name) { - Some(list) => list.iter().map(|&r| targets[r as usize]).collect(), - None => targets.to_vec(), - }; - carried.push((reference.to.clone(), ends)); - } - } - for (to, ends) in carried { - let (Some(list), Some(set)) = (wanted.get_mut(&to), drawn.get_mut(&to)) else { continue }; - for t in ends { - if t != UNWIRED && set.insert(t) { - list.push(t); - } - } - } - // Each table drawn in part: its rows laid out, and the versions it holds while they do. - let mut rows: Named> = Named::default(); - let mut versions: Named<(Version, Version)> = Named::default(); - for (name, list) in &wanted { - let gathered = self.tables.entry(name.clone()).or_default(); - let len = source.table(name).map_or(0, Table::len); - let writes = gathered.slots.lay(list, len, afresh); - let laid: Arc<[u32]> = match (&writes, &gathered.rows) { - (Some(writes), Some(before)) if writes.is_empty() => before.clone(), - _ => gathered.slots.rows.as_slice().into(), - }; - // Rows written in place keep the order the table's rows are drawn in, as every row still - // drawn keeps its place; rows laid out afresh draw in another. - let held = match (&writes, gathered.versions) { - (Some(writes), Some(held)) if writes.is_empty() => held, - (Some(_), Some((order, _))) => (order, Version::next()), - _ => { - let v = Version::next(); - (v, v) - } - }; - gathered.versions = Some(held); - versions.insert(name.clone(), held); - if writes.is_none() { - gathered.ids = None; - gathered.fields.clear(); - gathered.references.clear(); - } - gathered.writes = writes.map(|w| { - w.iter().map(|&(at, r)| (at as usize, if r == HOLE { crate::data::HOLE } else { r as usize })).collect() - }); - gathered.rows = Some(laid.clone()); - rows.insert(name.clone(), laid); - } - let mut tables = Vec::with_capacity(source.tables().count()); - for table in source.tables() { - tables.push(self.drawn(source, table, ask, &rows, &versions)); - } - (self.rows, self.versions) = (rows, versions); - source.drawn(tables) - } - - /// Every row's target by reference field `name` of table `table`, `field`, as of its version. - fn targets(&mut self, table: &Name, name: &Name, field: &Field) -> Arc<[u32]> { - let Field::Reference(reference) = field else { return Arc::from([]) }; - let gathered = self.tables.entry(table.clone()).or_default(); - if let Some((version, targets)) = gathered.targets.get(name) - && *version == reference.version + let mut served: Vec = + drawn.iter().map(|(table, _)| self.placed.get(&table.name).is_some_and(|p| p.serves(table))).collect(); + while let Some((i, j)) = + ends.iter().enumerate().find_map(|(i, end)| end.filter(|&j| served[i] != served[j]).map(|j| (i, j))) { - return targets.clone(); - } - let targets: Arc<[u32]> = reference.targets().into(); - gathered.targets.insert(name.clone(), (reference.version, targets.clone())); - targets - } - - /// Table `table` as it is, its references naming the rows drawn of the tables drawn in part, each - /// holding `versions` while they do: resolved again only when the reference or where the rows it - /// names are drawn changed. - fn remapped(&mut self, table: &Table, versions: &Named<(Version, Version)>) -> Table { - let mut out = table.clone(); - for (name, field) in &table.fields { - let Field::Reference(reference) = field else { continue }; - let Some(&(order, structure)) = versions.get(&reference.to) else { continue }; - let whole = stamp_of((reference.version, order, structure)); - let kept = self.tables.get(&table.name).and_then(|g| g.references.get(name)); - if let Some((_, _, field)) = kept.filter(|(_, w, _)| *w == whole) { - out.insert(name.clone(), field.clone()); - continue; - } - let targets = self.targets(&table.name, name, field); - let map = &self.tables[&reference.to].slots.at; - let mapped: Vec = targets.iter().map(|&t| remap(map, t)).collect(); - let kept = self.tables.get(&table.name).and_then(|g| g.references.get(name)).map(|(_, _, f)| f.clone()); - let kept = kept.as_ref().and_then(|f| if let Field::Reference(r) = f { Some(r) } else { None }); - let remapped = reference.resolved_to(reference.ids.clone(), &mapped, order, structure, kept); - let field = Field::Reference(remapped); - let gathered = self.tables.entry(table.name.clone()).or_default(); - gathered.references.insert(name.clone(), (stamp_of(reference.version), whole, field.clone())); - out.insert(name.clone(), field); - } - out - } - - /// Table `table` as drawn: as it is, its references naming the rows drawn; or its rows drawn - /// in part, with the fields marks read. Rows written in place write the ids and fields kept, - /// touching only their pages, and so do rows whose values changed; rows laid out afresh gather - /// them again. What nothing changed under is kept as it is. - fn drawn( - &mut self, - source: &Data, - table: &Table, - ask: &Ask<'_>, - rows: &Named>, - versions: &Named<(Version, Version)>, - ) -> Table { - // A table no mark draws stays as it is: only joins read it, by id. - let planned = ask.plans.contains_key(&(source.name().clone(), table.name.clone())); - let remaps = table.fields.iter().any(|(_, f)| matches!(f, Field::Reference(r) if rows.contains_key(&r.to))); - let Some(list) = rows.get(&table.name) else { - return if remaps && planned { self.remapped(table, versions) } else { table.clone() }; - }; - let plan = ask.plans.get(&(source.name().clone(), table.name.clone())); - let reads = plan.and_then(|p| p.reads.as_ref()); - let traced = |name: &Name| plan.is_some_and(|p| p.traced.contains(name)); - // The source row of each row drawn, picked only where something gathers afresh. - let picks = std::cell::OnceCell::new(); - let picked = || { - picks.get_or_init(|| { - list.iter().map(|&r| if r == HOLE { crate::data::HOLE } else { r as usize }).collect::>() - }) - }; - let (order, structure) = versions[&table.name]; - let gathered = self.tables.get_mut(&table.name).expect("made with its rows"); - let writes = gathered.writes.take(); - let len = list.len(); - let ids = match (&gathered.ids, writes.as_deref()) { - (Some(ids), Some([])) => ids.clone(), - (Some(ids), Some(writes)) => ids.write(len, writes, &table.ids).expect("ids of one kind"), - _ => table.ids.gather(picked()), - }; - gathered.ids = Some(ids.clone()); - let mut out = - Table { name: table.name.clone(), ids, index: Arc::default(), fields: Vec::new(), structure, order }; - // Fields not read now miss the rows written: gathered again once read. - if writes.as_ref().is_some_and(|w| !w.is_empty()) { - gathered.fields.retain(|name, _| reads.is_none_or(|r| r.contains(name))); - } - let frames = |samples: &crate::data::Samples| samples.frame_at(ask.at); - for (name, field) in &table.fields { - let read = reads.is_none_or(|r| r.contains(name)); - let gathered = self.tables.get_mut(&table.name).expect("made with its rows"); - let kept = gathered.fields.get(name); - let (stamp, made) = match field { - Field::Values(column, version) if read => { - let stamp = stamp_of(version); - let made = match (kept, writes.as_deref()) { - (Some(kept), Some([])) if kept.stamp == stamp => kept.drawn.clone(), - // The rows laid out in place, and the rows whose values changed since the field - // kept was gathered, wherever they are drawn; a field now of another kind - // gathers afresh. - ( - Some(Kept { stamp: was, source: Field::Values(before, _), drawn: Field::Values(drawn, _) }), - Some(laid), - ) if before.kind() == column.kind() => { - // What the pages remember was written, else what differs: a row written - // the value it held is written again, cheaper than comparing every row. - let changed = if *was == stamp { - Vec::new() - } else { - column.rows_since(before).unwrap_or_else(|| column.differing(before)) - }; - let at = &gathered.slots.at; - let mut written: Vec<(usize, usize)> = changed - .into_iter() - .filter_map(|r| at.get(r).filter(|&&a| a != UNWIRED).map(|&a| (a as usize, r))) - .collect(); - written.extend_from_slice(laid); - crate::order::by_key(&mut written, |w| w.0 as u64); - written.dedup_by_key(|w| w.0); - Field::Values(drawn.write(len, &written, column).expect("one storage"), Version::next()) - } - _ => Field::Values(column.gather(picked()), Version::next()), + (served[i], served[j]) = (false, false); + } + let (sample, steps) = samples(&drawn, &ends); + let mut lifted: Named<(u64, Lifted, u32)> = Named::default(); + for (i, (table, planned)) in drawn.iter().enumerate() { + let key = (source.name().clone(), table.name.clone()); + let len = table.len() as u32; + let placed = self.placed.get(&table.name).filter(|_| served[i]); + let chosen = match (placed, planned.draw) { + (None, Draw::Whole) => continue, + (None, Draw::Tiles { .. } | Draw::First { .. }) => { + self.sampling = true; + // The endpoint rows of sampled edges draw first, as in a pyramid. + let edges: Vec = (0..drawn.len()).filter(|&e| ends[e] == Some(i)).collect(); + let (rows, step) = if edges.is_empty() { + (Lifted::default(), steps[i]) + } else { + let was = self.lifted.get(&table.name); + let made = lift(&drawn, &edges, &steps, i, sample[i], was); + let chosen = (made.1.clone(), made.2); + lifted.insert(table.name.clone(), made); + chosen }; - (stamp, made) + let order = Order::Sampled { rows: len, step, lifted: rows }; + let sampled = order.sampled(); + Chosen { order, ranges: std::iter::once(0..sampled).collect(), capped: sampled < len } } - Field::Sampled(samples, version) if read => { - // A trace draws every frame; else only the frames about the coordinate shown. - let frame = if traced(name) { None } else { frames(samples) }; - let stamp = stamp_of((version, frame, traced(name))); - let made = match (kept, writes.as_deref()) { - (Some(kept), Some([])) if kept.stamp == stamp => kept.drawn.clone(), - _ if traced(name) => Field::Sampled(samples.gather(picked()), Version::next()), - _ => { - let span = frame.map_or(0..0, |f| f..f + 2); - Field::Sampled(samples.gather_frames(picked(), span), Version::next()) + (Some(placed), draw) => { + let pyramid = &placed.pyramid; + let made = pyramid.rows(); + let (ranges, capped) = match draw { + Draw::Whole => (std::iter::once(0..len).collect(), false), + // The rows that matter most are last in the order. Appended rows draw too. + Draw::First { rows } => (std::iter::once(made - rows.min(made)..len).collect(), rows < made), + Draw::Tiles { rows } => { + let (mut wanted, capped) = + ask.sight.map(|s| pyramid.wanted(s, &placed.domain, rows)).unwrap_or_default(); + // Finest first, so more important rows draw last, on top. + crate::order::by_key(&mut wanted, |(z, range)| { + !crate::order::pair(u32::from(*z), !range.start) + }); + let mut ranges: Vec> = wanted.into_iter().map(|(_, range)| range).collect(); + if len > made { + ranges.push(made..len); + } + (ranges, capped) } }; - (stamp, made) + Chosen { order: Order::Pyramid { pyramid: pyramid.clone(), rows: len }, ranges, capped } } - Field::Reference(reference) => { - // Where the rows it names are drawn: as laid out where drawn in part, else as they are. - let held = versions.get(&reference.to).copied(); - let named = held.unwrap_or_else(|| { - source.table(&reference.to).map_or((order, structure), |t| (t.order, t.structure)) - }); - let resolved = - self.reference(&table.name, name, field, list, writes.as_deref(), (held.is_some(), named)); - out.insert(name.clone(), resolved); - continue; - } - _ => continue, }; - let gathered = self.tables.get_mut(&table.name).expect("made with its rows"); - gathered.fields.insert(name.clone(), Kept { stamp, source: field.clone(), drawn: made.clone() }); - out.insert(name.clone(), made); - } - out + choices.insert(key, chosen); + } + self.lifted = lifted; + } + + /// Whether the last request still serves `placing`: the same tables, placed the same way, with + /// no rows written since except appended ones. With no job pending, each table's pyramid must + /// also serve it, with fewer rows appended since than an eighth of the rows it was made from + /// (at least a page). A pending job keeps running while only rows were appended, so appends + /// never starve the pyramids. Once it finishes, its pyramids serve until that many rows are + /// appended. + fn holds(&self, placing: &[(&Table, &Planned)], earth: bool) -> bool { + placing.len() == self.read.len() + && placing.iter().all(|(table, planned)| { + let Some(read) = self.read.get(&table.name) else { return false }; + if read.written || read.inputs != inputs_of(planned, earth) { + return false; + } + let served = planned.as_is() || self.placed.get(&table.name).is_some_and(|p| p.serves(table)); + let appended = table.len().saturating_sub(read.rows); + self.working.busy() || (served && appended < (read.rows / 8).max(PAGE_ROWS as usize)) + }) } - /// Reference field `name` of table `table`, `field`, for the rows `list` drawn, resolved to where - /// the rows it names are drawn: `named` says whether those are drawn in part, and the versions they - /// hold. Kept as it is where no row was written here, the reference is the same, and the rows it - /// names hold; else its ids are written where rows were written in place. - fn reference( - &mut self, - table: &Name, - name: &Name, - field: &Field, - list: &[u32], - writes: Option<&[(usize, usize)]>, - (in_part, (order, structure)): (bool, (Version, Version)), - ) -> Field { - let Field::Reference(reference) = field else { return field.clone() }; - let stamp = stamp_of(reference.version); - let whole = stamp_of((reference.version, order, structure)); - let gathered = self.tables.entry(table.clone()).or_default(); - if let Some((_, _, field)) = gathered.references.get(name).filter(|(_, w, _)| *w == whole) - && writes.is_some_and(<[_]>::is_empty) - { - return field.clone(); - } - let targets = self.targets(table, name, field); - let gathered = self.tables.entry(table.clone()).or_default(); - let kept = gathered.references.get(name).filter(|(s, _, _)| *s == stamp).map(|(_, _, f)| f.clone()); - let kept = kept.as_ref().and_then(|f| if let Field::Reference(r) = f { Some(r) } else { None }); - let len = list.len(); - let ids = match (kept, writes) { - (Some(kept), Some(writes)) => kept.ids.write(len, writes, &reference.ids).expect("ids of one kind"), - (Some(kept), None) if kept.ids.len() == len => kept.ids.clone(), - _ => { - let picked: Vec = - list.iter().map(|&r| if r == HOLE { crate::data::HOLE } else { r as usize }).collect(); - reference.ids.gather(&picked) + /// Takes the pyramids of a finished job. Returns true when a job succeeded. A failed job is not + /// retried until its inputs change. + fn land(&mut self) -> bool { + let Some((_, asked, Ok(bytes))) = self.working.landed() else { return false }; + for (index, pyramid) in Pyramid::read(&bytes) { + if let Some((_, name, order, bounds)) = asked.tables.iter().find(|t| t.0 == index) { + let placed = + Placed { order: *order, domain: asked.domain, pyramid: Arc::new(pyramid), bounds: *bounds }; + self.placed.insert(name.clone(), placed); } - }; - let target = self.tables.get(&reference.to).filter(|_| in_part); - let mapped: Vec = list + } + true + } + + /// Requests the pyramids of `placing`: computed at once when they fit the frame's budget, else + /// in one job for the dataset. + fn level(&mut self, placing: &[(&Table, &Planned)], ask: &Ask<'_>, jobs: &mut Jobs<'_>) { + // Drop any pending job. Its answer would no longer serve. + self.working.cancel(); + self.read = + placing.iter().map(|(table, planned)| (table.name.clone(), Read::new(table, planned, ask.earth))).collect(); + let (leveling, bounds) = leveling(ask, placing); + // Placing a row takes about a microsecond. Ordering one by priority alone takes a + // twentieth of that. + let cost: f64 = leveling + .tables .iter() - .map(|&r| match (r, target) { - (HOLE, _) => UNWIRED, - (r, Some(target)) => remap(&target.slots.at, targets[r as usize]), - (r, None) => targets[r as usize], - }) + .map(|t| f64::from(t.rows) * if t.leveled { PLACE_COST } else { PLACE_COST / 20.0 }) + .sum(); + let tables: Vec<(u32, Name, Version, Option<[f64; 4]>)> = placing + .iter() + .enumerate() + .filter(|(i, _)| leveling.tables[*i].answered) + .map(|(i, (table, _))| (i as u32, table.name.clone(), table.order, bounds.get(&table.name).copied())) .collect(); - let resolved = Field::Reference(reference.resolved_to(ids, &mapped, order, structure, kept)); - let gathered = self.tables.entry(table.clone()).or_default(); - gathered.references.insert(name.clone(), (stamp, whole, resolved.clone())); - resolved - } -} - -/// Rows, a bit each: which rows of a table are drawn. -struct Bitset(Vec); - -impl Bitset { - /// The rows of `list`, of a table of `len` rows. - fn of<'a>(list: impl IntoIterator, len: usize) -> Bitset { - let mut set = Bitset(vec![0; len.div_ceil(64)]); - for &r in list { - set.insert(r); - } - set - } - - /// Whether row `r` is there. - fn contains(&self, r: u32) -> bool { - self.0.get(r as usize / 64).is_some_and(|word| word >> (r % 64) & 1 == 1) - } - - /// Adds row `r`; whether it was not there before. - fn insert(&mut self, r: u32) -> bool { - let Some(word) = self.0.get_mut(r as usize / 64) else { return false }; - let bit = 1 << (r % 64); - let new = *word & bit == 0; - *word |= bit; - new - } -} - -/// What a table drawn in part draws of its pyramid. -#[derive(Clone, Debug, PartialEq, Hash)] -enum Choice { - /// Every row as it is, until its pyramid lands. - AsIs, - /// Every row a step apart, until its pyramid lands. - Sample(u32), - /// The rows of these tiles. - Tiles(Wanted), - /// The last rows of its one tile: those that matter most. - Last(u32), -} - -impl Choice { - /// The rows it draws of a table of `len` rows, by `placed`, its pyramid, in drawing order: finest - /// tiles first, so what matters more draws on top. `None` for a table drawn as it is. - fn rows(&self, placed: Option<&Placed>, len: u32) -> Option> { - Some(match (self, placed) { - (Choice::AsIs, _) => return None, - (Choice::Tiles(wanted), Some(placed)) => { - let pyramid = &placed.pyramid; - let mut tiles = wanted.tiles.clone(); - crate::order::by_key(&mut tiles, |&(t, _)| { - !crate::order::pair(u32::from(pyramid.tiles[t as usize].key.z), u32::MAX - t) - }); - let mut list = Vec::new(); - for (t, finest) in tiles { - let range = pyramid.tiles[t as usize].rows.clone(); - let rows = &pyramid.order[range.start as usize..range.end as usize]; - // A coarser tile draws only its rows reaching where the finest cover. - match wanted.cover.filter(|_| !finest && !pyramid.anchors.is_empty()) { - Some(cover) => list.extend(rows.iter().copied().filter(|&r| { - let extent = pyramid.extents.get(r as usize).copied().unwrap_or(0); - reaches(&cover, pyramid.anchors[r as usize], extent) - })), - None => list.extend_from_slice(rows), - } + let domain = leveling.domain; + if jobs.fits(cost as u64) { + for (index, pyramid) in answer(&leveling) { + if let Some((_, name, order, bounds)) = tables.iter().find(|t| t.0 == index) { + let placed = Placed { order: *order, domain, pyramid: Arc::new(pyramid), bounds: *bounds }; + self.placed.insert(name.clone(), placed); } - list } - (Choice::Last(n), Some(placed)) => { - let order = &placed.pyramid.order; - order[order.len() - *n as usize..].to_vec() - } - (Choice::Sample(step), _) => (0..len).step_by(*step as usize).collect(), - (_, None) => Vec::new(), - }) + return; + } + self.working.start(self.stamp, jobs.spawn(Job::pyramid(&leveling)), Pyramids { tables, domain }); } } -/// A row drawn that no source row fills: a hole a row no longer drawn left. -const HOLE: u32 = u32::MAX; -/// Updates without change after which rows laid out of drawing order are laid out again in order. -const SETTLE: u32 = 8; - -/// Where a table drawn in part draws each row: a row still drawn keeps its place while the camera -/// moves, so only arriving rows are gathered and evaluated. A row no longer drawn leaves a hole, which -/// a row arriving later takes, the first hole first, else a row past the last. The rows are laid out -/// afresh, in drawing order, when holes pass a quarter of them, and once the camera rests while they -/// lie out of that order. -#[derive(Default)] -struct Slots { - /// Each row drawn: its source row, or a [`HOLE`]. - rows: Vec, - /// Where each source row is drawn, `UNWIRED` where it is not. - at: Vec, - /// Holes, by row. - holes: Vec, - /// Whether the rows lie in drawing order with no hole, as laying out afresh lays them. - ordered: bool, +/// Each table's sample size until its pyramid arrives (its plan's row budget, at most `SAMPLE`), +/// and its sample step. The endpoints of sampled edges count toward their target table's sample +/// (per `ends`), so edges sample sparsely enough that their endpoints fill at most half the +/// target's sample. The target's own sample fills the rest. +fn samples(drawn: &[(&Table, &Planned)], ends: &[Option]) -> (Vec, Vec) { + let sample: Vec = drawn + .iter() + .map(|(table, planned)| match planned.draw { + Draw::Tiles { rows } | Draw::First { rows } => rows.clamp(1, SAMPLE), + Draw::Whole => table.len() as u32, + }) + .collect(); + let steps = drawn + .iter() + .enumerate() + .map(|(e, (table, planned))| { + let ends = ends[e].filter(|&i| planned.draw != Draw::Whole && drawn[i].1.draw != Draw::Whole); + let room = ends.map_or(u32::MAX, |i| { + let references = match &planned.place { + Place::Ends(fields) => fields.len().max(1) as u32, + _ => 1, + }; + (sample[i] / 2 / references).max(1) + }); + (table.len() as u32).div_ceil(sample[e].min(room)).max(1) + }) + .collect(); + (sample, steps) } -impl Slots { - /// Lays out `wanted`, rows of a table of `len` rows, in drawing order: each still drawn keeps its - /// place, each no longer drawn leaves a hole, and each arriving takes the first hole, else a row - /// past the last. Every row is laid out afresh where `afresh` says so or holes would pass a quarter - /// of the rows. The rows written, `(row, source row or HOLE)` by row; `None` where every row was - /// laid out afresh. - fn lay(&mut self, wanted: &[u32], len: usize, afresh: bool) -> Option> { - self.at.resize(len, UNWIRED); - if afresh || self.rows.is_empty() { - return self.afresh(wanted); - } - let keep = Bitset::of(wanted, len); - let mut writes: Vec<(u32, u32)> = Vec::new(); - let mut freed: Vec = Vec::new(); - for (i, slot) in self.rows.iter_mut().enumerate() { - if *slot != HOLE && !keep.contains(*slot) { - if let Some(at) = self.at.get_mut(*slot as usize) { - *at = UNWIRED; +/// The rows of table `drawn[at]` where the sampled rows of edge tables `edges` end, lifted into its +/// sample of `sample` rows. Each edge table samples every `steps[e]`th row. Also returns the step +/// of the target's own even sample of the rest. Returns `was` when its input stamp is unchanged, +/// else recomputes. The result starts with that stamp. +fn lift( + drawn: &[(&Table, &Planned)], + edges: &[usize], + steps: &[u32], + at: usize, + sample: u32, + was: Option<&(u64, Lifted, u32)>, +) -> (u64, Lifted, u32) { + let target = drawn[at].0; + let references = |e: usize| -> Vec<&Reference> { + let (table, planned) = drawn[e]; + let Place::Ends(fields) = &planned.place else { return Vec::new() }; + fields + .iter() + .filter_map(|f| match table.field_named(f) { + Some(Field::Reference(r)) if r.to == target.name => Some(r), + _ => None, + }) + .collect() + }; + let from = stamp_of(( + target.len(), + sample, + edges + .iter() + .map(|&e| (drawn[e].0.len(), steps[e], references(e).iter().map(|r| r.version).collect::>())) + .collect::>(), + )); + if let Some(was) = was.filter(|(had, ..)| *had == from) { + return was.clone(); + } + let mut rows: Vec = Vec::new(); + for &e in edges { + let references = references(e); + let len = drawn[e].0.len() as u32; + for k in 0..len.div_ceil(steps[e]) { + for reference in &references { + let t = reference.target_row((k * steps[e]) as usize); + if t != UNWIRED { + rows.push(t); } - *slot = HOLE; - freed.push(i as u32); - writes.push((i as u32, HOLE)); } } - if !freed.is_empty() { - self.holes.extend(freed); - self.holes.sort_unstable(); - } - let mut taken = 0; - for &source in wanted { - if self.at.get(source as usize).is_none_or(|&at| at != UNWIRED) { - continue; - } - let row = if let Some(&hole) = self.holes.get(taken) { - taken += 1; - hole - } else { - self.rows.push(HOLE); - self.rows.len() as u32 - 1 - }; - self.rows[row as usize] = source; - self.at[source as usize] = row; - writes.push((row, source)); - } - self.holes.drain(..taken); - if 4 * self.holes.len() > self.rows.len() { - return self.afresh(wanted); - } - // A row left and taken again writes what takes it. - crate::order::by_key(&mut writes, |w| u64::from(w.0)); - writes.dedup_by(|later, earlier| { - let same = later.0 == earlier.0; - if same { - earlier.1 = later.1; - } - same - }); - self.ordered = self.holes.is_empty() && self.rows == wanted; - Some(writes) - } - - /// Lays every row out afresh, in drawing order, with no hole. - fn afresh(&mut self, wanted: &[u32]) -> Option> { - for &old in self.rows.iter().filter(|&&r| r != HOLE) { - if let Some(at) = self.at.get_mut(old as usize) { - *at = UNWIRED; - } - } - for (row, &source) in wanted.iter().enumerate() { - if let Some(at) = self.at.get_mut(source as usize) { - *at = row as u32; - } - } - self.rows = wanted.to_vec(); - self.holes.clear(); - self.ordered = true; - None } + let (lifted, step) = Lifted::within(rows, target.len() as u32, sample); + (from, lifted, step) } -/// The row `t` is drawn at, by `map`. -fn remap(map: &[u32], t: u32) -> u32 { - map.get(t as usize).copied().unwrap_or(UNWIRED) -} - -/// The tables of `source` a pyramid places, each with its plan: every table drawn in part, and every -/// table the ends of one drawn through tiles name, which only anchors them. -fn placed<'a>(source: &'a Data, ask: &'a Ask<'_>) -> Vec<(&'a Table, &'a Planned)> { +/// The tables of `source` a pyramid places, each with its plan: every table not drawn as is, and +/// every table that a tiled edge table's endpoints reference, placed only to anchor those edges. +fn placed<'a>(source: &'a Snapshot, ask: &'a Ask<'_>) -> Vec<(&'a Table, &'a Planned)> { let plan = |table: &Table| ask.plans.get(&(source.name().clone(), table.name.clone())); let mut out: Vec<(&Table, &Planned)> = Vec::new(); for table in source.tables() { @@ -916,13 +500,13 @@ fn placed<'a>(source: &'a Data, ask: &'a Ask<'_>) -> Vec<(&'a Table, &'a Planned out.push((table, planned)); } } - // Tables an edge table's ends name come before it. + // Tables an edge table's endpoints reference come before it. crate::order::by_key(&mut out, |(_, p)| u64::from(matches!(p.place, Place::Ends(_)))); out } -/// The table the ends of `table` name, as `planned` places it. -fn ends_of<'t>(table: &'t Table, planned: &Planned) -> Option<&'t Name> { +/// The table that `table`'s endpoint references point to, when `planned` places it by endpoints. +pub(crate) fn ends_of<'t>(table: &'t Table, planned: &Planned) -> Option<&'t Name> { let Place::Ends(fields) = &planned.place else { return None }; match table.field_named(fields.first()?)? { Field::Reference(reference) => Some(&reference.to), @@ -931,94 +515,60 @@ fn ends_of<'t>(table: &'t Table, planned: &Planned) -> Option<&'t Name> { } /// The stamp of everything a dataset's pyramids are placed from. -fn stamp(source: &Data, ask: &Ask<'_>, placed: &[(&Table, &Planned)]) -> u64 { - let rows: Vec<_> = placed.iter().map(|(t, _)| (t.name.clone(), t.structure, t.order)).collect(); - let mut inputs = Vec::new(); - for (table, planned) in placed { - let mut fields = Vec::new(); - if let Some(priority) = &planned.priority { - fields_of(priority, &mut fields); - } - match &planned.place { - Place::Points(xy) => xy.iter().for_each(|e| fields_of(e, &mut fields)), - Place::Geometry(field) => fields.push(field.clone()), - Place::Ends(ends) => fields.extend(ends.iter().cloned()), - Place::Nowhere => {} - } - let versions: Vec<_> = fields.iter().map(|f| table.field_named(f).map(Field::version)).collect(); - let tiles = matches!(planned.draw, Draw::Tiles { .. }); - inputs.push((format!("{:?}{:?}", planned.priority, planned.place), versions, tiles, planned.as_is())); - } - stamp_of((source.name(), rows, inputs, ask.earth)) +fn stamp(source: &Snapshot, ask: &Ask<'_>, placed: &[(&Table, &Planned)]) -> u64 { + let tables: Vec<_> = placed + .iter() + .map(|(table, planned)| { + let versions: Vec<_> = + fields_read(planned).iter().map(|f| table.field_named(f).map(Field::version)).collect(); + (inputs_of(planned, ask.earth), table.structure, table.order, versions) + }) + .collect(); + stamp_of((source.name(), tables)) } -/// The fields `expr` reads of its mark's own table. -fn fields_of(expr: &Expr, out: &mut Vec) { - match &*expr.0 { - Node::Field(f) if f.of.is_none() => out.push(f.name.clone()), - Node::Unary(_, a) | Node::Is(a, _) => fields_of(a, out), - Node::Binary(_, a, b) => { - fields_of(a, out); - fields_of(b, out); - } - Node::When(c, a, b) => { - fields_of(c, out); - fields_of(a, out); - fields_of(b, out); - } - Node::Scale { input, .. } | Node::Colormap { input, .. } => fields_of(input, out), - _ => {} - } +/// A stamp of what a pyramid for `planned` depends on besides its rows: how rows are placed and +/// prioritized, whether they draw by tile or as is, and whether the space is the earth. +fn inputs_of(planned: &Planned, earth: bool) -> u64 { + let tiles = matches!(planned.draw, Draw::Tiles { .. }); + stamp_of((format!("{:?}{:?}", planned.priority, planned.place), tiles, planned.as_is(), earth)) } -/// The frame shown of each sampled field marks read, and do not trace, of a table drawn in part. -fn sampled_frames(source: &Data, ask: &Ask<'_>) -> Vec> { - let mut frames = Vec::new(); - for table in source.tables() { - let Some(planned) = ask.plans.get(&(source.name().clone(), table.name.clone())) else { continue }; - if planned.as_is() { - continue; - } - for (name, field) in &table.fields { - if let Field::Sampled(samples, _) = field - && planned.reads.as_ref().is_none_or(|r| r.contains(name)) - && !planned.traced.contains(name) - { - frames.push(samples.frame_at(ask.at)); - } - } +/// The fields of its own table a pyramid for `planned` reads: those of its priority and its place. +fn fields_read(planned: &Planned) -> Vec { + let mut fields = Vec::new(); + if let Some(priority) = &planned.priority { + fields_of(priority, &mut fields); } - frames + match &planned.place { + Place::Points(xy) => xy.iter().for_each(|e| fields_of(e, &mut fields)), + Place::Geometry(field) => fields.push(field.clone()), + Place::Ends(ends) => fields.extend(ends.iter().cloned()), + Place::Nowhere => {} + } + fields } -/// What decides which of `source`'s rows draw and where, beside the tiles and the rows pinned: each -/// table's rows and their order, and the rows its references name. -fn rows_of(source: &Data) -> Vec<(Version, Version, Vec)> { - source - .tables() - .map(|table| { - let references = table.fields.iter().filter(|(_, f)| matches!(f, Field::Reference(_))); - (table.structure, table.order, references.map(|(_, f)| f.version()).collect()) - }) - .collect() +/// The column a pyramid reads for field `name` of `table`: its values, or a reference's target ids. +fn column_of<'t>(table: &'t Table, name: &Name) -> Option<&'t Column> { + match table.field_named(name)? { + Field::Values(column, _) => Some(column), + Field::Reference(reference) => Some(&reference.ids), + Field::Sampled(..) => None, + } } -/// The fields read of each table drawn in part, in a stable order; `None` for every field. -fn reads_of(source: &Data, ask: &Ask<'_>) -> Vec>> { - source - .tables() - .filter_map(|t| ask.plans.get(&(source.name().clone(), t.name.clone()))) - .map(|p| { - p.reads.as_ref().map(|reads| { - let mut reads: Vec = reads.iter().cloned().collect(); - reads.sort(); - reads - }) - }) - .collect() +/// The fields `expr` reads of its mark's own table. +fn fields_of(expr: &Expr, out: &mut Vec) { + expr.for_each_field(&mut |f| { + if f.from.is_none() { + out.push(f.name.clone()); + } + }); } -/// The box every placed row of `places` lies in, west, south, east, north; inverted when none is. +/// The box (west, south, east, north) around every placed row of `places`. Inverted when no row +/// is placed. fn bounds_of(places: &Places) -> [f64; 4] { let mut b = [f64::INFINITY, f64::INFINITY, f64::NEG_INFINITY, f64::NEG_INFINITY]; let mut take = |r: [f64; 4]| { @@ -1034,8 +584,8 @@ fn bounds_of(places: &Places) -> [f64; 4] { b } -/// Where `table`'s rows lie, as `place` puts them. -fn locate(table: &Table, place: &Place, all: &[Data]) -> Places { +/// Where `table`'s rows lie, per `place`: a point or a geometry's box per row. +fn locate(table: &Table, place: &Place, all: &[Snapshot]) -> Places { match place { Place::Points([x, y]) => { let rows = Rows::new(table, all); @@ -1065,15 +615,16 @@ fn locate(table: &Table, place: &Place, all: &[Data]) -> Places { } } -/// What to ask pyramid jobs for `placed`, in the square they tile, and each table's bounds. +/// The pyramid job input for `placed`, in the square they tile, and each table's bounds. fn leveling(ask: &Ask<'_>, placed: &[(&Table, &Planned)]) -> (Leveling, Named<[f64; 4]>) { let names: Vec = placed.iter().map(|(t, _)| t.name.clone()).collect(); let mut tables: Vec = placed .iter() .map(|(table, planned)| { - // Tables drawn through tiles are placed; one only an edge table's ends name anchors them. + // Tiled tables are placed, and so are tables drawn as is, which are here only to anchor + // the tiled edges ending at them. let anchors = matches!(planned.draw, Draw::Tiles { .. }) || planned.as_is(); - let places = if anchors { locate(table, &planned.place, ask.sources) } else { Places::None }; + let places = if anchors { locate(table, &planned.place, ask.data) } else { Places::None }; Placing { rows: table.len() as u32, answered: !planned.as_is(), places, ..Placing::default() } }) .collect(); @@ -1105,7 +656,7 @@ fn leveling(ask: &Ask<'_>, placed: &[(&Table, &Planned)]) -> (Leveling, Named<[f placing.leveled = matches!(planned.draw, Draw::Tiles { .. }) && (!matches!(placing.places, Places::None) || placing.ends.is_some()); if let Some(priority) = &planned.priority { - placing.priority = Rows::new(table, ask.sources).numbers(priority, |p| p as f32).into(); + placing.priority = Rows::new(table, ask.data).numbers(priority, |p| p as f32).into(); } } (Leveling { domain, tables }, bounds) @@ -1114,30 +665,44 @@ fn leveling(ask: &Ask<'_>, placed: &[(&Table, &Planned)]) -> (Leveling, Named<[f #[cfg(test)] mod tests { use super::*; + use crate::{Data, Ids, Patch}; + + #[test] + fn a_sample_lifts_the_endpoints_of_its_sampled_edges() { + // A thousand vertices, and a thousand edges between vertices far apart in the table. + let n = 1000_u64; + let data = Data::new("g"); + data.apply( + Patch::new().set(Table::new("V", Ids::count(n as u32)).field("x", vec![0.0; n as usize])).set( + Table::new("E", Ids::count(n as u32)) + .reference("a", "V", (0..n).map(|e| (e * 7) % n).collect::>()) + .reference("b", "V", (0..n).map(|e| (e * 13 + 1) % n).collect::>()), + ), + ) + .expect("valid"); + let data = data.snapshot(); + let (vertices, edges) = (data.table("V").expect("set"), data.table("E").expect("set")); + let tiles = |place| Planned { draw: Draw::Tiles { rows: 100 }, priority: None, place }; + let (at, ends) = (tiles(Place::Nowhere), tiles(Place::Ends(vec!["a".into(), "b".into()]))); + let drawn = [(vertices, &at), (edges, &ends)]; + let (from, lifted, step) = lift(&drawn, &[1], &[10, 10], 0, 300, None); + let order = Order::Sampled { rows: n as u32, step, lifted: lifted.clone() }; + // Every sampled edge ends at rows the vertices' sample draws. + for e in (0..n).step_by(10) { + for t in [(e * 7) % n, (e * 13 + 1) % n] { + assert!(order.rank(t as u32) < order.sampled(), "vertex {t}, an endpoint of edge {e}, draws first"); + } + } + assert!(order.sampled() <= 300, "lifted rows count toward the sample's 300 rows"); + let again = lift(&drawn, &[1], &[10, 10], 0, 300, Some(&(from, lifted, step))); + assert_eq!(again.0, from, "computed once while its edges are unchanged"); + } #[test] - fn rows_still_drawn_keep_their_places_and_arriving_ones_fill_holes() { - let mut slots = Slots::default(); - let first: Vec = (0..20).collect(); - assert_eq!(slots.lay(&first, 100, false), None, "laid out afresh"); - assert!(slots.ordered); - // Rows 4..8 go: their places are holes, every other row stays. - let kept: Vec = (0..4).chain(8..20).collect(); - let writes = slots.lay(&kept, 100, false).expect("in place"); - assert_eq!(writes, (4..8).map(|r| (r, HOLE)).collect::>()); - assert_eq!((slots.rows[3], slots.rows[8], slots.at[9]), (3, 8, 9)); - assert!(!slots.ordered, "holes lie among the rows"); - // Rows 50..55 arrive: four take the holes, the fifth goes past the last row. - let more: Vec = kept.iter().copied().chain(50..55).collect(); - let writes = slots.lay(&more, 100, false).expect("in place"); - assert_eq!(writes, [(4, 50), (5, 51), (6, 52), (7, 53), (20, 54)]); - assert_eq!(slots.lay(&more, 100, false), Some(vec![]), "nothing changed, nothing written"); - // A row leaving and another arriving in one go: the place left is taken at once. - let swapped: Vec = more.iter().copied().filter(|&r| r != 10).chain([60]).collect(); - assert_eq!(slots.lay(&swapped, 100, false), Some(vec![(10, 60)])); - // Holes past a quarter of the rows lay every row out afresh, in order. - assert_eq!(slots.lay(&[60, 61], 100, false), None); - assert_eq!((slots.rows.as_slice(), slots.ordered, slots.at[10]), (&[60, 61][..], true, UNWIRED)); - assert_eq!(slots.lay(&[60, 61], 100, true), None, "and so does asking for it"); + fn a_pyramid_reads_every_field_its_priority_tests() { + let priority = crate::when(crate::field("name").contains("lake")).then(crate::field("size")).otherwise(0.0); + let x = [crate::field("x"), crate::field("y")]; + let planned = Planned { draw: Draw::Tiles { rows: 100 }, priority: Some(priority), place: Place::Points(x) }; + assert_eq!(fields_read(&planned), ["name", "size", "x", "y"].map(Name::from)); } } diff --git a/crates/latkit/src/derive/tiles/frames.rs b/crates/latkit/src/derive/tiles/frames.rs deleted file mode 100644 index 46dcc9d7..00000000 --- a/crates/latkit/src/derive/tiles/frames.rs +++ /dev/null @@ -1,242 +0,0 @@ -//! Time tiles: sampled fields' histories fetched in spans as a view asks for them, from the files -//! [`Data::to_tiles`](crate::Data::to_tiles) writes. Every field wants its coarsest spans, which stand -//! in for any span still loading; a line tracing it wants the spans of the level of detail its plot -//! window draws at, middle first; a mark showing it at the view's coordinate wants the frames about -//! that coordinate. A span drawn puts its entries in the dataset; one no longer drawn lets them go. - -use std::ops::Range; -use std::sync::Arc; - -use rustc_hash::{FxHashMap, FxHashSet}; - -use super::super::Jobs; -use super::super::tiling::{Decoding, Requested, Source, Tiling}; -use super::Reading; -use crate::data::{ - Data, Field, Listed, MAX_LEVEL, Manifest, Name, Op, Patch, Run, Samples, bucket_frames, detail, sides, -}; -use crate::host::Request; -use crate::tiles::Span; -use crate::{Error, Pending}; - -/// Spans a line wants at most, middle first: those its plot window draws need far fewer. -const MAX_SPANS: usize = 64; - -/// Sampled fields fetched in spans, drawn as a dataset the view owns. -pub(crate) struct Frames { - manifest: Manifest, - /// Where each span lies, resolved against the manifest's URL. - url: String, - /// The dataset without its sampled fields' entries, until it lands. - base: Option>>, - /// Per field listed, its spans. - fields: Vec, - failed: bool, -} - -/// One field's spans: fetched and decoded, and those drawn with the runs each put in. -#[derive(Default)] -struct Spans { - tiling: Tiling, - drawn: FxHashMap>, -} - -impl Frames { - /// The fields `manifest`, found at `url`, lists; their dataset is fetched at once. - pub(crate) fn open(url: &str, manifest: Manifest, jobs: &Jobs<'_>) -> Frames { - let base = jobs.fetch(&Request::new(crate::url::resolve(url, &manifest.data))); - Frames { - url: crate::url::resolve(url, &manifest.url), - base: Some(base), - fields: manifest.fields.iter().map(|_| Spans::default()).collect(), - manifest, - failed: false, - } - } - - /// Spans drawn. - pub(crate) fn drawn(&self) -> usize { - self.fields.iter().map(|f| f.drawn.len()).sum() - } - - /// Fetches and decodes under way. - pub(crate) fn under_way(&self) -> usize { - usize::from(self.base.is_some()) + self.fields.iter().map(|f| f.tiling.under_way()).sum::() - } - - /// Brings `data`, dataset `name`, up to what `reading` reads of it: whether it changed, whether a - /// span stands in, and a failure to tell. - pub(crate) fn update( - &mut self, - name: &Name, - data: &mut Data, - reading: &Reading, - jobs: &Jobs<'_>, - ) -> (bool, bool, Option) { - let mut changed = false; - if let Some(pending) = &self.base { - let Some(landed) = pending.take() else { return (false, true, None) }; - self.base = None; - match landed.and_then(|bytes| Data::from_bytes(&bytes)) { - Ok(base) => { - *data = base.renamed(name.clone()); - changed = true; - } - Err(error) => { - self.failed = true; - return (false, false, Some(error)); - } - } - } - if self.failed { - return (false, false, None); - } - let (mut standing_in, mut failure) = (false, None); - for (k, listed) in self.manifest.fields.iter().enumerate() { - let reads = reading.reads(name, &listed.table, &listed.field); - let spans = &mut self.fields[k]; - // Spans drawn may tell more finely where the window lies, which may want finer spans: ask - // again until what is wanted settles, so what stands in is never taken for what draws. - let (mut asked, mut standing) = (Vec::new(), false); - for _ in 0..=MAX_LEVEL { - let wanted = - wanted(&listed.frames, sampled(data, listed), reads, reading, self.manifest.chunk, listed.levels); - if wanted == asked { - break; - } - let drawn = spans.tiling.update(&wanted, 0, &mut At { url: &self.url, field: k }, jobs); - standing = drawn.standing_in; - failure = failure.or(drawn.failure); - changed |= spans.draw(&drawn.keys, &listed.table, &listed.field, data); - asked = wanted; - } - standing_in |= standing; - } - (changed, standing_in, failure) - } -} - -impl Spans { - /// Makes `data`'s field `field` of `table` hold the entries of spans `drawn` and of no other: - /// whether it changed. - fn draw(&mut self, drawn: &[Span], table: &Name, field: &Name, data: &mut Data) -> bool { - let gone: Vec = self.drawn.keys().filter(|s| !drawn.contains(s)).copied().collect(); - let arrived: Vec<(Span, &Patch)> = drawn - .iter() - .filter(|s| !self.drawn.contains_key(s)) - .filter_map(|s| Some((*s, self.tiling.decoded(s, 0)?))) - .collect(); - if gone.is_empty() && arrived.is_empty() { - return false; - } - let mut put: Vec<(Span, Vec)> = Vec::new(); - let drawn_runs = &mut self.drawn; - let changed = data.sampled(table, field, |samples| { - for span in &gone { - for run in drawn_runs.remove(span).unwrap_or_default() { - samples.remove(&run); - } - } - for (span, tile) in &arrived { - put.push((*span, held(tile, table, field, samples))); - } - true - }); - if changed { - self.drawn.extend(put); - } - changed - } -} - -/// Puts the runs of field `field` of `table` that `tile` holds in `samples`: those it put. -fn held(tile: &Patch, table: &Name, field: &Name, samples: &mut Samples) -> Vec { - let mut runs = Vec::new(); - for op in &tile.ops { - let Op::Frames { table: t, field: f, run, at, values } = op else { continue }; - if t != table || f != field { - continue; - } - let per_row = run.len() * sides(run.level); - let rows = values.len().checked_div(per_row).unwrap_or(0); - if samples.insert(run, at, values, rows).is_ok() { - runs.push(run.clone()); - } - } - runs -} - -/// The samples `data` holds of the field `listed` names. -fn sampled<'a>(data: &'a Data, listed: &Listed) -> Option<&'a Samples> { - match data.table(&listed.table)?.field_named(&listed.field)? { - Field::Sampled(samples, _) => Some(samples), - _ => None, - } -} - -/// The spans of a field reading `frames` a view wants, nearest first: those of its coarsest level -/// holding any entry always; where a mark shows it, `(_, true)`, the frames about the coordinate shown; -/// where a line traces it, `(true, _)`, those of the level of detail the plot window draws at, middle -/// first. Where the window lies among the frames, what is held tells, as finely as it is held. -fn wanted( - frames: &Range, - samples: Option<&Samples>, - (traced, shown): (bool, bool), - reading: &Reading, - chunk: u32, - levels: u32, -) -> Vec { - // The entries of each level a history of `frames` holds: whole buckets only. - let chunk = u64::from(chunk); - let entries = |level: u32| frames.start >> (3 * level)..frames.end >> (3 * level); - // The spans of `level` holding its entries over frames `window`. - let spans = |level: u32, window: &Range| { - let (held, span) = (entries(level), bucket_frames(level)); - let first = (window.start / span).max(held.start); - let end = window.end.div_ceil(span).min(held.end); - (first / chunk..end.div_ceil(chunk)).map(move |index| Span { level: level as u8, index }) - }; - let top = (0..=levels).rev().find(|&level| !entries(level).is_empty()).unwrap_or(0); - let mut wanted: Vec = spans(top, frames).take(MAX_SPANS).collect(); - let held = samples.filter(|s| !s.is_empty()); - if shown && let Some(samples) = held { - let frame = samples.bound(reading.at, true).saturating_sub(1); - wanted.extend(spans(0, &(frame..frame.saturating_add(2)))); - } - if traced { - let window = match (held, reading.across) { - (Some(samples), Some(across)) => samples.across(across), - _ => frames.clone(), - }; - let level = detail(window.end - window.start, reading.columns).min(top); - let (held, span) = (entries(level), bucket_frames(level)); - let first = (window.start / span).max(held.start) / chunk; - let end = window.end.div_ceil(span).min(held.end).div_ceil(chunk); - // Choose nearest spans before allocating: an absolute-index range may be enormous and sparse. - let count = end.saturating_sub(first); - let middle = first + count.saturating_sub(1) / 2; - wanted.extend((0..count.min(MAX_SPANS as u64)).map(|offset| { - let index = if offset % 2 == 0 { middle - offset / 2 } else { middle + offset.div_ceil(2) }; - Span { level: level as u8, index } - })); - } - let mut seen = FxHashSet::default(); - wanted.retain(|span| seen.insert(*span)); - wanted -} - -/// Where a field's spans lie: the field's place in the manifest, in the spans' URL template. -struct At<'a> { - url: &'a str, - field: usize, -} - -impl Source for At<'_> { - fn request(&mut self, span: Span, _: &Jobs<'_>) -> Requested { - Requested::Fetch(Request::new(Manifest::span(self.url, self.field, u32::from(span.level), span.index))) - } - - fn decode(&self, _: Span, bytes: &Arc>, _: &Jobs<'_>) -> Decoding { - Decoding::Now(Patch::from_bytes(bytes).unwrap_or_default()) - } -} diff --git a/crates/latkit/src/derive/tiles/mod.rs b/crates/latkit/src/derive/tiles/mod.rs deleted file mode 100644 index f6b11f7b..00000000 --- a/crates/latkit/src/derive/tiles/mod.rs +++ /dev/null @@ -1,263 +0,0 @@ -//! Tiles: what a view fetches in pieces as it asks for them, kept as datasets the view owns, which -//! marks draw like any other. A source's first bytes say what it holds: a manifest of time tiles, -//! whose spans of sampled fields the plot window and the shown coordinate ask for; or a PMTiles -//! archive, a TileJSON document, or a URL template, which hold vector tiles the camera asks for (the -//! `geo` part). - -mod frames; -mod vector; - -use super::sight::Sight; -use super::{Input, Jobs}; -use crate::data::{Data, Manifest, Name}; -use crate::host::Request; -use crate::{Error, Pending, Tiles}; - -/// What a view reads of sampled fields: the plot window lines trace across and the pixel columns it -/// spans, the coordinate fields are shown at, and each field marks read, by dataset, table, and -/// name, with whether a line traces it. -#[derive(Default)] -pub(crate) struct Reading { - pub(crate) across: Option<[f64; 2]>, - pub(crate) columns: u32, - pub(crate) at: f64, - pub(crate) fields: Vec<(Name, Name, Name, bool)>, -} - -impl Reading { - /// Whether a line traces field `field` of `table` in dataset `data`, and whether a mark shows it - /// at the coordinate. - fn reads(&self, data: &str, table: &str, field: &str) -> (bool, bool) { - let mut reads = (false, false); - for (d, t, f, traced) in &self.fields { - if d == data && t == table && f == field { - if *traced { - reads.0 = true; - } else { - reads.1 = true; - } - } - } - reads - } -} - -/// What a source holds, once its first bytes said. -enum Holds { - Frames(Box), - Vector(Box), -} - -impl Holds { - /// Pieces drawn. - fn drawn(&self) -> usize { - match self { - Holds::Frames(frames) => frames.drawn(), - Holds::Vector(vector) => vector.drawn(), - } - } - - /// Fetches and decodes under way. - fn under_way(&self) -> usize { - match self { - Holds::Frames(frames) => frames.under_way(), - Holds::Vector(vector) => vector.under_way(), - } - } - - /// Brings `data`, dataset `name`, up to what the view asks: whether it changed, whether a piece - /// stands in, and a failure to tell. - fn update( - &mut self, - input: &Input<'_>, - name: &Name, - data: &mut Data, - asked: &Asked<'_>, - jobs: &Jobs<'_>, - ) -> (bool, bool, Option) { - match self { - Holds::Frames(frames) => frames.update(name, data, asked.reading, jobs), - Holds::Vector(vector) => vector.update(input, name, data, asked.sight, jobs), - } - } -} - -/// What the view asks of its tiles: the tiles the camera shows, on the earth, and what it reads of -/// sampled fields. -pub(crate) struct Asked<'a> { - pub(crate) sight: Option<&'a Sight<'a>>, - pub(crate) reading: &'a Reading, -} - -/// Where a set's pieces come from, as its source opens. -enum Source { - Unopened, - /// The URL's first bytes, which say what it holds. - Opening(Pending>), - Open(Holds), - Failed, -} - -/// A source's pieces: where they come from, and the dataset they make. -struct TileSet { - tiles: Tiles, - source: Source, - data: Data, - /// Whether the dataset changed since it was last taken. - changed: bool, - /// Whether a failure was told: a source that went away tells it once. - told: bool, - /// Whether a piece the view wanted at the last update drew an ancestor, or nothing, instead. - standing_in: bool, -} - -impl TileSet { - fn new(tiles: Tiles) -> TileSet { - TileSet { - data: Data::new(tiles.name.clone()), - tiles, - source: Source::Unopened, - changed: false, - told: false, - standing_in: false, - } - } -} - -/// Every source of tiles a view was given. -#[derive(Default)] -pub(crate) struct TileSets { - sets: Vec, - /// Datasets the last update changed, for the view to take. - changed: Vec, - errors: Vec, -} - -impl TileSets { - /// Fetches tiles from `tiles` as the view asks; tiles of the same name replace these. - pub(crate) fn set(&mut self, tiles: Tiles) { - if self.sets.iter().any(|s| s.tiles == tiles) { - return; - } - self.sets.retain(|s| s.tiles.name != tiles.name); - self.sets.push(TileSet::new(tiles)); - } - - /// Whether every piece the view wants draws itself: none stands in by an ancestor, or by nothing - /// while its source opens. A piece that could not be fetched draws nothing, and is final. - pub(crate) fn complete(&self) -> bool { - self.sets.iter().all(|s| !matches!(s.source, Source::Unopened | Source::Opening(_)) && !s.standing_in) - } - - /// The pieces every source draws, and those being fetched or decoded. - pub(crate) fn counts(&self) -> [u64; 2] { - let mut counts = [0, 0]; - for set in &self.sets { - if let Source::Open(holds) = &set.source { - counts[0] += holds.drawn() as u64; - counts[1] += holds.under_way() as u64; - } - } - counts - } - - /// Whether the view owns dataset `name`: pieces fetched as the view asks, drawn as they are. - pub(crate) fn owns(&self, name: &str) -> bool { - self.sets.iter().any(|s| *s.tiles.name == *name) - } - - /// Stops fetching the pieces drawn as dataset `name`. - pub(crate) fn remove(&mut self, name: &str) { - self.sets.retain(|s| s.tiles.name != name); - } - - /// The datasets the last update changed. - pub(crate) fn take_changed(&mut self) -> Vec { - std::mem::take(&mut self.changed) - } - - /// Failures since last asked. - pub(crate) fn take_errors(&mut self) -> Vec { - std::mem::take(&mut self.errors) - } - - /// Brings every source up to what the view asks and the scene draws. - pub(crate) fn update(&mut self, input: &Input<'_>, asked: &Asked<'_>, jobs: &Jobs<'_>) { - for set in &mut self.sets { - if let Some(error) = set.update(input, asked, jobs) - && !set.told - { - set.told = true; - self.errors.push(Error::Io(format!("tiles {}: {error}", set.tiles.name).into())); - } - if std::mem::take(&mut set.changed) { - self.changed.push(set.data.clone()); - } - } - } -} - -impl TileSet { - /// Brings the pieces up to what the view asks; a failure to tell, if any. - fn update(&mut self, input: &Input<'_>, asked: &Asked<'_>, jobs: &Jobs<'_>) -> Option { - self.standing_in = false; - let failure = self.open(jobs); - let Source::Open(holds) = &mut self.source else { return failure }; - let (changed, standing_in, failed) = holds.update(input, &self.tiles.name, &mut self.data, asked, jobs); - self.changed |= changed; - self.standing_in = standing_in; - failure.or(failed) - } - - /// Opens the source as far as what landed allows; a failure to tell, if any. - fn open(&mut self, jobs: &Jobs<'_>) -> Option { - let url = self.tiles.url.clone(); - if matches!(self.source, Source::Unopened) { - if url.contains("{z}") { - return self.vector(|| Ok(vector::Vector::template(&url))); - } - // An archive is read in ranges, from its header; a document whole. - let archive = url.split(['?', '#']).next().is_some_and(|path| path.ends_with(".pmtiles")); - let request = if archive { - Request::range(url.clone(), 0..crate::tiles::ARCHIVE_FIRST) - } else { - Request::new(url.clone()) - }; - self.source = Source::Opening(jobs.fetch(&request)); - } - let Source::Opening(pending) = &self.source else { return None }; - let bytes = match pending.take()? { - Ok(bytes) => bytes, - Err(error) => { - self.source = Source::Failed; - return Some(error); - } - }; - match Manifest::from_bytes(&bytes) { - Some(Ok(manifest)) => { - self.source = Source::Open(Holds::Frames(Box::new(frames::Frames::open(&url, manifest, jobs)))); - return None; - } - Some(Err(error)) => { - self.source = Source::Failed; - return Some(error); - } - None => {} - } - self.vector(|| vector::Vector::open(&url, &bytes, jobs)) - } - - /// Opens the source as vector tiles `make` makes. - fn vector(&mut self, make: impl FnOnce() -> Result) -> Option { - match make() { - Ok(vector) => { - self.source = Source::Open(Holds::Vector(Box::new(vector))); - None - } - Err(reason) => { - self.source = Source::Failed; - Some(Error::Io(reason.into())) - } - } - } -} diff --git a/crates/latkit/src/derive/tiles/vector.rs b/crates/latkit/src/derive/tiles/vector.rs deleted file mode 100644 index 9c72d024..00000000 --- a/crates/latkit/src/derive/tiles/vector.rs +++ /dev/null @@ -1,387 +0,0 @@ -//! Vector tiles: the tiles the camera asks for, fetched from a PMTiles archive or a URL -//! template, given or from a TileJSON document, decoded in jobs, and patched into the dataset the view -//! owns. The camera chooses the tiles that draw no larger than their own size where they lie nearest -//! its focus, nearest first. Tiles decode only the layers marks draw, and again when the scene draws -//! others. - -use std::sync::Arc; - -use rustc_hash::{FxHashMap, FxHashSet}; - -use super::super::sight::{Sight, TILE_PX, meets}; -use super::super::tiling::{Decoding, Requested, Source, Tiling}; -use super::super::{Input, Jobs, stamp_of}; -use crate::data::{Data, Field, Id, Name, Op, Patch, Table, Values}; -use crate::geo::mvt::Kind as Geometry; -use crate::geo::pmtiles::{self, Directory, Entry, Header}; -use crate::geo::tiles::{Decoded, Template}; -use crate::host::{Job, Request}; -use crate::tiles::TileKey; -use crate::{Error, Kind, Mark, Pending, Scene}; - -/// Tiles drawn at once at most: past it, tiles stop splitting. -const MAX_TILES: usize = 256; - -/// The tiles to draw within `zooms` and `bounds`: those meeting what the camera shows, each split -/// while it draws larger than its size, nearest the view's middle first. -fn wanted(sight: &Sight<'_>, zooms: [u8; 2], bounds: [f64; 4]) -> Vec { - let limit = TILE_PX * 1.25; - let region = sight.region(); - let focus = sight.focus(); - let mut out: Vec<(TileKey, f64)> = Vec::new(); - let mut queue = vec![TileKey { z: 0, x: 0, y: 0 }]; - while !queue.is_empty() { - let mut next = Vec::new(); - for (k, &key) in queue.iter().enumerate() { - let b = key.bounds(); - if b[0] > bounds[2] || b[2] < bounds[0] || b[1] > bounds[3] || b[3] < bounds[1] || !meets(®ion, b) { - continue; - } - let (size, distance) = sight.measure(&focus, b); - let room = out.len() + next.len() + (queue.len() - k) * 4 < MAX_TILES; - if key.z < zooms[1] && (key.z < zooms[0] || size > limit) && room { - next.extend(key.children()); - } else if key.z >= zooms[0] { - out.push((key, distance)); - } - } - queue = next; - } - crate::order::by_key(&mut out, |a| crate::order::number(a.1)); - out.into_iter().map(|(key, _)| key).collect() -} - -/// Where vector tiles come from once their source opens. -enum Opened { - /// An archive whose root directory lies past the first bytes. - Root(Header, Pending>), - Template(Template), - Archive(Header, Directory), - Failed, -} - -/// An archive's or a template's tiles, as a tiling fetches and decodes them. -struct Tiles { - url: Arc, - opened: Opened, - /// Leaf directories, and those being fetched. - leaves: FxHashMap<(u64, u64), Directory>, - leaf_fetches: FxHashMap<(u64, u64), Pending>>, - /// The tables marks draw from the tiles, which decoding writes. - tables: Vec, -} - -/// Vector tiles drawn as a dataset the view owns. -pub(crate) struct Vector { - tiles: Tiles, - tiling: Tiling, - /// The tiles whose rows the dataset holds, with the rows of each table. - drawn: FxHashMap)>>, -} - -impl Vector { - /// The tiles a URL template names, `{z}`, `{x}`, and `{y}` in it. - pub(crate) fn template(url: &Arc) -> Vector { - Vector::opened(url, Opened::Template(Template::of(url))) - } - - /// The tiles whose source's first `bytes` are a PMTiles archive's or a TileJSON document's; the - /// archive's root directory is fetched when it lies past them. - pub(crate) fn open(url: &Arc, bytes: &[u8], jobs: &Jobs<'_>) -> Result { - if !pmtiles::is_archive(bytes) { - return Template::tilejson(bytes, url).map(|template| Vector::opened(url, Opened::Template(template))); - } - let header = pmtiles::header(bytes)?; - if !header.vector { - return Err(format!("{url} holds image tiles; only vector tiles draw")); - } - let (offset, length) = header.root; - let opened = match bytes.get(offset as usize..(offset + length) as usize) { - Some(root) => archive(header, root).ok_or_else(|| corrupt(url))?, - None => Opened::Root(header, jobs.fetch(&Request::range(url.clone(), offset..offset + length))), - }; - Ok(Vector::opened(url, opened)) - } - - fn opened(url: &Arc, opened: Opened) -> Vector { - let tiles = Tiles { - url: url.clone(), - opened, - leaves: FxHashMap::default(), - leaf_fetches: FxHashMap::default(), - tables: Vec::new(), - }; - Vector { tiles, tiling: Tiling::default(), drawn: FxHashMap::default() } - } - - /// Whether the source is still opening, its root directory not landed. - pub(crate) fn opening(&self) -> bool { - matches!(self.tiles.opened, Opened::Root(..)) - } - - /// Fetches and decodes under way. - pub(crate) fn under_way(&self) -> usize { - self.tiling.under_way() - } - - /// Tiles drawn. - pub(crate) fn drawn(&self) -> usize { - self.drawn.len() - } - - /// Brings `data` up to what `sight` shows and the scene draws of dataset `name`: whether it - /// changed, whether a tile stands in, and a failure to tell. - pub(crate) fn update( - &mut self, - input: &Input<'_>, - name: &Name, - data: &mut Data, - sight: Option<&Sight<'_>>, - jobs: &Jobs<'_>, - ) -> (bool, bool, Option) { - let tables = decoded_tables(input.scene, name); - if tables.is_empty() { - return (false, false, None); - } - let mut failure = self.tiles.open_root(); - failure = failure.or_else(|| self.tiles.land_leaves()); - let Some((zooms, bounds)) = self.tiles.zooms() else { return (false, self.opening(), failure) }; - let stamp = stamp_of(&tables); - self.tiles.tables = tables; - let wanted = sight.map(|s| wanted(s, zooms, bounds)).unwrap_or_default(); - let drawn = self.tiling.update(&wanted, stamp, &mut self.tiles, jobs); - let changed = self.draw(&drawn.keys, stamp, data); - (changed, drawn.standing_in, failure.or(drawn.failure)) - } - - /// Makes `data` hold the rows of `drawn` and no other tile's; whether it changed. - fn draw(&mut self, drawn: &[TileKey], stamp: u64, data: &mut Data) -> bool { - let keep: FxHashSet = drawn.iter().copied().collect(); - let mut patch = Patch::new(); - // Tiles no longer drawn go, and so do those decoded for other tables, which come back - // decoded again. - let gone: Vec = self - .drawn - .keys() - .filter(|k| !keep.contains(k) || self.tiling.decoded(k, stamp).is_none()) - .copied() - .collect(); - for key in &gone { - for (table, ids) in self.drawn.remove(key).unwrap_or_default() { - if data.table(&table).is_some() { - patch = patch.remove(table, ids); - } - } - } - for &key in drawn { - if self.drawn.contains_key(&key) { - continue; - } - let Some(tile) = self.tiling.decoded(&key, stamp) else { continue }; - let mut rows = Vec::new(); - for op in &tile.ops { - if let Op::Set(table) = op { - rows.push((table.name.clone(), table.ids().collect())); - patch = patch.set(harmonized(table, data)); - } - } - self.drawn.insert(key, rows); - } - if patch.is_empty() { - return false; - } - let mut next = data.clone(); - if next.apply(patch).is_ok() { - *data = next; - return true; - } - false - } -} - -impl Tiles { - /// The zooms and bounds of an open source. - fn zooms(&self) -> Option<([u8; 2], [f64; 4])> { - match &self.opened { - Opened::Template(t) => Some(([t.min_zoom, t.max_zoom], t.bounds)), - Opened::Archive(h, _) => Some(([h.min_zoom, h.max_zoom], h.bounds)), - Opened::Root(..) | Opened::Failed => None, - } - } - - /// Opens the archive once its root directory landed; a failure to tell, if any. - fn open_root(&mut self) -> Option { - let Opened::Root(header, pending) = &self.opened else { return None }; - let header = *header; - let landed = pending.take()?; - let opened = landed - .map_err(|error| error.to_string()) - .and_then(|bytes| archive(header, &bytes).ok_or_else(|| corrupt(&self.url))); - match opened { - Ok(opened) => { - self.opened = opened; - None - } - Err(reason) => { - self.opened = Opened::Failed; - Some(Error::Io(reason.into())) - } - } - } - - /// Takes leaf directories that landed; a failure to tell, if any. - fn land_leaves(&mut self) -> Option { - let mut failure = None; - let landed: Vec<_> = - self.leaf_fetches.iter().filter_map(|(at, pending)| pending.take().map(|r| (*at, r.ok()))).collect(); - for (at, bytes) in landed { - self.leaf_fetches.remove(&at); - let gzip = matches!(self.opened, Opened::Archive(h, _) if h.gzip_directories); - match bytes.and_then(|b| inflate(&b, gzip)).and_then(|b| Directory::decode(&b)) { - Some(leaf) => { - self.leaves.insert(at, leaf); - } - None => failure = Some(Error::Io("a tile directory could not be read".into())), - } - } - failure - } -} - -impl Source for Tiles { - fn request(&mut self, key: TileKey, jobs: &Jobs<'_>) -> Requested { - match &self.opened { - Opened::Template(template) => Requested::Fetch(Request::new(template.url_of(key))), - Opened::Archive(header, root) => { - let id = pmtiles::tile_id(key); - let mut directory = root; - for _ in 0..pmtiles::DEPTH { - match directory.find(id) { - Some(Entry::Leaf(offset, length)) => { - let at = (header.leaves + offset, length); - let Some(leaf) = self.leaves.get(&at) else { - let url = self.url.clone(); - self.leaf_fetches - .entry(at) - .or_insert_with(|| jobs.fetch(&Request::range(url, at.0..at.0 + at.1))); - return Requested::Wait; - }; - directory = leaf; - } - Some(Entry::Tile(offset, length)) => { - let start = header.data + offset; - return Requested::Fetch(Request::range(self.url.clone(), start..start + length)); - } - None => break, - } - } - // The archive has no such tile: it draws nothing. - Requested::Missing - } - Opened::Root(..) | Opened::Failed => Requested::Wait, - } - } - - fn decode(&self, key: TileKey, bytes: &Arc>, jobs: &Jobs<'_>) -> Decoding { - Decoding::Job(jobs.spawn(Job::tile(key, &self.tables, bytes))) - } -} - -/// The archive at `header` with root directory `bytes`, or `None` where it is corrupt. -fn archive(header: Header, bytes: &[u8]) -> Option { - let root = inflate(bytes, header.gzip_directories).and_then(|b| Directory::decode(&b))?; - Some(Opened::Archive(header, root)) -} - -fn corrupt(url: &str) -> String { - format!("the archive at {url} has a corrupt directory") -} - -/// The tables marks draw from dataset `name`: a table named after a layer holds the kind of feature -/// its marks draw, polygons first, then lines, then points; one named `layer:kind` holds that kind. -fn decoded_tables(scene: &Scene, name: &Name) -> Vec { - let mut tables: Vec = Vec::new(); - for mark in &scene.marks { - let style = mark.style(); - if style.data != *name { - continue; - } - let drawn = match mark { - Mark::Areas(_) => Geometry::Polygons, - Mark::Lines(_) => Geometry::Lines, - Mark::Vertices(_) => Geometry::Points, - Mark::Edges(_) | Mark::Faces(_) => continue, - }; - let table = style.table.clone(); - let (layer, kind) = match table.split_once(':') { - Some((layer, "polygons")) => (layer.to_owned(), Geometry::Polygons), - Some((layer, "lines")) => (layer.to_owned(), Geometry::Lines), - Some((layer, "points")) => (layer.to_owned(), Geometry::Points), - _ => (table.to_string(), drawn), - }; - let rank = |k: Geometry| match k { - Geometry::Polygons => 0, - Geometry::Lines => 1, - Geometry::Points => 2, - }; - match tables.iter_mut().find(|d| d.table == table) { - Some(d) if rank(kind) < rank(d.kind) => d.kind = kind, - Some(_) => {} - None => tables.push(Decoded { layer, kind, table }), - } - } - crate::order::by_name(&mut tables, |t| t.table.as_str()); - tables -} - -/// `table`'s fields made to hold what `data`'s table of its name holds where they differ: a field a -/// tile reads as numbers but the dataset holds as texts is written as text, and so on. -fn harmonized(table: &Table, data: &Data) -> Table { - let Some(existing) = data.table(&table.name) else { return table.clone() }; - let mut out = table.clone(); - for (name, field) in &table.fields { - let (Some(want), Field::Values(column, _)) = (existing.kind(name), field) else { continue }; - if column.kind() == want { - continue; - } - let values: Vec> = (0..column.len()).map(|r| column.get(r)).collect(); - let converted: Option = match want { - Kind::Text => Some(values.iter().map(|v| v.as_ref().map(text_of)).collect::>>().into()), - Kind::Number => Some(values.iter().map(|v| v.as_ref().and_then(number_of)).collect::>().into()), - Kind::Bool => { - Some(values.iter().map(|v| v.as_ref().and_then(number_of).map(|n| n != 0.0)).collect::>().into()) - } - _ => None, - }; - if let Some(values) = converted { - out = out.field(name.clone(), values); - } - } - out -} - -fn text_of(value: &crate::Value) -> String { - match value { - crate::Value::Text(t) => t.to_string(), - crate::Value::Number(n) => n.to_string(), - crate::Value::Bool(b) => b.to_string(), - other => format!("{other:?}"), - } -} - -fn number_of(value: &crate::Value) -> Option { - match value { - crate::Value::Number(n) => Some(*n), - crate::Value::Bool(b) => Some(f64::from(u8::from(*b))), - crate::Value::Text(t) => t.parse().ok(), - _ => None, - } -} - -/// `bytes` inflated when `gzip`. -fn inflate(bytes: &[u8], gzip: bool) -> Option> { - if !gzip { - return Some(bytes.to_vec()); - } - crate::geo::gunzipped(bytes).ok().map(std::borrow::Cow::into_owned) -} diff --git a/crates/latkit/src/derive/tiling.rs b/crates/latkit/src/derive/tiling.rs deleted file mode 100644 index 3f0ebb94..00000000 --- a/crates/latkit/src/derive/tiling.rs +++ /dev/null @@ -1,228 +0,0 @@ -//! Tilings: what a source holds, fetched in pieces by key as a view asks for them, decoded into -//! patches, and held under a budget of bytes, the least recently drawn going first. A key wanted draws -//! once it is decoded; until then its nearest decoded ancestor stands in, and what is missing is -//! fetched, nearest first. Tiles of the earth and spans of a sampled field's history share it. - -use std::sync::Arc; - -use rustc_hash::FxHashMap; - -use super::Jobs; -use crate::data::Patch; -use crate::host::Request; -use crate::tiles::{Key, ancestry}; -use crate::{Error, Pending}; - -/// Fetches, and decodes, a tiling has under way at once. -const REQUESTS: usize = 8; -/// Bytes of fetched pieces a tiling keeps, past which the least recently drawn go. -const KEPT_BYTES: usize = 64 << 20; - -/// How a source fetches a piece. -pub(crate) enum Requested { - /// The bytes `request` answers. - Fetch(Request), - /// Not yet: the source first fetches what finds the piece, as an archive its directories. - Wait, - /// Nowhere: the source holds no such piece, which draws nothing. - Missing, -} - -/// What a piece's bytes become, the patch it draws: at once, or the answer of a job. -pub(crate) enum Decoding { - Now(Patch), - Job(Pending>), -} - -/// Where a tiling's pieces come from, and what they become. -pub(crate) trait Source { - /// How to fetch `key`. - fn request(&mut self, key: K, jobs: &Jobs<'_>) -> Requested; - - /// What `key`'s fetched `bytes`, never empty, decode into. - fn decode(&self, key: K, bytes: &Arc>, jobs: &Jobs<'_>) -> Decoding; -} - -/// What an update draws: each wanted key decoded, else its nearest decoded ancestor, each once; and -/// whether any key stands in, by an ancestor or by nothing. -pub(crate) struct Drawn { - pub(crate) keys: Vec, - pub(crate) standing_in: bool, - /// A failure to tell: a piece that could not be fetched. - pub(crate) failure: Option, -} - -/// A tiling's pieces: their fetched bytes, and the patches they decoded into for a stamp of what -/// decoding reads. -pub(crate) struct Tiling { - /// Fetched bytes, as fetched, and when each last drew; empty where the source holds no piece. - held: FxHashMap>, u64)>, - fetching: FxHashMap>>, - /// Decodes under way and done, each for the stamp it was asked for. - decoding: FxHashMap>)>, - decoded: FxHashMap, - /// Counts updates, for drawn pieces to note when they last drew. - tick: u64, -} - -impl Default for Tiling { - fn default() -> Tiling { - Tiling { - held: FxHashMap::default(), - fetching: FxHashMap::default(), - decoding: FxHashMap::default(), - decoded: FxHashMap::default(), - tick: 0, - } - } -} - -impl Tiling { - /// Brings the pieces up to `wanted`, nearest first, decoding for `stamp`: what draws. - pub(crate) fn update( - &mut self, - wanted: &[K], - stamp: u64, - source: &mut impl Source, - jobs: &Jobs<'_>, - ) -> Drawn { - self.tick += 1; - let failure = self.land(stamp); - // Keys not decoded yet decode once held, and so do the held ancestors of those still - // loading, to stand in for them; the rest fetch. A piece decoded at once draws at once. - let mut decode: Vec = Vec::new(); - for &key in wanted { - if self.ready(&key, stamp) { - continue; - } - let held_up = ancestry(key).find(|k| self.held.contains_key(k)); - if let Some(k) = held_up - && !self.ready(&k, stamp) - && !decode.contains(&k) - { - decode.push(k); - } - if !self.held.contains_key(&key) && !decode.contains(&key) { - self.fetch(key, source, jobs); - } - } - for key in decode { - self.decode(key, stamp, source, jobs); - } - let mut drawn: Vec = Vec::new(); - let mut standing_in = false; - for &key in wanted { - let stands = ancestry(key).find(|k| self.ready(k, stamp)); - if let Some(stand) = stands - && !drawn.contains(&stand) - { - drawn.push(stand); - } - standing_in |= stands != Some(key); - } - let tick = self.tick; - for key in &drawn { - if let Some((_, used)) = self.held.get_mut(key) { - *used = tick; - } - } - self.evict(&drawn, wanted); - Drawn { keys: drawn, standing_in, failure } - } - - /// The patch `key` decoded into for `stamp`, once it did. - pub(crate) fn decoded(&self, key: &K, stamp: u64) -> Option<&Patch> { - self.decoded.get(key).filter(|(s, _)| *s == stamp).map(|(_, patch)| patch) - } - - /// Fetches and decodes under way. - pub(crate) fn under_way(&self) -> usize { - self.fetching.len() + self.decoding.len() - } - - /// Whether `key` is decoded for `stamp`. - fn ready(&self, key: &K, stamp: u64) -> bool { - self.decoded.get(key).is_some_and(|(s, _)| *s == stamp) - } - - /// Takes fetches and decodes that landed; a failure to tell, if any. - fn land(&mut self, stamp: u64) -> Option { - let mut failure = None; - let fetched: Vec<_> = self.fetching.iter().filter_map(|(key, p)| p.take().map(|r| (*key, r))).collect(); - for (key, result) in fetched { - self.fetching.remove(&key); - let bytes = result.unwrap_or_else(|error| { - failure.get_or_insert(error); - Vec::new() - }); - self.held.insert(key, (Arc::new(bytes), self.tick)); - } - let decoded: Vec<_> = - self.decoding.iter().filter_map(|(key, (s, p))| p.take().map(|r| (*key, *s, r))).collect(); - for (key, s, result) in decoded { - self.decoding.remove(&key); - let patch = result.ok().and_then(|bytes| Patch::from_bytes(&bytes).ok()).unwrap_or_default(); - if s == stamp { - self.decoded.insert(key, (s, patch)); - } - } - failure - } - - /// Starts fetching `key`, within the requests under way at once. - fn fetch(&mut self, key: K, source: &mut impl Source, jobs: &Jobs<'_>) { - if self.fetching.contains_key(&key) || self.fetching.len() >= REQUESTS { - return; - } - match source.request(key, jobs) { - Requested::Fetch(request) => { - self.fetching.insert(key, jobs.fetch(&request)); - } - Requested::Wait => {} - Requested::Missing => { - self.held.insert(key, (Arc::new(Vec::new()), self.tick)); - } - } - } - - /// Starts decoding `key` for `stamp`, within the decodes under way at once. - fn decode(&mut self, key: K, stamp: u64, source: &impl Source, jobs: &Jobs<'_>) { - if self.decoding.get(&key).is_some_and(|(s, _)| *s == stamp) || self.decoding.len() >= REQUESTS { - return; - } - let Some((bytes, _)) = self.held.get(&key) else { return }; - if bytes.is_empty() { - self.decoded.insert(key, (stamp, Patch::new())); - return; - } - match source.decode(key, bytes, jobs) { - Decoding::Now(patch) => { - self.decoded.insert(key, (stamp, patch)); - } - Decoding::Job(pending) => { - self.decoding.insert(key, (stamp, pending)); - } - } - } - - /// Lets go of the least recently drawn pieces' bytes past the budget, and of decodes no longer - /// drawn, wanted, or covering a wanted key. - fn evict(&mut self, drawn: &[K], wanted: &[K]) { - let kept = |k: &K| drawn.contains(k) || wanted.contains(k); - let mut bytes: usize = self.held.values().map(|(b, _)| b.len()).sum(); - if bytes > KEPT_BYTES { - let mut old: Vec<(u64, K)> = - self.held.iter().filter(|(k, _)| !kept(k)).map(|(k, (_, used))| (*used, *k)).collect(); - old.sort_unstable(); - for (_, key) in old { - if bytes <= KEPT_BYTES { - break; - } - if let Some((b, _)) = self.held.remove(&key) { - bytes -= b.len(); - } - } - } - self.decoded.retain(|k, _| kept(k) || wanted.iter().any(|w| ancestry(*w).skip(1).any(|a| a == *k))); - } -} diff --git a/crates/latkit/src/error.rs b/crates/latkit/src/error.rs index 0e3b4d3b..08c703bc 100644 --- a/crates/latkit/src/error.rs +++ b/crates/latkit/src/error.rs @@ -7,30 +7,30 @@ use crate::Name; #[derive(Clone, Debug, PartialEq, thiserror::Error)] #[non_exhaustive] pub enum Error { - /// An input does not fit what it describes; `path` names the part, such as `marks[2].vertices.color`. + /// An input is invalid. `path` names the invalid part, such as `marks[2].color`. #[error("{path}: {reason}")] Invalid { - /// Where in the input the problem is. + /// Location of the problem in the input. path: Path, - /// What is wrong there. + /// Why that part is invalid. reason: Cow<'static, str>, }, - /// The platform or device cannot do what was asked. + /// The platform or device does not support the request. #[error("unsupported: {0}")] Unsupported(Cow<'static, str>), /// A budget or device limit would be exceeded. #[error("over limit: {0}")] Limit(Cow<'static, str>), - /// Reading or writing external bytes failed: a fetch, a file, a worker, an encoder. + /// Reading or writing external bytes failed, such as a fetch, file, worker, or encoder. #[error("io: {0}")] Io(Cow<'static, str>), /// The operation did not complete before its deadline. #[error("operation timed out")] Timeout, - /// The GPU device was lost; create a new `Gpu`. + /// The GPU device was lost. Create a new `Gpu`. #[error("GPU device lost")] DeviceLost, - /// The owner of pending work was destroyed before it could answer. + /// The owner of pending work was dropped before the work finished. #[error("operation cancelled")] Cancelled, } @@ -39,6 +39,9 @@ pub enum Error { pub type Result = std::result::Result; impl Error { + /// Every kind's [`code`](Error::code). + const CODES: [&'static str; 7] = ["invalid", "unsupported", "limit", "io", "device-lost", "cancelled", "timeout"]; + /// An [`Error::Invalid`] at the root of the input. pub fn invalid(reason: impl Into>) -> Error { Error::Invalid { path: Path::default(), reason: reason.into() } @@ -58,7 +61,7 @@ impl Error { } } - /// Prefixes the path of an [`Error::Invalid`] with `step`; other errors pass through. + /// Prefixes the path of an [`Error::Invalid`] with `step`. Returns other errors unchanged. #[must_use] pub fn at(self, step: impl Into) -> Error { match self { @@ -69,9 +72,24 @@ impl Error { other => other, } } + + /// This error, of the same kind, with `source` named as where it happened. `source` becomes the + /// first step of an invalid input's path, or the start of another error's message. + #[cfg(feature = "gpu")] + #[must_use] + pub(crate) fn within(self, source: &str) -> Error { + let named = |message: Cow<'static, str>| Cow::Owned(format!("{source}: {message}")); + match self { + Error::Invalid { .. } => self.at(source), + Error::Unsupported(message) => Error::Unsupported(named(message)), + Error::Limit(message) => Error::Limit(named(message)), + Error::Io(message) => Error::Io(named(message)), + other => other, + } + } } -/// The error as `{ code, message, path? }`: the shape the npm package's `LatkitError` takes. +/// Serializes as `{ code, message, path? }`, the shape of the npm package's `LatkitError`. impl serde::Serialize for Error { fn serialize(&self, serializer: S) -> Result { use serde::ser::SerializeMap; @@ -85,7 +103,7 @@ impl serde::Serialize for Error { } } -/// The error as it serializes: a failure, as plain data. +/// Schema of the serialized error, named `Failure`. impl schemars::JsonSchema for Error { fn schema_name() -> Cow<'static, str> { "Failure".into() @@ -93,12 +111,18 @@ impl schemars::JsonSchema for Error { fn json_schema(_: &mut schemars::SchemaGenerator) -> schemars::Schema { schemars::json_schema!({ - "description": "What failed: its `code`, the message, and for invalid input the `path` to the part, such as `marks[2].vertices.color`.", + "description": "A failure as JSON, in the same form as the npm package's `LatkitError`.", "type": "object", "properties": { - "code": { "enum": ["invalid", "unsupported", "limit", "io", "device-lost", "cancelled", "timeout"] }, - "message": { "type": "string" }, - "path": { "type": "string" }, + "code": { + "description": "The kind of failure. See `LatkitError.code`.", + "enum": Error::CODES, + }, + "message": { "description": "A description of what failed.", "type": "string" }, + "path": { + "description": "For `invalid` failures, the path of the invalid input, such as `marks[2].color`.", + "type": "string", + }, }, "required": ["code", "message"], }) @@ -109,7 +133,7 @@ impl schemars::JsonSchema for Error { #[derive(Clone, Debug, Default, PartialEq, Eq)] pub struct Path(pub Vec); -/// One step of a [`Path`]: a named part or an index. +/// One step of a [`Path`]: a named part or a list index. #[derive(Clone, Debug, PartialEq, Eq)] pub enum Step { /// A named part, such as a table, field, or option. @@ -172,4 +196,18 @@ mod tests { fn only_invalid_errors_take_steps() { assert_eq!(Error::DeviceLost.at("marks"), Error::DeviceLost); } + + #[test] + fn the_schema_lists_every_code() { + let every = [ + Error::invalid(""), + Error::Unsupported("".into()), + Error::Limit("".into()), + Error::Io("".into()), + Error::DeviceLost, + Error::Cancelled, + Error::Timeout, + ]; + assert_eq!(every.map(|e| e.code()), Error::CODES); + } } diff --git a/crates/latkit/src/expr/color.rs b/crates/latkit/src/expr/color.rs index 9c821f19..b3501acf 100644 --- a/crates/latkit/src/expr/color.rs +++ b/crates/latkit/src/expr/color.rs @@ -1,6 +1,6 @@ use std::fmt; -/// An sRGB color with straight alpha, each channel in `0..=1`, as in CSS. +/// An sRGB color with straight (not premultiplied) alpha, each channel in `0..=1`, as in CSS. #[derive(Clone, Copy, PartialEq, Debug)] pub struct Color { /// Red. @@ -31,7 +31,7 @@ impl Color { Color { r, g, b, a: 1.0 } } - /// A color from `#rgb`, `#rgba`, `#rrggbb`, or `#rrggbbaa`. + /// A color from `#rgb`, `#rgba`, `#rrggbb`, or `#rrggbbaa`. `None` for any other text. pub fn hex(hex: &str) -> Option { let digits = hex.strip_prefix('#')?; if !digits.is_ascii() { @@ -50,7 +50,8 @@ impl Color { Some(Color::from_bytes([r, g, b, a])) } - /// A color from CSS: hex, `rgb(…)`, `rgba(…)`, or a named color. + /// A color from CSS: hex, `rgb(…)` or `rgba(…)` (channels 0 to 255 or percentages, alpha 0 to 1 + /// or a percentage), a named color, or `transparent`. Case-insensitive. `None` if unrecognized. pub fn css(css: &str) -> Option { let css = css.trim(); if css.starts_with('#') { @@ -92,7 +93,7 @@ impl Color { Color::rgba(red, green, blue, alpha) } - /// Red, green, blue, and alpha as bytes. + /// Red, green, blue, and alpha as bytes, clamped and rounded. pub fn to_rgba8(self) -> [u8; 4] { let q = |v: f32| (v.clamp(0.0, 1.0) * 255.0).round() as u8; [q(self.r), q(self.g), q(self.b), q(self.a)] diff --git a/crates/latkit/src/expr/colormap.rs b/crates/latkit/src/expr/colormap.rs index f860b514..d4e5b58c 100644 --- a/crates/latkit/src/expr/colormap.rs +++ b/crates/latkit/src/expr/colormap.rs @@ -7,10 +7,10 @@ use super::ramp::{self, Spread}; use crate::gpu::shaders as wgsl; use crate::{Error, Result}; -/// Colors for a range of values: one of the catalog's, or one of your own. +/// Colors for a range of values: a catalog colormap or a custom one. /// /// Sequential and diverging maps clamp, cyclic maps wrap, and categorical maps bin values by their -/// integer part: category `i` takes color `i`, and may be named with [`Colormap::categories`]. +/// integer part: category `i` takes color `i`. [`Colormap::categories`] names the categories. #[derive(Clone, PartialEq, Debug)] pub struct Colormap(Map); @@ -28,13 +28,14 @@ struct Custom { categories: Vec>, } -/// How a colormap spans values. +/// How a colormap maps values: `sequential` and `diverging` maps clamp, `cyclic` maps wrap, and +/// `categorical` maps give each integer its own color. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug, serde::Serialize, serde::Deserialize, schemars::JsonSchema)] #[serde(rename_all = "lowercase")] pub enum ColormapKind { /// Low to high. Sequential, - /// Below and above a middle. + /// Below and above a midpoint. Diverging, /// Around a circle: values wrap. Cyclic, @@ -51,7 +52,7 @@ impl ColormapKind { } } - /// The code shaders read it by. + /// The code shaders identify it by. #[cfg(feature = "gpu")] pub(crate) fn code(self) -> u32 { match self { @@ -63,8 +64,8 @@ impl ColormapKind { } } -/// A colormap's color, at a place from 0 to 1 or evenly between its neighbors: from a [`Color`] or -/// a `(place, Color)` pair. +/// A colormap color stop: a [`Color`], spaced evenly between its neighbors, or a `(place, Color)` +/// pair with a place from 0 to 1. #[derive(Clone, Copy, PartialEq, Debug)] pub struct Stop { at: Option, @@ -99,33 +100,35 @@ struct Entry { include!(concat!(env!("OUT_DIR"), "/colormaps.rs")); -/// The catalog's names, in order. -pub(crate) fn catalog() -> impl Iterator { - CATALOG.iter().map(|e| e.name) -} - impl Colormap { - /// The catalog colormap named `name`, such as `"viridis"`. + /// Every catalog colormap, in order. + pub fn catalog() -> impl Iterator { + (0..CATALOG.len()).map(|i| Colormap(Map::Catalog(i as u8))) + } + + /// The catalog colormap named `name`, such as `"viridis"`. `None` if the catalog has none. pub fn named(name: &str) -> Option { CATALOG.iter().position(|e| e.name == name).map(|i| Colormap(Map::Catalog(i as u8))) } - /// A colormap through `stops`, colors or `(place, color)` pairs: blended in OKLab, so lightness - /// runs evenly, a color without a place evenly between its neighbors as in a CSS gradient; or - /// for a categorical map, one color per category. Fails without a stop, for places outside - /// `0..=1` or out of order, and for a categorical map's places or more than 256 categories. + /// A colormap through `stops`, each a color or a `(place, color)` pair. Colors blend in OKLab, + /// so lightness changes evenly. A color without a place sits evenly between its neighbors, like + /// a CSS gradient. A categorical map takes one color per category. + /// + /// Errors when `stops` is empty, when places lie outside `0..=1` or out of order, and when a + /// categorical map has places or more than 256 colors. pub fn new>(kind: ColormapKind, stops: impl IntoIterator) -> Result { let stops: Vec<(Option, [f32; 4])> = stops.into_iter().map(Into::into).map(|s: Stop| (s.at, s.color.to_array())).collect(); let places = || stops.iter().filter_map(|s| s.0); let wrong: Option> = match kind { _ if stops.is_empty() => Some("a colormap needs a color".into()), - ColormapKind::Categorical if places().next().is_some() => Some("a category's color has no place".into()), + ColormapKind::Categorical if places().next().is_some() => Some("categories cannot have places".into()), ColormapKind::Categorical if stops.len() > 256 => { Some(format!("a categorical map has at most 256 colors, not {}", stops.len()).into()) } - _ if places().any(|p| !(0.0..=1.0).contains(&p)) => Some("a color's place is from 0 to 1".into()), - _ if !places().is_sorted() => Some("colors' places ascend".into()), + _ if places().any(|p| !(0.0..=1.0).contains(&p)) => Some("a color's place must be from 0 to 1".into()), + _ if !places().is_sorted() => Some("colors' places must ascend".into()), _ => None, }; if let Some(reason) = wrong { @@ -135,8 +138,8 @@ impl Colormap { Ok(Colormap(Map::Custom(Arc::new(Custom { kind, count: stops.len() as u32, texels, categories: Vec::new() })))) } - /// The same categorical map with its categories named in order, from category 0: what its legend - /// lists. Panics for a continuous map. + /// This categorical map with its categories named in order from category 0, for its legend. + /// Panics if the map is not categorical. #[must_use] pub fn categories>>(&self, names: impl IntoIterator) -> Colormap { assert_eq!(self.kind(), ColormapKind::Categorical, "only a categorical map has categories"); @@ -149,7 +152,7 @@ impl Colormap { }))) } - /// The names of the categories, from category 0; none unless named. + /// The names of the categories, from category 0. Empty unless named. pub(crate) fn named_categories(&self) -> &[Arc] { match &self.0 { Map::Catalog(_) => &[], @@ -157,7 +160,7 @@ impl Colormap { } } - /// How the colormap spans values. + /// How the colormap maps values. pub fn kind(&self) -> ColormapKind { match &self.0 { Map::Catalog(i) => CATALOG[*i as usize].kind, @@ -165,7 +168,7 @@ impl Colormap { } } - /// The catalog name, or `None` for a colormap of your own. + /// The catalog name, or `None` for a custom colormap. pub fn name(&self) -> Option<&'static str> { match &self.0 { Map::Catalog(i) => Some(CATALOG[*i as usize].name), @@ -173,7 +176,7 @@ impl Colormap { } } - /// A readable title, such as `"Red / Blue"`. + /// A human-readable title, such as `"Red / Blue"`. `"Custom"` for a custom colormap. pub fn label(&self) -> &str { match &self.0 { Map::Catalog(i) => CATALOG[*i as usize].label, @@ -199,7 +202,8 @@ impl Colormap { Colormap(Map::Custom(Arc::new(Custom { kind: self.kind(), count: self.count(), texels, categories }))) } - /// The color at `t` in `0..=1`: clamped, wrapped for cyclic maps, binned for categorical maps. + /// The color at `t`. A continuous map takes `t` in `0..=1`, clamping it, or wrapping it for a + /// cyclic map. A categorical map takes a category number, binned by its integer part. pub fn sample(&self, t: f32) -> Color { let texels = self.texels(); if self.kind() == ColormapKind::Categorical { @@ -214,7 +218,7 @@ impl Colormap { Color::rgba(red, green, blue, alpha) } - /// How many colors it was made from: a categorical map's categories. + /// The number of colors the map was made from. For a categorical map, its category count. pub fn count(&self) -> u32 { match &self.0 { Map::Catalog(i) => CATALOG[*i as usize].count, @@ -229,7 +233,7 @@ impl Colormap { } } - /// Identifies the colormap's colors, for sharing one texture layer. + /// Identifies the colormap's colors, so maps with the same colors share one texture layer. pub(crate) fn key(&self) -> ColormapKey { use std::hash::{Hash, Hasher}; match &self.0 { @@ -274,7 +278,7 @@ mod tests { #[test] fn custom_maps_blend_and_reverse() { let map = Colormap::new(ColormapKind::Sequential, [Color::BLACK, Color::WHITE]).unwrap(); - // Blended in OKLab: half lightness, a darker byte than sRGB's middle. + // Blended in OKLab, half lightness is a darker byte than sRGB's midpoint. assert!((95..105).contains(&map.sample(0.5).to_rgba8()[0])); assert_eq!(map.reversed().sample(0.0).to_rgba8(), [255, 255, 255, 255]); let placed = Colormap::new(ColormapKind::Sequential, [(0.0, Color::BLACK), (0.5, Color::WHITE)]).unwrap(); @@ -295,7 +299,7 @@ mod tests { assert!(Colormap::new(ColormapKind::Sequential, [Color::BLACK; 0]).is_err()); assert!(wrong(ColormapKind::Sequential, &[(0.5, Color::BLACK), (0.2, Color::WHITE)]).contains("ascend")); assert!(wrong(ColormapKind::Cyclic, &[(1.5, Color::BLACK)]).contains("0 to 1")); - assert!(wrong(ColormapKind::Categorical, &[(0.0, Color::BLACK)]).contains("no place")); + assert!(wrong(ColormapKind::Categorical, &[(0.0, Color::BLACK)]).contains("cannot have places")); assert!(Colormap::new(ColormapKind::Categorical, [Color::BLACK; 257]).is_err()); } } diff --git a/crates/latkit/src/expr/compile.rs b/crates/latkit/src/expr/compile.rs index 73ded0f3..b4324d0a 100644 --- a/crates/latkit/src/expr/compile.rs +++ b/crates/latkit/src/expr/compile.rs @@ -1,7 +1,7 @@ -//! Expressions to bytecode: each channel of a mark is a range of ops that ends in a store. +//! Compiles expressions to bytecode. Each channel of a mark is a range of ops ending in a store. //! -//! Compilation type-checks against the data's fields and records what each channel depends on, so -//! the view evaluates a channel only when one of its inputs changes. +//! Compilation type-checks expressions against the data's fields and records each channel's +//! inputs, so the view re-evaluates a channel only when one of its inputs changes. use std::ops::BitOr; use std::sync::Arc; @@ -10,7 +10,7 @@ use super::{Binary, Colormap, ColormapKind, Domain, Expr, FieldRef, Node, State, use crate::gpu::shaders as wgsl; use crate::{Error, Kind, Name, Result}; -/// What a channel reads, so the view knows when to evaluate it again. +/// The kinds of input a channel reads, so the view knows when to re-evaluate it. #[derive(Clone, Copy, Default, PartialEq, Eq, Hash, Debug)] pub(crate) struct Deps(u8); @@ -49,30 +49,30 @@ pub(crate) enum Words { Values, /// Text as dictionary codes. Codes, - /// Reference rows as `u32`, `NONE` where unwired. + /// Referenced rows as `u32`, `NONE` where the reference is unset. Bits, /// A point's x. X, /// A point's y. Y, - /// A geometry's points per row: a page's row count, each row's first point and the end, then - /// every point's x and y; parts are broken by a missing point. + /// A geometry's points per row: the page's row count, each row's first point index and the end + /// index, then every point's x and y. A missing point separates parts. Path, } -/// A column a program reads: of the mark's own table, of another dataset's table joined by row id, -/// or of the table the view derives beside the mark's own, row for row. +/// A column a program reads: from the mark's own table, from another dataset's table joined by row +/// id, or from the derived table the view keeps alongside the mark's table, row for row. #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub(crate) struct ColumnUse { - pub(crate) of: Option<(Name, Name)>, + pub(crate) from: Option<(Name, Name)>, pub(crate) derived: bool, pub(crate) name: Name, pub(crate) words: Words, pub(crate) sampled: bool, pub(crate) step: bool, - /// Also read the part `f32` drops of an `f64`, for positions. + /// Also read the low part that `f32` loses from an `f64`, for full-precision positions. pub(crate) low: bool, - /// Read every frame in view, as a line tracing the field does. + /// Read at every frame in view, like a line that traces the field. pub(crate) trace: bool, } @@ -80,7 +80,7 @@ impl ColumnUse { /// The mark's own field `name`, read as `words`. fn own(name: impl Into, words: Words) -> ColumnUse { ColumnUse { - of: None, + from: None, derived: false, name: name.into(), words, @@ -97,7 +97,15 @@ impl ColumnUse { } } -/// A constant holding a text's dictionary code in a column; the view writes it. +/// One end of each row of a mark: the vertex table it references, and the program column of the +/// reference field. The column is `NONE` when the end is the row's own vertex, as for a pole. +#[derive(Clone, PartialEq, Debug)] +pub(crate) struct EndUse { + pub(crate) table: Name, + pub(crate) column: u32, +} + +/// A constant that holds `text`'s dictionary code in `column`. The view fills it in. #[derive(Clone, PartialEq, Debug)] pub(crate) struct TextUse { pub(crate) constant: u32, @@ -105,7 +113,7 @@ pub(crate) struct TextUse { pub(crate) text: Arc, } -/// A mark's compiled channels and what they read. +/// A mark's compiled channels and their inputs. #[derive(Clone, Default, PartialEq, Debug)] pub(crate) struct Program { pub(crate) code: Vec<[u32; 4]>, @@ -113,23 +121,28 @@ pub(crate) struct Program { /// Op ranges of every channel, then of every reduction. pub(crate) ranges: Vec<[u32; 2]>, pub(crate) channels: Vec, - /// What each fitted domain's input reads. + /// The [`Reads`] of each fitted domain's input, then the inputs themselves. For a paged table, + /// a domain whose input reads only data (no view state or derived fields) is fitted on the CPU + /// over all rows. pub(crate) domains: Vec, + pub(crate) fitted: Vec, pub(crate) columns: Vec, pub(crate) texts: Vec, - /// Colormap ops, whose layer the view writes. + /// Each colormap op's index in `code`, with its colormap. The view writes the op's texture + /// layer. pub(crate) colormaps: Vec<(u32, Colormap)>, - /// Tables that ends point at, by end index. - pub(crate) ends: Vec, - /// State words stored in position units. + /// Each row's ends, in order. The end pass writes the GPU row holding end `i` into kind word + /// `i`. + pub(crate) ends: Vec, + /// Bit mask of the state words stored in position units. pub(crate) units: u32, } -/// What a channel or a fitted domain reads: kinds of input, and which of the program's columns. +/// The inputs of a channel or a fitted domain: kinds of input, and which of the program's columns. #[derive(Clone, Copy, Default, PartialEq, Eq, Debug)] pub(crate) struct Reads { pub(crate) deps: Deps, - /// Bit `i` for column `i`; the last bit stands for every column past it. + /// Bit `i` for column `i`. Bit 63 also covers every column after it. pub(crate) columns: u64, } @@ -146,7 +159,7 @@ pub(crate) struct Channel { } impl Program { - /// What all channels read together. + /// The kinds of input all channels read together. pub(crate) fn reads(&self) -> Deps { self.channels.iter().fold(Deps::default(), |all, c| all | c.reads.deps) } @@ -163,9 +176,9 @@ pub(crate) enum Store { /// The fields a mark's expressions can read. pub(crate) trait Fields { - /// The kind of `name` in the mark's table, or in `of`'s table. - fn kind(&self, of: Option<&(Name, Name)>, name: &str) -> Option; - /// The table a reference field of the mark's table names. + /// The kind of `name` in the mark's table, or in `from`'s table. + fn kind(&self, from: Option<&(Name, Name)>, name: &str) -> Option; + /// The table that reference field `name` of the mark's table points to. fn target(&self, name: &str) -> Option; } @@ -235,8 +248,8 @@ impl<'a> Compiler<'a> { Ok(()) } - /// Compiles a position channel reading the place the view derives for each row, at full - /// precision. + /// Compiles a position channel that reads each row's derived position (from a diagram's sheet + /// or the scene's layout), at full precision. pub(crate) fn derived_position(&mut self, word: u32) -> Result<()> { let start = self.begin(); let [x, y] = @@ -258,9 +271,9 @@ impl<'a> Compiler<'a> { Ok(()) } - /// The program column of the derived number field `field`, which shaders read directly. - pub(crate) fn derived_number(&mut self, field: &str) -> u32 { - self.column(ColumnUse::derived(field, Words::Values)) + /// The program column of the derived field `field`, read as `u32` words. + pub(crate) fn derived_bits(&mut self, field: &str) -> u32 { + self.column(ColumnUse::derived(field, Words::Bits)) } /// The program column of the derived geometry field `field`, read as points per row. @@ -273,7 +286,8 @@ impl<'a> Compiler<'a> { self.column(ColumnUse::own(field, Words::Path)) } - /// The program column of the mark's reference field `name`, read as the rows it names. + /// The program column of the mark's reference field `name`, read as the rows it references. + /// Errors if `name` is not a reference field. pub(crate) fn bits(&mut self, name: &str) -> Result { if self.fields.kind(None, name) != Some(Kind::Reference) { return Err(invalid("not a reference field")); @@ -281,89 +295,78 @@ impl<'a> Compiler<'a> { Ok(self.column(ColumnUse::own(name, Words::Bits))) } - /// Refuses `expr` for option `name` when it reads anything but static fields: the CPU works it - /// out from the data alone, as it does a row's priority. + /// Errors if `expr`, given for option `name`, reads anything but static fields. Options such as + /// a row's priority are computed on the CPU from the data alone. pub(crate) fn check_data(&self, name: &str, expr: &Expr) -> Result<()> { match self.unstructural(expr) { - Some(reason) => Err(invalid(format!("{name} reads data only, not {reason}")).at(name)), + Some(reason) => Err(invalid(format!("{name} can read only data, not {reason}")).at(name)), None => Ok(()), } } - /// Refuses `exprs` for option `name` when they read anything but static fields: where a - /// diagram's rows stand and how large they are is its structure, made from the data alone. + /// Errors if any of `exprs`, given for option `name`, reads anything but static fields. A + /// diagram's positions and sizes are its structure, which the sheet computes from data alone. pub(crate) fn check_structure<'e>(&self, name: &str, exprs: impl IntoIterator) -> Result<()> { for expr in exprs { if let Some(reason) = self.unstructural(expr) { - return Err(invalid(format!("a diagram's {name} reads data only, not {reason}")).at(name)); + return Err(invalid(format!("a diagram's {name} can read only data, not {reason}")).at(name)); } } Ok(()) } - /// What `expr` reads that structure cannot: view state or a sampled field. + /// The first input of `expr` that structure cannot depend on: view state or a sampled field. fn unstructural(&self, expr: &Expr) -> Option<&'static str> { match &*expr.0 { Node::State(State::Selected | State::Hovered) => Some("selection or hover"), Node::State(State::Pointer) => Some("the pointer"), Node::State(State::Elapsed) => Some("time"), Node::State(State::Zoom) => Some("the zoom"), - Node::Field(f) if self.fields.kind(f.of.as_ref(), &f.name) == Some(Kind::Sampled) => { + Node::Field(f) if self.fields.kind(f.from.as_ref(), &f.name) == Some(Kind::Sampled) => { Some("a sampled field") } - Node::Unary(_, a) | Node::Is(a, _) | Node::Contains(a, _) => self.unstructural(a), - Node::Binary(_, a, b) => self.unstructural(a).or_else(|| self.unstructural(b)), - Node::When(c, a, b) => { - self.unstructural(c).or_else(|| self.unstructural(a)).or_else(|| self.unstructural(b)) - } - Node::Scale { input, .. } | Node::Colormap { input, .. } => self.unstructural(input), - _ => None, + _ => expr.children().find_map(|child| self.unstructural(child)), } } - /// Compiles end `index` of an edge: the vertex the reference field `name` names. - pub(crate) fn end(&mut self, name: &str, word: u32) -> Result<()> { - let start = self.begin(); + /// Adds the next end of each row: the vertex that reference field `name` references. + pub(crate) fn end(&mut self, name: &str) -> Result<()> { if self.fields.kind(None, name) != Some(Kind::Reference) { return Err(invalid("not a reference field").at(name)); } - let target = self.fields.target(name).ok_or_else(|| invalid("not a reference field").at(name))?; + let table = self.fields.target(name).ok_or_else(|| invalid("not a reference field").at(name))?; let column = self.column(ColumnUse::own(name, Words::Bits)); - let index = self.program.ends.len() as u32; - self.program.ends.push(target); - self.deps |= Deps::DATA; - self.op(wgsl::OP_STORE_END, word, column, index, 0, 0)?; - self.close(name, start); + self.program.ends.push(EndUse { table, column }); Ok(()) } - /// Compiles an end storing each row's own vertex slot at `word`: the vertex mark drawing the - /// mark's own table, row for row, as a pole stands on. - pub(crate) fn own_end(&mut self, table: &Name, word: u32) -> Result<()> { - let start = self.begin(); - let index = self.program.ends.len() as u32; - self.program.ends.push(table.clone()); - self.op(wgsl::OP_STORE_END, word, wgsl::NONE, index, 0, 0)?; - self.close("poles", start); - Ok(()) + /// Adds the next end of each row: the row's own vertex, where the vertex mark of `table` (the + /// mark's own table) places it. A pole stands on it. + pub(crate) fn own_end(&mut self, table: &Name) { + self.program.ends.push(EndUse { table: table.clone(), column: wgsl::NONE }); } - /// Compiles channel `name` storing the code of text field `expr` at `word`, and returns the - /// program column whose dictionary names the codes. + /// Compiles channel `name`, which stores at `word` the dictionary code of text field `expr`. + /// `expr` can also be a `when` that returns that field or missing. A missing code draws no + /// label. Returns the program column whose dictionary decodes the codes. pub(crate) fn text(&mut self, name: &str, expr: &Expr, word: u32) -> Result { - let Node::Field(f) = &*expr.0 else { return Err(invalid("a label's text is a text field").at(name)) }; - if self.fields.kind(f.of.as_ref(), &f.name) != Some(Kind::Text) { + let Some(f) = text_field(expr, None).map_err(|e| e.at(name))? else { + return Err(invalid("a label's text needs a text field").at(name)); + }; + if self.fields.kind(f.from.as_ref(), &f.name) != Some(Kind::Text) { return Err(invalid("not a text field").at(&f.name).at(name)); } let start = self.begin(); + // The values compile like any conditional's, and a missing value as the missing code. + self.expect(expr, Ty::Text).map_err(|e| e.at(name))?; let column = self.field_column(f, Words::Codes, false); - self.op(wgsl::OP_FIELD, column, 0, 0, 0, 1)?; self.op(wgsl::OP_STORE_CODE, word, 0, 0, 1, 0)?; self.close(name, start); Ok(column) } - /// The column of sampled field `name` that a line traces over the axis. + /// The column of sampled field `name` that a line traces over the axis. Errors if `name` is not + /// a sampled field. pub(crate) fn trace(&mut self, name: &str) -> Result { if self.fields.kind(None, name) != Some(Kind::Sampled) { return Err(invalid("not a sampled field").at(name)); @@ -431,7 +434,7 @@ impl<'a> Compiler<'a> { /// The column of a plain number field or point component, with its low part. fn plain(&mut self, e: &Expr) -> Option { let Node::Field(f) = &*e.0 else { return None }; - if f.of.is_some() || f.step { + if f.from.is_some() || f.step { return None; } let words = match (self.fields.kind(None, &f.name)?, f.component) { @@ -445,7 +448,7 @@ impl<'a> Compiler<'a> { fn coordinate(&mut self, e: &Expr) -> Result<()> { if mentions_pointer(e) { - return Err(invalid("pointer() cannot place a position")); + return Err(invalid("a position cannot use pointer()")); } self.expect(e, Ty::Number) } @@ -463,7 +466,14 @@ impl<'a> Compiler<'a> { match (self.emit(e)?, want) { (got, want) if got == want => Ok(()), (Ty::Bool, Ty::Number) => Ok(()), - (got, want) => Err(invalid(format!("expected {}, found {}", ty_name(want), ty_name(got)))), + (got, want) => { + let hint = match (got, want) { + (Ty::Number, Ty::Color) => "; map a number to colors with colormap", + (Ty::Bool, Ty::Color) => "; choose between colors with when", + _ => "", + }; + Err(invalid(format!("expected {}, found {}{hint}", ty_name(want), ty_name(got)))) + } } } @@ -492,8 +502,8 @@ impl<'a> Compiler<'a> { Node::Binary(op, a, b) => self.binary(*op, a, b), Node::When(c, a, b) => { self.expect(c, Ty::Bool).map_err(|e| e.at("when"))?; - // A missing value takes the other branch's type, so it goes last: a missing first - // branch swaps the branches and negates the condition. + // A missing value takes the other branch's type, so it must compile second. A + // missing first branch swaps the branches and negates the condition. let (a, b) = match (&*a.0, &*b.0) { (Node::Missing, Node::Missing) => (a, b), (Node::Missing, _) => { @@ -511,16 +521,28 @@ impl<'a> Compiler<'a> { let ty = match (ta, self.emit(b)?) { (a, b) if a == b => a, (Ty::Bool, Ty::Number) | (Ty::Number, Ty::Bool) => Ty::Number, - (a, b) => return Err(invalid(format!("when chooses between {} and {}", ty_name(a), ty_name(b)))), + (a, b) => return Err(invalid(format!("when mixes {} and {}", ty_name(a), ty_name(b)))), }; self.op(wgsl::OP_SELECT, 0, 0, 0, 3, 1)?; Ok(ty) } - Node::Contains(..) => Err(invalid("contains reads texts, which only a query holds")), + Node::Contains(a, text) => { + // The view computes each row's match on the CPU, as a derived field alongside the + // mark's own table. + let Node::Field(f) = &*a.0 else { return Err(invalid("contains needs a text field")) }; + if f.from.is_some() || self.fields.kind(None, &f.name) != Some(Kind::Text) { + return Err(invalid("contains needs a text field of the mark's own table").at(&f.name)); + } + let name = crate::derive::matches::contains_of(&f.name, text); + let column = self.column(ColumnUse::derived(name, Words::Values)); + self.deps |= Deps::DATA; + self.op(wgsl::OP_FIELD, column, 0, 0, 0, 1)?; + Ok(Ty::Bool) + } Node::Is(a, text) => { - let Node::Field(f) = &*a.0 else { return Err(invalid("is compares a text field")) }; - if self.fields.kind(f.of.as_ref(), &f.name) != Some(Kind::Text) { - return Err(invalid("is compares a text field").at(&f.name)); + let Node::Field(f) = &*a.0 else { return Err(invalid("is needs a text field")) }; + if self.fields.kind(f.from.as_ref(), &f.name) != Some(Kind::Text) { + return Err(invalid("is needs a text field").at(&f.name)); } let column = self.field_column(f, Words::Codes, false); self.op(wgsl::OP_FIELD, column, 0, 0, 0, 1)?; @@ -558,7 +580,7 @@ impl<'a> Compiler<'a> { } fn field(&mut self, f: &FieldRef) -> Result { - let kind = self.fields.kind(f.of.as_ref(), &f.name).ok_or_else(|| match &f.of { + let kind = self.fields.kind(f.from.as_ref(), &f.name).ok_or_else(|| match &f.from { Some((data, table)) => invalid(format!("no field {} in {data}.{table}", f.name)), None => invalid(format!("no field {}", f.name)), })?; @@ -569,7 +591,7 @@ impl<'a> Compiler<'a> { (Kind::Sampled, None) => (Words::Values, Ty::Number, true), (Kind::Point, Some(0)) => (Words::X, Ty::Number, false), (Kind::Point, Some(1)) => (Words::Y, Ty::Number, false), - (Kind::Reference, _) => return Err(invalid("a reference names rows; read it as an end").at(&f.name)), + (Kind::Reference, _) => return Err(invalid("a reference field holds rows, not values").at(&f.name)), _ => return Err(invalid("not a number, boolean, or text field").at(&f.name)), }; let column = self.field_column(f, words, sampled); @@ -581,7 +603,7 @@ impl<'a> Compiler<'a> { fn field_column(&mut self, f: &FieldRef, words: Words, sampled: bool) -> u32 { self.deps |= if sampled { Deps::SAMPLES } else { Deps::DATA }; let step = f.step && sampled; - self.column(ColumnUse { of: f.of.clone(), sampled, step, ..ColumnUse::own(f.name.clone(), words) }) + self.column(ColumnUse { from: f.from.clone(), sampled, step, ..ColumnUse::own(f.name.clone(), words) }) } fn column(&mut self, use_: ColumnUse) -> u32 { @@ -624,7 +646,8 @@ impl<'a> Compiler<'a> { Ok(ty) } - /// A domain record in the constants; a fitted one adds a reduction of its input. + /// Adds a domain record to the constants and returns its index. A fitted domain also compiles a + /// reduction of its input. fn domain(&mut self, input: &Expr, domain: Domain) -> Result { let at = self.program.constants.len() as u32; match domain { @@ -633,12 +656,13 @@ impl<'a> Compiler<'a> { } Domain::Fit => { if self.reducing { - return Err(invalid("a fitted domain's input cannot fit another domain; set one with domain")); + return Err(invalid("nested fitted domains are not supported; give the inner one a domain")); } let index = self.program.domains.len() as u32; self.program.domains.push(Reads::default()); + self.program.fitted.push(input.clone()); self.program.constants.extend([1, 0, 0, index]); - // Compile the input again as a reduction; `finish` places it after the channels. + // Compile the input again as a reduction. `finish` places it after the channels. let saved = (std::mem::take(&mut self.program.code), self.depth, self.deps, self.columns_read); self.depth = 0; self.deps = Deps::default(); @@ -740,17 +764,27 @@ fn binary_name(op: Binary) -> &'static str { } } -fn mentions_pointer(e: &Expr) -> bool { - match &*e.0 { - Node::State(State::Pointer) => true, - Node::Unary(_, a) | Node::Is(a, _) | Node::Contains(a, _) => mentions_pointer(a), - Node::Binary(_, a, b) => mentions_pointer(a) || mentions_pointer(b), - Node::When(c, a, b) => mentions_pointer(c) || mentions_pointer(a) || mentions_pointer(b), - Node::Scale { input, .. } | Node::Colormap { input, .. } => mentions_pointer(input), - _ => false, +/// The one text field a label's text shows. It is `expr` itself, or the field a `when` returns when +/// every value is that field (same name and table) or missing. `seen` is the field found so far. +/// Returns `None` when every value is missing. +fn text_field<'e>(expr: &'e Expr, seen: Option<&'e FieldRef>) -> Result> { + match &*expr.0 { + Node::Missing => Ok(seen), + Node::Field(f) => match seen { + Some(g) if g.name != f.name || g.from != f.from => { + Err(invalid("a label's text chooses between one text field and missing, not two fields")) + } + _ => Ok(Some(f)), + }, + Node::When(_, a, b) => text_field(b, text_field(a, seen)?), + _ => Err(invalid("a label's text must be a text field, or a when choosing between one and missing")), } } +fn mentions_pointer(e: &Expr) -> bool { + e.any(&|e| matches!(&*e.0, Node::State(State::Pointer))) +} + fn ty_name(ty: Ty) -> &'static str { match ty { Ty::Number => "a number", diff --git a/crates/latkit/src/expr/cpu.rs b/crates/latkit/src/expr/cpu.rs index d057b0da..602104ac 100644 --- a/crates/latkit/src/expr/cpu.rs +++ b/crates/latkit/src/expr/cpu.rs @@ -1,6 +1,7 @@ -//! The expression machine on the CPU, for what a diagram's sheet reads (where rows stand, how large -//! they are, and the texts its blocks fit) and what a query keeps, sorts, and selects. The GPU runs -//! the same expressions for everything drawn; here they read fields, sampled ones at a coordinate. +//! Evaluates expressions on the CPU, for work that needs only data: a diagram's sheet +//! (positions, sizes, and block label texts), layouts, row priorities, fitted domains of paged +//! tables, and queries (filters, sort keys, and selections). The GPU evaluates the same expressions +//! for drawing. Here they read only fields, with sampled fields read at one axis coordinate. use std::cell::RefCell; use std::sync::Arc; @@ -9,37 +10,46 @@ use rustc_hash::FxHashMap; use super::{Binary, Domain, Expr, FieldRef, Node, Unary}; use crate::Kind; -use crate::data::{Data, Field, Geometry, Table, Value}; +use crate::data::{Field, Geometry, Snapshot, Table, Value}; /// The rows an expression reads: a table, and every dataset its joins may name. pub(crate) struct Rows<'a> { pub(crate) table: &'a Table, - pub(crate) all: &'a [Data], - /// The axis coordinate sampled fields are read at: their last frame by default. + pub(crate) all: &'a [Snapshot], + /// The axis coordinate at which sampled fields are read. Defaults to infinity, which reads + /// their last frame. at: f64, - /// Fitted domains already reduced, by the input they reduce. + /// Fitted domains already computed, keyed by their input node. fitted: RefCell>, + /// The texts `contains` looks for, lowercased once. + lowered: RefCell, Arc>>, } impl<'a> Rows<'a> { - pub(crate) fn new(table: &'a Table, all: &'a [Data]) -> Rows<'a> { - Rows { table, all, at: f64::INFINITY, fitted: RefCell::default() } + pub(crate) fn new(table: &'a Table, all: &'a [Snapshot]) -> Rows<'a> { + Rows { table, all, at: f64::INFINITY, fitted: RefCell::default(), lowered: RefCell::default() } } - /// The rows with sampled fields read at axis coordinate `at`. + /// `text` lowercased for `contains`, cached. + fn lowered(&self, text: &Arc) -> Arc { + self.lowered.borrow_mut().entry(text.clone()).or_insert_with(|| text.to_lowercase().into()).clone() + } + + /// These rows, with sampled fields read at axis coordinate `at`. pub(crate) fn at(mut self, at: f64) -> Rows<'a> { self.at = at; self } - /// `expr` for `row` as a number; `None` where it reads missing, or holds no number. + /// `expr` evaluated for `row`. `None` where the result is missing or not a finite number. pub(crate) fn number(&self, expr: &Expr, row: usize) -> Option { let v = self.eval(expr, row); v.is_finite().then_some(v) } - /// `expr` for every row, NaN where it reads missing, each made into a `T` by `make`: a plain - /// field read page by page, anything else row by row. + /// `expr` evaluated for every row and converted by `make`, with NaN for missing values. A + /// constant fills every row, and a plain field of the table is read page by page. Anything else + /// is evaluated row by row. pub(crate) fn numbers(&self, expr: &Expr, mut make: impl FnMut(f64) -> T) -> Vec { let n = self.table.len(); let mut out = Vec::with_capacity(n); @@ -48,12 +58,12 @@ impl<'a> Rows<'a> { out.resize_with(n, || make(*v)); return out; } - Node::Field(f) if f.of.is_none() => { + Node::Field(f) if f.from.is_none() => { if let Some(Field::Values(column, _)) = self.table.field_named(&f.name) { let pages = column.pages().iter().map(|p| &p.values); match (column.kind(), f.component) { (Kind::Number | Kind::Bool, None) => { - pages.for_each(|v| v.each_number(|x| out.push(make(x.unwrap_or(f64::NAN))))); + pages.for_each(|v| v.extend_numbers(&mut out, |x| make(x.unwrap_or(f64::NAN)))); return out; } (Kind::Point, Some(k)) => { @@ -76,18 +86,23 @@ impl<'a> Rows<'a> { out } - /// The text field `expr` names, for `row`. + /// The text `expr` gives `row`: a text field's value, or the value a `when` picks. `None` where + /// it is missing. pub(crate) fn text(&self, expr: &Expr, row: usize) -> Option> { - let Node::Field(f) = &*expr.0 else { return None }; - match self.value(f, row)? { - Value::Text(text) => Some(text), + match &*expr.0 { + Node::Field(f) => match self.value(f, row)? { + Value::Text(text) => Some(text), + _ => None, + }, + Node::When(c, a, b) => self.text(if self.eval(c, row) > 0.5 { a } else { b }, row), _ => None, } } - /// Field `f`'s value for `row`: of the table, or of a joined one by the row's id. + /// Field `f`'s value in `row`: from the table, or for a join, from the joined table's row with + /// the same id. fn value(&self, f: &FieldRef, row: usize) -> Option { - let value = match &f.of { + let value = match &f.from { None => self.table.value(row, &f.name, self.at)?, Some((data, table)) => { let other = self.all.iter().find(|d| d.name() == data)?.table(table)?; @@ -101,7 +116,7 @@ impl<'a> Rows<'a> { }) } - /// `expr` as the GPU computes it, NaN standing for missing. + /// `expr` computed as the GPU computes it, with NaN for missing. fn eval(&self, expr: &Expr, row: usize) -> f64 { match &*expr.0 { Node::Number(v) => *v, @@ -126,8 +141,8 @@ impl<'a> Rows<'a> { } Node::Contains(a, text) => { let Node::Field(f) = &*a.0 else { return f64::NAN }; - let held = |t: &str| t.to_lowercase().contains(&text.to_lowercase()); - f64::from(u8::from(matches!(self.value(f, row), Some(Value::Text(t)) if held(&t)))) + let needle = self.lowered(text); + f64::from(u8::from(matches!(self.value(f, row), Some(Value::Text(t)) if super::holds(&t, &needle)))) } Node::Scale { input, domain, range } => { let x = self.eval(input, row); @@ -146,7 +161,8 @@ impl<'a> Rows<'a> { } } - /// The extent of `input` over every row, as a fitted domain reduces it. + /// The extent of `input` over every row, as a fitted domain computes it. `[0, 1]` when no row + /// has a value. fn fit(&self, input: &Expr) -> [f64; 2] { let key = Arc::as_ptr(&input.0); if let Some(&extent) = self.fitted.borrow().get(&key) { @@ -209,7 +225,7 @@ mod tests { use crate::{Ids, Table, field, when}; #[test] - fn numbers_follow_the_gpu() { + fn cpu_evaluation_matches_the_gpu() { let table = Table::new("T", Ids::count(3)) .field("w", vec![Some(10.0), None, Some(30.0)]) .field("on", vec![true, false, true]); @@ -222,4 +238,13 @@ mod tests { let scaled: Expr = field("w").scale(0.0..=1.0).into(); assert_eq!(rows.number(&scaled, 2), Some(1.0), "a fitted domain spans the values"); } + + #[test] + fn a_conditional_text_gives_its_field_or_nothing() { + let table = Table::new("T", Ids::count(2)).field("name", vec!["a", "b"]).field("on", vec![true, false]); + let rows = Rows::new(&table, &[]); + let text = when(field("on")).then(field("name")).otherwise(Expr::missing()); + assert_eq!(rows.text(&text, 0).as_deref(), Some("a")); + assert_eq!(rows.text(&text, 1), None, "a missing text is no text"); + } } diff --git a/crates/latkit/src/expr/form.rs b/crates/latkit/src/expr/form.rs index bc25cc78..18bbf588 100644 --- a/crates/latkit/src/expr/form.rs +++ b/crates/latkit/src/expr/form.rs @@ -1,22 +1,30 @@ -//! Expressions, sizes, colors, and colormaps as JSON. A number or boolean is a constant, a string a -//! field (or a hex color when it starts with `#`), and an object one node, named by its key: +//! JSON forms of expressions, sizes, and colormaps. Every value follows one rule: a number or +//! boolean is a constant, `null` is a missing value, a string starting with `$` reads the field it +//! names, any other string is a CSS color, and an object is one node whose key names its kind. A +//! call has one key, which names its operation and holds its arguments: //! -//! | Form | Builder | -//! | ---------------------------------------------------------- | ------------------------------- | -//! | `{ "field": "x", "of": ["data", "Table"], "step": true }` | `field("x").of(..).step()` | -//! | `{ "color": "#ffb000" }` | a color | -//! | `{ "state": "selected" }` | `selected()`, `hovered()`, … | -//! | `{ "missing": true }` | `Expr::missing()` | -//! | `{ "op": "add", "args": [a, b] }` | `a + b`, `a.min(b)`, `!a`, … | -//! | `{ "when": [[c, v], …], "otherwise": e }` | `when(c).then(v)…otherwise(e)` | -//! | `{ "is": [a, "text"] }`, `{ "contains": [a, "text"] }` | `a.is("text")`, `a.contains(..)` | -//! | `{ "scale": [0, 1], "value": a, "domain": [lo, hi] }` | `a.scale(..).domain(..)` | -//! | `{ "colormap": "viridis", "value": a, "domain": [lo, hi] }` | `a.colormap(..).domain(..)` | -//! | `{ "units": a, "atLeast": 2, "atMost": 20 }` (sizes only) | `units(a).at_least(..)` | +//! | Form | Builder | +//! | ------------------------------------------------------------ | ---------------------------------- | +//! | `"$x"` | `field("x")` | +//! | `{ "field": "x", "from": ["data", "Table"], "step": true }` | `field("x").from_table(..).step()` | +//! | `"#ffb000"`, `"rebeccapurple"` | a [`Color`] | +//! | `null` | `Expr::missing()` | +//! | `{ "state": "selected" }` | `selected()`, `hovered()`, … | +//! | `{ "add": [a, b] }`, `{ "not": [a] }` | `a + b`, `!a`, `a.min(b)`, … | +//! | `{ "is": [a, "text"] }`, `{ "contains": [a, "text"] }` | `a.is("text")`, `a.contains(..)` | +//! | `{ "when": [[c, v], …], "otherwise": e }` | `when(c).then(v)…otherwise(e)` | +//! | `{ "scale": [0, 1], "value": a, "domain": [lo, hi] }` | `a.scale(..).domain(..)` | +//! | `{ "colormap": "viridis", "value": a, "domain": [lo, hi] }` | `a.colormap(..).domain(..)` | +//! | `{ "px": a }` (sizes only) | `px(a)` | +//! | `{ "units": a, "atLeast": 2, "atMost": 20 }` (sizes only) | `units(a).at_least(..)` | //! -//! Each object is its kind's form exactly: a key another kind has is refused. A map is a catalog name -//! or `{ "kind": "sequential", "colors": ["black", [0.2, "red"], "white"] }`, a color placed as a -//! `[place, color]` pair, and `categories` naming a categorical map's colors. +//! `neg`, `not`, `abs`, `sqrt`, `sin`, `cos`, `floor`, `ln`, and `exp` take one argument. `add`, +//! `sub`, `mul`, `div`, `rem`, `pow`, `min`, `max`, `lt`, `le`, `gt`, `ge`, `eq`, `ne`, `and`, and +//! `or` take two. `is` and `contains` take a value and a plain text, which is neither a field nor a +//! color. A field whose name starts with `$` is written `{ "field": "$x" }`. An object may have +//! only its kind's keys, and a key of another kind is an error. A colormap is a catalog name or +//! `{ "kind": "sequential", "colors": ["black", [0.2, "red"], "white"] }`. A `[place, color]` pair +//! places a color, and `categories` names a categorical map's colors. use std::borrow::Cow; use std::fmt; @@ -25,9 +33,9 @@ use std::sync::Arc; use schemars::{JsonSchema, Schema, SchemaGenerator, json_schema}; use serde::Deserialize; use serde::de::value::MapAccessDeserializer; -use serde::de::{self, Deserializer, IntoDeserializer, MapAccess, SeqAccess, Visitor}; +use serde::de::{self, Deserializer, MapAccess, SeqAccess, Visitor}; -use super::{Binary, Color, Colormap, ColormapKind, Expr, Node, Size, State, Stop, Unary, node}; +use super::{Binary, Color, Colormap, ColormapKind, Expr, Node, Size, State, Stop, Unary, Unit, node}; use crate::Name; impl<'de> Deserialize<'de> for Expr { @@ -36,23 +44,18 @@ impl<'de> Deserialize<'de> for Expr { } } -/// What an expression, a size, or a label is written as, read where it is written: a constant, a -/// text, or an object, whose keys say what it is once all are read. +/// The JSON of an expression, size, or label, before its kind is known: an expression already +/// read, or an object whose keys decide its kind once all are read. pub(crate) enum Written { - Constant(Expr), - Text(String), + Expr(Expr), Object(Box
), } impl Written { - /// The expression it writes: a text names a field, or a color when it starts with `#`. + /// The expression this JSON describes. pub(crate) fn expr(self) -> Result { match self { - Written::Constant(expr) => Ok(expr), - Written::Text(text) if text.starts_with('#') => { - Color::hex(&text).map(Expr::from).ok_or_else(|| format!("not a hex color: {text:?}")) - } - Written::Text(name) => Ok(super::field(name)), + Written::Expr(expr) => Ok(expr), Written::Object(form) => form.expr(), } } @@ -70,186 +73,213 @@ impl<'de> Visitor<'de> for Reading { type Value = Written; fn expecting(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str("a number, a boolean, a field's name, or an expression object") + f.write_str("a number, a boolean, null, a field or a color, or an expression object") } fn visit_bool(self, b: bool) -> Result { - Ok(Written::Constant(b.into())) + Ok(Written::Expr(b.into())) } fn visit_i64(self, n: i64) -> Result { - Ok(Written::Constant((n as f64).into())) + Ok(Written::Expr((n as f64).into())) } fn visit_u64(self, n: u64) -> Result { - Ok(Written::Constant((n as f64).into())) + Ok(Written::Expr((n as f64).into())) } fn visit_f64(self, n: f64) -> Result { - Ok(Written::Constant(n.into())) + Ok(Written::Expr(n.into())) + } + + fn visit_unit(self) -> Result { + Ok(Written::Expr(Expr::missing())) } fn visit_str(self, text: &str) -> Result { - Ok(Written::Text(text.to_owned())) + string(text).map(Written::Expr).map_err(E::custom) } - fn visit_map>(self, map: A) -> Result { - Form::deserialize(MapAccessDeserializer::new(map)).map(|form| Written::Object(Box::new(form))) + fn visit_map>(self, mut map: A) -> Result { + let mut form = Form::default(); + while let Some(key) = map.next_key::()? { + form.read(&key, &mut map)?; + } + Ok(Written::Object(Box::new(form))) } } -/// Every key an expression object, a size in units, or a label's options hold; each kind's own -/// alone make it. -#[derive(Deserialize)] -#[serde(rename_all = "camelCase", deny_unknown_fields)] -pub(crate) struct Form { - field: Option, - color: Option, - state: Option, - missing: Option, - op: Option, - when: Option>, - is: Option<(Expr, String)>, - contains: Option<(Expr, String)>, - scale: Option<[f64; 2]>, - colormap: Option, - of: Option<(Name, Name)>, - step: Option, - args: Option>, - otherwise: Option, - value: Option, - domain: Option<[f64; 2]>, - pub(crate) units: Option, - pub(crate) at_least: Option, - pub(crate) at_most: Option, - pub(crate) text: Option, - pub(crate) size: Option, - pub(crate) priority: Option, - pub(crate) max_width: Option, - pub(crate) font: Option>, - pub(crate) max: Option, +/// The expression a string writes: `$name` reads field `name`, and any other string is a CSS +/// color. +fn string(text: &str) -> Result { + match text.strip_prefix('$') { + Some("") => Err("\"$\" names no field".to_owned()), + Some(name) => Ok(super::field(name)), + None => Color::css(text) + .map(Expr::from) + .ok_or_else(|| format!("{text:?} is not a CSS color; \"${text}\" reads field {text}")), + } +} + +/// Declares `Form` and how it reads, from one list of keys, each with its JSON spelling. `null` +/// leaves a key unset, except a key marked `as value`, which holds an expression and reads `null` +/// as a missing value. +macro_rules! form { + (@read $map:ident, $ty:ty) => { + $map.next_value::>()? + }; + (@read $map:ident, $ty:ty, value) => { + Some($map.next_value::<$ty>()?) + }; + ($($vis:vis $key:ident $json:literal: $ty:ty $(as $how:ident)?),+ $(,)?) => { + /// Every key that an expression object, a size, or label options can have, and the call + /// made by a key that names an operation. Each kind accepts only its own keys. + #[derive(Default)] + pub(crate) struct Form { + $($vis $key: Option<$ty>,)+ + /// The operation a key names, and the call it makes with the key's arguments. + call: Option<(&'static str, Expr)>, + } + + impl Form { + /// The keys present, as JSON spells them. + fn held(&self) -> Vec<&'static str> { + let mut held = Vec::new(); + $(if self.$key.is_some() { + held.push($json); + })+ + if let Some((operation, _)) = &self.call { + held.push(*operation); + } + held + } + + /// Reads `key`'s value from `map`. Any other key names an operation, and this makes + /// its call. + fn read<'de, A: MapAccess<'de>>(&mut self, key: &str, map: &mut A) -> Result<(), A::Error> { + match key { + $($json => { + if self.$key.is_some() { + return Err(de::Error::custom(format!("{key} is given twice"))); + } + self.$key = form!(@read map, $ty $(, $how)?); + Ok(()) + })+ + _ => self.read_call(key, map), + } + } + } + }; +} + +form! { + field "field": Name, + state "state": State, + when "when": Vec<(Expr, Expr)>, + scale "scale": [f64; 2], + colormap "colormap": Colormap, + from "from": (Name, Name), + step "step": bool, + otherwise "otherwise": Expr as value, + value "value": Expr as value, + domain "domain": [f64; 2], + pub(crate) px "px": Expr as value, + pub(crate) units "units": Expr as value, + pub(crate) at_least "atLeast": f32, + pub(crate) at_most "atMost": f32, + pub(crate) text "text": Expr as value, + pub(crate) size "size": f32, + pub(crate) color "color": Expr as value, + pub(crate) max_width "maxWidth": f32, + pub(crate) font "font": Arc, } impl Form { - /// The keys it holds, as written. - fn held(&self) -> Vec<&'static str> { - [ - ("field", self.field.is_some()), - ("color", self.color.is_some()), - ("state", self.state.is_some()), - ("missing", self.missing.is_some()), - ("op", self.op.is_some()), - ("when", self.when.is_some()), - ("is", self.is.is_some()), - ("contains", self.contains.is_some()), - ("scale", self.scale.is_some()), - ("colormap", self.colormap.is_some()), - ("of", self.of.is_some()), - ("step", self.step.is_some()), - ("args", self.args.is_some()), - ("otherwise", self.otherwise.is_some()), - ("value", self.value.is_some()), - ("domain", self.domain.is_some()), - ("units", self.units.is_some()), - ("atLeast", self.at_least.is_some()), - ("atMost", self.at_most.is_some()), - ("text", self.text.is_some()), - ("size", self.size.is_some()), - ("priority", self.priority.is_some()), - ("maxWidth", self.max_width.is_some()), - ("font", self.font.is_some()), - ("max", self.max.is_some()), - ] - .into_iter() - .filter_map(|(key, held)| held.then_some(key)) - .collect() - } - - /// Why it is not `what`, holding only `keys`: the first other key it holds. + /// Errors unless every key present is one of `keys`, naming `what` and the first other key. pub(crate) fn only(&self, what: &str, keys: &[&str]) -> Result<(), String> { match self.held().into_iter().find(|key| !keys.contains(key)) { - Some(key) => Err(format!("{what} has no {key}")), + Some(key) => Err(format!("{what} takes no {key}")), None => Ok(()), } } - /// Its color: an expression's, or a label's. - pub(crate) fn color(&mut self) -> Result, String> { - self.color.take().map(Written::expr).transpose() + /// Reads from `map` the arguments of the operation `key` names, and makes the call. The + /// arguments are values, or a value and a text for `is` and `contains`. + fn read_call<'de, A: MapAccess<'de>>(&mut self, key: &str, map: &mut A) -> Result<(), A::Error> { + let Some(&(name, operation, _)) = OPERATIONS.iter().find(|(name, ..)| *name == key) else { + return Err(de::Error::custom(format!("{key:?} names no option or operation"))); + }; + if let Some((first, _)) = &self.call { + return Err(de::Error::custom(format!("an object names one operation, not {first} and {name}"))); + } + let count = |want: usize, got: usize| -> A::Error { + let plural = if want == 1 { "" } else { "s" }; + de::Error::custom(format!("{name} takes {want} argument{plural}, not {got}")) + }; + let call = match operation { + Operation::Unary(op) => { + let [a] = <[Expr; 1]>::try_from(map.next_value::>()?).map_err(|a| count(1, a.len()))?; + node(Node::Unary(op, a)) + } + Operation::Binary(op) => { + let [a, b] = <[Expr; 2]>::try_from(map.next_value::>()?).map_err(|a| count(2, a.len()))?; + node(Node::Binary(op, a, b)) + } + Operation::Is => { + let (a, text): (Expr, String) = map.next_value()?; + a.is(&text) + } + Operation::Contains => { + let (a, text): (Expr, String) = map.next_value()?; + a.contains(&text) + } + }; + self.call = Some((name, call)); + Ok(()) } - /// The expression it writes: the first kind it holds, and only that kind's keys. - pub(crate) fn expr(self) -> Result { + /// The expression this object describes: the call its operation makes, or the node of its + /// first kind key. It may have only that kind's keys. + pub(crate) fn expr(mut self) -> Result { + if let Some((operation, call)) = self.call.take() { + self.only(operation, &[])?; + return Ok(call); + } let held = self.held(); let Some(own) = held.iter().find_map(|key| KINDS.iter().find(|keys| keys[0] == *key)) else { - let mut kinds: Vec<&str> = KINDS.iter().map(|keys| keys[0]).collect(); - kinds.dedup(); - return Err(format!("an expression object has one of {}", kinds.join(", "))); + let kinds: Vec<&str> = KINDS.iter().map(|keys| keys[0]).collect(); + let found = if held.is_empty() { String::new() } else { format!("; found {}", held.join(", ")) }; + return Err(format!("an expression object needs an operation or one of {}{found}", kinds.join(", "))); }; let kind = own[0]; self.only(kind, own)?; - let Form { field, color, state, missing, op, when, is, contains, scale, colormap, .. } = self; + let Form { field, state, when, scale, colormap, from, step, otherwise, value, domain, .. } = self; let needs = |key: &str| format!("{kind} needs {key}"); Ok(match kind { "field" => { - let expr = super::field(field.expect("held")); - let expr = match self.of { - Some((data, table)) => expr.of(data, table), + let expr = super::field(field.expect("present")); + let expr = match from { + Some((data, table)) => expr.from_table(data, table), None => expr, }; - if self.step == Some(true) { expr.step() } else { expr } - } - "color" => match color { - Some(Written::Text(css)) => Color::css(&css).ok_or_else(|| format!("not a CSS color: {css:?}"))?.into(), - _ => return Err("a color is a CSS color".to_owned()), - }, - "state" => node(Node::State(state.expect("held"))), - "missing" => { - if missing != Some(true) { - return Err("missing is true".to_owned()); - } - Expr::missing() - } - "op" => { - let op = op.expect("held"); - let args = self.args.ok_or_else(|| needs("args"))?; - let name = || IntoDeserializer::::into_deserializer(op.as_str()); - if let Ok(unary) = Unary::deserialize(name()) { - let [a] = - <[Expr; 1]>::try_from(args).map_err(|a| format!("{op} takes 1 argument, not {}", a.len()))?; - node(Node::Unary(unary, a)) - } else { - let binary = Binary::deserialize(name()).map_err(|_| format!("no op {op:?}"))?; - let [a, b] = - <[Expr; 2]>::try_from(args).map_err(|a| format!("{op} takes 2 arguments, not {}", a.len()))?; - node(Node::Binary(binary, a, b)) - } + if step == Some(true) { expr.step() } else { expr } } + "state" => node(Node::State(state.expect("present"))), "when" => { - let otherwise = self.otherwise.ok_or_else(|| needs("otherwise"))?; - when.expect("held").into_iter().rev().fold(otherwise, |rest, (c, v)| node(Node::When(c, v, rest))) - } - "is" => { - let (a, text) = is.expect("held"); - a.is(&text) - } - "contains" => { - let (a, text) = contains.expect("held"); - a.contains(&text) + let otherwise = otherwise.ok_or_else(|| needs("otherwise"))?; + let cases = when.expect("present"); + cases.into_iter().rev().fold(otherwise, |rest, (c, v)| node(Node::When(c, v, rest))) } "scale" => { - let [lo, hi] = scale.expect("held"); - domained(self.value.ok_or_else(|| needs("a value"))?.scale(lo..=hi), self.domain) - } - _ => { - let value = self.value.ok_or_else(|| needs("a value"))?; - domained(value.colormap(colormap.expect("held")), self.domain) + let [lo, hi] = scale.expect("present"); + domained(value.ok_or_else(|| needs("a value"))?.scale(lo..=hi), domain) } + _ => domained(value.ok_or_else(|| needs("a value"))?.colormap(colormap.expect("present")), domain), }) } } -/// `mapped` with `domain` when given, else fitting the values shown. +/// `mapped` with `domain` when given, else fitted to the values shown. fn domained(mapped: super::Mapped, domain: Option<[f64; 2]>) -> Expr { match domain { Some([lo, hi]) => mapped.domain(lo..=hi).into(), @@ -257,24 +287,77 @@ fn domained(mapped: super::Mapped, domain: Option<[f64; 2]>) -> Expr { } } -/// Each kind of expression object, declared once: the schema of its JSON, which reading never builds, -/// and the keys reading lets an object of that kind hold, its own key first. +/// What a call computes from its arguments. +#[derive(Clone, Copy)] +enum Operation { + Unary(Unary), + Binary(Binary), + Is, + Contains, +} + +/// Each operation a call can name: the key that names it, what it computes, and its description. +const OPERATIONS: [(&str, Operation, &str); 27] = [ + ("neg", Operation::Unary(Unary::Neg), "The value negated."), + ("not", Operation::Unary(Unary::Not), "1 where the value is at most 0.5, else 0."), + ("abs", Operation::Unary(Unary::Abs), "The absolute value."), + ("sqrt", Operation::Unary(Unary::Sqrt), "The square root. Negative values give a missing value."), + ("sin", Operation::Unary(Unary::Sin), "The sine of the value, an angle in radians."), + ("cos", Operation::Unary(Unary::Cos), "The cosine of the value, an angle in radians."), + ("floor", Operation::Unary(Unary::Floor), "The largest whole number at most the value."), + ("ln", Operation::Unary(Unary::Ln), "The natural logarithm. Values at most 0 give a missing value."), + ("exp", Operation::Unary(Unary::Exp), "e raised to the value."), + ("add", Operation::Binary(Binary::Add), "The sum of the two values."), + ("sub", Operation::Binary(Binary::Sub), "The first value minus the second."), + ("mul", Operation::Binary(Binary::Mul), "The product of the two values."), + ( + "div", + Operation::Binary(Binary::Div), + "The first value divided by the second. Dividing by 0 gives a missing value.", + ), + ( + "rem", + Operation::Binary(Binary::Rem), + "The remainder of the first value divided by the second, with the sign of the second: -1 and 3 give 2.", + ), + ("pow", Operation::Binary(Binary::Pow), "The first value raised to the power of the second."), + ("min", Operation::Binary(Binary::Min), "The smaller of the two values."), + ("max", Operation::Binary(Binary::Max), "The larger of the two values."), + ("lt", Operation::Binary(Binary::Lt), "1 where the first value is less than the second, else 0."), + ("le", Operation::Binary(Binary::Le), "1 where the first value is at most the second, else 0."), + ("gt", Operation::Binary(Binary::Gt), "1 where the first value is greater than the second, else 0."), + ("ge", Operation::Binary(Binary::Ge), "1 where the first value is at least the second, else 0."), + ("eq", Operation::Binary(Binary::Eq), "1 where the two values are equal, else 0. Colors compare too."), + ("ne", Operation::Binary(Binary::Ne), "1 where the two values differ, else 0. Colors compare too."), + ("and", Operation::Binary(Binary::And), "1 where both values are above 0.5, else 0."), + ("or", Operation::Binary(Binary::Or), "1 where either value is above 0.5, else 0."), + ("is", Operation::Is, "Whether a text field equals the plain text exactly."), + ( + "contains", + Operation::Contains, + "Whether a text field contains the plain text, ignoring case. In a mark, the field must belong to the mark's own table.", + ), +]; + +/// Declares each kind of expression object once. Each kind gets a struct for its JSON schema (never +/// built, since reading goes through `Form`) and the list of keys an object of that kind may have, +/// its own key first. macro_rules! kinds { ($($(#[$meta:meta])* $node:ident { $($(#[$attr:meta])* $key:ident: $ty:ty),+ $(,)? })+) => { $( $(#[$meta])* #[derive(JsonSchema)] - #[expect(dead_code, reason = "its schema alone: expressions read through `Form`")] + #[expect(dead_code, reason = "used for its schema only; `Form` reads expressions")] #[serde(deny_unknown_fields)] struct $node { $($(#[$attr])* $key: $ty,)+ } )+ - /// Each kind's keys, its own first, in the order an object's keys name its kind. + /// Each kind's keys, its own key first. const KINDS: &[&[&str]] = &[$(&[$(stringify!($key)),+]),+]; - /// Each kind's schema, in that order. + /// Each kind's schema, in `KINDS` order. fn kinds(generator: &mut SchemaGenerator) -> Vec { vec![$(generator.subschema_for::<$node>()),+] } @@ -282,91 +365,88 @@ macro_rules! kinds { } kinds! { - /// A field of each row: of the mark's own table, or of another dataset's table joined by row id. + /// A field's value in each row, from the mark's own table. With `from`, it reads another + /// dataset's table, joined by row id. `"$name"` is the short form of `{ "field": "name" }`. FieldNode { + /// The field's name. field: Name, /// The dataset and table the field is read from, joined by row id. #[serde(default)] - #[schemars(extend("tsType" = "[data: string, table: string]"))] - of: Option<(Name, Name)>, - /// Reads a sampled field at its last frame without interpolating. + #[schemars(extend("tsType" = "TableInput"))] + from: Option<(Name, Name)>, + /// For a sampled field, reads the last frame at or before the shown coordinate instead of + /// interpolating. Defaults to `false`. #[serde(default)] step: bool, } - ColorNode { - color: Color, - } - /// What the view knows of each row. + /// View state of each row, such as whether it is selected. StateNode { + /// The state read. state: State, } - /// A missing value: options that read one take the theme's. - MissingNode { - #[schemars(extend("const" = true))] - missing: bool, - } - UnaryNode { - op: Unary, - args: [Expr; 1], - } - BinaryNode { - op: Binary, - args: [Expr; 2], - } /// The value of the first case whose condition holds, else `otherwise`. WhenNode { + /// The cases, as `[condition, value]` pairs tried in order. #[schemars(extend("tsType" = "[condition: ExprInput, value: ExprInput][]"))] when: Vec<(Expr, Expr)>, + /// The value where no condition holds. otherwise: Expr, } - /// Whether a text field equals the text. - IsNode { - #[schemars(extend("tsType" = "[field: ExprInput, text: string]"))] - is: (Expr, String), - } - /// Whether a text field holds the text, ignoring case: read by queries, which hold texts. - ContainsNode { - #[schemars(extend("tsType" = "[field: ExprInput, text: string]"))] - contains: (Expr, String), - } - /// `value` mapped linearly onto the range `scale`, clamped: from its `domain`, else from the extent of - /// the values shown. + /// A value mapped linearly onto a range, clamped to the range. ScaleNode { + /// The range. #[schemars(extend("tsType" = "[low: number, high: number]"))] scale: [f64; 2], + /// The value mapped. value: Expr, + /// The values mapped to the range's ends. Defaults to the extent of the values shown. #[serde(default)] #[schemars(extend("tsType" = "[low: number, high: number]"))] domain: Option<[f64; 2]>, } - /// `value` through `colormap`: from its `domain`, else from the extent of the values shown. + /// A value mapped through a colormap. ColormapNode { + /// The colormap. colormap: Colormap, + /// The value mapped. A categorical map takes each value's integer part as its category. value: Expr, + /// The values mapped to the colormap's ends. Defaults to the extent of the values shown. A + /// categorical map ignores it. #[serde(default)] #[schemars(extend("tsType" = "[low: number, high: number]"))] domain: Option<[f64; 2]>, } } -/// A size in position units, so it scales as the view zooms, kept within CSS pixels when given. +/// A size in CSS pixels, regardless of the space's unit. #[derive(JsonSchema)] -#[expect(dead_code, reason = "its schema alone: expressions read through `Form`")] +#[expect(dead_code, reason = "used for its schema only; `Form` reads sizes")] +#[serde(deny_unknown_fields)] +struct Px { + /// The size, in CSS pixels. + px: Expr, +} + +/// A size in position units (degrees on the earth), regardless of the space's unit. It scales with +/// the zoom. `atLeast` and `atMost` bound it in CSS pixels. +#[derive(JsonSchema)] +#[expect(dead_code, reason = "used for its schema only; `Form` reads sizes")] #[serde(rename_all = "camelCase", deny_unknown_fields)] #[allow(clippy::struct_field_names, reason = "its JSON is `{units, atLeast, atMost}`")] struct Units { + /// The size, in position units. units: Expr, - /// CSS pixels it keeps at least, however far the view zooms out. + /// The minimum on-screen size in CSS pixels, however far the view zooms out. #[serde(default)] #[schemars(range(min = 0))] at_least: Option, - /// CSS pixels it keeps at most, however far the view zooms in. + /// The maximum on-screen size in CSS pixels, however far the view zooms in. #[serde(default)] #[schemars(range(min = 0))] at_most: Option, } -/// An expression is its JSON, `ExprNode`; TypeScript also takes the builders that write it. +/// An expression's schema is `ExprNode`. Its TypeScript type also accepts the builders. impl JsonSchema for Expr { fn schema_name() -> Cow<'static, str> { "Expr".into() @@ -379,7 +459,7 @@ impl JsonSchema for Expr { } } -/// The JSON an expression is written as. +/// The JSON form of an expression. struct ExprNode; impl JsonSchema for ExprNode { @@ -388,27 +468,82 @@ impl JsonSchema for ExprNode { } fn json_schema(generator: &mut SchemaGenerator) -> Schema { - let forms = kinds(generator); let mut any = vec![ json_schema!({ "type": "number" }), json_schema!({ "type": "boolean" }), - json_schema!({ "type": "string", "description": "A field's name, or a hex color when it starts with `#`." }), + json_schema!({ + "type": "null", + "description": "A missing value. An option that is missing uses the theme's value.", + }), + json_schema!({ + "type": "string", + "description": "A field or a color. `\"$kv\"` reads field `kv`. Any other string is a CSS color, such as `#ffb000` or `rebeccapurple`. Write a field named `$x` as `{ \"field\": \"$x\" }`.", + }), ]; - any.extend(forms); + any.extend(kinds(generator)); + any.push(generator.subschema_for::()); json_schema!({ - "description": "A value for every row of a mark: a constant, a field, view state, or arithmetic of them.", + "description": "A value for each row. A number or boolean is a constant and `null` is missing. A string starting with `$` reads that field, and any other string is a CSS color. An object computes a value.", "anyOf": any, }) } } +/// The JSON form of a call: an object whose one key names an operation and holds its arguments. +struct CallNode; + +impl JsonSchema for CallNode { + fn schema_name() -> Cow<'static, str> { + "CallNode".into() + } + + fn json_schema(generator: &mut SchemaGenerator) -> Schema { + let value = generator.subschema_for::(); + let calls: Vec = OPERATIONS + .iter() + .map(|&(name, operation, description)| { + let mut arguments = match operation { + Operation::Unary(_) => { + json_schema!({ "type": "array", "prefixItems": [value], "minItems": 1, "maxItems": 1 }) + } + Operation::Binary(_) => { + json_schema!({ "type": "array", "prefixItems": [value, value], "minItems": 2, "maxItems": 2 }) + } + Operation::Is | Operation::Contains => json_schema!({ + "type": "array", + "prefixItems": [value, { "type": "string" }], + "minItems": 2, + "maxItems": 2, + "tsType": "[field: ExprInput, text: string]", + }), + }; + arguments.insert("description".into(), description.into()); + json_schema!({ + "type": "object", + "properties": { name: arguments }, + "required": [name], + "additionalProperties": false, + }) + }) + .collect(); + json_schema!({ + "description": "An operation, written as an object with one key: the operation's name, holding its arguments. For example `{ \"add\": [\"$a\", 1] }`. The second argument of `is` and `contains` is plain text, not a field or a color.", + "anyOf": calls, + }) + } +} + impl<'de> Deserialize<'de> for Size { fn deserialize>(deserializer: D) -> Result { let size = |written: Written| match written { + Written::Object(form) if form.px.is_some() => { + form.only("a size in CSS pixels", &["px"])?; + Ok(Size { expr: form.px.expect("present"), unit: Some(Unit::Px), px: Size::ANY }) + } Written::Object(form) if form.units.is_some() => { form.only("a size in units", &["units", "atLeast", "atMost"])?; let px = [form.at_least.unwrap_or(0.0).max(0.0), form.at_most.unwrap_or(f32::INFINITY)]; - Ok(Size { expr: form.units.expect("held"), units: true, px }) + Ok(Size { expr: form.units.expect("present"), unit: Some(Unit::Units), px }) } written => written.expr().map(Size::from), }; @@ -423,10 +558,11 @@ impl JsonSchema for Size { fn json_schema(generator: &mut SchemaGenerator) -> Schema { let expr = generator.subschema_for::(); + let px = generator.subschema_for::(); let units = generator.subschema_for::(); json_schema!({ - "description": "A size: CSS pixels, or position units with `units`.", - "anyOf": [expr, units], + "description": "A size: a value in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. `{ units }` is position units (degrees on the earth), which grow as the view zooms in.", + "anyOf": [expr, px, units], }) } } @@ -438,7 +574,8 @@ impl<'de> Deserialize<'de> for Color { } } -/// A color writes as CSS `rgba(…)`, read back exactly: whole channels as bytes, alpha as it is. +/// Writes a color as CSS `rgba(…)` that reads back exactly: channels as byte values, fractional +/// where needed, and alpha unchanged. impl serde::Serialize for Color { fn serialize(&self, serializer: S) -> Result { let channel = |v: f32| { @@ -468,15 +605,17 @@ impl JsonSchema for Color { } } -/// Your own colormap: its kind, its colors, and for a categorical map its categories' names. +/// A custom colormap: its kind, its colors, and for a categorical map, its category names. #[derive(Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] struct CustomColormap { + /// How the colors map values. kind: ColormapKind, - /// Colors blended in OKLab, each at its `[place, color]` from 0 to 1 or evenly between its - /// neighbors, as in a CSS gradient; a categorical map's, one per category. + /// The colors, blended in OKLab. Each is a color, spaced evenly between its neighbors like a + /// CSS gradient, or a `[place, color]` pair with a place from 0 to 1. Places must ascend. A + /// categorical map takes one color per category, without places, at most 256. colors: Vec, - /// The categories' names, in order, for a legend: a categorical map's alone. + /// The category names in category order, shown in legends. Categorical maps only. #[serde(default)] categories: Vec, } @@ -520,10 +659,10 @@ impl JsonSchema for Colormap { } fn json_schema(generator: &mut SchemaGenerator) -> Schema { - let names: Vec<&str> = super::colormap::catalog().collect(); + let names: Vec<&str> = Colormap::catalog().filter_map(|map| map.name()).collect(); generator.definitions_mut().insert( "ColormapName".into(), - serde_json::json!({ "description": "A colormap of the catalog.", "type": "string", "enum": names }), + serde_json::json!({ "description": "A colormap from the catalog.", "type": "string", "enum": names }), ); let colors = generator.subschema_for::(); json_schema!({ @@ -596,26 +735,45 @@ mod tests { serde_json::from_str(json).expect("an expression") } + fn error(json: &str) -> String { + serde_json::from_str::(json).expect_err("an error").to_string() + } + #[test] fn forms_match_builders() { - assert_eq!(parse("\"vm\""), field("vm")); + assert_eq!(parse("\"$vm\""), field("vm")); assert_eq!(parse("2.5"), Expr::from(2.5)); - assert_eq!(parse(r#"{"op": "add", "args": ["a", 1]}"#), field("a") + 1.0); - assert_eq!(parse(r#"{"op": "not", "args": ["on"]}"#), !field("on")); + assert_eq!(parse("null"), Expr::missing()); + assert_eq!(parse(r#"{"add": ["$a", 1]}"#), field("a") + 1.0); + assert_eq!(parse(r#"{"not": ["$on"]}"#), !field("on")); + assert_eq!(parse(r#"{"min": ["$a", {"abs": ["$b"]}]}"#), field("a").min(field("b").abs())); assert_eq!( - parse(r##"{"when": [[{"state": "selected"}, {"color": "#ff0000"}]], "otherwise": "c"}"##), + parse(r##"{"when": [[{"state": "selected"}, "#ff0000"]], "otherwise": "$c"}"##), when(selected()).then(Color::rgb(1.0, 0.0, 0.0)).otherwise(field("c")) ); assert_eq!( - parse(r#"{"colormap": "balance", "value": "vm", "domain": [0.9, 1.1]}"#), + parse(r#"{"when": [["$on", "$name"]], "otherwise": null}"#), + when(field("on")).then(field("name")).otherwise(Expr::missing()), + "a label's text where `on` holds, and none elsewhere" + ); + assert_eq!( + parse(r#"{"field": "vm", "from": ["weather", "Station"], "step": true}"#), + field("vm").from_table("weather", "Station").step() + ); + assert_eq!( + parse(r#"{"colormap": "balance", "value": "$vm", "domain": [0.9, 1.1]}"#), field("vm").colormap(Colormap::BALANCE).domain(0.9..=1.1).into() ); - assert_eq!(parse(r#"{"scale": [2, 8], "value": "kv"}"#), field("kv").scale(2.0..=8.0).into()); - assert_eq!(parse(r#"{"contains": ["name", "lake"]}"#), field("name").contains("lake")); - let size: Size = serde_json::from_str(r#"{"units": 3}"#).expect("a size"); - assert_eq!(size, crate::units(3.0)); - let size: Size = serde_json::from_str(r#"{"units": 3, "atLeast": 2, "atMost": 20}"#).expect("a size"); - assert_eq!(size, crate::units(3.0).at_least(2.0).at_most(20.0)); + assert_eq!(parse(r#"{"scale": [2, 8], "value": "$kv"}"#), field("kv").scale(2.0..=8.0).into()); + assert_eq!(parse(r#"{"is": ["$kind", "substation"]}"#), field("kind").is("substation")); + assert_eq!(parse(r#"{"contains": ["$name", "lake"]}"#), field("name").contains("lake")); + let size = |json: &str| serde_json::from_str::(json).expect("a size"); + assert_eq!(size("3"), Size::from(3.0)); + assert_eq!(size("\"$r\""), Size::from(field("r"))); + assert_eq!(size(r#"{"px": 3}"#), crate::px(3.0)); + assert_eq!(size(r#"{"units": 3}"#), crate::units(3.0)); + assert_eq!(size(r#"{"units": 3, "atLeast": 2, "atMost": 20}"#), crate::units(3.0).at_least(2.0).at_most(20.0)); + assert!(serde_json::from_str::(r#"{"px": 3, "atLeast": 2}"#).is_err(), "px sizes take no bounds"); let map: Colormap = serde_json::from_str(r#"{"kind": "sequential", "colors": ["black", [0.25, "red"], "white"]}"#) .expect("a map"); @@ -624,15 +782,47 @@ mod tests { } #[test] - fn errors_name_what_is_wrong() { - let error = serde_json::from_str::(r#"{"field": "a", "color": "red"}"#).expect_err("two kinds"); - assert!(error.to_string().contains("color"), "{error}"); - let error = serde_json::from_str::(r#"{"field": "a", "map": "viridis"}"#).expect_err("a stray key"); - assert!(error.to_string().contains("map"), "{error}"); - let error = serde_json::from_str::(r#"{"op": "frob", "args": []}"#).expect_err("unknown op"); - assert!(error.to_string().contains("frob"), "{error}"); - let error = serde_json::from_str::(r#"{"op": "add", "args": ["a"]}"#).expect_err("two arguments"); - assert!(error.to_string().contains('2'), "{error}"); + fn a_string_with_a_dollar_reads_a_field_and_any_other_is_a_color() { + let red = Expr::from(Color::rgb(1.0, 0.0, 0.0)); + assert_eq!(parse("\"red\""), red); + assert_eq!(parse("\"#ff0000\""), red); + assert_eq!(parse("\"$#ff0000\""), field("#ff0000"), "after the `$`, any text names a field"); + assert_eq!(parse(r#"{"field": "$x"}"#), field("$x"), "the long form names any field"); + let wrong = error("\"level\""); + assert!(wrong.starts_with(r#""level" is not a CSS color; "$level" reads field level"#), "{wrong}"); + assert!(error("\"$\"").contains("names no field")); + // A call's text is plain: neither a field nor a color. + assert_eq!(parse(r#"{"is": ["$kind", "$red"]}"#), field("kind").is("$red")); + assert_eq!(parse(r#"{"contains": ["$name", "level"]}"#), field("name").contains("level")); + } + + #[test] + fn null_is_a_missing_value_wherever_a_value_goes() { + let chosen = parse(r#"{"when": [["$on", null]], "otherwise": "$v"}"#); + assert_eq!(chosen, when(field("on")).then(Expr::missing()).otherwise(field("v"))); + assert_eq!(parse(r#"{"max": ["$v", null]}"#), field("v").max(Expr::missing())); + assert_eq!( + parse(r#"{"colormap": "viridis", "value": null}"#), + Expr::missing().colormap(Colormap::VIRIDIS).into() + ); + let size = serde_json::from_str::(r#"{"px": null}"#).expect("a size"); + assert_eq!(size, crate::px(Expr::missing())); + // An option of a node that is no value stays unset. + assert_eq!(parse(r#"{"field": "v", "from": null}"#), field("v")); + } + + #[test] + fn errors_name_the_offending_part() { + assert!(error(r#"{"field": "a", "state": "selected"}"#).contains("field takes no state")); + assert!(error(r#"{"field": "a", "map": "viridis"}"#).contains("map")); + assert!(error(r#"{"frob": []}"#).contains("frob")); + let arguments = error(r#"{"add": ["$a"]}"#); + assert!(arguments.contains("add takes 2 arguments, not 1"), "{arguments}"); + assert!(error(r#"{"add": ["$a", 1], "field": "b"}"#).contains("add takes no field")); + assert!(error(r#"{"add": [1, 2], "mul": [3, 4]}"#).contains("one operation")); + assert!(error(r#"{"color": "red"}"#).contains("found color"), "a color is a string, not a node"); + assert!(error(r#"{"missing": true}"#).contains("missing")); + assert!(error(r#"{"op": "add", "args": ["$a", 1]}"#).contains("op")); let error = serde_json::from_str::(r#"{"kind": "cyclic", "colors": [[0.5, "red"], [0.2, "blue"]]}"#) .expect_err("descending places"); assert!(error.to_string().contains("ascend"), "{error}"); diff --git a/crates/latkit/src/expr/mod.rs b/crates/latkit/src/expr/mod.rs index 309465ae..b6eac5a7 100644 --- a/crates/latkit/src/expr/mod.rs +++ b/crates/latkit/src/expr/mod.rs @@ -1,4 +1,5 @@ -//! Per-row values: expressions over fields and view state, compiled to bytecode the GPU runs. +//! Expressions: per-row values computed from fields and view state, compiled to bytecode the GPU +//! runs. mod color; mod colormap; @@ -18,9 +19,9 @@ pub use colormap::{Colormap, ColormapKind, Stop}; use crate::Name; -/// A value for every row of a mark: a constant, a field, view state, or arithmetic of them. -/// -/// A `&str` converts to the field of that name, so options read fields by name: +/// A per-row value of a mark: a constant, a color, a field, view state, or arithmetic on them. +/// Color options take an expression that computes a color: a [`Color`], a value through a +/// colormap, or a conditional that picks between colors. /// /// ``` /// # use latkit::{field, when, selected, Colormap, Color}; @@ -30,7 +31,8 @@ use crate::Name; /// let high_voltage = field("kv").ge(230.0) & field("kind").is("substation"); /// ``` /// -/// Expressions compile to bytecode; changing one never compiles a shader. +/// Expressions compile to bytecode, so changing one never recompiles a shader. In JSON, a string +/// starting with `$` reads the field it names, and any other string is a CSS color. #[derive(Clone, PartialEq, Debug)] pub struct Expr(pub(crate) Arc); @@ -51,17 +53,19 @@ pub(crate) enum Node { Colormap { input: Expr, colormap: Colormap, domain: Domain }, } -/// A field read: of the mark's own table, or of another dataset's table joined by row id. +/// A field reference: a field of the mark's own table, or of another dataset's table joined by row +/// id. #[derive(Clone, PartialEq, Debug)] pub(crate) struct FieldRef { pub(crate) name: Name, - pub(crate) of: Option<(Name, Name)>, + pub(crate) from: Option<(Name, Name)>, pub(crate) step: bool, - /// For a point geometry: 0 for x, 1 for y. + /// The component read from a point field: 0 for x, 1 for y. pub(crate) component: Option, } -/// What the view knows of each row. +/// View state of each row: `selected`, `hovered`, `pointer` (CSS pixels from a vertex to the +/// pointer), `elapsed` (seconds since the view was created), or `zoom` (the camera's zoom). #[derive(Clone, Copy, PartialEq, Eq, Debug, serde::Deserialize, schemars::JsonSchema)] #[serde(rename_all = "lowercase")] pub(crate) enum State { @@ -72,9 +76,9 @@ pub(crate) enum State { Zoom, } -/// An operation of one value. -#[derive(Clone, Copy, PartialEq, Eq, Debug, serde::Deserialize, schemars::JsonSchema)] -#[serde(rename_all = "lowercase")] +/// An operation on one value. `sin` and `cos` take radians. `not` treats values above 0.5 as +/// true. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) enum Unary { Neg, Not, @@ -87,9 +91,10 @@ pub(crate) enum Unary { Exp, } -/// An operation of two values. -#[derive(Clone, Copy, PartialEq, Eq, Debug, serde::Deserialize, schemars::JsonSchema)] -#[serde(rename_all = "lowercase")] +/// An operation on two values: arithmetic, `min` and `max`, comparisons (1 for true, 0 for +/// false), and logic. `rem` is a floored remainder, which takes the sign of the divisor. `and` and +/// `or` treat values above 0.5 as true. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) enum Binary { Add, Sub, @@ -111,7 +116,7 @@ pub(crate) enum Binary { #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) enum Domain { - /// The extent of the values shown, reduced on the GPU. + /// Fitted to the extent of the values shown, computed by a reduction on the GPU. Fit, Fixed([f64; 2]), } @@ -120,9 +125,21 @@ pub(crate) fn node(node: Node) -> Expr { Expr(Arc::new(node)) } -/// The field `name` of each row. +/// Whether `text` contains `needle`, ignoring case. `needle` must already be lowercase. Queries and +/// marks share this test for `contains`. +pub(crate) fn holds(text: &str, needle: &str) -> bool { + if needle.is_empty() { + return true; + } + if text.is_ascii() && needle.is_ascii() { + return text.as_bytes().windows(needle.len()).any(|w| w.eq_ignore_ascii_case(needle.as_bytes())); + } + text.to_lowercase().contains(needle) +} + +/// The value of field `name` in each row, written `"$name"` in JSON. pub fn field(name: impl Into) -> Expr { - node(Node::Field(FieldRef { name: name.into(), of: None, step: false, component: None })) + node(Node::Field(FieldRef { name: name.into(), from: None, step: false, component: None })) } /// Starts a conditional: `when(c).then(a).when(d).then(b).otherwise(e)`. @@ -135,31 +152,37 @@ pub fn selected() -> Expr { node(Node::State(State::Selected)) } -/// Whether the pointer is over the row. +/// Whether the row is hovered, by the pointer or by keyboard stepping. pub fn hovered() -> Expr { node(Node::State(State::Hovered)) } -/// CSS pixels from a vertex to the pointer; for vertices only, and not in their position. +/// CSS pixels from each vertex to the pointer. Use it only in vertex marks, and not in a position. pub fn pointer() -> Expr { node(Node::State(State::Pointer)) } -/// Seconds since the view was created; a mark that reads it redraws every frame. +/// Seconds since the view was created. A mark that reads it redraws every frame. pub fn elapsed() -> Expr { node(Node::State(State::Elapsed)) } -/// The camera's zoom: the web mercator level on the earth, log₂ CSS pixels per unit elsewhere. Options -/// that read it change with the view, as levels of detail do. +/// The camera's zoom: the web mercator zoom level on the earth, and log₂ of CSS pixels per unit +/// elsewhere. Options that read it update as the camera zooms, like levels of detail. pub fn zoom() -> Expr { node(Node::State(State::Zoom)) } -/// A size measured in position units, so it scales as the view zooms; [`Size::at_least`] and -/// [`Size::at_most`] keep it within CSS pixels. +/// A size in position units (degrees on the earth), regardless of the space's unit. It scales with +/// the zoom. [`Size::at_least`] and [`Size::at_most`] bound it in CSS pixels. pub fn units(size: impl Into) -> Size { - Size { expr: size.into(), units: true, px: Size::ANY } + Size { expr: size.into(), unit: Some(Unit::Units), px: Size::ANY } +} + +/// A size in CSS pixels, regardless of the space's unit. It keeps its size on screen as the view +/// zooms. In a diagram, this keeps a stroke's width constant on screen. +pub fn px(size: impl Into) -> Size { + Size { expr: size.into(), unit: Some(Unit::Px), px: Size::ANY } } macro_rules! unary { @@ -183,7 +206,7 @@ macro_rules! binary { } impl Expr { - /// A missing value: options that read one take the theme's value. + /// A missing value, `null` in JSON. An option that evaluates to missing uses the theme's value. pub fn missing() -> Expr { node(Node::Missing) } @@ -193,11 +216,11 @@ impl Expr { abs => Abs, /// The square root. sqrt => Sqrt, - /// The sine of radians. + /// The sine of an angle in radians. sin => Sin, - /// The cosine of radians. + /// The cosine of an angle in radians. cos => Cos, - /// The largest integer at most the value. + /// The largest integer not greater than the value. floor => Floor, /// The natural logarithm. ln => Ln, @@ -206,9 +229,9 @@ impl Expr { ); binary!( - /// The smaller value. + /// The smaller of the value and `other`. min => Min, - /// The larger value. + /// The larger of the value and `other`. max => Max, /// The value raised to `other`. pow => Pow, @@ -240,14 +263,14 @@ impl Expr { t.clone() * t.clone() * (3.0 - 2.0 * t) } - /// Whether a text field equals `text`. + /// Whether a text field equals `text` exactly. #[must_use] pub fn is(self, text: &str) -> Expr { node(Node::Is(self, text.into())) } - /// Whether a text field holds `text`, ignoring case. Only a [`Query`](crate::Query) reads it, as - /// the GPU holds no texts. + /// Whether a text field contains `text`, ignoring case. In a mark, the field must belong to the + /// mark's own table. The view recomputes matches on the CPU when the field changes. #[must_use] pub fn contains(self, text: &str) -> Expr { node(Node::Contains(self, text.into())) @@ -255,9 +278,9 @@ impl Expr { /// Reads this expression's fields from `table` of dataset `data`, joined by row id. #[must_use] - pub fn of(self, data: impl Into, table: impl Into) -> Expr { - let of = (data.into(), table.into()); - self.map_fields(&|f| FieldRef { of: Some(of.clone()), ..f.clone() }) + pub fn from_table(self, data: impl Into, table: impl Into) -> Expr { + let from = (data.into(), table.into()); + self.map_fields(&|f| FieldRef { from: Some(from.clone()), ..f.clone() }) } /// Reads sampled fields at the last frame at or before the shown coordinate, without @@ -267,21 +290,22 @@ impl Expr { self.map_fields(&|f| FieldRef { step: true, ..f.clone() }) } - /// The value through `colormap`; the domain fits the values shown unless set by - /// [`Mapped::domain`]. + /// The value mapped through `colormap`. The domain defaults to the extent of the values shown. + /// Set it with [`Mapped::domain`]. A categorical map ignores the domain and bins values by + /// their integer part. pub fn colormap(self, colormap: impl Into) -> Mapped { Mapped(node(Node::Colormap { input: self, colormap: colormap.into(), domain: Domain::Fit })) } - /// The value mapped linearly onto `range`, clamped; the domain fits the values shown unless set - /// by [`Mapped::domain`]. + /// The value mapped linearly onto `range`, clamped. The domain defaults to the extent of the + /// values shown. Set it with [`Mapped::domain`]. pub fn scale(self, range: RangeInclusive) -> Mapped { let range = [*range.start(), *range.end()]; Mapped(node(Node::Scale { input: self, domain: Domain::Fit, range })) } pub(crate) fn component(name: impl Into, component: u8) -> Expr { - node(Node::Field(FieldRef { name: name.into(), of: None, step: false, component: Some(component) })) + node(Node::Field(FieldRef { name: name.into(), from: None, step: false, component: Some(component) })) } fn map_fields(&self, f: &impl Fn(&FieldRef) -> FieldRef) -> Expr { @@ -302,53 +326,64 @@ impl Expr { } } - /// Calls `f` with every field the expression reads. - pub(crate) fn for_each_field<'e>(&'e self, f: &mut impl FnMut(&'e FieldRef)) { - match &*self.0 { - Node::Field(field) => f(field), + /// The expressions this one is made of, in order: an operand or input, a binary operation's + /// two sides, or a conditional's condition, value, and alternative. + pub(crate) fn children(&self) -> impl Iterator { + let (a, b, c) = match &*self.0 { Node::Unary(_, a) | Node::Is(a, _) | Node::Contains(a, _) | Node::Scale { input: a, .. } - | Node::Colormap { input: a, .. } => a.for_each_field(f), - Node::Binary(_, a, b) => { - a.for_each_field(f); - b.for_each_field(f); + | Node::Colormap { input: a, .. } => (Some(a), None, None), + Node::Binary(_, a, b) => (Some(a), Some(b), None), + Node::When(c, a, b) => (Some(c), Some(a), Some(b)), + Node::Number(_) | Node::Bool(_) | Node::Color(_) | Node::Missing | Node::Field(_) | Node::State(_) => { + (None, None, None) } - Node::When(c, a, b) => { - c.for_each_field(f); - a.for_each_field(f); - b.for_each_field(f); - } - Node::Number(_) | Node::Bool(_) | Node::Color(_) | Node::Missing | Node::State(_) => {} + }; + [a, b, c].into_iter().flatten() + } + + /// Whether `test` holds for this expression or any expression it is made of. + pub(crate) fn any(&self, test: &impl Fn(&Expr) -> bool) -> bool { + test(self) || self.children().any(|child| child.any(test)) + } + + /// Calls `f` with each field that a `contains` in the expression tests, and the text it looks + /// for. + pub(crate) fn for_each_contains<'e>(&'e self, f: &mut impl FnMut(&'e FieldRef, &'e str)) { + match &*self.0 { + Node::Contains(a, text) if let Node::Field(field) = &*a.0 => f(field, text), + _ => self.children().for_each(|child| child.for_each_contains(f)), } } - /// Whether a domain fits the values of every row: each row's value then reads them all. - pub(crate) fn fits(&self) -> bool { + /// Calls `f` with every field the expression reads. + pub(crate) fn for_each_field<'e>(&'e self, f: &mut impl FnMut(&'e FieldRef)) { match &*self.0 { - Node::Scale { domain: Domain::Fit, .. } | Node::Colormap { domain: Domain::Fit, .. } => true, - Node::Unary(_, a) - | Node::Is(a, _) - | Node::Contains(a, _) - | Node::Scale { input: a, .. } - | Node::Colormap { input: a, .. } => a.fits(), - Node::Binary(_, a, b) => a.fits() || b.fits(), - Node::When(c, a, b) => c.fits() || a.fits() || b.fits(), - Node::Number(_) | Node::Bool(_) | Node::Color(_) | Node::Missing | Node::Field(_) | Node::State(_) => false, + Node::Field(field) => f(field), + _ => self.children().for_each(|child| child.for_each_field(f)), } } - /// The field's name, when the expression is a field of its own table read as it is. + /// Whether the expression has a fitted domain, which makes each row's value depend on every + /// row. + pub(crate) fn fits(&self) -> bool { + self.any(&|e| { + matches!(&*e.0, Node::Scale { domain: Domain::Fit, .. } | Node::Colormap { domain: Domain::Fit, .. }) + }) + } + + /// The field's name, when the expression is a plain field of the mark's own table. pub(crate) fn field_name(&self) -> Option<&Name> { match &*self.0 { - Node::Field(f) if f.of.is_none() && f.component.is_none() => Some(&f.name), + Node::Field(f) if f.from.is_none() && f.component.is_none() => Some(&f.name), _ => None, } } - /// The first colormap a color maps through, and its domain when set: what a legend shows. A - /// conditional's general case, its `otherwise`, comes before the cases it makes exceptions for. + /// The first colormap the expression maps through, with its domain when fixed. A legend shows + /// it. A conditional's `otherwise` is searched before its cases, since it is the general case. pub(crate) fn colormap_of(&self) -> Option<(&Colormap, Option<[f64; 2]>)> { match &*self.0 { Node::Colormap { colormap, domain, .. } => { @@ -363,12 +398,12 @@ impl Expr { } } -/// A value mapped through a colormap or onto a range, whose domain can be set. +/// A value mapped through a colormap or onto a range. Its domain can be set. #[derive(Clone, PartialEq, Debug)] pub struct Mapped(Expr); impl Mapped { - /// The input values that map to the ends of the range or colormap. + /// Sets the input values that map to the ends of the range or colormap. #[must_use] pub fn domain(self, domain: RangeInclusive) -> Mapped { let fixed = Domain::Fixed([*domain.start(), *domain.end()]); @@ -390,13 +425,15 @@ impl From for Expr { } } -/// The open cases of a [`when`]. +/// A conditional started by [`when`], waiting for [`then`](When::then) to give the value for its +/// latest condition. pub struct When { cases: Vec<(Expr, Expr)>, condition: Expr, } -/// The cases of a [`when`], waiting for another case or the value otherwise. +/// A conditional started by [`when`], waiting for another case or for +/// [`otherwise`](Then::otherwise). pub struct Then { cases: Vec<(Expr, Expr)>, } @@ -421,28 +458,49 @@ impl Then { } } -/// A size: CSS pixels, or position units with [`units`], kept within a range of CSS pixels. +/// A size. A bare value is in the space's unit: CSS pixels, or units in a diagram. [`px`] makes it +/// CSS pixels, and [`units`] makes it position units, which scale with the zoom. +/// [`Size::at_least`] and [`Size::at_most`] bound it in CSS pixels. #[derive(Clone, PartialEq, Debug)] pub struct Size { pub(crate) expr: Expr, - pub(crate) units: bool, - /// The CSS pixels it keeps within, at any zoom. + /// Its explicit unit. `None` means the space's unit. + pub(crate) unit: Option, + /// The CSS pixel bounds it stays within at any zoom. pub(crate) px: [f32; 2], } +/// An explicit unit for a length. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub(crate) enum Unit { + /// CSS pixels, at any zoom. + Px, + /// Position units, which scale as the view zooms. + Units, +} + impl Size { - /// Any number of CSS pixels. + /// No bounds: from 0 to infinitely many CSS pixels. pub(crate) const ANY: [f32; 2] = [0.0, f32::INFINITY]; - /// Keeps at least `px` CSS pixels however far the view zooms out: `units(field("r")).at_least(2.0)` - /// grows with the map and never vanishes. A marker keeps its proportions, its larger side held. + /// Whether the size is in position units. `space_units` says whether the space's own unit is + /// position units, as a diagram's is. + pub(crate) fn in_units(&self, space_units: bool) -> bool { + self.unit.map_or(space_units, |unit| unit == Unit::Units) + } + + /// Sets a minimum size of `px` CSS pixels on screen, however far the view zooms out. For + /// example, `units(field("r")).at_least(2.0)` scales with the map but never shrinks below 2 + /// pixels. A marker is bounded on its larger side and keeps its proportions. Negative values + /// count as 0. #[must_use] pub fn at_least(mut self, px: f32) -> Size { self.px[0] = px.max(0.0); self } - /// Keeps at most `px` CSS pixels however far the view zooms in. + /// Sets a maximum on-screen size of `px` CSS pixels, however far the view zooms in. Negative + /// values count as 0. #[must_use] pub fn at_most(mut self, px: f32) -> Size { self.px[1] = px.max(0.0); @@ -452,25 +510,7 @@ impl Size { impl> From for Size { fn from(value: T) -> Size { - Size { expr: value.into(), units: false, px: Size::ANY } - } -} - -impl From<&str> for Expr { - fn from(name: &str) -> Expr { - field(name) - } -} - -impl From for Expr { - fn from(name: String) -> Expr { - field(name) - } -} - -impl From<&Name> for Expr { - fn from(name: &Name) -> Expr { - field(name) + Size { expr: value.into(), unit: None, px: Size::ANY } } } @@ -562,19 +602,19 @@ mod tests { } #[test] - fn domain_applies_to_the_mapping_whatever_the_order() { + fn domain_applies_to_the_mapping_regardless_of_order() { let a: Expr = field("vm").colormap(Colormap::BALANCE).domain(0.9..=1.1).into(); let Node::Colormap { domain, .. } = &*a.0 else { panic!("a colormap") }; assert_eq!(*domain, Domain::Fixed([0.9, 1.1])); } #[test] - fn of_reaches_every_field() { - let e = (field("a") + field("b")).of("weather", "Station"); + fn a_join_applies_to_every_field() { + let e = (field("a") + field("b")).from_table("weather", "Station"); let Node::Binary(_, a, b) = &*e.0 else { panic!("a sum") }; for side in [a, b] { let Node::Field(f) = &*side.0 else { panic!("a field") }; - assert_eq!(f.of, Some(("weather".into(), "Station".into()))); + assert_eq!(f.from, Some(("weather".into(), "Station".into()))); } } } diff --git a/crates/latkit/src/expr/ramp.rs b/crates/latkit/src/expr/ramp.rs index c416be0f..d7272c8a 100644 --- a/crates/latkit/src/expr/ramp.rs +++ b/crates/latkit/src/expr/ramp.rs @@ -1,6 +1,7 @@ -//! A colormap's 256 texels from its stops, made one way for the catalog the build bakes and for the -//! maps made at run time: blended in OKLab, so lightness runs evenly; wrapping when cyclic; a stop a -//! texel in turn when categorical. Free of the crate, so the build script includes it as it is. +//! Builds a colormap's 256 texels from its stops, both for the catalog baked at build time and for +//! maps made at run time. Colors blend in OKLab so lightness changes evenly. Cyclic ramps wrap, and +//! categorical ramps repeat their stops texel by texel. The file uses nothing from the crate, so +//! the build script includes it unchanged. /// How a ramp spans its stops. #[derive(Clone, Copy, PartialEq, Eq, Debug)] @@ -9,16 +10,17 @@ pub(crate) enum Spread { Continuous, /// Around a circle: past the last stop, back to the first. Cyclic, - /// A stop's color for each texel in turn. + /// Each texel takes the next stop's color, repeating. Categorical, } -/// The 256 texels through `stops`, each a place from 0 to 1, ascending, and a straight sRGB color -/// from 0 to 1. A stop without a place sits evenly between its neighbors' places, as in a CSS -/// gradient: the first at 0, the last at 1, or a turn after the first when cyclic. A categorical -/// ramp's texel `i` takes stop `i` modulo their count. +/// The 256 texels through `stops`: each an optional place from 0 to 1, ascending, and a +/// straight-alpha sRGB color with channels from 0 to 1. A stop without a place sits evenly between +/// its neighbors' places, like a CSS gradient. An unplaced first stop sits at 0. An unplaced last +/// stop sits at 1, except in a cyclic ramp. There, unplaced stops spread evenly up to one turn past +/// the first stop. A categorical ramp's texel `i` takes stop `i` modulo the stop count. pub(crate) fn texels(spread: Spread, stops: &[(Option, [f32; 4])]) -> [[u8; 4]; 256] { - assert!(!stops.is_empty(), "a ramp has a stop"); + assert!(!stops.is_empty(), "a ramp needs at least one stop"); if spread == Spread::Categorical || stops.len() == 1 { return std::array::from_fn(|t| bytes(stops[t % stops.len()].1)); } @@ -58,10 +60,10 @@ fn blend(stops: &[(f32, [f32; 4])], x: f32, cyclic: bool) -> [f32; 4] { let n = stops.len(); let (first, last) = (stops[0], stops[n - 1]); let (a, b) = match stops.partition_point(|s| s.0 <= x) { - // Before the first stop: from the last a turn back, when cyclic. + // Before the first stop: when cyclic, blend from the last stop, one turn back. 0 if cyclic => ((last.0 - 1.0, last.1), first), 0 => (first, first), - // Past the last stop: on to the first a turn on, when cyclic. + // Past the last stop: when cyclic, blend toward the first stop, one turn on. i if i == n && cyclic => (last, (first.0 + 1.0, first.1)), i if i == n => (last, last), i => (stops[i - 1], stops[i]), @@ -82,7 +84,7 @@ fn gamma(c: f32) -> f32 { if c <= 0.003_130_8 { c * 12.92 } else { 1.055 * c.max(0.0).powf(1.0 / 2.4) - 0.055 } } -/// A straight sRGB color in OKLab, its alpha kept. +/// Converts a straight-alpha sRGB color to OKLab, keeping alpha. fn oklab([r, g, b, a]: [f32; 4]) -> [f32; 4] { let [r, g, b] = [r, g, b].map(linear); let l = (0.412_221_47 * r + 0.536_332_55 * g + 0.051_445_995 * b).cbrt(); @@ -96,7 +98,7 @@ fn oklab([r, g, b, a]: [f32; 4]) -> [f32; 4] { ] } -/// An OKLab color in straight sRGB, its alpha kept. +/// Converts an OKLab color to straight-alpha sRGB, keeping alpha. fn srgb([l, a, b, alpha]: [f32; 4]) -> [f32; 4] { let l_ = (l + 0.396_337_78 * a + 0.215_803_76 * b).powi(3); let m_ = (l - 0.105_561_346 * a - 0.063_854_17 * b).powi(3); @@ -118,14 +120,14 @@ mod tests { const WHITE: [f32; 4] = [1.0, 1.0, 1.0, 1.0]; #[test] - fn stops_land_on_their_texels_and_blend_in_oklab() { + fn stops_sit_on_their_texels_and_blend_in_oklab() { let ramp = texels(Spread::Continuous, &[(None, BLACK), (None, WHITE)]); assert_eq!((ramp[0], ramp[255]), ([0, 0, 0, 255], [255, 255, 255, 255])); - // OKLab's half lightness is a darker byte than sRGB's middle: about 100, not 128. + // OKLab's half lightness is a darker byte than sRGB's midpoint: about 100, not 128. assert!((95..105).contains(&ramp[128][0]), "{:?}", ramp[128]); let red = [1.0, 0.0, 0.0, 1.0]; let placed = texels(Spread::Continuous, &[(None, BLACK), (Some(64.0 / 255.0), red), (None, WHITE)]); - assert_eq!(placed[64], [255, 0, 0, 255], "a placed stop lands on its texel"); + assert_eq!(placed[64], [255, 0, 0, 255], "a placed stop sets its texel's color"); assert_eq!(placed[0], [0, 0, 0, 255]); } diff --git a/crates/latkit/src/geo/earcut.rs b/crates/latkit/src/geo/earcut.rs index 719dcb56..59f8fdec 100644 --- a/crates/latkit/src/geo/earcut.rs +++ b/crates/latkit/src/geo/earcut.rs @@ -1,24 +1,26 @@ -//! Ear clipping: a polygon with holes cut into triangles, after Mapbox's earcut 3.2. Holes join the -//! outer ring through bridges; ears are clipped from a doubly linked ring, found by z-order hashing -//! on large polygons; where none clips, the ring is filtered while that removes points, cured once -//! of local self-intersections, then split in two. - -/// No node. +//! Polygon triangulation by ear clipping, ported from Mapbox's earcut 3.2. +//! +//! Each hole is joined to the outer ring by a bridge. Ears are clipped from a doubly linked ring. +//! Large polygons use a z-order index to find points that block an ear. When no ear can be +//! clipped, the ring is filtered while that removes points, then cured of local self-intersections +//! once, then split in two. + +/// The null node index. const NIL: u32 = u32::MAX; #[derive(Clone, Copy)] struct Node { - /// The vertex index. + /// Index into the input points. i: u32, x: f64, y: f64, prev: u32, next: u32, - /// Z-order of the vertex, and its neighbors in z-order. + /// Z-order code of the vertex. `prev_z` and `next_z` link the ring sorted by it. z: u32, prev_z: u32, next_z: u32, - /// Whether the vertex is a lone hole point, which may stay in the ring. + /// Whether the vertex is a single-point hole (a Steiner point), which filtering never removes. steiner: bool, } @@ -30,8 +32,8 @@ struct Rings<'a> { filtered: bool, } -/// The triangles of the polygon whose points are `points`: the outer ring first, then each hole from -/// where `holes` starts it. Each triangle is three indices into `points`. +/// Triangulates a polygon. `points` holds the outer ring, then each hole. `holes` lists the index +/// where each hole starts. Returns triangles as index triples into `points`. pub(crate) fn earcut(points: &[[f64; 2]], holes: &[usize]) -> Vec<[u32; 3]> { let nodes = Vec::with_capacity(points.len() * 3 / 2); let mut rings = Rings { nodes, points, triangles: Vec::new(), filtered: false }; @@ -43,7 +45,7 @@ pub(crate) fn earcut(points: &[[f64; 2]], holes: &[usize]) -> Vec<[u32; 3]> { if !holes.is_empty() { outer = rings.eliminate_holes(holes, outer); } - // Large polygons find ears through z-order hashing. + // Polygons over 80 points index their ring in z-order to speed up ear tests. let (mut min, mut size) = ([0.0; 2], 0.0); if points.len() > 80 { let mut max = [f64::NEG_INFINITY; 2]; @@ -75,7 +77,8 @@ impl Rings<'_> { (n.x, n.y) } - /// A circular list of points `start..end`, wound as `clockwise` asks. + /// Links points `start..end` into a circular list, reversed if needed so its winding matches + /// `clockwise`. Drops a closing point equal to the first. Returns the last node, or `NIL`. fn linked(&mut self, start: usize, end: usize, clockwise: bool) -> u32 { let mut last = NIL; if clockwise == (signed_area(&self.points[start..end]) > 0.0) { @@ -128,15 +131,17 @@ impl Rings<'_> { self.xy(a) == self.xy(b) } - /// Twice the signed area of the triangle `p q r`: negative turning one way. + /// Twice the signed area of triangle `p q r`. Negative where the ring turns convexly at `q`, + /// positive at a reflex turn. fn area(&self, p: u32, q: u32, r: u32) -> f64 { let ((px, py), (qx, qy), (rx, ry)) = (self.xy(p), self.xy(q), self.xy(r)); (qy - py) * (rx - qx) - (qx - px) * (ry - qy) } - /// Removes repeated points and straight runs: around the whole ring from `start` until none is - /// left when `end` is `NIL`; else from `start` to `end`, healing the cut between them. Answers a - /// point still in the ring. + /// Removes repeated and collinear points, except Steiner points. When `end` is `NIL` or + /// `start`, it circles the whole ring until a full pass removes nothing. Otherwise it checks + /// only the nodes from `start` through `end`, as around a new diagonal. Returns a node still in + /// the ring. fn filter(&mut self, start: u32, end: u32) -> u32 { if start == NIL { return start; @@ -165,8 +170,9 @@ impl Rings<'_> { } } - /// Clips ears from the ring at `ear`. Where none clips, filters while that removes points, cures - /// local self-intersections once, then splits the ring in two. + /// Clips ears from the ring starting at `ear`, adding their triangles. When a full pass finds + /// no ear, it filters the ring and retries while that removes points, then cures local + /// self-intersections once, then splits the ring in two. fn clip(&mut self, mut ear: u32, min: [f64; 2], size: f64) { if ear == NIL { return; @@ -274,7 +280,7 @@ impl Rings<'_> { true } - /// Clips the triangles of local self-intersections, returning the ring left. + /// Clips a triangle at each local self-intersection and returns a node of the remaining ring. fn cure(&mut self, start: u32) -> u32 { let (mut p, mut start, mut cured) = (start, start, false); loop { @@ -337,12 +343,13 @@ impl Rings<'_> { } queue.push(self.leftmost(list)); } - // Holes whose leftmost points meet go around that point, so each bridges to where they meet. + // Holes whose leftmost points coincide are ordered by the slope of their first edge, so + // each one bridges to the shared point. let slope = |p: u32| { let (n, next) = (self.n(p), self.n(self.n(p).next)); (next.y - n.y) / (next.x - n.x) }; - // By x, then y, then slope: the last key first, as each sort keeps the order of ties. + // Order by x, then y, then slope. The sorts are stable, so the last key sorts first. crate::order::by_key(&mut queue, |&p| crate::order::number(slope(p))); crate::order::by_wide_key(&mut queue, |&p| { let n = self.n(p); @@ -351,7 +358,7 @@ impl Rings<'_> { for hole in queue { outer = self.eliminate_hole(hole, outer); } - // Points made collinear or repeated across the merged ring go before clipping. + // Remove points the bridges made repeated or collinear before clipping. self.filter(outer, NIL) } @@ -362,13 +369,14 @@ impl Rings<'_> { self.filter(bridge, self.n(bridge).next) } - /// The outer ring's point a hole joins it at: seen from the hole's leftmost point, nothing between. + /// Finds the outer ring node to bridge a hole to: one visible from `hole`, the hole's leftmost + /// point. `None` if no ring edge lies to its left. fn hole_bridge(&self, hole: u32, outer: u32) -> Option { let (hx, hy) = self.xy(hole); let mut qx = f64::NEG_INFINITY; let mut m = NIL; let mut p = outer; - // A hole touching the outer ring at a point joins it there. + // A hole that touches the outer ring at a point bridges there. if self.equals(hole, p) { return Some(p); } @@ -407,8 +415,8 @@ impl Rings<'_> { if hx >= px && px >= mx && hx != px && inside_triangle([ax, hy], [mx, my], [cx, hy], [px, py]) { let tan = (hy - py).abs() / (hx - px); let (bx, _) = self.xy(m); - // The hole's point on `p`'s level edge, a T touch: a bridge along that edge is valid, - // though not locally inside. + // When the hole point lies on `p`'s horizontal edge (a T junction), a bridge along + // that edge is valid although `locally_inside` rejects it. let (nx, ny) = self.xy(self.n(p).next); let along = py == hy && ny == hy && nx > hx; if (self.locally_inside(p, hole) || along) @@ -445,7 +453,7 @@ impl Rings<'_> { left } - /// Links the ring in z-order, sorted. + /// Computes each node's z-order code and links the ring in z-order. fn index(&mut self, start: u32, min: [f64; 2], size: f64) { let mut p = start; loop { @@ -464,7 +472,7 @@ impl Rings<'_> { self.sort(p); } - /// Merge-sorts the z-order list from `list`. + /// Merge-sorts the z-order list starting at `list`. fn sort(&mut self, mut list: u32) { let mut in_size = 1; loop { @@ -515,9 +523,10 @@ impl Rings<'_> { } } - /// Whether diagonal `a b` lies inside the polygon: locally visible from both ends, crossing no - /// edge, its middle inside, and making no opposite-facing sectors; or of no length between two - /// convex corners. + /// Whether diagonal `a b` lies inside the polygon. `b` must not be the node after `a`, and the + /// diagonal must be locally inside at both ends, cross no edge, have its midpoint inside, and + /// create no opposite-facing sectors. A zero-length diagonal between two reflex corners is also + /// valid. fn valid_diagonal(&self, a: u32, b: u32) -> bool { let (na, nb) = (self.n(a), self.n(b)); let zero_length = @@ -531,14 +540,14 @@ impl Rings<'_> { && (zero_length || self.middle_inside(a, b)) } - /// Whether segments `p1 q1` and `p2 q2` cross, not counting a touch. + /// Whether segments `p1 q1` and `p2 q2` cross. Touching does not count. fn crosses(&self, p1: u32, q1: u32, p2: u32, q2: u32) -> bool { let (o1, o2) = (sign(self.area(p1, q1, p2)), sign(self.area(p1, q1, q2))); let (o3, o4) = (sign(self.area(p2, q2, p1)), sign(self.area(p2, q2, q1))); o1 * o2 < 0 && o3 * o4 < 0 } - /// Whether segments `p1 q1` and `p2 q2` meet, touches included. + /// Whether segments `p1 q1` and `p2 q2` intersect, including touching. fn intersects(&self, p1: u32, q1: u32, p2: u32, q2: u32) -> bool { let o1 = sign(self.area(p1, q1, p2)); let o2 = sign(self.area(p1, q1, q2)); @@ -551,13 +560,13 @@ impl Rings<'_> { || (o4 == 0 && self.on_segment(p2, q1, q2)) } - /// Whether `q` lies within the box of segment `p r`, on which it lies. + /// For `q` collinear with segment `p r`, whether it lies on the segment, within its box. fn on_segment(&self, p: u32, q: u32, r: u32) -> bool { let ((px, py), (qx, qy), (rx, ry)) = (self.xy(p), self.xy(q), self.xy(r)); qx <= px.max(rx) && qx >= px.min(rx) && qy <= py.max(ry) && qy >= py.min(ry) } - /// Whether diagonal `a b` meets an edge of the ring, other than those at its ends. + /// Whether diagonal `a b` intersects a ring edge not incident to `a` or `b`. fn intersects_polygon(&self, a: u32, b: u32) -> bool { let (ia, ib) = (self.n(a).i, self.n(b).i); let ((ax, ay), (bx, by)) = (self.xy(a), self.xy(b)); @@ -566,7 +575,7 @@ impl Rings<'_> { loop { let next = self.n(p).next; let ((px, py), (nx, ny)) = (self.xy(p), self.xy(next)); - // An edge beside the diagonal's box cannot meet it. + // Skip edges entirely outside the diagonal's bounding box. let apart = (px > x1 && nx > x1) || (px < x0 && nx < x0) || (py > y1 && ny > y1) || (py < y0 && ny < y0); let (ip, inext) = (self.n(p).i, self.n(next).i); if !apart && ip != ia && inext != ia && ip != ib && inext != ib && self.intersects(p, next, a, b) { @@ -607,7 +616,8 @@ impl Rings<'_> { } } - /// Splits the ring along the diagonal `a b` into two; returns the new copy of `b`. + /// Splits the ring in two along diagonal `a b` by duplicating both nodes. Returns the copy of + /// `b`, which lies in the second ring. fn split_polygon(&mut self, a: u32, b: u32) -> u32 { let (na, nb) = (*self.n(a), *self.n(b)); let a2 = self.nodes.len() as u32; @@ -628,7 +638,7 @@ impl Rings<'_> { } } -/// Twice the signed area of a ring, by the shoelace sum as earcut takes it. +/// Twice the signed area of a ring by the shoelace formula, with earcut's sign convention. fn signed_area(points: &[[f64; 2]]) -> f64 { let Some(&last) = points.last() else { return 0.0 }; let mut sum = 0.0; @@ -640,7 +650,7 @@ fn signed_area(points: &[[f64; 2]]) -> f64 { sum } -/// The z-order of point `(x, y)` in a box from `min` scaled by `size` to 15 bits a side. +/// The z-order (Morton) code of `(x, y)`, offset by `min` and scaled by `size` to 15 bits per axis. fn z_order(x: f64, y: f64, min: [f64; 2], size: f64) -> u32 { let spread = |v: f64| { let mut v = (v * size) as u32 & 0x7fff; @@ -658,8 +668,8 @@ fn inside_triangle(a: [f64; 2], b: [f64; 2], c: [f64; 2], p: [f64; 2]) -> bool { && (b[0] - p[0]) * (c[1] - p[1]) >= (c[0] - p[0]) * (b[1] - p[1]) } -/// Whether `p` lies in the triangle, other than at its first corner: a point where a bridge doubled -/// that corner blocks no ear. +/// Whether `p` lies in triangle `a b c` and is not at `a`. Bridges duplicate points, and a copy of +/// `a` must not block the ear. fn inside_except_first(a: [f64; 2], b: [f64; 2], c: [f64; 2], p: [f64; 2]) -> bool { a != p && inside_triangle(a, b, c, p) } @@ -678,7 +688,7 @@ fn sign(v: f64) -> i8 { mod tests { use super::*; - /// Twice the area the triangles cover. + /// Twice the total area of `triangles`. fn covered(points: &[[f64; 2]], triangles: &[[u32; 3]]) -> f64 { triangles .iter() @@ -708,15 +718,15 @@ mod tests { #[test] fn holes_touching_the_ring_or_each_other_are_left_out() { - /// Holes, where each starts, and the area left. + /// Hole points, the index where each hole starts, and the expected area. type Case<'a> = (&'a [[f64; 2]], &'a [usize], f64); let outer = [[0.0, 0.0], [10.0, 0.0], [10.0, 10.0], [0.0, 10.0]]; let cases: [Case<'_>; 3] = [ // A hole touching the outer ring at its corner. (&[[0.0, 0.0], [3.0, 1.0], [1.0, 3.0]], &[4], 96.0), - // One touching its edge. + // A hole touching the outer ring's edge. (&[[0.0, 5.0], [3.0, 4.0], [3.0, 6.0]], &[4], 97.0), - // Two meeting at their leftmost point. + // Two holes sharing their leftmost point. (&[[5.0, 5.0], [7.0, 4.0], [7.0, 6.0], [5.0, 5.0], [7.0, 7.0], [6.0, 9.0]], &[4, 7], 95.0), ]; for (holes, starts, area) in cases { @@ -728,7 +738,7 @@ mod tests { #[test] fn large_rings_hash_their_ears() { - // A star of 200 points: concave everywhere, so ear finding matters. + // A 200-point star: every other corner is reflex, which exercises the hashed ear test. let points: Vec<[f64; 2]> = (0..200) .map(|k| { let a = f64::from(k) / 200.0 * std::f64::consts::TAU; @@ -743,7 +753,7 @@ mod tests { } #[test] - fn degenerate_rings_answer_nothing() { + fn degenerate_rings_give_no_triangles() { let none: Vec<[u32; 3]> = Vec::new(); assert_eq!(earcut(&[[0.0, 0.0], [1.0, 1.0]], &[]), none); assert_eq!(earcut(&[[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]], &[]), none, "collinear"); diff --git a/crates/latkit/src/geo/geojson.rs b/crates/latkit/src/geo/geojson.rs index 11fa1b04..dfe46545 100644 --- a/crates/latkit/src/geo/geojson.rs +++ b/crates/latkit/src/geo/geojson.rs @@ -1,6 +1,5 @@ -//! GeoJSON as data: a feature collection's features become rows of three tables by what their -//! geometry is, `points`, `lines`, and `polygons`, each with its geometry as field `geometry` and its -//! properties as fields. +//! Reads GeoJSON into three tables by geometry type: `points`, `lines`, and `polygons`. Each row +//! holds its geometry in field `geometry` and each property in a field of its own. use std::fmt; @@ -8,22 +7,25 @@ use serde::Deserialize; use serde::de::{self, IgnoredAny, SeqAccess, Visitor}; use serde_json::{Map, Value as Json}; -use crate::data::{Data, Ids, Name, Patch, Table, Values}; +use crate::data::{Ids, Name, Patch, Snapshot, Table, Value, Values}; use crate::{Error, Result}; type Point = [f64; 2]; -/// The features of GeoJSON `bytes`, gzip-compressed or not, as dataset `name`: a feature collection, -/// one feature, or a bare geometry. +/// Reads GeoJSON `bytes`, optionally gzip-compressed, as dataset `name`. The document may be a +/// feature collection, a single feature, or a bare geometry. /// -/// Points become rows of table `points`, a point field `geometry`; line strings rows of `lines`, a -/// field of parts; polygons rows of `polygons`, a field of polygons. A multi-point is a row per point, -/// and a geometry collection a row per member. Properties become fields: numbers, booleans, and texts, -/// any other value written as its JSON. Rows take their features' ids where every row of a table has -/// one and no two share it, and are numbered in order otherwise. -pub(crate) fn geojson(name: impl Into, bytes: &[u8]) -> Result { +/// Points become rows of table `points` with a point field `geometry`. Line strings become rows of +/// `lines` (a parts field), and polygons rows of `polygons` (a polygons field). A multi-point adds +/// a row per point, and a geometry collection a row per member. Each property becomes a field of +/// numbers, booleans, or text, with any other value written as JSON. A null counts as missing. +/// +/// A table keeps its features' ids when every row has one, all are non-negative integers or all are +/// strings, and none repeat. Otherwise its rows are numbered in order. Errors if `bytes` are not +/// GeoJSON. +pub(crate) fn geojson(name: impl Into, bytes: &[u8]) -> Result { let bytes = super::gunzipped(bytes)?; - let top: Top = serde_json::from_slice(&bytes).map_err(|e| Error::invalid(format!("not GeoJSON: {e}")))?; + let top: Top = crate::json::from_bytes(&bytes).map_err(|e| Error::invalid(format!("not GeoJSON: {e}")))?; let mut tables = [Rows::default(), Rows::default(), Rows::default()]; let mut add = |feature: Feature| { if let Some(geometry) = feature.geometry { @@ -46,12 +48,11 @@ pub(crate) fn geojson(name: impl Into, bytes: &[u8]) -> Result { patch = patch.replace(table); } } - let mut data = Data::new(name); - data.apply(patch)?; - Ok(data) + Snapshot::new(name).applied(patch) } -/// The top of a document: a collection, a feature, or a geometry, as one struct of each's members. +/// A document's top-level object, with the members of a feature collection, a feature, and a +/// geometry in one struct. #[derive(Deserialize)] struct Top { #[serde(rename = "type")] @@ -79,13 +80,13 @@ struct Geometry { geometries: Option>, } -/// Which table a geometry's rows go to. +/// Indices into the table array, by geometry type. const POINTS: usize = 0; const LINES: usize = 1; const POLYGONS: usize = 2; impl Geometry { - /// Adds this geometry's rows, with the feature's id and properties, to the table of each kind. + /// Adds this geometry's rows, with the feature's id and properties, to the table for its type. fn into_rows(self, id: Option<&Json>, properties: &Map, tables: &mut [Rows; 3]) { let coordinates = self.coordinates.unwrap_or(Coordinates::Empty); match (self.kind.as_str(), coordinates) { @@ -108,7 +109,7 @@ impl Geometry { member.into_rows(id, properties, tables); } } - // A geometry without coordinates, or of a type GeoJSON does not have, adds no row. + // Unknown types, missing coordinates, and coordinates of the wrong depth add no row. _ => {} } } @@ -120,7 +121,7 @@ enum Shape { Polygons(Vec>>), } -/// One table's rows as they are read: each row's id, its properties, and its shape. +/// Rows read for one table: each row's id, properties, and shape. #[derive(Default)] struct Rows { ids: Vec>, @@ -135,7 +136,7 @@ impl Rows { self.shapes.push(shape); } - /// The rows as table `name`, or `None` without any. + /// Builds table `name` from the rows, or `None` if there are none. fn table(self, name: &str) -> Option
{ if self.shapes.is_empty() { return None; @@ -172,7 +173,7 @@ impl Rows { } } for key in keys { - // A property named like the geometry would replace it. + // A property named `geometry` would replace the geometry field, so skip it. if key == "geometry" { continue; } @@ -184,7 +185,8 @@ impl Rows { } } -/// The rows' ids: every one present, of one kind, and distinct; else `None`. +/// The rows' ids, if every row has one, all are non-negative integers or all are strings, and none +/// repeat. Else `None`. fn ids(ids: &[Option]) -> Option { let mut seen = std::collections::HashSet::with_capacity(ids.len()); if ids.iter().all(|id| id.as_ref().and_then(Json::as_u64).is_some_and(|n| seen.insert(n.to_string()))) { @@ -200,30 +202,25 @@ fn ids(ids: &[Option]) -> Option { None } -/// A property's values as a field: numbers where every value present is one, booleans likewise, and -/// texts otherwise, any value not a text written as its JSON. +/// Converts a property's values into a field, typed as [`Values::infer`] types them, with arrays +/// and objects written as JSON text. fn property(values: &[Option<&Json>]) -> Values { - let present = || values.iter().flatten(); - if present().all(|v| v.is_number()) { - return values.iter().map(|v| v.and_then(Json::as_f64)).collect::>().into(); - } - if present().all(|v| v.is_boolean()) { - return values.iter().map(|v| v.and_then(Json::as_bool)).collect::>().into(); - } - let texts: Vec> = values + let values: Vec> = values .iter() .map(|v| { v.map(|v| match v { - Json::String(text) => text.clone(), - other => other.to_string(), + Json::Number(n) => Value::Number(n.as_f64().unwrap_or(f64::NAN)), + Json::Bool(b) => Value::Bool(*b), + Json::String(text) => Value::Text(text.as_str().into()), + other => Value::Text(other.to_string().into()), }) }) .collect(); - texts.into() + Values::infer(&values) } -/// Coordinates of any depth: a position, a list of them, lists of those, or polygons. Positions read -/// their longitude and latitude and skip any altitude. +/// Coordinates of any nesting depth: a position, a list of positions, a list of those, or a list of +/// polygons. Positions keep longitude and latitude and drop any altitude. enum Coordinates { Number(f64), Position(Point), @@ -239,7 +236,7 @@ impl<'de> Deserialize<'de> for Coordinates { } } -/// Reads coordinates of any depth: the first element of each array says how deep the rest are. +/// Reads coordinates of any depth. The first element of each array sets the depth of the rest. struct Depth; impl<'de> Visitor<'de> for Depth { @@ -298,7 +295,7 @@ impl<'de> Visitor<'de> for Depth { } } -/// One position: longitude and latitude, any altitude skipped. +/// One position: longitude and latitude, ignoring any altitude. struct Position(Point); impl<'de> Deserialize<'de> for Position { @@ -361,7 +358,7 @@ mod tests { } #[test] - fn gzip_compressed_bytes_read_alike() { + fn gzip_compressed_geojson_reads_the_same() { use std::io::Write; let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast()); encoder.write_all(COLLECTION.as_bytes()).expect("written"); @@ -370,10 +367,10 @@ mod tests { } #[test] - fn bare_geometries_and_bad_documents() { + fn a_bare_geometry_reads_and_a_truncated_document_errors() { let data = geojson("g", br#"{"type": "Polygon", "coordinates": [[[0, 0], [1, 0], [0, 1], [0, 0]]]}"#) .expect("a geometry"); assert_eq!(data.table("polygons").map(Table::len), Some(1)); - assert!(geojson("g", b"{\"type\": \"FeatureCollection\"").is_err(), "cut short"); + assert!(geojson("g", b"{\"type\": \"FeatureCollection\"").is_err(), "a truncated document errors"); } } diff --git a/crates/latkit/src/geo/mercator.rs b/crates/latkit/src/geo/mercator.rs new file mode 100644 index 00000000..7b33c59f --- /dev/null +++ b/crates/latkit/src/geo/mercator.rs @@ -0,0 +1,42 @@ +//! Web Mercator: the projection of the earth's map camera, and of the square its tiles divide. + +use std::f64::consts::{FRAC_PI_4, PI, TAU}; + +/// The latitude Web Mercator reaches, north and south, in degrees: where its square ends. +pub(crate) const MAX_LATITUDE: f64 = 85.051_128_779_806_59; + +/// The projected y of `latitude` in degrees, clamped to [`MAX_LATITUDE`]: radians north of the +/// equator. +pub(crate) fn y(latitude: f64) -> f64 { + let lat = latitude.clamp(-MAX_LATITUDE, MAX_LATITUDE).to_radians(); + (FRAC_PI_4 + lat / 2.0).tan().ln() +} + +/// The latitude in degrees at projected y `y`, the inverse of [`y`]. +pub(crate) fn latitude(y: f64) -> f64 { + y.sinh().atan().to_degrees() +} + +/// How far down the square `latitude` lies, from 0 at its north edge to 1 at its south. +pub(crate) fn down(latitude: f64) -> f64 { + 0.5 - y(latitude) / TAU +} + +/// The latitude `t` of the way down the square, the inverse of [`down`]. +pub(crate) fn latitude_down(t: f64) -> f64 { + latitude(PI * (1.0 - 2.0 * t)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn latitudes_survive_the_round_trip_and_the_square_ends_at_the_limit() { + for lat in [-80.0, -37.5, 0.0, 12.25, 85.0] { + assert!((latitude(y(lat)) - lat).abs() < 1e-9, "{lat}"); + assert!((latitude_down(down(lat)) - lat).abs() < 1e-9, "{lat}"); + } + assert!(down(MAX_LATITUDE).abs() < 1e-12 && (down(-MAX_LATITUDE) - 1.0).abs() < 1e-12); + } +} diff --git a/crates/latkit/src/geo/mod.rs b/crates/latkit/src/geo/mod.rs index b335c69e..94166e92 100644 --- a/crates/latkit/src/geo/mod.rs +++ b/crates/latkit/src/geo/mod.rs @@ -1,10 +1,12 @@ -//! Geography: GeoJSON and vector tiles as data, and what shapes become to draw: an -//! area's triangles, and lines bent along the globe. +//! Geography: GeoJSON and vector tiles read as data, polygon triangulation, lines given extra +//! points so they follow the globe, and Web Mercator. #[cfg(feature = "gpu")] mod earcut; mod geojson; #[cfg(feature = "gpu")] +pub(crate) mod mercator; +#[cfg(feature = "gpu")] pub(crate) mod mvt; #[cfg(feature = "gpu")] pub(crate) mod pmtiles; @@ -20,7 +22,8 @@ pub(crate) use geojson::geojson; use crate::{Error, Result}; -/// `bytes` inflated where they are gzip-compressed, else as they are. +/// Decompresses `bytes` if they start with the gzip magic number, else returns them unchanged. +/// Errors if the gzip stream is corrupt. pub(crate) fn gunzipped(bytes: &[u8]) -> Result> { if !bytes.starts_with(&[0x1f, 0x8b]) { return Ok(Cow::Borrowed(bytes)); diff --git a/crates/latkit/src/geo/mvt.rs b/crates/latkit/src/geo/mvt.rs index 81f0d699..250f02e4 100644 --- a/crates/latkit/src/geo/mvt.rs +++ b/crates/latkit/src/geo/mvt.rs @@ -1,11 +1,13 @@ -//! Mapbox vector tiles: a tile's layers, their features' geometry in longitude and latitude, clipped -//! to the tile so neighbors never overlap, and their properties. +//! Mapbox vector tile decoding. Feature geometry is converted to longitude and latitude and clipped +//! to the tile, so neighboring tiles never overlap. +use crate::Value; use crate::tiles::TileKey; +use crate::wire::In; type Point = [f64; 2]; -/// What a feature's geometry is. +/// A feature's geometry type. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] pub(crate) enum Kind { Points, @@ -13,15 +15,9 @@ pub(crate) enum Kind { Polygons, } -/// A property value as a tile holds it. -#[derive(Clone, PartialEq, Debug)] -pub(crate) enum Property { - Text(String), - Number(f64), - Bool(bool), -} +crate::wire::codes!(Kind { Points = 0, Lines = 1, Polygons = 2 }); -/// A feature's geometry, in degrees. +/// A feature's geometry, in degrees of longitude and latitude. #[derive(Clone, PartialEq, Debug)] pub(crate) enum Shape { Points(Vec), @@ -39,14 +35,15 @@ impl Shape { } } -/// One feature: its place among its layer's features, its shape, and its properties by key. +/// One feature: its shape, and its properties as pairs of key index and value. A value is a text, +/// number, or boolean. #[derive(Clone, Debug)] pub(crate) struct Feature { pub(crate) shape: Shape, - pub(crate) properties: Vec<(u32, Property)>, + pub(crate) properties: Vec<(u32, Value)>, } -/// One layer: its name, its keys, and its features. +/// One layer: its name, its property keys, and its features. #[derive(Clone, Debug, Default)] pub(crate) struct Layer { pub(crate) name: String, @@ -54,8 +51,8 @@ pub(crate) struct Layer { pub(crate) features: Vec, } -/// The layers of tile `key`'s `bytes`; a feature's geometry clipped to the tile, and one left with -/// nothing inside it dropped. Bytes that are no tile answer `None`. +/// Decodes the layers of tile `key` from `bytes`. Geometry is clipped to the tile, and features +/// with nothing inside it are dropped. `None` if the tile or one of its layers is malformed. pub(crate) fn decode(bytes: &[u8], key: TileKey) -> Option> { let mut layers = Vec::new(); let mut tile = Reader::new(bytes); @@ -92,19 +89,19 @@ fn layer(bytes: &[u8], key: TileKey) -> Option { Some(out) } -fn value(bytes: &[u8]) -> Option { +fn value(bytes: &[u8]) -> Option { let mut reader = Reader::new(bytes); - let mut out = Property::Bool(false); + let mut out = Value::Bool(false); while !reader.done() { let (field, wire) = reader.tag()?; out = match (field, wire) { - (1, LENGTH) => Property::Text(String::from_utf8_lossy(reader.bytes()?).into_owned()), - (2, FIXED32) => Property::Number(f64::from(f32::from_le_bytes(reader.take()?))), - (3, FIXED64) => Property::Number(f64::from_le_bytes(reader.take()?)), - (4, VARINT) => Property::Number(reader.varint()? as i64 as f64), - (5, VARINT) => Property::Number(reader.varint()? as f64), - (6, VARINT) => Property::Number(zigzag(reader.varint()?) as f64), - (7, VARINT) => Property::Bool(reader.varint()? != 0), + (1, LENGTH) => Value::Text(String::from_utf8_lossy(reader.bytes()?).into()), + (2, FIXED32) => Value::Number(f64::from(f32::from_le_bytes(reader.take()?))), + (3, FIXED64) => Value::Number(f64::from_le_bytes(reader.take()?)), + (4, VARINT) => Value::Number(reader.varint()? as i64 as f64), + (5, VARINT) => Value::Number(reader.varint()? as f64), + (6, VARINT) => Value::Number(zigzag(reader.varint()?) as f64), + (7, VARINT) => Value::Bool(reader.varint()? != 0), _ => { reader.skip(wire)?; continue; @@ -114,9 +111,9 @@ fn value(bytes: &[u8]) -> Option { Some(out) } -/// A feature: `None` when nothing of it lies in the tile, or its bytes are no feature, which drops -/// it alone. -fn feature(bytes: &[u8], values: &[Property], frame: &Frame) -> Option { +/// Decodes one feature. `None`, which drops only this feature, if it lies wholly outside the tile +/// or is malformed. +fn feature(bytes: &[u8], values: &[Value], frame: &Frame) -> Option { let mut reader = Reader::new(bytes); let (mut tags, mut geometry, mut kind) = (Vec::new(), Vec::new(), 0); while !reader.done() { @@ -146,7 +143,7 @@ fn feature(bytes: &[u8], values: &[Property], frame: &Frame) -> Option Some(Feature { shape, properties }) } -/// A tile's frame: where its coordinates lie on the earth. +/// Converts a tile's coordinates to longitude and latitude. struct Frame { key: TileKey, extent: f64, @@ -157,16 +154,15 @@ impl Frame { Frame { key, extent } } - /// The longitude and latitude of tile coordinates `p`. + /// The longitude and latitude of tile coordinates `p` (web mercator, y down). fn degrees(&self, p: [f64; 2]) -> Point { let n = f64::from(1_u32 << self.key.z.min(31)); let x = (f64::from(self.key.x) + p[0] / self.extent) / n; let y = (f64::from(self.key.y) + p[1] / self.extent) / n; - let lat = (std::f64::consts::PI * (1.0 - 2.0 * y)).sinh().atan().to_degrees(); - [x * 360.0 - 180.0, lat] + [x * 360.0 - 180.0, super::mercator::latitude_down(y)] } - /// The tile's points, those outside it dropped. + /// Decodes points, dropping those outside the tile. fn points(&self, geometry: &[u32]) -> Option { let inside = |p: &[f64; 2]| p.iter().all(|&v| v >= 0.0 && v < self.extent); let points: Vec = @@ -174,7 +170,7 @@ impl Frame { (!points.is_empty()).then_some(Shape::Points(points)) } - /// The tile's lines, cut where they leave it. + /// Decodes lines, cut where they leave the tile. fn lines(&self, geometry: &[u32]) -> Option { let mut parts: Vec> = Vec::new(); for path in paths(geometry) { @@ -187,8 +183,8 @@ impl Frame { (!parts.is_empty()).then_some(Shape::Lines(parts)) } - /// The tile's polygons: each exterior ring, by its positive area, with the holes after it, - /// clipped to the tile as a whole. + /// Decodes polygons. A ring with positive area starts a polygon, and the negative-area rings + /// after it are its holes. Each polygon is clipped to the tile as a whole. fn polygons(&self, geometry: &[u32]) -> Option { let mut groups: Vec>> = Vec::new(); for ring in paths(geometry) { @@ -215,15 +211,15 @@ impl Frame { } } -/// A polygon, its exterior then its holes, clipped to the square `0..extent` as a whole: a ring -/// wholly inside it stays; the runs of the rest inside it join along its edge, where the polygon's -/// inside lies, into exteriors; and each hole inside goes to the exterior around it. Clipping each -/// ring alone would leave a hole crossing the edge running along its exterior there, which no -/// triangulation reads right. +/// Clips a polygon (exterior, then holes) to the square `0..extent` as a whole. Rings wholly inside +/// the square stay. The parts of other rings inside it are joined along the square's edge into +/// exteriors, and each hole wholly inside goes to the exterior containing it. Clipping rings +/// separately would leave a hole that crosses the edge running along its clipped exterior there, +/// which triangulation mishandles. fn clip_polygon(rings: &[Vec<[f64; 2]>], extent: f64) -> Vec>> { let inside = |p: &[f64; 2]| p.iter().all(|&v| (0.0..=extent).contains(&v)); let (mut exteriors, mut holes, mut runs) = (Vec::new(), Vec::new(), Vec::new()); - // Whether the square lies within the polygon, as the rings that miss it say. + // Whether the whole square lies inside the polygon, judged from rings that do not cross it. let mut covered = false; for (k, ring) in rings.iter().enumerate().filter(|(_, r)| r.len() >= 3) { if ring.iter().all(inside) { @@ -235,7 +231,8 @@ fn clip_polygon(rings: &[Vec<[f64; 2]>], extent: f64) -> Vec>> continue; } let crossing = runs_inside(ring, extent); - // A ring missing the square holds all of it or none: the exterior first, then a hole within. + // A ring that does not cross the square contains all of it or none. The exterior comes + // first, so a hole that contains the square clears `covered`. if crossing.is_empty() && contains(ring, [extent / 2.0; 2]) { covered = k == 0; } @@ -261,9 +258,9 @@ fn clip_polygon(rings: &[Vec<[f64; 2]>], extent: f64) -> Vec>> polygons } -/// The runs of closed `ring`, which has a point outside the square `0..extent`, inside it: each from -/// where it enters to where it leaves, on the edge. A run lying along the edge alone bounds nothing, -/// and drops. +/// The runs of closed `ring` inside the square `0..extent`, each from where it enters to where it +/// leaves, with both ends on the edge. Returns nothing if `ring` has no point outside the square. +/// A run lying entirely on the edge bounds nothing and is dropped. fn runs_inside(ring: &[[f64; 2]], extent: f64) -> Vec> { let outside = |p: &[f64; 2]| p.iter().any(|&v| !(0.0..=extent).contains(&v)); let Some(start) = ring.iter().position(outside) else { return Vec::new() }; @@ -280,7 +277,7 @@ fn runs_inside(ring: &[[f64; 2]], extent: f64) -> Vec> { run.push(q); continue; } - // The segment leaves the square: the run ends. + // The segment leaves the square, so the run ends. run.push(onto_edge(q, extent)); if run.len() >= 2 && !run.iter().all(on_edge) { runs.push(std::mem::take(&mut run)); @@ -290,7 +287,7 @@ fn runs_inside(ring: &[[f64; 2]], extent: f64) -> Vec> { runs } -/// `p`, near the square's edge, moved onto it. +/// Moves `p`, which lies near the square's edge, onto the nearest edge. fn onto_edge(p: [f64; 2], extent: f64) -> [f64; 2] { let gaps = [p[1].abs(), (p[0] - extent).abs(), (p[1] - extent).abs(), p[0].abs()]; let edge = (0..4).min_by(|&a, &b| gaps[a].total_cmp(&gaps[b])).unwrap_or(0); @@ -303,8 +300,8 @@ fn onto_edge(p: [f64; 2], extent: f64) -> [f64; 2] { } } -/// How far along the square's edge `p` lies, from the corner at 0 the way an exterior winds: along -/// y = 0, then x = extent, y = extent, and x = 0. +/// Distance of edge point `p` along the square's perimeter from the corner at the origin, in the +/// direction an exterior winds: along y = 0, then x = extent, y = extent, and x = 0. fn along(p: [f64; 2], extent: f64) -> f64 { let gaps = [p[1].abs(), (p[0] - extent).abs(), (p[1] - extent).abs(), p[0].abs()]; match (0..4).min_by(|&a, &b| gaps[a].total_cmp(&gaps[b])).unwrap_or(0) { @@ -315,8 +312,9 @@ fn along(p: [f64; 2], extent: f64) -> f64 { } } -/// The exteriors `runs` make: from where each leaves the square, along its edge the way the -/// polygon's inside lies to where the next enters, past the corners between, until the first again. +/// Joins `runs` into exterior rings. From each run's exit, a ring follows the square's edge in the +/// exterior's winding direction to the next run's entry, adding the corners it passes, until it +/// returns to its first run. fn join(runs: &[Vec<[f64; 2]>], extent: f64) -> Vec> { let around = 4.0 * extent; let entries: Vec = runs.iter().map(|r| along(r[0], extent)).collect(); @@ -362,7 +360,7 @@ fn join(runs: &[Vec<[f64; 2]>], extent: f64) -> Vec> { rings } -/// Whether closed `ring` holds `p`, by the crossings of a ray from it. +/// Whether closed `ring` contains `p`, by counting crossings of a ray from it. fn contains(ring: &[[f64; 2]], p: [f64; 2]) -> bool { let mut inside = false; let mut a = ring[ring.len() - 1]; @@ -375,8 +373,8 @@ fn contains(ring: &[[f64; 2]], p: [f64; 2]) -> bool { inside } -/// The paths a geometry's commands draw, in tile coordinates: each move starts one, a close returns -/// to its start (left implicit). +/// Decodes geometry commands into paths in tile coordinates. Each move-to starts a path, and a +/// close-path is left implicit. fn paths(geometry: &[u32]) -> Vec> { let mut paths: Vec> = Vec::new(); let (mut x, mut y) = (0_i64, 0_i64); @@ -398,7 +396,7 @@ fn paths(geometry: &[u32]) -> Vec> { if let Some(path) = paths.last_mut() { path.push([x as f64, y as f64]); } - // A move with several points is several points, each its own path. + // A move-to with several points (a multipoint) makes each point its own path. if id == 1 && count > 1 && k + 1 < count { paths.push(Vec::new()); } @@ -412,8 +410,8 @@ fn paths(geometry: &[u32]) -> Vec> { paths } -/// Twice a ring's area by the surveyor's formula in tile coordinates, y down: positive for an -/// exterior ring. +/// Twice a ring's signed area by the surveyor's formula in tile coordinates (y down). Positive for +/// an exterior ring. fn surveyor(ring: &[[f64; 2]]) -> f64 { let Some(&last) = ring.last() else { return 0.0 }; let mut sum = 0.0; @@ -425,7 +423,7 @@ fn surveyor(ring: &[[f64; 2]]) -> f64 { sum } -/// `line` cut to the square `0..extent`: the runs of it inside. +/// Clips `line` to the square `0..extent`, returning the parts inside. fn clip_line(line: &[[f64; 2]], extent: f64) -> Vec> { let mut parts: Vec> = Vec::new(); let mut current: Vec<[f64; 2]> = Vec::new(); @@ -445,7 +443,7 @@ fn clip_line(line: &[[f64; 2]], extent: f64) -> Vec> { current.push(a); } current.push(b); - // The segment left the tile: what follows starts a new run. + // The segment left the tile, so the next segment starts a new part. if b != w[1] { parts.push(std::mem::take(&mut current)); } @@ -456,7 +454,7 @@ fn clip_line(line: &[[f64; 2]], extent: f64) -> Vec> { parts } -/// Segment `a b` clipped to the square `0..extent`, by Liang and Barsky. +/// Clips segment `a b` to the square `0..extent` (Liang-Barsky). `None` if it lies outside. fn clip_segment(a: [f64; 2], b: [f64; 2], extent: f64) -> Option<([f64; 2], [f64; 2])> { let (dx, dy) = (b[0] - a[0], b[1] - a[1]); let (mut t0, mut t1) = (0.0_f64, 1.0_f64); @@ -506,31 +504,20 @@ const FIXED64: u8 = 1; const LENGTH: u8 = 2; const FIXED32: u8 = 5; -/// Protocol buffer fields, read in order; `None` past the end or on a malformed field. -struct Reader<'a> { - bytes: &'a [u8], -} +/// Reads protocol buffer fields in order. Reads return `None` past the end or on a malformed field. +struct Reader<'a>(In<'a>); impl<'a> Reader<'a> { fn new(bytes: &'a [u8]) -> Reader<'a> { - Reader { bytes } + Reader(In::new(bytes)) } fn done(&self) -> bool { - self.bytes.is_empty() + self.0.is_empty() } fn varint(&mut self) -> Option { - let mut value = 0_u64; - for shift in (0..64).step_by(7) { - let (&byte, rest) = self.bytes.split_first()?; - self.bytes = rest; - value |= u64::from(byte & 0x7f) << shift; - if byte < 0x80 { - return Some(value); - } - } - None + self.0.varint() } /// The next field's number and wire type. @@ -539,17 +526,14 @@ impl<'a> Reader<'a> { Some(((tag >> 3) as u32, (tag & 7) as u8)) } + /// A length-delimited field's bytes. fn bytes(&mut self) -> Option<&'a [u8]> { let len = usize::try_from(self.varint()?).ok()?; - let (head, rest) = self.bytes.split_at_checked(len)?; - self.bytes = rest; - Some(head) + self.0.bytes(len) } fn take(&mut self) -> Option<[u8; N]> { - let (head, rest) = self.bytes.split_first_chunk::()?; - self.bytes = rest; - Some(*head) + self.0.take() } fn skip(&mut self, wire: u8) -> Option<()> { @@ -567,7 +551,7 @@ impl<'a> Reader<'a> { pub(crate) mod tests { use super::*; - /// Protocol buffer fields, written. + /// Writes protocol buffer fields. #[derive(Default)] pub(crate) struct Writer(pub(crate) Vec); @@ -603,8 +587,8 @@ pub(crate) mod tests { ((v << 1) ^ (v >> 63)) as u32 } - /// A tile of one layer `name` holding one feature of `kind` drawn by `geometry`, with property - /// `name` set to `value`. + /// A tile with one layer named `layer` holding one feature of type `kind`, with geometry + /// commands `geometry` and property `name` set to `value`. pub(crate) fn tile(layer: &str, kind: u64, geometry: &[u32], value: &str) -> Vec { let mut feature = Writer::default(); feature.packed(2, &[0, 0]); @@ -627,7 +611,7 @@ pub(crate) mod tests { t.0 } - /// A square's commands, from `(x, y)` of `side`, clockwise in tile coordinates. + /// Geometry commands for a square of `side` from `(x, y)`, clockwise in tile coordinates. pub(crate) fn square(x: i64, y: i64, side: i64) -> Vec { vec![9, zig(x), zig(y), 26, zig(side), 0, 0, zig(side), zig(-side), 0, 15] } @@ -644,7 +628,7 @@ pub(crate) mod tests { assert_eq!(ring.len(), 5, "closed"); assert!((ring[0][0] + 90.0).abs() < 1e-9, "a quarter in: {:?}", ring[0]); assert!(ring[0][1] > 60.0 && ring[0][1] < 70.0, "mercator latitude: {:?}", ring[0]); - assert_eq!(feature.properties, vec![(0, Property::Text("lake".into()))]); + assert_eq!(feature.properties, vec![(0, Value::Text("lake".into()))]); } #[test] @@ -653,8 +637,8 @@ pub(crate) mod tests { let ring = vec![[x, y], [x + side, y], [x + side, y + side], [x, y + side]]; if exterior { ring } else { ring.into_iter().rev().collect() } }; - // A sea over the tile and its buffer, with islands across its left edge, across its corner, - // inside it, and outside it. + // A sea covering the tile and its buffer, with islands across its left edge, across its + // corner, inside it, and outside it. let rings = [ square(-64.0, -64.0, 4224.0, true), square(-200.0, 1800.0, 500.0, false), @@ -664,7 +648,7 @@ pub(crate) mod tests { ]; let polygons = clip_polygon(&rings, 4096.0); assert_eq!(polygons.len(), 1); - assert_eq!(polygons[0].len(), 2, "islands across the edge notch the sea; the one inside stays a hole"); + assert_eq!(polygons[0].len(), 2, "islands across the edge join the outer ring; the one inside stays a hole"); let points: Vec<[f64; 2]> = polygons[0].concat(); let triangles = super::super::earcut::earcut(&points, &[polygons[0][0].len()]); let area: f64 = triangles @@ -675,8 +659,8 @@ pub(crate) mod tests { }) .sum(); let sea = 4096.0 * 4096.0 - 300.0 * 500.0 - 196.0 * 300.0 - 200.0 * 200.0; - assert!((area - sea).abs() < 1e-6, "{area} drawn of {sea}"); - // A sea the tile lies wholly in is the tile; one beside it is nothing. + assert!((area - sea).abs() < 1e-6, "area {area}, expected {sea}"); + // A sea containing the whole tile clips to the tile. One beside it clips to nothing. assert_eq!(clip_polygon(&[square(-100.0, -100.0, 5000.0, true)], 4096.0)[0][0].len(), 4); assert_eq!(clip_polygon(&[square(5000.0, 0.0, 100.0, true)], 4096.0).len(), 0); } @@ -684,7 +668,7 @@ pub(crate) mod tests { #[test] fn shapes_are_clipped_to_the_tile() { let key = TileKey { z: 1, x: 1, y: 0 }; - // A square reaching past the tile's left edge: clipped there. + // A square reaching past the tile's left edge is clipped there. let layers = decode(&tile("land", 3, &square(-512, 1024, 1024), "x"), key).expect("a tile"); let Shape::Polygons(polygons) = &layers[0].features[0].shape else { panic!("polygons") }; assert!(polygons[0][0].iter().all(|p| p[0] >= -1e-9), "west of 0 is clipped: {:?}", polygons[0][0]); @@ -695,19 +679,20 @@ pub(crate) mod tests { } #[test] - fn a_polygon_wholly_outside_leaves_nothing() { + fn a_polygon_wholly_outside_the_tile_is_dropped() { let key = TileKey { z: 1, x: 1, y: 0 }; - // Commands move relative to where the last ring ended, (-3000, 600): the second starts at (100, 100). + // Commands are relative to where the last ring ended, (-3000, 600), so the second ring + // starts at (100, 100). let mut geometry = square(-3000, 100, 500); geometry.extend(square(3100, -500, 500)); let layers = decode(&tile("land", 3, &geometry, "x"), key).expect("a tile"); let Shape::Polygons(polygons) = &layers[0].features[0].shape else { panic!("polygons") }; - assert_eq!(polygons.len(), 1, "only the one inside: {polygons:?}"); + assert_eq!(polygons.len(), 1, "only the polygon inside remains: {polygons:?}"); assert!(polygons.iter().flatten().all(|ring| ring.len() >= 4)); } #[test] - fn lines_are_cut_where_they_leave() { + fn lines_are_cut_where_they_leave_the_tile() { let parts = clip_line(&[[-10.0, 5.0], [10.0, 5.0], [10.0, -10.0], [20.0, 5.0]], 100.0); assert_eq!(parts.len(), 2, "{parts:?}"); assert_eq!(parts[0][0], [0.0, 5.0]); diff --git a/crates/latkit/src/geo/pmtiles.rs b/crates/latkit/src/geo/pmtiles.rs index 1eeebf0f..806ba894 100644 --- a/crates/latkit/src/geo/pmtiles.rs +++ b/crates/latkit/src/geo/pmtiles.rs @@ -1,50 +1,52 @@ -//! PMTiles archives, version 3: a header, directories of tile entries along the Hilbert curve, and -//! the tiles, each found by range requests. +//! PMTiles version 3 archives: a header, directories of tile entries in Hilbert curve order, and +//! the tile data, each read with range requests. use crate::tiles::TileKey; +use crate::wire::In; -/// Bytes a header takes. +/// Size of the header in bytes. pub(crate) const HEADER: usize = 127; const MAGIC: &[u8] = b"PMTiles"; -/// How deep directories nest at most. +/// Maximum directory nesting depth. pub(crate) const DEPTH: usize = 4; -/// What an archive's header says. +/// An archive's header. #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) struct Header { pub(crate) root: (u64, u64), pub(crate) metadata: (u64, u64), pub(crate) leaves: u64, pub(crate) data: u64, - /// Whether directories, and tiles, are gzip-compressed. + /// Whether directories are gzip-compressed. `gzip_tiles` is the same for tiles. pub(crate) gzip_directories: bool, pub(crate) gzip_tiles: bool, pub(crate) vector: bool, pub(crate) min_zoom: u8, pub(crate) max_zoom: u8, - /// West, south, east, north. + /// West, south, east, north, in degrees. pub(crate) bounds: [f64; 4], } -/// Whether `bytes` begin a PMTiles archive. +/// Whether `bytes` start with the PMTiles magic number. pub(crate) fn is_archive(bytes: &[u8]) -> bool { bytes.starts_with(MAGIC) } -/// The header at the start of `bytes`: a version 3 archive's, with compression read or none. +/// Parses the header at the start of `bytes`. Errors if it is not a version 3 archive with +/// uncompressed or gzip-compressed directories and tiles. pub(crate) fn header(bytes: &[u8]) -> Result { if !is_archive(bytes) || bytes.len() < HEADER { return Err("these bytes are not a PMTiles archive".into()); } if bytes[7] != 3 { - return Err(format!("PMTiles version {} is not read; version 3 is", bytes[7])); + return Err(format!("unsupported PMTiles version {}; only version 3 is supported", bytes[7])); } let u64_at = |at: usize| u64::from_le_bytes(bytes[at..at + 8].try_into().expect("eight bytes")); let degrees = |at: usize| f64::from(i32::from_le_bytes(bytes[at..at + 4].try_into().expect("four bytes"))) / 1e7; let compression = |code: u8, what: &str| match code { 0 | 1 => Ok(false), 2 => Ok(true), - other => Err(format!("the archive's {what} use compression {other}; only gzip is read")), + other => Err(format!("the archive's {what} use compression {other}; only gzip or none is supported")), }; Ok(Header { root: (u64_at(8), u64_at(16)), @@ -60,8 +62,8 @@ pub(crate) fn header(bytes: &[u8]) -> Result { }) } -/// A directory's entries, sorted by tile id: each one's id, run length (0 for a leaf directory), -/// offset, and length. +/// A directory's entries, sorted by tile id: id, run length (0 for a leaf directory), offset, and +/// length. #[derive(Clone, Default, Debug)] pub(crate) struct Directory { ids: Vec, @@ -70,31 +72,20 @@ pub(crate) struct Directory { lengths: Vec, } -/// Where a directory sends a tile. +/// The location a directory gives for a tile. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) enum Entry { - /// The tile's bytes: offset from the data section, and length. + /// The tile's data: offset from the start of the data section, and length. Tile(u64, u64), - /// A leaf directory: offset from the leaf section, and length. + /// A leaf directory to search next: offset from the start of the leaf section, and length. Leaf(u64, u64), } impl Directory { - /// The directory in `bytes`, decompressed. + /// Decodes a directory from already decompressed `bytes`. `None` if malformed. pub(crate) fn decode(bytes: &[u8]) -> Option { - let mut at = 0; - let mut next = || -> Option { - let mut value = 0_u64; - for shift in (0..64).step_by(7) { - let byte = *bytes.get(at)?; - at += 1; - value |= u64::from(byte & 0x7f) << shift; - if byte < 0x80 { - return Some(value); - } - } - None - }; + let mut input = In::new(bytes); + let mut next = || input.varint(); let count = usize::try_from(next()?).ok()?; if count > bytes.len() { return None; @@ -123,7 +114,8 @@ impl Directory { Some(Directory { ids, runs, offsets, lengths }) } - /// Where the entry holding tile id `id` sends it, if any does. + /// Looks up tile `id`: the tile entry whose run covers it, or the leaf directory that may hold + /// it. `None` if neither. pub(crate) fn find(&self, id: u64) -> Option { let i = match self.ids.binary_search(&id) { Ok(i) => i, @@ -139,7 +131,8 @@ impl Directory { } } -/// Tile `key`'s id along the Hilbert curve, counting every tile of the zooms before it. +/// Tile `key`'s PMTiles id: its Hilbert curve index within its zoom, plus the number of tiles at +/// all lower zooms. pub(crate) fn tile_id(key: TileKey) -> u64 { let z = u32::from(key.z); let before: u64 = (0..z).map(|k| 1_u64 << (2 * k)).sum(); @@ -170,7 +163,7 @@ mod tests { #[test] fn tile_ids_follow_the_hilbert_curve() { assert_eq!(tile_id(TileKey { z: 0, x: 0, y: 0 }), 0); - // Zoom 1 goes around its four tiles from the top left, down, then right. + // Zoom 1 visits its four tiles from the top left: down, right, then up. let ids: Vec = [(0, 0), (0, 1), (1, 1), (1, 0)].map(|(x, y)| tile_id(TileKey { z: 1, x, y })).to_vec(); assert_eq!(ids, vec![1, 2, 3, 4]); assert_eq!(tile_id(TileKey { z: 2, x: 0, y: 0 }), 5); @@ -178,8 +171,8 @@ mod tests { #[test] fn directories_find_tiles_and_leaves() { - // Entries: ids 0 (a run of 2 tiles), 5 (a leaf); offsets written as offset + 1, then 0 for - // "right after the one before". + // Two entries: id 0 (a run of 2 tiles) and id 5 (a leaf). Offsets are stored as offset + 1, + // or 0 for "right after the previous entry". let bytes = [2, 0, 5, 2, 0, 10, 20, 1, 0]; let directory = Directory::decode(&bytes).expect("a directory"); assert_eq!(directory.find(0), Some(Entry::Tile(0, 10))); diff --git a/crates/latkit/src/geo/shapes.rs b/crates/latkit/src/geo/shapes.rs index ffade701..7fff2e8c 100644 --- a/crates/latkit/src/geo/shapes.rs +++ b/crates/latkit/src/geo/shapes.rs @@ -1,7 +1,9 @@ -//! What geometry becomes to draw: an area's triangles with the point its label hangs from, and on the -//! earth a path that bends with the globe. On the earth, longitudes unwrap across the antimeridian so -//! nothing streaks across the map, and nothing spans more than [`EARTH_STEP`] degrees: triangles -//! split along a grid of that pitch, paths gain points. +//! Converts geometry into drawable shapes: polygon triangles with a label anchor, and earth paths +//! that follow the globe. +//! +//! On the earth, longitudes are unwrapped across the antimeridian so no shape streaks across the +//! map. No piece spans more than [`EARTH_STEP`] degrees: triangles are split along a grid of that +//! pitch, and paths gain points. use std::collections::BinaryHeap; @@ -9,12 +11,14 @@ use super::earcut::earcut; type Point = [f64; 2]; -/// Degrees an earth triangle spans a side at most, and an earth path's segment: a chord this long -/// sags under a two-thousandth of the globe's radius. +/// Largest span in degrees, in longitude or latitude, of an earth triangle or path segment. A chord +/// this long sags less than 1/2000 of the globe's radius. pub(crate) const EARTH_STEP: f64 = 2.0; -/// The triangles of `polygons`, each its outer ring then its holes, as a point list: the point its -/// label hangs from first, then three points per triangle. Without triangles, no points at all. +/// Triangulates `polygons`, each an outer ring followed by holes, into a point list: the label +/// anchor first, then three points per triangle. The anchor lies in the polygon with the largest +/// outer ring. Returns an empty list when there are no triangles. With `earth`, rings are unwrapped +/// and triangles split along the [`EARTH_STEP`] grid. pub(crate) fn triangles(polygons: &[Vec>], earth: bool) -> Vec { let mut out = vec![[f64::NAN; 2]]; let mut largest: Option<(f64, Vec>)> = None; @@ -50,8 +54,8 @@ pub(crate) fn triangles(polygons: &[Vec>], earth: bool) -> Vec out } -/// `parts` as an earth path draws them: longitudes unwrapped across the antimeridian, and a point -/// every [`EARTH_STEP`] degrees along longer segments. `None` when nothing changes. +/// Prepares `parts` for an earth path: longitudes unwrapped across the antimeridian, and points +/// added every [`EARTH_STEP`] degrees along longer segments. `None` if no segment needs either. pub(crate) fn earth_parts(parts: &[Vec]) -> Option>> { let changes = parts.iter().any(|part| { part.windows(2).any(|w| (w[1][0] - w[0][0]).abs() > EARTH_STEP || (w[1][1] - w[0][1]).abs() > EARTH_STEP) @@ -59,8 +63,8 @@ pub(crate) fn earth_parts(parts: &[Vec]) -> Option>> { changes.then(|| parts.iter().map(|part| densified(&unwrap(part))).collect()) } -/// `polygon`'s rings with longitudes unwrapped: each ring continuous, and every ring on the side of -/// the antimeridian the outer ring starts on. +/// Unwraps each ring of `polygon` so its longitudes are continuous, then shifts it by whole turns +/// to start within 180 degrees of the outer ring's first point. fn unwrapped(polygon: &[Vec]) -> Vec> { let first = polygon.first().and_then(|r| r.first()).map_or(0.0, |p| p[0]); polygon @@ -80,7 +84,7 @@ fn unwrapped(polygon: &[Vec]) -> Vec> { .collect() } -/// `line` with each longitude moved by whole turns to within 180 degrees of the one before. +/// Shifts each longitude of `line` by whole turns to within 180 degrees of the previous point's. fn unwrap(line: &[Point]) -> Vec { let mut out: Vec = Vec::with_capacity(line.len()); for &p in line { @@ -93,14 +97,15 @@ fn unwrap(line: &[Point]) -> Vec { out } -/// The whole turns that bring a step of `degrees` within half a turn. A step of exactly half a turn -/// stays: a tile half the world wide has edges that long, and which way round they run is theirs. +/// Whole turns (360 degrees) that bring a longitude step of `degrees` within 180 degrees. A step of +/// exactly 180 is kept as given, because a zoom-1 tile's edges span exactly that and their +/// direction matters. fn turns(degrees: f64) -> f64 { degrees.signum() * ((degrees.abs() - 180.0) / 360.0).ceil().max(0.0) } -/// `line` with points every [`EARTH_STEP`] degrees along its longer segments, straight in longitude -/// and latitude. +/// Adds points every [`EARTH_STEP`] degrees along longer segments of `line`, interpolated linearly +/// in longitude and latitude. fn densified(line: &[Point]) -> Vec { let mut out: Vec = Vec::with_capacity(line.len()); for (k, &p) in line.iter().enumerate() { @@ -117,9 +122,9 @@ fn densified(line: &[Point]) -> Vec { out } -/// Triangle `t` cut along the grid of [`EARTH_STEP`] degrees into the pieces each cell holds, each -/// fanned into triangles. A crossing is worked out from its edge's ends in one order, so triangles -/// sharing an edge share its crossings and no crack opens. +/// Cuts triangle `t` along the [`EARTH_STEP`] grid and fans each cell's piece into triangles. A +/// triangle within one cell is kept whole. Each crossing is computed from its edge's endpoints in a +/// fixed order, so triangles sharing an edge get identical crossings and no cracks appear. fn split(t: [Point; 3], out: &mut Vec) { let cell = |v: f64| (v / EARTH_STEP).floor(); let lo = [0, 1].map(|k| cell(t[0][k].min(t[1][k]).min(t[2][k]))); @@ -157,8 +162,8 @@ fn split(t: [Point; 3], out: &mut Vec) { } } -/// The part of convex polygon `piece` on one side of the line where coordinate `axis` is `at`: at or -/// above it when `keep_above`, else at or below it. +/// Clips convex polygon `piece` to one side of the line where coordinate `axis` equals `at`: at or +/// above it when `keep_above`, else at or below it. Writes the result to `out`. fn clip(piece: &[Point], axis: usize, at: f64, keep_above: bool, out: &mut Vec) { out.clear(); let inside = |p: &Point| if keep_above { p[axis] >= at } else { p[axis] <= at }; @@ -174,8 +179,8 @@ fn clip(piece: &[Point], axis: usize, at: f64, keep_above: bool, out: &mut Vec

Point { let (a, b) = if (p[0], p[1]) <= (q[0], q[1]) { (p, q) } else { (q, p) }; let t = (at - a[axis]) / (b[axis] - a[axis]); @@ -198,8 +203,8 @@ fn ring_area(ring: &[Point]) -> f64 { sum } -/// The point a polygon's label hangs from: the point inside it farthest from its edges, found to a -/// fiftieth of its size, as polylabel finds it. +/// A polygon's label anchor: the interior point farthest from its edges, found to within 1/50 of +/// its longer side by polylabel's method. fn label_point(rings: &[Vec]) -> Point { let Some(outer) = rings.first() else { return [f64::NAN; 2] }; let (mut lo, mut hi) = ([f64::INFINITY; 2], [f64::NEG_INFINITY; 2]); @@ -213,7 +218,7 @@ fn label_point(rings: &[Vec]) -> Point { if short.is_nan() || short <= 0.0 { return [lo[0].midpoint(hi[0]), lo[1].midpoint(hi[1])]; } - // Cells start square, at most sixteen along the longer side. + // Initial cells are squares as wide as the shorter side, with at most 16 along the longer side. let size = short.max(long / 16.0); let precision = long / 50.0; let probe = |center: Point, half: f64| Cell::new(center, half, signed_distance(center, rings)); @@ -233,7 +238,7 @@ fn label_point(rings: &[Vec]) -> Point { if middle.distance > best.distance { best = middle; } - // Each cell probed costs a pass over the rings; cap them so a huge polygon stays bounded. + // Each probe is a pass over the rings, so cap the probes to bound the cost on huge polygons. let mut probes = 0; while let Some(cell) = queue.pop() { if cell.distance > best.distance { @@ -251,8 +256,9 @@ fn label_point(rings: &[Vec]) -> Point { best.center } -/// A square cell of polylabel's search: its center, half its side, the distance from its center to -/// the polygon's edges (negative outside), and the most any point of it could reach. +/// A square cell in polylabel's search: its center, half its side, the signed distance from its +/// center to the polygon's edges (negative outside), and `max`, an upper bound on that distance for +/// any point in the cell. #[derive(Clone, Copy)] struct Cell { center: Point, @@ -337,7 +343,7 @@ mod tests { vec![[x, y], [x + side, y], [x + side, y + side], [x, y + side], [x, y]] } - /// The area the triangles of a point list cover. + /// Total area of a point list's triangles, after its label anchor. fn covered(points: &[Point]) -> f64 { points[1..] .chunks(3) @@ -357,8 +363,8 @@ mod tests { } #[test] - fn a_half_world_tile_stays_where_it_lies_from_any_corner() { - // A tile at zoom 1 is half the world wide: its edges run exactly 180 degrees. + fn a_half_world_tile_does_not_wrap_regardless_of_its_first_corner() { + // A zoom-1 tile is half the world wide, so its edges span exactly 180 degrees. let corners = [[0.0, 0.0], [180.0, 0.0], [180.0, 80.0], [0.0, 80.0]]; for start in 0..4 { let mut ring: Vec = (0..4).map(|k| corners[(start + k) % 4]).collect(); @@ -388,7 +394,7 @@ mod tests { t.iter().map(|p| p[k]).fold(f64::INFINITY, f64::min), t.iter().map(|p| p[k]).fold(f64::NEG_INFINITY, f64::max), ); - assert!(hi - lo <= EARTH_STEP + 1e-9, "a piece spans {lo}..{hi}"); + assert!(hi - lo <= EARTH_STEP + 1e-9, "a triangle spans at most one grid step: {lo}..{hi}"); } } } @@ -403,8 +409,8 @@ mod tests { #[test] fn earth_parts_bend_and_unwrap() { - assert!(earth_parts(&[vec![[0.0, 0.0], [1.0, 1.0]]]).is_none(), "short segments stay"); - let parts = earth_parts(&[vec![[170.0, 10.0], [-170.0, 10.0]]]).expect("crosses"); + assert!(earth_parts(&[vec![[0.0, 0.0], [1.0, 1.0]]]).is_none(), "short segments need no bending"); + let parts = earth_parts(&[vec![[170.0, 10.0], [-170.0, 10.0]]]).expect("a long segment bends"); assert_eq!(parts[0].first(), Some(&[170.0, 10.0])); assert_eq!(parts[0].last(), Some(&[190.0, 10.0])); assert!(parts[0].windows(2).all(|w| (w[1][0] - w[0][0]).abs() <= EARTH_STEP + 1e-9)); diff --git a/crates/latkit/src/geo/tiles.rs b/crates/latkit/src/geo/tiles.rs index e6a4ada5..f24b0b43 100644 --- a/crates/latkit/src/geo/tiles.rs +++ b/crates/latkit/src/geo/tiles.rs @@ -1,34 +1,35 @@ -//! Vector tiles as data: where tiles come from, and a tile decoded into the rows of the tables marks -//! draw from it. +//! Vector tiles as data: tile URL templates, and decoding a tile into table rows with their shapes. -use super::mvt::{self, Kind, Property, Shape}; -use crate::data::{Ids, Name, Patch, Table, Values}; +use super::mvt::{self, Kind, Shape}; +use crate::data::{Ids, Name, Patch, Table, Value, Values}; +use crate::host::Shaping; use crate::tiles::TileKey; -/// Tiles fetched from a URL template, from a TileJSON document or given as one. +/// A tile URL template with its zoom range and bounds, read from TileJSON or given directly. #[derive(Clone, PartialEq, Debug)] pub(crate) struct Template { pub(crate) url: String, pub(crate) min_zoom: u8, pub(crate) max_zoom: u8, - /// West, south, east, north. + /// West, south, east, north, in degrees. pub(crate) bounds: [f64; 4], - /// Whether rows count from the south, as TMS has them. + /// Whether tile rows count from the south (the TMS scheme). pub(crate) tms: bool, } -/// The whole earth web mercator reaches. +/// Bounds of the web mercator world: west, south, east, north, in degrees. pub(crate) const WORLD: [f64; 4] = [-180.0, -85.051_128_78, 180.0, 85.051_128_78]; impl Template { - /// The template `url`, its zooms and bounds as most sources serve them. + /// Template `url` with common defaults: zooms 0 to 14, world bounds, and XYZ row order. pub(crate) fn of(url: &str) -> Template { Template { url: url.to_owned(), min_zoom: 0, max_zoom: 14, bounds: WORLD, tms: false } } - /// The TileJSON document `bytes`, found at `base`. + /// Reads the TileJSON document `bytes` fetched from `base`, resolving its first tile URL + /// against `base`. Errors if it is not TileJSON, names no tiles, or names image tiles. pub(crate) fn tilejson(bytes: &[u8], base: &str) -> Result { - /// What a TileJSON document says that tiles draw by. + /// The TileJSON members read. #[derive(serde::Deserialize)] struct TileJson { #[serde(default)] @@ -39,13 +40,12 @@ impl Template { bounds: Option>, scheme: Option, } - let text = std::str::from_utf8(bytes).map_err(|e| format!("{base} is not TileJSON: {e}"))?; - let json = crate::from_json::(text).map_err(|e| format!("{base} is not TileJSON: {e}"))?; - let template = json.tiles.first().ok_or_else(|| format!("the TileJSON at {base} names no tiles"))?; + let json = crate::json::from_bytes::(bytes).map_err(|e| format!("{base} is not TileJSON: {e}"))?; + let template = json.tiles.first().ok_or_else(|| format!("the TileJSON at {base} lists no tile URLs"))?; if let Some(format) = json.format.as_deref() && ["png", "jpg", "jpeg", "webp", "avif"].contains(&format) { - return Err(format!("the TileJSON at {base} names image tiles; only vector tiles draw")); + return Err(format!("the TileJSON at {base} has image tiles; only vector tiles are supported")); } let zoom = |z: Option, or: u8| z.filter(|z| *z >= 0.0).map_or(or, |z| z.min(24.0) as u8); let bounds = json.bounds.as_deref().and_then(|b| <[f64; 4]>::try_from(b).ok()).unwrap_or(WORLD); @@ -65,21 +65,33 @@ impl Template { } } -/// A table a tile decodes into: its layer's features of one kind, as table `table`. +/// A table decoded from each tile: the features of one `kind` in layer `layer`, as table `table`. +/// `shapes` lists the derived shape fields its marks draw. #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub(crate) struct Decoded { pub(crate) layer: String, pub(crate) kind: Kind, pub(crate) table: Name, + pub(crate) shapes: Vec, } -/// Features a tile's table numbers at most; ids beyond its place in the tile never collide. +/// The most rows one tile adds to a table. Further rows are dropped. Row ids are the tile's id +/// times this plus the row's index, so ids from different tiles never collide. pub(crate) const FEATURES: u64 = 1 << 20; -/// Tile `key`'s rows as a patch of every table of `tables`: each feature of the table's layer and -/// kind a row (a multi-point a row per point), its geometry field `geometry` and its properties -/// fields. A row's id is the tile's place along the Hilbert curve times [`FEATURES`], plus its own -/// place among the table's rows. `None` when the bytes are no tile. +/// Name of the field holding a tile row's geometry. +const GEOMETRY: &str = "geometry"; + +/// Whether `name` is a derived shape field of a tile's table. Properties of that name are skipped. +pub(crate) fn shapes_field(name: &str) -> bool { + [Shaping::Triangles { earth: true }, Shaping::Path].iter().any(|s| *s.field_of(GEOMETRY) == *name) +} + +/// Decodes tile `key` from `bytes`, optionally gzip-compressed, into a patch setting each table of +/// `tables`. Each feature of the table's layer and kind becomes a row (a multi-point, a row per +/// point) with field `geometry`, its derived shape fields, and a field per property. A row's id is +/// the tile's Hilbert curve id times [`FEATURES`], plus the row's index. `None` if `bytes` are not +/// a tile. pub(crate) fn decode(bytes: &[u8], key: TileKey, tables: &[Decoded]) -> Option { let bytes = super::gunzipped(bytes).ok()?; let layers = mvt::decode(&bytes, key)?; @@ -88,7 +100,7 @@ pub(crate) fn decode(bytes: &[u8], key: TileKey, tables: &[Decoded]) -> Option

= Vec::new(); - let mut properties: Vec<&[(u32, Property)]> = Vec::new(); + let mut properties: Vec<&[(u32, Value)]> = Vec::new(); for feature in layer.features.iter().filter(|f| f.shape.kind() == decoded.kind) { match &feature.shape { Shape::Points(points) => { @@ -107,7 +119,7 @@ pub(crate) fn decode(bytes: &[u8], key: TileKey, tables: &[Decoded]) -> Option

= (0..count as u64).map(|k| first + k).collect(); + let ids: Ids = (0..count as u64).map(|k| first + k).collect::>().into(); let geometry: Values = match decoded.kind { Kind::Points => shapes[..count] .iter() @@ -127,17 +139,22 @@ pub(crate) fn decode(bytes: &[u8], key: TileKey, tables: &[Decoded]) -> Option

>() .into(), }; - let mut table = Table::new(decoded.table.clone(), Ids::from(ids)).field("geometry", geometry); + let depth = geometry.geometry().map_or(0, crate::data::Geometry::depth); + let mut table = Table::new(decoded.table.clone(), ids); + for &shaping in decoded.shapes.iter().filter(|s| s.takes(depth)) { + table = table.field(shaping.field_of(GEOMETRY), shaping.shapes_of(&geometry)); + } + table = table.field(GEOMETRY, geometry); for (k, key) in layer.keys.iter().enumerate() { - if key == "geometry" { + if key == GEOMETRY || shapes_field(key) { continue; } - let values: Vec> = properties[..count] + let values: Vec> = properties[..count] .iter() - .map(|props| props.iter().find(|(i, _)| *i as usize == k).map(|(_, v)| v)) + .map(|props| props.iter().find(|(i, _)| *i as usize == k).map(|(_, v)| v.clone())) .collect(); if values.iter().any(Option::is_some) { - table = table.field(key.as_str(), field(&values)); + table = table.field(key.as_str(), Values::infer(&values)); } } patch = patch.set(table); @@ -145,36 +162,6 @@ pub(crate) fn decode(bytes: &[u8], key: TileKey, tables: &[Decoded]) -> Option

]) -> Values { - let present = || values.iter().flatten(); - if present().all(|v| matches!(v, Property::Number(_))) { - return values - .iter() - .map(|v| v.and_then(|v| if let Property::Number(n) = v { Some(*n) } else { None })) - .collect::>() - .into(); - } - if present().all(|v| matches!(v, Property::Bool(_))) { - return values - .iter() - .map(|v| v.and_then(|v| if let Property::Bool(b) = v { Some(*b) } else { None })) - .collect::>() - .into(); - } - values.iter().map(|v| v.map(text)).collect::>>().into() -} - -/// A property written as text. -pub(crate) fn text(property: &Property) -> String { - match property { - Property::Text(t) => t.clone(), - Property::Number(n) => n.to_string(), - Property::Bool(b) => b.to_string(), - } -} - #[cfg(test)] mod tests { use super::*; @@ -185,16 +172,22 @@ mod tests { fn a_tile_decodes_into_its_tables_rows() { let key = TileKey { z: 3, x: 2, y: 3 }; let bytes = tile("water", 3, &square(100, 100, 1000), "bay"); - let tables = [Decoded { layer: "water".into(), kind: Kind::Polygons, table: "water".into() }]; + let triangles = Shaping::Triangles { earth: true }; + let tables = + [Decoded { layer: "water".into(), kind: Kind::Polygons, table: "water".into(), shapes: vec![triangles] }]; let patch = decode(&bytes, key, &tables).expect("a tile"); - let mut data = Data::new("base"); + let data = Data::new("base"); data.apply(patch).expect("valid"); + let data = data.snapshot(); let water = data.table("water").expect("the table"); assert_eq!(water.len(), 1); let id = crate::geo::pmtiles::tile_id(key) * FEATURES; assert_eq!(water.get(id, "name"), Some(Value::Text("bay".into())), "stable ids"); assert!(matches!(water.get(id, "geometry"), Some(Value::Polygons(_)))); - let other = [Decoded { layer: "water".into(), kind: Kind::Lines, table: "water".into() }]; + // The derived triangles field: a label point and at least one triangle. + let triangles = water.get(id, "geometry.triangles"); + assert!(matches!(triangles, Some(Value::Path(points)) if points.len() >= 4)); + let other = [Decoded { layer: "water".into(), kind: Kind::Lines, table: "water".into(), shapes: Vec::new() }]; assert!(decode(&bytes, key, &other).expect("a tile").is_empty(), "no lines in it"); } @@ -208,7 +201,7 @@ mod tests { } #[test] - fn tile_bounds_meet_their_parents() { + fn tile_bounds_are_mercator_and_children_map_back_to_parents() { let key = TileKey { z: 1, x: 1, y: 0 }; let [w, s, e, n] = key.bounds(); assert_eq!([w, s, e], [0.0, 0.0, 180.0]); diff --git a/crates/latkit/src/gpu/arena.rs b/crates/latkit/src/gpu/arena.rs index 09ba3206..be75ff49 100644 --- a/crates/latkit/src/gpu/arena.rs +++ b/crates/latkit/src/gpu/arena.rs @@ -1,60 +1,126 @@ -//! Data pages on the GPU: one pool of 64 Ki-word chunks in one storage buffer, which doubles as it -//! fills, up to the budget. A chunk is carved into blocks of one power-of-two size, and no page is -//! larger than a chunk: a geometry's points split into pieces of a chunk each. So the room any page -//! leaves serves any other: a chunk whose blocks all come back is carved again for another size. +//! GPU memory for pages: one pool of 128 Ki-word chunks in a single storage buffer. The buffer +//! doubles as it fills, up to the budget. It holds uploaded data pages, and the per-row word pages +//! that each view's marks keep and the GPU writes. //! -//! A page is identified by its id, the words of it a program reads, and its piece. Page tables pin -//! the pages they reference, so a page in use is never evicted; unpinned pages stay cached, the least -//! recently used going first when a page needs their room. Uploads stream within a per-frame byte -//! budget. A page that finds no room waits until pins let room go: views plan what they draw to fit -//! the pool, so none waits long. +//! Each chunk is carved into blocks of one power-of-two size. No page is larger than a chunk, and a +//! geometry's points are split into pieces of one row group each. So space that any page frees can +//! serve any other: once all of a chunk's blocks are free, it can be carved again for another size. +//! With few sizes in few chunks, partly used chunks waste little space, which matters with a small +//! budget. +//! +//! A data page is keyed by its id and the words a program reads from it. Geometry pieces are keyed +//! by content, so a rewritten page finds the pieces of its unchanged rows already resident. A view +//! reserves all the pages its next draw reads at once, or none. Resident pages are pinned, and +//! missing ones get blocks, evicting the least recently used unpinned pages. So nothing is partly +//! pinned and no page waits for space. Page tables pin the pages they reference, so a page in use +//! is never evicted, and unpinned pages stay cached. Uploads stream into the reserved blocks within +//! a per-frame byte budget. Views plan their drawing to fit the pool, so a reservation rarely finds +//! no space. When it does, it waits for the next frame, which grows the pool by the shortfall. +//! +//! A per-row word page is a block of its own. It is cleared when handed out if it is read before +//! anything writes it, and freed only once the frame that released it is done. The pool grows only +//! before a frame records writes to the pages, since growing copies the buffer. A block that finds +//! no space after that waits for the next frame. -use std::collections::hash_map::Entry; use std::collections::{BTreeMap, BTreeSet, HashMap, VecDeque}; +use std::hash::{Hash, Hasher}; +use std::sync::Arc; -use rustc_hash::FxBuildHasher; +use rustc_hash::{FxBuildHasher, FxHasher}; -use crate::data::{AxisPage, FramePage, PageId, Value, Values}; +use crate::data::{AxisPage, FRAME_PAGE_WORDS, FramePage, PageId, Value, Values}; use crate::expr::compile::Words; use crate::gpu::shaders::{MISSING, NONE, POINT_BITS}; use crate::gpu::work::Queue; -/// Words in a chunk: the largest block, and the largest page. -pub(crate) const CHUNK_WORDS: u32 = 1 << 16; +/// Words in a chunk, as a power of two. A chunk is also the largest block and the largest page. +const CHUNK_BITS: u32 = 17; +pub(crate) const CHUNK_WORDS: u32 = 1 << CHUNK_BITS; /// Bytes in a chunk. pub(crate) const CHUNK_BYTES: u64 = CHUNK_WORDS as u64 * 4; const MIN_CLASS_BITS: u32 = 6; -const CLASSES: usize = (16 - MIN_CLASS_BITS + 1) as usize; -const INITIAL_CHUNKS: u32 = 16; -/// Points in a piece of a geometry's points: a chunk of x and y words. +const CLASSES: usize = (CHUNK_BITS - MIN_CLASS_BITS + 1) as usize; +/// 64-bit words in a chunk's free bitmap, one bit per block of the smallest class. +const FREE_WORDS: usize = 1 << (CHUNK_BITS - MIN_CLASS_BITS - 6); +const INITIAL_CHUNKS: u32 = 8; +/// Most points a geometry piece holds, each stored as an x and a y word. pub(crate) const PIECE_POINTS: usize = 1 << POINT_BITS; +const _: () = assert!(2 * PIECE_POINTS <= CHUNK_WORDS as usize, "a piece of points fits a chunk"); +/// Rows per group. A geometry piece holds the points of one group of this many rows, or fewer when +/// it fills, so rewriting a row changes only its group's piece. +const GROUP_ROWS: usize = 64; -/// A page as a program reads it. +/// Key of a resident page: the page and how a program reads it. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] pub(crate) struct PageKey { pub(crate) page: PageId, pub(crate) words: Words, - /// The part `f32` drops of an `f64`, rather than the value. + /// Whether the page holds the low part of `f64` values (what `f32` drops) instead of the + /// values. pub(crate) low: bool, - /// Which piece of the page: a geometry's first points, then its points a chunk at a time; 0 - /// for every other page, which is one piece. - pub(crate) piece: u32, } impl PageKey { - /// The one piece of a page that is not a geometry's. pub(crate) fn of(page: PageId, words: Words, low: bool) -> PageKey { - PageKey { page, words, low, piece: 0 } + PageKey { page, words, low } + } + + /// The key of geometry piece `words`, from a hash of its content. + pub(crate) fn piece(words: &[u32]) -> PageKey { + let mut hasher = FxHasher::default(); + words.hash(&mut hasher); + PageKey { page: PageId::of_content(hasher.finish()), words: Words::Path, low: false } + } +} + +/// Piece keys of recently drawn geometry pages, by page id: the key of the page's row table, then +/// the content keys of its point pieces, with its point count. A page drawn again reuses them +/// without rebuilding or recounting. A rewritten page rebuilds them and uploads the pieces the GPU +/// lacks. +#[derive(Default)] +pub(crate) struct Pieces { + keys: HashMap, usize), FxBuildHasher>, + order: VecDeque, +} + +/// Most geometry pages whose piece keys are kept. A page forgotten past this rebuilds its pieces +/// when drawn. +const PIECED_PAGES: usize = 1 << 14; + +impl Pieces { + /// The piece keys and point count of page `page`, if kept. + pub(crate) fn of(&self, page: PageId) -> Option<(Arc<[PageKey]>, usize)> { + self.keys.get(&page).cloned() + } + + /// Keeps `keys` as page `page`'s piece keys, with `points` points. Beyond [`PIECED_PAGES`] + /// pages, forgets the oldest. + pub(crate) fn keep(&mut self, page: PageId, keys: Arc<[PageKey]>, points: usize) { + if self.keys.insert(page, (keys, points)).is_none() { + self.order.push_back(page); + } + while self.order.len() > PIECED_PAGES { + let Some(old) = self.order.pop_front() else { break }; + self.keys.remove(&old); + } } } -/// What to upload for a page. +/// A page in a reservation: its key, the words it must hold, and the content to upload if the pool +/// lacks them. +pub(crate) struct Ask { + pub(crate) key: PageKey, + pub(crate) need: u32, + pub(crate) content: Option, +} + +/// The content uploaded for a page. pub(crate) enum Content { - /// Static values, converted to words for `words`. + /// Static values, converted to words as `words` reads them. Values { values: Values, words: Words, low: bool }, - /// A page of samples; one an append opened grows, and only its new tail uploads. + /// A page of samples. A page that an append opened grows, and only its new tail uploads. Frames(FramePage), - /// A page of where a level's entries draw along the axis; it grows like a page of samples. + /// A page of axis positions for a level's entries. It grows like a page of samples. Axis(AxisPage), /// Words computed on the CPU: text codes, join rows, a geometry's pieces. Computed(Vec), @@ -73,7 +139,7 @@ impl Content { len as u32 } - /// The page's words from word `from` on: a growing page uploads only its new tail. + /// The page's words from word `from` on, so a growing page uploads only its new tail. fn words(&self, from: usize) -> Vec { match self { Content::Values { values, words, low } => page_words(values, *words, *low).split_off(from), @@ -85,17 +151,24 @@ impl Content { } } - /// The class of the block it takes: a growing page takes a whole chunk to grow into. + /// The block class the page takes. A growing page takes a block of its kind's largest page + /// size, to grow into. fn class(&self, words: u32) -> usize { match self { - Content::Frames(page) if page.grows() => CLASSES - 1, - Content::Axis(page) if page.grows() => CLASSES - 1, + Content::Frames(page) if page.grows() => class_of(FRAME_PAGE_WORDS as u32), + Content::Axis(page) if page.grows() => class_of(FRAME_PAGE_WORDS as u32), _ => class_of(words), } } } -/// The class of a block holding `words`: the power of two at least that large. +/// Words in the block that holds `words`. +pub(crate) fn rounded(words: u32) -> u32 { + class_words(class_of(words)) +} + +/// The class of a block holding `words`: the smallest power of two at least that large, and at +/// least 64. fn class_of(words: u32) -> usize { let bits = words.max(1).next_power_of_two().trailing_zeros().max(MIN_CLASS_BITS); (bits - MIN_CLASS_BITS) as usize @@ -110,8 +183,9 @@ fn slots(class: usize) -> u32 { CHUNK_WORDS / class_words(class) } -/// What a page uploads as `words` reads it: numbers as `f32`, the low part of `f64`s, point -/// components, or reference rows as `u32`. Missing values are `MISSING`, or `NONE` for rows. +/// The words a page uploads for `words`: numbers as `f32` (with `low`, the low part of each `f64`), +/// point components, or reference rows as `u32`. Missing values become `MISSING`, or `NONE` for +/// rows. pub(crate) fn page_words(values: &Values, words: Words, low: bool) -> Vec { let n = values.len(); let word = |x: f64| { @@ -138,30 +212,24 @@ pub(crate) fn page_words(values: &Values, words: Words, low: bool) -> Vec { } } -/// Points a geometry page's rows take as shaders read them: every coordinate, and a break between a -/// row's parts or rings. -pub(crate) fn path_points(values: &Values) -> usize { - let Some(geometry) = values.geometry() else { return 0 }; - (0..values.len()).filter(|&row| values.is_valid(row)).map(|row| geometry.row_points(row)).sum() +/// A geometry page as `lib/read.wesl` reads it. The row table has two words per row. The first is +/// the index of the piece where the row's points begin, shifted left by `POINT_BITS`, bitwise-or +/// the index of its first point in that piece. The second is its point count. Then come the pieces, +/// with each point stored as an x and a y word. +pub(crate) struct PathPage { + pub(crate) rows: Vec, + pub(crate) pieces: Vec>, } -/// A geometry page's pieces, as `lib/read.wesl` reads them: each row's first point, then its points -/// a chunk at a time, every point an x and a y word. A row's parts, or its polygons' rings, are -/// broken by a missing point. -pub(crate) fn path_pieces(values: &Values) -> Vec> { - let mut firsts = Vec::with_capacity(values.len()); - let mut pieces: Vec> = vec![Vec::new()]; - let mut points = 0_u32; - let push = |point: [u32; 2], pieces: &mut Vec>, points: &mut u32| { - if pieces.last().is_some_and(|p| p.len() == 2 * PIECE_POINTS) { - pieces.push(Vec::with_capacity(2 * PIECE_POINTS)); - } - pieces.last_mut().expect("a piece").extend(point); - *points += 1; - }; +/// Builds a geometry page's row table and pieces. Each group of rows starts a new piece, as does a +/// row that does not fit in the current one, so rewriting a row changes only its group's piece. A +/// row larger than a piece starts one and fills as many as it needs. A missing point separates a +/// row's parts, or its polygons' rings. +pub(crate) fn path_pieces(values: &Values) -> PathPage { + let mut rows = Vec::with_capacity(2 * values.len()); + let mut pieces: Vec> = Vec::new(); let bits = |p: &[f64; 2]| p.map(|v| (v as f32).to_bits()); for row in 0..values.len() { - firsts.push(points); let parts: Vec> = match values.get(row) { Some(Value::Point(p)) => vec![vec![p]], Some(Value::Path(path)) => vec![path], @@ -169,42 +237,72 @@ pub(crate) fn path_pieces(values: &Values) -> Vec> { Some(Value::Polygons(polygons)) => polygons.into_iter().flatten().collect(), _ => Vec::new(), }; + let count = parts.iter().map(Vec::len).sum::() + parts.len().saturating_sub(1); + let begins = pieces.last().is_none_or(|piece| { + let held = piece.len() / 2; + held == PIECE_POINTS || (held > 0 && (row % GROUP_ROWS == 0 || count > PIECE_POINTS - held)) + }); + if begins { + pieces.push(Vec::new()); + } + let slot = pieces.len() - 1; + rows.extend([(slot as u32) << POINT_BITS | (pieces[slot].len() / 2) as u32, count as u32]); + let mut push = |point: [u32; 2]| { + if pieces.last().is_some_and(|piece| piece.len() == 2 * PIECE_POINTS) { + pieces.push(Vec::new()); + } + pieces.last_mut().expect("a piece").extend(point); + }; for (k, part) in parts.iter().enumerate() { if k > 0 { - push([MISSING, MISSING], &mut pieces, &mut points); + push([MISSING, MISSING]); } for p in part { - push(bits(p), &mut pieces, &mut points); + push(bits(p)); } } } if pieces.last().is_some_and(Vec::is_empty) { pieces.pop(); } - std::iter::once(firsts).chain(pieces).collect() + PathPage { rows, pieces } +} + +/// Approximate arena bytes for a geometry page of `rows` rows and `points` points: its row table, +/// plus its points in one piece per row group, or in full pieces when groups hold more. +pub(crate) fn path_page_bytes(rows: usize, points: usize) -> u64 { + let groups = rows.div_ceil(GROUP_ROWS).max(1); + let per_group = points.div_ceil(groups); + let pieces = if per_group <= PIECE_POINTS { + groups as u64 * u64::from(rounded(2 * per_group as u32)) + } else { + points.div_ceil(PIECE_POINTS) as u64 * u64::from(rounded(2 * PIECE_POINTS as u32)) + }; + (u64::from(rounded(2 * rows as u32)) + if points > 0 { pieces } else { 0 }) * 4 } struct Block { - /// Offset in words, or `NONE` until allocated. + /// Offset in words. offset: u32, class: usize, /// Words uploaded so far. written: u32, pins: u32, - /// When it was last unpinned, its place in the LRU. + /// When it was last unpinned: its position in the LRU order. tick: u64, } -/// A chunk of the pool: free, or carved into blocks of one class with a bit for each that is free; -/// the class is the one whose chunks list it. +/// A chunk of the pool: free, or carved into blocks of one class with a bit per free block. The +/// chunk does not store its class. `Arena::partial` lists it under its class while it has a free +/// block. #[derive(Clone, Copy)] struct Chunk { live: u32, - free: [u64; 16], + free: [u64; FREE_WORDS], } impl Chunk { - const FREE: Chunk = Chunk { live: 0, free: [0; 16] }; + const FREE: Chunk = Chunk { live: 0, free: [0; FREE_WORDS] }; /// Takes a free block, returning its slot. fn take(&mut self) -> u32 { @@ -219,11 +317,12 @@ impl Chunk { pub(crate) struct Arena { buffer: wgpu::Buffer, chunks: Vec, - /// The most chunks the pool grows to. + /// Most chunks the pool may grow to. limit: u32, /// Changes whenever the buffer is replaced, so views rebind it. pub(crate) generation: u64, - /// Free chunks, and each class's chunks with a free block, lowest first so blocks gather low. + /// Free chunks, and each class's chunks with a free block. The lowest is used first, so blocks + /// gather at low offsets. free: BTreeSet, partial: [BTreeSet; CLASSES], blocks: HashMap, @@ -234,12 +333,18 @@ pub(crate) struct Arena { contents: HashMap, /// Bytes of allocated blocks. resident: u64, - /// Whether the page at the front of the queue found no room, the last time it tried. - starved: bool, + /// Per-row blocks handed out and not yet cleared. `freed` holds released ones, freed when the + /// next frame begins. + cleared: Vec<(u32, usize)>, + freed: Vec<(u32, usize)>, + /// Whether this frame recorded writes to the pages, after which the pool keeps its buffer. + /// `short` counts the words of blocks that found no space. The next frame grows the pool by it. + writing: bool, + short: u64, } impl Arena { - /// A pool that grows to at most `bytes`, within what the device binds. + /// A pool that grows to at most `bytes`, within what the device can bind. pub(crate) fn new(device: &wgpu::Device, bytes: u64) -> Arena { let limit = limit_of(device, bytes); let chunks = INITIAL_CHUNKS.min(limit); @@ -256,66 +361,177 @@ impl Arena { uploads: VecDeque::new(), contents: HashMap::default(), resident: 0, - starved: false, + cleared: Vec::new(), + freed: Vec::new(), + writing: false, + short: 0, + } + } + + /// Begins a frame for all views drawing from the pool. Frees the blocks the last frame released + /// and grows the pool to cover its shortfall, at least doubling it, before anything this frame + /// writes the pages. + pub(crate) fn begin_frame(&mut self, device: &wgpu::Device, queue: &Queue) { + for (offset, class) in std::mem::take(&mut self.freed) { + self.release(offset, class); + } + self.writing = false; + let short = std::mem::take(&mut self.short).div_ceil(u64::from(CHUNK_WORDS)); + if short > 0 { + let chunks = self.chunks.len() as u64; + self.grow(device, queue, (chunks + short).max(2 * chunks).min(u64::from(u32::MAX)) as u32); } } + /// Reserves every page in `asks`, or none. Pins the resident ones and gives the missing ones + /// blocks, evicting least recently used unpinned pages but never these. Queues uploads for new + /// pages and for resident pages holding fewer words than asked. Returns `false`, pinning + /// nothing, if a missing page has no content to upload or finds no space. When it finds no + /// space, the next frame grows the pool by the shortfall. + pub(crate) fn reserve(&mut self, device: &wgpu::Device, queue: &Queue, asks: &mut [Ask]) -> bool { + // Pin resident pages first, so placing the rest evicts none of them. + let held: Vec = asks.iter().map(|ask| self.blocks.contains_key(&ask.key)).collect(); + for ask in asks.iter().zip(&held).filter(|&(_, &held)| held).map(|(ask, _)| ask) { + self.pin(ask.key); + } + let mut placed: HashMap = HashMap::default(); + let mut room = true; + for (ask, _) in asks.iter().zip(&held).filter(|&(_, &held)| !held) { + if placed.contains_key(&ask.key) { + continue; + } + let class = ask.content.as_ref().map(|content| content.class(content.len().max(ask.need))); + let Some(offset) = class.and_then(|class| self.allocate(device, queue, class)) else { + room = false; + break; + }; + placed.insert(ask.key, (offset, class.expect("placed"))); + } + if !room { + for (offset, class) in placed.into_values() { + self.release(offset, class); + } + for ask in asks.iter().zip(&held).filter(|&(_, &held)| held).map(|(ask, _)| ask) { + self.unpin(ask.key); + } + return false; + } + for (key, (offset, class)) in placed { + self.blocks.insert(key, Block { offset, class, written: 0, pins: 0, tick: 0 }); + } + for (ask, held) in asks.iter_mut().zip(held) { + if !held { + self.pin(ask.key); + } + if self.blocks[&ask.key].written < ask.need + && !self.contents.contains_key(&ask.key) + && let Some(content) = ask.content.take() + { + self.contents.insert(ask.key, content); + self.uploads.push_back(ask.key); + } + } + true + } + + /// Allocates a separate block of `words` for a per-row word page. With `cleared`, the next + /// [`clear`](Arena::clear) zeroes it. Returns `None` if the pool has no space now. The next + /// frame then grows it. + pub(crate) fn block(&mut self, device: &wgpu::Device, queue: &Queue, words: u32, cleared: bool) -> Option { + let class = class_of(words); + let offset = self.allocate(device, queue, class)?; + if cleared { + self.cleared.push((offset, class)); + } + Some(offset) + } + + /// Releases block `offset` of `words` from [`block`](Arena::block). It is freed when the next + /// frame begins, once this frame is done with it. + pub(crate) fn free(&mut self, offset: u32, words: u32) { + self.freed.push((offset, class_of(words))); + } + + /// Clears the blocks handed out since the last clear, immediately, before anything this frame + /// records or writes into them. + pub(crate) fn clear(&mut self, device: &wgpu::Device, queue: &Queue) { + if self.cleared.is_empty() { + return; + } + let mut encoder = + device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("latkit clear") }); + for (offset, class) in self.cleared.drain(..) { + queue.clear_buffer(&mut encoder, &self.buffer, u64::from(offset) * 4, u64::from(class_words(class)) * 4); + } + queue.submit([encoder.finish()]); + } + + /// Notes that this frame records writes to the pages. From now on the pool keeps its buffer, + /// since growing would copy it before those writes run. + pub(crate) fn written(&mut self) { + self.writing = true; + } + pub(crate) fn buffer(&self) -> &wgpu::Buffer { &self.buffer } - /// Lets the pool grow to at most `bytes` from now on; it never shrinks. + /// Whether the pool has grown as far as the budget lets it. + pub(crate) fn full(&self) -> bool { + self.chunks.len() as u32 >= self.limit + } + + /// Lets the pool grow to at most `bytes` from now on. It never shrinks. pub(crate) fn set_limit(&mut self, device: &wgpu::Device, bytes: u64) { self.limit = limit_of(device, bytes).max(self.chunks.len() as u32); } - /// The block holding at least `need` words of `key`, or `None` while it uploads. - pub(crate) fn get(&mut self, key: PageKey, need: u32, content: impl FnOnce() -> Content) -> Option { - if let Some(offset) = self.offset(key, need) { - return Some(offset); - } - if let Entry::Vacant(entry) = self.contents.entry(key) { - entry.insert(content()); - self.uploads.push_back(key); - } - None + /// The offset of the block holding at least `need` words of `key`, if any. + pub(crate) fn offset(&self, key: PageKey, need: u32) -> Option { + self.blocks.get(&key).filter(|b| b.written >= need).map(|b| b.offset) } - /// The block holding at least `need` words of `key`, if one does. - pub(crate) fn offset(&self, key: PageKey, need: u32) -> Option { - self.blocks.get(&key).filter(|b| b.offset != NONE && b.written >= need).map(|b| b.offset) + /// Bytes of the block `key` occupies, or 0. + pub(crate) fn bytes_of(&self, key: PageKey) -> u64 { + self.blocks.get(&key).map_or(0, |b| u64::from(class_words(b.class)) * 4) } - /// Keeps `key` resident while pinned; pins nest. - pub(crate) fn pin(&mut self, key: PageKey) { - let block = self.blocks.entry(key).or_insert(Block { offset: NONE, class: 0, written: 0, pins: 0, tick: 0 }); - if block.pins == 0 && block.offset != NONE { + /// Whether the pool holds, or is uploading, at least `need` words of `key`. A reservation + /// supplies content to upload only when it does not. + pub(crate) fn holds(&self, key: PageKey, need: u32) -> bool { + self.offset(key, need).is_some() || (self.blocks.contains_key(&key) && self.contents.contains_key(&key)) + } + + /// Pins resident page `key` so it is not evicted. Pins nest. + fn pin(&mut self, key: PageKey) { + let block = self.blocks.get_mut(&key).expect("a resident page"); + if block.pins == 0 { self.unpinned.remove(&block.tick); } block.pins += 1; } + /// Releases a pin a reservation took. A page no longer pinned stays cached, unless it is still + /// uploading: nothing waits for it now, so it is freed and its upload dropped. pub(crate) fn unpin(&mut self, key: PageKey) { let Some(block) = self.blocks.get_mut(&key) else { return }; block.pins -= 1; if block.pins > 0 { return; } - if block.offset == NONE { - self.blocks.remove(&key); - self.contents.remove(&key); + if self.contents.remove(&key).is_some() { + let block = self.blocks.remove(&key).expect("a resident page"); + self.release(block.offset, block.class); return; } self.tick += 1; block.tick = self.tick; self.unpinned.insert(self.tick, key); - // Room let go may be what a waiting page needs. - self.starved = false; } - /// Whether uploads are waiting that can make progress. + /// Whether uploads are waiting. pub(crate) fn uploading(&self) -> bool { - !self.uploads.is_empty() && !self.starved + !self.uploads.is_empty() } /// Bytes waiting to upload. @@ -328,65 +544,48 @@ impl Arena { self.resident } - /// Bytes the pool may grow to. - pub(crate) fn capacity(&self) -> u64 { - u64::from(self.limit) * CHUNK_BYTES - } - - /// Uploads queued pages, at least one and at most `budget` bytes; returns the bytes uploaded. - /// - /// A page that finds no room stays queued at the front, and later pages wait behind it, so the - /// pages of one table upload in order; it tries again once a pin lets room go. - pub(crate) fn pump(&mut self, device: &wgpu::Device, queue: &Queue, budget: u64) -> u64 { + /// Uploads queued pages into their reserved blocks until `budget` bytes are reached, always at + /// least one page unless `budget` is 0. Returns the bytes uploaded. + pub(crate) fn pump(&mut self, queue: &Queue, budget: u64) -> u64 { let mut bytes = 0; - while !self.starved - && (bytes == 0 || bytes < budget) + // Views drawn earlier in the frame spent the budget. + if budget == 0 { + return 0; + } + let mut writes: Vec<(u32, Vec)> = Vec::new(); + while (bytes == 0 || bytes < budget) && let Some(key) = self.uploads.pop_front() { let Some(content) = self.contents.remove(&key) else { continue }; - let resident = self.blocks.get(&key).filter(|b| b.offset != NONE).map(|b| (b.offset, b.class, b.written)); - let len = content.len(); - let (offset, written) = match resident { - Some((offset, class, written)) if class_words(class) >= len => (offset, written), - _ => { - let Some(offset) = self.place(device, queue, key, content.class(len)) else { - self.contents.insert(key, content); - self.uploads.push_front(key); - self.starved = true; - break; - }; - (offset, 0) - } - }; + let Some(block) = self.blocks.get_mut(&key) else { continue }; + let (offset, written) = (block.offset, block.written); let tail = content.words(written as usize); + block.written = written + tail.len() as u32; if !tail.is_empty() { - queue.upload(&self.buffer, u64::from(offset + written) * 4, bytemuck::cast_slice(&tail)); bytes += tail.len() as u64 * 4; + writes.push((offset + written, tail)); } - self.blocks.get_mut(&key).expect("placed").written = written + tail.len() as u32; } - bytes - } - - /// Allocates a block of `class` for `key`, releasing a smaller one it had. - fn place(&mut self, device: &wgpu::Device, queue: &Queue, key: PageKey, class: usize) -> Option { - let offset = self.allocate(device, queue, class, key)?; - if let Some(old) = self.blocks.get(&key).filter(|b| b.offset != NONE) { - let (old_offset, old_class) = (old.offset, old.class); - self.release(old_offset, old_class); + // Adjacent pages, such as a column's, upload in one write. + writes.sort_unstable_by_key(|&(at, _)| at); + let mut run: Option<(u32, Vec, u64)> = None; + for (at, words) in writes { + match &mut run { + Some((start, joined, pages)) if *start + joined.len() as u32 == at => { + joined.extend_from_slice(&words); + *pages += 1; + } + _ => { + if let Some((start, joined, pages)) = run.replace((at, words, 1)) { + queue.upload(&self.buffer, u64::from(start) * 4, bytemuck::cast_slice(&joined), pages); + } + } + } } - let block = self.blocks.entry(key).or_insert(Block { offset: NONE, class, written: 0, pins: 0, tick: 0 }); - let pinned = block.pins > 0; - block.offset = offset; - block.class = class; - block.written = 0; - self.resident += u64::from(class_words(class)) * 4; - if !pinned { - self.tick += 1; - block.tick = self.tick; - self.unpinned.insert(self.tick, key); + if let Some((start, joined, pages)) = run { + queue.upload(&self.buffer, u64::from(start) * 4, bytemuck::cast_slice(&joined), pages); } - Some(offset) + bytes } fn release(&mut self, offset: u32, class: usize) { @@ -405,10 +604,11 @@ impl Arena { self.resident -= u64::from(class_words(class)) * 4; } - /// A block of `class`: a free one of its size, a free chunk carved, the pool grown, or room the - /// least recently used unpinned blocks leave, whatever their size; `None` when every block left - /// is pinned. `keep` is the page it is for, which is never evicted for itself. - fn allocate(&mut self, device: &wgpu::Device, queue: &Queue, class: usize, keep: PageKey) -> Option { + /// Allocates a block of `class`. Tries, in order: a free block of that class, a newly carved + /// free chunk, growing the pool (only before this frame writes the pages), then evicting least + /// recently used unpinned blocks of any size. Returns `None` when every remaining block is + /// pinned or a per-row block. The next frame then grows the pool by the shortfall. + fn allocate(&mut self, device: &wgpu::Device, queue: &Queue, class: usize) -> Option { loop { if let Some(&k) = self.partial[class].first() { let chunk = &mut self.chunks[k as usize]; @@ -416,11 +616,12 @@ impl Arena { if chunk.live == slots(class) { self.partial[class].remove(&k); } + self.resident += u64::from(class_words(class)) * 4; return Some(k * CHUNK_WORDS + slot * class_words(class)); } if let Some(k) = self.free.pop_first() { let slots = slots(class); - let mut free = [0_u64; 16]; + let mut free = [0_u64; FREE_WORDS]; for (w, word) in free.iter_mut().enumerate() { let bits = slots.saturating_sub(w as u32 * 64).min(64); *word = if bits == 64 { u64::MAX } else { (1 << bits) - 1 }; @@ -429,20 +630,24 @@ impl Arena { self.partial[class].insert(k); continue; } - if self.grow(device, queue) { + if !self.writing && self.grow(device, queue, 2 * self.chunks.len() as u32) { continue; } - let (&tick, &key) = self.unpinned.iter().find(|&(_, k)| *k != keep)?; + let Some((&tick, &key)) = self.unpinned.iter().next() else { + self.short += u64::from(class_words(class)); + return None; + }; self.unpinned.remove(&tick); - let block = self.blocks.remove(&key).expect("listed"); + let block = self.blocks.remove(&key).expect("an unpinned page is resident"); self.release(block.offset, block.class); } } - /// Doubles the buffer up to the limit, copying what it holds. - fn grow(&mut self, device: &wgpu::Device, queue: &Queue) -> bool { + /// Grows the buffer to `chunks`, capped at the limit, copying its contents. Returns whether it + /// grew. + fn grow(&mut self, device: &wgpu::Device, queue: &Queue, chunks: u32) -> bool { let before = self.chunks.len() as u32; - let chunks = (before * 2).min(self.limit); + let chunks = chunks.min(self.limit); if chunks <= before { return false; } @@ -459,7 +664,8 @@ impl Arena { } } -/// Chunks a pool of at most `bytes` holds, within what the device binds: at least the first ones. +/// Chunks in a pool of at most `bytes`, within what the device can bind, and at least +/// `INITIAL_CHUNKS`. fn limit_of(device: &wgpu::Device, bytes: u64) -> u32 { let limits = device.limits(); let bytes = bytes.min(limits.max_storage_buffer_binding_size).min(limits.max_buffer_size); @@ -485,8 +691,9 @@ mod tests { assert_eq!(class_words(class_of(64)), 64); assert_eq!(class_words(class_of(65)), 128); assert_eq!(class_words(class_of(65536)), 65536); - assert_eq!(CLASSES, 11); - assert_eq!(slots(0), 1024); + assert_eq!(class_words(class_of(CHUNK_WORDS)), CHUNK_WORDS); + assert_eq!(CLASSES, 12); + assert_eq!(slots(0), 2048); assert_eq!(slots(CLASSES - 1), 1); } @@ -500,18 +707,41 @@ mod tests { assert!((back - (1.0 + 1e-9)).abs() < 1e-15); } + /// Row `row`'s piece index, first point, and point count, from a page's row table. + fn entry(page: &PathPage, row: usize) -> (u32, u32, u32) { + let [entry, count] = [page.rows[2 * row], page.rows[2 * row + 1]]; + (entry >> POINT_BITS, entry & ((1 << POINT_BITS) - 1), count) + } + #[test] - fn geometry_pieces_hold_a_chunk_of_points_each() { + fn a_row_larger_than_a_piece_fills_as_many_as_it_takes() { let long: Vec<[f64; 2]> = (0..PIECE_POINTS + 10).map(|i| [i as f64, 0.0]).collect(); let parts = vec![vec![[0.0, 0.0], [1.0, 1.0]], vec![[2.0, 2.0]]]; - let values = Values::from(vec![vec![long.clone()], Vec::new(), parts]); - let pieces = path_pieces(&values); - assert_eq!(pieces[0], vec![0, (PIECE_POINTS + 10) as u32, (PIECE_POINTS + 10) as u32], "first points"); - assert_eq!(pieces.len(), 3, "the first points, then two pieces of points"); - assert_eq!(pieces[1].len(), 2 * PIECE_POINTS); - // The last row's two parts are broken by a missing point. - assert_eq!(pieces[2].len(), 2 * (10 + 2 + 1 + 1)); - assert_eq!(path_points(&values), PIECE_POINTS + 10 + 4); - assert!(pieces.iter().all(|p| p.len() <= CHUNK_WORDS as usize)); + let values = Values::from(vec![vec![long], Vec::new(), parts]); + let page = path_pieces(&values); + assert_eq!(entry(&page, 0), (0, 0, (PIECE_POINTS + 10) as u32), "the long row begins the first piece"); + assert_eq!(page.pieces[0].len(), 2 * PIECE_POINTS, "and fills it"); + assert_eq!(entry(&page, 1), (1, 10, 0), "an empty row has no points and starts where the next begins"); + // A missing point separates the last row's two parts. + assert_eq!(entry(&page, 2), (1, 10, 4)); + assert_eq!(page.pieces[1].len(), 2 * (10 + 4)); + assert_eq!(values.points(), PIECE_POINTS + 10 + 4); + } + + #[test] + fn rewriting_a_row_changes_only_the_piece_of_its_group() { + let rows = |nudge: f64| { + let path = + |r: usize| (0..5).map(|k| [r as f64, f64::from(k) + if r == 70 { nudge } else { 0.0 }]).collect(); + Values::from((0..200).map(|r| vec![path(r)]).collect::>>>()) + }; + let (before, after) = (path_pieces(&rows(0.0)), path_pieces(&rows(0.5))); + assert_eq!(before.pieces.len(), 4, "one piece per group of 64 rows"); + assert_eq!(entry(&before, 64), (1, 0, 5), "each group starts a new piece"); + let keys = |page: &PathPage| page.pieces.iter().map(|p| PageKey::piece(p)).collect::>(); + let changed: Vec = keys(&before).iter().zip(keys(&after)).map(|(a, b)| *a != b).collect(); + assert_eq!(changed, [false, true, false, false], "row 70 lies in the second group"); + assert_eq!(before.rows, after.rows, "the row table stores where points are, not their values"); + assert!(path_page_bytes(200, 1000) < 16 * 1024, "the estimate counts one piece per row group"); } } diff --git a/crates/latkit/src/gpu/atlas.rs b/crates/latkit/src/gpu/atlas.rs index 71bb27c7..29724674 100644 --- a/crates/latkit/src/gpu/atlas.rs +++ b/crates/latkit/src/gpu/atlas.rs @@ -1,66 +1,66 @@ -//! Text on the GPU: runs set once by the host at `TEXT_BASE`, turned into distance fields reaching -//! `TEXT_SPREAD` texels past the text's edge, and packed on shelves of one texture. A distance field -//! draws crisp at any size and gives halos for free, so a run is set once whatever size it shows at. -//! The host sets coverage; the GPU makes the fields of every run set in a frame at once, before the -//! frame draws (`compute/fields.wesl`). When no shelf has room, the shelf used least recently is -//! cleared for the new run; only when none fits does the packing start over. A run counts as used -//! when it is set and whenever labels in view show it, so what the view shows stays. Views follow -//! evictions to drop what they pointed at. +//! The text atlas. The host rasterizes each text run once at `TEXT_BASE`. The GPU turns the raster +//! into a distance field that extends `TEXT_SPREAD` texels past the text's edge, and packs it on a +//! shelf of one texture. A distance field draws crisp at any size and also gives halos, so each run +//! is rasterized once whatever size it shows at. Before a frame draws, the GPU builds the fields of +//! every run rasterized in that frame in one batch (`compute/fields.wesl`). +//! +//! When no shelf has space, the least recently used shelf that is tall enough is cleared for the +//! new run. A run counts as used when it is rasterized and whenever labels in view show it. A shelf +//! used this frame is never cleared, so visible text is not evicted. When every shelf is in use, +//! the atlas is full for the rest of the frame and takes no more runs. It starts over only when +//! nothing in it was used this frame. Views track evictions to drop stale references. use std::collections::HashMap; use std::sync::Arc; -use crate::gpu::pipelines::Pipelines; +use crate::gpu::grown::Grown; +use crate::gpu::pipelines::{Pipelines, Variant}; use crate::gpu::shaders::{FIELD_AT, FIELD_COLUMNS, FIELD_COVERAGE, FIELD_HEIGHT, FIELD_ROW, FIELD_WIDTH, FIELD_WORDS}; use crate::gpu::shaders::{Shader, TEXT_BASE, TEXT_SPREAD}; use crate::gpu::work::{Count, Counters, Queue}; use crate::host::{Host, Raster, TextRun}; +/// The passes building distance fields: along columns, then along rows. +pub(crate) const FIELDS: [Variant; 2] = + [Variant::compute(Shader::Fields, "columns"), Variant::compute(Shader::Fields, "rows")]; + /// Texels across the atlas: room for about 1,400 labels of ten characters at the base size. const SIZE: u32 = 4096; /// Texels between runs, so filtering never reads a neighbor. const GAP: u32 = 1; /// Texels of field around a run's coverage. const PAD: u32 = TEXT_SPREAD as u32; -/// Runs whose fields one dispatch makes at most: one workgroup layer each. +/// The most runs one dispatch builds fields for, one workgroup layer each. const RUNS: usize = 65_535; -/// Where a run lies in the atlas. +/// Where `Atlas::hold` put a raster: at its glyphs on a shelf, nowhere because it has nothing to +/// draw or is malformed, or nowhere yet because the atlas is full this frame. +enum Held { + Placed(Glyphs, usize), + Nothing, + Full, +} + +/// A run's location in the atlas. #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) struct Glyphs { - /// Left, top, right, and bottom, texels, reaching `TEXT_SPREAD` past the text on every side. + /// Left, top, right, and bottom in texels, extending `TEXT_SPREAD` past the text on every side. pub(crate) rect: [u32; 4], /// Texels from the top of the rectangle to the baseline. pub(crate) baseline: f32, } impl Glyphs { - /// The rectangle as the two corner words shaders read: 16 bits of x, then 16 of y. + /// The rectangle as the two corner words shaders read: x in the low 16 bits, y in the high 16. pub(crate) fn corners(&self) -> [u32; 2] { let [l, t, r, b] = self.rect; [l | t << 16, r | b << 16] } } -/// A run table entry for `run` before it is set: the estimate flag, then the corner a rectangle of -/// its size by a rough measure would reach from the atlas's own corner, so its label can be placed -/// while only labels that show have their texts set. -pub(crate) fn estimate(run: &TextRun) -> [u32; 2] { - let size = run.size_px as f32; - let width = run.text.chars().count() as f32 * 0.58 * size; - let (width, lines) = match run.max_width.map(|w| w as f32) { - Some(wrap) if width > wrap => (wrap, (width / wrap).ceil()), - _ => (width, 1.0), - }; - // The host pads each run a little, and the field reaches past it. - let pad = 2.0 * (2.0 + PAD as f32); - let w = ((width + pad).ceil() as u32).min(0xffff); - let h = ((lines * 1.25 * size + pad).ceil() as u32).min(0xffff); - [crate::gpu::shaders::TEXT_ESTIMATE, w | h << 16] -} - -/// The run the atlas sets for `text` shown at `size`, wrapping at `max_width`, both length units: set -/// at the base size, wrapping where `max_width` falls at it. A sheet measures the same run it draws. +/// The run the atlas rasterizes for `text` shown at `size`, wrapping at `max_width`, both in length +/// units. The run is set at the base size, with `max_width` scaled to match. A sheet measures the +/// same run it draws. pub(crate) fn run(text: Arc, font: Arc, size: f32, max_width: Option) -> TextRun { let wrap = max_width.map(|w| (w * TEXT_BASE / size.max(1e-3)).round().max(1.0) as u32); TextRun { text, font, size_px: TEXT_BASE as u32, max_width: wrap } @@ -70,11 +70,12 @@ struct Shelf { top: u32, height: u32, used: u32, - /// When a run on it was last asked for. + /// The clock tick when a run on it was last used. tick: u64, } -/// A run set whose field waits to be made: its coverage, and the shelf and corner its field fills. +/// A rasterized run whose field is not built yet: its coverage, and the shelf and corner its field +/// goes to. struct Waiting { raster: Raster, shelf: usize, @@ -86,17 +87,21 @@ pub(crate) struct Atlas { texture: wgpu::Texture, pub(crate) view: wgpu::TextureView, shelves: Vec, - /// Every run asked for, and its shelf; `None` when the host could not set it. + /// Every run rasterized: its location and shelf, or `None` when it has nothing to draw or its + /// raster failed. Either way the host is not asked for it again. runs: HashMap>, - /// Runs set whose fields the next frame makes. + /// Rasterized runs whose fields the next frame builds. waiting: Vec, - /// What making fields works in, and its bind group. - work: Option<(wgpu::Buffer, wgpu::BindGroup)>, - /// Counts the frames runs are asked for in. + /// Scratch buffer for building fields, and its bind group once built. + work: Grown, + work_group: Option, + /// Frame counter for shelf use. clock: u64, - /// Changes whenever runs are evicted, so what pointed at them looks again. + /// Whether a run found no room this frame, so no more are placed until the next. + full: bool, + /// Changes whenever runs are evicted, so anything referencing them looks them up again. pub(crate) evictions: u64, - /// What counts the texts the host is asked to set. + /// Counts the texts the host is asked to rasterize. counters: Arc, } @@ -120,75 +125,99 @@ impl Atlas { shelves: Vec::new(), runs: HashMap::new(), waiting: Vec::new(), - work: None, + work: Grown::new(device, "latkit fields", wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC), + work_group: None, clock: 0, + full: false, evictions: 0, counters, } } - /// Starts a frame: runs asked for from now on count as used in it. + /// Starts a frame. Runs requested from now on count as used in it. pub(crate) fn tick(&mut self) { self.clock += 1; + self.full = false; } - /// Where `run` lies, set by `host` the first time it is asked for; its field is made before the - /// frame draws. + /// Whether a run found no room this frame, so the atlas takes no more until the next. + pub(crate) fn full(&self) -> bool { + self.full + } + + /// The location of `run` in the atlas, rasterized by `host` on first request. Its field is + /// built before the frame draws. `None` without a host, when the run has nothing to draw or its + /// raster failed (a failure is reported once), or while the atlas is full this frame. pub(crate) fn text(&mut self, host: Option<&dyn Host>, run: &TextRun) -> Option { if let Some(found) = self.runs.get(run) { let (glyphs, shelf) = (*found)?; self.shelves[shelf].tick = self.clock; return Some(glyphs); } - let host = host?; + let host = host.filter(|_| !self.full)?; self.counters.add(Count::Texts, 1); - let raster = match host.rasterize(run) { - Ok(raster) => raster, + let placed = match host.rasterize(run).map(|raster| self.hold(raster)) { + Ok(Held::Placed(glyphs, shelf)) => Some((glyphs, shelf)), + Ok(Held::Nothing) => None, + // No room this frame: the run is asked for again later. + Ok(Held::Full) => return None, Err(error) => { self.errors.push(error); - return None; + None } }; - let placed = self.hold(raster); - if placed.is_some() { - self.runs.insert(run.clone(), placed); - } + self.runs.insert(run.clone(), placed); placed.map(|(glyphs, _)| glyphs) } - /// Places `raster`'s field, which waits to be made: where it lies, and its shelf. - fn hold(&mut self, raster: Raster) -> Option<(Glyphs, usize)> { + /// Forgets the runs and failures the host made. A new host rasterizes texts again in its own + /// fonts. + pub(crate) fn rehost(&mut self) { + self.runs.clear(); + self.waiting.clear(); + self.shelves.clear(); + self.evictions += 1; + } + + /// Places `raster`'s field and queues it to be built. Returns where it was placed. + fn hold(&mut self, raster: Raster) -> Held { let [w, h] = raster.size; + if w == 0 || h == 0 { + return Held::Nothing; + } let (width, height) = (w.saturating_add(2 * PAD), h.saturating_add(2 * PAD)); - if w == 0 || h == 0 || width > SIZE || height > SIZE { + if width > SIZE || height > SIZE { self.errors.push(crate::Error::Limit("text raster does not fit the atlas".into())); - return None; + return Held::Nothing; } if raster.coverage.len() != w as usize * h as usize { self.errors.push(crate::Error::invalid("text raster coverage does not match its size")); - return None; + return Held::Nothing; } - let (shelf, [left, top]) = self.place(width, height); + let Some((shelf, [left, top])) = self.place(width, height) else { + self.full = true; + return Held::Full; + }; let glyphs = Glyphs { rect: [left, top, left + width, top + height], baseline: raster.baseline + PAD as f32 }; self.waiting.push(Waiting { raster, shelf, corner: [left, top] }); - Some((glyphs, shelf)) + Held::Placed(glyphs, shelf) } - /// Counts `run` as used now: labels in view show it. + /// Marks `run` as used now, because labels in view show it. pub(crate) fn touch(&mut self, run: &TextRun) { if let Some(Some((_, shelf))) = self.runs.get(run) { self.shelves[*shelf].tick = self.clock; } } - /// Whether `glyphs` still lie where they were for `run`. + /// Whether `run` is still at `glyphs`. pub(crate) fn holds(&self, run: &TextRun, glyphs: Glyphs) -> bool { matches!(self.runs.get(run), Some(Some((g, _))) if *g == glyphs) } - /// Makes the fields of the runs set since last time, recorded into `encoder` before what draws - /// them: their coverage goes up, each column and then each row takes its least squared distances - /// within the spread, and each field copies into its place. + /// Builds the fields of runs rasterized since the last call, recorded into `encoder` before the + /// draws that use them. Uploads their coverage, computes each column's and then each row's + /// least squared distances within the spread, and copies each field into its place. pub(crate) fn lay( &mut self, device: &wgpu::Device, @@ -202,8 +231,8 @@ impl Atlas { let rest = self.waiting.split_off(self.waiting.len().min(RUNS)); let laid = std::mem::replace(&mut self.waiting, rest); let size = |w: &Waiting| w.raster.size.map(|n| n + 2 * PAD); - // The runs and their coverage go up; their columns and fields follow. A field starts where a - // copy out of it may, and its rows are as far apart as copies need. + // Scratch layout: run entries, coverage, column distances, then fields. Each field starts + // at a valid copy offset, and its rows use the copy row pitch. let mut runs = vec![[0_u32; FIELD_WORDS as usize]; laid.len()]; let mut coverage = Vec::new(); let mut at = laid.len() as u32 * FIELD_WORDS; @@ -239,9 +268,10 @@ impl Atlas { pass.set_bind_group(0, &pipelines.empty, &[]); pass.set_bind_group(1, &pipelines.empty, &[]); pass.set_bind_group(2, &group, &[]); - pass.set_pipeline(&pipelines.compute(device, Shader::Fields, "columns")); + let [columns, rows] = FIELDS; + pass.set_pipeline(&pipelines.compute(device, columns)); pass.dispatch_workgroups(most[0].div_ceil(16), most[1].div_ceil(16), laid.len() as u32); - pass.set_pipeline(&pipelines.compute(device, Shader::Fields, "rows")); + pass.set_pipeline(&pipelines.compute(device, rows)); pass.dispatch_workgroups(most[0].div_ceil(4).div_ceil(16), most[1].div_ceil(16), laid.len() as u32); } for (w, run) in laid.iter().zip(&runs) { @@ -266,33 +296,26 @@ impl Atlas { } } - /// The buffer making fields works in, holding at least `bytes`, and its bind group. + /// The scratch buffer for building fields, at least `bytes` long, and its bind group. fn fit_work( &mut self, device: &wgpu::Device, pipelines: &mut Pipelines, bytes: u64, ) -> (wgpu::Buffer, wgpu::BindGroup) { - if let Some((buffer, group)) = self.work.as_ref().filter(|(buffer, _)| buffer.size() >= bytes) { - return (buffer.clone(), group.clone()); + if self.work.ensure(device, bytes) || self.work_group.is_none() { + self.work_group = Some(device.create_bind_group(&wgpu::BindGroupDescriptor { + label: Some("latkit fields"), + layout: &pipelines.pass_layout(device, Shader::Fields), + entries: &[wgpu::BindGroupEntry { binding: 0, resource: self.work.buffer.as_entire_binding() }], + })); } - let buffer = device.create_buffer(&wgpu::BufferDescriptor { - label: Some("latkit fields"), - size: (bytes + bytes / 2).next_multiple_of(256), - usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC | wgpu::BufferUsages::COPY_DST, - mapped_at_creation: false, - }); - let group = device.create_bind_group(&wgpu::BindGroupDescriptor { - label: Some("latkit fields"), - layout: &pipelines.pass_layout(device, Shader::Fields), - entries: &[wgpu::BindGroupEntry { binding: 0, resource: buffer.as_entire_binding() }], - }); - self.work = Some((buffer.clone(), group.clone())); - (buffer, group) + (self.work.buffer.clone(), self.work_group.clone().expect("built above")) } - /// The shelf and top left of room for `width` by `height` texels. - fn place(&mut self, width: u32, height: u32) -> (usize, [u32; 2]) { + /// Finds space for `width` by `height` texels and returns the shelf and top-left corner. + /// Returns `None` when every shelf that could hold it was used this frame. + fn place(&mut self, width: u32, height: u32) -> Option<(usize, [u32; 2])> { let fits = |s: &Shelf| s.height >= height && s.height <= height + height / 2 + 4; let at = self.shelves.iter().position(|s| fits(s) && s.used + width <= SIZE); let at = at.or_else(|| { @@ -303,43 +326,47 @@ impl Atlas { }) }); let at = at.or_else(|| { - // Clear the shelf of this height used least recently, if not in this frame. + // Clear the least recently used shelf that is tall enough, preferring a fitting height. + // Shelves used this frame are skipped. let clock = self.clock; let oldest = (0..self.shelves.len()) - .filter(|&i| fits(&self.shelves[i]) && self.shelves[i].tick < clock) - .min_by_key(|&i| self.shelves[i].tick)?; + .filter(|&i| self.shelves[i].height >= height && self.shelves[i].tick < clock) + .min_by_key(|&i| (!fits(&self.shelves[i]), self.shelves[i].tick))?; self.runs.retain(|_, placed| !matches!(placed, Some((_, shelf)) if *shelf == oldest)); self.waiting.retain(|w| w.shelf != oldest); self.shelves[oldest].used = 0; self.evictions += 1; Some(oldest) }); - let at = at.unwrap_or_else(|| { - // Nothing fits: start over. + let at = at.or_else(|| { + // No shelf is tall enough. Start over, unless text in view uses the atlas this frame. + if self.shelves.iter().any(|s| s.tick == self.clock) { + return None; + } self.runs.clear(); self.waiting.clear(); self.shelves.clear(); self.evictions += 1; self.shelves.push(Shelf { top: 0, height, used: 0, tick: self.clock }); - 0 - }); + Some(0) + })?; let shelf = &mut self.shelves[at]; let corner = [shelf.used, shelf.top]; shelf.used += width + GAP; shelf.tick = self.clock; - (at, corner) + Some((at, corner)) } } -/// Bytes a copy out of a buffer into a texture may start at a multiple of, on every backend. +/// Alignment in bytes of a buffer-to-texture copy's start offset, valid on every backend. const COPY_ALIGN: u32 = 512; #[cfg(test)] mod tests { use super::*; - /// The field of `width` by `height` coverage bytes as it is defined, texel by texel: the least - /// squared distance to a texel inside and outside over every texel within reach. + /// The exact field of `width` by `height` coverage bytes, computed texel by texel: the least + /// squared distance to inside and to outside over every texel within `PAD` texels. fn exact(coverage: &[u8], width: usize, height: usize) -> Vec { let pad = i64::from(PAD); let (w, h) = (width as i64 + 2 * pad, height as i64 + 2 * pad); @@ -378,7 +405,7 @@ mod tests { } #[test] - fn fields_made_on_the_gpu_are_the_exact_field() { + fn gpu_distance_fields_match_the_exact_field() { let gpu = match pollster::block_on(crate::Gpu::new()) { Ok(gpu) => gpu, Err(error) if std::env::var_os("LATKIT_REQUIRE_GPU").is_some() => panic!("GPU required: {error}"), @@ -399,14 +426,15 @@ mod tests { coverage[5 * width + 37] = 40; let mut inner = gpu.0.lock(); let raster = Raster { size: [width as u32, height as u32], coverage: coverage.clone(), baseline: 20.0 }; - let (glyphs, _) = inner.atlas.hold(raster).expect("placed"); + let Held::Placed(glyphs, _) = inner.atlas.hold(raster) else { panic!("placed") }; let (device, queue) = (&gpu.0.device, &gpu.0.queue); let mut encoder = device.create_command_encoder(&wgpu::CommandEncoderDescriptor::default()); let crate::gpu::Inner { atlas, pipelines, .. } = &mut *inner; atlas.lay(device, queue, &mut encoder, pipelines); // Another view can lay a smaller batch into the same scratch buffer before submission. // Its upload must not replace the first batch's inputs before that batch has executed. - atlas.hold(Raster { size: [9, 7], coverage: vec![255; 63], baseline: 5.0 }).expect("second batch"); + let held = atlas.hold(Raster { size: [9, 7], coverage: vec![255; 63], baseline: 5.0 }); + assert!(matches!(held, Held::Placed(..)), "a second batch"); atlas.lay(device, queue, &mut encoder, pipelines); let [l, t, r, b] = glyphs.rect; let (w, h) = (r - l, b - t); @@ -441,6 +469,43 @@ mod tests { let off = made.iter().zip(&want).filter(|(a, b)| a.abs_diff(**b) > 1).count(); assert_eq!(off, 0, "every texel within a step of the exact field"); let deepest = made.iter().max().copied().unwrap_or(0); - assert!(made.contains(&0) && deepest > 224, "the field runs from nothing outside to deep inside: {deepest}"); + assert!(made.contains(&0) && deepest > 224, "the field runs from 0 outside to over 224 inside: {deepest}"); + } + + #[test] + fn each_run_is_rasterized_once_however_it_turns_out() { + /// A host that counts its rasters: none for "", a failure for "bad", else a square. + struct Counting(std::sync::atomic::AtomicU32); + impl Host for Counting { + fn rasterize(&self, run: &TextRun) -> crate::Result { + self.0.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + match &*run.text { + "" => Ok(Raster { size: [0, 0], coverage: Vec::new(), baseline: 0.0 }), + "bad" => Err(crate::Error::Unsupported("no font".into())), + _ => Ok(Raster { size: [4, 4], coverage: vec![255; 16], baseline: 3.0 }), + } + } + } + let gpu = match pollster::block_on(crate::Gpu::new()) { + Ok(gpu) => gpu, + Err(error) if std::env::var_os("LATKIT_REQUIRE_GPU").is_some() => panic!("GPU required: {error}"), + Err(_) => return, + }; + let counting = Counting(std::sync::atomic::AtomicU32::new(0)); + let rasters = || counting.0.load(std::sync::atomic::Ordering::Relaxed); + let host: Option<&dyn Host> = Some(&counting); + let text = |text: &str| run(text.into(), "sans-serif".into(), 12.0, None); + let mut inner = gpu.0.lock(); + let atlas = &mut inner.atlas; + for _ in 0..2 { + assert!(atlas.text(host, &text("")).is_none(), "an empty raster draws nothing"); + assert!(atlas.text(host, &text("bad")).is_none(), "a failed raster draws nothing"); + assert!(atlas.text(host, &text("ok")).is_some(), "a raster is placed"); + } + assert_eq!(rasters(), 3, "each run is rasterized once"); + assert_eq!(atlas.errors.len(), 1, "only the failure is reported, once: {:?}", atlas.errors); + atlas.rehost(); + assert!(atlas.text(host, &text("ok")).is_some()); + assert_eq!(rasters(), 4, "a new host rasterizes the run again"); } } diff --git a/crates/latkit/src/gpu/colormaps.rs b/crates/latkit/src/gpu/colormaps.rs index cd78b52a..26bb89be 100644 --- a/crates/latkit/src/gpu/colormaps.rs +++ b/crates/latkit/src/gpu/colormaps.rs @@ -1,84 +1,19 @@ //! Colormaps on the GPU: one 256-texel layer per colormap a program uses, in one texture array. -use std::collections::HashMap; - +use super::layers::Layers; +use crate::Result; use crate::expr::{Colormap, ColormapKey}; use crate::gpu::work::Queue; -use crate::{Error, Result}; - -const INITIAL_LAYERS: u32 = 16; -pub(crate) struct Colormaps { - texture: wgpu::Texture, - pub(crate) view: wgpu::TextureView, - layers: HashMap, - capacity: u32, - /// Changes whenever the texture is replaced, so views rebind it. - pub(crate) generation: u64, -} - -fn create(device: &wgpu::Device, layers: u32) -> (wgpu::Texture, wgpu::TextureView) { - let texture = device.create_texture(&wgpu::TextureDescriptor { - label: Some("latkit colormaps"), - size: wgpu::Extent3d { width: 256, height: 1, depth_or_array_layers: layers }, - mip_level_count: 1, - sample_count: 1, - dimension: wgpu::TextureDimension::D2, - format: wgpu::TextureFormat::Rgba8Unorm, - usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST | wgpu::TextureUsages::COPY_SRC, - view_formats: &[], - }); - let view = texture.create_view(&wgpu::TextureViewDescriptor { - dimension: Some(wgpu::TextureViewDimension::D2Array), - ..Default::default() - }); - (texture, view) -} +pub(crate) type Colormaps = Layers; impl Colormaps { - pub(crate) fn new(device: &wgpu::Device) -> Colormaps { - let (texture, view) = create(device, INITIAL_LAYERS); - Colormaps { texture, view, layers: HashMap::new(), capacity: INITIAL_LAYERS, generation: 0 } + pub(crate) fn colormaps(device: &wgpu::Device) -> Colormaps { + Layers::new(device, "colormaps", [256, 1], 16) } /// The layer holding `colormap`, uploading it the first time. - pub(crate) fn layer(&mut self, device: &wgpu::Device, queue: &Queue, colormap: &Colormap) -> Result { - let key = colormap.key(); - if let Some(&layer) = self.layers.get(&key) { - return Ok(layer); - } - let layer = self.layers.len() as u32; - if layer >= self.capacity { - let capacity = (self.capacity * 2).min(device.limits().max_texture_array_layers); - if layer >= capacity { - return Err(Error::Limit(format!("more than {capacity} distinct colormaps").into())); - } - let (texture, view) = create(device, capacity); - let mut encoder = - device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("latkit colormaps") }); - encoder.copy_texture_to_texture( - self.texture.as_image_copy(), - texture.as_image_copy(), - wgpu::Extent3d { width: 256, height: 1, depth_or_array_layers: self.capacity }, - ); - queue.submit([encoder.finish()]); - self.texture = texture; - self.view = view; - self.capacity = capacity; - self.generation += 1; - } - queue.write_texture( - wgpu::TexelCopyTextureInfo { - texture: &self.texture, - mip_level: 0, - origin: wgpu::Origin3d { x: 0, y: 0, z: layer }, - aspect: wgpu::TextureAspect::All, - }, - bytemuck::cast_slice(colormap.texels()), - wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(256 * 4), rows_per_image: Some(1) }, - wgpu::Extent3d { width: 256, height: 1, depth_or_array_layers: 1 }, - ); - self.layers.insert(key, layer); - Ok(layer) + pub(crate) fn colormap(&mut self, device: &wgpu::Device, queue: &Queue, colormap: &Colormap) -> Result { + self.layer(device, queue, colormap.key(), || bytemuck::cast_slice(colormap.texels()).to_vec()) } } diff --git a/crates/latkit/src/gpu/dictionary.rs b/crates/latkit/src/gpu/dictionary.rs index b452fac1..66886cb8 100644 --- a/crates/latkit/src/gpu/dictionary.rs +++ b/crates/latkit/src/gpu/dictionary.rs @@ -1,6 +1,6 @@ -//! Text on the GPU: each text field's distinct texts get codes in the order they are first seen. -//! Codes only ever grow, so a page's codes never change and upload once. A page is coded once; one -//! written over a page coded takes its codes and codes only the rows written. +//! Text dictionaries. Each text field's distinct texts get codes in the order they are first seen. +//! Codes are never reassigned, so a page's codes never change and upload once. A page is coded +//! once. A page written over a coded page reuses that page's codes and codes only the rows written. use std::collections::HashMap; use std::sync::Arc; @@ -8,10 +8,10 @@ use std::sync::Arc; use rustc_hash::FxBuildHasher; use crate::Name; -use crate::data::{Keyed, Page, PageId, Values}; +use crate::data::{Column, Keyed, Page, PageId, Values}; use crate::gpu::shaders::MISSING; -/// Pages a dictionary keeps the codes of: past them, it forgets them all. +/// Most pages a dictionary keeps codes for. Past this, it forgets all of them. const PAGES: usize = 256; #[derive(Default)] @@ -24,7 +24,7 @@ pub(crate) struct Dictionary { codes: HashMap, u32, FxBuildHasher>, /// Each code's text. texts: Vec>, - /// The codes of pages coded, by page. + /// Codes of coded pages, by page. pages: HashMap, FxBuildHasher>, } @@ -36,14 +36,15 @@ impl Dictionaries { } impl Dictionary { - /// The codes of `page`, its texts added: `f32` words, `MISSING` where missing. A page coded - /// before costs nothing; one written over a page coded costs the rows written. + /// The codes of `page` as `f32` words, `MISSING` where missing, adding its new texts. A page + /// coded before needs no work. A page written over a coded page codes only the rows written. pub(crate) fn codes(&mut self, page: &Page) -> Arc<[u32]> { if let Some(codes) = self.pages.get(&page.id) { return codes.clone(); } let len = page.values.len(); - // The nearest page it was written over whose codes are kept, and the rows written since. + // Find the nearest page this one was written over that still has codes, and the rows + // written since. let mut reused = None; let mut at = page.written.as_deref(); while let Some(written) = at { @@ -55,7 +56,7 @@ impl Dictionary { } let codes: Arc<[u32]> = match reused { Some((from, before, rows)) => { - // The page written over is usually written over no more. + // The old page is rarely written over again, so drop its codes. self.pages.remove(&from); let mut codes = before.to_vec(); for row in rows { @@ -72,7 +73,13 @@ impl Dictionary { codes } - /// The code of value `i` of `values` as an `f32` word, its text added when it is new. + /// The codes of `column`'s rows `rows`, in that order, adding new texts. This fills a packed + /// page of a paged table. Each row costs one lookup, however its pages were coded. + pub(crate) fn rows(&mut self, column: &Column, rows: &[u32]) -> Vec { + rows.iter().map(|&r| column.at(r as usize).map_or(MISSING, |(values, i)| self.code(values, i))).collect() + } + + /// The code of value `i` of `values` as an `f32` word, adding its text if new. fn code(&mut self, values: &Values, i: usize) -> u32 { let Some(text) = values.text().filter(|_| values.is_valid(i)) else { return MISSING }; let t = text.get(i); @@ -88,7 +95,7 @@ impl Dictionary { (code as f32).to_bits() } - /// Codes given so far. + /// Number of codes assigned so far. pub(crate) fn len(&self) -> usize { self.texts.len() } @@ -98,7 +105,8 @@ impl Dictionary { self.texts.get(code as usize) } - /// The code of `text` as an `f32` word; one no text has when it was never seen. + /// The code of `text` as an `f32` word. Returns NaN, which matches no text, if `text` was never + /// seen. pub(crate) fn code_of(&self, text: &str) -> u32 { self.codes.get(text).map_or(f32::NAN.to_bits(), |&code| (code as f32).to_bits()) } @@ -110,7 +118,7 @@ mod tests { use crate::data::Column; #[test] - fn a_page_written_over_one_coded_codes_only_its_rows() { + fn rewriting_a_coded_page_codes_only_the_written_rows() { let column = Column::new(&Values::from(vec!["a", "b", "c"])); let mut dictionary = Dictionary::default(); let first = dictionary.codes(&column.pages()[0]); diff --git a/crates/latkit/src/gpu/grown.rs b/crates/latkit/src/gpu/grown.rs new file mode 100644 index 00000000..56ec14c8 --- /dev/null +++ b/crates/latkit/src/gpu/grown.rs @@ -0,0 +1,35 @@ +//! Buffers that grow. A buffer that runs out of space is replaced by one 1.5 times the size needed, +//! so a buffer that keeps growing is replaced only a few times. + +/// A buffer that is replaced by one 1.5 times the size needed when it runs out of space. +pub(crate) struct Grown { + pub(crate) buffer: wgpu::Buffer, + label: &'static str, + usage: wgpu::BufferUsages, +} + +impl Grown { + pub(crate) fn new(device: &wgpu::Device, label: &'static str, usage: wgpu::BufferUsages) -> Grown { + Grown { buffer: buffer(device, label, usage, 256), label, usage } + } + + /// Makes room for `bytes`. Returns true when it replaced the buffer, losing its contents. + pub(crate) fn ensure(&mut self, device: &wgpu::Device, bytes: u64) -> bool { + if bytes <= self.buffer.size() { + return false; + } + self.buffer = buffer(device, self.label, self.usage, (bytes + bytes / 2).next_multiple_of(256)); + true + } +} + +/// Creates a buffer for `usage` that writes can copy into. Its size is `bytes`, rounded up to at +/// least 256 and to whole words. +pub(crate) fn buffer(device: &wgpu::Device, label: &str, usage: wgpu::BufferUsages, bytes: u64) -> wgpu::Buffer { + device.create_buffer(&wgpu::BufferDescriptor { + label: Some(label), + size: bytes.max(256).next_multiple_of(4), + usage: usage | wgpu::BufferUsages::COPY_DST, + mapped_at_creation: false, + }) +} diff --git a/crates/latkit/src/gpu/icons.rs b/crates/latkit/src/gpu/icons.rs index b6b907c3..0c594b44 100644 --- a/crates/latkit/src/gpu/icons.rs +++ b/crates/latkit/src/gpu/icons.rs @@ -1,99 +1,48 @@ -//! Icons on the GPU: every image an icon marker shows, as a layer of one texture array of a common -//! size, scaled to fit. Markers name their images; the same image shares its layer. +//! Icons on the GPU. Each image an icon marker shows is scaled to a common size and stored as one +//! layer of a texture array. Markers showing the same image share its layer. -use std::collections::HashMap; +use std::hash::{Hash, Hasher}; use std::sync::Arc; +use super::layers::Layers; +use crate::Result; use crate::gpu::work::Queue; /// Texels across an icon layer. const SIZE: u32 = 64; -pub(crate) struct Icons { - texture: wgpu::Texture, - pub(crate) view: wgpu::TextureView, - /// Each image's layer, by the address of its pixels. - layers: HashMap, u32)>, - capacity: u32, - /// Changes whenever the texture is replaced, so views rebind it. - pub(crate) generation: u64, +/// An image's pixels, compared by address so that markers showing one image share its layer. +/// Holding the pixels keeps the address from being reused. +pub(crate) struct Pixels(Arc<[u8]>); + +impl PartialEq for Pixels { + fn eq(&self, other: &Pixels) -> bool { + Arc::ptr_eq(&self.0, &other.0) + } } -fn create(device: &wgpu::Device, layers: u32) -> (wgpu::Texture, wgpu::TextureView) { - let texture = device.create_texture(&wgpu::TextureDescriptor { - label: Some("latkit icons"), - size: wgpu::Extent3d { width: SIZE, height: SIZE, depth_or_array_layers: layers }, - mip_level_count: 1, - sample_count: 1, - dimension: wgpu::TextureDimension::D2, - format: wgpu::TextureFormat::Rgba8Unorm, - usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST | wgpu::TextureUsages::COPY_SRC, - view_formats: &[], - }); - let view = texture.create_view(&wgpu::TextureViewDescriptor { - dimension: Some(wgpu::TextureViewDimension::D2Array), - ..Default::default() - }); - (texture, view) +impl Eq for Pixels {} + +impl Hash for Pixels { + fn hash(&self, hasher: &mut H) { + Arc::as_ptr(&self.0).cast::().hash(hasher); + } } +pub(crate) type Icons = Layers; + impl Icons { - pub(crate) fn new(device: &wgpu::Device) -> Icons { - let (texture, view) = create(device, 1); - Icons { texture, view, layers: HashMap::new(), capacity: 1, generation: 0 } + pub(crate) fn icons(device: &wgpu::Device) -> Icons { + Layers::new(device, "icons", [SIZE, SIZE], 1) } /// The layer of each of `images`, uploading the ones not yet on the GPU. - pub(crate) fn layers(&mut self, device: &wgpu::Device, queue: &Queue, images: &[crate::Image]) -> Vec { - let needed = self.layers.len() as u32 + images.len() as u32; - if needed > self.capacity { - let capacity = needed.next_power_of_two().min(device.limits().max_texture_array_layers); - let (texture, view) = create(device, capacity); - let mut encoder = - device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("latkit icons") }); - encoder.copy_texture_to_texture( - self.texture.as_image_copy(), - texture.as_image_copy(), - wgpu::Extent3d { width: SIZE, height: SIZE, depth_or_array_layers: self.capacity }, - ); - queue.submit([encoder.finish()]); - self.texture = texture; - self.view = view; - self.capacity = capacity; - self.generation += 1; - } - images - .iter() - .map(|image| { - let key = image.rgba.as_ptr() as usize; - if let Some((_, layer)) = self.layers.get(&key) { - return *layer; - } - let layer = (self.layers.len() as u32).min(self.capacity - 1); - let texels = fit(image); - queue.write_texture( - wgpu::TexelCopyTextureInfo { - texture: &self.texture, - mip_level: 0, - origin: wgpu::Origin3d { x: 0, y: 0, z: layer }, - aspect: wgpu::TextureAspect::All, - }, - &texels, - wgpu::TexelCopyBufferLayout { - offset: 0, - bytes_per_row: Some(SIZE * 4), - rows_per_image: Some(SIZE), - }, - wgpu::Extent3d { width: SIZE, height: SIZE, depth_or_array_layers: 1 }, - ); - self.layers.insert(key, (image.rgba.clone(), layer)); - layer - }) - .collect() + pub(crate) fn of(&mut self, device: &wgpu::Device, queue: &Queue, images: &[crate::Image]) -> Result> { + images.iter().map(|image| self.layer(device, queue, Pixels(image.rgba.clone()), || fit(image))).collect() } } -/// `image` scaled to `SIZE` square by its nearest texels, centered with its aspect kept. +/// Scales `image` into a `SIZE` square by nearest texel, centered with its aspect ratio kept. fn fit(image: &crate::Image) -> Vec { let [w, h] = image.size.map(|v| v.max(1)); let scale = f64::from(w.max(h)) / f64::from(SIZE); diff --git a/crates/latkit/src/gpu/layers.rs b/crates/latkit/src/gpu/layers.rs new file mode 100644 index 00000000..9559419f --- /dev/null +++ b/crates/latkit/src/gpu/layers.rs @@ -0,0 +1,135 @@ +//! Texture arrays holding one layer per distinct image, such as each colormap or icon a program +//! uses. An array grows by copying itself into one twice as deep, up to the device's layer limit. + +use std::collections::HashMap; +use std::hash::Hash; + +use crate::gpu::work::Queue; +use crate::{Error, Result}; + +pub(crate) struct Layers { + /// What the layers hold, for labels and errors, such as "colormaps". + what: &'static str, + /// Texels across and down each layer. + size: [u32; 2], + texture: wgpu::Texture, + pub(crate) view: wgpu::TextureView, + /// The layer holding each key. + held: HashMap, + capacity: u32, + /// Changes whenever the texture is replaced, so views rebind it. + pub(crate) generation: u64, +} + +impl Layers { + /// An array of `capacity` layers of `size` texels, holding `what`. + pub(crate) fn new(device: &wgpu::Device, what: &'static str, size: [u32; 2], capacity: u32) -> Layers { + let (texture, view) = create(device, what, size, capacity); + Layers { what, size, texture, view, held: HashMap::new(), capacity, generation: 0 } + } + + /// The layer holding `key`, uploading `texels()` (RGBA, row after row) the first time. Errors + /// once the array reaches the device's limit on layers. + pub(crate) fn layer( + &mut self, + device: &wgpu::Device, + queue: &Queue, + key: K, + texels: impl FnOnce() -> Vec, + ) -> Result { + if let Some(&layer) = self.held.get(&key) { + return Ok(layer); + } + let layer = self.held.len() as u32; + if layer >= self.capacity { + self.grow(device, queue)?; + } + let [width, height] = self.size; + queue.write_texture( + wgpu::TexelCopyTextureInfo { + texture: &self.texture, + mip_level: 0, + origin: wgpu::Origin3d { x: 0, y: 0, z: layer }, + aspect: wgpu::TextureAspect::All, + }, + &texels(), + wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(width * 4), rows_per_image: Some(height) }, + wgpu::Extent3d { width, height, depth_or_array_layers: 1 }, + ); + self.held.insert(key, layer); + Ok(layer) + } + + /// Doubles the array's layers, within the device's limit, copying the layers it holds. + fn grow(&mut self, device: &wgpu::Device, queue: &Queue) -> Result<()> { + let capacity = (self.capacity * 2).min(device.limits().max_texture_array_layers); + if capacity <= self.capacity { + return Err(Error::Limit(format!("more than {capacity} distinct {}", self.what).into())); + } + let (texture, view) = create(device, self.what, self.size, capacity); + let label = format!("latkit {}", self.what); + let mut encoder = device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some(&label) }); + let [width, height] = self.size; + encoder.copy_texture_to_texture( + self.texture.as_image_copy(), + texture.as_image_copy(), + wgpu::Extent3d { width, height, depth_or_array_layers: self.capacity }, + ); + queue.submit([encoder.finish()]); + (self.texture, self.view, self.capacity) = (texture, view, capacity); + self.generation += 1; + Ok(()) + } +} + +fn create( + device: &wgpu::Device, + what: &str, + [width, height]: [u32; 2], + layers: u32, +) -> (wgpu::Texture, wgpu::TextureView) { + let texture = device.create_texture(&wgpu::TextureDescriptor { + label: Some(&format!("latkit {what}")), + size: wgpu::Extent3d { width, height, depth_or_array_layers: layers }, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format: wgpu::TextureFormat::Rgba8Unorm, + usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST | wgpu::TextureUsages::COPY_SRC, + view_formats: &[], + }); + let view = texture.create_view(&wgpu::TextureViewDescriptor { + dimension: Some(wgpu::TextureViewDimension::D2Array), + ..Default::default() + }); + (texture, view) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn layers_grow_for_new_keys_and_stop_at_the_device_limit() { + let gpu = match pollster::block_on(crate::Gpu::new()) { + Ok(gpu) => gpu, + Err(error) if std::env::var_os("LATKIT_REQUIRE_GPU").is_some() => panic!("GPU required: {error}"), + Err(_) => return, + }; + let (device, queue) = (&gpu.0.device, &gpu.0.queue); + let texel = || vec![255; 4]; + let mut layers = Layers::::new(device, "tests", [1, 1], 1); + let first: Vec = (0..5).map(|key| layers.layer(device, queue, key, texel).expect("room")).collect(); + assert_eq!(first, [0, 1, 2, 3, 4], "each key gets its own layer"); + assert_eq!(layers.generation, 3, "grown from 1 to 2, 4, then 8 layers"); + assert_eq!(layers.layer(device, queue, 2, texel).expect("held"), 2, "a key keeps its layer"); + assert_eq!(layers.generation, 3, "and nothing grows for it"); + let max = device.limits().max_texture_array_layers; + let mut full = Layers::::new(device, "tests", [1, 1], max); + for key in 0..max { + full.layer(device, queue, key, texel).expect("within the limit"); + } + let over = full.layer(device, queue, max, texel); + assert!(matches!(over, Err(Error::Limit(_))), "past the limit, an error, not a reused layer"); + } +} diff --git a/crates/latkit/src/gpu/mod.rs b/crates/latkit/src/gpu/mod.rs index 5e40934c..66c67435 100644 --- a/crates/latkit/src/gpu/mod.rs +++ b/crates/latkit/src/gpu/mod.rs @@ -1,11 +1,14 @@ -//! The GPU: one device shared by every view, with the page arena, pipelines, colormaps, and text -//! dictionaries every view draws from. +//! The GPU device that every view shares, and the page arena, pipelines, colormaps, and text +//! dictionaries the views draw from. pub(crate) mod arena; pub(crate) mod atlas; pub(crate) mod colormaps; pub(crate) mod dictionary; +pub(crate) mod grown; pub(crate) mod icons; +mod layers; +pub(crate) mod packed; pub(crate) mod pipelines; pub(crate) mod readback; pub(crate) mod shaders; @@ -19,6 +22,7 @@ use atlas::Atlas; use colormaps::Colormaps; use dictionary::Dictionaries; use icons::Icons; +use packed::Packed; use pipelines::Pipelines; use readback::Readbacks; use work::{Counters, Queue, Work}; @@ -26,11 +30,11 @@ use work::{Counters, Queue, Work}; use crate::host::Host; use crate::{Error, Result}; -/// The format views draw canvases and shots in, so both share one set of pipelines: every WebGPU -/// canvas takes it. +/// The texture format of canvases and shots, shared so both use one set of pipelines. Every WebGPU +/// canvas supports it. pub(crate) const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Bgra8Unorm; -/// The GPU device that views draw with. Cheap to clone; views of one `Gpu` share uploaded data. +/// The GPU device that views draw with. Cheap to clone. Views of one `Gpu` share uploaded data. #[derive(Clone)] pub struct Gpu(pub(crate) Arc); @@ -52,69 +56,113 @@ impl DeviceEvents { self.errors .lock() .unwrap_or_else(PoisonError::into_inner) - .push(Error::Unsupported(format!("the GPU refused: {error}").into())); + .push(Error::Unsupported(format!("GPU error: {error}").into())); } } pub(crate) struct Shared { - /// The instance and adapter `Gpu::new` created the device from, for surfaces. + /// The instance and adapter `Gpu::new` created the device from, for creating surfaces. pub(crate) instance: Option, pub(crate) adapter: Option, pub(crate) device: wgpu::Device, - /// The queue, counting what is written through it. + /// The queue, which counts what is written through it. pub(crate) queue: Queue, - /// What the GPU did, counted: views count the rows they evaluate and the jobs they hand out - /// through it, and [`Gpu::work`] reads it. + /// Work counters. Views count the rows they evaluate and the jobs they start here, and + /// [`Gpu::work`] reads them. pub(crate) counters: Arc, pub(crate) sampler: wgpu::Sampler, pub(crate) lost: Arc, - /// Errors the device reported outside any scope, for views to tell. + /// Errors the device reported outside any error scope, for views to report. pub(crate) errors: Arc>>, + /// Multisample count of every frame drawn with this GPU. It is set once from the screen's pixel + /// ratio, so a later change (browser zoom, a window moved to another screen) never rebuilds a + /// scene's pipelines. + samples: std::sync::OnceLock, inner: Mutex, } +/// Multisample count at `pixel_ratio`: 1 at 1.75 and above, else 4. Shader-drawn marks antialias +/// themselves and only triangle edges need multisampling, so dense screens skip it. +fn samples_at(pixel_ratio: f32) -> u32 { + if pixel_ratio >= 1.75 { 1 } else { 4 } +} + pub(crate) struct Inner { pub(crate) arena: Arena, + /// Piece keys of recently drawn geometry pages. + pub(crate) pieces: arena::Pieces, pub(crate) pipelines: Pipelines, pub(crate) colormaps: Colormaps, pub(crate) dictionaries: Dictionaries, + /// Pages of paged tables, packed in draw order and shared by every view drawing them. + pub(crate) packed: Packed, + /// Fetched and decoded tiles of each tile source, shared by every view drawing it. + pub(crate) tiles: crate::tiles::Cache, pub(crate) atlas: Atlas, pub(crate) icons: Icons, pub(crate) readbacks: Readbacks, pub(crate) host: Option>, pub(crate) budget: Budget, - /// What each view plans within: its share of the budget. + /// Each view's share of the budget, which it plans within. pub(crate) ledger: Ledger, + /// What the current frame may still upload and compute. Views drawn together share it. + pub(crate) frame: Allowance, +} + +/// What one frame may still do across a GPU's views: bytes to upload, and microseconds of work done +/// on the frame rather than in jobs. A frame ends when a view that drew in it draws again. So views +/// drawn together (a page's canvases, a shot's panels) share one allowance, and a view drawn alone +/// gets a new one each time, even when it redraws the same instant. +#[derive(Default)] +pub(crate) struct Allowance { + drawn: Vec, + pub(crate) upload: u64, + pub(crate) work: u32, } -/// The share of the budget the views of a GPU plan what they draw within, between them; the rest holds -/// pages a new version uploads while the one it replaces is still drawn, and pages cached. +impl Inner { + /// Begins view `view`'s part of a frame. If no view has drawn yet, or `view` already drew in + /// the current frame, starts a new one: resets the allowance, frees the blocks the last frame + /// released, and grows the pool to cover its shortfall. + pub(crate) fn begin_frame(&mut self, view: u64, device: &wgpu::Device, queue: &Queue) { + if self.frame.drawn.is_empty() || self.frame.drawn.contains(&view) { + self.frame = + Allowance { drawn: Vec::new(), upload: self.budget.upload_per_frame, work: crate::derive::FRAME_WORK }; + self.arena.begin_frame(device, queue); + } + self.frame.drawn.push(view); + } +} + +/// Fraction of the budget that all of a GPU's views plan their drawing within. The rest holds +/// cached pages, and the pages of a new data version that upload while the old version still draws. const PLANNED: f64 = 0.5; -/// What each view of a GPU plans what it draws within: shares of one part of the budget, granted -/// max-min fair, so views together never plan past it however many there are. Pages several views -/// draw are on the GPU once, so they count once, split between those views. +/// Each view's share of the planned part of the budget. Shares are max-min fair, so all views +/// together never plan past it, however many there are. A page that several views draw is resident +/// once, so it counts once, split between those views. #[derive(Default)] pub(crate) struct Ledger { - /// What each view would draw whole, by view. + /// Each view's request for drawing everything, by view. asks: rustc_hash::FxHashMap, } -/// What a view would draw whole: bytes of its own, and the pages it reads that any view drawing the -/// same data shares. +/// A view's memory request for drawing everything: its own bytes, plus the pieces it shares with +/// any view drawing the same data. #[derive(Clone, Default, PartialEq, Debug)] pub(crate) struct Ask { - /// Bytes no other view holds: its marks' state, focus, labels, traces, and derived columns. + /// Bytes no other view holds: its marks' state, focus, traces, and derived columns. pub(crate) own: f64, - /// Each piece any view drawing it shares, by what it holds, and its bytes. + /// Each shared piece, keyed by content, with its bytes. pub(crate) shared: Vec<(u64, f64)>, } impl Ledger { - /// Records what view `view` would draw whole, `whole` bytes as it counts them, and grants it its - /// share of what `budget` plans, as a part of `whole`. Each view asks its own bytes and a split of - /// each piece it shares between the views drawing it; each gets all it asks while an even split - /// of what is left covers it, else that split, views asking less having what they ask first. + /// Records view `view`'s request `ask` and returns its grant from the planned part of `budget`, + /// scaled to `whole` (the view's own count of its bytes). Each view asks for its own bytes plus + /// an even split of each piece it shares. Grants are max-min fair: smaller asks are granted in + /// full first, and once an ask exceeds an even split of what is left, each remaining view gets + /// that split. pub(crate) fn grant(&mut self, view: u64, whole: f64, ask: Ask, budget: u64) -> f64 { self.asks.insert(view, ask); let mut sharing: rustc_hash::FxHashMap = rustc_hash::FxHashMap::default(); @@ -148,18 +196,21 @@ impl Ledger { #[derive(Clone, Copy, PartialEq, Eq, Debug, serde::Serialize, serde::Deserialize, schemars::JsonSchema)] #[serde(rename_all = "camelCase", deny_unknown_fields)] pub struct Budget { - /// Bytes the data every view of the `Gpu` draws may take on the GPU: its pages, and each view's - /// rows. A view draws a table too large for its share through the tiles the camera asks for, so - /// what it draws always fits; pages no view draws stay cached in the room left, the least - /// recently drawn going first. By default half the device's largest storage buffer. The pool of - /// pages grows as data needs it, and never shrinks: set the budget before data draws. + /// GPU memory in bytes for the data all views of the `Gpu` draw: its pages and each view's + /// per-row state. A table too large for a view's share draws as a paged table, with only the + /// pages the camera needs, so it always fits. Pages no view draws stay cached in the rest of + /// the budget, and the least recently drawn are evicted first. Defaults to half the device's + /// largest storage buffer. The page pool grows as data needs it and never shrinks, so set the + /// budget before drawing data. pub bytes: u64, - /// Bytes uploaded per frame at most; larger data streams in over frames. + /// The most bytes uploaded per frame. Larger data streams in over several frames. Defaults to + /// 32 MiB. pub upload_per_frame: u64, } impl Budget { - /// Half the largest storage buffer `device` binds, and 32 MiB uploaded per frame. + /// The default budget: half the largest storage buffer `device` binds, and 32 MiB uploaded per + /// frame. fn of(device: &wgpu::Device) -> Budget { let limits = device.limits(); let largest = limits.max_storage_buffer_binding_size.min(limits.max_buffer_size); @@ -169,6 +220,7 @@ impl Budget { impl Gpu { /// Creates a device on the best adapter: WebGPU in browsers, Vulkan, Metal, or DX12 natively. + /// Errors if no adapter or device is available. pub async fn new() -> Result { let instance = wgpu::Instance::new(wgpu::InstanceDescriptor { backends: wgpu::Backends::PRIMARY | wgpu::Backends::BROWSER_WEBGPU, @@ -188,7 +240,7 @@ impl Gpu { max_storage_buffers_per_shader_stage: supported.max_storage_buffers_per_shader_stage, ..wgpu::Limits::default() }; - // Timestamps, where the adapter writes them, measure what each frame costs the GPU. + // Timestamp queries, where the adapter supports them, measure each frame's GPU time. let features = adapter.features() & wgpu::Features::TIMESTAMP_QUERY; let (device, queue) = adapter .request_device(&wgpu::DeviceDescriptor { @@ -202,19 +254,33 @@ impl Gpu { Ok(Gpu::create(Some(instance), Some(adapter), device, queue)) } - /// Queues, once, what nearly every scene draws with into the page's canvases, so views opened - /// later find it compiled: in a browser, as its first view opens, compiling once that view's - /// frames have all they draw with. A GPU no view uses compiles nothing. + /// Queues, once, the pipelines nearly every scene uses to draw into the page's canvases, so + /// views opened later find them compiled. In a browser this runs when the first view opens, and + /// these pipelines compile after that view's own draw pipelines. A GPU no view uses compiles + /// nothing. #[cfg(all(feature = "web", target_arch = "wasm32"))] pub(crate) fn warm(&self) { let pixel_ratio = web_sys::window().map_or(1.0, |w| w.device_pixel_ratio() as f32); - let samples = crate::view::samples(pixel_ratio); - self.0.lock().pipelines.warm(&self.0.device, || crate::view::common(FORMAT, samples)); + let samples = self.samples_for(pixel_ratio); + let waker = crate::web::waker(self); + self.0.lock().pipelines.warm(&self.0.device, &waker, || crate::view::common(FORMAT, samples)); + } + + /// The multisample count of its frames, once set. + pub(crate) fn samples(&self) -> Option { + self.0.samples.get().copied() + } + + /// The multisample count of its frames. If no screen set it when the GPU was created, the first + /// frame's `pixel_ratio` sets it, and it stays fixed from then on. + pub(crate) fn samples_for(&self, pixel_ratio: f32) -> u32 { + *self.0.samples.get_or_init(|| samples_at(pixel_ratio)) } /// Draws with an application-owned device and queue without replacing their callbacks. /// Forward device loss and uncaptured errors through [`Gpu::device_events`]. Clone this - /// `Gpu` to share its caches and budget between views. + /// `Gpu` to share its caches and budget between views. Errors if the device does not meet + /// WebGPU's default limits. pub fn from_wgpu(device: wgpu::Device, queue: wgpu::Queue) -> Result { if !wgpu::Limits::default().check_limits(&device.limits()) { return Err(Error::Unsupported("the device does not meet WebGPU's default limits".into())); @@ -222,7 +288,7 @@ impl Gpu { Ok(Gpu::create(None, None, device, queue)) } - /// A lightweight sink for the application's existing device callbacks; it does not own the GPU. + /// A handle that the application's own device callbacks report to. It does not own the GPU. pub fn device_events(&self) -> DeviceEvents { DeviceEvents { lost: self.0.lost.clone(), errors: self.0.errors.clone() } } @@ -253,29 +319,50 @@ impl Gpu { let queue = Queue::new(queue, counters.clone()); #[cfg_attr( target_arch = "wasm32", - allow(clippy::arc_with_non_send_sync, reason = "WebGPU objects stay on the one thread a page has") + allow(clippy::arc_with_non_send_sync, reason = "WebGPU objects stay on the page's single thread") )] let shared = Arc::new_cyclic(|gpu| { let inner = Inner { arena: Arena::new(&device, budget.bytes), + pieces: arena::Pieces::default(), pipelines: Pipelines::new(&device, counters.clone(), gpu.clone()), - colormaps: Colormaps::new(&device), + colormaps: Colormaps::colormaps(&device), dictionaries: Dictionaries::default(), + packed: Packed::new(counters.clone()), + tiles: crate::tiles::Cache::default(), atlas: Atlas::new(&device, counters.clone()), - icons: Icons::new(&device), + icons: Icons::icons(&device), readbacks: Readbacks::counted(counters.clone()), host: None, budget, ledger: Ledger::default(), + frame: Allowance::default(), }; - Shared { instance, adapter, device, queue, counters, sampler, lost, errors, inner: Mutex::new(inner) } + Shared { + instance, + adapter, + device, + queue, + counters, + sampler, + lost, + errors, + samples: std::sync::OnceLock::new(), + inner: Mutex::new(inner), + } }); + // In a page, set the multisample count now from the screen's pixel ratio. Every view and + // canvas of the GPU draws for that screen. + #[cfg(all(feature = "web", target_arch = "wasm32"))] + if let Some(window) = web_sys::window() { + let _ = shared.samples.set(samples_at(window.device_pixel_ratio() as f32)); + } Gpu(shared) } - /// What this GPU did since it was made, counted exactly. It repeats wherever the budget, the - /// device's limits, and the frames drawn do, so tests and benchmarks hold it to what it was. - /// Subtract two for what happened between. + /// Exact counts of the work this GPU has done since it was created. The counts repeat for the + /// same budget, device limits, and frames drawn, so tests and benchmarks can compare them with + /// a baseline. Subtract two snapshots to get the work between them. pub fn work(&self) -> Work { self.0.counters.work() } @@ -290,7 +377,8 @@ impl Gpu { &self.0.queue } - /// The instance a `Gpu` from [`Gpu::new`] was created with, for creating surfaces it can draw to. + /// The instance a `Gpu` from [`Gpu::new`] was created with, for creating surfaces it can draw + /// to. pub fn instance(&self) -> Option<&wgpu::Instance> { self.0.instance.as_ref() } @@ -300,7 +388,7 @@ impl Gpu { self.0.adapter.as_ref() } - /// Sets how much GPU memory latkit uses; bytes past what the device binds are not used. + /// Sets how much GPU memory latkit uses. Bytes beyond what the device can bind are not used. pub fn set_budget(&self, budget: Budget) { let mut inner = self.0.lock(); inner.arena.set_limit(&self.0.device, budget.bytes); @@ -312,12 +400,25 @@ impl Gpu { self.0.lock().budget } - /// Gives views the platform's services: setting text, and doing jobs off the frame. + /// Bytes resident on the GPU for all its views: data pages and each view's per-row state. + pub fn resident(&self) -> u64 { + self.0.lock().arena.resident() + } + + /// Bytes waiting to upload, across all its views. + pub fn uploading(&self) -> u64 { + self.0.lock().arena.pending() + } + + /// Gives views the platform's services: text, background jobs, fetching, and video encoding. + /// Text the previous host rasterized is rasterized again by this one. pub fn set_host(&self, host: impl Host) { - self.0.lock().host = Some(Arc::new(host)); + let mut inner = self.0.lock(); + inner.host = Some(Arc::new(host)); + inner.atlas.rehost(); } - /// Whether the device was lost; drawing then does nothing, and a new `Gpu` is needed. + /// Whether the device was lost. Drawing then does nothing, and a new `Gpu` is needed. pub fn lost(&self) -> bool { self.0.lost.load(Ordering::Acquire) } @@ -328,7 +429,7 @@ impl Shared { self.inner.lock().unwrap_or_else(PoisonError::into_inner) } - /// The lock, unless something holds it now. + /// The lock, or `None` if it is held now. #[cfg(all(feature = "web", target_arch = "wasm32"))] pub(crate) fn try_lock(&self) -> Option> { match self.inner.try_lock() { @@ -357,21 +458,21 @@ mod tests { fn views_share_the_planned_budget_max_min_fair() { let mut ledger = Ledger::default(); let grant = |ledger: &mut Ledger, view, bytes| ledger.grant(view, bytes, own(bytes), 1000); - assert_eq!(grant(&mut ledger, 1, 100.0), 100.0, "a view asking less than half the budget has all it asks"); - assert_eq!(grant(&mut ledger, 2, 900.0), 400.0, "the next has what is left"); - assert_eq!(grant(&mut ledger, 3, 900.0), 200.0, "two large views split what the small one leaves"); + assert_eq!(grant(&mut ledger, 1, 100.0), 100.0, "a view asking for under half the budget gets all it asks"); + assert_eq!(grant(&mut ledger, 2, 900.0), 400.0, "the next gets the rest"); + assert_eq!(grant(&mut ledger, 3, 900.0), 200.0, "two large views split the rest of the budget"); ledger.leave(2); - assert_eq!(grant(&mut ledger, 3, 900.0), 400.0, "a view that leaves gives its share back"); + assert_eq!(grant(&mut ledger, 3, 900.0), 400.0, "a view that leaves returns its share"); } #[test] - fn pages_views_share_count_once() { + fn pages_shared_by_views_count_once() { let mut ledger = Ledger::default(); - // Eight views of one dataset: 300 bytes of shared pages each reads, and 20 of its own. + // Eight views of one dataset, each reading 300 shared bytes and holding 20 of its own. let ask = Ask { own: 20.0, shared: vec![(7, 300.0)] }; let shares: Vec = (0..8).map(|view| ledger.grant(view, 320.0, ask.clone(), 1000)).collect(); - assert_eq!(shares[7], 320.0, "300 + 8 × 20 fits the 500 planned: every view draws whole"); - // Two views of other data each, too large together: each has the same part of what it asks. + assert_eq!(shares[7], 320.0, "300 + 8 × 20 fits the 500 planned, so every view gets its full ask"); + // Two views of different data, too large together, each get the same fraction of their ask. let mut ledger = Ledger::default(); ledger.grant(1, 400.0, Ask { own: 0.0, shared: vec![(1, 400.0)] }, 1000); assert_eq!(ledger.grant(2, 400.0, Ask { own: 0.0, shared: vec![(2, 400.0)] }, 1000), 250.0); diff --git a/crates/latkit/src/gpu/packed.rs b/crates/latkit/src/gpu/packed.rs new file mode 100644 index 00000000..361e241a --- /dev/null +++ b/crates/latkit/src/gpu/packed.rs @@ -0,0 +1,283 @@ +//! Packed pages of paged tables. A paged table's column is packed into the words its marks read, +//! one page of its draw order at a time, the first time a view draws that page. Packed pages are +//! keyed by their source. So views that draw a table the same way share its pages, a page the +//! camera returns to is the same page, and a write repacks only the pages its rows draw in. A table +//! drawn in its own order reads its data's pages directly. The exception is a reference to a table +//! drawn in another order: its pages are packed through that order's ranks. + +use std::sync::Arc; + +use rustc_hash::FxHashMap; + +use crate::data::{Column, Name, PageId, Reference, Samples, Table, UNWIRED, Version}; +use crate::derive::order::Order; +use crate::expr::compile::Words; +use crate::gpu::arena::page_words; +use crate::gpu::shaders::PAGE_ROWS; +use crate::gpu::shaders::{MISSING, NONE}; +use crate::gpu::work::{Count, Counters}; + +/// Most packed columns kept. Past this, the least recently used half is forgotten. +const KEPT: usize = 4096; + +/// The key of one packed column: the order, the dataset (`origin`) and table, the field and how it +/// is read, and anything else its words depend on (`reads`: a reference's target order, the frame +/// of samples shown, or the rows a join reads). +#[derive(Clone, PartialEq, Eq, Hash, Debug)] +pub(crate) struct Packing { + pub(crate) order: u64, + pub(crate) origin: Version, + pub(crate) table: Name, + pub(crate) field: Name, + pub(crate) words: Words, + pub(crate) low: bool, + pub(crate) reads: u64, +} + +/// A column's packed pages: their source column, each page's id once packed, the samples gathered +/// for each traced page, and when they were last used. +struct Pages { + source: Option, + ids: Vec>, + samples: Vec>, + used: u64, +} + +/// Every packed column, shared by the GPU's views, and the counters for rows packed. +pub(crate) struct Packed { + counters: Arc, + columns: FxHashMap, + /// Reference pages of tables drawn in their own order, packed through the ranks of the target + /// table's draw order. Keyed by page and that order. + ranked: FxHashMap<(PageId, u64), (PageId, u64)>, + uses: u64, +} + +impl Packed { + pub(crate) fn new(counters: Arc) -> Packed { + Packed { counters, columns: FxHashMap::default(), ranked: FxHashMap::default(), uses: 0 } + } + + /// Counts `rows` rows packed. + pub(crate) fn count(&self, rows: usize) { + self.counters.add(Count::Packed, rows as u64); + } + + /// The counters for rows packed, for packing done later. + pub(crate) fn counters(&self) -> Arc { + self.counters.clone() + } + + /// The pages of `packing`, packed from the current `source`. A page is kept while none of its + /// rows was written since. `page_of` maps a row to its page. Without a source, pages are kept + /// as long as `packing` matches. + fn pages( + &mut self, + packing: &Packing, + source: Option<&Column>, + page_of: impl Fn(u32) -> Option, + ) -> &mut Pages { + self.uses += 1; + if !self.columns.contains_key(packing) && self.columns.len() >= KEPT { + forget(&mut self.columns, |p| p.used); + } + let pages = self.columns.entry(packing.clone()).or_insert_with(|| Pages { + source: source.cloned(), + ids: Vec::new(), + samples: Vec::new(), + used: 0, + }); + pages.used = self.uses; + if let Some(source) = source + && !pages.source.as_ref().is_some_and(|s| s.shares(source)) + { + if let Some(before) = &pages.source { + for page in source.written_since(before).into_iter().filter_map(|row| page_of(row as u32)) { + if let Some(id) = pages.ids.get_mut(page) { + *id = None; + } + if let Some(samples) = pages.samples.get_mut(page) { + *samples = None; + } + } + } else { + pages.ids.clear(); + pages.samples.clear(); + } + pages.source = Some(source.clone()); + } + pages + } + + /// The id of page `page` of `packing`, packed from `source` in `order`. + pub(crate) fn page(&mut self, packing: &Packing, order: &Order, source: Option<&Column>, page: u32) -> PageId { + self.pages_of(packing, order, source, &[page])[0] + } + + /// The ids of pages `wanted` of `packing`, packed from `source` in `order`, looking the column + /// up once. + pub(crate) fn pages_of( + &mut self, + packing: &Packing, + order: &Order, + source: Option<&Column>, + wanted: &[u32], + ) -> Vec { + let pages = self.pages(packing, source, |row| Some((order.rank(row) / PAGE_ROWS) as usize)); + wanted + .iter() + .map(|&k| { + let k = k as usize; + if pages.ids.len() <= k { + pages.ids.resize(k + 1, None); + } + *pages.ids[k].get_or_insert_with(PageId::next) + }) + .collect() + } + + /// The id of `packing`'s page of kept rows, packed from `source`. The page is kept while none + /// of its rows (those `holds` accepts) was written since. + pub(crate) fn kept(&mut self, packing: &Packing, source: Option<&Column>, holds: impl Fn(u32) -> bool) -> PageId { + let pages = self.pages(packing, source, |row| holds(row).then_some(0)); + if pages.ids.is_empty() { + pages.ids.push(None); + } + *pages.ids[0].get_or_insert_with(PageId::next) + } + + /// The samples of `samples`' rows `rows`, for page `page` of `packing`, gathered once. + pub(crate) fn samples(&mut self, packing: &Packing, page: u32, rows: &[u32], samples: &Samples) -> Samples { + let counters = self.counters.clone(); + let pages = self.pages(packing, None, |_| None); + let k = page as usize; + if pages.samples.len() <= k { + pages.samples.resize(k + 1, None); + } + pages.samples[k] + .get_or_insert_with(|| { + let rows: Vec = rows.iter().map(|&r| r as usize).collect(); + counters.add(Count::Packed, rows.len() as u64); + samples.gather(&rows) + }) + .clone() + } + + /// The id of reference page `page` of a table drawn in its own order, packed through the ranks + /// of `order`, the target table's draw order. + pub(crate) fn ranked(&mut self, page: PageId, order: u64) -> PageId { + self.uses += 1; + if self.ranked.len() >= KEPT && !self.ranked.contains_key(&(page, order)) { + forget(&mut self.ranked, |&(_, used)| used); + } + let uses = self.uses; + let entry = self.ranked.entry((page, order)).or_insert_with(|| (PageId::next(), 0)); + entry.1 = uses; + entry.0 + } +} + +/// Drops the least recently used half of `kept`, by the use time `used` returns. +fn forget(kept: &mut FxHashMap, used: impl Fn(&V) -> u64) { + let mut uses: Vec = kept.values().map(&used).collect(); + let half = uses.len() / 2; + let (_, &mut cut, _) = uses.select_nth_unstable(half); + kept.retain(|_, v| used(v) > cut); +} + +/// A page of words for `column`'s rows `rows`, padded with missing values past the rows. +pub(crate) fn values(column: &Column, rows: &[u32], words: Words, low: bool) -> Vec { + let mut out = page_words(&column.pick(rows), words, low); + out.resize(PAGE_ROWS as usize, if words == Words::Bits { NONE } else { MISSING }); + out +} + +/// A page of the target rows that `reference` gives for rows `rows`. Each target is its place in +/// `rank` (its table's draw order), or the row itself when that table is drawn whole. `NONE` where +/// a row references nothing. +pub(crate) fn targets(reference: &Reference, rows: &[u32], rank: Option<&Order>) -> Vec { + let mut out: Vec = rows + .iter() + .map(|&r| match reference.target_row(r as usize) { + UNWIRED => NONE, + t => rank.map_or(t, |order| order.rank(t)), + }) + .collect(); + out.resize(PAGE_ROWS as usize, NONE); + out +} + +/// The words of reference page `page` of a table drawn in its own order. Each word is the target +/// row's place in `rank`, the target table's draw order. +pub(crate) fn ranked_page(reference: &Reference, page: usize, rank: &Order) -> Vec { + let start = page * PAGE_ROWS as usize; + let len = reference.rows.pages().get(page).map_or(0, |p| p.values.len()); + (start..start + len) + .map(|r| match reference.target_row(r) { + UNWIRED => NONE, + t => rank.rank(t), + }) + .collect() +} + +/// A page of the values of `samples`' rows `rows` at frame `frame`, padded with missing values. +pub(crate) fn frame(samples: &Samples, rows: &[u32], frame: Option) -> Vec { + let mut out: Vec = rows + .iter() + .map(|&r| frame.and_then(|f| samples.value(r as usize, f)).map_or(MISSING, |v| (v as f32).to_bits())) + .collect(); + out.resize(PAGE_ROWS as usize, MISSING); + out +} + +/// A page of the rows of `other` with the ids of `table`'s rows `rows`. `NONE` where `other` has +/// no such row. +pub(crate) fn joined(table: &Table, rows: &[u32], other: &Table) -> Vec { + let mut out: Vec = rows + .iter() + .map(|&r| table.ids.key(r as usize).and_then(|key| other.row(key)).map_or(NONE, |o| o as u32)) + .collect(); + out.resize(PAGE_ROWS as usize, NONE); + out +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::*; + use crate::data::Values; + use crate::derive::pyramid::Pyramid; + + fn packing() -> Packing { + Packing { + order: 1, + origin: Version::next(), + table: "T".into(), + field: "v".into(), + words: Words::Values, + low: false, + reads: 0, + } + } + + #[test] + fn a_write_repacks_only_the_pages_its_rows_draw_in() { + let n = 3 * PAGE_ROWS as usize; + let column = Column::new(&Values::from((0..n).map(|i| i as f64).collect::>())); + // The order draws the rows backward: the last page of rows is its first page. + let order = Order::Pyramid { + pyramid: Arc::new(Pyramid::new((0..n as u32).rev().collect(), Vec::new())), + rows: n as u32, + }; + let mut packed = Packed::new(Arc::default()); + let packing = packing(); + let ids: Vec = (0..3).map(|p| packed.page(&packing, &order, Some(&column), p)).collect(); + let written = column.write(n, &[(5, 0)], &Column::new(&Values::from(vec![-1.0_f64]))).expect("numbers"); + let after: Vec = (0..3).map(|p| packed.page(&packing, &order, Some(&written), p)).collect(); + assert_eq!((after[0], after[1]), (ids[0], ids[1]), "pages without row 5 are reused"); + assert_ne!(after[2], ids[2], "row 5 is in the last page, which is repacked"); + let words = values(&written, &order.page(2), Words::Values, false); + assert_eq!(f32::from_bits(words[PAGE_ROWS as usize - 1 - 5]), -1.0, "row 5 is packed at its place"); + } +} diff --git a/crates/latkit/src/gpu/pipelines.rs b/crates/latkit/src/gpu/pipelines.rs index 9b52238b..fe2f0843 100644 --- a/crates/latkit/src/gpu/pipelines.rs +++ b/crates/latkit/src/gpu/pipelines.rs @@ -1,12 +1,15 @@ -//! Pipelines, kept by [`Variant`]. Everything compiles through one queue per GPU -//! ([`Pipelines::ask`]), the most urgent first: what frames draw with, then their text, then what -//! changes, lit rows, and input need, then what the GPU readies for views to come, once it would -//! otherwise sit idle. In a browser what frames draw with compiles at once, their text as the -//! browser's compile workers free up, and the rest a few at a time, each off the GPU's thread, side -//! by side with the rest; what uses it waits until it landed, so no frame waits on a compile. Making -//! it afterwards finds it compiled. Natively a pipeline is made in process where first used, which -//! costs as little. Every pipeline shares the view and mark group layouts; each -//! shader's pass group comes from its WGSL. +//! Pipelines, cached by [`Variant`]. +//! +//! Every pipeline compiles through one queue per GPU ([`Pipelines::ask`]), most urgent first: draw +//! pipelines, then text pipelines, then the pipelines that changes, lit rows, and input need. Last +//! come pipelines warmed for future views, once the queue would otherwise sit idle. +//! +//! In a browser, compiles run in the background and in parallel. Draw pipelines start at once, +//! text pipelines as the browser's compile workers free up, and the rest a few at a time. Anything +//! that uses a pipeline waits until its compile finishes. So no frame waits on a compile, and +//! creating the pipeline afterwards finds it compiled. Natively, a pipeline is created in process +//! when first used, which is just as cheap. Every pipeline shares the view and mark group layouts. +//! Each shader's pass group layout comes from its WGSL. use std::collections::HashMap; use std::future::Future; @@ -21,8 +24,9 @@ use super::work::{Count, Counters}; use crate::Error; use crate::view::DEPTH; -/// How soon a pipeline asked for is needed, the soonest compiling first: what frames draw with, -/// their text, what changes, lit rows, and input need, and what the GPU readies for views to come. +/// How soon a requested pipeline is needed. Sooner compiles first. `Draw`: pipelines that frames +/// draw with. `Text`: pipelines for the frames' text. `Answer`: pipelines that changes, lit rows, +/// and input need. `Warm`: pipelines warmed for future views. #[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug)] pub(crate) enum Urgency { Draw, @@ -31,13 +35,14 @@ pub(crate) enum Urgency { Warm, } -/// How many pipelines a browser compiles side by side. Text asked while fewer compile takes a free -/// worker, where asked sooner it would hold back what frames draw with; the rest compile at most half -/// as many at once, so frames drawing meanwhile keep the CPU, the next asked as each lands. +/// How many pipelines a browser compiles in parallel. Text pipelines start only while fewer than +/// this are compiling, or while no draw pipeline is, since starting them sooner would delay the +/// draw pipelines. The rest compile at most half as many at once, so frames drawn meanwhile keep +/// the CPU. Each finished compile starts the next. const WORKERS: u64 = 8; -/// A check of a custom marker's shader, resolving to its error if it has one: sent between threads -/// natively, kept on the page's one thread in browsers. +/// A pending check of a custom marker's shader, resolving to its error if any. It is `Send` +/// natively, and stays on the page's single thread in browsers. #[cfg(not(target_arch = "wasm32"))] type Check = Pin> + Send>>; #[cfg(target_arch = "wasm32")] @@ -50,10 +55,10 @@ struct Custom { failed: bool, } -/// One pipeline: a shader's entry point made for one primitive's geometry (its `PRIMITIVE`), one kind -/// of path (its `PATH`), and what channels read (its `JOINED`, `TIMED`, and `VIEWED`), and for a render -/// shader, the format and samples per pixel it draws into. It holds a primitive and a path only where -/// its shader reads them, so no two of its pipelines compile the same. +/// One pipeline: a shader's entry point specialized for one primitive's geometry (`PRIMITIVE`), +/// one kind of path (`PATH`), and what channels read (`JOINED`, `TIMED`, and `VIEWED`). A render +/// variant also has the format and multisample count it draws into. It sets a primitive and a path +/// only when its shader reads them, so no two variants compile to the same pipeline. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] pub(crate) struct Variant { shader: Shader, @@ -64,9 +69,9 @@ pub(crate) struct Variant { target: Option<(wgpu::TextureFormat, u32)>, } -/// What a mark's channels read besides its own table's static fields: fields of other tables by row -/// id, fields sampled over an axis, and the view (selection, hover, the pointer, the zoom). Every -/// pipeline but the channels' reads all of them. +/// What a mark's channels read besides its own table's static fields: other tables' fields by row +/// id (`joined`), fields sampled over an axis (`timed`), and the view's selection, hover, pointer, +/// and zoom (`viewed`). Every pipeline except the channels pipeline reads all of them. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] pub(crate) struct Reads { pub(crate) joined: bool, @@ -76,11 +81,12 @@ pub(crate) struct Reads { impl Reads { pub(crate) const ALL: Reads = Reads { joined: true, timed: true, viewed: true }; + pub(crate) const NONE: Reads = Reads { joined: false, timed: false, viewed: false }; } impl Variant { - /// `shader` drawing into `format` with `samples` per pixel. - pub(crate) fn draw(shader: Shader, format: wgpu::TextureFormat, samples: u32) -> Variant { + /// `shader` drawing into `format` with `samples` samples per pixel. + pub(crate) const fn draw(shader: Shader, format: wgpu::TextureFormat, samples: u32) -> Variant { Variant { shader, entry: "vertex", @@ -92,22 +98,23 @@ impl Variant { } /// `shader`'s compute entry point `entry`. - pub(crate) fn compute(shader: Shader, entry: &'static str) -> Variant { + pub(crate) const fn compute(shader: Shader, entry: &'static str) -> Variant { Variant { shader, entry, primitive: 0, path: PATH_ENDS, reads: Reads::ALL, target: None } } - /// The probe program made for marks of `primitive`, which answers every question a step at a time. + /// The probe program for marks of `primitive`. It answers every query in steps, and each + /// dispatch's uniform selects the step. pub(crate) fn probe(shader: Shader, primitive: u32) -> Variant { Variant::compute(shader, "answer").primitive(primitive) } - /// The channels of marks reading what `reads` says. + /// The channels pipeline for marks whose channels read `reads`. pub(crate) fn channels(reads: Reads) -> Variant { Variant { reads, ..Variant::compute(Shader::Evaluate, "evaluate") } } - /// The variant made for marks of `primitive`, a `POINTS`, `STROKES`, or `TRIANGLES` code, where its - /// shader holds one primitive's geometry. + /// This variant specialized for marks of `primitive` (a `POINTS`, `STROKES`, or `TRIANGLES` + /// code), if its shader reads `PRIMITIVE`. #[must_use] pub(crate) fn primitive(mut self, primitive: u32) -> Variant { if self.reads_constant("PRIMITIVE") { @@ -116,7 +123,7 @@ impl Variant { self } - /// The variant whose strokes read `path`, one of the `PATH_` codes, where its shader reads one. + /// This variant with strokes reading `path` (a `PATH_` code), if its shader reads `PATH`. #[must_use] pub(crate) fn path(mut self, path: u32) -> Variant { if self.reads_constant("PATH") { @@ -129,7 +136,7 @@ impl Variant { self.shader.source().overrides.contains(&name) } - /// The pipeline constants its shader declares, set. + /// Values for the pipeline constants its shader declares. fn constants(&self) -> Vec<(&'static str, f64)> { let value = |name: &str| match name { "PRIMITIVE" => f64::from(self.primitive), @@ -143,8 +150,8 @@ impl Variant { } } -/// A pipeline asked for ahead: a variant of one of latkit's shaders, or of a custom marker's, whose -/// WGSL follows its shader's. +/// A pipeline requested ahead of use: a variant of a latkit shader, or of a custom marker. A custom +/// marker's WGSL is appended to its shader's WGSL. #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub(crate) enum Asked { Shader(Variant), @@ -152,7 +159,7 @@ pub(crate) enum Asked { } impl Asked { - /// Bytes of WGSL it compiles from: the more, the longer it takes. + /// Bytes of WGSL it compiles from. More bytes take longer to compile. pub(crate) fn size(&self) -> usize { match self { Asked::Shader(variant) => variant.shader.source().size(), @@ -161,8 +168,8 @@ impl Asked { } } -/// A pipeline asked for ahead: whether it compiled then, off the GPU's thread, whether that landed, -/// and what waits for it to. +/// The compile state of a pipeline requested ahead: whether it compiled in the background, whether +/// that compile finished (`landed`), and the wakers waiting for it. #[derive(Default)] struct Ahead { compiled: AtomicBool, @@ -175,12 +182,12 @@ impl Ahead { self.landed.load(Ordering::Acquire) } - /// Waits for it: `waker` wakes as it lands. + /// Registers `waker` to wake when the compile finishes. fn wait(&self, waker: &Waker) { self.waiting.lock().unwrap_or_else(PoisonError::into_inner).push(waker.clone()); } - /// Lands it, counted by `landed`, and wakes what waits. + /// Marks the compile finished, increments `landed`, and wakes the waiters. fn land(&self, landed: &AtomicU64) { self.landed.store(true, Ordering::Release); landed.fetch_add(1, Ordering::AcqRel); @@ -193,7 +200,7 @@ impl Ahead { pub(crate) struct Pipelines { pub(crate) view_layout: wgpu::BindGroupLayout, pub(crate) mark_layout: wgpu::BindGroupLayout, - /// What a shader reading neither shared group binds in their place. + /// The bind group bound in place of the shared groups for a shader that reads neither. pub(crate) empty: wgpu::BindGroup, empty_layout: wgpu::BindGroupLayout, pass_layouts: HashMap, wgpu::BindGroupLayout>, @@ -201,27 +208,28 @@ pub(crate) struct Pipelines { render: HashMap, compute: HashMap, custom: HashMap<(Arc, Variant), Custom>, - /// What views, shots, and the GPU asked for ahead, each with its compile. + /// Pipelines requested ahead by views, shots, and the GPU, each with its compile state. asked: HashMap>, - /// What was asked for and waits its turn: soonest first, the largest first within an urgency, - /// since a browser compiles in the order asked. + /// Requested pipelines waiting to start compiling: most urgent first, and largest first within + /// an urgency, since a browser compiles in request order. queue: Vec<(Urgency, Asked)>, - /// How many compiles asked for ahead landed, and in a browser how many are under way. + /// How many requested compiles have finished. In a browser, `compiling` counts those in + /// progress. landed: Arc, #[cfg(all(feature = "web", target_arch = "wasm32"))] compiling: Arc, - /// Of those, what frames draw with. + /// Of the compiles in progress, how many are draw pipelines. #[cfg(all(feature = "web", target_arch = "wasm32"))] drawing: Arc, - /// Whether frames were asked to draw with anything yet, and whether what the GPU warms starts. + /// Whether any draw pipeline has started compiling, and whether warm pipelines may start. drew: bool, warming: bool, - /// Whether what views to come draw with was queued. + /// Whether the pipelines for future views were queued. #[cfg(all(feature = "web", target_arch = "wasm32"))] warmed: bool, - /// What counts the pipelines compiled. + /// Counts the pipelines compiled. counters: Arc, - /// The GPU these belong to, for a compile landing to start the next. + /// The GPU these belong to, so a finished compile can start the next. #[cfg_attr( not(all(target_arch = "wasm32", feature = "web")), expect(dead_code, reason = "only a browser compiles ahead") @@ -229,65 +237,34 @@ pub(crate) struct Pipelines { gpu: Weak, } -/// Shaders that draw into the shadow map, custom markers' included. -const CASTERS: [Shader; 5] = - [Shader::PointsCast, Shader::StrokesCast, Shader::TrianglesCast, Shader::Shade, Shader::PointsCustomCast]; -/// The probe a frame of `samples` per pixel reads its depth with. +/// The probe shader that reads depth in frames of `samples` samples per pixel. pub(crate) fn probe(samples: u32) -> Shader { if samples > 1 { Shader::ProbeMultisampled } else { Shader::Probe } } -/// How a render shader's vertices assemble. +/// The primitive topology of a render shader. fn topology(shader: Shader) -> wgpu::PrimitiveTopology { - // Custom markers draw as points do. + // Custom markers draw like points, as triangle strips. match shader { - Shader::Backdrop | Shader::Triangles | Shader::TrianglesCast | Shader::Shade => { - wgpu::PrimitiveTopology::TriangleList - } + Shader::Backdrop | Shader::Triangles => wgpu::PrimitiveTopology::TriangleList, _ => wgpu::PrimitiveTopology::TriangleStrip, } } -/// Whether a shader draws into the shadow map, which has no depth. -fn casts(shader: Shader) -> bool { - CASTERS.contains(&shader) -} - -/// How a render shader meets depth, which is reversed: the backdrop writes the globe's, triangles -/// write theirs, the overlay and labels ignore it, and every other mark draws where nothing nearer -/// was drawn. -fn depth(shader: Shader) -> Option { - if casts(shader) { - return None; - } +/// A render shader's depth state, with reversed depth. The backdrop writes the globe's depth and +/// triangles write theirs. The overlay and glyphs ignore depth, and every other mark draws only +/// where nothing nearer was drawn. +fn depth(shader: Shader) -> wgpu::DepthStencilState { let compare = match shader { Shader::Backdrop | Shader::Overlay | Shader::Glyphs => wgpu::CompareFunction::Always, _ => wgpu::CompareFunction::GreaterEqual, }; - Some(wgpu::DepthStencilState { + wgpu::DepthStencilState { format: DEPTH, depth_write_enabled: Some(matches!(shader, Shader::Triangles | Shader::Backdrop)), depth_compare: Some(compare), stencil: wgpu::StencilState::default(), bias: wgpu::DepthBiasState::default(), - }) -} - -/// How a shader's fragments blend: premultiplied over what is drawn; casters keep the least of each -/// channel; map levels replace. -fn blend(shader: Shader) -> Option { - let component = |src, dst, operation| wgpu::BlendComponent { src_factor: src, dst_factor: dst, operation }; - match shader { - Shader::Shade => None, - Shader::PointsCast | Shader::StrokesCast | Shader::TrianglesCast | Shader::PointsCustomCast => { - let least = component(wgpu::BlendFactor::One, wgpu::BlendFactor::One, wgpu::BlendOperation::Min); - Some(wgpu::BlendState { color: least, alpha: least }) - } - _ => { - let over = - component(wgpu::BlendFactor::One, wgpu::BlendFactor::OneMinusSrcAlpha, wgpu::BlendOperation::Add); - Some(wgpu::BlendState { color: over, alpha: over }) - } } } @@ -330,18 +307,18 @@ impl Pipelines { } } - /// Asks for each of `needs` at `urgency`, `waker` waking as each lands: what frames draw with - /// compiles at once, their text as workers free up, the rest in turn, half the `WORKERS` at a time; - /// asking again for one still waiting its turn moves it up. Its shader module is made now: a - /// browser reads a module's WGSL on its GPU thread as it is made, so what compiles later is read - /// before anything draws. + /// Requests each of `needs` at `urgency`, waking `waker` as each compile finishes. Draw + /// pipelines start at once, text pipelines as workers free up, and the rest in turn, at most + /// half of `WORKERS` at a time. Requesting a queued pipeline again at a higher urgency moves it + /// up. Each shader module is created now, because a browser parses a module's WGSL on its GPU + /// thread at creation. This way, modules that compile later are parsed before anything draws. pub(crate) fn ask(&mut self, device: &wgpu::Device, needs: &[Asked], urgency: Urgency, waker: &Waker) { self.queue(device, needs, urgency, waker); self.pump(device); } - /// Queues each of `needs` at `urgency` as [`ask`](Pipelines::ask) does, starting none: the next - /// ask, landing, or [`pump`](Pipelines::pump) does. + /// Queues each of `needs` at `urgency` like [`ask`](Pipelines::ask), without starting any. The + /// next request, finished compile, or [`pump`](Pipelines::pump) starts them. pub(crate) fn queue(&mut self, device: &wgpu::Device, needs: &[Asked], urgency: Urgency, waker: &Waker) { for need in needs { if let Some(ahead) = self.asked.get(need) { @@ -361,26 +338,28 @@ impl Pipelines { } self.queue.push((urgency, need.clone())); } - // The largest first, so the last lands soonest; what the GPU warms in the order listed. + // Largest first, so the last one finishes soonest. Warm pipelines keep their listed order. crate::order::by_key(&mut self.queue, |(urgency, need)| { let size = if *urgency == Urgency::Warm { 0 } else { !(need.size().min(u32::MAX as usize) as u32) }; crate::order::pair(*urgency as u32, size) }); } - /// Queues what `needs` lists, what views to come draw with, once per GPU: it compiles once the - /// first scene's frames have all they draw with, so it never holds them back. + /// Queues `needs`, the pipelines future views will draw with, once per GPU. They compile after + /// the first scene's draw pipelines finish, so they never delay them. `waker` wakes as each + /// finishes, so anything waiting on the GPU's compiles sees it. #[cfg(all(feature = "web", target_arch = "wasm32"))] - pub(crate) fn warm(&mut self, device: &wgpu::Device, needs: impl FnOnce() -> Vec) { + pub(crate) fn warm(&mut self, device: &wgpu::Device, waker: &Waker, needs: impl FnOnce() -> Vec) { if !std::mem::replace(&mut self.warmed, true) { - self.queue(device, &needs(), Urgency::Warm, Waker::noop()); + self.queue(device, &needs(), Urgency::Warm, waker); } } - /// Starts what waits its turn, in order: all that frames draw with; their text while fewer than - /// `WORKERS` compile, or none of that does; the rest while fewer than half as many compile. + /// Starts queued compiles in order. Draw pipelines always start. Text pipelines start while + /// fewer than `WORKERS` are compiling or no draw pipeline is. The rest start while fewer than + /// half of `WORKERS` are compiling. pub(crate) fn pump(&mut self, device: &wgpu::Device) { - // What the GPU warms waits for the first scene's frames to have all they draw with. + // Warm pipelines wait until the first scene's draw pipelines have finished. self.warming |= self.drew && self.drawing() == 0; while let Some(&(urgency, _)) = self.queue.first() && match urgency { @@ -399,7 +378,7 @@ impl Pipelines { } } - /// Compiles under way, off the GPU's thread. + /// Compiles in progress in the background. #[cfg_attr( not(all(target_arch = "wasm32", feature = "web")), expect(clippy::unused_self, reason = "only a browser's device compiles ahead") @@ -411,7 +390,7 @@ impl Pipelines { 0 } - /// Compiles under way of what frames draw with. + /// Draw pipeline compiles in progress. #[cfg_attr( not(all(target_arch = "wasm32", feature = "web")), expect(clippy::unused_self, reason = "only a browser's device compiles ahead") @@ -423,25 +402,27 @@ impl Pipelines { 0 } - /// Whether every one of `needs`, asked for, has landed: what uses them can be made without a - /// compile. + /// Whether every one of `needs` was requested and has finished compiling, so pipelines using + /// them can be created without a compile. pub(crate) fn ready(&self, needs: &[Asked]) -> bool { needs.iter().all(|need| self.asked.get(need).is_some_and(|ahead| ahead.landed())) } - /// How many compiles asked for ahead have landed so far: what waits on one sees it change. + /// How many requested compiles have finished so far. Anything waiting on a compile sees this + /// change. pub(crate) fn landed(&self) -> u64 { self.landed.load(Ordering::Acquire) } - /// Compiles asked for ahead still under way or waiting their turn. + /// Requested compiles still in progress or queued. #[cfg(all(feature = "web", target_arch = "wasm32"))] pub(crate) fn compiling(&self) -> u64 { self.in_flight() + self.queue.len() as u64 } - /// Starts compiling `need` off the GPU's thread, where a browser can, counted among what frames - /// draw with when it `draws`: whether it did. Its landing starts the next waiting its turn. + /// Starts compiling `need` in the background where a browser supports it, counted as a draw + /// pipeline when `draws`. Returns whether it started. When it finishes, it starts the next + /// queued compile. #[cfg_attr( not(all(target_arch = "wasm32", feature = "web")), expect(clippy::unused_self, reason = "only a browser's device compiles ahead") @@ -460,13 +441,14 @@ impl Pipelines { drawing.fetch_add(1, Ordering::AcqRel); } wasm_bindgen_futures::spawn_local(async move { - // A descriptor the browser refuses lands too: making it then tells why, as it did. + // A descriptor the browser rejects also finishes. Creating the pipeline later + // reports why, as it would without compiling ahead. let _ = wasm_bindgen_futures::JsFuture::from(promise).await; compiling.fetch_sub(1, Ordering::AcqRel); if let Some(drawing) = drawing { drawing.fetch_sub(1, Ordering::AcqRel); } - // Room for the next before what waits wakes, so it finds that under way. + // Start the next compile before waking the waiters, so they find it in progress. if let Some(gpu) = gpu.upgrade() && let Some(mut inner) = gpu.try_lock() { @@ -483,8 +465,9 @@ impl Pipelines { } } - /// Makes `need` through wgpu, neither kept nor counted: what compiling it ahead captures. A - /// custom marker's WGSL fails in its own scope, so only making it for real tells why. + /// Creates `need` through wgpu without caching or counting it, for compiling ahead to capture. + /// A custom marker's WGSL errors are caught in a scope of their own here, so only creating it + /// for real reports why it fails. #[cfg(all(target_arch = "wasm32", feature = "web"))] fn create(&mut self, device: &wgpu::Device, need: &Asked) { match need { @@ -502,8 +485,8 @@ impl Pipelines { } } - /// Counts a pipeline made for `need`: compiled now unless it compiled when asked for, and late - /// unless it was asked for at all. + /// Counts a pipeline created for `need`: as compiled now unless it compiled ahead, and as late + /// if it was never requested. fn count(&self, need: &Asked) { let ahead = self.asked.get(need); if !ahead.is_some_and(|ahead| ahead.compiled.load(Ordering::Acquire)) { @@ -514,7 +497,7 @@ impl Pipelines { } } - /// The layout of `shader`'s pass group; shaders with the same bindings share one. + /// The layout of `shader`'s pass group. Shaders with the same bindings share one. pub(crate) fn pass_layout(&mut self, device: &wgpu::Device, shader: Shader) -> wgpu::BindGroupLayout { let entries: Vec<_> = shader.source().pass.iter().map(Binding::layout_entry).collect(); self.pass_layouts @@ -539,8 +522,8 @@ impl Pipelines { .clone() } - /// The layout of `shader`'s pipelines: the view and mark groups, empty where it reads neither, then - /// its pass group. + /// The layout of `shader`'s pipelines: the view and mark groups (empty if it reads neither), + /// then its pass group. fn pipeline_layout(&mut self, device: &wgpu::Device, shader: Shader) -> wgpu::PipelineLayout { let pass = (!shader.source().pass.is_empty()).then(|| self.pass_layout(device, shader)); let mut groups = if shader.source().shared { @@ -558,31 +541,29 @@ impl Pipelines { }) } - /// The pipeline drawing a custom marker of `source` with `shader` (one of the custom variants), - /// or `None` once its WGSL failed to compile. + /// The pipeline that draws a custom marker of `source`: `variant`, with `source` appended to + /// its shader. `None` once that WGSL failed to compile. pub(crate) fn custom( &mut self, device: &wgpu::Device, - source: &str, - shader: Shader, - format: wgpu::TextureFormat, - samples: u32, + source: &Arc, + variant: Variant, ) -> Option { - let key = (Arc::::from(source), Variant::draw(shader, format, samples)); + let key = (source.clone(), variant); if let Some(custom) = self.custom.get(&key) { return (!custom.failed).then(|| custom.pipeline.clone()); } self.count(&Asked::Custom(key.0.clone(), key.1)); let scope = device.push_error_scope(wgpu::ErrorFilter::Validation); - let module = custom_module(device, source, shader); - let pipeline = self.pipeline(device, key.1, &module); + let module = custom_module(device, source, variant.shader); + let pipeline = self.pipeline(device, variant, &module); let check: Check = Box::pin(scope.pop()); let custom = Custom { pipeline: pipeline.clone(), check: Some(check), failed: false }; self.custom.insert(key, custom); Some(pipeline) } - /// Errors custom markers' shaders turned out to have since last asked. + /// Errors found in custom markers' shaders since the last call. pub(crate) fn custom_errors(&mut self) -> Vec { let mut context = Context::from_waker(Waker::noop()); let mut errors = Vec::new(); @@ -599,19 +580,8 @@ impl Pipelines { errors } - /// The pipeline drawing `shader` into `format` with `samples` per pixel. - pub(crate) fn render( - &mut self, - device: &wgpu::Device, - shader: Shader, - format: wgpu::TextureFormat, - samples: u32, - ) -> wgpu::RenderPipeline { - self.render_pipeline(device, Variant::draw(shader, format, samples)) - } - /// The render pipeline `variant` names. - pub(crate) fn render_pipeline(&mut self, device: &wgpu::Device, variant: Variant) -> wgpu::RenderPipeline { + pub(crate) fn render(&mut self, device: &wgpu::Device, variant: Variant) -> wgpu::RenderPipeline { if let Some(pipeline) = self.render.get(&variant) { return pipeline.clone(); } @@ -643,7 +613,7 @@ impl Pipelines { buffers: &[], }, primitive: wgpu::PrimitiveState { topology: topology(shader), ..Default::default() }, - depth_stencil: depth(shader), + depth_stencil: Some(depth(shader)), multisample: wgpu::MultisampleState { count: samples, ..Default::default() }, fragment: Some(wgpu::FragmentState { module, @@ -651,7 +621,8 @@ impl Pipelines { compilation_options: options(), targets: &[Some(wgpu::ColorTargetState { format, - blend: blend(shader), + // Blend premultiplied colors over what is already drawn. + blend: Some(wgpu::BlendState::PREMULTIPLIED_ALPHA_BLENDING), write_mask: wgpu::ColorWrites::ALL, })], }), @@ -660,18 +631,8 @@ impl Pipelines { }) } - /// The compute pipeline of `shader`'s entry point `entry`. - pub(crate) fn compute( - &mut self, - device: &wgpu::Device, - shader: Shader, - entry: &'static str, - ) -> wgpu::ComputePipeline { - self.compute_pipeline(device, Variant::compute(shader, entry)) - } - /// The compute pipeline `variant` names. - pub(crate) fn compute_pipeline(&mut self, device: &wgpu::Device, variant: Variant) -> wgpu::ComputePipeline { + pub(crate) fn compute(&mut self, device: &wgpu::Device, variant: Variant) -> wgpu::ComputePipeline { if let Some(pipeline) = self.compute.get(&variant) { return pipeline.clone(); } @@ -696,7 +657,7 @@ impl Pipelines { } } -/// The module of a custom marker of `source` drawn with `shader`: its shader's WGSL, then the marker's. +/// The shader module for a custom marker: `shader`'s WGSL followed by the marker's `source`. fn custom_module(device: &wgpu::Device, source: &str, shader: Shader) -> wgpu::ShaderModule { device.create_shader_module(wgpu::ShaderModuleDescriptor { label: Some("latkit custom marker"), @@ -704,10 +665,10 @@ fn custom_module(device: &wgpu::Device, source: &str, shader: Shader) -> wgpu::S }) } -/// Compiling ahead in a browser. A browser compiles a pipeline made through `createRenderPipeline` -/// on the GPU's thread, holding back every frame and readback behind it, but one made through its -/// async twin on worker threads, side by side with others; Chrome keeps what it compiled by -/// descriptor, so making the same descriptor later finds it compiled. +/// Compiling ahead in a browser. A browser compiles a pipeline created with `createRenderPipeline` +/// on the GPU's thread, delaying every frame and readback behind it. One created with the async +/// variant compiles on worker threads, in parallel with others. Chrome caches compiled pipelines by +/// descriptor, so creating the same descriptor later finds it compiled. #[cfg(all(target_arch = "wasm32", feature = "web"))] mod compiled { use std::cell::RefCell; @@ -716,16 +677,16 @@ mod compiled { use wasm_bindgen::closure::Closure; use wasm_bindgen::{JsCast, JsValue}; - /// The device's pipeline makers, each with its async twin. + /// The device's pipeline creation methods, each with its async variant. const MAKERS: [(&str, &str); 2] = [ ("createRenderPipeline", "createRenderPipelineAsync"), ("createComputePipeline", "createComputePipelineAsync"), ]; - /// The browser compiling, off its GPU's thread, the pipeline `make` makes through wgpu: while it - /// runs, the device's makers are their async twins, handed the very descriptor wgpu builds, so - /// what compiles is what wgpu makes later, by construction. `None` where the device is no - /// browser's, or `make` made nothing. + /// Has the browser compile, in the background, the pipeline that `make` creates through wgpu. + /// While `make` runs, the device's creation methods are swapped for their async variants. These + /// receive the exact descriptor wgpu builds, so what compiles is exactly what wgpu creates + /// later. Returns `None` if the device is not a browser's or `make` created nothing. pub(super) fn capture(device: &wgpu::Device, make: impl FnOnce()) -> Option { let raw: &JsValue = device.as_webgpu()?.as_ref(); let twins: Vec = MAKERS diff --git a/crates/latkit/src/gpu/readback.rs b/crates/latkit/src/gpu/readback.rs index 1b353e96..e4b075c7 100644 --- a/crates/latkit/src/gpu/readback.rs +++ b/crates/latkit/src/gpu/readback.rs @@ -1,6 +1,6 @@ -//! Bytes read back from the GPU: copied in a frame's encoder and mapped when the caller submits it, -//! so reading never stalls a frame and never submits on its own; once mapped, it wakes what waits on -//! it. Map buffers come from a pool and go back to it once read, so hovering allocates nothing. +//! Readbacks from the GPU. Bytes are copied in a frame's encoder and mapped when the caller submits +//! it, so reading never stalls a frame or submits on its own. Once mapped, a readback wakes its +//! waker. Map buffers come from a pool and return to it once read, so hovering allocates nothing. use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex, PoisonError}; @@ -9,8 +9,8 @@ use std::task::Waker; use super::work::{Count, Counters}; use crate::{Error, Result}; -/// Map buffers no readback holds, by size, what counts the readbacks made from them, and how many -/// are still mapping. Every pool says whether it counts, so none stops counting by default. +/// A pool of free map buffers. It also holds the counters for its readbacks and the number still +/// mapping. Each pool is created as counted or uncounted, so no pool skips counting by default. #[derive(Clone)] pub(crate) struct Readbacks { free: Arc>>, @@ -19,17 +19,17 @@ pub(crate) struct Readbacks { } impl Readbacks { - /// A pool whose readbacks `counters` counts: the `Gpu`'s. + /// A pool whose readbacks are counted in `counters`, like the `Gpu`'s pool. pub(crate) fn counted(counters: Arc) -> Readbacks { Readbacks { free: Arc::default(), counters: Some(counters), mapping: Arc::default() } } - /// A pool whose readbacks count as no work: the frame timer's, which only some devices have. + /// A pool whose readbacks are not counted: the frame timer's, which only some devices support. pub(crate) fn uncounted() -> Readbacks { Readbacks { free: Arc::default(), counters: None, mapping: Arc::default() } } - /// Readbacks of this pool the GPU has yet to map. + /// Readbacks from this pool the GPU has not mapped yet. #[cfg(all(feature = "web", target_arch = "wasm32"))] pub(crate) fn mapping(&self) -> u64 { self.mapping.load(Ordering::Acquire) @@ -53,7 +53,7 @@ impl Readbacks { } } -/// Where a readback's map stands. +/// The state of a readback's mapping. #[derive(Clone, Copy, PartialEq, Eq)] enum Mapping { Waiting, @@ -70,8 +70,8 @@ pub(crate) struct Readback { } impl Readback { - /// Copies `size` bytes of `source` from `offset`, to read once `encoder` is submitted; `waker` - /// wakes once they can be read. + /// Copies `size` bytes of `source` from `offset`, to read once `encoder` is submitted. `waker` + /// wakes when they can be read. pub(crate) fn new( pool: &Readbacks, device: &wgpu::Device, @@ -97,7 +97,8 @@ impl Readback { Readback { buffer: Some(buffer), size, pool: pool.clone(), mapping } } - /// The bytes once mapped, or `None` while the GPU has not finished or they were taken. + /// The bytes once mapped, or `None` while the GPU has not finished or after they were taken. + /// Errors with `DeviceLost` if mapping failed. pub(crate) fn take(&self) -> Option>> { let mut mapping = self.mapping.lock().unwrap_or_else(PoisonError::into_inner); let buffer = self.buffer.as_ref()?; @@ -120,7 +121,7 @@ impl Readback { impl Drop for Readback { fn drop(&mut self) { let Some(buffer) = self.buffer.take() else { return }; - // A buffer still mapping cannot go back yet; it is dropped instead. + // A buffer still mapping cannot return to the pool, so it is dropped instead. let mapping = *self.mapping.lock().unwrap_or_else(PoisonError::into_inner); if mapping == Mapping::Ready { buffer.unmap(); diff --git a/crates/latkit/src/gpu/shaders.rs b/crates/latkit/src/gpu/shaders.rs index 58495b6a..7bfb2d43 100644 --- a/crates/latkit/src/gpu/shaders.rs +++ b/crates/latkit/src/gpu/shaders.rs @@ -1,9 +1,9 @@ -//! The GPU contract generated from the WESL shaders by `build/shaders.rs`: constants, `#[repr(C)]` -//! mirrors of shared structs, bind group layouts, and each shader's text. +//! Code that `build/shaders.rs` generates from the WESL shaders: constants, `#[repr(C)]` mirrors of +//! shared structs, bind group layouts, and each shader's text. #![allow(clippy::unreadable_literal)] -/// How a binding is read. +/// A binding's resource type. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) enum Kind { Uniform, @@ -27,7 +27,8 @@ const FRAGMENT: u32 = 2; const COMPUTE: u32 = 4; impl ShaderSource { - /// The shader's WGSL: its declarations, each stored once for every shader, joined in order. + /// The shader's WGSL: its declarations joined in order. Each declaration is stored once and + /// shared by every shader. pub(crate) fn wgsl(&self) -> String { let mut out = String::new(); for (i, &piece) in self.pieces.iter().enumerate() { @@ -40,8 +41,8 @@ impl ShaderSource { out } - /// Bytes of its WGSL: what a browser compiles for each of its pipelines, and so about how long - /// one takes. + /// The size of its WGSL in bytes, a rough measure of compile time. A browser compiles the WGSL + /// once per pipeline. pub(crate) fn size(&self) -> usize { self.pieces .iter() @@ -53,11 +54,11 @@ impl ShaderSource { } } -/// Every distinct top-level declaration of every linked shader, each once; [`PIECES`] says where each -/// lies. +/// Every distinct top-level declaration of every linked shader, each stored once. [`PIECES`] gives +/// each one's byte range. static WGSL: &str = include_str!(concat!(env!("OUT_DIR"), "/shaders.wgsl")); -/// One linked shader: its declarations, entry points, and the bindings of its pass group. +/// One linked shader: its declarations, entry points included, and the bindings of its pass group. pub(crate) struct ShaderSource { pub(crate) label: &'static str, /// Its declarations, as indices into [`PIECES`], in order. @@ -65,7 +66,8 @@ pub(crate) struct ShaderSource { pub(crate) pass: &'static [Binding], /// The pipeline constants it declares. pub(crate) overrides: &'static [&'static str], - /// Whether it reads the view and mark groups; one that does not binds its pass group alone. + /// Whether it reads the view and mark groups. A shader that does not gets empty groups in their + /// place. pub(crate) shared: bool, } @@ -114,5 +116,6 @@ impl Binding { include!(concat!(env!("OUT_DIR"), "/shaders.rs")); -// A block of rows the GPU evaluates is a page of every column. -const _: () = assert!(BLOCK_ROWS as usize == crate::data::PAGE_ROWS && 1 << PAGE_BITS == BLOCK_ROWS); +// Data, draw order, and mark evaluation all use the same number of rows per page. +const _: () = + assert!(1 << PAGE_BITS == PAGE_ROWS && PAGE_MASK + 1 == PAGE_ROWS && PAGE_ROWS as usize == crate::data::PAGE_ROWS); diff --git a/crates/latkit/src/gpu/work.rs b/crates/latkit/src/gpu/work.rs index e14cee58..af0ffba4 100644 --- a/crates/latkit/src/gpu/work.rs +++ b/crates/latkit/src/gpu/work.rs @@ -1,17 +1,21 @@ -//! What a `Gpu` did, counted exactly. Unlike time, it repeats from run to run and machine to machine -//! wherever the budget, the device's limits, and the frames drawn do, so the benchmark gates it and -//! tests assert it. Every write to the GPU goes through [`Queue`], which counts it; clippy refuses the -//! raw queue's writes anywhere else. Whatever else counts holds the counters itself: the pipelines, -//! the readback pool, the atlas, and each frame's jobs and evaluations through the `Gpu`. +//! Exact counts of the work a `Gpu` does. Unlike timings, the counts repeat across runs and +//! machines for the same budget, device limits, and frames drawn. The benchmark gates on them and +//! tests assert them. +//! +//! Every write to the GPU goes through [`Queue`], which counts it. Clippy forbids the raw queue's +//! writes anywhere else. The pipelines, the readback pool, and the atlas hold the counters +//! themselves, and views count each frame's jobs and evaluations through the `Gpu`. use std::ops::{Deref, Sub}; use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering::Relaxed}; -/// [`Work`], the [`Count`] naming each of its fields, and the [`Counters`] keeping them, from one list. +/// Defines [`Work`], a [`Count`] variant for each of its fields, and the [`Counters`] that hold +/// them, from one list. macro_rules! work { ($($(#[$doc:meta])* $field:ident: $count:ident,)*) => { - /// What a [`Gpu`](crate::Gpu) did since it was made; subtract two for what happened between. + /// Counts of the work a [`Gpu`](crate::Gpu) has done since it was created. Subtract two + /// snapshots to get the work between them. #[derive(Clone, Copy, Default, PartialEq, Eq, Debug, serde::Serialize, schemars::JsonSchema)] pub struct Work { $($(#[$doc])* pub $field: u64,)* @@ -23,7 +27,7 @@ macro_rules! work { $($count,)* } - /// The counts, shared by every view of a `Gpu`, from any thread. + /// The counters, shared by every view of a `Gpu` and updated from any thread. #[derive(Default)] pub(crate) struct Counters { $($field: AtomicU64,)* @@ -55,28 +59,41 @@ macro_rules! work { work! { /// Bytes of data pages uploaded. uploaded: Uploaded, - /// Bytes written besides pages: uniforms, programs, label runs, the selection, textures. + /// Data pages, or pieces of pages, uploaded. A change's uploads count pages here and bytes in + /// `uploaded`. + uploads: Uploads, + /// Bytes written other than pages: uniforms, programs, label runs, the selection, and textures. written: Written, /// Rows whose channels the GPU evaluated. evaluated: Evaluated, - /// Reads back from the GPU: probes, queries, fits, label placements, segment counts, pictures; - /// not the frame timer's, which only some devices have. + /// Edge and face rows whose endpoints the GPU resolved to resident vertex rows. These are rows + /// that changed, plus every row of a page that references a vertex now resident elsewhere. + resolved: Resolved, + /// Bytes of per-row word pages the GPU cleared: pages read before anything writes them, and the + /// normals that faces sum from scratch. + cleared: Cleared, + /// Rows packed into pages of a paged table's draw order. Each page is packed once, the first + /// time a view draws it, and again only when a write changes its rows. + packed: Packed, + /// Readbacks from the GPU: probes, queries, fits, label texts, segment counts, and pictures. + /// Frame timer readbacks, which only some devices support, are not counted. readbacks: Readbacks, - /// Jobs handed out: layouts, routes, lanes, triangulations, pyramids, tiles; to the host's - /// workers, or done at once without a host. + /// Jobs started: layouts, routes, lanes, triangulations, pyramids, and tiles. They run on the + /// host's workers, or immediately without a host. jobs: Jobs, - /// Label texts the host was asked to set. + /// Label texts the host was asked to rasterize. texts: Texts, - /// Pipelines compiled: in a browser, each as a view or a shot asks for it ahead, off the GPU's - /// thread; natively, where it is first made. + /// Pipelines compiled. In a browser, each compiles in the background when a view or shot + /// requests it ahead of use. Natively, each compiles when first created. pipelines: Pipelines, - /// Pipelines made without being asked for ahead, each compiled where it is first used: in a - /// browser on the GPU's thread, holding frames back. None, unless a scene uses what it did not - /// ask for. + /// Pipelines not requested ahead of use, compiled when first used. In a browser they compile + /// on the GPU's thread and delay frames. Zero unless a scene uses a pipeline it did not + /// request. late: Late, } -/// The device's queue, counting what is written through it; everything else it does as the queue. +/// The device's queue, which counts what is written through it. Everything else passes through to +/// the queue. pub(crate) struct Queue { raw: wgpu::Queue, counters: Arc, @@ -94,7 +111,9 @@ impl Queue { self.raw.write_buffer(buffer, offset, bytes); } - /// Writes in command order, so a reused scratch buffer keeps earlier passes' inputs intact. + /// Writes `bytes` into `buffer` at `offset` through a copy recorded in `encoder`, counted as + /// written. The write lands in command order, so a reused scratch buffer keeps earlier passes' + /// inputs intact, and the write takes effect after any copy recorded before it. pub(crate) fn write_buffer_in( &self, device: &wgpu::Device, @@ -113,10 +132,24 @@ impl Queue { encoder.copy_buffer_to_buffer(&upload, 0, buffer, offset, bytes.len() as u64); } - /// Writes a data page's words into `buffer` at `offset`, counted as uploaded. + /// Clears `size` bytes of `buffer` from `offset` in `encoder`, counted as cleared. + pub(crate) fn clear_buffer( + &self, + encoder: &mut wgpu::CommandEncoder, + buffer: &wgpu::Buffer, + offset: u64, + size: u64, + ) { + self.counters.add(Count::Cleared, size); + encoder.clear_buffer(buffer, offset, Some(size)); + } + + /// Writes `bytes` into `buffer` at `offset`, counted as uploaded. The bytes are the words of + /// `pages` adjacent data pages or pieces of pages. #[expect(clippy::disallowed_methods, reason = "the counted upload")] - pub(crate) fn upload(&self, buffer: &wgpu::Buffer, offset: u64, bytes: &[u8]) { + pub(crate) fn upload(&self, buffer: &wgpu::Buffer, offset: u64, bytes: &[u8], pages: u64) { self.counters.add(Count::Uploaded, bytes.len() as u64); + self.counters.add(Count::Uploads, pages); self.raw.write_buffer(buffer, offset, bytes); } diff --git a/crates/latkit/src/host/browser.rs b/crates/latkit/src/host/browser.rs index d0c04fcd..4e4833e4 100644 --- a/crates/latkit/src/host/browser.rs +++ b/crates/latkit/src/host/browser.rs @@ -3,43 +3,99 @@ use std::cell::RefCell; use wasm_bindgen::{JsCast, JsValue}; use web_sys::{OffscreenCanvas, OffscreenCanvasRenderingContext2d}; +use super::queue::Queue; use super::{Encoder, Unit}; use super::{Host, Job, Raster, Request, TextRun}; use crate::Encoding; use crate::{Error, Pending}; -/// The browser: text set with a 2D canvas in the page's fonts, jobs done by the page's workers once -/// [`set_jobs`](Browser::set_jobs) names them, else at once, bytes fetched with `fetch`, and video -/// encoded by WebCodecs once [`set_encoder`](Browser::set_encoder) names how. +/// The browser host. Text is rasterized with a 2D canvas in the page's fonts, and bytes are fetched +/// with `fetch`. Jobs run on the page's workers once [`set_jobs`](Browser::set_jobs) provides them, +/// else immediately. Video is encoded with WebCodecs once [`set_encoder`](Browser::set_encoder) +/// provides an encoder. #[derive(Clone, Copy, Debug, Default)] pub struct Browser; thread_local! { static CANVAS: RefCell> = const { RefCell::new(None) }; static JOBS: RefCell> = const { RefCell::new(None) }; + static QUEUE: RefCell = RefCell::new(Queue::default()); static ENCODERS: RefCell> = const { RefCell::new(None) }; } +/// How many jobs the page's workers run at once: one fewer than the number of cores, from 1 to 4. +/// The page keeps a worker for each, so it never queues. +fn workers() -> usize { + let cores = web_sys::window() + .map(|window| window.navigator().hardware_concurrency()) + .or_else(|| { + js_sys::global() + .dyn_into::() + .ok() + .map(|scope| scope.navigator().hardware_concurrency()) + }) + .unwrap_or(2.0) as usize; + cores.saturating_sub(1).clamp(1, 4) +} + +/// The latkit error for a value the page's script threw or rejected with. An error whose `code` is +/// `unsupported` or `timeout` keeps that code. Anything else becomes an [`Error::Io`] saying `what` +/// failed. +fn failure(error: &JsValue, what: &str) -> Error { + let field = |key: &str| js_sys::Reflect::get(error, &key.into()).ok().and_then(|v| v.as_string()); + let message = field("message").or_else(|| error.as_string()).unwrap_or_default(); + match field("code").as_deref() { + Some("unsupported") => Error::Unsupported(message.into()), + Some("timeout") => Error::Timeout, + _ => Error::Io(format!("{what}: {message}").into()), + } +} + +/// Hands waiting jobs to the page's workers while any is free, skipping cancelled jobs. +fn pump(run: &js_sys::Function) { + loop { + let next = QUEUE.with(|queue| queue.borrow_mut().next(workers())); + let Some((job, resolver)) = next else { return }; + let bytes = js_sys::Uint8Array::from(&job.into_bytes()[..]); + let promise = run.call1(&JsValue::NULL, &bytes).map(js_sys::Promise::from); + let run = run.clone(); + wasm_bindgen_futures::spawn_local(async move { + let answer = match promise { + Ok(promise) => wasm_bindgen_futures::JsFuture::from(promise) + .await + .map(|bytes| js_sys::Uint8Array::new(&bytes).to_vec()) + .map_err(|error| failure(&error, "a job failed")), + Err(error) => Err(failure(&error, "a job failed")), + }; + resolver.resolve(answer); + QUEUE.with(|queue| queue.borrow_mut().done()); + pump(&run); + }); + } +} + impl Browser { - /// Has views' jobs done by `run`, which takes a job's bytes as a `Uint8Array` and returns a - /// promise of its answer's: a pool of workers, each holding the page's compiled module. + /// Runs views' jobs with `run`, which takes a job's bytes as a `Uint8Array` and returns a + /// promise of its result's bytes. `run` should pass jobs to workers that each hold the page's + /// compiled module. Jobs queue here, and `run` gets one only when a worker is free. pub fn set_jobs(run: js_sys::Function) { JOBS.with(|jobs| *jobs.borrow_mut() = Some(run)); } - /// Has videos encoded by what `create` makes of `(width, height, fps, bitrate)`: an object whose - /// `close()` releases resources; `encode(nv12, time, key)` and `finish()` return promises of the units finished since, each - /// `[time, key, bytes]`, in order. The npm package makes one with WebCodecs. + /// Encodes videos with encoders made by `create(width, height, fps, bitrate)`. An encoder's + /// `encode(nv12, time, key)` and `finish()` return promises of the units finished since the + /// last call, in order, each `[time, key, bytes]`. Its `close()` releases its resources. The + /// npm package provides one built on WebCodecs. pub fn set_encoder(create: js_sys::Function) { ENCODERS.with(|encoders| *encoders.borrow_mut() = Some(create)); } } -/// An encoder the page made: its `encode` and `finish` answer promises of units. +/// An encoder made by the page. Its `encode` and `finish` return promises of units. struct Script(JsValue); impl Script { - /// Calls method `name` with `args`, resolving to the units its promise answers. + /// Calls method `name` with `args`, resolving to the units its promise returns. fn call(&self, name: &str, args: &js_sys::Array) -> Pending> { let promise = js_sys::Reflect::get(&self.0, &name.into()) .and_then(JsCast::dyn_into::) @@ -47,31 +103,27 @@ impl Script { .map(|answer| js_sys::Promise::resolve(&answer)); let promise = match promise { Ok(promise) => promise, - Err(error) => return Pending::ready(Err(Error::Unsupported(format!("the encoder: {error:?}").into()))), + Err(error) => return Pending::ready(Err(failure(&error, "the encoder failed"))), }; let (pending, resolver) = Pending::new(); wasm_bindgen_futures::spawn_local(async move { let answer = wasm_bindgen_futures::JsFuture::from(promise).await; - resolver.resolve( - answer - .map_err(|error| Error::Unsupported(format!("the encoder failed: {error:?}").into())) - .and_then(units), - ); + resolver.resolve(answer.map_err(|error| failure(&error, "the encoder failed")).and_then(units)); }); pending } } -/// The units an encoder answered: `[time, key, bytes]` each. +/// Reads the units an encoder returned, each `[time, key, bytes]`. fn units(answer: JsValue) -> crate::Result> { let units: js_sys::Array = - answer.dyn_into().map_err(|_| Error::Unsupported("the encoder answered no units".into()))?; + answer.dyn_into().map_err(|_| Error::Io("the encoder did not return an array of units".into()))?; units .iter() .map(|unit| { let unit: js_sys::Array = - unit.dyn_into().map_err(|_| Error::Unsupported("a unit is [time, key, bytes]".into()))?; - let time = unit.get(0).as_f64().ok_or_else(|| Error::Unsupported("a unit's time".into()))?; + unit.dyn_into().map_err(|_| Error::Io("an encoder unit is not an array [time, key, bytes]".into()))?; + let time = unit.get(0).as_f64().ok_or_else(|| Error::Io("an encoder unit has no numeric time".into()))?; let bytes = js_sys::Uint8Array::new(&unit.get(2)); Ok(Unit { time: time as u64, key: unit.get(1).is_truthy(), annex_b: bytes.to_vec() }) }) @@ -98,10 +150,11 @@ impl Encoder for Script { } } -/// Pixels of room around each run, so antialiasing and overhangs are kept. +/// Padding in pixels around each run, so antialiasing and overhangs are not clipped. const PAD: f64 = 2.0; -/// `text` broken at spaces into lines no wider than `max_width`, each with its width. +/// Breaks `text` at spaces into lines no wider than `max_width`, each with its width. A word wider +/// than `max_width` gets a line of its own. fn lines( context: &OffscreenCanvasRenderingContext2d, text: &str, @@ -125,13 +178,13 @@ fn lines( Some(lines) } -/// The page's 2D context for setting and measuring text, made on first use. +/// Calls `f` with the 2D context that rasterizes and measures text, created on first use. fn with_context(f: impl FnOnce(&OffscreenCanvas, &OffscreenCanvasRenderingContext2d) -> Option) -> Option { CANVAS.with(|cell| { let mut cell = cell.borrow_mut(); if cell.is_none() { let canvas = OffscreenCanvas::new(256, 64).ok()?; - // Each run is read back once it is set. + // Each run is read back after drawing, so ask for a canvas suited to readback. let options = js_sys::Object::new(); js_sys::Reflect::set(&options, &"willReadFrequently".into(), &true.into()).ok()?; let context = canvas.get_context_with_context_options("2d", &options).ok()??; @@ -143,7 +196,7 @@ fn with_context(f: impl FnOnce(&OffscreenCanvas, &OffscreenCanvasRenderingCon }) } -/// A run laid out: its lines, each with its width, and its ascent and line height. +/// A run's layout: its lines with their widths, its ascent, and its line height. struct Layout { lines: Vec<(String, f64)>, ascent: f64, @@ -151,7 +204,7 @@ struct Layout { } impl Layout { - /// `run` laid out in `context`, set to its font. + /// Lays out `run` in `context`, setting the context's font to the run's. fn of(context: &OffscreenCanvasRenderingContext2d, run: &TextRun) -> Option { context.set_font(&format!("{}px {}", run.size_px, run.font)); let lines = lines(context, &run.text, run.max_width.map(f64::from))?; @@ -160,7 +213,7 @@ impl Layout { Some(Layout { lines, ascent, line_height: ascent + metrics.font_bounding_box_descent() }) } - /// Its width and height, with room around it. + /// Its width and height, including padding. fn size(&self) -> [f64; 2] { let widest = self.lines.iter().map(|(_, w)| *w).fold(0.0, f64::max); [widest + 2.0 * PAD, self.line_height * self.lines.len() as f64 + 2.0 * PAD] @@ -175,20 +228,9 @@ impl Host for Browser { fn spawn(&self, job: Job) -> Pending> { let Some(run) = JOBS.with(|jobs| jobs.borrow().clone()) else { return Pending::ready(Ok(job.run())) }; - let bytes = js_sys::Uint8Array::from(&job.into_bytes()[..]); - let promise = match run.call1(&JsValue::NULL, &bytes).map(js_sys::Promise::from) { - Ok(promise) => promise, - Err(error) => return Pending::ready(Err(Error::Unsupported(format!("jobs: {error:?}").into()))), - }; let (pending, resolver) = Pending::new(); - wasm_bindgen_futures::spawn_local(async move { - let answer = wasm_bindgen_futures::JsFuture::from(promise).await; - resolver.resolve( - answer - .map(|bytes| js_sys::Uint8Array::new(&bytes).to_vec()) - .map_err(|error| Error::Unsupported(format!("a job failed: {error:?}").into())), - ); - }); + QUEUE.with(|queue| queue.borrow_mut().push(job, resolver)); + pump(&run); pending } @@ -209,7 +251,7 @@ impl Host for Browser { canvas.set_width(canvas.width().max(size[0].next_power_of_two())); canvas.set_height(canvas.height().max(size[1].next_power_of_two())); } - // Resizing resets the context. + // Resizing the canvas resets the context's state, so set it again. context.set_font(&format!("{}px {}", run.size_px, run.font)); context.set_text_baseline("alphabetic"); context.set_fill_style_str("#fff"); @@ -226,7 +268,7 @@ impl Host for Browser { fn encoder(&self, encoding: &Encoding) -> crate::Result> { let unsupported = - || Error::Unsupported("videos in a browser encode with WebCodecs, which the page did not give".into()); + || Error::Unsupported("browser videos need a WebCodecs encoder, and the page provided none".into()); let create = ENCODERS.with(|encoders| encoders.borrow().clone()).ok_or_else(unsupported)?; let [width, height] = encoding.size; let args = js_sys::Array::of4( @@ -235,15 +277,14 @@ impl Host for Browser { &encoding.fps.into(), &f64::from(encoding.bitrate).into(), ); - let encoder = create - .apply(&JsValue::NULL, &args) - .map_err(|error| Error::Unsupported(format!("no encoder for {width}×{height}: {error:?}").into()))?; + let encoder = + create.apply(&JsValue::NULL, &args).map_err(|error| failure(&error, "the encoder failed to start"))?; Ok(Box::new(Script(encoder))) } } -/// The bytes `request` asks for: none where there is nothing at its URL, and only the range asked for -/// where a server sends every byte. +/// Fetches the bytes of `request`. Returns no bytes on status 204 or 404. When a server ignores the +/// requested range and sends every byte, slices out the range. async fn fetched(request: &Request) -> crate::Result> { let failed = |what: &str| Error::Io(format!("{} could not be read: {what}", request.url).into()); let init = web_sys::RequestInit::new(); @@ -254,19 +295,19 @@ async fn fetched(request: &Request) -> crate::Result> { } init.set_headers(&headers); let fetch = web_sys::Request::new_with_str_and_init(&request.url, &init).map_err(|e| failed(&format!("{e:?}")))?; - // The page's fetch, or a worker's. + // Use the window's fetch, or the worker's. let global = js_sys::global(); let promise = match (global.dyn_ref::(), global.dyn_ref::()) { (Some(window), _) => window.fetch_with_request(&fetch), (None, Some(worker)) => worker.fetch_with_request(&fetch), - (None, None) => return Err(failed("no fetch here")), + (None, None) => return Err(failed("no fetch in this context")), }; let response = wasm_bindgen_futures::JsFuture::from(promise).await.map_err(|e| failed(&format!("{e:?}")))?; let response: web_sys::Response = response.dyn_into().map_err(|_| failed("not a response"))?; match response.status() { 200 | 206 => {} 204 | 404 => return Ok(Vec::new()), - status => return Err(failed(&format!("status {status}"))), + status => return Err(failed(&format!("HTTP status {status}"))), } let body = response.array_buffer().map_err(|e| failed(&format!("{e:?}")))?; let body = wasm_bindgen_futures::JsFuture::from(body).await.map_err(|e| failed(&format!("{e:?}")))?; diff --git a/crates/latkit/src/host/job.rs b/crates/latkit/src/host/job.rs index e34c7fb4..e9687725 100644 --- a/crates/latkit/src/host/job.rs +++ b/crates/latkit/src/host/job.rs @@ -1,52 +1,59 @@ -//! Jobs: work too large for the frame (laying out in ranks or by stress, routing wires, cutting areas -//! into triangles, grouping a large table's rows into tiles, writing a picture as a file), handed to the -//! host to do off the frame. A job is bytes in and bytes out, so a thread, or a worker holding its own -//! copy of latkit, can do it with nothing shared. A view's job answers with a [`Patch`]: the rows it -//! worked out, by id, with their places, the paths their wires run along, or their shapes; an image -//! job answers with the file. +//! Jobs: work too large for a frame, handed to the host to run in the background. Jobs lay out +//! graphs (layered or stress), route wires, triangulate areas, assign edge lanes, group a large +//! table's rows into tiles, decode vector tiles and GeoJSON, and encode images. +//! +//! A job is bytes in and bytes out, so a thread, or a worker with its own copy of latkit, can run +//! it with nothing shared. Layout, route, tile, and GeoJSON jobs return a [`Patch`] of rows by id. +//! Other jobs return their own format: lane words, shape columns, each table's pyramid (order and +//! tiles), or an encoded image file. use std::sync::Arc; use crate::Side; -use crate::data::{Field, GeometryBuilder, Id, Name, Op, Patch, Table, Value, Values, Version}; +use crate::data::{GeometryBuilder, Id, Name, Patch, Table, Value, Values, read_values, values_bytes}; use crate::derive::pyramid::{Leveling, Places, Placing}; use crate::derive::sight::Domain; use crate::geo::mvt::Kind as Geometry; use crate::geo::shapes::{earth_parts, triangles}; use crate::geo::tiles::{Decoded, decode}; +use crate::layout::boxes::Boxes; use crate::layout::route::{Net, Terminal, route}; use crate::layout::{layered::layered, levels::Graph, stress::stress}; use crate::tiles::TileKey; +use crate::wire::{In, Out, codes}; use crate::{Direction, Layered, Layout}; -/// Work latkit hands its [`Host`](super::Host) to do off the frame: bytes in, bytes out. +/// Work latkit hands its [`Host`](super::Host) to run in the background: bytes in, bytes out. /// -/// A host that does jobs elsewhere sends [`into_bytes`](Job::into_bytes) there, and answers with -/// what [`run`](Job::run) makes of [`from_bytes`](Job::from_bytes) of them. +/// A host that runs jobs elsewhere sends the job's [`into_bytes`](Job::into_bytes) there, rebuilds +/// it with [`from_bytes`](Job::from_bytes), and returns the result of [`run`](Job::run). #[derive(Clone, PartialEq, Eq)] pub struct Job(Vec); -/// What a job does, its first word. +/// A job's type, stored as its first word. #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Kind { - Layered = 1, - Stress = 2, - Routes = 3, - Shapes = 4, - Tile = 5, - GeoJson = 6, - Lanes = 7, - Pyramid = 8, - Image = 9, + Layered, + Stress, + Routes, + Shapes, + Tile, + GeoJson, + Lanes, + Pyramid, + Image, } -/// What a shapes job works out of each row's geometry. +codes!(Kind { Layered = 1, Stress = 2, Routes = 3, Shapes = 4, Tile = 5, GeoJson = 6, Lanes = 7, Pyramid = 8, Image = 9 }); + +/// The shapes computed from each row's geometry. #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] pub(crate) enum Shaping { - /// An area's triangles, the point its label hangs from first; on the earth, unwrapped and split - /// along its grid. + /// An area's triangles, after its label anchor. With `earth`, rings are unwrapped and + /// triangles split along the earth grid. Triangles { earth: bool }, - /// An earth line's path: unwrapped, and bent with the globe; cheap enough that no job makes it. + /// An earth line's path, unwrapped and given extra points to follow the globe. Cheap enough + /// to compute on the frame, so it never needs a shapes job. Path, } @@ -68,7 +75,7 @@ impl Shaping { } } - /// The depth of the shapes made from geometry of `depth`. + /// The geometry depth of shapes made from geometry of `depth`. pub(crate) fn depth(self, depth: u8) -> u8 { match self { Shaping::Triangles { .. } => 1, @@ -76,7 +83,41 @@ impl Shaping { } } - /// The shape of `value`, or `None` where it stays as it is: a path that needs no bending. + /// Whether geometry of `depth` gets this shaping. Triangles apply to parts or polygons, and + /// paths to anything but points. + pub(crate) fn takes(self, depth: u8) -> bool { + match self { + Shaping::Triangles { .. } => matches!(depth, 2 | 3), + Shaping::Path => depth >= 1, + } + } + + /// Name of the derived field holding the shapes of geometry field `field`. + pub(crate) fn field_of(self, field: &str) -> Name { + match self { + Shaping::Triangles { .. } => crate::scene::areas::triangles_of(field), + Shaping::Path => crate::scene::lines::path_of(field), + } + } + + /// Computes the shapes of every row of geometry `values`, in order. A missing row gets an empty + /// shape, and a path that needs no extra points is copied unchanged. + pub(crate) fn shapes_of(self, values: &Values) -> Values { + let depth = values.geometry().map_or(0, crate::data::Geometry::depth); + let mut shapes = GeometryBuilder::new(self.depth(depth)); + for row in 0..values.len() { + match values.get(row) { + Some(value) => { + let shaped = self.shape(&value); + shapes.push_value(shaped.as_ref().unwrap_or(&value)); + } + None => shapes.push_empty(), + } + } + Values::of_geometry(shapes.finish()) + } + + /// The shape of `value`, or `None` if it is unchanged (a path that needs no extra points). pub(crate) fn shape(self, value: &Value) -> Option { match (self, value) { (Shaping::Triangles { earth }, Value::Polygons(polygons)) => Some(Value::Path(triangles(polygons, earth))), @@ -105,12 +146,13 @@ impl Shaping { } } -/// A row a job answers for: its table and id. +/// A row in a job's result: its table and id. pub(crate) type Key = (Name, Id); -/// Vertices to place: each one's size, its center when it has one, its group and how far a group's -/// box reaches past its members (left, bottom, right, top), links between them, and the key of -/// each vertex the answer places; vertices without a key are only placed around. Plane units, y up. +/// Vertices to lay out: each one's size, pinned center if any, group, and how far a group's box +/// reaches past its members (left, bottom, right, top), plus the links between vertices. `keys` +/// names the row that receives each vertex's center. Vertices without a key are laid out but not +/// written. Plane units, y up. pub(crate) struct Placement<'a> { pub(crate) sizes: &'a [[f64; 2]], pub(crate) pinned: &'a [Option<[f64; 2]>], @@ -120,8 +162,9 @@ pub(crate) struct Placement<'a> { pub(crate) keys: &'a [Option], } -/// Wires to route: around `obstacles` and among `groups`, each net's terminals with the key of the row -/// it draws, runs beside `fixed` wires already routed, tracks `spacing` apart. Plane units, y up. +/// Wires to route around `obstacles` and among `groups`. `keys` names the row that receives each +/// net's route. Routes run beside the already routed `fixed` wires, on tracks `spacing` apart. +/// Plane units, y up. pub(crate) struct Wiring<'a> { pub(crate) obstacles: &'a [[f64; 4]], pub(crate) groups: &'a [[f64; 4]], @@ -132,7 +175,7 @@ pub(crate) struct Wiring<'a> { } impl Job { - /// A job from its bytes. + /// Rebuilds a job from its bytes. pub fn from_bytes(bytes: Vec) -> Job { Job(bytes) } @@ -142,142 +185,133 @@ impl Job { self.0 } - /// Does the job, answering with a patch's bytes. Bytes that are no job answer nothing. + /// Runs the job and returns its result: a patch of rows by id, lane words, shape columns, a + /// pyramid's order and tiles, or an encoded image file. Returns no bytes if the job is + /// malformed or fails. pub fn run(self) -> Vec { - let mut input = Reader::new(&self.0); - let kind = input.u32(); - if kind == Some(Kind::Image as u32) { - return run_image(&mut input).unwrap_or_default(); - } + let mut input = In::new(&self.0); + let Some(kind) = input.u32().and_then(Kind::from_code) else { return Vec::new() }; let answer = match kind { - Some(k) if k == Kind::Layered as u32 || k == Kind::Stress as u32 => run_layout(k, &mut input), - Some(k) if k == Kind::Routes as u32 => run_routes(&mut input), - Some(k) if k == Kind::Shapes as u32 => run_shapes(&mut input), - Some(k) if k == Kind::Tile as u32 => run_tile(&mut input), - Some(k) if k == Kind::GeoJson as u32 => run_geojson(&mut input), - Some(k) if k == Kind::Lanes as u32 => run_lanes(&mut input), - Some(k) if k == Kind::Pyramid as u32 => run_pyramid(&mut input), - _ => None, + Kind::Image => return run_image(&mut input).unwrap_or_default(), + Kind::Lanes => return run_lanes(&mut input).unwrap_or_default(), + Kind::Pyramid => return run_pyramid(&mut input).unwrap_or_default(), + Kind::Shapes => return run_shapes(&mut input).unwrap_or_default(), + Kind::Layered | Kind::Stress => run_layout(kind, &mut input), + Kind::Routes => run_routes(&mut input), + Kind::Tile => run_tile(&mut input), + Kind::GeoJson => run_geojson(&mut input), }; answer.map(|patch| patch.to_bytes()).unwrap_or_default() } - /// Placing `placement` by `layout`, which is layered or stress; it answers each keyed vertex's - /// center as fields `x` and `y`, y up. + /// A job laying out `placement` with `layout` (layered or stress). Its result sets each keyed + /// vertex's center as fields `x` and `y`, y up. pub(crate) fn layout(layout: &Layout, placement: &Placement<'_>) -> Job { let mut out = match *layout { Layout::Stress(stress) => { - let mut out = Writer::new(Kind::Stress); + let mut out = job(Kind::Stress); out.f64(stress.gap); out } Layout::Layered(layered) => { - let mut out = Writer::new(Kind::Layered); - out.u32(match layered.flow { - Direction::Right => 0, - Direction::Left => 1, - Direction::Down => 2, - Direction::Up => 3, - }); + let mut out = job(Kind::Layered); + out.u32(layered.direction.code()); out.f64(layered.gap); out.f64(layered.rank_gap); out } - Layout::None => unreachable!("a sheet asks for a layout only when its scene has one"), + Layout::None => unreachable!("a sheet requests a layout only when its scene has one"), }; - out.u32(placement.sizes.len() as u32); + out.len(placement.sizes.len()); for v in 0..placement.sizes.len() { - out.f64s(&placement.sizes[v]); + out.pod(&placement.sizes[v]); out.u32(u32::from(placement.pinned[v].is_some())); - out.f64s(&placement.pinned[v].unwrap_or_default()); + out.pod(&placement.pinned[v].unwrap_or_default()); out.u32(placement.groups[v].unwrap_or(u32::MAX)); - out.f64s(&placement.pads[v]); + out.pod(&placement.pads[v]); out.u32(u32::from(placement.keys[v].is_some())); if let Some(key) = &placement.keys[v] { - out.key(key); + write_key(&mut out, key); } } - out.u32(placement.links.len() as u32); - for link in placement.links { - out.u32(link[0]); - out.u32(link[1]); - } + out.len(placement.links.len()); + out.pod(placement.links); Job(out.0) } - /// Working out `shaping` for every row of `rows`, whose field `geometry` holds each row's - /// geometry; it answers each row's shape as field `shape`. - pub(crate) fn shapes(shaping: Shaping, rows: Table) -> Job { - let mut out = Writer::new(Kind::Shapes); + /// A job computing `shaping` for every row of each geometry column in `parts`. Its result holds + /// each part's shapes, read by [`shapes_answer`]. + pub(crate) fn shapes(shaping: Shaping, parts: &[Values]) -> Job { + let mut out = job(Kind::Shapes); out.u32(shaping.code()); - out.0.extend(Patch::new().set(rows).to_bytes()); + write_columns(&mut out, parts); Job(out.0) } - /// Decoding tile `key` from `bytes`, gzip-compressed or not, into the rows of `tables`; it answers - /// those rows. + /// A job decoding tile `key` from `bytes`, optionally gzip-compressed, into rows of `tables` + /// with their derived shapes. Its result is a patch of those rows. pub(crate) fn tile(key: TileKey, tables: &[Decoded], bytes: &[u8]) -> Job { - let mut out = Writer::new(Kind::Tile); + let mut out = job(Kind::Tile); out.u32(u32::from(key.z)); out.u32(key.x); out.u32(key.y); - out.u32(tables.len() as u32); + out.len(tables.len()); for decoded in tables { - out.text(&decoded.layer); - out.u32(match decoded.kind { - Geometry::Points => 0, - Geometry::Lines => 1, - Geometry::Polygons => 2, - }); - out.text(&decoded.table); + out.str(&decoded.layer); + out.u32(decoded.kind.code()); + out.str(&decoded.table); + out.len(decoded.shapes.len()); + for shaping in &decoded.shapes { + out.u32(shaping.code()); + } } - out.0.extend_from_slice(bytes); + out.bytes(bytes); Job(out.0) } - /// The job writing `image` as a file of `format`, which it answers with; nothing where it - /// cannot be written. + /// A job encoding `image` as a file of `format`. Its result is the file's bytes, or no bytes if + /// the image cannot be encoded. pub fn encode(image: &crate::Image, format: crate::ImageFormat) -> Job { - let mut out = Writer::new(Kind::Image); - out.u32(match format { - crate::ImageFormat::Png => 0, - crate::ImageFormat::Jpeg { quality } => 0x100 | u32::from(quality), - crate::ImageFormat::WebP => 1, - }); + let mut out = job(Kind::Image); + out.u32(format.code()); out.u32(image.size[0]); out.u32(image.size[1]); - out.0.extend_from_slice(&image.rgba); + out.bytes(&image.rgba); Job(out.0) } - /// Reading GeoJSON `bytes` as dataset `name`, as [`Data::read`](crate::Data::read) does; it - /// answers the patch that builds the dataset, or nothing when the bytes are no GeoJSON. - pub fn geojson(name: &str, bytes: &[u8]) -> Job { - let mut out = Writer::new(Kind::GeoJson); - out.text(name); - out.0.extend_from_slice(bytes); - Job(out.0) + /// A job that reads GeoJSON `bytes` as dataset `name` in the background, like + /// [`Data::read`](crate::Data::read), since GeoJSON can be large. `None` if `bytes` are + /// latkit's own bytes or a dataset document, which are read on the calling thread. Its result + /// is the patch that builds the dataset, or no bytes if the GeoJSON is invalid. + pub fn geojson(name: &str, bytes: &[u8]) -> Option { + if !crate::data::geojson(bytes) { + return None; + } + let mut out = job(Kind::GeoJson); + out.str(name); + out.bytes(bytes); + Some(Job(out.0)) } - /// Finding every edge row's lane among the rows joining the same two vertices, from each row's two - /// ends; it answers a table of every row's lane as field `lane`, without rows when none share. + /// A job assigning each edge row a lane among the rows joining the same two vertices, from the + /// rows' endpoints `from` and `to`. Its result is every row's lane word as a little-endian + /// `u32`, or no bytes when no two rows share endpoints. pub(crate) fn lanes(from: &[u32], to: &[u32]) -> Job { - let mut out = Writer::new(Kind::Lanes); - out.u32(from.len() as u32); - out.words(from); - out.words(to); + let mut out = job(Kind::Lanes); + out.len(from.len()); + out.pod(from); + out.pod(to); Job(out.0) } - /// Grouping `leveling`'s tables into tiles; it answers each table's pyramid, as - /// [`Pyramid::read`](crate::derive::pyramid::Pyramid::read) reads it. + /// A job grouping the rows of `leveling`'s tables into tiles, all of one dataset's tables at + /// once. Its result holds each table's pyramid, read by + /// [`Pyramid::read`](crate::derive::pyramid::Pyramid::read). pub(crate) fn pyramid(leveling: &Leveling) -> Job { - let mut out = Writer::new(Kind::Pyramid); - match leveling.domain { - Domain::Earth => out.f64s(&[0.0, 0.0, 0.0, 0.0]), - Domain::Plane { min, size } => out.f64s(&[1.0, min[0], min[1], size]), - } - out.u32(leveling.tables.len() as u32); + let mut out = job(Kind::Pyramid); + out.pod(&leveling.domain.words()); + out.len(leveling.tables.len()); for table in &leveling.tables { let places = match &table.places { Places::None => 0, @@ -293,44 +327,40 @@ impl Job { out.u32(flags); match &table.places { Places::None => {} - Places::Points(points) => out.words(bytemuck::cast_slice(&points[..])), - Places::Boxes(boxes) => out.words(bytemuck::cast_slice(&boxes[..])), + Places::Points(points) => out.pod(&points[..]), + Places::Boxes(boxes) => out.pod(&boxes[..]), } - out.words(bytemuck::cast_slice(&table.priority)); + out.pod(&table.priority[..]); if let Some((target, ends)) = &table.ends { out.u32(*target); - out.u32(ends.len() as u32); + out.len(ends.len()); for end in ends { - out.words(end); + out.pod(&end[..]); } } } Job(out.0) } - /// Routing `wiring`; it answers each keyed row's wire as field `route`, its polylines. + /// A job routing `wiring`. Its result sets each keyed row's wire polylines as field `route`. pub(crate) fn routes(wiring: &Wiring<'_>) -> Job { - let mut out = Writer::new(Kind::Routes); + let mut out = job(Kind::Routes); out.f64(wiring.spacing); for boxes in [wiring.obstacles, wiring.groups] { - out.u32(boxes.len() as u32); - for b in boxes { - out.f64s(b); - } + out.len(boxes.len()); + out.pod(boxes); } - out.u32(wiring.fixed.len() as u32); + out.len(wiring.fixed.len()); for line in wiring.fixed { - out.u32(line.len() as u32); - for point in line { - out.f64s(point); - } + out.len(line.len()); + out.pod(line.as_slice()); } - out.u32(wiring.nets.len() as u32); + out.len(wiring.nets.len()); for (net, key) in wiring.nets.iter().zip(wiring.keys) { - out.key(key); - out.u32(net.terminals.len() as u32); + write_key(&mut out, key); + out.len(net.terminals.len()); for terminal in &net.terminals { - out.f64s(&terminal.at); + out.pod(&terminal.at); out.u32(terminal.side.map_or(u32::MAX, Side::code)); out.u32(terminal.owner.unwrap_or(u32::MAX)); } @@ -339,13 +369,20 @@ impl Job { } } +/// A job's bytes so far: its kind. +fn job(kind: Kind) -> Out { + let mut out = Out::default(); + out.u32(kind.code()); + out +} + impl std::fmt::Debug for Job { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_tuple("Job").field(&self.0.len()).finish() } } -/// A patch writing `values` for `keys`, one table at a time in the order the keys name them. +/// Builds a patch writing `values` for `keys`, grouped by table in order of first appearance. fn keyed(keys: &[Key], values: Vec, write: impl Fn(Table, Vec) -> Table) -> Patch { let mut tables: Vec<(Name, Vec, Vec)> = Vec::new(); for ((table, id), value) in keys.iter().zip(values) { @@ -360,17 +397,12 @@ fn keyed(keys: &[Key], values: Vec, write: impl Fn(Table, Vec) -> Table tables.into_iter().fold(Patch::new(), |patch, (name, ids, values)| patch.set(write(Table::new(name, ids), values))) } -fn run_layout(kind: u32, input: &mut Reader<'_>) -> Option { - let options = if kind == Kind::Stress as u32 { +fn run_layout(kind: Kind, input: &mut In<'_>) -> Option { + let options = if kind == Kind::Stress { Layout::Stress(crate::Stress { gap: input.f64()? }) } else { - let flow = match input.u32()? { - 0 => Direction::Right, - 1 => Direction::Left, - 2 => Direction::Down, - _ => Direction::Up, - }; - Layout::Layered(Layered { flow, gap: input.f64()?, rank_gap: input.f64()? }) + let direction = Direction::from_code(input.u32()?)?; + Layout::Layered(Layered { direction, gap: input.f64()?, rank_gap: input.f64()? }) }; let count = input.u32()? as usize; let (mut sizes, mut pinned, mut groups, mut pads, mut keys) = @@ -383,7 +415,7 @@ fn run_layout(kind: u32, input: &mut Reader<'_>) -> Option { groups.push(Some(input.u32()?).filter(|&g| (g as usize) < count)); pads.push([input.f64()?, input.f64()?, input.f64()?, input.f64()?]); let keyed = input.u32()? != 0; - keys.push(if keyed { Some(input.key()?) } else { None }); + keys.push(if keyed { Some(read_key(input)?) } else { None }); } let links = input.u32()?; let links = (0..links).map(|_| Some([input.u32()?, input.u32()?])).collect::>>()?; @@ -405,7 +437,7 @@ fn run_layout(kind: u32, input: &mut Reader<'_>) -> Option { })) } -fn run_routes(input: &mut Reader<'_>) -> Option { +fn run_routes(input: &mut In<'_>) -> Option { let spacing = input.f64()?; let mut boxes = || { let count = input.u32()?; @@ -422,7 +454,7 @@ fn run_routes(input: &mut Reader<'_>) -> Option { let count = input.u32()?; let (mut nets, mut keys) = (Vec::new(), Vec::new()); for _ in 0..count { - keys.push(input.key()?); + keys.push(read_key(input)?); let terminals = input.u32()?; let terminals = (0..terminals) .map(|_| { @@ -434,69 +466,51 @@ fn run_routes(input: &mut Reader<'_>) -> Option { .collect::>>()?; nets.push(Net { terminals }); } - let routes = route(&obstacles, &groups, &nets, &fixed, spacing); + let routes = route(&Boxes::new(obstacles), &groups, &nets, &fixed, spacing); Some(keyed(&keys, routes, |table, routes| table.field("route", routes))) } -fn run_shapes(input: &mut Reader<'_>) -> Option { +/// Reads a shapes job's result: a column per part in the job. `None` if `bytes` are malformed. +pub(crate) fn shapes_answer(bytes: &[u8]) -> Option> { + read_columns(&mut In::new(bytes)) +} + +fn run_shapes(input: &mut In<'_>) -> Option> { let shaping = Shaping::from_code(input.u32()?)?; - let Op::Set(rows) = Patch::from_bytes(input.rest()).ok()?.ops.into_iter().next()? else { return None }; - let Some(Field::Values(column, _)) = rows.field_named("geometry") else { return None }; - let depth = column.kind().depth()?; - let mut shapes = GeometryBuilder::new(shaping.depth(depth)); - for row in 0..rows.len() { - match column.get(row) { - Some(value) => { - let shaped = shaping.shape(&value); - shapes.push_value(shaped.as_ref().unwrap_or(&value)); - } - None => shapes.push_empty(), - } - } - let shapes = Values::of_geometry(shapes.finish()); - let mut answer = rows.clone(); - answer.fields.clear(); - answer.insert("shape".into(), Field::Values(crate::data::Column::new(&shapes), Version::next())); - Some(Patch::new().set(answer)) + let parts = read_columns(input)?; + let mut out = Out::default(); + write_columns(&mut out, &parts.iter().map(|part| shaping.shapes_of(part)).collect::>()); + Some(out.0) } -fn run_tile(input: &mut Reader<'_>) -> Option { +fn run_tile(input: &mut In<'_>) -> Option { let key = TileKey { z: u8::try_from(input.u32()?).ok()?, x: input.u32()?, y: input.u32()? }; let count = input.u32()?; let tables = (0..count) .map(|_| { - let layer = input.text()?; - let kind = match input.u32()? { - 0 => Geometry::Points, - 1 => Geometry::Lines, - _ => Geometry::Polygons, - }; - Some(Decoded { layer, kind, table: input.text()?.into() }) + let layer = input.str()?.to_owned(); + let kind = Geometry::from_code(input.u32()?)?; + let table = input.str()?.into(); + let shapes = (0..input.u32()?).map(|_| Shaping::from_code(input.u32()?)).collect::>>()?; + Some(Decoded { layer, kind, table, shapes }) }) .collect::>>()?; decode(input.rest(), key, &tables) } -fn run_image(input: &mut Reader<'_>) -> Option> { - let format = match input.u32()? { - 0 => crate::ImageFormat::Png, - 1 => crate::ImageFormat::WebP, - jpeg => crate::ImageFormat::Jpeg { quality: (jpeg & 0xff) as u8 }, - }; +fn run_image(input: &mut In<'_>) -> Option> { + let format = crate::ImageFormat::from_code(input.u32()?)?; let size = [input.u32()?, input.u32()?]; crate::Image::new(size, input.rest()).ok()?.encode(format).ok() } -fn run_geojson(input: &mut Reader<'_>) -> Option { - let name = input.text()?; +fn run_geojson(input: &mut In<'_>) -> Option { + let name = input.str()?; crate::geo::geojson(name, input.rest()).ok().map(|data| data.to_patch()) } -fn run_pyramid(input: &mut Reader<'_>) -> Option { - let domain = match [input.f64()?, input.f64()?, input.f64()?, input.f64()?] { - [0.0, ..] => Domain::Earth, - [_, x, y, size] => Domain::Plane { min: [x, y], size }, - }; +fn run_pyramid(input: &mut In<'_>) -> Option> { + let domain = Domain::from_words(input.pod::(4)?.try_into().ok()?); let count = input.u32()?; let mut tables = Vec::with_capacity(count as usize); for _ in 0..count { @@ -505,135 +519,76 @@ fn run_pyramid(input: &mut Reader<'_>) -> Option { let n = rows as usize; let has = |bit: u32| flags & 1 << bit != 0; let places = match flags >> 2 & 3 { - 1 => Places::Points(input.words(2 * n)?.as_chunks::<2>().0.iter().map(|w| w.map(f32::from_bits)).collect()), - 2 => Places::Boxes(input.words(4 * n)?.as_chunks::<4>().0.iter().map(|w| w.map(f32::from_bits)).collect()), + 1 => Places::Points(input.pod::<[f32; 2]>(n)?.into()), + 2 => Places::Boxes(input.pod::<[f32; 4]>(n)?.into()), _ => Places::None, }; - let priority = if has(4) { input.words(n)?.into_iter().map(f32::from_bits).collect() } else { Arc::from([]) }; + let priority = if has(4) { input.pod::(n)?.into() } else { Arc::from([]) }; let ends = if has(5) { let target = input.u32()?; let count = input.u32()?; - Some((target, (0..count).map(|_| input.words(n).map(Arc::from)).collect::>>()?)) + Some((target, (0..count).map(|_| input.pod::(n).map(Arc::from)).collect::>>()?)) } else { None }; tables.push(Placing { rows, leveled: has(0), answered: has(1), places, priority, ends }); } - Some(crate::derive::pyramid::answer(&Leveling { domain, tables })) + Some(crate::derive::pyramid::encode(&crate::derive::pyramid::answer(&Leveling { domain, tables }))) } -fn run_lanes(input: &mut Reader<'_>) -> Option { +fn run_lanes(input: &mut In<'_>) -> Option> { let rows = input.u32()? as usize; - let (from, to) = (input.words(rows)?, input.words(rows)?); - Some(crate::derive::lanes_answer(&from, &to)) + let (from, to) = (input.pod::(rows)?, input.pod::(rows)?); + let words = crate::derive::lane_words(&from, &to)?; + let mut out = Out::default(); + out.pod(&words); + Some(out.0) } -/// Little-endian words, floats, and keys, written. -struct Writer(Vec); - -impl Writer { - fn new(kind: Kind) -> Writer { - let mut out = Writer(Vec::new()); - out.u32(kind as u32); - out - } - - fn u32(&mut self, v: u32) { - self.0.extend(v.to_le_bytes()); - } - - fn f64(&mut self, v: f64) { - self.0.extend(v.to_le_bytes()); +/// Writes columns, each as its byte length followed by its values' bytes. +fn write_columns(out: &mut Out, columns: &[Values]) { + out.len(columns.len()); + for values in columns { + let bytes = values_bytes(values); + out.len(bytes.len()); + out.bytes(&bytes); } +} - /// Words, copied whole where the machine is little-endian too. - fn words(&mut self, words: &[u32]) { - if cfg!(target_endian = "little") { - self.0.extend_from_slice(bytemuck::cast_slice(words)); - } else { - self.0.extend(words.iter().flat_map(|w| w.to_le_bytes())); - } - } +/// Reads columns written by [`write_columns`]. +fn read_columns(input: &mut In<'_>) -> Option> { + let count = input.u32()?; + (0..count) + .map(|_| { + let len = input.u32()? as usize; + read_values(input.bytes(len)?).ok() + }) + .collect() +} - fn f64s(&mut self, vs: &[f64]) { - for &v in vs { - self.f64(v); +/// Writes a key: 1 and a numeric id, or 2 and a text id, then the table name. +fn write_key(out: &mut Out, (table, id): &Key) { + match id { + Id::Number(n) => { + out.u32(1); + out.u64(*n); } - } - - fn text(&mut self, text: &str) { - self.u32(text.len() as u32); - self.0.extend(text.as_bytes()); - } - - /// A key: 1 then a numbered id, or 2 then a texted one; the table's name after. - fn key(&mut self, (table, id): &Key) { - match id { - Id::Number(n) => { - self.u32(1); - self.0.extend(n.to_le_bytes()); - } - Id::Text(t) => { - self.u32(2); - self.text(t); - } + Id::Text(text) => { + out.u32(2); + out.str(text); } - self.text(table); } + out.str(table); } -/// Little-endian words, floats, and keys, read; `None` past the end. -struct Reader<'a> { - bytes: &'a [u8], -} - -impl<'a> Reader<'a> { - fn new(bytes: &'a [u8]) -> Reader<'a> { - Reader { bytes } - } - - /// Every byte not read yet. - fn rest(&mut self) -> &'a [u8] { - std::mem::take(&mut self.bytes) - } - - fn take(&mut self) -> Option<[u8; N]> { - let (head, rest) = self.bytes.split_first_chunk::()?; - self.bytes = rest; - Some(*head) - } - - fn u32(&mut self) -> Option { - self.take().map(u32::from_le_bytes) - } - - fn f64(&mut self) -> Option { - self.take().map(f64::from_le_bytes) - } - - /// `n` words. - fn words(&mut self, n: usize) -> Option> { - let (head, rest) = self.bytes.split_at_checked(n.checked_mul(4)?)?; - self.bytes = rest; - Some(head.as_chunks::<4>().0.iter().map(|&w| u32::from_le_bytes(w)).collect()) - } - - fn text(&mut self) -> Option { - let len = self.u32()? as usize; - let (head, rest) = self.bytes.split_at_checked(len)?; - self.bytes = rest; - String::from_utf8(head.to_vec()).ok() - } - - /// A key as [`Writer::key`] wrote it. - fn key(&mut self) -> Option { - let id = match self.u32()? { - 1 => Id::Number(u64::from_le_bytes(self.take()?)), - 2 => Id::from(self.text()?), - _ => return None, - }; - Some((self.text()?.into(), id)) - } +/// Reads a key written by [`write_key`]. +fn read_key(input: &mut In<'_>) -> Option { + let id = match input.u32()? { + 1 => Id::Number(input.u64()?), + 2 => Id::from(input.str()?), + _ => return None, + }; + Some((input.str()?.into(), id)) } #[cfg(test)] @@ -645,15 +600,14 @@ mod tests { (table.into(), Id::Number(id)) } - /// Field `field` of `table`'s row `id` in the answer. + /// Field `field` of row `id` of `table` in a job's result. fn answered(answer: &[u8], table: &str, id: u64, field: &str) -> Option { - let mut data = crate::Data::new("d"); - data.apply(Patch::from_bytes(answer).ok()?).ok()?; - data.get(&crate::Row::new("d", table, id), field) + let data = crate::Snapshot::new("d").applied(Patch::from_bytes(answer).ok()?).ok()?; + data.get(table, id, field, None) } #[test] - fn layout_jobs_answer_the_rows_they_place() { + fn layout_jobs_return_the_rows_they_place() { let sizes = [[10.0, 10.0]; 3]; let pinned = [None, Some([5.0, 5.0]), None]; let keys = [Some(key("V", 7)), None, Some(key("V", 9))]; @@ -679,11 +633,11 @@ mod tests { assert_eq!(answered(&answer, "V", 9, "x"), Some(Value::Number(ranked[2][0]))); assert_eq!(answered(&answer, "V", 7, "y"), Some(Value::Number(ranked[0][1]))); let stress = Job::layout(&Layout::Stress(crate::Stress::default()), &placement).run(); - assert!(matches!(answered(&stress, "V", 7, "x"), Some(Value::Number(_))), "stress answers alike"); + assert!(matches!(answered(&stress, "V", 7, "x"), Some(Value::Number(_))), "stress layouts return rows too"); } #[test] - fn route_jobs_answer_routed_wires() { + fn route_jobs_return_routed_wires() { let obstacles = [[0.0, 0.0, 10.0, 10.0], [40.0, 0.0, 50.0, 10.0]]; let terminal = |x: f64, side, owner| Terminal { at: [x, 5.0], side: Some(side), owner: Some(owner) }; let nets = [Net { terminals: vec![terminal(10.0, Side::Right, 0), terminal(40.0, Side::Left, 1)] }]; @@ -696,12 +650,12 @@ mod tests { spacing: 2.0, }; let answer = Job::routes(&wiring).run(); - let want = route(&obstacles, &[], &nets, &[], 2.0).swap_remove(0); + let want = route(&Boxes::new(obstacles.to_vec()), &[], &nets, &[], 2.0).swap_remove(0); assert_eq!(answered(&answer, "Wire", 3, "route"), Some(Value::Parts(want))); } #[test] - fn bytes_that_are_no_job_answer_nothing() { + fn bytes_that_are_not_a_job_return_nothing() { assert_eq!(Job::from_bytes(vec![1, 0, 0, 0, 9]).run(), Vec::::new()); assert_eq!(Job::from_bytes(Vec::new()).run(), Vec::::new()); } diff --git a/crates/latkit/src/host/mod.rs b/crates/latkit/src/host/mod.rs index 80422815..55922444 100644 --- a/crates/latkit/src/host/mod.rs +++ b/crates/latkit/src/host/mod.rs @@ -1,16 +1,20 @@ -//! What the platform does for latkit: set and measure text in its fonts, do jobs off the frame, fetch -//! bytes (tiles, and the files JSON documents name), and encode video with its [`Encoder`]. A -//! [`Gpu`](crate::Gpu) is given its host with [`Gpu::set_host`](crate::Gpu::set_host); without one, -//! views draw no text, do their jobs at once, fetch nothing, and encode no video. A browser's host is -//! `Browser`, built with the `web` feature for `wasm32`. Natively, the `native` feature supplies -//! `Native`: lazy fonts, a worker pool doing jobs and reading files, and Media Foundation's H.264 on -//! Windows, without a window or network client. +//! The host: platform services latkit uses. A host rasterizes and measures text in the platform's +//! fonts, runs jobs in the background, fetches bytes (tiles, and files that JSON documents name), +//! and encodes video with an [`Encoder`]. +//! +//! A [`Gpu`](crate::Gpu) gets its host from [`Gpu::set_host`](crate::Gpu::set_host). Without one, +//! views draw no text, run jobs immediately, fetch nothing, and encode no video. In a browser the +//! host is `Browser` (the `web` feature on `wasm32`). Natively, the `native` feature supplies +//! `Native`. It loads fonts on first use, runs jobs and reads files on a worker pool, and encodes +//! H.264 with Media Foundation on Windows. It needs no window or network client. #[cfg(all(feature = "web", target_arch = "wasm32"))] mod browser; mod job; #[cfg(all(feature = "native", not(target_arch = "wasm32")))] mod native; +#[cfg(any(test, all(feature = "web", target_arch = "wasm32")))] +mod queue; use std::sync::Arc; @@ -19,7 +23,7 @@ pub use browser::Browser; pub use job::Job; pub(crate) use job::Placement; pub(crate) use job::Wiring; -pub(crate) use job::{Key, Shaping}; +pub(crate) use job::{Key, Shaping, shapes_answer}; #[cfg(all(feature = "native", not(target_arch = "wasm32")))] pub use native::Native; @@ -27,77 +31,79 @@ use crate::Encoding; pub use crate::resource::Request; use crate::{Error, Pending, Result}; -/// Ticks of a second an encoder's times count. +/// Ticks per second of encoder times. pub const TIMESCALE: u32 = 90_000; -/// A platform's H.264 encoder: NV12 pictures (BT.709, limited range) in, Annex B units out, no -/// B-frames, so every platform's encoder fits and the muxer never reorders. +/// A platform's H.264 encoder: NV12 pictures (BT.709, limited range) in, Annex B units out. It uses +/// no B-frames, so every platform's encoder can comply and the muxer never reorders frames. pub trait Encoder { - /// Encodes one NV12 picture at `time` ticks, a keyframe when `key`; resolves, once it has room - /// for another picture, to the units it finished since the last call, in order. + /// Encodes one NV12 picture at `time` ticks, as a keyframe when `key`. Once the encoder can + /// take another picture, resolves to the units finished since the last call, in order. fn encode(&mut self, nv12: Vec, time: u64, key: bool) -> Pending>; - /// Resolves to the units it still holds once every picture given is encoded. + /// Resolves to the remaining units once every picture given has been encoded. fn finish(&mut self) -> Pending>; } -/// One encoded access unit of H.264: Annex B bytes, at `time` ticks of [`TIMESCALE`], decode order -/// equal to display order. +/// One encoded H.264 access unit: Annex B bytes at `time` ticks of [`TIMESCALE`]. Decode order +/// equals display order. #[derive(Clone, PartialEq, Eq, Debug)] pub struct Unit { - /// Ticks at which this frame shows. + /// Presentation time in ticks of [`TIMESCALE`]. pub time: u64, - /// Whether it is a keyframe, which carries its parameter sets in band. + /// Whether this is a keyframe. Keyframes carry their parameter sets in band. pub key: bool, - /// Its NAL units, each after a start code. + /// NAL units, each preceded by a start code. pub annex_b: Vec, } -/// The platform services a view uses. +/// Platform services for text, background jobs, fetching, and video encoding. pub trait Host: Send + Sync + 'static { - /// Sets `run` as coverage, reporting unavailable fonts or failed rasterization. + /// Rasterizes `run` to a coverage mask. Errors if its font is unavailable or rasterization + /// fails. fn rasterize(&self, run: &TextRun) -> Result; - /// The width and height `run` takes once set, in pixels; by default what setting it takes. + /// The width and height of `run` once rasterized, in pixels. The default implementation + /// rasterizes it. fn measure(&self, run: &TextRun) -> Result<[f32; 2]> { self.rasterize(run).map(|raster| raster.size.map(|v| v as f32)) } - /// Does `job` off the frame, resolving to what [`Job::run`] answers; by default at once, on the - /// calling thread. + /// Runs `job` in the background, resolving to the result of [`Job::run`]. The default + /// implementation runs it immediately on the calling thread. fn spawn(&self, job: Job) -> Pending> { Pending::ready(Ok(job.run())) } - /// Fetches `request`, resolving to its bytes; to no bytes where there is nothing at its URL, and - /// to an error where it cannot be read. By default a host fetches nothing. + /// Fetches `request`, resolving to its bytes. Resolves to no bytes if nothing exists at the + /// URL, and to an error if it cannot be read. The default implementation fails as unsupported. fn fetch(&self, request: &Request) -> Pending> { let _ = request; - Pending::ready(Err(Error::Unsupported("this host does not fetch".into()))) + Pending::ready(Err(Error::Unsupported("this host cannot fetch".into()))) } - /// An H.264 encoder for pictures as `encoding` says, hardware where the platform has one: what an - /// [`Mp4`](crate::Mp4) encodes with. By default a host encodes no video. + /// An H.264 encoder for `encoding`, in hardware where the platform has one. [`Mp4`](crate::Mp4) + /// encodes with it. The default implementation fails as unsupported. fn encoder(&self, encoding: &Encoding) -> crate::Result> { let _ = encoding; - Err(Error::Unsupported("this host encodes no video".into())) + Err(Error::Unsupported("this host cannot encode video".into())) } } -/// A line of text to set. +/// Text to rasterize: one line, or a paragraph wrapped at `max_width`. #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub struct TextRun { /// The text. pub text: Arc, /// A CSS font family list. pub font: Arc, - /// The size in pixels. + /// Font size in pixels. pub size_px: u32, - /// Pixels a line runs to before it wraps at a space; `None` keeps one line. + /// Width in pixels at which lines wrap at a space. `None` keeps a single line. pub max_width: Option, } -/// Text set as coverage: one byte per pixel, rows top to bottom. +/// Rasterized text as a coverage mask: one byte per pixel, rows top to bottom. #[derive(Clone, PartialEq, Debug)] pub struct Raster { /// Width and height in pixels. diff --git a/crates/latkit/src/host/native.rs b/crates/latkit/src/host/native.rs index efca2c85..0c7ff020 100644 --- a/crates/latkit/src/host/native.rs +++ b/crates/latkit/src/host/native.rs @@ -1,5 +1,5 @@ -//! Native text, a small, lazily started worker pool, local files, and the platform's H.264 encoder. -//! No window or async runtime is needed. +//! The native host: text, a small worker pool started on first use, local file reads, and the +//! platform's H.264 encoder. It needs no window or async runtime. #[cfg(windows)] #[allow(unsafe_code, reason = "Media Foundation is COM: the FFI stays in this module, behind Encoder")] @@ -16,12 +16,13 @@ use super::{Host, Job, Raster, Request, TextRun}; use crate::Encoding; use crate::{Error, Pending, Resolver, Result}; -/// Native platform services: shaped text, background jobs, local files, and video. A handle: clones -/// share its fonts and workers. System fonts are discovered only on the first text request; -/// [`from_fonts`](Self::from_fonts) uses only the supplied font bytes. A pool of at most four -/// workers starts on the first job or fetch; a fetch reads a file path, or a `file://` URL, on it. -/// Video encodes with Media Foundation on Windows, hardware where the machine has it; elsewhere this -/// host encodes none. +/// Native platform services: shaped text, background jobs, local files, and video. +/// +/// `Native` is a handle, and clones share its fonts and workers. System fonts are discovered on the +/// first text request. [`from_fonts`](Self::from_fonts) uses only the supplied font bytes instead. +/// A pool of at most four workers starts on the first job or fetch. A fetch reads a file path or a +/// `file://` URL on a worker. Video is encoded with Media Foundation on Windows, in hardware where +/// the machine has it. Other platforms encode no video. #[derive(Clone, Default)] pub struct Native(Arc); @@ -32,13 +33,15 @@ struct Services { } impl Native { - /// A host that discovers system fonts lazily. Constructing it does no I/O and starts no threads. + /// A host that discovers system fonts on first use. Constructing it does no I/O and starts no + /// threads. pub fn new() -> Self { Self::default() } - /// A host using only these TTF/OTF/TTC bytes, in order; the first family is the generic fallback. - /// No system fonts are discovered. Each item must contain at least one usable face. + /// A host using only these TTF/OTF/TTC font bytes, in order. The first family is the fallback + /// for the generic families. No system fonts are discovered. Errors if no fonts are given or an + /// item has no usable face. pub fn from_fonts(fonts: impl IntoIterator>) -> Result { let mut db = fontdb::Database::new(); for (i, font) in fonts.into_iter().enumerate() { @@ -71,7 +74,7 @@ impl Native { f(text) } - /// Does `work` on a worker, resolving to what it answers. + /// Runs `work` on a worker, resolving to its result. fn work(&self, work: impl FnOnce() -> Result> + Send + 'static) -> Pending> { let workers = match self.0.workers.get_or_init(Workers::new) { Ok(workers) => workers, @@ -128,20 +131,21 @@ impl Host for Native { { let _ = encoding; Err(Error::Unsupported( - "native H.264 encoding is Windows' (Media Foundation); this platform has none".into(), + "native H.264 encoding is available only on Windows, through Media Foundation".into(), )) } } } -/// The bytes `request` names of a local file: none where there is no file. +/// Reads the local file, or its byte range, that `request` names by path or `file://` URL. Returns +/// no bytes if the file does not exist. Errors on other URL schemes. fn read(request: &Request) -> Result> { let url = &*request.url; let path = if url.starts_with("file:") { crate::url::file_path(url).ok_or_else(|| Error::invalid(format!("invalid file URL: {url}")))? } else { if url.contains("://") { - return Err(Error::Unsupported(format!("the native host reads files, not {url}").into())); + return Err(Error::Unsupported(format!("the native host reads only local files, not {url}").into())); } std::path::PathBuf::from(url) }; @@ -233,10 +237,10 @@ impl Text { } } -/// Points each generic family at an installed one. fontdb names a generic after fontconfig's last -/// alias for it, installed or not: stock Ubuntu's last `sans-serif` alias prefers `FreeSans`, which -/// it does not ship. A generic naming nothing installed takes the first installed of its common -/// families, else any installed family. +/// Points each generic family at an installed family. fontdb names a generic after fontconfig's +/// last alias for it, installed or not. On stock Ubuntu, the last `sans-serif` alias prefers +/// `FreeSans`, which Ubuntu does not ship. A generic naming no installed family gets the first +/// installed of its common families, else any installed family. fn installed_generics(db: &mut fontdb::Database) { fn has(db: &fontdb::Database, family: Family<'_>) -> bool { db.query(&fontdb::Query { families: &[family], ..Default::default() }).is_some() @@ -367,7 +371,7 @@ mod tests { let url = path.to_string_lossy().into_owned(); assert_eq!(fetch(Request::new(url.as_str())).unwrap(), b"0123456789"); assert_eq!(fetch(Request::range(format!("file://{url}"), 3..6)).unwrap(), b"345"); - assert_eq!(fetch(Request::range(url.as_str(), 8..20)).unwrap(), b"89", "a range past the end"); + assert_eq!(fetch(Request::range(url.as_str(), 8..20)).unwrap(), b"89", "a range past the end is cut short"); assert_eq!(fetch(Request::new(dir.join("none").to_string_lossy().into_owned())).unwrap(), b""); assert!(fetch(Request::new("https://example.com/tiles")).is_err()); std::fs::remove_dir_all(dir).unwrap(); diff --git a/crates/latkit/src/host/native/media_foundation.rs b/crates/latkit/src/host/native/media_foundation.rs index 58cce9a6..66e48b23 100644 --- a/crates/latkit/src/host/native/media_foundation.rs +++ b/crates/latkit/src/host/native/media_foundation.rs @@ -1,6 +1,6 @@ -//! H.264 through Windows Media Foundation: the platform's encoder transform, hardware where the -//! machine has one and the software encoder otherwise, on a thread of its own so encoding overlaps -//! drawing. NV12 pictures in, Annex B units out, no B-frames, keyframes where asked. +//! H.264 encoding through Windows Media Foundation. Uses the machine's hardware encoder transform +//! if it has one, else the software encoder. The transform runs on its own thread so encoding +//! overlaps drawing. NV12 pictures in, Annex B units out, no B-frames, keyframes on request. use std::collections::VecDeque; use std::mem::ManuallyDrop; @@ -33,27 +33,27 @@ use crate::video::nals; use crate::{Encoding, Error, Pending, Resolver, Result}; use tell::Tell; -/// The longest the transform may take to start, to want a picture, or to drain. +/// Timeout for the transform to start, to accept a picture, or to drain. const STALL: Duration = Duration::from_secs(20); -/// Seconds between the keyframes the encoder places itself; the video asks for its own besides. +/// Seconds between the keyframes the encoder inserts on its own. The video also requests some. const KEYFRAMES: f64 = 2.0; -/// 100-nanosecond ticks a second, Media Foundation's clock. +/// Media Foundation clock ticks (100 ns) per second. const HNS: u64 = 10_000_000; -/// An encoder on a thread of its own, where Media Foundation lives. +/// An encoder whose Media Foundation objects live on a thread of their own. struct MediaFoundation { work: Option>, } -/// What the encoder's thread is handed: a picture, with what tells the encoder it has room for -/// another; or the end. +/// Work for the encoder's thread: a picture, whose `room` resolves once the encoder can take +/// another, or the request to finish. enum Work { Picture { nv12: Vec, time: u64, key: bool, room: Resolver> }, Finish(Resolver>), } -/// An encoder for pictures as `encoding` says: the machine's hardware encoder where one starts, else -/// Windows' software encoder. +/// An encoder for `encoding`: the machine's hardware encoder if one starts, else Windows' software +/// encoder. pub(super) fn encoder(encoding: &Encoding) -> Result> { open(*encoding, &[hardware(), software()]).map(|e| Box::new(e) as Box) } @@ -66,7 +66,8 @@ fn software() -> MFT_ENUM_FLAG { MFT_ENUM_FLAG_SYNCMFT | MFT_ENUM_FLAG_SORTANDFILTER } -/// An encoder from the first transform of `kinds`, in order, that starts. +/// Opens an encoder with the first transform of `kinds`, tried in order, that starts. Times out if +/// starting takes longer than [`STALL`]. fn open(encoding: Encoding, kinds: &[MFT_ENUM_FLAG]) -> Result { let (started, ready) = mpsc::channel(); let (work, pictures) = mpsc::channel(); @@ -104,9 +105,9 @@ impl Encoder for MediaFoundation { } } -/// The encoder's thread: starts Media Foundation and the transform, then encodes what it is handed -/// until it finishes or the encoder is dropped. Taking a picture tells there is room for another, -/// with the units finished before it; after an error, every picture is answered with it. +/// The encoder's thread. It starts Media Foundation and the transform, then encodes pictures until +/// told to finish or the encoder is dropped. Taking a picture resolves its `room` with the units +/// finished before it. After an error, every later picture resolves with that error. fn run(encoding: Encoding, kinds: &[MFT_ENUM_FLAG], started: &Sender>, work: &Receiver) { let session = match Session::new() { Ok(session) => session, @@ -155,12 +156,12 @@ fn run(encoding: Encoding, kinds: &[MFT_ENUM_FLAG], started: &Sender> drop(session); } -/// A failed Media Foundation call, named. +/// The error for a failed Media Foundation call, naming the call. fn failed(call: &str, error: &windows::core::Error) -> Error { - Error::Unsupported(format!("the Media Foundation encoder's {call}: {error}").into()) + Error::Unsupported(format!("the Media Foundation encoder's {call} failed: {error}").into()) } -/// COM and Media Foundation, started on this thread and shut down with it. +/// COM and Media Foundation, started on the current thread and shut down when the session drops. struct Session { media: bool, } @@ -179,7 +180,7 @@ impl Session { impl Drop for Session { fn drop(&mut self) { - // SAFETY: every transform, sample, and buffer of this thread was released before. + // SAFETY: every transform, sample, and buffer on this thread was already released. unsafe { if self.media { let _ = MFShutdown(); @@ -189,25 +190,27 @@ impl Drop for Session { } } -/// One H.264 encoder transform, streaming. +/// One streaming H.264 encoder transform. struct Transform { activate: IMFActivate, mft: IMFTransform, codec: Option, - /// An asynchronous transform's events: hardware encoders are asynchronous. + /// An asynchronous transform's events. Hardware encoders are asynchronous. events: Option, - /// Inputs an asynchronous transform asked for and was not given. + /// Inputs an asynchronous transform requested and has not yet received. wanted: u32, input: u32, output: u32, - /// Whether the transform makes its output samples; else, how large a buffer it takes. + /// Whether the transform allocates its output samples. If not, `output_size` is the size of + /// the buffer to allocate. provides: bool, output_size: u32, - /// The times of the pictures given and not yet encoded, in order: no B-frames reorder them. + /// Times of pictures given but not yet encoded, in order. Without B-frames, output order + /// matches input order. times: VecDeque, - /// The SPS and PPS, Annex B, for keyframes that leave them out. + /// The SPS and PPS in Annex B, added to keyframes that lack them. header: Vec, - /// Parameter sets the transform gave as units of their own, for the next picture. + /// Parameter sets the transform output as separate units, prepended to the next picture. prefix: Vec, encoding: Encoding, } @@ -260,7 +263,7 @@ impl Transform { (transform.input, transform.output) = (inputs[0], outputs[0]); } if let Some(codec) = &transform.codec { - // What an encoder cannot set it does without: these only shape its output. + // Settings an encoder rejects are skipped. They only tune its output. let gop = (encoding.fps * KEYFRAMES).round().max(1.0) as u32; let rate = eAVEncCommonRateControlMode_UnconstrainedVBR.0 as u32; for (api, value) in [ @@ -284,7 +287,7 @@ impl Transform { transform.mft.SetOutputType(transform.output, &output, 0).map_err(|e| { if e.code() == MF_E_INVALIDMEDIATYPE { Error::Unsupported( - format!("this H.264 encoder takes no {width}×{height} pictures at {num}/{den} fps").into(), + format!("this H.264 encoder rejects {width}×{height} pictures at {num}/{den} fps").into(), ) } else { failed("SetOutputType", &e) @@ -306,7 +309,7 @@ impl Transform { } } - /// The NV12 input type: the transform's own where it offers one, else a type of ours. + /// The NV12 input type: the one the transform offers, if any, else one built here. fn input_type(&self, width: u32, height: u32, num: u32, den: u32) -> Result { // SAFETY: owned COM handles only. unsafe { @@ -327,7 +330,7 @@ impl Transform { } } - /// Whether the transform makes its own output samples, and how large ours must be otherwise. + /// Reads whether the transform allocates its output samples, and the buffer size otherwise. fn output_info(&mut self) -> Result<()> { // SAFETY: owned COM handles only. let info = @@ -338,8 +341,8 @@ impl Transform { Ok(()) } - /// Encodes one NV12 picture at `time` ticks, a keyframe when `key`: the units the transform - /// finished meanwhile. + /// Encodes one NV12 picture at `time` ticks, as a keyframe when `key`. Returns the units the + /// transform finished meanwhile. fn encode(&mut self, nv12: &[u8], time: u64, key: bool) -> Result> { let sample = self.sample(nv12, time)?; if key && let Some(codec) = &self.codec { @@ -355,7 +358,7 @@ impl Transform { } self.wanted -= 1; self.input(&sample)?; - // Whatever it told meanwhile: outputs ready, and inputs wanted. + // Handle the events queued meanwhile: outputs ready and inputs requested. while self.event(false, &mut units)? {} } else { let started = Instant::now(); @@ -372,7 +375,7 @@ impl Transform { Ok(units) } - /// The units the transform still holds once every picture given is encoded. + /// Drains the transform, returning its remaining units once every picture given is encoded. fn finish(&mut self) -> Result> { let mut units = Vec::new(); // SAFETY: plain messages to an owned transform. @@ -396,14 +399,15 @@ impl Transform { Ok(units) } - /// Gives a synchronous transform `sample`, or an asynchronous one that asked for it. + /// Passes `sample` to the transform. An asynchronous transform must have requested input. fn input(&self, sample: &IMFSample) -> Result<()> { // SAFETY: an owned sample, borrowed for the call. unsafe { self.mft.ProcessInput(self.input, sample, 0) }.map_err(|e| failed("ProcessInput", &e)) } - /// Handles the asynchronous transform's next event, waiting for it, up to the stall, when `wait`: - /// an input wanted is counted, an output ready is taken into `units`. Whether there was one. + /// Handles the asynchronous transform's next event, waiting up to [`STALL`] when `wait`. An + /// input request is counted, and a ready output is taken into `units`. Returns whether there + /// was an event. fn event(&mut self, wait: bool, units: &mut Vec) -> Result { let events = self.events.as_mut().expect("an asynchronous transform"); match events.next(wait)? { @@ -421,12 +425,12 @@ impl Transform { Ok(()) } - /// Takes the output an asynchronous transform said it has. + /// Takes the output an asynchronous transform reported ready. fn drain_one(&mut self, units: &mut Vec) -> Result<()> { self.output(units).map(|_| ()) } - /// Takes one output into `units`: whether there was one, rather than a want of input. + /// Takes one output into `units`. Returns `false` if the transform needs more input instead. fn output(&mut self, units: &mut Vec) -> Result { for _ in 0..4 { let sample = if self.provides { None } else { Some(empty(self.output_size)?) }; @@ -437,7 +441,7 @@ impl Transform { pEvents: ManuallyDrop::new(None), }; let mut status = 0; - // SAFETY: the buffer's sample and events are taken back below, whatever the result. + // SAFETY: the buffer's sample and events are taken back below, on success or failure. let result = unsafe { self.mft.ProcessOutput(0, std::slice::from_mut(&mut buffer), &raw mut status) }; let sample = unsafe { ManuallyDrop::take(&mut buffer.pSample) }; drop(unsafe { ManuallyDrop::take(&mut buffer.pEvents) }); @@ -454,7 +458,8 @@ impl Transform { self.mft.SetOutputType(self.output, &changed, 0).map_err(|e| failed("SetOutputType", &e))?; self.output_info()?; } - // An asynchronous transform says when the output is ready in its new type. + // An asynchronous transform sends another event once output in the new type is + // ready. if self.events.is_some() { return Ok(true); } @@ -462,11 +467,11 @@ impl Transform { } Err(e) => return Err(failed("ProcessOutput", &e)), } - let Some(sample) = sample else { return Err(Error::Io("the encoder gave no sample".into())) }; + let Some(sample) = sample else { return Err(Error::Io("the encoder returned no sample".into())) }; let bytes = bytes(&sample)?; let types: Vec = nals(&bytes).map(|nal| nal[0] & 0x1f).collect(); if !types.iter().any(|t| (1..=5).contains(t)) { - // Parameter sets alone: they lead the next picture. + // Only parameter sets: prepend them to the next picture. self.prefix.extend_from_slice(&bytes); return Ok(true); } @@ -476,14 +481,14 @@ impl Transform { annex_b.extend_from_slice(&self.header); } annex_b.extend_from_slice(&bytes); - let time = self.times.pop_front().ok_or_else(|| Error::Io("the encoder gave more than given".into()))?; + let time = self.times.pop_front().ok_or_else(|| Error::Io("the encoder returned extra pictures".into()))?; units.push(Unit { time, key, annex_b }); return Ok(true); } - Err(Error::Io("the encoder kept changing its output".into())) + Err(Error::Io("the encoder kept changing its output type".into())) } - /// A sample of NV12 `bytes` showing at `time` ticks for a frame's duration. + /// A sample of NV12 `bytes` at `time` ticks, lasting one frame. fn sample(&self, bytes: &[u8], time: u64) -> Result { let length = u32::try_from(bytes.len()).map_err(|_| Error::invalid("the picture is too large to encode"))?; let sample = empty(length)?; @@ -517,12 +522,12 @@ fn empty(length: u32) -> Result { impl Drop for Transform { fn drop(&mut self) { - // SAFETY: an asynchronous transform is shut down through what activated it, once. + // SAFETY: the transform is shut down once, through the activation object that created it. let _ = unsafe { self.activate.ShutdownObject() }; } } -/// The transforms encoding NV12 into H.264 of `kind`, the best first. +/// The transforms of `kind` that encode NV12 into H.264, best first. fn activates(kind: MFT_ENUM_FLAG) -> Result> { let input = MFT_REGISTER_TYPE_INFO { guidMajorType: MFMediaType_Video, guidSubtype: MFVideoFormat_NV12 }; let output = MFT_REGISTER_TYPE_INFO { guidMajorType: MFMediaType_Video, guidSubtype: MFVideoFormat_H264 }; @@ -549,7 +554,8 @@ fn activates(kind: MFT_ENUM_FLAG) -> Result> { } } -/// Sets what a picture is on `kind`: its size and rate, progressive, square pixels, BT.709 limited. +/// Sets the picture format on media type `kind`: size, frame rate, progressive, square pixels, and +/// BT.709 limited range. fn picture(kind: &IMFMediaType, width: u32, height: u32, num: u32, den: u32) -> Result<()> { // SAFETY: owned COM handles, values borrowed for each call. unsafe { @@ -568,7 +574,8 @@ fn picture(kind: &IMFMediaType, width: u32, height: u32, num: u32, den: u32) -> Ok(()) } -/// The SPS and PPS the transform's output type carries, Annex B; none where it carries them in band. +/// The SPS and PPS in Annex B from the transform's output type. Empty if the type has no Annex B +/// sequence header, in which case they come in band. fn header(mft: &IMFTransform, output: u32) -> Vec { // SAFETY: owned COM handles; the blob is copied into a buffer of its size. unsafe { @@ -582,12 +589,12 @@ fn header(mft: &IMFTransform, output: u32) -> Vec { } } -/// An asynchronous transform's events, told as they come to the callback each request names. +/// Events from an asynchronous transform, delivered to the callback passed with each request. struct Events { generator: IMFMediaEventGenerator, callback: IMFAsyncCallback, told: Receiver>, - /// Whether an event was asked for and not yet told. + /// Whether an event was requested and has not yet arrived. asked: bool, } @@ -598,8 +605,8 @@ impl Events { Events { generator, callback, told, asked: false } } - /// The type of the next event: waiting for it up to the stall when `wait`, else `None` while - /// none was told. + /// The type of the next event. With `wait`, waits up to [`STALL`] and errors on timeout. + /// Otherwise returns `None` if no event has arrived. fn next(&mut self, wait: bool) -> Result> { if !self.asked { // SAFETY: owned COM handles; the callback outlives the request it answers. @@ -619,9 +626,9 @@ impl Events { } } -/// What a transform tells its next event to: a COM object of ours, as `windows::core::implement` -/// writes one. -#[allow(clippy::ref_as_ptr, clippy::inline_always, reason = "what windows::core::implement writes")] +/// The callback a transform sends its next event to: a COM object generated by +/// `windows::core::implement`. +#[allow(clippy::ref_as_ptr, clippy::inline_always, reason = "code generated by windows::core::implement")] mod tell { use std::sync::mpsc::Sender; @@ -637,7 +644,7 @@ mod tell { impl IMFAsyncCallback_Impl for Tell_Impl { fn GetParameters(&self, _: *mut u32, _: *mut u32) -> windows::core::Result<()> { - // E_NOTIMPL: the default work queue calls back. + // E_NOTIMPL: call back on the default work queue. Err(windows::core::Error::from_hresult(windows::core::HRESULT(0x8000_4001_u32 as i32))) } @@ -701,16 +708,16 @@ mod tests { assert!(units[0].key, "the first is a keyframe"); let first: Vec = nals(&units[0].annex_b).map(|nal| nal[0] & 0x1f).collect(); assert!(first.contains(&7) && first.contains(&8), "the first keyframe carries its parameter sets: {first:?}"); - assert!(units[6].key, "a keyframe where asked"); + assert!(units[6].key, "a keyframe where requested"); } #[test] - fn the_software_encoder_gives_a_unit_a_picture() { + fn the_software_encoder_returns_a_unit_per_picture() { encodes_a_short_video(&[software()]); } #[test] - fn the_hardware_encoder_gives_a_unit_a_picture() { + fn the_hardware_encoder_returns_a_unit_per_picture() { encodes_a_short_video(&[hardware()]); } } diff --git a/crates/latkit/src/host/queue.rs b/crates/latkit/src/host/queue.rs new file mode 100644 index 00000000..717cdde9 --- /dev/null +++ b/crates/latkit/src/host/queue.rs @@ -0,0 +1,67 @@ +//! A first-in, first-out queue of jobs waiting for a worker. A job reaches a worker only when one +//! is free, so a job whose result is no longer awaited, such as a pyramid superseded by newer data, +//! is dropped before it uses a worker. + +use std::collections::VecDeque; + +use super::Job; +use crate::Resolver; + +#[derive(Default)] +pub(crate) struct Queue { + waiting: VecDeque<(Job, Resolver>)>, + running: usize, +} + +impl Queue { + /// Queues `job`. `resolver` receives its result. + pub(crate) fn push(&mut self, job: Job, resolver: Resolver>) { + self.waiting.push_back((job, resolver)); + } + + /// The next job for a worker, if fewer than `workers` jobs are running. Drops cancelled jobs. + pub(crate) fn next(&mut self, workers: usize) -> Option<(Job, Resolver>)> { + while let Some((job, resolver)) = self.waiting.pop_front() { + if resolver.is_cancelled() { + continue; + } + if self.running >= workers { + self.waiting.push_front((job, resolver)); + return None; + } + self.running += 1; + return Some((job, resolver)); + } + None + } + + /// Records that a worker finished its job. + pub(crate) fn done(&mut self) { + self.running = self.running.saturating_sub(1); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Pending; + + #[test] + fn a_job_cancelled_while_queued_never_reaches_a_worker() { + let mut queue = Queue::default(); + let (kept, first) = Pending::>::new(); + let (dropped, second) = Pending::>::new(); + let (later, third) = Pending::>::new(); + queue.push(Job::from_bytes(vec![1]), first); + queue.push(Job::from_bytes(vec![2]), second); + queue.push(Job::from_bytes(vec![3]), third); + let (job, _) = queue.next(1).expect("a worker is free"); + assert_eq!(job.into_bytes(), [1]); + assert!(queue.next(1).is_none(), "the one worker is busy"); + drop(dropped); + queue.done(); + let (job, _) = queue.next(1).expect("the worker is free again"); + assert_eq!(job.into_bytes(), [3], "the cancelled job was skipped"); + drop((kept, later)); + } +} diff --git a/crates/latkit/src/image.rs b/crates/latkit/src/image.rs index 36bba992..b6aecfdd 100644 --- a/crates/latkit/src/image.rs +++ b/crates/latkit/src/image.rs @@ -1,12 +1,12 @@ -//! Images: RGBA texels, what an icon marker draws and what a shot reads its picture back as, written -//! as PNG, JPEG, or lossless WebP and read from PNG. +//! RGBA images: icon marker textures and the pictures shots read back. Images encode to PNG, JPEG, +//! or lossless WebP, and decode from PNG. use std::io::{Cursor, Write}; use std::sync::Arc; use crate::{Error, Result}; -/// RGBA texels, straight alpha, sRGB, rows top to bottom. +/// An image of RGBA texels: straight alpha, sRGB, rows top to bottom. #[derive(Clone, PartialEq, Eq, Debug, serde::Deserialize, schemars::JsonSchema)] #[serde(remote = "Self", deny_unknown_fields)] #[schemars(rename = "Bitmap")] @@ -17,7 +17,7 @@ pub struct Image { pub rgba: Arc<[u8]>, } -/// An image reads as `{ size, rgba }`, four bytes a texel. +/// Deserializes from `{ size, rgba }`, four bytes per texel, and checks the length. impl<'de> serde::Deserialize<'de> for Image { fn deserialize>(deserializer: D) -> Result { let Image { size, rgba } = Image::deserialize(deserializer)?; @@ -26,7 +26,8 @@ impl<'de> serde::Deserialize<'de> for Image { } impl Image { - /// An image of `size` texels holding `rgba`, four bytes a texel. + /// An image of `size` texels from `rgba`, four bytes per texel. Fails if a dimension is zero + /// or `rgba` has the wrong length. pub fn new(size: [u32; 2], rgba: impl Into>) -> Result { let rgba = rgba.into(); let expected = u64::from(size[0]).checked_mul(u64::from(size[1])).and_then(|pixels| pixels.checked_mul(4)); @@ -36,14 +37,15 @@ impl Image { Ok(Image { size, rgba }) } - /// The image written as a file of `format`. + /// Encodes the image as a `format` file. pub fn encode(&self, format: ImageFormat) -> Result> { let mut out = Vec::new(); self.write(format, &mut out)?; Ok(out) } - /// Writes the image as a file of `format` to a file or stream, reporting write failures. + /// Writes the image as a `format` file to a file or stream. Write failures return + /// [`Error::Io`]. pub fn write(&self, format: ImageFormat, mut output: impl Write) -> Result<()> { Self::new(self.size, self.rgba.clone())?; let [width, height] = self.size; @@ -54,7 +56,7 @@ impl Image { encoder.set_color(png::ColorType::Rgba); encoder.set_depth(png::BitDepth::Eight); encoder.set_source_srgb(png::SrgbRenderingIntent::Perceptual); - // A picture is written far more often than it is stored long. + // Favor speed over size: pictures are written often and rarely stored long. encoder.set_compression(png::Compression::Fast); let mut writer = encoder.write_header().map_err(|e| failed(&e))?; writer.write_image_data(&self.rgba).map_err(|e| failed(&e))?; @@ -62,9 +64,9 @@ impl Image { } ImageFormat::Jpeg { quality } => { if !(1..=100).contains(&quality) { - return Err(Error::invalid("a JPEG's quality runs from 1 to 100").at("quality")); + return Err(Error::invalid("JPEG quality must be from 1 to 100").at("quality")); } - let side = |v: u32| u16::try_from(v).map_err(|_| failed(&"a JPEG is at most 65535 pixels a side")); + let side = |v: u32| u16::try_from(v).map_err(|_| failed(&"each side must be at most 65535 pixels")); let mut bytes = Vec::new(); jpeg_encoder::Encoder::new(&mut bytes, quality) .encode(&self.rgba, side(width)?, side(height)?, jpeg_encoder::ColorType::Rgba) @@ -77,13 +79,13 @@ impl Image { } } - /// The image a PNG file holds, any color type and depth made eight bit RGBA. + /// Decodes a PNG file. Any color type and bit depth converts to 8-bit RGBA. pub fn from_png(bytes: &[u8]) -> Result { let failed = |e: png::DecodingError| Error::invalid(format!("not a PNG image: {e}")); let mut decoder = png::Decoder::new(Cursor::new(bytes)); decoder.set_transformations(png::Transformations::normalize_to_color8()); let mut reader = decoder.read_info().map_err(failed)?; - let mut buffer = vec![0; reader.output_buffer_size().ok_or_else(|| Error::invalid("a PNG too large"))?]; + let mut buffer = vec![0; reader.output_buffer_size().ok_or_else(|| Error::invalid("PNG too large to decode"))?]; let info = reader.next_frame(&mut buffer).map_err(failed)?; let texels = (info.width * info.height) as usize; let rgba: Vec = match info.color_type { @@ -101,23 +103,24 @@ impl Image { } } -/// How an image is written as a file. +/// An image file format. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default)] pub enum ImageFormat { - /// PNG: lossless, alpha kept, tagged sRGB. + /// PNG: lossless, keeps alpha, tagged sRGB. #[default] Png, - /// JPEG at `quality` from 1 to 100: small, its alpha dropped. + /// JPEG at `quality` 1 to 100: small files, alpha dropped. Jpeg { - /// From 1, smallest, to 100, sharpest; 90 is good for pictures of views. + /// 1 (smallest) to 100 (sharpest). 90 suits pictures of views. quality: u8, }, - /// WebP, lossless: smaller than PNG, alpha kept. + /// Lossless WebP: smaller than PNG, keeps alpha. WebP, } impl ImageFormat { - /// The format a file's `extension` names: `png`, `jpg` or `jpeg` at quality 90, or `webp`. + /// The format for a file `extension`, ignoring case: `png`, `jpg` or `jpeg` (quality 90), or + /// `webp`. pub fn of_extension(extension: &str) -> Option { match extension.to_ascii_lowercase().as_str() { "png" => Some(ImageFormat::Png), @@ -126,6 +129,26 @@ impl ImageFormat { _ => None, } } + + /// The code an encoding job writes the format as: 0 for PNG, 1 for WebP, and `0x100` plus the + /// quality for JPEG. + pub(crate) fn code(self) -> u32 { + match self { + ImageFormat::Png => 0, + ImageFormat::WebP => 1, + ImageFormat::Jpeg { quality } => 0x100 | u32::from(quality), + } + } + + /// The format written as `code`. + pub(crate) fn from_code(code: u32) -> Option { + match code { + 0 => Some(ImageFormat::Png), + 1 => Some(ImageFormat::WebP), + 0x100..=0x1ff => Some(ImageFormat::Jpeg { quality: (code & 0xff) as u8 }), + _ => None, + } + } } impl std::fmt::Display for ImageFormat { @@ -143,9 +166,9 @@ mod tests { use super::*; #[test] - fn png_holds_what_it_was_given() { + fn png_round_trips_losslessly() { let rgba: Vec = (0..3 * 2 * 4).map(|i| (i * 11) as u8).collect(); - let image = Image::new([3, 2], rgba).expect("whole"); + let image = Image::new([3, 2], rgba).expect("a valid image"); let back = Image::from_png(&image.encode(ImageFormat::Png).expect("written")).expect("a PNG"); assert_eq!(back, image); } @@ -153,7 +176,7 @@ mod tests { #[test] fn every_format_writes_its_file() { let rgba: Vec = (0..16 * 8 * 4).map(|i| (i * 7) as u8).collect(); - let image = Image::new([16, 8], rgba).expect("whole"); + let image = Image::new([16, 8], rgba).expect("a valid image"); let jpeg = image.encode(ImageFormat::Jpeg { quality: 90 }).expect("written"); assert_eq!(jpeg[..3], [0xff, 0xd8, 0xff], "a JPEG starts with its marker"); let webp = image.encode(ImageFormat::WebP).expect("written"); @@ -163,7 +186,7 @@ mod tests { } #[test] - fn an_image_holds_four_bytes_a_texel() { + fn images_need_a_positive_size_and_four_bytes_per_texel() { assert!(Image::new([2, 2], vec![0; 15]).is_err()); assert!(Image::from_png(b"not a png").is_err()); assert!(Image::new([0, 0], Vec::new()).is_err()); diff --git a/crates/latkit/src/json.rs b/crates/latkit/src/json.rs index 10013920..25ab2ca5 100644 --- a/crates/latkit/src/json.rs +++ b/crates/latkit/src/json.rs @@ -1,5 +1,5 @@ -//! JSON as latkit reads it: any input from its text, an error naming the path to what it could not -//! read, such as `marks[2].vertices.color`. +//! Reading inputs from JSON. Errors name the path to the part that could not be read, such as +//! `marks[2].color`. use std::fmt::Display; @@ -8,8 +8,8 @@ use serde_path_to_error::Segment; use crate::{Error, Path, Result, Step}; -/// What JSON `text` holds, read as `T`: a scene, a camera, a fit, a query, rows, a budget, a shot, or -/// any type that deserializes. An error names the path to what could not be read. +/// Deserializes JSON `text` as `T`, such as a scene, camera, fit, query, rows, budget, or shot. +/// Fails with [`Error::Invalid`], whose path names the part that could not be read. /// /// ``` /// # #[cfg(feature = "gpu")] { @@ -26,19 +26,49 @@ pub fn from_json(text: &str) -> Result { Ok(value) } -/// What `deserializer` holds, read as `T`, an error naming the path to what could not be read. +/// Deserializes JSON `bytes` as `T`, like [`from_json`]. Bytes that are not UTF-8 are invalid. +pub(crate) fn from_bytes(bytes: &[u8]) -> Result { + from_json(std::str::from_utf8(bytes).map_err(|e| Error::invalid(format!("not UTF-8: {e}")))?) +} + +/// Deserializes `T` from `deserializer`. Errors name the path to the part that could not be read. pub(crate) fn read<'de, T: Deserialize<'de>, D: Deserializer<'de>>(deserializer: D) -> Result where D::Error: Display, { - serde_path_to_error::deserialize(deserializer).map_err(|error| { + // A failed `nested` read leaves the path within its value. Clear one left by a read that did not + // pass through here, such as a test's `serde_json::from_str`. + WITHIN.take(); + let read = serde_path_to_error::deserialize(deserializer); + let within = WITHIN.take(); + read.map_err(|error| { let steps = error.path().iter().filter_map(|segment| match segment { Segment::Seq { index } => Some(Step::Index(*index)), Segment::Map { key } => Some(Step::Name(key.as_str().into())), Segment::Enum { variant } => Some(Step::Name(variant.as_str().into())), Segment::Unknown => None, }); - Error::Invalid { path: Path(steps.collect()), reason: error.into_inner().to_string().into() } + let path = steps.chain(within.into_iter().flat_map(|path| path.0)).collect(); + Error::Invalid { path: Path(path), reason: error.into_inner().to_string().into() } + }) +} + +thread_local! { + /// The path within a buffered value at which [`nested`] failed, for [`read`] to append to the + /// path of the value itself. + static WITHIN: std::cell::Cell> = const { std::cell::Cell::new(None) }; +} + +/// Deserializes `T` from `value`, a buffered part of the input [`read`] is reading. Marks are +/// buffered, since the key naming a mark's kind may come after its options. An error's path +/// continues from the value's own path. +pub(crate) fn nested(value: serde_json::Value) -> std::result::Result { + read(value).map_err(|error| match error { + Error::Invalid { path, reason } => { + WITHIN.set(Some(path)); + E::custom(reason) + } + error => E::custom(error), }) } @@ -51,8 +81,14 @@ mod tests { let at = |json: &str| from_json::(json).unwrap_err().to_string(); let grid = at(r#"{"space": {"plane": {"grid": "bogus"}}, "marks": []}"#); assert!(grid.starts_with("space.plane.grid:"), "{grid}"); - let mark = at(r#"{"space": "plane", "marks": [{"vertices": {"of": ["d", "t"], "colour": "x"}}]}"#); - assert!(mark.starts_with("marks[0].vertices"), "{mark}"); + let mark = at(r#"{"space": "plane", "marks": [{"vertices": ["d", "t"], "colour": "x"}]}"#); + assert!(mark.starts_with("marks[0].colour:"), "{mark}"); + // An option before the key naming the mark's kind still names its own path. + let radius = + at(r#"{"space": "plane", "marks": [{"edges": ["d", "e"]}, {"radius": [1], "vertices": ["d", "t"]}]}"#); + assert!(radius.starts_with("marks[1].radius"), "{radius}"); + let none = at(r#"{"space": "plane", "marks": [{"of": ["d", "t"]}]}"#); + assert!(none.starts_with("marks[0]:") && none.contains("vertices"), "{none}"); assert!(from_json::(r#"{"space": "plane", "marks": []} trailing"#).is_err()); } } diff --git a/crates/latkit/src/layout/boxes.rs b/crates/latkit/src/layout/boxes.rs new file mode 100644 index 00000000..598b6cf9 --- /dev/null +++ b/crates/latkit/src/layout/boxes.rs @@ -0,0 +1,221 @@ +//! A spatial index of boxes that change. It is a packed R-tree whose leaves refit as their boxes +//! change, plus a list of items given boxes since packing. Every query also scans that list until +//! it is long enough to repack. Queries and changes take about O(log n). The root holds the extent +//! of all boxes, so keeping the index current never walks every box, except when repacking. + +use super::rect::{EMPTY, Rect, meets, union, valid}; + +/// Children per node. +const NODE: usize = 16; + +/// How many items may get boxes after packing before the index is repacked. +const FRESH: usize = 64; + +/// Boxes by item (`EMPTY` where an item has none), with a spatial index. +pub(crate) struct Boxes { + rects: Vec, + index: Index, + /// Each item's leaf, or `usize::MAX` for an item that had no box when packed. + leaves: Vec, + /// Items given a box since packing. They have no leaf. + fresh: Vec, +} + +impl Default for Boxes { + fn default() -> Boxes { + Boxes::new(Vec::new()) + } +} + +impl Boxes { + /// Packs `boxes`, by item, into a new index. + pub(crate) fn new(boxes: Vec) -> Boxes { + let index = Index::new(&boxes); + let mut leaves = vec![usize::MAX; boxes.len()]; + for (leaf, &item) in index.items.iter().enumerate() { + leaves[item] = leaf; + } + Boxes { rects: boxes, index, leaves, fresh: Vec::new() } + } + + /// Sets item `item`'s box to `rect` (`EMPTY` for none), adding items as needed. + pub(crate) fn set(&mut self, item: usize, rect: Rect) { + if item >= self.rects.len() { + self.rects.resize(item + 1, EMPTY); + self.leaves.resize(item + 1, usize::MAX); + } + if self.rects[item] == rect { + return; + } + self.rects[item] = rect; + match self.leaves[item] { + usize::MAX if valid(&rect) && !self.fresh.contains(&item) => { + self.fresh.push(item); + if self.fresh.len() > FRESH { + *self = Boxes::new(std::mem::take(&mut self.rects)); + } + } + usize::MAX => {} + leaf => self.index.refit(leaf, if valid(&rect) { rect } else { EMPTY }), + } + } + + /// Item `item`'s box, where it has one. + pub(crate) fn get(&self, item: usize) -> Option { + self.rects.get(item).copied().filter(valid) + } + + /// Visits the items whose boxes meet `region` until `visit` returns true. Returns whether it + /// did. + pub(crate) fn some(&self, region: Rect, visit: &mut impl FnMut(usize) -> bool) -> bool { + let mut seen = |item: usize| self.get(item).is_some_and(|b| meets(&b, ®ion)) && visit(item); + self.index.some(region, &mut seen) || self.fresh.iter().any(|&item| seen(item)) + } + + /// The box around all boxes. + pub(crate) fn extent(&self) -> Rect { + self.fresh.iter().fold(self.index.root(), |extent, &item| union(extent, self.rects[item])) + } + + /// Every item's box, `EMPTY` where it has none, in the form a job takes them. + pub(crate) fn all(&self) -> &[Rect] { + &self.rects + } +} + +/// A packed R-tree over the valid boxes: leaves sorted in Z-order of their centers, [`NODE`] +/// children per node. +struct Index { + /// The item at each leaf. + items: Vec, + /// Each node's box: the leaves, then each level up to the root. + nodes: Vec, + /// Where each level's nodes start, leaves first. The last is the node count. + starts: Vec, +} + +impl Index { + /// An index over `boxes`, leaving out those that are not [`valid`]. + fn new(boxes: &[Rect]) -> Index { + let mut items: Vec = (0..boxes.len()).filter(|&item| valid(&boxes[item])).collect(); + let extent = items.iter().fold(EMPTY, |extent, &item| union(extent, boxes[item])); + let cell = + |v: f64, lo: f64, hi: f64| if hi > lo { ((v - lo) / (hi - lo) * f64::from(u16::MAX)) as u32 } else { 0 }; + crate::order::by_key(&mut items, |&item| { + let middle = [boxes[item][0].midpoint(boxes[item][2]), boxes[item][1].midpoint(boxes[item][3])]; + let at = morton(cell(middle[0], extent[0], extent[2]), cell(middle[1], extent[1], extent[3])); + crate::order::pair(at, item as u32) + }); + let mut nodes: Vec = items.iter().map(|&item| boxes[item]).collect(); + let mut starts = vec![0]; + let mut level = 0; + loop { + let next = nodes.len(); + starts.push(next); + for first in (level..next).step_by(NODE) { + let node = nodes[first..(first + NODE).min(next)].iter().fold(EMPTY, |node, child| union(node, *child)); + nodes.push(node); + } + if nodes.len() - next <= 1 { + break; + } + level = next; + } + starts.push(nodes.len()); + Index { items, nodes, starts } + } + + /// Visits the items meeting `region` until `visit` returns true. Returns whether it did. + fn some(&self, region: Rect, visit: &mut impl FnMut(usize) -> bool) -> bool { + let top = self.starts.len() - 2; + !self.items.is_empty() + && meets(&self.nodes[self.starts[top]], ®ion) + && self.descend(top, self.starts[top], region, visit) + } + + /// [`Index::some`] within the subtree of `node` at `level`. + fn descend(&self, level: usize, node: usize, region: Rect, visit: &mut impl FnMut(usize) -> bool) -> bool { + let first = self.starts[level - 1] + (node - self.starts[level]) * NODE; + let last = (first + NODE).min(self.starts[level]); + (first..last).any(|child| { + meets(&self.nodes[child], ®ion) + && if level > 1 { self.descend(level - 1, child, region, visit) } else { visit(self.items[child]) } + }) + } + + /// Sets leaf `leaf`'s box to `rect`, and refits every node above it to its children. + fn refit(&mut self, leaf: usize, rect: Rect) { + self.nodes[leaf] = rect; + let mut child = leaf; + for level in 1..self.starts.len() - 1 { + let parent = self.starts[level] + (child - self.starts[level - 1]) / NODE; + let first = self.starts[level - 1] + (parent - self.starts[level]) * NODE; + let last = (first + NODE).min(self.starts[level]); + self.nodes[parent] = self.nodes[first..last].iter().fold(EMPTY, |node, child| union(node, *child)); + child = parent; + } + } + + /// The box around its leaves. + fn root(&self) -> Rect { + if self.items.is_empty() { EMPTY } else { self.nodes[self.starts[self.starts.len() - 2]] } + } +} + +/// Position along a Z-order curve on a 65536² grid. +fn morton(x: u32, y: u32) -> u32 { + let spread = |v: u32| { + let v = v & 0xffff; + let v = (v | (v << 8)) & 0x00ff_00ff; + let v = (v | (v << 4)) & 0x0f0f_0f0f; + let v = (v | (v << 2)) & 0x3333_3333; + (v | (v << 1)) & 0x5555_5555 + }; + spread(x) | (spread(y) << 1) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::layout::rect::grow; + + /// The next pseudo-random whole number below `n`, from `seed`. + fn random(seed: &mut u64, n: u32) -> f64 { + *seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1_442_695_040_888_963_407); + f64::from(((*seed >> 33) % u64::from(n)) as u32) + } + + /// A random box within 1000 units, at most 30 across. + fn somewhere(seed: &mut u64) -> Rect { + let [x, y] = [random(seed, 1000), random(seed, 1000)]; + [x, y, x + random(seed, 30), y + random(seed, 30)] + } + + #[test] + fn queries_and_extent_match_a_brute_force_scan_after_edits() { + let mut seed = 3; + let mut all: Vec = (0..500).map(|_| somewhere(&mut seed)).collect(); + let mut boxes = Boxes::new(all.clone()); + for round in 0..300 { + // A box moves, is removed, or is added past the last item. + let item = random(&mut seed, 560) as usize; + let rect = if round % 7 == 0 { EMPTY } else { somewhere(&mut seed) }; + if item >= all.len() { + all.resize(item + 1, EMPTY); + } + all[item] = rect; + boxes.set(item, rect); + let region = grow(somewhere(&mut seed), 40.0); + let mut found = Vec::new(); + boxes.some(region, &mut |item| { + found.push(item); + false + }); + found.sort_unstable(); + let want: Vec = (0..all.len()).filter(|&i| valid(&all[i]) && meets(&all[i], ®ion)).collect(); + assert_eq!(found, want, "round {round}"); + let extent = all.iter().filter(|r| valid(r)).fold(EMPTY, |e, &r| union(e, r)); + assert_eq!(boxes.extent(), extent, "round {round}"); + } + } +} diff --git a/crates/latkit/src/layout/layered.rs b/crates/latkit/src/layout/layered.rs index 4669dbc3..1ad60f06 100644 --- a/crates/latkit/src/layout/layered.rs +++ b/crates/latkit/src/layout/layered.rs @@ -5,31 +5,33 @@ use super::pack; use super::rect::union; use crate::scene::{Direction, Layered}; -/// Width over height of the rows that parts nothing pins pack into. +/// The target width over height of the rows that parts with no pinned vertex pack into. const ASPECT: f64 = 16.0 / 9.0; /// Crossing-reduction sweeps, alternately down and up the ranks. const SWEEPS: usize = 8; -/// Every vertex's center: pinned vertices where they are; the others in ranks along `options.flow` -/// (`Direction::Up` toward +y), `options.gap` apart within a rank and `options.rank_gap` between -/// ranks, ordered to cross few links, each pointing from one vertex to another; cycles are broken. -/// Each connected part is laid out alone, and parts nothing pins pack into rows (aspect 16:9) beside -/// the pinned ones. Across the flow, each vertex lines up with its sources; in a part nothing pins, -/// each source then settles amid the vertices it fans out to. Groups nest as [`nested`] lays them -/// out. +/// Every vertex's center from a layered layout. Pinned vertices stay put. The others go in ranks +/// along `options.direction` (`Direction::Up` is toward +y), `options.gap` apart within a rank and +/// `options.rank_gap` between ranks, ordered to reduce crossings. Links are directed, and cycles +/// are broken. +/// +/// Each connected part is laid out separately, and parts with no pinned vertex pack into rows +/// (aspect 16:9) below the pinned ones. Across the direction, each vertex lines up with its +/// sources. In a part with no pinned vertex, each source then centers among the vertices it fans +/// out to. Groups nest as [`nested`] lays them out. pub(crate) fn layered(graph: &Graph<'_>, options: &Layered) -> Vec<[f64; 2]> { nested(graph, |level| arrange(level, options)) } -/// Places one level: each part that its links join laid out alone in ranks, then the parts nothing -/// pins packed in rows below the pinned ones, tallest first. +/// Places one level: lays out each connected part in ranks, then packs the parts with no pinned +/// vertex in rows below the pinned ones, tallest first. fn arrange(level: &Level, options: &Layered) -> Vec<[f64; 2]> { let (sizes, pins) = (&level.sizes, &level.pinned); let mut centers = vec![[0.0; 2]; sizes.len()]; let (part, count) = parts(sizes.len(), &level.links); let inside = Runs::new(count, (0..sizes.len()).map(|i| (part[i] as usize, i as u32))); let joins = Runs::new(count, level.links.iter().map(|&link| (part[link[0] as usize] as usize, link))); - // Each member's place in its part, whose members run in index order. + // Each member's index within its part. A part's members are in index order. let mut slot = vec![0_u32; sizes.len()]; for p in 0..count { for (s, &i) in inside.get(p).iter().enumerate() { @@ -37,7 +39,7 @@ fn arrange(level: &Level, options: &Layered) -> Vec<[f64; 2]> { } } let vertex = |i: u32| i as usize; - let flow = options.flow; + let direction = options.direction; let mut anchored: Option<[f64; 4]> = None; let mut loose: Vec<(usize, [f64; 4])> = Vec::new(); for p in 0..count { @@ -46,14 +48,14 @@ fn arrange(level: &Level, options: &Layered) -> Vec<[f64; 2]> { if free > 0 && part_members.len() == 1 { centers[vertex(part_members[0])] = [0.0; 2]; } else if free > 0 { - let extents: Vec<[f64; 2]> = part_members.iter().map(|&i| extent(flow, sizes[vertex(i)])).collect(); + let extents: Vec<[f64; 2]> = part_members.iter().map(|&i| extent(direction, sizes[vertex(i)])).collect(); let fixed: Vec> = - part_members.iter().map(|&i| pins[vertex(i)].map(|pin| to_flow(flow, pin))).collect(); + part_members.iter().map(|&i| pins[vertex(i)].map(|pin| to_ranks(direction, pin))).collect(); let part_links: Vec<[u32; 2]> = joins.get(p).iter().map(|&[from, to]| [slot[from as usize], slot[to as usize]]).collect(); let placed = ranked(&extents, &fixed, &part_links, options.gap, options.rank_gap); for (&i, &at) in part_members.iter().zip(&placed) { - centers[vertex(i)] = from_flow(flow, at); + centers[vertex(i)] = from_ranks(direction, at); } } for &i in part_members { @@ -71,7 +73,7 @@ fn arrange(level: &Level, options: &Layered) -> Vec<[f64; 2]> { if loose.is_empty() { return centers; } - // Tallest first, then by first member, so like parts line up. + // Tallest first, ties in part order (by first member), so similar parts line up. crate::order::by_key(&mut loose, |a| !crate::order::number(a.1[3] - a.1[1])); let boxes: Vec<[f64; 2]> = loose.iter().map(|(_, b)| [b[2] - b[0], b[3] - b[1]]).collect(); let gap = options.rank_gap; @@ -87,8 +89,8 @@ fn arrange(level: &Level, options: &Layered) -> Vec<[f64; 2]> { centers } -/// Each of `count` vertices' part, the parts that `links` join numbered by their first vertex, and how many parts -/// there are. +/// Each of `count` vertices' connected part under `links`, with parts numbered in order of their +/// first vertex, and the number of parts. fn parts(count: usize, links: &[[u32; 2]]) -> (Vec, usize) { fn find(root: &mut [u32], mut vertex: u32) -> u32 { while root[vertex as usize] != vertex { @@ -122,10 +124,11 @@ fn parts(count: usize, links: &[[u32; 2]]) -> (Vec, usize) { (part, parts) } -/// Plane coordinates turned so `flow` runs along the first: `[along, across]`, across running down a sideways flow and -/// rightward across an upward or downward one. -fn to_flow(flow: Direction, [x, y]: [f64; 2]) -> [f64; 2] { - match flow { +/// Rank coordinates `[along, across]` of plane coordinates, rotated so `direction` runs along the +/// first axis. Across runs down for a sideways direction, and right for an upward or +/// downward one. +fn to_ranks(direction: Direction, [x, y]: [f64; 2]) -> [f64; 2] { + match direction { Direction::Right => [x, -y], Direction::Left => [-x, -y], Direction::Down => [-y, x], @@ -133,9 +136,9 @@ fn to_flow(flow: Direction, [x, y]: [f64; 2]) -> [f64; 2] { } } -/// Plane coordinates of flow coordinates `[along, across]`; undoes [`to_flow`]. -fn from_flow(flow: Direction, [along, across]: [f64; 2]) -> [f64; 2] { - match flow { +/// Plane coordinates of rank coordinates `[along, across]`. Undoes [`to_ranks`]. +fn from_ranks(direction: Direction, [along, across]: [f64; 2]) -> [f64; 2] { + match direction { Direction::Right => [along, -across], Direction::Left => [-along, -across], Direction::Down => [across, -along], @@ -143,16 +146,17 @@ fn from_flow(flow: Direction, [along, across]: [f64; 2]) -> [f64; 2] { } } -/// A width and height as a length along `flow` and a span across it. -fn extent(flow: Direction, [width, height]: [f64; 2]) -> [f64; 2] { - match flow { +/// A width and height as a length along `direction` and a span across it. +fn extent(direction: Direction, [width, height]: [f64; 2]) -> [f64; 2] { + match direction { Direction::Right | Direction::Left => [width, height], Direction::Down | Direction::Up => [height, width], } } -/// One part's centers in flow coordinates (see [`to_flow`]). `extents` are each vertex's length along the flow and -/// span across it, `pins` the centers of those that stay, and `links` point from one vertex to another. +/// One part's centers in rank coordinates (see [`to_ranks`]). `extents` are each vertex's length +/// along the direction and span across it, `pins` the centers of pinned vertices, and `links` are +/// directed. fn ranked( extents: &[[f64; 2]], pins: &[Option<[f64; 2]>], @@ -164,7 +168,8 @@ fn ranked( let rank = ranks(count, links); let rank_count = rank.iter().max().map_or(0, |&last| last as usize + 1); let mut layers = Runs::new(rank_count, (0..count).map(|i| (rank[i] as usize, i as u32))); - // Each link as its end in the earlier rank, then its end in the later; ranking leaves none within a rank. + // Each link as its end in the earlier rank, then its end in the later. Ranking leaves no link + // within a rank. let ends = |&[from, to]: &[u32; 2]| if rank[from as usize] < rank[to as usize] { [from, to] } else { [to, from] }; let above = Runs::new(count, links.iter().map(ends).map(|[early, late]| (late as usize, early))); let below = Runs::new(count, links.iter().map(ends).map(|[early, late]| (early as usize, late))); @@ -184,8 +189,9 @@ fn ranked( centers } -/// Each vertex's rank: the longest path to it from a vertex nothing points to, once the links that close cycles are -/// set aside. A walk depth first from the vertices nothing points to finds those links. +/// Each vertex's rank: the longest path to it from a source (a vertex no link points to), ignoring +/// the links that close cycles. A depth-first walk from the sources, then from any vertex left, +/// finds those links. fn ranks(count: usize, links: &[[u32; 2]]) -> Vec { const NEW: u8 = 0; const WALKING: u8 = 1; @@ -198,7 +204,7 @@ fn ranks(count: usize, links: &[[u32; 2]]) -> Vec { } let mut roots: Vec = (0..count as u32).collect(); crate::order::by_key(&mut roots, |&vertex| u64::from(pointed[vertex as usize])); - // A link into a vertex still on the walk closes a cycle; every other link keeps its direction. + // A link into a vertex still on the walk closes a cycle. Every other link keeps its direction. let mut state = vec![NEW; count]; let mut forward: Vec<[u32; 2]> = Vec::with_capacity(links.len()); let mut walk: Vec<(u32, usize)> = Vec::new(); @@ -247,9 +253,9 @@ fn ranks(count: usize, links: &[[u32; 2]]) -> Vec { rank } -/// Orders each rank to cross few edges. Barycenter sweeps run alternately down the ranks, sorting each by where its -/// vertices' neighbors in earlier ranks sit, and up them by their neighbors in later ranks; the order that crosses the -/// fewest edges between adjacent ranks stays. +/// Orders each rank to reduce edge crossings. Barycenter sweeps alternate: down the ranks, sorting +/// each by the mean position of its vertices' neighbors in earlier ranks, then up, by neighbors in +/// later ranks. The order with the fewest crossings between adjacent ranks is kept. fn reorder(layers: &mut Runs, above: &Runs, below: &Runs, adjacent: &Runs<[u32; 2]>) { let mut position = vec![0_u32; above.len()]; for k in 0..layers.len() { @@ -294,18 +300,18 @@ fn settle(layer: &[u32], position: &mut [u32]) { } } -/// Counts edges crossing between adjacent ranks, keeping its buffers from one count to the next. +/// Counts edge crossings between adjacent ranks, reusing its buffers from one count to the next. #[derive(Default)] struct Crossings { - /// Where each edge between two ranks starts and ends in them. + /// Each edge's position in the earlier rank and in the later rank. ends: Vec<(u32, u32)>, - /// A Fenwick tree of how many edges so far end at each place of the later rank. + /// A Fenwick tree counting the edges seen so far that end at each position of the later rank. ended: Vec, } impl Crossings { - /// Edges between adjacent ranks that cross, given where each vertex sits in its rank: once sorted by where they - /// start, each edge crosses the earlier ones ending past it. + /// The number of crossing edge pairs between adjacent ranks, given each vertex's position in + /// its rank. With edges sorted by start, each edge crosses the earlier ones that end after it. fn count(&mut self, layers: &Runs, position: &[u32], adjacent: &Runs<[u32; 2]>) -> u64 { let mut total = 0; for k in 1..layers.len() { @@ -337,10 +343,11 @@ impl Crossings { } } -/// Each vertex's center in flow coordinates. Ranks run `rank_gap` apart along the flow, each vertex centered in its -/// rank's band; across it, each vertex sits in line with the median of its neighbors in earlier ranks unless that -/// would bring it within `gap` of the vertex before it in its rank. A rank holding pinned vertices starts where they -/// do, and vertices step past the pinned ones they would come within `gap` of. +/// Each vertex's center in rank coordinates. Ranks run `rank_gap` apart along the direction, each +/// vertex centered in its rank's band. Across the direction, each vertex lines up with the median +/// of its neighbors in earlier ranks, unless that brings it within `gap` of the previous vertex in +/// its rank. A rank holding pinned vertices starts where they do, and vertices step past pinned +/// ones they would come within `gap` of. fn coordinates( extents: &[[f64; 2]], pins: &[Option<[f64; 2]>], @@ -353,12 +360,12 @@ fn coordinates( let length_of = |vertex: u32| extents[vertex as usize][0]; let bands: Vec = (0..layers.len()).map(|k| layers.get(k).iter().map(|&i| length_of(i)).fold(0.0, f64::max)).collect(); - // Where a rank's pinned vertices begin along the flow. + // Where a rank's pinned vertices begin along the direction. let lead = |k: usize| { layers.get(k).iter().filter_map(|&i| pins[i as usize].map(|pin| pin[0] - length_of(i) / 2.0)).reduce(f64::min) }; let first = (0..layers.len()).find_map(|k| lead(k).map(|at| (k, at))); - // Ranks before the first pinned one end where it starts. + // Ranks before the first rank with a pinned vertex end where that rank starts. let mut major = first.map_or(0.0, |(k, at)| at - bands[..k].iter().map(|band| band + rank_gap).sum::()); let mut obstacles = first.map(|_| Obstacles::new(extents, pins, [gap.min(rank_gap), gap])); let mut wanted: Vec = Vec::new(); @@ -403,9 +410,9 @@ fn coordinates( centers } -/// Moves each vertex across the flow toward the middle of its neighbors in later ranks, the last ranks first, as far -/// as the vertices beside it in its rank allow, `gap` apart: a source settles amid the vertices it fans out to, and a -/// chain stays in line. +/// Moves each vertex across the direction toward the median of its neighbors in later ranks, last +/// ranks first, as far as its neighbors in its rank allow at `gap` apart. A source centers among +/// the vertices it fans out to, and a chain stays in line. fn balance(centers: &mut [[f64; 2]], extents: &[[f64; 2]], layers: &Runs, below: &Runs, gap: f64) { let mut across: Vec = Vec::new(); let mut goals: Vec> = Vec::new(); @@ -451,11 +458,11 @@ fn balance(centers: &mut [[f64; 2]], extents: &[[f64; 2]], layers: &Runs, b } } -/// Placed boxes in flow coordinates, bucketed in square cells to find those near a box quickly. +/// Placed boxes in rank coordinates, bucketed in square cells to find those near a box quickly. struct Obstacles { /// A cell's side. cell: f64, - /// The room boxes keep between them along the flow and across it. + /// The clearance boxes keep between them, along the direction and across it. room: [f64; 2], /// Each box: its least along and across, then its greatest. boxes: Vec<[f64; 4]>, @@ -464,7 +471,8 @@ struct Obstacles { } impl Obstacles { - /// The pinned vertices' boxes, keeping `room` between boxes. Cells span a few vertices, so a box reaches few. + /// Obstacles holding the pinned vertices' boxes, keeping `room` between boxes. Cells span a few + /// vertices, so each box reaches few cells. fn new(extents: &[[f64; 2]], pins: &[Option<[f64; 2]>], room: [f64; 2]) -> Obstacles { let largest = extents.iter().flatten().fold(0.0, |largest: f64, &side| largest.max(side)); let cell = (room[1] * 8.0).max(largest * 2.0); @@ -491,7 +499,7 @@ impl Obstacles { } } - /// The greatest across of the boxes that `query` comes within room of, if any. + /// The greatest across edge among the boxes that `query` comes within `room` of, if any. fn reach(&self, query: [f64; 4]) -> Option { let [along, across] = self.room; let grown = [query[0] - along, query[1] - across, query[2] + along, query[3] + across]; @@ -526,8 +534,8 @@ mod tests { const SIZE: [f64; 2] = [20.0, 10.0]; - fn options(flow: Direction) -> Layered { - Layered { flow, gap: 10.0, rank_gap: 40.0 } + fn options(direction: Direction) -> Layered { + Layered { direction, gap: 10.0, rank_gap: 40.0 } } /// A graph's columns, owned. @@ -540,8 +548,8 @@ mod tests { } impl Columns { - /// `count` vertices of [`SIZE`] joined by `edges`, none pinned or grouped; a group pads its members by - /// half a gap. + /// `count` vertices of [`SIZE`] joined by `edges`, none pinned or grouped. A group pads its + /// members by half a gap. fn new(count: usize, edges: &[[u32; 2]]) -> Columns { Columns { sizes: vec![SIZE; count], @@ -584,7 +592,7 @@ mod tests { (a - b).abs() < 1e-9 } - /// Edges that cross between vertices in the same two ranks of a rightward flow. + /// How many edge pairs cross between the same two ranks of a rightward layout. fn crossings(edges: &[[u32; 2]], centers: &[[f64; 2]]) -> usize { let mut count = 0; for (k, first) in edges.iter().enumerate() { @@ -601,11 +609,11 @@ mod tests { } #[test] - fn a_chain_ranks_along_the_flow_in_order() { + fn a_chain_ranks_along_the_direction_in_order() { let graph = Columns::new(4, &[[2, 3], [0, 1], [1, 2]]); - for flow in [Direction::Right, Direction::Left, Direction::Down, Direction::Up] { - let centers = graph.layout(&options(flow)); - let (along, across, sign) = match flow { + for direction in [Direction::Right, Direction::Left, Direction::Down, Direction::Up] { + let centers = graph.layout(&options(direction)); + let (along, across, sign) = match direction { Direction::Right => (0, 1, 1.0), Direction::Left => (0, 1, -1.0), Direction::Down => (1, 0, -1.0), @@ -614,8 +622,8 @@ mod tests { let length = SIZE[along]; for pair in centers.windows(2) { let step = (pair[1][along] - pair[0][along]) * sign; - assert!(near(step, length + 40.0), "{flow:?}: ranks a rank gap apart: {centers:?}"); - assert!(near(pair[1][across], pair[0][across]), "{flow:?}: in line with its source: {centers:?}"); + assert!(near(step, length + 40.0), "{direction:?}: ranks a rank gap apart: {centers:?}"); + assert!(near(pair[1][across], pair[0][across]), "{direction:?}: in line with its source: {centers:?}"); } } } @@ -624,7 +632,7 @@ mod tests { fn a_source_centers_on_its_fan() { // 0 fans out to 1, 2, and 3; 3 runs on to 4. let centers = Columns::new(5, &[[0, 1], [0, 2], [0, 3], [3, 4]]).layout(&options(Direction::Down)); - assert!(near(centers[0][0], centers[2][0]), "the source sits amid its fan: {centers:?}"); + assert!(near(centers[0][0], centers[2][0]), "the source is centered on its fan: {centers:?}"); assert!(near(centers[3][0], centers[4][0]), "a chain stays in line: {centers:?}"); assert!(centers[1][0] < centers[2][0] && centers[2][0] < centers[3][0], "{centers:?}"); } @@ -671,7 +679,8 @@ mod tests { #[test] fn sweeps_untangle_crossings() { - // A K2,2 short of one edge in two ranks of two, and three ranks of three; both cross in index order. + // A K2,2 minus one edge in two ranks of two, and a graph in three ranks of three. Both + // cross when laid out in index order. let small = [[0, 3], [1, 2], [1, 3]]; let large = [[0, 5], [1, 4], [2, 3], [2, 4], [3, 8], [4, 7], [5, 6]]; for (count, width, edges) in [(4, 2, &small[..]), (9, 3, &large[..])] { @@ -683,8 +692,9 @@ mod tests { } #[test] - fn groups_hold_their_members_apart_from_the_rest() { - // Group 0 holds 1 → 2 → 3 and group 6, which holds 7 → 8; 4 feeds the group and 5 drains it. + fn groups_contain_their_members_and_keep_other_vertices_out() { + // Group 0 holds 1 → 2 → 3 and group 6, which holds 7 → 8. Vertex 4 feeds group 0, and 5 + // drains it. let mut graph = Columns::new(9, &[[1, 2], [2, 3], [4, 1], [3, 5], [2, 7], [7, 8]]); graph.groups = vec![None, Some(0), Some(0), Some(0), None, None, Some(0), Some(6), Some(6)]; let options = options(Direction::Right); @@ -707,7 +717,7 @@ mod tests { } #[test] - fn a_pinned_member_holds_its_group_and_a_pinned_group_carries_its_members() { + fn a_pinned_member_fixes_its_group_and_a_pinned_group_moves_its_members() { let mut graph = Columns::new(7, &[[1, 2], [3, 0], [5, 6], [4, 5]]); graph.groups = vec![None, Some(0), Some(0), None, None, Some(4), Some(4)]; graph.pinned[1] = Some([300.0, 200.0]); @@ -723,8 +733,8 @@ mod tests { } #[test] - fn large_graphs_lay_out_apart() { - // Fifty groups of twenty, and edges joining random vertices, many against the flow. + fn large_graphs_lay_out_without_overlaps() { + // Fifty groups of twenty, and edges joining random vertices, many against the direction. let count = 10_000; let mut seed = 0x9e37_79b9_u32; let mut random = move |below: usize| { diff --git a/crates/latkit/src/layout/levels.rs b/crates/latkit/src/layout/levels.rs index 198ed620..49eab447 100644 --- a/crates/latkit/src/layout/levels.rs +++ b/crates/latkit/src/layout/levels.rs @@ -1,12 +1,14 @@ -//! Nested graphs laid out a level at a time. A level is the vertices of the top, or the members of -//! one group. A group's members are placed first, innermost groups first; the group then stands as -//! one vertex of its box's size in the level holding it, with links between members of different -//! groups lifted to the groups they meet in. Every layout places one level at a time, so every -//! layout nests alike. +//! Nested graphs, laid out one level at a time. +//! +//! A level is the top-level vertices, or the members of one group. Groups are laid out innermost +//! first: a group's members are placed, then the group acts as one vertex, sized to its box, in +//! the level holding it. Links between members of different groups are lifted to the groups that +//! share a level. Every layout places one level at a time, so all layouts nest the same way. -/// A graph to lay out, plane units with y up: each vertex's width and height, its center when it has -/// one, the group holding it, and how far a group's box reaches past its members (left, bottom, -/// right, top); and links between vertices, from one to another where a layout reads direction. +/// A graph to lay out, in plane units with y up. It holds each vertex's width and height, its +/// pinned center if any, the group holding it, and its padding as a group (how far its box reaches +/// past its members: left, bottom, right, top). It also holds the links between vertices, directed +/// where a layout reads direction. pub(crate) struct Graph<'a> { pub(crate) sizes: &'a [[f64; 2]], pub(crate) links: &'a [[u32; 2]], @@ -22,15 +24,16 @@ pub(crate) struct Level { pub(crate) pinned: Vec>, } -/// Every vertex's center, `place` answering each level's. A group's center is the middle of its box: -/// its members' bounds grown by its pads. A pinned member holds its group where its members settled; -/// a group's own pin moves its members only when none of them is pinned. Groups out of range, of -/// themselves, or closing a cycle hold nothing. +/// Every vertex's center, with `place` laying out each level. A group's center is the middle of +/// its box: its members' bounds grown by its pads. A group with a pinned member is pinned where its +/// members settled. A group's own pin moves its members only when none of them is pinned. Group +/// references out of range, to the vertex itself, or closing a cycle are ignored. Panics unless +/// `pinned`, `groups`, and `pads` have an entry per vertex. pub(crate) fn nested(graph: &Graph<'_>, mut place: impl FnMut(&Level) -> Vec<[f64; 2]>) -> Vec<[f64; 2]> { let count = graph.sizes.len(); assert!( graph.pinned.len() == count && graph.groups.len() == count && graph.pads.len() == count, - "a pin, a group, and pads for each vertex" + "pinned, groups, and pads need an entry per vertex" ); let parent = tree(graph.groups); let depth = depths(&parent); @@ -54,7 +57,7 @@ pub(crate) fn nested(graph: &Graph<'_>, mut place: impl FnMut(&Level) -> Vec<[f6 graph.sizes.iter().map(|&[width, height]| [width.max(0.0), height.max(0.0)]).collect(); let mut pins: Vec> = graph.pinned.iter().map(|pin| pin.filter(|at| at[0].is_finite() && at[1].is_finite())).collect(); - // Centers within each vertex's level, and each group's members' middle within its own. + // Each vertex's center within its level, and each group's box middle in its members' frame. let mut centers = vec![[0.0; 2]; count]; let mut middles = vec![[0.0; 2]; count]; let mut local = vec![0_u32; count]; @@ -104,8 +107,8 @@ pub(crate) fn nested(graph: &Graph<'_>, mut place: impl FnMut(&Level) -> Vec<[f6 .collect() } -/// Each vertex's group: groups out of range or of themselves dropped, and each cycle of groups cut where a walk up it -/// closes. +/// Each vertex's group, dropping groups out of range or equal to the vertex itself, and cutting +/// each cycle of groups where a walk up it closes. fn tree(groups: &[Option]) -> Vec> { const NEW: u8 = 0; const WALKING: u8 = 1; @@ -123,7 +126,7 @@ fn tree(groups: &[Option]) -> Vec> { loop { if state[at] != NEW { if state[at] == WALKING { - parent[*walk.last().expect("a walk reaches itself only after a step")] = None; + parent[*walk.last().expect("a walk that closes a cycle is not empty")] = None; } break; } @@ -154,7 +157,7 @@ fn depths(parent: &[Option]) -> Vec { None => break, } } - // The walk ends at a vertex already counted, or at the top, which it holds. + // The walk ends at a vertex already counted, or at a top-level vertex it includes. let mut next = if depth[at] == u32::MAX { 0 } else { depth[at] + 1 }; while let Some(vertex) = walk.pop() { depth[vertex] = next; @@ -164,13 +167,14 @@ fn depths(parent: &[Option]) -> Vec { depth } -/// The link `from` → `to` between the outermost groups (or vertices) that hold each end and share a level; `None` for -/// a link within one of them or with an end out of range. +/// The link `from` → `to` lifted to the level where it is laid out: between the ancestors of its +/// ends (or the ends themselves) that share a parent. `None` for a link inside one group or vertex, +/// or with an end out of range. fn lift(from: u32, to: u32, parent: &[Option], depth: &[u32]) -> Option<[u32; 2]> { if from as usize >= parent.len() || to as usize >= parent.len() { return None; } - let up = |vertex: usize| parent[vertex].expect("a held vertex has a group") as usize; + let up = |vertex: usize| parent[vertex].expect("a nested vertex has a group") as usize; let (mut from, mut to) = (from as usize, to as usize); while depth[from] > depth[to] { from = up(from); @@ -190,20 +194,21 @@ pub(crate) fn frame(center: [f64; 2], size: [f64; 2]) -> [f64; 4] { [center[0] - half[0], center[1] - half[1], center[0] + half[0], center[1] + half[1]] } -/// The box holding `boxes`; a point at the origin for none. +/// The box holding `boxes`, or a point at the origin for none. pub(crate) fn bounds(boxes: impl Iterator) -> [f64; 4] { boxes.reduce(super::rect::union).unwrap_or([0.0; 4]) } -/// Items bucketed by key, each bucket's in the order given: bucket `k` is `items[offsets[k]..offsets[k + 1]]`. +/// Items bucketed by key, each bucket in the order given: bucket `k` is +/// `items[offsets[k]..offsets[k + 1]]`. pub(crate) struct Runs { offsets: Vec, - /// Every bucket's items, bucket after bucket; reordered within buckets, they stay bucketed. + /// Every bucket's items, bucket after bucket. Reordering within a bucket keeps them bucketed. pub(crate) items: Vec, } impl Runs { - /// `keyed` items in `buckets` buckets, each item's key below `buckets`. + /// Buckets `keyed` items into `buckets` buckets. Each key must be below `buckets`. pub(crate) fn new(buckets: usize, keyed: impl Iterator + Clone) -> Runs { let mut offsets = vec![0; buckets + 1]; for (key, _) in keyed.clone() { @@ -224,7 +229,7 @@ impl Runs { } impl Runs { - /// How many buckets. + /// The number of buckets. pub(crate) fn len(&self) -> usize { self.offsets.len() - 1 } @@ -234,7 +239,7 @@ impl Runs { &self.items[self.offsets[k]..self.offsets[k + 1]] } - /// Bucket `k`'s items, to change. + /// Bucket `k`'s items, mutably. pub(crate) fn get_mut(&mut self, k: usize) -> &mut [T] { &mut self.items[self.offsets[k]..self.offsets[k + 1]] } diff --git a/crates/latkit/src/layout/mod.rs b/crates/latkit/src/layout/mod.rs index 94e56c18..15e1fa6a 100644 --- a/crates/latkit/src/layout/mod.rs +++ b/crates/latkit/src/layout/mod.rs @@ -1,5 +1,7 @@ -//! Layouts on the CPU: layered drawing and stress over nested graphs, packing of parts, overlap -//! removal, and a diagram's orthogonal routes. Each is pure, so it runs as well in a host's job. +//! CPU graph layouts: layered and stress layouts of nested graphs, packing of disconnected parts, +//! overlap removal, orthogonal wire routing for diagrams, and an R-tree of boxes. Each is a pure +//! function, so it runs the same on the frame or in a host job. +pub(crate) mod boxes; pub(crate) mod layered; pub(crate) mod levels; pub(crate) mod overlap; diff --git a/crates/latkit/src/layout/overlap.rs b/crates/latkit/src/layout/overlap.rs index 541c80ac..9d6c36f6 100644 --- a/crates/latkit/src/layout/overlap.rs +++ b/crates/latkit/src/layout/overlap.rs @@ -1,22 +1,25 @@ -//! Overlap removal: boxes pushed apart until no two come within a clearance of each other. +//! Overlap removal: pushes boxes apart until no two are within a clearance of each other. use rustc_hash::FxHashMap; -/// Rounds of pairwise pushing, before a sweep parts what they leave. +/// Rounds of pairwise pushing before a sweep separates the overlaps that remain. const ROUNDS: usize = 40; -/// Each box's center once no two of `boxes` (center, then half width and height) come within -/// `clearance` of each other, pinned ones aside. First rounds of pushing: a pair too close parts -/// along the axis it overlaps least, each half the way, or a free box all of it when the other is -/// pinned. That moves boxes least, but a crowd jams; a sweep along x then places each box after the -/// pinned ones, in order, right of any placed box it overlaps. Pairs are found in a grid of cells as -/// wide as the widest box, so each round, and the sweep, is linear in the boxes for boxes of like -/// sizes. +/// Each box's center once no two of `boxes` (center x, center y, half width, half height) are +/// within `clearance` of each other. Pinned boxes never move, and two pinned boxes may still +/// overlap. +/// +/// First come rounds of pushing. A pair too close parts along the axis where it overlaps least. +/// Each box moves half the overlap, or the free box moves all of it when the other is pinned. This +/// moves boxes least, but a crowd can jam. If overlaps remain after `ROUNDS`, a sweep places the +/// free boxes in order of x after the pinned ones. Each is pushed right past any placed box it +/// overlaps. Pairs are found in a grid of cells as wide as the widest box, so each round and the +/// sweep are linear in the number of boxes when their sizes are similar. pub(crate) fn separate(boxes: &[[f64; 4]], pinned: &[bool], clearance: f64) -> Vec<[f64; 2]> { let mut centers: Vec<[f64; 2]> = boxes.iter().map(|b| [b[0], b[1]]).collect(); let half: Vec<[f64; 2]> = boxes.iter().map(|b| [b[2], b[3]].map(|h| h.max(0.0) + clearance / 2.0)).collect(); let grid = Grid { cell: 2.0 * half.iter().flatten().copied().fold(1e-9, f64::max) }; - // Overlaps this small are rounding, not overlap. + // Overlaps this small are rounding error. let slack = grid.cell * 1e-9; let overlap = |centers: &[[f64; 2]], i: usize, j: usize| { let over = [0, 1].map(|k| half[i][k] + half[j][k] - (centers[j][k] - centers[i][k]).abs()); @@ -33,7 +36,7 @@ pub(crate) fn separate(boxes: &[[f64; 4]], pinned: &[bool], clearance: f64) -> V } let Some(over) = overlap(¢ers, i, j) else { continue }; let d = [0, 1].map(|k| centers[j][k] - centers[i][k]); - // Boxes at one point part by their order, across and along by turns. + // Coincident boxes part along x or y, alternating with the parity of `i + j`. let k = if d == [0.0; 2] { (i + j) % 2 } else { usize::from(over[1] < over[0]) }; let sign = if d[k] == 0.0 { 1.0 } else { d[k].signum() }; let share = if pinned[i] { @@ -56,7 +59,7 @@ pub(crate) fn separate(boxes: &[[f64; 4]], pinned: &[bool], clearance: f64) -> V crate::order::by_key(&mut order, |&a| crate::order::number(centers[a][0])); grid.fill(&mut cells, ¢ers, (0..centers.len()).filter(|&i| pinned[i])); for i in order { - // Right past the placed boxes it overlaps, until it overlaps none. + // Push it right past the placed boxes it overlaps until it overlaps none. while let Some(x) = grid .near(&cells, centers[i]) .filter_map(|j| overlap(¢ers, j, i).map(|_| centers[j][0] + half[j][0] + half[i][0])) @@ -69,7 +72,8 @@ pub(crate) fn separate(boxes: &[[f64; 4]], pinned: &[bool], clearance: f64) -> V centers } -/// Square cells of boxes' centers, wide enough that boxes that overlap stand in neighboring cells. +/// A grid of square cells holding box centers, wide enough that overlapping boxes are in the same +/// or neighboring cells. struct Grid { cell: f64, } @@ -79,7 +83,7 @@ impl Grid { ((c[0] / self.cell).floor() as i64, (c[1] / self.cell).floor() as i64) } - /// Fills `cells` with the boxes `which`. + /// Empties `cells`, then fills them with the boxes `which`. fn fill( &self, cells: &mut FxHashMap<(i64, i64), Vec>, @@ -92,7 +96,7 @@ impl Grid { } } - /// The boxes in the cells around `at`. + /// The boxes in the 3 × 3 cells around `at`. fn near<'a>(&self, cells: &'a FxHashMap<(i64, i64), Vec>, at: [f64; 2]) -> impl Iterator + 'a { let (x, y) = self.key(at); [-1, 0, 1] @@ -119,7 +123,7 @@ mod tests { } #[test] - fn a_pair_parts_along_its_lesser_overlap() { + fn a_pair_separates_along_its_smaller_overlap() { // Each half grows by half the clearance: 12 and 12 overlap 15 apart by 9 across, 22 along. let boxes = [[0.0, 0.0, 10.0, 10.0], [15.0, 2.0, 10.0, 10.0]]; let centers = separate(&boxes, &[false, false], 4.0); @@ -127,7 +131,7 @@ mod tests { } #[test] - fn pinned_boxes_hold() { + fn pinned_boxes_stay_put() { let boxes = [[0.0, 0.0, 10.0, 10.0], [15.0, 0.0, 10.0, 10.0]]; let centers = separate(&boxes, &[true, false], 0.0); assert_eq!(centers, vec![[0.0, 0.0], [20.0, 0.0]]); @@ -139,6 +143,6 @@ mod tests { let pinned: Vec = (0..60).map(|i| i == 7).collect(); let centers = separate(&boxes, &pinned, 2.0); assert!(clear(&boxes, ¢ers, 2.0), "{centers:?}"); - assert_eq!(centers[7], [0.0, 0.0], "the pinned box holds"); + assert_eq!(centers[7], [0.0, 0.0], "the pinned box stays put"); } } diff --git a/crates/latkit/src/layout/pack.rs b/crates/latkit/src/layout/pack.rs index 25cb7317..09d3502b 100644 --- a/crates/latkit/src/layout/pack.rs +++ b/crates/latkit/src/layout/pack.rs @@ -1,19 +1,19 @@ -/// Offsets that pack boxes (each `[width, height]`) into rows `gap` apart whose width over height approaches `aspect`, -/// in the order given; the first box's top left lands at the origin. Returns each box's top-left corner (y up: top is -/// the greater y). +/// Packs boxes (each `[width, height]`) in the given order into rows `gap` apart, so the result's +/// width over height approaches `aspect`. Returns each box's top-left corner, with y up (top is the +/// greater y). The first box's top left is at the origin. pub(crate) fn pack(boxes: &[[f64; 2]], gap: f64, aspect: f64) -> Vec<[f64; 2]> { rows(boxes, gap, span(boxes, gap, aspect)) } -/// How wide rows of `boxes` run for the rows' width over height to approach `aspect`: as wide as a rectangle of that -/// aspect holding all the boxes and their gaps. +/// The row width at which rows of `boxes` approach width over height `aspect`: the width of a +/// rectangle of that aspect whose area is the boxes' total area, gaps included. pub(super) fn span(boxes: &[[f64; 2]], gap: f64, aspect: f64) -> f64 { let area: f64 = boxes.iter().map(|[width, height]| (width + gap) * (height + gap)).sum(); (area * aspect).sqrt() } -/// Each box's top-left corner in rows `gap` apart, going down from the origin. A row breaks before a box that would -/// run it past `span`, so only a box wider than `span` makes a row wider. +/// Each box's top-left corner, in rows `gap` apart going down from the origin. A row breaks before +/// a box that would extend it past `span`, so only a box wider than `span` makes a row wider. pub(super) fn rows(boxes: &[[f64; 2]], gap: f64, span: f64) -> Vec<[f64; 2]> { let (mut x, mut top, mut tallest) = (0.0, 0.0, 0.0_f64); boxes @@ -37,7 +37,7 @@ pub(super) fn rows(boxes: &[[f64; 2]], gap: f64, span: f64) -> Vec<[f64; 2]> { mod tests { use super::*; - /// Boxes of uneven sizes, the same each run. + /// Boxes of uneven sizes, the same on every run. fn uneven(count: usize) -> Vec<[f64; 2]> { let mut seed = 0x2545_f491_u32; let mut next = move || { diff --git a/crates/latkit/src/layout/rect.rs b/crates/latkit/src/layout/rect.rs index 08c31932..fe5a2d54 100644 --- a/crates/latkit/src/layout/rect.rs +++ b/crates/latkit/src/layout/rect.rs @@ -1,4 +1,4 @@ -//! Boxes on the plane, least x and y then greatest, and the points they hold. +//! Axis-aligned boxes on the plane, as least x and y then greatest x and y, and points. /// A point on the plane. pub(crate) type Point = [f64; 2]; @@ -6,20 +6,20 @@ pub(crate) type Point = [f64; 2]; /// A box: least x, least y, greatest x, greatest y. pub(crate) type Rect = [f64; 4]; -/// A box holding nothing, which any union replaces. +/// The empty box. It is inverted, so a union with it yields the other box. pub(crate) const EMPTY: Rect = [f64::INFINITY, f64::INFINITY, f64::NEG_INFINITY, f64::NEG_INFINITY]; -/// The box holding `a` and `b`. +/// The smallest box containing `a` and `b`. pub(crate) fn union(a: Rect, b: Rect) -> Rect { [a[0].min(b[0]), a[1].min(b[1]), a[2].max(b[2]), a[3].max(b[3])] } -/// The box holding only `p`. +/// The zero-size box at `p`. pub(crate) fn dot(p: Point) -> Rect { [p[0], p[1], p[0], p[1]] } -/// The box `half` its width and height each way about `center`. +/// The box centered on `center` that reaches `half` (half width, half height) each way. pub(crate) fn around(center: Point, half: Point) -> Rect { [center[0] - half[0], center[1] - half[1], center[0] + half[0], center[1] + half[1]] } @@ -29,7 +29,12 @@ pub(crate) fn grow(r: Rect, by: f64) -> Rect { [r[0] - by, r[1] - by, r[2] + by, r[3] + by] } -/// Whether two boxes meet, touching included. +/// Whether two boxes overlap or touch. pub(crate) fn meets(a: &Rect, b: &Rect) -> bool { a[0] <= b[2] && a[2] >= b[0] && a[1] <= b[3] && a[3] >= b[1] } + +/// Whether a box is finite and not inverted. +pub(crate) fn valid(r: &Rect) -> bool { + r.iter().all(|v| v.is_finite()) && r[0] <= r[2] && r[1] <= r[3] +} diff --git a/crates/latkit/src/layout/route.rs b/crates/latkit/src/layout/route.rs index 5e86d274..90f55e50 100644 --- a/crates/latkit/src/layout/route.rs +++ b/crates/latkit/src/layout/route.rs @@ -1,27 +1,28 @@ -//! Orthogonal routes: wires in horizontal and vertical runs around obstacles. +//! Orthogonal routes: wires of horizontal and vertical runs around obstacles. //! -//! Each net grows as a tree from its first terminal. Every terminal leaves its vertex along its side -//! by a stub, and each further terminal, nearest first, branches off the tree by the cheapest clear -//! elbow, or else by an A* search over a sparse visibility graph: the free stretches of the lines -//! through the nearby obstacles' corners, joined where they cross. Tracks then set apart the runs of -//! different nets that share a line. +//! Each net grows as a tree from its first terminal. Every terminal leaves its vertex by a short +//! stub along its side. Each further terminal, nearest first, branches off the tree by the +//! cheapest clear elbow, or else by an A* search over a sparse visibility graph: the free +//! stretches of the lines through nearby obstacles' corners, joined where they cross. Finally, +//! runs of different nets that share a line are spread onto separate tracks. //! -//! Sides are in plane units with y up: a top side faces +y. +//! Coordinates are plane units with y up: a top side faces +y. use std::cmp::Ordering; use std::collections::BinaryHeap; -use super::rect::{EMPTY, Point, Rect, dot, grow, meets, union}; +use super::boxes::Boxes; +use super::rect::{Point, Rect, dot, grow, meets, union, valid}; use crate::scene::Side; -/// Where a wire meets a vertex: the point on its boundary, the side it leaves by (None: any), and the -/// obstacle it belongs to (an index into `obstacles`), which its own wire may touch. +/// Where a wire meets a vertex: the point on its boundary, the side it leaves by (`None`: any), and +/// the obstacle it belongs to (an index into `obstacles`), which its own wire may touch. #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) struct Terminal { /// The point on the vertex's boundary. pub(crate) at: [f64; 2], - /// The side the wire leaves by; with none, the owner's side nearest `at`, or else the way toward - /// the rest of the net. + /// The side the wire leaves by. With none, the wire leaves by the owner's side nearest `at`, or + /// without an owner, along the axis toward the rest of the net. pub(crate) side: Option, /// The obstacle the terminal belongs to. pub(crate) owner: Option, @@ -30,22 +31,23 @@ pub(crate) struct Terminal { /// A net: every terminal it joins. Two terminals make a plain edge. #[derive(Clone, PartialEq, Debug, Default)] pub(crate) struct Net { - /// The terminals; the route grows from the first. + /// The terminals. The route grows from the first. pub(crate) terminals: Vec, } -/// Each net's route as polylines (a tree for nets of more than two terminals, sharing junction points), -/// running in horizontal and vertical segments around `obstacles` (each `[min x, min y, max x, max y]`), -/// leaving each terminal along its side, with parallel runs of different nets `spacing` apart. Runs of -/// `fixed` polylines, wires already routed, keep their tracks, and the new runs take tracks beside -/// them. A net whose route cannot avoid the obstacles still gets an orthogonal route through them. +/// Each net's route as polylines: a tree for nets of more than two terminals, sharing junction +/// points. Routes run in horizontal and vertical segments around `obstacles` (each +/// `[min x, min y, max x, max y]`), leave each terminal along its side, and keep parallel runs of +/// different nets `spacing` apart. Runs of `fixed` polylines (wires already routed) keep their +/// tracks, and new runs take tracks beside them. A net whose route cannot avoid the obstacles +/// still gets an orthogonal route through them. /// -/// Wires keep twice `spacing` from obstacles where they can. `groups` are soft: wires cross their -/// borders at a cost, so they enter and leave a group once, and keep off its borders. A net of fewer -/// than two terminals at finite points gets no polylines; obstacles with a non-finite or inverted box -/// are left out. +/// Wires keep twice `spacing` from obstacles where they can. `groups` are soft obstacles: crossing +/// a border costs extra, so wires enter and leave a group once and keep off its borders. A net +/// with fewer than two terminals at finite points gets no polylines. Obstacles with a non-finite +/// or inverted box are ignored. pub(crate) fn route( - obstacles: &[[f64; 4]], + obstacles: &Boxes, groups: &[[f64; 4]], nets: &[Net], fixed: &[Vec<[f64; 2]>], @@ -60,13 +62,13 @@ pub(crate) fn route( trees.iter().map(|tree| tree.as_ref().map_or_else(Vec::new, Tree::polylines)).collect() } -/// Arc points a rounded bend takes between its ends. +/// Points a rounded bend's arc has between its ends. const ARC: usize = 3; -/// A net's polylines as drawn: each bend between perpendicular runs rounded by up to `radius`, and -/// each part ending where another starts doubled at its end, so that point draws as a junction and -/// takes no arrowhead. No point repeats anywhere else, as where two bends' arcs meet on a short run: -/// it would draw as a junction too. +/// A net's polylines as drawn. Each bend between perpendicular runs is rounded with a radius of up +/// to `radius`. A part ending where another starts repeats its last point, so that point draws as +/// a junction without an arrowhead. No point repeats anywhere else (for example where two arcs +/// meet on a short run), since a repeat would draw as a junction too. pub(crate) fn finish(lines: &[Vec], radius: f64) -> Vec> { lines .iter() @@ -102,7 +104,7 @@ pub(crate) fn finish(lines: &[Vec], radius: f64) -> Vec> { .collect() } -/// Pushes `p` onto `out` unless it is the last point there again. +/// Pushes `p` onto `out` unless it repeats the last point (within `EPSILON`). fn push_new(out: &mut Vec, p: Point) { if out.last().is_none_or(|&q| manhattan(q, p) > EPSILON) { out.push(p); @@ -112,40 +114,37 @@ fn push_new(out: &mut Vec, p: Point) { /// How far wires keep from obstacles, in spacings. const CLEARANCE: f64 = 2.0; -/// Offsets below this are rounding: a point this near a box's side is on it. +/// Offsets below this are rounding error: a point this near a box's side is on it. const EPSILON: f64 = 1e-6; -/// Points nearer than this are one point. +/// Points closer than this are the same point. const SAME: f64 = 1e-9; -/// Children per index node. -const NODE: usize = 16; - -/// Attach points a branch tries, nearest first. +/// How many attach points a branch tries, nearest first. const ATTACHES: usize = 8; -/// Attach points a branch searches from when no elbow is clear. +/// How many attach points a search starts from when no elbow is clear. const SEARCHED: usize = 3; -/// Crossings past which a search gives up. +/// The number of visibility-graph crossings beyond which a search gives up. const MAX_CROSSINGS: usize = 1 << 20; -/// Steps a search takes before it gives up. +/// The number of states a search expands before it gives up. const MAX_VISITS: usize = 1 << 17; -/// Crossings a flood from a search's goal reaches before it takes the goal for open. +/// How many crossings a flood from the goal must reach before the goal counts as open. const POCKET: usize = 256; -/// Times a search widens its window over the obstacles that wall it in. +/// How many times a search widens its window to cover the obstacles walling it in. const WIDENINGS: usize = 1; -/// What crossing a group's border costs, in clearances of length. +/// The cost of crossing a group's border, in clearances of length. const CROSSING: f64 = 4.0; -/// No joint: the root's parent, or a joint without children. +/// No joint: the root's parent, or the end of a child list. const NONE: usize = usize::MAX; -/// No state: where a search starts. +/// No state: marks a search's starting states. const NOWHERE: u32 = u32::MAX; impl Side { @@ -160,14 +159,15 @@ impl Side { } } -/// A terminal as routing reads it: where its wire meets the vertex, the way out, and its owner's box. +/// A terminal resolved for routing: where its wire meets the vertex, its outward direction, and +/// its owner's box. struct End { at: Point, normal: Point, owner: Option, } -/// Plans nets among the obstacles, then sets their runs apart. +/// Routes nets among the obstacles, then spreads their shared runs onto tracks. struct Router<'a> { obstacles: Obstacles<'a>, groups: Groups, @@ -179,22 +179,22 @@ struct Router<'a> { } impl<'a> Router<'a> { - /// A router among `boxes` and `groups`, tracks `spacing` apart; a negative or non-finite spacing - /// is zero. - fn new(boxes: &'a [Rect], groups: &[Rect], spacing: f64) -> Router<'a> { + /// A router among `boxes` and `groups`, with tracks `spacing` apart. A negative or non-finite + /// spacing counts as zero. + fn new(boxes: &'a Boxes, groups: &[Rect], spacing: f64) -> Router<'a> { let spacing = if spacing.is_finite() { spacing.max(0.0) } else { 0.0 }; let clearance = spacing * CLEARANCE; let groups = Groups { boxes: groups.iter().copied().filter(valid).collect(), clearance }; - Router { obstacles: Obstacles::new(boxes), groups, spacing, clearance, search: Search::new(2.0 * clearance) } + Router { obstacles: Obstacles { boxes }, groups, spacing, clearance, search: Search::new(2.0 * clearance) } } - /// The shortest stub, and the least a track keeps a stub's end off its terminal. + /// The shortest stub, and the least distance a track keeps a stub's end from its terminal. fn least(&self) -> f64 { self.spacing / 8.0 } - /// A net's tree: the first terminal's stub, then each further terminal's branch, nearest first; - /// `None` for a net of fewer than two terminals at finite points. + /// A net's tree: the first terminal's stub, then each further terminal's branch, nearest + /// first. `None` for a net with fewer than two terminals at finite points. fn net(&mut self, net: &Net) -> Option { let terminals: Vec = net.terminals.iter().filter(|terminal| terminal.at.iter().all(|v| v.is_finite())).copied().collect(); @@ -226,8 +226,8 @@ impl<'a> Router<'a> { Some(tree) } - /// How routing reads a terminal. Its way out is its side; else its owner's side nearest it; else - /// the axis toward `toward`. + /// Resolves a terminal for routing. Its outward direction is its side's normal, else the normal + /// of its owner's side nearest it, else the axis toward `toward`. fn end(&self, terminal: Terminal, toward: Point) -> End { let owner = self.obstacles.owned(terminal.owner); let at = terminal.at; @@ -242,8 +242,8 @@ impl<'a> Router<'a> { End { at, normal, owner } } - /// Where a terminal's stub ends: clear of its owner by the clearance, or halfway to an obstacle - /// in the way. + /// Where a terminal's stub ends: past its owner by the clearance, or halfway to an obstacle in + /// the way, but at least `least` past the owner. fn stub(&self, end: &End) -> Point { let axis = usize::from(end.normal[0] == 0.0); let sign = end.normal[axis]; @@ -270,9 +270,9 @@ impl<'a> Router<'a> { point } - /// The cheapest clear way from the tree to a reader's stub end `end`: an elbow from one of the - /// nearest attach points; else a search around the obstacles at less and less clearance; else a - /// plain elbow through them. + /// The cheapest clear path from the tree to a reader's stub end `end`. It is an elbow from one + /// of the nearest attach points, else a search around the obstacles at decreasing clearance, + /// else a plain elbow through them. fn branch(&mut self, tree: &Tree, end: Point, reader: &End, root: &End) -> Branch { let mut attaches = tree.attaches(end); attaches.truncate(ATTACHES); @@ -288,13 +288,13 @@ impl<'a> Router<'a> { best.consider(&[from, [middle[0], from[1]], [middle[0], end[1]], end], attach, clear); best.consider(&[from, [from[0], middle[1]], [end[0], middle[1]], end], attach, clear); } - // An elbow that keeps out of every group it need not cross is as good as any search finds. + // An elbow with no group cost is as good as anything a search would find. match best.branch.take() { Some(branch) if branch.points.windows(2).all(|w| self.groups.cost(w[0], w[1]) == 0.0) => return branch, elbow => best.branch = elbow, } - // The window frames the boxes a branch may have to go around: its terminal's, and the root's - // when the branch may leave the root's stub. + // The search window frames the boxes a branch may have to go around: its terminal's owner, + // and the root's owner when the branch may leave from the root's stub. let searched = &attaches[..attaches.len().min(SEARCHED)]; let from_root = searched.iter().any(|attach| attach.point == tree.points[1]); let owners: Vec = [reader.owner, root.owner.filter(|_| from_root)].into_iter().flatten().collect(); @@ -313,10 +313,10 @@ impl<'a> Router<'a> { Branch { points: simplify(&[attach.point, [end[0], attach.point[1]], end]), attach } } - /// Sets apart the runs of different nets that share a line: each takes its own track, `spacing` - /// apart where the corridor between obstacles has room, ordered by where its ends turn so runs - /// nest rather than cross. A run carries its joints, so branches stay joined; terminals never - /// move, and stubs only grow. + /// Spreads runs of different nets that share a line onto separate tracks, `spacing` apart where + /// the corridor between obstacles has room. Tracks are ordered by where the runs' ends turn, so + /// runs nest rather than cross. A run moves its joints, so branches stay connected. Terminals + /// never move, and stubs only grow. fn separate(&self, trees: &mut [Option]) { let tolerance = self.spacing / 16.0; for axis in 0..2 { @@ -360,8 +360,8 @@ impl<'a> Router<'a> { false }); let mut order: Vec<&Run> = cluster.iter().collect(); - // By turn, then length (longest first where ends turn lower), then start: the last key - // first, as each sort keeps the order of ties. + // Order by turn, then length (longest first where ends turn lower), then start. Sort by the + // last key first, since each sort is stable. crate::order::by_key(&mut order, |a| crate::order::number(a.along[0])); crate::order::by_wide_key(&mut order, |a| { let length = crate::order::number(a.along[1] - a.along[0]); @@ -386,26 +386,26 @@ impl<'a> Router<'a> { } } -/// The axis direction an offset from a box's middle leaves it by, measured across the box's `size`. +/// The axis direction by which an offset from a box's center exits it, relative to its `size`. fn outward(offset: Point, size: Point) -> Point { let across = |k: usize| (offset[k] / size[k].max(f64::MIN_POSITIVE)).abs(); let away = |v: f64| if v < 0.0 { -1.0 } else { 1.0 }; if across(0) > across(1) { [away(offset[0]), 0.0] } else { [0.0, away(offset[1])] } } -/// What a joint of a tree is, which says how far tracks may move it. +/// The kind of a tree's joint, which limits how tracks may move it. #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Joint { /// A bend or a junction, free to move. Bend, /// A terminal, which never moves. Terminal, - /// A stub's outer end, whose stub from the terminal joint given only grows. + /// A stub's outer end. Its stub from the given terminal joint may only grow. Stub(usize), } -/// A net while it is planned: joints, each joined to its parent by an axis-aligned segment. The -/// first joint is the root's terminal. +/// A net being routed: joints, each joined to its parent by an axis-aligned segment. The first +/// joint is the root terminal. #[derive(Default)] struct Tree { points: Vec, @@ -414,7 +414,7 @@ struct Tree { } impl Tree { - /// A wire already routed, as a chain of joints none of which moves: its runs hold their tracks. + /// A wire already routed, as a chain of joints that never move, so its runs keep their tracks. fn fixed(line: &[Point]) -> Tree { let mut tree = Tree::default(); let mut parent = NONE; @@ -432,14 +432,14 @@ impl Tree { self.points.len() - 1 } - /// A joint at `point` on the segment from `child` to its parent. + /// Inserts a joint at `point` on the segment from `child` to its parent. fn split(&mut self, child: usize, point: Point) -> usize { let joint = self.add(point, self.parent[child], Joint::Bend); self.parent[child] = joint; joint } - /// Where a branch may attach, nearest `toward` first: each joint but a terminal, and each + /// Where a branch may attach, nearest to `toward` first: each joint except terminals, and each /// segment's nearest point to `toward`. Segments into terminals are stubs, which no branch /// attaches inside. fn attaches(&self, toward: Point) -> Vec { @@ -468,7 +468,8 @@ impl Tree { out } - /// The way from a joint `attach` is at along its stub into a terminal, which no branch leaves by. + /// When `attach` is at a joint with a stub into a terminal, the direction along that stub, + /// which no branch may leave by. fn stub_way(&self, attach: Attach) -> Option { if attach.interior { return None; @@ -481,8 +482,8 @@ impl Tree { .map(|terminal| direction(attach.point, self.points[terminal])) } - /// Moves a path's start along the tree while its first step runs along a segment of the tree, so - /// the path attaches where it leaves the tree. Stubs, which no branch attaches inside, stop it. + /// Moves a path's start along the tree while its first step runs along a tree segment, so the + /// path attaches where it leaves the tree. Stubs stop it, since no branch attaches inside one. fn slide(&self, path: &mut Vec, mut attach: Attach) -> Attach { for _ in 0..self.points.len() { let [start, next, ..] = path[..] else { break }; @@ -557,7 +558,7 @@ struct Attach { interior: bool, } -/// A branch from a tree to a terminal's stub end: its path, from where it attaches. +/// A branch from a tree to a terminal's stub end: its path, starting where it attaches. struct Branch { points: Vec, attach: Attach, @@ -566,9 +567,9 @@ struct Branch { /// The cheapest branch found so far. struct Best<'t> { tree: &'t Tree, - /// The way a branch enters the stub end: against the terminal's side. + /// The direction a branch enters the stub end: opposite the terminal's outward normal. into: Point, - /// What a bend costs, in length. + /// The cost of a bend, in length. bend: f64, groups: &'t Groups, cost: f64, @@ -576,8 +577,8 @@ struct Best<'t> { } impl Best<'_> { - /// Keeps the path through `points` from `attach` when it turns well, costs less than the best so - /// far, and, with `clear`, keeps out of the obstacles grown by the clearance given. + /// Keeps the path through `points` from `attach` if it turns well, costs less than the best so + /// far, and, with `clear`, avoids the obstacles grown by the given clearance. fn consider(&mut self, points: &[Point], attach: Attach, clear: Option<(&Obstacles, f64)>) { let mut path = simplify(points); let attach = self.tree.slide(&mut path, attach); @@ -597,9 +598,9 @@ impl Best<'_> { } } - /// Whether a path from the tree to the stub end turns well: it never doubles back along the - /// stub, leaves a segment it attaches inside across it, and leaves a joint along none of the - /// segments already there. + /// Whether a path from the tree to the stub end turns well: it does not double back along the + /// stub, it leaves a segment it attaches inside at a right angle, and it leaves a joint along + /// none of the joint's segments. fn fits(&self, path: &[Point], attach: Attach) -> bool { let (Some(&[start, next]), Some(&[before, last])) = (path.first_chunk(), path.last_chunk()) else { return path.len() == 1; @@ -607,7 +608,8 @@ impl Best<'_> { direction(before, last) != [-self.into[0], -self.into[1]] && self.leaves_well(start, next, attach) } - /// Whether a path's first step, from `start` toward `next`, leaves the tree well. + /// Whether a path's first step, from `start` toward `next`, leaves the tree well: at a right + /// angle to a segment it attaches inside, or along none of a joint's segments. fn leaves_well(&self, start: Point, next: Point, attach: Attach) -> bool { let tree = self.tree; let parent = tree.parent[attach.child]; @@ -625,8 +627,8 @@ impl Best<'_> { } } -/// What a search joins: the points it may leave from, each with a way it may not leave by, and the -/// point it reaches, with a way it may not arrive moving. +/// A search's ends: the points it may start from, each with a direction it may not leave by, and +/// its goal, with a direction it may not arrive moving in. struct Leg<'p> { from: &'p [(Point, Option)], to: Point, @@ -635,26 +637,27 @@ struct Leg<'p> { /// How a search over one window ended. enum Outcome { - /// The path found, from the leg's start given. + /// The path found, with the index of the leg's start it begins at. Found(usize, Vec), - /// No way inside the window, though a wider one may hold one. + /// No path inside the window, though a wider window may have one. Walled, - /// No way at all, or too much to look through. + /// No path at all, or too much to search. Unreachable, } -/// What a flood from a search's goal found before the search spreads over the window. +/// What a flood from the goal found before the full search. enum Reach { - /// A start, or more than a pocket: the search is worth running. + /// A start, or more than `POCKET` crossings: the search is worth running. Open, - /// A pocket without a start that reaches the window's edge: a wider window may join them. + /// A small region around the goal, without a start, touching the window's edge: a wider + /// window may reach a start. Apart, - /// A pocket without a start, walled in inside the window. + /// A small region around the goal, without a start, walled in inside the window. Enclosed, } -/// A state waiting in a search: its estimated total cost, its cost so far, and the state, a crossing -/// and the axis of the step that reached it. +/// A state queued in a search: its estimated total cost, its cost so far, and the state (a +/// crossing and the axis of the step that reached it). #[derive(Clone, Copy)] struct Step { estimate: f64, @@ -663,7 +666,7 @@ struct Step { } impl Ord for Step { - // The heap pops its greatest: the least estimate, then the farthest along, then the first state. + // The max-heap pops the least estimate first, then the greatest cost, then the lowest state. fn cmp(&self, other: &Step) -> Ordering { other .estimate @@ -687,7 +690,7 @@ impl PartialEq for Step { impl Eq for Step {} -/// A free stretch of a line among a search's obstacles: the line's place across, and where the +/// A free stretch of a line among a search's obstacles: the line's coordinate, and where the /// stretch starts and ends along it. #[derive(Clone, Copy, PartialEq)] struct Span { @@ -702,17 +705,17 @@ struct Crossing { point: Point, /// The horizontal span, then the vertical one. spans: [u32; 2], - /// Its place in the search's columns. + /// Its index in the search's `columns`. place: u32, } -/// Searches for orthogonal paths around obstacles over a sparse visibility graph: the free spans of -/// the lines through the obstacles' corners, the path's two ends, and the window's edges, which meet -/// where they cross. Its scratch is shared by every search, grown and never shrunk. +/// Searches for orthogonal paths around obstacles over a sparse visibility graph: the free spans +/// of the lines through the obstacles' corners, the path's ends, and the window's edges, with +/// vertices where they cross. Every search reuses its scratch buffers, which never shrink. struct Search { - /// What a bend costs, in length. + /// The cost of a bend, in length. bend: f64, - /// How far a window reaches past what it frames. + /// How far a window extends past the points and boxes it frames. margin: f64, /// The graph's vertices: horizontal span by span, each from left to right. crossings: Vec, @@ -720,18 +723,19 @@ struct Search { columns: Vec, /// Where each vertical span's crossings start in `columns`, then where the last ends. starts: Vec, - /// The cheapest cost found to each state: a crossing, and the axis of the step that reached it. + /// The cheapest cost found to each state (a crossing and the axis of the step that reached it). costs: Vec, /// The state each state was reached from. came: Vec, heap: BinaryHeap, - /// Which crossings a flood has reached, and the order it reached them in. + /// Which crossings a flood has reached. seen: Vec, + /// The crossings a flood reached, in the order it reached them. queue: Vec, } impl Search { - /// A search whose bends cost, and whose windows reach, `reach`. + /// A search where a bend costs `reach`, and windows extend `reach` past what they frame. fn new(reach: f64) -> Search { Search { bend: reach, @@ -747,10 +751,10 @@ impl Search { } } - /// The cheapest orthogonal path for `leg` around the obstacles grown by `by`, from whichever start - /// is cheapest, in a window framing the leg and the boxes `around`. The window widens over - /// obstacles that wall it in. `None` when the leg's end or every start is inside an obstacle, or no - /// way is found. + /// The cheapest orthogonal path for `leg` around the obstacles grown by `by`, from the cheapest + /// start, within a window framing the leg and the boxes `around`. The window widens to cover + /// obstacles that wall it in. `None` when the goal or every start is inside an obstacle, or no + /// path is found. fn path( &mut self, obstacles: &Obstacles, @@ -781,8 +785,8 @@ impl Search { None } - /// Builds the window's visibility graph through `ends` among `boxes`, the grown obstacles that - /// meet it; false when it holds too many crossings to search. + /// Builds the window's visibility graph through `ends` among `boxes` (the grown obstacles that + /// meet the window). Returns false when it has too many crossings to search. fn build(&mut self, boxes: &[Rect], window: Rect, ends: &[Point]) -> bool { let horizontal = spans(boxes, window, ends, 0); let vertical = spans(boxes, window, ends, 1); @@ -801,7 +805,7 @@ impl Search { return false; } } - // Each vertical span's crossings, which come bottom to top as the rows do. + // Index each vertical span's crossings. They come bottom to top, as the rows do. self.starts.clear(); self.starts.resize(vertical.len() + 1, 0); for crossing in &self.crossings { @@ -823,10 +827,10 @@ impl Search { true } - /// A* over the window's visibility graph for `leg` among `boxes`, the grown obstacles that meet - /// it, its steps costing their crossings of `groups`' borders: each state a crossing and the axis - /// of the step that reached it. The groups' lanes, just inside and outside their borders, are - /// lines of the graph too. + /// Runs A* over the window's visibility graph for `leg` among `boxes` (the grown obstacles that + /// meet the window). Each state is a crossing plus the axis of the step that reached it, and + /// each step also pays its `groups` border cost. The groups' lanes, just inside and outside + /// their borders, are lines of the graph too. fn graph(&mut self, boxes: &[Rect], groups: &Groups, window: Rect, leg: &Leg) -> Outcome { let groups = groups.within(window); let mut ends: Vec = leg.from.iter().map(|&(start, _)| start).chain([leg.to]).collect(); @@ -878,7 +882,7 @@ impl Search { } let crossing = self.crossings[vertex]; edge |= on_edge(crossing.point, &window); - // A start's first step leaves by any way but its banned one. + // A start's first step may leave in any direction but its banned one. let first = self.came[state] == NOWHERE; let banned = if first { sources.iter().find(|source| source.0 == vertex).and_then(|source| source.1) } else { None }; @@ -902,12 +906,13 @@ impl Search { self.heap.push(Step { estimate, cost: value, state: reached as u32 }); } } - // A start walled in short of the window's edge stays walled in however wide the window. + // If the search never reached the window's edge, a wider window would not help. if edge { Outcome::Walled } else { Outcome::Unreachable } } - /// The least a path can cost from `point`, reached along `axis` (`None` at a start, whose first - /// step turns for free), to `to`: its length, and a bend for the turn it still needs. + /// A lower bound on the cost from `point`, reached along `axis`, to `to`: the Manhattan + /// distance, plus a bend if a turn is still needed. `axis` is `None` at a start, whose first + /// step turns for free. fn estimate(&self, point: Point, axis: Option, to: Point) -> f64 { let turns = match ([point[0] != to[0], point[1] != to[1]], axis) { ([true, true], _) | ([true, false], Some(1)) | ([false, true], Some(0)) => 1.0, @@ -932,8 +937,8 @@ impl Search { ] } - /// Floods the graph from `goal` for a pocket's worth of crossings, to fail fast where the goal is - /// walled in apart from every start. + /// Floods the graph from `goal` for up to `POCKET` crossings, to fail fast when the goal is + /// walled off from every start. fn flood(&mut self, goal: usize, sources: &[(usize, Option)], window: Rect) -> Reach { self.seen.clear(); self.seen.resize(self.crossings.len(), false); @@ -957,7 +962,7 @@ impl Search { if edge { Reach::Apart } else { Reach::Enclosed } } - /// The path that reached state `goal`, from the start. + /// The simplified path to state `goal`, from its start. fn trace(&self, goal: usize) -> Vec { let mut path = Vec::new(); let mut state = goal; @@ -973,15 +978,16 @@ impl Search { } } -/// The free spans of lines running along `along` (0: x, 1: y) within `window`, among `boxes`, the -/// grown obstacles that meet it: through each box's corners, or where its sides' lines enter the -/// window, through each of `ends`, and every span along the window's edges. Sorted by line, then -/// start. +/// The free spans of lines running along axis `along` (0: x, 1: y) within `window`, among `boxes` +/// (the grown obstacles that meet it). Lines pass through each box's corners (clamped into the +/// window) and through each of `ends`, taking the free span at that point. Lines along the +/// window's edges take every free span. Sorted by line, then start. fn spans(boxes: &[Rect], window: Rect, ends: &[Point], along: usize) -> Vec { let held = 1 - along; let (start, end) = (window[along], window[along + 2]); let (low, high) = (window[held], window[held + 2]); - // A seed is a line and a point along it its span holds; NaN takes every span of the line. + // A seed is a line and a point along it, which selects the free span holding that point. NaN + // selects every free span of the line. let mut seeds: Vec = boxes .iter() .flat_map(|r| { @@ -995,7 +1001,7 @@ fn spans(boxes: &[Rect], window: Rect, ends: &[Point], along: usize) -> Vec = (0..boxes.len()).collect(); crate::order::by_key(&mut order, |&a| crate::order::number(boxes[a][held])); let mut order = order.into_iter().peekable(); @@ -1009,7 +1015,7 @@ fn spans(boxes: &[Rect], window: Rect, ends: &[Point], along: usize) -> Vec Vec, - /// How far wires keep from obstacles: a border's lanes lie half of it inside and outside. + /// How far wires keep from obstacles. A border's lanes lie half this inside and outside it. clearance: f64, } impl Groups { - /// What a run from `a` to `b` costs past its length: [`CROSSING`] clearances for each border it - /// crosses, and its length again where it runs along a border, nearer than half a clearance. + /// The cost of a run from `a` to `b` beyond its length: [`CROSSING`] clearances per border it + /// crosses, plus the length it runs along a border closer than half a clearance. fn cost(&self, a: Point, b: Point) -> f64 { let along = usize::from(a[0] == b[0]); let held = 1 - along; @@ -1085,8 +1091,8 @@ impl Groups { Groups { boxes: self.boxes.iter().copied().filter(|r| meets(r, &window)).collect(), clearance: self.clearance } } - /// The corners of the lanes along every group's borders, half a clearance inside and outside each: - /// points the search's lines run through. + /// The corners of the lanes along every group's borders, half a clearance inside and outside, + /// which the search's lines pass through. fn lanes(&self) -> Vec { let half = self.clearance / 2.0; let mut out = Vec::new(); @@ -1102,27 +1108,21 @@ impl Groups { } } -/// The obstacles routing avoids, with an index over them. +/// The obstacles routing avoids, read from an index of boxes. struct Obstacles<'a> { - boxes: &'a [Rect], - index: Index, + boxes: &'a Boxes, } -impl<'a> Obstacles<'a> { - /// The obstacles `boxes`, indexed. - fn new(boxes: &'a [Rect]) -> Obstacles<'a> { - Obstacles { boxes, index: Index::new(boxes) } - } - +impl Obstacles<'_> { /// The box of obstacle `owner`, when there is one. fn owned(&self, owner: Option) -> Option { - owner.and_then(|owner| self.boxes.get(owner as usize)).copied().filter(valid) + owner.and_then(|owner| self.boxes.get(owner as usize)) } - /// Visits the obstacles meeting `region`, each grown by `by`, until `visit` returns true; whether - /// it did. + /// Visits the obstacles meeting `region`, each grown by `by`, until `visit` returns true. + /// Returns whether it did. fn some(&self, region: Rect, by: f64, mut visit: impl FnMut(Rect) -> bool) -> bool { - self.index.some(region, &mut |item| visit(grow(self.boxes[item], by))) + self.boxes.some(region, &mut |item| visit(grow(self.boxes.get(item).expect("a box visited"), by))) } /// The obstacles meeting `region`, each grown by `by`. @@ -1135,91 +1135,13 @@ impl<'a> Obstacles<'a> { out } - /// Whether the axis-aligned segment from `a` to `b` keeps out of every obstacle grown by `by`. + /// Whether the axis-aligned segment from `a` to `b` avoids every obstacle grown by `by`. fn clear(&self, a: Point, b: Point, by: f64) -> bool { let span = grow([a[0].min(b[0]), a[1].min(b[1]), a[0].max(b[0]), a[1].max(b[1])], by); !self.some(span, by, |obstacle| crosses(a, b, &obstacle)) } } -/// A packed R-tree over the valid boxes: leaves in Z-order of their middles, [`NODE`] children a -/// node. -struct Index { - /// The box at each leaf. - items: Vec, - /// Each node's box: the leaves, then each level up to the root. - nodes: Vec, - /// Where each level's nodes start, leaves first; the last is the node count. - starts: Vec, -} - -impl Index { - /// An index over `boxes`, leaving out those that are not [`valid`]. - fn new(boxes: &[Rect]) -> Index { - let mut items: Vec = (0..boxes.len()).filter(|&item| valid(&boxes[item])).collect(); - let extent = items.iter().fold(EMPTY, |extent, &item| union(extent, boxes[item])); - let cell = - |v: f64, lo: f64, hi: f64| if hi > lo { ((v - lo) / (hi - lo) * f64::from(u16::MAX)) as u32 } else { 0 }; - crate::order::by_key(&mut items, |&item| { - let middle = [boxes[item][0].midpoint(boxes[item][2]), boxes[item][1].midpoint(boxes[item][3])]; - let at = morton(cell(middle[0], extent[0], extent[2]), cell(middle[1], extent[1], extent[3])); - crate::order::pair(at, item as u32) - }); - let mut nodes: Vec = items.iter().map(|&item| boxes[item]).collect(); - let mut starts = vec![0]; - let mut level = 0; - loop { - let next = nodes.len(); - starts.push(next); - for first in (level..next).step_by(NODE) { - let node = nodes[first..(first + NODE).min(next)].iter().fold(EMPTY, |node, child| union(node, *child)); - nodes.push(node); - } - if nodes.len() - next <= 1 { - break; - } - level = next; - } - starts.push(nodes.len()); - Index { items, nodes, starts } - } - - /// Visits the boxes meeting `region` until `visit` returns true; whether it did. - fn some(&self, region: Rect, visit: &mut impl FnMut(usize) -> bool) -> bool { - let top = self.starts.len() - 2; - !self.items.is_empty() - && meets(&self.nodes[self.starts[top]], ®ion) - && self.descend(top, self.starts[top], region, visit) - } - - /// [`Index::some`] below `node` of `level`. - fn descend(&self, level: usize, node: usize, region: Rect, visit: &mut impl FnMut(usize) -> bool) -> bool { - let first = self.starts[level - 1] + (node - self.starts[level]) * NODE; - let last = (first + NODE).min(self.starts[level]); - (first..last).any(|child| { - meets(&self.nodes[child], ®ion) - && if level > 1 { self.descend(level - 1, child, region, visit) } else { visit(self.items[child]) } - }) - } -} - -/// Position along a Z-order curve on a 65536² grid. -fn morton(x: u32, y: u32) -> u32 { - let spread = |v: u32| { - let v = v & 0xffff; - let v = (v | (v << 8)) & 0x00ff_00ff; - let v = (v | (v << 4)) & 0x0f0f_0f0f; - let v = (v | (v << 2)) & 0x3333_3333; - (v | (v << 1)) & 0x5555_5555 - }; - spread(x) | (spread(y) << 1) -} - -/// Whether a box is finite and not inverted. -fn valid(r: &Rect) -> bool { - r.iter().all(|v| v.is_finite()) && r[0] <= r[2] && r[1] <= r[3] -} - /// Whether box `a` lies within box `b`. fn within(a: &Rect, b: &Rect) -> bool { a[0] >= b[0] && a[1] >= b[1] && a[2] <= b[2] && a[3] <= b[3] @@ -1230,14 +1152,14 @@ fn on_edge(p: Point, r: &Rect) -> bool { p[0] == r[0] || p[0] == r[2] || p[1] == r[1] || p[1] == r[3] } -/// Whether a point is strictly inside a box. +/// Whether a point is strictly inside a box, by more than `EPSILON`. fn inside(p: Point, r: &Rect) -> bool { p[0] > r[0] + EPSILON && p[0] < r[2] - EPSILON && p[1] > r[1] + EPSILON && p[1] < r[3] - EPSILON } /// Whether the axis-aligned segment from `a` to `b` passes through a box's interior. fn crosses(a: Point, b: Point, r: &Rect) -> bool { - // A vertical segment holds its x and runs along y; a horizontal one holds its y. + // A vertical segment has a fixed x and runs along y. A horizontal one has a fixed y. let held = usize::from(a[0] != b[0]); let runs = 1 - held; a[held] > r[held] + EPSILON @@ -1251,7 +1173,7 @@ fn between(v: f64, a: f64, b: f64) -> bool { a.min(b) <= v && v <= a.max(b) } -/// The length of an orthogonal path between two points. +/// The Manhattan distance between two points: the length of an orthogonal path between them. fn manhattan(a: Point, b: Point) -> f64 { (a[0] - b[0]).abs() + (a[1] - b[1]).abs() } @@ -1267,12 +1189,12 @@ fn sign(v: f64) -> f64 { } } -/// The axis signs of the way from one point to another. +/// The sign of each axis of the direction from `from` to `to`. fn direction(from: Point, to: Point) -> Point { [sign(to[0] - from[0]), sign(to[1] - from[1])] } -/// A path's points without repeats or in-line middles. +/// A path's points with repeats and collinear middle points removed. pub(crate) fn simplify(path: &[Point]) -> Vec { let mut out: Vec = Vec::with_capacity(path.len()); for &p in path { @@ -1292,27 +1214,27 @@ pub(crate) fn simplify(path: &[Point]) -> Vec { /// A net's chain of segments along one line, which a track moves as one. struct Run { net: usize, - /// The joints it carries. + /// The joints a track moves with it. joints: Vec, /// The line's coordinate on the axis tracks move it along. line: f64, /// Where it starts and ends along the line. along: [f64; 2], - /// Where its ends turn: below zero toward lower lines, above toward higher. + /// Which way its ends turn: negative toward lower lines, positive toward higher. turn: f64, - /// The lines tracks may move it between; one line for a run holding a terminal. + /// The lines tracks may move it between, or a single line for a run holding a terminal. limits: [f64; 2], } impl Run { - /// Whether it holds a terminal, so no track moves it. + /// Whether its limits allow one line only (it holds a terminal), so no track moves it. fn held(&self) -> bool { self.limits[0] == self.limits[1] } } -/// Every net's runs on lines across `axis`: maximal chains of segments along one line through -/// joints. A stub's end keeps `least` off its terminal. +/// Every net's runs whose line has a fixed `axis` coordinate: maximal chains of segments along one +/// line. A stub's end keeps at least `least` from its terminal. fn runs(trees: &[Option], axis: usize, least: f64) -> Vec { let mut runs = Vec::new(); for (net, tree) in trees.iter().enumerate() { @@ -1413,7 +1335,7 @@ mod tests { segments(wire).iter().all(|[a, b]| boxes.iter().all(|r| !crosses(*a, *b, r))) } - /// The point next to the terminal at `at` on its wire: the way the wire leaves it. + /// The point after the terminal at `at` on its wire, which shows the way the wire leaves it. fn leaving(wire: &[Vec], at: Point) -> Point { wire.iter() .find_map(|line| match line[..] { @@ -1424,8 +1346,8 @@ mod tests { .expect("the wire ends at the terminal") } - /// Whether the polylines make one tree reaching every terminal: each starts at the first terminal - /// or where another ends. + /// Whether the polylines make one tree reaching every terminal: each starts at the first + /// terminal or where another ends. fn one_tree(wire: &[Vec], terminals: &[Terminal]) -> bool { let mut reached = vec![terminals[0].at]; let mut joined = vec![false; wire.len()]; @@ -1440,14 +1362,17 @@ mod tests { fn side_by_side_boxes_join_by_a_straight_run() { let boxes = [[0.0, 0.0, 20.0, 20.0], [100.0, 0.0, 120.0, 20.0]]; let nets = [Net { terminals: vec![on([20.0, 10.0], Side::Right, 0), on([100.0, 10.0], Side::Left, 1)] }]; - assert_eq!(route(&boxes, &[], &nets, &[], SPACING), vec![vec![vec![[20.0, 10.0], [100.0, 10.0]]]]); + assert_eq!( + route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING), + vec![vec![vec![[20.0, 10.0], [100.0, 10.0]]]] + ); } #[test] fn a_wire_bends_around_an_obstacle_between() { let boxes = [[0.0, 0.0, 20.0, 20.0], [200.0, 0.0, 220.0, 20.0], [80.0, -30.0, 140.0, 50.0]]; let nets = [Net { terminals: vec![on([20.0, 10.0], Side::Right, 0), on([200.0, 10.0], Side::Left, 1)] }]; - let wire = &route(&boxes, &[], &nets, &[], SPACING)[0]; + let wire = &route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING)[0]; assert!(orthogonal(wire), "{wire:?}"); assert!(segments(wire).len() > 2, "it bends: {wire:?}"); assert!(outside(wire, &boxes), "{wire:?}"); @@ -1456,31 +1381,32 @@ mod tests { #[test] fn wires_go_around_groups_they_do_not_join() { - // A thin group on the straight way between two blocks: through it is shorter, around keeps out. + // A thin group lies on the straight path between two blocks: going through is shorter, + // going around stays outside it. let boxes = [[20.0, 45.0, 40.0, 55.0], [160.0, 45.0, 180.0, 55.0]]; let group = [90.0, 44.0, 110.0, 56.0]; let nets = [Net { terminals: vec![on([40.0, 50.0], Side::Right, 0), on([160.0, 50.0], Side::Left, 1)] }]; - let through = &route(&boxes, &[], &nets, &[], SPACING)[0]; + let through = &route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING)[0]; assert_eq!(segments(through).len(), 1, "without the group, straight: {through:?}"); - let around = &route(&boxes, &[group], &nets, &[], SPACING)[0]; + let around = &route(&Boxes::new(boxes.to_vec()), &[group], &nets, &[], SPACING)[0]; assert!(outside(around, &[group]), "with it, around: {around:?}"); assert!(orthogonal(around) && one_tree(around, &nets[0].terminals), "{around:?}"); } #[test] - fn bends_meeting_on_a_short_run_repeat_no_point() { + fn bends_meeting_on_a_short_run_do_not_repeat_points() { // An S-bend whose middle run is shorter than two radii: the arcs of its bends meet halfway. let wire = finish(&[vec![[0.0, 0.0], [20.0, 0.0], [20.0, 10.0], [40.0, 10.0]]], 8.0); let line = &wire[0]; - assert!(line.windows(2).all(|w| manhattan(w[0], w[1]) > EPSILON), "no point twice: {line:?}"); + assert!(line.windows(2).all(|w| manhattan(w[0], w[1]) > EPSILON), "no repeated point: {line:?}"); assert_eq!((line[0], line[line.len() - 1]), ([0.0, 0.0], [40.0, 10.0])); - // A part ending where another starts still doubles its end: a junction. + // A part ending where another starts still repeats its last point: a junction. let tree = finish(&[vec![[0.0, 0.0], [10.0, 0.0]], vec![[10.0, 0.0], [10.0, 10.0]]], 8.0); assert_eq!(tree[0], [[0.0, 0.0], [10.0, 0.0], [10.0, 0.0]]); } #[test] - fn crossing_and_running_along_a_border_cost() { + fn crossing_or_running_along_a_group_border_adds_cost() { let groups = Groups { boxes: vec![[0.0, 0.0, 100.0, 90.0]], clearance: 2.0 * SPACING }; assert_eq!(groups.cost([-10.0, 10.0], [10.0, 10.0]), CROSSING * 2.0 * SPACING, "a crossing"); assert_eq!(groups.cost([10.0, 10.0], [20.0, 10.0]), 0.0, "inside, away from the border"); @@ -1488,7 +1414,7 @@ mod tests { } #[test] - fn a_wall_too_long_for_the_window_is_gone_around() { + fn a_wall_longer_than_the_search_window_is_routed_around() { let boxes = [ [0.0, 0.0, 20.0, 20.0], [200.0, 0.0, 220.0, 20.0], @@ -1496,7 +1422,7 @@ mod tests { [100.0, -400.0, 120.0, 90.0], ]; let nets = [Net { terminals: vec![on([20.0, 10.0], Side::Right, 0), on([200.0, 10.0], Side::Left, 1)] }]; - let wire = &route(&boxes, &[], &nets, &[], SPACING)[0]; + let wire = &route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING)[0]; assert!(orthogonal(wire) && outside(wire, &boxes), "{wire:?}"); } @@ -1512,7 +1438,7 @@ mod tests { Net { terminals: vec![on([10.0, 20.0], Side::Top, 0), on([100.0, 110.0], Side::Left, 1)] }, Net { terminals: vec![on([10.0, 200.0], Side::Bottom, 2), on([120.0, -90.0], Side::Right, 3)] }, ]; - let routes = route(&boxes, &[], &nets, &[], SPACING); + let routes = route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING); for (net, wire) in nets.iter().zip(&routes) { assert!(orthogonal(wire) && outside(wire, &boxes), "{wire:?}"); for terminal in &net.terminals { @@ -1535,7 +1461,7 @@ mod tests { on([100.0, 70.0], Side::Left, 2), ], }; - let wire = &route(&boxes, &[], std::slice::from_ref(&net), &[], SPACING)[0]; + let wire = &route(&Boxes::new(boxes.to_vec()), &[], std::slice::from_ref(&net), &[], SPACING)[0]; assert!(wire.len() >= 3, "a junction splits it: {wire:?}"); assert!(orthogonal(wire) && outside(wire, &boxes), "{wire:?}"); assert!(one_tree(wire, &net.terminals), "{wire:?}"); @@ -1549,7 +1475,7 @@ mod tests { Net { terminals: vec![on([20.0, 10.0], Side::Right, 0), on([100.0, 70.0], Side::Left, 2)] }, Net { terminals: vec![on([20.0, 40.0], Side::Right, 1), on([100.0, 100.0], Side::Left, 3)] }, ]; - let routes = route(&boxes, &[], &nets, &[], SPACING); + let routes = route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING); // A segment as the axis it runs along, its line, and its extent along it. let run = |[a, b]: [Point; 2]| { let axis = usize::from(a[0] == b[0]); @@ -1581,12 +1507,12 @@ mod tests { let mut boxes = vec![[0.0, 0.0, 20.0, 20.0], [100.0, 0.0, 120.0, 20.0]]; boxes.extend(ring); let nets = [Net { terminals: vec![on([20.0, 10.0], Side::Right, 0), on([100.0, 10.0], Side::Left, 1)] }]; - let wire = &route(&boxes, &[], &nets, &[], SPACING)[0]; + let wire = &route(&Boxes::new(boxes), &[], &nets, &[], SPACING)[0]; assert!(orthogonal(wire) && one_tree(wire, &nets[0].terminals), "{wire:?}"); } #[test] - fn a_crowded_grid_routes_alike_every_time() { + fn a_crowded_grid_routes_the_same_every_time() { let mut boxes = Vec::new(); for row in 0..30 { for column in 0..30 { @@ -1621,8 +1547,9 @@ mod tests { Net { terminals } }) .collect(); - let routes = route(&boxes, &[], &nets, &[], SPACING); - assert_eq!(routes, route(&boxes, &[], &nets, &[], SPACING), "the same input routes the same"); + let obstacles = Boxes::new(boxes.clone()); + let routes = route(&obstacles, &[], &nets, &[], SPACING); + assert_eq!(routes, route(&obstacles, &[], &nets, &[], SPACING), "the same input routes the same"); for (net, wire) in nets.iter().zip(&routes) { assert!(orthogonal(wire), "{wire:?}"); assert!(one_tree(wire, &net.terminals), "{net:?}: {wire:?}"); @@ -1635,14 +1562,14 @@ mod tests { } #[test] - fn nets_without_two_terminals_get_nothing() { + fn nets_without_two_valid_terminals_get_no_wire() { let free = Terminal { at: [0.0, 0.0], side: None, owner: None }; let nets = [ Net::default(), Net { terminals: vec![free] }, Net { terminals: vec![free, Terminal { at: [f64::NAN, 0.0], ..free }] }, ]; - assert_eq!(route(&[], &[], &nets, &[], SPACING), vec![Vec::>::new(); 3]); + assert_eq!(route(&Boxes::default(), &[], &nets, &[], SPACING), vec![Vec::>::new(); 3]); } #[test] @@ -1654,7 +1581,7 @@ mod tests { Terminal { at: [100.0, 50.0], side: None, owner: None }, ], }]; - let wire = &route(&boxes, &[], &nets, &[], SPACING)[0]; + let wire = &route(&Boxes::new(boxes.to_vec()), &[], &nets, &[], SPACING)[0]; assert!(orthogonal(wire) && outside(wire, &boxes), "{wire:?}"); assert_eq!(leaving(wire, [20.0, 10.0])[1], 10.0, "it leaves by the side it is on: {wire:?}"); } diff --git a/crates/latkit/src/layout/stress.rs b/crates/latkit/src/layout/stress.rs index 81fed7b0..2cca902a 100644 --- a/crates/latkit/src/layout/stress.rs +++ b/crates/latkit/src/layout/stress.rs @@ -1,9 +1,10 @@ -//! Stress majorization: vertices at graph distances, then their boxes parted. Each vertex's terms are -//! its neighbors a gap away and a few pivots at their graph distance, so a round costs O(vertices × -//! pivots). Pinned vertices hold the rest; vertices start near the pinned ones they link to, or -//! around their part's most linked vertex, and parts nothing pins pack apart. Groups nest as -//! [`nested`] lays them out: a group's members settle by stress among themselves, and the group by -//! stress among the vertices and groups beside it. +//! Stress majorization: places vertices at their graph distances, then separates their boxes. +//! +//! Each vertex's terms are its neighbors, a gap away, and a few pivots at their graph distance, so +//! a round costs O(vertices × pivots). Pinned vertices stay put and anchor the rest. Vertices start +//! near the pinned vertices they connect to, or around their part's most linked vertex, and parts +//! with no pinned vertex pack apart. Groups nest as [`nested`] lays them out: a group's members +//! settle by stress among themselves, and the group among the vertices and groups beside it. use std::collections::VecDeque; @@ -11,21 +12,22 @@ use super::levels::{Graph, Level, nested}; use super::overlap::separate; use super::pack::pack; -/// Jacobi rounds stress runs. +/// Jacobi iterations per level. const ROUNDS: usize = 240; -/// Pivots stress measures graph distances from, and the distances it keeps in all. +/// The most pivots stress measures graph distances from. const PIVOTS: usize = 32; +/// The most graph distances it stores in all (pivots × vertices). const DISTANCES: usize = 8_000_000; -/// The share of the gap vertices' boxes keep clear of each other. +/// The fraction of the gap that vertices' boxes keep clear of each other. const CLEARANCE: f64 = 0.25; -/// Every vertex's center: neighbors `gap` apart, pinned vertices where they are, boxes a quarter of -/// the gap clear of each other; links' directions do not matter. +/// Every vertex's center from a stress layout: neighbors `gap` apart, pinned vertices fixed, and +/// boxes a quarter gap clear of each other. Link direction is ignored. pub(crate) fn stress(graph: &Graph<'_>, gap: f64) -> Vec<[f64; 2]> { nested(graph, |level| place(level, gap)) } -/// One level's centers. +/// One level's centers: `ROUNDS` Jacobi iterations, then overlap removal. fn place(level: &Level, gap: f64) -> Vec<[f64; 2]> { let n = level.sizes.len(); if n == 0 { @@ -47,7 +49,7 @@ fn place(level: &Level, gap: f64) -> Vec<[f64; 2]> { let there = position[other as usize]; let apart = [here[0] - there[0], here[1] - there[1]]; let length = apart[0].hypot(apart[1]); - // Vertices at one point part along a direction of their own. + // Coincident vertices separate along a direction derived from their index. let toward = if length > 1e-9 { [apart[0] / length, apart[1] / length] } else { @@ -70,11 +72,14 @@ fn place(level: &Level, gap: f64) -> Vec<[f64; 2]> { separate(&boxes, &pinned, gap * CLEARANCE) } -/// Each vertex's terms: its neighbors one `gap` away, weighted by the inverse square, and pivots at -/// their graph distance, weighted by the vertices nearer the pivot than half that distance. +/// Each vertex's terms as (other vertex, target distance, weight). Its neighbors are one `gap` +/// away, weighted by the inverse square of that distance. Pivots are at their graph distance times +/// `gap`, weighted by the inverse square times how many vertices of the pivot's region lie within +/// half that distance of it. fn terms(n: usize, gap: f64, offsets: &[u32], targets: &[u32]) -> Vec> { let degree = |v: usize| offsets[v + 1] - offsets[v]; - // Pivots far from each other: each the vertex farthest from those before, other parts first. + // Pivots far apart: each is the vertex farthest from the earlier ones, so vertices they cannot + // reach (other parts) come first. let count = PIVOTS.min(n).min((DISTANCES / n.max(1)).max(2)); let mut pivots: Vec<(usize, Vec)> = Vec::with_capacity(count); let mut nearest = vec![u32::MAX; n]; @@ -121,9 +126,10 @@ fn terms(n: usize, gap: f64, offsets: &[u32], targets: &[u32]) -> Vec Vec<[f64; 2]> { let n = level.sizes.len(); let mut start = vec![[0.0; 2]; n]; @@ -161,7 +167,7 @@ fn start(level: &Level, gap: f64, offsets: &[u32], targets: &[u32]) -> Vec<[f64; }; for (p, inside) in parts.iter().enumerate() { let pins: Vec<[f64; 2]> = inside.iter().filter_map(|&v| level.pinned[v]).collect(); - // Hops from the pinned vertices, or from the part's first, most linked one. + // Hops from the pinned vertices, or else from the part's first vertex, its most linked. let sources: Vec = if pins.is_empty() { vec![inside[0]] } else { @@ -204,7 +210,7 @@ fn start(level: &Level, gap: f64, offsets: &[u32], targets: &[u32]) -> Vec<[f64; if loose.is_empty() { return start; } - // Parts nothing pins pack in rows below whatever is pinned. + // Parts with no pinned vertex pack in rows below the pinned vertices. let pinned: Vec<[f64; 2]> = level.pinned.iter().flatten().copied().collect(); let below = pinned.iter().map(|p| p[1]).fold(f64::INFINITY, f64::min); let left = pinned.iter().map(|p| p[0]).fold(f64::INFINITY, f64::min); @@ -219,7 +225,8 @@ fn start(level: &Level, gap: f64, offsets: &[u32], targets: &[u32]) -> Vec<[f64; start } -/// Each vertex's neighbors, once each. +/// Each vertex's neighbors, each once, as offsets into one list of targets. Ignores direction, +/// self-links, and links out of range. fn neighbors(n: usize, links: &[[u32; 2]]) -> (Vec, Vec) { let mut pairs: Vec<(u32, u32)> = links .iter() @@ -238,7 +245,7 @@ fn neighbors(n: usize, links: &[[u32; 2]]) -> (Vec, Vec) { (offsets, pairs.into_iter().map(|(_, b)| b).collect()) } -/// Hops from `from` to every vertex; `u32::MAX` where none lead. +/// Hop counts from `from` to every vertex, or `u32::MAX` where unreachable. fn hops(from: usize, offsets: &[u32], targets: &[u32]) -> Vec { let mut distance = vec![u32::MAX; offsets.len() - 1]; let mut queue = VecDeque::from([from]); @@ -262,7 +269,7 @@ mod tests { (a[0] - b[0]).hypot(a[1] - b[1]) } - /// `sizes` laid out by stress with `links`, `pinned`, and `groups`, groups padding their members by 5. + /// `sizes` laid out by stress with `links`, `pinned`, and `groups` (groups pad members by 5). fn laid( sizes: &[[f64; 2]], links: &[[u32; 2]], @@ -285,7 +292,7 @@ mod tests { } #[test] - fn pins_hold_and_boxes_part() { + fn pins_stay_put_and_boxes_do_not_overlap() { let pinned = [Some([-100.0, 60.0]), None, None, None]; let p = laid(&[[60.0, 20.0]; 4], &[[0, 1], [1, 2], [1, 3]], &pinned, &[None; 4], 10.0); assert_eq!(p[0], [-100.0, 60.0]); @@ -298,7 +305,7 @@ mod tests { } #[test] - fn groups_hold_their_members_apart_from_each_other() { + fn group_boxes_do_not_overlap() { // Groups 0 and 1 hold three linked vertices each, and link through their members. let sizes = [[0.0; 2], [0.0; 2], [20.0, 10.0], [20.0, 10.0], [20.0, 10.0], [20.0, 10.0], [20.0, 10.0], [20.0, 10.0]]; @@ -312,6 +319,6 @@ mod tests { }; let (a, b) = (frame(&[2, 3, 4]), frame(&[5, 6, 7])); let overlap = a[0] < b[2] && b[0] < a[2] && a[1] < b[3] && b[1] < a[3]; - assert!(!overlap, "the groups' boxes stand apart: {a:?} {b:?}"); + assert!(!overlap, "the groups' boxes do not overlap: {a:?} {b:?}"); } } diff --git a/crates/latkit/src/lib.rs b/crates/latkit/src/lib.rs index 3bf70b28..e17f9e78 100644 --- a/crates/latkit/src/lib.rs +++ b/crates/latkit/src/lib.rs @@ -1,15 +1,16 @@ -//! Columnar data drawn as vertices, edges, lines, and areas on the GPU. +//! Columnar data drawn on the GPU as vertices, edges, lines, and areas. //! -//! A [`Data`] is a named collection of [`Table`]s, changed in place by [`Patch`]es; a clone is an -//! O(1) snapshot. A [`Scene`] describes what to draw and how it answers input: marks that name tables -//! of data, in a space, with a theme and controls. A [`View`] draws a scene into any wgpu frame, is -//! given the data by name, and turns platform [`Input`] into [`Event`]s. +//! A [`Data`] is a named collection of [`Table`]s, changed in place by [`Patch`]es. A [`Snapshot`] +//! is one immutable version of a `Data`, taken in O(1). A [`Scene`] describes what to draw and how +//! it responds to input: marks that reference tables of datasets, a space, a theme, and controls. A +//! [`View`] draws a scene into any wgpu frame from its [`Source`]s, one per dataset name, and turns +//! platform [`Input`] into [`Event`]s. //! -//! Every input is plain data that deserializes: [`from_json`] reads one from JSON text, an error -//! naming the path to what it could not read, and [`schema`] gives its JSON Schema. +//! Every input is plain data that deserializes. [`from_json`] reads one from JSON text and reports +//! errors with the path to the part it could not read. [`schema`] returns its JSON Schema. -// Without `gpu`, what only drawing reads of the data, expressions, and sorts has no reader; builds -// with it, which lint every feature, find what nothing reads. +// Without `gpu`, the parts of data, expressions, and sorts that only drawing uses are dead code. +// Builds with `gpu` lint every feature, so they still catch code nothing uses. #![cfg_attr(not(feature = "gpu"), allow(dead_code))] mod data; @@ -49,16 +50,19 @@ mod video; mod view; #[cfg(all(feature = "web", target_arch = "wasm32"))] pub mod web; +mod wire; +#[cfg(feature = "gpu")] +pub use data::Source; pub use data::{ - Data, DecodeLimits, Description, Envelope, FieldDescription, Id, Ids, Kind, Name, Patch, Row, Rows, Table, - TableDescription, Value, Values, + Data, DecodeLimits, Description, Envelope, EnvelopeRequest, FieldDescription, Id, Ids, Kind, Name, Patch, Row, + Rows, Snapshot, Table, TableDescription, Value, Values, }; -pub use data::{Order, Patches, Query}; +pub use data::{Order, Patches, Query, QueryResult, Reading}; pub use error::{Error, Path, Result, Step}; pub use expr::{ - Color, Colormap, ColormapKind, Expr, Mapped, Size, Stop, Then, When, elapsed, field, hovered, pointer, selected, - units, when, zoom, + Color, Colormap, ColormapKind, Expr, Mapped, Size, Stop, Then, When, elapsed, field, hovered, pointer, px, + selected, units, when, zoom, }; pub use image::{Image, ImageFormat}; pub use json::from_json; @@ -71,11 +75,12 @@ pub use { gpu::{Budget, DeviceEvents, Gpu, work::Work}, panels::Panels, scene::{ - Areas, Axis, Controls, Diagram, Direction, Earth, Edges, Faces, Flow, Format, Grid, Interpolation, Label, - Layered, Layout, Lines, Mark, Marker, Motion, Plane, Plot, Port, Projection, Route, Scene, Shadow, Shape, Side, - Space, Stress, Theme, Vertices, Zoom, + AreaTheme, Areas, Axis, Controls, Diagram, DiagramTheme, Direction, Earth, EarthTheme, EdgeTheme, Edges, Faces, + FlowTheme, FocusTheme, Format, Grid, Interpolation, Label, Layered, Layout, Length, LineTheme, Lines, Mark, + Marker, Motion, Plane, Plot, PlotTheme, Port, Projection, Route, Scene, ShadowTheme, Shape, Side, Space, + Stress, TextTheme, Theme, VertexTheme, Vertices, Zoom, }, - shot::Shot, + shot::{Capture, Shot}, tiles::Tiles, video::{Encoding, Mp4, Pixels, Sink, Video}, view::{ diff --git a/crates/latkit/src/order.rs b/crates/latkit/src/order.rs index 8d4161e4..0c18e6e3 100644 --- a/crates/latkit/src/order.rs +++ b/crates/latkit/src/order.rs @@ -1,26 +1,28 @@ -//! The sorts latkit compiles. Each sort by a closure compiles the standard library's sort anew, so -//! every keyed sort ends in one of a few sorts of keys: items ordered by a key, least first, ties in -//! the order they came. +//! Stable sorts that share a few compiled sort functions. +//! +//! Each closure passed to a standard library sort compiles another copy of the sort. So keyed sorts +//! here build an array of keys, sort it with one of a few non-generic functions, and permute the +//! items to match. Keys sort ascending, and ties keep their input order. -/// Sorts keys packed with the index they came from: the one sort of 64-bit keys. +/// Sorts 64-bit keys packed above their 32-bit source index. Every 64-bit keyed sort uses it. #[inline(never)] fn sort_narrow(keys: &mut [u128]) { keys.sort_unstable(); } -/// Sorts 128-bit keys with the index they came from. +/// Sorts 128-bit keys paired with their source index. #[inline(never)] fn sort_wide(keys: &mut [(u128, u32)]) { keys.sort_unstable(); } -/// Sorts names with the index they came from. +/// Sorts names paired with their source index. #[inline(never)] fn sort_names(keys: &mut [(&str, u32)]) { keys.sort_unstable(); } -/// `items` ordered by `key`, least first, ties in the order they came. +/// Sorts `items` by `key`, ascending. Ties keep their input order. pub(crate) fn by_key(items: &mut [T], mut key: impl FnMut(&T) -> u64) { let mut keys: Vec = items.iter().zip(0u32..).map(|(item, i)| u128::from(key(item)) << 32 | u128::from(i)).collect(); @@ -28,14 +30,14 @@ pub(crate) fn by_key(items: &mut [T], mut key: impl FnMut(&T) -> u64) { permute(items, keys.into_iter().map(|k| k as u32 as usize).collect()); } -/// `items` ordered by a 128-bit `key`, least first, ties in the order they came. +/// Sorts `items` by a 128-bit `key`, ascending. Ties keep their input order. pub(crate) fn by_wide_key(items: &mut [T], mut key: impl FnMut(&T) -> u128) { let mut keys: Vec<(u128, u32)> = items.iter().zip(0u32..).map(|(item, i)| (key(item), i)).collect(); sort_wide(&mut keys); permute(items, keys.into_iter().map(|(_, i)| i as usize).collect()); } -/// `items` ordered by `name`, ties in the order they came. +/// Sorts `items` by `name`. Ties keep their input order. pub(crate) fn by_name(items: &mut [T], name: impl Fn(&T) -> &str) { let mut keys: Vec<(&str, u32)> = items.iter().zip(0u32..).map(|(item, i)| (name(item), i)).collect(); sort_names(&mut keys); @@ -43,29 +45,43 @@ pub(crate) fn by_name(items: &mut [T], name: impl Fn(&T) -> &str) { permute(items, from); } -/// `values` ordered as `f64::total_cmp` orders them. +/// Sorts `values` by `f64::total_cmp`. #[inline(never)] pub(crate) fn numbers(values: &mut [f64]) { values.sort_unstable_by(f64::total_cmp); } -/// A number as a key ordering as `f64::total_cmp` does. +/// Maps `x` to a key that orders as `f64::total_cmp` does. pub(crate) fn number(x: f64) -> u64 { let bits = x.to_bits(); if bits >> 63 == 1 { !bits } else { bits | 1 << 63 } } -/// Two keys as one, `high` first. +/// Maps `x` to a key that orders as `f32::total_cmp` does: the words `lib/reduce.wesl` reduces +/// values to. +#[cfg(feature = "gpu")] +pub(crate) fn number32(x: f32) -> u32 { + let bits = x.to_bits(); + if bits >> 31 == 1 { !bits } else { bits | 1 << 31 } +} + +/// The value whose key is `key`. The inverse of [`number32`]. +#[cfg(feature = "gpu")] +pub(crate) fn from_number32(key: u32) -> f32 { + f32::from_bits(if key >> 31 == 1 { key & !(1 << 31) } else { !key }) +} + +/// Packs two keys into one that orders by `high`, then `low`. pub(crate) fn pair(high: u32, low: u32) -> u64 { u64::from(high) << 32 | u64::from(low) } -/// Two keys as one, `high` first. +/// Packs two keys into one that orders by `high`, then `low`. pub(crate) fn wide(high: u64, low: u64) -> u128 { u128::from(high) << 64 | u128::from(low) } -/// Moves `items` into the order `from` lists: item `k` becomes what was at `from[k]`. +/// Reorders `items` in place so that item `k` becomes the item previously at `from[k]`. fn permute(items: &mut [T], mut from: Vec) { for k in 0..items.len() { let mut at = k; @@ -101,4 +117,17 @@ mod tests { by_name(&mut names, |n| n); assert_eq!(names, ["a", "a", "b", "c"]); } + + #[cfg(feature = "gpu")] + #[test] + fn narrow_keys_order_like_their_values_and_invert() { + use super::{from_number32, number32}; + let values = [-1e9_f32, -2.5, -0.0, 0.0, 1e-9, 3.0, 1e9]; + for pair in values.windows(2) { + assert!(number32(pair[0]) <= number32(pair[1]), "{pair:?}"); + } + for v in values { + assert_eq!(from_number32(number32(v)), v); + } + } } diff --git a/crates/latkit/src/panels.rs b/crates/latkit/src/panels.rs index 48b8ff59..dce19118 100644 --- a/crates/latkit/src/panels.rs +++ b/crates/latkit/src/panels.rs @@ -1,11 +1,11 @@ -//! Panels: where views draw in a target they share. A canvas showing several views and a picture or -//! video composing them lay their views out the same way. +//! Panels: the region of a shared target each view draws in. A canvas with several views and a +//! picture or video composed from them use the same layout. use crate::{Error, Result}; -/// Where each of several views draws in a target: x, y, width, and height as fractions of it, a -/// panel per view, in order; a later panel draws over an earlier one. None given is one view filling -/// the target. +/// The region of a shared target each view draws in, one panel per view, in order. A panel is +/// `[x, y, width, height]` as fractions of the target, from its top left. Later panels draw over +/// earlier ones. With no panels, one view fills the target. /// /// ``` /// use latkit::Panels; @@ -18,17 +18,18 @@ use crate::{Error, Result}; pub struct Panels(Vec<[f32; 4]>); impl Panels { - /// How many views the panels take: one when none are given. + /// Number of views the panels lay out. One when no panels are given. pub fn len(&self) -> usize { self.0.len().max(1) } - /// Whether no parts were given, so one view fills the target. + /// Whether no panels were given, so one view fills the target. pub fn is_empty(&self) -> bool { self.0.is_empty() } - /// The pixels of a target of `size` each view draws in, edges on multiples of `align`. + /// Each view's pixel rectangle in a target of `size`, with edges rounded to multiples of + /// `align`. Fails if a panel covers no pixels. pub(crate) fn rects(&self, size: [u32; 2], align: [u32; 2]) -> Result> { let whole = [[0.0, 0.0, 1.0, 1.0]]; let parts = if self.0.is_empty() { &whole[..] } else { &self.0[..] }; @@ -62,7 +63,7 @@ impl From<[[f32; 4]; N]> for Panels { } } -/// A view's panel of a target in pixels: its top left corner and its size. +/// A view's panel in target pixels: its top left corner and its size. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) struct Rect { pub(crate) origin: [u32; 2], @@ -83,7 +84,7 @@ mod tests { use super::*; #[test] - fn panels_split_a_target_on_whole_blocks() { + fn panel_edges_round_to_multiples_of_the_alignment() { let panels = Panels::from([[0.0, 0.0, 1.0, 0.7], [0.0, 0.7, 1.0, 0.3]]); let rects = panels.rects([1920, 1080], [4, 2]).expect("valid"); assert_eq!(rects, [Rect { origin: [0, 0], size: [1920, 756] }, Rect { origin: [0, 756], size: [1920, 324] }]); diff --git a/crates/latkit/src/pending.rs b/crates/latkit/src/pending.rs index 5ad3d34d..29a1732f 100644 --- a/crates/latkit/src/pending.rs +++ b/crates/latkit/src/pending.rs @@ -1,5 +1,5 @@ -//! The one future latkit hands out: what a view will answer once the GPU has, and what a host's job -//! or fetch will bring. +//! The future latkit returns for every deferred result: a view's answer once the GPU produces +//! it, or the result of a host's job or fetch. use std::future::Future; use std::pin::Pin; @@ -8,10 +8,11 @@ use std::task::{Context, Poll, Waker}; use crate::{Error, Result}; -/// A value still to come: a future its [`Resolver`] resolves, from a view's frames, a host's worker, -/// or the network. A view's answers resolve to an error when the device is lost, and stay pending -/// while no frames are drawn. Destroying their owner cancels them. Await it, or [`take`](Pending::take) -/// it from a frame loop. +/// A future value, resolved by its [`Resolver`] from a view's frames, a host's worker, or the +/// network. Await it, or poll it with [`take`](Pending::take) from a frame loop. +/// +/// A view's answers resolve to an error when the device is lost, and stay pending while no frames +/// are drawn. Dropping their owner cancels them. #[must_use = "a pending value does nothing unless awaited or taken"] pub struct Pending(Arc>>); @@ -21,23 +22,23 @@ enum Slot { Taken, } -/// What resolves a [`Pending`]: a host keeps it until its job or fetch is done. Dropping it without -/// resolving cancels the answer and wakes its waiter. +/// The write side of a [`Pending`]. A host keeps it until its job or fetch finishes. Dropping it +/// without resolving resolves the value to [`Error::Cancelled`] and wakes its waiter. pub struct Resolver(Arc>>); impl Pending { - /// A pending value, and what resolves it. + /// Creates a pending value and its resolver. pub fn new() -> (Pending, Resolver) { let slot = Arc::new(Mutex::new(Slot::Waiting { waker: None, errors: None })); (Pending(slot.clone()), Resolver(slot)) } - /// A value already here. + /// A pending value that is already resolved. pub fn ready(result: Result) -> Pending { Pending(Arc::new(Mutex::new(Slot::Ready(result)))) } - /// Wakes `waker` once the value is resolved: at once when it already is. + /// Wakes `waker` when the value resolves, or immediately if it already has. #[cfg(feature = "gpu")] pub(crate) fn notify(&self, waker: &Waker) { let mut slot = self.0.lock().unwrap_or_else(PoisonError::into_inner); @@ -51,8 +52,9 @@ impl Pending { } } - /// Reports a host job's failure even if its derivation keeps an earlier result. Called once - /// before handing the pending value to a derivation; superseded/dropped consumers report nothing. + /// Pushes this host job's failure to `errors`, even if the derivation consuming it keeps an + /// earlier result. Call once, before handing the pending value to the derivation. Nothing is + /// reported after the pending value is dropped, such as when a newer job supersedes it. #[cfg(any(feature = "gpu", test))] pub(crate) fn report_errors(&self, errors: &Arc>>) { let mut slot = self.0.lock().unwrap_or_else(PoisonError::into_inner); @@ -63,7 +65,7 @@ impl Pending { } } - /// The value once resolved, taking it; `None` while it waits, and after it was taken. + /// Takes the value once resolved. Returns `None` while it is pending and after it was taken. pub fn take(&self) -> Option> { let mut slot = self.0.lock().unwrap_or_else(PoisonError::into_inner); match std::mem::replace(&mut *slot, Slot::Taken) { @@ -78,12 +80,12 @@ impl Pending { } impl Resolver { - /// Whether the consumer dropped its pending value, so unstarted work can be skipped. + /// Whether the consumer dropped its pending value. Work not yet started can then be skipped. pub fn is_cancelled(&self) -> bool { Arc::strong_count(&self.0) == 1 } - /// Resolves the pending value with `result`, waking whatever awaits it. + /// Resolves the pending value with `result` and wakes its waiter. pub fn resolve(self, result: Result) { let mut slot = self.0.lock().unwrap_or_else(PoisonError::into_inner); if let Slot::Waiting { errors: Some(errors), .. } = &*slot @@ -138,6 +140,52 @@ impl Future for Pending { } } +/// The wakers of everything watching one thing, keyed by watcher id. Each watcher keeps only its +/// latest waker. Waking wakes every watcher. +#[derive(Default)] +pub(crate) struct Watchers(Mutex>); + +impl Watchers { + /// Registers `waker` for watcher `id`, replacing its previous one. + #[cfg(feature = "gpu")] + pub(crate) fn watch(&self, id: u64, waker: &Waker) { + let mut watchers = self.0.lock().unwrap_or_else(PoisonError::into_inner); + match watchers.iter_mut().find(|(watcher, _)| *watcher == id) { + Some((_, kept)) => kept.clone_from(waker), + None => watchers.push((id, waker.clone())), + } + } + + /// Removes watcher `id`'s waker. Returns how many watchers are left. + #[cfg(feature = "gpu")] + pub(crate) fn unwatch(&self, id: u64) -> usize { + let mut watchers = self.0.lock().unwrap_or_else(PoisonError::into_inner); + watchers.retain(|(watcher, _)| *watcher != id); + watchers.len() + } + + /// Wakes every watcher. + pub(crate) fn wake_all(&self) { + let wakers: Vec = + self.0.lock().unwrap_or_else(PoisonError::into_inner).iter().map(|(_, waker)| waker.clone()).collect(); + for waker in wakers { + waker.wake(); + } + } +} + +/// A waker that wakes every watcher. +#[cfg(feature = "gpu")] +impl std::task::Wake for Watchers { + fn wake(self: Arc) { + self.wake_all(); + } + + fn wake_by_ref(self: &Arc) { + self.wake_all(); + } +} + impl std::fmt::Debug for Pending { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.write_str("Pending") diff --git a/crates/latkit/src/resource.rs b/crates/latkit/src/resource.rs index 8ea8f821..1643b56b 100644 --- a/crates/latkit/src/resource.rs +++ b/crates/latkit/src/resource.rs @@ -3,7 +3,7 @@ use crate::Pending; use std::ops::Range; use std::sync::Arc; -/// A source of bytes used by JSON data documents and their binary slices. +/// A source of bytes for JSON data documents and their binary slices. pub trait Fetch: Send + Sync { /// Resolves to the requested bytes, or an error. A missing resource returns empty bytes. fn fetch(&self, request: &Request) -> Pending>; @@ -16,22 +16,22 @@ impl Fetch for T { } } -/// Bytes to fetch: a URL, and the range of its bytes, or all of them. +/// Bytes to fetch: a URL and an optional byte range. #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub struct Request { /// The URL. pub url: Arc, - /// Bytes `start..end`; `None` fetches every byte. + /// Bytes `start..end`. `None` fetches every byte. pub range: Option>, } impl Request { - /// Every byte at `url`. + /// A request for every byte at `url`. pub fn new(url: impl Into>) -> Request { Request { url: url.into(), range: None } } - /// Bytes `range` at `url`. + /// A request for bytes `range` at `url`. pub fn range(url: impl Into>, range: Range) -> Request { Request { url: url.into(), range: Some(range) } } diff --git a/crates/latkit/src/scene/areas.rs b/crates/latkit/src/scene/areas.rs index 01ad02f2..a3a5a13c 100644 --- a/crates/latkit/src/scene/areas.rs +++ b/crates/latkit/src/scene/areas.rs @@ -4,17 +4,18 @@ use crate::gpu::shaders as wgsl; use crate::{Error, Kind, Name, Result}; mark!( -/// Rows drawn as the areas a polygon field holds: parcels, states, lakes, the land and water of a -/// map. The view cuts each row's polygons into triangles once, off the frame when they are large; -/// on the earth they bend with the globe and stay whole across the antimeridian. Outline areas with -/// a [`Lines`](crate::Lines) mark of the same table, which draws their rings. +/// Rows drawn as the areas of a polygon field: parcels, states, lakes, the land and water of a map. +/// The view triangulates each row's polygons once, in the background when they are large. On the +/// earth, areas follow the globe's curve and stay unbroken across the antimeridian. To outline +/// areas, add a [`Lines`](crate::Lines) mark of the same table. It draws their rings. /// /// ``` /// # use latkit::{field, Areas, Colormap}; /// let states = Areas::of("map", "states").color(field("load").colormap(Colormap::MAKO)); /// ``` -Areas { - /// The polygon field each row fills: one polygon's rings, or polygons. `geometry` unless set. +Areas("areas") { + /// The polygon field each row fills. It holds one polygon's rings or several polygons. Defaults + /// to the field named `geometry`. geometry: Option, }); @@ -24,15 +25,15 @@ impl Areas { Areas { style: Style::new(data.into(), table.into()), geometry: None } } - /// The polygon field each row fills: one polygon's rings, or polygons. Unless set, the field - /// `geometry`. + /// Sets the polygon field each row fills. It holds one polygon's rings or several polygons. + /// Defaults to the field named `geometry`. #[must_use] pub fn geometry(mut self, field: impl Into) -> Areas { self.geometry = Some(field.into()); self } - /// The field each row fills. + /// The polygon field each row fills. pub(crate) fn field(&self) -> Name { self.geometry.clone().unwrap_or_else(|| GEOMETRY.into()) } @@ -51,14 +52,14 @@ impl Areas { None => c.theme("color", wgsl::THEME_AREA, color)?, } c.channel("visible", &self.style.visible, Store::Flag { word: wgsl::AREA_FLAGS, bit: wgsl::VISIBLE })?; - let mut drawing = Drawing::new(MarkKind::Areas, wgsl::AREA_WORDS); + let mut drawing = Drawing::new(MarkKind::Areas); drawing.geometry = c.derived_path(&triangles_of(&field)); Ok(drawing) } } -/// The derived field holding the triangles of polygon field `field`: per row the point its label -/// hangs from, then three points per triangle. +/// The name of the derived field that holds the triangles of polygon field `field`. Each row holds +/// its label's anchor point, then three points per triangle. pub(crate) fn triangles_of(field: &str) -> Name { format!("{field}.triangles").into() } diff --git a/crates/latkit/src/scene/compile.rs b/crates/latkit/src/scene/compile.rs index 12966f07..a7c5be51 100644 --- a/crates/latkit/src/scene/compile.rs +++ b/crates/latkit/src/scene/compile.rs @@ -1,10 +1,10 @@ -//! One schema compiler for validation and interactive rendering. +//! Compiles marks against dataset schemas, for both validation and views. use std::sync::Arc; use super::mark::{Context, Drawing}; use crate::expr::compile::{Compiler, Fields, Program}; -use crate::{Data, Error, Kind, Mark, Name, Result, Table}; +use crate::{Error, Kind, Mark, Name, Result, Snapshot, Table}; /// A mark compiled against the schemas of its dataset and any datasets it joins. pub(crate) struct Compiled { @@ -13,8 +13,11 @@ pub(crate) struct Compiled { pub(crate) schemas: Vec<(Name, u64)>, } -/// Missing data stays pending in a live view; validation requires it before calling this. -pub(crate) fn compile(mark: &Mark, all: &[Data], context: Context<'_>) -> Result> { +/// Compiles `mark` against the schemas in `all`. Returns `None` while the mark's dataset or table +/// is missing. A view keeps such a mark pending, and validation reports the missing table before +/// calling this. Errors when an option fails to compile or the mark compiles to more than 32 +/// channels. +pub(crate) fn compile(mark: &Mark, all: &[Snapshot], context: Context<'_>) -> Result> { let style = mark.style(); let Some(data) = all.iter().find(|d| *d.name() == style.data) else { return Ok(None) }; let Some(table) = data.table(&style.table) else { return Ok(None) }; @@ -27,7 +30,7 @@ pub(crate) fn compile(mark: &Mark, all: &[Data], context: Context<'_>) -> Result } let mut schemas = vec![(data.name().clone(), data.schema())]; for column in &program.columns { - if let Some((name, _)) = &column.of + if let Some((name, _)) = &column.from && !schemas.iter().any(|(n, _)| n == name) && let Some(other) = all.iter().find(|d| d.name() == name) { @@ -39,12 +42,12 @@ pub(crate) fn compile(mark: &Mark, all: &[Data], context: Context<'_>) -> Result struct Schema<'a> { table: &'a Table, - all: &'a [Data], + all: &'a [Snapshot], } impl Fields for Schema<'_> { - fn kind(&self, of: Option<&(Name, Name)>, name: &str) -> Option { - match of { + fn kind(&self, from: Option<&(Name, Name)>, name: &str) -> Option { + match from { None => self.table.kind(name), Some((data, table)) => self.all.iter().find(|d| d.name() == data)?.table(table)?.kind(name), } diff --git a/crates/latkit/src/scene/controls.rs b/crates/latkit/src/scene/controls.rs index 014ef9e1..d03d4d66 100644 --- a/crates/latkit/src/scene/controls.rs +++ b/crates/latkit/src/scene/controls.rs @@ -1,103 +1,95 @@ use std::time::Duration; use schemars::JsonSchema; -use serde::{Deserialize, Deserializer}; +use serde::Deserialize; -use super::form::{Presets, preset_or, presets, seconds}; +use super::form::{preset_schema, presets, seconds}; -/// How a view answers the pointer and keyboard, and how much it moves on its own: a preset, or options -/// over the defaults. Each mark says whether its rows pick, select, and drag; these say what the view -/// itself does. +/// How a view responds to the pointer and keyboard, and how much it animates on its own: a preset +/// (`"none"` or `"inspect"`), or options that change the defaults. Each mark sets whether its own +/// rows pick, select, and drag. These options control the view itself. #[derive(Clone, Copy, PartialEq, Debug, Deserialize, JsonSchema)] #[serde(remote = "Self", default, rename_all = "camelCase", deny_unknown_fields)] -#[schemars(transform = presets::)] +#[schemars(transform = preset_schema::)] pub struct Controls { - /// Dragging the background moves the camera. + /// Whether dragging the background pans the camera. Defaults to `true`. pub pan: bool, - /// What zooms: the wheel and pinching, or the wheel only with Control or Meta held (as a - /// trackpad's pinch sends it), leaving the plain wheel to scroll the page; or nothing. + /// Which input zooms the view. `wheel` zooms with the wheel and pinching. `modifier` zooms with + /// pinching, and with the wheel only while Control or Meta is held (as trackpad pinches send + /// it), so the plain wheel scrolls the page. `off` turns zooming off. Defaults to `wheel`. pub zoom: Zoom, - /// Dragging with the secondary button, or with Shift, turns and tilts a map or plane. + /// Whether dragging with the secondary button, or with Shift, rotates and tilts a map or plane. + /// Defaults to `true`. pub turn: bool, - /// Tab steps through the rows nearest the view's middle and arrows to the nearest row their way; - /// Enter opens the row stepped to and Space selects it. Otherwise arrows pan, `+` and `-` zoom, - /// Home fits, and Escape clears the selection. + /// Whether the keyboard controls the view. Defaults to `true`. Tab and Shift+Tab step through + /// the rows nearest the view's center. Arrow keys step to the nearest row in their direction. + /// Enter opens the row stepped to, and Space selects it. With no row stepped to, arrow keys pan + /// (Shift+arrows turn), `+` and `-` zoom, Home fits everything, and Escape clears the + /// selection. pub keyboard: bool, - /// The row the user points at glows. It is hovered either way: events tell it, and `hovered()` - /// reads it. + /// Whether the hovered row glows and a hovered vertex sized in CSS pixels grows. Defaults to + /// `true`. Hover is tracked either way: [`Event::Hovered`](crate::Event) still fires and + /// `hovered()` still returns the row. pub hover: bool, - /// CSS pixels around the pointer that picks reach; touch reaches 8 more. + /// How far a row can be from the pointer and still be picked, in CSS pixels. Touch adds 8. + /// Defaults to 6. pub pick_radius: f32, - /// Dragging with Shift selects the rows in a box, instead of turning; a Shift click still toggles - /// its row. + /// Whether Shift+drag selects the rows inside a box instead of turning the camera. Shift+click + /// still toggles one row. Defaults to `false`. pub select_box: bool, - /// Dragging from a port, from a vertex with Alt, or an edge off its end draws a connection, - /// told as [`Event::Connect`](crate::Event). + /// Whether the user can draw connections by dragging from a port, from a vertex with Alt held, + /// or an edge's end off its vertex. Connections are reported as + /// [`Event::Connect`](crate::Event). Defaults to `false`. pub connect: bool, - /// Delete and Backspace tell [`Event::Delete`](crate::Event) with the selection. + /// Whether Delete and Backspace report the selection as [`Event::Delete`](crate::Event). + /// Defaults to `false`. pub delete: bool, - /// Clicking again where nothing moved selects the next row under the point, nearest first, so - /// rows drawn over one another can each be selected. + /// Whether clicking again at the same point, with the camera unmoved, selects the next row + /// under the pointer, nearest first. This lets overlapping rows each be selected. Defaults to + /// `false`. pub cycle: bool, - /// Seconds a touch or a pen held still asks for a menu, as the secondary button does; zero never - /// does. + /// How many seconds a touch or pen must hold still to request a menu + /// ([`Event::Menu`](crate::Event)), like the secondary button. 0 turns this off. Defaults to + /// 0.5. #[serde(deserialize_with = "seconds")] #[schemars(with = "f64")] pub press: Duration, - /// How much the view moves on its own: eases, flights, coasting, orbits, label fades, comets, and - /// what reads `elapsed()`. + /// How much the view animates on its own: eases, camera flights, coasting, orbits, comets, and + /// options that read `elapsed()`. `system` follows the user's reduced-motion setting, `full` + /// animates everything, and `reduced` animates only motion the user drives. Defaults to + /// `system`. pub motion: Motion, } -/// What zooms a view. +/// Which input zooms a view. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default, Deserialize, JsonSchema)] #[serde(rename_all = "camelCase")] pub enum Zoom { /// The wheel, pinching, and the keys. #[default] Wheel, - /// Pinching and the keys, and the wheel only with Control or Meta held, as a trackpad's pinch - /// sends it: the plain wheel scrolls the page. + /// Pinching and the keys, and the wheel only with Control or Meta held (as trackpad pinches + /// send it), so the plain wheel scrolls the page. Modifier, - /// Nothing. + /// No input zooms. Off, } -/// How much a view moves on its own. +/// How much a view animates on its own. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default, Deserialize, JsonSchema)] #[serde(rename_all = "camelCase")] pub enum Motion { - /// As the platform asks: reduced where the user asked their system for less motion. + /// Follows the system setting: reduced when the user has asked their system for less motion. #[default] System, - /// Every motion. + /// All motion. Full, - /// Only what the user moves: changes apply at once, flights jump, nothing coasts or orbits, - /// labels show without fading, and comets and what reads `elapsed()` hold still. + /// Only motion the user drives: changes apply at once, flights jump, nothing coasts or orbits, + /// and comets and options that read `elapsed()` stay still. Reduced, } -impl Presets for Controls { - const NAMES: &'static [&'static str] = &["none", "inspect"]; - - fn preset(name: &str) -> Option { - match name { - "none" => Some(Controls::none()), - "inspect" => Some(Controls::inspect()), - _ => None, - } - } - - fn options<'de, D: serde::Deserializer<'de>>(deserializer: D) -> Result { - Controls::deserialize(deserializer) - } -} - -impl<'de> Deserialize<'de> for Controls { - fn deserialize>(deserializer: D) -> Result { - preset_or(deserializer) - } -} +presets!(Controls { "none" => Controls::none(), "inspect" => Controls::inspect() }); impl Default for Controls { fn default() -> Controls { @@ -119,7 +111,8 @@ impl Default for Controls { } impl Controls { - /// A view that answers nothing: a still picture, its camera where the application puts it. + /// Controls for a still picture: no panning, zooming, turning, keyboard, hover glow, or press + /// menus, so only the application moves the camera. Marks that select still select on click. pub fn none() -> Controls { Controls { pan: false, @@ -132,8 +125,8 @@ impl Controls { } } - /// A view to inspect rows in: they glow under the pointer, select, and step with the keys, while - /// the camera stays where the application puts it. + /// Controls for inspecting rows: rows glow under the pointer, select on click, and step with + /// the keys. Only the application moves the camera. pub fn inspect() -> Controls { Controls { pan: false, zoom: Zoom::Off, turn: false, ..Controls::default() } } diff --git a/crates/latkit/src/scene/edges.rs b/crates/latkit/src/scene/edges.rs index e3d03361..c8d1303c 100644 --- a/crates/latkit/src/scene/edges.rs +++ b/crates/latkit/src/scene/edges.rs @@ -5,28 +5,30 @@ use crate::gpu::shaders as wgsl; use crate::{Error, Kind, Name, Result}; mark!( -/// Rows drawn as strokes between the vertices two reference fields name: branches, links, wires. In a -/// diagram, a row without ends is a net: it joins every vertex port whose reference field names it. +/// Rows drawn as strokes between two vertices named by reference fields: branches, links, wires. +/// In a diagram, a row without ends is a net. It joins every vertex port whose reference field +/// names it. /// /// ``` /// # use latkit::{field, Edges}; /// let branches = Edges::of("grid", "Branch").ends("from", "to").width(field("rating") / 100.0).flow(field("p")); /// ``` -Edges + stroke { +Edges("edges") + stroke { /// The reference fields naming each row's two vertices. ends: Option<(Name, Name)> as "[from: string, to: string]", - /// A path field of points each row runs through between its ends. + /// A path field whose points each row passes through between its ends. bends: Option, - /// How each row runs between its ends; on the earth geodesic, elsewhere straight, unless set. - /// Orthogonal routes run around blocks, bends rounded; diagrams only. + /// How each row runs between its ends. Defaults to geodesic on the earth and straight + /// elsewhere. `orthogonal` routes around blocks with rounded bends, in diagrams only. route: Option, - /// Whether each row ends in an arrowhead at its second end; a net's at each input it reaches. + /// Whether each row has an arrowhead at its second end. A net has one at each input it + /// reaches. Defaults to `false`. arrows: bool, - /// Draws each row as a pole from its own vertex down to the ground beneath it, on a plane or the - /// earth; it has no ends. + /// Draws each row as a pole from its own vertex down to the ground, on a plane or the earth. + /// Poles take no `ends`, `bends`, or `joins`. Defaults to `false`. poles: bool, - /// Draws each row as a net on a plane or the earth: it joins the vertex rows of `table` whose - /// reference field names it, as a star from their middle to each. + /// Draws each row as a net on a plane or the earth. The net joins the vertex rows of `table` + /// whose `reference` field names it, as a star from their center to each member. joins: Option<(Name, Name)> as "[table: string, reference: string]", }); @@ -45,46 +47,48 @@ impl Edges { } } - /// Draws each row as a net on a plane or the earth: it joins the vertex rows of `table` whose - /// reference field `reference` names it, as a star from their middle to each. In a diagram nets - /// join vertex ports instead. + /// Draws each row as a net on a plane or the earth. The net joins the vertex rows of `table` + /// whose field `reference` names it, as a star from their center to each member. In a diagram, + /// nets join vertex ports. #[must_use] pub fn joins(mut self, table: impl Into, reference: impl Into) -> Edges { self.joins = Some((table.into(), reference.into())); self } - /// Draws each row as a pole: a stroke from the row's own vertex, as the vertex mark drawing the - /// table places it, down to the ground beneath it, on a plane or the earth. Dashes, comets, picks, - /// and labels run along it as along any edge. + /// Draws each row as a pole on a plane or the earth: a stroke from the row's own vertex down to + /// the ground. The vertex is where the vertex mark of the same table places it. Dashes, comets, + /// picking, and labels work as on any edge. #[must_use] pub fn poles(mut self) -> Edges { self.poles = true; self } - /// A path field of points each row runs through between its ends. + /// Sets a path field whose points each row passes through between its ends. #[must_use] pub fn bends(mut self, field: impl Into) -> Edges { self.bends = Some(field.into()); self } - /// How each row runs between its ends; on the earth geodesic, elsewhere straight, unless set. + /// Sets how each row runs between its ends. Defaults to geodesic on the earth and straight + /// elsewhere. #[must_use] pub fn route(mut self, route: Route) -> Edges { self.route = Some(route); self } - /// Whether each row ends in an arrowhead at its second end; a net's at each input it reaches. + /// Sets whether each row has an arrowhead at its second end. A net has one at each input it + /// reaches. #[must_use] pub fn arrows(mut self, on: bool) -> Edges { self.arrows = on; self } - /// The reference fields naming each row's two vertices. + /// Sets the reference fields naming each row's two vertices. #[must_use] pub fn ends(mut self, from: impl Into, to: impl Into) -> Edges { self.ends = Some((from.into(), to.into())); @@ -94,22 +98,22 @@ impl Edges { pub(crate) fn compile(&self, c: &mut Compiler<'_>, context: Context<'_>) -> Result { if self.joins.is_some() { if self.ends.is_some() || self.bends.is_some() || self.poles { - return Err(Error::invalid("a net joins its members: it has no ends, bends, or poles").at("joins")); + return Err(Error::invalid("a net cannot have ends, bends, or poles").at("joins")); } } else if self.poles { if self.ends.is_some() || self.bends.is_some() { return Err( - Error::invalid("a pole stands on its own row's vertex: it has no ends or bends").at("poles") + Error::invalid("a pole stands on its own row's vertex and cannot have ends or bends").at("poles") ); } - // Both ends are the row's own vertex, which lights as its pole does. - c.own_end(&self.style.table, wgsl::EDGE_FROM)?; - c.own_end(&self.style.table, wgsl::EDGE_TO)?; + // Both ends are the row's own vertex, so the vertex lights up with its pole. + c.own_end(&self.style.table); + c.own_end(&self.style.table); } else if let Some((from, to)) = &self.ends { - c.end(from, wgsl::EDGE_FROM).map_err(|e| e.at("ends"))?; - c.end(to, wgsl::EDGE_TO).map_err(|e| e.at("ends"))?; + c.end(from).map_err(|e| e.at("ends"))?; + c.end(to).map_err(|e| e.at("ends"))?; } else if self.bends.is_some() { - return Err(Error::invalid("bends run between ends; name them").at("bends")); + return Err(Error::invalid("bends need ends").at("bends")); } if let Some(bends) = &self.bends && !matches!(c.kind(bends), Some(Kind::Path)) @@ -118,10 +122,11 @@ impl Edges { } let mut flags = if self.arrows { wgsl::ARROWS } else { 0 }; let color = Store::Color { word: wgsl::EDGE_COLOR, fallback: wgsl::THEME_EDGE }; - match (&self.style.color, context.theme.edge) { + match (&self.style.color, context.theme.edge.color) { (Some(expr), _) => c.channel("color", expr, color)?, (None, Some(_)) => c.theme("color", wgsl::THEME_EDGE, color)?, - // A diagram's wires, and nets without ends to blend, take the text color. + // A diagram's wires, and edges without ends to blend (nets and poles), take the text + // color. (None, None) if context.diagram() || self.ends.is_none() => { c.theme( "color", @@ -139,22 +144,15 @@ impl Edges { theme_width: wgsl::THEME_EDGE_WIDTH, }; self.stroke.compile(c, &words, context.diagram())?; - let mut drawing = Drawing::new(MarkKind::Edges, wgsl::EDGE_WORDS); + let mut drawing = Drawing::new(MarkKind::Edges); drawing.size_px = self.stroke.px(); drawing.flags = flags; - if self.poles { - drawing.path = wgsl::PATH_POLES; - } else if self.joins.is_some() { - // Each net's star runs along the path the view works out from its members. - drawing.path = wgsl::PATH_GEOMETRY; - drawing.geometry = c.derived_path("route"); - } else if context.diagram() { - // Every wire runs along the path its sheet worked out. - drawing.path = wgsl::PATH_GEOMETRY; + if self.joins.is_some() || context.diagram() { + // A net's star and every wire in a diagram follow the route the view computes. drawing.geometry = c.derived_path("route"); - } else { + } else if !self.poles { drawing.geometry = self.bends.as_ref().map_or(wgsl::NONE, |bends| c.path(bends)); - drawing.lane = c.derived_number("lane"); + drawing.lane = c.derived_bits("lane"); } Ok(drawing) } diff --git a/crates/latkit/src/scene/faces.rs b/crates/latkit/src/scene/faces.rs index f29f4008..09b09892 100644 --- a/crates/latkit/src/scene/faces.rs +++ b/crates/latkit/src/scene/faces.rs @@ -4,17 +4,17 @@ use crate::gpu::shaders as wgsl; use crate::{Error, Name, Result}; mark!( -/// Rows drawn as triangles between the vertices three reference fields name: meshes, surfaces, and -/// fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen from -/// above shows its exact colors. +/// Rows drawn as triangles between three vertices named by reference fields: meshes, surfaces, and +/// fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen +/// from above shows its exact colors. /// /// ``` /// # use latkit::Faces; /// let surface = Faces::of("mesh", "Face").corners("a", "b", "c"); /// ``` -Faces { - /// The reference fields naming each row's three vertices. Without a color, a face blends the - /// colors of its corners. +Faces("faces") { + /// The reference fields naming each row's three vertices. Required. Without a color, a face + /// blends the colors of its corners. corners: Option<[Name; 3]> as "[a: string, b: string, c: string]", }); @@ -24,8 +24,8 @@ impl Faces { Faces { style: Style::new(data.into(), table.into()), corners: None } } - /// The reference fields naming each row's three vertices. Without a color, a face blends the - /// colors of its corners. + /// Sets the reference fields naming each row's three vertices. Required. Without a color, a + /// face blends the colors of its corners. #[must_use] pub fn corners(mut self, a: impl Into, b: impl Into, c: impl Into) -> Faces { self.corners = Some([a.into(), b.into(), c.into()]); @@ -36,8 +36,8 @@ impl Faces { let Some(corners) = &self.corners else { return Err(Error::invalid("name the reference fields of each row's three corners").at("corners")); }; - for (corner, word) in corners.iter().zip([wgsl::FACE_A, wgsl::FACE_B, wgsl::FACE_C]) { - c.end(corner, word).map_err(|e| e.at("corners"))?; + for corner in corners { + c.end(corner).map_err(|e| e.at("corners"))?; } let mut flags = 0; match &self.style.color { @@ -47,7 +47,7 @@ impl Faces { None => flags |= wgsl::ENDS_COLOR, } c.channel("visible", &self.style.visible, Store::Flag { word: wgsl::FACE_FLAGS, bit: wgsl::VISIBLE })?; - let mut drawing = Drawing::new(MarkKind::Faces, wgsl::FACE_WORDS); + let mut drawing = Drawing::new(MarkKind::Faces); drawing.flags = flags; Ok(drawing) } diff --git a/crates/latkit/src/scene/form.rs b/crates/latkit/src/scene/form.rs index 2817f120..b439d207 100644 --- a/crates/latkit/src/scene/form.rs +++ b/crates/latkit/src/scene/form.rs @@ -1,6 +1,5 @@ -//! What scenes read as JSON. Each type deserializes as itself and its schema is derived from it; a -//! type a name can stand for also reads one of its presets' names, such as `"inspect"`, `"globe"`, or -//! `"stress"`. +//! JSON reading for scenes. Each type deserializes and takes its schema from its own definition. A +//! type with presets also accepts a preset's name, such as `"inspect"`, `"earth"`, or `"stress"`. use std::fmt; use std::marker::PhantomData; @@ -12,20 +11,20 @@ use serde::de::value::MapAccessDeserializer; use serde::de::{self, Deserializer, MapAccess, Visitor}; use serde_json::Value; -/// A type a name can stand for. +/// A type with named presets. pub(crate) trait Presets: Sized { - /// The names, as the schema lists them. + /// The preset names, in the order the schema lists them. const NAMES: &'static [&'static str]; - /// The value `name` stands for. + /// The preset named `name`, if any. fn preset(name: &str) -> Option; - /// The value its options, written out, make: what its derive reads. + /// Reads the type from its options object with the derived implementation. fn options<'de, D: Deserializer<'de>>(deserializer: D) -> Result; } -/// `T` read from one of its presets' names, or from its options, read where they are written so an -/// error names the path to the option it could not read. +/// Reads `T` from a preset name or from its options. Options are read in place, so an error names +/// the path to the option that failed. pub(crate) fn preset_or<'de, D: Deserializer<'de>, T: Presets>(deserializer: D) -> Result { struct Named(PhantomData); @@ -49,11 +48,44 @@ pub(crate) fn preset_or<'de, D: Deserializer<'de>, T: Presets>(deserializer: D) deserializer.deserialize_any(Named(PhantomData)) } -/// `T`'s schema as one of its presets' names or what it derived, its description kept on the whole. -pub(crate) fn presets(schema: &mut Schema) { +/// Declares a type's presets, each name once, as `Presets`. The type reads its options with the +/// `Deserialize` that serde derives as `remote = "Self"`. Unless it ends `; read by hand`, it also +/// gets a `Deserialize` that takes a preset's name or options. +macro_rules! presets { + ($type:ident { $($name:literal => $value:expr),+ $(,)? }) => { + presets!($type { $($name => $value),+ }; read by hand); + impl<'de> serde::Deserialize<'de> for $type { + fn deserialize>(deserializer: D) -> Result<$type, D::Error> { + crate::scene::form::preset_or(deserializer) + } + } + }; + ($type:ident { $($name:literal => $value:expr),+ $(,)? }; read by hand) => { + impl crate::scene::form::Presets for $type { + const NAMES: &'static [&'static str] = &[$($name),+]; + + fn preset(name: &str) -> Option<$type> { + match name { + $($name => Some($value),)+ + _ => None, + } + } + + fn options<'de, D: serde::Deserializer<'de>>(deserializer: D) -> Result<$type, D::Error> { + $type::deserialize(deserializer) + } + } + }; +} + +pub(crate) use presets; + +/// Rewrites `T`'s derived schema to accept a preset name or the derived form. The description +/// moves to the combined schema. +pub(crate) fn preset_schema(schema: &mut Schema) { let mut derived = std::mem::replace(schema, json_schema!({})); let description = derived.remove("description"); - // A variant without options that a preset names already is the preset. + // Drop unit variants that a preset already names, since the preset's name covers them. for key in ["oneOf", "anyOf"] { if let Some(variants) = derived.get_mut(key).and_then(Value::as_array_mut) { variants.retain(|v| !v.get("const").and_then(Value::as_str).is_some_and(|name| T::NAMES.contains(&name))); @@ -65,8 +97,8 @@ pub(crate) fn presets(schema: &mut Schema) { } } -/// A duration read as seconds. +/// Reads a duration in seconds. Errors on negative or out-of-range values. pub(crate) fn seconds<'de, D: Deserializer<'de>>(deserializer: D) -> Result { let seconds = f64::deserialize(deserializer)?; - Duration::try_from_secs_f64(seconds).map_err(|_| de::Error::custom("seconds from 0, within range")) + Duration::try_from_secs_f64(seconds).map_err(|_| de::Error::custom("seconds must be finite and not negative")) } diff --git a/crates/latkit/src/scene/label.rs b/crates/latkit/src/scene/label.rs index 7f7d3964..7a8b2c07 100644 --- a/crates/latkit/src/scene/label.rs +++ b/crates/latkit/src/scene/label.rs @@ -7,56 +7,57 @@ use serde::de::{self, Deserializer}; use crate::Expr; use crate::expr::form::Written; -/// A row's text, placed beside, inside, or along what its mark draws, following it every frame: a -/// text field, or options. Labels that would overlap give way by priority; among equals, those -/// already shown keep their place. In a diagram a label is part of the drawing: it scales with the -/// zoom, fades out once too small to read, and shows wherever it fits its vertex. +/// Text for each row, placed beside, inside, or along the row's shape and moving with it: a text +/// field, or options. A label shows when its row is visible and its text fits, for example inside +/// a block. +/// +/// Labels beside or along their rows never overlap. Where they would, the row drawn on top keeps +/// its label (so `priority` decides), and a dense view labels as many rows as fit. A row whose text +/// is missing or empty has no label, so a `when` that returns the field or missing labels only +/// some rows. In a diagram, a label scales with the zoom, fades out when too small to read, and +/// shows wherever it fits its vertex. /// /// ``` -/// # use latkit::{field, Label, Vertices}; -/// let buses = Vertices::of("grid", "Bus").position("lon", "lat").label(Label::new("name").priority(field("kv"))); +/// # use latkit::{field, when, zoom, Expr, Label, Vertices}; +/// let named = when(zoom().ge(6.0)).then(field("name")).otherwise(Expr::missing()); +/// let buses = Vertices::of("grid", "Bus").position(field("lon"), field("lat")); +/// let labeled = buses.label(Label::new(named)); /// ``` #[derive(Clone, PartialEq, Debug, JsonSchema)] #[serde(rename_all = "camelCase", deny_unknown_fields)] #[schemars(transform = text_or_options)] pub struct Label { - /// A text field. + /// The text field to show, or a `when` that returns that field or missing. A row whose text is + /// missing or empty has no label. In a diagram, the text may read only data, since the sheet + /// sizes blocks around it. pub(crate) text: Expr, - /// CSS pixels, or units in a diagram; the theme's font size unless set. + /// Font size in CSS pixels, or diagram units in a diagram. Defaults to the theme's + /// `text.size`. #[serde(default)] pub(crate) size: Option, - /// Each row's text color; the theme's text color unless set. + /// Each row's text color. Defaults to the theme's text color. #[serde(default)] pub(crate) color: Option, - /// Where labels would overlap, the higher shows; the mark's priority unless set. - #[serde(default)] - pub(crate) priority: Option, - /// How far a line runs before it wraps: CSS pixels, or units in a diagram. + /// The width at which lines wrap, in CSS pixels, or diagram units in a diagram. #[serde(default)] pub(crate) max_width: Option, - /// The CSS font family list its texts are set in; the theme's font unless set. + /// The CSS font family list for the text. Defaults to the theme's `text.font`. #[serde(default)] pub(crate) font: Option>, - /// The most labels shown at once, those of highest priority among the rows in view; where they - /// would overlap, fewer. - #[serde(default)] - pub(crate) max: Option, } -/// A label reads a text field as its text alone, or an object with `text` as its options. +/// Reads a label from a text field alone, or from an options object with `text`. impl<'de> Deserialize<'de> for Label { fn deserialize>(deserializer: D) -> Result { let label = |written: Written| match written { - Written::Object(mut form) if form.text.is_some() => { - form.only("a label", &["text", "size", "color", "priority", "maxWidth", "font", "max"])?; + Written::Object(form) if form.text.is_some() => { + form.only("a label", &["text", "size", "color", "maxWidth", "font"])?; Ok(Label { - color: form.color()?, - text: form.text.expect("held"), + text: form.text.expect("present"), + color: form.color, size: form.size, - priority: form.priority, max_width: form.max_width, font: form.font, - max: form.max, }) } written => written.expr().map(Label::new), @@ -65,59 +66,49 @@ impl<'de> Deserialize<'de> for Label { } } -/// A label's schema: a text field, or its options. +/// A label's schema: a text field or its options. The description moves to the combined schema. fn text_or_options(schema: &mut schemars::Schema) { - let options = std::mem::replace(schema, schemars::json_schema!({})); + let mut options = std::mem::replace(schema, schemars::json_schema!({})); + let description = options.remove("description"); *schema = schemars::json_schema!({ "anyOf": [{ "$ref": "#/$defs/Expr" }, options] }); + if let Some(description) = description { + schema.insert("description".into(), description); + } } impl Label { - /// Labels each row with a text field's value. + /// Labels each row with a text field's value, or with a `when` that returns that field or + /// missing. A row whose text is missing or empty has no label. pub fn new(text: impl Into) -> Label { - Label { text: text.into(), size: None, color: None, priority: None, max_width: None, font: None, max: None } + Label { text: text.into(), size: None, color: None, max_width: None, font: None } } - /// Text size, CSS pixels, or units in a diagram; the theme's font size unless set. + /// Sets the font size in CSS pixels, or diagram units in a diagram. Defaults to the theme's + /// text size. #[must_use] pub fn size(mut self, size: f32) -> Label { self.size = Some(size); self } - /// Each row's text color; the theme's text color unless set. + /// Sets each row's text color. Defaults to the theme's text color. #[must_use] pub fn color(mut self, color: impl Into) -> Label { self.color = Some(color.into()); self } - /// Each row's priority: where labels would overlap, the higher shows. The mark's priority - /// unless set. - #[must_use] - pub fn priority(mut self, priority: impl Into) -> Label { - self.priority = Some(priority.into()); - self - } - - /// How far a line of text runs before it wraps: CSS pixels, or units in a diagram. + /// Sets the width at which lines wrap, in CSS pixels, or diagram units in a diagram. #[must_use] pub fn max_width(mut self, width: f32) -> Label { self.max_width = Some(width); self } - /// The CSS font family list its texts are set in; the theme's font unless set. + /// Sets the CSS font family list for the text. Defaults to the theme's font. #[must_use] pub fn font(mut self, font: impl Into>) -> Label { self.font = Some(font.into()); self } - - /// Shows at most `max` labels at once, those of highest priority among the rows in view; where - /// they would overlap, fewer. - #[must_use] - pub fn max(mut self, max: u32) -> Label { - self.max = Some(max); - self - } } diff --git a/crates/latkit/src/scene/layout.rs b/crates/latkit/src/scene/layout.rs index 00010459..8ab92636 100644 --- a/crates/latkit/src/scene/layout.rs +++ b/crates/latkit/src/scene/layout.rs @@ -1,54 +1,40 @@ use schemars::JsonSchema; -use serde::{Deserialize, Deserializer}; +use serde::Deserialize; -use super::form::{Presets, preset_or, presets}; +use super::form::{preset_schema, presets}; -/// How the view places vertex rows without positions: a diagram's blocks, sized, or the points of a -/// plane or the earth, among the rows linked to them, gaps in the space's units: a name, or `{ stress -/// | layered: options }`. A place, once given, holds: rows added later are placed around the rest, -/// and [`View::arrange`](crate::View::arrange) lays every row out again. +/// How the view places vertex rows that have no position: a name, or +/// `{ stress | layered: options }`. In a diagram it places blocks. On a plane or the earth it +/// places points among the rows they link to. Gaps are in position units: diagram units, plane +/// units, or degrees on the earth. +/// +/// A placed row keeps its place. Rows added later are placed around the others, and +/// [`View::arrange`](crate::View::arrange) lays out every row again. #[derive(Clone, Copy, PartialEq, Debug, Default, Deserialize, JsonSchema)] #[serde(remote = "Self", rename_all = "camelCase", deny_unknown_fields)] -#[schemars(transform = presets::)] +#[schemars(transform = preset_schema::)] pub enum Layout { - /// Unplaced vertices are not drawn. + /// No layout: vertices without a position are not drawn. The default. #[default] None, - /// Distances on the drawing follow distances in the graph. + /// Drawn distances follow graph distances. Stress(Stress), - /// Ranks along a direction, for flows and hierarchies. + /// Ranks along a direction, for processes and hierarchies. Layered(Layered), } -impl Presets for Layout { - const NAMES: &'static [&'static str] = &["none", "stress", "layered"]; +presets!(Layout { + "none" => Layout::None, + "stress" => Layout::Stress(Stress::default()), + "layered" => Layout::Layered(Layered::default()), +}); - fn preset(name: &str) -> Option { - match name { - "none" => Some(Layout::None), - "stress" => Some(Layout::Stress(Stress::default())), - "layered" => Some(Layout::Layered(Layered::default())), - _ => None, - } - } - - fn options<'de, D: serde::Deserializer<'de>>(deserializer: D) -> Result { - Layout::deserialize(deserializer) - } -} - -impl<'de> Deserialize<'de> for Layout { - fn deserialize>(deserializer: D) -> Result { - preset_or(deserializer) - } -} - -/// Stress majorization: each pair of vertices pulls toward its graph distance; then their boxes -/// part, keeping a quarter of the gap clear. +/// Stress majorization: vertices move until drawn distances match graph distances. Then +/// overlapping boxes separate, keeping a quarter of `gap` between them. #[derive(Clone, Copy, PartialEq, Debug, Deserialize, JsonSchema)] #[serde(default, rename_all = "camelCase", deny_unknown_fields)] pub struct Stress { - /// Units between adjacent vertices' centers. + /// Distance between the centers of linked vertices, in position units. Defaults to 40. #[schemars(range(min = 0))] pub gap: f64, } @@ -59,21 +45,22 @@ impl Default for Stress { } } -/// Layered drawing: vertices in ranks, edges pointing along `flow`, and nets from their outputs. +/// Layered drawing: vertices in ranks, edges pointing along `direction`, and nets ranked from their +/// outputs. #[derive(Clone, Copy, PartialEq, Debug, Deserialize, JsonSchema)] #[serde(default, rename_all = "camelCase", deny_unknown_fields)] pub struct Layered { - /// Where edges point. - pub flow: Direction, - /// Units between vertices of a rank. + /// The direction edges point. Defaults to `right`. + pub direction: Direction, + /// Distance between vertices within a rank, in position units. Defaults to 30. pub gap: f64, - /// Units between ranks. + /// Distance between ranks, in position units. Defaults to 80. pub rank_gap: f64, } impl Default for Layered { fn default() -> Layered { - Layered { flow: Direction::Right, gap: 30.0, rank_gap: 80.0 } + Layered { direction: Direction::Right, gap: 30.0, rank_gap: 80.0 } } } @@ -81,12 +68,12 @@ impl Default for Layered { #[derive(Clone, Copy, PartialEq, Eq, Debug, Deserialize, JsonSchema)] #[serde(rename_all = "camelCase")] pub enum Route { - /// Straight across the screen. + /// A straight line on screen. Straight, /// Along the great circle, on the earth. Geodesic, - /// In horizontal and vertical runs around vertices, leaving and entering at their sides, bends - /// rounded. Diagrams only. + /// Horizontal and vertical runs around vertices that leave and enter at their sides, with + /// rounded bends. Diagrams only. Orthogonal, } @@ -97,9 +84,10 @@ pub enum Interpolation { /// Straight from each sample to the next. #[default] Linear, - /// Each value holds from the sample before it: steps rise at their samples' ends. + /// Each value holds from the previous sample up to its own, so the line steps at the previous + /// sample. StepBefore, - /// Each value holds until the next sample: steps rise at the next sample. + /// Each value holds until the next sample, where the line steps. StepAfter, } @@ -118,6 +106,8 @@ pub enum Direction { Up, } +crate::wire::codes!(Direction { Right = 0, Left = 1, Down = 2, Up = 3 }); + impl From for Layout { fn from(stress: Stress) -> Layout { Layout::Stress(stress) diff --git a/crates/latkit/src/scene/lines.rs b/crates/latkit/src/scene/lines.rs index 8cfe55d6..834df07d 100644 --- a/crates/latkit/src/scene/lines.rs +++ b/crates/latkit/src/scene/lines.rs @@ -7,22 +7,23 @@ use crate::gpu::shaders as wgsl; use crate::{Error, Kind, Name, Result, Space}; mark!( -/// Rows drawn as lines: along a path field (roads, rivers, borders) or a polygon field's rings, -/// outlining its areas; on a plot, through a sampled field's history across the data's axis, what a -/// monitor shows. On the earth paths bend with the globe and stay whole across the antimeridian. +/// Rows drawn as lines along a path field (roads, rivers, borders) or a polygon field's rings (area +/// outlines). On a plot, each row traces a sampled field's history across the data's axis, like a +/// monitor. On the earth, paths follow the globe's curve and stay unbroken across the antimeridian. /// /// ``` /// # use latkit::Lines; /// let roads = Lines::of("city", "roads").width(2.0); /// let height = Lines::of("body", "Body").trace("height"); /// ``` -Lines + stroke { - /// The path or polygon field each row runs along; `geometry` unless set. Not on a plot. +Lines("lines") + stroke { + /// The path or polygon field each row runs along. Defaults to the field named `geometry`. Not + /// allowed on a plot. geometry: Option, - /// The sampled field each row traces across the data's axis, on a plot alone. A field not recorded - /// yet draws nothing until its first frame. + /// The sampled field each row traces across the data's axis. Required on a plot, and allowed + /// only there. A field with no frames yet draws nothing until its first frame arrives. trace: Option, - /// How each row runs between its samples, on a plot. + /// How each row runs between its samples on a plot. Defaults to `linear`. interpolation: Interpolation, }); @@ -38,22 +39,23 @@ impl Lines { } } - /// The path or polygon field each row runs along; the field `geometry` unless set. Not on a plot. + /// Sets the path or polygon field each row runs along. Defaults to the field named `geometry`. + /// Not allowed on a plot. #[must_use] pub fn geometry(mut self, field: impl Into) -> Lines { self.geometry = Some(field.into()); self } - /// The sampled field each row traces across the data's axis, on a plot. Unless named, the mark - /// takes the field's name. + /// Sets the sampled field each row traces across the data's axis. Plots only. Unless the mark + /// is [named](Lines::named), the field's name identifies it across scenes. #[must_use] pub fn trace(mut self, field: impl Into) -> Lines { self.trace = Some(field.into()); self } - /// How each row runs between its samples, on a plot. + /// Sets how each row runs between its samples on a plot. Defaults to linear. #[must_use] pub fn interpolation(mut self, interpolation: Interpolation) -> Lines { self.interpolation = interpolation; @@ -66,17 +68,18 @@ impl Lines { } pub(crate) fn compile(&self, c: &mut Compiler<'_>, context: Context<'_>) -> Result { - let mut drawing = Drawing::new(MarkKind::Lines, wgsl::LINE_WORDS); + let mut drawing = Drawing::new(MarkKind::Lines); drawing.size_px = self.stroke.px(); if matches!(context.space, Space::Plot(_)) { if self.geometry.is_some() { - return Err(Error::invalid("a plot traces sampled fields; name one with trace").at("geometry")); + return Err(Error::invalid("in a plot, use trace instead of geometry").at("geometry")); } let Some(field) = &self.trace else { return Err(Error::invalid("name the sampled field each row traces").at("trace")); }; drawing.trace = match c.kind(field) { - // Streamed data appends its first frame later; the mark compiles again then. + // Streamed data may not have its first frame yet. The mark recompiles when it + // arrives. None => wgsl::NONE, _ => c.trace(field).map_err(|e| e.at("trace"))?, }; @@ -87,7 +90,7 @@ impl Lines { }; } else { if self.trace.is_some() { - return Err(Error::invalid("only a plot traces a sampled field; name a path with geometry").at("trace")); + return Err(Error::invalid("outside a plot, use geometry instead of trace").at("trace")); } let field = self.geometry.clone().unwrap_or_else(|| GEOMETRY.into()); match c.kind(&field) { @@ -98,8 +101,7 @@ impl Lines { } None => return Err(Error::invalid("name the path field each row runs along").at("geometry")), } - drawing.path = wgsl::PATH_GEOMETRY; - // On the earth a line runs along its path as it bends with the globe. + // On the earth, a line follows a derived copy of its path that bends with the globe. drawing.geometry = if matches!(context.space, Space::Earth(_)) { c.derived_path(&path_of(&field)) } else { @@ -123,8 +125,8 @@ impl Lines { } } -/// The derived field holding path field `field` as an earth line draws it: unwrapped across the -/// antimeridian, with a point every few degrees along longer segments. +/// The name of the derived field holding path field `field` as drawn on the earth: unwrapped across +/// the antimeridian, with a point added every few degrees along long segments. pub(crate) fn path_of(field: &str) -> Name { format!("{field}.path").into() } diff --git a/crates/latkit/src/scene/mark.rs b/crates/latkit/src/scene/mark.rs index 329925b3..41a41be5 100644 --- a/crates/latkit/src/scene/mark.rs +++ b/crates/latkit/src/scene/mark.rs @@ -5,9 +5,9 @@ use crate::expr::compile::{Compiler, Store}; use crate::gpu::shaders as wgsl; use crate::{Error, Expr, Name, Result, Size, Space}; -/// One drawn table of a scene: `{ vertices | edges | faces | lines | areas: options }` as JSON. -#[derive(Clone, PartialEq, Debug, serde::Deserialize, schemars::JsonSchema)] -#[serde(rename_all = "camelCase", deny_unknown_fields)] +/// One table a scene draws. In JSON, the mark's kind is the key that holds its dataset and table, +/// next to its options: `{ "vertices": ["grid", "Bus"], "radius": 4 }`. +#[derive(Clone, PartialEq, Debug)] #[non_exhaustive] pub enum Mark { /// Rows drawn as markers. @@ -22,6 +22,48 @@ pub enum Mark { Areas(Areas), } +impl<'de> serde::Deserialize<'de> for Mark { + /// Reads all the options first, since the key naming the kind can come after them. + fn deserialize>(deserializer: D) -> std::result::Result { + use serde::de::Error as _; + let form = serde_json::Map::deserialize(deserializer)?; + let mut kinds = MarkKind::ALL.into_iter().filter(|kind| form.contains_key(kind.name())); + let (Some(kind), None) = (kinds.next(), kinds.next()) else { + return Err(D::Error::custom( + "a mark names its dataset and table under exactly one of vertices, edges, faces, lines, or areas", + )); + }; + let form = serde_json::Value::Object(form); + match kind { + MarkKind::Vertices => crate::json::nested(form).map(Mark::Vertices), + MarkKind::Edges => crate::json::nested(form).map(Mark::Edges), + MarkKind::Faces => crate::json::nested(form).map(Mark::Faces), + MarkKind::Lines => crate::json::nested(form).map(Mark::Lines), + MarkKind::Areas => crate::json::nested(form).map(Mark::Areas), + } + } +} + +impl schemars::JsonSchema for Mark { + fn schema_name() -> std::borrow::Cow<'static, str> { + "Mark".into() + } + + fn json_schema(generator: &mut schemars::SchemaGenerator) -> schemars::Schema { + let kinds = [ + generator.subschema_for::(), + generator.subschema_for::(), + generator.subschema_for::(), + generator.subschema_for::(), + generator.subschema_for::(), + ]; + schemars::json_schema!({ + "description": "One table a scene draws. The mark's kind is the key that holds its table, next to its options: `{ vertices: [data, table], radius: 4 }`.", + "oneOf": kinds, + }) + } +} + impl From for Mark { fn from(mark: Vertices) -> Mark { Mark::Vertices(mark) @@ -52,7 +94,7 @@ impl From for Mark { } } -/// The field a mark reads its shape from unless told otherwise: what geography's tables hold. +/// The default name of the field a mark reads geometry from, the name geographic tables use. pub(crate) const GEOMETRY: &str = "geometry"; /// Options every mark has. @@ -67,7 +109,6 @@ pub(crate) struct Style { pub(crate) name: Option, pub(crate) interaction: Interaction, pub(crate) label: Option

, +} + +impl Drawn { + /// The drawing marks sync to: `next` if any, else `current`. + pub(crate) fn latest(&self) -> Option<&Arc> { + self.next.as_ref().or(self.current.as_ref()) + } + + /// The slots of `drawing` (this table's edge drawing) whose rows end at any of `places` in the + /// vertex table, ascending: resident pages by their cached endpoints, and every kept page. + pub(crate) fn naming(&self, drawing: &Drawing, places: &[u32]) -> Vec { + (0..drawing.slot_count()) + .filter(|&s| match drawing.holds(s) { + Holds::Page(page) => self.ends.lie_in(page, places), + Holds::Kept(_) => true, + Holds::Free => false, + }) + .map(|s| s as u32) + .collect() + } + + /// The merged needs of every edge table ending at this table, merged again only when one + /// changes. + fn ending(&mut self, endings: &[Arc]) -> Arc { + match endings { + [] => Arc::default(), + [one] => one.clone(), + many => { + let stamp = stamp_of(many.iter().map(|e| e.stamp).collect::>()); + if stamp != self.ending.0 { + self.ending = (stamp, Arc::new(Ending::merged(many))); + } + self.ending.1.clone() + } + } + } + + /// The pinned rows: `own` (selected or framed by a fit) and the endpoints of kept edges in + /// `ending`. Rebuilt only when those change. + fn pin(&mut self, own: Vec, ending: &Ending) -> Pinned { + let stamp = stamp_of((&own, &ending.kept)); + if stamp != self.pinned.stamp { + let mut places = own; + places.extend_from_slice(&ending.kept); + places.sort_unstable(); + places.dedup(); + self.pinned = Pinned { places: places.into(), stamp }; + } + self.pinned.clone() + } +} + +impl View { + /// Lays out each paged table's drawing from the camera's choices. A new drawing replaces the + /// current one when its inputs changed, or when the current one finished fading. A table that + /// stops being paged loses its drawing and draws all its rows, in its own order. + pub(super) fn lay_out_drawings(&mut self) { + let choices = std::mem::take(&mut self.choices); + self.drawings.retain(|key, _| choices.contains_key(key)); + let fades = !self.ease().is_zero(); + let ease = self.ease().as_secs_f64(); + // Edge tables go first, so their vertex tables know which endpoints to make resident and + // which kept-edge endpoints to draw. + let mut keys: Vec<&(Name, Name)> = choices.keys().collect(); + crate::order::by_key(&mut keys, |key| { + u64::from(!matches!(self.planning.plans().get(*key).map(|p| &p.place), Some(Place::Ends(_)))) + }); + let mut ends: crate::data::Keyed<(Name, Name), Vec>> = crate::data::Keyed::default(); + for key in keys { + let chosen = &choices[key]; + let endings = ends.remove(key).unwrap_or_default(); + let own = self.pinned(key, &chosen.order); + let (scene, room) = (self.scene_version, self.room(key)); + let table = self.snapshot(&key.0).and_then(|d| d.table(&key.1)).cloned(); + let drawn = self.drawings.entry(key.clone()).or_default(); + let written = match (&drawn.table, &table) { + (Some(before), Some(now)) => Written::since(before, now, &chosen.order), + _ => Written::Every, + }; + drawn.table = table; + let ending = drawn.ending(&endings); + let pinned = drawn.pin(own, &ending); + let lay = Lay { chosen, pinned: &pinned, ends: &ending, room, scene, written: &written }; + let stamp = lay.stamp(); + let current = drawn.current.clone(); + let latest = drawn.next.as_ref().or(current.as_ref()); + // Idle pages are released when the scene changes or one of their rows is written. + let stale = latest.is_some_and(|d| { + !d.idle.is_empty() && (d.scene != scene || d.idle.iter().any(|&page| written.holds(page))) + }); + if latest.map(|d| d.laid) != Some(stamp) || stale { + drawn.next = Some(Arc::new(Drawing::lay(current.as_deref(), &lay, fades))); + } else if drawn.next.is_none() + && let Some(current) = current.filter(|c| c.fades && self.clock.done(drawn.since, ease)) + { + drawn.next = Some(Arc::new(current.settled(&lay))); + } + // Record what this table's resident edges need of the vertex table they end at. + let Some(place) = self.planning.plans().get(key).map(|p| &p.place) else { continue }; + let Place::Ends(fields) = place else { continue }; + let Some(table) = self.snapshots.iter().find(|d| *d.name() == key.0).and_then(|d| d.table(&key.1)) else { + continue; + }; + let references: Vec<&Reference> = fields + .iter() + .filter_map(|f| match table.field_named(f) { + Some(Field::Reference(r)) => Some(r), + _ => None, + }) + .collect(); + let Some(to) = references.first().map(|r| r.to.clone()) else { continue }; + let target_key = (key.0.clone(), to); + let Some(target) = choices.get(&target_key) else { continue }; + let drawn = self.drawings.get_mut(key).expect("laid out"); + let drawing = drawn.latest().expect("laid out").clone(); + let ending = drawn.ends.ending(&drawing, &target.order, &references); + ends.entry(target_key).or_default().push(ending); + } + self.choices = choices; + } + + /// Sets each mark's `reveal` for this frame: how far its paged table's tiles have faded in or + /// out, from 0 to 1. 1 when they are not fading. + pub(super) fn reveal(&mut self) { + let ease = self.ease().as_secs_f64(); + for shown in &mut self.marks { + let key = (shown.mark.style().data.clone(), shown.mark.style().table.clone()); + shown.reveal = match self.drawings.get(&key) { + Some(drawn) if drawn.current.as_ref().is_some_and(|c| c.fades) => { + self.clock.eased(drawn.since, ease) as f32 + } + _ => 1.0, + }; + } + } + + /// What each paged table of lines or areas draws this frame, for shaping: the pages of its + /// order the camera chose, and the pinned rows outside those pages. + pub(super) fn paged(&self) -> crate::data::Keyed<(Name, Name), crate::derive::Paged> { + let shaped = |(data, table): &(Name, Name)| { + self.scene.marks.iter().any(|mark| { + let style = mark.style(); + matches!(mark, crate::Mark::Lines(_) | crate::Mark::Areas(_)) + && style.data == *data + && style.table == *table + }) + }; + let mut paged = crate::data::Keyed::default(); + for (key, chosen) in self.choices.iter().filter(|(key, _)| shaped(key)) { + let ranges = chosen.ranges.iter().filter(|r| r.start < r.end); + let mut pages: Vec = ranges.flat_map(|r| r.start / PAGE_ROWS..=(r.end - 1) / PAGE_ROWS).collect(); + pages.sort_unstable(); + pages.dedup(); + let pinned = self.pinned(key, &chosen.order); + let kept: Vec = + pinned.into_iter().filter(|&d| pages.binary_search(&(d / PAGE_ROWS)).is_err()).collect(); + paged.insert(key.clone(), crate::derive::Paged { order: chosen.order.clone(), pages, kept }); + } + paged + } + + /// The most pages `key`'s drawing may keep resident. For a table drawn by tiles, as many as the + /// plan's rows fill. Otherwise 0, so no idle page stays. + fn room(&self, key: &(Name, Name)) -> u32 { + match self.planning.plans().get(key).map(|p| p.draw) { + Some(Draw::Tiles { rows }) => rows.div_ceil(PAGE_ROWS), + _ => 0, + } + } + + /// The places in `order` of `key`'s rows drawn regardless of tiles: selected rows and rows a + /// fit frames, ascending. + fn pinned(&self, key: &(Name, Name), order: &Order) -> Vec { + let Some(table) = self.snapshot(&key.0).and_then(|d| d.table(&key.1)) else { return Vec::new() }; + let mut rows: Vec = self + .focus + .selection + .iter() + .chain(&self.focus.fitting) + .filter(|row| row.data == key.0 && row.table == key.1) + .filter_map(|row| table.row(row.id.as_ref())) + .map(|r| order.rank(r as u32)) + .collect(); + rows.sort_unstable(); + rows.dedup(); + rows + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::derive::pyramid::Pyramid; + + fn order(rows: u32) -> Order { + Order::Pyramid { pyramid: Arc::new(Pyramid::new((0..rows).collect(), Vec::new())), rows } + } + + fn chosen(order: &Order, ranges: &[Range]) -> Chosen { + Chosen { order: order.clone(), ranges: ranges.to_vec(), capped: false } + } + + /// Pinned rows at `places`. + fn pinned(places: &[u32]) -> Pinned { + Pinned { places: places.into(), stamp: stamp_of(places) } + } + + /// A choice drawing one range of `order`. + fn one(order: &Order, range: Range) -> Chosen { + Chosen { order: order.clone(), ranges: vec![range], capped: false } + } + + #[test] + fn written_rows_mark_the_pages_that_contain_them() { + let order = order(3 * PAGE_ROWS); + let n = 3 * PAGE_ROWS as usize; + let data = crate::data::Data::new("d"); + data.apply(Table::new("T", crate::data::Ids::count(n as u32)).field("v", vec![0.0; n])).expect("valid"); + let table = || data.snapshot().table("T").expect("a table").clone(); + let before = table(); + assert_eq!(Written::since(&before, &table(), &order), Written::Nothing, "nothing written"); + data.apply(Table::new("T", vec![u64::from(PAGE_ROWS) + 5]).field("v", vec![1.0])).expect("valid"); + assert_eq!(Written::since(&before, &table(), &order), Written::Pages(vec![false, true, false])); + data.apply(Table::new("T", vec![0_u64]).field("w", vec![1.0])).expect("valid"); + assert_eq!(Written::since(&before, &table(), &order), Written::Every, "a new field marks every row written"); + } + + #[test] + fn pages_keep_their_slots_as_tiles_come_and_go() { + let order = order(8 * PAGE_ROWS); + let first = Drawing::lay( + None, + &Lay { + chosen: &chosen(&order, &[0..PAGE_ROWS, 3 * PAGE_ROWS..4 * PAGE_ROWS]), + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 0, + scene: 0, + written: &Written::Nothing, + }, + false, + ); + assert_eq!(&*first.slots, [0, 3]); + assert_eq!(first.items, 2 * PAGE_ROWS); + // Page 0 leaves and page 5 arrives: page 5 takes page 0's slot, and page 3 keeps its own. + let next = Drawing::lay( + Some(&first), + &Lay { + chosen: &chosen(&order, &[3 * PAGE_ROWS..4 * PAGE_ROWS, 5 * PAGE_ROWS..6 * PAGE_ROWS]), + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 0, + scene: 0, + written: &Written::Nothing, + }, + false, + ); + assert_eq!(&*next.slots, [5, 3]); + assert_eq!(next.held_of(3 * PAGE_ROWS + 7), Some(PAGE_ROWS + 7)); + assert_eq!(next.source(7), Some(5 * PAGE_ROWS + 7)); + assert_eq!(next.item_of(PAGE_ROWS + 7), Some(7), "page 3's rows draw first"); + } + + #[test] + fn idle_pages_stay_resident_while_there_is_room() { + let order = order(8 * PAGE_ROWS); + let lay = |ranges: &[Range], room: u32, before: Option<&Drawing>| { + Drawing::lay( + before, + &Lay { + chosen: &chosen(&order, ranges), + pinned: &Pinned::default(), + ends: &Ending::default(), + room, + scene: 0, + written: &Written::Nothing, + }, + false, + ) + }; + let first = lay(&[0..PAGE_ROWS, PAGE_ROWS..2 * PAGE_ROWS], 3, None); + assert_eq!(&*first.slots, [0, 1]); + // Page 1 stops drawing and page 4 starts. With room for three pages, page 1 stays resident. + let next = lay(&[0..PAGE_ROWS, 4 * PAGE_ROWS..5 * PAGE_ROWS], 3, Some(&first)); + assert_eq!((&*next.slots, next.items), (&[0, 1, 4][..], 2 * PAGE_ROWS), "page 1 is idle: resident, not drawn"); + // Drawn again, page 1 draws from its old slot, so no slot takes a page of other rows. + let back = lay(&[0..PAGE_ROWS, PAGE_ROWS..2 * PAGE_ROWS], 3, Some(&next)); + assert_eq!(&*back.slots, [0, 1, 4]); + assert_eq!(crate::view::transition::bound(Some(&next), &back), Vec::::new()); + // With room for two, the most recently needed idle page stays. + let tight = lay(std::slice::from_ref(&(6 * PAGE_ROWS..7 * PAGE_ROWS)), 2, Some(&back)); + assert_eq!(&*tight.slots, [6, 1]); + // In another scene, idle pages are released. + let ranges = [0..PAGE_ROWS, 4 * PAGE_ROWS..5 * PAGE_ROWS]; + let changed = Lay { + chosen: &chosen(&order, &ranges), + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 3, + scene: 1, + written: &Written::Nothing, + }; + let changed = Drawing::lay(Some(&back), &changed, false); + assert_eq!((&*changed.slots, &*changed.idle), (&[0, wgsl::NONE, 4][..], &[][..])); + // A row written in an idle page releases that page and no other. + let mut pages = vec![false; 8]; + pages[4] = true; + let written = Lay { + chosen: &chosen(&order, std::slice::from_ref(&(0..PAGE_ROWS))), + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 3, + scene: 0, + written: &Written::Pages(pages), + }; + let written = Drawing::lay(Some(&back), &written, false); + assert_eq!((&*written.slots, &*written.idle), (&[0, 1][..], &[1][..])); + } + + #[test] + fn tiles_fade_in_and_out_then_settle() { + let order = order(4 * PAGE_ROWS); + let first = Drawing::lay( + None, + &Lay { + chosen: &one(&order, 0..10), + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 0, + scene: 0, + written: &Written::Nothing, + }, + true, + ); + assert!(!first.fades, "a first drawing has nothing to fade from"); + let lay = Lay { + chosen: &one(&order, PAGE_ROWS..PAGE_ROWS + 10), + pinned: &pinned(&[3]), + ends: &Ending::default(), + room: 0, + scene: 0, + written: &Written::Nothing, + }; + let next = Drawing::lay(Some(&first), &lay, true); + assert!(next.fades); + let fades: Vec<(u32, u32)> = next.runs.iter().map(|r| (r.len, r.fade)).collect(); + // Rows 0..10 fade out except pinned row 3, which draws steadily after the entering tile. + assert_eq!(fades, [(3, wgsl::FADE_OUT), (6, wgsl::FADE_OUT), (10, wgsl::FADE_IN), (1, 0)]); + assert_eq!(next.leaving_items, 9); + let settled = next.settled(&lay); + assert!(!settled.fades); + assert_eq!(settled.runs.iter().map(|r| r.fade).collect::>(), [0, 0]); + assert_eq!(settled.items, 11); + } + + #[test] + fn table_order_draws_ranges_in_the_order_chosen_and_keeps_slots_as_rows_are_appended() { + let lay = |version: Version, rows: u32, ranges: &[Range], before: Option<&Drawing>| { + let order = Order::Table { rows, order: version }; + let lay = Lay { + chosen: &chosen(&order, ranges), + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 0, + scene: 0, + written: &Written::Nothing, + }; + Drawing::lay(before, &lay, true) + }; + let version = Version::next(); + // A coarse tile's rows on page 1 draw first, under a finer tile's rows on page 0. + let first = lay(version, PAGE_ROWS + 50, &[PAGE_ROWS..PAGE_ROWS + 50, 0..10], None); + assert_eq!(&*first.slots, [0, 1]); + assert_eq!((first.item_of(PAGE_ROWS), first.item_of(0)), (Some(0), Some(50))); + // A tile arriving appends its rows, which fade in; every page keeps its slot. + let ranges = [PAGE_ROWS..PAGE_ROWS + 50, 0..10, PAGE_ROWS + 50..PAGE_ROWS + 80]; + let appended = lay(version, PAGE_ROWS + 80, &ranges, Some(&first)); + assert_eq!(&*appended.slots, [0, 1]); + assert!(appended.fades && appended.leaving.is_empty()); + assert_eq!(appended.items, 90); + // Removing rows makes another order: nothing carries over, and nothing fades. + let removed = lay(Version::next(), 30, std::slice::from_ref(&(0..30)), Some(&appended)); + assert!(!removed.fades); + assert_eq!((&*removed.slots, removed.items), (&[0][..], 30)); + } + + #[test] + fn sparse_endpoints_are_kept_individually() { + let order = order(4 * PAGE_ROWS); + // Many endpoints lie in page 3 and two in page 2: page 3 is resident whole, the two rows + // are kept individually, and none draws. + let ends = Ending::new(vec![3], vec![2 * PAGE_ROWS + 5, 2 * PAGE_ROWS + 9], Vec::new()); + let lay = Lay { + chosen: &one(&order, 0..10), + pinned: &Pinned::default(), + ends: &ends, + room: 0, + scene: 0, + written: &Written::Nothing, + }; + let drawing = Drawing::lay(None, &lay, false); + assert_eq!(&*drawing.slots, [0, 3, KEPT], "the drawn page, the endpoint page, and a page of kept rows"); + assert_eq!(drawing.items, 10, "rows resident only for endpoints are not drawn"); + assert_eq!(drawing.held_of(2 * PAGE_ROWS + 9), Some(2 * PAGE_ROWS + 1)); + assert_eq!(drawing.held_of(2 * PAGE_ROWS + 6), None, "rows no endpoint references are not resident"); + // Another endpoint is added: the rows kept before keep their positions. + let more = Ending::new(vec![3], vec![PAGE_ROWS, 2 * PAGE_ROWS + 5, 2 * PAGE_ROWS + 9], Vec::new()); + let next = Drawing::lay(Some(&drawing), &Lay { ends: &more, ..lay }, false); + assert_eq!(next.held_of(2 * PAGE_ROWS + 9), Some(2 * PAGE_ROWS + 1)); + assert_eq!(next.held_of(PAGE_ROWS), Some(2 * PAGE_ROWS + 2)); + assert_eq!(next.moved, drawing.moved, "no endpoint moved"); + // Appending rows to the order changes no kept-row input, so the kept rows stay unchanged. + let Order::Pyramid { pyramid, .. } = &order else { unreachable!("a pyramid") }; + let grown = Order::Pyramid { pyramid: pyramid.clone(), rows: 4 * PAGE_ROWS + 1 }; + let appended = Drawing::lay(Some(&next), &Lay { chosen: &one(&grown, 0..10), ends: &more, ..lay }, false); + assert!(Arc::ptr_eq(&appended.kept, &next.kept), "the kept rows are unchanged"); + assert_eq!((&appended.kept_table, &appended.moved), (&next.kept_table, &next.moved)); + // Page 2 is drawn: its individually kept rows move into it, and the edges ending there + // resolve their endpoints again. The kept row of page 1 stays put. + let drawn = Lay { chosen: &chosen(&order, &[0..10, 2 * PAGE_ROWS..2 * PAGE_ROWS + 10]), ends: &more, ..lay }; + let paged = Drawing::lay(Some(&next), &drawn, false); + assert_eq!(paged.held_of(2 * PAGE_ROWS + 9), Some(paged.pages[2] * PAGE_ROWS + 9)); + assert_eq!(paged.held_of(PAGE_ROWS), next.held_of(PAGE_ROWS)); + assert_ne!(paged.moved.0, next.moved.0); + assert_eq!(&*paged.moved.1, [2 * PAGE_ROWS + 5, 2 * PAGE_ROWS + 9], "page 2's kept rows are listed as moved"); + } + + #[test] + fn edge_pages_ending_at_moved_rows_resolve_their_endpoints_again() { + let order = order(3 * PAGE_ROWS); + let whole = one(&order, 0..3 * PAGE_ROWS); + let lay = Lay { + chosen: &whole, + pinned: &Pinned::default(), + ends: &Ending::default(), + room: 0, + scene: 0, + written: &Written::Nothing, + }; + let edges = Drawing::lay(None, &lay, false); + let at = |places: &[u32]| Some((PAGE_ROWS, Arc::new(Places { places: places.to_vec(), runs: Vec::new() }))); + let pages = vec![at(&[3, 9]), at(&[PAGE_ROWS + 1]), at(&[7, 2 * PAGE_ROWS])]; + let drawn = Drawn { ends: Ends { pages, ..Ends::default() }, ..Drawn::default() }; + assert_eq!(drawn.naming(&edges, &[9]), [0], "the page of edges ending at place 9"); + assert_eq!(drawn.naming(&edges, &[7, PAGE_ROWS + 1]), [1, 2]); + assert!(drawn.naming(&edges, &[5]).is_empty(), "no resident edge ends at place 5"); + } + + #[test] + fn kept_rows_fill_pages_of_their_own_and_endpoint_pages_are_resident() { + let order = order(4 * PAGE_ROWS); + let lay = Lay { + chosen: &one(&order, 0..10), + pinned: &pinned(&[2 * PAGE_ROWS + 1]), + ends: &Ending::new(vec![3], Vec::new(), Vec::new()), + room: 0, + scene: 0, + written: &Written::Nothing, + }; + let drawing = Drawing::lay(None, &lay, false); + assert_eq!(&*drawing.slots, [0, 3, KEPT], "the drawn page, the endpoint page, and a page of kept rows"); + assert_eq!(drawing.items, 11, "endpoint pages are resident, not drawn"); + let held = drawing.held_of(2 * PAGE_ROWS + 1).expect("kept"); + assert_eq!((held, drawing.source(held)), (2 * PAGE_ROWS, Some(2 * PAGE_ROWS + 1))); + assert_eq!(drawing.held_pages().collect::>(), [0, 3]); + // Another pinned row takes the next kept position; the first keeps its own. + let more = Lay { pinned: &pinned(&[PAGE_ROWS + 7, 2 * PAGE_ROWS + 1]), ..lay }; + let next = Drawing::lay(Some(&drawing), &more, false); + assert_eq!( + (next.held_of(2 * PAGE_ROWS + 1), next.held_of(PAGE_ROWS + 7)), + (Some(2 * PAGE_ROWS), Some(2 * PAGE_ROWS + 1)) + ); + assert_eq!(crate::view::transition::bound(Some(&drawing), &next), [2], "only the kept-row slot changed"); + // A kept row whose page starts drawing draws from that page. + let drawn = Lay { chosen: &chosen(&order, &[0..10, 2 * PAGE_ROWS..2 * PAGE_ROWS + 10]), ..more }; + let paged = Drawing::lay(Some(&next), &drawn, false); + assert_eq!(paged.held_of(2 * PAGE_ROWS + 1), Some(paged.pages[2] * PAGE_ROWS + 1)); + assert_eq!(paged.held_of(PAGE_ROWS + 7), Some(2 * PAGE_ROWS + 1), "the other kept row stays put"); + } +} diff --git a/crates/latkit/src/view/ends.rs b/crates/latkit/src/view/ends.rs new file mode 100644 index 00000000..ee1a1664 --- /dev/null +++ b/crates/latkit/src/view/ends.rs @@ -0,0 +1,246 @@ +//! The end table: where each endpoint of an edge or face row is held on the GPU. +//! +//! A row's ends reference places in the vertex table's order, which do not change as the camera +//! moves. The held row of a place does change: the vertex mark's drawing assigns resident pages to +//! slots and keeps sparse endpoints individually. So no channel stores held rows. Instead an +//! expression-machine step (`lib/ends.wesl`) resolves them and writes each into the row's kind +//! word for that end, where draws, probes, and normals read it directly. The step runs for rows +//! whose ends changed, for rows that reference vertices the drawing moved, and for every row when +//! another mark or another order holds the vertices. + +use std::sync::Arc; + +use super::View; +use super::kept::KeptTable; +use super::marks::{Blocks, Shown}; +use super::passes::{dispatch_blocks, offset}; +use super::resources::step; +use super::words::Kept; +use crate::data::Version; +use crate::expr::compile::EndUse; +use crate::gpu::Inner; +use crate::gpu::pipelines::Variant; +use crate::gpu::shaders as wgsl; +use crate::gpu::work::Count; + +/// Where the vertices that one end of a mark's rows references are held: the vertex mark drawing +/// their table (`mark`), its state pages and words per row (`state`, `stride`), and its pages of +/// the normals faces sum into (`normals`). For a paged vertex table, also its order's id, each +/// page's slot (`pages`), the kept rows by order place (`kept`), and the places its drawing moved. +#[derive(Clone, PartialEq, Debug)] +pub(crate) struct End { + pub(crate) mark: Option, + pub(crate) state: Arc<[u32]>, + pub(crate) stride: u32, + pub(crate) normals: Option>, + pub(crate) order: u64, + pub(crate) pages: Option>, + pub(crate) kept: Option, + pub(crate) moved: Option<(Version, Arc<[u32]>)>, +} + +impl End { + /// An end whose table no vertex mark draws. + pub(crate) fn unattached() -> End { + End { + mark: None, + state: Arc::new([]), + stride: 0, + normals: None, + order: 0, + pages: None, + kept: None, + moved: None, + } + } + + /// Whether every referenced vertex may be held elsewhere than in `before`: another mark or + /// another order holds them. If only the vertex mark's word pages or its drawing's moved places + /// changed, the other vertices stay where they were. + fn moved_whole(&self, before: &End) -> bool { + self.mark != before.mark || self.order != before.order + } + + /// The places its drawing moved since `before` was attached. Empty if none moved. + fn moved_since(&self, before: &End) -> &[u32] { + match (&self.moved, &before.moved) { + (Some((version, places)), Some((was, _))) if version != was => places, + _ => &[], + } + } + + /// Words its tables take in a mark's program beyond its `END_WORDS`: the state page table, the + /// page slots and kept-row table, and the normals page table. + fn tables(&self) -> usize { + let kept = |k: &KeptTable| 2 * k.buckets().len(); + self.state.len() + + self.pages.as_ref().map_or(0, |p| 2 + p.len() + self.kept.as_ref().map_or(0, kept)) + + self.normals.as_ref().map_or(0, |n| n.len()) + } +} + +/// An earlier end of `ends` equal to end `c`, whose tables `c` shares, as both ends of an edge +/// between vertices of one mark do. +fn shared(ends: &[End], c: usize) -> Option { + ends[..c].iter().position(|e| *e == ends[c]) +} + +/// Words a mark's `ends` take in its program. +pub(super) fn words(ends: &[End]) -> usize { + (0..ends.len()) + .map(|c| wgsl::END_WORDS as usize + if shared(ends, c).is_some() { 0 } else { ends[c].tables() }) + .sum() +} + +/// Appends the words of a mark's `ends`, whose columns `uses` gives, starting at word `at` of the +/// program buffer. First `END_WORDS` per end, then each end's tables: the state page table of its +/// vertex mark; the page count, then each page's slot; the kept-row bucket count, then each +/// bucket's place and held row; and the normals page table. Equal ends share tables. +pub(super) fn pack(ends: &[End], uses: &[EndUse], at: u32, out: &mut Vec) { + let mut heads: Vec<[u32; wgsl::END_WORDS as usize]> = Vec::with_capacity(ends.len()); + let mut table = at + wgsl::END_WORDS * ends.len() as u32; + let mut take = |len: usize| { + let at = table; + table += len as u32; + at + }; + for (c, (end, used)) in ends.iter().zip(uses).enumerate() { + let mut head = if let Some(earlier) = shared(ends, c) { + heads[earlier] + } else { + let state = if end.mark.is_some() { take(end.state.len()) } else { wgsl::NONE }; + let kept = end.kept.as_ref().map_or(0, |k| 2 * k.buckets().len()); + let slots = end.pages.as_ref().map_or(wgsl::NONE, |p| take(2 + p.len() + kept)); + let normals = end.normals.as_ref().map_or(wgsl::NONE, |n| take(n.len())); + [state, end.stride, slots, normals, wgsl::NONE] + }; + head[wgsl::END_COLUMN as usize] = used.column; + heads.push(head); + out.extend(head); + } + for (c, end) in ends.iter().enumerate() { + if shared(ends, c).is_some() { + continue; + } + if end.mark.is_some() { + out.extend(end.state.iter()); + } + if let Some(pages) = &end.pages { + out.push(pages.len() as u32); + out.extend(pages.iter()); + let kept = end.kept.as_ref().map_or(&[][..], KeptTable::buckets); + out.push(kept.len() as u32); + out.extend(kept.iter().flatten()); + } + if let Some(normals) = &end.normals { + out.extend(normals.iter()); + } + } +} + +impl View { + /// Points every edge and face mark's ends at the vertex marks drawing their tables. If another + /// mark or order now holds an end's vertices, every row resolves its endpoints again. If the + /// vertex drawing moved some vertices, only the rows that reference them do. If only the vertex + /// mark's word pages moved, the ends read the new pages and nothing is resolved. + pub(super) fn attach_ends(&mut self) { + for i in 0..self.marks.len() { + let ends = ends(&self.marks, i); + let was = &self.marks[i].ends; + if ends == *was { + continue; + } + let whole = ends.len() != was.len() || ends.iter().zip(was).any(|(end, before)| end.moved_whole(before)); + let mut places: Vec = + ends.iter().zip(was).flat_map(|(end, before)| end.moved_since(before).iter().copied()).collect(); + places.sort_unstable(); + places.dedup(); + let blocks = match (whole, places.is_empty()) { + (true, _) => None, + (false, true) => Some(Vec::new()), + (false, false) => self.naming(i, &places), + }; + let shown = &mut self.marks[i]; + match blocks { + Some(blocks) => shown.unresolved.add(Some(&blocks)), + None => shown.unresolved = Blocks::All, + } + shown.ends = ends; + shown.packed = false; + } + } + + /// The row blocks of mark `i` whose rows end at any of `places` in the attached vertex table, + /// ascending: the slots of its drawing whose endpoints include them. `None` for a fully drawn + /// table, which keeps no per-page endpoints, so every block resolves. + fn naming(&self, i: usize, places: &[u32]) -> Option> { + let shown = &self.marks[i]; + let style = shown.mark.style(); + let drawn = self.drawings.get(&(style.data.clone(), style.table.clone()))?; + Some(drawn.naming(shown.drawing()?, places)) + } + + /// Resolves the held rows of the ends of `marks`' unresolved rows, over the row blocks each + /// lists, by running the end step on the pipeline each mark's channels use. `group` is the + /// expression machine's step group, left set to the channel step. + pub(super) fn resolve_ends( + &mut self, + pass: &mut wgpu::ComputePass<'_>, + inner: &mut Inner, + mark: &wgpu::BindGroup, + group: &wgpu::BindGroup, + marks: &[usize], + ) { + if marks.is_empty() { + return; + } + let device = &self.gpu.0.device; + pass.set_bind_group(2, group, &[step(wgsl::STEP_ENDS)]); + for &i in marks { + let shown = &mut self.marks[i]; + let rows = shown.live(); + pass.set_pipeline(&inner.pipelines.compute(device, Variant::channels(shown.reads()))); + pass.set_bind_group(1, mark, &[offset(i)]); + dispatch_blocks(pass, rows, shown.end_block_count); + let blocks = shown.unresolved.of(rows); + let resolved = blocks.iter().map(|&b| u64::from((rows - b * wgsl::PAGE_ROWS).min(wgsl::PAGE_ROWS))); + self.gpu.0.counters.add(Count::Resolved, resolved.sum()); + shown.unresolved = Blocks::default(); + // Resolving writes its state and can move where its rows stand, which places labels. + shown.version += 1; + shown.place_version += 1; + } + pass.set_bind_group(2, group, &[step(wgsl::STEP_CHANNELS)]); + } +} + +/// Where each of mark `i`'s ends finds its vertices: the vertex mark it attaches to, that mark's +/// state and normals pages, and for a paged table, its drawing's page slots and kept rows. An end +/// attaches to the unnamed vertex mark drawing its table, otherwise the first one, but only while +/// that mark has word pages for all its held rows. +fn ends(marks: &[Shown], i: usize) -> Vec { + let Some(compiled) = &marks[i].compiled else { return Vec::new() }; + let data = &marks[i].mark.style().data; + compiled + .program + .ends + .iter() + .map(|end| match super::graph::end_mark(marks, data, &end.table) { + // A vertex mark without word pages for its rows draws nothing. Nothing attaches to it. + Some(m) if marks[m].pages_held() => { + let (state, normals) = (marks[m].words.pages(Kept::State), marks[m].words.pages(Kept::Normals)); + End { + mark: Some(m), + state: state.blocks.clone(), + stride: state.width, + normals: (normals.width > 0).then(|| normals.blocks.clone()), + order: marks[m].drawing().map_or(0, |d| d.order.id()), + pages: marks[m].drawing().map(|d| d.pages.clone()), + kept: marks[m].drawing().map(|d| d.kept_table.clone()), + moved: marks[m].drawing().map(|d| d.moved.clone()), + } + } + _ => End::unattached(), + }) + .collect() +} diff --git a/crates/latkit/src/view/event.rs b/crates/latkit/src/view/event.rs index f3598372..85e47be4 100644 --- a/crates/latkit/src/view/event.rs +++ b/crates/latkit/src/view/event.rs @@ -1,8 +1,10 @@ -//! What a view tells the application. An intent asks something of it: a click, an open, a menu, a -//! drag, a connection, a deletion, each told once, in the order the user acted. A change tells that -//! state the view keeps moved: hover, selection, camera, what a frame waits on. Only the latest change -//! of each is told, and the state can always be read, so a listener added late reads what it missed. -//! A view tells only what the application did not do itself: its setters change state silently. +//! Events a view reports to the application. +//! +//! An intent asks the application to act: a click, open, menu, drag, connection, or deletion. Each +//! is reported once, in the order the user acted. A change reports that view state changed and is +//! named after that state: `hovered`, `selection`, `camera`, `waiting`. Only the latest change of +//! each is reported, and the state can always be read, so a late listener can catch up. A view +//! reports only what the application did not do itself. Its setters change state silently. use std::collections::VecDeque; @@ -10,8 +12,7 @@ use super::Camera; use super::input::{Device, Modifiers}; use crate::{Error, Name, Pending, Row, Rows}; -/// What happened in a view, for the application to act on. Drain them with -/// [`View::events`](crate::View::events). +/// Something that happened in a view, as [`View::events`](crate::View::events) reports it. #[derive(Clone, PartialEq, Debug, serde::Serialize, schemars::JsonSchema)] #[serde(tag = "type", content = "detail", rename_all = "lowercase")] #[schemars(rename = "ViewEvent")] @@ -24,93 +25,95 @@ pub enum Event { /// How the user clicked it. trigger: Trigger, }, - /// A row was double clicked, or Enter pressed on the row the keys point at. + /// A row was double-clicked, or Enter was pressed on the row the keyboard points at. Open { /// The row. hit: Hit, /// How the user opened it. trigger: Trigger, }, - /// The secondary button clicked, or the menu key pressed: what lies under the trigger's point, - /// for a menu of what to do with it. + /// A context menu was requested: by a secondary click, the menu key, or a touch or pen held + /// still. Lists the rows under the trigger's point, for a menu of actions on them. Menu { - /// What lies under the point, nearest first. + /// The rows under the point, nearest first. hits: Vec, /// How the user asked. trigger: Trigger, }, - /// A row of a draggable mark follows the pointer: the drag starts once the pointer moves a few + /// A row of a draggable mark is dragged. The drag starts once the pointer moves more than 4 CSS /// pixels from where it pressed the row. Drag { /// Whether the drag starts, moves, or ends. phase: Phase, /// The row dragged, as it was hit. hit: Hit, - /// Where the row goes, in the space's coordinates: where it was hit, moved as the pointer moved - /// since the press, at the depth the drag started; a vertex keeps where it was grabbed. On a - /// diagram's dots when its grid snaps. + /// The row's new position in the space's coordinates. It is the hit point moved with the + /// pointer since the press, at the depth where the drag started, so a vertex keeps its + /// offset from the pointer. Snapped to a diagram's grid dots when the grid snaps. position: [f64; 3], /// The pointer. trigger: Trigger, }, - /// A connection was drawn with [`Controls::connect`](crate::Controls): from a port, a vertex - /// with Alt held, or an edge whose end was dragged off it. + /// A connection was drawn with [`Controls::connect`](crate::Controls) on: from a port, from a + /// vertex with Alt held, or by dragging an edge's end off it. Connect { - /// The end it was drawn from: for an edge, the end that stayed. + /// The end it was drawn from. For a dragged edge end, the edge's other end. from: End, /// The vertex it was released over, if any. to: Option, - /// The edge whose other end was dragged, which `to` replaces. + /// The edge whose end was dragged off. `to` replaces that end. edge: Option, /// Where it was released. trigger: Trigger, }, - /// Delete or Backspace with rows selected and [`Controls::delete`](crate::Controls) on. + /// Delete or Backspace was pressed with rows selected and [`Controls::delete`](crate::Controls) + /// on. Delete { /// The rows selected. rows: Vec, /// The key. trigger: Trigger, }, - /// What the user points at changed: [`View::hovered`](crate::View::hovered). - Hover { - /// The row now pointed at; `None` once it left every row. + /// The hovered row changed: [`View::hovered`](crate::View::hovered). + Hovered { + /// The row now hovered, or `None` if no row is. hit: Option, /// The pointer, or the keys. trigger: Trigger, }, /// The selection changed: [`View::selection`](crate::View::selection). - Select { + Selection { /// The rows selected now. rows: Vec, - /// How the user changed it; `None` when the data dropped rows from it. + /// How the user changed it, or `None` when the data dropped rows from it. trigger: Option, }, - /// The camera moved by a gesture, a fit, a flight, coasting, or an orbit: - /// [`View::camera`](crate::View::camera). + /// The camera moved: [`View::camera`](crate::View::camera). A gesture, fit, flight, coast, or + /// orbit moved it, or it moved back within its zoom and pan limits after the view's size, + /// insets, or space changed. Camera(Camera), - /// What the last frame stood in for changed; nothing once it is complete: - /// [`View::complete`](crate::View::complete). + /// The work the last frame waited on changed: [`View::waiting`](crate::View::waiting). Empty + /// once frames are complete. Waiting(Waiting), - /// Something failed, told once. A mark its data no longer fits also lists in + /// Something failed. Reported once. A mark that no longer fits its data is also listed in /// [`View::failures`](crate::View::failures). Error(Error), } -/// Where, with what keys, and with what the user acted. For keys, the point is the row they point -/// at, else the pointer, else the view's middle. +/// Where the user acted, with which modifier keys and device. For keyboard input, the point is the +/// row the keys point at, or else the pointer, or else the view's middle. #[derive(Clone, Copy, PartialEq, Debug, Default, serde::Serialize, schemars::JsonSchema)] pub struct Trigger { /// CSS pixels from the top left. pub point: [f32; 2], - /// The keys held. + /// The modifier keys held. pub modifiers: Modifiers, - /// What acted. + /// The device used. pub device: Device, } impl Event { - /// Whether it asks something of the application, so keeps its place among intents. + /// Whether it is an intent, which keeps its order among intents. fn intent(&self) -> bool { matches!( self, @@ -123,13 +126,13 @@ impl Event { ) } - /// Whether it leaves `older`, still untold, nothing to say: a change replaces the change of the - /// same state, a drag's move the move before it, an error the same error. + /// Whether it makes the unreported `older` redundant: a change replaces an earlier change of + /// the same state, a drag move the previous move, and an error an identical error. fn replaces(&self, older: &Event) -> bool { match (self, older) { (Event::Drag { phase: Phase::Move, .. }, Event::Drag { phase: Phase::Move, .. }) => true, (Event::Error(error), Event::Error(old)) => error == old, - (Event::Hover { .. } | Event::Select { .. } | Event::Camera(_) | Event::Waiting(_), _) => { + (Event::Hovered { .. } | Event::Selection { .. } | Event::Camera(_) | Event::Waiting(_), _) => { std::mem::discriminant(self) == std::mem::discriminant(older) } _ => false, @@ -137,19 +140,19 @@ impl Event { } } -/// What a view has yet to tell, in the order it happened. An intent waiting on a pick holds its -/// place, and the intents after it wait behind it; changes and errors never wait. +/// The events a view has yet to report, in order. An intent waiting on a pick keeps its place, and +/// later intents wait behind it. Changes and errors never wait. #[derive(Default)] pub(crate) struct Events(VecDeque); enum Slot { Ready(Event), Picking(Pending>, Awaited), - /// A pick that made nothing, dropped once settled. + /// A pick that produced no event, removed once settled. Gone, } -/// What an intent waits on a pick for. +/// An intent waiting for its pick's hits. #[derive(Clone)] pub(crate) enum Awaited { Menu(Trigger), @@ -158,8 +161,8 @@ pub(crate) enum Awaited { edge: Option, trigger: Trigger, }, - /// A click again where nothing moved: the selection moves to the row under the point after - /// `after`. + /// A repeated click where nothing moved: the selection moves to the next row under the point + /// after `after`. Cycle { after: Option, trigger: Trigger, @@ -167,18 +170,19 @@ pub(crate) enum Awaited { } impl Events { - /// Queues `event`, dropping what it leaves nothing to say. + /// Queues `event`, dropping the unreported events it makes redundant. pub(crate) fn tell(&mut self, event: Event) { self.0.retain(|slot| !matches!(slot, Slot::Ready(older) if event.replaces(older))); self.0.push_back(Slot::Ready(event)); } - /// Queues the intent `awaited` makes once `hits` answers. + /// Queues the intent `awaited` produces once `hits` resolves. pub(crate) fn tell_after(&mut self, hits: Pending>, awaited: Awaited) { self.0.push_back(Slot::Picking(hits, awaited)); } - /// Makes, in place, each event whose pick answered; one that makes nothing leaves no place. + /// Turns each intent whose pick resolved into its event, in place. One that produces no event + /// is removed. pub(crate) fn settle(&mut self, mut make: impl FnMut(Vec, Awaited) -> Option) { let mut made = false; for slot in &mut self.0 { @@ -197,7 +201,7 @@ impl Events { } } - /// The first error untold: a shot fails on it, and it stays for the view's owner. + /// The first unreported error. A shot fails on it and leaves it queued for the view's owner. pub(crate) fn error(&self) -> Option<&Error> { self.0.iter().find_map(|slot| match slot { Slot::Ready(Event::Error(error)) => Some(error), @@ -205,7 +209,8 @@ impl Events { }) } - /// Takes what can be told: all but the intents behind one still waiting on its pick. + /// Takes every event ready to report, except intents queued behind one still waiting on its + /// pick. pub(crate) fn drain(&mut self) -> std::vec::IntoIter { let (mut told, mut held) = (Vec::new(), VecDeque::new()); for slot in self.0.drain(..) { @@ -225,7 +230,7 @@ impl Extend for Events { } } -/// A vertex a connection ends at, and its port when it has the one the connection meets. +/// A vertex a connection ends at, and the port the connection meets, if any. #[derive(Clone, PartialEq, Debug, serde::Serialize, schemars::JsonSchema)] pub struct End { /// The vertex. @@ -240,23 +245,24 @@ pub struct End { pub struct Hit { /// The row. pub row: Row, - /// The name of the mark that drew it, when it has one: what tells two marks of one table apart. + /// The name of the mark that drew it, if the mark has one. It tells apart two marks of one + /// table. #[serde(skip_serializing_if = "Option::is_none")] pub mark: Option, /// The index of the mark in the scene the frame drew. #[serde(skip)] pub(crate) index: usize, - /// The point of the row nearest the probe, CSS pixels from the top left: a vertex's or a port's - /// middle, the nearest point of a stroke or a triangle. + /// The row's point nearest the probe, in CSS pixels from the top left: a vertex's or port's + /// middle, or the nearest point of a stroke or triangle. pub point: [f32; 2], /// That point in the space's coordinates: x, y, and elevation. pub position: [f64; 3], /// CSS pixels from the probe to the row. pub distance: f32, - /// The port hit, when a vertex's port was. + /// The port hit, if the hit is on a vertex's port. #[serde(skip_serializing_if = "Option::is_none")] pub port: Option, - /// The sample hit, when a line tracing a sampled field was. + /// The sample hit, if the hit is on a line tracing a sampled field. #[serde(skip_serializing_if = "Option::is_none")] pub sample: Option, } @@ -282,13 +288,13 @@ pub enum Phase { Start, /// The pointer moved. Move, - /// The pointer let go, or the drag was cancelled. + /// The pointer was released, or the drag was cancelled. End, } /// What [`View::fit`](crate::View::fit) frames. #[derive(Clone, PartialEq, Debug, serde::Deserialize, schemars::JsonSchema)] -#[schemars(description = "What a view's fit frames.")] +#[schemars(description = "The target of a view's fit.")] #[serde(rename_all = "camelCase")] pub enum Fit { /// Everything drawn. @@ -299,14 +305,14 @@ pub enum Fit { Reveal(Row), /// The box from `min` to `max`: x, y, and elevation. Bounds { - /// The least corner. + /// The minimum corner. min: [f64; 3], - /// The greatest corner. + /// The maximum corner. max: [f64; 3], }, } -/// The pointer a view asks for over what it shows. +/// The cursor a view wants shown over it. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default)] pub enum Cursor { /// The platform's default. @@ -314,15 +320,15 @@ pub enum Cursor { Default, /// Over a row a click selects. Pointer, - /// Over a row that drags. + /// Over a draggable row. Grab, /// While dragging a row or the camera. Grabbing, - /// While selecting a box. + /// While selecting a box or drawing a connection. Crosshair, } -/// What a view does each frame. +/// Counters describing a view's frames. #[derive(Clone, Copy, PartialEq, Debug, Default, serde::Serialize, schemars::JsonSchema)] #[serde(rename_all = "camelCase")] pub struct Stats { @@ -330,48 +336,40 @@ pub struct Stats { pub frames: u64, /// Rows of every mark's table. pub rows: u64, - /// Rows every mark draws: every row of a table drawn whole, the rows of the tiles in view of one - /// drawn through tiles. + /// Rows the marks draw: every row of a fully drawn table, and the rows in the tiles in view of + /// a paged table. pub drawn: u64, - /// Draw calls the last frame recorded, shadows and labels with the marks. + /// Draw calls the last frame recorded, label draws included. pub draws: u64, - /// Instances the last frame's draws drew: points, segments, triangles, glyphs; at most, for - /// segments the GPU culls. + /// Instances the last frame's draws drew: points, segments, triangles, and glyphs. An upper + /// bound for segments, which the GPU may cull. pub instances: u64, - /// Fetched tiles the view draws, of every tile source. + /// Fetched tiles the view draws, across all tile sources. pub tiles: u64, - /// Tiles being fetched or decoded, of every tile source. + /// Tiles being fetched or decoded, across all tile sources. pub tiles_loading: u64, - /// Bytes of data waiting to upload. - pub uploading: u64, - /// Bytes of data pages on the GPU, for every view of its `Gpu`. - pub resident: u64, - /// Bytes the GPU's [`Budget`](crate::Budget) lets data take. - pub budget: u64, - /// The work under way the last frame stood in for; nothing when it drew everything as the data, - /// scene, and camera say: see [`View::complete`](crate::View::complete). - pub waiting: Waiting, /// Milliseconds the last frame took to record on the CPU. pub cpu_ms: f32, - /// Milliseconds the GPU takes on a frame: the median of the last frames it measured; 0 where the - /// device cannot measure. + /// Milliseconds the GPU takes per frame: the median of recently measured frames. 0 if the + /// device cannot measure GPU time. pub gpu_ms: f32, } -/// The work under way a frame stood in for, by kind; nothing when the frame was complete. +/// The work a frame was still waiting on, by kind. Empty when the frame was complete. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default)] pub struct Waiting { - /// Pipelines the scene draws with still compiling: until they land the view draws nothing new. + /// Pipelines the view uses are still compiling. Until its draw pipelines finish, the view + /// draws nothing new. pub compiling: bool, - /// Pages its marks draw still uploading, or tables not yet built from them. + /// Pages its marks draw are still uploading, or tables are not yet built from them. pub data: bool, - /// Layouts, routes, lanes, shapes, tiles, or pyramids under way. + /// Layouts, routes, lanes, shapes, tiles, or pyramids in progress. pub derived: bool, - /// A label placement being read back, or texts asked for and not yet set. + /// The label texts in view being read back, or texts requested and not yet set. pub labels: bool, /// A fit due or being measured. pub fit: bool, - /// Row regions laid out again, segment counts being read back, or a transition beginning. + /// Segment counts being read back, or a transition beginning. pub marks: bool, } @@ -381,7 +379,7 @@ impl Waiting { *self == Waiting::default() } - /// The names of what it waits on, as its fields are named. + /// The names of the fields that are set. pub fn names(&self) -> impl Iterator { let Waiting { compiling, data, derived, labels, fit, marks } = *self; [ @@ -403,7 +401,7 @@ impl std::fmt::Display for Waiting { } } -/// The names of what it waits on, as [`Waiting::names`] gives them. +/// Serializes as the list of names [`Waiting::names`] gives. impl serde::Serialize for Waiting { fn serialize(&self, serializer: S) -> Result { serializer.collect_seq(self.names()) @@ -417,7 +415,7 @@ impl schemars::JsonSchema for Waiting { fn json_schema(_: &mut schemars::SchemaGenerator) -> schemars::Schema { schemars::json_schema!({ - "description": "Work under way a frame stands in for: pipelines the scene draws with compiling, until which the view draws nothing new (`compiling`); pages uploading or tables not yet built from them (`data`); layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); a label placement or texts not yet set (`labels`); a fit due or being measured (`fit`); row regions laid out again, segment counts, or a transition beginning (`marks`). Empty once a frame is complete.", + "description": "What a frame still waits on: pipelines the scene draws with, which compile before the view draws anything new (`compiling`); pages uploading, or tables not yet built from them (`data`); layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); label texts being read back, or not yet set (`labels`); a fit due or being measured (`fit`); segment counts, or a transition starting (`marks`). Empty once the frame is complete.", "type": "array", "items": { "enum": ["compiling", "data", "derived", "labels", "fit", "marks"] }, }) @@ -469,7 +467,7 @@ mod tests { } #[test] - fn a_change_replaces_the_one_still_untold() { + fn a_change_replaces_the_undelivered_one() { let mut events = Events::default(); let mut waiting = Waiting { data: true, ..Waiting::default() }; events.tell(Event::Waiting(waiting)); diff --git a/crates/latkit/src/view/fit.rs b/crates/latkit/src/view/fit.rs index 12ebfe54..7839cf02 100644 --- a/crates/latkit/src/view/fit.rs +++ b/crates/latkit/src/view/fit.rs @@ -1,13 +1,16 @@ -//! Framing what is drawn. A box frames at once, and so does what the CPU already knows: a diagram's -//! sheet holds every box and wire, and tiles hold the bounds of every row they draw. Anything else is -//! measured on the GPU, read back, and framed a frame or two later. A plot without a value range -//! measures its traces whenever they or the camera change, and grows its range with headroom so the -//! axes stay still. +//! Fits: moving the camera to frame what is drawn. +//! +//! A box fits at once, as does anything whose extent the CPU already knows. A diagram's sheet has +//! every box and wire, and tiles have the bounds of every row they draw. Anything else is measured +//! on the GPU, read back, and framed a frame or two later. A plot without a value range measures +//! its traces whenever they or the camera change. It grows its range with headroom so the axes +//! stay still. use super::View; use super::camera::Bounds; use super::event::Fit; use super::marks::Shown; +use crate::derive::Working; use crate::gpu::readback::Readback; use crate::gpu::shaders as wgsl; use crate::scene::mark::MarkKind; @@ -17,8 +20,8 @@ use crate::{Camera, Space}; pub(crate) struct Fitting { /// A fit waiting for its frame. pub(crate) wanted: Option, - /// Extents in flight, and the fit they were measured for. - reading: Option<(Readback, Option)>, + /// The extents being read back, and the fit they are measured for. + reading: Working, Readback>, /// Whether a plot's traces changed since they were last measured. pub(crate) plot_due: bool, } @@ -26,17 +29,25 @@ pub(crate) struct Fitting { impl Fitting { /// Whether extents are being read back. pub(crate) fn reading(&self) -> bool { - self.reading.is_some() + self.reading.busy() } - /// Whether a fit waits for a frame to measure it: none while extents are being read back. + /// Whether a fit is waiting to be measured. Always false while extents are being read back. pub(crate) fn due(&self) -> bool { - self.wanted.is_some() && self.reading.is_none() + self.wanted.is_some() && !self.reading.busy() } /// Whether a fit is about to move the camera: one waiting, or its extents being read back. pub(crate) fn framing(&self) -> bool { - self.wanted.is_some() || self.reading.as_ref().is_some_and(|(_, fit)| fit.is_some()) + self.wanted.is_some() || self.reading.context().is_some_and(Option::is_some) + } + + /// Drops the fit about to move the camera, keeping a plot's measure of its traces. + fn cancel(&mut self) { + self.wanted = None; + if self.reading.context().is_some_and(Option::is_some) { + self.reading.cancel(); + } } } @@ -47,7 +58,7 @@ pub(crate) struct Measure { } impl View { - /// Applies a box fit, a plot's fit, or measured extents that landed. + /// Applies a box fit, a plot's fit to its traces' span, or measured extents that arrived. pub(super) fn settle_fit(&mut self) { if self.size[0] <= 0.0 { return; @@ -66,16 +77,20 @@ impl View { } _ => {} } - let Some((readback, _)) = &self.fitting.reading else { return }; - let Some(bytes) = readback.take() else { return }; - let fit = self.fitting.reading.take().and_then(|(_, fit)| fit); + let Some((_, fit, bytes)) = self.fitting.reading.landed() else { return }; if let Ok(bytes) = bytes { - self.measured(bytemuck::cast_slice(&bytes), fit.as_ref()); + self.measured(&crate::wire::words(&bytes), fit.as_ref()); } self.fitted(); } - /// Lets go of the rows marked for a fit, once it framed them. + /// Drops a fit not yet applied, as a camera set or flown afterwards replaces it. + pub(super) fn cancel_fit(&mut self) { + self.fitting.cancel(); + self.fitted(); + } + + /// Clears the rows marked for a fit once it framed them. fn fitted(&mut self) { if !self.focus.fitting.is_empty() { self.focus.fitting.clear(); @@ -85,31 +100,33 @@ impl View { /// What this frame measures, if anything: a fit whose marks are drawn, or a plot's traces. pub(super) fn measuring(&mut self) -> Option { - if self.fitting.reading.is_some() || self.size[0] <= 0.0 { + if self.fitting.reading.busy() || self.size[0] <= 0.0 { return None; } - // A plot that traces frames everything by its traces' whole span, which settles once their - // data is drawn; measuring them first would frame only what shows up to the coordinate shown. + // A plot with traces fits everything to their whole span once their data is available. + // Measuring instead would frame only what is drawn up to the coordinate shown. let plot = matches!(self.scene.space, Space::Plot(_)); if plot && matches!(self.fitting.wanted, Some(Fit::All)) && self.traces() { return None; } - // A fit frames what shows once data has streamed, its shapes are triangulated and bent, and any - // layout has placed it; in a diagram, once vertices stand where their sheet puts them. + // A fit waits until data has streamed, shapes are triangulated and bent, and any layout has + // placed the rows. In a diagram it also waits until vertices reach where the sheet puts + // them. let easing = matches!(self.scene.space, Space::Diagram(_)) && self .marks .iter() .any(|m| m.kind() == MarkKind::Vertices && (m.transition.is_some() || m.begin.is_some())); let all = matches!(self.fitting.wanted, Some(Fit::All)); - // Everything a table drawn through tiles holds frames by its rows' bounds, whatever is drawn. A - // mark with no rows has nothing to frame: it holds a fit only while no mark has rows. + // Fitting everything uses a tiled table's row bounds, regardless of what is drawn. A mark + // with no rows has nothing to frame. It only delays a fit while no mark has rows. let compiled = || self.marks.iter().filter(|m| m.compiled.is_some()); - let drawn = |m: &Shown| (m.drawable() && !m.tables.waiting()) || (all && self.tiled_bounds(m).is_some()); - let empty = |m: &Shown| m.tables.synced() && m.rows + m.exits == 0; + let staged = |m: &Shown| self.staged.get(&m.mark.style().data).is_some_and(|s| !s.is_empty()); + let drawn = |m: &Shown| (m.drawable() && !staged(m)) || (all && self.tiled_bounds(m).is_some()); + let empty = |m: &Shown| m.tables.synced() && m.held() == 0; let ready = compiled().any(&drawn) && compiled().all(|m| drawn(m) || empty(m)) - && !self.derived.sheet.placing() + && !self.derived.placing() && !self.derived.shaping() && !easing; if let Some(fit) = self.fitting.wanted.clone().filter(|f| matches!(f, Fit::All | Fit::Rows(_) | Fit::Reveal(_))) @@ -140,7 +157,7 @@ impl View { } pub(super) fn measuring_started(&mut self, readback: Readback, fit: Option) { - self.fitting.reading = Some((readback, fit)); + self.fitting.reading.start(0, readback, fit); } /// Frames the extents measured for `fit`, or fits a plot's value range to its traces. @@ -162,16 +179,17 @@ impl View { continue; } for k in 0..3 { - lo[k] = lo[k].min(f64::from(unordered(!e[k]))); - hi[k] = hi[k].max(f64::from(unordered(e[3 + k]))); + lo[k] = lo[k].min(f64::from(crate::order::from_number32(!e[k]))); + hi[k] = hi[k].max(f64::from(crate::order::from_number32(e[3 + k]))); } } self.frame_fit(lo, hi, fit); } - /// The box `fit` frames when it is known without measuring, least x and y then greatest: in a - /// diagram once its sheet stands, its vertices and wires, or the boxes of the rows fitted; else, - /// with every mark `ready`, where only tables drawn through tiles are measured, their bounds. + /// The box `fit` frames, if known without measuring, as `[min x, min y, max x, max y]`. In a + /// diagram whose sheet is placed, it is the sheet's extent or the union of the fitted rows' + /// boxes. Otherwise, for `Fit::All` once every mark is `ready` and only tiled tables are + /// measured, it is their bounds. fn known(&self, fit: &Fit, ready: bool) -> Option<[f64; 4]> { let union = |a: [f64; 4], b: [f64; 4]| [a[0].min(b[0]), a[1].min(b[1]), a[2].max(b[2]), a[3].max(b[3])]; let sheet = &self.derived.sheet; @@ -189,8 +207,8 @@ impl View { measured().filter_map(|m| self.tiled_bounds(m)).reduce(union) } - /// Frames `lo..hi` as `fit` asks: without one, a plot's value range; to reveal a row, the camera - /// brought to it where it lies out of view; else the box. + /// Frames `lo..hi` as `fit` asks. With no fit, fits a plot's value range. For `Fit::Reveal`, + /// centers the camera on the row if it is out of view. Otherwise frames the box. fn frame_fit(&mut self, lo: [f64; 3], hi: [f64; 3], fit: Option<&Fit>) { if lo[0] > hi[0] { return; @@ -206,7 +224,7 @@ impl View { .is_some_and(|s| s[0] >= area[0] && s[0] <= area[2] && s[1] >= area[1] && s[1] <= area[3]); if !inside { let camera = self.camera.centered(middle); - self.fly(camera); + self.fly(camera, None); } } Some(fit) => { @@ -218,14 +236,15 @@ impl View { } } - /// The bounds of every row `shown` draws through tiles, whatever tiles are drawn. + /// The bounds of every row of `shown`'s tiled table, regardless of which tiles are drawn. fn tiled_bounds(&self, shown: &Shown) -> Option<[f64; 4]> { let style = shown.mark.style(); self.derived.tiled.bounds(&style.data, &style.table) } - /// Frames the box `min..max` at once. Framing everything sets how far the camera can go from it; - /// framing part of it widens that reach, never narrows it. + /// Frames the box `min..max` at once, within the camera's zoom and pan limits. A fit to + /// everything sets the box that narrows the limits. A fit to part of it grows that box and + /// never shrinks it. fn frame_box(&mut self, min: [f64; 3], max: [f64; 3], everything: bool) { let camera = self.camera.framing(&self.scene.space, min, max, self.size, self.area()); self.rig.hold(); @@ -235,14 +254,14 @@ impl View { }; self.rig.bounds = match (self.rig.bounds, framed) { _ if everything => framed, - (Some(reach), Some(framed)) => Some(reach.with(framed)), - (reach, _) => reach, + (Some(bounds), Some(framed)) => Some(bounds.with(framed)), + (bounds, _) => bounds, }; self.move_camera(camera); } - /// Grows the value range a plot shows to hold `lo..hi` with headroom, and shrinks it once the - /// values fill less than a third of it. + /// Grows the value range a plot shows to hold `lo..hi` with 12% headroom on each side, and + /// shrinks it once the values fill less than a third of it. fn fit_values(&mut self, lo: f64, hi: f64) { let holds = self.plot_y.is_some_and(|[a, b]| lo >= a && hi <= b && (hi - lo) * 3.0 >= b - a); if !holds { @@ -252,17 +271,17 @@ impl View { } } - /// The coordinates a plot's traces span. /// Whether a mark traces a sampled field. fn traces(&self) -> bool { self.marks.iter().any(|shown| matches!(&shown.mark, crate::Mark::Lines(lines) if lines.trace.is_some())) } + /// The axis range a plot's traces span, widened to a unit range around a single point. fn traced_span(&self) -> Option<[f64; 2]> { let spans = self.marks.iter().filter_map(|shown| { let crate::Mark::Lines(lines) = &shown.mark else { return None }; let style = shown.mark.style(); - let table = self.drawn(&style.data)?.table(&style.table)?; + let table = self.snapshot(&style.data)?.table(&style.table)?; match table.field_named(&lines.traced())? { crate::data::Field::Sampled(samples, _) => samples.span(), _ => None, @@ -275,35 +294,8 @@ impl View { } impl super::marks::Shown { - /// Whether this mark's extents frame a fit: vertices, lines, and areas. + /// Whether fits frame this mark's extents, which depends on its kind. pub(crate) fn measured(&self) -> bool { - matches!(self.kind(), MarkKind::Vertices | MarkKind::Lines | MarkKind::Areas) - } -} - -/// The `f32` an ordered word stands for (see `lib/reduce.wesl`). -fn unordered(u: u32) -> f32 { - f32::from_bits(if u & 0x8000_0000 != 0 { u & 0x7fff_ffff } else { !u }) -} - -#[cfg(test)] -mod tests { - use super::unordered; - - /// The inverse of `lib/reduce.wesl`'s `ordered`. - fn ordered(x: f32) -> u32 { - let b = x.to_bits(); - if b & 0x8000_0000 != 0 { !b } else { b | 0x8000_0000 } - } - - #[test] - fn ordered_words_sort_like_their_values() { - let values = [-1e9_f32, -2.5, -0.0, 0.0, 1e-9, 3.0, 1e9]; - for pair in values.windows(2) { - assert!(ordered(pair[0]) <= ordered(pair[1]), "{pair:?}"); - } - for v in values { - assert_eq!(unordered(ordered(v)), v); - } + self.kind().bounds().is_some() } } diff --git a/crates/latkit/src/view/focus.rs b/crates/latkit/src/view/focus.rs index 425c02a5..a94e5703 100644 --- a/crates/latkit/src/view/focus.rs +++ b/crates/latkit/src/view/focus.rs @@ -1,14 +1,17 @@ -//! Selection, hover, and the rows a fit measures, as focus words: one per row of every mark, written -//! only where they change. The selection keeps only rows a pickable mark shows. The ends of a selected -//! or hovered edge are lit too. Each mark's lit rows are listed for its focus draw, which draws them -//! again over everything, glowing. Hover is state either way, which expressions read; whether it -//! glows is the controls' to say. +//! Focus words: selection, hover, and the rows a fit measures, as bits in one word per held row of +//! every mark, kept in its focus word pages and written only where they change. +//! +//! The selection keeps only rows a pickable mark shows. The endpoints of a selected or hovered edge +//! are lit too. Each mark's lit rows are listed by draw index for its focus draw, which draws them +//! again over everything with a glow. Hover is always state that expressions can read. The controls +//! decide whether it glows. use rustc_hash::FxHashMap; use super::View; use super::event::{Event, Hit}; -use super::sync::{Source, table_of}; +use super::sync::Source; +use super::words::Kept; use crate::data::{Field, PAGE_ROWS, Reference}; use crate::expr::compile::Deps; use crate::gpu::shaders as wgsl; @@ -16,25 +19,29 @@ use crate::gpu::work::Queue; use crate::scene::mark::Primitive; use crate::{Row, Rows}; -/// Changed words past which the whole buffer is written at once. -const WHOLE: usize = 512; +/// The row gap at which two changed words of a page are written separately. Closer ones share one +/// write, along with the unchanged words between them. +const GAP: u32 = 16; + +/// A focus word's mark and held row. +type At = (usize, u32); #[derive(Default)] pub(crate) struct Focus { pub(crate) selection: Vec, - /// What the user points at, as it was hit. + /// The hovered row, as hit. pub(crate) hovered: Option, /// Rows a fit measures. pub(crate) fitting: Vec, - /// Words written, by index into the focus buffer. - written: FxHashMap, + /// The words last written, by mark and held row. + written: FxHashMap, pub(crate) dirty: bool, - /// Every mark's lit rows, laid out since they were last written. + /// Every mark's lit rows, if laid out since last taken. lit: Option>, } impl Focus { - /// The lit rows laid out since they were last taken, to write. + /// Takes the lit rows laid out since last taken, for writing to the lit buffer. pub(crate) fn take_lit(&mut self) -> Option> { self.lit.take() } @@ -49,9 +56,10 @@ pub(crate) struct Lit { } impl View { - /// Writes the focus words that changed; after `fresh`, a new buffer, every word. Channels reading - /// selection or hover evaluate again, and every mark's lit rows are laid out again. - pub(super) fn write_focus(&mut self, queue: &Queue, buffer: &wgpu::Buffer, capacity: u64, fresh: bool) { + /// Writes changed focus words into `pages`, or every word when `fresh` (focus word pages new or + /// moved). Channels reading focus evaluate again where words changed, and every mark's lit rows + /// are laid out again. + pub(super) fn write_focus(&mut self, queue: &Queue, pages: &wgpu::Buffer, fresh: bool) { if fresh { self.focus.written.clear(); self.focus.dirty = true; @@ -59,7 +67,7 @@ impl View { if !std::mem::take(&mut self.focus.dirty) { return; } - let mut words: FxHashMap = FxHashMap::default(); + let mut words: FxHashMap = FxHashMap::default(); let glows = self.scene.controls.hover; let hovered = self.focus.hovered.as_ref().map(|hit| hit.row.clone()); let selection = self.focus_rows(&self.focus.selection, wgsl::SELECTED, true, true, &mut words); @@ -68,71 +76,60 @@ impl View { // Rows the data no longer holds, or no mark shows, leave the selection. if selection.len() < self.focus.selection.len() { self.focus.selection.clone_from(&selection); - self.events.tell(Event::Select { rows: Rows::group(selection), trigger: None }); + self.events.tell(Event::Selection { rows: Rows::group(selection), trigger: None }); } - let cleared = self.focus.written.keys().filter(|w| !words.contains_key(w)).map(|&w| (w, 0)); - let set = words.iter().filter(|(w, b)| self.focus.written.get(w) != Some(b)).map(|(&w, &b)| (w, b)); - let changes: Vec<(u32, u32)> = cleared.chain(set).filter(|&(w, _)| u64::from(w) < capacity).collect(); - if changes.len() > WHOLE { - let mut all = vec![0_u32; capacity as usize]; - for (&w, &b) in &words { - if let Some(word) = all.get_mut(w as usize) { - *word = b; + let cleared = self.focus.written.keys().filter(|at| !words.contains_key(at)).copied(); + let set = words.iter().filter(|(at, b)| self.focus.written.get(at) != Some(b)).map(|(&at, _)| at); + let mut changes: Vec = cleared.chain(set).collect(); + changes.sort_unstable(); + // Write each page's changed words in runs, unchanged words between them included. + for run in changes.chunk_by(|a, b| a.0 == b.0 && a.1 >> wgsl::PAGE_BITS == b.1 >> wgsl::PAGE_BITS) { + let (mark, page) = (run[0].0, run[0].1 >> wgsl::PAGE_BITS); + let Some(&block) = self.marks[mark].words.pages(Kept::Focus).blocks.get(page as usize) else { continue }; + let mut start = 0; + for end in 1..=run.len() { + if end < run.len() && run[end].1 - run[end - 1].1 < GAP { + continue; } - } - queue.write_buffer(buffer, 0, bytemuck::cast_slice(&all)); - } else { - for &(w, b) in &changes { - queue.write_buffer(buffer, u64::from(w) * 4, &b.to_le_bytes()); + let (first, last) = (run[start].1, run[end - 1].1); + let bits: Vec = (first..=last).map(|row| words.get(&(mark, row)).copied().unwrap_or(0)).collect(); + let at = block + (first & wgsl::PAGE_MASK); + queue.write_buffer(pages, u64::from(at) * 4, bytemuck::cast_slice(&bits)); + start = end; } } if !changes.is_empty() { - // Channels reading focus evaluate the blocks of the rows whose words changed; every row - // after the words were laid out again. - for shown in &mut self.marks { - let (focus, rows) = (shown.region.focus, shown.rows + shown.exits); - let blocks: Vec = changes - .iter() - .filter(|&&(w, _)| w >= focus && w - focus < rows) - .map(|&(w, _)| (w - focus) / wgsl::BLOCK_ROWS) - .collect(); - shown.dirty_rows(shown.channels_reading(Deps::FOCUS), (!fresh).then_some(&blocks[..])); + // Channels reading focus evaluate the pages of rows whose words changed, or every row + // when the focus word pages are new. + for (i, shown) in self.marks.iter_mut().enumerate() { + let pages: Vec = + changes.iter().filter(|&&(mark, _)| mark == i).map(|&(_, row)| row / wgsl::PAGE_ROWS).collect(); + shown.dirty_rows(shown.channels_reading(Deps::FOCUS), (!fresh).then_some(&pages[..])); } } - // Rows lit only by hover draw no glow when it does not glow. + // When hover does not glow, rows lit only by hover are left out of the focus draw. self.lay_out_lit(&words, if glows { wgsl::LIT } else { wgsl::LIT & !wgsl::HOVERED }); self.focus.written = words; } - /// Sets `bit` in the focus words of `rows` in every mark showing them (with `picked`, every - /// pickable one), and with `ends` lights the ends of each edge among them; returns the rows some - /// mark shows. A row its data holds stays shown while a table drawn in part does not draw it. - fn focus_rows( - &self, - rows: &[Row], - bit: u32, - picked: bool, - ends: bool, - words: &mut FxHashMap, - ) -> Vec { + /// Sets `bit` in the focus words of `rows` in every mark drawing their table (only pickable + /// marks with `picked`), and with `ends` also lights the endpoints of each edge among them. + /// Returns the rows some mark shows. A row in the data stays shown even while its paged table + /// has no resident page for it. + fn focus_rows(&self, rows: &[Row], bit: u32, picked: bool, ends: bool, words: &mut FxHashMap) -> Vec { let mut kept = Vec::new(); for row in rows { - let held = self.data(&row.data).and_then(|d| d.table(&row.table)).and_then(|t| t.row(row.id.as_ref())); - if held.is_none() { - continue; - } - let table = self.drawn(&row.data).and_then(|d| d.table(&row.table)); - let index = table.and_then(|t| t.row(row.id.as_ref())); + let Some(table) = self.snapshot(&row.data).and_then(|d| d.table(&row.table)) else { continue }; + let Some(index) = table.row(row.id.as_ref()) else { continue }; let mut shows = false; - for shown in &self.marks { + for (i, shown) in self.marks.iter().enumerate() { let style = shown.mark.style(); if (picked && !shown.mark.picks()) || style.data != row.data || style.table != row.table { continue; } shows = true; - let (Some(table), Some(index)) = (table, index) else { continue }; - if (index as u32) < shown.rows { - *words.entry(shown.region.focus + index as u32).or_default() |= bit; + if let Some(held) = shown.held_row(index as u32) { + *words.entry((i, held)).or_default() |= bit; } let crate::Mark::Edges(edges) = &shown.mark else { continue }; let Some((from, to)) = edges.ends.as_ref().filter(|_| ends) else { continue }; @@ -148,49 +145,55 @@ impl View { kept } - /// Lights the row `reference` names for row `index` in every vertex mark drawing its table. - fn light_end(&self, data: &crate::Name, reference: &Reference, index: usize, words: &mut FxHashMap) { + /// Sets the `END` bit of the vertex `reference` names for row `index`, in every vertex mark + /// holding it. + fn light_end(&self, data: &crate::Name, reference: &Reference, index: usize, words: &mut FxHashMap) { let Some(target) = reference.target(index) else { return }; - for shown in &self.marks { + for (i, shown) in self.marks.iter().enumerate() { let style = shown.mark.style(); if shown.kind() == crate::scene::mark::MarkKind::Vertices && style.data == *data && style.table == reference.to - && target < shown.rows + && let Some(held) = shown.held_row(target) { - *words.entry(shown.region.focus + target).or_default() |= wgsl::END; + *words.entry((i, held)).or_default() |= wgsl::END; } } } - /// Lists every mark's rows with a bit of `glowing` set for its focus draw, with the instances - /// each takes. - fn lay_out_lit(&mut self, words: &FxHashMap, glowing: u32) { + /// Lists each mark's drawn rows with any `glowing` bit set, by draw index, for its focus draw, + /// with the instances each takes. + fn lay_out_lit(&mut self, words: &FxHashMap, glowing: u32) { let mut all = Vec::new(); for i in 0..self.marks.len() { - let (focus, rows) = (self.marks[i].region.focus, self.marks[i].rows); - let mut lit: Vec = words + let shown = &self.marks[i]; + let live = shown.live(); + let mut lit: Vec<(u32, u32)> = words .iter() - .filter(|&(&w, &bits)| w >= focus && w - focus < rows && bits & glowing != 0) - .map(|(&w, _)| w - focus) + .filter(|&(&(mark, held), &bits)| mark == i && held < live && bits & glowing != 0) + .filter_map(|(&(_, held), _)| Some((shown.item_of(held)?, held))) .collect(); lit.sort_unstable(); - let pieces = if lit.is_empty() { 0 } else { self.pieces(i, &lit) }; + let held: Vec = lit.iter().map(|&(_, h)| h).collect(); + let pieces = if lit.is_empty() { 0 } else { self.pieces(i, &held) }; let rows = if pieces == 0 { 0 } else { lit.len() as u32 }; self.marks[i].lit = Lit { at: all.len() as u32, rows, pieces }; if rows > 0 { - all.extend(lit); + all.extend(lit.into_iter().map(|(item, _)| item)); } } self.focus.lit = Some(all); } - /// Instances mark `i`'s focus draw takes for each of its `lit` rows: a point one; a path as many - /// as the longest of theirs has pieces, past which a row's own draw nothing; none for faces. + /// Instances each lit row takes in mark `i`'s focus draw, for lit held rows `lit`. One for a + /// point. For paths and areas, enough for the longest lit row's pieces (a shorter row's extra + /// instances draw nothing). None for faces. fn pieces(&self, i: usize, lit: &[u32]) -> u32 { let shown = &self.marks[i]; - let Some(compiled) = &shown.compiled else { return 0 }; - match compiled.drawing.primitive { + if shown.compiled.is_none() { + return 0; + } + match shown.kind().primitive() { Primitive::Points => 1, Primitive::Strokes if !shown.listed() => shown.segments_per_row().max(1), Primitive::Strokes if shown.path() == wgsl::PATH_ENDS => (self.longest(i, lit) + 1).max(wgsl::MAX_PIECES), @@ -200,23 +203,21 @@ impl View { } } - /// The most points any of mark `i`'s `lit` rows has in its geometry column: its path, its bends, - /// or its triangles. + /// The most points any of mark `i`'s lit held rows `lit` has in its geometry column: path + /// points, bends, or triangle corners. fn longest(&self, i: usize, lit: &[u32]) -> u32 { let shown = &self.marks[i]; let Some(compiled) = &shown.compiled else { return 0 }; let Some(column) = compiled.program.columns.get(compiled.drawing.geometry as usize) else { return 0 }; - let style = shown.mark.style(); - let Some(data) = self.drawn(&style.data) else { return 0 }; - let Some(table) = data.table(&style.table) else { return 0 }; - let source = Source { data, table, derived: self.derived.beside(&style.data, &style.table) }; - let Some((_, table)) = table_of(column, &source, &self.data) else { return 0 }; + let Some(source) = Source::of(shown.mark.style(), &self.snapshots, &self.derived) else { return 0 }; + let Some((_, table)) = source.table_of(column, &self.snapshots) else { return 0 }; let Some(Field::Values(values, _)) = table.field_named(&column.name) else { return 0 }; let pages = values.pages(); lit.iter() - .filter_map(|&row| { - let page = pages.get(row as usize / PAGE_ROWS)?; - Some(page.values.geometry()?.row_coords(row as usize % PAGE_ROWS).len() as u32) + .filter_map(|&held| { + let row = shown.source_row(held)? as usize; + let page = pages.get(row / PAGE_ROWS)?; + Some(page.values.geometry()?.row_coords(row % PAGE_ROWS).len() as u32) }) .max() .unwrap_or(0) diff --git a/crates/latkit/src/view/frame.rs b/crates/latkit/src/view/frame.rs index 173b1279..64c99c76 100644 --- a/crates/latkit/src/view/frame.rs +++ b/crates/latkit/src/view/frame.rs @@ -1,38 +1,57 @@ -//! One frame, in order: tick the clock, settle readbacks, advance the camera, derive what marks read -//! beside their data, sync columns, stream uploads, fit rows into their regions, freeze transitions, -//! pack the program buffer, write focus and uniforms, then record the compute, shadow, label, render, -//! and probe passes into the caller's encoder. Nothing here submits. +//! One frame, in order: tick the clock, settle readbacks, advance the camera, derive tables from +//! the data, lay out drawings, sync columns, stream uploads, give marks word pages for their rows, +//! start transitions, pack the program buffer, write focus words and uniforms, then record the +//! compute, texts, render, probe, and query passes into the caller's encoder. No pass is recorded +//! before the program buffer is packed, since packing may move a mark's words with copies +//! recorded in the encoder. +//! +//! The frame submits nothing itself except setup that must run before its recorded passes: +//! clearing newly allocated pages and copying buffers that grew. use web_time::Instant; -use super::marks::Shown; -use super::passes::Drawn; -use super::resources::{EyeSlot, MARK_STRIDE, Resources}; -use super::sync::{Source, Synced, Window, table_of}; -use super::transition::{remap, replaced, rewritten}; +use super::drawing::Drawn; +use super::marks::Holding; +use super::resources::{MARK_STRIDE, Resources}; +use super::sync::{Inputs, Rebuilt, Rows, Source, Window}; +use super::transition::{bound, holds_alike, in_place, moved}; +use super::words::Kept; use super::{Camera, Event, Frame, View, timer, uniforms}; +use crate::data::Keyed; +use crate::derive::order::Order; use crate::expr::compile::Deps; use crate::gpu::Inner; use crate::gpu::shaders as wgsl; use crate::scene::mark::MarkKind; -use crate::{Error, derive}; +use crate::{Error, Name, derive}; -/// The most seconds comets travel in one frame: after a pause they carry on, not leap. +/// Whether a frame's pages fit in the arena. If some did not, the next frame makes space. `Short` +/// grows the arena or plans less. `Starved` means the arena is full and the plan is already +/// minimal, so the next frame releases the drawn tables of datasets waiting for space. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Default, Debug)] +pub(super) enum Room { + #[default] + Found, + Short, + Starved, +} + +/// The most seconds comets travel in one frame, so after a pause they resume instead of jumping. const TRAVEL: f64 = 0.1; impl View { - /// Records `frame` of the scene into `encoder`; the caller submits it. Until the view is - /// [ready](View::ready) for the frame's target it records nothing, and the target keeps what it - /// holds. + /// Records `frame` of the scene into `encoder`. The caller submits it. Until the view is + /// [ready](View::ready) for the frame's target, it records nothing and the target keeps its + /// contents. /// - /// Uploads stream within the GPU's budget per frame. A mark shows a new data version once all of - /// its changed pages are on the GPU, and keeps drawing the previous one until then. + /// Uploads stream within the GPU's per-frame budget. A mark shows a new data version once all + /// of its changed pages are on the GPU, and draws the previous version until then. pub fn render(&mut self, encoder: &mut wgpu::CommandEncoder, frame: &Frame<'_>) { self.record(encoder, frame, true); } - /// Records `frame` for a picture, which takes no input: it asks for what it draws and changes - /// with alone. + /// Records `frame` for a picture. Pictures take no input, so they compile only the pipelines + /// they draw and update with, never those that probes and queries use. pub(crate) fn render_picture(&mut self, encoder: &mut wgpu::CommandEncoder, frame: &Frame<'_>) { self.record(encoder, frame, false); } @@ -45,43 +64,53 @@ impl View { } return; } + // Set the frame's size first, so the sources load the tiles this frame shows. Clear the + // arrival flag before that, so work they start that finishes immediately wakes the next + // frame. + self.pixel_ratio = frame.pixel_ratio.max(0.01); + let ratio = f64::from(self.pixel_ratio); + self.size = [f64::from(frame.size[0]) / ratio, f64::from(frame.size[1]) / ratio]; + let landed = self.signal.take(); if !self.ready_for(frame.format, frame.pixel_ratio, input) { + self.signal.leave(landed); return; } let started = Instant::now(); - self.signal.clear(); + let starved = std::mem::take(&mut self.room) == Room::Starved; let dt = self.clock.tick(frame.time); let gpu = self.gpu.clone(); let (device, queue) = (&gpu.0.device, &gpu.0.queue); if let Some(timer) = &mut self.timer { timer.begin(encoder); } - // Runs the callbacks of finished readbacks; a no-op in browsers, which run them themselves. + // Runs the callbacks of finished readbacks. A no-op in browsers, which run them themselves. let _ = device.poll(wgpu::PollType::Poll); self.tell_gpu_errors(); - self.pixel_ratio = frame.pixel_ratio.max(0.01); - let ratio = f64::from(self.pixel_ratio); - self.size = [f64::from(frame.size[0]) / ratio, f64::from(frame.size[1]) / ratio]; + self.measure_gutters(); self.settle(); self.advance(dt); let mut inner = gpu.0.lock(); + // Release before `begin_frame`, which frees the pages before this frame reads any of them. + if starved { + self.start_over(&mut inner); + } + inner.begin_frame(self.id, device, queue); inner.atlas.tick(); self.settle_labels(&mut inner); self.derive(&mut inner); + self.tell_failures(); self.answer_arranging(); self.stream(&mut inner); - let moved = self.lay_out(encoder, &mut inner); + let fresh = self.lay_out(encoder, &mut inner); let (elapsed, travel) = self.effect_clock(); - let resources = self.resources.as_ref().expect("created"); - let (buffer, capacity) = (resources.focus().clone(), resources.focus_words()); - self.focus.dirty |= moved; - self.write_focus(queue, &buffer, capacity, moved); + let pages = inner.arena.buffer().clone(); + self.write_focus(queue, &pages, fresh); if let Some(rows) = self.focus.take_lit() { self.resources.as_mut().expect("created").write_lit(device, queue, &rows); } let measure = self.measuring(); - // A picture takes no input; frames that do probe and answer queries once what those use - // compiled. + // A picture takes no input. Other frames probe and answer queries once their pipelines + // have compiled. let answers = input && self.answers(); let probe = self.probe_point(answers); let query = self.lay_out_lists(answers); @@ -102,23 +131,16 @@ impl View { if let Some((words, _)) = query { uniform.query = words; } - self.label_words(&mut uniform, dt); - let main = uniforms::eye(&projected, self.size, self.pixel_ratio); - let light = self.shade(&mut uniform, &main); + let eye = uniforms::eye(&projected, self.size, self.pixel_ratio); let sprites = self.overlay(&mut inner); self.lay_out_blocks(); - self.write_uniforms(&uniform, &main, light.as_ref(), &sprites); + self.write_uniforms(&uniform, &eye, &sprites); if let Some(readback) = self.compute(encoder, &mut inner, measure.is_some(), travel > 0.0) { self.measuring_started(readback, measure.and_then(|m| m.fit)); } self.count_found(encoder, &inner); - let mut drawn = Drawn::default(); - if let Some(light) = &light { - self.cast(encoder, &mut inner, &mut drawn, light); - } else { - self.shadow_map.mark(0); - } - self.place_labels(encoder, &mut inner, &main); + let mut drawn = super::passes::Drawn::default(); + self.place_labels(encoder, &mut inner, &eye); let depth = probe.is_some() || query.is_some_and(|(_, depth)| depth); self.draw(encoder, &mut inner, &mut drawn, frame, sprites.len() as u32, depth); self.stats.draws = drawn.draws; @@ -132,8 +154,9 @@ impl View { timer.end(encoder, device); } for shown in &mut self.marks { - shown.held = shown.rows + shown.exits; - shown.moved = None; + shown.before = shown.held(); + shown.words.settle(); + shown.remap = wgsl::NONE; shown.carry = false; } self.events.extend(inner.atlas.errors.drain(..).map(Event::Error)); @@ -142,15 +165,15 @@ impl View { self.end(started, input); } - /// The time effects show this frame, and the seconds comets travel to it. What reads the clock - /// shows anything new only when it moved; a frame drawn again at the same instant, as a shot - /// draws while it waits, evaluates none of it. Under reduced motion the effect clock holds where - /// it first stood; a picture's still sets it. + /// The effect time for this frame, and the seconds comets travel to reach it. Channels reading + /// the clock evaluate again only when it moves, so a frame redrawn at the same instant (as a + /// shot draws while it waits) evaluates none of them. Under reduced motion the effect time + /// stays at its first value. A shot's fixed time overrides both. fn effect_clock(&mut self) -> (f64, f64) { let held = !self.moves() && !self.effect_time.is_nan(); let elapsed = self.signal.elapsed().unwrap_or_else(|| if held { self.effect_time } else { self.clock.now() }); - // Comets travel the clock's step since the last frame: none at first or back in time, and - // never more than a pause's worth. + // Comets travel the clock's step since the last frame: none on the first frame or when time + // runs backward, and at most `TRAVEL`. let travel = if self.effect_time.is_nan() { 0.0 } else { (elapsed - self.effect_time).clamp(0.0, TRAVEL) }; if self.effect_time != elapsed { self.effect_time = elapsed; @@ -161,9 +184,10 @@ impl View { (elapsed, travel) } - /// Ends a frame begun at `started`: what it stood in for, what it took, and the next frame its - /// work asks for, as work landing between frames does. A frame taking no `input`, a picture's, - /// asks none for probes and queries, which only frames that take input run. + /// Ends a frame begun at `started`. Reports what it still waits on, records its stats, and + /// requests the next frame if work remains (as work arriving between frames does). A frame + /// without `input` (a picture's) requests no frame for probes and queries, which only input + /// frames run. fn end(&mut self, started: Instant, input: bool) { let waiting = self.under_way(); if waiting != self.waiting { @@ -178,9 +202,9 @@ impl View { } } - /// Lays the frame out: where every mark's rows sit, the transitions starting, the ends edges - /// attach to, fitted domains, the buffers they fit, the program, and the labels; whether rows - /// moved. + /// Lays out the frame: each mark's word pages, starting transitions, edge end attachments, + /// fitted domains and the buffers sized for them, the program buffer, and labels. Returns + /// whether any focus word pages are new. fn lay_out(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner) -> bool { let gpu = self.gpu.clone(); let device = &gpu.0.device; @@ -188,28 +212,121 @@ impl View { let geodesic = self.geodesic(&self.marks[i]); self.marks[i].geodesic = geodesic; } - let resources = self.resources.get_or_insert_with(|| Resources::new(device)); - resources.forget_retired(); - let moved = self.fit_rows(encoder); - self.transitions(encoder, inner); + self.resources.get_or_insert_with(|| Resources::new(device)); + let fresh = self.fit_words(inner); + self.fit_easing(inner); + inner.arena.clear(device, &gpu.0.queue); + self.transitions(inner); self.attach_ends(); let domains = self.place_domains(); let resources = self.resources.as_mut().expect("created"); - let vertices: u64 = self - .marks - .iter() - .filter(|m| m.kind() == MarkKind::Vertices) - .map(|m| u64::from(m.region.state / wgsl::VERTEX_WORDS + m.region.capacity)) - .max() - .unwrap_or(0); - resources.fit(device, self.marks.len() as u64, domains, vertices); - self.pack(inner); + resources.fit(device, self.marks.len() as u64, domains); + self.pack(encoder, inner); self.lay_out_labels(inner); - moved + fresh + } + + /// Gives every mark word pages for its held rows: state, focus words, and, for a vertex mark + /// that faces attach to, the normals the faces sum. Rows in new state pages evaluate from + /// scratch. A mark whose pages do not fit in the arena draws nothing until they do. Returns + /// whether any focus word pages are new. + fn fit_words(&mut self, inner: &mut Inner) -> bool { + let gpu = self.gpu.clone(); + let device = (&gpu.0.device, &gpu.0.queue); + let summing = super::graph::summing(&self.marks); + let (mut fresh, mut short, mut changed) = (false, false, false); + for (shown, summing) in self.marks.iter_mut().zip(summing) { + let rows = shown.page_rows(); + for (kept, width) in shown.kept_widths(summing) { + let Some(fitted) = shown.words.fit(kept, &rows, width, device, &mut inner.arena) else { + short = true; + continue; + }; + if !fitted.changed { + continue; + } + shown.packed = false; + changed = true; + fresh |= kept == Kept::Focus; + // Rows in new state pages evaluate every channel, writing every word first. + if kept == Kept::State { + shown.slots_bound(&fitted.new); + shown.fresh.extend(&fitted.new); + } + } + } + if short { + self.found_no_room(inner); + } + if changed { + self.hold_to_share(inner); + } + fresh + } + + /// Records that some pages did not fit in the arena. Marks missing pages draw nothing until the + /// next frame makes space. If the arena can grow, or the plan can shrink (it shrinks here), + /// that is `Short`. Otherwise it is `Starved`, which releases the drawn tables of datasets + /// waiting for space, unless an earlier release freed nothing (`limited`). + fn found_no_room(&mut self, inner: &Inner) { + let room = if !inner.arena.full() || self.planning.overflowed() { + Room::Short + } else if self.limited { + Room::Found + } else { + Room::Starved + }; + self.room = self.room.max(room); + } + + /// Releases the drawn tables of each dataset waiting for space, when the arena is full and the + /// plan is already minimal. Their marks draw nothing until the staged tables upload into the + /// freed space. If nothing was released, the view's data does not fit the budget. It reports a + /// `Limit` error once and waits for a new data version to publish. + fn start_over(&mut self, inner: &mut Inner) { + let mut freed = false; + for shown in &mut self.marks { + if !self.staged.contains_key(&shown.mark.style().data) || shown.holding.take().is_none() { + continue; + } + shown.release(inner); + (shown.rows, shown.exits, shown.before) = (0, 0, 0); + (shown.begin, shown.transition) = (None, None); + shown.packed = false; + freed = true; + } + for (key, drawn) in self.drawings.iter_mut() { + if self.staged.contains_key(&key.0) { + drawn.current = None; + } + } + if !freed { + self.limited = true; + let reason = "the view does not fit the GPU's budget, even drawing as little as possible"; + self.events.tell(Event::Error(Error::Limit(reason.into()))); + } + } + + /// Keeps the view's GPU memory within its budget share. Counts each page its tables pin once, + /// plus the marks' word pages, minus idle pages. Idle pages stay only while there is space and + /// go first when the plan shrinks. + fn hold_to_share(&mut self, inner: &Inner) { + let mut pins: Vec = self.marks.iter().flat_map(|m| m.tables.pins()).collect(); + pins.sort_unstable_by_key(|key| (key.page, key.low, key.words as u8)); + pins.dedup(); + let pages: u64 = pins.into_iter().map(|key| inner.arena.bytes_of(key)).sum(); + let words: u64 = self.marks.iter().map(|m| m.words.bytes()).sum(); + let idle: f64 = self + .drawings + .iter() + .filter_map(|(key, drawn)| Some(self.planning.page_bytes(key, drawn.current.as_ref()?.idle.len() as u32))) + .sum(); + let needed = ((pages + words) as f64 - idle).max(0.0); + self.planning.held(needed as u64, self.choices.values().any(|c| c.capped)); } - /// Takes what landed since the last frame: probes, fits, queries, the intents and keys waiting on - /// picks; and tells, once, why marks the data given since no longer fits failed. + /// Takes the results that arrived since the last frame: GPU timing, fits, probes, queries, and + /// the intents and key steps waiting on picks. fn settle(&mut self) { if let Some(ms) = self.timer.as_mut().and_then(timer::Timer::landed) { self.stats.gpu_ms = ms; @@ -218,126 +335,109 @@ impl View { self.settle_probe(); self.settle_query(); self.settle_gestures(); + } + + /// Reports, once each, why marks failed against the data drawn. Runs after the frame takes the + /// latest data versions, so a dataset filled between frames reports nothing about the version + /// before. + fn tell_failures(&mut self) { for (i, shown) in self.marks.iter_mut().enumerate() { if let Some((error, told)) = &mut shown.failure && !*told { *told = true; - self.events.tell(Event::Error(shown.mark.within(i, error.clone()))); + self.events.tell(Event::Error(error.clone().at(i).at("marks"))); } } } - /// Derives what marks read beside their data, from the data and the scene alone, handing the - /// host what the frame cannot afford. First the tiles the camera and the plot window ask for: - /// the datasets tiles make, then what each dataset drawn in part draws, which the rest derive - /// from. + /// Derives tables from the data and scene alone, handing the host any work the frame cannot + /// afford. First come the orders of paged tables and the ranges of them the camera needs. These + /// come from the tables' pyramids or, for tables of vector tiles, from the vector tiles drawn. + /// Every other derivation follows. fn derive(&mut self, inner: &mut Inner) { let host = inner.host.clone(); let projected = self.projected(); let earth = matches!(self.scene.space, crate::Space::Earth(_)); let placing = earth || matches!(self.scene.space, crate::Space::Plane(_)); let sight = derive::sight::Sight { projected: &projected, size: self.size, earth }; - // One budget for the frame: the tiles, what is drawn in part, then every other derivation. + // One work budget for the frame, spent on paged tables first, then on other derivations. let (waker, counters) = (self.waker.clone(), self.gpu.0.counters.clone()); let errors = self.gpu.0.errors.clone(); - let mut jobs = derive::Jobs::new(host.as_deref(), &waker, &counters, &errors); - let mut tiles = std::mem::take(&mut self.derived.tiles); - let reading = self.reading(); - let asked = derive::tiles::Asked { sight: earth.then_some(&sight), reading: &reading }; - tiles.update(&self.derive_input(host.as_deref()), &asked, &jobs); - for data in tiles.take_changed() { - self.take_data(data, inner); - } - self.derived.tiles = tiles; + let mut jobs = derive::Jobs::new(host.as_deref(), &waker, &counters, &errors, inner.frame.work); let budget = inner.budget.bytes; self.update_plan(&mut inner.ledger, budget); self.trace_columns = self.planning.trace_columns(); - let pinned: Vec = self.focus.selection.iter().chain(&self.focus.fitting).cloned().collect(); let ask = derive::tiled::Ask { - sources: &self.sources, + data: &self.snapshots, plans: self.planning.plans(), + tiles: &self.derived.tiles, sight: placing.then_some(&sight), earth, - pinned: &pinned, - at: self.at, }; let mut tiled = std::mem::take(&mut self.derived.tiled); - let (changed, _) = tiled.update(&ask, &mut jobs); + self.choices = tiled.update(&ask, &mut jobs); self.derived.tiled = tiled; - self.follow_sources(&changed); + self.derived.tiles.choose(self.planning.plans(), &mut self.choices); + let paged = self.paged(); let mut derived = std::mem::take(&mut self.derived); - if derived.update(&self.derive_input(host.as_deref()), &mut jobs) { + let input = derive::Input { paged: Some(&paged), ..self.derive_input(host.as_deref()) }; + if derived.update(&input, &mut jobs) { self.changed = true; } self.derived = derived; + inner.frame.work = jobs.left() as u32; } - /// Makes what marks draw follow the sources: each dataset as it is, or as drawn in part, which - /// `changed` holds the new versions of. - fn follow_sources(&mut self, changed: &[crate::Data]) { - let mut next = Vec::with_capacity(self.sources.len()); - for source in &self.sources { - let name = source.name(); - let drawn = match changed.iter().find(|d| d.name() == name).cloned() { - Some(drawn) => drawn, - None if self.derived.tiled.draws(name) => match self.drawn(name) { - Some(drawn) => drawn.clone(), - None => continue, - }, - None => source.clone(), - }; - next.push(drawn); - } - if next != self.data { - self.data = next; - // Rows may have moved under the focus. - self.focus.dirty = true; - self.fitting.plot_due = true; - } + /// Updates every tile set to what the camera and plot window need, from the GPU's tile cache + /// and the pieces that arrived since. Tiles fetch and decode as they arrive, so this spends + /// none of the frame's work budget. + pub(super) fn update_tiles(&mut self, inner: &mut Inner) { + let host = inner.host.clone(); + let projected = self.projected(); + let earth = matches!(self.scene.space, crate::Space::Earth(_)); + let sight = derive::sight::Sight { projected: &projected, size: self.size, earth }; + let reading = self.reading(); + let asked = crate::tiles::Asked { sight: earth.then_some(&sight), reading: &reading }; + let (waker, counters) = (self.waker.clone(), self.gpu.0.counters.clone()); + let errors = self.gpu.0.errors.clone(); + let jobs = derive::Jobs::new(host.as_deref(), &waker, &counters, &errors, 0); + let mut tiles = std::mem::take(&mut self.derived.tiles); + tiles.update(&self.derive_input(host.as_deref()), &asked, &mut inner.tiles, &jobs); + self.derived.tiles = tiles; } - /// What derivations read this frame. + /// The input derivations read this frame. fn derive_input<'a>(&'a self, host: Option<&'a dyn crate::host::Host>) -> derive::Input<'a> { derive::Input { scene: &self.scene, scene_version: self.scene_version, - data: &self.data, + data: &self.snapshots, host, dragging: self.dragging(), + paged: None, } } - /// Lays out the lists buffer, the marks' segment lists, and takes this frame's query when it - /// `answers` them, returning its words and whether it reads depth. A buffer replaced empty counts - /// every segment again. + /// Lays out the lists buffer and the marks' segment lists. If the frame `answers` queries, + /// takes this frame's query, returning its words and whether it reads depth. A replaced buffer + /// starts empty, so every segment is counted again. fn lay_out_lists(&mut self, answers: bool) -> Option<([f32; 4], bool)> { self.lay_out_segments(); let device = &self.gpu.0.device; - self.resources.as_mut().expect("created").fit_lists(device, u64::from(self.segment_words)); + self.resources.as_mut().expect("created").fit_lists(device, u64::from(self.list_words)); if answers { self.next_query() } else { None } } - /// Writes the frame's uniform, its eyes, every mark's uniform, and the overlay's sprites. - fn write_uniforms( - &mut self, - frame: &wgsl::Frame, - main: &wgsl::Eye, - light: Option<&wgsl::Eye>, - sprites: &[wgsl::Sprite], - ) { + /// Writes the frame's uniform, its eye, every mark's uniform, and the overlay's sprites. + fn write_uniforms(&mut self, frame: &wgsl::Frame, eye: &wgsl::Eye, sprites: &[wgsl::Sprite]) { let gpu = self.gpu.clone(); let (device, queue) = (&gpu.0.device, &gpu.0.queue); + self.reveal(); let marks = self.mark_uniforms(); let resources = self.resources.as_mut().expect("created"); queue.write_buffer(&resources.frame, 0, bytemuck::bytes_of(frame)); - let mut eyes = vec![0_u8; 2 * MARK_STRIDE as usize]; - let light_at = EyeSlot::Light.offset() as usize; - eyes[..size_of::()].copy_from_slice(bytemuck::bytes_of(main)); - if let Some(light) = light { - eyes[light_at..light_at + size_of::()].copy_from_slice(bytemuck::bytes_of(light)); - } - queue.write_buffer(resources.eyes(), 0, &eyes); + queue.write_buffer(resources.eye(), 0, bytemuck::bytes_of(eye)); if !marks.is_empty() { queue.write_buffer(resources.marks(), 0, &marks); } @@ -347,15 +447,18 @@ impl View { } } - /// Syncs every mark's columns and streams the pages they need within the frame's budget. + /// Lays out each paged table's drawing, syncs every mark's columns, uploads the pages they need + /// within the frame's remaining upload budget, and swaps in each dataset whose pages have all + /// arrived, including those that just did. fn stream(&mut self, inner: &mut Inner) { let gpu = self.gpu.clone(); - let waiting = self.sync(inner); - let budget = inner.budget; - inner.arena.pump(&gpu.0.device, &gpu.0.queue, budget.upload_per_frame); - if waiting { - // Pages that just uploaded can show this frame. - self.sync(inner); + self.lay_out_drawings(); + self.sync(inner); + let uploaded = inner.arena.pump(&gpu.0.queue, inner.frame.upload); + inner.frame.upload = inner.frame.upload.saturating_sub(uploaded); + if self.publish(inner) { + self.limited = false; + self.hold_to_share(inner); } } @@ -363,14 +466,15 @@ impl View { pub(super) fn mark_uniforms(&self) -> Vec { let mut marks = vec![0_u8; self.marks.len() * MARK_STRIDE as usize]; for (i, shown) in self.marks.iter().enumerate() { - let uniform = uniforms::mark(i, shown, self.labels.slots(i), &self.clock); + let uniform = uniforms::mark(i, shown, &self.labels, &self.clock); let at = i * MARK_STRIDE as usize; marks[at..at + size_of::()].copy_from_slice(bytemuck::bytes_of(&uniform)); } marks } - /// Tells errors the device reported outside any scope, once each. + /// Reports errors the device raised outside any error scope, and custom pipeline errors, once + /// each. pub(super) fn tell_gpu_errors(&mut self) { let errors: Vec = std::mem::take(&mut *self.gpu.0.errors.lock().unwrap_or_else(std::sync::PoisonError::into_inner)); @@ -379,8 +483,8 @@ impl View { self.events.extend(custom.into_iter().map(Event::Error)); } - /// The plot window lines trace across, and the pixel columns it spans within what the plan - /// affords. + /// The plot's x range that traces span, and its width in device pixel columns, capped by the + /// plan. fn trace_window(&self) -> (Option<[f64; 2]>, u32) { let across = match self.camera { Camera::Plot { x, .. } => Some(x), @@ -390,93 +494,225 @@ impl View { (across, (((area[2] - area[0]) * f64::from(self.pixel_ratio)).max(1.0) as u32).min(self.trace_columns)) } - /// What marks read of sampled fields, for time tiles to fetch: the plot window, the coordinate - /// shown, and each field a mark's program reads, with whether a line traces it. - fn reading(&self) -> derive::tiles::Reading { + /// What marks read of sampled fields, for time tiles to fetch: the plot window and its pixel + /// columns, the coordinate shown, and each field a mark's program reads, with whether a line + /// traces it. + fn reading(&self) -> crate::tiles::Reading { let (across, columns) = self.trace_window(); let mut fields = Vec::new(); for shown in &self.marks { let Some(compiled) = &shown.compiled else { continue }; let style = shown.mark.style(); for column in compiled.program.columns.iter().filter(|c| !c.derived) { - let (data, table) = column.of.clone().unwrap_or_else(|| (style.data.clone(), style.table.clone())); + let (data, table) = column.from.clone().unwrap_or_else(|| (style.data.clone(), style.table.clone())); fields.push((data, table, column.name.clone(), column.trace)); } } - derive::tiles::Reading { across, columns, at: self.at, fields } + crate::tiles::Reading { across, columns, at: self.at, fields } } - /// Syncs every mark's columns; changed columns dirty the channels reading them, and changed - /// static values or rows start a transition. True when a mark waits for uploads. - fn sync(&mut self, inner: &mut Inner) -> bool { + /// Syncs every mark's columns to its data and, for a paged table, to its latest drawing. Column + /// tables whose inputs changed are rebuilt once and staged with their dataset. The dataset + /// reserves every page they read all at once, or waits for space. + fn sync(&mut self, inner: &mut Inner) { let at = self.at; let (across, columns) = self.trace_window(); let window = Window { across, columns, steps: 0 }; - // A drag moves rows at once. - let easing = !self.ease().is_zero() && !self.dragging(); - let mut waiting = false; - for shown in &mut self.marks { + // Each paged table's order, by dataset. + let mut orders: Keyed> = Keyed::default(); + for ((data, table), drawn) in &self.drawings { + if let Some(drawing) = drawn.latest() { + orders.entry(data.clone()).or_default().push((table.clone(), drawing.order.clone())); + } + } + for (i, shown) in self.marks.iter_mut().enumerate() { let Some(compiled) = &shown.compiled else { continue }; - let program = compiled.program.clone(); - let geometry = compiled.drawing.geometry; - let window = Window { steps: compiled.drawing.flags & (wgsl::STEP_BEFORE | wgsl::STEP_AFTER), ..window }; let style = shown.mark.style(); - let Some(data) = self.data.iter().find(|d| *d.name() == style.data) else { continue }; - let Some(table) = data.table(&style.table) else { continue }; - let derived = self.derived.beside(&style.data, &style.table); - let source = Source { data, table, derived }; - let synced = shown.tables.sync(&program, &source, &self.data, at, window, geometry, inner); - waiting |= matches!(synced, Synced::Waiting); - let Synced::Changed { columns, blocks } = synced else { continue }; - let moved = shown.drawn.as_ref().and_then(|(origin, old)| { - if *origin == data.origin() { remap(old, table) } else { Some(replaced(old, table)) } - }); - // A path morphs from the one its row ran through only while every row keeps its place; - // rows that moved, as tiles' do when finer ones replace them, fade instead. - let kept = moved.as_ref().is_none_or(|(sources, exits)| { - exits.is_empty() && sources.iter().enumerate().all(|(r, &s)| s == r as u32 || s == wgsl::NONE) - }); - // Rows that came, went, or moved evaluate every channel, constants included: where every row - // kept its place, as a table drawn through tiles keeps those of the tiles still drawn, over - // the row blocks rewritten; else over every row. - let in_place = moved - .as_ref() - .is_some_and(|(sources, _)| sources.iter().enumerate().all(|(r, &s)| s == r as u32 || s == wgsl::NONE)); - if moved.is_some() || shown.rows != table.len() as u32 { - match shown.drawn.as_ref().filter(|_| in_place).and_then(|(_, old)| rewritten(old, table)) { - Some(blocks) => shown.rows_rewritten(&blocks), - None => shown.all_dirty(), - } + let Some(mut source) = Source::of(style, &self.snapshots, &self.derived) else { continue }; + source.paged = self.derived.paged(&style.data, &style.table); + let drawing = self.drawings.get(&(style.data.clone(), style.table.clone())).and_then(Drawn::latest); + let rows = drawing.map_or(Rows::Own, |drawing| Rows::Held(drawing)); + let window = Window { steps: compiled.drawing.flags & (wgsl::STEP_BEFORE | wgsl::STEP_AFTER), ..window }; + let orders = orders.get(&style.data).map_or(&[][..], Vec::as_slice); + let inputs = Inputs { all: &self.snapshots, at, window, orders }; + let staged = self.staged.get(&style.data).and_then(|s| s.of(i)); + if let Some(rebuilt) = shown.tables.rebuild(&compiled.program, &source, rows, &inputs, staged, inner) { + self.staged.entry(style.data.clone()).or_default().put(i, rebuilt, inner); + } + } + let gpu = self.gpu.clone(); + let mut short = false; + for (_, staged) in self.staged.iter_mut() { + short |= !staged.reserve(inner, &gpu.0.device, &gpu.0.queue); + } + if short { + self.found_no_room(inner); + } + } + + /// Swaps in each dataset once all of it is ready and every mark has word pages for its new + /// rows. A dataset's rebuilt tables and its paged tables' drawings swap in together, so edges + /// never end at another drawing's vertices and a data version appears all at once. Returns + /// whether any tables were swapped in. + fn publish(&mut self, inner: &mut Inner) -> bool { + let mut datasets: Vec = + self.marks.iter().filter(|m| m.compiled.is_some()).map(|m| m.mark.style().data.clone()).collect(); + datasets.sort(); + datasets.dedup(); + let now = self.clock.now(); + let mut published = false; + for name in datasets { + let marks: Vec = (0..self.marks.len()) + .filter(|&i| self.marks[i].compiled.is_some() && self.marks[i].mark.style().data == name) + .collect(); + if self.staged.get_mut(&name).is_some_and(|s| !s.ready(inner)) { + continue; } - // Without an ease, a flowing mark whose rows moved still begins, a transition of no length, - // so its comets follow the rows. - let carries = shown.mark.flows() && moved.is_some(); - if (easing || carries) && shown.held > 0 && (moved.is_some() || shown.static_columns(columns)) { - shown.begin(moved); - } else if moved.is_some() { - shown.exits = 0; + if !marks.iter().all(|&i| self.room_for(i, inner)) { + self.found_no_room(inner); + continue; } - if !kept || (shown.begin.is_none() && shown.transition.is_none()) { - shown.tables.thaw(inner); + let mut staged = self.staged.remove(&name).unwrap_or_default(); + for (key, drawn) in self.drawings.iter_mut() { + if key.0 == name + && let Some(next) = drawn.next.take() + { + if next.fades { + drawn.since = now; + } + drawn.current = Some(next); + } + } + for i in marks { + let rebuilt = staged.take(i); + published |= rebuilt.is_some(); + self.took(i, rebuilt, inner); } - shown.rows = table.len() as u32; - shown.drawn = Some((data.origin(), table.clone())); - shown.columns_changed(columns, blocks.as_deref()); - if shown.kind() == MarkKind::Lines { - self.fitting.plot_due = true; + staged.release(inner); + } + published + } + + /// Whether mark `i` has word pages for the rows it will hold once its dataset publishes, + /// allocating any it lacks. Its existing pages stay unchanged until then. + fn room_for(&mut self, i: usize, inner: &mut Inner) -> bool { + let shown = &self.marks[i]; + let style = shown.mark.style(); + let key = (style.data.clone(), style.table.clone()); + let drawing = self.drawings.get(&key).and_then(Drawn::latest); + let live = match drawing { + Some(drawing) => drawing.live(), + None => self.snapshot(&key.0).and_then(|d| d.table(&key.1)).map_or(0, |t| t.len() as u32), + }; + // New pages are sized as they will be once the rows arrive. + let mut rows = shown.page_rows(); + let had = rows.len(); + rows.extend(super::marks::page_rows(live, live + shown.exits, drawing.is_some()).into_iter().skip(had)); + if rows.len() == had { + return true; + } + let widths = shown.kept_widths(super::graph::summing(&self.marks)[i]); + let gpu = self.gpu.clone(); + let shown = &mut self.marks[i]; + widths.into_iter().all(|(kept, width)| { + let fitted = shown.words.fit(kept, &rows, width, (&gpu.0.device, &gpu.0.queue), &mut inner.arena); + shown.packed &= fitted.as_ref().is_some_and(|f| !f.changed); + fitted.is_some() + }) + } + + /// Updates mark `i` after its dataset publishes: its rebuilt tables, its table's drawing, and + /// the rows they hold. Changed columns dirty the channels reading them. Rows that came, went, + /// or moved evaluate every channel. Moved rows or changed static values begin a transition. + fn took(&mut self, i: usize, rebuilt: Option, inner: &mut Inner) { + // During a drag, rows move without easing. + let easing = !self.ease().is_zero() && !self.dragging(); + let shown = &mut self.marks[i]; + let Some(compiled) = &shown.compiled else { return }; + let (program, geometry) = (compiled.program.clone(), compiled.drawing.geometry); + let key = (shown.mark.style().data.clone(), shown.mark.style().table.clone()); + let Some(data) = self.snapshots.iter().find(|d| *d.name() == key.0) else { return }; + let Some(table) = data.table(&key.1) else { return }; + let drawing = self.drawings.get(&key).and_then(|d| d.current.clone()); + let published = rebuilt.map(|rebuilt| shown.tables.publish(rebuilt, &program, geometry, inner)); + let holding = Holding { origin: data.origin(), table: table.clone(), drawing: drawing.clone() }; + if published.is_none() && shown.holding.as_ref().is_some_and(|h| holds_alike(h, &holding)) { + return; + } + let before = shown.holding.take(); + let alike = before.as_ref().is_some_and(|old| holds_alike(old, &holding)); + let moved = before.as_ref().and_then(|old| moved(old, &holding)); + // A path morphs from its previous shape only while every row keeps its place. Rows that + // moved, as they do between orders, fade instead. + let kept = moved.as_ref().is_none_or(|(sources, exits)| exits.is_empty() && in_place(sources)); + // Slots now holding pages of other rows: their rows evaluate every channel. + let bound = match &drawing { + Some(new) => bound(before.as_ref().and_then(|h| h.drawing.as_deref()).filter(|_| moved.is_none()), new), + None => Vec::new(), + }; + match &before { + // Rows that came, went, or moved evaluate every channel, constants included: over the + // rewritten row blocks if every row kept its place, otherwise over every row. + Some(old) if old.drawing.is_none() && drawing.is_none() => { + if moved.is_some() || shown.rows != table.len() as u32 { + let rewritten = old.table.len() <= table.len() && moved.as_ref().is_none_or(|(s, _)| in_place(s)); + match rewritten.then(|| super::transition::rewritten(&old.table, table)).flatten() { + Some(pages) => shown.rows_rewritten(&pages), + None => shown.all_dirty(), + } + } } + Some(_) if moved.is_none() => shown.slots_bound(&bound), + _ => shown.all_dirty(), + } + shown.rows = drawing.as_ref().map_or(table.len() as u32, |d| d.items); + shown.holding = Some(holding); + // Without an ease, a flowing mark whose rows moved still begins a zero-length transition, + // so its comets follow the rows. + let carries = shown.mark.flows() && moved.is_some(); + let eased = published.as_ref().is_some_and(|p| shown.static_columns(p.data)); + if (easing || carries) && shown.before > 0 && (moved.is_some() || eased) { + shown.begin(moved); + } else if moved.is_some() { + shown.exits = 0; + } else if shown.transition.is_some() && !bound.is_empty() { + // Rows in slots rebound mid-transition enter. The rest keep easing from their current + // state. + shown.restart(&bound); + } + if !kept || (shown.begin.is_none() && shown.transition.is_none()) { + shown.tables.thaw(inner); + } + // If where rows draw changed, bump the versions so normals, labels, and segment lists + // follow. A changed column alone evaluates the channels reading it, which bump them. + if !alike { + shown.version += 1; + shown.place_version += 1; + } + shown.packed = false; + if let Some(published) = &published { + shown.columns_changed(published.data, published.blocks.as_deref()); + let source = Source::of(shown.mark.style(), &self.snapshots, &self.derived); for text in &program.texts { let column = &program.columns[text.column as usize]; - let code = table_of(column, &source, &self.data).map_or(wgsl::NONE, |(d, t)| { - inner.dictionaries.field(d, &t.name, &column.name).code_of(&text.text) - }); + let code = + source.as_ref().and_then(|s| s.table_of(column, &self.snapshots)).map_or(wgsl::NONE, |(d, t)| { + inner.dictionaries.field(d, &t.name, &column.name).code_of(&text.text) + }); shown.constants[text.constant as usize] = code; } } - waiting + if shown.kind() == MarkKind::Lines { + self.fitting.plot_due = true; + } + // Rows may have moved under the focus words. + if !alike { + self.focus.dirty = true; + } } - /// Where each mark's fitted domains start; returns how many there are. + /// Assigns each mark the index of its first fitted domain, dirtying marks whose index moved. + /// Returns the total number of domains. fn place_domains(&mut self) -> u64 { let mut domains = 0; for shown in &mut self.marks { @@ -488,32 +724,4 @@ impl View { } u64::from(domains) } - - /// Points every edge and face mark's ends at the vertex marks drawing their tables. - fn attach_ends(&mut self) { - for i in 0..self.marks.len() { - let ends = ends(&self.marks, i); - if ends != self.marks[i].ends { - self.marks[i].ends = ends; - self.marks[i].packed = false; - self.marks[i].all_dirty(); - } - } - } -} - -/// The graph slot of vertex 0 of the vertex mark each of mark `i`'s ends attaches to, or `NONE`. An -/// end attaches to the unnamed vertex mark drawing its table, else to the first one. -fn ends(marks: &[Shown], i: usize) -> Vec { - let Some(compiled) = &marks[i].compiled else { return Vec::new() }; - let data = &marks[i].mark.style().data; - compiled - .program - .ends - .iter() - .map(|table| { - super::graph::end_mark(marks, data, table) - .map_or(wgsl::NONE, |m| marks[m].region.state / wgsl::VERTEX_WORDS) - }) - .collect() } diff --git a/crates/latkit/src/view/gestures.rs b/crates/latkit/src/view/gestures.rs index 3a20016f..868d4f32 100644 --- a/crates/latkit/src/view/gestures.rs +++ b/crates/latkit/src/view/gestures.rs @@ -1,22 +1,25 @@ -//! Gestures: what the pointer and keys mean. A press decides once the pointer moves a few pixels: with -//! Shift it selects in a box when boxes select, else turns a map or plane; on a port, an Alt-held -//! vertex, or an edge it draws a connection when connecting is on; on a row of a draggable mark it -//! drags the row; elsewhere it pans. A press that does not move clicks, Shift, Control, or Meta -//! toggling its row in the selection; with cycling on, a click again where nothing moved selects the -//! next row under the point. A secondary press turns once it moves and asks for a menu when it does -//! not, as a touch or a pen held still does. The wheel and pinches zoom, as the controls allow, and -//! keys step. A diagram never turns, and its drags snap -//! to its grid when it says so. Every camera move is about the point under the pointer and goes -//! through the rig, which carries a released drag on and keeps the camera within reach. +//! Gestures: how a view interprets pointer and key input. //! -//! The point under a press comes from the latest probe when it was taken where the press is, -//! otherwise from a probe of its own a frame or two later; the press waits for it and catches up. -//! Menus and connections tell what lies under their point once a pick answers, in their place among -//! the intents. +//! A press decides what it does once the pointer moves a few pixels. With Shift it selects a box +//! when box selection is on, otherwise it turns a map or plane. On a port, a vertex with Alt held, +//! or an edge, it draws a connection when connecting is on. On a row of a draggable mark it drags +//! the row. Elsewhere it pans. A press that does not move clicks. With Shift, Control, or Meta it +//! toggles its row in the selection. With cycling on, clicking again where nothing moved selects +//! the next row under the point. A secondary press turns once it moves and opens a menu when it +//! does not, as does a touch or pen held still. The wheel and pinches zoom, as the controls allow, +//! and keys step. A diagram never turns, and its drags snap to its grid when the grid snaps. Every +//! camera move is about the point under the pointer and goes through the rig, which lets a released +//! drag coast and keeps the camera within its zoom and pan limits. //! -//! The keys point too. Tab steps through the rows nearest the view's middle, and past the last one -//! lets focus move on; an arrow steps to the nearest row its way; Enter opens the row and Space -//! selects it. The row the keys point at is hovered until the pointer acts. +//! The point under a press comes from the latest probe if it was taken where the press is, +//! otherwise from its own probe a frame or two later. The press waits for it, then catches up. +//! Menus and connections report the rows under their point once a pick resolves, in their place +//! among the intents. +//! +//! Keys can point at rows too. Tab steps through the rows nearest the view's middle, and past the +//! last one lets focus leave the view. An arrow steps to the nearest row in its direction. Enter +//! opens the row and Space toggles it in the selection. The row the keys point at stays hovered +//! until the pointer acts. use super::View; use super::event::{Awaited, Cursor, End, Event, Fit, Hit, Phase, Trigger}; @@ -33,9 +36,9 @@ const DOUBLE: f64 = 0.4; const DOUBLE_REACH: f32 = 6.0; /// A drag released this many seconds after its last move does not coast. const HELD: f64 = 0.06; -/// Degrees a pixel of turning drag turns. +/// Degrees turned per CSS pixel of turning drag. const TURN: f64 = 0.3; -/// What a step of the keys moves: CSS pixels, degrees, levels. +/// How far one key step moves: CSS pixels of pan, degrees of turn, and zoom levels. const STEP: f64 = 100.0; const STEP_TURN: f64 = 15.0; const STEP_ZOOM: f64 = 0.5; @@ -44,9 +47,9 @@ const STEP_ZOOM: f64 = 0.5; pub(crate) enum Gesture { #[default] Idle, - /// A pointer pressed at `from`, now at `trigger`; `answer` is `None` while its probe is in - /// flight. `earlier` is a press released here before its own answer came, which clicks first on - /// this one's. + /// A pointer pressed at `from`, now at `trigger`. `answer` is `None` while its probe is in + /// flight. `earlier` is a previous press released here before its own answer arrived. It clicks + /// first, using this press's answer. Press { pointer: u32, button: Button, @@ -58,12 +61,12 @@ pub(crate) enum Gesture { /// When it began, on the view's clock. since: f64, }, - /// A touch or pen held still until it asked for a menu: its release does nothing more. + /// A touch or pen held still long enough to open a menu. Its release does nothing more. Held { pointer: u32, }, - /// A connection drawn from `from` toward the pointer at `to`; `anchor` is where it starts on - /// screen, once known. + /// A connection drawn from `from` toward the pointer at `to`. `anchor` is where it starts on + /// screen, once known. `locating` finds it when the press did not give it. Connect { pointer: u32, from: End, @@ -72,8 +75,8 @@ pub(crate) enum Gesture { locating: Option>>, to: [f32; 2], }, - /// A row dragged by `pointer`: `grab` is where the row was hit less where the press was, CSS - /// pixels, so the row keeps where it was grabbed. + /// A row dragged by `pointer`. `grab` is the hit point minus the press point, in CSS pixels, so + /// the row keeps its offset from the pointer. Drag { pointer: u32, probed: Probed, @@ -92,7 +95,8 @@ pub(crate) enum Gesture { Pinch { pointers: [(u32, [f32; 2]); 2], }, - /// A Shift drag where boxes select: the rows in the box between the press and the pointer. + /// A Shift drag with box selection on, selecting the rows in the box between the press and the + /// pointer. Box { pointer: u32, from: [f32; 2], @@ -105,31 +109,32 @@ pub(crate) struct Gestures { pub(crate) now: Gesture, /// The rows the keys step through. pub(crate) keys: Stepping, - /// The time of the input being taken, on the clock of whatever feeds the view its input. + /// The time of the input being handled, on the clock of whatever feeds the view its input. time: f64, /// The last click: when, where, on which row, and where the camera was. clicked: Option<(f64, [f32; 2], Option, Camera)>, - /// How fast the pan or turn under way moves, per second, and when it last moved. + /// How fast the current pan or turn moves, per second, and when it last moved. velocity: [f64; 2], moved: Option, } -/// The rows the keys step through: what a pick around the view's middle found, nearest first, as -/// the frame it was taken in drew them; the row stepped to, which holds the hover until the pointer -/// acts; and keys pressed while the rows were being picked. +/// The rows the keys step through. Holds the hits of a pick around the view's middle, nearest +/// first, as drawn in the frame it was taken in. Also holds the index stepped to, which keeps the +/// hover until the pointer acts, and keys pressed while the pick was pending. #[derive(Default)] pub(crate) struct Stepping { - /// The rows last picked; `None` until the keys first step. + /// The rows last picked. `None` until the keys first step. rows: Option>, picking: Option>>, - /// Whether the rows are where the frame now draws them. + /// Whether the rows match where the current frame draws them. fresh: bool, pub(crate) at: Option, queued: Vec<(Key, Modifiers)>, } impl Stepping { - /// The frame changed: the rows are picked again before an arrow steps. + /// Marks the rows stale after the frame changed, so they are picked again before an arrow + /// steps. pub(crate) fn stale(&mut self) { self.fresh = false; } @@ -155,7 +160,7 @@ impl Gestures { } } - /// Follows the velocity of a move of `delta` per second-scaled step. + /// Blends a move of `delta` into the velocity, smoothed over about 50 ms of input time. fn track(&mut self, delta: [f64; 2]) { let dt = self.moved.map_or(0.0, |at| self.time - at); self.moved = Some(self.time); @@ -165,7 +170,8 @@ impl Gestures { } } - /// The velocity a release coasts with: none when the pointer stopped before letting go. + /// The velocity a release coasts with: zero if the pointer never moved, or stopped for over + /// `HELD` before the release. fn release(&mut self) -> [f64; 2] { let held = self.moved.is_none_or(|at| self.time - at > HELD); let velocity = std::mem::take(&mut self.velocity); @@ -175,10 +181,10 @@ impl Gestures { } impl View { - /// Takes one pointer, wheel, or key input at `time`, seconds on the clock its frames give; - /// events it causes queue for [`events`](View::events). Double clicks and flings measure the - /// times between inputs. Returns whether the view used it, so a platform stops its default - /// handling: scrolling the page, its context menu, focus moving on. + /// Handles one pointer, wheel, or key input at `time`, in seconds on the same clock as frame + /// times. Events it causes queue for [`events`](View::events). Double clicks and flings measure + /// the time between inputs. Returns whether the view used the input, so the platform can skip + /// its default handling: scrolling the page, its context menu, or moving focus. pub fn input(&mut self, input: &Input, time: f64) -> bool { if self.size[0] <= 0.0 { return false; @@ -214,8 +220,8 @@ impl View { used } - /// The pointer acts at `trigger`: what reads it evaluates again, and it takes the hover back from - /// the keys. + /// Records the pointer at `trigger`: channels reading the pointer evaluate again, and the + /// pointer takes the hover back from the keys. fn follow_pointer(&mut self, trigger: Trigger) { self.pointer = Some(trigger); self.gestures.keys.at = None; @@ -224,12 +230,14 @@ impl View { } } - /// Where the pointer last acted, the keys it held, and what it is. + /// Where the pointer last acted, the modifier keys held, and its device. The default once it + /// leaves. fn pointing(&self) -> Trigger { self.pointer.unwrap_or_default() } - /// The pointer left: nothing it hovered stays hovered. + /// Handles the pointer leaving. If no gesture is in progress, its probes are dropped, and the + /// hover clears unless the keys point at a row. fn left(&mut self) { let trigger = self.pointing(); self.pointer = None; @@ -297,14 +305,14 @@ impl View { let delta = [f64::from(point[0] - from[0]), f64::from(point[1] - from[1])]; self.gestures.track(delta); let camera = self.camera.panned(&self.projected(), anchor, delta); - self.steer(camera); + self.move_camera(camera); } Gesture::Turn { pointer: p, anchor, last } if *p == pointer => { let (anchor, from) = (*anchor, std::mem::replace(last, point)); let degrees = [f64::from(point[0] - from[0]) * TURN, -f64::from(point[1] - from[1]) * TURN]; self.gestures.track(degrees); let camera = self.camera.turned(&self.projected(), anchor, degrees); - self.steer(camera); + self.move_camera(camera); } Gesture::Pinch { pointers } => { let Some(i) = pointers.iter().position(|(p, _)| *p == pointer) else { return }; @@ -320,8 +328,9 @@ impl View { let delta = [f64::from(to[0] - from[0]), f64::from(to[1] - from[1])]; let camera = self.camera.panned(&self.projected(), anchor, delta); let view = self.projected_with(&camera); - let camera = camera.zoomed(&view, anchor, f64::from(spread(after) / spread(before)).log2()); - self.steer(camera); + let levels = f64::from(spread(after) / spread(before)).log2(); + let camera = camera.zoomed(&view, anchor, levels, &self.reach()); + self.move_camera(camera); } _ => {} } @@ -382,8 +391,8 @@ impl View { true } - /// A probe landed: a press waiting for it decides; otherwise it is what the pointer hovers, - /// unless the keys point at a row. + /// Handles a probe's answer at `point`. A press waiting for it decides what to do. Otherwise + /// the answer sets the hover, unless the keys point at a row. pub(super) fn landed(&mut self, point: [f32; 2], answer: Answer) { if let Gesture::Press { from, answer: waiting @ None, .. } = &mut self.gestures.now && near(*from, point, 0.5) @@ -400,19 +409,19 @@ impl View { } } - /// Points at `hit`, telling the application when the row changed. + /// Hovers `hit`, reporting a change when the row differs. fn hover(&mut self, hit: Option, trigger: Trigger) { if hit.as_ref().map(|h| &h.row) != self.focus.hovered.as_ref().map(|h| &h.row) { self.focus.dirty = true; - self.events.tell(Event::Hover { hit: hit.clone(), trigger }); + self.events.tell(Event::Hovered { hit: hit.clone(), trigger }); self.wake(); } self.focus.hovered = hit; } - /// Decides a press: once it moves with Shift and boxes select, a box, whatever lies under it; - /// otherwise once what lies under it is known, a release clicks or asks for a menu, and a move - /// connects, drags, pans, or turns. + /// Decides what a press does. A Shift drag with box selection on starts a box, regardless of + /// what lies under it. Otherwise, once the probe's answer is known, a release clicks or opens a + /// menu, and a move connects, drags, pans, or turns. fn pressed(&mut self) { let Gesture::Press { pointer, button, from, trigger, answer, released, earlier, .. } = &mut self.gestures.now else { @@ -462,7 +471,7 @@ impl View { && let Some((from, edge, anchor)) = answer.hit.as_ref().and_then(|probed| self.connect_from(probed, trigger.modifiers)) { - let locating = anchor.is_none().then(|| self.locate(&from.row)); + let locating = anchor.is_none().then(|| self.screen_of(&from.row)); self.gestures.now = Gesture::Connect { pointer, from, edge, anchor, locating, to }; return; } @@ -479,13 +488,13 @@ impl View { let delta = [f64::from(to[0] - from[0]), f64::from(to[1] - from[1])]; self.gestures.now = Gesture::Pan { pointer, anchor, last: to }; let camera = self.camera.panned(&self.projected(), anchor, delta); - self.steer(camera); + self.move_camera(camera); } _ => self.gestures.now = Gesture::Idle, } } - /// A press let go where it began: a click, or a menu for the secondary button. + /// A press released where it began: a click, or a menu for the secondary button. fn tap(&mut self, button: Button, hit: Option, trigger: Trigger) { if button == Button::Secondary { self.menu(trigger); @@ -513,7 +522,7 @@ impl View { self.events.tell(Event::Open { hit: hit.clone(), trigger }); } if again && !double && !toggle && self.scene.controls.cycle { - // The next row under the point, once a pick lists them nearest first. + // Select the next row under the point once a pick lists them, nearest first. let hits = self.pick(trigger.point, self.scene.controls.pick_radius); let after = self.focus.selection.last().cloned(); self.events.tell_after(hits, Awaited::Cycle { after, trigger }); @@ -527,7 +536,7 @@ impl View { } } - /// Selects `row` alone, or with `toggle` adds it to the selection or takes it out. + /// Selects `row` alone, or with `toggle` toggles it in the selection. fn select_row(&mut self, row: Row, toggle: bool, trigger: Trigger) { let mut selection = if toggle { self.focus.selection.clone() } else { Vec::new() }; match selection.iter().position(|r| *r == row) { @@ -540,18 +549,19 @@ impl View { self.reselect(selection, Some(trigger)); } - /// Selects `selection` for the user, or without a trigger for the data, telling the application - /// when it changed. + /// Sets the selection, reporting a change if it differs. `trigger` is `None` when the data, + /// not the user, changed it. pub(super) fn reselect(&mut self, selection: Vec, trigger: Option) { if selection != self.focus.selection { self.focus.selection.clone_from(&selection); self.focus.dirty = true; - self.events.tell(Event::Select { rows: Rows::group(selection), trigger }); + self.events.tell(Event::Selection { rows: Rows::group(selection), trigger }); } } - /// Tells where the row dragged goes with the pointer at `trigger`: where it was hit, moved as the - /// pointer moved since the press, at the depth the drag started; on a diagram's dots when it snaps. + /// Reports where the dragged row goes with the pointer at `trigger`: the hit point moved with + /// the pointer since the press, at the depth where the drag started, snapped to a diagram's + /// grid dots when the grid snaps. fn drag(&mut self, phase: Phase, trigger: Trigger) { let Gesture::Drag { probed, grab, .. } = &self.gestures.now else { return }; let view = self.projected(); @@ -568,7 +578,7 @@ impl View { self.events.tell(Event::Drag { phase, hit: probed.hit.clone(), position, trigger }); } - /// Whether a row is being dragged: what it moves follows at once. + /// Whether a row is being dragged. Rows it moves then update without easing. pub(super) fn dragging(&self) -> bool { matches!(self.gestures.now, Gesture::Drag { .. }) } @@ -578,13 +588,14 @@ impl View { self.scene.controls.turn && !matches!(self.scene.space, Space::Diagram(_)) } - /// Whether the rows of mark `mark` select. + /// Whether clicking a row of mark `mark` selects it. fn selects(&self, mark: usize) -> bool { self.scene.marks.get(mark).is_some_and(|m| m.style().interaction.select) } - /// Where a connection starts from what a press hit: a port, or a vertex with Alt, starts one - /// there; an edge with ends lets go of the end it was pressed nearer, from the one that stays. + /// Where a connection starts from what a press hit: at a port, at a vertex with Alt held, or, + /// for an edge with ends, at the end that stays (the end pressed nearer detaches). Also returns + /// the edge, and the start's screen point when known. fn connect_from(&self, probed: &Probed, modifiers: Modifiers) -> Option<(End, Option, Option<[f32; 2]>)> { let hit = &probed.hit; let shown = self.marks.get(hit.index)?; @@ -603,14 +614,16 @@ impl View { } } - /// Asks what lies under the trigger's point, for a menu told in its place once the pick answers. + /// Picks what lies under the trigger's point, for a menu reported in order once the pick + /// resolves. fn menu(&mut self, trigger: Trigger) { let hits = self.pick(trigger.point, self.scene.controls.pick_radius); self.events.tell_after(hits, Awaited::Menu(trigger)); } - /// Makes the intents whose picks answered, in their places among the intents; takes the keys' - /// rows once picked; and starts a connection being drawn where the row it stays at lies. + /// Turns intents whose picks resolved into events, in order. Also takes the keys' rows once + /// picked, anchors a connection being drawn at its start once located, and opens a menu for a + /// long press. pub(super) fn settle_gestures(&mut self) { if let Gesture::Connect { anchor, locating: Some(locating), .. } = &mut self.gestures.now && let Some(found) = locating.take() @@ -627,8 +640,8 @@ impl View { self.held_long(); } - /// The event `awaited` makes of the hits its pick found, if any: an intent, or the selection a - /// cycling click moves to. + /// The event `awaited` produces from its pick's hits, if any: an intent, or the selection + /// change of a cycling click. fn finish(&mut self, hits: Vec, awaited: Awaited) -> Option { Some(match awaited { Awaited::Menu(trigger) => Event::Menu { hits, trigger }, @@ -653,12 +666,12 @@ impl View { } self.focus.selection.clone_from(&selection); self.focus.dirty = true; - Event::Select { rows: Rows::group(selection), trigger: Some(trigger) } + Event::Selection { rows: Rows::group(selection), trigger: Some(trigger) } } }) } - /// A touch or a pen held still for the controls' press asks for a menu where it is. + /// Opens a menu where a touch or pen has been held still for the controls' `press` duration. fn held_long(&mut self) { let press = self.scene.controls.press.as_secs_f64(); let Gesture::Press { pointer, from, trigger, released: false, since, .. } = self.gestures.now else { return }; @@ -669,14 +682,16 @@ impl View { } } - /// Whether a touch or a pen is held still, toward a menu: frames go on to notice it. + /// Whether a touch or pen is pressed while long presses open menus, so frames keep running to + /// detect a long press. pub(super) fn holding(&self) -> bool { matches!(self.gestures.now, Gesture::Press { released: false, trigger, .. } if matches!(trigger.device, Device::Touch | Device::Pen)) && !self.scene.controls.press.is_zero() } - /// The wheel zooms about the point under it, as the controls allow; otherwise the page has it. + /// Zooms about the point under the wheel, as the controls allow. Otherwise leaves the wheel to + /// the page. fn wheel(&mut self, point: [f32; 2], delta: [f32; 2], modifiers: Modifiers) -> bool { let zooms = match self.scene.controls.zoom { Zoom::Wheel => true, @@ -765,9 +780,10 @@ impl View { } /// Steps the row the keys point at: Tab to the next row nearest the view's middle, Shift+Tab - /// back, an arrow to the nearest row its way. Tab steps through the rows as last picked and waits - /// only for the first pick; an arrow waits for rows picked where the frame now draws them. Past - /// either end Tab is not used, so focus moves on. Whether the key was used. + /// back, an arrow to the nearest row in its direction. Tab uses the rows as last picked and + /// waits only for the first pick. An arrow waits for rows picked where the current frame draws + /// them. Past either end, Tab is not used, so focus leaves the view. Returns whether the key + /// was used. fn step(&mut self, key: Key, modifiers: Modifiers) -> bool { let tab = key == Key::Tab; if !self.gestures.keys.fresh && self.gestures.keys.picking.is_none() { @@ -803,13 +819,13 @@ impl View { true } - /// Takes the keys' rows once their pick answers, and steps the keys that waited for them. + /// Takes the keys' rows once their pick resolves, and steps the keys that waited for them. fn settle_keys(&mut self) { let keys = &mut self.gestures.keys; let Some(rows) = keys.picking.as_ref().and_then(Pending::take) else { return }; let held = keys.row().map(|hit| hit.row.clone()); let rows = rows.unwrap_or_default(); - // The row stepped to keeps its place among the rows as now drawn. + // The row stepped to keeps its position among the newly picked rows. keys.at = held.as_ref().and_then(|row| rows.iter().position(|hit| hit.row == *row)); keys.rows = Some(rows); keys.picking = None; @@ -830,7 +846,7 @@ impl View { } } -/// The row of `rows` nearest row `i` within 45° of the arrow's way from it. +/// The row of `rows` nearest row `i` within 45° of the arrow's direction from it. fn toward(rows: &[Hit], i: usize, arrow: Key) -> Option { let way = match arrow { Key::Left => [-1.0, 0.0], diff --git a/crates/latkit/src/view/graph.rs b/crates/latkit/src/view/graph.rs index fb95f1cf..6ca42698 100644 --- a/crates/latkit/src/view/graph.rs +++ b/crates/latkit/src/view/graph.rs @@ -1,5 +1,5 @@ -//! The graph a view's data holds: the rows joined to a row across edges and nets, the row a reference -//! names, and which vertex mark an end attaches to. +//! Graph queries over a view's data: a row's neighbors across edges and nets, the row a reference +//! names, and the vertex mark an end attaches to. use rustc_hash::FxHashSet; @@ -18,12 +18,12 @@ fn reference<'t>(table: &'t Table, name: &str) -> Option<&'t Reference> { } impl View { - /// The rows joined to `row`: a vertex's neighbors across edges and nets, or an edge's or a net's - /// vertices, a set per table. It reads the references' inverses, so it costs what the row's degree - /// does. + /// The rows joined to `row`: a vertex's neighbors across edges and nets, or an edge's or net's + /// vertices, grouped by table. It reads the references' inverses, so its cost is proportional + /// to the row's degree. pub fn neighbors(&self, row: &Row) -> Vec { let mut found: Vec = Vec::new(); - let Some(data) = self.data(&row.data) else { return Vec::new() }; + let Some(data) = self.snapshot(&row.data) else { return Vec::new() }; let Some(index) = data.table(&row.table).and_then(|t| t.row(row.id.as_ref())) else { return Vec::new() }; let at = |table: &Name, r: u32| { let id = data.table(table)?.ids.id(r as usize)?; @@ -72,10 +72,11 @@ impl View { Rows::group(found) } - /// Every vertex mark's port reference field that names rows of net mark `i`'s table. + /// Each vertex mark's port reference fields that point at net mark `i`'s table, as + /// `(mark, field)`. fn ports_naming(&self, i: usize) -> Vec<(usize, Name)> { let style = self.marks[i].mark.style(); - let Some(data) = self.data(&style.data) else { return Vec::new() }; + let Some(data) = self.snapshot(&style.data) else { return Vec::new() }; let mut found = Vec::new(); for (m, shown) in self.marks.iter().enumerate() { let s = shown.mark.style(); @@ -95,14 +96,15 @@ impl View { /// The row `row`'s reference field `field` names. pub(super) fn referenced(&self, row: &Row, field: &str) -> Option { - let data = self.data(&row.data)?; + let data = self.snapshot(&row.data)?; let table = data.table(&row.table)?; let r = reference(table, field)?; let id = data.table(&r.to)?.ids.id(r.target(table.row(row.id.as_ref())?)? as usize)?; Some(Row { data: row.data.clone(), table: r.to.clone(), id }) } - /// Marks each edge mark on the earth whose route is not straight as following great circles. + /// Whether `shown` follows great circles: an edge mark with ends, on the earth, whose route is + /// unset or geodesic. pub(super) fn geodesic(&self, shown: &Shown) -> bool { let Space::Earth(_) = self.scene.space else { return false }; match &shown.mark { @@ -112,7 +114,22 @@ impl View { } } -/// The vertex mark drawing `table` that ends attach to: the unnamed one, else the first. +/// For each of `marks`, whether faces sum their normals at its rows: whether it is the vertex mark +/// some face mark's corners attach to. +pub(super) fn summing(marks: &[Shown]) -> Vec { + let mut summing = vec![false; marks.len()]; + for shown in marks.iter().filter(|m| m.kind() == MarkKind::Faces) { + let Some(compiled) = &shown.compiled else { continue }; + for end in &compiled.program.ends { + if let Some(m) = end_mark(marks, &shown.mark.style().data, &end.table) { + summing[m] = true; + } + } + } + summing +} + +/// The vertex mark drawing `table` that ends attach to: the unnamed one, or else the first. pub(super) fn end_mark(marks: &[Shown], data: &Name, table: &Name) -> Option { let drawing = |i: &usize| { let m = &marks[*i]; diff --git a/crates/latkit/src/view/input.rs b/crates/latkit/src/view/input.rs index a39688b3..1f7f7eb4 100644 --- a/crates/latkit/src/view/input.rs +++ b/crates/latkit/src/view/input.rs @@ -1,42 +1,42 @@ -//! Input as a view reads it, from any platform: a browser canvas translates its events into these. +//! Platform-independent input: a browser canvas translates its events into these. /// A pointer, wheel, or key event, with points in CSS pixels from the view's top left. #[derive(Clone, Copy, PartialEq, Debug)] pub enum Input { - /// A pointer pressed. + /// A pointer was pressed. Down { - /// Tells pointers apart while several are down. + /// Identifies the pointer while several are down. pointer: u32, - /// Where. + /// Where it was pressed. point: [f32; 2], /// Which button. button: Button, - /// Keys held. + /// Modifier keys held. modifiers: Modifiers, - /// What pressed: a finger's picks reach farther. + /// The device that pressed. A finger picks rows from farther away. device: Device, }, /// A pointer moved. Move { /// Which pointer. pointer: u32, - /// Where to. + /// Where it moved to. point: [f32; 2], - /// Keys held. + /// Modifier keys held. modifiers: Modifiers, - /// What moved. + /// The device that moved. device: Device, }, - /// A pointer let go. + /// A pointer was released. Up { /// Which pointer. pointer: u32, - /// Where. + /// Where it was released. point: [f32; 2], - /// Keys held. + /// Modifier keys held. modifiers: Modifiers, }, - /// The platform took a pointer away. + /// The platform cancelled a pointer. Cancel { /// Which pointer. pointer: u32, @@ -47,7 +47,7 @@ pub enum Input { point: [f32; 2], /// How far. delta: [f32; 2], - /// Keys held. + /// Modifier keys held. modifiers: Modifiers, }, /// A key went down or up while the view had focus. @@ -56,7 +56,7 @@ pub enum Input { key: Key, /// Whether it went down. down: bool, - /// Keys held. + /// Modifier keys held. modifiers: Modifiers, }, /// The pointer left the view. @@ -68,13 +68,13 @@ pub enum Input { pub enum Button { /// The main button: left, or a touch. Primary, - /// The context button: right. + /// The context-menu button: right. Secondary, /// The middle button or wheel press. Middle, } -/// What the user acts with. +/// The input device. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default, serde::Serialize, schemars::JsonSchema)] #[serde(rename_all = "lowercase")] pub enum Device { @@ -83,13 +83,13 @@ pub enum Device { Mouse, /// A pen. Pen, - /// A finger: picks reach 8 CSS pixels farther. + /// A finger. Picks find rows up to 8 CSS pixels farther away. Touch, - /// The keys. + /// The keyboard. Keyboard, } -/// Keys held during an input. +/// Modifier keys held during an input. #[derive(Clone, Copy, PartialEq, Eq, Debug, Default, serde::Serialize, schemars::JsonSchema)] pub struct Modifiers { /// Shift. @@ -103,13 +103,13 @@ pub struct Modifiers { } impl Modifiers { - /// Whether a click adds its row to the selection or takes it out: Shift, Control, or Meta held. + /// Whether a click toggles its row in the selection: Shift, Control, or Meta is held. pub fn toggles(self) -> bool { self.shift || self.control || self.meta } } -/// The keys a view answers. +/// The keys a view responds to. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub enum Key { /// Escape. diff --git a/crates/latkit/src/view/kept.rs b/crates/latkit/src/view/kept.rs new file mode 100644 index 00000000..0d726250 --- /dev/null +++ b/crates/latkit/src/view/kept.rs @@ -0,0 +1,197 @@ +//! The kept-row table: a drawing's kept rows, keyed by their place in its order. +//! +//! It is an open-addressed hash table with linear probing. Each bucket holds a place and its held +//! row, `[NONE, NONE]` when empty. A new table is built three fifths full, with at least `LEAST` +//! buckets and any count above that, not just powers of two. Updating from a previous table +//! removes the rows that left and inserts those that came, as long as it stays between three +//! tenths and four fifths full. Only their buckets, and the few after a removed row that shift +//! back, change, so the end table's words change only where rows did. Past those bounds the table +//! is rebuilt, slightly larger or smaller, never larger than needed. A lookup ends within a few +//! buckets, on the GPU (`lib/ends.wesl`) as here. + +use std::sync::Arc; + +use crate::gpu::shaders as wgsl; + +/// The fewest buckets a table has once it has any. +const LEAST: usize = 64; +const EMPTY: [u32; 2] = [wgsl::NONE, wgsl::NONE]; + +/// A drawing's kept rows: buckets of `[place, held row]`, as the module docs describe. +#[derive(Clone, Default, Debug, PartialEq)] +pub(crate) struct KeptTable(Arc<[[u32; 2]]>); + +impl KeptTable { + /// The table of `rows`, each `[place, held row]`, updated from `before`. If `before`'s size + /// suits `rows`, its buckets are reused with places `gone` removed and rows `came` inserted. + /// Otherwise the table is rebuilt. + pub(crate) fn laid(before: &KeptTable, rows: &[[u32; 2]], gone: &[u32], came: &[[u32; 2]]) -> KeptTable { + let buckets = before.0.len(); + // A table of the fewest buckets is updated in place however few rows it keeps. + if buckets > 0 && rows.len() * 5 <= buckets * 4 && (rows.len() * 10 >= buckets * 3 || buckets == LEAST) { + let mut table = before.0.to_vec(); + for &d in gone { + remove(&mut table, d); + } + for &row in came { + insert(&mut table, row); + } + return KeptTable(table.into()); + } + if rows.is_empty() { + return KeptTable::default(); + } + let mut table = vec![EMPTY; (rows.len() * 5).div_ceil(3).max(LEAST)]; + for &row in rows { + insert(&mut table, row); + } + KeptTable(table.into()) + } + + /// The held row of place `d`, if kept. + pub(crate) fn held(&self, d: u32) -> Option { + if self.0.is_empty() { + return None; + } + let mut b = home(d, self.0.len()); + loop { + match self.0[b] { + [place, held] if place == d => return Some(held), + [wgsl::NONE, _] => return None, + _ => b = next(b, self.0.len()), + } + } + } + + /// Its buckets, as the end table packs them. + pub(crate) fn buckets(&self) -> &[[u32; 2]] { + &self.0 + } +} + +/// The bucket, of `buckets`, where a probe for place `d` starts: a hash of `d`'s bits. Must match +/// `home` in `lib/ends.wesl`. +fn home(d: u32, buckets: usize) -> usize { + let mut h = d ^ (d >> 16); + h = h.wrapping_mul(0x7feb_352d); + h ^= h >> 15; + h = h.wrapping_mul(0x846c_a68b); + (h ^ (h >> 16)) as usize % buckets +} + +/// The bucket after `b`, wrapping to 0. +fn next(b: usize, buckets: usize) -> usize { + if b + 1 == buckets { 0 } else { b + 1 } +} + +/// Puts `row` in the first empty bucket from where its place's probe starts. +fn insert(table: &mut [[u32; 2]], row: [u32; 2]) { + let mut b = home(row[0], table.len()); + while table[b][0] != wgsl::NONE { + b = next(b, table.len()); + } + table[b] = row; +} + +/// Removes place `d`, shifting back each later row whose probe passed its bucket, so every probe +/// still ends at the first empty bucket. +fn remove(table: &mut [[u32; 2]], d: u32) { + let mut hole = home(d, table.len()); + while table[hole][0] != d { + if table[hole][0] == wgsl::NONE { + return; + } + hole = next(hole, table.len()); + } + let mut b = hole; + loop { + b = next(b, table.len()); + let [place, _] = table[b]; + if place == wgsl::NONE { + break; + } + // A row whose probe starts after the hole and at or before its own bucket stays: its probe + // never passed the hole. + let home = home(place, table.len()); + let stays = if hole <= b { hole < home && home <= b } else { hole < home || home <= b }; + if !stays { + table[hole] = table[b]; + hole = b; + } + } + table[hole] = EMPTY; +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use super::*; + + /// The next pseudo-random number below `n`, advancing `seed`. + fn random(seed: &mut u64, n: u32) -> u32 { + *seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1_442_695_040_888_963_407); + ((*seed >> 33) % u64::from(n)) as u32 + } + + #[test] + fn a_table_holds_every_kept_row_and_no_other() { + let mut seed = 7; + let mut kept: BTreeMap = BTreeMap::new(); + let mut table = KeptTable::default(); + for round in 0..200 { + // Each round, places come or go, each at most once. + let (mut gone, mut came) = (Vec::new(), Vec::new()); + for _ in 0..random(&mut seed, 40) { + let d = random(&mut seed, 5_000); + if gone.contains(&d) || came.iter().any(|row: &[u32; 2]| row[0] == d) { + continue; + } + if kept.remove(&d).is_some() { + gone.push(d); + } else { + kept.insert(d, round); + came.push([d, round]); + } + } + let rows: Vec<[u32; 2]> = kept.iter().map(|(&d, &held)| [d, held]).collect(); + table = KeptTable::laid(&table, &rows, &gone, &came); + let buckets = table.buckets().len(); + assert!(buckets == 0 || kept.len() * 5 <= buckets * 4, "at most four fifths full"); + for d in 0..5_000 { + assert_eq!(table.held(d), kept.get(&d).copied(), "place {d} after round {round}"); + } + } + } + + #[test] + fn rows_coming_and_going_change_only_nearby_buckets() { + let rows: Vec<[u32; 2]> = (0..1_000).map(|d| [d * 7, d]).collect(); + let table = KeptTable::laid(&KeptTable::default(), &rows, &[], &[]); + let gone: Vec = (0..10).map(|d| d * 70).collect(); + let came: Vec<[u32; 2]> = (0..10).map(|d| [100_000 + d, d]).collect(); + let now: Vec<[u32; 2]> = + rows.iter().filter(|r| !gone.contains(&r[0])).copied().chain(came.iter().copied()).collect(); + let next = KeptTable::laid(&table, &now, &gone, &came); + assert_eq!(next.buckets().len(), table.buckets().len(), "updated in place"); + let changed = table.buckets().iter().zip(next.buckets()).filter(|(a, b)| a != b).count(); + assert!(changed <= 60, "{changed} buckets changed for 20 rows that came or went"); + } + + #[test] + fn a_table_is_rebuilt_past_four_fifths_full_and_under_three_tenths() { + let rows = |n: u32| (0..n).map(|d| [d, d]).collect::>(); + let small = KeptTable::laid(&KeptTable::default(), &rows(10), &[], &[]); + assert_eq!(small.buckets().len(), LEAST, "few rows get the fewest buckets"); + let full = KeptTable::laid(&small, &rows(100), &[], &(10..100).map(|d| [d, d]).collect::>()); + assert_eq!(full.buckets().len(), 167, "past four fifths of 64, rebuilt three fifths full"); + let more = KeptTable::laid(&full, &rows(130), &[], &(100..130).map(|d| [d, d]).collect::>()); + assert_eq!(more.buckets().len(), 167, "up to four fifths full, updated in place"); + let gone: Vec = (51..130).collect(); + assert_eq!(KeptTable::laid(&more, &rows(51), &gone, &[]).buckets().len(), 167, "three tenths, in place"); + let gone: Vec = (20..130).collect(); + assert_eq!(KeptTable::laid(&more, &rows(20), &gone, &[]).buckets().len(), LEAST, "under three tenths, rebuilt"); + let gone: Vec = (0..130).collect(); + assert_eq!(KeptTable::laid(&more, &[], &gone, &[]), KeptTable::default(), "no rows, no buckets"); + } +} diff --git a/crates/latkit/src/view/labels.rs b/crates/latkit/src/view/labels.rs index 3a4a529c..a84562f7 100644 --- a/crates/latkit/src/view/labels.rs +++ b/crates/latkit/src/view/labels.rs @@ -1,94 +1,104 @@ -//! Labels: each labeled mark keeps, in the labels buffer, a candidate per row and a run table from -//! its text codes to where the atlas holds each text, or, until the text is set, an estimate of its -//! size. Draws place every label from its anchor each frame. Placement decides which show: it runs -//! on the GPU when the camera, a labeled mark's state, or a run table changed, as candidates, then -//! rounds of claim and decide on a grid of cells, and labels fade in and out after it. Only placed -//! labels ask for their texts, so the host sets no more texts than fit on screen however many rows -//! are labeled. A diagram's labels scale with the sheet and show wherever they fit, so there -//! placement only gathers the texts the atlas lacks. Those are read back, with the texts of labels -//! shown, which the atlas keeps; the host sets a few each frame, and they show the frame after. +//! Labels. +//! +//! In the labels buffer, each labeled mark keeps a run table that maps its text codes to where the +//! atlas holds each text. Both words are 0 until the text is set. Draws place every label from its +//! anchor each frame. A label shows when its text is set, its row is visible, and the text fits, +//! for example inside a block. No label depends on another label or on an earlier frame. A row +//! whose text is missing has no label. +//! +//! Labels beside their rows compete for screen space. When the camera, a labeled mark's places, or +//! a run table changes, the claim pass writes each such label's claim into the screen cells its +//! text covers. Until its text is set, a label claims the cells a short text would cover. Each cell +//! keeps the claim of the row drawn on top. A label shows only if it kept every cell it covers, so +//! labels never overlap. Only labels that keep their cells need texts, so however large the data, +//! the texts in view are only as many as fit on the screen. The claims depend on that frame alone. +//! +//! The texts pass then finds the texts those labels need. Each label near the view whose text the +//! atlas lacks asks for it, and each one in view whose text is set marks it seen. The host reads +//! both back, sets a few of the asked texts each frame within a budget, and touches the seen ones +//! so the atlas keeps them. A text shows the frame after it is set. When the atlas has no space +//! left beyond what the texts in view use, the rest wait. A run table grows with its mark's texts, +//! and the new one starts with every text already set, so labels keep showing. use std::collections::HashMap; -use std::ops::Range; +use std::sync::Arc; use std::time::Duration; use web_time::Instant; use super::View; use super::passes::{dispatch, offset}; -use super::resources::EyeSlot; -use super::sync::{Source, table_of}; -use crate::Space; +use super::sync::Source; +use super::variants::MarkVariants; +use crate::Name; use crate::derive::{Derived, stamp_of}; use crate::gpu::Inner; use crate::gpu::atlas::{self, Glyphs}; +use crate::gpu::dictionary::Dictionary; use crate::gpu::pipelines::Variant; use crate::gpu::readback::Readback; use crate::gpu::shaders::{self as wgsl, Shader}; use crate::host::TextRun; -/// Rounds of claim and decide each placement runs. -const ROUNDS: u32 = 6; -/// Words of the select capping a mark's labels: a digit's buckets, then its state. -const SELECT_WORDS: u32 = 2048 + 3; -/// Digits of a key the select finds, one pass each. -const DIGITS: u32 = 3; -/// Seconds labels fade in or out over. -const FADE: f64 = 0.2; -/// Texts the host sets per frame at most, and the time setting them and making their distance -/// fields may take. +/// The most texts the host sets per frame, and the most time setting them and building their +/// distance fields may take. const BUDGET: usize = 48; const BUDGET_TIME: Duration = Duration::from_millis(4); /// Where one labeled mark's words lie in the labels buffer. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) struct Slots { - pub(crate) candidates: u32, pub(crate) runs: u32, pub(crate) wanted: u32, pub(crate) seen: u32, - pub(crate) id: u32, - /// The select capping the mark's labels, or `NONE` without a cap. - pub(crate) select: u32, - /// Rows and codes the slots hold. - rows: u32, + /// The number of codes its run table and seen bits have space for. codes: u32, } +/// How a labeled mark sets its texts: the field its codes index, as dataset, table, and field, and +/// the font, size, and wrap width of its runs. A run table holds texts set one way. +#[derive(Clone, PartialEq, Debug)] +struct Setting { + field: [Name; 3], + font: Arc, + size: f32, + max_width: Option, +} + +impl Setting { + /// The dictionary of the field's codes. + fn dictionary<'g>(&self, inner: &'g mut Inner) -> &'g mut Dictionary { + let [data, table, field] = &self.field; + inner.dictionaries.field(data, table, field) + } +} + #[derive(Default)] pub(crate) struct Labels { - /// Each labeled mark's slots, by mark. + /// Each labeled mark's slots and setting, by mark. slots: Vec>, - /// The cell grid: where it starts, its columns, and its rows. - pub(crate) cells: [u32; 3], - words: u32, - /// What placement last ran from, and the readback of the texts it asked for. - placed: Derived<()>, - /// The labeled marks' slots when placement asked. + settings: Vec>, + /// The stamp the claim pass last ran from. + claimed: Option, + /// The stamp the texts pass last ran from, and the readback of the texts it gathered. + gathered: Derived<()>, + /// The labeled marks' slots when the texts pass ran. asked: Vec<(usize, Slots)>, - /// Labels fade until then, on the view's clock. - fading: Option, - /// Whether the last placement that landed asked for texts, which the next one shows. + /// Whether the last readback led to texts being set (or left to set), so the next texts pass + /// runs to ask again. wanting: bool, - /// Each labeled mark's written codes, with the run and where it lay. + /// Each labeled mark's codes whose texts are set, with the run and its glyphs in the atlas. written: HashMap>, - /// Each labeled mark's codes whose run table entries hold a run or an estimate. - estimated: HashMap, evictions: u64, /// Bumps whenever a run table changes. changes: u64, } impl Labels { - /// Whether every label placed shows its text: no placement is read back, and the last asked - /// for no text it lacked. + /// Whether every label in view shows its text: no texts pass is being read back, and the last + /// one asked for no missing text. pub(crate) fn complete(&self) -> bool { - !self.placed.busy() && !self.wanting - } - - /// Whether labels are fading in or out at `now` on the view's clock. - pub(crate) fn fading(&self, now: f64) -> bool { - self.fading.is_some_and(|until| now < until) + !self.gathered.busy() && !self.wanting } pub(crate) fn slots(&self, mark: usize) -> Option { @@ -97,41 +107,43 @@ impl Labels { } impl View { - /// The run each code of mark `i`'s label text is set as: at the base size, whatever size it shows - /// at, as a diagram's sheet measures it. - fn run_of(&self, i: usize, code: u32, inner: &mut Inner) -> Option { + /// How mark `i` sets its label texts, wherever its text field is: in the mark's table, a joined + /// one, or a table derived beside it. `None` while the mark has no label or the table with its + /// text is missing. + fn setting(&self, i: usize) -> Option { let shown = &self.marks[i]; let compiled = shown.compiled.as_ref()?; let labeling = compiled.drawing.label.as_ref()?; let column = &compiled.program.columns[labeling.column as usize]; - let style = shown.mark.style(); - let data = self.drawn(&style.data)?; - let source = Source { data, table: data.table(&style.table)?, derived: [None; 2] }; - let (d, t) = table_of(column, &source, &self.data)?; - let text = inner.dictionaries.field(d, &t.name, &column.name).text(code)?.clone(); - Some(atlas::run(text, labeling.font.clone(), labeling.size, labeling.max_width)) + let source = Source::of(shown.mark.style(), &self.snapshots, &self.derived)?; + let (data, table) = source.table_of(column, &self.snapshots)?; + Some(Setting { + field: [data.clone(), table.name.clone(), column.name.clone()], + font: labeling.font.clone(), + size: labeling.size, + max_width: labeling.max_width, + }) } - /// Codes mark `i`'s label text has so far. + /// The text run for code `code` of mark `i`'s label text, at the base size regardless of the + /// size it shows at, as a diagram's sheet measures it. + fn run_of(&self, i: usize, code: u32, inner: &mut Inner) -> Option { + let setting = self.setting(i)?; + let text = setting.dictionary(inner).text(code)?.clone(); + Some(atlas::run(text, setting.font, setting.size, setting.max_width)) + } + + /// The number of codes mark `i`'s label text has so far. fn codes_of(&self, i: usize, inner: &mut Inner) -> u32 { - let shown = &self.marks[i]; - let Some(compiled) = &shown.compiled else { return 0 }; - let Some(labeling) = &compiled.drawing.label else { return 0 }; - let column = &compiled.program.columns[labeling.column as usize]; - let style = shown.mark.style(); - let Some(data) = self.drawn(&style.data) else { return 0 }; - let Some(table) = data.table(&style.table) else { return 0 }; - let source = Source { data, table, derived: [None; 2] }; - let Some((d, t)) = table_of(column, &source, &self.data) else { return 0 }; - inner.dictionaries.field(d, &t.name, &column.name).len() as u32 + self.setting(i).map_or(0, |setting| setting.dictionary(inner).len() as u32) } - /// Sets the texts a placement asked for, within the frame's budget; placement asks again for - /// those left. + /// Sets the texts the last texts pass asked for, within the frame's budget. The next pass asks + /// for any left over. pub(super) fn settle_labels(&mut self, inner: &mut Inner) { - let Some((_, bytes)) = self.labels.placed.landed() else { return }; + let Some((_, bytes)) = self.labels.gathered.landed() else { return }; let asked = std::mem::take(&mut self.labels.asked); - let words: &[u32] = bytemuck::cast_slice(&bytes); + let words = crate::wire::words(&bytes); let started = Instant::now(); let (mut set, mut left) = (0, false); 'marks: for (i, slots) in asked { @@ -139,7 +151,8 @@ impl View { continue; } let first = self.labels.slots.iter().flatten().map(|s| s.wanted).min().unwrap_or(0); - // Texts labels show stay in the atlas. + // Touch the texts shown labels use, so the atlas keeps them. The seen bit of a text not + // set marks it asked for, and touches nothing. let seen = (slots.seen - first) as usize; if let (Some(bits), Some(written)) = (words.get(seen..seen + (slots.codes / 32) as usize), self.labels.written.get(&i)) @@ -166,10 +179,22 @@ impl View { break 'marks; } let Some(run) = self.run_of(i, code, inner) else { continue }; + // An empty text gives no label, so it claims no screen cells. A host pads even an + // empty run, so it is never rasterized. + let blank = run.text.trim().is_empty(); let host = inner.host.clone(); - let Some(glyphs) = inner.atlas.text(host.as_deref(), &run) else { continue }; + let glyphs = if blank { None } else { inner.atlas.text(host.as_deref(), &run) }; + // With no atlas space left this frame, the remaining texts are asked for again once + // the view changes. + if glyphs.is_none() && !blank && inner.atlas.full() { + break 'marks; + } set += 1; - self.write_run(i, slots, code, run, glyphs); + match glyphs { + Some(glyphs) => self.write_run(i, slots, code, run, glyphs), + // A text that draws nothing gives no label too. + None => self.write_words(slots, code, [wgsl::NONE; 2]), + } } } self.labels.wanting = set > 0 || left; @@ -180,16 +205,22 @@ impl View { } fn write_run(&mut self, i: usize, slots: Slots, code: u32, run: TextRun, glyphs: Glyphs) { - let words = glyphs.corners(); + self.write_words(slots, code, glyphs.corners()); + self.labels.written.entry(i).or_default().insert(code, (run, glyphs)); + } + + /// Writes `code`'s run table entry. + fn write_words(&self, slots: Slots, code: u32, words: [u32; 2]) { if let Some(resources) = &self.resources { let at = u64::from(slots.runs + 2 * code) * 4; self.gpu.0.queue.write_buffer(resources.labels(), at, bytemuck::cast_slice(&words)); } - self.labels.written.entry(i).or_default().insert(code, (run, glyphs)); } - /// Lays each labeled mark's slots out in the labels buffer, starting it over when they moved; - /// drops run table entries the atlas evicted. + /// Lays out each labeled mark's slots in the labels buffer, and fixes run table entries whose + /// texts the atlas evicted. When the layout or a mark's setting changed, it replaces the buffer + /// and writes back every text still set as before, so labels keep showing. A mark that stopped + /// being labeled, or sets its texts another way, starts over. pub(super) fn lay_out_labels(&mut self, inner: &mut Inner) { let mut slots: Vec> = vec![None; self.marks.len()]; let labeled: Vec = (0..self.marks.len()) @@ -197,21 +228,12 @@ impl View { self.marks[i].drawable() && self.marks[i].compiled.as_ref().is_some_and(|c| c.drawing.label.is_some()) }) .collect(); - // Wanted codes, then seen codes, so one copy reads them all; then the cells; then each mark's - // candidates and run table. + // Every mark's wanted codes, then every mark's seen bits, so one copy reads them all. Then + // each mark's run table. let mut at = 0; let codes: Vec = labeled.iter().map(|&i| self.codes_of(i, inner).next_power_of_two().max(64)).collect(); for &i in &labeled { - slots[i] = Some(Slots { - candidates: 0, - runs: 0, - wanted: at, - seen: 0, - id: 0, - select: wgsl::NONE, - rows: 0, - codes: 0, - }); + slots[i] = Some(Slots { runs: 0, wanted: at, seen: 0, codes: 0 }); at += 1 + wgsl::WANTED; } for (&i, &codes) in labeled.iter().zip(&codes) { @@ -220,102 +242,60 @@ impl View { s.codes = codes; at += codes / 32; } - let columns = (self.size[0] / f64::from(wgsl::CELL)).ceil().max(1.0) as u32; - let rows = (self.size[1] / f64::from(wgsl::CELL)).ceil().max(1.0) as u32; - let cells = [at, columns, rows]; - at += columns * rows; - let mut id = 0; for &i in &labeled { - let mark_rows = self.marks[i].region.capacity; let s = slots[i].as_mut().expect("labeled"); - s.candidates = at; - s.rows = mark_rows; - at += mark_rows * wgsl::CANDIDATE_WORDS; s.runs = at; at += 2 * s.codes; - s.id = id; - id += mark_rows; - if self.marks[i].compiled.as_ref().and_then(|c| c.drawing.label.as_ref()).is_some_and(|l| l.max.is_some()) { - s.select = at; - at += SELECT_WORDS; - } } - if slots != self.labels.slots || cells != self.labels.cells { + let settings: Vec> = + (0..self.marks.len()).map(|i| slots[i].and_then(|_| self.setting(i))).collect(); + if slots != self.labels.slots || settings != self.labels.settings { let device = self.gpu.0.device.clone(); if let Some(resources) = &mut self.resources { resources.replace_labels(&device, u64::from(at)); } + let before = std::mem::take(&mut self.labels.settings); + let kept = |i: usize| settings.get(i).is_some_and(|s| s.is_some() && before.get(i) == Some(s)); + self.labels.written.retain(|&i, _| kept(i)); self.labels.slots = slots; - self.labels.cells = cells; - self.labels.words = at; - self.labels.written.clear(); - self.labels.estimated.clear(); - self.labels.placed = Derived::default(); + self.labels.settings = settings; + for &i in &labeled { + self.write_back(i); + } + self.labels.claimed = None; + self.labels.gathered = Derived::default(); self.labels.asked.clear(); self.labels.changes += 1; } if inner.atlas.evictions != self.labels.evictions { self.labels.evictions = inner.atlas.evictions; self.drop_evicted(inner); - // Port labels look for their texts again where the atlas holds them now. + // Repack port labels so they find their texts where the atlas now holds them. for shown in self.marks.iter_mut().filter(|m| m.port_count() > 0) { shown.packed = false; } } - self.estimate_runs(inner); } - /// Writes the run table entries of each labeled mark's codes new since the last frame: where the - /// atlas holds a text, else an estimate of its size, so its label is placed before it is set. - fn estimate_runs(&mut self, inner: &mut Inner) { - for i in 0..self.labels.slots.len() { - let Some(slots) = self.labels.slots[i] else { continue }; - let from = self.labels.estimated.get(&i).copied().unwrap_or(0); - let codes = self.codes_of(i, inner).min(slots.codes); - if codes <= from { - continue; - } - let words = self.run_words(i, from..codes, inner); - if let Some(resources) = &self.resources { - let at = u64::from(slots.runs + 2 * from) * 4; - self.gpu.0.queue.write_buffer(resources.labels(), at, bytemuck::cast_slice(&words)); - } - self.labels.estimated.insert(i, codes); - self.labels.changes += 1; - } - } - - /// The run table words of mark `i`'s codes `codes`: where the atlas holds each text set, and an - /// estimate of the rest. - fn run_words(&self, i: usize, codes: Range, inner: &mut Inner) -> Vec { - let shown = &self.marks[i]; - let written = self.labels.written.get(&i); - let texts = 'texts: { - let Some(compiled) = &shown.compiled else { break 'texts None }; - let Some(labeling) = &compiled.drawing.label else { break 'texts None }; - let column = &compiled.program.columns[labeling.column as usize]; - let style = shown.mark.style(); - let Some(data) = self.drawn(&style.data) else { break 'texts None }; - let Some(table) = data.table(&style.table) else { break 'texts None }; - let source = Source { data, table, derived: [None; 2] }; - let Some((d, t)) = table_of(column, &source, &self.data) else { break 'texts None }; - Some((inner.dictionaries.field(d, &t.name, &column.name), labeling)) + /// Writes the run table entries of every text set for mark `i`, as into a new labels buffer. + /// Consecutive codes write together. + fn write_back(&self, i: usize) { + let (Some(resources), Some(slots), Some(written)) = + (&self.resources, self.labels.slots(i), self.labels.written.get(&i)) + else { + return; }; - let mut words = Vec::with_capacity(2 * codes.len()); - for code in codes { - let entry = match (written.and_then(|w| w.get(&code)), &texts) { - (Some((_, glyphs)), _) => glyphs.corners(), - (None, Some((dictionary, labeling))) => dictionary.text(code).map_or([0, 0], |text| { - atlas::estimate(&atlas::run(text.clone(), labeling.font.clone(), labeling.size, labeling.max_width)) - }), - (None, None) => [0, 0], - }; - words.extend(entry); + let mut codes: Vec = written.keys().copied().filter(|&code| code < slots.codes).collect(); + codes.sort_unstable(); + for consecutive in codes.chunk_by(|a, b| *b == *a + 1) { + let words: Vec = consecutive.iter().flat_map(|code| written[code].1.corners()).collect(); + let at = u64::from(slots.runs + 2 * consecutive[0]) * 4; + self.gpu.0.queue.write_buffer(resources.labels(), at, bytemuck::cast_slice(&words)); } - words } - /// Puts estimates back in run table entries whose runs the atlas no longer holds where they were. + /// Clears run table entries whose texts the atlas evicted or moved, so their labels hide until + /// their texts are set again. fn drop_evicted(&mut self, inner: &Inner) { let mut cleared = Vec::new(); for (&i, written) in &mut self.labels.written { @@ -323,7 +303,7 @@ impl View { written.retain(|&code, (run, glyphs)| { let held = inner.atlas.holds(run, *glyphs); if !held { - cleared.push((slots.runs + 2 * code, atlas::estimate(run))); + cleared.push(slots.runs + 2 * code); } held }); @@ -332,24 +312,17 @@ impl View { return; } if let Some(resources) = &self.resources { - for (at, words) in &cleared { - self.gpu.0.queue.write_buffer(resources.labels(), u64::from(*at) * 4, bytemuck::cast_slice(words)); + for &at in &cleared { + self.gpu.0.queue.write_buffer(resources.labels(), u64::from(at) * 4, bytemuck::cast_slice(&[0_u32; 2])); } } self.labels.changes += 1; } - /// The labels' words of the frame uniform, the clock having moved `dt` seconds. - pub(super) fn label_words(&mut self, frame: &mut wgsl::Frame, dt: f64) { - let [cells, columns, rows] = self.labels.cells; - frame.cells = cells; - frame.cell_columns = columns; - frame.cell_rows = rows; - frame.label_fade = if self.moves() { (dt / FADE).min(1.0) as f32 } else { 1.0 }; - } - - /// Places labels when what they would show changed, and fades them; reads back the texts - /// placement asked for. Labels wait for what places them to compile. + /// Claims the screen cells labels cover, and runs the texts pass, when their inputs changed. + /// Claims run in every such frame, since that frame's labels show from them. The texts pass + /// waits while its last readback is in flight, and while a fit is about to move the camera. + /// Both wait until the text pipelines have compiled. pub(super) fn place_labels(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner, eye: &wgsl::Eye) { if !self.texts() { return; @@ -359,109 +332,62 @@ impl View { if labeled.is_empty() { return; } - // A diagram's labels follow the camera without placement; the camera only says which texts - // they ask for. - let diagram = matches!(self.scene.space, Space::Diagram(_)); - // What places labels: a color changing every frame moves none, and a fit about to move the - // camera places them once it did. + // What the passes read: a color changing every frame moves no label. let versions: Vec = labeled.iter().map(|&(i, _)| self.marks[i].place_version).collect(); let stamp = stamp_of((bytemuck::bytes_of(eye), self.labels.changes, versions)); - let due = self.labels.placed.due(stamp) && !self.labels.placed.busy() && !self.fitting.framing(); - let now = self.clock.now(); - let fading = !diagram && self.labels.fading(now); - if !due && !fading { + let claim = self.labels.claimed != Some(stamp); + let gather = self.labels.gathered.due(stamp) && !self.labels.gathered.busy() && !self.fitting.framing(); + if !claim && !gather { return; } let gpu = self.gpu.clone(); let device = &gpu.0.device; let resources = self.resources.as_mut().expect("created before labels"); - if due { - self.labels.placed.mark(stamp); - if !diagram { - self.labels.fading = Some(now + FADE + 0.05); - } - for &(_, slots) in &labeled { - encoder.clear_buffer(resources.labels(), u64::from(slots.wanted) * 4, Some(4)); - encoder.clear_buffer( - resources.labels(), - u64::from(slots.seen) * 4, - Some(u64::from(slots.codes / 32) * 4), - ); - } - } + // The grid spans the view and a margin around it, a word per cell. One more cell each way + // covers rounding. + let cells = + |side: f32| u64::from(((side + 2.0 * wgsl::LABEL_MARGIN * side) / wgsl::LABEL_CELL).ceil() as u32 + 1); + let words = cells(eye.size[0]) * cells(eye.size[1]); + let grid = resources.label_cells(device, words).clone(); let groups = resources.groups(&gpu.0, inner); - let measuring = resources.pass(device, inner, Shader::Candidates); - let group = resources.pass(device, inner, Shader::Labels); - let [_, columns, rows] = self.labels.cells; - { - let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { - label: Some("latkit labels"), - timestamp_writes: None, - }); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Main.offset()]); - let marks: Vec<(usize, u32, u32)> = - labeled.iter().map(|&(i, _)| (i, self.marks[i].rows, self.marks[i].path())).collect(); - // Candidates are measured where their anchors put them, by a program made for each mark's - // primitive; placement reads what it wrote. - if due { - pass.set_bind_group(2, &measuring, &[]); - for &(i, rows, path) in &marks { - let primitive = self.marks[i].primitive(); - let variant = Variant::compute(Shader::Candidates, "candidates").path(path).primitive(primitive); - pass.set_pipeline(&inner.pipelines.compute_pipeline(device, variant)); - pass.set_bind_group(1, &groups.mark, &[offset(i)]); - dispatch(&mut pass, rows); - } - } + let group = resources.pass(device, inner, Shader::Texts); + let mut run = |encoder: &mut wgpu::CommandEncoder, label: &str, entry: usize| { + let mut pass = + encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { label: Some(label), timestamp_writes: None }); + pass.set_bind_group(0, &groups.view, &[]); pass.set_bind_group(2, &group, &[]); - // A diagram's labels claim no cells. - let cleared = (!diagram).then(|| { - ["clear_cells", "reset_cells"].map(|entry| inner.pipelines.compute(device, Shader::Labels, entry)) - }); - let capped: Vec<(usize, u32)> = - labeled.iter().filter(|(_, s)| s.select != wgsl::NONE).map(|&(i, _)| (i, self.marks[i].rows)).collect(); - let selecting = (!capped.is_empty() && !diagram).then(|| { - ["select_start", "select_count", "select_find", "select_cut"] - .map(|entry| inner.pipelines.compute(device, Shader::Labels, entry)) - }); - let mut each = |pass: &mut wgpu::ComputePass<'_>, entry: &'static str| { - for &(i, rows, path) in &marks { - let variant = Variant::compute(Shader::Labels, entry).path(path); - pass.set_pipeline(&inner.pipelines.compute_pipeline(device, variant)); - pass.set_bind_group(1, &groups.mark, &[offset(i)]); - dispatch(pass, rows); - } - }; - let cells = |pass: &mut wgpu::ComputePass<'_>, pipeline: &wgpu::ComputePipeline| { - pass.set_bind_group(1, &groups.mark, &[offset(labeled[0].0)]); - pass.set_pipeline(pipeline); - dispatch(pass, columns * rows); - }; - if due && let Some([clear, reset]) = &cleared { - // A capped mark's labels past its cap give up their room before any claims. - if let Some(pipelines) = &selecting { - select(&mut pass, pipelines, &groups.mark, &capped); - } - cells(&mut pass, clear); - for _ in 0..ROUNDS { - each(&mut pass, "claim"); - each(&mut pass, "decide"); - each(&mut pass, "occupy"); - cells(&mut pass, reset); - } - each(&mut pass, "ask"); - } - if !diagram { - each(&mut pass, "fade"); + // A pipeline for each mark's primitive, which finds its rows' anchors. + for &(i, _) in &labeled { + let shown = &self.marks[i]; + pass.set_pipeline(&inner.pipelines.compute(device, MarkVariants::shown(shown).label_passes()[entry])); + pass.set_bind_group(1, &groups.mark, &[offset(i)]); + dispatch(&mut pass, shown.rows); } + }; + if claim { + encoder.clear_buffer(&grid, 0, Some(words * 4)); + run(encoder, "latkit label claims", 0); + self.labels.claimed = Some(stamp); + } + if gather { + self.labels.gathered.mark(stamp); + self.forget_asked(encoder, &labeled); + run(encoder, "latkit texts", 1); + self.read_gathered(encoder, inner, labeled, stamp); } - if due { - self.read_placed(encoder, inner, labeled, stamp); + } + + /// Clears the texts each of `labeled` asked for and saw, before a texts pass asks again. + fn forget_asked(&self, encoder: &mut wgpu::CommandEncoder, labeled: &[(usize, Slots)]) { + let labels = self.resources.as_ref().expect("created before labels").labels(); + for &(_, slots) in labeled { + encoder.clear_buffer(labels, u64::from(slots.wanted) * 4, Some(4)); + encoder.clear_buffer(labels, u64::from(slots.seen) * 4, Some(u64::from(slots.codes / 32) * 4)); } } - /// Reads back the texts the placement of `labeled`, from `stamp`, asked for. - fn read_placed( + /// Reads back the texts the texts pass over `labeled` (from `stamp`) asked for and saw. + fn read_gathered( &mut self, encoder: &mut wgpu::CommandEncoder, inner: &Inner, @@ -474,41 +400,15 @@ impl View { let words = labeled.len() as u64 * u64::from(1 + wgsl::WANTED) + seen; let (device, offset) = (&self.gpu.0.device, u64::from(first) * 4); let readback = Readback::new(&inner.readbacks, device, encoder, labels, offset, words * 4, &self.waker); - self.labels.placed.start(stamp, readback); + self.labels.gathered.start(stamp, readback); self.labels.asked = labeled; } } -/// Keeps each `capped` mark's labels to its cap, those of the highest keys: a radix select over the -/// candidates' keys, a digit at a time, then a cut of those below. -fn select( - pass: &mut wgpu::ComputePass<'_>, - [start, count, find, cut]: &[wgpu::ComputePipeline; 4], - marks: &wgpu::BindGroup, - capped: &[(usize, u32)], -) { - let mut each = |pipeline: &wgpu::ComputePipeline, threads: Option| { - pass.set_pipeline(pipeline); - for &(i, rows) in capped { - pass.set_bind_group(1, marks, &[offset(i)]); - match threads { - Some(n) => dispatch(pass, n.max(rows)), - None => pass.dispatch_workgroups(1, 1, 1), - } - } - }; - each(start, Some(SELECT_WORDS)); - for _ in 0..DIGITS { - each(count, Some(0)); - each(find, None); - } - each(cut, Some(0)); -} - impl View { - /// Draws every labeled mark's labels, then every vertex mark's port labels: `glyphs` are the - /// pass groups of the labels' and the port labels' shaders. - #[allow(clippy::too_many_arguments, reason = "the main pass's state, as it draws")] + /// Draws every labeled mark's labels, then every vertex mark's port labels. `glyphs` holds the + /// pass bind groups of their glyph shaders. + #[allow(clippy::too_many_arguments, reason = "the main pass's draw state")] pub(super) fn draw_labels( &self, pass: &mut wgpu::RenderPass<'_>, @@ -517,17 +417,15 @@ impl View { format: wgpu::TextureFormat, samples: u32, groups: &super::resources::Groups, - glyphs: [(Shader, &wgpu::BindGroup); 2], + [labels_group, ports_group]: [&wgpu::BindGroup; 2], ) { let device = &self.gpu.0.device; - let [(labels, labels_group), (ports, ports_group)] = glyphs; for (i, slots) in self.labels.slots.iter().enumerate() { let shown = &self.marks[i]; if slots.is_none() || shown.rows == 0 { continue; } - let variant = Variant::draw(labels, format, samples).path(shown.path()).primitive(shown.primitive()); - pass.set_pipeline(&inner.pipelines.render_pipeline(device, variant)); + pass.set_pipeline(&inner.pipelines.render(device, MarkVariants::shown(shown).glyphs(format, samples))); pass.set_bind_group(2, labels_group, &[]); pass.set_bind_group(1, &groups.mark, &[offset(i)]); drawn.draw(pass, 4, shown.rows); @@ -538,7 +436,7 @@ impl View { if !labeled || !shown.drawable() || shown.rows == 0 { continue; } - pass.set_pipeline(&inner.pipelines.render(device, ports, format, samples)); + pass.set_pipeline(&inner.pipelines.render(device, Variant::draw(Shader::GlyphsPorts, format, samples))); pass.set_bind_group(2, ports_group, &[]); pass.set_bind_group(1, &groups.mark, &[offset(i)]); drawn.draw(pass, 4, shown.rows * shown.port_count()); diff --git a/crates/latkit/src/view/legends.rs b/crates/latkit/src/view/legends.rs index 1d9becde..1bd7bd35 100644 --- a/crates/latkit/src/view/legends.rs +++ b/crates/latkit/src/view/legends.rs @@ -1,7 +1,9 @@ -//! A scene's title and its marks' legends, written over the marks: the title and subtitle at the top -//! left, and the legends stacked up from the bottom right, each under its title: a bar of its mark's -//! colormap across the domain, ticked; a cell of color for each category; or, where categories are -//! named, a swatch beside each name. Rebuilt each frame as overlay sprites, O(legends). +//! A scene's title and its marks' legends, drawn over the marks. +//! +//! The title and subtitle sit at the top left. Legends stack upward from the bottom right of the +//! drawing area (inside a plot's axes), each under its title. A legend is a ticked bar of the +//! mark's colormap across its domain, a color cell per category, or, for named categories, a swatch +//! beside each name. Rebuilt each frame as overlay sprites, in O(legends). use std::sync::Arc; @@ -11,9 +13,9 @@ use crate::gpu::Inner; use crate::gpu::shaders as wgsl; use crate::{Axis, Color, Colormap, ColormapKind, Theme}; -/// CSS pixels between the view's edge and what is written there. +/// CSS pixels between the view's inset edge and the title or legends. const MARGIN: f64 = 16.0; -/// A legend's bar: width and height, CSS pixels. +/// A legend's bar width and height, in CSS pixels. const BAR: [f64; 2] = [220.0, 8.0]; /// Cells a continuous bar is drawn in. const CELLS: usize = 64; @@ -21,18 +23,19 @@ const CELLS: usize = 64; const CATEGORIES: usize = 24; /// The most names a column of a legend lists. const ROWS: usize = 8; -/// The size of what a legend writes beside its colors, ticks and names, for the theme's text size. +/// Text scale of a legend's tick labels and names, relative to the theme's text size. const SMALL: f64 = 0.9; -/// The sizes of the title and subtitle for the theme's text size, and the line each takes for its size. +/// Text scales of the title and subtitle, relative to the theme's text size, and their line height +/// relative to their own size. const TITLE: f64 = 1.75; const SUBTITLE: f64 = 1.1; const LEADING: f64 = 1.35; impl View { - /// CSS pixels the scene's title and subtitle take below the top inset, with the margin above them - /// and half of it below: none when it has neither. + /// CSS pixels the title and subtitle take below the top inset, including the margin above them + /// and half a margin below. 0 when the scene has neither. pub(super) fn title_band(&self) -> f64 { - let font = f64::from(self.scene.theme.font_size); + let font = f64::from(self.scene.theme.text.size); let lines: f64 = [(self.scene.title.is_some(), TITLE), (self.scene.subtitle.is_some(), SUBTITLE)] .into_iter() .filter_map(|(shown, scale)| shown.then_some(font * scale * LEADING)) @@ -53,28 +56,32 @@ impl View { let [top, right, bottom, left] = self.insets.map(f64::from); let area = [left + MARGIN, top + MARGIN, w - right - MARGIN, h - bottom - MARGIN]; let ratio = f64::from(self.pixel_ratio); - let font = f64::from(theme.font_size); + let font = f64::from(theme.text.size); let writer = |inner: &mut Inner, scale: f64, color: Color, f: &mut dyn FnMut(&mut Writer<'_>)| { let size_px = (font * scale * ratio).round().max(1.0) as u32; - f(&mut Writer { inner, font: theme.font.clone(), size_px, ratio, color }); + f(&mut Writer { inner, font: theme.text.font.clone(), size_px, ratio, color }); }; let mut sprites = Vec::new(); let mut y = area[1]; for (text, scale, color) in - [(&scene.title, TITLE, theme.text), (&scene.subtitle, SUBTITLE, theme.text.alpha(0.7))] + [(&scene.title, TITLE, theme.text.color), (&scene.subtitle, SUBTITLE, theme.text.color.alpha(0.7))] { let Some(text) = text else { continue }; writer(inner, scale, color, &mut |w| sprites.extend(w.text(text, [area[0], y], Align::BelowRight))); y += font * scale * LEADING; } - let mut base = area[3]; + // Legends stack up from the drawing area's bottom right. On a plot that is inside its axes, + // clear of their labels. + let drawing = self.area(); + let right = drawing[2] - MARGIN; + let mut base = drawing[3] - MARGIN; for (title, color) in legends() { let Some((colormap, domain)) = color.colormap_of() else { continue }; let write = |s, color, f: &mut dyn FnMut(&mut Writer<'_>)| writer(inner, s, color, f); let height = if !colormap.named_categories().is_empty() { - categories(&mut sprites, theme, [area[2], base], colormap, ratio, write, title) + categories(&mut sprites, theme, [right, base], colormap, ratio, write, title) } else if let Some(domain) = domain { - legend(&mut sprites, theme, [area[2], base], colormap, domain, write, title) + legend(&mut sprites, theme, [right, base], colormap, domain, write, title) } else { continue; }; @@ -84,9 +91,9 @@ impl View { } } -/// Draws a legend of `colormap`'s named categories ending at `corner`, its right edge at its x and its -/// bottom at its y: a swatch of each category's color beside its name, in columns of up to -/// `ROWS`, under `title`, placed whole on pixels of `ratio`. How tall it is. +/// Draws a legend of `colormap`'s named categories with its bottom-right corner at `corner`: a +/// swatch of each category's color beside its name, in columns of up to `ROWS`, under `title`, +/// snapped to device pixels of `ratio`. Returns its height. fn categories( sprites: &mut Vec, theme: &Theme, @@ -96,14 +103,14 @@ fn categories( mut write: impl FnMut(f64, Color, &mut dyn FnMut(&mut Writer<'_>)), title: &Arc, ) -> f64 { - let font = f64::from(theme.font_size); + let font = f64::from(theme.text.size); let names = colormap.named_categories(); let names = &names[..names.len().min(CATEGORIES)]; let [row, swatch] = [font * 1.5, font * 0.8]; let rows = names.len().min(ROWS); let top = corner[1] - rows as f64 * row; let start = sprites.len(); - // Laid out from x = 0, each column clear of the names before it, then moved into place. + // Lay out from x = 0, each column clear of the names before it, then shift it all into place. let mut x = 0.0; for (column, names) in names.chunks(ROWS).enumerate() { let mut right = x; @@ -112,7 +119,7 @@ fn categories( let category = column * ROWS + r; let color = Color::from_bytes(colormap.texels()[category]); sprites.push(fill([x, middle - swatch / 2.0], [x + swatch, middle + swatch / 2.0], color)); - write(SMALL, theme.text.alpha(0.85), &mut |w| { + write(SMALL, theme.text.color.alpha(0.85), &mut |w| { if let Some(text) = w.text(name, [x + swatch + font * 0.5, middle], Align::Right) { right = f64::max(right, f64::from(text.b[0])); sprites.push(text); @@ -123,7 +130,7 @@ fn categories( } let mut height = corner[1] - top; if !title.is_empty() { - write(1.0, theme.text, &mut |w| sprites.extend(w.text(title, [0.0, top - 5.0], Align::Above))); + write(1.0, theme.text.color, &mut |w| sprites.extend(w.text(title, [0.0, top - 5.0], Align::Above))); height += font * 1.4 + 5.0; } let right = sprites[start..].iter().fold(0.0_f32, |right, s| right.max(s.a[0]).max(s.b[0])); @@ -135,8 +142,8 @@ fn categories( height } -/// Draws a legend whose bar ends at `corner`'s x, its tick labels' bottom at its y: `colormap` across -/// `domain` under `title`. How tall it is. +/// Draws a bar legend of `colormap` across `domain`, under `title`, with the bar's right end at +/// `corner`'s x and the tick labels' bottom at its y. Returns its height. fn legend( sprites: &mut Vec, theme: &Theme, @@ -146,11 +153,11 @@ fn legend( mut write: impl FnMut(f64, Color, &mut dyn FnMut(&mut Writer<'_>)), title: &Arc, ) -> f64 { - let font = f64::from(theme.font_size); + let font = f64::from(theme.text.size); let bar_bottom = corner[1] - font * SMALL * 1.4 - 4.0; let [left, top] = [corner[0] - BAR[0], bar_bottom - BAR[1]]; let color = |t: usize| Color::from_bytes(colormap.texels()[t.min(255)]); - let text = theme.text.alpha(0.85); + let text = theme.text.color.alpha(0.85); let across = |v: f64| left + (v - domain[0]) / (domain[1] - domain[0]).max(f64::MIN_POSITIVE) * BAR[0]; let mut labels: Vec<(String, f64)> = Vec::new(); if colormap.kind() == ColormapKind::Categorical { @@ -190,13 +197,13 @@ fn legend( }); let mut height = corner[1] - top; if !title.is_empty() { - write(1.0, theme.text, &mut |w| sprites.extend(w.text(title, [left, top - 5.0], Align::Above))); + write(1.0, theme.text.color, &mut |w| sprites.extend(w.text(title, [left, top - 5.0], Align::Above))); height += font * 1.4 + 5.0; } height } -/// A rectangle filled with `color`, corner `a` to corner `b`, CSS pixels. +/// A rectangle filled with `color`, from corner `a` to corner `b`, in CSS pixels. fn fill(a: [f64; 2], b: [f64; 2], color: Color) -> wgsl::Sprite { let mut sprite = line(a, b, 0.0, color); sprite.kind = wgsl::SPRITE_FILL; diff --git a/crates/latkit/src/view/lists.rs b/crates/latkit/src/view/lists.rs index 15c5612a..239958d3 100644 --- a/crates/latkit/src/view/lists.rs +++ b/crates/latkit/src/view/lists.rs @@ -1,6 +1,10 @@ -//! Lists on the GPU: each item counts its entries, a scan turns the counts into where each item's -//! entries start, and each item writes its entries there. The total lands in the list's header as -//! indirect draw and dispatch arguments, so nothing waits on a readback to know it. +//! Lists on the GPU: each item counts its entries, a scan turns the counts into start offsets, and +//! each item writes its entries there. The total goes into the list's header as indirect draw and +//! dispatch arguments, so nothing waits on a readback for it. +//! +//! After the segment lists lie the drawn-row words of each mark drawing a paged table: the held +//! row of each row drawn, expanded from the drawing's runs when they change, so every pass reads a +//! drawn row in one word. use super::View; use super::marks::Shown; @@ -8,14 +12,14 @@ use super::passes::dispatch; use crate::derive::{Derived, stamp_of}; use crate::gpu::Inner; use crate::gpu::readback::Readback; -use crate::gpu::shaders::{self as wgsl, Shader}; +use crate::gpu::shaders as wgsl; use crate::scene::mark::MarkKind; -/// A stroke mark's segment list, when its rows' paths have different numbers of segments: where it -/// lies, a stamp of what it was last built from (the list, the paths, and for great circles where -/// the vertices are drawn), and its count. Each entry is a segment's row and how far along its path it -/// starts. A curved mark's count is read back, one at a time, and its list makes room for what it -/// found. +/// A listed mark's segment list (see `Shown::listed`), used when its rows have different numbers +/// of segments, or for areas, of triangles: where it lies, a stamp of its last build's inputs (the +/// list, the paths, and for great circles where the vertices are drawn), and its count. Each entry +/// is a segment's row and how far along its path it starts. A curved mark's count is read back, +/// one readback at a time, and its list grows to fit what was found. #[derive(Default)] pub(crate) struct Segments { pub(crate) list: Option, @@ -26,13 +30,21 @@ pub(crate) struct Segments { /// Words a segment list keeps per segment: its row, and its distance along. const SEGMENT_WORDS: u32 = 2; +/// Where a paged mark's drawn-row words lie in the lists buffer, `[start, room]` in words, and the +/// stamp of their last expansion's inputs: the drawing, their location, and the buffer. +#[derive(Default)] +pub(crate) struct DrawnRows { + pub(crate) at: Option<[u32; 2]>, + expanded: u64, +} + /// Where a list lies in the lists buffer, in words. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) struct List { pub(crate) header: u32, pub(crate) items: u32, pub(crate) entries: u32, - /// Entries kept, and words per entry. + /// The most entries it keeps, and words per entry. pub(crate) kept: u32, pub(crate) stride: u32, sums: u32, @@ -58,7 +70,7 @@ impl List { items.div_ceil(wgsl::SCAN_WORKGROUP).max(1) + items + 1 } - /// Where the scan leaves the list's total, before it is held to what the list keeps. + /// Where the scan writes the list's total, before it is capped at `kept`. pub(crate) fn total(&self) -> u32 { self.counts + self.items } @@ -79,13 +91,13 @@ impl List { } impl View { - /// Lays out every listed stroke mark's segment list at the front of the lists buffer, taking - /// counts read back: a list too small for what its rows were found to have grows. + /// Lays out every listed mark's segment list at the front of the lists buffer, using counts + /// read back. A list smaller than its rows need grows, with a quarter to spare. pub(super) fn lay_out_segments(&mut self) { let mut at = 0; for shown in &mut self.marks { if let Some((stamp, bytes)) = shown.segments.found.landed() { - let found = bytes.get(..4).map_or(0, |w| u32::from_le_bytes([w[0], w[1], w[2], w[3]])); + let found = crate::wire::In::new(&bytes).u32().unwrap_or(0); let kept = shown.segments.list.map_or(0, |l| l.kept); let room = if found > kept { found + found / 4 } else { shown.segments.found.value }; shown.segments.found.make(stamp, room); @@ -106,22 +118,58 @@ impl View { at += List::front_words(list.kept, SEGMENT_WORDS) + List::back_words(items); shown.segments.list = Some(list); } - self.segment_words = at; + // Then one word per row each paged mark draws, rows that left included. + for shown in &mut self.marks { + if shown.drawing().is_none() { + shown.drawn_rows.at = None; + continue; + } + let items = shown.rows + shown.exits; + let region = match shown.drawn_rows.at { + Some([start, room]) if start == at && room >= items => [start, room], + _ => [at, items + items / 4], + }; + at += region[1]; + shown.drawn_rows.at = Some(region); + } + self.list_words = at; + } + + /// The marks whose drawn-row words are rewritten this frame: those whose drawing, rows, word + /// location, or lists buffer changed. + pub(super) fn drawn_rows_due(&mut self) -> Vec { + let generation = self.resources.as_ref().map_or(0, super::resources::Resources::lists_generation); + let mut due = Vec::new(); + for (i, shown) in self.marks.iter_mut().enumerate() { + let Some(at) = shown.drawn_rows.at.filter(|_| shown.drawable()) else { continue }; + let drawn = shown.drawing().map(|d| d.made); + let stamp = stamp_of((at, shown.rows, shown.exits, drawn, generation)); + if stamp != shown.drawn_rows.expanded { + shown.drawn_rows.expanded = stamp; + due.push(i); + } + } + due } - /// The marks whose segment lists build again this frame: their list, the lists buffer, or their - /// paths changed, or, on great circles, where the vertices their rows join are drawn. A color - /// changing every frame builds none. + /// The marks whose segment lists rebuild this frame: those whose list, lists buffer, paths, or + /// drawing changed, or, along great circles or lanes, where the vertices their rows join are + /// drawn. A color changing every frame rebuilds none. pub(super) fn segments_due(&mut self) -> Vec { let vertices: Vec<(bool, u64)> = self.marks.iter().filter(|m| m.kind() == MarkKind::Vertices).map(Shown::places_drawn).collect(); let generation = self.resources.as_ref().map_or(0, super::resources::Resources::lists_generation); + // Lanes are laid out on screen, so they count again whenever the camera moves. + let projected = self.projected(); + let sight = (projected.clip_from_world.to_cols_array().map(f64::to_bits), self.size.map(f64::to_bits)); let mut due = Vec::new(); for (i, shown) in self.marks.iter_mut().enumerate() { let Some(list) = shown.segments.list.filter(|_| shown.drawable()) else { continue }; - let arcs = shown.geodesic.then(|| (&vertices, shown.places_drawn())); + let laned = shown.laned().then_some(sight); + let arcs = (shown.geodesic || laned.is_some()).then(|| (&vertices, shown.places_drawn(), laned)); let tables = shown.tables.generation(); - let stamp = stamp_of((list.header, list.items, list.kept, tables, generation, arcs)); + let drawn = shown.drawing().map(|d| d.made); + let stamp = stamp_of((list.header, list.items, list.kept, tables, generation, arcs, drawn)); if stamp != shown.segments.built { shown.segments.built = stamp; due.push(i); @@ -130,13 +178,16 @@ impl View { due } - /// Reads back how many segments each curved mark's list holds, so a list too small grows: one - /// count at a time, never dropped for a newer list, which is counted once it lands. + /// Reads back how many segments each great-circle mark needs, so a list too small grows. Each + /// mark has one readback at a time: a newer list does not cancel it, and is counted after it + /// arrives. A mark only bowed into lanes always has space for every piece of every lane, so it + /// reads back nothing. pub(super) fn count_found(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &Inner) { let gpu = self.gpu.clone(); let Some(resources) = &self.resources else { return }; for shown in &mut self.marks { - let Some(list) = shown.segments.list.filter(|_| shown.curved() && shown.drawable()) else { continue }; + let geodesic = shown.curved() && shown.geodesic; + let Some(list) = shown.segments.list.filter(|_| geodesic && shown.drawable()) else { continue }; let built = shown.segments.built; if shown.segments.found.busy() || !shown.segments.found.due(built) { continue; @@ -150,10 +201,10 @@ impl View { } impl Shown { - /// Whether how many segments its list holds is being read back, or is due to be. + /// Whether its segment count is being read back, or is due to be. pub(crate) fn counting(&self) -> bool { let found = &self.segments.found; - found.busy() || (self.curved() && self.drawable() && found.due(self.segments.built)) + found.busy() || (self.curved() && self.geodesic && self.drawable() && found.due(self.segments.built)) } } @@ -164,10 +215,11 @@ pub(super) fn scan( device: &wgpu::Device, items: u32, ) { - pass.set_pipeline(&pipelines.compute(device, Shader::Lists, "scan_blocks")); + let [blocks, sums, add] = super::variants::SCAN; + pass.set_pipeline(&pipelines.compute(device, blocks)); dispatch(pass, items.max(1)); - pass.set_pipeline(&pipelines.compute(device, Shader::Lists, "scan_sums")); + pass.set_pipeline(&pipelines.compute(device, sums)); pass.dispatch_workgroups(1, 1, 1); - pass.set_pipeline(&pipelines.compute(device, Shader::Lists, "add_sums")); + pass.set_pipeline(&pipelines.compute(device, add)); dispatch(pass, items); } diff --git a/crates/latkit/src/view/marks.rs b/crates/latkit/src/view/marks.rs index 96546e8e..bc89a5da 100644 --- a/crates/latkit/src/view/marks.rs +++ b/crates/latkit/src/view/marks.rs @@ -1,19 +1,21 @@ -//! Committed marks: a scene's mark, its program compiled against its data, and its GPU state. +//! Shown marks: a scene's mark, its program compiled against its data, and its GPU state. use std::sync::Arc; -use super::rows::Region; +use super::drawing::Drawing; +use super::ends::End; use super::sync::Tables; use super::transition::{Begin, Transition}; -use crate::data::Table; +use super::words::{Kept, Words}; +use crate::data::{Table, Version}; use crate::expr::compile::{Deps, Reads}; use crate::gpu::shaders as wgsl; use crate::gpu::{Inner, atlas}; use crate::scene::compile::{self, Compiled}; use crate::scene::mark::{Context, Key, MarkKind, Primitive}; -use crate::{Data, Error, Mark, Port, Result, Shape}; +use crate::{Error, Mark, Port, Result, Shape, Snapshot}; -/// Where a mark's parts sit in the program buffer, as absolute word offsets. +/// Where each part of a mark's region lies in the program buffer, as absolute word offsets. #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) struct Offsets { pub(crate) columns: u32, @@ -24,11 +26,54 @@ pub(crate) struct Offsets { pub(crate) palette: u32, pub(crate) port_info: u32, pub(crate) blends: u32, + /// The runs of held rows it draws. `NONE` without a drawing, when the held rows are the table's + /// rows. + pub(crate) runs: u32, + /// The page table of each kind of per-row words, indexed by `Kept`. `NONE` for a kind the mark + /// does not keep. + pub(crate) pages: [u32; 5], +} + +impl Offsets { + /// Where the page table of `kept` lies, or `NONE`. + pub(crate) fn table(&self, kept: Kept) -> u32 { + self.pages[kept as usize] + } } impl Default for Offsets { fn default() -> Offsets { - Offsets { columns: 0, ranges: 0, code: 0, constants: 0, ends: 0, palette: 0, port_info: wgsl::NONE, blends: 0 } + Offsets { + columns: 0, + ranges: 0, + code: 0, + constants: 0, + ends: 0, + palette: 0, + port_info: wgsl::NONE, + blends: 0, + runs: wgsl::NONE, + pages: [wgsl::NONE; 5], + } + } +} + +/// The rows a mark's state holds: a version of its table, laid out by a drawing or in table order, +/// and the origin of its dataset. Rows that move are followed by id only within one origin. +#[derive(Clone)] +pub(crate) struct Holding { + pub(crate) origin: Version, + pub(crate) table: Table, + pub(crate) drawing: Option>, +} + +impl Holding { + /// The table row at held row `h`, if it holds one there. + pub(crate) fn source_row(&self, h: u32) -> Option { + match &self.drawing { + Some(drawing) => drawing.source(h), + None => (h < self.table.len() as u32).then_some(h), + } } } @@ -37,62 +82,84 @@ pub(crate) struct Shown { pub(crate) key: Key, pub(crate) compiled: Option, pub(crate) tables: Tables, - /// Rows of the mark's table when last synced, and rows past them that left and fade out. + /// Rows the mark draws, as of the last published tables. pub(crate) rows: u32, + /// Rows that left and are fading out, held after the live rows. pub(crate) exits: u32, - /// Rows the region held when the frame began: what moves and freezes. - pub(crate) held: u32, - /// The table whose rows the mark's state holds, for following rows by id when they move, and the - /// origin of its dataset: rows of another dataset are not followed but replaced. - pub(crate) drawn: Option<(crate::data::Version, Table)>, - /// Where its rows sit in every per-row buffer, and where they sat before this frame moved them. - pub(crate) region: Region, - pub(crate) moved: Option, - /// First fitted domain. + /// Rows it held when the frame began. A starting transition freezes these. + pub(crate) before: u32, + /// The rows its state holds, as of the last published tables. + pub(crate) holding: Option, + /// How far its paged table's tiles have faded in or out, from 0 to 1. + pub(crate) reveal: f32, + /// For a paged table, each fitted domain computed on the CPU over every row: the stamp of its + /// inputs, and its words. + pub(crate) fitted: Vec<(u64, [u32; 2])>, + /// Its per-row words, in pages of the arena. + pub(crate) words: Words, + /// The index of its first fitted domain in the domains buffer. pub(crate) domains: u32, - /// Channels to evaluate, the row blocks they evaluate over, and whether fitted domains reduce, - /// next frame. Set through `dirty_rows` and `all_dirty`, so every channel made dirty names its rows. + /// The channels to evaluate next frame, as a bit mask. `blocks` holds the row blocks to + /// evaluate them over, and `reduce` whether fitted domains reduce. Set only through + /// `dirty_rows` and `all_dirty`, so every dirty channel has row blocks. pub(crate) dirty: u32, pub(crate) blocks: Blocks, pub(crate) reduce: bool, - /// Where this frame's evaluation lists the row blocks it covers, and how many. + /// Row blocks whose state words start over in newly allocated pages. Evaluation writes every + /// word of their rows before any channel runs. + pub(crate) fresh: Vec, + /// Where this frame's evaluation lists its row blocks in the blocks buffer, and how many. pub(crate) block: u32, pub(crate) block_count: u32, - /// Whether this frame's ease pass writes what the mark shows from its frozen and settled state. + /// The row blocks whose endpoints the end pass resolves next frame: blocks whose rows or + /// endpoint references changed, and blocks whose rows end at vertices that moved. + /// `end_block` and `end_block_count` locate this frame's list in the blocks buffer. + pub(crate) unresolved: Blocks, + pub(crate) end_block: u32, + pub(crate) end_block_count: u32, + /// Whether this frame's ease pass writes the mark's shown state from its frozen and settled + /// words. pub(crate) easing: bool, - /// The program's code and constants with colormap layers and text codes written in. + /// The program's code and constants, with colormap layers and text codes filled in. pub(crate) code: Vec<[u32; 4]>, pub(crate) constants: Vec, - /// Graph slot of vertex 0 of each table the program's ends name. - pub(crate) ends: Vec, - /// A vertex mark's ports, `PORT_WORDS` each, their labels' places in the atlas written in. + /// The end table of each endpoint reference: where the vertex rows it references are held. + pub(crate) ends: Vec, + /// A vertex mark's port words, `PORT_WORDS` per port, including the atlas corners of its label. pub(crate) ports: Vec, - /// The mark's region of the program buffer, `[start, capacity]` in words, and where its parts - /// are; `packed` is false until the region holds the current words. - pub(crate) words: Option<[u32; 2]>, + /// The mark's region of the program buffer, `[start, capacity]` in words. `offsets` locates its + /// parts, `packed` is false until the region holds the current words, and `written` holds the + /// words last written there. + pub(crate) region: Option<[u32; 2]>, pub(crate) offsets: Offsets, pub(crate) packed: bool, - /// A transition waiting to freeze, the one running, and where its remap words start; and whether - /// this frame's froze rows where they moved, so their comets' travel follows them. + pub(crate) written: Vec, + /// A transition waiting to start. `transition` is the running one. `remap` is where its remap + /// words start in the frame it starts, which the compute pass freezes it from (`NONE` in other + /// frames). `carry` is whether this frame froze rows at new places, so their comets follow + /// them. pub(crate) begin: Option, pub(crate) transition: Option, pub(crate) remap: u32, pub(crate) carry: bool, - /// Where the header of the list this frame's query builds over its instances starts. + /// The probe-buffer word where this frame's query writes its result for this mark. pub(crate) query: u32, - /// The list numbering a stroke mark's segments when rows have different numbers of them. + /// The list that numbers a stroke mark's segments when rows have different segment counts. pub(crate) segments: super::lists::Segments, + /// For a paged table, where each drawn row's held-row word lies in the lists buffer. + pub(crate) drawn_rows: super::lists::DrawnRows, /// Whether an edge mark follows great circles. pub(crate) geodesic: bool, - /// Bumps whenever the mark's state is written, so what derives from it knows to follow; and - /// whenever what places its labels is: where its rows stand, how large and whether they show, - /// and their labels' texts and priorities. A color that changes every frame moves no label. + /// Incremented whenever the mark's state is written, so anything derived from it updates. + /// `place_version` is incremented whenever the channels that place its rows and labels are + /// written: row positions, sizes, visibility, and label texts. A color that changes every frame + /// moves no label. pub(crate) version: u64, pub(crate) place_version: u64, /// The rows its focus draw lights. pub(crate) lit: super::focus::Lit, - /// Why it no longer compiles against the data, kept until data fits it again, and whether a frame - /// told it: the next frame does, unless a scene shown before then replaced the mark. + /// Why it no longer compiles against the data, kept until the data fits again, and whether a + /// frame has reported it. The next frame reports it unless a new scene replaces the mark first. pub(crate) failure: Option<(Error, bool)>, } @@ -105,30 +172,37 @@ impl Shown { tables: Tables::default(), rows: 0, exits: 0, - held: 0, - drawn: None, - region: Region::EMPTY, - moved: None, + before: 0, + holding: None, + reveal: 1.0, + fitted: Vec::new(), + words: Words::default(), domains: 0, dirty: 0, blocks: Blocks::default(), reduce: false, + fresh: Vec::new(), block: 0, block_count: 0, + unresolved: Blocks::default(), + end_block: 0, + end_block_count: 0, easing: false, code: Vec::new(), constants: Vec::new(), ends: Vec::new(), ports: Vec::new(), - words: None, + region: None, offsets: Offsets::default(), packed: false, + written: Vec::new(), begin: None, transition: None, - remap: 0, + remap: wgsl::NONE, carry: false, query: wgsl::NONE, segments: super::lists::Segments::default(), + drawn_rows: super::lists::DrawnRows::default(), geodesic: false, version: 0, place_version: 0, @@ -141,14 +215,96 @@ impl Shown { self.key.kind } - /// Whether the mark has a program, rows, and columns synced at least once. + /// Live held rows, not counting exits. Every per-row word belongs to a held row. An unpaged + /// table's held rows are its rows. A paged table holds a page of rows per slot. + pub(crate) fn live(&self) -> u32 { + self.drawing().map_or(self.rows, |d| d.live()) + } + + /// The drawing of its paged table, as last published. + pub(crate) fn drawing(&self) -> Option<&Arc> { + self.holding.as_ref().and_then(|h| h.drawing.as_ref()) + } + + /// The table row at held row `h`. + pub(crate) fn source_row(&self, h: u32) -> Option { + self.holding.as_ref()?.source_row(h) + } + + /// The held row of table row `r`, if the mark holds it. + pub(crate) fn held_row(&self, r: u32) -> Option { + match self.drawing() { + Some(drawing) => drawing.held_row(r), + None => (r < self.rows).then_some(r), + } + } + + /// The position of held row `h` among the rows the mark draws, if it draws it. + pub(crate) fn item_of(&self, h: u32) -> Option { + match self.drawing() { + Some(drawing) => drawing.item_of(h), + None => (h < self.rows).then_some(h), + } + } + + /// Rows it draws that are not fading out with their tiles. + pub(crate) fn steady_rows(&self) -> u32 { + self.drawing().map_or(self.rows, |d| d.items - d.leaving_items) + } + + /// The runs of held rows it draws, in the layout `lib/drawn.wesl` reads: the count, then each + /// run's first drawn row, first held row, and fade. Exits come last, as one steady run. Empty + /// without a drawing. + fn runs(&self) -> Vec { + let Some(drawing) = self.drawing() else { return Vec::new() }; + let exits = (self.exits > 0).then_some([drawing.items, drawing.live(), 0]); + let count = drawing.runs.len() + usize::from(exits.is_some()); + let mut words = Vec::with_capacity(1 + count * wgsl::RUN_WORDS as usize); + words.push(count as u32); + for run in &drawing.runs { + words.extend([run.item, run.held, run.fade]); + } + words.extend(exits.into_iter().flatten()); + words + } + + /// Held rows, exits included. + pub(crate) fn held(&self) -> u32 { + self.live() + self.exits + } + + /// Whether the mark draws faces, which need normals. + pub(crate) fn faces(&self) -> bool { + self.kind() == MarkKind::Faces + } + + /// Whether the mark can evaluate and draw: it has a program, held rows with pages of words for + /// them, and columns synced at least once. A mark holding only the rows that drawn edges end at + /// evaluates them but draws none. pub(crate) fn drawable(&self) -> bool { - self.compiled.is_some() && self.rows + self.exits > 0 && self.tables.synced() + self.compiled.is_some() && self.held() > 0 && self.tables.synced() && self.pages_held() + } + + /// Whether its state pages have the current row width and cover every held row. + pub(crate) fn pages_held(&self) -> bool { + let state = self.words.pages(Kept::State); + state.width == self.width() && state.blocks.len() as u32 >= self.held().div_ceil(wgsl::PAGE_ROWS) + } + + /// The row count of each page of its held rows, as [`page_rows`] lays them out. + pub(crate) fn page_rows(&self) -> Vec { + page_rows(self.live(), self.held(), self.drawing().is_some()) + } + + /// Releases its tables and pages on the GPU, as when the mark is removed. + pub(crate) fn release(&mut self, gpu: &mut Inner) { + self.tables.release(gpu); + self.words.release(&mut gpu.arena); } - /// Kind words per row. + /// Words per row that the mark's kind uses. pub(crate) fn stride(&self) -> u32 { - self.compiled.as_ref().map_or_else(|| kind_stride(self.kind()), |c| c.drawing.stride) + self.kind().stride() } /// Extra words per row. @@ -156,75 +312,85 @@ impl Shown { self.compiled.as_ref().map_or(0, |c| c.drawing.extras) } - /// What its channels and fitted domains read besides its own table's static fields, which their - /// pipeline is made for: before its data says, neither joins, samples, nor the view. + /// Words per row: the kind's, then the extras. + pub(crate) fn width(&self) -> u32 { + self.stride() + self.extras() + } + + /// Words per row for each kind of word page: state, focus, and normals (when faces are + /// `summing` normals at its rows). 0 for a kind the mark keeps none of. + pub(crate) fn kept_widths(&self, summing: bool) -> [(Kept, u32); 3] { + [(Kept::State, self.width()), (Kept::Focus, 1), (Kept::Normals, if summing { wgsl::NORMAL_WORDS } else { 0 })] + } + + /// What its channels and fitted domains read besides its own table's static fields. This picks + /// their pipeline variant. Before the mark compiles, it reads no joins, samples, or view state. pub(crate) fn reads(&self) -> crate::gpu::pipelines::Reads { let Some(program) = self.compiled.as_ref().map(|c| &c.program) else { - return crate::gpu::pipelines::Reads { joined: false, timed: false, viewed: false }; + return crate::gpu::pipelines::Reads::NONE; }; let viewing = [wgsl::OP_SELECTED, wgsl::OP_HOVERED, wgsl::OP_POINTER, wgsl::OP_ZOOM]; crate::gpu::pipelines::Reads { - joined: program.columns.iter().any(|c| c.of.is_some()), + joined: program.columns.iter().any(|c| c.from.is_some()), timed: program.columns.iter().any(|c| c.sampled), viewed: program.code.iter().any(|op| viewing.contains(&op[0])), } } - /// The path its strokes read, or where its triangles' corners come from: one of the `PATH_` codes, - /// which its pipelines are made for. + /// The `PATH_` code its pipelines are built for, as [`Mark::path`](crate::Mark) decides it. pub(crate) fn path(&self) -> u32 { - match &self.compiled { - Some(c) if c.drawing.primitive != Primitive::Points => c.drawing.path, - _ => wgsl::PATH_ENDS, - } + self.compiled.as_ref().map_or(wgsl::PATH_ENDS, |c| c.drawing.path) } - /// Whether it numbers its instances with a list: strokes whose rows' paths have different numbers - /// of segments, running through a geometry column, along great circles, or bowed into lanes; and - /// areas, whose rows have different numbers of triangles. + /// Whether a list numbers its instances: strokes whose rows have different segment counts + /// (through a geometry column, along great circles, or bowed into lanes), and areas, whose rows + /// have different triangle counts. pub(crate) fn listed(&self) -> bool { let Some(compiled) = &self.compiled else { return false }; - match compiled.drawing.primitive { + match self.kind().primitive() { Primitive::Strokes => compiled.drawing.geometry != wgsl::NONE || self.curved(), Primitive::Triangles => self.path() == wgsl::PATH_GEOMETRY, Primitive::Points => false, } } - /// Whether its edges curve between their ends, along great circles or bowed into lanes: how many - /// pieces each takes is found on the GPU and read back. + /// Whether its edges curve between their endpoints, along great circles or bowed into lanes. + /// The GPU counts each edge's pieces. For great circles, the total is read back to size the + /// segment list. pub(crate) fn curved(&self) -> bool { + self.path() == wgsl::PATH_ENDS && (self.geodesic || self.laned()) + } + + /// Whether its edges bow into lanes among the edges joining the same two vertices. + pub(crate) fn laned(&self) -> bool { let Some(compiled) = &self.compiled else { return false }; - let laned = compiled.drawing.lane != wgsl::NONE && self.tables.holds(compiled.drawing.lane); - self.path() == wgsl::PATH_ENDS && (self.geodesic || laned) + compiled.drawing.lane != wgsl::NONE && self.tables.holds(compiled.drawing.lane) } - /// Instances its list must have room for: every path's segments, each curve's pieces as last - /// found; or every area's triangles. + /// Instances its list needs space for: every path's segments plus each curve's pieces (as last + /// counted), or every area's triangles. pub(crate) fn segment_capacity(&self) -> u32 { let rows = (self.rows + self.exits) as usize; let Some(compiled) = &self.compiled else { return 1 }; let points = self.tables.points(compiled.drawing.geometry) as usize; - if compiled.drawing.primitive == Primitive::Triangles { + if self.kind().primitive() == Primitive::Triangles { return (points / 3 + rows).max(1) as u32; } - let pieces = if self.curved() { (self.segments.found.value as usize).max(rows * 4) } else { 0 }; + let pieces = if !self.curved() { + 0 + } else if self.geodesic { + (self.segments.found.value as usize).max(rows * 4) + } else { + rows * wgsl::BOW_PIECES as usize + }; (points + rows + pieces).max(1) as u32 } - /// Vertices each of its instances draws: a quad per point or segment, three corners per triangle. - pub(crate) fn corners(&self) -> u32 { - match self.compiled.as_ref().map(|c| c.drawing.primitive) { - Some(Primitive::Triangles) => 3, - _ => 4, - } - } - - /// Its channels that say where its rows' labels go and whether they show: where rows stand, how - /// large they are and whether they show, and their labels' texts and priorities. + /// A bit mask of the channels that place its rows and labels: row position, size, and + /// visibility, and label text. pub(crate) fn placing_channels(&self) -> u32 { let Some(compiled) = &self.compiled else { return 0 }; - let shaping = ["position", "radius", "width", "height", "elevation", "visible", "text", "priority"]; + let shaping = ["position", "radius", "width", "height", "elevation", "visible", "text"]; compiled .program .channels @@ -233,8 +399,8 @@ impl Shown { .fold(0, |mask, (k, c)| if shaping.contains(&&*c.name) { mask | 1 << k } else { mask }) } - /// A version of where its rows are drawn: their places, and while a transition runs, every frame - /// that eases them. + /// A version of where its rows are drawn: `place_version`, or while a transition runs, + /// `version`, which changes every frame that eases them. The flag says which one it is. pub(crate) fn places_drawn(&self) -> (bool, u64) { if self.transition.is_some() { (true, self.version) } else { (false, self.place_version) } } @@ -249,7 +415,7 @@ impl Shown { self.compiled.as_ref().map_or(0, |c| c.program.domains.len() as u32) } - /// Channels whose inputs include any of `deps`. + /// A bit mask of the channels whose inputs include any of `deps`. pub(crate) fn channels_reading(&self, deps: Deps) -> u32 { let Some(compiled) = &self.compiled else { return 0 }; compiled @@ -260,9 +426,8 @@ impl Shown { .fold(0, |mask, (k, c)| if c.reads.deps.intersects(deps) { mask | 1 << k } else { mask }) } - /// Channels and domains reading any column of `columns` become dirty: over the row blocks - /// `blocks` names, the pages that changed, while the channels read only their own rows; else over - /// every row. + /// Marks the channels and domains that read any of `columns` dirty. They evaluate over `blocks` + /// (the changed pages) when the channels read only their own rows, otherwise over every row. pub(crate) fn columns_changed(&mut self, columns: u64, blocks: Option<&[u32]>) { let Some(compiled) = &self.compiled else { return }; let program = &compiled.program; @@ -273,25 +438,34 @@ impl Shown { channels |= 1 << k; } } - // A fitted domain, or a table read through a join, reaches past a row's own block. - let joined = program.columns.iter().enumerate().any(|(i, c)| c.of.is_some() && columns & Reads::column(i) != 0); + // A fitted domain, or a table read through a join, reads beyond a row's own block. + let joined = + program.columns.iter().enumerate().any(|(i, c)| c.from.is_some() && columns & Reads::column(i) != 0); + // Rows whose endpoint references changed resolve their endpoints again. + let ends = program + .ends + .iter() + .any(|end| end.column != wgsl::NONE && columns & Reads::column(end.column as usize) != 0); self.dirty_rows(channels, blocks.filter(|_| !reduce && !joined)); self.reduce |= reduce; + if ends { + self.unresolved.add(blocks); + } self.packed = false; } - /// Channels `channels` evaluate again on the next frame, over the row blocks `blocks` names, or - /// over every row for `None`. + /// Marks `channels` to evaluate next frame over the row blocks `blocks`, or over every row for + /// `None`. An empty block list marks nothing. pub(crate) fn dirty_rows(&mut self, channels: u32, blocks: Option<&[u32]>) { - if channels != 0 { + if channels != 0 && blocks.is_none_or(|b| !b.is_empty()) { self.dirty |= channels; self.blocks.add(blocks); } } - /// Whether any column of `columns` holds static values that show eased: values a channel reads, - /// or the path a stroke runs along. Changing one eases; a column only shaders read, such as a - /// lane, shows at once. + /// Whether any of `columns` holds static values that ease when they change: values a channel + /// reads, or the path a stroke follows. A column only shaders read, such as a lane, changes at + /// once. pub(crate) fn static_columns(&self, columns: u64) -> bool { let Some(compiled) = &self.compiled else { return false }; let program = &compiled.program; @@ -301,30 +475,46 @@ impl Shown { program.columns.iter().enumerate().any(|(i, c)| !c.sampled && columns & eased & (1 << i.min(63)) != 0) } - /// Rows rewritten in place in row blocks `blocks`, every other row where it was: those blocks - /// evaluate every channel, constants included. A fitted domain reads every row, so with one every - /// row does. + /// Rows in row blocks `blocks` were rewritten in place, and every other row is unchanged. Those + /// blocks evaluate every channel, constants included. A fitted domain reads every row, so with + /// one, every row evaluates. pub(crate) fn rows_rewritten(&mut self, blocks: &[u32]) { let Some(compiled) = &self.compiled else { return }; if !compiled.program.domains.is_empty() { return self.all_dirty(); } + self.slots_bound(blocks); + } + + /// Slots `blocks` received pages of other rows. Those blocks evaluate every channel, constants + /// included, and resolve their endpoints. The data itself is unchanged, so fitted domains stay + /// valid. + pub(crate) fn slots_bound(&mut self, blocks: &[u32]) { + let Some(compiled) = &self.compiled else { return }; let n = compiled.program.channels.len(); + let ends = !compiled.program.ends.is_empty() && !blocks.is_empty(); self.dirty_rows(if n >= 32 { u32::MAX } else { (1 << n) - 1 }, Some(blocks)); + if ends { + self.unresolved.add(Some(blocks)); + } } + /// Marks every channel dirty over every row, and every row's endpoints unresolved. pub(crate) fn all_dirty(&mut self) { if let Some(compiled) = &self.compiled { let n = compiled.program.channels.len(); self.dirty = if n >= 32 { u32::MAX } else { (1 << n) - 1 }; self.blocks = Blocks::All; self.reduce = !compiled.program.domains.is_empty(); + if !compiled.program.ends.is_empty() { + self.unresolved = Blocks::All; + } } } - /// Compiles the mark against `all` in `context`; it stays uncompiled while its data or table is - /// not set. - pub(crate) fn compile(&mut self, all: &[Data], context: Context<'_>, gpu: &mut Inner) -> Result<()> { + /// Compiles the mark against the datasets `all` in `context`. It stays uncompiled while its + /// dataset or table is missing. + pub(crate) fn compile(&mut self, all: &[Snapshot], context: Context<'_>, gpu: &mut Inner) -> Result<()> { self.tables.release(gpu); self.compiled = None; self.packed = false; @@ -336,8 +526,9 @@ impl Shown { Ok(()) } - /// Whether `data` changed a schema the mark compiled against, or brought data it waited for. - pub(crate) fn stale(&self, data: &Data) -> bool { + /// Whether `data` changes a schema the mark compiled against, or is the dataset an uncompiled + /// mark waits for. + pub(crate) fn stale(&self, data: &Snapshot) -> bool { match &self.compiled { None => *data.name() == self.mark.style().data, Some(compiled) => { @@ -346,8 +537,8 @@ impl Shown { } } - /// Writes its ports' words: each one's code, shape, and reference column, and where the atlas - /// holds its label, set in `font` by the host the first time. + /// Writes its port words: each port's code, shape, reference column, and the atlas corners of + /// its label. The host sets each label in `font` the first time. pub(crate) fn write_ports(&mut self, font: &Arc, gpu: &mut Inner) { self.ports.clear(); let Some(compiled) = &self.compiled else { return }; @@ -364,26 +555,28 @@ impl Shown { } } - /// Words the mark's program region needs: descriptors and tables, ranges, code, constants, ends, - /// its markers' palette, its ports, and how each word of its rows eases. + /// Words the mark's program region needs: column descriptors and tables, ranges, code, + /// constants, end tables, the marker palette, ports, runs, page tables, and blends. pub(crate) fn packed_len(&self) -> u32 { let Some(compiled) = &self.compiled else { return 0 }; (self.tables.len() + 2 * compiled.program.ranges.len() + 4 * self.code.len() + self.constants.len() - + self.ends.len() + + super::ends::words(&self.ends) + compiled.drawing.markers.palette.len() - + self.ports.len()) as u32 + + self.ports.len() + + self.drawing().map_or(0, |d| 1 + (d.runs.len() + 1) * wgsl::RUN_WORDS as usize) + + self.words.len()) as u32 + self.stride() + self.extras() } - /// How each word of a row eases while the mark eases toward what its channels evaluate: a number - /// or a color its channels store mixes, a position's high words move by their offset, and what - /// else they store shows as evaluated; a word no channel stores keeps what it holds. + /// How each word of a row blends while the mark eases toward what its channels evaluate. Stored + /// numbers and colors interpolate, a position's high words move by their offset, and other + /// stored words snap to the new value. A word no channel stores keeps its value. fn blends(&self) -> Vec { - let mut blends = vec![wgsl::BLEND_KEEP; (self.stride() + self.extras()) as usize]; + let mut blends = vec![wgsl::BLEND_KEEP; self.width() as usize]; let mut set = |word: u32, blend: u32| { if let Some(b) = blends.get_mut(word as usize) { *b = blend; @@ -400,11 +593,11 @@ impl Shown { set(word + 3, wgsl::BLEND_SNAP); set(word + wgsl::VERTEX_FLAGS - wgsl::VERTEX_POSITION, wgsl::BLEND_SNAP); } - wgsl::OP_STORE_FLAG | wgsl::OP_STORE_CODE | wgsl::OP_STORE_END => set(word, wgsl::BLEND_SNAP), + wgsl::OP_STORE_FLAG | wgsl::OP_STORE_CODE => set(word, wgsl::BLEND_SNAP), _ => {} } } - // What evaluation writes of every vertex row: its flags' units, and its size's range. + // Evaluation writes these for every vertex row: its flags' units and its size range. if self.kind() == MarkKind::Vertices { set(wgsl::VERTEX_FLAGS, wgsl::BLEND_SNAP); set(wgsl::VERTEX_RANGE, wgsl::BLEND_SNAP); @@ -412,8 +605,8 @@ impl Shown { blends } - /// Appends the mark's words, which start at word `base` of the program buffer, and records where - /// each part lands. + /// Appends the mark's words, which start at program-buffer word `base`, and records each part's + /// offset. pub(crate) fn pack(&mut self, base: u32, out: &mut Vec) { let Some(compiled) = &self.compiled else { return }; let start = out.len(); @@ -427,19 +620,26 @@ impl Shown { let constants = at(out); out.extend(&self.constants); let ends = at(out); - out.extend(&self.ends); + super::ends::pack(&self.ends, &compiled.program.ends, ends, out); let palette = at(out); out.extend(&compiled.drawing.markers.palette); let port_info = if self.ports.is_empty() { wgsl::NONE } else { at(out) }; out.extend(&self.ports); let blends = at(out); out.extend(self.blends()); - self.offsets = Offsets { columns, ranges, code, constants, ends, palette, port_info, blends }; + let runs = if self.drawing().is_some() { at(out) } else { wgsl::NONE }; + out.extend(self.runs()); + let mut pages = [wgsl::NONE; 5]; + for (kept, blocks) in self.words.tables() { + pages[kept as usize] = at(out); + out.extend(blocks); + } + self.offsets = Offsets { columns, ranges, code, constants, ends, palette, port_info, blends, runs, pages }; } } -/// The row blocks of a mark whose rows evaluate next: every block, or those listed, the blocks of the -/// pages that changed. A block is the `BLOCK_ROWS` rows a page of every column holds. +/// The row blocks (pages of held rows) to process next: all of them, or a list of those that +/// changed. #[derive(Clone, PartialEq, Eq, Debug)] pub(crate) enum Blocks { All, @@ -454,7 +654,7 @@ impl Default for Blocks { impl Blocks { /// Adds the blocks `blocks` names, or every block for `None`. - fn add(&mut self, blocks: Option<&[u32]>) { + pub(crate) fn add(&mut self, blocks: Option<&[u32]>) { match (&mut *self, blocks) { (Blocks::All, _) => {} (Blocks::Listed(listed), Some(more)) => listed.extend_from_slice(more), @@ -462,9 +662,9 @@ impl Blocks { } } - /// The blocks of a mark of `rows` rows to evaluate, in order, each once. + /// The blocks within a mark's `rows` held rows, ascending and without repeats. pub(crate) fn of(&self, rows: u32) -> Vec { - let count = rows.div_ceil(wgsl::BLOCK_ROWS); + let count = rows.div_ceil(wgsl::PAGE_ROWS); match self { Blocks::All => (0..count).collect(), Blocks::Listed(listed) => { @@ -477,15 +677,16 @@ impl Blocks { } } -/// Kind words per row of a mark of `kind`. -pub(crate) fn kind_stride(kind: MarkKind) -> u32 { - match kind { - MarkKind::Vertices => wgsl::VERTEX_WORDS, - MarkKind::Edges => wgsl::EDGE_WORDS, - MarkKind::Faces => wgsl::FACE_WORDS, - MarkKind::Lines => wgsl::LINE_WORDS, - MarkKind::Areas => wgsl::AREA_WORDS, - } +/// The row count of each page of `held` rows, of which the first `live` are not exits. A `paged` +/// table gets a full page per slot, since each slot evaluates as a whole page. An unpaged table's +/// rows are split into pages. Exits follow in either case. +pub(crate) fn page_rows(live: u32, held: u32, paged: bool) -> Vec { + (0..held.div_ceil(wgsl::PAGE_ROWS)) + .map(|p| { + let start = p * wgsl::PAGE_ROWS; + if paged && start < live { wgsl::PAGE_ROWS } else { held.min(start + wgsl::PAGE_ROWS) - start } + }) + .collect() } fn shape_code(shape: Shape) -> u32 { diff --git a/crates/latkit/src/view/mod.rs b/crates/latkit/src/view/mod.rs index 5a629eb6..17c61b5f 100644 --- a/crates/latkit/src/view/mod.rs +++ b/crates/latkit/src/view/mod.rs @@ -1,8 +1,10 @@ -//! Views: a scene drawn from named data into any wgpu frame, answering input. +//! The view. Draws a scene from its sources into any wgpu frame and handles input. mod axes; mod camera; mod clock; +mod drawing; +mod ends; mod event; mod fit; mod focus; @@ -10,6 +12,7 @@ mod frame; mod gestures; mod graph; mod input; +mod kept; mod labels; mod legends; mod lists; @@ -21,14 +24,13 @@ mod program; mod query; mod resources; mod rig; -mod rows; -mod shadows; mod sync; mod timer; mod transition; mod uniforms; mod variants; mod wake; +mod words; use std::sync::Arc; use std::task::Waker; @@ -38,10 +40,10 @@ pub use camera::Camera; pub(crate) use camera::Projected; pub use event::{Cursor, End, Event, Fit, Hit, Phase, Sample, Stats, Trigger, Waiting}; pub use input::{Button, Device, Input, Key, Modifiers}; -pub(crate) use resources::{DEPTH, SHADE}; -pub(crate) use wake::Signal; +pub(crate) use resources::DEPTH; #[cfg(all(feature = "web", target_arch = "wasm32"))] -pub(crate) use {resources::samples, variants::common}; +pub(crate) use variants::common; +pub(crate) use wake::Signal; use camera::Area; use clock::Clock; @@ -58,127 +60,147 @@ use rig::Rig; use crate::derive::{Derivations, Derived}; use crate::expr::compile::Deps; use crate::scene::mark::Context; -use crate::{Data, Gpu, Name, Patch, Pending, Result, Row, Rows, Scene, Space}; +use crate::{Error, Gpu, Name, Patch, Pending, Result, Row, Rows, Scene, Snapshot, Source, Space}; -/// A frame to draw: where, a texture view of a format and size in physical pixels at a ratio of -/// physical pixels to CSS pixels, and when. +/// A frame to draw: the target texture view, its format and size, the pixel ratio, and the time. #[derive(Clone, Copy, Debug)] pub struct Frame<'a> { /// The texture view to draw into. pub target: &'a wgpu::TextureView, - /// Its format; sRGB formats are handled. + /// The target's format. Both sRGB and non-sRGB formats work. pub format: wgpu::TextureFormat, - /// Its width and height in physical pixels. + /// The target's width and height in physical pixels. pub size: [u32; 2], /// Physical pixels per CSS pixel. pub pixel_ratio: f32, - /// Seconds on the clock of whatever draws the frames: an animation frame's timestamp, a video - /// frame's time, a test's step. The view's motion moves by the steps between them, a quarter of - /// a second at most, and never back. + /// Seconds on the caller's clock: an animation frame's timestamp, a video frame's time, or a + /// test's step. Motion advances by the time since the previous frame, clamped to between 0 and + /// 0.25 seconds, so it never jumps or runs backward. pub time: f64, } -/// A scene drawn from named data: the application owns data and scene; the view owns camera, -/// coordinate, selection, and what the frame needs. It has no loop: [`render`](View::render) records -/// a frame into an encoder the application submits, and [`input`](View::input) takes the platform's -/// pointer and keys. +/// Draws a scene from its sources (named datasets) and handles input. +/// +/// The application owns the data and the scene. The view owns the camera, the axis coordinate, the +/// selection, and the GPU state for drawing. It has no loop of its own: +/// [`render`](View::render) records a frame into an encoder the application submits, and +/// [`input`](View::input) takes the platform's pointer and key events. /// /// ```no_run /// # async fn example() -> latkit::Result<()> { -/// use latkit::{Data, Gpu, Ids, Plane, Scene, Table, Vertices, View}; +/// use latkit::{Data, Gpu, Ids, Plane, Scene, Table, Vertices, View, field}; /// let gpu = Gpu::new().await?; -/// let mut data = Data::new("graph"); -/// data.apply(Table::new("Node", Ids::count(3)).field("x", vec![0.0, 1.0, 2.0]).field("y", vec![0.0, 1.0, 0.0]))?; -/// let mut view = View::new(&gpu, Scene::new(Plane::default()).draw(Vertices::of("graph", "Node").position("x", "y")))?; -/// view.set_data(data); +/// let graph = Data::new("graph"); +/// graph.apply(Table::new("Node", Ids::count(3)).field("x", vec![0.0, 1.0, 2.0]).field("y", vec![0.0, 1.0, 0.0]))?; +/// let nodes = Vertices::of("graph", "Node").position(field("x"), field("y")); +/// let mut view = View::new(&gpu, Scene::new(Plane::default()).draw(nodes))?; +/// view.set_sources([&graph])?; /// view.fit(latkit::Fit::All); +/// // The next frame draws this change: a view draws the latest version of each Data it is given. +/// graph.apply(Table::new("Node", [1_u64]).field("y", vec![2.0]))?; /// # Ok(()) } /// ``` pub struct View { gpu: Gpu, - /// Which of its GPU's views it is: what the GPU's ledger grants its share to. + /// This view's id among the GPU's views. The GPU's ledger keys budget shares by it. id: u64, scene: Scene, - /// Bumps whenever the scene changes. + /// Incremented whenever the scene changes. scene_version: u64, - /// The datasets given, and those the view fetches as the camera asks, by name. - sources: Vec, - /// What marks draw of each: the dataset as it is, or as drawn in part. - data: Vec, - /// What the view derives from the data and the scene: tables beside the tables it draws, row for - /// row, which marks read like data. + /// The sources the application gave, each a named dataset. + sources: Vec, + /// The version of each dataset the view draws, sorted by name: a given snapshot, the latest + /// version of a `Data` as of the last refresh, or the tiles loaded so far. + snapshots: Vec, + /// The parts of each paged table the camera chose to draw this frame. + choices: crate::derive::tiled::Choices, + /// Each paged table's drawing, by dataset and table. + drawings: crate::data::Keyed<(Name, Name), drawing::Drawn>, + /// The marks' rebuilt tables, per dataset. They replace the current tables once their pages + /// arrive. + staged: crate::data::Keyed, + /// Whether this frame's pages fit in the arena, and if not, how the next frame makes space. + room: frame::Room, + /// Set once the view has reported that its data exceeds the GPU budget. Until a new data + /// version publishes, a full arena and a due fit (with nothing drawn to measure) stop + /// requesting frames. + limited: bool, + /// Tables derived from the data and the scene, row for row with the drawn tables. Marks read + /// them like data. derived: Derivations, marks: Vec, camera: Camera, rig: Rig, /// CSS pixels kept clear at the top, right, bottom, and left. insets: [f32; 4], + /// The space a plot's axes take around the plot area, and the stamp of the scene and pixel + /// ratio it was measured for. + gutters: (u64, axes::Gutters), fitting: Fitting, - /// The value range a plot without one shows, fitted to its traces. + /// The value range fitted to the traces, shown when the plot camera sets none. plot_y: Option<[f64; 2]>, at: f64, clock: Clock, effect_time: f64, events: Events, - /// What landing work wakes, and the waker it is: readbacks, jobs, and fetches hold it. + /// The signal that finished work wakes, and a waker for it. Readbacks, jobs, and fetches keep + /// clones of the waker. signal: Arc, waker: Waker, resources: Option, - /// Marks' regions of the program buffer. + /// Each mark's word range in the program buffer. regions: Regions, - /// Marks came or went: row regions are laid out again. - regions_dirty: bool, /// Something visible changed since the last frame. changed: bool, - /// Whether the application was told the device was lost. + /// Whether the device loss has been reported. told_lost: bool, - /// The work under way the last frame stood in for. + /// The pending work the last frame showed placeholders for. waiting: Waiting, - /// What the scene asked the GPU to compile, by the targets its frames draw into. + /// The pipelines requested for each target the frames draw into. asked: Vec, - /// While what its frames draw with compiles, how many compiles had landed when it last looked: - /// it draws nothing new until they all have, and sleeps until the next lands. + /// While the draw pipelines compile, how many compiles had finished at the last check. The view + /// draws nothing new until they all finish, and sleeps until the next one finishes. compiling: Option, - /// Likewise while what lights rows, probes, and answers queries compiles: its frames draw without - /// lit rows, and probes and queries wait. + /// The same, while the pipelines for lit rows, probes, and queries compile. Meanwhile frames + /// draw without lit rows, and probes and queries wait. answering: Option, - /// Likewise while what its text draws with compiles: its frames draw without labels, glyphs, and - /// the overlay. + /// The same, while the text pipelines compile. Meanwhile frames draw without labels, glyphs, + /// and the overlay. texting: Option, - /// Who asked for the arranging under way. + /// Callers waiting for the arrangement in progress. arranging: Vec>>, - /// Whether the platform asks for reduced motion, which `Motion::System` follows. + /// Whether the platform requests reduced motion, which `Motion::System` follows. reduced_motion: bool, - /// The shadow map and the face normals, made from what they read. - shadow_map: Derived<()>, + /// The stamp of the inputs the face normals were last summed from. normals: Derived<()>, - /// Measures what frames cost the GPU, where it can. + /// Measures the GPU time of frames, where the device supports it. timer: Option, focus: Focus, probing: Probing, queries: query::Queries, - /// Words the marks' segment lists take at the front of the lists buffer. - segment_words: u32, + /// The number of words the marks' segment lists use at the start of the lists buffer. + list_words: u32, labels: labels::Labels, gestures: Gestures, - /// The pointer over the view: where, with what keys, and with what; `None` once it left. + /// The pointer over the view: its point, pressed modifier keys, and device. `None` once it + /// leaves. pointer: Option, - /// The size of the last frame, CSS pixels, and its pixel ratio. + /// The size of the last frame in CSS pixels, and its pixel ratio. size: [f64; 2], pixel_ratio: f32, - /// How each table draws within the view's share of the budget, and what that weighs. + /// How each table draws within the view's budget share, and the costs the plan used. planning: plan::Planning, - /// Pixel columns a trace spans at most: as many as the plan holds. + /// The most pixel columns a trace may span, set by the plan. trace_columns: u32, stats: Stats, } -/// Views made so far, which numbers each. +/// The number of views created, used to give each view its id. static VIEWS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); impl View { - /// A view drawing `scene` with `gpu`. Its marks draw once [`set_data`](View::set_data) gives them - /// their data. + /// Creates a view that draws `scene` with `gpu`. Its marks draw nothing until + /// [`set_sources`](View::set_sources) provides their datasets. Errors if the scene is invalid. pub fn new(gpu: &Gpu, scene: Scene) -> Result { scene.check()?; let camera = Camera::default_for(&scene.space); @@ -189,12 +211,18 @@ impl View { scene: Scene::new(scene.space.clone()), scene_version: 0, sources: Vec::new(), - data: Vec::new(), + snapshots: Vec::new(), + choices: crate::derive::tiled::Choices::default(), + drawings: crate::data::Keyed::default(), + staged: crate::data::Keyed::default(), + room: frame::Room::Found, + limited: false, derived: Derivations::default(), marks: Vec::new(), camera, rig: Rig::default(), insets: [0.0; 4], + gutters: (0, axes::Gutters::default()), fitting: Fitting::default(), plot_y: None, at: f64::INFINITY, @@ -205,7 +233,6 @@ impl View { signal, resources: None, regions: Regions::default(), - regions_dirty: true, changed: true, told_lost: false, waiting: Waiting::default(), @@ -215,13 +242,12 @@ impl View { texting: None, arranging: Vec::new(), reduced_motion: false, - shadow_map: Derived::default(), normals: Derived::default(), timer: timer::Timer::new(&gpu.0.device, &gpu.0.queue), focus: Focus::default(), probing: Probing::default(), queries: query::Queries::default(), - segment_words: 0, + list_words: 0, labels: labels::Labels::default(), gestures: Gestures::default(), pointer: None, @@ -231,29 +257,35 @@ impl View { trace_columns: u32::MAX, stats: Stats::default(), }; - view.show(scene)?; - // Views to come find what nearly every scene draws with compiled, once this one draws. + view.set_scene(scene)?; + // Queue the pipelines nearly every scene uses, so later views find them compiled. They + // compile once this view's draw pipelines are ready. #[cfg(all(feature = "web", target_arch = "wasm32"))] gpu.warm(); - // Until it draws, a view waits on everything its first frame would. + // Until its first frame, the view reports all the work that frame would wait on. view.waiting = view.under_way(); Ok(view) } - /// The scene drawn. + /// The scene the view draws. pub fn scene(&self) -> &Scene { &self.scene } - /// Draws `scene` instead. Marks that match one of the current scene by kind, data, table, and name - /// keep their state, and ease to a changed description; an invalid scene changes nothing. - pub fn show(&mut self, scene: Scene) -> Result<()> { + /// Replaces the scene, compiling its marks against the latest version of each + /// [`Data`](crate::Data) source. A mark that matches a current mark by kind, data, table, and + /// name keeps its state and eases to its new options. Errors if the scene is invalid or a mark + /// fails to compile, and the current scene stays. + pub fn set_scene(&mut self, scene: Scene) -> Result<()> { scene.check()?; let gpu = self.gpu.clone(); let mut inner = gpu.0.lock(); - // A mark compiles again when its theme or space changed. + self.refresh(&mut inner); + // Every mark recompiles if the theme, the kind of space, or the presence of a layout + // changed. + let kind_changed = std::mem::discriminant(&scene.space) != std::mem::discriminant(&self.scene.space); let context_changed = scene.theme != self.scene.theme - || std::mem::discriminant(&scene.space) != std::mem::discriminant(&self.scene.space) + || kind_changed || matches!(scene.layout, crate::Layout::None) != matches!(self.scene.layout, crate::Layout::None); let context = Context { theme: &scene.theme, space: &scene.space, layout: &scene.layout }; // Compile every new or changed mark before changing anything. @@ -265,7 +297,7 @@ impl View { continue; } let mut shown = Shown::new(mark.clone()); - shown.compile(&self.sources, context, &mut inner).map_err(|e| mark.within(i, e))?; + shown.compile(&self.snapshots, context, &mut inner).map_err(|e| e.at(i).at("marks"))?; fresh.push(Some(shown)); } let mut old: Vec> = std::mem::take(&mut self.marks).into_iter().map(Some).collect(); @@ -280,16 +312,17 @@ impl View { .iter_mut() .find(|s| s.as_ref().is_some_and(|s| s.mark.draws_like(mark))) .and_then(Option::take) - .expect("kept"); + .expect("the unchanged mark's previous state"); kept.mark = mark.clone(); return kept; }; - // A changed mark takes over its predecessor's rows, and eases from them. + // A changed mark takes over the rows of the old mark with its key, and eases from + // them. let before = old.iter_mut().find(|s| s.as_ref().is_some_and(|s| s.key == shown.key)); if let Some(mut before) = before.and_then(Option::take) { before.tables.release(&mut inner); - shown.inherit(&before); - if easing && shown.compiled.as_ref().map(|c| c.drawing.stride) == Some(before.stride()) { + shown.inherit(&mut before); + if easing && shown.compiled.is_some() { shown.begin(None); } } @@ -297,126 +330,205 @@ impl View { }) .collect(); for mut dropped in old.into_iter().flatten() { - dropped.tables.release(&mut inner); + dropped.release(&mut inner); + } + // Mark indices may have changed, so staged tables are dropped and rebuilt. + for (_, staged) in std::mem::take(&mut self.staged) { + staged.release(&mut inner); } for shown in self.marks.iter_mut().filter(|s| s.failure.is_none()) { - shown.failure = absent(shown, &self.sources, &self.derived.tiles).map(|error| (error, false)); + shown.failure = absent(shown, &self.snapshots, &self.derived.tiles).map(|error| (error, false)); } - self.regions_dirty = true; drop(inner); if !self.camera.fits(&scene.space) { self.camera = Camera::default_for(&scene.space); self.plot_y = None; } + // Fit bounds and running motion refer to the old space's positions, so they do not carry + // over to another kind of space. The next frame places the camera within the new space's + // zoom and pan limits. + if kind_changed { + self.rig.bounds = None; + self.rig.hold(); + } if matches!(scene.space, Space::Diagram(_)) { self.rig.orbit = 0.0; } self.scene = scene; self.scene_version += 1; - // What the new scene uses compiles from now, while its data syncs and its sheet arranges; what - // it shares landed already. + // Request the new scene's pipelines again. They compile while its data syncs and its sheet + // is arranged. Pipelines shared with the old scene are already compiled. self.asked.clear(); self.compiling = None; self.answering = None; self.texting = None; - self.ask(&mut gpu.0.lock(), (crate::gpu::FORMAT, resources::samples(self.pixel_ratio)), true); + // If the GPU's sample count is known, start compiling now. Otherwise the first frame asks. + if let Some(samples) = gpu.samples() { + self.ask(&mut gpu.0.lock(), (crate::gpu::FORMAT, samples), true); + } self.focus.dirty = true; self.gestures.keys.stale(); self.invalidate(); Ok(()) } - /// Gives the view the dataset `data`, replacing the one of the same name. Cheap: unchanged pages - /// are shared and only changed fields are evaluated again. A mark it no longer fits draws nothing, - /// and the next frame reports why, unless a scene shown before then replaces the mark: new data - /// and the scene reading it can be given in either order. - pub fn set_data(&mut self, data: Data) { + /// Replaces the view's sources, each a named dataset. A [`Data`](crate::Data) draws its latest + /// version, starting the frame after each patch applies. A [`Snapshot`] draws as given. + /// [`Tiles`](crate::Tiles) loads the pieces in view. + /// + /// This is cheap: unchanged pages are shared and only changed fields evaluate again. A mark + /// whose dataset no longer fits it draws nothing, and the next frame reports why unless a new + /// scene replaces that mark first. So the sources and the scene can be set in either order. + /// Errors if two sources share a name. + pub fn set_sources>(&mut self, sources: impl IntoIterator) -> Result<()> { + let sources: Vec = sources.into_iter().map(Into::into).collect(); + for (i, source) in sources.iter().enumerate() { + if sources[..i].iter().any(|other| other.name() == source.name()) { + return Err(Error::invalid(format!("two sources named '{}'", source.name())).at(i).at("sources")); + } + } let gpu = self.gpu.clone(); let mut inner = gpu.0.lock(); - self.take_data(data, &mut inner); + // Forget datasets no source provides any more, and those whose source switched to or from + // tiles: the view draws the new source from scratch. + let tiled = |source: &Source| matches!(source, Source::Tiles(_)); + let gone: Vec = self + .sources + .iter() + .filter(|old| !sources.iter().any(|new| new.name() == old.name() && tiled(new) == tiled(old))) + .map(|old| old.name().clone()) + .collect(); + let forgot = !gone.is_empty(); + for name in gone { + self.forget(&name, &mut inner); + } + for old in &self.sources { + if let Source::Data(data) = old { + data.watchers().unwatch(self.id); + } + } + for source in &sources { + match source { + Source::Data(data) => { + data.watchers().watch(self.id, &self.waker); + self.take_newer(data.snapshot(), &mut inner); + } + Source::Snapshot(snapshot) => self.take_newer(snapshot.clone(), &mut inner), + Source::Tiles(tiles) => self.derived.tiles.set(tiles.clone(), &mut inner.tiles), + } + } + self.sources = sources; + drop(inner); + // Marks that lost their tables need the program laid out again. + if forgot { + self.invalidate(); + } else { + self.wake(); + } + Ok(()) + } + + /// The sources the view draws. + pub fn sources(&self) -> &[Source] { + &self.sources } - /// Takes `data` with the GPU's state locked: marks whose schema it changed compile again; one that - /// fails keeps why until data fits it again, and the frame tells it. The frame draws the data, as - /// it is or in part. - fn take_data(&mut self, data: Data, inner: &mut crate::gpu::Inner) { - let at = match self.sources.binary_search_by(|d| d.name().cmp(data.name())) { + /// The version of dataset `name` the view draws: the latest version it took of a + /// [`Data`](crate::Data), a given snapshot, or the pieces of [`Tiles`](crate::Tiles) loaded for + /// its camera. `None` if the view has no such dataset. + pub fn snapshot(&self, name: &str) -> Option<&Snapshot> { + let i = self.snapshots.binary_search_by(|s| s.name().as_str().cmp(name)).ok()?; + Some(&self.snapshots[i]) + } + + /// Updates every source before the frame reads it: the tiles the camera and the plot window + /// need, the pieces that arrived for them, and the latest version of each changed `Data`. A + /// frame compiles marks, requests pipelines, derives, and draws from one version of each + /// dataset, so no pipeline reads a field added after the pipeline was requested. While a shot + /// keeps the view's data unchanged, `Data` sources stay at the versions taken when it began, + /// but tiles still arrive. + fn refresh(&mut self, inner: &mut crate::gpu::Inner) { + self.update_tiles(inner); + for data in self.derived.tiles.take_changed() { + self.take_data(data, inner); + } + if self.signal.holding() { + return; + } + for i in 0..self.sources.len() { + let Source::Data(data) = &self.sources[i] else { continue }; + let now = data.snapshot(); + self.take_newer(now, inner); + } + } + + /// Takes `snapshot` unless the view draws that version already. + fn take_newer(&mut self, snapshot: Snapshot, inner: &mut crate::gpu::Inner) { + if self.snapshot(snapshot.name()).is_none_or(|drawn| drawn.version() != snapshot.version()) { + self.take_data(snapshot, inner); + } + } + + /// Takes a new version of a dataset, with the GPU's state locked. Marks whose schema it changed + /// recompile. A mark that fails keeps its error until the data fits it again, and the next + /// frame reports the error. A caller outside a frame requests a frame to draw the data. + fn take_data(&mut self, data: Snapshot, inner: &mut crate::gpu::Inner) { + let at = match self.snapshots.binary_search_by(|d| d.name().cmp(data.name())) { Ok(i) => { - self.sources[i] = data; + self.snapshots[i] = data; i } Err(i) => { - self.sources.insert(i, data); + self.snapshots.insert(i, data); i } }; let context = Context { theme: &self.scene.theme, space: &self.scene.space, layout: &self.scene.layout }; - for shown in &mut self.marks { - if shown.stale(&self.sources[at]) { - let failed = shown.compile(&self.sources, context, inner).err(); - let failed = failed.or_else(|| absent(shown, &self.sources, &self.derived.tiles)); + for (i, shown) in self.marks.iter_mut().enumerate() { + if shown.stale(&self.snapshots[at]) { + if let Some(staged) = self.staged.get_mut(&shown.mark.style().data) { + staged.forget(i, inner); + } + let failed = shown.compile(&self.snapshots, context, inner).err(); + let failed = failed.or_else(|| absent(shown, &self.snapshots, &self.derived.tiles)); shown.failure = failed.map(|error| (error, false)); } } - // Rows may have moved under the focus and the keys. + // Rows may have moved under the focus and the keyboard-stepped hover. self.focus.dirty = true; self.gestures.keys.stale(); self.fitting.plot_due = true; - self.wake(); - } - - /// The dataset named `name`, as given, or as the view fetches it. - pub fn data(&self, name: &str) -> Option<&Data> { - self.source(name) } - /// The dataset named `name`, whole. - fn source(&self, name: &str) -> Option<&Data> { - self.sources.iter().find(|d| d.name() == name) - } - - /// What marks draw of the dataset named `name`: it as it is, or as drawn in part. - fn drawn(&self, name: &str) -> Option<&Data> { - self.data.iter().find(|d| d.name() == name) - } - - /// Fetches `tiles` as the camera asks for them, kept as the dataset they name, which the view - /// owns: marks draw it like any other, [`data`](View::data) reads it, and - /// [`remove_data`](View::remove_data) stops it. Tiles of the same name replace these. - pub fn set_tiles(&mut self, tiles: crate::Tiles) { - self.derived.tiles.set(tiles); - self.wake(); - } - - /// Stops drawing the dataset named `name`, given or fetched as tiles. - pub fn remove_data(&mut self, name: &str) { - self.derived.tiles.remove(name); - self.sources.retain(|d| d.name() != name); - self.data.retain(|d| d.name() != name); - let gpu = self.gpu.clone(); - let mut inner = gpu.0.lock(); - for shown in &mut self.marks { + /// Stops drawing dataset `name` until a source provides it again, and releases the GPU tables + /// its marks use. + fn forget(&mut self, name: &str, inner: &mut crate::gpu::Inner) { + self.derived.tiles.remove(name, &mut inner.tiles); + self.snapshots.retain(|d| d.name() != name); + self.drawings.retain(|key, _| key.0 != name); + for (i, shown) in self.marks.iter_mut().enumerate() { if shown.mark.style().data == *name { shown.failure = None; } if shown.compiled.as_ref().is_some_and(|c| c.schemas.iter().any(|(n, _)| n == name)) { - shown.tables.release(&mut inner); + if let Some(staged) = self.staged.get_mut(&shown.mark.style().data) { + staged.forget(i, inner); + } + shown.tables.release(inner); shown.compiled = None; shown.rows = 0; shown.exits = 0; - shown.drawn = None; + shown.holding = None; } } - drop(inner); - self.invalidate(); } - /// The axis coordinate shown: every sampled field reads its last frame at or before it. + /// The axis coordinate shown. Every sampled field shows its last frame at or before it. pub fn at(&self) -> f64 { self.at } - /// Shows axis coordinate `at`; only options that read sampled fields evaluate again. + /// Shows axis coordinate `at`. Only options that read sampled fields evaluate again. pub fn set_at(&mut self, at: f64) { if at.to_bits() != self.at.to_bits() { self.at = at; @@ -424,21 +536,31 @@ impl View { } } - /// Where the view looks. + /// The view's camera. pub fn camera(&self) -> &Camera { &self.camera } - /// Looks through `camera` at once. Like every setter, it tells nothing: the application knows. + /// Moves to `camera` at once, stopping any flight or coasting and replacing a fit not yet + /// applied. Like every setter, it emits no event. Ignored unless `camera` + /// [fits](Camera::fits) the scene's space. + /// + /// Every camera move stays within the camera's zoom and pan limits. On the earth, the limits + /// are those [`Earth`](crate::Earth) describes, for the part of the view inside the + /// [`insets`](View::insets). After a fit, the camera zooms at most 3 levels out and 8 in from + /// the fit, and its center stays within one box size of the box the fit framed. The view learns + /// its size from its first frame. When its size, insets, or space change, the next frame moves + /// the camera back within the limits and emits a [`Camera`](Event::Camera) event. pub fn set_camera(&mut self, camera: Camera) { if camera.fits(&self.scene.space) { + self.cancel_fit(); self.rig.hold(); self.place_camera(camera); } } - /// Tells the view whether the platform asks for reduced motion, as a browser's - /// `prefers-reduced-motion` does: `Motion::System` follows it. A canvas tells it itself. + /// Sets whether the platform requests reduced motion, like a browser's + /// `prefers-reduced-motion`. `Motion::System` follows it. A canvas sets it automatically. pub fn set_reduced_motion(&mut self, reduced: bool) { if reduced != self.reduced_motion { self.reduced_motion = reduced; @@ -446,8 +568,8 @@ impl View { } } - /// Whether the view moves on its own: eases, flights, coasting, orbits, label fades, comets, and - /// what reads `elapsed()`. + /// Whether motion is enabled, so the view animates on its own: eases, flights, coasting, + /// orbits, comets, and expressions that read `elapsed()`. pub(super) fn moves(&self) -> bool { self.moves_with(&self.scene.controls) } @@ -460,39 +582,49 @@ impl View { } } - /// What changes ease over: the scene's ease, or nothing where the view does not move on its own. + /// How long changes ease: the scene's ease, or zero when motion is disabled. pub(super) fn ease(&self) -> Duration { if self.moves() { self.scene.ease } else { Duration::ZERO } } - /// Eases to `camera` over twice the scene's ease, on the view's clock. - pub fn fly(&mut self, camera: Camera) { + /// Eases to `camera` on the view's clock, over `seconds`, or over twice the scene's ease when + /// `None`, replacing a fit not yet applied. The target stays within the zoom and pan limits + /// [`set_camera`](View::set_camera) describes. Ignored unless `camera` [fits](Camera::fits) the + /// scene's space. + pub fn fly(&mut self, camera: Camera, seconds: Option) { if camera.fits(&self.scene.space) { - let seconds = (self.ease().max(Duration::from_millis(1)) * 2).as_secs_f64(); - self.rig.fly(self.camera, camera, self.clock.now(), seconds); + self.cancel_fit(); + let least = Duration::from_millis(1); + let seconds = seconds.unwrap_or_else(|| self.ease().max(least) * 2).max(least).as_secs_f64(); + let to = camera.within(&self.reach()); + self.rig.fly(self.camera, to, self.clock.now(), seconds); self.wake(); } } - /// Where `position` (x, y, and elevation) lands on screen, CSS pixels from the top left; `None` - /// behind the camera. Overlays anchor to what the view draws with it. - pub fn point(&self, position: [f64; 3]) -> Option<[f32; 2]> { + /// The screen point of `position` (x, y, and elevation), in CSS pixels from the top left. + /// `None` behind the camera. Use it to anchor overlays to what the view draws. + pub fn screen(&self, position: [f64; 3]) -> Option<[f32; 2]> { let view = self.projected(); view.screen(view.world(position)).map(|s| [s[0] as f32, s[1] as f32]) } - /// The position on the ground under `point`: on a plane at elevation 0, on the earth its surface, - /// on a plot the values there; `None` where the pointer looks past it. - pub fn ground(&self, point: [f32; 2]) -> Option<[f64; 3]> { + /// The ground position under screen point `point` (CSS pixels from the top left). On a plane + /// it is the point at elevation 0, on the earth the point on the map or globe, and on a plot + /// the values there. `None` where the point misses the ground. + pub fn position(&self, point: [f32; 2]) -> Option<[f64; 3]> { let view = self.projected(); - view.surface(point.map(f64::from)).map(|w| view.position(w)) + view.ground(point.map(f64::from)).map(|w| view.position(w)) } - /// Lays every vertex row without a position out again by the scene's layout: a diagram's blocks, - /// or the points of a plane or the earth. Places a layout gave otherwise hold, so rows added later - /// are placed around the rest. Once the layout lands, the answer is a patch to each dataset - /// writing each row laid out into its mark's position fields (or its point `geometry`); applied, - /// the places are the data's own. A mark whose position is not fields is left out. + /// Runs the scene's layout again on every vertex row without a position: a diagram's blocks, or + /// the points on a plane or the earth. Without this call, laid-out positions persist and rows + /// added later are placed around them. + /// + /// Once the layout finishes, resolves to one patch per dataset. Each patch writes the laid-out + /// positions into the mark's position fields (or its point `geometry`). Applying the patches + /// makes the positions part of the data. Marks whose position is not a pair of fields are + /// skipped. pub fn arrange(&mut self) -> Pending> { let (pending, resolver) = Pending::new(); self.arranging.push(resolver); @@ -501,7 +633,7 @@ impl View { pending } - /// Answers the arranging asked for, once its layout landed. + /// Resolves the pending `arrange` calls once their layout has finished. fn answer_arranging(&mut self) { let Some(placed) = self.derived.take_arranged() else { return }; let patches = arranged(&self.scene, placed); @@ -522,13 +654,13 @@ impl View { self.wake(); } - /// CSS pixels kept clear at the top, right, bottom, and left: the camera centers and fits in what - /// remains, as when panels cover part of the view. + /// CSS pixels kept clear at the top, right, bottom, and left, for panels that cover part of the + /// view. The camera centers and fits within the rest. pub fn insets(&self) -> [f32; 4] { self.insets } - /// Keeps `insets` clear. + /// Sets the [`insets`](View::insets). pub fn set_insets(&mut self, insets: [f32; 4]) { if insets != self.insets { self.insets = insets; @@ -536,26 +668,28 @@ impl View { } } - /// Degrees of bearing a map or plane turns each second about its center. + /// The orbit speed: degrees of bearing per second that a map or plane turns about its center. pub fn orbit(&self) -> f64 { self.rig.orbit } - /// Turns the camera `degrees_per_second` about its center; zero stops it. A diagram never turns. + /// Turns the camera about its center at `degrees_per_second`. Zero stops it. A diagram never + /// turns. pub fn set_orbit(&mut self, degrees_per_second: f64) { let turns = degrees_per_second.is_finite() && !matches!(self.scene.space, Space::Diagram(_)); self.rig.orbit = if turns { degrees_per_second } else { 0.0 }; self.wake(); } - /// The rows selected, a set per table in the order each was first selected. + /// The selected rows, one set per table, in the order each table was first selected. pub fn selection(&self) -> Vec { Rows::group(self.focus.selection.iter().cloned()) } - /// Selects `rows`, and no others, telling nothing. Rows the data drops, or no pickable mark shows, - /// leave the selection with a [`Select`](Event::Select) event without a trigger. - pub fn select(&mut self, rows: impl IntoIterator) { + /// Selects exactly `rows`, without emitting an event. Rows that the data drops, or that no + /// pickable mark shows, later leave the selection with a [`Selection`](Event::Selection) event + /// that has no trigger. + pub fn set_selection(&mut self, rows: impl IntoIterator) { let rows = Rows::flatten(rows); if rows != self.focus.selection { self.focus.selection = rows; @@ -564,80 +698,74 @@ impl View { } } - /// What the user points at: the row under the pointer, or the row the keys stepped to. + /// The hovered row: the row under the pointer, or the row the keyboard stepped to. pub fn hovered(&self) -> Option<&Hit> { self.focus.hovered.as_ref() } - /// Every mark that draws nothing because its data no longer fits it, and why. + /// An error for every mark that draws nothing because its data no longer fits it. pub fn failures(&self) -> impl Iterator + '_ { - self.marks - .iter() - .enumerate() - .filter_map(|(i, shown)| Some(shown.mark.within(i, shown.failure.as_ref()?.0.clone()))) + self.marks.iter().enumerate().filter_map(|(i, shown)| Some(shown.failure.as_ref()?.0.clone().at(i).at("marks"))) } - /// The pointer the view asks for over what it shows. + /// The cursor the view requests for what is under the pointer. pub fn cursor(&self) -> Cursor { self.gestures.cursor(self.probing.hovered(), &self.scene) } - /// What the view did. + /// Counters of the view's work and its last frame. pub fn stats(&self) -> Stats { - let inner = self.gpu.0.lock(); let rows = |m: &Shown| { let style = m.mark.style(); - self.source(&style.data).and_then(|d| d.table(&style.table)).map_or(0, |t| t.len() as u64) - }; - let [tiles, tiles_loading] = self.derived.tiles.counts(); - // A table drawn through tiles leaves holes where tiles drawn before were: they draw nothing. - let drawn = |m: &Shown| match &m.drawn { - Some((_, table)) => (table.len() - table.ids.holes()) as u64, - None => u64::from(m.rows), + self.snapshot(&style.data).and_then(|d| d.table(&style.table)).map_or(0, |t| t.len() as u64) }; + let [tiles, tiles_loading] = self.derived.tiles.counts(&self.gpu.0.lock().tiles); Stats { rows: self.marks.iter().map(rows).sum(), - drawn: self.marks.iter().map(drawn).sum(), + drawn: self.marks.iter().map(|m| u64::from(m.steady_rows())).sum(), tiles, tiles_loading, - uploading: inner.arena.pending(), - resident: inner.arena.resident(), - budget: inner.budget.bytes.min(inner.arena.capacity()), - waiting: self.waiting, ..self.stats } } - /// Whether the last frame drew everything as the data, scene, and camera say, with nothing - /// standing in for work under way: no pipeline it draws with was compiling, no changed page - /// waited to upload, no tile drew its ancestor, - /// no table its first sample, no row stood where it was until a layout landed, no wire drew its - /// elbows, no area its earlier triangles, no label its estimate, and no fit was due. A shot or a - /// video waits for it; [`Stats::waiting`] says what it waits on. + /// Whether the last frame drew everything the data, scene, and camera describe, with no + /// placeholder for pending work. A frame is incomplete while a pipeline it draws with compiles, + /// a changed page waits to upload, a tile shows its ancestor, a large table shows an even + /// sample of its rows, a row keeps its old position until a layout arrives, a wire shows an + /// elbow until its route arrives, an area shows its previous triangles, a label shows its + /// estimated size, or a fit is due. Shots and videos wait until it is true. + /// [`waiting`](View::waiting) lists what is pending. pub fn complete(&self) -> bool { self.waiting.is_empty() } - /// What happened since the last call, in order: see [`Event`]. An intent whose detail a pick has - /// yet to bring waits for it, and the intents after it wait too. + /// The kinds of pending work the last frame showed placeholders for. Empty once the view is + /// [complete](View::complete). [`Waiting`](Event::Waiting) events report changes to it. + pub fn waiting(&self) -> Waiting { + self.waiting + } + + /// The events since the last call, in order. See [`Event`]. An intent that needs a pick result + /// waits until the pick finishes, and later intents wait behind it. pub fn events(&mut self) -> impl Iterator + use<> { self.events.drain() } - /// Woken when the view needs a frame without being asked: work it waited on lands, data or the - /// camera changes. + /// Sets the waker called when the view needs a frame on its own: when pending work arrives, or + /// the data or the camera changes. pub fn set_waker(&mut self, waker: Waker) { self.signal.set_waker(waker); } - /// What wakes the view: a shot drawing it waits on it. + /// The view's wake signal. A shot drawing the view waits on it. pub(crate) fn signal(&self) -> &Arc { &self.signal } - /// Whether a [`Shot`](crate::Shot) or a [`Video`](crate::Video) draws the view: nothing else - /// should draw it meanwhile, as a canvas or a window holds its last frame, since a view draws at - /// one size at a time. + /// Whether a [`Shot`](crate::Shot) or a [`Video`](crate::Video) is drawing the view. Nothing + /// else should draw it meanwhile, since a view draws at one size at a time. A canvas or window + /// keeps showing its last frame. pub fn lent(&self) -> bool { self.signal.lent() } @@ -647,30 +775,34 @@ impl View { &self.gpu } - /// The error a shot or a video fails on: the first one still untold, which stays for the view's - /// owner. + /// The error a shot or video fails with: the first error not yet taken from the events. It + /// stays queued for the view's owner. pub(crate) fn shot_error(&mut self) -> Option { self.tell_gpu_errors(); self.events.error().cloned() } - /// Whether the view can draw frames into `format` at `pixel_ratio` now: every pipeline its scene - /// draws with compiled. The first call for a target asks for every pipeline the scene uses, all at - /// once, what it draws with first; in a browser they compile side by side off the GPU's thread, - /// and each that lands wakes the view. Until what it draws with has, it draws nothing new, so - /// whatever shows its frames keeps the last, as a canvas does; until the rest has, it draws - /// without lit rows, and probes and queries wait. Natively it can always draw. - /// [`render`](View::render) asks for itself, drawing nothing until then. + /// Whether the view can draw frames into `format` at `pixel_ratio` now, because the scene's + /// draw pipelines for that target are compiled. + /// + /// The first call for a target requests every pipeline the scene uses, draw pipelines first. In + /// a browser they compile in parallel off the GPU thread, and each one that finishes wakes the + /// view. Until the draw pipelines are ready, the view draws nothing new, so a canvas keeps + /// showing the last frame. Until the rest are ready, frames omit text and lit rows, and probes + /// and queries wait. Natively the view can always draw. [`render`](View::render) checks this + /// itself and records nothing until it is true. pub fn ready(&mut self, format: wgpu::TextureFormat, pixel_ratio: f32) -> bool { self.ready_for(format, pixel_ratio, true) } - /// Whether frames into `format` at `pixel_ratio`, taking `input` or not, can draw now: see - /// [`ready`](View::ready). + /// Like [`ready`](View::ready), for frames that take `input` or not. It refreshes the sources + /// first, so the pipelines a newly added field needs are requested before a frame reads it. pub(crate) fn ready_for(&mut self, format: wgpu::TextureFormat, pixel_ratio: f32, input: bool) -> bool { let gpu = self.gpu.clone(); + let samples = gpu.samples_for(pixel_ratio); let mut inner = gpu.0.lock(); - let draws = self.pipelines_ready(&mut inner, format, resources::samples(pixel_ratio), input); + self.refresh(&mut inner); + let draws = self.pipelines_ready(&mut inner, format, samples, input); let answers = self.answers_ready(&inner); let texts = self.texts_ready(&inner); let landed = inner.pipelines.landed(); @@ -686,74 +818,79 @@ impl View { draws } - /// Whether probes and queries can run: what they and the lit rows' draws use compiled. + /// Whether probes and queries can run: the probe, query, and lit-row pipelines are compiled. fn answers(&self) -> bool { self.answering.is_none() } - /// Whether frames draw text: what labels, glyphs, and the overlay use compiled. + /// Whether frames draw text: the label, glyph, and overlay pipelines are compiled. fn texts(&self) -> bool { self.texting.is_none() } - /// Whether what its frames draw with is still compiling: it draws nothing new until all of it - /// landed. + /// Whether the draw pipelines are still compiling. The view draws nothing new until they have + /// all finished. #[cfg(all(feature = "web", target_arch = "wasm32"))] pub(crate) fn compiling(&self) -> bool { self.compiling.is_some() } - /// Whether a frame would differ from the last one: something changed, moves, or streams in, or - /// work the view waited on landed. While it only waits, it needs none: what it waits on wakes it. + /// Whether a new frame would differ from the last one: something changed, animates, or streams + /// in, or pending work arrived. While the view only waits, it needs no frame. The work it waits + /// on wakes it. pub fn needs_render(&self) -> bool { let landed = || self.gpu.0.lock().pipelines.landed(); - // While what it draws with compiles, it draws nothing new: each compile landing wakes it. + // While draw pipelines compile, the view draws nothing new. Each finished compile wakes it. if let Some(seen) = self.compiling { return landed() != seen; } - // What moves on its own, only while the view moves on its own: under reduced motion an orbit, - // comets, and what reads `elapsed()` hold still. + // Self-driven motion counts only while motion is enabled: under reduced motion, orbits, + // comets, and `elapsed()` hold still. let moves = self.moves(); self.changed || self.signal.landed() + || self.derived.tiles.pending() || self.rig.easing() || (self.rig.orbit != 0.0 && moves) - || self.fitting.due() - // Probes and queries wait for what answers them to compile, which wakes the view as it lands. + || (self.fitting.due() && !self.limited) + // Probes and queries wait for their pipelines, which wake the view as they compile. || (self.answers() && (self.probing.due() || self.queries.due())) || self.answering.is_some_and(|seen| landed() != seen) || self.texting.is_some_and(|seen| landed() != seen) - || self.derived.tiled.due() - || self.labels.fading(self.clock.now()) - // A mark evaluates once its columns synced: until then, the pages they wait for wake it. + || self.fading() + // A mark evaluates once its columns are synced. Until then, the pages they wait for + // wake the view. || self.marks.iter().any(|m| (m.drawable() && (m.dirty != 0 || m.reduce)) || m.begin.is_some() || m.transition.is_some()) || (moves && self.marks.iter().any(|m| m.compiled.as_ref().is_some_and(|c| c.program.reads().intersects(Deps::TIME)))) || (moves && self.flowing()) || self.gpu.0.lock().arena.uploading() + || self.room != frame::Room::Found } - /// Whether the view wants a frame: one that would differ ([`needs_render`](View::needs_render)), - /// or one noticing a touch held long enough for a menu. A loop drawing on demand draws while this - /// holds, as a canvas does; a video holds the frames that only notice. + /// Whether the view wants a frame: one that would differ + /// ([`needs_render`](View::needs_render)), or one that only checks whether a touch has been + /// pressed long enough to open a menu. A loop that draws on demand, like a canvas, draws while + /// this is true. A video repeats its last frame rather than draw frames that only check. pub fn wants_frame(&self) -> bool { self.needs_render() || (self.compiling.is_none() && self.holding()) } - /// Whether comets travel along a mark's rows: one that flows draws, and the theme shows them. + /// Whether comets travel along some mark's rows: a flowing mark is drawable, and the theme's + /// flow color is visible. fn flowing(&self) -> bool { self.scene.theme.flow.color.a > 0.0 && self.marks.iter().any(|m| m.drawable() && m.mark.flows()) } - /// Whether the next frame, taking `input` or not, does work this one left: pages to upload past - /// the frame's budget, rows to lay out, a fit, probe, or query due, the tiles the camera wants, or - /// a transition to begin. Nothing else would wake the view for it: jobs, texts, fetches, and - /// readbacks do as they land. + /// Whether this frame left work for the next one (taking `input` or not): pages over the + /// frame's upload budget, pages that did not fit in the arena, a due fit, probe, or query, tile + /// pieces to take, or a transition to begin. Nothing else would wake the view for these. Jobs, + /// texts, fetches, and readbacks wake it themselves when they arrive. fn due(&self, input: bool) -> bool { self.gpu.0.lock().arena.uploading() - || self.regions_dirty - || self.fitting.due() + || self.room != frame::Room::Found + || (self.fitting.due() && !self.limited) || (input && self.answers() && (self.probing.due() || self.queries.due())) - || self.derived.tiled.due() + || self.derived.tiles.pending() || self.marks.iter().any(|m| m.begin.is_some()) } @@ -762,54 +899,61 @@ impl View { self.signal.ask(); } - /// The work under way a frame drawn now stands in for: see [`complete`](View::complete). A mark - /// waits for a dataset it names until it is given; one that failed, or that draws tiles the view - /// fetches, which may never hold its table, stands in for nothing. + /// The pending work a frame drawn now would show placeholders for. See + /// [`complete`](View::complete). A mark waits for its dataset until a source provides it. A + /// mark that failed, or whose dataset comes from tiles (which may never contain its table), + /// waits for nothing. fn under_way(&self) -> Waiting { let marks = || self.marks.iter().filter(|m| m.compiled.is_some()); let given = |m: &Shown| { let data = &m.mark.style().data; - m.compiled.is_some() || m.failure.is_some() || self.source(data).is_some() || self.derived.tiles.owns(data) + m.compiled.is_some() + || m.failure.is_some() + || self.snapshot(data).is_some() + || self.derived.tiles.owns(data) }; Waiting { compiling: self.asked.is_empty() || self.compiling.is_some() || self.answering.is_some() || self.texting.is_some(), - data: !self.marks.iter().all(given) || marks().any(|m| !m.tables.synced() || m.tables.waiting()), + data: !self.marks.iter().all(given) + || marks().any(|m| !m.tables.synced()) + || self.staged.values().any(|s| !s.is_empty()), derived: !self.derived.complete(), labels: !self.labels.complete(), fit: self.fitting.due() || self.fitting.reading(), - marks: self.regions_dirty || marks().any(|m| m.counting() || m.begin.is_some()), + marks: marks().any(|m| m.counting() || m.begin.is_some()), } } - /// Whether something eases toward where it goes: a transition, a flight, coasting, or a label - /// fade. An orbit, a comet, and anything else that moves for as long as it is shown never settles, - /// and is not counted. + /// Whether something is easing to rest: a transition, a flight, coasting, or a tile fade. + /// Orbits, comets, and other continuous motion never settle and do not count. pub(crate) fn easing(&self) -> bool { - self.rig.easing() - || self.labels.fading(self.clock.now()) - || self.marks.iter().any(|m| m.begin.is_some() || m.transition.is_some()) + self.rig.easing() || self.fading() || self.marks.iter().any(|m| m.begin.is_some() || m.transition.is_some()) + } + + /// Whether tiles of a paged table are fading in or out. + fn fading(&self) -> bool { + self.drawings.values().any(|d| d.current.as_ref().is_some_and(|c| c.fades)) } fn invalidate(&mut self) { - self.regions_dirty = true; self.regions.relayout = true; self.fitting.plot_due = true; self.wake(); } - /// The part of the view the camera frames: inside the insets, and on a plot inside its axes. + /// The part of the view the camera frames, in CSS pixels: inside the insets, and on a plot + /// inside its axes. At least one pixel each way. fn area(&self) -> Area { let [w, h] = self.size; let [top, right, bottom, left] = self.insets.map(f64::from); let mut area = [left, top, w - right, h - bottom]; - if let Space::Plot(plot) = &self.scene.space { + if matches!(self.scene.space, Space::Plot(_)) { // A plot's axes start below its title, which the other spaces draw over. - let gutters = axes::gutters(plot, &self.scene.theme); - let top = area[1] + self.title_band() + gutters[0]; - area = [area[0] + gutters[3], top, area[2] - gutters[1], area[3] - gutters[2]]; + let [top, right, bottom, left] = self.gutters.1.room; + area = [area[0] + left, area[1] + self.title_band() + top, area[2] - right, area[3] - bottom]; } if area[2] - area[0] < 1.0 { area[2] = area[0] + 1.0; @@ -832,7 +976,8 @@ impl View { self.projected_with(&self.drawn_camera()) } - /// What `camera` would show, a plot's missing value range filled as the frame fills it. + /// The projection through `camera`. A plot camera without a value range uses the fitted one, + /// as the frame does. fn projected_with(&self, camera: &Camera) -> Projected { let camera = match *camera { Camera::Plot { x, y: None } => Camera::Plot { x, y: Some(self.plot_y.unwrap_or([0.0, 1.0])) }, @@ -848,9 +993,19 @@ impl Drop for View { let gpu = self.gpu.clone(); let mut inner = gpu.0.lock(); for shown in &mut self.marks { - shown.tables.release(&mut inner); + shown.release(&mut inner); + } + for (_, staged) in std::mem::take(&mut self.staged) { + staged.release(&mut inner); } + self.derived.tiles.release(&mut inner.tiles); inner.ledger.leave(self.id); + drop(inner); + for source in &self.sources { + if let Source::Data(data) = source { + data.watchers().unwatch(self.id); + } + } } } @@ -864,10 +1019,10 @@ impl std::fmt::Debug for View { } } -/// The patches writing rows laid out into the data, by dataset: each row's place into the position -/// fields of the vertex mark drawing its table, or its point geometry. +/// One patch per dataset that writes each laid-out row's position into the position fields of the +/// vertex mark drawing its table, or into its point geometry. fn arranged(scene: &Scene, placed: Vec<(Row, [f64; 2])>) -> Vec { - /// A table's rows laid out: its dataset and name, their ids, and their places. + /// A table's laid-out rows: its dataset and table names, the row ids, and their positions. type Laid = ((Name, Name), Vec, Vec<[f64; 2]>); let mut tables: Vec = Vec::new(); for (row, place) in placed { @@ -905,10 +1060,10 @@ fn arranged(scene: &Scene, placed: Vec<(Row, [f64; 2])>) -> Vec { patches } -/// Why `shown`, whose dataset is given, draws nothing: the table it names is absent. A dataset the -/// view fetches as tiles fills its tables as they land, so a table it lacks is not missing. -fn absent(shown: &Shown, sources: &[Data], tiles: &crate::derive::tiles::TileSets) -> Option { +/// The error for a mark `shown` whose dataset is present but lacks the table it names. A dataset +/// from tiles gains tables as pieces arrive, so a table it lacks is not an error. +fn absent(shown: &Shown, snapshots: &[Snapshot], tiles: &crate::tiles::TileSets) -> Option { let style = shown.mark.style(); - let data = sources.iter().find(|d| *d.name() == style.data)?; + let data = snapshots.iter().find(|d| *d.name() == style.data)?; (shown.compiled.is_none() && !tiles.owns(&style.data)).then(|| shown.mark.absent_from(data)).flatten() } diff --git a/crates/latkit/src/view/passes.rs b/crates/latkit/src/view/passes.rs index 72a02fba..97e4e2da 100644 --- a/crates/latkit/src/view/passes.rs +++ b/crates/latkit/src/view/passes.rs @@ -1,25 +1,37 @@ -//! A frame's passes: compute evaluates dirty channels over the row blocks that changed, eases what -//! marks show toward what changed, reduces fitted domains, numbers segments, sums -//! face normals when their corners moved, and measures extents when asked; render draws the -//! backdrop, every mark's primitive and ports in scene order with depth, the labels, and the -//! overlay; a probe, when one waits, reads what the frame drew through the same geometry. +//! A frame's GPU passes. +//! +//! The compute pass freezes the rows' current state for transitions starting this frame, expands +//! paged marks' drawn rows from their runs, evaluates dirty channels over the changed row blocks +//! (reducing fitted domains first), resolves endpoints, eases marks in transition, moves comets, +//! numbers segments, sums face normals when their corners moved, and measures extents when asked. +//! +//! The render pass draws the backdrop, then every mark's primitive and ports in scene order with +//! depth, then the labels and the overlay. A pending probe then reads what the frame drew, through +//! the same geometry. -use super::resources::{EyeSlot, MARK_STRIDE}; +use super::resources::MARK_STRIDE; +use super::sync::Source; +use super::variants::{CARRY, EASE, EXPAND_RUNS, GATHER, MarkVariants, NORMALS, TRAVEL}; +use super::words::Kept; use super::{Frame, View}; use crate::Space; +use crate::data::Field; use crate::derive::stamp_of; +use crate::expr::compile::{Deps, Reads}; +use crate::expr::cpu::Rows; use crate::gpu::Inner; -use crate::gpu::pipelines::Variant; +use crate::gpu::pipelines::{Asked, Variant}; use crate::gpu::readback::Readback; use crate::gpu::shaders::{self as wgsl, Shader}; use crate::gpu::work::Count; use crate::scene::mark::{MarkKind, Primitive}; const WORKGROUP: u32 = 256; -/// Workgroups one dimension of a dispatch holds on every device. +/// The most workgroups per dispatch dimension that every device supports. const MAX_GROUPS: u32 = 65_535; -/// The target pixels of `area`: a plot's lines draw only inside its axes. +/// The scissor rectangle `[x, y, width, height]`, in target pixels, covering `area`. A plot's lines +/// draw only inside its axes. fn scissor(area: super::camera::Area, frame: &Frame<'_>) -> [u32; 4] { let [w, h] = frame.size.map(f64::from); let r = f64::from(frame.pixel_ratio); @@ -30,8 +42,8 @@ fn scissor(area: super::camera::Area, frame: &Frame<'_>) -> [u32; 4] { [x0 as u32, y0 as u32, (x1 - x0) as u32, (y1 - y0) as u32] } -/// The main render pass: cleared to `background`, multisampled into `view` and resolved into -/// `resolve` when there is one, its depth kept when `probing` reads it. +/// Begins the main render pass: cleared to `background`, drawn into `view` and resolved into +/// `resolve` when multisampling, with depth stored only when `probing` reads it. fn main_pass<'e>( encoder: &'e mut wgpu::CommandEncoder, view: &wgpu::TextureView, @@ -66,8 +78,8 @@ fn main_pass<'e>( }) } -/// The draws a frame recorded, and the instances they draw: at most, for a draw its segment list -/// counts on the GPU. +/// The draws a frame recorded and the instances they draw. For a draw whose segment list counts its +/// instances on the GPU, the count is an upper bound. #[derive(Clone, Copy, Default, Debug)] pub(super) struct Drawn { pub(super) draws: u64, @@ -75,7 +87,7 @@ pub(super) struct Drawn { } impl Drawn { - /// Draws `instances` of `corners` corners each, counted. + /// Draws and counts `instances` instances of `corners` vertices each. pub(super) fn draw(&mut self, pass: &mut wgpu::RenderPass<'_>, corners: u32, instances: u32) { pass.draw(0..corners, 0..instances); self.count(instances); @@ -87,8 +99,8 @@ impl Drawn { } } -/// What every mark's draw in the main pass shares: the device and target, the bind groups, the -/// lists buffer, and a plot's axes, which its lines clip to. +/// State that every mark's draw in the main pass shares: the device and target, the bind groups, +/// the lists buffer, and the plot area that lines clip to. struct Marks<'a> { device: &'a wgpu::Device, format: wgpu::TextureFormat, @@ -101,7 +113,7 @@ struct Marks<'a> { } impl Marks<'_> { - /// Draws mark `i`, `shown`: every instance, or with `lit` the rows its focus lights, glowing. + /// Draws mark `i`: every instance, or with `lit`, only the rows its focus lights, glowing. fn draw( &self, pass: &mut wgpu::RenderPass<'_>, @@ -111,29 +123,19 @@ impl Marks<'_> { shown: &super::marks::Shown, lit: bool, ) { - let Some(compiled) = &shown.compiled else { return }; - let custom = compiled.drawing.markers.custom.is_some(); - let areas = shown.path() == wgsl::PATH_GEOMETRY; - let (shader, corners) = match (compiled.drawing.primitive, lit) { - (Primitive::Points, false) if custom => (Shader::PointsCustom, 4), - (Primitive::Points, false) => (Shader::Points, 4), - (Primitive::Points, true) if custom => (Shader::PointsCustomFocus, 4), - (Primitive::Points, true) => (Shader::PointsFocus, 4), - (Primitive::Strokes, false) => (Shader::Strokes, 4), - (Primitive::Strokes, true) => (Shader::StrokesFocus, 4), - (Primitive::Triangles, false) if areas => (Shader::Areas, 3), - (Primitive::Triangles, false) => (Shader::Triangles, 3), - (Primitive::Triangles, true) if areas => (Shader::AreasFocus, 3), - // A face lights by its color alone. - (Primitive::Triangles, true) => return, + if shown.compiled.is_none() { + return; + } + let variants = MarkVariants::shown(shown); + let (Some(shader), Some(asked)) = (variants.shader(lit), variants.draw(lit, self.format, self.samples)) else { + return; }; - let pipeline = if let Some(source) = &compiled.drawing.markers.custom { - inner.pipelines.custom(self.device, source, shader, self.format, self.samples) - } else { - let variant = Variant::draw(shader, self.format, self.samples).path(shown.path()); - Some(inner.pipelines.render_pipeline(self.device, variant)) + let pipeline = match asked { + Asked::Shader(variant) => Some(inner.pipelines.render(self.device, variant)), + Asked::Custom(source, variant) => inner.pipelines.custom(self.device, &source, variant), }; let Some(pipeline) = pipeline else { return }; + let corners = shown.kind().primitive().corners(); let group = &self.passes.iter().find(|(s, _)| *s == shader).expect("listed").1; pass.set_bind_group(1, &self.groups.mark, &[offset(i)]); let clip = self.clip.filter(|_| shown.kind() == MarkKind::Lines); @@ -152,8 +154,8 @@ impl Marks<'_> { } } - /// Draws the ports of every row of mark `i`, `shown`, where its rows have them: after its rows, and - /// again after the rows its focus lights, which they sit on. + /// Draws the ports of every row of mark `i`, if its rows have ports. Called after its rows, and + /// again after its lit rows, since ports sit on top of them. fn ports( &self, pass: &mut wgpu::RenderPass<'_>, @@ -163,44 +165,73 @@ impl Marks<'_> { shown: &super::marks::Shown, ) { let ports = shown.port_count(); - if ports == 0 || !shown.compiled.as_ref().is_some_and(|c| c.drawing.primitive == Primitive::Points) { + if ports == 0 || shown.compiled.is_none() || shown.kind() != MarkKind::Vertices { return; } let group = &self.passes.iter().find(|(s, _)| *s == Shader::PointsPorts).expect("listed").1; pass.set_bind_group(1, &self.groups.mark, &[offset(i)]); - pass.set_pipeline(&inner.pipelines.render(self.device, Shader::PointsPorts, self.format, self.samples)); + let variant = Variant::draw(Shader::PointsPorts, self.format, self.samples); + pass.set_pipeline(&inner.pipelines.render(self.device, variant)); pass.set_bind_group(2, group, &[]); drawn.draw(pass, 4, (shown.rows + shown.exits) * ports); } } -/// What a frame's compute pass does. +/// The marks each step of a frame's compute pass covers. struct Work { + /// Marks whose transitions start this frame, freezing their rows' current state. + frozen: Vec, evaluated: Vec, - /// Marks whose ease pass writes what they show from their frozen and settled state. + /// Marks whose rows resolve changed endpoints. + resolved: Vec, + /// Marks whose ease pass writes their shown state from their frozen and settled words. eased: Vec, segments: Vec, faces: Vec, measured: Vec, - /// Marks whose comets carry their travel over to rows a transition moved, and that travel. + /// Marks whose comets carry their travel over to rows a transition moved. `travelled` lists + /// marks whose comets move this frame. carried: Vec, travelled: Vec, + /// Marks whose drawn rows' words are rebuilt from their runs. + expanded: Vec, } -/// Dispatches a workgroup for every `WORKGROUP` of `items`, if there are any; past one dimension's -/// workgroups, blocks of `BLOCK_ROWS` items along the second, as `item` in `lib/types.wesl` reads them. +/// Dispatches one workgroup per `WORKGROUP` items, if there are any. Past one dimension's limit, it +/// dispatches rows of `DISPATCH_ROWS` items along the second dimension, as `item` in +/// `lib/types.wesl` expects. pub(super) fn dispatch(pass: &mut wgpu::ComputePass<'_>, items: u32) { let groups = items.div_ceil(WORKGROUP); if groups > MAX_GROUPS { - pass.dispatch_workgroups(wgsl::BLOCK_ROWS / WORKGROUP, items.div_ceil(wgsl::BLOCK_ROWS), 1); + pass.dispatch_workgroups(wgsl::DISPATCH_ROWS / WORKGROUP, items.div_ceil(wgsl::DISPATCH_ROWS), 1); } else if groups > 0 { pass.dispatch_workgroups(groups, 1, 1); } } -/// Dispatches over `blocks` listed row blocks of a mark of `rows` rows: a block's rows wide. -fn dispatch_blocks(pass: &mut wgpu::ComputePass<'_>, rows: u32, blocks: u32) { - let wide = rows.min(wgsl::BLOCK_ROWS).div_ceil(WORKGROUP); +/// The row blocks `shown` evaluates: those its dirty channels cover, and those whose state words +/// start over, flagged with `FRESH`. +fn listed(shown: &super::marks::Shown) -> Vec { + let count = shown.live().div_ceil(wgsl::PAGE_ROWS); + let mut fresh: Vec = shown.fresh.iter().copied().filter(|&b| b < count).collect(); + fresh.sort_unstable(); + fresh.dedup(); + let mut blocks = if shown.dirty != 0 || shown.reduce { shown.blocks.of(shown.live()) } else { Vec::new() }; + blocks.extend(&fresh); + blocks.sort_unstable(); + blocks.dedup(); + for block in &mut blocks { + if fresh.binary_search(block).is_ok() { + *block |= wgsl::FRESH; + } + } + blocks +} + +/// Dispatches over `blocks` listed row blocks of a mark with `rows` held rows: x spans one block's +/// rows, and y the blocks. +pub(super) fn dispatch_blocks(pass: &mut wgpu::ComputePass<'_>, rows: u32, blocks: u32) { + let wide = rows.min(wgsl::PAGE_ROWS).div_ceil(WORKGROUP); if wide > 0 && blocks > 0 { pass.dispatch_workgroups(wide, blocks, 1); } @@ -211,39 +242,71 @@ pub(super) fn offset(index: usize) -> u32 { (index as u64 * MARK_STRIDE) as u32 } +/// The minimum and maximum finite value of `input` over `rows`, encoded as `lib/reduce.wesl` keeps +/// them: the complement of the minimum's key, then the maximum's key. `[0, 0]` when no row has a +/// finite value. +fn fit(rows: &Rows<'_>, input: &crate::Expr) -> [u32; 2] { + use crate::order::number32; + let (mut least, mut most) = (f32::INFINITY, f32::NEG_INFINITY); + for x in rows.numbers(input, |v| v as f32) { + if x.is_finite() { + (least, most) = (least.min(x), most.max(x)); + } + } + if least > most { [0, 0] } else { [!number32(least), number32(most)] } +} + impl super::marks::Shown { - /// Segments of each row's path, for strokes whose rows all have as many. + /// Whether its fitted domains are computed on the CPU over every row of its paged table: it has + /// a drawing, and the domains read only its own data, nothing the view shows or derives. + pub(crate) fn fits_data(&self) -> bool { + let Some(compiled) = &self.compiled else { return false }; + let program = &compiled.program; + let viewed = Deps::TIME | Deps::POINTER | Deps::FOCUS | Deps::CAMERA | Deps::DOMAIN; + let derived = |reads: &Reads| { + program + .columns + .iter() + .enumerate() + .any(|(c, column)| column.derived && reads.columns & Reads::column(c) != 0) + }; + self.drawing().is_some() + && !program.domains.is_empty() + && program.domains.iter().all(|reads| !reads.deps.intersects(viewed) && !derived(reads)) + } + + /// Segments per row, for strokes whose rows all have the same count. pub(crate) fn segments_per_row(&self) -> u32 { if self.compiled.is_none() { return 0; } match self.kind() { - // A trace's points count the corners steps make. + // A trace's point count includes the corners its steps make. MarkKind::Lines => self.tables.traced().saturating_sub(1), _ => 1, } } - /// Instances its primitive draws: rows and exits, or every segment of their paths; at most what - /// its segment list keeps. + /// Instances its primitive draws: rows and exits, or every segment of their paths. With a + /// segment list, the entries the list keeps, an upper bound. pub(crate) fn instances(&self) -> u32 { if let Some(list) = &self.segments.list { return list.kept; } let rows = self.rows + self.exits; - match self.compiled.as_ref().map(|c| c.drawing.primitive) { + match self.compiled.as_ref().map(|_| self.kind().primitive()) { Some(Primitive::Strokes) => rows * self.segments_per_row(), Some(_) => rows, None => 0, } } - /// Instances a probe or a measure reaches: live rows and their ports only. + /// Instances that probes and measures cover: live rows and their ports, not exits. pub(crate) fn live_instances(&self) -> u32 { if let Some(list) = &self.segments.list { return list.kept; } - match self.compiled.as_ref().map(|c| c.drawing.primitive) { + match self.compiled.as_ref().map(|_| self.kind().primitive()) { Some(Primitive::Strokes) => self.rows * self.segments_per_row(), Some(Primitive::Points) => self.rows * (1 + self.port_count()), Some(_) => self.rows, @@ -251,12 +314,7 @@ impl super::marks::Shown { } } - /// The code of the primitive it draws. - pub(crate) fn primitive(&self) -> u32 { - self.compiled.as_ref().map_or(0, |c| c.drawing.primitive.code()) - } - - /// Records the draw of its instances: as many as its segment list holds, when it has one. + /// Records the draw of its instances, indirect from its segment list when it has one. pub(super) fn record( &self, pass: &mut wgpu::RenderPass<'_>, @@ -275,15 +333,15 @@ impl super::marks::Shown { } impl View { - /// Evaluates the dirty channels of marks `work`, each over its dirty row blocks, its fitted - /// domains reduced first where they changed: into its state, bound by `state`, or while it eases - /// into its settled mirror, bound by `settling`. Leaves `state` bound. + /// Evaluates the dirty channels of marks `work` over their dirty row blocks, first reducing the + /// fitted domains that changed. Channels write a mark's state, or its settled words while it + /// eases. `state` is the expression machine's pass group. fn evaluate( &mut self, pass: &mut wgpu::ComputePass<'_>, inner: &mut Inner, group: &wgpu::BindGroup, - (state, settling): (&wgpu::BindGroup, Option<&wgpu::BindGroup>), + state: &wgpu::BindGroup, work: &[usize], ) { if work.is_empty() { @@ -293,29 +351,34 @@ impl View { let step = super::resources::step; for &i in work { let shown = &mut self.marks[i]; - let into = settling.filter(|_| shown.transition.is_some()).unwrap_or(state); - pass.set_pipeline(&inner.pipelines.compute_pipeline(device, Variant::channels(shown.reads()))); + let into = state; + pass.set_pipeline(&inner.pipelines.compute(device, Variant::channels(shown.reads()))); pass.set_bind_group(1, group, &[offset(i)]); - if shown.reduce && shown.rows > 0 { + let rows = shown.live(); + // A changed domain dirties every block, so the reductions run over the listed blocks. A + // paged table whose domains read only its data fits them on the CPU, over every row. + if shown.reduce && rows > 0 && !shown.fits_data() { + debug_assert_eq!(shown.blocks, super::marks::Blocks::All, "a changed domain dirties every block"); pass.set_bind_group(2, into, &[step(wgsl::STEP_REDUCE)]); - dispatch(pass, shown.rows); + dispatch_blocks(pass, rows, shown.block_count); } pass.set_bind_group(2, into, &[step(wgsl::STEP_CHANNELS)]); - let rows = shown.rows; dispatch_blocks(pass, rows, shown.block_count); let blocks = shown.blocks.of(rows); - let evaluated = blocks.iter().map(|&b| u64::from((rows - b * wgsl::BLOCK_ROWS).min(wgsl::BLOCK_ROWS))); + let evaluated = blocks.iter().map(|&b| u64::from((rows - b * wgsl::PAGE_ROWS).min(wgsl::PAGE_ROWS))); self.gpu.0.counters.add(Count::Evaluated, evaluated.sum()); shown.dirty = 0; shown.blocks = super::marks::Blocks::default(); shown.reduce = false; + shown.fresh.clear(); } pass.set_bind_group(2, state, &[step(wgsl::STEP_CHANNELS)]); } - /// Evaluates dirty channels, and sums face normals when what they rest on changed; with - /// `measure`, also measures every measured mark's extents and reads them back. With `travel`, - /// comets travel on by the frame's step. + /// Records the frame's compute pass: freezes starting transitions, evaluates dirty channels, + /// and sums face normals when their inputs changed. With `measure`, it also measures the + /// extents of every measured mark and returns their readback. With `travel`, comets move by the + /// frame's step. pub(super) fn compute( &mut self, encoder: &mut wgpu::CommandEncoder, @@ -325,41 +388,41 @@ impl View { ) -> Option { let gpu = self.gpu.clone(); let device = &gpu.0.device; - let Work { evaluated: work, eased, segments, faces, measured, carried, travelled } = self.work(measure, travel); - if [&work, &eased, &segments, &faces, &measured, &carried, &travelled].iter().all(|marks| marks.is_empty()) { + let Work { frozen, evaluated: work, resolved, eased, segments, faces, measured, carried, travelled, expanded } = + self.work(measure, travel); + let all = [&frozen, &work, &resolved, &eased, &segments, &faces, &measured, &carried, &travelled, &expanded]; + if all.iter().all(|marks| marks.is_empty()) { return None; } + self.start_domains(encoder, &work); let resources = self.resources.as_mut().expect("created before passes"); - for &i in &work { - let shown = &self.marks[i]; - if shown.reduce { - let bytes = u64::from(shown.reductions()) * 8; - encoder.clear_buffer(resources.domains(), u64::from(shown.domains) * 8, Some(bytes)); - } - } let extent_bytes = self.marks.len() as u64 * u64::from(wgsl::EXTENT_WORDS) * 4; if !measured.is_empty() { encoder.clear_buffer(resources.extents(), 0, Some(extent_bytes)); } + inner.arena.written(); if !faces.is_empty() { - encoder.clear_buffer(resources.normals(), 0, None); + // Faces sum their normals from zero into the pages of the vertex marks they attach to. + let words = u64::from(wgsl::PAGE_ROWS * wgsl::NORMAL_WORDS) * 4; + for shown in &self.marks { + for &block in shown.words.pages(Kept::Normals).blocks.iter() { + gpu.0.queue.clear_buffer(encoder, inner.arena.buffer(), u64::from(block) * 4, words); + } + } } let groups = resources.groups(&gpu.0, inner); + let freeze_group = (!frozen.is_empty()).then(|| resources.pass(device, inner, Shader::Freeze)); let evaluate_group = resources.pass(device, inner, Shader::Evaluate); - let settling_group = work - .iter() - .any(|&i| self.marks[i].transition.is_some()) - .then(|| resources.settling_group(device, inner)) - .flatten(); let normals_group = (!faces.is_empty()).then(|| resources.pass(device, inner, Shader::Normals)); let bounds_group = (!measured.is_empty()).then(|| resources.pass(device, inner, Shader::Bounds)); - let lists_group = (!segments.is_empty()).then(|| resources.pass(device, inner, Shader::Lists)); + let lists_group = + (!segments.is_empty() || !expanded.is_empty()).then(|| resources.pass(device, inner, Shader::Lists)); for &i in &segments { let list = self.marks[i].segments.list.expect("listed"); gpu.0.queue.write_buffer( resources.lists(), u64::from(list.header) * 4, - bytemuck::cast_slice(&list.header_words(self.marks[i].corners())), + bytemuck::cast_slice(&list.header_words(self.marks[i].kind().primitive().corners())), ); } { @@ -367,36 +430,41 @@ impl View { label: Some("latkit compute"), timestamp_writes: None, }); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Main.offset()]); + pass.set_bind_group(0, &groups.view, &[]); + let held = super::marks::Shown::held; + // Starting transitions freeze their rows' state as the frame began, before any pass + // writes it, and after packing placed their page tables. + if let Some(group) = &freeze_group { + pass.set_bind_group(2, group, &[]); + self.over_marks(&mut pass, inner, &groups.mark, GATHER, &frozen, held); + } + // Each drawn row's word first, exits included: every later pass reads it. + if let Some(group) = &lists_group { + pass.set_bind_group(2, group, &[]); + self.over_marks(&mut pass, inner, &groups.mark, EXPAND_RUNS, &expanded, |s| s.rows + s.exits); + } pass.set_bind_group(2, &evaluate_group, &[super::resources::step(wgsl::STEP_CHANNELS)]); - self.every_row(&mut pass, inner, &groups.mark, "carry", &carried); - self.evaluate(&mut pass, inner, &groups.mark, (&evaluate_group, settling_group.as_ref()), &work); - // After evaluating, so what a mark shows eases toward what it evaluated this frame. - self.every_row(&mut pass, inner, &groups.mark, "ease", &eased); - // After evaluating, so comets travel at the flow this frame shows. - self.every_row(&mut pass, inner, &groups.mark, "travel", &travelled); + self.over_marks(&mut pass, inner, &groups.mark, CARRY, &carried, held); + self.evaluate(&mut pass, inner, &groups.mark, &evaluate_group, &work); + // Before anything that reads where rows' endpoints are held. + self.resolve_ends(&mut pass, inner, &groups.mark, &evaluate_group, &resolved); + // After evaluating, so marks ease toward this frame's values. + self.over_marks(&mut pass, inner, &groups.mark, EASE, &eased, held); + // After evaluating, so comets travel at this frame's flow. + self.over_marks(&mut pass, inner, &groups.mark, TRAVEL, &travelled, held); if let Some(group) = &lists_group { pass.set_bind_group(2, group, &[]); self.count_segments(&mut pass, inner, &groups.mark, &segments); } if let Some(group) = &normals_group { - pass.set_pipeline(&inner.pipelines.compute(device, Shader::Normals, "accumulate")); pass.set_bind_group(2, group, &[]); - for &i in &faces { - pass.set_bind_group(1, &groups.mark, &[offset(i)]); - dispatch(&mut pass, self.marks[i].rows); - } + self.over_marks(&mut pass, inner, &groups.mark, NORMALS, &faces, |s| s.rows); } if let Some(group) = &bounds_group { pass.set_bind_group(2, group, &[]); for &i in &measured { - let entry = match self.marks[i].kind() { - MarkKind::Vertices => "points", - MarkKind::Areas => "areas", - _ => "strokes", - }; - let variant = Variant::compute(Shader::Bounds, entry).path(self.marks[i].path()); - pass.set_pipeline(&inner.pipelines.compute_pipeline(device, variant)); + let Some(bounds) = MarkVariants::shown(&self.marks[i]).bounds() else { continue }; + pass.set_pipeline(&inner.pipelines.compute(device, bounds)); pass.set_bind_group(1, &groups.mark, &[offset(i)]); dispatch(&mut pass, self.marks[i].live_instances()); } @@ -408,20 +476,45 @@ impl View { }) } - /// What this frame's compute pass does: the marks it evaluates, the segment lists it counts, the - /// faces whose normals it sums, the marks it measures, and with `travel` the marks whose comets - /// travel. + /// Prepares the fitted domains that marks `work` reduce. Domains computed on the CPU over a + /// paged table's rows are written. The rest are cleared for the GPU to fit. + fn start_domains(&mut self, encoder: &mut wgpu::CommandEncoder, work: &[usize]) { + let mut fitted: Vec<(usize, Vec)> = Vec::new(); + for &i in work { + if self.marks[i].reduce && self.marks[i].fits_data() { + fitted.push((i, self.data_domains(i))); + } + } + let resources = self.resources.as_mut().expect("created before passes"); + for &i in work { + let shown = &self.marks[i]; + if !shown.reduce { + continue; + } + let at = u64::from(shown.domains) * 8; + match fitted.iter().find(|(m, _)| *m == i) { + Some((_, words)) => self.gpu.0.queue.write_buffer(resources.domains(), at, bytemuck::cast_slice(words)), + None => encoder.clear_buffer(resources.domains(), at, Some(u64::from(shown.reductions()) * 8)), + } + } + } + + /// Chooses the marks each compute step covers this frame: freezing, evaluation, endpoint + /// resolution, easing, segment counting, face normals, measuring, comet carry, run expansion, + /// and with `travel`, comet travel. fn work(&mut self, measure: bool, travel: bool) -> Work { + let frozen = (0..self.marks.len()).filter(|&i| self.marks[i].remap != wgsl::NONE).collect(); for shown in &mut self.marks { - // A mark without rows has nothing to evaluate. - if shown.rows + shown.exits == 0 && shown.tables.synced() { + // A mark without rows has nothing to evaluate and no endpoints to resolve. + if shown.held() == 0 && shown.tables.synced() { shown.dirty = 0; shown.reduce = false; + shown.unresolved = super::marks::Blocks::default(); + shown.fresh.clear(); } } - let work: Vec = (0..self.marks.len()) - .filter(|&i| self.marks[i].drawable() && (self.marks[i].dirty != 0 || self.marks[i].reduce)) - .collect(); + let work: Vec = (0..self.marks.len()).filter(|&i| self.marks[i].block_count > 0).collect(); + let resolved: Vec = (0..self.marks.len()).filter(|&i| self.marks[i].end_block_count > 0).collect(); for &i in &work { let shown = &mut self.marks[i]; shown.version += 1; @@ -429,7 +522,7 @@ impl View { shown.place_version += 1; } } - // A mark easing toward a change writes what it shows as the clock moves, and as it evaluates. + // A mark in transition rewrites its shown state when the clock moves and when it evaluates. let eased: Vec = (0..self.marks.len()) .filter(|&i| { let shown = &self.marks[i]; @@ -454,45 +547,52 @@ impl View { let flowing = |i: &usize| self.marks[*i].drawable() && self.marks[*i].mark.flows(); let carried = (0..self.marks.len()).filter(|i| self.marks[*i].carry && flowing(i)).collect(); let travelled = (0..self.marks.len()).filter(|i| travel && flowing(i)).collect(); - Work { evaluated: work, eased, segments, faces, measured, carried, travelled } + let expanded = self.drawn_rows_due(); + Work { frozen, evaluated: work, resolved, eased, segments, faces, measured, carried, travelled, expanded } } - /// Lists the row blocks each mark evaluates this frame where its uniform finds them: every block of - /// a mark some channel of which changed for every row, else the blocks of the pages that changed. + /// Writes the blocks buffer, which each mark's uniform points into. For each mark, it lists the + /// row blocks it evaluates this frame (every block when a channel changed for every row, + /// otherwise the changed pages), then the row blocks whose endpoints it resolves. pub(super) fn lay_out_blocks(&mut self) { let mut words = Vec::new(); for shown in &mut self.marks { - let evaluates = shown.drawable() && (shown.dirty != 0 || shown.reduce); - let blocks = if evaluates { shown.blocks.of(shown.rows) } else { Vec::new() }; + let evaluates = shown.drawable() && (shown.dirty != 0 || shown.reduce || !shown.fresh.is_empty()); + let blocks = if evaluates { listed(shown) } else { Vec::new() }; shown.block = words.len() as u32; shown.block_count = blocks.len() as u32; words.extend(blocks); + let ends = if shown.drawable() { shown.unresolved.of(shown.live()) } else { Vec::new() }; + shown.end_block = words.len() as u32; + shown.end_block_count = ends.len() as u32; + words.extend(ends); } let gpu = self.gpu.clone(); let resources = self.resources.as_mut().expect("created before passes"); resources.write_blocks(&gpu.0.device, &gpu.0.queue, &words); } - /// Runs the expression machine's entry point `entry` over every row and exit of each of `marks`. - fn every_row( + /// Runs `variant` over each of `marks`, one invocation per item `items` counts for the mark. + fn over_marks( &self, pass: &mut wgpu::ComputePass<'_>, inner: &mut Inner, mark: &wgpu::BindGroup, - entry: &'static str, + variant: Variant, marks: &[usize], + items: impl Fn(&super::marks::Shown) -> u32, ) { if marks.is_empty() { return; } - pass.set_pipeline(&inner.pipelines.compute(&self.gpu.0.device, Shader::Evaluate, entry)); + pass.set_pipeline(&inner.pipelines.compute(&self.gpu.0.device, variant)); for &i in marks { pass.set_bind_group(1, mark, &[offset(i)]); - dispatch(pass, self.marks[i].rows + self.marks[i].exits); + dispatch(pass, items(&self.marks[i])); } } - /// Records the count → scan → write of each of `marks`' segment lists. + /// Records the count, scan, and write steps of each of `marks`' segment lists. fn count_segments( &self, pass: &mut wgpu::ComputePass<'_>, @@ -503,20 +603,53 @@ impl View { let device = &self.gpu.0.device; for &i in marks { let items = self.marks[i].rows + self.marks[i].exits; - let path = self.marks[i].path(); + let [count, write] = MarkVariants::shown(&self.marks[i]).segments(); pass.set_bind_group(1, mark, &[offset(i)]); - let count = Variant::compute(Shader::Lists, "count_segments").path(path); - pass.set_pipeline(&inner.pipelines.compute_pipeline(device, count)); + pass.set_pipeline(&inner.pipelines.compute(device, count)); dispatch(pass, items); super::lists::scan(pass, &mut inner.pipelines, device, items); - let write = Variant::compute(Shader::Lists, "write_segments").path(path); - pass.set_pipeline(&inner.pipelines.compute_pipeline(device, write)); + pass.set_pipeline(&inner.pipelines.compute(device, write)); dispatch(pass, items); } } - /// Whether face normals must be summed again: a face mark or a vertex mark wrote its state since - /// they were, or the globe turned under them. + /// The words of mark `i`'s fitted domains, as `lib/reduce.wesl` keeps them, each computed on + /// the CPU over every row of its paged table. So a domain stays fixed however little of the + /// table the camera shows. Each is cached by the stamp of its inputs, so a camera move refits + /// none. + fn data_domains(&mut self, i: usize) -> Vec { + let shown = &self.marks[i]; + let Some(compiled) = &shown.compiled else { return Vec::new() }; + let program = &compiled.program; + let Some(source) = Source::of(shown.mark.style(), &self.snapshots, &self.derived) else { + return vec![0; 2 * program.domains.len()]; + }; + let table = source.table; + let rows = Rows::new(table, &self.snapshots).at(self.at); + let mut fitted = Vec::with_capacity(program.domains.len()); + for (d, (reads, input)) in program.domains.iter().zip(&program.fitted).enumerate() { + let versions: Vec> = (0..program.columns.len()) + .filter(|&c| reads.columns & Reads::column(c) != 0) + .map(|c| { + let column = &program.columns[c]; + source.table_of(column, &self.snapshots)?.1.field_named(&column.name).map(Field::version) + }) + .collect(); + let sampled = reads.deps.intersects(Deps::SAMPLES).then_some(self.at.to_bits()); + let stamp = stamp_of((versions, table.structure, sampled)); + let words = match shown.fitted.get(d) { + Some(&(kept, words)) if kept == stamp => words, + _ => fit(&rows, input), + }; + fitted.push((stamp, words)); + } + let words = fitted.iter().flat_map(|(_, w)| *w).collect(); + self.marks[i].fitted = fitted; + words + } + + /// Whether face normals need summing again: a face or vertex mark wrote its state since the + /// last sum, or the globe turned. fn normals_due(&mut self) -> bool { let faces = self.marks.iter().any(|m| m.drawable() && m.faces()); if !faces { @@ -540,8 +673,10 @@ impl View { true } - /// Draws the backdrop, every drawable mark in scene order, then `sprites` overlay sprites, counted - /// in `drawn`, once the text set this frame has its fields. Depth is kept when `probing` reads it. + /// Records the main render pass, counting draws in `drawn`: the backdrop, every drawable mark + /// in scene order, and the lit rows. Once the text pipelines are ready, it also builds the + /// distance fields of text set this frame, and draws the labels and `sprites` overlay sprites. + /// Depth is stored when `probing` reads it. pub(super) fn draw( &mut self, encoder: &mut wgpu::CommandEncoder, @@ -562,7 +697,7 @@ impl View { if self.signal.transparent() { [0.0; 4] } else { self.scene.theme.background.to_array().map(f64::from) }; let plot = matches!(self.scene.space, Space::Plot(_)); let scissor = scissor(self.area(), frame); - let samples = super::resources::samples(frame.pixel_ratio); + let samples = gpu.samples_for(frame.pixel_ratio); let resources = self.resources.as_mut().expect("created before passes"); let (color, depth) = resources.targets(device, frame.size, frame.format, samples); let groups = resources.groups(&gpu.0, inner); @@ -592,9 +727,9 @@ impl View { None => (frame.target, None), }; let mut pass = main_pass(encoder, view, resolve, &depth, background, probing); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Main.offset()]); + pass.set_bind_group(0, &groups.view, &[]); pass.set_bind_group(1, &groups.mark, &[0]); - pass.set_pipeline(&inner.pipelines.render(device, Shader::Backdrop, frame.format, samples)); + pass.set_pipeline(&inner.pipelines.render(device, Variant::draw(Shader::Backdrop, frame.format, samples))); pass.set_bind_group(2, group(Shader::Backdrop), &[]); drawn.draw(&mut pass, 3, 1); let marks = Marks { @@ -614,8 +749,8 @@ impl View { marks.draw(&mut pass, drawn, inner, i, shown, false); marks.ports(&mut pass, drawn, inner, i, shown); } - // The rows focus lights draw again over every mark but labels, glowing: nothing covers them but - // the ports they carry. They draw once what draws them compiled. + // Lit rows draw again, glowing, over every mark but labels. Only their own ports cover + // them. They draw once their pipelines have compiled. for (i, shown) in self.marks.iter().enumerate() { if self.answers() && shown.drawable() && shown.lit.rows > 0 { marks.draw(&mut pass, drawn, inner, i, shown, true); @@ -625,80 +760,66 @@ impl View { if !texts { return; } - let glyphs = [Shader::Glyphs, Shader::GlyphsPorts].map(|shader| (shader, group(shader))); + let glyphs = [Shader::Glyphs, Shader::GlyphsPorts].map(group); self.draw_labels(&mut pass, drawn, inner, frame.format, samples, &groups, glyphs); if sprites > 0 { - pass.set_pipeline(&inner.pipelines.render(device, Shader::Overlay, frame.format, samples)); + pass.set_pipeline(&inner.pipelines.render(device, Variant::draw(Shader::Overlay, frame.format, samples))); pass.set_bind_group(2, group(Shader::Overlay), &[]); drawn.draw(&mut pass, 4, sprites); } } - /// Finds the row nearest the probe among what the frame drew, and the nearest depth drawn there, - /// and reads them back. + /// Records the probe pass: finds the drawn row nearest the probe point and the nearest depth + /// drawn there, and returns their readback. pub(super) fn probe(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner) -> Readback { - let gpu = self.gpu.clone(); - let device = &gpu.0.device; let probed: Vec = (0..self.marks.len()).filter(|&i| self.marks[i].probed()).collect(); - // Every mark drawn has its probe program, which reads the depth under the probe as well as any. - let under = self.marks.iter().position(super::marks::Shown::drawable); - let resources = self.resources.as_mut().expect("created before passes"); - let shader = crate::gpu::pipelines::probe(resources.samples()); + // Any drawn mark's probe program can read the depth under the probe. The first one does. + let under: Vec = self.marks.iter().position(super::marks::Shown::drawable).into_iter().collect(); let mut reset = [0_u32; wgsl::PROBE_WORDS as usize]; reset[wgsl::PROBE_KEY as usize] = wgsl::NONE; - gpu.0.queue.write_buffer(resources.probe(), 0, bytemuck::cast_slice(&reset)); - let groups = resources.groups(&gpu.0, inner); - let group = resources.pass(device, inner, shader); - { - let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { - label: Some("latkit probe"), - timestamp_writes: None, - }); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Main.offset()]); - // Each step over every mark before the next: the nearest key, then its topmost instance, - // then where that lies. - for step in [wgsl::STEP_NEAREST, wgsl::STEP_TOPMOST, wgsl::STEP_DESCRIBE] { - pass.set_bind_group(2, &group, &[super::resources::step(step)]); - for &i in &probed { - self.probe_step(&mut pass, inner, shader, &groups.mark, i, step); - } - } - if let Some(i) = under { - pass.set_bind_group(2, &group, &[super::resources::step(wgsl::STEP_UNDER)]); - self.probe_step(&mut pass, inner, shader, &groups.mark, i, wgsl::STEP_UNDER); - } - } + let probe = self.resources.as_ref().expect("created before passes").probe(); + self.gpu.0.queue.write_buffer(probe, 0, bytemuck::cast_slice(&reset)); + // Each step runs over every mark before the next: the nearest key, then its topmost + // instance, then where that lies. + let steps = [wgsl::STEP_NEAREST, wgsl::STEP_TOPMOST, wgsl::STEP_DESCRIBE].map(|step| (step, &probed[..])); + self.probe_pass(encoder, inner, "latkit probe", &[&steps[..], &[(wgsl::STEP_UNDER, &under[..])]].concat()); let resources = self.resources.as_ref().expect("created before passes"); - Readback::new( - &inner.readbacks, - device, - encoder, - resources.probe(), - 0, - u64::from(wgsl::PROBE_WORDS) * 4, - &self.waker, - ) + let (device, bytes) = (&self.gpu.0.device, u64::from(wgsl::PROBE_WORDS) * 4); + Readback::new(&inner.readbacks, device, encoder, resources.probe(), 0, bytes, &self.waker) } - /// Records step `step` of mark `i`'s probe program, `shader`, the step's pass group bound: over - /// every instance it reaches, or once for a step that reads one thing. - pub(super) fn probe_step( - &self, - pass: &mut wgpu::ComputePass<'_>, + /// Records a compute pass named `label` that runs each of `steps` of the probe program in turn: + /// a `STEP_` code, over each of the marks listed with it. + pub(super) fn probe_pass( + &mut self, + encoder: &mut wgpu::CommandEncoder, inner: &mut Inner, - shader: Shader, - mark: &wgpu::BindGroup, - i: usize, - step: u32, + label: &str, + steps: &[(u32, &[usize])], ) { - let shown = &self.marks[i]; - let variant = Variant::probe(shader, shown.primitive()).path(shown.path()); - pass.set_pipeline(&inner.pipelines.compute_pipeline(&self.gpu.0.device, variant)); - pass.set_bind_group(1, mark, &[offset(i)]); - if matches!(step, wgsl::STEP_DESCRIBE | wgsl::STEP_LOCATE | wgsl::STEP_UNDER) { - pass.dispatch_workgroups(1, 1, 1); - } else { - dispatch(pass, shown.live_instances()); + let gpu = self.gpu.clone(); + let device = &gpu.0.device; + let resources = self.resources.as_mut().expect("created before passes"); + let shader = crate::gpu::pipelines::probe(resources.samples()); + let groups = resources.groups(&gpu.0, inner); + let group = resources.pass(device, inner, shader); + let mut pass = + encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { label: Some(label), timestamp_writes: None }); + pass.set_bind_group(0, &groups.view, &[]); + for &(step, marks) in steps { + pass.set_bind_group(2, &group, &[super::resources::step(step)]); + for &i in marks { + let shown = &self.marks[i]; + pass.set_pipeline(&inner.pipelines.compute(device, MarkVariants::shown(shown).probe(shader))); + pass.set_bind_group(1, &groups.mark, &[offset(i)]); + // A step that reads one result runs once. The rest run over every instance probes + // cover. + if matches!(step, wgsl::STEP_DESCRIBE | wgsl::STEP_LOCATE | wgsl::STEP_UNDER) { + pass.dispatch_workgroups(1, 1, 1); + } else { + dispatch(&mut pass, shown.live_instances()); + } + } } } } diff --git a/crates/latkit/src/view/plan.rs b/crates/latkit/src/view/plan.rs index 79b7c88d..dc43216e 100644 --- a/crates/latkit/src/view/plan.rs +++ b/crates/latkit/src/view/plan.rs @@ -1,70 +1,97 @@ -//! The plan: how each table drawn fits the budget. A table draws whole, in priority order when a mark -//! drawing it gives one, unless it is too large to: denser than a screen shows, or more than the -//! view's share of the budget holds with the rest. The GPU's ledger grants each view its share, by -//! what it would draw whole. What the tables cost is worked out again only when the scene or a drawn -//! table's rows or geometry change, and the plan only when those costs or the share do. Then a table a mark places on a map or a plane -//! draws through tiles, the rows of the tiles the camera asks for, as many as its share holds, and -//! edges ending at it draw through tiles too, so they show with their ends; a table a mark traces -//! draws whole, its traces spanning as few pixel columns as fit; any other table draws the rows that -//! matter most, as many as fit. What a view draws always fits. +//! The plan: how each drawn table fits the GPU budget. +//! +//! A table draws all its rows (in priority order when a mark gives one) unless it is too large. A +//! table a mark places on a map or a plane is paged when it is denser than a screen can show. The +//! largest such tables are paged until the rest fit the view's budget share. A table of vector +//! tiles is always paged, with the vector tiles drawn as its tiles. Other tables drawn from tiles +//! never are. A paged table draws the rows of the tiles the camera needs, as many as its share +//! holds. Edges ending at it are paged too, so they show with their endpoints. If the unpaged +//! tables still do not fit, a table a line traces still draws all its rows, with its traces +//! spanning only as many pixel columns as fit. Any other table draws the rows that matter most, as +//! many as fit. What a view draws always fits. +//! +//! A table costs what the arena allocates for it: its columns' pages, the arena blocks of its +//! marks' per-row words, and its geometry. The GPU's ledger grants each view a share based on its +//! cost with every table drawn in full. Costs are recomputed only when the scene, or a drawn +//! table's rows or geometry, change. The plan is recomputed only when the costs or the share +//! change. +//! +//! Paged tables can need more than the costs predict, such as the kept rows that drawn edges end +//! at. When the view's memory exceeds its share, or its pages do not fit in a full arena, paged +//! tables are planned within a smaller part of the share until the share changes. That way the +//! current drawing and its replacement both fit, with space left to keep idle pages. -use std::hash::{Hash, Hasher}; +use std::hash::Hash; -use rustc_hash::{FxHashMap, FxHashSet, FxHasher}; +use rustc_hash::FxHashMap; use super::View; use super::marks::Shown; -use crate::data::{Field, Geometry, Keyed, Name, Table}; +use crate::data::{Field, Keyed, Name, Table}; use crate::derive::tiled::{Draw, Place, Planned, Plans}; -use crate::derive::{Derived, stamp_of}; +use crate::derive::{Derived, stamp_of, stamp_with}; use crate::expr::compile::Words; +use crate::gpu::arena::{path_page_bytes, rounded}; use crate::gpu::shaders as wgsl; use crate::gpu::{Ask, Ledger}; use crate::{Expr, Kind, Mark, Space}; -/// Rows past which a table a mark places draws through tiles: more than a screen shows one to a cell. +/// Rows past which a table a mark places is paged: more than a screen shows at one row per cell. pub(crate) const DENSE: usize = 1 << 18; -/// Pixel columns a view asks its traces' room for: a wide screen's. +/// Pixel columns a view requests trace space for: a wide screen's width. const ASKED_COLUMNS: f64 = 4096.0; -/// Rows a table drawn through tiles draws at least: a tile's. +/// The fewest rows a paged table draws: one tile's worth. const LEAST: u32 = 1 << 14; -/// Bytes a row's trace takes for each pixel column it spans: the least and greatest values of two -/// buckets. +/// Bytes a row's trace takes per pixel column: the minimum and maximum values of two buckets. const TRACE_COLUMN_BYTES: f64 = 16.0; -/// Pixel columns a trace spans at least, however little room is left. +/// The fewest pixel columns a trace spans, however little budget is left. const TRACE_COLUMNS: u32 = 32; -/// What a table costs: its rows, bytes each row takes on the GPU, bytes its geometry takes, and bytes -/// its traces take for each pixel column they span. +/// A table's cost as the arena allocates it: its row count, bytes per page of rows, bytes of +/// geometry, and bytes its traces take per pixel column. struct Cost { rows: usize, - per_row: f64, + per_page: f64, geometry: f64, traced: f64, } impl Cost { fn bytes(&self) -> f64 { - self.rows as f64 * self.per_row + self.geometry + self.rows.div_ceil(wgsl::PAGE_ROWS as usize) as f64 * self.per_page + self.geometry + } + + /// Bytes a row takes on average. + fn per_row(&self) -> f64 { + self.per_page / f64::from(wgsl::PAGE_ROWS) + self.geometry / self.rows.max(1) as f64 } } -/// Each table the scene draws, drawn whole, and what it costs. +/// Each table the scene draws, planned as drawn in full, and its cost. #[derive(Default)] struct Costs { whole: Plans, tables: Keyed<(Name, Name), Cost>, - /// The pages the costs count that any view drawing the same data shares: each field's by its - /// version and how it is read, the bytes the costs count of it, and its bytes once. + /// Pages in the costs that every view drawing the same data shares, keyed by field version and + /// read mode: the bytes the costs count for each, and its size counted once. shared: FxHashMap, } -/// The plan, and the costs it weighs, each worked out again only when what it reads changes. -#[derive(Default)] +/// The plan and the costs it uses, each recomputed only when its inputs change. pub(crate) struct Planning { costs: Derived, - /// How each table draws, and the pixel columns a trace spans at most. + /// How each table draws, and the most pixel columns a trace spans. plan: Derived<(Plans, u32)>, + /// The fraction of the remaining budget the plan gives paged tables. Lowered each time their + /// pages do not fit in a full arena or exceed the share. Reset when the share changes. + fit: f64, + share: f64, +} + +impl Default for Planning { + fn default() -> Planning { + Planning { costs: Derived::default(), plan: Derived::default(), fit: 1.0, share: 0.0 } + } } impl Planning { @@ -73,15 +100,46 @@ impl Planning { &self.plan.value.0 } - /// The pixel columns a trace spans at most. + /// Bytes `pages` pages of `key`'s table take, as its costs count them. + pub(crate) fn page_bytes(&self, key: &(Name, Name), pages: u32) -> f64 { + self.costs.value.tables.get(key).map_or(0.0, |cost| f64::from(pages) * cost.per_page) + } + + /// The most pixel columns a trace spans. pub(crate) fn trace_columns(&self) -> u32 { self.plan.value.1 } + + /// Records that paged tables did not fit in a full arena, so they draw within less while any + /// draws more than its minimum. Returns whether the plan shrank. + pub(crate) fn overflowed(&mut self) -> bool { + self.tighten(0.75) + } + + /// Records that the view uses `bytes` for what it draws. If that exceeds the share and the plan + /// left out some of the rows the camera needs from a paged table (`capped`), paged tables draw + /// within less, in proportion to the excess. If nothing was left out, a smaller plan would draw + /// no less and only release idle pages, so the plan stays. + pub(crate) fn held(&mut self, bytes: u64, capped: bool) { + if capped && self.share > 0.0 && bytes as f64 > self.share { + self.tighten(self.share / bytes as f64); + } + } + + /// Scales the budget for paged tables by `part`, if any draws more than its minimum. Returns + /// whether it did. + fn tighten(&mut self, part: f64) -> bool { + let shrinks = self.plans().values().any(|p| matches!(p.draw, Draw::Tiles { rows } if rows > LEAST)); + if shrinks { + self.fit *= part; + } + shrinks + } } impl View { - /// Brings the plan up to the scene, the data, and the share of `budget` the GPU's `ledger` - /// grants the view by what it would draw whole. + /// Updates the plan for the scene, the data, and the share of `budget` that the GPU's `ledger` + /// grants the view based on its cost with every table drawn in full. pub(super) fn update_plan(&mut self, ledger: &mut Ledger, budget: u64) { let stamp = self.costs_stamp(); if self.planning.costs.due(stamp) { @@ -97,33 +155,38 @@ impl View { shared: costs.shared.iter().map(|(piece, (_, bytes))| (*piece, *bytes)).collect(), }; let share = ledger.grant(self.id, whole, ask, budget); - let stamp = stamp_of((self.planning.costs.stamp(), share.to_bits())); + if share != self.planning.share { + (self.planning.share, self.planning.fit) = (share, 1.0); + } + let stamp = stamp_of((self.planning.costs.stamp(), share.to_bits(), self.planning.fit.to_bits())); if self.planning.plan.due(stamp) { - let planned = self.plan(costs, share); + let planned = self.plan(costs, share, self.planning.fit); self.planning.plan.make(stamp, planned); } } - /// What the costs read, stamped: the scene, and of each table a mark draws, its dataset's - /// schema, its rows, its geometry, and whether the view fetches it as tiles. + /// A stamp of the costs' inputs: the scene, and for each table a mark draws, its dataset's + /// schema, its row structure, its geometry's version, and whether it comes from tiles, and + /// from vector tiles. fn costs_stamp(&self) -> u64 { - let mut hasher = FxHasher::default(); - self.scene_version.hash(&mut hasher); - for shown in &self.marks { - let style = shown.mark.style(); - let data = self.source(&style.data); - let table = data.and_then(|d| d.table(&style.table)); - let geometry = geometry_of(&shown.mark).and_then(|f| table?.field_named(&f)).map(Field::version); - let owned = self.derived.tiles.owns(&style.data); - (data.map(crate::Data::schema), table.map(|t| t.structure), geometry, owned).hash(&mut hasher); - } - hasher.finish().max(1) + stamp_with(|hasher| { + self.scene_version.hash(hasher); + for shown in &self.marks { + let style = shown.mark.style(); + let data = self.snapshot(&style.data); + let table = data.and_then(|d| d.table(&style.table)); + let geometry = geometry_of(&shown.mark).and_then(|f| table?.field_named(&f)).map(Field::version); + let owned = (self.derived.tiles.owns(&style.data), self.derived.tiles.vector(&style.data)); + (data.map(crate::Snapshot::schema), table.map(|t| t.structure), geometry, owned).hash(hasher); + } + }) } - /// How each table the scene draws draws, within `share` bytes, from what `costs` says each - /// costs, and the pixel columns a trace spans at most. - fn plan(&self, costs: &Costs, share: f64) -> (Plans, u32) { + /// Plans how each table the scene draws fits in `share` bytes given `costs`, giving paged + /// tables `fit` of the remaining budget. Also returns the most pixel columns a trace spans. + fn plan(&self, costs: &Costs, share: f64, fit: f64) -> (Plans, u32) { let (mut plans, costs) = (costs.whole.clone(), &costs.tables); + let vector = |key: &(Name, Name)| self.derived.tiles.vector(&key.0); let tileable = |key: &(Name, Name), planned: &Planned| planned.place != Place::Nowhere && !self.derived.tiles.owns(&key.0); let mut whole: f64 = costs.values().map(Cost::bytes).sum(); @@ -135,10 +198,11 @@ impl View { tiled.push(key.clone()); } }; - // Dense tables draw through tiles, then the largest until what draws whole fits. + // Page tables of vector tiles and dense tables, then the largest ones until the unpaged + // tables fit. let keys: Vec<(Name, Name)> = plans.keys().cloned().collect(); for key in &keys { - if tileable(key, &plans[key]) && costs[key].rows > DENSE { + if vector(key) || (tileable(key, &plans[key]) && costs[key].rows > DENSE) { tile(key, &mut plans, &mut whole); } } @@ -150,14 +214,15 @@ impl View { let Some(key) = largest.cloned() else { break }; tile(&key, &mut plans, &mut whole); } - // Edges ending at a table drawn through tiles draw through tiles, so they show with their ends. + // Edges ending at a paged table are paged too, so they show with their endpoints. for key in &keys { if self.ends_tiled(key, &plans) { tile(key, &mut plans, &mut whole); } } - // What still does not fit draws the rows that matter most, as many as fit; a table a mark - // traces draws whole, its traces thinning in time instead. + // If the unpaged tables still do not fit, each draws the rows that matter most, as many as + // fit. A table a line traces still draws all its rows, and its traces get coarser in time + // instead. if whole > share { let kept: f64 = keys .iter() @@ -174,12 +239,13 @@ impl View { } whole = share.max(kept); } - // Tables drawn through tiles, and traces, share what is left. + // Paged tables and traces share the remaining budget. let traced: f64 = costs.values().map(|c| c.traced).sum(); - let left = (share - whole).max(share / 4.0); + let left = (share - whole).max(share / 4.0) * fit; let each = left / (tiled.len() + usize::from(traced > 0.0)).max(1) as f64; for key in &tiled { - let per_row = costs[key].per_row.max(1.0) + costs[key].geometry / costs[key].rows.max(1) as f64; + // A paged table's drawn row takes one more word: its held row. + let per_row = costs[key].per_row().max(1.0) + 4.0; let rows = (each / per_row).min(f64::from(u32::MAX)) as u32; plans.get_mut(key).expect("listed").draw = Draw::Tiles { rows: rows.max(LEAST) }; } @@ -187,25 +253,24 @@ impl View { (plans, columns) } - /// Each table the scene draws, drawn whole, placed and ordered by the first mark drawing it to - /// give a place or a priority, with the fields its marks read; and what it costs. + /// Plans each table the scene draws as drawn in full, with the place and priority of the first + /// mark that gives each, and computes each table's cost from the fields its marks read. fn costs(&self) -> Costs { let space = &self.scene.space; let placing = matches!(space, Space::Earth(_) | Space::Plane(_)); + let easing = !self.scene.ease.is_zero(); let mut plans = Plans::default(); let mut costs: Keyed<(Name, Name), Cost> = Keyed::default(); let mut shared: FxHashMap = FxHashMap::default(); - let mut joined: Vec<((Name, Name), Name)> = Vec::new(); - for shown in &self.marks { + let summing = super::graph::summing(&self.marks); + for (shown, summing) in self.marks.iter().zip(summing) { let style = shown.mark.style(); - let Some(table) = self.source(&style.data).and_then(|d| d.table(&style.table)) else { continue }; + let Some(table) = self.snapshot(&style.data).and_then(|d| d.table(&style.table)) else { continue }; let key = (style.data.clone(), style.table.clone()); let planned = plans.entry(key.clone()).or_insert_with(|| Planned { draw: Draw::Whole, priority: None, place: Place::Nowhere, - reads: (!matches!(space, Space::Diagram(_))).then(FxHashSet::default), - traced: FxHashSet::default(), }); if planned.priority.is_none() { planned.priority.clone_from(&style.priority); @@ -213,26 +278,9 @@ impl View { if placing && planned.place == Place::Nowhere { planned.place = place_of(&shown.mark, table); } - if let Some(compiled) = &shown.compiled { - let traces = compiled.program.columns.iter().filter(|c| c.trace && c.of.is_none() && !c.derived); - planned.traced.extend(traces.map(|c| c.name.clone())); - } - if let (Some(reads), Some(compiled)) = (&mut planned.reads, &shown.compiled) { - for column in &compiled.program.columns { - match &column.of { - None if !column.derived => { - reads.insert(column.name.clone()); - } - Some(of) => joined.push((of.clone(), column.name.clone())), - None => {} - } - } - if let Some(field) = geometry_of(&shown.mark) { - reads.insert(field); - } - } - let cost = costs.entry(key).or_insert(Cost { rows: table.len(), per_row: 0.0, geometry: 0.0, traced: 0.0 }); - cost.per_row += row_bytes(shown); + let cost = + costs.entry(key).or_insert(Cost { rows: table.len(), per_page: 0.0, geometry: 0.0, traced: 0.0 }); + cost.per_page += page_bytes(shown, easing, summing); cost.geometry += geometry_bytes(shown, table); cost.traced += traced_bytes(shown, table); for (piece, bytes) in self.shared_pages(shown, table) { @@ -241,17 +289,12 @@ impl View { *once = bytes; } } - for (key, field) in joined { - if let Some(reads) = plans.get_mut(&key).and_then(|p| p.reads.as_mut()) { - reads.insert(field); - } - } Costs { whole: plans, tables: costs, shared } } - /// The pages `shown` reads of `table`, and of the tables it joins, that every view drawing the - /// same data shares, as `row_bytes` and `geometry_bytes` count them: each field's, by its version - /// and how it is read. + /// The pages `shown` reads of `table` and its joined tables that every view drawing the same + /// data shares, sized as `page_bytes` and `geometry_bytes` count them. Each field's pages are + /// keyed by its version and read mode. fn shared_pages(&self, shown: &Shown, table: &Table) -> Vec<(u64, f64)> { let Some(compiled) = &shown.compiled else { return Vec::new() }; let mut pieces: Vec<(u64, f64)> = compiled @@ -260,12 +303,12 @@ impl View { .iter() .filter(|c| c.words != Words::Path && !c.derived) .filter_map(|c| { - let read = match &c.of { + let read = match &c.from { None => table.field_named(&c.name), - Some((data, joined)) => self.source(data)?.table(joined)?.field_named(&c.name), + Some((data, joined)) => self.snapshot(data)?.table(joined)?.field_named(&c.name), }; - let width = if c.low { 8.0 } else { 4.0 }; - Some((stamp_of((read?.version(), c.low)), table.len() as f64 * width)) + let pages = table.len().div_ceil(wgsl::PAGE_ROWS as usize) as f64; + Some((stamp_of((read?.version(), c.low)), pages * column_page_bytes(c.low))) }) .collect(); if let Some(field) = geometry_of(&shown.mark).and_then(|f| table.field_named(&f)) { @@ -275,10 +318,10 @@ impl View { pieces } - /// Whether the table `key` names is edges or faces whose ends lie in a table drawn through tiles. + /// Whether table `key` holds edges or faces whose endpoints lie in a paged table. fn ends_tiled(&self, key: &(Name, Name), plans: &Plans) -> bool { let Place::Ends(fields) = &plans[key].place else { return false }; - let Some(table) = self.source(&key.0).and_then(|d| d.table(&key.1)) else { return false }; + let Some(table) = self.snapshot(&key.0).and_then(|d| d.table(&key.1)) else { return false }; let target = fields.first().and_then(|f| match table.field_named(f) { Some(Field::Reference(reference)) => Some(reference.to.clone()), _ => None, @@ -315,33 +358,45 @@ fn geometry_of(mark: &Mark) -> Option { } } -/// Bytes a row of `shown` takes on the GPU: its state and the frozen mirror a transition keeps, its -/// focus word, a word for each column it reads and two for a position's, and its label's candidate. -fn row_bytes(shown: &Shown) -> f64 { +/// Bytes of the arena block for a page of rows at `width` words per row. +fn block_bytes(width: u32) -> f64 { + f64::from(rounded(wgsl::PAGE_ROWS * width)) * 4.0 +} + +/// Bytes a page of a column's rows takes: its values, and its low parts where it reads them. +fn column_page_bytes(low: bool) -> f64 { + block_bytes(1) * if low { 2.0 } else { 1.0 } +} + +/// Bytes a page of `shown`'s rows takes, as the arena allocates them: a page of each column it +/// reads, plus each kind of word page it keeps (with the normals faces are `summing` at its rows), +/// and when `easing`, a transition's frozen and settled words. +fn page_bytes(shown: &Shown, easing: bool, summing: bool) -> f64 { let Some(compiled) = &shown.compiled else { return 0.0 }; - let state = f64::from(compiled.drawing.stride + compiled.drawing.extras) * 8.0; let columns: f64 = - compiled.program.columns.iter().filter(|c| c.words != Words::Path).map(|c| if c.low { 8.0 } else { 4.0 }).sum(); - let label = if compiled.drawing.label.is_some() { f64::from(wgsl::CANDIDATE_WORDS) * 4.0 } else { 0.0 }; - state + 4.0 + columns + label + compiled.program.columns.iter().filter(|c| c.words != Words::Path).map(|c| column_page_bytes(c.low)).sum(); + let kept: f64 = + shown.kept_widths(summing).iter().filter(|&&(_, width)| width > 0).map(|&(_, width)| block_bytes(width)).sum(); + let eased = if easing { 2.0 * block_bytes(shown.width()) } else { 0.0 }; + columns + kept + eased } -/// Bytes the traces `shown` draws of `table` take for each pixel column they span: a row's for each -/// field it traces. +/// Bytes the traces `shown` draws of `table` take per pixel column: `TRACE_COLUMN_BYTES` for each +/// row of each traced field. fn traced_bytes(shown: &Shown, table: &Table) -> f64 { let Some(compiled) = &shown.compiled else { return 0.0 }; let traced = compiled.program.columns.iter().filter(|c| { - c.trace && c.of.is_none() && !c.derived && matches!(table.field_named(&c.name), Some(Field::Sampled(..))) + c.trace && c.from.is_none() && !c.derived && matches!(table.field_named(&c.name), Some(Field::Sampled(..))) }); traced.count() as f64 * table.len() as f64 * TRACE_COLUMN_BYTES } -/// Bytes the geometry `shown` draws of `table` takes: its points, and an area's triangles about -/// three times as many. +/// Bytes the geometry `shown` draws of `table` takes, as the arena allocates it: per page of rows, +/// a row table and the points in pieces. An area's triangles take about three times the points of +/// its outlines. fn geometry_bytes(shown: &Shown, table: &Table) -> f64 { let Some(field) = geometry_of(&shown.mark) else { return 0.0 }; let Some(Field::Values(column, _)) = table.field_named(&field) else { return 0.0 }; - let points: usize = column.pages().iter().filter_map(|p| p.values.geometry()).map(Geometry::coords_spanned).sum(); - let scale = if matches!(shown.mark, Mark::Areas(_)) { 3.0 } else { 1.0 }; - points as f64 * 8.0 * scale + let scale = if matches!(shown.mark, Mark::Areas(_)) { 3 } else { 1 }; + column.pages().iter().map(|p| path_page_bytes(p.values.len(), p.points() * scale) as f64).sum() } diff --git a/crates/latkit/src/view/probe.rs b/crates/latkit/src/view/probe.rs index c0daf779..8469d6c2 100644 --- a/crates/latkit/src/view/probe.rs +++ b/crates/latkit/src/view/probe.rs @@ -1,17 +1,25 @@ -//! Probes: what lies under a point, answered by a compute pass over what the frame drew and read back -//! a frame or two later: the nearest row, and the nearest depth drawn. One probe is in flight at a -//! time; a newer point waits for it. +//! Probes: the nearest row and the nearest drawn depth under a point. +//! +//! A compute pass runs over what the frame drew, and its result is read back a frame or two later. +//! One probe is in flight at a time, and a newer point waits for it. The answer is decoded against +//! the frame that drew it (its marks' held rows and its camera), regardless of changes since. +use std::sync::Arc; + +use super::Projected; use super::event::{Hit, Sample}; use super::gestures::Gesture; -use super::marks::Shown; +use super::marks::{Holding, Shown}; +use super::sync::Plan; use super::{Device, View}; +use crate::derive::Working; use crate::gpu::readback::Readback; use crate::gpu::shaders as wgsl; -use crate::{Camera, Row}; +use crate::scene::mark::MarkKind; +use crate::{Camera, Name, Row, wire}; -/// A hit, with the depth it lies at for dragging, and for an edge which end it is nearer: 0 its -/// first, 1 its second. +/// A hit, with its depth (for dragging) and, for an edge, the nearer endpoint: 0 for the first, +/// 1 for the second. #[derive(Clone, Debug)] pub(crate) struct Probed { pub(crate) hit: Hit, @@ -23,33 +31,121 @@ pub(crate) struct Probed { #[derive(Clone, Debug, Default)] pub(crate) struct Answer { pub(crate) hit: Option, - /// The nearest depth drawn around the point; 0 where nothing was drawn. + /// The nearest depth drawn around the point. 0 where nothing was drawn. pub(crate) under: f32, } +/// A record of what a frame drew, for decoding answers about it: each mark's held rows, and the +/// frame's camera. +#[derive(Clone)] +pub(crate) struct Drew { + marks: Arc<[Held]>, + projected: Projected, +} + +/// One mark as its frame drew it: its dataset, table, and name, whether its rows are vertices, +/// whether selection picks them, the rows its state held, its port names, and, for a line tracing a +/// field, the field and its trace plan. +struct Held { + data: Name, + table: Name, + name: Option, + vertices: bool, + selects: bool, + holding: Option, + ports: Vec, + traced: Option<(Name, Plan)>, +} + +impl Drew { + /// The row at held row `held` of mark `i`, and its index in its table. + pub(super) fn row(&self, i: usize, held: u32) -> Option<(Row, usize)> { + let mark = self.marks.get(i)?; + let holding = mark.holding.as_ref()?; + let r = holding.source_row(held)?; + let id = holding.table.ids.id(r as usize)?; + Some((Row { data: mark.data.clone(), table: mark.table.clone(), id }, r as usize)) + } + + /// The hit at held row `held` of mark `i` as the frame drew it: at screen `point` and `depth`, + /// `distance` 1/64 CSS pixels from where it was asked, on `part` (a port's number from 1, or a + /// segment's index along its path). `None` when the frame held no such row. + pub(super) fn hit( + &self, + i: usize, + held: u32, + point: [f32; 2], + depth: f32, + distance: u32, + part: u32, + ) -> Option { + let (row, at) = self.row(i, held)?; + Some(Hit { + row, + mark: self.marks.get(i)?.name.clone(), + index: i, + point, + position: self.projected.position(self.projected.unproject(point.map(f64::from), f64::from(depth))), + distance: distance as f32 / 64.0, + port: self.port(i, part), + sample: self.sample(i, at, part), + }) + } + + /// Whether a box's selection picks mark `i`'s rows. + pub(super) fn selects(&self, i: usize) -> bool { + self.marks.get(i).is_some_and(|m| m.selects) + } + + /// The name of port `number` of vertex mark `i`, numbered from 1. `None` for 0, the marker. + fn port(&self, i: usize, number: u32) -> Option { + let mark = self.marks.get(i)?; + if !mark.vertices || number == 0 { + return None; + } + mark.ports.get(number as usize - 1).cloned() + } + + /// The sample that point `point` of row `row`'s traced path shows, on mark `i` tracing a field. + fn sample(&self, i: usize, row: usize, point: u32) -> Option { + let mark = self.marks.get(i)?; + let (field, plan) = mark.traced.as_ref()?; + let (level, index, side) = plan.entry(point)?; + let table = &mark.holding.as_ref()?.table; + let crate::data::Field::Sampled(samples, _) = table.field_named(field)? else { return None }; + // A bucket's two points show its middle frame, with its minimum or maximum value. + let span = crate::data::bucket_frames(level); + let frame = index * span + span / 2; + let value = samples.entry(row, level, side, index)?; + Some(Sample { frame, at: samples.drawn_at(level, index)?, value }) + } +} + #[derive(Default)] pub(crate) struct Probing { /// The next point to probe. pub(crate) wanted: Option<[f32; 2]>, - reading: Option<([f32; 2], Readback)>, + /// The probe being read back: the point it asked about, and what its frame drew. + reading: Working<([f32; 2], Drew), Readback>, /// The latest answer and the point it was asked for. pub(crate) latest: Option<([f32; 2], Answer)>, } impl Probing { - /// Whether a point waits for a frame to probe it: none while a probe is being read back. + /// Whether a point is waiting for a frame to probe it. Never while a probe is being read back. pub(crate) fn due(&self) -> bool { - self.wanted.is_some() && self.reading.is_none() + self.wanted.is_some() && !self.reading.busy() } pub(crate) fn hovered(&self) -> Option<&Probed> { self.latest.as_ref().and_then(|(_, answer)| answer.hit.as_ref()) } - /// The latest answer when it was asked within a pixel or two of `point` and nothing newer waits. + /// The latest answer, if it was for a point within 2 CSS pixels of `point` and no newer probe + /// is pending. pub(crate) fn at(&self, point: [f32; 2]) -> Option { let (asked, answer) = self.latest.as_ref()?; - (self.wanted.is_none() && self.reading.is_none() && near(*asked, point, 2.0)).then(|| answer.clone()) + (self.wanted.is_none() && !self.reading.busy() && near(*asked, point, 2.0)).then(|| answer.clone()) } } @@ -58,9 +154,9 @@ pub(crate) fn near(a: [f32; 2], b: [f32; 2], within: f32) -> bool { } impl View { - /// The point this frame probes, when it `answers` probes: one waits and none is in flight. What - /// the pointer rests on is probed again whenever the frame changed under it, so a moving frame - /// costs a probe per readback and a still one nothing. + /// The point this frame probes, if it `answers` probes, a point is waiting, and no probe is in + /// flight. The point under a resting pointer is probed again whenever the frame changes under + /// it, so a moving frame runs one probe per readback and a still frame runs none. pub(super) fn probe_point(&mut self, answers: bool) -> Option<[f32; 2]> { if (self.changed || self.easing()) && matches!(self.gestures.now, Gesture::Idle) @@ -69,7 +165,7 @@ impl View { { self.probing.wanted.get_or_insert(pointer.point); } - if self.probing.reading.is_some() || !answers { + if self.probing.reading.busy() || !answers { return None; } // Where nothing is drawn, there is nothing to probe. @@ -77,86 +173,54 @@ impl View { point.filter(|_| self.marks.iter().any(Shown::drawable)) } - /// CSS pixels a probe reaches around its point: a finger reaches farther. + /// The probe radius in CSS pixels: the scene's pick radius, plus 8 for touch. pub(super) fn probe_radius(&self) -> f32 { let touch = self.pointer.is_some_and(|pointer| pointer.device == Device::Touch); self.scene.controls.pick_radius + if touch { 8.0 } else { 0.0 } } pub(super) fn probing(&mut self, point: [f32; 2], readback: Readback) { - self.probing.reading = Some((point, readback)); + let drew = self.drew(); + self.probing.reading.start(0, readback, (point, drew)); } - /// Hands a landed probe's answer to hover and gestures. - pub(super) fn settle_probe(&mut self) { - let Some((point, readback)) = &self.probing.reading else { return }; - let Some(bytes) = readback.take() else { return }; - let point = *point; - self.probing.reading = None; - let answer = bytes.map_or_else(|_| Answer::default(), |bytes| self.answer(bytemuck::cast_slice(&bytes))); - self.landed(point, answer); - } - - /// What the probe's words say. - fn answer(&self, words: &[u32]) -> Answer { - let word = |i: u32| words.get(i as usize).copied().unwrap_or(0); - Answer { hit: self.hit(&word), under: f32::from_bits(word(wgsl::PROBE_UNDER)) } + /// A record of what this frame drew, for decoding answers about it. + pub(super) fn drew(&self) -> Drew { + let marks: Vec = self + .marks + .iter() + .map(|shown| { + let style = shown.mark.style(); + let ports = shown.compiled.as_ref().map(|c| c.drawing.ports.iter().map(|(name, ..)| name.clone())); + let traced = match &shown.mark { + crate::Mark::Lines(lines) => shown.tables.plan().map(|plan| (lines.traced(), plan.clone())), + _ => None, + }; + Held { + data: style.data.clone(), + table: style.table.clone(), + name: style.name.clone(), + vertices: shown.kind() == MarkKind::Vertices, + selects: style.interaction.select, + holding: shown.holding.clone(), + ports: ports.map(Iterator::collect).unwrap_or_default(), + traced, + } + }) + .collect(); + Drew { marks: marks.into(), projected: self.projected() } } - fn hit(&self, word: &impl Fn(u32) -> u32) -> Option { - let key = word(wgsl::PROBE_KEY); - if key == wgsl::NONE || word(wgsl::PROBE_INSTANCE) == 0 { - return None; - } - let row = word(wgsl::PROBE_ROW); - let mark = 255 - (key & 255) as usize; - let shown = self.marks.get(mark)?; - let style = shown.mark.style(); - let id = self.drawn(&style.data)?.table(&style.table)?.ids.id(row as usize)?; - let depth = f32::from_bits(word(wgsl::PROBE_DEPTH)); - let point = [f32::from_bits(word(wgsl::PROBE_X)), f32::from_bits(word(wgsl::PROBE_Y))]; - let view = self.projected(); - let position = view.position(view.unproject(point.map(f64::from), f64::from(depth))); - let hit = Hit { - row: Row { data: style.data.clone(), table: style.table.clone(), id }, - mark: style.name.clone(), - index: mark, - point, - position, - distance: (key >> 8) as f32 / 64.0, - port: self.port(mark, word(wgsl::PROBE_SEGMENT)), - sample: self.sample(mark, row, word(wgsl::PROBE_SEGMENT)), - }; - Some(Probed { hit, depth, end: word(wgsl::PROBE_SEGMENT) }) - } - - /// The port a vertex mark's probe reached: its number from 1, or 0 for the marker. - pub(super) fn port(&self, mark: usize, number: u32) -> Option { - let shown = self.marks.get(mark)?; - if shown.kind() != crate::scene::mark::MarkKind::Vertices || number == 0 { - return None; - } - let compiled = shown.compiled.as_ref()?; - compiled.drawing.ports.get(number as usize - 1).map(|(name, ..)| name.clone()) - } - - /// The sample point `point` of a traced row's path stands for, on a mark tracing a field. - pub(super) fn sample(&self, mark: usize, row: u32, point: u32) -> Option { - let shown = self.marks.get(mark)?; - let crate::Mark::Lines(lines) = &shown.mark else { return None }; - let (level, index, side) = shown.tables.plan()?.entry(point)?; - let style = shown.mark.style(); - let table = self.drawn(&style.data)?.table(&style.table)?; - let crate::data::Field::Sampled(samples, _) = table.field_named(&lines.traced())? else { return None }; - // A bucket's points stand for its middle frame, its least or greatest value. - let span = crate::data::bucket_frames(level); - let frame = index * span + span / 2; - let value = samples.entry(row as usize, level, side, index)?; - Some(Sample { frame, at: samples.drawn_at(level, index)?, value }) + /// Hands a finished probe's answer to hover and gestures. + pub(super) fn settle_probe(&mut self) { + let Some((_, (point, drew), bytes)) = self.probing.reading.landed() else { return }; + let answer = bytes.map_or_else(|_| Answer::default(), |bytes| answer(&wire::words(&bytes), &drew)); + self.landed(point, answer); } - /// The point gestures move about under `point`: what the frame drew there, else the level the - /// camera looks at, else a point as far away as the camera's center. + /// The position gestures pivot on under screen point `point`: what the frame drew there, or + /// else the ground level the camera looks at, or else a point as far away as the camera's + /// center. pub(super) fn anchor(&self, point: [f32; 2], answer: Option<&Answer>) -> [f64; 3] { let view = self.projected(); let p = point.map(f64::from); @@ -168,7 +232,7 @@ impl View { return view.on_plane(p, 0.0).map_or([0.0; 3], |w| view.position(w)); }; let reach = 50.0 * (view.eye - view.world(center)).length(); - let level = if view.globe >= 0.5 { view.surface(p) } else { view.on_plane(p, center[2]) }; + let level = if view.globe >= 0.5 { view.ground(p) } else { view.on_plane(p, center[2]) }; if let Some(w) = level.filter(|w| view.grazing(p) > 0.1 && (*w - view.eye).length() < reach) { view.position(w) } else { @@ -178,8 +242,26 @@ impl View { } } +/// Decodes a probe's words against what its frame drew. +fn answer(words: &[u32], drew: &Drew) -> Answer { + let word = |i: u32| words.get(i as usize).copied().unwrap_or(0); + Answer { hit: hit(&word, drew), under: f32::from_bits(word(wgsl::PROBE_UNDER)) } +} + +fn hit(word: &impl Fn(u32) -> u32, drew: &Drew) -> Option { + let key = word(wgsl::PROBE_KEY); + if key == wgsl::NONE || word(wgsl::PROBE_INSTANCE) == 0 { + return None; + } + let depth = f32::from_bits(word(wgsl::PROBE_DEPTH)); + let point = [f32::from_bits(word(wgsl::PROBE_X)), f32::from_bits(word(wgsl::PROBE_Y))]; + let part = word(wgsl::PROBE_SEGMENT); + let hit = drew.hit(255 - (key & 255) as usize, word(wgsl::PROBE_ROW), point, depth, key >> 8, part)?; + Some(Probed { hit, depth, end: part }) +} + impl super::marks::Shown { - /// Whether probes reach this mark's rows. + /// Whether probes cover this mark's rows. pub(crate) fn probed(&self) -> bool { self.drawable() && self.mark.picks() && self.rows > 0 } diff --git a/crates/latkit/src/view/program.rs b/crates/latkit/src/view/program.rs index 982c1e12..d48d0afe 100644 --- a/crates/latkit/src/view/program.rs +++ b/crates/latkit/src/view/program.rs @@ -1,6 +1,11 @@ -//! The program buffer: each mark's descriptors, tables, ranges, code, constants, ends, and ports in a -//! region of its own, rewritten alone when only that mark changed. A region that outgrows its room -//! moves to the end of the buffer; when the end passes the buffer, every region is laid out again. +//! The program buffer: each mark's column descriptors and tables, ranges, code, constants, end +//! tables, and ports, in a region of its own. +//! +//! When a mark changes, only its region is rewritten, and only the words that differ. A region +//! that outgrows its capacity moves to the end of the buffer. If the buffer is too small, it grows +//! and the GPU copies its contents over, so no other region is rewritten. Once moved regions leave +//! more of the buffer unused than the regions take, every region is laid out again from the start, +//! each writing only its own words. use super::{Event, View}; use crate::gpu::Inner; @@ -12,69 +17,123 @@ pub(crate) struct Regions { pub(crate) relayout: bool, } -/// The room a region of `len` words gets, so it can grow a little in place. +/// Changed words separated by fewer unchanged words than this are written in one span. +const GAP: usize = 16; + +/// The capacity given to a region of `len` words, with slack to grow in place. fn room(len: u32) -> u32 { len + len / 4 + 16 } +/// The spans of `words` that differ from `was` (the words last written there), as `[start, end)`. +/// Spans closer than `GAP` words merge. +fn changed(was: &[u32], words: &[u32]) -> Vec<[usize; 2]> { + let mut spans: Vec<[usize; 2]> = Vec::new(); + for i in (0..words.len()).filter(|&i| was.get(i) != Some(&words[i])) { + match spans.last_mut() { + Some(span) if i - span[1] < GAP => span[1] = i + 1, + _ => spans.push([i, i + 1]), + } + } + spans +} + impl View { - /// Writes every mark whose words changed into its region of the program buffer. - pub(super) fn pack(&mut self, inner: &mut Inner) { + /// Writes every changed mark's words into its region of the program buffer. When the buffer + /// grows, `encoder` copies the old contents over, and later writes are ordered after the copy. + pub(super) fn pack(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner) { let gpu = self.gpu.clone(); - let font = self.scene.theme.font.clone(); + let font = self.scene.theme.text.font.clone(); for shown in self.marks.iter_mut().filter(|m| !m.packed) { shown.write_ports(&font, inner); let Some(compiled) = &mut shown.compiled else { continue }; let markers = &mut compiled.drawing.markers; if !markers.icons.is_empty() { - let layers = inner.icons.layers(&gpu.0.device, &gpu.0.queue, &markers.icons); - markers.palette = std::iter::once(layers.len() as u32).chain(layers).collect(); + match inner.icons.of(&gpu.0.device, &gpu.0.queue, &markers.icons) { + Ok(layers) => markers.palette = std::iter::once(layers.len() as u32).chain(layers).collect(), + Err(error) => self.events.tell(Event::Error(error)), + } } for (at, colormap) in &compiled.program.colormaps { - match inner.colormaps.layer(&gpu.0.device, &gpu.0.queue, colormap) { + match inner.colormaps.colormap(&gpu.0.device, &gpu.0.queue, colormap) { Ok(layer) => shown.code[*at as usize][1] = layer | colormap.kind().code() << 16, Err(error) => self.events.tell(Event::Error(error)), } } } let resources = self.resources.as_mut().expect("created before packing"); - let capacity = resources.program_words() as u32; + let (device, queue) = (&gpu.0.device, &gpu.0.queue); + // Whether `encoder` copied words this frame. Writes after a copy must be ordered after it. + let mut ordered = false; + let write = + |encoder: &mut wgpu::CommandEncoder, ordered: bool, buffer: &wgpu::Buffer, at: u32, words: &[u32]| { + let (at, bytes) = (u64::from(at) * 4, bytemuck::cast_slice(words)); + if ordered { + queue.write_buffer_in(device, encoder, buffer, at, bytes); + } else { + queue.write_buffer(buffer, at, bytes); + } + }; if !self.regions.relayout { + // The space the regions take. Moved regions left the rest of the buffer unused, up to + // `end`. + let mut live: u32 = self.marks.iter().filter_map(|m| m.region).map(|[_, room]| room).sum(); for shown in self.marks.iter_mut().filter(|m| !m.packed && m.compiled.is_some()) { let len = shown.packed_len(); - let start = match shown.words { + let start = match shown.region { Some([start, room]) if len <= room => start, - _ if self.regions.end + room(len) <= capacity => { - let start = self.regions.end; - shown.words = Some([start, room(len)]); - self.regions.end += room(len); + region => { + let (start, end) = (self.regions.end, self.regions.end + room(len)); + live = live - region.map_or(0, |[_, room]| room) + room(len); + if end > 2 * live { + self.regions.relayout = true; + break; + } + if u64::from(end) > resources.program_words() { + let kept = u64::from(self.regions.end); + ordered |= resources.grow_program(device, encoder, u64::from(end), kept); + } + // The GPU copies the region's words to the new place, so only changed words + // are written. + match region { + Some([from, _]) if !shown.written.is_empty() => { + let words = shown.written.len() as u32; + resources.move_program(device, encoder, from, start, words); + ordered = true; + } + _ => shown.written.clear(), + } + shown.region = Some([start, room(len)]); + self.regions.end = end; start } - _ => { - self.regions.relayout = true; - break; - } }; let mut words = Vec::with_capacity(len as usize); shown.pack(start, &mut words); - gpu.0.queue.write_buffer(resources.program(), u64::from(start) * 4, bytemuck::cast_slice(&words)); + for [from, to] in changed(&shown.written, &words) { + write(encoder, ordered, resources.program(), start + from as u32, &words[from..to]); + } + shown.written = words; shown.packed = true; } } if std::mem::take(&mut self.regions.relayout) { let total: u64 = self.marks.iter().map(|m| u64::from(room(m.packed_len()))).sum(); - resources.fit_program(&gpu.0.device, total); - let mut words = Vec::with_capacity(total as usize); + resources.fit_program(device, total); + // Write each region's words at its start. The slack after them is not written: nothing + // reads it until the region grows into it and writes it. + let mut start = 0; for shown in self.marks.iter_mut().filter(|m| m.compiled.is_some()) { - let start = words.len() as u32; let len = shown.packed_len(); + let mut words = Vec::with_capacity(len as usize); shown.pack(start, &mut words); - words.resize((start + room(len)) as usize, 0); - shown.words = Some([start, room(len)]); + write(encoder, ordered, resources.program(), start, &words); + shown.written = words; + shown.region = Some([start, room(len)]); shown.packed = true; + start += room(len); } - self.regions.end = words.len() as u32; - gpu.0.queue.write_buffer(resources.program(), 0, bytemuck::cast_slice(&words)); + self.regions.end = start; } } } diff --git a/crates/latkit/src/view/query.rs b/crates/latkit/src/view/query.rs index b18b88c0..0c78b75a 100644 --- a/crates/latkit/src/view/query.rs +++ b/crates/latkit/src/view/query.rs @@ -1,9 +1,12 @@ -//! Queries: every row within reach of a point, every row in a box, and where a row lies on screen, -//! answered on the GPU by each mark's probe program through the geometry the frame drew, and read back -//! a frame or two later. What a query finds of each mark follows the probe's words: the hits it keeps, -//! after how many it found and how many it keeps, or a bit per row. Hits land in any order; they are -//! told in one, and a mark that found more than it kept asks again with room for them all. One query -//! runs at a time; the rest wait their turn. +//! Queries: every row within a radius of a point, every row in a box, and where a row lies on +//! screen. +//! +//! Each mark's probe program answers on the GPU, through the geometry the frame drew. The result is +//! read back a frame or two later and decoded against that frame. A query's results for each mark +//! follow the probe's words in the probe buffer: a hit count and capacity followed by the hits, or +//! a bit per row. Hits arrive in any order and are sorted before they are returned. A mark that +//! found more hits than it kept is queried again with space for all of them. One query runs at a +//! time. The rest wait in order, and a query nobody awaits any more is skipped. use std::collections::VecDeque; @@ -11,27 +14,39 @@ use rustc_hash::FxHashSet; use super::View; use super::event::{Hit, Trigger}; -use super::resources::{EyeSlot, step}; +use super::probe::Drew; +use crate::derive::Working; use crate::gpu::Inner; use crate::gpu::readback::Readback; use crate::gpu::shaders as wgsl; use crate::{Pending, Resolver, Row}; -/// Hits a mark keeps at first: past them, a query counts hits but keeps none, and asks again. +/// Hits kept per mark on the first try. Hits past these are counted but not kept, and the query +/// runs again with space for them. const HITS: u32 = 1024; -/// A question for the GPU. +/// A query for the GPU. pub(crate) enum Ask { /// Every row within `radius` CSS pixels of `point`, keeping up to `room` hits of each mark. Hits { point: [f32; 2], radius: f32, room: u32, reply: Resolver> }, - /// Every row in the box between two corners; they become the selection, as `trigger` chose. + /// Every row in the box between two corners. The rows become the selection, with `trigger` as + /// its cause. Boxed { corners: [[f32; 2]; 2], trigger: Trigger }, - /// Where a row lies. + /// Where a row lies on screen. Locate { row: Row, reply: Resolver> }, } impl Ask { - /// The frame's query words for it. + /// Whether nobody awaits its answer any more. A box query selects rows, so it always runs. + fn cancelled(&self) -> bool { + match self { + Ask::Hits { reply, .. } => reply.is_cancelled(), + Ask::Locate { reply, .. } => reply.is_cancelled(), + Ask::Boxed { .. } => false, + } + } + + /// The frame uniform's query words for it. fn words(&self, marks: &[usize], view: &View) -> [f32; 4] { match self { Ask::Hits { point, radius, .. } => [point[0], point[1], *radius, 0.0], @@ -52,8 +67,8 @@ impl Ask { } } - /// What it keeps of a mark of `instances` instances and `rows` rows, hits or rows, and the words - /// that takes. + /// How many hits or rows it keeps of a mark with `instances` instances and `rows` rows, and the + /// words that takes. fn room(&self, instances: u32, rows: u32) -> (u32, u32) { match self { Ask::Hits { room, .. } => { @@ -66,7 +81,8 @@ impl Ask { } } -/// What a query finds of mark `mark`: from word `at` of the probe's, `room` hits or rows. +/// Where a query's results for mark `mark` lie: from probe-buffer word `at`, with space for `room` +/// hits or rows. #[derive(Clone, Copy)] struct Found { mark: usize, @@ -74,37 +90,39 @@ struct Found { room: u32, } -/// A query this frame runs: what it asks, what it finds of each mark it asks, and the words that -/// takes past the probe's. +/// The query this frame runs: the query, where each mark's results go, and the words they take +/// after the probe's. struct Running { ask: Ask, found: Vec, words: u32, } -/// A query whose answer the GPU is copying back. +/// What reading a query's answer needs: the query, where each mark's results are, and what its +/// frame drew. struct Reading { ask: Ask, found: Vec, - readback: Readback, + drew: Drew, } #[derive(Default)] pub(crate) struct Queries { waiting: VecDeque, running: Option, - reading: Option, + /// The query whose answer the GPU is copying back. + reading: Working, } impl Queries { - /// Whether a query waits to run. + /// Whether a query is running, or waiting to run with someone still awaiting it. pub(crate) fn due(&self) -> bool { - !self.waiting.is_empty() || self.running.is_some() + self.running.is_some() || self.waiting.iter().any(|ask| !ask.cancelled()) } } impl View { - /// Every row of a mark that picks within `radius` CSS pixels of `point`, nearest first, as the + /// Every row of a pickable mark within `radius` CSS pixels of `point`, nearest first, as the /// next frames draw them. pub fn pick(&mut self, point: [f32; 2], radius: f32) -> Pending> { let (answer, reply) = Pending::new(); @@ -113,9 +131,9 @@ impl View { answer } - /// Where `row` lies on screen, CSS pixels from the top left, as the next frames draw it; `None` - /// when no mark draws it in view. A diagram's vertices answer at once from its sheet. - pub fn locate(&mut self, row: &Row) -> Pending> { + /// Where `row` lies on screen, in CSS pixels from the top left, as the next frames draw it. + /// `None` when no mark draws it in view. A diagram's vertices resolve at once from its sheet. + pub fn screen_of(&mut self, row: &Row) -> Pending> { let diagram = matches!(self.scene.space, crate::Space::Diagram(_)); if let Some(b) = self.derived.sheet.bounds(row).filter(|_| diagram) { let view = self.projected(); @@ -128,36 +146,31 @@ impl View { answer } - /// Selects every row in the box between two screen points, as `trigger` chose. + /// Selects every row in the box between two screen points, with `trigger` as the cause. pub(super) fn select_box(&mut self, corners: [[f32; 2]; 2], trigger: Trigger) { self.queries.waiting.push_back(Ask::Boxed { corners, trigger }); self.wake(); } - /// The index of `row` in mark `i`'s table. + /// The position of `row` among the rows mark `i` draws, if it draws it. pub(super) fn row_index(&self, i: usize, row: &Row) -> Option { - let style = self.marks.get(i)?.mark.style(); + let shown = self.marks.get(i)?; + let style = shown.mark.style(); if style.data != row.data || style.table != row.table { return None; } - let index = self.drawn(&style.data)?.table(&style.table)?.row(row.id.as_ref())?; - (index < self.marks[i].rows as usize).then_some(index as u32) - } - - /// The row at `index` of mark `i`'s table. - pub(super) fn row_at(&self, i: usize, index: u32) -> Option { - let style = self.marks.get(i)?.mark.style(); - let id = self.drawn(&style.data)?.table(&style.table)?.ids.id(index as usize)?; - Some(Row { data: style.data.clone(), table: style.table.clone(), id }) + let index = self.snapshot(&style.data)?.table(&style.table)?.row(row.id.as_ref())?; + shown.item_of(shown.held_row(index as u32)?).filter(|&item| item < shown.rows) } - /// Takes the next query for this frame when none runs, makes room for what it finds past the - /// probe's words, and returns its query words and whether it reads depth. + /// Starts the next query this frame if none is in progress, skipping those nobody awaits. + /// Reserves space for its results after the probe's words, and returns its query words and + /// whether it reads depth. pub(super) fn next_query(&mut self) -> Option<([f32; 4], bool)> { - if self.queries.running.is_some() || self.queries.reading.is_some() { + if self.queries.running.is_some() || self.queries.reading.busy() { return None; } - let ask = self.queries.waiting.pop_front()?; + let ask = std::iter::from_fn(|| self.queries.waiting.pop_front()).find(|ask| !ask.cancelled())?; let marks: Vec = match &ask { Ask::Locate { row, .. } => (0..self.marks.len()) .filter(|&i| self.marks[i].drawable() && self.row_index(i, row).is_some()) @@ -176,7 +189,7 @@ impl View { let mut at = wgsl::PROBE_WORDS; let mut found = Vec::with_capacity(marks.len()); for i in marks { - let (room, taken) = ask.room(self.marks[i].live_instances(), self.marks[i].rows); + let (room, taken) = ask.room(self.marks[i].live_instances(), self.marks[i].live()); self.marks[i].query = at; found.push(Found { mark: i, at, room }); at += taken; @@ -188,15 +201,14 @@ impl View { Some((words, depth)) } - /// Records the running query's step over every mark it asks, and the copy of what it found. + /// Records the running query's step over every mark it covers, and the copy of its results. pub(super) fn run_query(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner) { let Some(Running { ask, found, words }) = self.queries.running.take() else { return }; let gpu = self.gpu.clone(); let (device, queue) = (&gpu.0.device, &gpu.0.queue); - let resources = self.resources.as_mut().expect("created before queries"); - let probe = resources.probe().clone(); + let probe = self.resources.as_ref().expect("created before queries").probe().clone(); let start = u64::from(wgsl::PROBE_WORDS) * 4; - // What it finds starts out empty: no hits, with room for those each mark keeps; no rows. + // Results start empty: for hits, a zero count and each mark's capacity; for a box, no rows. match &ask { Ask::Hits { .. } => { for f in &found { @@ -206,35 +218,22 @@ impl View { Ask::Boxed { .. } if words > 0 => encoder.clear_buffer(&probe, start, Some(u64::from(words) * 4)), Ask::Boxed { .. } | Ask::Locate { .. } => {} } - let shader = crate::gpu::pipelines::probe(resources.samples()); - let groups = resources.groups(&gpu.0, inner); - let group = resources.pass(device, inner, shader); - { - let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { - label: Some("latkit query"), - timestamp_writes: None, - }); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Main.offset()]); - pass.set_bind_group(2, &group, &[step(ask.step())]); - for f in &found { - self.probe_step(&mut pass, inner, shader, &groups.mark, f.mark, ask.step()); - } - } + let marks: Vec = found.iter().map(|f| f.mark).collect(); + self.probe_pass(encoder, inner, "latkit query", &[(ask.step(), &marks)]); let (at, bytes) = match &ask { Ask::Locate { .. } => (u64::from(wgsl::PROBE_LOCATE_X) * 4, 16), _ => (start, u64::from(words.max(1)) * 4), }; let readback = Readback::new(&inner.readbacks, device, encoder, &probe, at, bytes, &self.waker); - self.queries.reading = Some(Reading { ask, found, readback }); + let drew = self.drew(); + self.queries.reading.start(0, readback, Reading { ask, found, drew }); } - /// Answers a query whose readback landed. + /// Answers a query whose readback has arrived. pub(super) fn settle_query(&mut self) { - let Some(reading) = &self.queries.reading else { return }; - let Some(bytes) = reading.readback.take() else { return }; - let Reading { ask, found, .. } = self.queries.reading.take().expect("reading"); - let words: Vec = match bytes { - Ok(bytes) => bytemuck::cast_slice(&bytes).to_vec(), + let Some((_, Reading { ask, found, drew }, bytes)) = self.queries.reading.landed() else { return }; + let words = match bytes { + Ok(bytes) => crate::wire::words(&bytes), Err(error) => { match ask { Ask::Hits { reply, .. } => reply.resolve(Err(error)), @@ -246,23 +245,23 @@ impl View { }; match ask { Ask::Hits { point, radius, room, reply } => { - // A mark that found more than it kept asks again, with room for every hit it found. + // If a mark found more hits than it kept, run again with space for all of them. let most = found.iter().map(|f| words_at(&words, f.at, 1).first().copied().unwrap_or(0)).max(); match most.filter(|&most| most > room) { Some(room) => self.queries.waiting.push_front(Ask::Hits { point, radius, room, reply }), - None => reply.resolve(Ok(self.hits(&found, &words))), + None => reply.resolve(Ok(hits(&found, &words, &drew))), } } Ask::Boxed { trigger, .. } => { let mut rows = Vec::new(); let mut seen = FxHashSet::default(); - for f in found.iter().filter(|f| self.marks[f.mark].mark.style().interaction.select) { + for f in found.iter().filter(|f| drew.selects(f.mark)) { for (k, &bits) in words_at(&words, f.at, f.room.div_ceil(32)).iter().enumerate() { let mut bits = bits; while bits != 0 { let index = k as u32 * 32 + bits.trailing_zeros(); bits &= bits - 1; - if let Some(row) = self.row_at(f.mark, index) + if let Some((row, _)) = drew.row(f.mark, index) && seen.insert(row.clone()) { rows.push(row); @@ -280,49 +279,39 @@ impl View { } self.wake(); } +} - fn hits(&self, found: &[Found], words: &[u32]) -> Vec { - let view = self.projected(); - // Nearest first, the mark drawn last first among equals, and a port before its marker; then - // by row and where it lies, so the order never depends on which hit landed first. Each row - // answers once, where it is nearest. - let mut kept: Vec<(usize, &[u32; wgsl::HIT_WORDS as usize])> = found - .iter() - .flat_map(|f| { - let count = words_at(words, f.at, 1).first().map_or(0, |&n| n.min(f.room)); - let entries = words_at(words, f.at + 2, count * wgsl::HIT_WORDS); - entries.as_chunks().0.iter().map(move |entry| (f.mark, entry)) - }) - .collect(); - // The last keys first, as each sort keeps the order of ties. - let key = |high: u32, low: u32| u128::from(crate::order::pair(high, low)); - crate::order::by_wide_key(&mut kept, |&(_, entry)| key(entry[3], entry[4]) << 32 | u128::from(entry[5])); - crate::order::by_wide_key(&mut kept, |&(i, entry)| key(entry[1], !(i as u32)) << 64 | key(!entry[2], entry[0])); - let mut seen = FxHashSet::default(); - let mut hits = Vec::new(); - for (i, entry) in kept { - if !seen.insert((i, entry[0])) { - continue; - } - let Some(row) = self.row_at(i, entry[0]) else { continue }; - let point = [f32::from_bits(entry[3]), f32::from_bits(entry[4])]; - let depth = f64::from(f32::from_bits(entry[5])); - hits.push(Hit { - row, - mark: self.marks[i].mark.style().name.clone(), - index: i, - point, - position: view.position(view.unproject(point.map(f64::from), depth)), - distance: entry[1] as f32 / 64.0, - port: self.port(i, entry[2]), - sample: self.sample(i, entry[0], entry[2]), - }); +/// The hits in `words` for `found`, decoded against `drew`, the frame they came from. +fn hits(found: &[Found], words: &[u32], drew: &Drew) -> Vec { + // Nearest first. Among equals, the mark drawn last comes first, and a port before its marker. + // Then by row and position, so the order never depends on which hit arrived first. Each row + // appears once, at its nearest hit. + let mut kept: Vec<(usize, &[u32; wgsl::HIT_WORDS as usize])> = found + .iter() + .flat_map(|f| { + let count = words_at(words, f.at, 1).first().map_or(0, |&n| n.min(f.room)); + let entries = words_at(words, f.at + 2, count * wgsl::HIT_WORDS); + entries.as_chunks().0.iter().map(move |entry| (f.mark, entry)) + }) + .collect(); + // Sort by the minor keys first: each sort is stable, so the last one keeps their order in ties. + let key = |high: u32, low: u32| u128::from(crate::order::pair(high, low)); + crate::order::by_wide_key(&mut kept, |&(_, entry)| key(entry[3], entry[4]) << 32 | u128::from(entry[5])); + crate::order::by_wide_key(&mut kept, |&(i, entry)| key(entry[1], !(i as u32)) << 64 | key(!entry[2], entry[0])); + let mut seen = FxHashSet::default(); + let mut hits = Vec::new(); + for (i, entry) in kept { + if !seen.insert((i, entry[0])) { + continue; } - hits + let point = [f32::from_bits(entry[3]), f32::from_bits(entry[4])]; + hits.extend(drew.hit(i, entry[0], point, f32::from_bits(entry[5]), entry[1], entry[2])); } + hits } -/// `len` words from word `at` of the probe's, in a query's answer read from past the probe's words. +/// `len` words from probe-buffer word `at`, in a query answer read back from past the probe's +/// words. fn words_at(words: &[u32], at: u32, len: u32) -> &[u32] { let start = (at - wgsl::PROBE_WORDS) as usize; words.get(start..start + len as usize).unwrap_or(&[]) diff --git a/crates/latkit/src/view/resources.rs b/crates/latkit/src/view/resources.rs index 0628e226..8c732de2 100644 --- a/crates/latkit/src/view/resources.rs +++ b/crates/latkit/src/view/resources.rs @@ -1,48 +1,32 @@ -//! A view's buffers, textures, and bind groups. Grown buffers grow by half again; replacing one bumps -//! the generation, and bind groups are rebuilt when a generation they were built from changes. +//! A view's buffers, textures, and bind groups. A buffer that runs out of room grows +//! ([`Grown`]). Replacing a buffer increments the generation, and bind groups are rebuilt when a +//! generation they were built from changes. //! -//! Every shader's pass group (group 2) binds by one convention: 0 state, 1 frozen state, 2 extents, -//! 3 normals, 4 domains, 5 probe words and what queries find, 6 depth, 7 sprites, 8 the text atlas, -//! 9 lists, 10 the shadow map, 11 a level of it, 12 remap words, 13 labels, 14 the settled state, -//! 15 icons, 16 row blocks, 17 the probe's step. +//! Every shader's pass group (group 2) uses the same bindings: 0 the arena pages (the data, and the +//! marks' per-row words), 2 extents, 4 domains, 5 probe words and query results, 6 depth, +//! 7 sprites, 8 the text atlas, 9 lists, 12 remap words, 13 labels, 15 icons, 16 the row blocks +//! each mark evaluates, 17 the probe step. use std::collections::HashMap; use std::num::NonZeroU64; -use super::rows::Sizes; +use crate::gpu::grown::{self, Grown}; use crate::gpu::shaders::{self as wgsl, Shader}; use crate::gpu::work::Queue; use crate::gpu::{Inner, Shared}; -/// Bytes between marks' uniforms and between eyes: the dynamic offset alignment every device supports. +/// Bytes between marks' uniforms: the dynamic offset alignment every device supports. pub(crate) const MARK_STRIDE: u64 = 256; const _: () = assert!(size_of::() as u64 <= MARK_STRIDE, "a mark's uniform fits its stride"); -/// The depth format; depth is reversed, nearer is greater. +/// The depth format. Depth is reversed: nearer is greater. pub(crate) const DEPTH: wgpu::TextureFormat = wgpu::TextureFormat::Depth32Float; -/// The shadow map's format. -pub(crate) const SHADE: wgpu::TextureFormat = wgpu::TextureFormat::Rgba16Float; -/// Texels across the shadow map. -pub(crate) const SHADE_SIZE: u32 = 1024; - -/// The eyes a frame draws through, by index. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub(crate) enum EyeSlot { - Main = 0, - Light = 1, -} - -impl EyeSlot { - pub(crate) fn offset(self) -> u32 { - (self as u64 * MARK_STRIDE) as u32 - } -} - /// The dynamic offset of the probe's step `step`, one of the `STEP_` codes. pub(crate) fn step(step: u32) -> u32 { (u64::from(step) * MARK_STRIDE) as u32 } -/// The probe's steps, each at its own dynamic offset: a dispatch binds the one it takes. +/// A uniform buffer holding each probe step's code at its own dynamic offset. A dispatch binds the +/// offset of its step. fn steps(device: &wgpu::Device) -> wgpu::Buffer { use wgpu::util::DeviceExt as _; let mut contents = vec![0_u8; step(wgsl::STEPS) as usize]; @@ -57,43 +41,9 @@ fn steps(device: &wgpu::Device) -> wgpu::Buffer { }) } -/// Samples per pixel: shader-drawn marks antialias themselves, so dense screens skip multisampling. -pub(crate) fn samples(pixel_ratio: f32) -> u32 { - if pixel_ratio >= 1.75 { 1 } else { 4 } -} - -/// A storage buffer that grows by half again when it runs out of room. -struct Grown { - buffer: wgpu::Buffer, - label: &'static str, - usage: wgpu::BufferUsages, -} - -impl Grown { - fn new(device: &wgpu::Device, label: &'static str, usage: wgpu::BufferUsages) -> Grown { - Grown { buffer: create(device, label, usage, 256), label, usage } - } - - /// Makes room for `bytes`; true when the buffer was replaced and its contents lost. - fn ensure(&mut self, device: &wgpu::Device, bytes: u64) -> bool { - if bytes <= self.buffer.size() { - return false; - } - self.buffer = create(device, self.label, self.usage, (bytes + bytes / 2).next_multiple_of(256)); - true - } -} - -fn create(device: &wgpu::Device, label: &str, usage: wgpu::BufferUsages, bytes: u64) -> wgpu::Buffer { - device.create_buffer(&wgpu::BufferDescriptor { - label: Some(label), - size: bytes.max(256).next_multiple_of(4), - usage: usage | wgpu::BufferUsages::COPY_DST, - mapped_at_creation: false, - }) -} - -const ROWS: wgpu::BufferUsages = wgpu::BufferUsages::STORAGE.union(wgpu::BufferUsages::COPY_SRC); +/// The labels buffer's usages: storage for the texts pass's atomics, and copy source for reading +/// back the texts it asks for. +const LABELS: wgpu::BufferUsages = wgpu::BufferUsages::STORAGE.union(wgpu::BufferUsages::COPY_SRC); /// The bind groups every pipeline shares. #[derive(Clone)] @@ -102,8 +52,8 @@ pub(crate) struct Groups { pub(crate) mark: wgpu::BindGroup, } -/// What a frame draws into: multisampled color resolved into the target when it multisamples, and -/// depth. +/// The textures a frame draws into: multisampled color (resolved into the target) when it +/// multisamples, and depth. struct Targets { color: Option, depth: wgpu::TextureView, @@ -112,85 +62,36 @@ struct Targets { samples: u32, } -/// The ground's shadow map, its levels, and a view of each level. -pub(crate) struct ShadowMap { - pub(crate) view: wgpu::TextureView, - pub(crate) levels: Vec, -} - -impl ShadowMap { - fn new(device: &wgpu::Device, size: u32) -> ShadowMap { - let count = size.max(1).ilog2() + 1; - let texture = device.create_texture(&wgpu::TextureDescriptor { - label: Some("latkit shadows"), - size: wgpu::Extent3d { width: size, height: size, depth_or_array_layers: 1 }, - mip_level_count: count, - sample_count: 1, - dimension: wgpu::TextureDimension::D2, - format: SHADE, - usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::TEXTURE_BINDING, - view_formats: &[], - }); - let levels = (0..count) - .map(|level| { - texture.create_view(&wgpu::TextureViewDescriptor { - base_mip_level: level, - mip_level_count: Some(1), - ..Default::default() - }) - }) - .collect(); - ShadowMap { view: texture.create_view(&wgpu::TextureViewDescriptor::default()), levels } - } -} - -/// Buffers a layout of rows replaced, kept until their rows are copied out. -pub(crate) struct OldRows { - pub(crate) state: wgpu::Buffer, - pub(crate) frozen: Option, - pub(crate) settled: Option, -} - pub(crate) struct Resources { pub(crate) frame: wgpu::Buffer, - eyes: wgpu::Buffer, + eye: wgpu::Buffer, marks: Grown, program: Grown, - focus: wgpu::Buffer, /// The rows each mark's focus draw lights. lit: Grown, - state: wgpu::Buffer, - frozen: Option, - /// The state's mirror that channels of marks easing toward a change store into, while one does: - /// where their rows settle. - settled: Option, - retired: Option, - /// The row blocks each mark's evaluation covers this frame. + /// The row blocks each mark evaluates, and whose endpoints it resolves, this frame. blocks: Grown, remap: Grown, domains: Grown, extents: Grown, - normals: Grown, - /// The probe's words, then what the running query finds. + /// The probe's words, then the running query's results. probe: Grown, steps: wgpu::Buffer, sprites: Grown, lists: Grown, labels: wgpu::Buffer, - empty: wgpu::Buffer, - /// What the settled state binds without one: writable, so it never shares a buffer bound - /// read-only. - scratch: wgpu::Buffer, - pub(crate) shadows: ShadowMap, - /// Changes whenever one of the buffers or textures above is replaced; and whenever the lists - /// buffer is, emptied. + /// The labels' claim grid: a word per screen cell. + cells: Grown, + /// Incremented whenever a buffer or texture above is replaced. `lists_generation` is also + /// incremented when the lists buffer is replaced, which empties it. generation: u64, lists_generation: u64, - /// Built from (arena, colormaps, icons, own) generations. + /// The shared bind groups, with the (arena, colormaps, icons, own) generations they were built + /// from. groups: Option<((u64, u64, u64, u64), Groups)>, - /// Pass groups, built at `passes_generation`. + /// Pass groups by shader, built at `passes_generation`: the arena, icons, and own generations. passes: HashMap, - passes_generation: (u64, u64), + passes_generation: (u64, u64, u64), targets: Option, } @@ -199,21 +100,16 @@ impl Resources { let storage = wgpu::BufferUsages::STORAGE; let uniform = wgpu::BufferUsages::UNIFORM; Resources { - frame: create(device, "latkit frame", uniform, size_of::() as u64), - eyes: create(device, "latkit eyes", uniform, 2 * MARK_STRIDE), + frame: grown::buffer(device, "latkit frame", uniform, size_of::() as u64), + eye: grown::buffer(device, "latkit eye", uniform, size_of::() as u64), marks: Grown::new(device, "latkit marks", uniform), - program: Grown::new(device, "latkit program", storage), - focus: create(device, "latkit focus", storage, 256), + // A copy source, so growing keeps its contents. + program: Grown::new(device, "latkit program", storage | wgpu::BufferUsages::COPY_SRC), lit: Grown::new(device, "latkit lit", storage), - state: create(device, "latkit state", ROWS, 256), - frozen: None, - settled: None, - retired: None, blocks: Grown::new(device, "latkit blocks", storage), remap: Grown::new(device, "latkit remap", storage), domains: Grown::new(device, "latkit domains", storage), extents: Grown::new(device, "latkit extents", storage | wgpu::BufferUsages::COPY_SRC), - normals: Grown::new(device, "latkit normals", storage), probe: Grown::new(device, "latkit probe", storage | wgpu::BufferUsages::COPY_SRC), steps: steps(device), sprites: Grown::new(device, "latkit sprites", storage), @@ -222,87 +118,18 @@ impl Resources { "latkit lists", storage | wgpu::BufferUsages::INDIRECT | wgpu::BufferUsages::COPY_SRC, ), - labels: create(device, "latkit labels", ROWS, 256), - empty: create(device, "latkit empty", ROWS, 256), - scratch: create(device, "latkit scratch", ROWS, 256), - shadows: ShadowMap::new(device, 1), + labels: grown::buffer(device, "latkit labels", LABELS, 256), + cells: Grown::new(device, "latkit label cells", storage), generation: 0, lists_generation: 0, groups: None, passes: HashMap::new(), - passes_generation: (u64::MAX, 0), + passes_generation: (u64::MAX, 0, 0), targets: None, } } - /// Lays rows out into new buffers of `sizes`, returning the buffers they replaced so their rows - /// can be copied over; the frozen mirror, while one exists, is replaced alike. - pub(crate) fn replace_rows(&mut self, device: &wgpu::Device, sizes: Sizes) -> OldRows { - let words = |n: u64| (n + 1) * 4; - let state = create(device, "latkit state", ROWS, words(sizes.state)); - self.focus = create(device, "latkit focus", wgpu::BufferUsages::STORAGE, words(sizes.focus)); - let frozen = self.frozen.as_ref().map(|_| create(device, "latkit frozen", ROWS, words(sizes.state))); - let settled = self.settled.as_ref().map(|_| create(device, "latkit settled", ROWS, words(sizes.state))); - let old = OldRows { - state: std::mem::replace(&mut self.state, state), - frozen: std::mem::replace(&mut self.frozen, frozen), - settled: std::mem::replace(&mut self.settled, settled), - }; - self.generation += 1; - old - } - - /// Keeps the buffers rows moved out of until the frame that reads them is submitted. - pub(crate) fn retire(&mut self, old: OldRows) { - self.retired = Some(old); - } - - /// Drops buffers a previous frame retired. - pub(crate) fn forget_retired(&mut self) { - self.retired = None; - } - - pub(crate) fn retired_state(&self) -> Option<&wgpu::Buffer> { - self.retired.as_ref().map(|old| &old.state) - } - - /// Makes the frozen mirror of the state buffer exist. - pub(crate) fn ensure_frozen(&mut self, device: &wgpu::Device) { - if self.frozen.is_none() { - self.frozen = Some(create(device, "latkit frozen", ROWS, self.state.size())); - self.generation += 1; - } - } - - pub(crate) fn drop_frozen(&mut self) { - if self.frozen.take().is_some() { - self.generation += 1; - } - } - - pub(crate) fn frozen(&self) -> Option<&wgpu::Buffer> { - self.frozen.as_ref() - } - - /// Makes the settled mirror of the state buffer exist. - pub(crate) fn ensure_settled(&mut self, device: &wgpu::Device) { - if self.settled.is_none() { - self.settled = Some(create(device, "latkit settled", ROWS, self.state.size())); - self.generation += 1; - } - } - - pub(crate) fn drop_settled(&mut self) { - if self.settled.take().is_some() { - self.generation += 1; - } - } - - pub(crate) fn settled(&self) -> Option<&wgpu::Buffer> { - self.settled.as_ref() - } - - /// Writes the row blocks every mark's evaluation covers this frame. + /// Writes this frame's row-block lists for every mark. pub(crate) fn write_blocks(&mut self, device: &wgpu::Device, queue: &Queue, blocks: &[u32]) { if self.blocks.ensure(device, (blocks.len() as u64 + 1) * 4) { self.generation += 1; @@ -312,31 +139,6 @@ impl Resources { } } - /// Copies `rows` rows of a region's kind and extra words from `source` (the state buffer when - /// `None`) at `from` into the frozen mirror at `to`. - pub(crate) fn copy_frozen( - &self, - encoder: &mut wgpu::CommandEncoder, - source: Option<&wgpu::Buffer>, - from: super::rows::Region, - to: super::rows::Region, - rows: u32, - ) { - let Some(frozen) = &self.frozen else { return }; - let source = source.unwrap_or(&self.state); - let mut copy = |a: u32, b: u32, words: u32| { - if words > 0 { - encoder.copy_buffer_to_buffer(source, u64::from(a) * 4, frozen, u64::from(b) * 4, u64::from(words) * 4); - } - }; - if from.stride == to.stride { - copy(from.state, to.state, rows * to.stride); - } - if from.extras == to.extras { - copy(from.extra, to.extra, rows * to.extras); - } - } - /// Writes the rows every mark's focus draw lights. pub(crate) fn write_lit(&mut self, device: &wgpu::Device, queue: &Queue, rows: &[u32]) { if self.lit.ensure(device, (rows.len() as u64 + 1) * 4) { @@ -354,20 +156,20 @@ impl Resources { queue.write_buffer(&self.remap.buffer, 0, bytemuck::cast_slice(words)); } - /// Makes room for every mark's uniforms, domains, extents, and normals for `vertices` slots. - pub(crate) fn fit(&mut self, device: &wgpu::Device, marks: u64, domains: u64, vertices: u64) { + /// Makes room for every mark's uniforms, domains, and extents. + pub(crate) fn fit(&mut self, device: &wgpu::Device, marks: u64, domains: u64) { let words = |n: u64| (n + 1) * 4; let count = marks + 1; let mut replaced = self.domains.ensure(device, words(domains)); replaced |= self.extents.ensure(device, words(u64::from(wgsl::EXTENT_WORDS) * count)); replaced |= self.marks.ensure(device, MARK_STRIDE * count); - replaced |= self.normals.ensure(device, words(3 * vertices)); if replaced { self.generation += 1; } } - /// Makes room for `words` of program; true when the buffer was replaced and must be rewritten. + /// Makes room for `words` of program. Returns true when the buffer was replaced and must be + /// rewritten. pub(crate) fn fit_program(&mut self, device: &wgpu::Device, words: u64) -> bool { let replaced = self.program.ensure(device, (words + 1) * 4); if replaced { @@ -376,7 +178,45 @@ impl Resources { replaced } - /// Makes room for `words` probe words, what the running query finds included. + /// Makes room for `words` of program, keeping its first `kept` words. A replaced buffer has + /// them copied over in `encoder`, so this frame's writes must follow the copy. Returns whether + /// it was replaced. + pub(crate) fn grow_program( + &mut self, + device: &wgpu::Device, + encoder: &mut wgpu::CommandEncoder, + words: u64, + kept: u64, + ) -> bool { + let old = self.program.buffer.clone(); + if !self.program.ensure(device, (words + 1) * 4) { + return false; + } + if kept > 0 { + encoder.copy_buffer_to_buffer(&old, 0, &self.program.buffer, 0, kept * 4); + } + self.generation += 1; + true + } + + /// Copies `words` program words from word `from` to word `to` in `encoder`, through a temporary + /// buffer, since a buffer cannot be copied into itself. + pub(crate) fn move_program( + &self, + device: &wgpu::Device, + encoder: &mut wgpu::CommandEncoder, + from: u32, + to: u32, + words: u32, + ) { + let size = u64::from(words) * 4; + let between = grown::buffer(device, "latkit program move", wgpu::BufferUsages::COPY_SRC, size); + let program = &self.program.buffer; + encoder.copy_buffer_to_buffer(program, u64::from(from) * 4, &between, 0, size); + encoder.copy_buffer_to_buffer(&between, 0, program, u64::from(to) * 4, size); + } + + /// Makes room for `words` probe words, including the running query's results. pub(crate) fn fit_probe(&mut self, device: &wgpu::Device, words: u32) { if self.probe.ensure(device, u64::from(words.max(wgsl::PROBE_WORDS)) * 4) { self.generation += 1; @@ -390,8 +230,8 @@ impl Resources { } } - /// Makes the lists buffer hold `words`; replacing it, emptied, changes the lists generation, so - /// every list counts again. + /// Makes room for `words` in the lists buffer. Replacing it empties it and changes the lists + /// generation, so every list is counted again. pub(crate) fn fit_lists(&mut self, device: &wgpu::Device, words: u64) { if self.lists.ensure(device, (words + 1) * 4) { self.generation += 1; @@ -401,7 +241,7 @@ impl Resources { /// Replaces the labels buffer with `words` of zeros. pub(crate) fn replace_labels(&mut self, device: &wgpu::Device, words: u64) { - self.labels = create(device, "latkit labels", ROWS, (words + 1) * 4); + self.labels = grown::buffer(device, "latkit labels", LABELS, (words + 1) * 4); self.generation += 1; } @@ -409,16 +249,16 @@ impl Resources { &self.labels } - /// Makes the shadow map exist at full size. - pub(crate) fn ensure_shadows(&mut self, device: &wgpu::Device) { - if self.shadows.levels.len() == 1 { - self.shadows = ShadowMap::new(device, SHADE_SIZE); + /// Makes room for `words` in the labels' claim grid, and returns it. + pub(crate) fn label_cells(&mut self, device: &wgpu::Device, words: u64) -> &wgpu::Buffer { + if self.cells.ensure(device, words * 4) { self.generation += 1; } + &self.cells.buffer } - pub(crate) fn eyes(&self) -> &wgpu::Buffer { - &self.eyes + pub(crate) fn eye(&self) -> &wgpu::Buffer { + &self.eye } pub(crate) fn program(&self) -> &wgpu::Buffer { @@ -433,18 +273,6 @@ impl Resources { &self.marks.buffer } - pub(crate) fn focus(&self) -> &wgpu::Buffer { - &self.focus - } - - pub(crate) fn focus_words(&self) -> u64 { - self.focus.size() / 4 - } - - pub(crate) fn state(&self) -> &wgpu::Buffer { - &self.state - } - pub(crate) fn domains(&self) -> &wgpu::Buffer { &self.domains.buffer } @@ -453,10 +281,6 @@ impl Resources { &self.extents.buffer } - pub(crate) fn normals(&self) -> &wgpu::Buffer { - &self.normals.buffer - } - pub(crate) fn probe(&self) -> &wgpu::Buffer { &self.probe.buffer } @@ -465,7 +289,8 @@ impl Resources { &self.sprites.buffer } - /// Changes whenever the lists buffer is replaced, emptied, so every list counts again. + /// Changes whenever the lists buffer is replaced (which empties it), so every list is counted + /// again. pub(crate) fn lists_generation(&self) -> u64 { self.lists_generation } @@ -484,20 +309,16 @@ impl Resources { } let device = &gpu.device; let entry = |binding: u32, resource| wgpu::BindGroupEntry { binding, resource }; - let eye = - wgpu::BufferBinding { buffer: &self.eyes, offset: 0, size: NonZeroU64::new(size_of::() as u64) }; let view = device.create_bind_group(&wgpu::BindGroupDescriptor { label: Some("latkit view"), layout: &inner.pipelines.view_layout, entries: &[ entry(0, self.frame.as_entire_binding()), - entry(1, inner.arena.buffer().as_entire_binding()), - entry(2, self.program.buffer.as_entire_binding()), - entry(3, self.focus.as_entire_binding()), - entry(4, wgpu::BindingResource::TextureView(&inner.colormaps.view)), - entry(5, wgpu::BindingResource::Sampler(&gpu.sampler)), - entry(6, wgpu::BindingResource::Buffer(eye)), - entry(7, self.lit.buffer.as_entire_binding()), + entry(1, self.program.buffer.as_entire_binding()), + entry(2, wgpu::BindingResource::TextureView(&inner.colormaps.view)), + entry(3, wgpu::BindingResource::Sampler(&gpu.sampler)), + entry(4, self.eye.as_entire_binding()), + entry(5, self.lit.buffer.as_entire_binding()), ], }); let binding = wgpu::BufferBinding { @@ -515,9 +336,9 @@ impl Resources { groups } - /// `shader`'s pass group, binding what its WGSL declares. + /// `shader`'s pass group, cached until a generation it binds changes. pub(crate) fn pass(&mut self, device: &wgpu::Device, inner: &mut Inner, shader: Shader) -> wgpu::BindGroup { - let generation = (self.generation, inner.icons.generation); + let generation = (inner.arena.generation, inner.icons.generation, self.generation); if self.passes_generation != generation { self.passes.clear(); self.passes_generation = generation; @@ -525,52 +346,13 @@ impl Resources { if let Some(group) = self.passes.get(&shader) { return group.clone(); } - let group = self.bind(device, inner, shader, &self.state); + let group = self.bind(device, inner, shader); self.passes.insert(shader, group.clone()); group } - /// The expression machine's group for a mark easing: its channels store into the settled mirror - /// where the state binds, and the writable stand-in binds where the mirror does. - pub(crate) fn settling_group(&self, device: &wgpu::Device, inner: &mut Inner) -> Option { - let settled = self.settled.as_ref()?; - Some(self.bind_into(device, inner, Shader::Evaluate, settled, &self.scratch)) - } - - /// The freeze pass's group: state read from the buffer rows moved out of when they moved. - pub(crate) fn freeze_group(&mut self, device: &wgpu::Device, inner: &mut Inner, moved: bool) -> wgpu::BindGroup { - match self.retired.as_ref().filter(|_| moved) { - Some(old) => self.bind(device, inner, Shader::Freeze, &old.state), - None => self.pass(device, inner, Shader::Freeze), - } - } - - /// A shadow map level's group, reading the level below it. - pub(crate) fn level_group(&self, device: &wgpu::Device, inner: &mut Inner, below: usize) -> wgpu::BindGroup { - let layout = inner.pipelines.pass_layout(device, Shader::Shade); - device.create_bind_group(&wgpu::BindGroupDescriptor { - label: Some("latkit shadow level"), - layout: &layout, - entries: &[wgpu::BindGroupEntry { - binding: 11, - resource: wgpu::BindingResource::TextureView(&self.shadows.levels[below]), - }], - }) - } - - fn bind(&self, device: &wgpu::Device, inner: &mut Inner, shader: Shader, state: &wgpu::Buffer) -> wgpu::BindGroup { - self.bind_into(device, inner, shader, state, self.settled.as_ref().unwrap_or(&self.scratch)) - } - - /// `shader`'s pass group, `state` bound as the state and `settled` as the settled mirror. - fn bind_into( - &self, - device: &wgpu::Device, - inner: &mut Inner, - shader: Shader, - state: &wgpu::Buffer, - settled: &wgpu::Buffer, - ) -> wgpu::BindGroup { + /// Builds `shader`'s pass group from the bindings its WGSL declares. + fn bind(&self, device: &wgpu::Device, inner: &mut Inner, shader: Shader) -> wgpu::BindGroup { let layout = inner.pipelines.pass_layout(device, shader); let entries: Vec> = shader .source() @@ -578,10 +360,8 @@ impl Resources { .iter() .map(|binding| { let resource = match binding.slot { - 0 => state.as_entire_binding(), - 1 => self.frozen.as_ref().unwrap_or(&self.empty).as_entire_binding(), + 0 => inner.arena.buffer().as_entire_binding(), 2 => self.extents.buffer.as_entire_binding(), - 3 => self.normals.buffer.as_entire_binding(), 4 => self.domains.buffer.as_entire_binding(), 5 => self.probe.buffer.as_entire_binding(), 6 => wgpu::BindingResource::TextureView( @@ -590,10 +370,9 @@ impl Resources { 7 => self.sprites.buffer.as_entire_binding(), 8 => wgpu::BindingResource::TextureView(&inner.atlas.view), 9 => self.lists.buffer.as_entire_binding(), - 10 => wgpu::BindingResource::TextureView(&self.shadows.view), 12 => self.remap.buffer.as_entire_binding(), 13 => self.labels.as_entire_binding(), - 14 => settled.as_entire_binding(), + 14 => self.cells.buffer.as_entire_binding(), 15 => wgpu::BindingResource::TextureView(&inner.icons.view), 16 => self.blocks.buffer.as_entire_binding(), 17 => wgpu::BindingResource::Buffer(wgpu::BufferBinding { @@ -613,8 +392,8 @@ impl Resources { }) } - /// The color and depth a frame of `size` and `format` draws into with `samples` per pixel: color - /// is `None` without multisampling, when marks draw straight into the target. + /// The color and depth textures for a frame of `size` and `format` at `samples` per pixel. + /// Color is `None` without multisampling, when marks draw straight into the target. pub(crate) fn targets( &mut self, device: &wgpu::Device, diff --git a/crates/latkit/src/view/rig.rs b/crates/latkit/src/view/rig.rs index 199965bf..62a894b2 100644 --- a/crates/latkit/src/view/rig.rs +++ b/crates/latkit/src/view/rig.rs @@ -1,32 +1,37 @@ -//! How the camera moves. A gesture moves it at once, about the point under the pointer. The wheel, -//! the keys, and a released drag leave motion owed about such a point, which the next frames pay out -//! exponentially, so a fling coasts and a wheel notch glides by the same rule. A flight eases over a -//! set time, an orbit turns steadily, and what a fit framed bounds how far the rest can take it. All -//! of it moves by the view's clock. +//! Camera motion. A gesture moves the camera at once, about the point under the pointer. The wheel, +//! the keys, and a released drag add pending motion about such a point. The following frames apply +//! it with exponential decay, so a fling coasts and a wheel notch glides by the same rule. A flight +//! eases over a set time, and an orbit turns steadily. All of it runs on the view's clock. +//! +//! The camera stays within its zoom and pan limits (`Reach`): what the space allows for the area +//! the camera frames, narrowed by fits. Every move is clamped to these limits: gestures, pending +//! motion, flights, orbits, fits, and the setter. The limits change with the view's size and +//! insets, its space, and fits, so each frame clamps the camera again before it moves. -use super::camera::{Bounds, Camera}; +use super::camera::{Bounds, Camera, Reach}; use super::{Event, View}; -/// Seconds over which owed motion pays out: a fling's pan and turn coast longer than a zoom glides. +/// Time constants, in seconds, for applying pending motion: a fling's pan and turn coast longer +/// than a zoom glides. const COAST: f64 = 0.3; const GLIDE: f64 = 0.1; #[derive(Default)] pub(crate) struct Rig { - /// The point owed motion moves about. + /// The point pending motion moves about. anchor: [f64; 3], - /// Motion owed: CSS pixels of pan, degrees of bearing and pitch, levels of zoom. + /// Pending motion: CSS pixels of pan, degrees of bearing and pitch, and levels of zoom. pan: [f64; 2], turn: [f64; 2], zoom: f64, flight: Option, /// Degrees of bearing per second. pub(crate) orbit: f64, - /// What the last fit framed. + /// The box fits have framed, which narrows the zoom and pan limits. pub(crate) bounds: Option, } -/// A flight from one camera to another, begun at a time on the view's clock. +/// A flight from one camera to another, started at `start` on the view's clock. struct Flight { from: Camera, to: Camera, @@ -35,7 +40,7 @@ struct Flight { } impl Rig { - /// Whether the camera eases toward where it goes: a flight, or motion owed. An orbit never settles. + /// Whether the camera is easing to rest: a flight or pending motion. An orbit never settles. pub(crate) fn easing(&self) -> bool { self.flight.is_some() || self.owed() } @@ -44,7 +49,7 @@ impl Rig { self.pan != [0.0; 2] || self.turn != [0.0; 2] || self.zoom != 0.0 } - /// Drops motion owed and any flight: a gesture takes the camera. + /// Drops pending motion and any flight, as when a gesture takes the camera. pub(crate) fn hold(&mut self) { self.pan = [0.0; 2]; self.turn = [0.0; 2]; @@ -52,7 +57,8 @@ impl Rig { self.flight = None; } - /// Owes `pan`, `turn`, and `zoom` more about `anchor`. + /// Adds `pan`, `turn`, and `zoom` to the pending motion, now about `anchor`, and cancels any + /// flight. pub(crate) fn owe(&mut self, anchor: [f64; 3], pan: [f64; 2], turn: [f64; 2], zoom: f64) { self.flight = None; self.anchor = anchor; @@ -61,7 +67,8 @@ impl Rig { self.zoom += zoom; } - /// Owes what a drag released at `velocity` coasts: its velocity over the coasting time. + /// Adds the coasting motion of a drag released with velocities `pan` and `turn`: each velocity + /// times the coasting time constant. pub(crate) fn fling(&mut self, anchor: [f64; 3], pan: [f64; 2], turn: [f64; 2]) { self.owe(anchor, pan.map(|v| v * COAST), turn.map(|v| v * COAST), 0.0); } @@ -74,18 +81,21 @@ impl Rig { } impl View { - /// Moves the camera by a gesture or the rig, kept within reach of what was fitted. - pub(super) fn steer(&mut self, camera: Camera) { - let camera = match &self.rig.bounds { - Some(bounds) => camera.within(bounds), - None => camera, - }; - self.move_camera(camera); + /// The camera's zoom and pan limits: the space's limits for the area the camera frames, + /// narrowed by the box fits have framed. Until a frame gives the view its size, only fits set + /// limits. + pub(super) fn reach(&self) -> Reach { + let area = self.area(); + let sized = self.size[0] > 0.0 && self.size[1] > 0.0; + let room = sized.then(|| [area[2] - area[0], area[3] - area[1]]); + Reach::new(&self.scene.space, room, self.rig.bounds) } - /// Pays out the share of motion owed for the `dt` seconds the clock moved, advances a flight, - /// and orbits. + /// Clamps the camera to its zoom and pan limits, which may have changed since the last frame. + /// Then applies the share of pending motion due for the `dt` seconds the clock moved, advances + /// a flight, and orbits. pub(super) fn advance(&mut self, dt: f64) { + self.move_camera(self.camera); if let Some(flight) = &self.rig.flight { let eased = self.clock.eased(flight.start, flight.seconds); let camera = flight.from.ease(&flight.to, eased, &self.scene.space); @@ -98,7 +108,7 @@ impl View { return; } if self.rig.owed() { - // Without motion of its own, what is owed is paid at once. + // When motion is disabled, pending motion applies at once. let moves = self.moves(); let coast = if moves { 1.0 - (-dt / COAST).exp() } else { 1.0 }; let glide = if moves { 1.0 - (-dt / GLIDE).exp() } else { 1.0 }; @@ -108,12 +118,12 @@ impl View { let view = self.projected(); let mut camera = self.camera.panned(&view, anchor, pan); camera = camera.turned(&self.projected_with(&camera), anchor, turn); - camera = camera.zoomed(&self.projected_with(&camera), anchor, zoom); + camera = camera.zoomed(&self.projected_with(&camera), anchor, zoom, &self.reach()); let settle = |v: f64, least: f64| if v.abs() < least { 0.0 } else { v }; self.rig.pan = [settle(self.rig.pan[0] - pan[0], 0.05), settle(self.rig.pan[1] - pan[1], 0.05)]; self.rig.turn = [settle(self.rig.turn[0] - turn[0], 0.01), settle(self.rig.turn[1] - turn[1], 0.01)]; self.rig.zoom = settle(self.rig.zoom - zoom, 1e-4); - self.steer(camera); + self.move_camera(camera); } if self.rig.orbit != 0.0 && matches!(self.camera, Camera::Map { .. }) && self.moves() { let Camera::Map { center, .. } = self.camera else { return }; @@ -123,15 +133,18 @@ impl View { } } - /// Moves the camera by a gesture, a fit, or a motion over frames, telling the application. + /// Moves the camera for a gesture, a fit, or an animation, or back within limits that changed, + /// and emits a `Camera` event if it moved. pub(super) fn move_camera(&mut self, camera: Camera) { if self.place_camera(camera) { - self.events.tell(Event::Camera(camera)); + self.events.tell(Event::Camera(self.camera)); } } - /// Moves the camera, evaluating again what reads it; whether it moved. + /// Moves the camera to `camera`, clamped to its zoom and pan limits, and dirties the channels + /// that read the camera or the pointer. Returns whether it moved. pub(super) fn place_camera(&mut self, camera: Camera) -> bool { + let camera = camera.within(&self.reach()); if camera == self.camera { return false; } diff --git a/crates/latkit/src/view/rows.rs b/crates/latkit/src/view/rows.rs deleted file mode 100644 index a66ac260..00000000 --- a/crates/latkit/src/view/rows.rs +++ /dev/null @@ -1,117 +0,0 @@ -//! Row regions: each mark's rows in every per-row buffer (kind words, extra words, and focus words), -//! laid out from one capacity per mark with room to grow. When a mark outgrows its room, every region -//! is laid out again into new buffers and each mark's rows move by copy, so what they hold survives: -//! eased state, a transition's frozen rows. -//! -//! Vertex marks come first, so a vertex's slot (its kind words' offset over `VERTEX_WORDS`) counts -//! every vertex mark's rows. - -use super::View; -use crate::scene::mark::MarkKind; - -/// Where a mark's rows sit in each per-row buffer, in words. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub(crate) struct Region { - /// Rows the region holds. - pub(crate) capacity: u32, - pub(crate) stride: u32, - pub(crate) extras: u32, - pub(crate) state: u32, - pub(crate) extra: u32, - pub(crate) focus: u32, -} - -impl Region { - pub(crate) const EMPTY: Region = Region { capacity: 0, stride: 0, extras: 0, state: 0, extra: 0, focus: 0 }; -} - -/// Room for `rows` rows and some to spare, so streaming rows in rarely lays regions out again. -fn room(rows: u32) -> u32 { - rows + rows / 4 + 64 -} - -/// Totals of a layout: words in each buffer. -#[derive(Clone, Copy, Default, PartialEq, Eq, Debug)] -pub(crate) struct Sizes { - pub(crate) state: u64, - pub(crate) focus: u64, -} - -impl View { - /// Fits every mark's rows and exits into its region. When one outgrows its room, or a mark's row - /// shape changed, lays every region out again into new buffers, copying each mark's rows over in - /// `encoder`. True when regions moved: vertex slots changed and focus words are written again. - pub(super) fn fit_rows(&mut self, encoder: &mut wgpu::CommandEncoder) -> bool { - let shape = |m: &super::marks::Shown| (m.stride(), m.extras()); - let fits = self.marks.iter().all(|m| { - let r = &m.region; - m.rows + m.exits <= r.capacity && (r.stride, r.extras) == shape(m) - }); - if fits && !self.regions_dirty { - return false; - } - self.regions_dirty = false; - let order: Vec = (0..self.marks.len()) - .filter(|&i| self.marks[i].kind() == MarkKind::Vertices) - .chain((0..self.marks.len()).filter(|&i| self.marks[i].kind() != MarkKind::Vertices)) - .collect(); - let mut sizes = Sizes::default(); - let mut regions = vec![Region::EMPTY; self.marks.len()]; - for &i in &order { - let m = &self.marks[i]; - let (stride, extras) = shape(m); - let need = m.rows + m.exits; - let capacity = if need <= m.region.capacity && (m.region.stride, m.region.extras) == (stride, extras) { - m.region.capacity - } else { - room(need) - }; - regions[i] = - Region { capacity, stride, extras, state: sizes.state as u32, extra: 0, focus: sizes.focus as u32 }; - sizes.state += u64::from(capacity) * u64::from(stride); - sizes.focus += u64::from(capacity); - } - // Extra words follow every mark's kind words. - for &i in &order { - regions[i].extra = sizes.state as u32; - sizes.state += u64::from(regions[i].capacity) * u64::from(regions[i].extras); - } - let device = self.gpu.0.device.clone(); - let resources = self.resources.as_mut().expect("created before rows"); - let old = resources.replace_rows(&device, sizes); - for (shown, new) in self.marks.iter_mut().zip(regions) { - let was = shown.region; - let kept = was.capacity.min(new.capacity).min(shown.held); - let copy = |encoder: &mut wgpu::CommandEncoder, - from: &wgpu::Buffer, - to: &wgpu::Buffer, - a: u32, - b: u32, - words: u32| { - if words > 0 { - encoder.copy_buffer_to_buffer(from, u64::from(a) * 4, to, u64::from(b) * 4, u64::from(words) * 4); - } - }; - if was.stride == new.stride { - copy(encoder, &old.state, resources.state(), was.state, new.state, kept * new.stride); - } - if was.extras == new.extras { - copy(encoder, &old.state, resources.state(), was.extra, new.extra, kept * new.extras); - } - // A running transition's frozen and settled rows move with its rows. - if shown.transition.is_some() && (was.stride, was.extras) == (new.stride, new.extras) { - let rows = was.capacity.min(new.capacity).min(shown.rows + shown.exits); - for (from, to) in [(&old.frozen, resources.frozen()), (&old.settled, resources.settled())] { - if let (Some(from), Some(to)) = (from, to) { - copy(encoder, from, to, was.state, new.state, rows * new.stride); - copy(encoder, from, to, was.extra, new.extra, rows * new.extras); - } - } - } - shown.moved = Some(was); - shown.region = new; - } - resources.retire(old); - true - } -} diff --git a/crates/latkit/src/view/shadows.rs b/crates/latkit/src/view/shadows.rs deleted file mode 100644 index b4a778c6..00000000 --- a/crates/latkit/src/view/shadows.rs +++ /dev/null @@ -1,160 +0,0 @@ -//! Shadows: marks that cast draw into one map of the ground through the light's eye, blended by MIN, -//! then each level of the map is made from the one below. The map covers the ground the plane's grid -//! fades over, around what the camera looks at, and is drawn again only when a caster or the camera -//! moved. - -use glam::DVec3; - -use super::View; -use super::resources::{EyeSlot, SHADE, SHADE_SIZE}; -use super::uniforms; -use crate::Space; -use crate::derive::stamp_of; -use crate::gpu::Inner; -use crate::gpu::pipelines::Variant; -use crate::gpu::shaders::{self as wgsl, Shader}; -use crate::scene::mark::{MarkKind, Primitive}; - -impl View { - /// Indices of marks that cast shadows and draw. - fn casters(&self) -> Vec { - let ground = match &self.scene.space { - Space::Plane(_) => true, - Space::Earth(_) => self.projected().globe < 0.5, - Space::Plot(_) | Space::Diagram(_) => false, - }; - if !ground { - return Vec::new(); - } - (0..self.marks.len()) - .filter(|&i| { - let m = &self.marks[i]; - m.drawable() && m.compiled.as_ref().is_some_and(|c| c.drawing.flags & wgsl::CASTS != 0) - }) - .collect() - } - - /// The light's eye and the frame's shade words for this frame, or `None` when nothing casts. - pub(super) fn shade(&self, frame: &mut wgsl::Frame, main: &wgsl::Eye) -> Option { - let casters = self.casters(); - if casters.is_empty() { - return None; - } - let projected = self.projected(); - let shadow = self.scene.theme.shadow; - let light = DVec3::from(shadow.light.map(f64::from)).normalize_or(DVec3::Z); - let light = DVec3::new(light.x, light.y, light.z.max(0.1)).normalize(); - let radius = uniforms::grid_fade(&self.scene, &projected, self.size).max(1e-9); - // Snapped to whole texels so the map does not shimmer as the camera pans. - let texel = 2.0 * radius / f64::from(SHADE_SIZE); - let snap = |v: f64| (v / texel).round() * texel; - let rect = [snap(-radius), snap(-radius), snap(-radius) + 2.0 * radius, snap(-radius) + 2.0 * radius]; - let lift = f64::from(shadow.lift) / projected.px_per_unit[0].max(1e-30); - frame.shade.light = [light.x as f32, light.y as f32, light.z as f32, shadow.softness]; - frame.shade.rect = rect.map(|v| v as f32); - frame.shade.lift = lift as f32; - frame.shade.texel = texel as f32; - frame.shade.levels = (SHADE_SIZE.ilog2() + 1) as f32; - frame.shade.on = 1; - Some(uniforms::light_eye(main, light, rect, lift, SHADE_SIZE)) - } - - /// Draws every caster into the map through `light`, then the map's levels, when what the map - /// would show changed; counted in `drawn`. - pub(super) fn cast( - &mut self, - encoder: &mut wgpu::CommandEncoder, - inner: &mut Inner, - drawn: &mut super::passes::Drawn, - light: &wgsl::Eye, - ) { - let casters = self.casters(); - // What the map is drawn from: the light's eye and each caster's state. - let versions: Vec<(usize, u64)> = casters.iter().map(|&i| (i, self.marks[i].version)).collect(); - let stamp = stamp_of((bytemuck::bytes_of(light), versions)); - if !self.shadow_map.due(stamp) { - return; - } - self.shadow_map.mark(stamp); - let gpu = self.gpu.clone(); - let device = &gpu.0.device; - let resources = self.resources.as_mut().expect("created before shadows"); - resources.ensure_shadows(device); - let groups = resources.groups(&gpu.0, inner); - let lists = resources.lists().clone(); - let mut draws = Vec::new(); - for &i in &casters { - let shown = &self.marks[i]; - let Some(compiled) = &shown.compiled else { continue }; - let custom = compiled.drawing.markers.custom.as_deref(); - let (shader, corners) = match compiled.drawing.primitive { - Primitive::Points if custom.is_some() => (Shader::PointsCustomCast, 4), - Primitive::Points => (Shader::PointsCast, 4), - Primitive::Strokes => (Shader::StrokesCast, 4), - Primitive::Triangles => (Shader::TrianglesCast, 3), - }; - let pipeline = match custom { - Some(source) => inner.pipelines.custom(device, source, shader, SHADE, 1), - None => { - Some(inner.pipelines.render_pipeline(device, Variant::draw(shader, SHADE, 1).path(shown.path()))) - } - }; - let Some(pipeline) = pipeline else { continue }; - draws.push((i, pipeline, corners, resources.pass(device, inner, shader))); - } - let base = resources.shadows.levels[0].clone(); - let open = wgpu::Color { r: 1.0, g: 65504.0, b: 65504.0, a: 0.0 }; - { - let mut pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor { - label: Some("latkit cast"), - color_attachments: &[Some(wgpu::RenderPassColorAttachment { - view: &base, - depth_slice: None, - resolve_target: None, - ops: wgpu::Operations { load: wgpu::LoadOp::Clear(open), store: wgpu::StoreOp::Store }, - })], - depth_stencil_attachment: None, - timestamp_writes: None, - occlusion_query_set: None, - multiview_mask: None, - }); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Light.offset()]); - for (i, pipeline, corners, group) in &draws { - pass.set_pipeline(pipeline); - pass.set_bind_group(1, &groups.mark, &[super::passes::offset(*i)]); - pass.set_bind_group(2, group, &[]); - self.marks[*i].record(&mut pass, drawn, *corners, &lists); - } - } - let resources = self.resources.as_ref().expect("created"); - let pipeline = inner.pipelines.render(device, Shader::Shade, SHADE, 1); - for level in 1..resources.shadows.levels.len() { - let group = resources.level_group(device, inner, level - 1); - let mut pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor { - label: Some("latkit shadow level"), - color_attachments: &[Some(wgpu::RenderPassColorAttachment { - view: &resources.shadows.levels[level], - depth_slice: None, - resolve_target: None, - ops: wgpu::Operations { load: wgpu::LoadOp::Clear(open), store: wgpu::StoreOp::Store }, - })], - depth_stencil_attachment: None, - timestamp_writes: None, - occlusion_query_set: None, - multiview_mask: None, - }); - pass.set_pipeline(&pipeline); - pass.set_bind_group(0, &groups.view, &[EyeSlot::Light.offset()]); - pass.set_bind_group(1, &groups.mark, &[0]); - pass.set_bind_group(2, &group, &[]); - drawn.draw(&mut pass, 3, 1); - } - } -} - -impl super::marks::Shown { - /// Whether the mark draws faces, which receive shadows and need normals. - pub(crate) fn faces(&self) -> bool { - self.kind() == MarkKind::Faces - } -} diff --git a/crates/latkit/src/view/sync.rs b/crates/latkit/src/view/sync.rs index dc86ddf0..60f86e0f 100644 --- a/crates/latkit/src/view/sync.rs +++ b/crates/latkit/src/view/sync.rs @@ -1,43 +1,60 @@ -//! A mark's columns on the GPU: for every column its program reads, a descriptor and a page table of -//! arena offsets, or for sampled columns a cursor per row block at the shown coordinate. +//! A mark's columns on the GPU. Each column the program reads gets a descriptor and a page table of +//! arena offsets. A sampled column instead gets a cursor per page of rows at the shown coordinate. //! -//! A column a line traces holds what is in view instead: runs of entries, frames while there are at -//! most two per pixel column, else the buckets of the level of detail that fit, and where those are -//! not held, the coarser ones standing in or the finer ones there are. Each level it draws lists its -//! axis pages and, per row block, its pages of values. +//! An unpaged table's rows are paged as its data is, and its columns read the data's pages. A paged +//! table's rows are paged by slot, one page of its order per slot. Its columns read pages packed in +//! that order, cached on the GPU by what they were packed from. A reference column to a paged table +//! holds the referenced rows' places in that table's order. //! -//! Only columns whose inputs changed are rebuilt: moving the shown coordinate rebuilds sampled -//! columns' cursors and nothing else. A rebuilt column that lists pages says which row blocks hold -//! pages that changed, so only their rows evaluate again. A mark keeps drawing its current tables until every page of -//! its rebuilt ones is resident, then swaps them in together, so a data version appears whole. -//! Tables pin the pages they reference. While a mark eases, the table its geometry column had when -//! the transition began stays, packed after the rest, so paths morph from it. +//! A column a line traces holds only what is in view: runs of entries, as raw frames while there +//! are at most two per pixel column, otherwise as buckets of the level of detail that fits. Where +//! the samples lack those, coarser buckets are shown as placeholders, or else the finer ones +//! available. Each level drawn lists its axis pages and, per page of rows, its pages of values. +//! +//! Only columns whose inputs changed are rebuilt, once each. Moving the shown coordinate rebuilds +//! only sampled columns' cursors, and a paged table rebuilds only where its marks' resident pages +//! changed. Rebuilt tables are staged together with those of every other mark of the dataset. The +//! dataset reserves every page they read all at once, or none and waits for space. Once every page +//! has arrived, the view swaps them all in, so a data version, and what a camera move draws, appear +//! complete. Staged tables are rebuilt while they wait only if their inputs change again. A rebuilt +//! column that lists pages reports which pages of rows read changed pages, so only those rows +//! evaluate again. While a mark eases, its geometry column keeps the table it had when the +//! transition began, packed after the rest, so paths morph from it. use std::ops::Range; use std::sync::Arc; -use crate::data::{Column, Field, FramePage, PAGE_ROWS, PageId, Run, Samples, Version, detail}; -use crate::derive::Beside; +use super::drawing::{Drawing, Holds}; +use crate::data::{Column, Field, FramePage, PageId, Run, Samples, Values, Version, detail}; +use crate::derive::order::Order; +use crate::derive::shapes::PagedShapes; +use crate::derive::stamp_of; +use crate::derive::{Beside, Derivations}; use crate::expr::compile::{ColumnUse, Program, Reads, Words}; use crate::gpu::Inner; -use crate::gpu::arena::{Content, PIECE_POINTS, PageKey, path_pieces, path_points}; +use crate::gpu::arena::{Ask, Content, PageKey, PathPage, path_pieces}; +use crate::gpu::packed::{self, Packing}; use crate::gpu::shaders as wgsl; -use crate::{Data, Name, Table}; +use crate::gpu::work::Count; +use crate::scene::mark::Style; +use crate::{Name, Snapshot, Table}; const WORDS: usize = wgsl::COLUMN_WORDS as usize; -/// What one column's table was built from; a change in it means rebuilding the table. +/// The inputs one column's table was built from. When they change, the table is rebuilt. #[derive(Clone, PartialEq, Debug)] enum Stamp { Missing, - Field(Version), + /// A field's version, and the id of the order it depends on: the order of a reference's target + /// table, or the order paged shapes were made in. Otherwise 0. + Field(Version, u64), Sampled(Version, Option, u32), Trace(Version, Plan), Join(Version, Version), } -/// What a trace shows: the window across, the pixel columns it spans, and the steps a line draws -/// between frames (`STEP_BEFORE` or `STEP_AFTER`, or 0). +/// What a trace shows: the axis window, the pixel columns it spans, and how a line steps between +/// frames (`STEP_BEFORE`, `STEP_AFTER`, or 0). #[derive(Clone, Copy, PartialEq, Debug)] pub(crate) struct Window { pub(crate) across: Option<[f64; 2]>, @@ -45,8 +62,8 @@ pub(crate) struct Window { pub(crate) steps: u32, } -/// What a traced column draws: runs of entries left to right, each with the point it starts at; the -/// level the window asks for; and the points the trace runs through in all. +/// What a traced column draws: runs of entries left to right, each with its first point, the level +/// of detail the window needs, and the total point count. #[derive(Clone, PartialEq, Eq, Debug)] pub(crate) struct Plan { pub(crate) level: u32, @@ -56,8 +73,8 @@ pub(crate) struct Plan { } impl Plan { - /// The entry point `point` draws: its level, its index, and its side, a bucket's least (0) or - /// greatest (1); `None` for a point that breaks the line. + /// The entry that point `point` draws: its level, its index, and its side, a bucket's minimum + /// (0) or maximum (1). `None` for a point that breaks the line. pub(crate) fn entry(&self, point: u32) -> Option<(u32, u64, usize)> { let i = self.runs.partition_point(|(_, first)| *first <= point).checked_sub(1)?; let (run, first) = &self.runs[i]; @@ -74,61 +91,280 @@ impl Plan { } } -/// One column on the GPU: its descriptor, whose table offsets are relative to `words`, its page table -/// or cursors, and the pages it pins. +/// How a mark's columns of its own table page its rows: by the data's pages, or by its drawing's +/// slots, one page of the order per slot. +#[derive(Clone, Copy)] +pub(crate) enum Rows<'a> { + Own, + Held(&'a Drawing), +} + +impl Rows<'_> { + /// A stamp of how it pages rows: 0 for the data's own pages. + fn stamp(self) -> u64 { + match self { + Rows::Own => 0, + Rows::Held(drawing) => drawing.layout(), + } + } +} + +/// One column on the GPU: its descriptor (whose table offsets are relative to `words`), its page +/// table or cursors in `words`, and the pages it reads. Until its dataset reserves them, `asks` +/// lists the pages it needs, each with the word that receives its offset. After that, `pins` lists +/// its pinned pages, and `missing` the words still waiting for an offset. struct ColumnTable { stamp: Stamp, + /// How it pages its rows: 0 for the data's pages, otherwise the stamp of its drawing's slots. + held: u64, descriptor: [u32; WORDS], words: Vec, + asks: Vec<(usize, Ask)>, pins: Vec, + /// Words waiting for a page's offset: the word, the page, and the words the page must hold. + missing: Vec<(usize, PageKey, u32)>, /// What a traced column draws. plan: Option, - /// At most how many points a geometry column's paths take. + /// The most points a geometry column's paths take. points: u32, - /// The page of each row block the table lists, for a column whose rows read their own block's - /// page; `None` for one whose every row changes when it does (cursors, traces, joins). - pages: Option>, + /// For a column whose rows each read their own page, the page each page of rows reads. `None` + /// where any change affects every row (cursors, traces, joins). + pages: Option>>, +} + +impl ColumnTable { + /// Fills in the offsets of waiting pages that have arrived. Returns whether none is still + /// waiting. + fn resolve(&mut self, gpu: &Inner) -> bool { + let words = &mut self.words; + self.missing.retain(|&(at, key, need)| match gpu.arena.offset(key, need) { + Some(offset) => { + words[at] = offset; + false + } + None => true, + }); + self.missing.is_empty() + } } /// Rows of the mark's table joined by id to rows of another table. struct Join { - of: (Name, Name), + from: (Name, Name), structure: (Version, Version), pages: Vec<(PageId, Arc<[u32]>)>, } -/// What a mark's program reads, column by column: the program's columns, then one per join. +/// A mark's rebuilt tables, by index into its current ones, and whether every table was rebuilt, +/// which replaces them all. +pub(crate) struct Rebuilt { + tables: Vec<(usize, ColumnTable)>, + reshape: bool, +} + +impl Rebuilt { + /// Unpins its tables' pages. + fn release(self, gpu: &mut Inner) { + for (_, table) in self.tables { + unpin(&table.pins, gpu); + } + } + + /// Whether every page each table reads is resident, filling in the offsets of those that + /// arrived. + fn ready(&mut self, gpu: &Inner) -> bool { + let mut ready = true; + for (_, table) in &mut self.tables { + ready &= table.asks.is_empty() && table.resolve(gpu); + } + ready + } +} + +/// The tables rebuilt for a dataset's marks since the dataset last published, by mark. They replace +/// the current tables once every page they read has arrived. +#[derive(Default)] +pub(crate) struct Staged { + marks: Vec<(usize, Rebuilt)>, +} + +impl Staged { + /// Mark `mark`'s staged tables, if any. + pub(crate) fn of(&self, mark: usize) -> Option<&Rebuilt> { + self.marks.iter().find(|(m, _)| *m == mark).map(|(_, r)| r) + } + + /// Stages mark `mark`'s `rebuilt` tables. A table rebuilt again replaces its staged version, + /// which is unpinned. A full rebuild replaces all of the mark's staged tables. + pub(crate) fn put(&mut self, mark: usize, rebuilt: Rebuilt, gpu: &mut Inner) { + let Some(at) = self.marks.iter().position(|(m, _)| *m == mark) else { + self.marks.push((mark, rebuilt)); + return; + }; + if rebuilt.reshape { + let (_, old) = std::mem::replace(&mut self.marks[at], (mark, rebuilt)); + old.release(gpu); + return; + } + let staged = &mut self.marks[at].1; + for (i, table) in rebuilt.tables { + match staged.tables.iter().position(|(j, _)| *j == i) { + Some(k) => { + let (_, old) = std::mem::replace(&mut staged.tables[k], (i, table)); + unpin(&old.pins, gpu); + } + None => staged.tables.push((i, table)), + } + } + } + + /// Reserves every page the staged tables need, all or none. Returns `false` when the arena has + /// no space for them now. A table that needs a page the arena has since dropped, and has no + /// content to upload for it, is discarded and rebuilt later. + pub(crate) fn reserve(&mut self, gpu: &mut Inner, device: &wgpu::Device, queue: &crate::gpu::work::Queue) -> bool { + let gone = |table: &ColumnTable| { + table.asks.iter().any(|(_, a)| a.content.is_none() && !gpu.arena.holds(a.key, a.need)) + }; + for (_, rebuilt) in &mut self.marks { + rebuilt.tables.retain(|(_, table)| !gone(table)); + } + let mut asks: Vec = Vec::new(); + let mut owners: Vec<(usize, usize, usize)> = Vec::new(); + for (m, (_, rebuilt)) in self.marks.iter_mut().enumerate() { + for (t, (_, table)) in rebuilt.tables.iter_mut().enumerate() { + for (at, ask) in table.asks.drain(..) { + owners.push((m, t, at)); + asks.push(ask); + } + } + } + if asks.is_empty() { + return true; + } + let reserved = gpu.arena.reserve(device, queue, &mut asks); + for ((m, t, at), ask) in owners.into_iter().zip(asks) { + let table = &mut self.marks[m].1.tables[t].1; + if reserved { + table.pins.push(ask.key); + table.missing.push((at, ask.key, ask.need)); + } else { + table.asks.push((at, ask)); + } + } + reserved + } + + /// Whether every page each staged table reads is resident, filling in the offsets of those that + /// arrived. + pub(crate) fn ready(&mut self, gpu: &Inner) -> bool { + self.marks.iter_mut().fold(true, |ready, (_, rebuilt)| rebuilt.ready(gpu) && ready) + } + + /// Removes and returns mark `mark`'s staged tables, to publish them. + pub(crate) fn take(&mut self, mark: usize) -> Option { + let at = self.marks.iter().position(|(m, _)| *m == mark)?; + Some(self.marks.swap_remove(at).1) + } + + /// Drops mark `mark`'s staged tables, after it recompiled or was removed. + pub(crate) fn forget(&mut self, mark: usize, gpu: &mut Inner) { + if let Some(rebuilt) = self.take(mark) { + rebuilt.release(gpu); + } + } + + /// Unpins every staged table. + pub(crate) fn release(self, gpu: &mut Inner) { + for (_, rebuilt) in self.marks { + rebuilt.release(gpu); + } + } + + pub(crate) fn is_empty(&self) -> bool { + self.marks.is_empty() + } +} + +/// The column tables a mark's program reads: one per program column, then one per join. #[derive(Default)] pub(crate) struct Tables { current: Vec, - /// Rebuilt tables waiting for their pages, by index into `current`. - pending: Vec<(usize, ColumnTable)>, /// The geometry column's table before it was last rebuilt, kept while the mark eases from it. frozen: Option, joins: Vec, synced: bool, - /// The rows of the table last synced, which a program reading no columns follows alone. - structure: Option, - /// Bumps whenever a table is swapped in. + /// The row structure of the table last synced. A program that reads no columns tracks only + /// this. + structure: Option, + /// Incremented whenever a table is swapped in. generation: u64, } -/// What a sync did. -pub(crate) enum Synced { - /// Nothing changed. - Same, - /// Pages are uploading; the current tables still draw. - Waiting, - /// New tables: bit `i` of `columns` for each program column that changed, and the row blocks - /// whose pages changed; `None` when every row did. - Changed { columns: u64, blocks: Option> }, +/// What publishing rebuilt tables changed. Bit `i` of `columns` is set for each program column +/// whose table changed. Bit `i` of `data` is set for each whose data changed, not only its paging. +/// `blocks` lists the row blocks whose pages changed, or is `None` when every row changed. +pub(crate) struct Published { + pub(crate) columns: u64, + pub(crate) data: u64, + pub(crate) blocks: Option>, } -/// Where a mark's own rows come from, and the tables the view derives beside them. +/// A mark's own dataset and table, the tables derived beside it, and, for a paged table, the shapes +/// made one page of its order at a time, keyed by the field each is written as. pub(crate) struct Source<'a> { - pub(crate) data: &'a Data, + pub(crate) data: &'a Snapshot, pub(crate) table: &'a Table, pub(crate) derived: Beside<'a>, + pub(crate) paged: Vec<(Name, &'a PagedShapes)>, +} + +impl<'a> Source<'a> { + /// The source of a mark drawing `style`'s table: its dataset among `all`, and the tables + /// `derived` holds beside it, without paged shapes, which only syncing reads. `None` until the + /// dataset has the table. + pub(crate) fn of(style: &Style, all: &'a [Snapshot], derived: &'a Derivations) -> Option> { + let data = all.iter().find(|d| *d.name() == style.data)?; + let table = data.table(&style.table)?; + Some(Source { data, table, derived: derived.beside(&style.data, &style.table), paged: Vec::new() }) + } + + /// The shapes written as field `name`, made one page of the drawn order at a time. + fn paged(&self, name: &Name) -> Option<&'a PagedShapes> { + self.paged.iter().find(|(field, _)| field == name).map(|&(_, shapes)| shapes) + } + + /// The dataset and table `column` reads: the mark's own, another one among `all` joined by id, + /// or the derived table beside the mark's own that holds the field. + pub(crate) fn table_of(&self, column: &ColumnUse, all: &'a [Snapshot]) -> Option<(&'a Name, &'a Table)> { + if column.derived { + let mut beside = self.derived.iter().flatten(); + let table = beside.find(|t| t.field_named(&column.name).is_some()).copied().or(self.derived[0])?; + return Some((self.data.name(), table)); + } + match &column.from { + None => Some((self.data.name(), self.table)), + Some((data, table)) => { + let data = all.iter().find(|d| d.name() == data)?; + Some((data.name(), data.table(table)?)) + } + } + } +} + +/// What a sync reads besides the mark's own rows: every dataset, the shown coordinate, the trace +/// window, and the order of each paged table in the mark's dataset. +pub(crate) struct Inputs<'a> { + pub(crate) all: &'a [Snapshot], + pub(crate) at: f64, + pub(crate) window: Window, + pub(crate) orders: &'a [(Name, Order)], +} + +impl Inputs<'_> { + /// The order of table `table` in the mark's dataset, if that table is paged. + fn order_of(&self, table: &Name) -> Option<&Order> { + self.orders.iter().find(|(name, _)| name == table).map(|(_, order)| order) + } } /// One column a program reads: a field, or the rows a join maps to. @@ -138,100 +374,107 @@ enum Use<'a> { } impl Tables { - /// Brings the tables up to date with `source` and every dataset in `all` at coordinate `at`; - /// traced columns hold the frames across `window`, or every frame without one. When column - /// `geometry` is rebuilt, its table before stays frozen until [`thaw`](Tables::thaw). - #[allow(clippy::too_many_arguments, reason = "one sync of a mark's tables")] - pub(crate) fn sync( + /// Rebuilds the tables whose inputs changed, for `source` paged as `rows`, with `inputs`. + /// Changes are measured against the `staged` tables if any, otherwise the current ones. Each + /// table is built once and requests its pages. `None` when nothing changed. + pub(crate) fn rebuild( &mut self, program: &Program, source: &Source<'_>, - all: &[Data], - at: f64, - window: Window, - geometry: u32, + rows: Rows<'_>, + inputs: &Inputs<'_>, + staged: Option<&Rebuilt>, gpu: &mut Inner, - ) -> Synced { + ) -> Option { let joins = joins(program); let fields = program.columns.iter().map(|column| { - let join = column.of.as_ref().map_or(wgsl::NONE, |of| { - (program.columns.len() + joins.iter().position(|j| *j == of).expect("listed")) as u32 + let join = column.from.as_ref().map_or(wgsl::NONE, |from| { + (program.columns.len() + joins.iter().position(|j| *j == from).expect("listed")) as u32 }); Use::Field { column, join } }); - let uses: Vec> = fields.chain(joins.iter().map(|of| Use::Join(of))).collect(); - let reshape = !self.synced - || self.current.len() != uses.len() - || (uses.is_empty() && self.structure != Some(source.table.structure)); + let uses: Vec> = fields.chain(joins.iter().map(|from| Use::Join(from))).collect(); + // Rebuild every table when there are no tables of this shape yet. If a reshape is already + // staged, compare against it instead. + let reshaped = staged.is_some_and(|s| s.reshape && s.tables.len() == uses.len()); + let reshape = !reshaped + && (!self.synced + || self.current.len() != uses.len() + || (uses.is_empty() && self.structure != Some(source.table.structure))); self.structure = Some(source.table.structure); - let mut rebuilt = Vec::new(); - let mut ready = true; + let held = rows.stamp(); + let mut rebuilt: Vec<(usize, ColumnTable)> = Vec::new(); for (i, use_) in uses.iter().enumerate() { - let stamp = match use_ { - Use::Field { column, .. } => stamp(column, source, all, at, window), - Use::Join(of) => match joined(of, all) { - Some(other) => Stamp::Join(source.table.structure, other.structure), - None => Stamp::Missing, + let (stamp, paged) = match use_ { + Use::Field { column, .. } => { + (stamp(column, source, inputs), if column.from.is_none() { held } else { 0 }) + } + Use::Join(from) => match joined(from, inputs.all) { + Some(other) => (Stamp::Join(source.table.structure, other.structure), held), + None => (Stamp::Missing, held), }, }; - if !reshape && self.current[i].stamp == stamp { + // The staged replacement for table `i` if any, else the current table. + let waiting = staged.and_then(|s| s.tables.iter().find(|(j, _)| *j == i)); + let latest = waiting.map(|(_, t)| t).or(self.current.get(i)); + if !reshape && latest.is_some_and(|t| t.stamp == stamp && t.held == paged) { continue; } - let mut builder = Builder { - words: Vec::new(), - pins: Vec::new(), - ready: true, - plan: None, - points: 0, - pages: Some(Vec::new()), - gpu: &mut *gpu, - }; + let mut builder = Builder::new(gpu); let descriptor = match use_ { Use::Field { column, join } => { - let mut descriptor = builder.column(column, source, all, at, window); + let rows = if column.from.is_none() { rows } else { Rows::Own }; + let mut descriptor = builder.column(column, source, rows, inputs); descriptor[wgsl::COLUMN_JOIN as usize] = *join; descriptor } - Use::Join(of) => { - let join = self.join(of, source, all); - let descriptor = builder.join(&join, source.table.len()); + Use::Join(from) => { + let join = self.join(from, source, inputs.all); + let descriptor = builder.join(&join, from, source, rows, inputs.all); self.remember(join); descriptor } }; - let Builder { words, pins, ready: column_ready, plan, points, pages, gpu: _ } = builder; - for &key in &pins { - gpu.arena.pin(key); - } - ready &= column_ready; - rebuilt.push((i, ColumnTable { stamp, descriptor, words, pins, plan, points, pages })); - } - for (_, table) in std::mem::take(&mut self.pending) { - unpin(&table.pins, gpu); - } - if rebuilt.is_empty() && !reshape { - return Synced::Same; - } - if !ready { - self.pending = rebuilt; - return Synced::Waiting; + rebuilt.push((i, builder.finish(stamp, paged, descriptor))); } + (reshape || !rebuilt.is_empty()).then_some(Rebuilt { tables: rebuilt, reshape }) + } + + /// Swaps in the `rebuilt` tables, whose pages are all resident, and unpins the replaced ones. + /// The replaced table of geometry column `geometry` stays frozen instead, for the mark to ease + /// from. + pub(crate) fn publish(&mut self, rebuilt: Rebuilt, program: &Program, geometry: u32, gpu: &mut Inner) -> Published { + let Rebuilt { tables: rebuilt, reshape } = rebuilt; self.generation += 1; - let mut blocks = Some(Vec::new()); - let changed = if reshape { + let joins = joins(program); + let published = if reshape { self.thaw(gpu); for table in std::mem::take(&mut self.current) { unpin(&table.pins, gpu); } - self.current = rebuilt.into_iter().map(|(_, table)| table).collect(); - blocks = None; - u64::MAX + let mut tables: Vec> = Vec::new(); + for (i, table) in rebuilt { + if tables.len() <= i { + tables.resize_with(i + 1, || None); + } + tables[i] = Some(table); + } + self.current = tables.into_iter().map(|t| t.expect("every table rebuilt")).collect(); + Published { columns: u64::MAX, data: u64::MAX, blocks: None } } else { - let mut changed = 0; + let mut published = Published { columns: 0, data: 0, blocks: Some(Vec::new()) }; for (i, table) in rebuilt { - match (blocks.as_mut(), changed_blocks(&self.current[i], &table)) { + match (published.blocks.as_mut(), changed_blocks(&self.current[i], &table)) { (Some(blocks), Some(more)) => blocks.extend(more), - _ => blocks = None, + _ => published.blocks = None, + } + let bits = match program.columns.get(i) { + Some(_) => Reads::column(i), + None => columns_from(program, joins[i - program.columns.len()]), + }; + published.columns |= bits; + if table.stamp != self.current[i].stamp { + published.data |= bits; } let old = std::mem::replace(&mut self.current[i], table); if i as u32 == geometry { @@ -240,35 +483,31 @@ impl Tables { } else { unpin(&old.pins, gpu); } - changed |= match &uses[i] { - Use::Field { .. } => Reads::column(i), - Use::Join(of) => columns_of(program, of), - }; } - changed + published }; self.synced = true; - Synced::Changed { columns: changed, blocks } + published } - /// Drops the frozen geometry table: the mark no longer eases from it. + /// Drops the frozen geometry table, so the mark stops easing from it. pub(crate) fn thaw(&mut self, gpu: &mut Inner) { if let Some(table) = self.frozen.take() { unpin(&table.pins, gpu); } } - /// The program column the frozen geometry table packs as, once there is one. + /// The column index the frozen geometry table is packed at, if there is one. pub(crate) fn frozen(&self) -> Option { self.frozen.as_ref().map(|_| self.current.len() as u32) } - /// Bumps whenever a table is swapped in. + /// Incremented whenever a table is swapped in. pub(crate) fn generation(&self) -> u64 { self.generation } - /// At most how many points the paths of geometry column `column` take; 0 for other columns. + /// The most points the paths of geometry column `column` take. 0 for other columns. pub(crate) fn points(&self, column: u32) -> u32 { self.current.get(column as usize).map_or(0, |t| t.points) } @@ -293,11 +532,6 @@ impl Tables { self.plan().map_or(0, |plan| plan.points) } - /// Whether rebuilt tables wait for pages to upload. - pub(crate) fn waiting(&self) -> bool { - !self.pending.is_empty() - } - /// Words the descriptors and tables take in the program buffer. pub(crate) fn len(&self) -> usize { self.packed().map(|t| WORDS + t.words.len()).sum() @@ -308,8 +542,13 @@ impl Tables { self.current.iter().chain(&self.frozen) } - /// Appends every descriptor, then every table; `base` is the word of the program buffer the - /// first descriptor lands at, so table offsets are absolute. + /// The pages the current and frozen tables pin. + pub(crate) fn pins(&self) -> impl Iterator + '_ { + self.packed().flat_map(|t| t.pins.iter().copied()) + } + + /// Appends every descriptor, then every table. `base` is the program-buffer word of the first + /// descriptor, which makes table offsets absolute. pub(crate) fn pack(&self, base: u32, out: &mut Vec) { let mut at = base + (self.packed().count() * WORDS) as u32; for table in self.packed() { @@ -326,43 +565,43 @@ impl Tables { } } - /// Releases every pin. + /// Unpins every table and marks the tables unsynced. pub(crate) fn release(&mut self, gpu: &mut Inner) { self.thaw(gpu); - for table in self.current.drain(..).chain(self.pending.drain(..).map(|(_, t)| t)) { + for table in self.current.drain(..) { unpin(&table.pins, gpu); } self.synced = false; } - /// The join rows for `of`, reused while neither table's rows change. - fn join(&self, of: &(Name, Name), source: &Source<'_>, all: &[Data]) -> Join { - let Some(other) = joined(of, all) else { + /// The join rows for `from`, reused while neither table's rows change. + fn join(&self, from: &(Name, Name), source: &Source<'_>, all: &[Snapshot]) -> Join { + let Some(other) = joined(from, all) else { return Join { - of: of.clone(), + from: from.clone(), structure: (source.table.structure, source.table.structure), pages: Vec::new(), }; }; let structure = (source.table.structure, other.structure); - if let Some(join) = self.joins.iter().find(|j| &j.of == of && j.structure == structure) { - return Join { of: of.clone(), structure, pages: join.pages.clone() }; + if let Some(join) = self.joins.iter().find(|j| &j.from == from && j.structure == structure) { + return Join { from: from.clone(), structure, pages: join.pages.clone() }; } let rows: Vec = (0..source.table.len()) .map(|r| source.table.ids.key(r).and_then(|key| other.row(key)).map_or(wgsl::NONE, |row| row as u32)) .collect(); - let pages = rows.chunks(PAGE_ROWS).map(|chunk| (PageId::next(), Arc::from(chunk))).collect(); - Join { of: of.clone(), structure, pages } + let pages = rows.chunks(wgsl::PAGE_ROWS as usize).map(|chunk| (PageId::next(), Arc::from(chunk))).collect(); + Join { from: from.clone(), structure, pages } } fn remember(&mut self, join: Join) { - self.joins.retain(|j| j.of != join.of); + self.joins.retain(|j| j.from != join.from); self.joins.push(join); } } -/// The row blocks whose pages differ between table `before` and table `now`; `None` when either reads -/// no block's page alone, so every row changed. +/// The pages of rows whose page differs between table `before` and table `now`. `None` when either +/// table has no page list (see `ColumnTable::pages`), so every row changed. fn changed_blocks(before: &ColumnTable, now: &ColumnTable) -> Option> { let (Some(old), Some(new)) = (&before.pages, &now.pages) else { return None }; Some((0..old.len().max(new.len())).filter(|&p| old.get(p) != new.get(p)).map(|p| p as u32).collect()) @@ -377,48 +616,31 @@ fn unpin(pins: &[PageKey], gpu: &mut Inner) { /// The distinct tables the program's joins read, in first-read order. fn joins(program: &Program) -> Vec<&(Name, Name)> { let mut joins: Vec<&(Name, Name)> = Vec::new(); - for of in program.columns.iter().filter_map(|c| c.of.as_ref()) { - if !joins.contains(&of) { - joins.push(of); + for from in program.columns.iter().filter_map(|c| c.from.as_ref()) { + if !joins.contains(&from) { + joins.push(from); } } joins } -/// Bits of the program's columns that read through the join to `of`. -fn columns_of(program: &Program, of: &(Name, Name)) -> u64 { +/// Bits of the program's columns that read through the join from `from`. +fn columns_from(program: &Program, from: &(Name, Name)) -> u64 { program .columns .iter() .enumerate() - .filter(|(_, c)| c.of.as_ref() == Some(of)) + .filter(|(_, c)| c.from.as_ref() == Some(from)) .fold(0, |bits, (i, _)| bits | Reads::column(i)) } -/// The dataset and table a column reads: the mark's own, another one joined by id, or the derived -/// table beside the mark's own that holds the field. -pub(crate) fn table_of<'a>(column: &ColumnUse, source: &Source<'a>, all: &'a [Data]) -> Option<(&'a Name, &'a Table)> { - if column.derived { - let mut beside = source.derived.iter().flatten(); - let table = beside.find(|t| t.field_named(&column.name).is_some()).copied().or(source.derived[0])?; - return Some((source.data.name(), table)); - } - match &column.of { - None => Some((source.data.name(), source.table)), - Some((data, table)) => { - let data = all.iter().find(|d| d.name() == data)?; - Some((data.name(), data.table(table)?)) - } - } -} - /// The table a join reads. -fn joined<'a>((data, table): &(Name, Name), all: &'a [Data]) -> Option<&'a Table> { +fn joined<'a>((data, table): &(Name, Name), all: &'a [Snapshot]) -> Option<&'a Table> { all.iter().find(|d| d.name() == data)?.table(table) } -/// What a trace across `window` draws: the runs covering its frames at the level they ask for, a -/// point left out between runs that do not meet. +/// What a trace across `window` draws: the runs covering its frames at the level of detail they +/// need, with one point left out between runs that do not meet, which breaks the line. fn plan(samples: &Samples, window: Window) -> Plan { let frames = window.across.map_or(samples.start()..samples.end(), |across| samples.across(across)); let level = detail(frames.end - frames.start, window.columns); @@ -441,67 +663,136 @@ fn plan(samples: &Samples, window: Window) -> Plan { Plan { level, runs, points: point, steps: window.steps } } -fn stamp(column: &ColumnUse, source: &Source<'_>, all: &[Data], at: f64, window: Window) -> Stamp { - let Some((_, table)) = table_of(column, source, all) else { return Stamp::Missing }; +/// Where sampled `column` reads `samples` at axis coordinate `at`: the frame shown there, and how +/// far toward the next one it is (0 for a column that steps). +fn shown_frame(samples: &Samples, column: &ColumnUse, at: f64) -> (Option, f32) { + let frame = samples.frame_at(at); + let t = if column.step { 0.0 } else { frame.map_or(0.0, |f| samples.t(f, at)) }; + (frame, t) +} + +/// A column descriptor with `flags`, reading no low parts and no join. +fn descriptor(flags: u32) -> [u32; WORDS] { + let mut d = [0; WORDS]; + d[wgsl::COLUMN_FLAGS as usize] = flags; + d[wgsl::COLUMN_LOW as usize] = wgsl::NONE; + d[wgsl::COLUMN_JOIN as usize] = wgsl::NONE; + d +} + +/// The cache key of a column of `table` packed in `drawing`'s order: field `field` as `words`. +fn packing(drawing: &Drawing, source: &Source<'_>, table: &Name, field: Name, words: Words) -> Packing { + Packing { + order: drawing.order.id(), + origin: source.data.origin(), + table: table.clone(), + field, + words, + low: false, + reads: 0, + } +} + +fn stamp(column: &ColumnUse, source: &Source<'_>, inputs: &Inputs<'_>) -> Stamp { + if let Some(shapes) = source.paged(&column.name).filter(|_| column.derived) { + return Stamp::Field(shapes.version, shapes.order); + } + let Some((_, table)) = source.table_of(column, inputs.all) else { return Stamp::Missing }; match table.field_named(&column.name) { None => Stamp::Missing, - Some(Field::Sampled(samples, version)) if column.trace => Stamp::Trace(*version, plan(samples, window)), + Some(Field::Sampled(samples, version)) if column.trace => Stamp::Trace(*version, plan(samples, inputs.window)), Some(Field::Sampled(samples, version)) => { - let frame = samples.frame_at(at); - let t = if column.step { 0.0 } else { frame.map_or(0.0, |f| samples.t(f, at)) }; + let (frame, t) = shown_frame(samples, column, inputs.at); Stamp::Sampled(*version, frame, t.to_bits()) } - Some(field) => Stamp::Field(field.version()), + Some(Field::Reference(reference)) => { + let to = if column.from.is_none() { inputs.order_of(&reference.to).map_or(0, Order::id) } else { 0 }; + Stamp::Field(reference.version, to) + } + Some(field) => Stamp::Field(field.version(), 0), } } -/// Builds one column's table, asking the arena for each page. +/// Builds one column's table, requesting each page it reads. struct Builder<'g> { words: Vec, - pins: Vec, - ready: bool, + asks: Vec<(usize, Ask)>, plan: Option, points: u32, - /// The page of each row block, while every row reads its own block's. - pages: Option>, + /// The page each page of rows reads, while every row reads only its own page. + pages: Option>>, gpu: &'g mut Inner, } -impl Builder<'_> { - fn column( - &mut self, - column: &ColumnUse, - source: &Source<'_>, - all: &[Data], - at: f64, - window: Window, - ) -> [u32; WORDS] { - let mut d = [0; WORDS]; - d[wgsl::COLUMN_LOW as usize] = wgsl::NONE; - if column.words == Words::Bits { - d[wgsl::COLUMN_FLAGS as usize] = wgsl::BITS; +impl<'g> Builder<'g> { + fn new(gpu: &'g mut Inner) -> Builder<'g> { + Builder { words: Vec::new(), asks: Vec::new(), plan: None, points: 0, pages: Some(Vec::new()), gpu } + } + + fn finish(self, stamp: Stamp, held: u64, descriptor: [u32; WORDS]) -> ColumnTable { + let Builder { words, asks, plan, points, pages, gpu: _ } = self; + ColumnTable { stamp, held, descriptor, words, asks, pins: Vec::new(), missing: Vec::new(), plan, points, pages } + } + + /// Requests page `key`, holding `need` words, whose offset goes to word `at` once it arrives. + /// Until then the word is `NONE` and the table waits. `content` builds the upload when the + /// arena lacks the page. + fn place(&mut self, at: usize, key: PageKey, need: u32, content: impl FnOnce() -> Content) { + let content = (!self.gpu.arena.holds(key, need)).then(content); + self.asks.push((at, Ask { key, need, content })); + self.words[at] = wgsl::NONE; + } + + /// Appends a word for page `key`'s offset, which [`place`](Builder::place) fills. + fn push(&mut self, key: PageKey, need: u32, content: impl FnOnce() -> Content) { + self.words.push(wgsl::NONE); + let at = self.words.len() - 1; + self.place(at, key, need, content); + } + + /// Records `page` as the page the next page of rows reads. + fn list(&mut self, page: Option) { + if let Some(pages) = &mut self.pages { + pages.push(page); } - let Some((data, table)) = table_of(column, source, all) else { + } + + fn column(&mut self, column: &ColumnUse, source: &Source<'_>, rows: Rows<'_>, inputs: &Inputs<'_>) -> [u32; WORDS] { + let mut d = descriptor(if column.words == Words::Bits { wgsl::BITS } else { 0 }); + if let Rows::Held(drawing) = rows + && let Some(shapes) = source.paged(&column.name).filter(|_| column.derived) + { + d[wgsl::COLUMN_ROWS as usize] = drawing.live(); + let packing = packing(drawing, source, &source.table.name, column.name.clone(), Words::Path); + self.shaped(shapes, drawing, &packing); + return d; + } + let Some((data, table)) = source.table_of(column, inputs.all) else { self.pages = None; return d; }; d[wgsl::COLUMN_ROWS as usize] = table.len() as u32; - match table.field_named(&column.name) { + let field = table.field_named(&column.name); + if let Rows::Held(drawing) = rows { + d[wgsl::COLUMN_ROWS as usize] = drawing.live(); + let packing = packing(drawing, source, &table.name, column.name.clone(), column.words); + self.held(column, data, field, drawing, packing, inputs, &mut d); + return d; + } + match field { Some(Field::Values(values, _)) if column.words == Words::Codes => { let field = (data, &table.name, &column.name); - self.listed(values); for page in values.pages() { + self.list(Some(page.id)); let key = PageKey::of(page.id, Words::Codes, false); - // Coded once, whether or not the page is on the GPU, so the dictionary holds its texts. + // Each page is coded once, resident or not, so the dictionary holds its texts. let codes = self.gpu.dictionaries.field(field.0, field.1, field.2).codes(page); - let offset = - self.gpu.arena.get(key, page.values.len() as u32, || Content::Computed(codes.to_vec())); - self.push(key, offset); + self.push(key, page.values.len() as u32, || Content::Computed(codes.to_vec())); } } Some(Field::Values(values, _)) if column.words == Words::Path => self.path(values), Some(Field::Values(values, _)) => { - self.listed(values); + values.pages().iter().for_each(|page| self.list(Some(page.id))); self.pages(values, column.words, false); if column.low { d[wgsl::COLUMN_LOW as usize] = self.words.len() as u32; @@ -509,20 +800,27 @@ impl Builder<'_> { } } Some(Field::Reference(reference)) => { - self.listed(&reference.rows); - self.pages(&reference.rows, Words::Bits, false); + reference.rows.pages().iter().for_each(|page| self.list(Some(page.id))); + match inputs.order_of(&reference.to).filter(|_| column.from.is_none()) { + // The referenced rows' places in their table's order. + Some(order) => { + for (p, page) in reference.rows.pages().iter().enumerate() { + let id = self.gpu.packed.ranked(page.id, order.id()); + let key = PageKey::of(id, Words::Bits, false); + let need = page.values.len() as u32; + self.push(key, need, || Content::Computed(packed::ranked_page(reference, p, order))); + } + } + None => self.pages(&reference.rows, Words::Bits, false), + } } Some(Field::Sampled(samples, _)) if column.trace => { - self.pages = None; - d[wgsl::COLUMN_FLAGS as usize] |= wgsl::SAMPLED; - self.trace(samples, plan(samples, window), table.len()); + let blocks: Vec<(Samples, usize)> = + (0..table.len().div_ceil(wgsl::PAGE_ROWS as usize)).map(|b| (samples.clone(), b)).collect(); + self.trace(&mut d, samples, &blocks, inputs.window); } Some(Field::Sampled(samples, _)) => { - self.pages = None; - d[wgsl::COLUMN_FLAGS as usize] |= wgsl::SAMPLED | if column.step { wgsl::STEP } else { 0 }; - let frame = samples.frame_at(at); - let t = if column.step { 0.0 } else { frame.map_or(0.0, |f| samples.t(f, at)) }; - d[wgsl::COLUMN_T as usize] = t.to_bits(); + let frame = self.sampled(&mut d, column, samples, inputs.at); self.cursors(samples, frame, table.len()); } None => self.pages = None, @@ -530,78 +828,311 @@ impl Builder<'_> { d } - /// A join's table: a bits column of the mark's rows. - fn join(&mut self, join: &Join, rows: usize) -> [u32; WORDS] { + /// Sets descriptor `d` to read sampled `column` at axis coordinate `at`, through a cursor per + /// page of rows. Returns the frame shown there. + fn sampled(&mut self, d: &mut [u32; WORDS], column: &ColumnUse, samples: &Samples, at: f64) -> Option { self.pages = None; - let mut d = [0; WORDS]; - d[wgsl::COLUMN_FLAGS as usize] = wgsl::BITS; - d[wgsl::COLUMN_ROWS as usize] = rows as u32; - d[wgsl::COLUMN_LOW as usize] = wgsl::NONE; - d[wgsl::COLUMN_JOIN as usize] = wgsl::NONE; - for (id, words) in &join.pages { - let key = PageKey::of(*id, Words::Bits, false); - let offset = self.gpu.arena.get(key, words.len() as u32, || Content::Computed(words.to_vec())); - self.push(key, offset); + d[wgsl::COLUMN_FLAGS as usize] |= wgsl::SAMPLED | if column.step { wgsl::STEP } else { 0 }; + let (frame, t) = shown_frame(samples, column, at); + d[wgsl::COLUMN_T as usize] = t.to_bits(); + frame + } + + /// A paged table's column: one page packed in the table's order per slot, cached under + /// `packing`. + #[allow(clippy::too_many_arguments, reason = "one column's pages in its order")] + fn held( + &mut self, + column: &ColumnUse, + data: &Name, + field: Option<&Field>, + drawing: &Drawing, + packing: Packing, + inputs: &Inputs<'_>, + d: &mut [u32; WORDS], + ) { + let order = &drawing.order; + let slots = drawing.slot_count(); + match field { + Some(Field::Values(values, _)) if column.words == Words::Codes => { + let field = (packing.table.clone(), packing.field.clone()); + self.held_pages(drawing, &packing, Some(values), |gpu, rows| { + let mut codes = gpu.dictionaries.field(data, &field.0, &field.1).rows(values, rows); + codes.resize(wgsl::PAGE_ROWS as usize, wgsl::MISSING); + codes + }); + } + Some(Field::Values(values, _)) if column.words == Words::Path => self.held_path(values, drawing, &packing), + Some(Field::Values(values, _)) => { + let words = column.words; + self.held_pages(drawing, &packing, Some(values), |_, rows| packed::values(values, rows, words, false)); + if column.low { + d[wgsl::COLUMN_LOW as usize] = self.words.len() as u32; + let low = Packing { low: true, ..packing }; + let listed = self.pages.take(); + self.held_pages(drawing, &low, Some(values), |_, rows| packed::values(values, rows, words, true)); + self.pages = listed; + } + } + Some(Field::Reference(reference)) => { + let target = inputs.order_of(&reference.to); + let packing = Packing { reads: target.map_or(0, Order::id), ..packing }; + self.held_pages(drawing, &packing, Some(&reference.rows), |_, rows| { + packed::targets(reference, rows, target) + }); + } + Some(Field::Sampled(samples, version)) if column.trace => { + let packing = Packing { reads: stamp_of(version), ..packing }; + let blocks: Vec<(Samples, usize)> = (0..slots) + .map(|s| match slot_key(&packing, drawing, s) { + None => (Samples::default(), 0), + Some((packing, page)) => { + (self.gpu.packed.samples(&packing, page, &drawing.rows(s), samples), 0) + } + }) + .collect(); + self.trace(d, samples, &blocks, inputs.window); + } + Some(Field::Sampled(samples, version)) => { + let frame = self.sampled(d, column, samples, inputs.at); + // A cursor per slot, reading the frame packed in the slot's rows. + for s in 0..slots { + let at = self.cursor_words(); + let Some((base, page)) = slot_key(&packing, drawing, s) else { continue }; + for (k, f) in [frame, frame.map(|f| f + 1)].into_iter().enumerate() { + let Some(f) = f.filter(|&f| samples.held(0, f)) else { continue }; + let packing = Packing { reads: stamp_of((version, f)), ..base.clone() }; + let id = self.gpu.packed.page(&packing, order, None, page); + let key = PageKey::of(id, Words::Values, false); + let rows = drawing.rows(s); + let counters = self.gpu.packed.counters(); + let content = || { + counters.add(Count::Packed, rows.len() as u64); + Content::Computed(packed::frame(samples, &rows, Some(f))) + }; + self.cursor(at, k, key, 0, wgsl::PAGE_ROWS, content); + } + } + } + None => self.pages = None, } - d } - /// Lists 's pages as the row blocks the table reads. - fn listed(&mut self, column: &Column) { - if let Some(pages) = &mut self.pages { - pages.extend(column.pages().iter().map(|page| page.id)); + /// Appends the page of `packing` in each slot of `drawing`, packed from `source` by `pack` the + /// first time it is needed, or `NONE` for a free slot. The order's pages are looked up in one + /// pass over the column, and each page of kept rows separately. + fn held_pages( + &mut self, + drawing: &Drawing, + packing: &Packing, + source: Option<&Column>, + pack: impl Fn(&mut Inner, &[u32]) -> Vec, + ) { + let slots = drawing.slot_count(); + let pages: Vec = (0..slots) + .filter_map(|s| match drawing.holds(s) { + Holds::Page(page) => Some(page), + _ => None, + }) + .collect(); + let mut ids = self.gpu.packed.pages_of(packing, &drawing.order, source, &pages).into_iter(); + for s in 0..slots { + let id = match drawing.holds(s) { + Holds::Free => None, + Holds::Page(_) => ids.next(), + Holds::Kept(_) => slot_page(&mut self.gpu.packed, packing, drawing, source, s), + }; + let Some(id) = id else { + self.words.push(wgsl::NONE); + self.list(None); + continue; + }; + self.list(Some(id)); + let key = PageKey::of(id, packing.words, packing.low); + let words = if self.gpu.arena.holds(key, wgsl::PAGE_ROWS) { + Vec::new() + } else { + let rows = drawing.rows(s); + self.gpu.packed.count(rows.len()); + pack(self.gpu, &rows) + }; + self.push(key, wgsl::PAGE_ROWS, || Content::Computed(words)); } } + /// A join's table: a bits column holding, for each of the mark's rows, the row of the other + /// table with the same id. + fn join( + &mut self, + join: &Join, + from: &(Name, Name), + source: &Source<'_>, + rows: Rows<'_>, + all: &[Snapshot], + ) -> [u32; WORDS] { + self.pages = None; + let mut d = descriptor(wgsl::BITS); + d[wgsl::COLUMN_ROWS as usize] = source.table.len() as u32; + if let Rows::Held(drawing) = rows { + d[wgsl::COLUMN_ROWS as usize] = drawing.live(); + let Some(other) = joined(from, all) else { return d }; + let field = format!("{}/{}", from.0, from.1).into(); + let packing = Packing { + reads: stamp_of((source.table.structure, other.structure, other.order)), + ..packing(drawing, source, &source.table.name, field, Words::Bits) + }; + self.held_pages(drawing, &packing, None, |_, rows| packed::joined(source.table, rows, other)); + return d; + } + for (id, words) in &join.pages { + let key = PageKey::of(*id, Words::Bits, false); + self.push(key, words.len() as u32, || Content::Computed(words.to_vec())); + } + d + } + fn pages(&mut self, column: &Column, words: Words, low: bool) { for page in column.pages() { let key = PageKey::of(page.id, words, low); let values = page.values.clone(); - let offset = self.gpu.arena.get(key, values.len() as u32, || Content::Values { values, words, low }); - self.push(key, offset); + self.push(key, values.len() as u32, || Content::Values { values, words, low }); } } - /// A geometry column's table, as `lib/read.wesl` reads it: per page, the block of each row's - /// first point, where the page's blocks of points are listed in the table, and how many points - /// it has; then those lists. A page's points take as many blocks of a chunk as they need. + /// A geometry column's table, in the layout `lib/read.wesl` reads: per page, the block of its + /// row table (each row's first point), where its blocks of points are listed, its point count, + /// and its row count, followed by those lists. A page's points take as many blocks as they + /// need. fn path(&mut self, column: &Column) { - self.listed(column); let pages = column.pages(); - self.words.resize(3 * pages.len(), 0); + self.words.resize(wgsl::PATH_PAGE_WORDS as usize * pages.len(), 0); for (p, page) in pages.iter().enumerate() { - let points = path_points(&page.values); - self.points += points as u32; - let mut made: Option>> = None; - let mut blocks = Vec::with_capacity(1 + points.div_ceil(PIECE_POINTS)); - for piece in 0..=points.div_ceil(PIECE_POINTS) { - let key = PageKey { page: page.id, words: Words::Path, low: false, piece: piece as u32 }; - let offset = self.gpu.arena.get(key, 1, || { - let pieces = made.get_or_insert_with(|| path_pieces(&page.values)); - Content::Computed(std::mem::take(&mut pieces[piece])) - }); - self.pins.push(key); - self.ready &= offset.is_some(); - blocks.push(offset.unwrap_or(wgsl::NONE)); - } - let listed = self.words.len() as u32; - self.words[3 * p..3 * p + 3].copy_from_slice(&[blocks[0], listed, points as u32]); - self.words.extend(&blocks[1..]); - } - } - - /// What a line draws, laid out as `lib/trace.wesl` reads it: a header (how many runs, row blocks, - /// and levels, and the points in all); per run its level's place, first entry, entries, and first - /// point; per level the level, how many axis pages and where they are listed, and where its row - /// blocks' lists start; per level and row block how many pages of frames or least values and where - /// they are listed, then of greatest values; then those lists, axis pages as (offset, first entry) - /// and pages of values as (offset, first entry, words per entry). Offsets within the table are - /// relative to it. - fn trace(&mut self, samples: &Samples, plan: Plan, rows: usize) { - let blocks = rows.div_ceil(PAGE_ROWS); - // Each run has its own local origin. Even sparse runs billions of frames apart use - // compact GPU offsets, while CPU plans retain their absolute 64-bit indexes. - self.words.extend([plan.runs.len() as u32, blocks as u32, plan.runs.len() as u32, plan.points]); + self.list(Some(page.id)); + self.path_page(p, page.id, page.values.len() as u32, || page.points(), || path_pieces(&page.values)); + } + } + + /// A paged table's geometry column: one page of points per slot, cached under `packing`. + fn held_path(&mut self, column: &Column, drawing: &Drawing, packing: &Packing) { + self.words.resize(wgsl::PATH_PAGE_WORDS as usize * drawing.slot_count(), 0); + for s in 0..drawing.slot_count() { + let Some(id) = slot_page(&mut self.gpu.packed, packing, drawing, Some(column), s) else { + self.no_path_page(s); + continue; + }; + self.list(Some(id)); + let rows = drawing.rows(s); + let counters = self.gpu.packed.counters(); + let pieces = || { + counters.add(Count::Packed, rows.len() as u64); + path_pieces(&column.pick(&rows)) + }; + self.path_page(s, id, rows.len() as u32, || rows_points(column, &rows), pieces); + } + } + + /// A paged table's derived shapes, made one page of its order at a time: those of each resident + /// page, and of its kept rows individually, in pages cached under `packing` by their contents. + /// A page draws nothing until its shapes are made. + fn shaped(&mut self, shapes: &PagedShapes, drawing: &Drawing, packing: &Packing) { + self.words.resize(wgsl::PATH_PAGE_WORDS as usize * drawing.slot_count(), 0); + let alike = shapes.order == drawing.order.id(); + for s in 0..drawing.slot_count() { + let page: Option<(PageId, std::borrow::Cow<'_, Values>)> = match drawing.holds(s) { + Holds::Page(page) if alike => { + shapes.pages.get(&page).map(|p| (p.id, std::borrow::Cow::Borrowed(&p.shapes))) + } + Holds::Kept(_) if alike => { + let places = drawing.places(s); + let kept = places.iter().map(|d| Some((shapes.kept.get(d).and_then(Values::geometry)?, 0))); + let kept = Values::of_rows(shapes.depth, kept); + let packing = + Packing { order: stamp_of((packing.order, places, shapes.version)), ..packing.clone() }; + let id = self.gpu.packed.page(&packing, &drawing.order, None, 0); + Some((id, std::borrow::Cow::Owned(kept))) + } + _ => None, + }; + let Some((id, shaped)) = page else { + self.no_path_page(s); + continue; + }; + self.list(Some(id)); + let rows = shaped.len() as u32; + let counters = self.gpu.packed.counters(); + let pieces = || { + counters.add(Count::Packed, u64::from(rows)); + path_pieces(&shaped) + }; + self.path_page(s, id, rows, || shaped.points(), pieces); + } + } + + /// Page `p` of a geometry column's table, holding no rows: no row table and no points. + fn no_path_page(&mut self, p: usize) { + let per = wgsl::PATH_PAGE_WORDS as usize; + self.words[per * p..per * (p + 1)].copy_from_slice(&[wgsl::NONE, 0, 0, 0]); + self.list(None); + } + + /// Page `p` (`id`, with `rows` rows) of a geometry column's table: its row table, then its + /// pieces of points, each keyed by its contents, so a rewritten page uploads only the pieces + /// whose points changed. `pieces` builds them when the GPU lacks one, and `points` counts the + /// page's points when its pieces are not cached. + fn path_page( + &mut self, + p: usize, + id: PageId, + rows: u32, + points: impl FnOnce() -> usize, + pieces: impl FnOnce() -> PathPage, + ) { + let per = wgsl::PATH_PAGE_WORDS as usize; + let arena = &self.gpu.arena; + let known = self.gpu.pieces.of(id); + let points = known.as_ref().map_or_else(points, |&(_, points)| points); + self.points += points as u32; + let kept = known.map(|(keys, _)| keys).filter(|keys| keys.iter().all(|&key| arena.holds(key, 1))); + let (keys, mut made) = if let Some(keys) = kept { + (keys, Vec::new()) + } else { + let PathPage { rows, pieces } = pieces(); + let made: Vec> = std::iter::once(rows).chain(pieces).collect(); + let keys: Arc<[PageKey]> = made.iter().map(|words| PageKey::piece(words)).collect(); + self.gpu.pieces.keep(id, keys.clone(), points); + (keys, made) + }; + let listed = self.words.len(); + self.words.resize(listed + keys.len() - 1, wgsl::NONE); + for (piece, &key) in keys.iter().enumerate() { + let at = if piece == 0 { per * p } else { listed + piece - 1 }; + let words = made.get_mut(piece).map(std::mem::take).unwrap_or_default(); + self.place(at, key, 1, || Content::Computed(words)); + } + self.words[per * p + 1] = listed as u32; + self.words[per * p + 2] = points as u32; + self.words[per * p + 3] = rows; + } + + /// What a line draws, in the layout `lib/trace.wesl` reads: + /// + /// - a header: the run count, pages of rows, level count, and total points; + /// - per run: its level's index, first entry, entry count, and first point; + /// - per level: the level, its axis page count and list offset, and its page lists' start; + /// - per level and page of rows: the count and list offset of its pages of frames or minimum + /// values, then the same for maximum values; + /// - the lists: axis pages as (offset, first entry), and pages of values as (offset, first + /// entry, words per entry). + /// + /// Offsets within the table are relative to it. `blocks` gives each page of rows the samples it + /// reads and its block among them. Descriptor `d` reads it as sampled, across `window`. + fn trace(&mut self, d: &mut [u32; WORDS], axis: &Samples, blocks: &[(Samples, usize)], window: Window) { + self.pages = None; + d[wgsl::COLUMN_FLAGS as usize] |= wgsl::SAMPLED; + let plan = plan(axis, window); + // Each run has its own origin, so runs billions of frames apart still get 32-bit GPU + // offsets. CPU plans keep absolute 64-bit indexes. + self.words.extend([plan.runs.len() as u32, blocks.len() as u32, plan.runs.len() as u32, plan.points]); let origins: Vec = plan .runs .iter() @@ -614,25 +1145,23 @@ impl Builder<'_> { self.words.resize(at + 4 * plan.runs.len(), 0); for (i, (run, _)) in plan.runs.iter().enumerate() { let origin = origins[i]; - let axis = samples.axis_pages(run.level, run.entries.clone()); + let axis_pages = axis.axis_pages(run.level, run.entries.clone()); let listed_at = self.words.len() as u32; - self.words[at + 4 * i..at + 4 * i + 3].copy_from_slice(&[run.level, axis.len() as u32, listed_at]); - for (page, held) in &axis { + self.words[at + 4 * i..at + 4 * i + 3].copy_from_slice(&[run.level, axis_pages.len() as u32, listed_at]); + for (page, held) in &axis_pages { let key = PageKey::of(page.id, Words::Values, false); - let offset = - self.gpu.arena.get(key, ((held.end - page.first) * 2) as u32, || Content::Axis(page.clone())); - self.pins.push(key); - self.ready &= offset.is_some(); - self.words.extend([offset.unwrap_or(wgsl::NONE), (page.first - origin) as u32]); + let need = ((held.end - page.first) * 2) as u32; + self.push(key, need, || Content::Axis(page.clone())); + self.words.push((page.first - origin) as u32); } let lists = self.words.len(); self.words[at + 4 * i + 3] = lists as u32; - self.words.resize(lists + 4 * blocks, 0); - for block in 0..blocks { + self.words.resize(lists + 4 * blocks.len(), 0); + for (b, (samples, block)) in blocks.iter().enumerate() { for side in 0..crate::data::sides(run.level) { - let pages = samples.pages(block, run.level, side, run.entries.clone()); - self.words[lists + 4 * block + 2 * side] = pages.len() as u32; - self.words[lists + 4 * block + 2 * side + 1] = self.words.len() as u32; + let pages = samples.pages(*block, run.level, side, run.entries.clone()); + self.words[lists + 4 * b + 2 * side] = pages.len() as u32; + self.words[lists + 4 * b + 2 * side + 1] = self.words.len() as u32; self.sample_pages(&pages, origin); } } @@ -640,40 +1169,86 @@ impl Builder<'_> { self.plan = Some(plan); } - /// Appends `pages` as (offset, first entry, words per entry), each with the entries it holds. + /// Appends `pages` as (offset, first entry, words per entry), requesting each page with the + /// entries it must hold. fn sample_pages(&mut self, pages: &[(FramePage, Range)], origin: u64) { for (page, held) in pages { let key = PageKey::of(page.id, Words::Values, false); let need = ((held.end - page.first) * u64::from(page.per)) as u32; - let offset = self.gpu.arena.get(key, need, || Content::Frames(page.clone())); - self.pins.push(key); - self.ready &= offset.is_some(); - self.words.extend([offset.unwrap_or(wgsl::NONE), (page.first - origin) as u32, page.per]); + self.push(key, need, || Content::Frames(page.clone())); + self.words.extend([(page.first - origin) as u32, page.per]); } } - /// Six words per row block: this frame's block and word, the next frame's, and their row counts. + /// A cursor per page of `rows` rows, at `frame` and the frame after it. fn cursors(&mut self, samples: &Samples, frame: Option, rows: usize) { - for block in 0..rows.div_ceil(PAGE_ROWS) { - let mut cursor = [wgsl::NONE, 0, wgsl::NONE, 0, 0, 0]; - for (i, f) in [frame, frame.map(|f| f + 1)].into_iter().enumerate() { + for block in 0..rows.div_ceil(wgsl::PAGE_ROWS as usize) { + let at = self.cursor_words(); + for (k, f) in [frame, frame.map(|f| f + 1)].into_iter().enumerate() { let Some((page, base)) = f.and_then(|f| samples.cursor(block, f)) else { continue }; let key = PageKey::of(page.id, Words::Values, false); - let rows_in_page = page.per; - let offset = self.gpu.arena.get(key, base + rows_in_page, || Content::Frames(page.clone())); - self.pins.push(key); - self.ready &= offset.is_some(); - cursor[2 * i] = offset.unwrap_or(wgsl::NONE); - cursor[2 * i + 1] = base; - cursor[4 + i] = rows_in_page; + self.cursor(at, k, key, base, page.per, || Content::Frames(page.clone())); } - self.words.extend(cursor); } } - fn push(&mut self, key: PageKey, offset: Option) { - self.pins.push(key); - self.ready &= offset.is_some(); - self.words.push(offset.unwrap_or(wgsl::NONE)); + /// Appends a cursor's six words, as `lib/read.wesl` reads them: the block and word of the + /// frame shown, the same for the next frame, and both frames' row counts. Each frame reads + /// nothing until [`cursor`](Builder::cursor) sets it. Returns where the words start. + fn cursor_words(&mut self) -> usize { + let at = self.words.len(); + self.words.extend([wgsl::NONE, 0, wgsl::NONE, 0, 0, 0]); + at + } + + /// Sets frame `k` (0 the frame shown, 1 the next) of the cursor at `at` to `rows` rows from + /// word `base` of page `key`, built by `content` when the arena lacks it. + fn cursor(&mut self, at: usize, k: usize, key: PageKey, base: u32, rows: u32, content: impl FnOnce() -> Content) { + self.place(at + 2 * k, key, base + rows, content); + self.words[at + 2 * k + 1] = base; + self.words[at + 4 + k] = rows; } } + +/// The cache key for the page in slot `s` of `drawing`, for a column packed as `packing`: a page of +/// the order by its number, or a page of kept rows by the places it holds. `None` for a free slot. +fn slot_key(packing: &Packing, drawing: &Drawing, s: usize) -> Option<(Packing, u32)> { + match drawing.holds(s) { + Holds::Free => None, + Holds::Page(page) => Some((packing.clone(), page)), + Holds::Kept(_) => Some((kept_packing(packing, drawing.kept_places(s)), 0)), + } +} + +/// The cache key for a page of kept rows at `places` of the order, in a column packed as `packing`. +fn kept_packing(packing: &Packing, places: &[u32]) -> Packing { + Packing { order: stamp_of((packing.order, places)), ..packing.clone() } +} + +/// The id of the page of `packing` in slot `s` of `drawing`, packed from `source`: a page of its +/// order, or of its kept rows, each cached until one of its rows is written. `None` if free. +fn slot_page( + packed: &mut crate::gpu::packed::Packed, + packing: &Packing, + drawing: &Drawing, + source: Option<&Column>, + s: usize, +) -> Option { + match drawing.holds(s) { + Holds::Free => None, + Holds::Page(page) => Some(packed.page(packing, &drawing.order, source, page)), + Holds::Kept(_) => { + let holds = |row: u32| drawing.held_row(row).is_some_and(|h| (h / wgsl::PAGE_ROWS) as usize == s); + Some(packed.kept(&kept_packing(packing, drawing.kept_places(s)), source, holds)) + } + } +} + +/// The points that rows `rows` of geometry column `column` take, as shaders read them. +fn rows_points(column: &Column, rows: &[u32]) -> usize { + rows.iter() + .filter_map(|&r| column.at(r as usize)) + .filter(|(values, i)| values.is_valid(*i)) + .map(|(values, i)| values.geometry().map_or(0, |g| g.row_points(i))) + .sum() +} diff --git a/crates/latkit/src/view/timer.rs b/crates/latkit/src/view/timer.rs index 5f2f3ffb..001179f7 100644 --- a/crates/latkit/src/view/timer.rs +++ b/crates/latkit/src/view/timer.rs @@ -1,15 +1,16 @@ -//! The GPU's time on frames, where the device writes timestamps: an empty pass begins a frame and -//! another ends it, and what lies between is the frame's. Each measure is resolved and read back -//! without stalling, a few at a time, and the time told is the median of the last frames measured, so -//! it holds steady while a frame now and then does more. +//! GPU time per frame, on devices that support timestamp queries. An empty pass writes a timestamp +//! at the start of a frame and another at its end. Each measurement is resolved and read back +//! without stalling, a few at a time. The reported time is the median of the last frames measured, +//! so it stays steady when an occasional frame does more work. use crate::gpu::readback::{Readback, Readbacks}; -/// Measures under way at most, and frames the time told is the median of. +/// The most measurements in flight. const SLOTS: usize = 4; +/// Frames the reported median covers. const KEPT: usize = 15; -/// A query set measuring one frame, and its readback while under way. +/// A query set measuring one frame, and its readback while in flight. struct Slot { set: wgpu::QuerySet, resolved: wgpu::Buffer, @@ -24,12 +25,12 @@ pub(crate) struct Timer { period: f64, /// Milliseconds of the last frames measured, oldest first. measured: Vec, - /// Its own map buffers, whose readbacks count as no work: only some devices have a timer. + /// Its own readback buffers, left out of the work counts, since only some devices have a timer. pool: Readbacks, } impl Timer { - /// A timer, where the device writes timestamps. + /// A timer, or `None` where the device lacks timestamp queries. pub(crate) fn new(device: &wgpu::Device, queue: &wgpu::Queue) -> Option { if !device.features().contains(wgpu::Features::TIMESTAMP_QUERY) { return None; @@ -57,7 +58,7 @@ impl Timer { }) } - /// Begins measuring the frame `encoder` records, while a slot is free. + /// Starts measuring the frame `encoder` records, if a slot is free. pub(crate) fn begin(&mut self, encoder: &mut wgpu::CommandEncoder) { self.current = self.slots.iter().position(|s| s.pending.is_none()); if let Some(i) = self.current { @@ -65,7 +66,7 @@ impl Timer { } } - /// Ends the measure this frame began, reading it back once submitted. + /// Ends this frame's measurement and records its readback, which completes after submission. pub(crate) fn end(&mut self, encoder: &mut wgpu::CommandEncoder, device: &wgpu::Device) { let Some(i) = self.current.take() else { return }; let slot = &mut self.slots[i]; @@ -75,7 +76,8 @@ impl Timer { slot.pending = Some(Readback::new(&self.pool, device, encoder, &slot.resolved, 0, 16, waker)); } - /// The median milliseconds of the last frames measured, once a measure landed since last asked. + /// The median milliseconds of the last frames measured, if a measurement arrived since the last + /// call. pub(crate) fn landed(&mut self) -> Option { let mut landed = false; for slot in &mut self.slots { @@ -98,12 +100,12 @@ impl Timer { /// Milliseconds between the two timestamps of `bytes`, ticks `period` nanoseconds long. fn elapsed_ms(bytes: &[u8], period: f64) -> Option { - let tick = |i: usize| Some(u64::from_le_bytes(bytes.get(i * 8..i * 8 + 8)?.try_into().ok()?)); - let ticks = tick(1)?.checked_sub(tick(0)?)?; - Some((ticks as f64 * period / 1e6) as f32) + let mut ticks = crate::wire::In::new(bytes); + let (start, end) = (ticks.u64()?, ticks.u64()?); + Some((end.checked_sub(start)? as f64 * period / 1e6) as f32) } -/// An empty pass writing timestamp `beginning` as it begins or `end` as it ends. +/// Records an empty pass that writes timestamp `beginning` at its start, or `end` at its end. fn mark(set: &wgpu::QuerySet, encoder: &mut wgpu::CommandEncoder, beginning: Option, end: Option) { encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { label: Some("latkit timer"), diff --git a/crates/latkit/src/view/transition.rs b/crates/latkit/src/view/transition.rs index 1e6df114..2aff47dc 100644 --- a/crates/latkit/src/view/transition.rs +++ b/crates/latkit/src/view/transition.rs @@ -1,29 +1,36 @@ -//! Transitions: a mark whose description, static data, or rows change eases from what it showed. +//! Transitions: a mark whose options, static data, or rows change eases from its previous state. //! -//! When one starts, what each row shows is frozen into the frozen buffer at the row's new place: -//! copied when the rows are the same, gathered by id when rows were added, removed, or reordered. -//! Rows that left are frozen past the new rows and fade out there. While the mark eases, its channels -//! store into the settled state, a second mirror of the state, once per change; each frame the clock -//! moves, the ease pass writes what every row shows from its frozen and settled state by the mark's -//! progress, and which rows fade, so draws, probes, bounds, and shadows read eased state without -//! knowing a transition runs, and no channel runs to ease. Where no row is the same as one shown -//! before, as when another dataset replaces the one drawn, every row enters, shows what it evaluated, -//! and only fades. The frozen and settled buffers mirror the state buffer's layout and exist only -//! while a transition does. A stroke's geometry column keeps the table it had as the transition -//! began, and its paths morph from it point by point. +//! When a transition starts, each row's shown state is frozen into the mark's frozen words at the +//! row's new index: from the same index when no row moved, or gathered by id when rows were added, +//! removed, or reordered. Rows that left are frozen after the new rows and fade out there. //! -//! Without an ease, a flowing mark whose rows moved still begins a transition, of no length: it -//! shows the new state at once, and its comets carry their travel over to the rows' new places. +//! While the mark eases, its channels write its settled words, once per change. Each frame the +//! clock moves, the ease pass writes every row's shown state from its frozen and settled words by +//! the mark's progress, and sets which rows fade. So draws, probes, and bounds read eased state +//! without knowing a transition runs, and no channel runs just to ease. When no row matches one +//! shown before, as when another dataset replaces the drawn one, every row enters with its +//! evaluated state and only fades in. +//! +//! The frozen and settled words are paged like the state, and kept only while a transition runs. A +//! stroke's geometry column keeps its table from the start of the transition, and its paths morph +//! from it point by point. +//! +//! Without an ease, a flowing mark whose rows moved still begins a zero-length transition: it shows +//! the new state at once, and its comets carry their travel over to the rows' new places. + +use std::sync::Arc; use super::View; use super::clock::Clock; -use super::marks::Shown; -use crate::data::{PAGE_ROWS, RowIds, Table}; +use super::drawing::{Drawing, Holds}; +use super::marks::{Holding, Shown}; +use super::words::Kept; +use crate::data::{RowIds, Table}; use crate::gpu::Inner; -use crate::gpu::shaders::{self as wgsl, Shader}; +use crate::gpu::shaders::{self as wgsl, PAGE_ROWS}; use crate::scene::mark::MarkKind; -/// A mark easing from what it showed, begun at a time on the view's clock. +/// A running transition: its start on the view's clock and its length in seconds. pub(crate) struct Transition { start: f64, seconds: f64, @@ -31,14 +38,15 @@ pub(crate) struct Transition { /// A transition waiting for the frame that freezes it. pub(crate) struct Begin { - /// Each new row's old row, then each exit's, when rows moved; `None` when they kept their places. + /// Each new row's old row, then each exit's old row, when rows moved. `None` when every row + /// kept its index. sources: Option>, - /// Whether no row was shown before, so rows only fade in and out. + /// Whether no new row matches one shown before, so rows only fade in and out. fades: bool, } impl Transition { - /// Eased progress on `clock`, `0..=1`; a transition of no length is done as it begins. + /// Eased progress on `clock`, from 0 to 1. A zero-length transition is at 1 from the start. pub(crate) fn motion(&self, clock: &Clock) -> f32 { if self.seconds <= 0.0 { 1.0 } else { clock.eased(self.start, self.seconds) as f32 } } @@ -49,19 +57,18 @@ impl Transition { } /// How rows moved from `old` to `new`: each new row's old row (`NONE` if it entered), and the old -/// rows that left. `None` when every row kept its place. +/// rows that left. `None` when every row kept its index. pub(crate) fn remap(old: &Table, new: &Table) -> Option<(Vec, Vec)> { if old.structure == new.structure && old.order == new.order { return None; } if old.order == new.order && new.len() >= old.len() { - // Rows kept in place, those of pages rewritten aside: there a row whose id changed enters, as - // one a table drawn through tiles writes where a tile drawn before was, and the row it - // replaces leaves. + // Rows keep their index, except in rewritten pages: there a row whose id changed enters, + // and the row it replaced leaves. let mut sources: Vec = (0..new.len()).map(|r| if r < old.len() { r as u32 } else { wgsl::NONE }).collect(); let mut exits = Vec::new(); for block in rewritten(old, new).unwrap_or_default() { - let rows = block as usize * PAGE_ROWS..((block as usize + 1) * PAGE_ROWS).min(old.len()); + let rows = (block * PAGE_ROWS) as usize..(((block + 1) * PAGE_ROWS) as usize).min(old.len()); for r in rows.filter(|&r| old.ids.key(r) != new.ids.key(r)) { sources[r] = wgsl::NONE; if old.ids.key(r).is_some() { @@ -94,15 +101,13 @@ pub(crate) fn remap(old: &Table, new: &Table) -> Option<(Vec, Vec)> { Some((sources, exits)) } -/// The row blocks of `new` whose rows `old` did not hold where `new` does, every other row kept in -/// place: those whose ids' pages changed, and those past `old`'s rows. `None` where that does not -/// tell, as between ids counted and ids listed. +/// The row blocks of `new` holding rows that `old` did not hold at the same index, assuming every +/// other row kept its index: blocks whose id pages changed, and blocks past `old`'s rows. `None` +/// when the ids cannot tell, as between counted and listed ids. pub(crate) fn rewritten(old: &Table, new: &Table) -> Option> { - let blocks = |len: usize| len.div_ceil(PAGE_ROWS) as u32; + let pages = |len: usize| (len as u32).div_ceil(PAGE_ROWS); match (&old.ids, &new.ids) { - (RowIds::Count(a), RowIds::Count(b)) if b >= a => { - Some(((*a as usize / PAGE_ROWS) as u32..blocks(*b as usize)).collect()) - } + (RowIds::Count(a), RowIds::Count(b)) if b >= a => Some((*a / PAGE_ROWS..pages(*b as usize)).collect()), (RowIds::Listed(a), RowIds::Listed(b)) if b.len() >= a.len() => { let (a, b) = (a.pages(), b.pages()); Some((0..b.len()).filter(|&p| a.get(p).is_none_or(|page| page.id != b[p].id)).map(|p| p as u32).collect()) @@ -111,26 +116,119 @@ pub(crate) fn rewritten(old: &Table, new: &Table) -> Option> { } } -/// The rows of `new` replacing every row of `old`, as when another dataset of the same name replaces -/// the one drawn: each enters, and each old row leaves, whatever its id. -pub(crate) fn replaced(old: &Table, new: &Table) -> (Vec, Vec) { - (vec![wgsl::NONE; new.len()], (0..old.len() as u32).collect()) +/// How held rows moved from `old` to `new`: each new held row's old held row (`NONE` if it +/// entered), and the old drawn rows that left. `None` when every row kept its held row, as in a +/// paged table whose order is unchanged, regardless of what the camera draws. +pub(crate) fn moved(old: &Holding, new: &Holding) -> Option<(Vec, Vec)> { + if old.origin != new.origin { + // Another dataset of the same name: every row enters, and every row drawn leaves. + return Some((vec![wgsl::NONE; held(new) as usize], drawn(old))); + } + match (&old.drawing, &new.drawing) { + (None, None) => remap(&old.table, &new.table), + (Some(a), Some(b)) if a.order.id() == b.order.id() && old.table.order == new.table.order => None, + _ => Some(follow(old, new)), + } +} + +/// Held rows: the table's rows, or its drawing's live held rows. +fn held(holding: &Holding) -> u32 { + holding.drawing.as_ref().map_or(holding.table.len() as u32, |d| d.live()) +} + +/// The held rows `holding` draws, except those fading out with their tiles. +fn drawn(holding: &Holding) -> Vec { + match &holding.drawing { + None => (0..holding.table.len() as u32).collect(), + Some(drawing) => drawing + .runs + .iter() + .filter(|run| run.fade != wgsl::FADE_OUT) + .flat_map(|run| run.held..run.held + run.len) + .collect(), + } +} + +/// How rows moved between holdings with different orders: each new held row is traced to its old +/// held row, by index while the table's row order is unchanged, otherwise by id. +fn follow(old: &Holding, new: &Holding) -> (Vec, Vec) { + let same = old.table.order == new.table.order; + let source = |h: u32| match &new.drawing { + Some(drawing) => drawing.source(h), + None => (h < new.table.len() as u32).then_some(h), + }; + let held_of = |r: usize| match &old.drawing { + Some(drawing) => drawing.held_row(r as u32), + None => (r < old.table.len()).then_some(r as u32), + }; + let mut kept = vec![false; held(old) as usize]; + let sources: Vec = (0..held(new)) + .map(|h| { + let Some(r) = source(h).map(|r| r as usize) else { return wgsl::NONE }; + let was = if same { + (r < old.table.len()).then_some(r) + } else { + new.table.ids.key(r).and_then(|key| old.table.row(key)) + }; + let Some(at) = was.and_then(held_of) else { return wgsl::NONE }; + if let Some(k) = kept.get_mut(at as usize) { + *k = true; + } + at + }) + .collect(); + let exits = drawn(old).into_iter().filter(|&h| !kept.get(h as usize).copied().unwrap_or(false)).collect(); + (sources, exits) +} + +/// Whether every new row kept its index or entered. +pub(crate) fn in_place(sources: &[u32]) -> bool { + sources.iter().enumerate().all(|(r, &s)| s == r as u32 || s == wgsl::NONE) +} + +/// The slots of `new` holding rows that `old` did not hold there: slots whose page is new to them, +/// slots whose page gained appended rows, and slots of kept rows that changed. +pub(crate) fn bound(old: Option<&Drawing>, new: &Drawing) -> Vec { + let grown = old.map_or(u32::MAX, |old| old.order.len() / PAGE_ROWS); + let appended = old.is_some_and(|o| o.order.len() != new.order.len()); + (0..new.slot_count()) + .filter(|&s| match new.holds(s) { + Holds::Free => false, + held => { + let alike = old.is_some_and(|o| o.holds_alike(new, s)); + !alike || matches!(held, Holds::Page(page) if appended && page >= grown) + } + }) + .map(|s| s as u32) + .collect() +} + +/// Whether two holdings hold the same rows in the same layout: the same origin, row structure, row +/// order, and drawing. +pub(crate) fn holds_alike(a: &Holding, b: &Holding) -> bool { + let same_drawing = match (&a.drawing, &b.drawing) { + (None, None) => true, + (Some(x), Some(y)) => Arc::ptr_eq(x, y), + _ => false, + }; + a.origin == b.origin && a.table.structure == b.table.structure && a.table.order == b.table.order && same_drawing } impl Shown { - /// Takes over the rows `before` holds, as a changed description of the same mark. - pub(crate) fn inherit(&mut self, before: &Shown) { - self.region = before.region; - self.held = before.held; + /// Takes over `before`'s held rows and their words, as a changed version of the same mark. + pub(crate) fn inherit(&mut self, before: &mut Shown) { + self.words = std::mem::take(&mut before.words); + self.before = before.before; self.rows = before.rows; - self.drawn.clone_from(&before.drawn); + self.holding.clone_from(&before.holding); self.version = before.version + 1; self.place_version = before.place_version + 1; } - /// Starts easing from what the mark shows: its rows moved as `moved` says, if they did. + /// Begins a transition from the mark's current state, with rows moved as `moved` says, if they + /// moved. Does nothing when the mark showed no rows. pub(crate) fn begin(&mut self, moved: Option<(Vec, Vec)>) { - if self.held == 0 { + if self.before == 0 { return; } let fades = moved.as_ref().is_some_and(|(sources, _)| sources.iter().all(|&s| s == wgsl::NONE)); @@ -145,17 +243,31 @@ impl Shown { self.begin = Some(Begin { sources, fades }); self.all_dirty(); } + + /// Restarts the running transition from the mark's current state, as slots `bound` receive + /// pages of entering rows. Other rows keep easing toward their settled state, which their + /// channels already evaluated, and exits keep fading out after the live rows. + pub(crate) fn restart(&mut self, bound: &[u32]) { + let mut sources: Vec = (0..self.live()).collect(); + for &slot in bound { + let rows = slot * PAGE_ROWS..((slot + 1) * PAGE_ROWS).min(sources.len() as u32); + sources[rows.start as usize..rows.end as usize].fill(wgsl::NONE); + } + // Exits were held after the live rows when the frame began. + sources.extend(self.before.saturating_sub(self.exits)..self.before); + self.begin = Some(Begin { sources: Some(sources), fades: false }); + } } impl View { - /// Freezes every transition starting this frame and advances the running ones; true while any - /// runs. - pub(super) fn transitions(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner) -> bool { + /// Starts every transition beginning this frame, which this frame's compute pass freezes, and + /// advances the running ones. + pub(super) fn transitions(&mut self, inner: &mut Inner) { // A drag moves rows at once. let seconds = if self.dragging() { 0.0 } else { self.ease().as_secs_f64() }; let mut starting: Vec = (0..self.marks.len()).filter(|&i| self.marks[i].begin.is_some()).collect(); if seconds == 0.0 { - // Without an ease only a flowing mark whose rows moved begins one, for its comets. + // Without an ease, only a flowing mark whose rows moved begins one, for its comets. starting.retain(|&i| { let shown = &mut self.marks[i]; let carries = @@ -168,131 +280,130 @@ impl View { }); } if !starting.is_empty() { - self.freeze(encoder, inner, &starting); + self.write_remap(&starting); let start = self.clock.now(); for &i in &starting { self.marks[i].transition = Some(Transition { start, seconds }); } } - let mut running = false; for (i, shown) in self.marks.iter_mut().enumerate() { let Some(transition) = &shown.transition else { continue }; - // One that begins this frame runs through it, as one of no length must for its comets. + // A transition beginning this frame runs through it, as a zero-length one must for its + // comets. if transition.done(&self.clock) && !starting.contains(&i) { - // One more evaluation writes every row's settled state. + // One more evaluation writes every row's final state. shown.transition = None; shown.exits = 0; shown.all_dirty(); + shown.packed = false; if shown.tables.frozen().is_some() { shown.tables.thaw(inner); - shown.packed = false; } } else { - running = true; - // Rows ease as the clock moves, and only then: their channels evaluated once. + // Rows ease only when the clock moves. Their channels evaluated once already. shown.easing = self.clock.moved(); } } - if !running && self.marks.iter().all(|m| m.transition.is_none()) { - let resources = self.resources.as_mut().expect("created"); - resources.drop_frozen(); - resources.drop_settled(); - } - running } - /// Records what each starting mark shows into the frozen buffer. - fn freeze(&mut self, encoder: &mut wgpu::CommandEncoder, inner: &mut Inner, starting: &[usize]) { + /// Writes the remap words from which the compute pass freezes each starting mark's shown state + /// into its frozen words, per mark: its state pages as the frame began, how many rows showed, + /// each new row's old row when rows moved, and where the vertices its endpoints reference + /// moved, if they did. + fn write_remap(&mut self, starting: &[usize]) { let gpu = self.gpu.clone(); let (device, queue) = (&gpu.0.device, &gpu.0.queue); - let slots = self.slot_table(starting); - let resources = self.resources.as_mut().expect("created before transitions"); - resources.ensure_frozen(device); - resources.ensure_settled(device); let mut words: Vec = Vec::new(); - let table = slots.map(|table| { - words.extend(table); - 0 - }); - let mut gathers = Vec::new(); + // For each starting vertex mark whose rows moved: how many it held, then each one's new + // row. + let mut moves = vec![wgsl::NONE; self.marks.len()]; + for &i in starting { + let m = &self.marks[i]; + let Some(sources) = m.begin.as_ref().and_then(|b| b.sources.as_ref()) else { continue }; + if m.kind() != MarkKind::Vertices { + continue; + } + moves[i] = words.len() as u32; + let at = words.len() + 1; + words.push(m.before); + words.resize(at + m.before as usize, wgsl::NONE); + for (new, &old) in sources.iter().enumerate() { + if old < m.before { + words[at + old as usize] = new as u32; + } + } + } for &i in starting { let shown = &mut self.marks[i]; let begin = shown.begin.take().expect("starting"); - let was = shown.moved.unwrap_or(shown.region); - let now = shown.region; - match begin.sources { - None => { - let source = if shown.moved.is_some() { resources.retired_state() } else { None }; - let rows = shown.held.min(shown.rows); - resources.copy_frozen(encoder, source, was, now, rows); - } + let before = shown.words.state_before().clone(); + let width = shown.width(); + // If the row width changed, there is nothing to ease from: every row enters, fading in. + let sources = + if before.width == width { begin.sources } else { Some(vec![wgsl::NONE; shown.held() as usize]) }; + shown.carry = sources.is_some(); + let header = words.len(); + words.extend([0; 4]); + words[header] = words.len() as u32; + words.extend(before.blocks.iter()); + words[header + 1] = shown.before.min(shown.live()); + words[header + 2] = match sources { Some(sources) => { let at = words.len() as u32; - let slot_words = match shown.kind() { - MarkKind::Edges => (1 << wgsl::EDGE_FROM) | (1 << wgsl::EDGE_TO), - MarkKind::Faces => (1 << wgsl::FACE_A) | (1 << wgsl::FACE_B) | (1 << wgsl::FACE_C), - _ => 0, - }; - words.extend([was.state, was.extra, slot_words, table.unwrap_or(wgsl::NONE)]); words.extend(sources); - shown.remap = at; - shown.carry = true; - gathers.push(i); + at } - } - } - if gathers.is_empty() { - return; + None => wgsl::NONE, + }; + // Each endpoint, stored in the kind word of its number, follows its vertex where the + // vertex mark's rows moved. + let ends: Vec<(u32, u32)> = (0..shown.ends.len()) + .filter_map(|k| Some((k as u32, moves[shown.ends[k].mark?]))) + .filter(|&(_, moved)| moved != wgsl::NONE) + .collect(); + words[header + 3] = if ends.is_empty() { + wgsl::NONE + } else { + let at = words.len(); + words.resize(at + shown.stride() as usize, wgsl::NONE); + for (word, moved) in ends { + words[at + word as usize] = moved; + } + at as u32 + }; + shown.remap = header as u32; } + let resources = self.resources.as_mut().expect("created before transitions"); resources.write_remap(device, queue, &words); - let moved = gathers.iter().any(|&i| self.marks[i].moved.is_some()); - let groups = resources.groups(&gpu.0, inner); - let group = resources.freeze_group(device, inner, moved); - let pipeline = inner.pipelines.compute(device, Shader::Freeze, "gather"); - // The marks' uniforms this frame writes before it submits hold each mark's remap offset. - let mut pass = encoder - .begin_compute_pass(&wgpu::ComputePassDescriptor { label: Some("latkit freeze"), timestamp_writes: None }); - pass.set_pipeline(&pipeline); - pass.set_bind_group(0, &groups.view, &[0]); - pass.set_bind_group(2, &group, &[]); - for &i in &gathers { - let shown = &self.marks[i]; - pass.set_bind_group(1, &groups.mark, &[super::passes::offset(i)]); - super::passes::dispatch(&mut pass, shown.rows + shown.exits); - } } - /// Where each old vertex slot moved, for the ends of rows easing out; `None` when no mark with - /// ends gathers. - fn slot_table(&self, starting: &[usize]) -> Option> { - let needed = starting.iter().any(|&i| { - let m = &self.marks[i]; - m.kind() != MarkKind::Vertices && m.begin.as_ref().is_some_and(|b| b.sources.is_some()) - }); - if !needed { - return None; - } - let vertices = self.marks.iter().filter(|m| m.kind() == MarkKind::Vertices); - let old_slot = |m: &Shown| m.moved.unwrap_or(m.region).state / wgsl::VERTEX_WORDS; - let size = vertices.clone().map(|m| old_slot(m) + m.held).max().unwrap_or(0); - let mut table = vec![wgsl::NONE; size as usize]; - for m in vertices { - let (from, to) = (old_slot(m), m.region.state / wgsl::VERTEX_WORDS); - match m.begin.as_ref().and_then(|b| b.sources.as_ref()) { - Some(sources) => { - for (new_row, &old_row) in sources.iter().enumerate() { - if old_row != wgsl::NONE && old_row < m.held { - table[(from + old_row) as usize] = to + new_row as u32; - } - } + /// Allocates frozen and settled words for every held row of each mark whose transition starts + /// or runs, and frees them once it ends. A transition whose words do not fit in the arena is + /// cancelled, and its rows show their evaluated state at once. + pub(super) fn fit_easing(&mut self, inner: &mut Inner) { + let gpu = self.gpu.clone(); + let device = (&gpu.0.device, &gpu.0.queue); + for shown in &mut self.marks { + let easing = shown.begin.is_some() || shown.transition.is_some(); + let rows = shown.page_rows(); + let width = if easing { shown.width() } else { 0 }; + let mut fits = true; + for kept in [Kept::Frozen, Kept::Settled] { + match shown.words.fit(kept, &rows, width, device, &mut inner.arena) { + Some(fitted) => shown.packed &= !fitted.changed, + None => fits = false, } - None => { - for row in 0..m.held.min(m.rows) { - table[(from + row) as usize] = to + row; - } + } + if easing && !fits { + for kept in [Kept::Frozen, Kept::Settled] { + shown.words.fit(kept, &[], 0, device, &mut inner.arena); } + shown.begin = None; + shown.transition = None; + shown.exits = 0; + shown.all_dirty(); + shown.packed = false; } } - Some(table) } } diff --git a/crates/latkit/src/view/uniforms.rs b/crates/latkit/src/view/uniforms.rs index 6b15e7f4..e6b3f70f 100644 --- a/crates/latkit/src/view/uniforms.rs +++ b/crates/latkit/src/view/uniforms.rs @@ -1,13 +1,13 @@ -//! What a frame tells its shaders: the frame uniform, the eyes its passes draw through, and each -//! mark's uniform. +//! The uniforms a frame writes for its shaders: the frame's, the eye's, and each mark's. -use glam::{DMat4, DVec3, DVec4}; +use glam::{DMat4, DVec3}; use super::camera::{Projected, split}; use super::marks::Shown; +use super::words::Kept; use crate::gpu::shaders as wgsl; use crate::scene::mark::MarkKind; -use crate::{Color, Grid, Scene, Space, Theme}; +use crate::{Color, Grid, Length, Scene, Space, Theme}; fn matrix(m: DMat4) -> [[f32; 4]; 4] { m.to_cols_array_2d().map(|c| c.map(|v| v as f32)) @@ -17,10 +17,11 @@ fn vector(v: DVec3) -> [f32; 4] { [v.x as f32, v.y as f32, v.z as f32, 0.0] } -/// The frame uniform for `scene` seen through `projected` in a target of `size` CSS pixels, showing -/// axis coordinate `at`, `elapsed` seconds of motion into the view's clock. Pointer, probe, focus, -/// and shade words start empty. In a diagram every length but glow and halo is a unit, so the -/// theme's lengths scale with the zoom. +/// The frame uniform for `scene` seen through `projected` in a target of `size` CSS pixels, at axis +/// coordinate `at` and `elapsed` seconds on the view's clock. `linear` says the target is sRGB and +/// encodes color itself, so shaders write linear color. Pointer, probe, and focus words start +/// empty. Theme lengths are in the space's unit, so in a diagram they scale with the zoom. The halo +/// and the focus glow are in CSS pixels everywhere. pub(crate) fn frame( scene: &Scene, projected: &Projected, @@ -42,20 +43,17 @@ pub(crate) fn frame( Grid::Dots { pitch, .. } => (wgsl::GRID_DOTS, pitch), }; let unit = if projected.space == wgsl::SPACE_DIAGRAM { projected.px_per_unit[0] as f32 } else { 1.0 }; - let (surface, graticule) = match &scene.space { - Space::Earth(earth) => (earth.surface.unwrap_or(theme.surface), earth.graticule), - _ => (theme.surface, false), - }; - // The sun stands where it does at the earth's daylight instant, on by the axis coordinate shown. + let graticule = matches!(&scene.space, Space::Earth(earth) if earth.graticule); + // The sun's position at the earth's daylight instant, advanced by the shown axis coordinate. let (sun, atmosphere) = match &scene.space { Space::Earth(earth) => ( earth.daylight.map(|t| crate::scene::sun(t + if at.is_finite() { at } else { 0.0 })), - earth.atmosphere.then(|| theme.atmosphere.to_array()), + earth.atmosphere.then(|| theme.earth.atmosphere.to_array()), ), _ => (None, None), }; - // A selected color with a negative red asks for each row's own color. - let selected = theme.selected.map_or([-1.0, 0.0, 0.0, 0.9], Color::to_array); + // A negative red in the selected color means each row keeps its own color. + let selected = theme.focus.selected.map_or([-1.0, 0.0, 0.0, 0.9], Color::to_array); let mut frame: wgsl::Frame = bytemuck::Zeroable::zeroed(); frame.light = vector(projected.light); frame.center_hi = [cx.0, cy.0]; @@ -66,44 +64,48 @@ pub(crate) fn frame( frame.globe = projected.globe as f32; frame.mercator_y = projected.mercator_y as f32; frame.elapsed = elapsed as f32; - frame.glow = theme.glow; - frame.dim = theme.dim; + frame.glow = theme.focus.glow; + frame.dim = theme.focus.dim; frame.dash = theme.dash * unit; - // Comets travel in their own lengths, which a stroke turns into pixels where it draws them. - frame.flow_spacing = theme.flow.spacing; - frame.flow_length = theme.flow.length; + // Comet spacing and length keep their own unit. A stroke converts them to pixels as it draws. + frame.flow_spacing = theme.flow.spacing.value(); + frame.flow_length = theme.flow.length.value(); frame.flow_width = theme.flow.width * unit; - frame.flow_units = u32::from(theme.flow.units); - frame.lane_bow = theme.lane_bow; - // Hover is state either way; the controls say only whether it glows. + frame.flow_unit = match theme.flow.spacing { + Length::Space(_) => wgsl::FLOW_LENGTHS, + Length::Units(_) => wgsl::FLOW_UNITS, + Length::Px(_) => wgsl::FLOW_PX, + }; + frame.bow = theme.edge.bow; + // Hover is tracked either way. The controls only decide whether it glows. let glows = scene.controls.hover; - frame.hover_scale = if glows { theme.hover_scale } else { 1.0 }; + frame.grow = if glows { theme.focus.grow } else { 1.0 }; frame.shading = theme.shading; - frame.arrow = theme.arrow * unit; - frame.halo = theme.halo; - frame.group_padding = theme.group_padding * unit; + frame.arrow = theme.edge.arrow * unit; + frame.halo = theme.text.halo; + frame.group_padding = theme.diagram.group_padding * unit; frame.unit = unit; frame.zoom = projected.zoom as f32; frame.linear = u32::from(linear); - frame.hover = if glows { theme.hover.to_array() } else { [0.0; 4] }; + frame.hover = if glows { theme.focus.hover.to_array() } else { [0.0; 4] }; frame.selected = selected; frame.flow = theme.flow.color.to_array(); frame.background = theme.background.to_array(); - frame.surface = surface.to_array(); + frame.ground = theme.earth.ground.to_array(); frame.grid_color = theme.grid.to_array(); frame.grid_kind = grid_kind; frame.grid_pitch = grid_pitch as f32; frame.grid_fade = grid_fade(scene, projected, size) as f32; frame.graticule = u32::from(graticule); - frame.sun = sun.map_or([0.0; 4], |s| [s[0] as f32, s[1] as f32, s[2] as f32, theme.night.clamp(0.0, 1.0)]); + frame.sun = sun.map_or([0.0; 4], |s| [s[0] as f32, s[1] as f32, s[2] as f32, theme.earth.night.clamp(0.0, 1.0)]); frame.atmosphere = atmosphere.unwrap_or([0.0; 4]); frame.theme = theme_slots(theme); - frame.shade.color = theme.shadow.color.to_array(); + frame.shadow = theme.diagram.shadow.color.to_array(); frame } -/// The world distance from the center a plane's grid fades out by: what the view shows looking -/// straight down, and a little more. Shadows cover the same ground. +/// The world distance from the center at which a plane's grid has faded out: a bit more than the +/// view shows looking straight down. 0 on a plot or the globe. pub(crate) fn grid_fade(scene: &Scene, projected: &Projected, size: [f64; 2]) -> f64 { match &scene.space { Space::Plot(_) => 0.0, @@ -112,7 +114,7 @@ pub(crate) fn grid_fade(scene: &Scene, projected: &Projected, size: [f64; 2]) -> } } -/// The eye of the view: `projected` into a target of `size` CSS pixels at `pixel_ratio`. +/// The eye uniform: `projected` into a target of `size` CSS pixels at `pixel_ratio`. pub(crate) fn eye(projected: &Projected, size: [f64; 2], pixel_ratio: f32) -> wgsl::Eye { let mut eye: wgsl::Eye = bytemuck::Zeroable::zeroed(); eye.clip_from_world = matrix(projected.clip_from_world); @@ -124,95 +126,68 @@ pub(crate) fn eye(projected: &Projected, size: [f64; 2], pixel_ratio: f32) -> wg eye } -/// The light's eye, looking down along `light` onto the square of ground `rect` (least x, y, greatest -/// x, y, world units), and the casters' lift in world units. Its CSS pixels are the view's: a -/// marker's quad keeps its size across the map. -pub(crate) fn light_eye(view: &wgsl::Eye, light: DVec3, rect: [f64; 4], lift: f64, texels: u32) -> wgsl::Eye { - let [x0, y0, x1, y1] = rect; - let (sx, sy) = (2.0 / (x1 - x0), 2.0 / (y1 - y0)); - let tilt = -light.truncate() / light.z.max(1e-3); - // Ground point along the light, mapped to clip space; z lifted first. - let cast = DMat4::from_cols( - DVec4::new(sx, 0.0, 0.0, 0.0), - DVec4::new(0.0, sy, 0.0, 0.0), - DVec4::new(tilt.x * sx, tilt.y * sy, 0.0, 0.0), - DVec4::new(-1.0 - x0 * sx, -1.0 - y0 * sy, 0.5, 1.0), - ) * DMat4::from_translation(DVec3::new(0.0, 0.0, lift)); - let texels_per_unit = f64::from(texels) / (x1 - x0); - let px = texels_per_unit / f64::from(view.px_per_unit[0]).max(1e-30); - let mut eye = *view; - eye.clip_from_world = matrix(cast); - eye.world_from_clip = matrix(DMat4::IDENTITY); - eye.size = [(f64::from(texels) / px) as f32; 2]; - eye.pixel_ratio = px as f32; - eye.lift = lift as f32; - eye -} - /// The theme's values in slot order: colors packed, sizes as `f32` bits. fn theme_slots(theme: &Theme) -> [[u32; 4]; 4] { let mut slots = [0_u32; wgsl::THEME_SLOTS as usize]; - let edge = theme.edge.unwrap_or(theme.vertex); - slots[wgsl::THEME_VERTEX as usize] = theme.vertex.packed(); - slots[wgsl::THEME_VERTEX_RADIUS as usize] = theme.vertex_radius.to_bits(); + let edge = theme.edge.color.unwrap_or(theme.vertex.color); + slots[wgsl::THEME_VERTEX as usize] = theme.vertex.color.packed(); + slots[wgsl::THEME_VERTEX_RADIUS as usize] = theme.vertex.radius.to_bits(); slots[wgsl::THEME_EDGE as usize] = edge.packed(); - slots[wgsl::THEME_EDGE_WIDTH as usize] = theme.edge_width.to_bits(); - slots[wgsl::THEME_LINE as usize] = theme.line.packed(); - slots[wgsl::THEME_LINE_WIDTH as usize] = theme.line_width.to_bits(); - slots[wgsl::THEME_AREA as usize] = theme.area.packed(); - slots[wgsl::THEME_TEXT as usize] = theme.text.packed(); + slots[wgsl::THEME_EDGE_WIDTH as usize] = theme.edge.width.to_bits(); + slots[wgsl::THEME_LINE as usize] = theme.line.color.packed(); + slots[wgsl::THEME_LINE_WIDTH as usize] = theme.line.width.to_bits(); + slots[wgsl::THEME_AREA as usize] = theme.area.color.packed(); + slots[wgsl::THEME_TEXT as usize] = theme.text.color.packed(); slots[wgsl::THEME_MISSING as usize] = theme.missing.packed(); slots[wgsl::THEME_CLEAR as usize] = 0; - slots[wgsl::THEME_SURFACE as usize] = theme.surface.packed(); - slots[wgsl::THEME_OUTLINE as usize] = theme.outline.map_or(0, Color::packed); - slots[wgsl::THEME_CORNER as usize] = theme.corner.to_bits(); - slots[wgsl::THEME_PORT_SIZE as usize] = theme.port_size.to_bits(); - slots[wgsl::THEME_LIFT as usize] = theme.shadow.lift.to_bits(); + slots[wgsl::THEME_GROUP as usize] = theme.diagram.group.packed(); + slots[wgsl::THEME_OUTLINE as usize] = theme.diagram.outline.map_or(0, Color::packed); + slots[wgsl::THEME_CORNER as usize] = theme.diagram.corner.to_bits(); + slots[wgsl::THEME_PORT_SIZE as usize] = theme.diagram.port_size.to_bits(); + slots[wgsl::THEME_LIFT as usize] = theme.diagram.shadow.lift.to_bits(); std::array::from_fn(|i| std::array::from_fn(|j| slots[i * 4 + j])) } -/// What `mark` evaluates and how it eases by `clock`: the row blocks this frame's evaluation covers, -/// and while it eases, where its frozen rows start, its progress, and how each word eases. +/// Sets `mark`'s evaluation and easing fields: where the row-block lists of this frame's evaluation +/// and end pass start, how each word blends, and while it eases, the page tables of its frozen and +/// settled words and its progress on `clock`. fn ease(mark: &mut wgsl::Mark, shown: &Shown, clock: &super::clock::Clock) { - let region = &shown.region; mark.block = shown.block; - mark.blocks = shown.block_count; + mark.end_block = shown.end_block; mark.blends = shown.offsets.blends; mark.frozen = wgsl::NONE; - mark.frozen_extra = wgsl::NONE; + mark.settled = wgsl::NONE; mark.motion = 1.0; if let Some(transition) = &shown.transition { - mark.frozen = region.state; - mark.frozen_extra = region.extra; + mark.frozen = shown.offsets.table(Kept::Frozen); + mark.settled = shown.offsets.table(Kept::Settled); mark.motion = transition.motion(clock); } } -/// The uniform of mark `index`, easing by `clock`. +/// The uniform of mark `index`, with its label slots from `labels` and its easing progress from +/// `clock`. pub(crate) fn mark( index: usize, shown: &Shown, - labels: Option, + labels: &super::labels::Labels, clock: &super::clock::Clock, ) -> wgsl::Mark { let mut mark: wgsl::Mark = bytemuck::Zeroable::zeroed(); - let region = &shown.region; mark.index = index as u32; - mark.kind = match shown.kind() { - MarkKind::Vertices => wgsl::MARK_VERTICES, - MarkKind::Edges => wgsl::MARK_EDGES, - MarkKind::Faces => wgsl::MARK_FACES, - MarkKind::Lines => wgsl::MARK_LINES, - MarkKind::Areas => wgsl::MARK_AREAS, - }; + mark.kind = shown.kind().code(); mark.rows = shown.rows; mark.exits = shown.exits; - mark.state = region.state; + mark.held = shown.held(); + mark.row_runs = shown.offsets.runs; + mark.drawn = shown.drawn_rows.at.map_or(wgsl::NONE, |[start, _]| start); + mark.reveal = shown.reveal; + mark.state = shown.offsets.table(Kept::State); mark.stride = shown.stride(); - mark.extra = region.extra; mark.extras = shown.extras(); ease(&mut mark, shown, clock); - mark.focus = region.focus; + mark.focus = shown.offsets.table(Kept::Focus); + mark.normals = shown.offsets.table(Kept::Normals); mark.domains = shown.domains; mark.columns = shown.offsets.columns; mark.ranges = shown.offsets.ranges; @@ -236,20 +211,15 @@ pub(crate) fn mark( mark.lane = wgsl::NONE; mark.label = wgsl::NONE; [mark.size_min, mark.size_max] = crate::Size::ANY; - mark.label_max = wgsl::NONE; - mark.label_select = wgsl::NONE; - if let Some(slots) = labels { - mark.label_select = slots.select; - mark.candidates = slots.candidates; + if let Some(slots) = labels.slots(index) { mark.runs = slots.runs; mark.wanted = slots.wanted; mark.seen = slots.seen; - mark.label_id = slots.id; } if let Some(compiled) = &shown.compiled { let program = &compiled.program; let drawing = &compiled.drawing; - mark.primitive = drawing.primitive.code(); + mark.primitive = shown.kind().primitive().code(); mark.channels = program.channels.len() as u32; mark.reductions = program.domains.len() as u32; mark.units = program.units; @@ -272,7 +242,6 @@ pub(crate) fn mark( if let Some(labeling) = &drawing.label { mark.label = labeling.extra; mark.label_size = labeling.size; - mark.label_max = labeling.max.unwrap_or(wgsl::NONE); } if shown.kind() == MarkKind::Lines { mark.per_row = shown.segments_per_row(); diff --git a/crates/latkit/src/view/variants.rs b/crates/latkit/src/view/variants.rs index 1fc777e4..1a017f6e 100644 --- a/crates/latkit/src/view/variants.rs +++ b/crates/latkit/src/view/variants.rs @@ -1,31 +1,147 @@ -//! The pipelines a scene uses, asked for as it is shown, each at its urgency: what its frames draw -//! with (for each kind of mark, its draw and the compute passes it needs, a stroke mark's for the -//! path it runs along, shadows when something casts, and each mark's channels for what they read); -//! then their text (labels, glyphs, the fields text is set in, the overlay); then what changes need -//! (the passes transitions ease with, comets carried on), what lights rows (the lit rows' draws) and, -//! for frames that take input, what the hover and the intents need (each kind of mark's probe -//! program, which answers the hover, picks, boxes, and where a row lies). The GPU compiles them -//! through its queue, the most urgent first ([`crate::gpu::pipelines`]): the view draws once what it -//! draws with landed, adds text once that did, and lights rows, probes, and answers queries once the -//! rest did, so no frame waits on a compile and nothing compiles where it is first used; what the -//! scene never uses is never asked for. A picture takes no input: it draws once all it draws, writes, -//! changes, and lights rows with landed. - -use super::{SHADE, View}; +//! The pipelines a scene uses, requested when the scene is shown, in three urgency tiers. +//! +//! Draw: for each kind of mark, its draw pipeline and the compute passes it needs (for a stroke +//! mark, those for the path it follows), and each mark's channel pipeline for what its channels +//! read. Text: labels, glyphs, the distance fields text is set in, and the overlay. Answer: the +//! passes that transitions and comet carry use, the lit rows' draws, and for frames that take +//! input, each kind of mark's probe program, which answers hover, picks, box selection, and row +//! location. +//! +//! The GPU compiles them through its queue, most urgent first ([`crate::gpu::pipelines`]). The view +//! draws once its draw pipelines are ready, adds text once those are, and lights rows, probes, and +//! answers queries once the rest are. So no frame waits on a compile, and nothing compiles at first +//! use. Pipelines the scene never uses are never requested. A picture takes no input. It requests +//! everything it draws with, text, transitions, and lit rows at draw urgency, and draws once all of +//! them are ready. +//! +//! Each pipeline is named once, here: the passes shared by every mark as constants, and each mark's +//! own through [`MarkVariants`]. The view requests and runs them by these same names, so it runs +//! only what it requested. + +use std::sync::Arc; + +use super::View; +use super::marks::Shown; use crate::gpu::Inner; +use crate::gpu::atlas::FIELDS; use crate::gpu::pipelines::{self, Asked, Urgency, Variant}; use crate::gpu::shaders::{PATH_ENDS, PATH_GEOMETRY, Shader}; +use crate::scene::mark::MarkKind; use crate::{Mark, Marker, Scene, Space}; -/// The passes of the select capping a mark's labels. -const SELECT: [&str; 4] = ["select_start", "select_count", "select_find", "select_cut"]; +/// Writes the word of each drawn row of a paged mark from its runs. +pub(super) const EXPAND_RUNS: Variant = Variant::compute(Shader::Lists, "expand_runs"); + +/// The scan of a segment list's counts into each row's first instance. +pub(super) const SCAN: [Variant; 3] = [ + Variant::compute(Shader::Lists, "scan_blocks"), + Variant::compute(Shader::Lists, "scan_sums"), + Variant::compute(Shader::Lists, "add_sums"), +]; + +/// Eases a mark in transition from its frozen words toward its settled ones. +pub(super) const EASE: Variant = Variant::compute(Shader::Evaluate, "ease"); + +/// Moves comets along their rows at this frame's flow. +pub(super) const TRAVEL: Variant = Variant::compute(Shader::Evaluate, "travel"); + +/// Carries comets' travel over to the rows a transition moved. +pub(super) const CARRY: Variant = Variant::compute(Shader::Evaluate, "carry"); + +/// Freezes the words a transition eases from. +pub(super) const GATHER: Variant = Variant::compute(Shader::Freeze, "gather"); -/// The passes placing each labeled mark's labels once its candidates are measured; a diagram's labels -/// need no placement. -pub(super) const PLACING: [&str; 5] = ["claim", "decide", "occupy", "ask", "fade"]; +/// Sums each face's normal into its corners. +pub(super) const NORMALS: Variant = Variant::compute(Shader::Normals, "accumulate"); -/// What a view asked the GPU to compile for frames into one target, taking input or not: what they -/// draw with, their text, and what changes, lights rows, probes, and answers queries. +/// What a mark's pipelines are specialized for: its kind, the `PATH_` code its rows follow, and a +/// custom marker's WGSL. +#[derive(Clone, Copy)] +pub(super) struct MarkVariants<'a> { + kind: MarkKind, + path: u32, + custom: Option<&'a Arc>, +} + +impl<'a> MarkVariants<'a> { + /// The variants `mark` uses in `space`, requested before it compiles. + fn of(mark: &'a Mark, space: &Space) -> MarkVariants<'a> { + let custom = match mark { + Mark::Vertices(vertices) => match &vertices.marker { + Marker::Wgsl { source, .. } => Some(source), + _ => None, + }, + _ => None, + }; + MarkVariants { kind: mark.kind(), path: mark.path(space), custom } + } + + /// The variants `shown` runs. + pub(super) fn shown(shown: &'a Shown) -> MarkVariants<'a> { + let custom = shown.compiled.as_ref().and_then(|c| c.drawing.markers.custom.as_ref()); + MarkVariants { kind: shown.kind(), path: shown.path(), custom } + } + + fn primitive(self) -> u32 { + self.kind.primitive().code() + } + + /// The shader drawing its rows, or with `lit`, the rows its focus lights. `None` for a face's + /// lit rows. + pub(super) fn shader(self, lit: bool) -> Option { + let (every, focus) = self.kind.shaders(self.custom.is_some()); + if lit { focus } else { Some(every) } + } + + /// The pipeline drawing its rows, or with `lit`, the rows its focus lights. A custom marker's + /// WGSL compiles together with its shader's. + pub(super) fn draw(self, lit: bool, format: wgpu::TextureFormat, samples: u32) -> Option { + let variant = Variant::draw(self.shader(lit)?, format, samples).path(self.path); + Some(match self.custom { + Some(source) => Asked::Custom(source.clone(), variant), + None => Asked::Shader(variant), + }) + } + + /// The pass measuring its rows' extents for a fit. `None` for edges and faces. + pub(super) fn bounds(self) -> Option { + Some(Variant::compute(Shader::Bounds, self.kind.bounds()?).path(self.path)) + } + + /// Whether a list may number its instances, for rows whose counts differ: edges between ends + /// (which may pass through bends, follow great circles, or bow into lanes), and every path + /// through geometry, including areas' triangles. + fn listed(self) -> bool { + self.path == PATH_GEOMETRY || (self.kind == MarkKind::Edges && self.path == PATH_ENDS) + } + + /// The passes counting each row's instances into its list, then writing them, around a + /// [`SCAN`]. + pub(super) fn segments(self) -> [Variant; 2] { + ["count_segments", "write_segments"].map(|entry| Variant::compute(Shader::Lists, entry).path(self.path)) + } + + /// Its program for the probe `shader`, which answers hover, queries, and the depth under the + /// pointer for its geometry. + pub(super) fn probe(self, shader: Shader) -> Variant { + Variant::probe(shader, self.primitive()).path(self.path) + } + + /// The label passes: `claim`, which claims the screen cells its labels would cover, then + /// `texts`, which finds the texts needed by the labels that kept their cells. + pub(super) fn label_passes(self) -> [Variant; 2] { + ["claim", "texts"] + .map(|entry| Variant::compute(Shader::Texts, entry).path(self.path).primitive(self.primitive())) + } + + /// The pipeline drawing its labels' glyphs. + pub(super) fn glyphs(self, format: wgpu::TextureFormat, samples: u32) -> Variant { + Variant::draw(Shader::Glyphs, format, samples).path(self.path).primitive(self.primitive()) + } +} + +/// The pipelines a view requested for frames into one target, with or without input, by tier: draw, +/// text, and answer (transitions, lit rows, probes, and queries). pub(super) struct Asking { target: (wgpu::TextureFormat, u32), input: bool, @@ -34,35 +150,37 @@ pub(super) struct Asking { answering: Vec, } -/// What nearly every scene draws with, which the GPU compiles for views to come, in this order: the -/// channels, strokes between ends and through geometry, points and the backdrop, their bounds and -/// lists, ports; then text. +/// The pipelines nearly every scene draws with, which the GPU compiles ahead for later views: +/// channels, edges, lines along paths, vertices and their labels, and the backdrop, ports, and +/// overlay. #[cfg(all(feature = "web", target_arch = "wasm32"))] pub(crate) fn common(format: wgpu::TextureFormat, samples: u32) -> Vec { - let draw = |shader| Variant::draw(shader, format, samples); - let mut needs = vec![ - Variant::channels(pipelines::Reads { joined: false, timed: false, viewed: false }), - draw(Shader::Strokes).path(PATH_ENDS), - draw(Shader::Strokes).path(PATH_GEOMETRY), - draw(Shader::Points), - draw(Shader::Backdrop), - Variant::compute(Shader::Bounds, "points"), - ]; - for path in [PATH_ENDS, PATH_GEOMETRY] { - needs.push(Variant::compute(Shader::Bounds, "strokes").path(path)); - needs.extend(["count_segments", "write_segments"].map(|e| Variant::compute(Shader::Lists, e).path(path))); + let draw = |shader| Asked::Shader(Variant::draw(shader, format, samples)); + let marks = [(MarkKind::Edges, PATH_ENDS), (MarkKind::Lines, PATH_GEOMETRY), (MarkKind::Vertices, PATH_ENDS)] + .map(|(kind, path)| MarkVariants { kind, path, custom: None }); + let mut needs = vec![Asked::Shader(Variant::channels(pipelines::Reads::NONE))]; + needs.extend(marks.iter().filter_map(|m| m.draw(false, format, samples))); + needs.push(draw(Shader::Backdrop)); + for m in &marks { + needs.extend(m.bounds().map(Asked::Shader)); + if m.listed() { + needs.extend(m.segments().map(Asked::Shader)); + } } - needs.extend(["scan_blocks", "scan_sums", "add_sums"].map(|e| Variant::compute(Shader::Lists, e))); - needs.extend([draw(Shader::PointsPorts), draw(Shader::Overlay), draw(Shader::Glyphs), draw(Shader::GlyphsPorts)]); - needs.push(Variant::compute(Shader::Candidates, "candidates")); - needs.extend(["columns", "rows"].map(|e| Variant::compute(Shader::Fields, e))); - needs.into_iter().map(Asked::Shader).collect() + needs.extend(SCAN.map(Asked::Shader)); + needs.extend([draw(Shader::PointsPorts), draw(Shader::Overlay), draw(Shader::GlyphsPorts)]); + let vertices = marks[2]; + needs.push(Asked::Shader(vertices.glyphs(format, samples))); + needs.extend(vertices.label_passes().map(Asked::Shader)); + needs.extend(FIELDS.map(Asked::Shader)); + needs } impl View { - /// Asks for what frames of the scene into `target`, taking `input` or not, use, each at its - /// urgency, the first time; and for the channels of every mark, which compile for what they read - /// once their data says. Each wakes the view as it lands. + /// Requests the pipelines the scene's frames into `target` use, with or without `input`, each + /// at its urgency, on the first call for that pair. Every call also requests each mark's + /// channel pipeline, whose variant depends on what the channels read, known once the data is. + /// Each finished compile wakes the view. Returns the request's index in `asked`. pub(super) fn ask(&mut self, inner: &mut Inner, target: (wgpu::TextureFormat, u32), input: bool) -> usize { let device = &self.gpu.0.device; let at = if let Some(at) = self.asked.iter().position(|a| a.target == target && a.input == input) { @@ -74,7 +192,7 @@ impl View { } else { ([draw, text, changes, lit].concat(), Vec::new(), Vec::new()) }; - // Marks alike share pipelines, each asked for once. + // Several marks can need the same pipeline. Request each once. for asked in [&mut drawing, &mut texts, &mut answering] { let mut seen = rustc_hash::FxHashSet::default(); asked.retain(|need| seen.insert(need.clone())); @@ -89,8 +207,8 @@ impl View { at } - /// Each mark's channels, made for what they read: before its data says, neither joins, samples, nor - /// the view. + /// Each mark's channel pipeline, built for what its channels read. Before the mark's data is + /// known, it reads no joins, samples, or view state. fn channels(&self) -> Vec { let mut channels: Vec = Vec::new(); for shown in &self.marks { @@ -102,8 +220,8 @@ impl View { channels } - /// Whether frames into `format` with `samples` per pixel, taking `input` or not, can draw now: - /// whether what they draw with compiled. + /// Whether frames into `format` at `samples` per pixel, with or without `input`, can draw now: + /// whether their draw pipelines are compiled. pub(super) fn pipelines_ready( &mut self, inner: &mut Inner, @@ -115,21 +233,20 @@ impl View { inner.pipelines.ready(&self.asked[at].drawing) && inner.pipelines.ready(&self.channels()) } - /// Whether frames draw text: labels, their glyphs and fields, and the overlay. Until then they - /// draw without, and labels fade in once it lands, as after any change. + /// Whether frames can draw text: labels, their glyphs and distance fields, and the overlay. + /// Until then frames draw without text. pub(super) fn texts_ready(&self, inner: &Inner) -> bool { self.asked.iter().all(|a| inner.pipelines.ready(&a.texts)) } - /// Whether every target the view's frames taking input draw into can ease, light rows, probe, - /// and answer queries: whether what those use compiled. + /// Whether, for every target that frames taking input draw into, the pipelines for easing, lit + /// rows, probes, and queries are compiled. pub(super) fn answers_ready(&self, inner: &Inner) -> bool { self.asked.iter().filter(|a| a.input).all(|a| inner.pipelines.ready(&a.answering)) } } -/// What a scene's frames use: what they draw with, their text, what changes need, what lights rows, -/// and what input needs. +/// The pipelines a scene's frames use, by stage: drawing, text, transitions, lit rows, and input. #[derive(Default)] struct Stages { draw: Vec, @@ -139,46 +256,42 @@ struct Stages { input: Vec, } -/// What frames of `scene` into `format` with `samples` per pixel use, but for each mark's channels, -/// which compile for what they read. +/// The pipelines frames of `scene` into `format` at `samples` per pixel use, except each mark's +/// channel pipeline, which depends on what its channels read. fn stages(scene: &Scene, format: wgpu::TextureFormat, samples: u32) -> Stages { - let any = |test: fn(&Mark) -> bool| scene.marks.iter().any(test); + let any = |test: &dyn Fn(&Mark) -> bool| scene.marks.iter().any(test); let draw = |shader| Asked::Shader(Variant::draw(shader, format, samples)); - let compute = |shader, entry| Asked::Shader(Variant::compute(shader, entry)); let mut stages = Stages { draw: vec![draw(Shader::Backdrop)], - changes: vec![compute(Shader::Evaluate, "ease"), compute(Shader::Freeze, "gather")], + changes: vec![Asked::Shader(EASE), Asked::Shader(GATHER)], ..Stages::default() }; let controls = scene.controls; - let diagram = matches!(scene.space, Space::Diagram(_)); - // The overlay writes a plot's axes, the title, and the legends, and the box and the connection a - // gesture draws. - let legends = any(|m| m.style().legend.is_some() && m.style().color.is_some()); + // The overlay draws a plot's axes, the title, the legends, and the selection box and connection + // line that gestures draw. + let legends = any(&|m| m.style().legend.is_some() && m.style().color.is_some()); let written = matches!(scene.space, Space::Plot(_)) || scene.title.is_some() || scene.subtitle.is_some() || legends; if written || controls.select_box || controls.connect { stages.text.push(draw(Shader::Overlay)); } - // The distance fields of the text frames set are made before they draw. - let labeled = any(|m| { + // The distance fields of text a frame sets are built before the frame draws. + let labeled = any(&|m| { m.style().label.is_some() || matches!(m, Mark::Vertices(v) if v.ports.iter().any(|(_, port)| port.label.is_some())) }); if written || labeled { - stages.text.extend(["columns", "rows"].map(|e| compute(Shader::Fields, e))); - } - if any(|m| m.style().label.is_some()) && !diagram { - stages.text.extend([compute(Shader::Labels, "clear_cells"), compute(Shader::Labels, "reset_cells")]); + stages.text.extend(FIELDS.map(Asked::Shader)); } - if any(|m| m.style().shadow) { - stages.draw.push(Asked::Shader(Variant::draw(Shader::Shade, SHADE, 1))); + if any(&Mark::flows) { + stages.draw.push(Asked::Shader(TRAVEL)); + stages.changes.push(Asked::Shader(CARRY)); } - if any(Mark::flows) { - stages.draw.push(compute(Shader::Evaluate, "travel")); - stages.changes.push(compute(Shader::Evaluate, "carry")); + if any(&|m| MarkVariants::of(m, &scene.space).listed()) { + stages.draw.extend(SCAN.map(Asked::Shader)); } - if any(|m| matches!(m, Mark::Edges(_) | Mark::Lines(_) | Mark::Areas(_))) { - stages.draw.extend(["scan_blocks", "scan_sums", "add_sums"].map(|e| compute(Shader::Lists, e))); + // Any table may become paged: its data can outgrow the budget, or a mark can give a priority. + if !scene.marks.is_empty() { + stages.draw.push(Asked::Shader(EXPAND_RUNS)); } for mark in &scene.marks { mark_stages(mark, scene, format, samples, &mut stages); @@ -186,82 +299,53 @@ fn stages(scene: &Scene, format: wgpu::TextureFormat, samples: u32) -> Stages { stages } -/// What `mark` adds to the stages: its draw and compute passes, for the path a stroke mark runs -/// along or an area's triangles' geometry; its lit rows, which hover, selection, and the ends of -/// edges light; its probes; its labels. +/// Adds `mark`'s pipelines to `stages`: its draw and compute passes (for the path a stroke follows +/// or an area's triangle geometry), its lit-row draw (for the rows that hover, selection, and edge +/// endpoints light), its probe, and its labels. fn mark_stages(mark: &Mark, scene: &Scene, format: wgpu::TextureFormat, samples: u32, stages: &mut Stages) { - let draw = |shader| Variant::draw(shader, format, samples); - let compute = |shader, entry| Asked::Shader(Variant::compute(shader, entry)); - let style = mark.style(); - let primitive = mark.kind().primitive().code(); - let path = mark.path(&scene.space).unwrap_or(PATH_ENDS); - let (shader, cast, lit) = match mark { - Mark::Vertices(v) => { - if let Marker::Wgsl { source, .. } = &v.marker { - // A custom marker's WGSL follows its shader's. - let custom = - |shader, format, samples| Asked::Custom(source.clone(), Variant::draw(shader, format, samples)); - stages.draw.push(custom(Shader::PointsCustom, format, samples)); - stages.lit.push(custom(Shader::PointsCustomFocus, format, samples)); - if style.shadow { - stages.draw.push(custom(Shader::PointsCustomCast, SHADE, 1)); - } - (None, None, None) - } else { - (Some(Shader::Points), Some(Shader::PointsCast), Some(Shader::PointsFocus)) - } - } - Mark::Faces(_) => (Some(Shader::Triangles), Some(Shader::TrianglesCast), None), - Mark::Areas(_) => (Some(Shader::Areas), None, Some(Shader::AreasFocus)), - Mark::Edges(_) | Mark::Lines(_) => { - (Some(Shader::Strokes), Some(Shader::StrokesCast), Some(Shader::StrokesFocus)) - } - }; - stages.draw.extend(shader.map(|s| Asked::Shader(draw(s).path(path)))); - stages.lit.extend(lit.map(|s| Asked::Shader(draw(s).path(path)))); - if style.shadow { - stages.draw.extend(cast.map(|s| Asked::Shader(Variant::draw(s, SHADE, 1).path(path)))); - } - let bounds = match mark { - Mark::Vertices(_) => "points", - Mark::Areas(_) => "areas", - _ => "strokes", - }; - stages.draw.push(Asked::Shader(Variant::compute(Shader::Bounds, bounds).path(path))); - // Rows running through different numbers of segments, or areas of different numbers of - // triangles, are numbered by a list. - if matches!(mark, Mark::Edges(_)) || path == PATH_GEOMETRY { - stages.draw.extend( - ["count_segments", "write_segments"].map(|e| Asked::Shader(Variant::compute(Shader::Lists, e).path(path))), - ); + let variants = MarkVariants::of(mark, &scene.space); + stages.draw.extend(variants.draw(false, format, samples)); + stages.lit.extend(variants.draw(true, format, samples)); + stages.draw.extend(variants.bounds().map(Asked::Shader)); + if variants.listed() { + stages.draw.extend(variants.segments().map(Asked::Shader)); } - // One program answers the hover, the queries, and the depth under the pointer for the mark's - // geometry: any row drawn can be located, whether or not it picks. - stages.input.push(Asked::Shader(Variant::probe(pipelines::probe(samples), primitive).path(path))); - if style.label.is_some() { - // Measuring where labels hang and drawing them are made for the mark's primitive; a diagram's - // labels need no placement. - stages.text.push(Asked::Shader(draw(Shader::Glyphs).path(path).primitive(primitive))); - stages - .text - .push(Asked::Shader(Variant::compute(Shader::Candidates, "candidates").path(path).primitive(primitive))); - if !matches!(scene.space, Space::Diagram(_)) { - stages.text.extend(PLACING.map(|e| Asked::Shader(Variant::compute(Shader::Labels, e).path(path)))); - } - if style.label.as_ref().is_some_and(|l| l.max.is_some()) && !matches!(scene.space, Space::Diagram(_)) { - stages.text.extend(SELECT.map(|e| compute(Shader::Labels, e))); - } + // One probe program answers hover, queries, and the depth under the pointer for the mark's + // geometry. Every mark gets one, so any drawn row can be located, pickable or not. + stages.input.push(Asked::Shader(variants.probe(pipelines::probe(samples)))); + if mark.style().label.is_some() { + stages.text.push(Asked::Shader(variants.glyphs(format, samples))); + stages.text.extend(variants.label_passes().map(Asked::Shader)); } match mark { - Mark::Faces(_) => stages.draw.push(compute(Shader::Normals, "accumulate")), + Mark::Faces(_) => stages.draw.push(Asked::Shader(NORMALS)), Mark::Vertices(vertices) => { if !vertices.ports.is_empty() { - stages.draw.push(Asked::Shader(draw(Shader::PointsPorts))); + stages.draw.push(Asked::Shader(Variant::draw(Shader::PointsPorts, format, samples))); } if vertices.ports.iter().any(|(_, port)| port.label.is_some()) { - stages.text.push(Asked::Shader(draw(Shader::GlyphsPorts))); + stages.text.push(Asked::Shader(Variant::draw(Shader::GlyphsPorts, format, samples))); } } Mark::Edges(_) | Mark::Lines(_) | Mark::Areas(_) => {} } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::{Edges, Lines, Plane, Vertices}; + + #[test] + fn only_marks_that_frame_a_fit_ask_for_a_bounds_pass() { + let scene = Scene::new(Plane::default()) + .draw(Vertices::of("g", "V")) + .draw(Edges::of("g", "E").ends("from", "to")) + .draw(Lines::of("g", "L")); + let Stages { draw, .. } = stages(&scene, wgpu::TextureFormat::Rgba8Unorm, 1); + let asks = |entry, path| draw.contains(&Asked::Shader(Variant::compute(Shader::Bounds, entry).path(path))); + assert!(asks("points", PATH_ENDS), "vertices frame a fit"); + assert!(asks("strokes", PATH_GEOMETRY), "so do lines along their paths"); + assert!(!asks("strokes", PATH_ENDS), "edges lie between the vertices that frame them"); + } +} diff --git a/crates/latkit/src/view/wake.rs b/crates/latkit/src/view/wake.rs index 20536a3d..c1a4afd7 100644 --- a/crates/latkit/src/view/wake.rs +++ b/crates/latkit/src/view/wake.rs @@ -1,52 +1,68 @@ -//! Sleeping while waiting: work a view waits on (a readback the GPU copies, a job a host does, bytes -//! a host fetches) lands between frames and wakes the view's signal, which marks a frame due and -//! wakes whatever draws its frames. Until something lands or changes, the view draws nothing. +//! Waking a view. Work the view waits on (a GPU readback, a host job, a host fetch) arrives between +//! frames and wakes the view's signal. The signal marks a frame due and wakes whatever draws the +//! view's frames. Until something arrives or changes, the view draws nothing. use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex, PoisonError}; use std::task::{Wake, Waker}; -/// What wakes a view: work landing, or the view itself changing. +/// Wakes a view when work arrives or the view itself changes. #[derive(Default)] pub(crate) struct Signal { landed: AtomicBool, + /// The effect time a shot fixes, if any. elapsed: Mutex>, + /// Whether the shot draws a transparent background. transparent: AtomicBool, /// Whether a shot draws the view, so nothing else does. lent: AtomicBool, - /// Whatever draws the view's frames, as `View::set_waker` gave it. + /// Whether the shot keeps the view's data unchanged from when it was lent, as a picture does. + holding: AtomicBool, + /// The waker of whatever draws the view's frames, as `View::set_waker` set it. waker: Mutex>, } impl Signal { - /// Whether work landed since the frame began. + /// Whether work arrived since the frame began. pub(crate) fn landed(&self) -> bool { self.landed.load(Ordering::Acquire) } - /// Takes what landed: the frame beginning settles it. - pub(crate) fn clear(&self) { - self.landed.store(false, Ordering::Release); + /// Clears and returns the arrival flag at the start of a frame. Work that arrives afterwards, + /// including work this frame starts that the host finishes at once, wakes the next frame. + pub(crate) fn take(&self) -> bool { + self.landed.swap(false, Ordering::AcqRel) + } + + /// Restores the arrival flag `landed` for the next frame, as a frame that is not ready to draw + /// does. + pub(crate) fn leave(&self, landed: bool) { + if landed { + self.landed.store(true, Ordering::Release); + } } pub(crate) fn set_waker(&self, waker: Waker) { *self.waker.lock().unwrap_or_else(PoisonError::into_inner) = Some(waker); } - /// Lends the view to a shot: `waker` wakes the shot in place of whatever drew the view, which - /// draws it no more until it is given back; returns the waker it had. - pub(crate) fn lend(&self, waker: Waker, elapsed: Option, transparent: bool) -> Option { + /// Lends the view to a shot. `waker` wakes the shot instead of whatever drew the view, which + /// stops drawing it until it is given back. With `holding`, the view's data stays unchanged + /// meanwhile. Returns the previous waker. + pub(crate) fn lend(&self, waker: Waker, elapsed: Option, transparent: bool, holding: bool) -> Option { *self.elapsed.lock().unwrap_or_else(PoisonError::into_inner) = elapsed; self.transparent.store(transparent, Ordering::Release); + self.holding.store(holding, Ordering::Release); self.lent.store(true, Ordering::Release); self.waker.lock().unwrap_or_else(PoisonError::into_inner).replace(waker) } - /// Gives the view back to whatever drew it, `waker` waking it again, and asks it for a frame. + /// Returns the view to whatever drew it, restoring `waker`, and requests a frame. pub(crate) fn give_back(&self, waker: Option) { *self.waker.lock().unwrap_or_else(PoisonError::into_inner) = waker; *self.elapsed.lock().unwrap_or_else(PoisonError::into_inner) = None; self.transparent.store(false, Ordering::Release); + self.holding.store(false, Ordering::Release); self.lent.store(false, Ordering::Release); self.landed.store(true, Ordering::Release); self.ask(); @@ -65,7 +81,12 @@ impl Signal { self.lent.load(Ordering::Acquire) } - /// Asks whatever draws the view for a frame. + /// Whether the shot drawing the view keeps its data unchanged from when it was lent. + pub(crate) fn holding(&self) -> bool { + self.holding.load(Ordering::Acquire) + } + + /// Requests a frame from whatever draws the view. pub(crate) fn ask(&self) { if let Some(waker) = self.waker.lock().unwrap_or_else(PoisonError::into_inner).as_ref() { waker.wake_by_ref(); diff --git a/crates/latkit/src/view/words.rs b/crates/latkit/src/view/words.rs new file mode 100644 index 00000000..d623ca90 --- /dev/null +++ b/crates/latkit/src/view/words.rs @@ -0,0 +1,160 @@ +//! A mark's per-row words, in pages of the arena. For each page of held rows, the mark has one +//! arena block per kind of words, sized for the page's rows. A row's state is its kind words, then +//! its extra words. Alongside the state are the row's focus word, the normals faces sum at a +//! vertex, and while the mark eases, its frozen state (as shown when the transition began) and its +//! settled state. A page keeps its blocks while the mark holds rows there, so rows keep their words +//! as the camera moves, and adding held rows moves none. Every kind counts toward the GPU budget +//! along with the data. + +use std::sync::Arc; + +use crate::gpu::arena::Arena; +use crate::gpu::work::Queue; + +/// A kind of words a mark keeps per row. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub(crate) enum Kept { + State, + Focus, + Normals, + Frozen, + Settled, +} + +const KINDS: [Kept; 5] = [Kept::State, Kept::Focus, Kept::Normals, Kept::Frozen, Kept::Settled]; + +impl Kept { + /// Whether a new block of this kind is cleared on allocation, because something reads it before + /// anything writes it. State and frozen words need no clearing, since evaluation and freezing + /// write them first, and normals are cleared before each sum. + fn cleared(self) -> bool { + matches!(self, Kept::Focus | Kept::Settled) + } +} + +/// One kind's pages: the words per row, and each page's arena block and its size in words. +#[derive(Clone, PartialEq, Eq, Debug)] +pub(crate) struct Pages { + pub(crate) width: u32, + pub(crate) blocks: Arc<[u32]>, + words: Vec, +} + +impl Default for Pages { + fn default() -> Pages { + Pages { width: 0, blocks: Arc::new([]), words: Vec::new() } + } +} + +/// What fitting one kind's pages changed: the pages given new blocks, whose words start over, and +/// whether its page table changed at all. +pub(crate) struct Fitted { + pub(crate) new: Vec, + pub(crate) changed: bool, +} + +/// A mark's pages of every kind. +#[derive(Default)] +pub(crate) struct Words { + pages: [Pages; KINDS.len()], + /// The state pages from before this frame changed them, kept until the frame is recorded: a + /// transition starting this frame freezes rows from them. + before: Option, +} + +impl Words { + pub(crate) fn pages(&self, kept: Kept) -> &Pages { + &self.pages[kept as usize] + } + + /// The state pages as the frame began, which a starting transition freezes rows from. + pub(crate) fn state_before(&self) -> &Pages { + self.before.as_ref().unwrap_or(&self.pages[Kept::State as usize]) + } + + /// Drops the frame-start state pages once the frame is recorded. + pub(crate) fn settle(&mut self) { + self.before = None; + } + + /// Fits `kept`'s pages to hold `rows[p]` rows on page `p`, at `width` words per row. A `width` + /// of 0 frees them all. A page whose block is too small gets a new block, as does every page + /// when the width changes. Unused blocks are freed. Returns what changed, or `None` (with + /// nothing changed) when the arena has no space now. + pub(crate) fn fit( + &mut self, + kept: Kept, + rows: &[u32], + width: u32, + (device, queue): (&wgpu::Device, &Queue), + arena: &mut Arena, + ) -> Option { + let rows = if width == 0 { &[][..] } else { rows }; + let have = &self.pages[kept as usize]; + let holds = |p: usize| have.width == width && have.words.get(p).is_some_and(|&w| w >= rows[p] * width); + if have.blocks.len() == rows.len() && (0..rows.len()).all(holds) { + return Some(Fitted { new: Vec::new(), changed: false }); + } + let mut next = Pages { width, blocks: Arc::new([]), words: Vec::with_capacity(rows.len()) }; + let mut blocks = Vec::with_capacity(rows.len()); + let mut new = Vec::new(); + for (p, &n) in rows.iter().enumerate() { + if holds(p) { + blocks.push(have.blocks[p]); + next.words.push(have.words[p]); + continue; + } + let words = (n * width).max(1); + let Some(block) = arena.block(device, queue, words, kept.cleared()) else { + for &p in &new { + arena.free(blocks[p as usize], next.words[p as usize]); + } + return None; + }; + blocks.push(block); + next.words.push(crate::gpu::arena::rounded(words)); + new.push(p as u32); + } + next.blocks = blocks.into(); + let have = std::mem::replace(&mut self.pages[kept as usize], next); + let next = &self.pages[kept as usize]; + for (p, (&block, &words)) in have.blocks.iter().zip(&have.words).enumerate() { + if next.blocks.get(p) != Some(&block) { + arena.free(block, words); + } + } + if kept == Kept::State && self.before.is_none() { + self.before = Some(have); + } + Some(Fitted { new, changed: true }) + } + + /// Frees every page, as when the mark is removed. + pub(crate) fn release(&mut self, arena: &mut Arena) { + for kept in KINDS { + let pages = std::mem::take(&mut self.pages[kept as usize]); + for (&block, &words) in pages.blocks.iter().zip(&pages.words) { + arena.free(block, words); + } + } + self.before = None; + } + + /// The page tables the program buffer holds: each kind the mark keeps, with its blocks. + pub(crate) fn tables(&self) -> impl Iterator { + KINDS + .into_iter() + .filter(|&kept| self.pages[kept as usize].width > 0) + .map(|kept| (kept, &self.pages[kept as usize].blocks[..])) + } + + /// Words the page tables take in the program. + pub(crate) fn len(&self) -> usize { + self.tables().map(|(_, blocks)| blocks.len()).sum() + } + + /// Bytes its pages take in the arena. + pub(crate) fn bytes(&self) -> u64 { + self.pages.iter().flat_map(|p| &p.words).map(|&w| u64::from(w) * 4).sum() + } +} diff --git a/crates/latkit/src/web.rs b/crates/latkit/src/web.rs index 5a5eac15..27ba61fc 100644 --- a/crates/latkit/src/web.rs +++ b/crates/latkit/src/web.rs @@ -1,10 +1,13 @@ -//! Views in a browser. A [`PageView`] is a view the page holds: the page reads and changes it, hears -//! what it tells, and lends it to shots, with or without a canvas showing it. A [`Canvas`] shows views -//! in the panels of a `` element, or of an `OffscreenCanvas` in a worker, and feeds each view -//! the pointer, wheel, and keys that land on its panel. Every canvas of one [`Gpu`] draws in one -//! animation frame, into one command buffer, after the frame listeners of [`on_frame`] run: linked -//! views always show the same moment, and an application's per-frame work lands in the frame it was -//! done for. +//! Views in a browser. +//! +//! A [`PageView`] is a view owned by the page. The page reads and changes it, listens to its +//! events, and lends it to shots, whether or not a canvas shows it. A [`Canvas`] shows views in +//! the panels of its layout, in a `` element or in an `OffscreenCanvas` in a worker. It +//! feeds each view the pointer, wheel, and key input for its panel. +//! +//! Every canvas of one [`Gpu`] draws in the same animation frame and command buffer, after the +//! [`on_frame`] listeners run. So linked views always show the same moment, and an application's +//! per-frame work appears in the frame it was done for. use std::cell::RefCell; use std::rc::{Rc, Weak}; @@ -28,7 +31,7 @@ use crate::{Image, Shot, Video}; pub struct Listener(Option>); impl Listener { - /// Keeps listening for as long as what it listens to lives. + /// Keeps the listener running for as long as its source lives, instead of stopping it on drop. pub fn forget(mut self) { self.0 = None; } @@ -45,8 +48,8 @@ impl Drop for Listener { type FrameListener = Rc>; type EventListener = Rc>; -/// The animation frames of one `Gpu`: its listeners, then every canvas that needs a frame, then what -/// each of its views told. +/// The animation frames of one `Gpu`. Each frame runs the listeners, then draws every canvas that +/// needs a frame, then dispatches each view's events. struct Frames { gpu: Gpu, canvases: Vec>>, @@ -54,15 +57,15 @@ struct Frames { listeners: Vec<(u64, FrameListener)>, next: u64, scheduled: Option, - /// Whether a harness steps the frames itself ([`step`]): no animation frame is asked for. + /// Whether a harness steps the frames itself with [`step`], so no animation frame is requested. stepped: bool, - /// What waits for work a view waits on to land ([`landed`]), woken as it asks for a frame, and - /// how many times that was. + /// Wakers of [`landed`] futures, woken whenever a frame is requested, and the number of such + /// wakes. landing: Vec, wakes: u64, tick: Option>, - /// The page's `prefers-reduced-motion`, which each view is told, and what draws again when it - /// changes; none in a worker. + /// The page's `prefers-reduced-motion` query, passed to each view, and the closure that + /// requests a frame when it changes. `None` in a worker. reduced: Option<(web_sys::MediaQueryList, Closure)>, } @@ -112,7 +115,7 @@ fn frames_of(gpu: &Gpu) -> Rc> { }) } -/// The page's or the worker's global scope, which hands out animation frames. +/// The page's or worker's global scope, which provides animation frames. enum Scope { Window(web_sys::Window), Worker(web_sys::DedicatedWorkerGlobalScope), @@ -126,8 +129,8 @@ fn scope() -> Option { global.dyn_ref::().map(|worker| Scope::Worker(worker.clone())) } -/// Asks for an animation frame, unless one is asked for or a harness steps the frames, and wakes what -/// waits for work to land. +/// Requests an animation frame, unless one is already requested or a harness steps the frames. Also +/// wakes every [`landed`] future. fn schedule(frames: &Rc>) { let landing = { let mut f = frames.borrow_mut(); @@ -170,7 +173,7 @@ fn tick(frames: &Rc>, now: f64) { let mut encoder = gpu.device().create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("latkit") }); let shown: Vec = canvases.iter().filter_map(|c| c.borrow_mut().record(&mut encoder, now / 1000.0)).collect(); - // A frame no canvas drew submits nothing: listeners alone keep frames coming. + // A frame in which no canvas drew submits nothing. Listeners alone still keep frames coming. if !shown.is_empty() { gpu.queue().submit([encoder.finish()]); } @@ -191,7 +194,8 @@ fn tick(frames: &Rc>, now: f64) { } } -/// Releases the registry's GPU and cancels its frame as soon as its final owner disappears. +/// Removes the `Gpu`'s frames from the registry and cancels its requested frame once no canvas, +/// view, or listener remains. fn release_idle(frames: &Rc>) { let mut f = frames.borrow_mut(); f.canvases.retain(|c| c.strong_count() > 0); @@ -210,12 +214,18 @@ fn release_idle(frames: &Rc>) { FRAMES.with(|all| all.borrow_mut().retain(|f| !Rc::ptr_eq(f, frames))); } -/// Asks for an animation frame of the `Gpu` whose shared state lies at `gpu`: what a view's waker does -/// when work it waited on lands, always on the page's one thread. +/// A view's waker. When work the view waited on arrives, it requests an animation frame of the +/// `Gpu` whose shared state is at address `gpu`, and wakes its [`landed`] futures. It always runs +/// on the page's single thread. struct Wakes { gpu: usize, } +/// A waker that, when work arrives, requests a frame of `gpu` and wakes its [`landed`] futures. +pub(crate) fn waker(gpu: &Gpu) -> std::task::Waker { + std::task::Waker::from(Arc::new(Wakes { gpu: Arc::as_ptr(&gpu.0) as usize })) +} + impl std::task::Wake for Wakes { fn wake(self: Arc) { self.wake_by_ref(); @@ -230,8 +240,9 @@ impl std::task::Wake for Wakes { } } -/// Calls `listener` with the frame's timestamp, milliseconds as `requestAnimationFrame` gives it, -/// every animation frame before `gpu`'s canvases draw, until the returned listener is dropped. +/// Calls `listener` every animation frame, before `gpu`'s canvases draw, with the frame's timestamp +/// in milliseconds as `requestAnimationFrame` gives it. Stops when the returned [`Listener`] is +/// dropped. pub fn on_frame(gpu: &Gpu, listener: impl FnMut(f64) + 'static) -> Listener { let frames = frames_of(gpu); let id = { @@ -251,10 +262,12 @@ pub fn on_frame(gpu: &Gpu, listener: impl FnMut(f64) + 'static) -> Listener { }))) } -/// Draws one frame of `gpu` now, at `now`, milliseconds as an animation frame's timestamp: its -/// listeners, then every canvas that needs a frame, then what its views told. From the first step -/// `gpu` asks for no animation frames: a harness steps every frame itself, so a run draws the same -/// frames however fast the device is, and work landing wakes [`landed`] alone. +/// Draws one frame of `gpu` immediately, with `now` as its timestamp in milliseconds. Runs the +/// frame listeners, draws every canvas that needs a frame, then dispatches the views' events. +/// +/// After the first call, `gpu` requests no animation frames. The harness steps every frame itself, +/// so a run draws the same frames regardless of device speed, and arriving work only wakes +/// [`landed`]. pub fn step(gpu: &Gpu, now: f64) { let frames = frames_of(gpu); let scheduled = { @@ -272,9 +285,9 @@ pub fn step(gpu: &Gpu, now: f64) { tick(&frames, now); } -/// Waits until work a view of `gpu` waits on lands after this call: a readback, a compile, a job, a -/// text, a fetch; or until a view asks for a frame. What lands between the call and the first wait -/// counts, as a view a harness changes right after asking does. +/// Waits until work a view of `gpu` depends on arrives after this call, such as a readback, +/// compile, job, text, or fetch, or until a view requests a frame. Arrivals between this call and +/// the first poll count, so a harness may change a view right after calling it. pub fn landed(gpu: &Gpu) -> impl Future + 'static { let frames = frames_of(gpu); let seen = frames.borrow().wakes; @@ -288,8 +301,8 @@ pub fn landed(gpu: &Gpu) -> impl Future + 'static { }) } -/// Waits until `gpu` finished the work submitted to it, every readback of it mapped, and every -/// pipeline asked of it compiled: a frame stepped then draws all that landed. +/// Waits until `gpu` has finished its submitted work, mapped every readback, and compiled every +/// requested pipeline. A frame stepped afterwards draws all the results. pub async fn done(gpu: &Gpu) { let (submitted, finished) = crate::Pending::new(); gpu.queue().on_submitted_work_done(move || finished.resolve(Ok(()))); @@ -306,27 +319,27 @@ pub async fn done(gpu: &Gpu) { } } -/// A view the page holds: it reads and changes the view, hears what it tells, shows it in a canvas, -/// and lends it to shots. Changing it asks its canvas for a frame. +/// A view owned by the page, which reads and changes it, listens to its events, shows it in a +/// canvas, and lends it to shots. Changing it requests a frame. #[derive(Clone)] pub struct PageView(Rc>); struct Held { view: View, frames: Rc>, - /// Whether it was changed since it last drew. + /// Whether the view changed since it last drew. due: bool, listeners: Vec<(u64, EventListener)>, next: u64, } impl PageView { - /// Holds `view` for the page. Its `Gpu` must come from [`Gpu::new`]; it sets text with the - /// browser's fonts unless it was given a host. + /// Wraps `view` for the page. Its `Gpu` must come from [`Gpu::new`]. Unless the `Gpu` already + /// has a host, text is set with the browser's fonts. pub fn new(mut view: View) -> PageView { let gpu = view.gpu().clone(); - // The view sleeps while it waits; what it waits on asks for the next frame as it lands. - view.set_waker(std::task::Waker::from(Arc::new(Wakes { gpu: Arc::as_ptr(&gpu.0) as usize }))); + // The view sleeps while it waits. The work it waits on requests the next frame on arrival. + view.set_waker(waker(&gpu)); if gpu.0.lock().host.is_none() { gpu.set_host(crate::host::Browser); } @@ -337,7 +350,7 @@ impl PageView { PageView(held) } - /// Calls `f` with the view, then draws a frame when it needs one. + /// Calls `f` with the view, then requests a frame to draw any change. pub fn with(&self, f: impl FnOnce(&mut View) -> R) -> R { let result = f(&mut self.0.borrow_mut().view); self.0.borrow_mut().due = true; @@ -346,14 +359,19 @@ impl PageView { result } - /// Calls `f` with the view, to read it without asking for a frame. + /// Calls `f` with the view to read it, without requesting a frame. pub fn read(&self, f: impl FnOnce(&View) -> R) -> R { f(&self.0.borrow().view) } - /// Calls `listener` with what the view tells, a batch at a time and in order, until the returned - /// listener is dropped. A listener hears what is told while it listens; what lasts is state the - /// view reads out. + /// The GPU the view draws with. + pub fn gpu(&self) -> Gpu { + self.0.borrow().view.gpu().clone() + } + + /// Calls `listener` with the view's events, in order, one batch per call, until the returned + /// [`Listener`] is dropped. A listener receives only the events sent while it listens. Read + /// lasting state from the view itself. pub fn on(&self, listener: impl FnMut(&[Event]) + 'static) -> Listener { let id = { let mut held = self.0.borrow_mut(); @@ -370,7 +388,7 @@ impl PageView { }))) } - /// Hands what the view told to its listeners, one batch. + /// Sends the view's pending events to its listeners as one batch. fn dispatch(&self) { let (events, listeners) = { let mut held = self.0.borrow_mut(); @@ -387,8 +405,8 @@ impl PageView { } } - /// Whether it wants a frame and no shot holds it: it changed, or its view wants one. While what - /// it draws with compiles, only a compile landing does. + /// Whether it needs a frame and is not lent to a shot: it changed, or its view wants one. While + /// its pipelines compile, a change waits until they finish. pub fn wants_frame(&self) -> bool { let held = self.0.borrow(); !held.view.lent() && (held.view.wants_frame() || (held.due && !held.view.compiling())) @@ -399,20 +417,31 @@ impl PageView { self.0.borrow_mut().view.ready(format, pixel_ratio) } - /// Whether a shot holds it. + /// Whether it is lent to a shot. fn lent(&self) -> bool { self.0.borrow().view.lent() } } -/// Views shown in a `` element or an `OffscreenCanvas`, one per panel. An element sizes -/// itself to its CSS box at the device's pixel ratio and passes each view the pointer, wheel, and -/// keys that land on its panel; an offscreen canvas keeps the size it is given and takes no input. A -/// frame draws only the views that need one, or all of them when the canvas changes size. +/// A view's panel in a canvas. Later panels draw over earlier ones. +#[derive(Clone)] +pub struct Panel { + /// The view shown. + pub view: PageView, + /// Where it draws: `[x, y, width, height]` as fractions of the canvas, from its top left. + pub at: [f32; 4], +} + +/// Views shown in a `` element or an `OffscreenCanvas`, in the panels of its layout. +/// +/// An element canvas sizes itself to its CSS box at the device pixel ratio, and passes each view +/// the pointer, wheel, and key input for its panel. An offscreen canvas keeps the size it is given +/// and takes no input. A frame redraws only the views that need it, or all of them when the canvas +/// changes size or layout. #[derive(Clone)] pub struct Canvas(Rc>); -/// What a canvas draws into. +/// The canvas a [`Canvas`] draws into: an element, or an offscreen canvas and its pixel ratio. enum Target { Element(HtmlCanvasElement), Offscreen { canvas: OffscreenCanvas, pixel_ratio: f32 }, @@ -424,13 +453,14 @@ struct Painted { frames: Rc>, panels: Panels, views: Vec, - /// Each panel's texture while the canvas composes several views, and its size. + /// Each panel's texture and its size, used when the canvas shows several views. textures: Vec>, - /// Each panel in CSS pixels, as last drawn: what input lands on. + /// Each panel in CSS pixels as last drawn, used to route input. rects: Vec<[f32; 4]>, - /// Whether every view draws in the next frame: the canvas changed size or what it shows. + /// Whether every view draws in the next frame, because the canvas changed size or layout. due: bool, - /// The panel each pointer down went to, the panel under the pointer, and the panel keys go to. + /// The panel that captured each pressed pointer, the panel under the pointer, and the panel + /// that receives keys. captured: Vec<(u32, usize)>, hovered: Option, focus: usize, @@ -439,22 +469,23 @@ struct Painted { resize: Option, } -/// A listener of one kind of the element's events. +/// A listener for one kind of element event. type Handler = (&'static str, Closure); -/// What follows the element's size. +/// The observer that tracks the element's size, and its callback. type Resizing = (ResizeObserver, Closure); impl Canvas { - /// A canvas showing views in `panels` of `element`. The `Gpu` must come from [`Gpu::new`]. - pub fn new(gpu: &Gpu, element: &HtmlCanvasElement, panels: impl Into) -> Result { + /// A canvas showing the views of `layout` in `element`. The `Gpu` must come from [`Gpu::new`]. + pub fn new(gpu: &Gpu, element: &HtmlCanvasElement, layout: Vec) -> Result { let size = [element.width(), element.height()]; let surface = Surface::new(gpu, wgpu::SurfaceTarget::Canvas(element.clone()), size)?; let _ = element.style().set_property("touch-action", "none"); - // Keys reach the views without the page's help. + // Make the element focusable so keys reach the views directly. if !element.has_attribute("tabindex") { element.set_tab_index(0); } - let canvas = Canvas::with_target(gpu, Target::Element(element.clone()), surface, panels.into()); + let canvas = Canvas::with_target(gpu, Target::Element(element.clone()), surface); + canvas.set_layout(layout)?; let weak = Rc::downgrade(&canvas.0); let resized = Closure::::new(move |_: JsValue| { if let Some(state) = weak.upgrade() { @@ -470,30 +501,28 @@ impl Canvas { Ok(canvas) } - /// A canvas showing views in `panels` of `canvas`, `pixel_ratio` of its pixels a CSS pixel: in a - /// worker, or wherever no page feeds it input. It keeps the size the canvas is given. - pub fn offscreen( - gpu: &Gpu, - canvas: &OffscreenCanvas, - panels: impl Into, - pixel_ratio: f32, - ) -> Result { + /// A canvas showing the views of `layout` in `canvas`, at `pixel_ratio` pixels per CSS pixel. + /// Use it in a worker, or wherever the page does not feed it input. It keeps the size it is + /// given. + pub fn offscreen(gpu: &Gpu, canvas: &OffscreenCanvas, layout: Vec, pixel_ratio: f32) -> Result { if !(pixel_ratio.is_finite() && pixel_ratio > 0.0) { - return Err(Error::invalid("a canvas's pixel ratio is positive").at("pixelRatio")); + return Err(Error::invalid("a canvas's pixel ratio must be positive").at("pixelRatio")); } let size = [canvas.width(), canvas.height()]; let surface = Surface::new(gpu, wgpu::SurfaceTarget::OffscreenCanvas(canvas.clone()), size)?; let target = Target::Offscreen { canvas: canvas.clone(), pixel_ratio }; - Ok(Canvas::with_target(gpu, target, surface, panels.into())) + let made = Canvas::with_target(gpu, target, surface); + made.set_layout(layout)?; + Ok(made) } - fn with_target(gpu: &Gpu, target: Target, surface: Surface, panels: Panels) -> Canvas { + fn with_target(gpu: &Gpu, target: Target, surface: Surface) -> Canvas { let frames = frames_of(gpu); let state = Rc::new(RefCell::new(Painted { target, surface, frames: frames.clone(), - panels, + panels: Panels::default(), views: Vec::new(), textures: Vec::new(), rects: Vec::new(), @@ -510,12 +539,10 @@ impl Canvas { Canvas(state) } - /// Shows `views`, one per panel, in place of those it showed. A view shows in one canvas at a time. - pub fn show(&self, views: &[PageView]) -> Result<()> { - let expected = self.0.borrow().panels.len(); - if views.len() != expected { - return Err(Error::invalid(format!("a canvas of {expected} panels shows {} views", views.len()))); - } + /// Replaces the shown views with those of `layout`. A view shows in one panel of one canvas + /// at a time. Fails if a view repeats or shows in another canvas, or a panel covers no pixels. + pub fn set_layout(&self, layout: Vec) -> Result<()> { + let views: Vec<&PageView> = layout.iter().map(|panel| &panel.view).collect(); let twice = views.iter().enumerate().any(|(i, a)| views[i + 1..].iter().any(|b| Rc::ptr_eq(&a.0, &b.0))); let frames = self.0.borrow().frames.clone(); let elsewhere = frames.borrow().canvases.iter().filter_map(Weak::upgrade).any(|other| { @@ -523,10 +550,13 @@ impl Canvas { && other.borrow().views.iter().any(|shown| views.iter().any(|v| Rc::ptr_eq(&v.0, &shown.0))) }); if twice || elsewhere { - return Err(Error::invalid("a view shows in one panel of one canvas")); + return Err(Error::invalid("a view can show in only one panel of one canvas").at("layout")); } + let panels = Panels::from(layout.iter().map(|panel| panel.at).collect::>()); + panels.rects([1 << 16, 1 << 16], [1, 1]).map_err(|e| e.at("layout"))?; let mut state = self.0.borrow_mut(); - state.views = views.to_vec(); + state.panels = panels; + state.views = layout.into_iter().map(|panel| panel.view).collect(); state.textures.clear(); state.captured.clear(); state.hovered = None; @@ -544,10 +574,16 @@ impl Canvas { Target::Offscreen { .. } => None, } } + + /// The canvas's size in CSS pixels: its pixels over its pixels per CSS pixel. + pub fn css_size(&self) -> [f32; 2] { + let (size, ratio) = self.0.borrow().measure(); + size.map(|side| side as f32 / ratio) + } } impl Painted { - /// The canvas's size in pixels and pixels a CSS pixel. + /// The canvas's size in pixels, and its pixels per CSS pixel. fn measure(&self) -> ([u32; 2], f32) { match &self.target { Target::Element(element) => { @@ -561,16 +597,16 @@ impl Painted { } } - /// Whether a frame is due: the canvas changed, or a view it shows wants to draw; never while shots - /// hold every view it shows, whose last frames it keeps. + /// Whether a frame is due: the canvas changed, or a view it shows wants to draw. Never while + /// every view it shows is lent to a shot, since it keeps their last frames. fn wants_frame(&self) -> bool { self.views.iter().any(|v| !v.lent()) && (self.due || self.views.iter().any(PageView::wants_frame)) } - /// Records a frame when one is needed, sizing the canvas to its box first: each view that needs - /// to draws into its panel, straight into the canvas when one view fills it. A view a shot holds, - /// or whose scene's pipelines still compile, draws nothing new and keeps its panel; while no view - /// it shows can draw, the canvas keeps the frame it shows. + /// Records a frame if one is needed, first sizing the canvas to its CSS box. Each view that + /// needs to draw renders into its panel, or straight into the canvas when one view fills it. A + /// view lent to a shot, or whose pipelines still compile, keeps its panel's previous contents. + /// While no view can draw, the canvas keeps its current frame. fn record(&mut self, encoder: &mut wgpu::CommandEncoder, time: f64) -> Option { let (size, ratio) = self.measure(); let resized = size != self.surface.size(); @@ -605,7 +641,7 @@ impl Painted { if fresh { self.textures[i] = Some((panel_texture(self.surface.gpu.device(), format, rect.size), rect.size)); } - let (panel, _) = self.textures[i].as_ref().expect("made"); + let (panel, _) = self.textures[i].as_ref().expect("the panel texture exists"); let mut held = self.views[i].0.borrow_mut(); if ready[i] && (fresh || self.due || held.due || held.view.needs_render()) { let target = panel.create_view(&wgpu::TextureViewDescriptor::default()); @@ -629,7 +665,7 @@ impl Painted { Some(texture) } - /// Sets the element's cursor to the one the view under the pointer asks for. + /// Sets the element's cursor to the one the view under the pointer requests. fn show_cursor(&mut self) { let Target::Element(element) = &self.target else { return }; let cursor = self.hovered.and_then(|i| self.views.get(i)).map_or(Cursor::Default, |v| v.read(View::cursor)); @@ -646,9 +682,9 @@ impl Painted { } } - /// The panels `input` goes to, each with the input in its own CSS pixels: a pointer goes to the - /// panel it pressed until it lets go, else to the panel under it, which hears when it leaves; keys - /// go to the panel pressed last. + /// The panels `input` goes to, each with the input in that panel's CSS pixels. A pressed + /// pointer goes to the panel it pressed until release. Otherwise it goes to the panel under + /// it, which receives a leave when the pointer moves off. Keys go to the panel pressed last. fn route(&mut self, input: Input) -> Vec<(usize, Input)> { let rects = if self.rects.is_empty() { self.css_rects() } else { self.rects.clone() }; let under = |point: [f32; 2]| { @@ -695,7 +731,8 @@ impl Painted { routed } - /// Each panel in CSS pixels, from the canvas's size now: before its first frame records them. + /// Each panel in CSS pixels at the canvas's current size, used before the first frame records + /// them. fn css_rects(&self) -> Vec<[f32; 4]> { let (size, ratio) = self.measure(); self.panels.rects(size, [1, 1]).map(|rects| rects.iter().map(|r| css(*r, ratio)).collect()).unwrap_or_default() @@ -709,7 +746,7 @@ fn css(rect: Rect, ratio: f32) -> [f32; 4] { [x, y, w, h] } -/// `input` with its point taken from `origin`, so it reads from its panel's top left. +/// `input` with its point made relative to `origin`, its panel's top left. fn within(input: Input, origin: [f32; 2]) -> Input { let at = |point: [f32; 2]| [point[0] - origin[0], point[1] - origin[1]]; match input { @@ -725,7 +762,7 @@ fn within(input: Input, origin: [f32; 2]) -> Input { } } -/// A panel's texture: views draw into it, and the canvas copies it into place. +/// A panel's texture. A view draws into it, and the canvas copies it into place. fn panel_texture(device: &wgpu::Device, format: wgpu::TextureFormat, size: [u32; 2]) -> wgpu::Texture { device.create_texture(&wgpu::TextureDescriptor { label: Some("latkit panel"), @@ -739,9 +776,9 @@ fn panel_texture(device: &wgpu::Device, format: wgpu::TextureFormat, size: [u32; }) } -/// A canvas's surface: in the one format views draw in wherever the browser offers it, so canvases and -/// shots share their pipelines; alpha premultiplied; panels copied into it; configured again whenever -/// its size changes. +/// A canvas's surface. It uses the views' standard format when the browser offers it, so canvases +/// and shots share pipelines. Alpha is premultiplied where supported, panels are copied in, and the +/// surface is reconfigured whenever its size changes. struct Surface { gpu: Gpu, raw: wgpu::Surface<'static>, @@ -749,10 +786,10 @@ struct Surface { } impl Surface { - /// The surface of `target`, `size` pixels, drawn with `gpu`, which must come from [`Gpu::new`]. + /// Creates the surface of `target` at `size` pixels. `gpu` must come from [`Gpu::new`]. fn new(gpu: &Gpu, target: wgpu::SurfaceTarget<'static>, size: [u32; 2]) -> Result { let (Some(instance), Some(adapter)) = (gpu.instance(), gpu.adapter()) else { - return Err(Error::Unsupported("a canvas draws with a Gpu made by Gpu::new".into())); + return Err(Error::Unsupported("a canvas needs a Gpu created by Gpu::new".into())); }; let raw = instance.create_surface(target).map_err(|e| Error::Unsupported(format!("no WebGPU canvas: {e}").into()))?; @@ -764,14 +801,15 @@ impl Surface { .formats .first() .copied() - .ok_or_else(|| Error::Unsupported("the canvas offers no format".into()))? + .ok_or_else(|| Error::Unsupported("the canvas supports no texture format".into()))? }; let alpha_mode = if capabilities.alpha_modes.contains(&wgpu::CompositeAlphaMode::PreMultiplied) { wgpu::CompositeAlphaMode::PreMultiplied } else { wgpu::CompositeAlphaMode::Auto }; - // Panels are drawn apart and copied in: a WebGPU canvas takes copies, whatever wgpu reports. + // Panels are drawn separately and copied in. A WebGPU canvas accepts copies, regardless of + // what wgpu reports. let config = wgpu::SurfaceConfiguration { usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::COPY_DST, format, @@ -787,13 +825,13 @@ impl Surface { Ok(Surface { gpu: gpu.clone(), raw, config }) } - /// Its width and height in pixels, as last configured. + /// Width and height in pixels, as last configured. fn size(&self) -> [u32; 2] { [self.config.width, self.config.height] } - /// The texture to draw a frame of `size` pixels into, configuring the surface again first when - /// its size changed; `None` when the surface had none to give, which a later frame tries again. + /// The texture for a frame of `size` pixels, reconfiguring the surface first if its size + /// changed. `None` if the surface has no texture available. A later frame tries again. fn texture(&mut self, size: [u32; 2]) -> Option { let size = size.map(|v| v.max(1)); if size != self.size() { @@ -812,16 +850,16 @@ impl Surface { } } -/// Views the page holds, lent to a shot a step at a time, so the page reads them between steps and -/// hears what each step told. +/// Page views lent to a shot one step at a time, so the page can read them between steps and +/// receives each step's events. struct Lending<'a>(&'a [PageView]); impl Lending<'_> { - /// The views, each once. + /// Lends `views`. Fails if a view appears twice. fn of(views: &[PageView]) -> Result> { let twice = views.iter().enumerate().any(|(i, a)| views[i + 1..].iter().any(|b| Rc::ptr_eq(&a.0, &b.0))); if twice { - return Err(Error::invalid("a shot draws each view in one panel")); + return Err(Error::invalid("a view can appear in only one panel of a shot")); } Ok(Lending(views)) } @@ -841,14 +879,14 @@ impl Views for Lending<'_> { } } -/// The picture `shot` takes of `views`, one per panel, once they are complete and still. Canvases -/// showing them hold their last frames meanwhile. +/// Takes `shot`'s picture of `views`, one per panel, when they are complete, as its +/// [`Capture`](crate::Capture) says. Canvases showing them keep their last frames meanwhile. pub async fn image(shot: &Shot, views: &[PageView]) -> Result { shot.image_of(&mut Lending::of(views)?).await } -/// Draws `video`'s next frame of `views`, one per panel. Canvases showing them hold their last frames -/// until the video is finished or dropped. +/// Draws `video`'s next frame of `views`, one per panel. Canvases showing them keep their last +/// frames until the video is finished or dropped. pub async fn frame(video: &mut Video, views: &[PageView]) -> Result<()> { video.frame_of(&mut Lending::of(views)?).await } @@ -867,7 +905,7 @@ impl Drop for Painted { } } -/// Feeds the element's pointer, wheel, and keys to the views of the panels they land on. +/// Feeds the element's pointer, wheel, and key events to the views of the panels they target. fn listen(state: &Rc>, element: &HtmlCanvasElement) -> Result<()> { let add = |kind: &'static str, handle: fn(&HtmlCanvasElement, &web_sys::Event) -> Option| { let weak = Rc::downgrade(state); @@ -885,7 +923,7 @@ fn listen(state: &Rc>, element: &HtmlCanvasElement) -> Result<( view.dispatch(); } } - // A wheel a view zooms with does not scroll the page; a key it uses does not either. + // A wheel or key event a view used must not also scroll the page. if used && matches!(kind, "wheel" | "keydown") { event.prevent_default(); } @@ -899,7 +937,7 @@ fn listen(state: &Rc>, element: &HtmlCanvasElement) -> Result<( handler.as_ref().unchecked_ref(), &options, ) - .map_err(|_| Error::Unsupported("the canvas takes no input".into()))?; + .map_err(|_| Error::Unsupported("cannot listen for input on the canvas".into()))?; state.borrow_mut().handlers.push((kind, handler)); Ok::<(), Error>(()) }; @@ -959,7 +997,7 @@ fn point(element: &HtmlCanvasElement, event: &web_sys::MouseEvent) -> [f32; 2] { ] } -/// The keys a key or mouse event says are held. +/// The modifier keys a key or mouse event reports as held. macro_rules! modifiers { ($event:expr) => { Modifiers { @@ -972,7 +1010,7 @@ macro_rules! modifiers { } use modifiers; -/// A pointer press, focusing the element and capturing the pointer. +/// Reads a pointer press, focusing the element and capturing the pointer. fn pressed(element: &HtmlCanvasElement, event: &web_sys::Event) -> Option { let event = event.dyn_ref::()?; let button = match event.button() { @@ -992,7 +1030,7 @@ fn pressed(element: &HtmlCanvasElement, event: &web_sys::Event) -> Option }) } -/// What a pointer event's pointer is. +/// The kind of device behind a pointer event. fn device(event: &PointerEvent) -> Device { match event.pointer_type().as_str() { "touch" => Device::Touch, @@ -1001,7 +1039,7 @@ fn device(event: &PointerEvent) -> Device { } } -/// A key going down or up. +/// Reads a key press or release. fn keyed(event: &web_sys::Event) -> Option { let event = event.dyn_ref::()?; let key = match event.key().as_str() { diff --git a/crates/latkit/src/wire.rs b/crates/latkit/src/wire.rs new file mode 100644 index 00000000..666d6117 --- /dev/null +++ b/crates/latkit/src/wire.rs @@ -0,0 +1,159 @@ +//! Little-endian bytes: the encoding of latkit's byte format, of the jobs it hands its host, and of +//! the binary formats it reads. [`Out`] writes values. [`In`] reads them back, returning `None` +//! past the end. + +#[cfg(target_endian = "big")] +compile_error!("latkit's byte formats require a little-endian target"); + +/// Bytes being written. +#[derive(Default)] +pub(crate) struct Out(pub(crate) Vec); + +impl Out { + pub(crate) fn u8(&mut self, v: u8) { + self.0.push(v); + } + + pub(crate) fn u32(&mut self, v: u32) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + + /// A count or a length, as a `u32`. + pub(crate) fn len(&mut self, n: usize) { + self.u32(u32::try_from(n).expect("counts fit in u32")); + } + + pub(crate) fn u64(&mut self, v: u64) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + + pub(crate) fn f64(&mut self, v: f64) { + self.0.extend_from_slice(&v.to_le_bytes()); + } + + /// A text: its byte length, then its UTF-8. + pub(crate) fn str(&mut self, text: &str) { + self.len(text.len()); + self.0.extend_from_slice(text.as_bytes()); + } + + /// Plain values, as their bytes. + pub(crate) fn pod(&mut self, values: &[T]) { + self.0.extend_from_slice(bytemuck::cast_slice(values)); + } + + /// Bytes as they are. + pub(crate) fn bytes(&mut self, bytes: &[u8]) { + self.0.extend_from_slice(bytes); + } +} + +/// Gives a fieldless enum the `u32` code it is written as, `code`, and its reverse, `from_code`, +/// from one list of its variants and their codes. +#[cfg(feature = "gpu")] +macro_rules! codes { + ($type:ty { $($variant:ident = $code:literal),+ $(,)? }) => { + impl $type { + /// The code this is written as. + pub(crate) fn code(self) -> u32 { + match self { + $(Self::$variant => $code,)+ + } + } + + /// The value written as `code`. `None` for a code no value has. + pub(crate) fn from_code(code: u32) -> Option { + match code { + $($code => Some(Self::$variant),)+ + _ => None, + } + } + } + }; +} + +#[cfg(feature = "gpu")] +pub(crate) use codes; + +/// Bytes being read, front to back. +pub(crate) struct In<'a>(&'a [u8]); + +impl<'a> In<'a> { + pub(crate) fn new(bytes: &'a [u8]) -> In<'a> { + In(bytes) + } + + /// The bytes not read yet. + pub(crate) fn rest(&self) -> &'a [u8] { + self.0 + } + + pub(crate) fn is_empty(&self) -> bool { + self.0.is_empty() + } + + /// Whether at least `n` bytes are left. + pub(crate) fn has(&self, n: usize) -> bool { + n <= self.0.len() + } + + pub(crate) fn bytes(&mut self, n: usize) -> Option<&'a [u8]> { + let (head, rest) = self.0.split_at_checked(n)?; + self.0 = rest; + Some(head) + } + + pub(crate) fn take(&mut self) -> Option<[u8; N]> { + let (head, rest) = self.0.split_first_chunk::()?; + self.0 = rest; + Some(*head) + } + + pub(crate) fn u8(&mut self) -> Option { + self.take().map(u8::from_le_bytes) + } + + pub(crate) fn u32(&mut self) -> Option { + self.take().map(u32::from_le_bytes) + } + + pub(crate) fn u64(&mut self) -> Option { + self.take().map(u64::from_le_bytes) + } + + pub(crate) fn f64(&mut self) -> Option { + self.take().map(f64::from_le_bytes) + } + + /// A text written by [`Out::str`]. `None` unless it is UTF-8. + pub(crate) fn str(&mut self) -> Option<&'a str> { + let n = self.u32()? as usize; + std::str::from_utf8(self.bytes(n)?).ok() + } + + /// `n` plain values, copied out of their bytes. + pub(crate) fn pod(&mut self, n: usize) -> Option> { + let bytes = self.bytes(n.checked_mul(size_of::())?)?; + let mut values = vec![T::zeroed(); n]; + bytemuck::cast_slice_mut::(&mut values).copy_from_slice(bytes); + Some(values) + } + + /// A varint: seven bits a byte, lowest first, as protocol buffers and PMTiles write them. + pub(crate) fn varint(&mut self) -> Option { + let mut value = 0_u64; + for shift in (0..64).step_by(7) { + let byte = self.u8()?; + value |= u64::from(byte & 0x7f) << shift; + if byte < 0x80 { + return Some(value); + } + } + None + } +} + +/// The words of `bytes`, ignoring a trailing partial word. +pub(crate) fn words(bytes: &[u8]) -> Vec { + bytes.as_chunks::<4>().0.iter().map(|&word| u32::from_le_bytes(word)).collect() +} diff --git a/crates/latkit/tests/areas.rs b/crates/latkit/tests/areas.rs index 0d8194f1..6e6c179c 100644 --- a/crates/latkit/tests/areas.rs +++ b/crates/latkit/tests/areas.rs @@ -1,6 +1,7 @@ -//! Areas: polygons filled through the triangles the view derives for them, holes left open, picked -//! where they cover, framed by a fit, outlined by lines along their rings; on the earth whole across -//! the antimeridian, and shaped in a job when too large for the frame. +//! Areas: polygons filled through triangles the view derives, with holes left open. Areas pick +//! where they cover, a fit frames them, and lines trace their rings. On the earth they stay whole +//! across the antimeridian. Polygons too large for one frame are triangulated in a job, and ease in +//! with each row's color when it lands. #![cfg(feature = "gpu")] @@ -29,22 +30,22 @@ fn dark(pixel: [u8; 4]) -> bool { pixel[0].max(pixel[1]).max(pixel[2]) < 30 } -/// A dataset of one table of polygons, each row its rings. +/// A dataset with one table of polygons, each row a list of rings. fn polygons(rows: Vec>) -> Data { let ids: Vec = (1..=rows.len() as u64).collect(); - let mut data = Data::new("map"); + let data = Data::new("map"); data.apply(Patch::new().set(Table::new("Zone", ids).field("geometry", rows))).expect("valid"); data } -fn show(gpu: &latkit::Gpu, scene: Scene, data: Data) -> View { +fn view(gpu: &latkit::Gpu, scene: Scene, data: &Data) -> View { let mut view = View::new(gpu, scene.theme(plain_theme())).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([data]).expect("a source"); view } -/// Renders frames until `pending` resolves, within the settle deadline. +/// Renders frames until `pending` resolves, within `SETTLE` frames. fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future>) -> T { let mut pending = pin!(pending); let mut context = Context::from_waker(Waker::noop()); @@ -62,7 +63,7 @@ fn areas_fill_their_polygons_and_leave_their_holes() { let Some(gpu) = gpu() else { return }; let data = polygons(vec![vec![square(-40.0, -40.0, 80.0), square(-10.0, -10.0, 20.0)]]); let scene = Scene::new(plane()).draw(Areas::of("map", "Zone").color(Color::hex("#ff0000").expect("hex"))); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); assert!(red(at(&pixels, C - 25, C)), "filled: {:?}", at(&pixels, C - 25, C)); assert!(dark(at(&pixels, C, C)), "the hole is open: {:?}", at(&pixels, C, C)); @@ -71,18 +72,47 @@ fn areas_fill_their_polygons_and_leave_their_holes() { } #[test] -fn areas_asked_to_pick_pick_their_rows_and_frame_a_fit() { +fn areas_triangulated_in_a_job_ease_in_with_each_row_s_color() { + let Some(gpu) = gpu() else { return }; + // A page of unit squares, more than one frame triangulates, valued from 0 to 1 row by row. + let side = 64_u32; + let cells: Vec> = + (0..side * side).map(|i| vec![square(f64::from(i % side), f64::from(i / side), 1.0)]).collect(); + let values: Vec = (0..side * side).map(|i| f64::from(i) / f64::from(side * side - 1)).collect(); + let data = Data::new("map"); + data.apply(Table::new("Zone", Ids::count(side * side)).field("geometry", cells).field("v", values)).expect("valid"); + let color = latkit::field("v").colormap(latkit::Colormap::VIRIDIS).domain(0.0..=1.0); + let mut view = view(&gpu, Scene::new(plane()).draw(Areas::of("map", "Zone").color(color)), &data); + let middle = f64::from(side) / 2.0; + view.set_camera(Camera::Map { center: [middle, middle, 0.0], zoom: 0.75, pitch: 0.0, bearing: 0.0 }); + // The triangles landing start a transition. Once it settles, each row shows its own color. + let pixels = frame(&gpu, &mut view); + let cell = |x: f64, y: f64| { + let [px, py] = view.screen([x + 0.5, y + 0.5, 0.0]).expect("in view"); + at(&pixels, px as u32, py as u32) + }; + // Viridis runs from dark purple through teal to yellow. + let far = f64::from(side - 3); + let [low, half, high] = [cell(2.0, 2.0), cell(2.0, middle), cell(far, far)]; + assert!(low[0] < 90 && low[1] < 20 && low[2] > 60, "0 is purple: {low:?}"); + assert!(half[1] > 120 && half[2] > 120 && half[0] < 60, "0.5 is teal: {half:?}"); + assert!(high[0] > 200 && high[1] > 200 && high[2] < 80, "1 is yellow: {high:?}"); + assert_eq!(errors(&mut view), Vec::new()); +} + +#[test] +fn pickable_areas_return_their_rows_and_a_fit_frames_them() { let Some(gpu) = gpu() else { return }; let data = polygons(vec![vec![square(100.0, 100.0, 40.0)], vec![square(300.0, 100.0, 40.0)]]); - // Areas are geometry: they answer the pointer only when asked to. + // Areas respond to the pointer only when `pick` is on. let areas = Areas::of("map", "Zone").color(Color::hex("#ff0000").expect("hex")).pick(true); let scene = Scene::new(plane()).draw(areas); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); view.fit(Fit::All); frame(&gpu, &mut view); let Camera::Map { center, .. } = *view.camera() else { panic!("a map camera") }; assert!((center[0] - 220.0).abs() < 1.0 && (center[1] - 120.0).abs() < 1.0, "framed: {center:?}"); - let left = view.point([120.0, 120.0, 0.0]).expect("in view"); + let left = view.screen([120.0, 120.0, 0.0]).expect("in view"); let pending = view.pick(left, 2.0); let hits = answer(&gpu, &mut view, pending); assert_eq!(hits.first().map(|h| h.row.id.clone()), Some(1_u64.into()), "{hits:?}"); @@ -94,7 +124,7 @@ fn lines_trace_the_rings_of_polygons() { let data = polygons(vec![vec![square(-40.0, -40.0, 80.0)]]); let scene = Scene::new(plane()).draw(Lines::of("map", "Zone").width(3.0).color(Color::hex("#ff0000").expect("hex"))); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); assert!(red(at(&pixels, C - 40, C)), "the ring: {:?}", at(&pixels, C - 40, C)); assert!(dark(at(&pixels, C, C)), "inside stays open"); @@ -106,20 +136,23 @@ fn earth_areas_stay_whole_across_the_antimeridian() { let Some(gpu) = gpu() else { return }; let ring = vec![[170.0, -10.0], [-170.0, -10.0], [-170.0, 10.0], [170.0, 10.0], [170.0, -10.0]]; let data = polygons(vec![vec![ring]]); - let earth = Earth { graticule: false, ..Earth::mercator() }; + let earth = Earth { graticule: false, ..Earth::default() }; let scene = Scene::new(earth).draw(Areas::of("map", "Zone").color(Color::hex("#ff0000").expect("hex"))); - let mut view = show(&gpu, scene, data); - view.set_camera(Camera::Map { center: [180.0, 0.0, 0.0], zoom: 2.0, pitch: 0.0, bearing: 0.0 }); + let mut view = view(&gpu, scene, &data); + // A flat map is one world, so the camera looks just short of the antimeridian, whose edge lies + // past the view's at zoom 4. An area streaked the other way around the world would leave this + // side of it empty. + view.set_camera(Camera::Map { center: [174.0, 0.0, 0.0], zoom: 4.0, pitch: 0.0, bearing: 0.0 }); let pixels = frame(&gpu, &mut view); - assert!(red(at(&pixels, C, C)), "across the antimeridian: {:?}", at(&pixels, C, C)); - assert!(red(at(&pixels, C - 10, C)) && red(at(&pixels, C + 10, C)), "on both sides"); + assert!(red(at(&pixels, C, C)), "whole up to the antimeridian: {:?}", at(&pixels, C, C)); + assert!(red(at(&pixels, C - 10, C)) && red(at(&pixels, C + 10, C)), "on both sides of the middle"); assert_eq!(errors(&mut view), Vec::new()); } #[test] fn large_areas_triangulate_in_a_job_and_then_draw() { let Some(gpu) = gpu() else { return }; - // A ring of 40,000 points: past what a frame triangulates. + // A ring of 40,000 points, more than one frame triangulates. let ring: Ring = (0..=40_000) .map(|k| { let a = f64::from(k) / 40_000.0 * std::f64::consts::TAU; @@ -128,21 +161,21 @@ fn large_areas_triangulate_in_a_job_and_then_draw() { .collect(); let data = polygons(vec![vec![ring]]); let scene = Scene::new(plane()).draw(Areas::of("map", "Zone").color(Color::hex("#ff0000").expect("hex"))); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); - assert!(red(at(&pixels, C, C)), "drawn once the job lands: {:?}", at(&pixels, C, C)); + assert!(red(at(&pixels, C, C)), "drawn once the job finishes: {:?}", at(&pixels, C, C)); assert_eq!(errors(&mut view), Vec::new()); } #[test] -fn a_polygon_field_is_what_areas_fill() { +fn areas_require_a_polygon_field() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("map"); + let data = Data::new("map"); data.apply( Patch::new().set(Table::new("Zone", Ids::count(1)).field("geometry", vec![vec![[0.0, 0.0], [1.0, 1.0]]])), ) .expect("valid"); - let mut view = show(&gpu, Scene::new(plane()), data); - let refused = view.show(Scene::new(plane()).draw(Areas::of("map", "Zone"))); - assert!(refused.is_err(), "a path field is no polygon field"); + let mut view = view(&gpu, Scene::new(plane()), &data); + let refused = view.set_scene(Scene::new(plane()).draw(Areas::of("map", "Zone"))); + assert!(refused.is_err(), "areas reject a path field"); } diff --git a/crates/latkit/tests/common/mod.rs b/crates/latkit/tests/common/mod.rs index 55cda128..d6bc5a59 100644 --- a/crates/latkit/tests/common/mod.rs +++ b/crates/latkit/tests/common/mod.rs @@ -1,26 +1,28 @@ -//! What every GPU test shares: a device, frames rendered headless on a clock the test steps and +//! Shared helpers for GPU tests: a device, headless frames rendered on a test-stepped clock and //! read back, and plain scenes. #![allow(dead_code, unreachable_pub, reason = "each test file uses some")] use std::cell::Cell; -use latkit::{Camera, Color, Diagram, Frame, Gpu, Grid, Plane, Theme, View}; +use latkit::host::{Host, Job, Raster, TextRun}; +use latkit::{Camera, Color, Diagram, Frame, Gpu, Grid, Pending, Plane, Theme, View}; -/// Seconds a frame steps the test's clock: frames run as a display's would, however fast they draw. +/// Seconds each frame advances the test clock, so frames run at display rate regardless of draw +/// speed. pub const TICK: f64 = 1.0 / 60.0; thread_local! { - /// The test's clock: every frame steps it, and inputs read it. + /// The test clock. Every frame advances it, and inputs read it. static NOW: Cell = const { Cell::new(0.0) }; } -/// The test's clock, for inputs. +/// The test clock's time, for inputs. pub fn now() -> f64 { NOW.with(Cell::get) } -/// Steps the test's clock `seconds` without drawing. +/// Advances the test clock by `seconds` without drawing. pub fn wait(seconds: f64) { NOW.with(|now| now.set(now.get() + seconds)); } @@ -28,8 +30,23 @@ pub fn wait(seconds: f64) { pub const SIZE: u32 = 128; pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; -/// A device, or `None` where the machine has no adapter at all, which only `LATKIT_REQUIRE_GPU` -/// makes a failure. +/// A device, or `None` if the machine has no adapter. With `LATKIT_REQUIRE_GPU` set, a missing +/// adapter panics instead. +/// A host that runs jobs on separate threads, finishing frames later, as a page's workers do. +pub struct Threads; + +impl Host for Threads { + fn rasterize(&self, _: &TextRun) -> latkit::Result { + Err(latkit::Error::Unsupported("these views draw no text".into())) + } + + fn spawn(&self, job: Job) -> Pending> { + let (pending, resolver) = Pending::new(); + std::thread::spawn(move || resolver.resolve(Ok(job.run()))); + pending + } +} + pub fn gpu() -> Option { match pollster::block_on(Gpu::new()) { Ok(gpu) => Some(gpu), @@ -43,7 +60,7 @@ pub fn gpu() -> Option { /// The longest the GPU may take to finish a frame in these tests. pub const DEADLINE: std::time::Duration = std::time::Duration::from_secs(10); -/// The most frames a view may take to settle: uploads, probes, and a transition of two seconds. +/// The most frames a view may take to settle: uploads, probes, and a two-second transition. pub const SETTLE: u32 = 600; pub fn finish(gpu: &Gpu) { @@ -51,7 +68,7 @@ pub fn finish(gpu: &Gpu) { gpu.device().poll(wait).expect("the GPU finishes within the deadline"); } -/// Renders frames until the view is settled, transitions included, then reads the target's pixels. +/// Renders frames until the view settles, including transitions, then returns the target's pixels. pub fn frame(gpu: &Gpu, view: &mut View) -> Vec<[u8; 4]> { let mut pixels = step(gpu, view); for _ in 0..SETTLE { @@ -63,7 +80,7 @@ pub fn frame(gpu: &Gpu, view: &mut View) -> Vec<[u8; 4]> { panic!("the view settles within {SETTLE} frames"); } -/// Steps the test's clock a frame, renders one frame, and reads the target's pixels. +/// Advances the test clock one tick, renders one frame, and returns the target's pixels. pub fn step(gpu: &Gpu, view: &mut View) -> Vec<[u8; 4]> { wait(TICK); let device = gpu.device(); @@ -106,7 +123,7 @@ pub fn step(gpu: &Gpu, view: &mut View) -> Vec<[u8; 4]> { bytemuck::cast_slice::(&slice.get_mapped_range().expect("mapped")).to_vec() } -/// The errors a view reported since its events were last drained, other events dropped. +/// The errors a view reported since its events were last drained. Other events are discarded. pub fn errors(view: &mut View) -> Vec { view.events().filter_map(|e| if let latkit::Event::Error(e) = e { Some(e) } else { None }).collect() } @@ -123,11 +140,23 @@ pub fn plain_theme() -> Theme { Theme { background: Color::BLACK, ..Theme::dark() } } +/// The plain theme with green edges. +pub fn green_edges() -> Theme { + let plain = plain_theme(); + Theme { edge: latkit::EdgeTheme { color: Some(Color::rgb(0.0, 1.0, 0.0)), ..plain.edge }, ..plain } +} + +/// The plain theme with label halos off. +pub fn haloless() -> Theme { + let plain = plain_theme(); + Theme { text: latkit::TextTheme { halo: 0.0, ..plain.text.clone() }, ..plain } +} + pub fn plane() -> Plane { Plane { grid: Grid::Off } } -/// A diagram without a grid: y runs down, and at zoom 0 a unit is a pixel. +/// A diagram without a grid. Y points down, and at zoom 0 one unit is one pixel. pub fn diagram() -> Diagram { Diagram { grid: Grid::Off } } diff --git a/crates/latkit/tests/fixtures/latk-v1.hex b/crates/latkit/tests/fixtures/latk-v1.hex deleted file mode 100644 index a0217b3a..00000000 --- a/crates/latkit/tests/fixtures/latk-v1.hex +++ /dev/null @@ -1 +0,0 @@ -4c41544b010000004c000000000000000000000001000000060100000054010000007601000000000300000001000000020000000000000000000000000000000000f03f0000803f00000040 diff --git a/crates/latkit/tests/fixtures/patch.hex b/crates/latkit/tests/fixtures/patch.hex new file mode 100644 index 00000000..48b23cde --- /dev/null +++ b/crates/latkit/tests/fixtures/patch.hex @@ -0,0 +1 @@ +4c41544b48000000000000000000000001000000060100000054010000007601000000000300000001000000020000000000000000000000000000000000f03f0000803f00000040 diff --git a/crates/latkit/tests/gestures.rs b/crates/latkit/tests/gestures.rs index f0d9423d..92dd4d71 100644 --- a/crates/latkit/tests/gestures.rs +++ b/crates/latkit/tests/gestures.rs @@ -1,26 +1,33 @@ -//! Interaction events: a secondary click asks for a menu of what lies under it, as a touch held still -//! does; dragging from a port draws a connection to the vertex it is released over, an edge's end can -//! be dragged off it, a dragged row keeps where it was grabbed, a double click opens what it hits, a -//! click again can cycle through the rows under it, and Delete tells the selection. The keys point at -//! rows too; hover is state whether or not it glows and follows the frame, and a lit row keeps its -//! ports on top; setters tell nothing. A scene's first frame asks for every pipeline its input uses. -//! The wheel zooms as the controls say, and reduced motion pays every move at once. +//! Interaction events. A secondary click, or a touch held still, requests a menu of the rows under +//! it. Dragging from a port connects to the vertex it is released over, and dragging an edge +//! detaches its nearer end. A dragged row keeps its grab point under the pointer. A double click +//! opens the row it hits, clicking again cycles through the rows under the point, and Delete +//! reports the selection. Keys can step through rows too. +//! +//! Hover is reported whether or not it glows, and follows the frame. A lit row keeps its ports on +//! top. Setters emit no events. A scene's first frame requests every pipeline its input can use. +//! The wheel zooms as the controls say, and reduced motion applies every move at once. A flight +//! takes the seconds it is given. A flat map keeps the camera over its one world, and a frame +//! reports moving it there. #![cfg(feature = "gpu")] mod common; -use common::{SIZE, camera, diagram, frame, gpu, plain_theme, step}; +use std::time::Duration; + +use common::{SIZE, TICK, camera, diagram, frame, gpu, plain_theme, step}; use latkit::{ Button, Camera, Controls, Data, Device, Edges, End, Event, Ids, Input, Key, Modifiers, Motion, Patch, Phase, Row, - Rows, Scene, Side, Table, Vertices, View, Zoom, + Rows, Scene, Side, Table, Vertices, View, Zoom, field, }; const C: f32 = SIZE as f32 / 2.0; -/// Three nodes, a link from node 1 to node 3, and a port on each node's right. +/// Three nodes and a link from node 1 to node 3. Each node has an empty `net` reference for its +/// port. fn graph() -> Data { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set( @@ -40,19 +47,24 @@ fn graph() -> Data { data } -/// The graph in a diagram, its nodes 16 units across. +/// The graph in a diagram, with nodes 16 units across and a port on each node's right. fn scene(controls: Controls) -> Scene { Scene::new(diagram()) .theme(plain_theme()) .controls(controls) .draw(Edges::of("graph", "Link").ends("from", "to").width(3.0)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(8.0).port("net", Side::Right)) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(8.0).port("net", Side::Right)) } fn view(gpu: &latkit::Gpu, controls: Controls) -> View { + view_of(gpu, controls, &graph()) +} + +/// A view of `graph`, rendered until settled. +fn view_of(gpu: &latkit::Gpu, controls: Controls, graph: &Data) -> View { let mut view = View::new(gpu, scene(controls)).expect("valid scene"); view.set_camera(camera()); - view.set_data(graph()); + view.set_sources([graph]).expect("a source"); frame(gpu, &mut view); view } @@ -61,7 +73,7 @@ fn editing() -> Controls { Controls { connect: true, delete: true, ..Controls::default() } } -/// Feeds `inputs`, rendering a frame after each, then renders until `wanted` finds its event. +/// Feeds `inputs`, rendering two frames after each, then renders until `wanted` matches an event. fn after(gpu: &latkit::Gpu, view: &mut View, inputs: &[Input], wanted: impl Fn(&Event) -> Option) -> T { for input in inputs { view.input(input, common::now()); @@ -98,7 +110,7 @@ fn connected(e: &Event) -> Option<(End, Option, Option)> { } #[test] -fn secondary_clicks_ask_for_a_menu() { +fn secondary_clicks_request_a_menu() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); let at = [C - 40.0, C]; @@ -117,7 +129,7 @@ fn secondary_clicks_ask_for_a_menu() { fn dragging_from_a_port_connects() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); - // Node 1's port sits on its right side, 8 units from its middle. + // Node 1's port sits on its right side, 8 units from its center. let port = [C - 32.0, C]; let target = [C + 40.0, C]; let inputs = [ @@ -135,33 +147,33 @@ fn dragging_from_a_port_connects() { } #[test] -fn a_dragged_row_keeps_where_it_was_grabbed() { +fn a_dragged_row_keeps_its_grab_point_under_the_pointer() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", vec![1_u64]).field("x", vec![-40.0]).field("y", vec![0.0])).expect("valid"); let scene = Scene::new(diagram()) .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(8.0).drag(true)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(8.0).drag(true)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); - // Grabbed five units right of its middle, then moved 30 right and 20 down: y runs down. + // Grabbed 5 units right of its center, then moved 30 right and 20 down (y points down). let grab = [C - 35.0, C]; let inputs = [moved(grab), down(grab, Button::Primary), moved([C - 25.0, C + 10.0]), moved([C - 5.0, C + 20.0])]; let dragged = |e: &Event| if let Event::Drag { phase: Phase::Move, position, .. } = e { Some(*position) } else { None }; let position = after(&gpu, &mut view, &inputs, dragged); let off = (position[0] + 10.0).hypot(position[1] - 20.0); - assert!(off < 0.5, "the row goes where the pointer took the point it was grabbed by: {position:?}"); + assert!(off < 0.5, "the grab point stays under the pointer: {position:?}"); } #[test] fn a_lit_block_keeps_its_ports_on_top() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, Controls::default()); - // Node 1's port is hollow, on its right side 8 units from its middle; a unit inside the block, the - // ground shows through it, lit or not. + // Node 1's port is hollow, centered on its right edge 8 units from its center. One unit inside + // the block, the pixel within the port stays the same, lit or not. let port = |pixels: &[[u8; 4]]| common::at(pixels, C as u32 - 33, C as u32); let before = port(&step(&gpu, &mut view)); view.input(&moved([C - 40.0, C]), common::now()); @@ -169,19 +181,19 @@ fn a_lit_block_keeps_its_ports_on_top() { for _ in 0..30 { pixels = step(&gpu, &mut view); } - assert_eq!(view.hovered().map(|hit| hit.row.clone()), Some(Row::new("graph", "Node", 1_u64)), "lit"); - assert!(common::close(port(&pixels), before), "{:?}, where unlit it was {before:?}", port(&pixels)); + assert_eq!(view.hovered().map(|hit| hit.row.clone()), Some(Row::new("graph", "Node", 1_u64)), "node 1 is lit"); + assert!(common::close(port(&pixels), before), "the port is unchanged: lit {:?}, unlit {before:?}", port(&pixels)); } #[test] -fn a_scene_asks_for_all_it_uses_at_its_first_frame() { +fn the_first_frame_requests_every_pipeline_the_scene_can_use() { let Some(gpu) = gpu() else { return }; - // Its first frame asks for every pipeline the scene can use: a menu, a connection, a selection, - // the keys, and a transition after them make none where it is first used. + // The first frame requests every pipeline the scene can use. A menu, a connection, a selection, + // the keys, and a later transition then need no pipeline requested late. let scene = scene(editing()).ease(std::time::Duration::from_millis(200)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(graph()); + view.set_sources([graph()]).expect("a source"); frame(&gpu, &mut view); let at = [C - 40.0, C]; let menu = |e: &Event| matches!(e, Event::Menu { .. }).then_some(()); @@ -193,23 +205,23 @@ fn a_scene_asks_for_all_it_uses_at_its_first_frame() { &[moved(port), down(port, Button::Primary), moved([C, C]), moved(target), up(target)], connected, ); - let selected = |e: &Event| matches!(e, Event::Select { .. }).then_some(()); + let selected = |e: &Event| matches!(e, Event::Selection { .. }).then_some(()); after(&gpu, &mut view, &[moved(at), down(at, Button::Primary), up(at)], selected); after(&gpu, &mut view, &[key(Key::Menu)], menu); - let mut moved_graph = graph(); + let moved_graph = graph(); moved_graph .apply(Patch::new().set(Table::new("Node", vec![1_u64, 2, 3]).field("x", vec![-20.0, 20.0, 0.0]))) .expect("valid"); - view.set_data(moved_graph); + view.set_sources([&moved_graph]).expect("a source"); frame(&gpu, &mut view); - assert_eq!(gpu.work().late, 0, "every pipeline was asked for at the first frame"); + assert_eq!(gpu.work().late, 0, "every pipeline was requested at the first frame"); } #[test] -fn dragging_an_edge_lets_go_of_its_nearer_end() { +fn dragging_an_edge_detaches_its_nearer_end() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); - // The link runs from node 1 up to node 3; pressed nearer node 3, it lets go of that end. + // The link runs from node 1 up to node 3. Pressed nearer node 3, it detaches that end. let near_three = [C - 10.0, C - 30.0]; let target = [C + 40.0, C]; let inputs = [ @@ -227,10 +239,10 @@ fn dragging_an_edge_lets_go_of_its_nearer_end() { } #[test] -fn a_double_click_faster_than_its_probe_opens() { +fn a_double_click_before_any_probe_answer_still_opens() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); - // Both clicks land before any frame answers what lies under them. + // Both clicks arrive before any frame reports the rows under them. let at = [C + 40.0, C]; for input in [down(at, Button::Primary), up(at), down(at, Button::Primary), up(at)] { view.input(&input, common::now()); @@ -241,24 +253,23 @@ fn a_double_click_faster_than_its_probe_opens() { } #[test] -fn the_selection_lets_go_of_rows_the_data_drops() { +fn the_selection_drops_rows_removed_from_the_data() { let Some(gpu) = gpu() else { return }; - let mut view = view(&gpu, editing()); - view.select([Rows::new("graph", "Node", [2_u64, 3])]); - let mut data = view.data("graph").expect("shown").clone(); + let data = graph(); + let mut view = view_of(&gpu, editing(), &data); + view.set_selection([Rows::new("graph", "Node", [2_u64, 3])]); data.apply(Patch::new().remove("Node", vec![2_u64])).expect("valid"); - view.set_data(data); let selected = after(&gpu, &mut view, &[], |e| match e { - Event::Select { rows, trigger } => Some((rows.clone(), *trigger)), + Event::Selection { rows, trigger } => Some((rows.clone(), *trigger)), _ => None, }); let kept = vec![Rows::new("graph", "Node", [3_u64])]; - assert_eq!(selected, (kept.clone(), None), "the data's doing, no trigger"); + assert_eq!(selected, (kept.clone(), None), "a data change has no trigger"); assert_eq!(view.selection(), kept); } #[test] -fn delete_tells_the_selection() { +fn delete_reports_the_selection() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); let at = [C + 40.0, C]; @@ -271,14 +282,14 @@ fn delete_tells_the_selection() { } #[test] -fn a_shift_click_toggles_where_boxes_select() { +fn a_shift_click_toggles_when_box_select_is_on() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, Controls { select_box: true, ..Controls::default() }); let (at, shift) = ([C + 40.0, C], Modifiers { shift: true, ..Modifiers::default() }); let press = Input::Down { pointer: 1, point: at, button: Button::Primary, modifiers: shift, device: Device::Mouse }; let release = Input::Up { pointer: 1, point: at, modifiers: shift }; let toggled = after(&gpu, &mut view, &[moved(at), press, release], |e| match e { - Event::Select { rows, trigger: Some(trigger) } => Some((rows.clone(), trigger.modifiers.shift)), + Event::Selection { rows, trigger: Some(trigger) } => Some((rows.clone(), trigger.modifiers.shift)), _ => None, }); assert_eq!(toggled, (vec![Rows::new("graph", "Node", [2_u64])], true), "a click, not an empty box"); @@ -289,9 +300,9 @@ fn the_keys_step_open_and_select() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); let link = Row::new("graph", "Link", 0_u64); - // Tab waits for the rows nearest the middle, then points at the nearest: the link. + // Tab waits for the rows nearest the center, then hovers the nearest one: the link. let hovered = after(&gpu, &mut view, &[key(Key::Tab)], |e| match e { - Event::Hover { hit: Some(hit), trigger } => Some((hit.row.clone(), trigger.device)), + Event::Hovered { hit: Some(hit), trigger } => Some((hit.row.clone(), trigger.device)), _ => None, }); assert_eq!(hovered, (link.clone(), Device::Keyboard)); @@ -302,73 +313,95 @@ fn the_keys_step_open_and_select() { }); assert_eq!(opened, (link.clone(), Device::Keyboard)); let selected = after(&gpu, &mut view, &[key(Key::Space)], |e| match e { - Event::Select { rows, .. } => Some(rows.clone()), + Event::Selection { rows, .. } => Some(rows.clone()), _ => None, }); assert_eq!(selected, vec![Rows::from(link)]); - // Before the first row, Shift+Tab is not used, so focus moves on, and nothing stays pointed at. + // Shift+Tab past the first row goes unused, so focus leaves the view and the hover clears. let back = Input::Key { key: Key::Tab, down: true, modifiers: Modifiers { shift: true, ..Modifiers::default() } }; - assert!(!view.input(&back, common::now()), "focus moves on"); - let left = after(&gpu, &mut view, &[], |e| if let Event::Hover { hit, .. } = e { Some(hit.clone()) } else { None }); + assert!(!view.input(&back, common::now()), "focus leaves the view"); + let left = + after(&gpu, &mut view, &[], |e| if let Event::Hovered { hit, .. } = e { Some(hit.clone()) } else { None }); assert_eq!(left, None); } #[test] -fn hover_is_told_without_its_glow_and_follows_the_frame() { +fn hover_is_reported_without_glow_and_follows_the_frame() { let Some(gpu) = gpu() else { return }; - let mut view = view(&gpu, Controls { hover: false, ..Controls::default() }); + let data = graph(); + let mut view = view_of(&gpu, Controls { hover: false, ..Controls::default() }, &data); let at = [C + 40.0, C]; let hovered = after(&gpu, &mut view, &[moved(at)], |e| match e { - Event::Hover { hit: Some(hit), trigger } => Some((hit.row.clone(), trigger.device)), + Event::Hovered { hit: Some(hit), trigger } => Some((hit.row.clone(), trigger.device)), _ => None, }); assert_eq!(hovered, (Row::new("graph", "Node", 2_u64), Device::Mouse)); - // The node moves out from under the still pointer: the hover follows. - let mut data = view.data("graph").expect("shown").clone(); + // The node moves out from under the still pointer, and the hover follows. data.apply(Patch::new().set(Table::new("Node", vec![2_u64]).field("x", vec![200.0]))).expect("valid"); - view.set_data(data); - let left = after(&gpu, &mut view, &[], |e| if let Event::Hover { hit, .. } = e { Some(hit.clone()) } else { None }); + let left = + after(&gpu, &mut view, &[], |e| if let Event::Hovered { hit, .. } = e { Some(hit.clone()) } else { None }); assert_eq!(left, None); } #[test] -fn setters_tell_nothing_while_motion_tells_the_camera() { +fn setters_emit_no_events_but_a_flight_reports_the_camera() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); let set = Camera::Map { center: [10.0, 0.0, 0.0], zoom: 0.0, pitch: 0.0, bearing: 0.0 }; view.set_camera(set); - view.select([Rows::new("graph", "Node", [1_u64])]); + view.set_selection([Rows::new("graph", "Node", [1_u64])]); frame(&gpu, &mut view); - let told: Vec = view.events().filter(|e| matches!(e, Event::Camera(_) | Event::Select { .. })).collect(); - assert_eq!(told, Vec::new(), "setters tell nothing"); - view.fly(Camera::Map { center: [-10.0, 0.0, 0.0], zoom: 0.0, pitch: 0.0, bearing: 0.0 }); + let told: Vec = view.events().filter(|e| matches!(e, Event::Camera(_) | Event::Selection { .. })).collect(); + assert_eq!(told, Vec::new(), "setters emit no events"); + view.fly(Camera::Map { center: [-10.0, 0.0, 0.0], zoom: 0.0, pitch: 0.0, bearing: 0.0 }, None); let flown = after(&gpu, &mut view, &[], |e| if let Event::Camera(camera) = e { Some(*camera) } else { None }); - assert_ne!(flown, set, "a flight tells where it moved"); + assert_ne!(flown, set, "a flight reports where it moved"); +} + +#[test] +fn a_flight_takes_the_seconds_it_is_given() { + let Some(gpu) = gpu() else { return }; + let mut view = view(&gpu, editing()); + let Camera::Map { center, zoom, pitch, bearing } = *view.camera() else { unreachable!("a plane's camera") }; + let x = |view: &View| if let Camera::Map { center, .. } = view.camera() { center[0] } else { f64::NAN }; + let to = Camera::Map { center: [center[0] + 1.0, center[1], center[2]], zoom, pitch, bearing }; + view.fly(to, Some(Duration::from_secs(2))); + // One second of frames in, halfway through the two seconds, smoothstep puts the camera halfway. + let second = (1.0 / TICK).round() as usize; + for _ in 0..second { + step(&gpu, &mut view); + } + assert!((x(&view) - (center[0] + 0.5)).abs() < 1e-9, "halfway: {}", x(&view)); + for _ in 0..second { + step(&gpu, &mut view); + } + assert_eq!(*view.camera(), to, "arrives on time"); } #[test] -fn waiting_tells_when_the_frame_is_complete() { +fn waiting_reports_when_the_frame_is_complete() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, editing()); let waiting = after(&gpu, &mut view, &[], |e| if let Event::Waiting(waiting) = e { Some(*waiting) } else { None }); - assert!(waiting.is_empty() && view.complete(), "the last change told is the complete frame: {waiting}"); + assert!(waiting.is_empty() && view.complete(), "the latest waiting event reports a complete frame: {waiting}"); } fn selected(e: &Event) -> Option> { - if let Event::Select { rows, .. } = e { Some(rows.clone()) } else { None } + if let Event::Selection { rows, .. } = e { Some(rows.clone()) } else { None } } #[test] fn clicking_again_cycles_through_the_rows_under_the_point() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, Controls { cycle: true, ..Controls::default() }); - // Inside node 1, 6 units toward node 3, where the link leaves node 1's outline 2 units away. + // Inside node 1, 6 units toward node 3: 2 units short of where the link leaves its outline. let s = 6.0 / std::f32::consts::SQRT_2; let at = [C - 40.0 + s, C - s]; let (node, link) = (Row::new("graph", "Node", 1_u64), Row::new("graph", "Link", 0_u64)); let click = [down(at, Button::Primary), up(at)]; assert_eq!(after(&gpu, &mut view, &[&[moved(at)][..], &click].concat(), selected), vec![Rows::from(node.clone())]); - // Slower than a double click, where nothing moved: the next row under the point, then back. + // Clicks slower than a double click, without moving, select the next row under the point, then + // wrap back. common::wait(1.0); assert_eq!(after(&gpu, &mut view, &click, selected), vec![Rows::from(link)]); common::wait(1.0); @@ -376,7 +409,7 @@ fn clicking_again_cycles_through_the_rows_under_the_point() { } #[test] -fn a_touch_held_still_asks_for_a_menu_and_does_not_click() { +fn a_touch_held_still_requests_a_menu_and_does_not_click() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, Controls::default()); let at = [C - 40.0, C]; @@ -397,8 +430,8 @@ fn a_touch_held_still_asks_for_a_menu_and_does_not_click() { step(&gpu, &mut view); } assert!( - !view.events().any(|e| matches!(e, Event::Click { .. } | Event::Select { .. })), - "the release only lets go" + !view.events().any(|e| matches!(e, Event::Click { .. } | Event::Selection { .. })), + "the release neither clicks nor selects" ); } @@ -411,12 +444,34 @@ fn the_wheel_zooms_as_the_controls_say() { modifiers: Modifiers { control, ..Modifiers::default() }, }; let mut view = view(&gpu, Controls { zoom: Zoom::Modifier, ..Controls::default() }); - assert!(!view.input(&wheel(false), common::now()), "the plain wheel is the page's to scroll"); - assert!(view.input(&wheel(true), common::now()), "a trackpad's pinch, the wheel with Control, zooms"); + assert!(!view.input(&wheel(false), common::now()), "the plain wheel is left to scroll the page"); + assert!(view.input(&wheel(true), common::now()), "Control+wheel (a trackpad pinch) zooms"); let mut view = view_off(&gpu); assert!(!view.input(&wheel(true), common::now()), "nothing zooms"); } +#[test] +fn zooming_at_the_zoom_limit_moves_nothing() { + let Some(gpu) = gpu() else { return }; + let mut view = view(&gpu, Controls { motion: Motion::Reduced, ..Controls::default() }); + view.fit(latkit::Fit::All); + frame(&gpu, &mut view); + let Camera::Map { zoom: framed, .. } = *view.camera() else { panic!("a diagram's camera") }; + // Just short of the deepest zoom the framed content allows, the wheel zooms toward a corner. + let near = Camera::Map { center: [5.0, -3.0, 0.0], zoom: framed + 7.95, pitch: 0.0, bearing: 0.0 }; + view.set_camera(near); + let wheel = Input::Wheel { point: [C + 30.0, C - 20.0], delta: [0.0, -400.0], modifiers: Modifiers::default() }; + view.input(&wheel, common::now()); + step(&gpu, &mut view); + let at_limit = *view.camera(); + assert!((zoom(&view) - (framed + 8.0)).abs() < 1e-9, "the zoom stops at its limit: {}", zoom(&view)); + for _ in 0..3 { + view.input(&wheel, common::now()); + step(&gpu, &mut view); + assert_eq!(*view.camera(), at_limit, "past the limit neither the zoom nor the center moves"); + } +} + fn view_off(gpu: &latkit::Gpu) -> View { view(gpu, Controls { zoom: Zoom::Off, ..Controls::default() }) } @@ -427,10 +482,10 @@ fn zoom(view: &View) -> f64 { } #[test] -fn reduced_motion_pays_every_move_at_once() { +fn reduced_motion_applies_every_move_at_once() { let Some(gpu) = gpu() else { return }; let wheel = Input::Wheel { point: [C, C], delta: [0.0, -120.0], modifiers: Modifiers::default() }; - // With motion, a wheel notch glides over frames; without, it lands in one. + // With motion, a wheel notch glides over several frames. Without, it applies in one. let mut moving = view(&gpu, Controls { motion: Motion::Full, ..Controls::default() }); let before = zoom(&moving); moving.input(&wheel, common::now()); @@ -440,14 +495,33 @@ fn reduced_motion_pays_every_move_at_once() { let mut still = view(&gpu, Controls { motion: Motion::Reduced, ..Controls::default() }); still.input(&wheel, common::now()); step(&gpu, &mut still); - assert!((zoom(&still) - before - 0.3).abs() < 1e-6, "lands at once: {}", zoom(&still) - before); - // The system's preference reduces it too, and a flight jumps. + assert!((zoom(&still) - before - 0.3).abs() < 1e-6, "applies at once: {}", zoom(&still) - before); + // The system preference reduces motion too, and a flight jumps to its end. let mut system = view(&gpu, Controls::default()); system.set_reduced_motion(true); let to = Camera::Map { center: [-10.0, 5.0, 0.0], zoom: 1.0, pitch: 0.0, bearing: 0.0 }; - system.fly(to); + system.fly(to, None); step(&gpu, &mut system); step(&gpu, &mut system); assert_eq!(*system.camera(), to); assert!(!system.needs_render(), "nothing moves on its own"); } + +#[test] +fn a_flat_map_keeps_the_camera_over_its_one_world() { + let Some(gpu) = gpu() else { return }; + let mut view = View::new(&gpu, Scene::new(latkit::Earth::default()).theme(plain_theme())).expect("valid scene"); + // Set before the first frame gives the view its size, the camera moves within its limits on + // that frame, which reports it. At zoom 4 the world is 4096 pixels across, and the center keeps + // half the view's width from the antimeridian. + view.set_camera(Camera::Map { center: [180.0, 0.0, 0.0], zoom: 4.0, pitch: 0.0, bearing: 0.0 }); + frame(&gpu, &mut view); + let Camera::Map { center, .. } = *view.camera() else { panic!("a map camera") }; + assert!((center[0] - 180.0 * (1.0 - f64::from(SIZE) / 4096.0)).abs() < 1e-9, "{center:?}"); + assert!(view.events().any(|e| matches!(e, Event::Camera(_))), "the frame reports the move"); + // Once the view has its size, a camera set past its limits is kept within them at once. Zoomed + // all the way out, the whole world just fits the view, centered. + view.set_camera(Camera::Map { center: [90.0, 60.0, 0.0], zoom: -3.0, pitch: 0.0, bearing: 0.0 }); + let Camera::Map { center, zoom, .. } = *view.camera() else { panic!("a map camera") }; + assert!((zoom + 1.0).abs() < 1e-9 && center[0].abs() < 1e-9 && center[1].abs() < 1e-9, "{center:?} {zoom}"); +} diff --git a/crates/latkit/tests/graph.rs b/crates/latkit/tests/graph.rs index a57a7c03..114fa12c 100644 --- a/crates/latkit/tests/graph.rs +++ b/crates/latkit/tests/graph.rs @@ -1,7 +1,8 @@ -//! The graph: edges joining the same two vertices bow into lanes, edges bend through the points of -//! a path field, edges end in arrowheads, and great circles settle while their vertices animate; in a -//! diagram, nets join the ports of the vertices naming them, a block makes room for its ports, and -//! orthogonal wires route around blocks. +//! Graphs. Edges between the same two vertices bow into lanes, edges bend through the points of a +//! path field, edges end in arrowheads, and great-circle edges settle while their vertices animate. +//! In a diagram, nets join the ports of the vertices that reference them, a block grows to fit its +//! ports and drops its shadow onto the sheet, and orthogonal wires route around blocks. Ports pick +//! by name, and neighbors follow edges and nets. #![cfg(feature = "gpu")] @@ -12,7 +13,7 @@ use std::pin::pin; use std::task::{Context, Poll, Waker}; use common::{SIZE, at, camera, diagram, errors, frame, gpu, plain_theme, plane, step}; -use latkit::{Data, Edges, Ids, Patch, Port, Scene, Side, Table, Vertices, View}; +use latkit::{Data, Edges, Ids, Patch, Port, Scene, Side, Table, Vertices, View, field}; const C: u32 = SIZE / 2; @@ -20,10 +21,10 @@ fn lit(pixel: [u8; 4]) -> bool { pixel[0].max(pixel[1]).max(pixel[2]) > 60 } -fn show(gpu: &latkit::Gpu, scene: Scene, data: Data) -> View { +fn view(gpu: &latkit::Gpu, scene: Scene, data: &Data) -> View { let mut view = View::new(gpu, scene.theme(plain_theme())).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([data]).expect("a source"); view } @@ -32,7 +33,7 @@ fn nodes(x: Vec, y: Vec) -> Table { Table::new("Node", ids).field("x", x).field("y", y) } -/// Renders frames until `pending` resolves, within the settle deadline. +/// Renders frames until `pending` resolves, within `SETTLE` frames. fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future>) -> T { let mut pending = pin!(pending); let mut context = Context::from_waker(Waker::noop()); @@ -48,7 +49,7 @@ fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future> = vec![vec![[0.0, 30.0]], vec![]]; data.apply( Patch::new().set(nodes(vec![-40.0, 40.0, 40.0], vec![0.0, 0.0, -40.0])).set( @@ -88,8 +89,8 @@ fn edges_bend_through_their_points_and_pick_there() { .expect("valid"); let scene = Scene::new(plane()) .draw(Edges::of("graph", "Link").ends("from", "to").bends("bends").width(3.0).color(latkit::Color::WHITE)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(2.0)); - let mut view = show(&gpu, scene, data); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(2.0)); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); assert!(lit(at(&pixels, C, C - 30)), "through its bend"); assert!(!lit(at(&pixels, C, C)), "not straight across"); @@ -104,9 +105,9 @@ fn edges_bend_through_their_points_and_pick_there() { } #[test] -fn nets_join_the_ports_naming_them() { +fn nets_join_the_ports_that_reference_them() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set(nodes(vec![-40.0, 40.0, 0.0, 0.0], vec![0.0, 0.0, -40.0, 50.0]).reference( @@ -117,10 +118,10 @@ fn nets_join_the_ports_naming_them() { .set(Table::new("Net", vec![1_u64, 2])), ) .expect("valid"); - let scene = Scene::new(diagram()) - .draw(Edges::of("graph", "Net").width(3.0).color(latkit::Color::WHITE)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(5.0).port("net", Port::new(Side::Top))); - let mut view = show(&gpu, scene, data); + let scene = Scene::new(diagram()).draw(Edges::of("graph", "Net").width(3.0).color(latkit::Color::WHITE)).draw( + Vertices::of("graph", "Node").position(field("x"), field("y")).radius(5.0).port("net", Port::new(Side::Top)), + ); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); // Net 1 joins the tops of nodes 1 and 2, five units above their centers. assert!(lit(at(&pixels, C, C - 5)), "between the ports"); @@ -133,7 +134,7 @@ fn nets_join_the_ports_naming_them() { #[test] fn nets_of_more_than_two_meet_at_a_junction() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set(nodes(vec![-40.0, 40.0, 0.0], vec![0.0, 0.0, 45.0]).reference("net", "Net", vec![1_u64, 1, 1])) @@ -142,10 +143,10 @@ fn nets_of_more_than_two_meet_at_a_junction() { .expect("valid"); let scene = Scene::new(diagram()) .draw(Edges::of("graph", "Net").width(3.0).color(latkit::Color::WHITE)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(2.0).port("net", Side::Left)); - let mut view = show(&gpu, scene, data); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(2.0).port("net", Side::Left)); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); - // The junction sits at the middle of the three ports, 15 units down and 2 left. + // The junction sits at the center of the three ports, 15 units down and 2 left. assert!(lit(at(&pixels, C - 2, C + 15)), "at the junction"); assert!(lit(at(&pixels, C - 2, C + 30)), "to the third terminal"); assert!(!lit(at(&pixels, C, C - 20)), "nothing above"); @@ -155,7 +156,7 @@ fn nets_of_more_than_two_meet_at_a_junction() { #[test] fn an_unlabeled_block_makes_room_for_its_ports() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); let mut node = nodes(vec![0.0], vec![0.0]); for port in ["a", "b", "c", "d"] { node = node.reference(port, "Net", vec![None::]); @@ -163,19 +164,48 @@ fn an_unlabeled_block_makes_room_for_its_ports() { data.apply(Patch::new().set(node).set(Table::new("Net", Ids::count(0)))).expect("valid"); let block = ["a", "b", "c", "d"] .into_iter() - .fold(Vertices::of("graph", "Node").position("x", "y"), |v, port| v.port(port, Side::Left)); - let mut view = show(&gpu, Scene::new(diagram()).draw(block), data); + .fold(Vertices::of("graph", "Node").position(field("x"), field("y")), |v, port| v.port(port, Side::Left)); + let mut view = view(&gpu, Scene::new(diagram()).draw(block), &data); let pixels = frame(&gpu, &mut view); - // Four ports a pitch apart down its left side, a pitch to spare: about 77 units tall. + // Four ports one pitch apart down its left side, plus a pitch of margin: about 77 units tall. assert!(lit(at(&pixels, C, C + 30)) && lit(at(&pixels, C, C - 30)), "tall enough for its ports"); assert!(!lit(at(&pixels, C, C + 45)), "and no taller"); assert_eq!(errors(&mut view), Vec::new()); } +#[test] +fn a_block_drops_its_shadow_onto_the_sheet_below_it() { + let Some(gpu) = gpu() else { return }; + let gray = latkit::Color::rgb(0.5, 0.5, 0.5); + let plain = plain_theme(); + let shadow = latkit::ShadowTheme { color: latkit::Color::BLACK.alpha(0.9), lift: 6.0 }; + let theme = latkit::Theme { background: gray, diagram: latkit::DiagramTheme { shadow, ..plain.diagram }, ..plain }; + let below = |shadow: bool| { + let data = Data::new("graph"); + data.apply(Patch::new().set(nodes(vec![0.0], vec![0.0]))).expect("valid"); + let block = Vertices::of("graph", "Node") + .position(field("x"), field("y")) + .size(40.0, 24.0) + .marker(latkit::Marker::Block { corner: None }) + .color(latkit::Color::WHITE) + .shadow(shadow); + let mut view = View::new(&gpu, Scene::new(diagram()).draw(block).theme(theme.clone())).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), Vec::new()); + // Y points down the sheet. Sample just under the block's bottom edge, 12 units below its + // center. + at(&pixels, C, C + 15)[0] + }; + let (dropped, bare) = (below(true), below(false)); + assert!(dropped + 20 < bare, "the shadow darkens the sheet below: {dropped} with, {bare} without"); +} + #[test] fn edges_along_great_circles_settle_while_their_vertices_pulse() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set( @@ -190,18 +220,22 @@ fn edges_along_great_circles_settle_while_their_vertices_pulse() { )), ) .expect("valid"); - // The vertices' color reads the clock, so they evaluate every frame; nothing their arcs bend by moves. + // The vertices' color reads the clock, so they evaluate every frame, but nothing the arcs + // depend on moves. let pulse = latkit::elapsed().sin().colormap(latkit::Colormap::VIRIDIS).domain(-1.0..=1.0); - let scene = Scene::new(latkit::Earth { graticule: false, ..latkit::Earth::globe() }) - .draw(Edges::of("graph", "Link").ends("from", "to").width(2.0)) - .draw(Vertices::of("graph", "Node").position("lon", "lat").color(pulse)); + let scene = Scene::new(latkit::Earth { + graticule: false, + ..latkit::Earth::default().projection(latkit::Projection::Globe) + }) + .draw(Edges::of("graph", "Link").ends("from", "to").width(2.0)) + .draw(Vertices::of("graph", "Node").position(field("lon"), field("lat")).color(pulse)); let mut view = View::new(&gpu, scene.theme(plain_theme())).expect("valid scene"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view.fit(latkit::Fit::All); for _ in 0..60 { step(&gpu, &mut view); } - assert!(view.complete(), "waiting on {}", view.stats().waiting); + assert!(view.complete(), "waiting on {}", view.waiting()); assert_eq!(errors(&mut view), Vec::new()); } @@ -209,7 +243,7 @@ fn edges_along_great_circles_settle_while_their_vertices_pulse() { fn arrows_draw_heads_at_second_ends() { let Some(gpu) = gpu() else { return }; let count = |arrows: bool| { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Patch::new().set(nodes(vec![-40.0, 40.0], vec![0.0, 0.0])).set( Table::new("Link", Ids::count(1)).reference("from", "Node", vec![1_u64]).reference( "to", @@ -220,8 +254,8 @@ fn arrows_draw_heads_at_second_ends() { .expect("valid"); let scene = Scene::new(plane()) .draw(Edges::of("graph", "Link").ends("from", "to").width(2.0).arrows(arrows).color(latkit::Color::WHITE)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(1.0)); - let mut view = show(&gpu, scene, data); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(1.0)); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); assert_eq!(errors(&mut view), Vec::new()); pixels.iter().filter(|&&p| lit(p)).count() @@ -233,7 +267,7 @@ fn arrows_draw_heads_at_second_ends() { #[test] fn ports_draw_on_their_sides_and_pick_by_name() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set(nodes(vec![-40.0, 40.0], vec![0.0, 0.0]).reference("in", "Net", vec![1_u64, 1]).reference( @@ -246,13 +280,13 @@ fn ports_draw_on_their_sides_and_pick_by_name() { .expect("valid"); let scene = Scene::new(diagram()).draw( Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .radius(8.0) .color(latkit::Color::WHITE) .port("in", Side::Left) .port("out", Port::output(Side::Right)), ); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); assert!(lit(at(&pixels, C - 50, C)) && lit(at(&pixels, C - 30, C)), "a port on each side"); assert!(!lit(at(&pixels, C - 40, C - 11)), "none on top"); @@ -268,7 +302,7 @@ fn ports_draw_on_their_sides_and_pick_by_name() { #[test] fn orthogonal_routes_run_around_vertices() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Patch::new().set(nodes(vec![-50.0, 50.0, 0.0], vec![0.0, 0.0, 0.0])).set( Table::new("Link", Ids::count(1)).reference("from", "Node", vec![1_u64]).reference("to", "Node", vec![2_u64]), )) @@ -283,25 +317,25 @@ fn orthogonal_routes_run_around_vertices() { ) .draw( Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .size(20.0, 20.0) .marker(latkit::Shape::Rectangle) .color(latkit::Color::rgb(1.0, 0.0, 0.0)), ); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let pixels = frame(&gpu, &mut view); let green = |y: u32| at(&pixels, C, y)[1] > 120 && at(&pixels, C, y)[0] < 60; - // A straight edge would run under node 3, hidden; the route goes over or under it. + // A straight edge would run hidden under node 3. The route goes above or below it. let around = (C - 45..C - 11).chain(C + 11..C + 45).any(green); assert!(around, "around the vertex between"); assert_eq!(errors(&mut view), Vec::new()); } #[test] -fn many_routes_route_in_a_job_and_land() { +fn many_routes_route_in_a_job_and_arrive() { let Some(gpu) = gpu() else { return }; let links = |n: usize| { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Patch::new().set(nodes(vec![-50.0, 50.0, 0.0], vec![0.0, 0.0, 0.0])).set( Table::new("Link", Ids::count(n as u32)).reference("from", "Node", vec![1_u64; n]).reference( "to", @@ -314,11 +348,16 @@ fn many_routes_route_in_a_job_and_land() { }; let scene = Scene::new(diagram()) .draw(Edges::of("graph", "Link").ends("from", "to").route(latkit::Route::Orthogonal).width(2.0)) - .draw(Vertices::of("graph", "Node").position("x", "y").size(20.0, 20.0).marker(latkit::Shape::Rectangle)); - let mut view = show(&gpu, scene, links(1)); + .draw( + Vertices::of("graph", "Node") + .position(field("x"), field("y")) + .size(20.0, 20.0) + .marker(latkit::Shape::Rectangle), + ); + let mut view = view(&gpu, scene, &links(1)); frame(&gpu, &mut view); - // More wires than route on the frame: they draw as elbows until the job's routes land. - view.set_data(links(64)); + // More wires than one frame routes: they draw as elbows until the job's routes arrive. + view.set_sources([links(64)]).expect("a source"); for _ in 0..4 { step(&gpu, &mut view); } @@ -329,7 +368,7 @@ fn many_routes_route_in_a_job_and_land() { #[test] fn neighbors_follow_edges_and_nets() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set(nodes(vec![-40.0, 0.0, 40.0], vec![0.0, 0.0, 0.0]).reference("net", "Net", vec![1_u64, 1, 2])) @@ -344,8 +383,8 @@ fn neighbors_follow_edges_and_nets() { let scene = Scene::new(diagram()) .draw(Edges::of("graph", "Link").ends("from", "to")) .draw(Edges::of("graph", "Net")) - .draw(Vertices::of("graph", "Node").position("x", "y").port("net", Side::Top)); - let mut view = show(&gpu, scene, data); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).port("net", Side::Top)); + let mut view = view(&gpu, scene, &data); frame(&gpu, &mut view); let nodes = |ids: &[u64]| vec![latkit::Rows::new("graph", "Node", ids.iter().copied())]; let node = |id: u64| latkit::Row::new("graph", "Node", id); diff --git a/crates/latkit/tests/groups.rs b/crates/latkit/tests/groups.rs index 9fc0f4f1..9883fce7 100644 --- a/crates/latkit/tests/groups.rs +++ b/crates/latkit/tests/groups.rs @@ -1,19 +1,19 @@ -//! Groups: a diagram's vertex mark whose rows contain rows draws each as a container around its -//! members, padded and with its title's header on top, the same at every zoom; lit, it glows about -//! its border and hides nothing it holds. +//! Groups: a diagram vertex mark whose rows contain other rows draws each row as a padded container +//! around its members, with a title header on top. It scales only with the zoom. When lit, it glows +//! around its border and hides none of its members. #![cfg(feature = "gpu")] mod common; use common::{SIZE, at, errors, frame, gpu, plain_theme}; -use latkit::{Camera, Color, Data, Edges, Ids, Patch, Scene, Table, Theme, Vertices, View}; +use latkit::{Camera, Color, Data, Edges, Ids, Patch, Scene, Table, Theme, Vertices, View, field}; const C: u32 = SIZE / 2; -/// Two nodes in area 1 and one outside it, linked in a chain; y runs down. +/// Two nodes in area 1 and one outside it, linked in a chain. Y points down. fn graph() -> Data { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set(Table::new("Area", vec![1_u64])) @@ -34,15 +34,19 @@ fn graph() -> Data { } fn view(gpu: &latkit::Gpu, zoom: f64) -> View { - let theme = Theme { surface: Color::rgb(0.0, 0.0, 1.0), group_padding: 10.0, ..plain_theme() }; + let plain = plain_theme(); + let diagram = latkit::DiagramTheme { group: Color::rgb(0.0, 0.0, 1.0), group_padding: 10.0, ..plain.diagram }; + let theme = Theme { diagram, ..plain }; let scene = Scene::new(common::diagram()) .theme(theme) .draw(Vertices::of("graph", "Area").contains("Node", "area")) .draw(Edges::of("graph", "Link").ends("from", "to").width(2.0).color(Color::rgb(0.0, 1.0, 0.0))) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(4.0).color(Color::rgb(1.0, 0.0, 0.0))); + .draw( + Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0).color(Color::rgb(1.0, 0.0, 0.0)), + ); let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0; 3], zoom, pitch: 0.0, bearing: 0.0 }); - view.set_data(graph()); + view.set_sources([graph()]).expect("a source"); view } @@ -52,11 +56,11 @@ fn blue(pixels: &[[u8; 4]], x: u32, y: u32) -> bool { } #[test] -fn groups_stand_around_their_members() { +fn groups_enclose_their_members() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, 0.0); let pixels = frame(&gpu, &mut view); - // Members span −19…19 across and −4…4 down; the group pads them by 10 units. + // Members span −19…19 across and −4…4 down. The group pads them by 10 units. assert!(blue(&pixels, C - 27, C) && blue(&pixels, C + 27, C) && blue(&pixels, C, C - 12), "around the members"); assert!(!blue(&pixels, C - 31, C) && !blue(&pixels, C + 31, C) && !blue(&pixels, C, C + 16), "and no further"); assert!(at(&pixels, C - 15, C)[0] > 120, "members draw over it"); @@ -64,11 +68,11 @@ fn groups_stand_around_their_members() { } #[test] -fn a_lit_group_glows_about_its_border_and_hides_nothing() { +fn a_lit_group_glows_around_its_border_and_hides_nothing() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, 0.0); frame(&gpu, &mut view); - // The pointer over the group's padding, away from its members: the group lights. + // Hovering the group's padding, away from its members, lights the group. let point = [C as f32 - 26.0, C as f32 + 12.0]; let device = latkit::Device::Mouse; view.input(&latkit::Input::Move { pointer: 1, point, modifiers: latkit::Modifiers::default(), device }, 0.0); @@ -83,9 +87,9 @@ fn a_lit_group_glows_about_its_border_and_hides_nothing() { } #[test] -fn groups_scale_with_the_zoom_and_nothing_else() { +fn groups_scale_only_with_the_zoom() { let Some(gpu) = gpu() else { return }; - // At twice the zoom the container is twice as wide: the sheet is one fixed picture. + // At twice the zoom the container is twice as wide, since the sheet scales as one picture. let mut view = view(&gpu, 1.0); let pixels = frame(&gpu, &mut view); assert!(blue(&pixels, C - 56, C) && blue(&pixels, C + 56, C), "twice as far out"); diff --git a/crates/latkit/tests/labels.rs b/crates/latkit/tests/labels.rs index 03030859..c1ca6a2d 100644 --- a/crates/latkit/tests/labels.rs +++ b/crates/latkit/tests/labels.rs @@ -1,6 +1,8 @@ -//! Labels show beside their rows once the host sets their texts, or inside a block while they fit; -//! where two would overlap, the one of higher priority shows, and only labels that show have their -//! texts set. A diagram's labels scale with the sheet and fade out once too small to read. +//! Labels show beside their rows once the host rasterizes their text, or inside a block while they +//! fit. Where labels would overlap, the label of the row drawn on top shows. An empty text gives no +//! label. Only labels that show near the view have their text rasterized, so a dense view settles. +//! A conditional text labels only the rows it names, and a label keeps showing while its mark gains +//! texts. A diagram's labels scale with the sheet and fade out once too small to read. #![cfg(feature = "gpu")] @@ -9,12 +11,12 @@ mod common; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; -use common::{SIZE, at, camera, diagram, errors, frame, gpu, plain_theme, plane}; +use common::{SETTLE, SIZE, at, camera, diagram, errors, frame, gpu, haloless, plain_theme, plane, step}; use latkit::Ids; use latkit::host::{Host, Raster, TextRun}; -use latkit::{Camera, Color, Data, Label, Marker, Scene, Table, Vertices, View, field}; +use latkit::{Camera, Color, Data, Expr, Label, Marker, Scene, Table, Vertices, View, field, when}; -/// Sets a text as a solid box six tenths of its size wide per character, as tall as its size. +/// A host that rasterizes text as a solid box, 0.6 × size wide per character and `size` tall. struct Boxes; impl Host for Boxes { @@ -24,7 +26,24 @@ impl Host for Boxes { } } -/// Boxes, counting the texts it sets. +/// `Boxes` with 2 blank pixels on each side, as a font host pads every run, so an empty text +/// rasterizes to a blank box. +struct Padded; + +impl Host for Padded { + fn rasterize(&self, run: &TextRun) -> latkit::Result { + let inner = Boxes.rasterize(run)?; + let size = [inner.size[0] + 4, inner.size[1]]; + let mut coverage = vec![0; (size[0] * size[1]) as usize]; + for (row, line) in inner.coverage.chunks(inner.size[0].max(1) as usize).enumerate() { + let at = row * size[0] as usize + 2; + coverage[at..at + line.len()].copy_from_slice(line); + } + Ok(Raster { size, coverage, baseline: inner.baseline }) + } +} + +/// `Boxes`, counting the texts it rasterizes. struct Counted(Arc); impl Host for Counted { @@ -34,45 +53,41 @@ impl Host for Counted { } } -fn view(gpu: &latkit::Gpu, x: Vec, priority: Vec) -> View { - shown(gpu, x, priority, |v| v.radius(2.0)) +fn view(gpu: &latkit::Gpu, x: Vec) -> View { + shown(gpu, x, |v| v.radius(2.0)) } -fn shown(gpu: &latkit::Gpu, x: Vec, priority: Vec, shape: impl Fn(Vertices) -> Vertices) -> View { +fn shown(gpu: &latkit::Gpu, x: Vec, shape: impl Fn(Vertices) -> Vertices) -> View { gpu.set_host(Boxes); - let mut data = Data::new("d"); + let data = Data::new("d"); let n = x.len(); data.apply( Table::new("V", Ids::count(n as u32)) .field("x", x) .field("y", vec![0.0; n]) .field("name", vec!["abcdef"; n]) - .field("p", priority) .field("hue", (0..n).map(|i| i as f64).collect::>()), ) .expect("valid"); let red = Color::rgb(1.0, 0.0, 0.0); let blue = Color::rgb(0.0, 0.0, 1.0); - let label = Label::new("name") - .size(10.0) - .priority(field("p")) - .color(latkit::when(field("hue").lt(0.5)).then(red).otherwise(blue)); + let label = Label::new(field("name")).size(10.0).color(when(field("hue").lt(0.5)).then(red).otherwise(blue)); let scene = Scene::new(plane()) - .theme(latkit::Theme { halo: 0.0, ..plain_theme() }) - .draw(shape(Vertices::of("d", "V").position("x", "y")).pick(false).label(label)); + .theme(haloless()) + .draw(shape(Vertices::of("d", "V").position(field("x"), field("y"))).pick(false).label(label)); let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view } #[test] fn labels_show_beside_their_rows() { let Some(gpu) = gpu() else { return }; - let mut view = view(&gpu, vec![-50.0, 10.0], vec![0.0, 0.0]); + let mut view = view(&gpu, vec![-50.0, 10.0]); let pixels = frame(&gpu, &mut view); let c = SIZE / 2; - // A label starts four pixels past its marker's right edge, centered on it. + // A label starts four pixels past its marker's right edge, vertically centered on it. let first = at(&pixels, c - 50 + 12, c); let second = at(&pixels, c + 10 + 12, c); assert!(first[0] > 200 && first[2] < 60, "the first row's label: {first:?}"); @@ -83,84 +98,144 @@ fn labels_show_beside_their_rows() { #[test] fn a_color_changing_every_frame_leaves_labels_placed() { let Some(gpu) = gpu() else { return }; - // A pulse reads the clock, so the mark evaluates every frame; nothing that places its labels moved. + // A pulse reads the clock, so the mark evaluates every frame, but nothing that places its + // labels moves. let pulse = latkit::elapsed().sin().colormap(latkit::Colormap::VIRIDIS).domain(-1.0..=1.0); - let mut view = shown(&gpu, vec![-50.0, 10.0], vec![0.0, 0.0], |v| v.radius(2.0).color(pulse.clone())); + let mut view = shown(&gpu, vec![-50.0, 10.0], |v| v.radius(2.0).color(pulse.clone())); for _ in 0..30 { - common::step(&gpu, &mut view); + step(&gpu, &mut view); } - assert!(view.complete(), "waiting on {}", view.stats().waiting); + assert!(view.complete(), "waiting on {}", view.waiting()); } #[test] -fn a_block_holds_its_label_while_it_fits() { +fn a_block_shows_its_label_inside_while_it_fits() { let Some(gpu) = gpu() else { return }; let c = SIZE / 2; let green = Color::rgb(0.0, 1.0, 0.0); - // The label is 36 by 10 pixels: a 60 by 20 block holds it, a 20 by 20 one does not. - let mut wide = shown(&gpu, vec![0.0], vec![0.0], |v| v.size(60.0, 20.0).color(green)); + // The label is 36 by 10 pixels. A 60 by 20 block fits it, and a 20 by 20 block does not. + let mut wide = shown(&gpu, vec![0.0], |v| v.size(60.0, 20.0).color(green)); let middle = at(&frame(&gpu, &mut wide), c, c); assert!(middle[0] > 200 && middle[1] < 60, "the label inside: {middle:?}"); - let mut narrow = shown(&gpu, vec![0.0], vec![0.0], |v| v.size(20.0, 20.0).color(green)); + let mut narrow = shown(&gpu, vec![0.0], |v| v.size(20.0, 20.0).color(green)); let middle = at(&frame(&gpu, &mut narrow), c, c); assert!(middle[1] > 200 && middle[0] < 60, "only the block: {middle:?}"); } #[test] -fn overlapping_labels_give_way_by_priority() { +fn where_labels_would_overlap_the_row_drawn_on_top_shows_its_label() { let Some(gpu) = gpu() else { return }; - let mut view = view(&gpu, vec![0.0, 4.0], vec![1.0, 5.0]); - let pixels = frame(&gpu, &mut view); let c = SIZE / 2; - let shared = at(&pixels, c + 20, c); - assert!(shared[2] > 200 && shared[0] < 60, "the higher priority label shows: {shared:?}"); - let only_first = at(&pixels, c + 7, c); - assert!(only_first[0] < 60, "the lower one gave way: {only_first:?}"); + // The two labels, each 36 pixels wide, would start 4 pixels apart: the first red, the second + // blue. The second row draws on top, so its label shows and the first's does not. + let pixels = frame(&gpu, &mut view(&gpu, vec![0.0, 4.0])); + let first = at(&pixels, c + 7, c); + assert!(first[0] < 60, "no first label: {first:?}"); + let second = at(&pixels, c + 20, c); + assert!(second[2] > 200 && second[0] < 60, "the second label: {second:?}"); + // Given a higher priority, the first row draws on top instead, and its label shows. + let mut first_on_top = shown(&gpu, vec![0.0, 4.0], |v| v.radius(2.0).priority(-field("hue"))); + let pixels = frame(&gpu, &mut first_on_top); + let first = at(&pixels, c + 7, c); + assert!(first[0] > 200 && first[2] < 60, "the first label: {first:?}"); + let after = at(&pixels, c + 44, c); + assert!(after[2] < 60, "no second label past the first: {after:?}"); } #[test] -fn only_labels_that_show_have_their_texts_set() { +fn an_empty_text_gives_no_label_and_claims_no_room() { + let Some(gpu) = gpu() else { return }; + gpu.set_host(Padded); + let c = SIZE / 2; + // The rows of the overlap above, with the second row's text empty: it draws on top but has no + // label, so the first row's label shows. + let data = Data::new("d"); + let rows = Table::new("V", Ids::count(2)).field("x", vec![0.0, 4.0]).field("y", vec![0.0; 2]); + data.apply(rows.field("name", vec!["abcdef", ""])).expect("valid"); + let label = Label::new(field("name")).size(10.0).color(Color::rgb(1.0, 0.0, 0.0)); + let mark = Vertices::of("d", "V").position(field("x"), field("y")).radius(2.0).pick(false).label(label); + let mut view = View::new(&gpu, Scene::new(plane()).theme(haloless()).draw(mark)).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + let first = at(&frame(&gpu, &mut view), c + 7, c); + assert!(first[0] > 200, "the first label: {first:?}"); + assert!(view.complete(), "waiting on {}", view.waiting()); +} + +#[test] +fn only_labels_that_show_near_the_view_have_their_text_rasterized() { let Some(gpu) = gpu() else { return }; let set = Arc::new(AtomicUsize::new(0)); gpu.set_host(Counted(set.clone())); - // Two thousand rows in view, each named apart: far more labels than fit. + // Two thousand rows, each with a distinct name: forty crowded in view, the rest far beyond it. let n = 2000; - let x: Vec = (0..n).map(|i| f64::from((i * 37) % 120) - 60.0).collect(); + let x: Vec = + (0..n).map(|i| if i < 40 { f64::from((i * 37) % 120) - 60.0 } else { 10_000.0 + f64::from(i) }).collect(); let y: Vec = (0..n).map(|i| f64::from((i * 53) % 120) - 60.0).collect(); let names: Vec = (0..n).map(|i| format!("row {i}")).collect(); - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("V", Ids::count(n as u32)).field("x", x).field("y", y).field("name", names)).expect("valid"); let scene = Scene::new(plane()).theme(plain_theme()).draw( Vertices::of("d", "V") - .position("x", "y") + .position(field("x"), field("y")) .radius(1.0) .color(Color::rgb(0.0, 0.0, 1.0)) .pick(false) - .label(Label::new("name").size(10.0).color(Color::rgb(1.0, 0.0, 0.0))), + .label(Label::new(field("name")).size(10.0).color(Color::rgb(1.0, 0.0, 0.0))), ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); - let count = set.load(Ordering::Relaxed); - assert!(count > 0 && count < 200, "texts set: {count}"); + // Only the labels that keep their room among the forty need texts. + let texts = set.load(Ordering::Relaxed); + assert!((1..40).contains(&texts), "some of the texts of the rows in view, and no others: {texts}"); let red = pixels.iter().filter(|p| p[0] > 200 && p[1] < 60).count(); assert!(red > 200, "labels show: {red} pixels"); assert_eq!(errors(&mut view), Vec::new()); } +#[test] +fn a_dense_view_settles_with_texts_for_the_labels_that_fit() { + let Some(gpu) = gpu() else { return }; + let set = Arc::new(AtomicUsize::new(0)); + gpu.set_host(Counted(set.clone())); + // Twenty thousand rows with distinct names, all in view: far more texts than the atlas holds. + let n = 20_000; + let x: Vec = (0..n).map(|i| f64::from((i * 37) % 128) - 64.0).collect(); + let y: Vec = (0..n).map(|i| f64::from((i * 53) % 128) - 64.0).collect(); + let names: Vec = (0..n).map(|i| format!("row {i}")).collect(); + let data = Data::new("d"); + data.apply(Table::new("V", Ids::count(n as u32)).field("x", x).field("y", y).field("name", names)).expect("valid"); + let scene = Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("d", "V") + .position(field("x"), field("y")) + .radius(1.0) + .pick(false) + .label(Label::new(field("name")).size(10.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + frame(&gpu, &mut view); + assert!(view.complete(), "waiting on {}", view.waiting()); + // A 128 pixel square holds a few dozen labels 10 pixels tall. + let texts = set.load(Ordering::Relaxed); + assert!(texts < 200, "texts for the labels that fit: {texts}"); +} + /// A block in a diagram, 60 by 20 units, labeled in red at size 10, seen at `zoom`. fn diagram_block(gpu: &latkit::Gpu, zoom: f64) -> Vec<[u8; 4]> { gpu.set_host(Boxes); - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply( Table::new("V", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0]).field("name", vec!["abcdef"]), ) .expect("valid"); - let label = Label::new("name").size(10.0).color(Color::rgb(1.0, 0.0, 0.0)); - let scene = Scene::new(diagram()).theme(latkit::Theme { halo: 0.0, ..plain_theme() }).draw( + let label = Label::new(field("name")).size(10.0).color(Color::rgb(1.0, 0.0, 0.0)); + let scene = Scene::new(diagram()).theme(haloless()).draw( Vertices::of("d", "V") - .position("x", "y") + .position(field("x"), field("y")) .size(60.0, 20.0) .marker(Marker::block()) .color(Color::rgb(0.0, 1.0, 0.0)) @@ -169,7 +244,7 @@ fn diagram_block(gpu: &latkit::Gpu, zoom: f64) -> Vec<[u8; 4]> { ); let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0; 3], zoom, pitch: 0.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(gpu, &mut view); assert_eq!(errors(&mut view), Vec::new()); pixels @@ -180,7 +255,7 @@ fn diagram_labels_scale_with_the_sheet() { let Some(gpu) = gpu() else { return }; let c = SIZE / 2; let red = |p: [u8; 4]| p[0] > 200 && p[1] < 60; - // At zoom 0 the label is 36 by 10 pixels; at zoom 1 twice that, still inside its block. + // At zoom 0 the label is 36 by 10 pixels. At zoom 1 it is twice that, still inside its block. let near = diagram_block(&gpu, 0.0); assert!(red(at(&near, c + 15, c)) && !red(at(&near, c + 21, c)), "36 wide at zoom 0"); let closer = diagram_block(&gpu, 1.0); @@ -191,20 +266,21 @@ fn diagram_labels_scale_with_the_sheet() { fn diagram_labels_fade_once_too_small_to_read() { let Some(gpu) = gpu() else { return }; let c = SIZE / 2; - // At zoom −2 the text is 2.5 pixels tall: past reading, it is gone, and the block shows. + // At zoom −2 the text is 2.5 pixels tall, too small to read, so it hides and the block shows. let far = diagram_block(&gpu, -2.0); let middle = at(&far, c, c); assert!(middle[1] > 120 && middle[0] < 60, "only the block: {middle:?}"); } #[test] -fn a_cap_shows_only_the_labels_of_highest_priority() { +fn a_conditional_text_labels_only_the_rows_it_names() { let Some(gpu) = gpu() else { return }; let set = Arc::new(AtomicUsize::new(0)); gpu.set_host(Counted(set.clone())); - // Four rows one above another, apart enough that none would give way, priorities rising upward. + // Four rows stacked vertically, far enough apart not to overlap, `p` rising upward. Only rows + // with `p` of 3 or more have a text. let ys = [-45.0, -15.0, 15.0, 45.0]; - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply( Table::new("V", Ids::count(4)) .field("x", vec![-30.0; 4]) @@ -213,21 +289,62 @@ fn a_cap_shows_only_the_labels_of_highest_priority() { .field("p", vec![1.0, 2.0, 3.0, 4.0]), ) .expect("valid"); - let label = Label::new("name").size(10.0).priority(field("p")).color(Color::rgb(1.0, 0.0, 0.0)).max(2); - let marks = Vertices::of("d", "V").position("x", "y").radius(1.0).color(Color::rgb(0.0, 0.0, 1.0)).pick(false); - let scene = Scene::new(plane()).theme(latkit::Theme { halo: 0.0, ..plain_theme() }).draw(marks.label(label)); + let text = when(field("p").ge(3.0)).then(field("name")).otherwise(Expr::missing()); + let label = Label::new(text).size(10.0).color(Color::rgb(1.0, 0.0, 0.0)); + let marks = Vertices::of("d", "V") + .position(field("x"), field("y")) + .radius(1.0) + .color(Color::rgb(0.0, 0.0, 1.0)) + .pick(false); + let scene = Scene::new(plane()).theme(haloless()).draw(marks.label(label)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let c = SIZE / 2; - // The plane runs y up; each label starts beside its marker. + // Y points up on the plane. Each label starts beside its marker. let labeled = |y: f64| { let row = (f64::from(c) - y) as u32; (c - 30 + 4..c - 30 + 20).any(|x| at(&pixels, x, row)[0] > 200) }; - assert!(labeled(45.0) && labeled(15.0), "the two of highest priority show"); - assert!(!labeled(-15.0) && !labeled(-45.0), "the rest give way to the cap"); - assert_eq!(set.load(Ordering::Relaxed), 2, "only the labels shown have their texts set"); + assert!(labeled(45.0) && labeled(15.0), "the rows the text names are labeled"); + assert!(!labeled(-15.0) && !labeled(-45.0), "the rest have no label"); + assert_eq!(set.load(Ordering::Relaxed), 2, "only the texts shown are rasterized"); assert_eq!(errors(&mut view), Vec::new()); } + +#[test] +fn a_label_keeps_showing_while_its_mark_gains_texts() { + let Some(gpu) = gpu() else { return }; + gpu.set_host(Boxes); + // Row 0 in view, and `n - 1` more rows with names of their own far beyond it. + let table = |n: u32| { + let x: Vec = (0..n).map(|i| if i == 0 { -50.0 } else { 10_000.0 + f64::from(i) }).collect(); + let names: Vec = (0..n).map(|i| if i == 0 { "abcdef".into() } else { format!("row {i}") }).collect(); + Table::new("V", Ids::count(n)).field("x", x).field("y", vec![0.0; n as usize]).field("name", names) + }; + let data = Data::new("d"); + data.apply(table(1)).expect("valid"); + let label = Label::new(field("name")).size(10.0).color(Color::rgb(1.0, 0.0, 0.0)); + let marks = Vertices::of("d", "V").position(field("x"), field("y")).radius(2.0).pick(false); + let scene = Scene::new(plane()).theme(haloless()).draw(marks.label(label)); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + let c = SIZE / 2; + let red = |pixels: &[[u8; 4]]| { + let p = at(pixels, c - 50 + 12, c); + p[0] > 200 && p[2] < 60 + }; + assert!(red(&frame(&gpu, &mut view)), "the label shows"); + // Seventy more names outgrow the mark's run table of 64 texts. The table grows, and the label + // shows in every frame while it does. + data.apply(table(71)).expect("valid"); + for _ in 0..SETTLE { + assert!(red(&step(&gpu, &mut view)), "the label shows in every frame"); + if !view.needs_render() { + return; + } + } + panic!("the view settles within {SETTLE} frames"); +} diff --git a/crates/latkit/tests/layouts.rs b/crates/latkit/tests/layouts.rs index 08e71ede..ccba63f8 100644 --- a/crates/latkit/tests/layouts.rs +++ b/crates/latkit/tests/layouts.rs @@ -1,8 +1,10 @@ -//! Layouts place a diagram's vertices its data gives no position: layered in ranks along the flow, -//! nets running from their outputs, stress at graph distances with boxes parted; on a plane, points -//! among their neighbors. Vertices with positions stay where they are, places hold once given, and -//! rows added later are placed around the rest until the view arranges every row again, answering -//! with a patch that writes the places into the data. +//! Layouts position the diagram vertices whose data has no position. Layered places them in ranks +//! along a direction, with nets running from their outputs. Stress places them at graph distances +//! and separates overlapping boxes. On a plane, a layout places points among their neighbors. +//! +//! Vertices with positions stay put, and computed positions stay fixed. Rows added later are +//! placed around the rest until the view arranges every row again, which resolves to a patch that +//! writes the positions into the data. The same graph always gets the same layout. #![cfg(feature = "gpu")] @@ -12,10 +14,12 @@ use std::future::Future; use std::pin::pin; use std::task::{Context, Poll, Waker}; -use common::{diagram, errors, frame, gpu, plain_theme, step}; -use latkit::{Camera, Data, Edges, Ids, Layered, Layout, Patch, Port, Row, Scene, Side, Stress, Table, Vertices, View}; +use common::{Threads, diagram, errors, frame, gpu, plain_theme, step}; +use latkit::{ + Camera, Data, Edges, Ids, Layered, Layout, Patch, Port, Row, Scene, Side, Stress, Table, Vertices, View, field, +}; -/// Renders frames until `pending` resolves, within the settle deadline. +/// Renders frames until `pending` resolves, within `SETTLE` frames. fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future>) -> T { let mut pending = pin!(pending); let mut context = Context::from_waker(Waker::noop()); @@ -28,13 +32,13 @@ fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future) -> Data { let (x, y) = match pin { Some([x, y]) => (vec![Some(x), None, None, None], vec![Some(y), None, None, None]), None => (vec![None; 4], vec![None; 4]), }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Patch::new().set(Table::new("Node", vec![1_u64, 2, 3, 4]).field("x", x).field("y", y)).set( Table::new("Link", Ids::count(3)).reference("from", "Node", vec![1_u64, 2, 2]).reference( "to", @@ -51,36 +55,36 @@ fn scene(layout: impl Into) -> Scene { .theme(plain_theme()) .layout(layout) .draw(Edges::of("graph", "Link").ends("from", "to")) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(4.0)) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0)) } -fn show(gpu: &latkit::Gpu, scene: Scene, data: Data) -> View { +fn view(gpu: &latkit::Gpu, scene: Scene, data: &Data) -> View { let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0; 3], zoom: -2.0, pitch: 0.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([data]).expect("a source"); frame(gpu, &mut view); view } -/// Where each of nodes 1 to `count` stands, units with y down, as the view locates it on screen. +/// The positions of nodes 1 to `count`, in units with y down, from where the view draws them. fn located(gpu: &latkit::Gpu, view: &mut View, count: u64) -> Vec<[f64; 2]> { let projected = |p: [f32; 2]| { - // A quarter pixel per unit, about the middle of the view. + // A quarter pixel per unit, relative to the view's center. let c = f64::from(common::SIZE) / 2.0; [(f64::from(p[0]) - c) * 4.0, (f64::from(p[1]) - c) * 4.0] }; (1..=count) .map(|id| { - let pending = view.locate(&Row::new("graph", "Node", id)); + let pending = view.screen_of(&Row::new("graph", "Node", id)); answer(gpu, view, pending).map_or([f64::NAN; 2], projected) }) .collect() } -/// Where each node settled under `layout`, units. +/// Each node's settled position under `layout`, in units. fn placed(layout: impl Into, pin: Option<[f64; 2]>) -> Vec<[f64; 2]> { let Some(gpu) = gpu() else { return Vec::new() }; - let mut view = show(&gpu, scene(layout), graph(pin)); + let mut view = view(&gpu, scene(layout), &graph(pin)); let p = located(&gpu, &mut view, 4); assert_eq!(errors(&mut view), Vec::new()); p @@ -91,14 +95,14 @@ fn apart(a: [f64; 2], b: [f64; 2]) -> f64 { } #[test] -fn layered_ranks_follow_the_flow() { +fn layered_ranks_follow_the_direction() { let p = placed(Layered::default(), None); if p.is_empty() { return; } assert!(p.iter().all(|v| v[0].is_finite()), "every node placed and drawn: {p:?}"); - // Markers of radius 4 are 8 units across; ranks and rows of a rank stand their gaps apart edge to - // edge. + // Markers of radius 4 are 8 units across. Ranks, and rows within a rank, are their gaps apart + // edge to edge. let size = 8.0; let rank_gap = Layered::default().rank_gap; assert!((p[1][0] - p[0][0] - rank_gap - size).abs() < 1.0, "a rank apart: {p:?}"); @@ -109,9 +113,9 @@ fn layered_ranks_follow_the_flow() { #[test] fn nets_run_from_their_outputs() { - // Nodes 1 and 2 drive the net from their outputs, 3 reads it, and 4 drives 1. + // Nodes 1 and 2 drive net 1 from their outputs, node 3 reads it, and node 4 drives node 1. let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set( @@ -126,17 +130,17 @@ fn nets_run_from_their_outputs() { .expect("valid"); let scene = Scene::new(diagram()).layout(Layered::default()).draw(Edges::of("graph", "Net")).draw( Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .radius(4.0) .port("in", Side::Left) .port("out", Port::output(Side::Right)), ); - let mut view = show(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let p = located(&gpu, &mut view, 4); let gap = Layered::default().rank_gap; assert!(p[0][0] - p[3][0] > gap, "1 a rank after 4, which drives it: {p:?}"); assert!((p[1][0] - p[3][0]).abs() < 1.0, "2 drives the net 3 reads, so it shares 4's rank: {p:?}"); - assert!(p[2][0] - p[0][0] > gap, "3 reads after both drivers: {p:?}"); + assert!(p[2][0] - p[0][0] > gap, "3 ranks after both drivers: {p:?}"); assert_eq!(errors(&mut view), Vec::new()); } @@ -156,16 +160,17 @@ fn stress_puts_neighbors_a_gap_apart() { } #[test] -fn stress_parts_the_boxes_it_crowds() { +fn stress_separates_overlapping_boxes() { let Some(gpu) = gpu() else { return }; let scene = Scene::new(diagram()) .theme(plain_theme()) .layout(Stress::default()) .draw(Edges::of("graph", "Link").ends("from", "to")) - .draw(Vertices::of("graph", "Node").position("x", "y").size(60.0, 20.0)); - let mut view = show(&gpu, scene, graph(None)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).size(60.0, 20.0)); + let mut view = view(&gpu, scene, &graph(None)); let p = located(&gpu, &mut view, 4); - // Stress puts neighbors 40 units apart; their 60 by 20 boxes part, a quarter of the gap clear. + // Stress puts neighbors 40 units apart. Their 60 by 20 boxes separate, keeping a quarter of the + // gap clear. for i in 0..4 { for j in i + 1..4 { let (dx, dy) = ((p[i][0] - p[j][0]).abs(), (p[i][1] - p[j][1]).abs()); @@ -190,10 +195,11 @@ fn pinned_vertices_stay_put() { #[test] fn rows_added_later_move_nothing_else() { let Some(gpu) = gpu() else { return }; - let mut view = show(&gpu, scene(Layered::default()), graph(None)); + let data = graph(None); + let mut view = view(&gpu, scene(Layered::default()), &data); let before = located(&gpu, &mut view, 4); - // A fifth node joins the end of the chain: the four placed hold, and it is placed after them. - let mut data = view.data("graph").expect("shown").clone(); + // A fifth node joins the end of the chain. The four placed nodes stay put, and it goes after + // them. data.apply( Patch::new() .set(Table::new("Node", vec![5_u64]).field("x", vec![None::]).field("y", vec![None::])) @@ -204,14 +210,13 @@ fn rows_added_later_move_nothing_else() { )), ) .expect("valid"); - view.set_data(data); frame(&gpu, &mut view); let after = located(&gpu, &mut view, 5); for k in 0..4 { - assert!(apart(before[k], after[k]) < 1e-3, "node {} held: {before:?} → {after:?}", k + 1); + assert!(apart(before[k], after[k]) < 1e-3, "node {} stays put: {before:?} → {after:?}", k + 1); } - assert!(after[4][0] > after[2][0], "the new node after the one it follows: {after:?}"); - // Arranged, every row is laid out again, and the answer writes the places into the data. + assert!(after[4][0] > after[2][0], "the new node is placed after node 3: {after:?}"); + // Arranging lays out every row again, and its patch writes the positions into the data. let pending = view.arrange(); let patches = answer(&gpu, &mut view, pending); frame(&gpu, &mut view); @@ -219,15 +224,13 @@ fn rows_added_later_move_nothing_else() { assert!(arranged.iter().all(|p| p[0].is_finite()), "every node placed again: {arranged:?}"); let [patch] = &patches[..] else { panic!("one dataset's patch: {patches:?}") }; assert_eq!(patch.data().map(latkit::Name::as_str), Some("graph")); - let mut data = view.data("graph").expect("shown").clone(); data.apply(patch.clone()).expect("valid"); - let node = data.table("Node").expect("nodes"); - assert!(node.ids().all(|id| node.get(id, "x").is_some()), "every node has its place as data"); - view.set_data(data); + let node = data.snapshot().query(&latkit::Query::of("Node")).expect("nodes").table; + assert!(node.ids().all(|id| node.get(id, "x").is_some()), "every node's position is in the data"); frame(&gpu, &mut view); let pinned = located(&gpu, &mut view, 5); for k in 0..5 { - assert!(apart(arranged[k], pinned[k]) < 0.5, "node {} holds its place: {arranged:?} → {pinned:?}", k + 1); + assert!(apart(arranged[k], pinned[k]) < 0.5, "node {} stays put: {arranged:?} → {pinned:?}", k + 1); } assert_eq!(errors(&mut view), Vec::new()); } @@ -240,23 +243,64 @@ fn a_plane_places_points_without_positions_by_its_layout() { .theme(plain_theme()) .layout(layout) .draw(Edges::of("graph", "Link").ends("from", "to")) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(4.0)) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0)) }; - let mut view = show(&gpu, scene(Stress { gap: 40.0 }.into()), graph(Some([-100.0, 60.0]))); + let mut view = view(&gpu, scene(Stress { gap: 40.0 }.into()), &graph(Some([-100.0, 60.0]))); let p = located(&gpu, &mut view, 4); - // The plane runs y up: on screen node 1 stands 60 units above the middle. - assert!(apart(p[0], [-100.0, -60.0]) < 1.0, "node 1 where its data says: {p:?}"); + // Y points up on the plane, so on screen node 1 is 60 units above the center. + assert!(apart(p[0], [-100.0, -60.0]) < 1.0, "node 1 at its data position: {p:?}"); assert!(p.iter().all(|v| v[0].is_finite()), "every node placed: {p:?}"); assert!((apart(p[0], p[1]) - 40.0).abs() < 15.0, "node 2 a gap from node 1: {p:?}"); // Without a layout, rows without positions are not drawn. - view.show(scene(Layout::None)).expect("valid scene"); + view.set_scene(scene(Layout::None)).expect("valid scene"); frame(&gpu, &mut view); - assert!(located(&gpu, &mut view, 2)[1][0].is_nan(), "unplaced"); + assert!(located(&gpu, &mut view, 2)[1][0].is_nan(), "node 2 is not drawn"); assert_eq!(errors(&mut view), Vec::new()); } #[test] -fn layouts_answer_the_same_for_the_same_graph() { +fn a_fit_on_a_plane_frames_the_places_its_layout_gives() { + let Some(gpu) = gpu() else { return }; + // A ring of two thousand nodes, laid out in a job that lands frames later, as in a page's + // worker. Nodes 40 units apart span far beyond the 128 pixel view unless the fit waits for + // their places. + gpu.set_host(Threads); + let n = 2000_u32; + let data = Data::new("graph"); + data.apply( + Patch::new() + .set( + Table::new("Node", Ids::count(n)) + .field("x", vec![None::; n as usize]) + .field("y", vec![None::; n as usize]), + ) + .set( + Table::new("Link", Ids::count(n)) + .reference("from", "Node", (0..n).map(u64::from).collect::>()) + .reference("to", "Node", (0..n).map(|i| u64::from((i + 1) % n)).collect::>()), + ), + ) + .expect("valid"); + let scene = Scene::new(latkit::Plane::default()) + .theme(plain_theme()) + .layout(Stress { gap: 40.0 }) + .draw(Edges::of("graph", "Link").ends("from", "to")) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(1.0)); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_sources([&data]).expect("a source"); + view.fit(latkit::Fit::All); + frame(&gpu, &mut view); + let size = common::SIZE as f32; + for id in [0_u64, 500, 1000, 1500] { + let pending = view.screen_of(&Row::new("graph", "Node", id)); + let p = answer(&gpu, &mut view, pending).expect("on screen"); + assert!(p.iter().all(|v| (0.0..=size).contains(v)), "node {id} in view: {p:?}"); + } + assert_eq!(errors(&mut view), Vec::new()); +} + +#[test] +fn layouts_place_the_same_graph_the_same_way() { for layout in [Layout::from(Stress::default()), Layout::from(Layered::default())] { let (a, b) = (placed(layout, None), placed(layout, None)); assert_eq!(a, b, "{layout:?}"); diff --git a/crates/latkit/tests/markers.rs b/crates/latkit/tests/markers.rs index b19248c9..b676669f 100644 --- a/crates/latkit/tests/markers.rs +++ b/crates/latkit/tests/markers.rs @@ -1,5 +1,6 @@ -//! Markers draw what their inputs say: a gauge filled so far, a pie's slices, an icon, a custom WGSL -//! marker, and a status ring; a custom marker that does not compile is told, not drawn. +//! Markers draw from their inputs: a gauge filled to its value, a pie's slices, an icon, a custom +//! WGSL marker, and a status ring. A custom marker that fails to compile is reported as an error, +//! not drawn. #![cfg(feature = "gpu")] @@ -12,15 +13,15 @@ use latkit::{Color, Data, Event, Expr, Ids, Image, Marker, Scene, Shape, Table, fn one(marker: Marker) -> (View, latkit::Gpu) { let gpu = gpu().expect("a GPU"); - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("V", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0]).field("v", vec![0.5])) .expect("valid"); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("d", "V").position("x", "y").radius(20.0).color(Color::WHITE).marker(marker)); + .draw(Vertices::of("d", "V").position(field("x"), field("y")).radius(20.0).color(Color::WHITE).marker(marker)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); (view, gpu) } @@ -56,7 +57,7 @@ fn a_pie_cuts_slices_in_their_colors() { } #[test] -fn icons_show_the_image_their_index_picks() { +fn icons_show_the_image_at_their_index() { if common::gpu().is_none() { return; } @@ -69,7 +70,7 @@ fn icons_show_the_image_their_index_picks() { } #[test] -fn custom_markers_draw_their_wgsl_and_bad_ones_are_told() { +fn custom_markers_draw_their_wgsl_and_bad_ones_are_reported() { if common::gpu().is_none() { return; } @@ -89,20 +90,24 @@ fn custom_markers_draw_their_wgsl_and_bad_ones_are_told() { frame(&gpu, &mut view); frame(&gpu, &mut view); let told = view.events().any(|e| matches!(e, Event::Error(error) if error.to_string().contains("custom marker"))); - assert!(told, "the broken marker is told"); + assert!(told, "the broken marker is reported"); } #[test] fn status_rings_surround_their_markers() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("V", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); let scene = Scene::new(plane()).theme(plain_theme()).draw( - Vertices::of("d", "V").position("x", "y").radius(10.0).color(Color::WHITE).ring(Color::rgb(1.0, 0.0, 0.0)), + Vertices::of("d", "V") + .position(field("x"), field("y")) + .radius(10.0) + .color(Color::WHITE) + .ring(Color::rgb(1.0, 0.0, 0.0)), ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let c = SIZE / 2; let ring = at(&pixels, c + 11, c); @@ -111,28 +116,28 @@ fn status_rings_surround_their_markers() { } #[test] -fn a_missing_status_draws_no_ring_whichever_branch_it_is() { +fn a_missing_status_draws_no_ring_in_either_branch() { let Some(gpu) = gpu() else { return }; let red = Color::rgb(1.0, 0.0, 0.0); - // Only the alarmed row on the left rings. + // Only the alarmed row, on the left, gets a ring. let forms: [Expr; 2] = [ when(field("alarm").gt(0.5)).then(red).otherwise(Expr::missing()), when(field("alarm").lt(0.5)).then(Expr::missing()).otherwise(red), ]; for status in forms { - let mut data = Data::new("d"); + let data = Data::new("d"); let table = Table::new("V", Ids::count(2)).field("x", vec![-30.0, 30.0]).field("y", vec![0.0, 0.0]); data.apply(table.field("alarm", vec![1.0, 0.0])).expect("valid"); - let scene = Scene::new(plane()) - .theme(plain_theme()) - .draw(Vertices::of("d", "V").position("x", "y").radius(10.0).color(Color::WHITE).ring(status)); + let scene = Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("d", "V").position(field("x"), field("y")).radius(10.0).color(Color::WHITE).ring(status), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let c = SIZE / 2; let alarmed = at(&pixels, c - 30 + 11, c); - assert!(alarmed[0] > 200 && alarmed[1] < 80, "the alarmed row rings: {alarmed:?}"); + assert!(alarmed[0] > 200 && alarmed[1] < 80, "the alarmed row has a ring: {alarmed:?}"); let quiet = at(&pixels, c + 30 + 11, c); assert!(quiet[0] < 80, "the quiet row does not: {quiet:?}"); assert_eq!(errors(&mut view), Vec::new()); diff --git a/crates/latkit/tests/picks.rs b/crates/latkit/tests/picks.rs index 7215c5e9..f94b1bb6 100644 --- a/crates/latkit/tests/picks.rs +++ b/crates/latkit/tests/picks.rs @@ -1,5 +1,5 @@ -//! Queries answer through what the frame drew: every row within reach of a point, nearest first; the -//! rows in a box a Shift drag draws; where a row lies; and the sample a traced line passes through. +//! Queries answered from what the frame drew: every row near a point, nearest first, the rows in a +//! Shift-drag box, a row's screen position, and the sample a traced line passes through. #![cfg(feature = "gpu")] @@ -11,11 +11,11 @@ use std::task::{Context, Poll, Waker}; use common::{SIZE, camera, errors, frame, gpu, plain_theme, plane, step}; use latkit::{ - Axis, Button, Camera, Controls, Data, Device, Edges, Event, Ids, Input, Lines, Modifiers, Patch, Plot, Scene, - Table, Vertices, View, + Axis, Button, Camera, Controls, Data, Device, Edges, Event, Faces, Ids, Input, Lines, Modifiers, Patch, Plot, + Scene, Table, Vertices, View, field, }; -/// Renders frames until `pending` resolves, within the settle deadline. +/// Renders frames until `pending` resolves, within `SETTLE` frames. fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future>) -> T { let mut pending = pin!(pending); let mut context = Context::from_waker(Waker::noop()); @@ -29,7 +29,7 @@ fn answer(gpu: &latkit::Gpu, view: &mut View, pending: impl Future Data { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set( @@ -52,10 +52,10 @@ fn view(gpu: &latkit::Gpu, controls: Controls) -> View { .theme(plain_theme()) .controls(controls) .draw(Edges::of("graph", "Link").ends("from", "to").width(3.0)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(4.0)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0)); let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(graph()); + view.set_sources([graph()]).expect("a source"); frame(gpu, &mut view); view } @@ -76,11 +76,33 @@ fn picks_list_every_row_in_reach_nearest_first() { assert_eq!(errors(&mut view), Vec::new()); } +#[test] +fn a_hit_reports_the_row_and_position_from_its_frame() { + let Some(gpu) = gpu() else { return }; + let mut view = view(&gpu, Controls::default()); + let c = SIZE as f32 / 2.0; + let pending = view.pick([c, c], 6.0); + // The answering frame draws node 1 at the center. Before its answer arrives, the nodes are + // reordered and the camera moves away. + step(&gpu, &mut view); + let data = graph(); + let reordered = + Table::new("Node", vec![3_u64, 2, 1]).field("x", vec![-40.0, 10.0, 0.0]).field("y", vec![30.0, 0.0, 0.0]); + data.apply(Patch::new().replace(reordered)).expect("valid"); + view.set_sources([&data]).expect("a source"); + view.set_camera(Camera::Map { center: [500.0, 0.0, 0.0], zoom: 0.0, pitch: 0.0, bearing: 0.0 }); + let hits = answer(&gpu, &mut view, pending); + let first = hits.first().expect("a hit"); + assert_eq!(first.row, latkit::Row::new("graph", "Node", 1_u64), "{hits:?}"); + assert!(first.position[0].abs() < 1.0, "where its frame drew it: {:?}", first.position); + assert_eq!(errors(&mut view), Vec::new()); +} + #[test] fn every_hit_of_geodesic_edges_lies_in_reach() { let Some(gpu) = gpu() else { return }; - // A lattice on the globe: geodesic edges drawn through a segment list, which draws fewer segments - // than the query's list is sized for. + // A lattice on the globe. Geodesic edges draw through a segment list, which holds fewer + // segments than the query's list is sized for. let side = 24_usize; let (mut lon, mut lat, mut from, mut to) = (Vec::new(), Vec::new(), Vec::new(), Vec::new()); for i in 0..side * side { @@ -95,7 +117,7 @@ fn every_hit_of_geodesic_edges_lies_in_reach() { to.push((i + side) as u64); } } - let mut data = Data::new("grid"); + let data = Data::new("grid"); let lines = from.len(); data.apply( Patch::new().set(Table::new("Bus", Ids::count((side * side) as u32)).field("lon", lon).field("lat", lat)).set( @@ -103,23 +125,24 @@ fn every_hit_of_geodesic_edges_lies_in_reach() { ), ) .expect("valid"); - let earth = latkit::Earth { graticule: false, ..latkit::Earth::globe() }; + let earth = latkit::Earth { graticule: false, ..latkit::Earth::default().projection(latkit::Projection::Globe) }; let scene = Scene::new(earth) .theme(plain_theme()) .draw(Edges::of("grid", "Line").ends("from", "to")) - .draw(Vertices::of("grid", "Bus").position("lon", "lat")); + .draw(Vertices::of("grid", "Bus").position(field("lon"), field("lat"))); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view.fit(latkit::Fit::All); frame(&gpu, &mut view); let c = SIZE as f32 / 2.0; - // A wide pick first leaves its counts in the lists buffer; a narrow one after must not sum them. + // A wide pick leaves its counts in the lists buffer. A later narrow pick must not add to them. for radius in [40.0, 4.0] { let pending = view.pick([c, c], radius); let hits = answer(&gpu, &mut view, pending); assert!(!hits.is_empty(), "the middle of the lattice is in reach"); for hit in &hits { - // A vertex is reached at its outline and lies at its center, a marker's radius further. + // A vertex is within the radius at its outline, but its point is its center, a marker + // radius further. let reach = (hit.point[0] - c).hypot(hit.point[1] - c); let what = format!("{} {} at {:?}, {reach} away", hit.row.table, hit.row.id, hit.point); assert!(hit.distance <= radius && reach <= radius + 8.0, "{what}"); @@ -131,14 +154,50 @@ fn every_hit_of_geodesic_edges_lies_in_reach() { fn locate_finds_a_row_on_screen() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu, Controls::default()); - let pending = view.locate(&latkit::Row::new("graph", "Node", 3_u64)); + let pending = view.screen_of(&latkit::Row::new("graph", "Node", 3_u64)); let point = answer(&gpu, &mut view, pending).expect("in view"); let c = SIZE as f32 / 2.0; assert!((point[0] - (c - 40.0)).abs() < 0.5 && (point[1] - (c - 30.0)).abs() < 0.5, "{point:?}"); - let pending = view.locate(&latkit::Row::new("graph", "Node", 99_u64)); + let pending = view.screen_of(&latkit::Row::new("graph", "Node", 99_u64)); assert_eq!(answer(&gpu, &mut view, pending), None, "no such row"); } +#[test] +fn hidden_rows_have_no_screen_position() { + let Some(gpu) = gpu() else { return }; + let data = graph(); + let links = Table::new("Link", vec![0_u64, 1]) + .reference("from", "Node", vec![1_u64, 1]) + .reference("to", "Node", vec![3_u64, 2]) + .field("on", vec![false, true]); + let faces = Table::new("Face", vec![0_u64, 1]) + .reference("a", "Node", vec![1_u64, 1]) + .reference("b", "Node", vec![2_u64, 2]) + .reference("c", "Node", vec![3_u64, 3]) + .field("on", vec![false, true]); + data.apply(Patch::new().replace(links).set(faces)).expect("valid"); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .draw(Faces::of("graph", "Face").corners("a", "b", "c").visible(field("on"))) + .draw(Edges::of("graph", "Link").ends("from", "to").width(3.0).visible(field("on"))) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0)); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + frame(&gpu, &mut view); + let mut screen_of = |table: &str, id: u64| { + let pending = view.screen_of(&latkit::Row::new("graph", table, id)); + answer(&gpu, &mut view, pending) + }; + let c = SIZE as f32 / 2.0; + let link = screen_of("Link", 1).expect("a shown edge"); + assert!((link[0] - (c + 5.0)).abs() < 1.0 && (link[1] - c).abs() < 1.0, "the edge's middle: {link:?}"); + assert_eq!(screen_of("Link", 0), None, "a hidden edge"); + let face = screen_of("Face", 1).expect("a shown face"); + assert!((face[0] - (c - 10.0)).abs() < 1.0 && (face[1] - (c - 10.0)).abs() < 1.0, "the centroid: {face:?}"); + assert_eq!(screen_of("Face", 0), None, "a hidden face"); +} + #[test] fn a_shift_drag_selects_the_rows_in_its_box() { let Some(gpu) = gpu() else { return }; @@ -152,7 +211,7 @@ fn a_shift_drag_selects_the_rows_in_its_box() { frame(&gpu, &mut view); let selected: Vec> = view .events() - .filter_map(|e| if let Event::Select { rows, trigger: Some(_) } = e { Some(rows) } else { None }) + .filter_map(|e| if let Event::Selection { rows, trigger: Some(_) } = e { Some(rows) } else { None }) .collect(); let rows = selected.last().expect("a selection"); let mut ids: Vec = @@ -163,12 +222,12 @@ fn a_shift_drag_selects_the_rows_in_its_box() { } #[test] -fn hovering_a_trace_names_its_sample() { +fn picking_a_trace_returns_its_sample() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Table::new("Signal", Ids::count(1))).expect("valid"); for t in 0..=10 { - data.apply(Patch::new().append([f64::from(t)], "Signal", [("v", vec![f64::from(t) / 10.0])])).expect("frame"); + data.apply(Patch::new().append("Signal", [f64::from(t)], [("v", vec![f64::from(t) / 10.0])])).expect("frame"); } let axis = Axis { hidden: true, grid: false, ..Axis::default() }; let scene = Scene::new(Plot { x: axis.clone(), y: axis }) @@ -176,9 +235,9 @@ fn hovering_a_trace_names_its_sample() { .draw(Lines::of("signals", "Signal").trace("v").width(3.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Plot { x: [0.0, 10.0], y: Some([0.0, 1.0]) }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); - let at = view.point([7.0, 0.7, 0.0]).expect("on the plot"); + let at = view.screen([7.0, 0.7, 0.0]).expect("on the plot"); let pending = view.pick(at, 6.0); let hits = answer(&gpu, &mut view, pending); let sample = hits.first().and_then(|h| h.sample).expect("a sample"); @@ -189,7 +248,7 @@ fn hovering_a_trace_names_its_sample() { #[test] fn a_trace_keeps_large_absolute_indexes_on_the_gpu() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Table::new("Signal", Ids::count(1))).expect("valid"); let first = (1_u64 << 32) + 123; data.apply(Patch::new().frames( @@ -207,9 +266,9 @@ fn a_trace_keeps_large_absolute_indexes_on_the_gpu() { .draw(Lines::of("signals", "Signal").trace("v").width(3.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Plot { x: [0.0, 10.0], y: Some([0.0, 1.0]) }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); - let at = view.point([7.0, 0.7, 0.0]).expect("on the plot"); + let at = view.screen([7.0, 0.7, 0.0]).expect("on the plot"); let pending = view.pick(at, 6.0); let hits = answer(&gpu, &mut view, pending); let sample = hits.first().and_then(|h| h.sample).expect("a sample"); diff --git a/crates/latkit/tests/render.rs b/crates/latkit/tests/render.rs index 24295f1a..d7f36941 100644 --- a/crates/latkit/tests/render.rs +++ b/crates/latkit/tests/render.rs @@ -1,10 +1,10 @@ -//! Frames rendered headless and read back: what each mark draws lands where and how it should. +//! Frames rendered headless and read back to check where and how each mark draws. #![cfg(feature = "gpu")] mod common; -use common::{SIZE, at, camera, close, frame, gpu, plain_theme, plane, step}; +use common::{SIZE, at, camera, close, frame, gpu, green_edges, plain_theme, plane, step}; use latkit::{ Axis, Button, Camera, Color, Colormap, Data, Device, Earth, Edges, Event, Faces, Ids, Input, Lines, Modifiers, Patch, Phase, Plot, Scene, Table, Theme, Vertices, View, field, @@ -13,7 +13,7 @@ use latkit::{ #[test] fn vertices_draw_at_their_positions_in_their_colors() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Table::new("Node", Ids::count(2)) .field("x", vec![0.0, 30.0]) @@ -25,13 +25,13 @@ fn vertices_draw_at_their_positions_in_their_colors() { let blue = Color::rgb(0.0, 0.0, 1.0); let scene = Scene::new(plane()).theme(plain_theme()).draw( Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .radius(6.0) .color(latkit::when(field("kind").is("a")).then(red).otherwise(blue)), ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!(close(at(&pixels, center, center), [255, 0, 0, 255]), "{:?}", at(&pixels, center, center)); @@ -43,31 +43,89 @@ fn vertices_draw_at_their_positions_in_their_colors() { assert!(close(at(&pixels, center + 15, center), [0, 0, 0, 255]), "background between vertices"); } +#[test] +fn contains_ignores_case_and_rechecks_changed_texts() { + let Some(gpu) = gpu() else { return }; + let data = Data::new("graph"); + data.apply( + Table::new("Node", Ids::count(3)) + .field("x", vec![-30.0, 0.0, 30.0]) + .field("y", vec![0.0, 0.0, 0.0]) + .field("name", vec!["Alpha", "Beta", "Gamma"]), + ) + .expect("valid"); + let scene = Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("graph", "Node") + .position(field("x"), field("y")) + .radius(6.0) + .color(Color::rgb(1.0, 0.0, 0.0)) + .visible(field("name").contains("TA")), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + let center = SIZE / 2; + let shown = |pixels: &[[u8; 4]]| [center - 30, center, center + 30].map(|x| at(pixels, x, center)[0] > 128); + assert_eq!(shown(&frame(&gpu, &mut view)), [false, true, false], "ignoring case, only Beta contains ta"); + data.apply(Table::new("Node", [0_u64]).field("name", vec!["Delta"])).expect("valid"); + assert_eq!(shown(&frame(&gpu, &mut view)), [true, true, false], "a changed text is checked again"); +} + +#[test] +fn a_stroke_in_pixels_keeps_its_width_as_a_diagram_zooms() { + let Some(gpu) = gpu() else { return }; + let data = Data::new("graph"); + data.apply( + Patch::new() + .set(Table::new("Node", Ids::count(2)).field("x", vec![-50.0, 50.0]).field("y", vec![0.0, 0.0])) + .set(Table::new("Link", Ids::count(1)).reference("from", "Node", vec![0_u64]).reference( + "to", + "Node", + vec![1_u64], + )), + ) + .expect("valid"); + let thickness = |width: latkit::Size| { + let scene = Scene::new(common::diagram()) + .theme(green_edges()) + .draw(Edges::of("graph", "Link").ends("from", "to").width(width)) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).size(8.0, 8.0)); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [0.0; 3], zoom: 1.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + (0..SIZE).filter(|&y| at(&pixels, SIZE / 2, y)[1] > 128).count() + }; + let (px, units) = (thickness(latkit::px(2.0)), thickness(2.0.into())); + assert!((1..=3).contains(&px), "a 2 px width stays 2 pixels at zoom 1: {px}"); + assert!((4..=5).contains(&units), "a 2-unit width is 4 pixels at zoom 1: {units}"); +} + #[test] fn channels_follow_the_selection_and_the_zoom() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(2)).field("x", vec![0.0, 30.0]).field("y", vec![0.0, 0.0])) .expect("valid"); let [red, green, blue] = [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]].map(|[r, g, b]| Color::rgb(r, g, b)); let color = latkit::when(latkit::selected()).then(red).when(latkit::zoom().ge(1.0)).then(green).otherwise(blue); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(6.0).color(color)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(6.0).color(color)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let center = SIZE / 2; let pixels = frame(&gpu, &mut view); assert!(close(at(&pixels, center + 30, center), [0, 0, 255, 255]), "{:?}", at(&pixels, center + 30, center)); - view.select([latkit::Rows::new("graph", "Node", [1_u64])]); + view.set_selection([latkit::Rows::new("graph", "Node", [1_u64])]); let pixels = frame(&gpu, &mut view); assert!( close(at(&pixels, center + 30, center), [255, 0, 0, 255]), "selected: {:?}", at(&pixels, center + 30, center) ); - // What the selection leaves out dims, in its own color. + // Rows outside the selection dim, keeping their own color. let [r, g, b, _] = at(&pixels, center, center); assert!(b > 60 && r.max(g) < 20, "not selected: {:?}", [r, g, b]); view.set_camera(Camera::Map { center: [0.0; 3], zoom: 1.0, pitch: 0.0, bearing: 0.0 }); @@ -79,7 +137,7 @@ fn channels_follow_the_selection_and_the_zoom() { #[test] fn edges_join_their_ends_and_colormaps_fit_the_values() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply( Patch::new() .set( @@ -95,13 +153,16 @@ fn edges_join_their_ends_and_colormaps_fit_the_values() { )), ) .expect("valid"); - let scene = Scene::new(plane()) - .theme(Theme { edge: Some(Color::rgb(0.0, 1.0, 0.0)), ..plain_theme() }) - .draw(Edges::of("grid", "Line").ends("from", "to").width(4.0)) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(5.0).color(field("v").colormap(Colormap::GRAYS))); + let scene = + Scene::new(plane()).theme(green_edges()).draw(Edges::of("grid", "Line").ends("from", "to").width(4.0)).draw( + Vertices::of("grid", "Bus") + .position(field("x"), field("y")) + .radius(5.0) + .color(field("v").colormap(Colormap::GRAYS)), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!( @@ -124,14 +185,49 @@ fn edges_join_their_ends_and_colormaps_fit_the_values() { assert!(middle[0] > 80 && middle[0] < 130, "the middle value is grey: {middle:?}"); } +#[test] +fn a_colormap_draws_a_missing_value_in_the_themes_missing_color() { + let Some(gpu) = gpu() else { return }; + let data = Data::new("grid"); + data.apply( + Table::new("Bus", Ids::count(3)) + .field("x", vec![-40.0, 0.0, 40.0]) + .field("y", vec![0.0; 3]) + .field("v", vec![Some(0.0), None, Some(1.0)]), + ) + .expect("valid"); + let [red, blue] = [Color::rgb(1.0, 0.0, 0.0), Color::rgb(0.0, 0.0, 1.0)]; + let plain = plain_theme(); + let theme = Theme { missing: red, vertex: latkit::VertexTheme { color: blue, ..plain.vertex }, ..plain }; + let scene = |color: latkit::Expr| { + Scene::new(plane()) + .theme(theme.clone()) + .draw(Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(6.0).color(color)) + }; + let mut view = + View::new(&gpu, scene(field("v").colormap(Colormap::GRAYS).domain(0.0..=1.0).into())).expect("valid"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + let center = SIZE / 2; + assert!(close(at(&pixels, center - 40, center), [0, 0, 0, 255]), "0 maps to black"); + assert!(close(at(&pixels, center + 40, center), [255, 255, 255, 255]), "1 maps to white"); + let gap = at(&pixels, center, center); + assert!(close(gap, [255, 0, 0, 255]), "a missing value takes the theme's missing color: {gap:?}"); + // An option given `missing()` takes the mark's theme color instead. + view.set_scene(scene(latkit::Expr::missing())).expect("valid"); + let pixel = at(&frame(&gpu, &mut view), center, center); + assert!(close(pixel, [0, 0, 255, 255]), "missing() takes the vertex color: {pixel:?}"); +} + #[test] fn edges_ending_far_ahead_and_behind_the_eye_draw_whole() { let Some(gpu) = gpu() else { return }; - // A plane seen steeply from close by: from the middle one edge runs far ahead, its far end deep in - // the view, the other far back, under and past the eye. Each draws all the way across the view: - // the one ahead unbroken however differently deep its ends are, the one back cut where it leaves - // the view, not dropped. - let mut data = Data::new("grid"); + // A plane seen steeply from close by. From the center, one edge runs far ahead, its far end + // deep in the view, and the other runs far back, under and past the eye. Each draws all the + // way across the view. The one ahead is unbroken however much its ends' depths differ. The one + // behind is clipped where it leaves the view, not dropped. + let data = Data::new("grid"); data.apply( Patch::new() .set(Table::new("Bus", vec![1_u64, 2, 3]).field("x", vec![0.0; 3]).field("y", vec![0.0, 1000.0, -1000.0])) @@ -143,23 +239,23 @@ fn edges_ending_far_ahead_and_behind_the_eye_draw_whole() { ) .expect("valid"); let scene = Scene::new(plane()) - .theme(Theme { edge: Some(Color::rgb(0.0, 1.0, 0.0)), ..plain_theme() }) + .theme(green_edges()) .draw(Edges::of("grid", "Line").ends("from", "to").width(4.0)) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(1.0)); + .draw(Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(1.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0; 3], zoom: 0.0, pitch: 60.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let middle = SIZE / 2; let column: String = (0..SIZE).map(|y| if at(&pixels, middle, y)[1] > 128 { '#' } else { '.' }).collect(); - assert!(!column.contains('.'), "the edges down the middle, top to bottom: {column}"); + assert!(!column.contains('.'), "the edges run down the middle, top to bottom: {column}"); } #[test] fn great_circles_longer_than_their_list_coarsen_instead_of_stopping_short() { let Some(gpu) = gpu() else { return }; - // One edge a third of the way around the equator: sixty arcs, where its list first keeps six. - let mut data = Data::new("grid"); + // One edge a third of the way around the equator: sixty arcs, where its list first holds six. + let data = Data::new("grid"); data.apply( Patch::new() .set(Table::new("Bus", vec![1_u64, 2]).field("lon", vec![-60.0, 60.0]).field("lat", vec![0.0, 0.0])) @@ -170,15 +266,16 @@ fn great_circles_longer_than_their_list_coarsen_instead_of_stopping_short() { )), ) .expect("valid"); - let scene = Scene::new(Earth { surface: Some(Color::BLACK), graticule: false, ..Earth::mercator() }) - .theme(Theme { edge: Some(Color::rgb(0.0, 1.0, 0.0)), ..plain_theme() }) + let theme = green_edges(); + let scene = Scene::new(Earth { graticule: false, ..Earth::default() }) + .theme(Theme { earth: latkit::EarthTheme { ground: Color::BLACK, ..theme.earth }, ..theme }) .draw(Edges::of("grid", "Line").ends("from", "to").width(4.0)) - .draw(Vertices::of("grid", "Bus").position("lon", "lat").radius(1.0)); + .draw(Vertices::of("grid", "Bus").position(field("lon"), field("lat")).radius(1.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0; 3], zoom: -1.5, pitch: 0.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let green = |view: &View, pixels: &[[u8; 4]], lon: f64| { - let [x, y] = view.point([lon, 0.0, 0.0]).expect("in view").map(|v| v.round() as u32); + let [x, y] = view.screen([lon, 0.0, 0.0]).expect("in view").map(|v| v.round() as u32); let [r, g, ..] = at(pixels, x.min(SIZE - 1), y.min(SIZE - 1)); g > 150 && r < 80 }; @@ -197,28 +294,47 @@ fn great_circles_longer_than_their_list_coarsen_instead_of_stopping_short() { #[test] fn new_data_versions_show_and_fit_frames_them() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![1000.0]).field("y", vec![1000.0])).expect("valid"); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(8.0).color(Color::WHITE)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(8.0).color(Color::WHITE)); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); view.fit(latkit::Fit::All); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!(close(at(&pixels, center, center), [255, 255, 255, 255]), "fit centers the only vertex"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![1010.0])).expect("valid"); - view.set_data(data); let pixels = frame(&gpu, &mut view); assert!(close(at(&pixels, center, center), [0, 0, 0, 255]), "the vertex moved away"); } #[test] -fn a_fit_frames_what_draws_while_another_mark_has_no_rows() { +fn a_camera_set_after_a_fit_replaces_it() { let Some(gpu) = gpu() else { return }; - // The links table is empty: it has nothing to frame, and holds no fit back. - let mut data = Data::new("graph"); + let data = Data::new("graph"); + data.apply(Table::new("Node", Ids::count(1)).field("x", vec![1000.0]).field("y", vec![1000.0])).expect("valid"); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(8.0).color(Color::WHITE)); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_sources([&data]).expect("a source"); + // The fit has not framed the vertex yet when the camera is set, so the camera stays. + view.fit(latkit::Fit::All); + let set = camera(); + view.set_camera(set); + let pixels = frame(&gpu, &mut view); + assert_eq!(*view.camera(), set, "the later camera wins"); + let center = SIZE / 2; + assert!(close(at(&pixels, center, center), [0, 0, 0, 255]), "the vertex stays out of view"); +} + +#[test] +fn a_mark_with_no_rows_does_not_delay_a_fit() { + let Some(gpu) = gpu() else { return }; + // The links table is empty. It has nothing to frame and does not delay the fit. + let data = Data::new("graph"); data.apply( Patch::new().set(Table::new("Node", Ids::count(1)).field("x", vec![1000.0]).field("y", vec![1000.0])).set( Table::new("Link", Ids::count(0)).reference("from", "Node", Vec::::new()).reference( @@ -232,30 +348,31 @@ fn a_fit_frames_what_draws_while_another_mark_has_no_rows() { let scene = Scene::new(plane()) .theme(plain_theme()) .draw(Edges::of("graph", "Link").ends("from", "to").width(2.0)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(8.0).color(Color::WHITE)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(8.0).color(Color::WHITE)); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view.fit(latkit::Fit::All); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!(close(at(&pixels, center, center), [255, 255, 255, 255]), "fit centers the only vertex"); - assert!(view.complete(), "waiting on {}", view.stats().waiting); + assert!(view.complete(), "waiting on {}", view.waiting()); } #[test] -fn a_fit_frames_data_whatever_its_units_and_its_markers_pixels() { +fn a_fit_frames_data_regardless_of_units_and_marker_sizes() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(2)).field("x", vec![0.0, 1.0]).field("y", vec![0.0, 1.0])).expect("valid"); - let scene = Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position("x", "y")); + let scene = + Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position(field("x"), field("y"))); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view.fit(latkit::Fit::All); for _ in 0..4 { frame(&gpu, &mut view); } - // Markers a few pixels across leave a unit square filling the view, not lost in its middle. - let [a, b] = [[0.0, 0.0, 0.0], [1.0, 1.0, 0.0]].map(|p| view.point(p).expect("in view")); + // Markers a few pixels across do not shrink the fit: the unit square fills most of the view. + let [a, b] = [[0.0, 0.0, 0.0], [1.0, 1.0, 0.0]].map(|p| view.screen(p).expect("in view")); let across = (b[0] - a[0]).abs().min((b[1] - a[1]).abs()); assert!(across > SIZE as f32 * 0.5, "the unit square spans {across} of {SIZE} pixels"); } @@ -263,33 +380,34 @@ fn a_fit_frames_data_whatever_its_units_and_its_markers_pixels() { #[test] fn sampled_fields_read_at_the_shown_coordinate() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("grid"); + let data = Data::new("grid"); data.apply(Table::new("Bus", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); - data.apply(Patch::new().append([0.0], "Bus", [("v", vec![0.0])]).append([10.0], "Bus", [("v", vec![1.0])])) + data.apply(Patch::new().append("Bus", [0.0], [("v", vec![0.0])]).append("Bus", [10.0], [("v", vec![1.0])])) .expect("frames"); let gray = |v: latkit::Expr| v.colormap(Colormap::GRAYS).domain(0.0..=1.0); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(8.0).color(gray(field("v")))); + .draw(Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(8.0).color(gray(field("v")))); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let center = SIZE / 2; // Halfway is OKLab's half lightness: about 100. for (t, want) in [(0.0, 0), (10.0, 255), (5.0, 100), (20.0, 255), (-1.0, 0)] { view.set_at(t); let pixel = at(&frame(&gpu, &mut view), center, center); if t < 0.0 { - // Before the first frame there is no value: the vertex takes the theme's color. - assert_ne!(pixel, [0, 0, 0, 255], "at {t}"); + // Before the first frame there is no value, so the vertex takes the theme's missing + // color. + assert!(close(pixel, plain_theme().missing.to_rgba8()), "at {t}: {pixel:?}"); } else { assert!(pixel[0].abs_diff(want) <= 3, "at {t}: {pixel:?}"); } } - view.show( - Scene::new(plane()) - .theme(plain_theme()) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(8.0).color(gray(field("v").step()))), + view.set_scene( + Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(8.0).color(gray(field("v").step())), + ), ) .expect("valid scene"); view.set_at(5.0); @@ -298,9 +416,9 @@ fn sampled_fields_read_at_the_shown_coordinate() { } #[test] -fn faces_blend_their_corners_and_hide_what_lies_behind() { +fn faces_blend_their_corners_and_hide_vertices_behind_them() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("mesh"); + let data = Data::new("mesh"); data.apply( Patch::new() .set( @@ -322,14 +440,14 @@ fn faces_blend_their_corners_and_hide_what_lies_behind() { let blue = Color::rgb(0.0, 0.0, 1.0); let scene = Scene::new(plane()).theme(plain_theme()).draw(Faces::of("mesh", "Face").corners("a", "b", "c")).draw( Vertices::of("mesh", "Vertex") - .position("x", "y") - .elevation("z") + .position(field("x"), field("y")) + .elevation(field("z")) .radius(4.0) .color(latkit::when(field("hue").gt(0.5)).then(blue).otherwise(red)), ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!(close(at(&pixels, center - 10, center + 10), [255, 0, 0, 255]), "a face seen from above shows its color"); @@ -340,15 +458,15 @@ fn faces_blend_their_corners_and_hide_what_lies_behind() { #[test] fn presses_click_and_drag_the_rows_they_hit() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", vec![7_u64, 8]).field("x", vec![0.0, 40.0]).field("y", vec![0.0, 0.0])) .expect("valid"); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(6.0).drag(true)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(6.0).drag(true)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); let center = SIZE as f32 / 2.0; let modifiers = Modifiers::default(); @@ -362,7 +480,7 @@ fn presses_click_and_drag_the_rows_they_hit() { let events: Vec = view.events().collect(); let clicked = events.iter().find_map(|e| if let Event::Click { hit, .. } = e { Some(hit) } else { None }); assert_eq!(clicked.map(|hit| hit.row.id.clone()), Some(8_u64.into()), "{events:?}"); - assert!(events.iter().any(|e| matches!(e, Event::Select { rows, trigger: Some(_) } if rows.len() == 1))); + assert!(events.iter().any(|e| matches!(e, Event::Selection { rows, trigger: Some(_) } if rows.len() == 1))); press(&mut view, [center, center]); frame(&gpu, &mut view); @@ -374,17 +492,17 @@ fn presses_click_and_drag_the_rows_they_hit() { .collect(); assert_eq!(drags.iter().map(|d| d.0).collect::>(), [Phase::Start, Phase::End]); let end = drags.last().expect("an end").1; - assert!((end[0] - 10.0).abs() < 0.5 && (end[1] - 20.0).abs() < 0.5, "the pointer's position: {end:?}"); + assert!((end[0] - 10.0).abs() < 0.5 && (end[1] - 20.0).abs() < 0.5, "the drag ends at the pointer: {end:?}"); assert_eq!(view.camera(), &camera(), "dragging a row does not pan"); } #[test] fn lines_trace_their_samples_across_a_plot() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Table::new("Signal", Ids::count(1))).expect("valid"); for t in 0..=10 { - data.apply(Patch::new().append([f64::from(t)], "Signal", [("v", vec![0.5])])).expect("frame"); + data.apply(Patch::new().append("Signal", [f64::from(t)], [("v", vec![0.5])])).expect("frame"); } let axis = Axis { grid: false, hidden: true, ..Axis::default() }; let scene = Scene::new(Plot { x: axis.clone(), y: axis }) @@ -392,7 +510,7 @@ fn lines_trace_their_samples_across_a_plot() { .draw(Lines::of("signals", "Signal").trace("v").color(Color::rgb(1.0, 0.0, 0.0)).width(4.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Plot { x: [0.0, 10.0], y: Some([0.0, 1.0]) }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!(close(at(&pixels, center, center), [255, 0, 0, 255]), "{:?}", at(&pixels, center, center)); @@ -402,11 +520,11 @@ fn lines_trace_their_samples_across_a_plot() { #[test] fn lines_traced_in_priority_order_draw_their_whole_span() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Patch::new().axis("time", "s").set(Table::new("Signal", Ids::count(1)).field("rank", vec![1.0]))) .expect("valid"); for t in 0..=10 { - data.apply(Patch::new().append([f64::from(t)], "Signal", [("v", vec![0.5])])).expect("frame"); + data.apply(Patch::new().append("Signal", [f64::from(t)], [("v", vec![0.5])])).expect("frame"); } let axis = Axis { grid: false, hidden: true, ..Axis::default() }; let scene = Scene::new(Plot { x: axis.clone(), y: axis }).theme(plain_theme()).draw( @@ -415,27 +533,27 @@ fn lines_traced_in_priority_order_draw_their_whole_span() { let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Plot { x: [0.0, 10.0], y: Some([0.0, 1.0]) }); view.set_at(0.0); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; - // Ordered by priority, the table draws in part; its trace still spans every frame, not only - // those about the coordinate shown, at its start. + // Ordered by priority, the table is drawn as a paged table. Its trace still spans every frame, + // not only those near the shown coordinate at its start. for x in [center - 40, center, center + 40] { assert!(close(at(&pixels, x, center), [255, 0, 0, 255]), "at {x}: {:?}", at(&pixels, x, center)); } } #[test] -fn a_plot_frames_its_traces_whole_span_wherever_it_shows() { +fn a_plot_fit_frames_the_whole_trace_at_any_shown_coordinate() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Table::new("Signal", Ids::count(1))).expect("valid"); for t in 0..=10 { - data.apply(Patch::new().append([f64::from(t) * 6.0], "Signal", [("v", vec![f64::from(t)])])).expect("frame"); + data.apply(Patch::new().append("Signal", [f64::from(t) * 6.0], [("v", vec![f64::from(t)])])).expect("frame"); } let scene = Scene::new(Plot::default()).draw(Lines::of("signals", "Signal").trace("v")); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view.set_at(0.0); view.fit(latkit::Fit::All); frame(&gpu, &mut view); @@ -446,19 +564,18 @@ fn a_plot_frames_its_traces_whole_span_wherever_it_shows() { #[test] fn lines_wait_for_the_first_frame_of_the_field_they_trace() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Patch::new().axis("time", "s").set(Table::new("Signal", Ids::count(1)))).expect("valid"); let scene = Scene::new(Plot::default()) .theme(plain_theme()) .draw(Lines::of("signals", "Signal").trace("v").color(Color::rgb(1.0, 0.0, 0.0)).width(4.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); assert_eq!(common::errors(&mut view), Vec::new(), "a field not recorded yet is no error"); for t in 0..3 { - data.apply(Patch::new().append([f64::from(t)], "Signal", [("v", vec![f64::from(t)])])).expect("valid"); + data.apply(Patch::new().append("Signal", [f64::from(t)], [("v", vec![f64::from(t)])])).expect("valid"); } - view.set_data(data); view.set_camera(Camera::Plot { x: [0.0, 2.0], y: Some([0.0, 2.0]) }); let pixels = frame(&gpu, &mut view); assert!(pixels.iter().any(|p| p[0] > 200 && p[1] < 60), "the trace draws once recorded"); @@ -469,7 +586,7 @@ fn lines_wait_for_the_first_frame_of_the_field_they_trace() { fn new_data_and_the_scene_reading_it_change_together() { let Some(gpu) = gpu() else { return }; let dataset = |field_name: &str| { - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply( Table::new("T", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0]).field(field_name, vec![1.0]), ) @@ -479,56 +596,57 @@ fn new_data_and_the_scene_reading_it_change_together() { let reading = |field_name: &str| { Scene::new(plane()).theme(plain_theme()).draw( Vertices::of("d", "T") - .position("x", "y") + .position(field("x"), field("y")) .radius(6.0) .color(field(field_name).colormap(Colormap::GRAYS).domain(0.0..=1.0)), ) }; let mut view = View::new(&gpu, reading("a")).expect("valid scene"); view.set_camera(camera()); - view.set_data(dataset("a")); + view.set_sources([dataset("a")]).expect("a source"); frame(&gpu, &mut view); - // Data the mark no longer fits, then the scene that fits it, before the frame: nothing to report. - view.set_data(dataset("b")); - view.show(reading("b")).expect("valid scene"); + // Data the mark no longer fits, then a scene that fits it, both before the frame: no error. + view.set_sources([dataset("b")]).expect("a source"); + view.set_scene(reading("b")).expect("valid scene"); let pixels = frame(&gpu, &mut view); assert_eq!(common::errors(&mut view), Vec::new()); let center = SIZE / 2; assert!(at(&pixels, center, center)[0] > 200, "the new mark draws: {:?}", at(&pixels, center, center)); - // Data no mark fits by the frame is told then, once, and listed until data fits it again. - view.set_data(dataset("a")); + // Data that no mark fits by the frame is reported then, once, and listed until data fits again. + view.set_sources([dataset("a")]).expect("a source"); frame(&gpu, &mut view); let errors = common::errors(&mut view); assert!(errors.len() == 1 && errors[0].to_string().contains("no field b"), "{errors:?}"); - assert_eq!(view.failures().collect::>(), errors, "listed as it was told"); + assert_eq!(view.failures().collect::>(), errors, "failures() lists the reported error"); view.set_camera(Camera::Map { center: [5.0, 0.0, 0.0], zoom: 0.0, pitch: 0.0, bearing: 0.0 }); frame(&gpu, &mut view); - assert_eq!(common::errors(&mut view), Vec::new(), "told once"); - view.set_data(dataset("b")); + assert_eq!(common::errors(&mut view), Vec::new(), "reported once"); + view.set_sources([dataset("b")]).expect("a source"); frame(&gpu, &mut view); - assert_eq!(view.failures().count(), 0, "data that fits clears it"); + assert_eq!(view.failures().count(), 0, "data that fits clears the failure"); } #[test] -fn a_dataset_not_given_is_waited_on_and_a_table_it_lacks_fails() { +fn a_missing_dataset_is_waited_for_and_a_missing_table_fails() { let Some(gpu) = gpu() else { return }; - let scene = Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("d", "Bus").position("x", "y")); + let scene = + Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("d", "Bus").position(field("x"), field("y"))); let mut view = View::new(&gpu, scene).expect("valid scene"); frame(&gpu, &mut view); - assert!(view.stats().waiting.data, "waits for the dataset it names"); - let mut data = Data::new("d"); + assert!(view.waiting().data, "waits for the dataset it names"); + let data = Data::new("d"); data.apply(Table::new("Buses", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); let errors = common::errors(&mut view); assert!(errors.len() == 1 && errors[0].to_string().contains("no table 'Bus'"), "{errors:?}"); - assert!(view.complete(), "a mark that failed stands in for nothing"); + assert!(view.complete(), "a failed mark leaves nothing to wait on"); } #[test] -fn hover_reaches_what_faces_leave_in_view() { +fn hover_skips_vertices_hidden_behind_faces() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("mesh"); + let data = Data::new("mesh"); data.apply( Patch::new() .set( @@ -548,17 +666,23 @@ fn hover_reaches_what_faces_leave_in_view() { let scene = Scene::new(plane()) .theme(plain_theme()) .draw(Faces::of("mesh", "Face").corners("a", "b", "c").pick(false)) - .draw(Vertices::of("mesh", "Vertex").position("x", "y").elevation("z").radius(3.0).drag(true)); + .draw( + Vertices::of("mesh", "Vertex") + .position(field("x"), field("y")) + .elevation(field("z")) + .radius(3.0) + .drag(true), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); let center = SIZE as f32 / 2.0; let hover = |view: &mut View, point: [f32; 2]| { let modifiers = Modifiers::default(); view.input(&Input::Move { pointer: 1, point, modifiers, device: Device::Mouse }, common::now()); frame(&gpu, view); - view.events().filter_map(|e| if let Event::Hover { hit, .. } = e { Some(hit) } else { None }).last() + view.events().filter_map(|e| if let Event::Hovered { hit, .. } = e { Some(hit) } else { None }).last() }; let above = hover(&mut view, [center + 20.0, center]).expect("a hover event"); assert_eq!(above.map(|hit| hit.row.id), Some(4_u64.into()), "the vertex above the face"); @@ -575,7 +699,7 @@ fn a_streamed_tilted_mesh_answers_hover_and_drags() { let faces = [[0, 1, 2], [0, 2, 3], [0, 3, 4], [0, 4, 1], [5, 2, 1], [5, 3, 2], [5, 4, 3], [5, 1, 4]]; let column = |k: usize| corners.iter().map(|c| c[k]).collect::>(); let corner = |k: usize| faces.iter().map(|f| f[k] as u64).collect::>(); - let mut data = Data::new("mesh"); + let data = Data::new("mesh"); data.apply( Patch::new() .set(Table::new("Vertex", Ids::count(6))) @@ -585,23 +709,30 @@ fn a_streamed_tilted_mesh_answers_hover_and_drags() { .reference("b", "Vertex", corner(1)) .reference("c", "Vertex", corner(2)), ) - .append([0.0], "Vertex", [("x", column(0)), ("y", column(1)), ("z", column(2))]), + .append("Vertex", [0.0], [("x", column(0)), ("y", column(1)), ("z", column(2))]), ) .expect("valid"); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Faces::of("mesh", "Face").corners("a", "b", "c").pick(false).shadow(true)) - .draw(Vertices::of("mesh", "Vertex").position("x", "y").elevation("z").radius(2.0).select(false).drag(true)); + .draw(Faces::of("mesh", "Face").corners("a", "b", "c").pick(false)) + .draw( + Vertices::of("mesh", "Vertex") + .position(field("x"), field("y")) + .elevation(field("z")) + .radius(2.0) + .select(false) + .drag(true), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0; 3], zoom: 0.0, pitch: 58.0, bearing: 30.0 }); view.fit(latkit::Fit::Bounds { min: [-2.0, -2.0, 0.0], max: [2.0, 2.0, 2.0] }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); - let top = view.point([0.0, 0.0, 2.0]).expect("the top is in view"); + let top = view.screen([0.0, 0.0, 2.0]).expect("the top is in view"); let (modifiers, device) = (Modifiers::default(), Device::Mouse); view.input(&Input::Move { pointer: 1, point: top, modifiers, device }, common::now()); frame(&gpu, &mut view); - let hovered = view.events().find_map(|e| if let Event::Hover { hit, .. } = e { hit } else { None }); + let hovered = view.events().find_map(|e| if let Event::Hovered { hit, .. } = e { hit } else { None }); assert_eq!(hovered.map(|hit| hit.row.id), Some(5_u64.into()), "the top vertex is hovered"); assert_eq!(view.hovered().map(|hit| hit.row.id.clone()), Some(5_u64.into()), "and reads as hovered"); assert_eq!(view.cursor(), latkit::Cursor::Grab); @@ -618,13 +749,13 @@ fn a_streamed_tilted_mesh_answers_hover_and_drags() { #[test] fn long_histories_draw_from_their_levels_of_detail() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Table::new("Signal", Ids::count(1))).expect("valid"); // Twenty thousand frames on a 128 pixel plot: buckets of 64 frames, with a spike in one. let mut patch = Patch::new(); for t in 0..20_000 { let v = if t == 10_000 { 0.9 } else { 0.1 }; - patch = patch.append([f64::from(t)], "Signal", [("v", vec![v])]); + patch = patch.append("Signal", [f64::from(t)], [("v", vec![v])]); } data.apply(patch).expect("frames"); let axis = Axis { grid: false, hidden: true, ..Axis::default() }; @@ -633,29 +764,30 @@ fn long_histories_draw_from_their_levels_of_detail() { .draw(Lines::of("signals", "Signal").trace("v").color(Color::rgb(1.0, 0.0, 0.0)).width(2.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Plot { x: [0.0, 20_000.0], y: Some([0.0, 1.0]) }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); - let spike = view.point([10_000.0, 0.85, 0.0]).expect("on the plot"); + let spike = view.screen([10_000.0, 0.85, 0.0]).expect("on the plot"); let pixel = at(&pixels, spike[0].round() as u32, spike[1].round() as u32); assert!(pixel[0] > 150, "the one-frame spike survives in its bucket: {pixel:?}"); - let base = view.point([5_000.0, 0.1, 0.0]).expect("on the plot"); + let base = view.screen([5_000.0, 0.1, 0.0]).expect("on the plot"); assert!(at(&pixels, base[0].round() as u32, base[1].round() as u32)[0] > 150, "and the level line draws"); let stats = view.stats(); assert_eq!(stats.rows, 1); } #[test] -fn frames_tell_what_they_cost_the_gpu_where_it_measures() { +fn frames_report_gpu_time_where_timestamps_are_supported() { let Some(gpu) = gpu() else { return }; if !gpu.device().features().contains(wgpu::Features::TIMESTAMP_QUERY) { return; } - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); - let scene = Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position("x", "y")); + let scene = + Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position(field("x"), field("y"))); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); for _ in 0..common::SETTLE { if view.stats().gpu_ms > 0.0 { break; @@ -663,8 +795,8 @@ fn frames_tell_what_they_cost_the_gpu_where_it_measures() { common::finish(&gpu); step(&gpu, &mut view); } - // Milliseconds, measured: no frame passes the deadline `finish` holds it to, even on a CPU - // renderer under load, while a count of nanoseconds would. + // The value is in milliseconds. No frame exceeds the deadline `finish` enforces, even on a + // loaded CPU renderer, whereas a count of nanoseconds would. let ms = view.stats().gpu_ms; assert!(ms > 0.0 && ms < common::DEADLINE.as_secs_f32() * 1000.0, "{ms} ms"); } @@ -672,13 +804,14 @@ fn frames_tell_what_they_cost_the_gpu_where_it_measures() { #[test] fn frames_count_their_draws_and_instances() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(3)).field("x", vec![-1.0, 0.0, 1.0]).field("y", vec![0.0; 3])) .expect("valid"); - let scene = Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position("x", "y")); + let scene = + Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position(field("x"), field("y"))); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); let stats = view.stats(); // The backdrop's one triangle, then the mark's three points. @@ -689,36 +822,38 @@ fn frames_count_their_draws_and_instances() { #[test] fn dashes_run_on_across_a_paths_segments() { let Some(gpu) = gpu() else { return }; - // A path of a hundred segments a pixel long, dashed. Were each segment to start its dashes - // afresh, each would show only the blank first half of a dash, and the path nothing. + // A dashed path of a hundred segments, each a pixel long. If each segment restarted its dashes, + // each would show only a dash's blank first half, and the path would show nothing. let points: Vec<[f64; 2]> = (0..=100).map(|i| [f64::from(i) - 50.0, 0.0]).collect(); - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("Path", Ids::count(1)).field("geometry", vec![points])).expect("valid"); let scene = Scene::new(plane()).theme(plain_theme()).draw(Lines::of("d", "Path").width(3.0).dash(true).color(Color::WHITE)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let c = SIZE / 2; let on = (c - 40..c + 40).filter(|&x| at(&pixels, x, c)[0] > 128).count(); - assert!(on > 20 && on < 60, "about half of the path dashes: {on} of 80 pixels"); + assert!(on > 20 && on < 60, "dashes cover about half the path: {on} of 80 pixels"); } -/// A view of a gray line a pixel wide from x -50 to 50, its comets red and 8 pixels wide, 30 apart -/// with tails of 8, in position units when `units`; flowing at `speed`, or not at all. +/// A view of a 1-pixel gray line from x -50 to 50, with red comets 8 pixels wide, spaced 30 apart +/// with tails of 8 (in position units if `units`). Comets flow at `speed`, or not at all if `None`. fn comets(gpu: &latkit::Gpu, units: bool, speed: Option) -> View { - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("Path", Ids::count(1)).field("geometry", vec![vec![[-50.0, 0.0], [50.0, 0.0]]])) .expect("valid"); let mut view = View::new(gpu, comets_scene(units, speed)).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view } fn comets_scene(units: bool, speed: Option) -> Scene { - let flow = latkit::Flow { color: Color::rgb(1.0, 0.0, 0.0), spacing: 30.0, length: 8.0, width: 8.0, units }; + let length = |v| if units { latkit::Length::Units(v) } else { latkit::Length::Space(v) }; + let flow = + latkit::FlowTheme { color: Color::rgb(1.0, 0.0, 0.0), spacing: length(30.0), length: length(8.0), width: 8.0 }; let line = Lines::of("d", "Path").width(1.0).color(Color::rgb(0.25, 0.25, 0.25)); let line = match speed { Some(speed) => line.flow(speed), @@ -727,7 +862,7 @@ fn comets_scene(units: bool, speed: Option) -> Scene { Scene::new(plane()).theme(Theme { flow, ..plain_theme() }).ease(std::time::Duration::ZERO).draw(line) } -/// Where comets' heads are along row `y`, CSS pixels: the reddest pixel of each run of red. +/// The x of each comet head along pixel row `y`: the reddest pixel of each run of red. fn heads(pixels: &[[u8; 4]], y: u32) -> Vec { let red = |x: u32| at(pixels, x, y)[0]; let mut heads = Vec::new(); @@ -752,13 +887,13 @@ fn comets_run_wider_than_their_line() { let mut view = comets(&gpu, false, Some(40.0)); let c = SIZE / 2; let red = |pixels: &[[u8; 4]], y: u32| (c - 45..c + 45).filter(|&x| at(pixels, x, y)[0] > 100).count(); - // Comets never settle: a few frames draw them. + // Comets never settle, so render a few frames instead of waiting. let pixels = (0..3).map(|_| step(&gpu, &mut view)).last().expect("frames"); assert!(view.needs_render(), "comets travel every frame"); let beside = red(&pixels, c + 2); assert!(beside > 0 && beside < 60, "comets reach past the line, spaced apart: {beside} of 90 pixels"); assert_eq!(red(&pixels, c + 6), 0, "and no further than their width"); - view.show(comets_scene(false, None)).expect("valid scene"); + view.set_scene(comets_scene(false, None)).expect("valid scene"); assert_eq!(red(&frame(&gpu, &mut view), c + 2), 0, "a line without flow has no comets"); } @@ -769,39 +904,44 @@ fn comets_carry_on_from_where_they_are_when_their_flow_changes() { for _ in 0..3 { step(&gpu, &mut view); } - // Long after the view began, as a page left running is, a pause moves them a tenth of a second. + // Long after the view began, as with a page left open, a long pause moves them only a tenth + // of a second's travel. common::wait(100_000.0); let paused = heads(&step(&gpu, &mut view), SIZE / 2 + 2); let before = heads(&step(&gpu, &mut view), SIZE / 2 + 2); let moved = |a: &[u32], b: &[u32]| a.iter().zip(b).map(|(a, b)| a.abs_diff(*b)).max().unwrap_or(99); assert!(!before.is_empty() && moved(&paused, &before) <= 2, "{paused:?} then {before:?}"); - view.show(comets_scene(false, Some(41.0))).expect("valid scene"); + view.set_scene(comets_scene(false, Some(41.0))).expect("valid scene"); let after = heads(&step(&gpu, &mut view), SIZE / 2 + 2); assert_eq!(before.len(), after.len(), "{before:?} then {after:?}"); - assert!(moved(&before, &after) <= 2, "a new speed moves comets a frame's way: {before:?} then {after:?}"); + assert!(moved(&before, &after) <= 2, "a new speed moves comets one frame's travel: {before:?} then {after:?}"); } #[test] fn comets_follow_their_rows_when_rows_move_without_easing() { let Some(gpu) = gpu() else { return }; - // Two lines whose comets travel at different speeds, so each row's travel is its own. - let mut data = Data::new("d"); + // Two lines whose comets travel at different speeds, so each row's travel distance differs. + let data = Data::new("d"); let paths = vec![vec![[-50.0, 30.0], [50.0, 30.0]], vec![[-50.0, 0.0], [50.0, 0.0]]]; data.apply(Table::new("Path", vec!["a", "b"]).field("geometry", paths).field("speed", vec![17.0, 40.0])) .expect("valid"); - let flow = latkit::Flow { color: Color::rgb(1.0, 0.0, 0.0), spacing: 30.0, length: 8.0, width: 8.0, units: false }; + let flow = latkit::FlowTheme { + color: Color::rgb(1.0, 0.0, 0.0), + spacing: latkit::Length::Space(30.0), + length: latkit::Length::Space(8.0), + width: 8.0, + }; let line = Lines::of("d", "Path").width(1.0).color(Color::rgb(0.25, 0.25, 0.25)).flow(field("speed")); let scene = Scene::new(plane()).theme(Theme { flow, ..plain_theme() }).ease(std::time::Duration::ZERO).draw(line); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); for _ in 0..40 { step(&gpu, &mut view); } let before = heads(&step(&gpu, &mut view), SIZE / 2 + 2); - // Row b moves up to row 0 at once: its comets carry on from where they were, not from a's. + // Row b moves to row 0 at once. Its comets continue from where they were, not from a's. data.apply(Patch::new().remove("Path", vec!["a"])).expect("valid"); - view.set_data(data); let after = heads(&step(&gpu, &mut view), SIZE / 2 + 2); let moved = before.iter().zip(&after).map(|(a, b)| a.abs_diff(*b)).max().unwrap_or(99); assert!(!before.is_empty() && before.len() == after.len() && moved <= 2, "{before:?} then {after:?}"); @@ -811,7 +951,7 @@ fn comets_follow_their_rows_when_rows_move_without_easing() { fn comets_in_units_stay_on_their_lines_as_the_view_zooms() { let Some(gpu) = gpu() else { return }; let c = f64::from(SIZE / 2); - // Where comets' heads stand in position units at zoom 0 and zoom 1, so slow they stand still. + // Comet heads in position units at zoom 0 and zoom 1, moving so slowly they stay still. let heads_at = |units: bool| { let mut view = comets(&gpu, units, Some(1e-4)); [0.0, 1.0].map(|zoom: f64| { @@ -830,35 +970,54 @@ fn comets_in_units_stay_on_their_lines_as_the_view_zooms() { } #[test] -fn sizes_in_units_keep_within_their_pixels() { +fn sizes_in_units_clamp_to_their_pixel_limits() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); - // Pixels across a red marker through the middle, at zoom 0: a unit a pixel. + // The width in pixels of a red marker through the center, at zoom 0 (one unit per pixel). let across = |radius: latkit::Size| { - let marker = Vertices::of("d", "Node").position("x", "y").radius(radius).color(Color::rgb(1.0, 0.0, 0.0)); + let marker = + Vertices::of("d", "Node").position(field("x"), field("y")).radius(radius).color(Color::rgb(1.0, 0.0, 0.0)); let mut view = View::new(&gpu, Scene::new(plane()).theme(plain_theme()).draw(marker)).expect("valid scene"); view.set_camera(camera()); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); (0..SIZE).filter(|&x| at(&pixels, x, SIZE / 2)[0] > 128).count() }; - assert!(across(latkit::units(0.01)) <= 1, "a hundredth of a unit is no pixel at all"); + assert!(across(latkit::units(0.01)) <= 1, "a hundredth of a unit covers at most a pixel"); let held = across(latkit::units(0.01).at_least(5.0)); assert!((9..=11).contains(&held), "a radius of at least 5 pixels: {held} across"); let capped = across(latkit::units(1000.0).at_most(6.0)); assert!((11..=13).contains(&capped), "a radius of at most 6 pixels: {capped} across"); } +#[test] +fn sizes_in_units_are_degrees_on_the_earth() { + let Some(gpu) = gpu() else { return }; + let data = Data::new("d"); + data.apply(Table::new("Node", Ids::count(1)).field("lon", vec![0.0]).field("lat", vec![0.0])).expect("valid"); + let marker = + Vertices::of("d", "Node").position(field("lon"), field("lat")).radius(latkit::units(0.25)).color(Color::WHITE); + let ground = Theme { earth: latkit::EarthTheme { ground: Color::BLACK, ..plain_theme().earth }, ..plain_theme() }; + let scene = Scene::new(Earth { graticule: false, ..Earth::default() }).theme(ground).draw(marker); + let mut view = View::new(&gpu, scene).expect("valid scene"); + // At zoom 6 the world is 256 · 2⁶ pixels around, so a degree spans 45.5 pixels. + view.set_camera(Camera::Map { center: [0.0; 3], zoom: 6.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + let across = (0..SIZE).filter(|&x| at(&pixels, x, SIZE / 2)[0] > 128).count(); + assert!((21..=25).contains(&across), "a radius of a quarter degree is 23 pixels across: {across}"); +} + #[test] fn poles_stand_from_each_vertex_down_to_the_ground() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0]).field("z", vec![40.0])) .expect("valid"); let scene = |poles: bool| { let vertices = Vertices::of("d", "Node") - .position("x", "y") + .position(field("x"), field("y")) .elevation(field("z")) .radius(3.0) .color(Color::rgb(0.0, 0.0, 1.0)); @@ -870,7 +1029,7 @@ fn poles_stand_from_each_vertex_down_to_the_ground() { let mut view = View::new(&gpu, scene(poles)).expect("valid scene"); // Tilted, so a pole stands up the screen. view.set_camera(Camera::Map { center: [0.0, 0.0, 20.0], zoom: 0.0, pitch: 60.0, bearing: 0.0 }); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); (0..SIZE).filter(|&y| (SIZE / 2 - 2..SIZE / 2 + 3).any(|x| at(&pixels, x, y)[0] > 128)).count() }; @@ -880,9 +1039,9 @@ fn poles_stand_from_each_vertex_down_to_the_ground() { } #[test] -fn nets_off_a_diagram_join_their_members_as_stars() { +fn nets_outside_a_diagram_join_their_members_as_stars() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("d"); + let data = Data::new("d"); data.apply( Patch::new().set(Table::new("Net", Ids::count(1))).set( Table::new("Bus", Ids::count(3)) @@ -892,15 +1051,15 @@ fn nets_off_a_diagram_join_their_members_as_stars() { ), ) .expect("valid"); - let buses = Vertices::of("d", "Bus").position("x", "y").radius(2.0).color(Color::rgb(0.0, 0.0, 1.0)); + let buses = Vertices::of("d", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(0.0, 0.0, 1.0)); let net = Edges::of("d", "Net").joins("Bus", "net").width(2.0).color(Color::rgb(1.0, 0.0, 0.0)); let mut view = View::new(&gpu, Scene::new(plane()).theme(plain_theme()).draw(net).draw(buses)).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let red = |x: u32, y: u32| at(&pixels, x, y)[0] > 128; - // The members' middle is the origin: the spokes meet there, and run out to each member. + // The members' center is the origin. The spokes meet there and run out to each member. let c = SIZE / 2; assert!(red(c, c), "the star's middle"); assert!(red(c, c - 15), "the spoke up to the member at the top (y up)"); diff --git a/crates/latkit/tests/scale.rs b/crates/latkit/tests/scale.rs index 3085fa33..1f5e64b4 100644 --- a/crates/latkit/tests/scale.rs +++ b/crates/latkit/tests/scale.rs @@ -1,19 +1,19 @@ -//! Scale: what a view draws always fits. Tables too large to draw whole draw the tiles in view, -//! traces too large thin in time, what matters more draws on top, a page never outgrows a chunk, and -//! a view sleeps while it waits. +//! Scale: a view's drawing always fits its budget. Paged tables draw the tiles in view, traces that +//! are too large thin in time, and higher-priority rows draw on top. A path longer than an arena +//! chunk still draws in full, and a view sleeps while it waits. #![cfg(feature = "gpu")] mod common; -use common::{SIZE, at, camera, close, errors, frame, gpu, plain_theme, plane}; +use common::{SIZE, Threads, at, camera, close, errors, frame, gpu, plain_theme, plane, step}; use latkit::{ - Axis, Budget, Camera, Color, Data, Edges, Fit, Frame, Ids, Lines, Patch, Plot, Row, Scene, Table, Vertices, View, - field, when, + Areas, Axis, Budget, Camera, Color, Data, Edges, Fit, Frame, Ids, Lines, Patch, Plot, Row, Scene, Table, Vertices, + View, Waiting, field, when, }; -/// A grid of `columns` by `rows` vertices a unit apart, from the origin, and an edge from each to -/// the next in its row. +/// A grid of `columns` by `rows` vertices one unit apart, starting at the origin, each with +/// `v` = 0, and an edge from each vertex to the next in its row. fn lattice(columns: usize, rows: usize) -> Data { let n = columns * rows; let x: Vec = (0..n).map(|i| (i % columns) as f64).collect(); @@ -21,8 +21,9 @@ fn lattice(columns: usize, rows: usize) -> Data { let links: Vec = (0..n).filter(|i| i % columns + 1 < columns).collect(); let from: Vec = links.iter().map(|&i| i as u64).collect(); let to: Vec = links.iter().map(|&i| i as u64 + 1).collect(); - let mut data = Data::new("grid"); - data.apply(Patch::new().set(Table::new("Bus", Ids::count(n as u32)).field("x", x).field("y", y)).set( + let data = Data::new("grid"); + let buses = Table::new("Bus", Ids::count(n as u32)).field("x", x).field("y", y).field("v", vec![0.0; n]); + data.apply(Patch::new().set(buses).set( Table::new("Line", Ids::count(links.len() as u32)).reference("from", "Bus", from).reference("to", "Bus", to), )) .expect("valid"); @@ -30,9 +31,9 @@ fn lattice(columns: usize, rows: usize) -> Data { } #[test] -fn what_matters_more_draws_on_top() { +fn higher_priority_rows_draw_on_top() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Table::new("Node", Ids::count(2)) .field("x", vec![0.0, 0.0]) @@ -44,7 +45,7 @@ fn what_matters_more_draws_on_top() { let red = Color::rgb(1.0, 0.0, 0.0); let blue = Color::rgb(0.0, 0.0, 1.0); let mark = Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .radius(8.0) .color(when(field("kind").is("a")).then(red).otherwise(blue)); let center = SIZE / 2; @@ -52,7 +53,7 @@ fn what_matters_more_draws_on_top() { let mark = if priority { mark.clone().priority(field("p")) } else { mark.clone() }; let mut view = View::new(&gpu, Scene::new(plane()).theme(plain_theme()).draw(mark)).expect("valid scene"); view.set_camera(camera()); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); assert!(close(at(&pixels, center, center), want), "priority {priority}: {:?}", at(&pixels, center, center)); } @@ -61,34 +62,33 @@ fn what_matters_more_draws_on_top() { #[test] fn a_table_that_starts_empty_draws_its_rows() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(0)).field("x", Vec::::new()).field("y", Vec::::new())) .expect("valid"); - let scene = Scene::new(plane()) - .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(4.0).color(Color::rgb(1.0, 1.0, 1.0))); + let scene = Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); - view.set_data(data); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; assert!(close(at(&pixels, center, center), [255, 255, 255, 255]), "its constant color and visibility evaluate"); } #[test] -fn a_dense_table_draws_the_tiles_in_view_and_keeps_what_is_selected() { +fn a_paged_table_draws_the_tiles_in_view_and_its_selected_rows() { let Some(gpu) = gpu() else { return }; let (columns, rows) = (600, 500); let data = lattice(columns, rows); - let scene = Scene::new(plane()) - .theme(plain_theme()) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(2.0).color(Color::rgb(1.0, 1.0, 1.0))); + let scene = Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); assert_eq!(errors(&mut view), []); let stats = view.stats(); @@ -100,44 +100,77 @@ fn a_dense_table_draws_the_tiles_in_view_and_keeps_what_is_selected() { assert!(close(at(&pixels, center + 4, center + 4), [0, 0, 0, 255]), "and none between rows"); // A row far from the tiles in view stays selected, and draws. let far = Row::new("grid", "Bus", 7_u64); - view.select([far.clone().into()]); + view.set_selection([far.clone().into()]); frame(&gpu, &mut view); assert_eq!(view.selection(), [latkit::Rows::from(far)]); - assert_eq!(view.stats().drawn, stats.drawn + 1, "the selected row draws whatever its tile"); + assert_eq!(view.stats().drawn, stats.drawn + 1, "the selected row draws regardless of its tile"); } #[test] -fn panning_a_dense_table_draws_every_row_in_view_as_tiles_come_and_go() { +fn a_paged_table_on_the_globe_draws_its_finest_tiles_close_up() { + let Some(gpu) = gpu() else { return }; + // 300,000 buses 0.0002° apart, all in one cell of the coarsest tiles, so only fine tiles hold + // the buses in view. + let (columns, rows) = (600_usize, 500_usize); + let n = columns * rows; + let data = Data::new("grid"); + data.apply( + Table::new("Bus", Ids::count(n as u32)) + .field("lon", (0..n).map(|i| -74.06 + 0.0002 * (i % columns) as f64).collect::>()) + .field("lat", (0..n).map(|i| 40.65 + 0.0002 * (i / columns) as f64).collect::>()), + ) + .expect("valid"); + let earth = latkit::Earth { graticule: false, ..latkit::Earth::default() }.projection(latkit::Projection::Globe); + let scene = Scene::new(earth) + .theme(plain_theme()) + .ease(std::time::Duration::ZERO) + .draw(Vertices::of("grid", "Bus").position(field("lon"), field("lat")).radius(2.0).color(Color::WHITE)); + let mut view = View::new(&gpu, scene).expect("valid scene"); + // So close that the camera's horizon is nearer than a sixty-fourth of the coarsest tile. + view.set_camera(Camera::Map { center: [-74.0, 40.7, 0.0], zoom: 15.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + let lit = pixels.iter().filter(|p| p[0] > 128).count(); + assert!(view.stats().drawn > 100, "the buses in view draw: {}", view.stats().drawn); + assert!(lit > 1000, "the buses in view show: {lit} lit pixels"); +} + +#[test] +fn panning_a_paged_table_draws_every_row_in_view_as_tiles_change() { let Some(gpu) = gpu() else { return }; let (columns, rows) = (600, 500); - // Without an ease, rows arriving draw at once. + // Without an ease, arriving rows draw at once. let scene = Scene::new(plane()) .theme(plain_theme()) .ease(std::time::Duration::ZERO) .draw(Edges::of("grid", "Line").ends("from", "to").width(2.0).color(Color::rgb(0.0, 1.0, 0.0))) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(2.0).color(Color::rgb(1.0, 1.0, 1.0))); + .draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(lattice(columns, rows)); + let data = lattice(columns, rows); + view.set_sources([&data]).expect("a source"); let center = SIZE / 2; let camera = |x: f64| Camera::Map { center: [x, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }; - let mut in_place = 0; + let mut whole = 0; for step in 0..6 { view.set_camera(camera(100.0 + 60.0 * f64::from(step))); - // While the rows still drawn stay where they were, those arriving in holes and past the last - // row, every row in view draws. + // Every frame not waiting on a page draws every row in view. After a camera move, the + // view is complete, with edges and their endpoints. for _ in 0..4 { let pixels = common::step(&gpu, &mut view); - let waiting = view.stats().waiting; - if step > 0 && !waiting.data && !waiting.marks && !view.complete() { + let waiting = view.waiting(); + if step > 0 && !waiting.data && !waiting.marks { let pixel = |dx: u32, dy: u32| at(&pixels, center + dx, center + dy); - assert!(close(pixel(0, 0), [255, 255, 255, 255]), "step {step}, in place: {:?}", pixel(0, 0)); - assert!(close(pixel(4, 0), [0, 255, 0, 255]), "step {step}, in place: {:?}", pixel(4, 0)); - in_place += 1; + assert!(close(pixel(0, 0), [255, 255, 255, 255]), "step {step}, as pages arrive: {:?}", pixel(0, 0)); + assert!(close(pixel(4, 0), [0, 255, 0, 255]), "step {step}, as pages arrive: {:?}", pixel(4, 0)); + whole += 1; } } let pixels = frame(&gpu, &mut view); assert_eq!(errors(&mut view), []); - assert!(view.complete(), "every row laid out in drawing order once the camera rests"); + assert!(view.complete(), "every row in view draws once the camera rests"); let stats = view.stats(); assert!(stats.drawn > 0 && stats.drawn < stats.rows / 4, "step {step}: {} of {}", stats.drawn, stats.rows); let pixel = |dx: u32, dy: u32| at(&pixels, center + dx, center + dy); @@ -145,25 +178,378 @@ fn panning_a_dense_table_draws_every_row_in_view_as_tiles_come_and_go() { assert!(close(pixel(4, 4), [0, 0, 0, 255]), "step {step}: none between rows: {:?}", pixel(4, 4)); assert!(close(pixel(4, 0), [0, 255, 0, 255]), "step {step}: an edge between two: {:?}", pixel(4, 0)); } - assert!(in_place > 0, "rows were written in place"); - // A row picked by id is found among the rows drawn, holes and all. + assert!(whole > 0, "frames drew as their pages arrived"); + // A row written far from the camera's starting point leaves the pages there resident. + let last = (columns * rows - 1) as u64; + data.apply(Table::new("Bus", vec![last]).field("v", vec![1.0])).expect("valid"); + frame(&gpu, &mut view); + // Back at the starting point, its pages are still resident, so nothing packs, uploads, or + // evaluates again. + let before = gpu.work(); + view.set_camera(camera(100.0)); + let pixels = frame(&gpu, &mut view); + let work = gpu.work() - before; + assert_eq!((work.packed, work.uploaded, work.evaluated), (0, 0, 0), "returning needs no work"); + assert!(close(at(&pixels, center, center), [255, 255, 255, 255]), "{:?}", at(&pixels, center, center)); + // A row selected by id is found among the drawn rows. let selected = Row::new("grid", "Bus", (250 * columns + 400) as u64); - view.select([selected.clone().into()]); + view.set_selection([selected.clone().into()]); view.set_camera(camera(380.0)); frame(&gpu, &mut view); - assert_eq!(view.selection(), [latkit::Rows::from(selected)], "what is selected stays so"); + assert_eq!(view.selection(), [latkit::Rows::from(selected)], "the selected row stays selected"); + assert_eq!(errors(&mut view), []); +} + +#[test] +fn appending_to_a_paged_table_never_thins_it() { + let Some(gpu) = gpu() else { return }; + let (columns, rows) = (600, 500); + let data = lattice(columns, rows); + let scene = Scene::new(plane()).theme(plain_theme()).ease(std::time::Duration::ZERO).draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + frame(&gpu, &mut view); + assert!(view.complete(), "complete, drawing from its pyramid"); + let drawn = view.stats().drawn; + let center = SIZE / 2; + let before = gpu.work(); + for added in 0..4_u32 { + // One row per step, far from the view. Until a pyramid includes it, it draws last. + let id = (columns * rows) as u64 + u64::from(added); + let row = Table::new("Bus", vec![id]).field("x", vec![-50.0]).field("y", vec![f64::from(added)]); + data.apply(row).expect("valid"); + let pixels = common::step(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + let pixel = |dx: u32| at(&pixels, center + dx, center); + assert!(close(pixel(0), [255, 255, 255, 255]), "append {added}: a row at the middle: {:?}", pixel(0)); + assert!(close(pixel(8), [255, 255, 255, 255]), "append {added}: and its neighbor: {:?}", pixel(8)); + assert!(view.stats().drawn >= drawn, "append {added}: {} drawn, {drawn} before", view.stats().drawn); + } + // The pyramid serves appended rows as they arrive, and only the page they join evaluates. + let work = gpu.work() - before; + assert_eq!(work.jobs, 0, "a few appended rows start no job"); + assert!(work.evaluated <= 4 << 12, "at most a page per append evaluates: {}", work.evaluated); +} + +#[test] +fn a_change_to_one_rows_color_moves_no_row() { + let Some(gpu) = gpu() else { return }; + // A grid on the globe, drawn as a paged table. Its lines are great circles whose pieces depend + // on the bus positions. + let (columns, rows) = (600, 500); + let n = columns * rows; + let links: Vec = (0..n).filter(|i| i % columns + 1 < columns).collect(); + let data = Data::new("grid"); + data.apply( + Patch::new() + .set( + Table::new("Bus", Ids::count(n as u32)) + .field("lon", (0..n).map(|i| -170.0 + 340.0 * (i % columns) as f64 / 599.0).collect::>()) + .field("lat", (0..n).map(|i| -75.0 + 150.0 * (i / columns) as f64 / 499.0).collect::>()) + .field("v", vec![0.0; n]), + ) + .set( + Table::new("Line", Ids::count(links.len() as u32)) + .reference("from", "Bus", links.iter().map(|&i| i as u64).collect::>()) + .reference("to", "Bus", links.iter().map(|&i| i as u64 + 1).collect::>()), + ), + ) + .expect("valid"); + let scene = Scene::new(latkit::Earth::default().projection(latkit::Projection::Globe)) + .theme(plain_theme()) + .ease(std::time::Duration::ZERO) + .draw(Edges::of("grid", "Line").ends("from", "to")) + .draw( + Vertices::of("grid", "Bus") + .position(field("lon"), field("lat")) + .color(field("v").colormap(latkit::Colormap::GRAYS).domain(0.0..=1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [10.0, 10.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + assert!(view.stats().drawn < view.stats().rows / 4, "the grid is paged"); + // One bus far out of view, one in it. Neither changes a line's pieces, so nothing is read back. + for id in [0, 250 * columns + 300] { + let before = gpu.work(); + data.apply(Table::new("Bus", vec![id as u64]).field("v", vec![1.0])).expect("valid"); + frame(&gpu, &mut view); + let work = gpu.work() - before; + assert_eq!(work.readbacks, 0, "bus {id}: no line counts its pieces again"); + assert!(work.evaluated <= 1 << 12, "bus {id}: at most a page evaluates: {}", work.evaluated); + assert_eq!(work.resolved, 0, "bus {id}: no line resolves its endpoints again"); + } +} + +#[test] +fn a_frame_waiting_for_pages_packs_nothing_again() { + let Some(gpu) = gpu() else { return }; + let budget = gpu.budget(); + let scene = Scene::new(plane()).theme(plain_theme()).ease(std::time::Duration::ZERO).draw( + Vertices::of("grid", "Bus") + .position(field("x"), field("y")) + .radius(2.0) + .color(field("x").colormap(latkit::Colormap::GRAYS)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([lattice(600, 500)]).expect("a source"); + frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + // Pages upload a few at a time, so the area the camera moves to waits for them. + gpu.set_budget(Budget { upload_per_frame: 16 << 10, ..budget }); + view.set_camera(Camera::Map { center: [80.0, 60.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + let before = gpu.work(); + common::step(&gpu, &mut view); + let packed = (gpu.work() - before).packed; + assert!(packed > 0 && view.waiting().data, "the pages at the new camera pack, then wait to upload"); + let mut waited = 0; + while view.waiting().data && waited < common::SETTLE { + common::step(&gpu, &mut view); + waited += 1; + } + assert!(waited > 1, "it waited several frames for pages"); + assert_eq!((gpu.work() - before).packed, packed, "the waiting frames packed nothing again"); + gpu.set_budget(budget); +} + +#[test] +fn a_fitted_domain_covers_every_row_of_a_paged_table() { + let Some(gpu) = gpu() else { return }; + // A lattice whose rows have v = x / 600, plus one far in a corner with v = 10 and the lowest + // priority. It falls among the finest rows, in no page drawn around the center. + let (columns, rows) = (600_usize, 500_usize); + let n = columns * rows; + let mut x: Vec = (0..n).map(|i| (i % columns) as f64).collect(); + let mut y: Vec = (0..n).map(|i| (i / columns) as f64).collect(); + let mut v: Vec = x.iter().map(|x| x / 600.0).collect(); + x.push(599.5); + y.push(499.5); + v.push(10.0); + let p: Vec = v.iter().map(|v| -v).collect(); + let data = Data::new("grid"); + data.apply(Table::new("Bus", Ids::count(n as u32 + 1)).field("x", x).field("y", y).field("v", v).field("p", p)) + .expect("valid"); + // A blue background, so a gray row is distinguishable from no row. + let theme = latkit::Theme { background: Color::rgb(0.0, 0.0, 1.0), ..plain_theme() }; + let scene = Scene::new(plane()).theme(theme).ease(std::time::Duration::ZERO).draw( + Vertices::of("grid", "Bus") + .position(field("x"), field("y")) + .radius(3.0) + .priority(field("p")) + .color(field("v").colormap(latkit::Colormap::GRAYS)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + assert!(view.stats().drawn < view.stats().rows / 4, "the table is paged"); + // The domain is fitted over every row, 0 to 10, so the center row's 0.5 draws near black, not + // the mid gray a domain fitted to the resident rows would give. + let center = at(&pixels, SIZE / 2, SIZE / 2); + let gray = center[0].abs_diff(center[2]) <= 3 && center[1].abs_diff(center[2]) <= 3; + assert!(gray && center[0] < 90, "the middle row draws near black: {center:?}"); +} + +#[test] +fn a_join_into_a_paged_table_reads_every_row() { + let Some(gpu) = gpu() else { return }; + let (columns, rows) = (600, 500); + let n = columns * rows; + let data = lattice(columns, rows); + // Bus 7 lies far from the view. A probe at the center reads its value by id. + let v: Vec = (0..n).map(|i| if i == 7 { 1.0 } else { 0.0 }).collect(); + data.apply( + Patch::new() + .set(Table::new("Bus", Ids::count(n as u32)).field("v", v)) + .set(Table::new("Probe", vec![7_u64]).field("x", vec![300.5]).field("y", vec![250.5])), + ) + .expect("valid"); + let gray = Color::rgb(0.2, 0.2, 0.2); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .draw(Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(1.0).color(gray)) + .draw( + Vertices::of("grid", "Probe") + .position(field("x"), field("y")) + .radius(4.0) + .color(field("v").from_table("grid", "Bus").colormap(latkit::Colormap::GRAYS).domain(0.0..=1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&data]).expect("a source"); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + assert!(view.stats().drawn < view.stats().rows / 4, "the buses are paged"); + let probe = view.screen([300.5, 250.5, 0.0]).expect("in view"); + let pixel = at(&pixels, probe[0] as u32, probe[1] as u32); + assert!(close(pixel, [255, 255, 255, 255]), "bus 7's value, though no tile in view contains it: {pixel:?}"); +} + +#[test] +fn pages_made_resident_mid_ease_enter_as_the_rest_keep_easing() { + let Some(gpu) = gpu() else { return }; + let camera = |zoom: f64| Camera::Map { center: [300.0, 250.0, 0.0], zoom, pitch: 0.0, bearing: 0.0 }; + let view = |zoom: f64| { + let scene = Scene::new(plane()).theme(plain_theme()).draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(camera(zoom)); + view.set_sources([lattice(600, 500)]).expect("a source"); + view + }; + let mut pulled = view(1.0); + let want = frame(&gpu, &mut pulled); + // A sample draws first. Then the pyramid arrives, and its rows ease in from the sample... + let mut easing = view(3.0); + common::step(&gpu, &mut easing); + common::step(&gpu, &mut easing); + // ...while the camera pulls back, needing more resident pages than before. + easing.set_camera(camera(1.0)); + let pixels = frame(&gpu, &mut easing); + assert_eq!(errors(&mut easing), []); + let differ = pixels.iter().zip(&want).filter(|(a, b)| !close(**a, **b)).count(); + assert_eq!(differ, 0, "settles to the frame of a view pulled back at once"); +} + +#[test] +fn paged_areas_fill_the_drawn_pages_as_the_camera_moves() { + let Some(gpu) = gpu() else { return }; + // A lattice of cells one unit apart, each a square half a unit across: too many to draw whole. + let (columns, rows) = (600_usize, 500_usize); + let cells: Vec>> = (0..columns * rows) + .map(|i| { + let (x, y) = ((i % columns) as f64, (i / columns) as f64); + vec![vec![[x, y], [x + 0.5, y], [x + 0.5, y + 0.5], [x, y + 0.5], [x, y]]] + }) + .collect(); + let data = Data::new("map"); + data.apply(Table::new("Cell", Ids::count((columns * rows) as u32)).field("geometry", cells)).expect("valid"); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .ease(std::time::Duration::ZERO) + .draw(Areas::of("map", "Cell").color(Color::rgb(1.0, 0.0, 0.0))); + let mut view = View::new(&gpu, scene).expect("valid scene"); + let camera = |x: f64, y: f64| Camera::Map { center: [x, y, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }; + view.set_sources([&data]).expect("a source"); + for (x, y) in [(300.25, 250.25), (80.25, 60.25)] { + view.set_camera(camera(x, y)); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + let stats = view.stats(); + assert!(stats.drawn < stats.rows / 4, "the cells are paged: {} of {} draw", stats.drawn, stats.rows); + // The cell at the center fills, and the gap beside it does not. + let center = SIZE / 2; + assert!(close(at(&pixels, center, center), [255, 0, 0, 255]), "at {x}, {y}: {:?}", at(&pixels, center, center)); + assert!( + close(at(&pixels, center + 4, center), [0, 0, 0, 255]), + "at {x}, {y}: {:?}", + at(&pixels, center + 4, center) + ); + } +} + +#[test] +fn selected_rows_off_screen_pack_into_kept_row_pages() { + let Some(gpu) = gpu() else { return }; + let scene = Scene::new(plane()).theme(plain_theme()).ease(std::time::Duration::ZERO).draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([lattice(600, 500)]).expect("a source"); + frame(&gpu, &mut view); + let (drawn, before) = (view.stats().drawn, gpu.work()); + // A hundred rows near the grid's bottom, far from the view, each in its own page of the order. + let ids: Vec = (0..100_u64).map(|i| latkit::Id::from(600 * (i % 50) + i * 6)).collect(); + view.set_selection([latkit::Rows { data: "grid".into(), table: "Bus".into(), ids }]); + frame(&gpu, &mut view); assert_eq!(errors(&mut view), []); + assert_eq!(view.stats().drawn, drawn + 100, "every kept row draws"); + let packed = (gpu.work() - before).packed; + assert!(packed < 4096, "kept rows pack together, not one order page each: {packed} rows packed"); } #[test] -fn a_change_to_one_row_drawn_in_part_writes_that_row_alone() { +fn a_kept_edge_draws_between_its_endpoints() { let Some(gpu) = gpu() else { return }; - // Four pages of rows drawn whole in priority order, so each is drawn far from its place in the - // table; every row lies off view but one, at the middle. + let budget = gpu.budget(); + // So small a budget that only the coarsest edges draw. + gpu.set_budget(Budget { bytes: 8 << 20, ..budget }); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .ease(std::time::Duration::ZERO) + .draw(Edges::of("grid", "Line").ends("from", "to").width(2.0).color(Color::rgb(0.0, 1.0, 0.0))) + .draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(1.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.5, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([lattice(600, 500)]).expect("a source"); + let center = SIZE / 2; + let green = |p: [u8; 4]| p[1] > 150 && p[0] < 100 && p[2] < 100; + let before = frame(&gpu, &mut view); + assert!(!green(at(&before, center, center)), "the edge is not drawn: {:?}", at(&before, center, center)); + // Line 150050 runs from bus 150300 to bus 150301, (300, 250) to (301, 250). Selecting it keeps + // it and its endpoints. + view.set_selection([Row::new("grid", "Line", 150_050_u64).into()]); + let after = frame(&gpu, &mut view); + gpu.set_budget(budget); + assert_eq!(errors(&mut view), []); + assert!(green(at(&after, center, center)), "kept, it draws between its ends: {:?}", at(&after, center, center)); +} + +#[test] +fn placeholder_lanes_of_a_paged_table_read_missing_past_its_rows() { + let Some(gpu) = gpu() else { return }; + // Two buses joined twice: their lanes are derived during the frame. + let pair = Data::new("grid"); + pair.apply( + Patch::new().set(Table::new("Bus", Ids::count(2)).field("x", vec![0.0, 1.0]).field("y", vec![0.0, 0.0])).set( + Table::new("Line", Ids::count(2)).reference("from", "Bus", vec![0_u64, 0]).reference( + "to", + "Bus", + vec![1_u64, 1], + ), + ), + ) + .expect("valid"); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .ease(std::time::Duration::ZERO) + .draw(Edges::of("grid", "Line").ends("from", "to").width(2.0).color(Color::rgb(0.0, 1.0, 0.0))) + .draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(2.0).color(Color::rgb(1.0, 1.0, 1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(Camera::Map { center: [300.0, 250.0, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); + view.set_sources([&pair]).expect("a source"); + frame(&gpu, &mut view); + // A grid too large to draw whole replaces it. Its lanes take a job, and until the job finishes + // the pair's lanes are the placeholder, covering none of the grid's rows past the first two. + view.set_sources([lattice(600, 500)]).expect("a source"); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + assert!(view.stats().drawn < view.stats().rows / 4, "the grid is paged"); + let pixel = at(&pixels, SIZE / 2 + 4, SIZE / 2); + assert!(close(pixel, [0, 255, 0, 255]), "an edge between two buses: {pixel:?}"); +} + +#[test] +fn a_change_to_one_row_of_a_paged_table_updates_only_its_page() { + let Some(gpu) = gpu() else { return }; + // 200,000 rows (49 pages) drawn whole in priority order, so each row draws far from its place + // in the table. Every row lies out of view except one, at the center. let (rows, middle) = (200_000_u32, 12_345_u32); let n = rows as usize; let off = |r: usize| if r == middle as usize { 0.0 } else { 1000.0 }; - let mut data = Data::new("points"); + let data = Data::new("points"); data.apply( Table::new("P", Ids::count(rows)) .field("x", (0..n).map(off).collect::>()) @@ -174,33 +560,33 @@ fn a_change_to_one_row_drawn_in_part_writes_that_row_alone() { .expect("valid"); let scene = Scene::new(plane()).theme(plain_theme()).ease(std::time::Duration::ZERO).draw( Vertices::of("points", "P") - .position("x", "y") + .position(field("x"), field("y")) .radius(4.0) .priority(field("p")) .color(field("v").colormap(latkit::Colormap::GRAYS).domain(0.0..=1.0)), ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); assert_eq!(errors(&mut view), []); assert_eq!(view.stats().drawn, u64::from(rows), "every row draws, in priority order"); let before = gpu.work(); data.apply(Table::new("P", vec![u64::from(middle)]).field("v", vec![1.0])).expect("valid"); - view.set_data(data); let pixels = frame(&gpu, &mut view); let work = gpu.work() - before; let center = SIZE / 2; assert!(close(at(&pixels, center, center), [255, 255, 255, 255]), "{:?}", at(&pixels, center, center)); - assert!(work.evaluated <= 1 << 16, "one page of the rows drawn evaluates: {}", work.evaluated); - assert!(work.uploaded <= 8 << 16, "one page of the field drawn uploads: {} bytes", work.uploaded); + // A page is 4,096 rows, and the field takes 8 bytes a row. + assert!(work.evaluated <= 4096, "only one page evaluates: {}", work.evaluated); + assert!(work.uploaded <= 8 * 4096, "only one page of the field uploads: {} bytes", work.uploaded); } #[test] -fn a_field_drawn_in_part_that_changes_kind_gathers_afresh() { +fn a_field_of_a_paged_table_that_changes_kind_is_gathered_again() { let Some(gpu) = gpu() else { return }; - // Two marks of one table drawn in part: one reads `v` as a number, the other as a text, so `v` is - // read before and after it changes kind. + // Two marks of one paged table. One reads `v` as a number and the other as text, so `v` is read + // both before and after it changes kind. let n = 1000_usize; let table = |v: latkit::Values| { Table::new("P", Ids::count(n as u32)) @@ -209,23 +595,22 @@ fn a_field_drawn_in_part_that_changes_kind_gathers_afresh() { .field("p", (0..n).map(|r| r as f64).collect::>()) .field("v", v) }; - let mut data = Data::new("points"); + let data = Data::new("points"); data.apply(table(vec![0.5; n].into())).expect("valid"); let number = Vertices::of("points", "P") - .position("x", "y") + .position(field("x"), field("y")) .priority(field("p")) .color(field("v").colormap(latkit::Colormap::GRAYS)); let text = Vertices::of("points", "P") .named("text") - .position("x", "y") + .position(field("x"), field("y")) .color(when(field("v").is("a")).then(Color::WHITE).otherwise(Color::BLACK)); let mut view = View::new(&gpu, Scene::new(plane()).theme(plain_theme()).draw(number).draw(text)).expect("valid"); view.set_camera(camera()); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); assert_eq!(view.failures().count(), 1, "the text mark fails while `v` holds numbers"); data.apply(Patch::new().replace(table(vec!["a"; n].into()))).expect("valid"); - view.set_data(data); for _ in 0..10 { common::step(&gpu, &mut view); } @@ -236,21 +621,23 @@ fn a_field_drawn_in_part_that_changes_kind_gathers_afresh() { } #[test] -fn edges_drawn_through_tiles_show_with_their_ends() { +fn paged_edges_draw_with_their_endpoints() { let Some(gpu) = gpu() else { return }; let data = lattice(600, 500); let scene = Scene::new(plane()) .theme(plain_theme()) .draw(Edges::of("grid", "Line").ends("from", "to").width(2.0).color(Color::rgb(0.0, 1.0, 0.0))) - .draw(Vertices::of("grid", "Bus").position("x", "y").radius(1.0).color(Color::rgb(0.0, 0.0, 1.0))); + .draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(1.0).color(Color::rgb(0.0, 0.0, 1.0)), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [300.0, 250.5, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); assert_eq!(errors(&mut view), []); assert!(view.stats().drawn < view.stats().rows / 4, "edges and vertices both draw the tiles in view"); - // Half a unit up from the middle, between rows of edges; at the middle's row, an edge between - // two vertices. + // The center pixel row lies half a unit above a row of vertices, between rows of edges. Four + // pixels down, on that row, an edge runs between two vertices. let center = SIZE / 2; assert!(close(at(&pixels, center + 4, center + 4), [0, 255, 0, 255]), "{:?}", at(&pixels, center + 4, center + 4)); assert!(close(at(&pixels, center + 4, center), [0, 0, 0, 255]), "{:?}", at(&pixels, center + 4, center)); @@ -261,7 +648,7 @@ fn a_tiny_budget_never_fails() { let Some(gpu) = gpu() else { return }; gpu.set_budget(Budget { bytes: 8 << 20, upload_per_frame: 32 << 20 }); let n = 200_000_u32; - let mut data = Data::new("points"); + let data = Data::new("points"); data.apply( Table::new("Point", Ids::count(n)) .field("x", (0..n).map(|i| f64::from(i % 500)).collect::>()) @@ -271,33 +658,168 @@ fn a_tiny_budget_never_fails() { .expect("valid"); let scene = Scene::new(plane()).theme(plain_theme()).draw( Vertices::of("points", "Point") - .position("x", "y") + .position(field("x"), field("y")) .radius(field("v") / 1e5 + 1.0) .color(Color::rgb(1.0, 1.0, 1.0)), ); let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); view.fit(Fit::All); frame(&gpu, &mut view); assert!(errors(&mut view).is_empty(), "nothing fails"); let stats = view.stats(); - assert!(stats.resident <= 8 << 20, "{} bytes resident", stats.resident); + assert!(gpu.resident() <= 8 << 20, "{} bytes resident", gpu.resident()); assert!(stats.drawn > 0 && stats.drawn < u64::from(n), "{} of {n} rows draw", stats.drawn); let Camera::Map { center, .. } = *view.camera() else { panic!("a map camera") }; assert!((center[0] - 249.5).abs() < 5.0 && (center[1] - 199.5).abs() < 5.0, "a fit frames every row: {center:?}"); } +#[test] +fn a_first_look_stays_within_budget_while_pyramids_are_computed() { + let Some(gpu) = gpu() else { return }; + gpu.set_budget(Budget { bytes: 8 << 20, upload_per_frame: 4 << 20 }); + gpu.set_host(Threads); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .draw(Edges::of("grid", "Line").ends("from", "to")) + .draw(Vertices::of("grid", "Bus").position(field("x"), field("y"))); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_sources([lattice(1000, 500)]).expect("a source"); + view.fit(Fit::All); + // Until the pyramids arrive, a sample draws, with its edges' endpoints among its vertex rows. + // Then its replacement finds room beside it. + let mut settled = false; + for _ in 0..600 { + step(&gpu, &mut view); + if view.waiting() == Waiting::default() && !view.needs_render() { + settled = true; + break; + } + if !view.needs_render() { + std::thread::sleep(std::time::Duration::from_millis(5)); + } + } + assert!(errors(&mut view).is_empty(), "nothing fails"); + assert!(settled, "the sample's replacement arrives: {:?}", view.waiting()); + assert!(gpu.resident() <= 8 << 20, "{} bytes resident", gpu.resident()); +} + +/// A grid of 500 by 500 vertices one unit apart, each with `v` = 0, and an edge from each vertex +/// `i` to vertex `far(i, random)`, with `random` drawn from a seeded stream. +fn far_apart(far: impl Fn(u32, u64) -> u64) -> Data { + let n = 250_000_u32; + let mut seed = 1_u64; + let to: Vec = (0..n) + .map(|i| { + seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1_442_695_040_888_963_407); + far(i, seed >> 33) + }) + .collect(); + let data = Data::new("grid"); + data.apply( + Patch::new() + .set( + Table::new("Bus", Ids::count(n)) + .field("x", (0..n).map(|i| f64::from(i % 500)).collect::>()) + .field("y", (0..n).map(|i| f64::from(i / 500)).collect::>()) + .field("v", vec![0.0; n as usize]), + ) + .set( + Table::new("Line", Ids::count(n)) + .reference("from", "Bus", (0..u64::from(n)).collect::>()) + .reference("to", "Bus", to), + ), + ) + .expect("valid"); + data +} + +#[test] +fn edges_ending_far_apart_make_their_endpoints_kept_rows() { + let Some(gpu) = gpu() else { return }; + gpu.set_budget(Budget { bytes: 8 << 20, upload_per_frame: 32 << 20 }); + // Each edge joins a vertex to a far one. Even the fewest edges drawn end in every vertex page, + // more pages than the budget allows, so their endpoints become kept rows. + let data = far_apart(|_, random| random % 250_000); + let scene = Scene::new(plane()).theme(plain_theme()).draw(Edges::of("grid", "Line").ends("from", "to")).draw( + Vertices::of("grid", "Bus") + .position(field("x"), field("y")) + .color(field("v").colormap(latkit::Colormap::GRAYS).domain(0.0..=1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_sources([&data]).expect("a source"); + view.fit(Fit::All); + let before = gpu.work(); + frame(&gpu, &mut view); + assert_eq!(errors(&mut view), [], "the drawing fits the budget"); + assert!(view.stats().drawn > 0, "edges and vertices draw"); + assert!((gpu.work() - before).resolved > 0, "edges resolve their endpoints"); + assert!(gpu.resident() <= 8 << 20, "{} bytes resident", gpu.resident()); + // Every vertex's value changes but none moves. The kept rows stay in place, so no edge resolves + // its endpoints again. + let before = gpu.work(); + data.apply(Table::new("Bus", Ids::count(250_000)).field("v", vec![1.0; 250_000])).expect("valid"); + frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + let work = gpu.work() - before; + assert!(work.uploaded > 0, "the values upload"); + assert_eq!(work.resolved, 0, "no edge resolves its endpoints again"); +} + +#[test] +fn edges_ending_far_apart_draw_between_their_ends() { + let Some(gpu) = gpu() else { return }; + gpu.set_budget(Budget { bytes: 8 << 20, upload_per_frame: 32 << 20 }); + // Each edge runs along its row to a far vertex in the same row, so every edge is horizontal. An + // edge whose endpoint resolved to the wrong vertex would slant across the gaps between rows. + let data = far_apart(|i, random| u64::from(i - i % 500) + random % 500); + let green = Color::rgb(0.0, 1.0, 0.0); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .ease(std::time::Duration::ZERO) + .draw(Edges::of("grid", "Line").ends("from", "to").width(2.0).color(green)) + .draw( + Vertices::of("grid", "Bus").position(field("x"), field("y")).radius(1.0).color(Color::rgb(0.0, 0.0, 1.0)), + ); + let mut view = View::new(&gpu, scene).expect("valid scene"); + // Rows are eight pixels apart, and the center pixel row lies between two of them. + let near = Camera::Map { center: [250.0, 250.5, 0.0], zoom: 3.0, pitch: 0.0, bearing: 0.0 }; + view.set_camera(near); + view.set_sources([&data]).expect("a source"); + let level = |pixels: &[[u8; 4]], when: &str| { + let center = SIZE / 2; + for x in 0..SIZE { + let between = at(pixels, x, center); + assert!(close(between, [0, 0, 0, 255]), "{when}: nothing crosses between the rows at {x}: {between:?}"); + } + let along = (0..SIZE).filter(|&x| close(at(pixels, x, center + 4), [0, 255, 0, 255])).count(); + assert!(along > 0, "{when}: edges run along a row"); + }; + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + level(&pixels, "first"); + // Zoomed out, the drawn edges end at more kept rows, so the end table grows, and so does the + // program buffer that holds it. Back at the first camera, every edge still resolves its + // endpoints. + view.set_camera(Camera::Map { center: [250.0, 250.5, 0.0], zoom: 1.0, pitch: 0.0, bearing: 0.0 }); + frame(&gpu, &mut view); + view.set_camera(near); + let pixels = frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + level(&pixels, "after the table grew"); +} + #[test] fn traces_too_large_for_the_budget_thin_in_time() { let Some(gpu) = gpu() else { return }; gpu.set_budget(Budget { bytes: 8 << 20, upload_per_frame: 32 << 20 }); - // Every frame of every row would take 12 MB: past the budget, so the traces draw coarser. + // Every frame of every row would take 12 MB, over the budget, so the traces draw coarser. let (rows, frames) = (20_000_usize, 150_u32); let mut patch = Patch::new().set(Table::new("Signal", Ids::count(rows as u32))); for t in 0..frames { - patch = patch.append([f64::from(t)], "Signal", [("v", vec![0.5; rows])]); + patch = patch.append("Signal", [f64::from(t)], [("v", vec![0.5; rows])]); } - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(patch).expect("valid"); let axis = Axis { grid: false, hidden: true, ..Axis::default() }; let scene = Scene::new(Plot { x: axis.clone(), y: axis }) @@ -305,32 +827,60 @@ fn traces_too_large_for_the_budget_thin_in_time() { .draw(Lines::of("signals", "Signal").trace("v").color(Color::rgb(1.0, 0.0, 0.0)).width(4.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Plot { x: [0.0, f64::from(frames - 1)], y: Some([0.0, 1.0]) }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); assert_eq!(errors(&mut view), []); let stats = view.stats(); assert_eq!(stats.drawn, rows as u64, "every row traces"); - assert_eq!(stats.uploading, 0, "nothing waits for room"); - assert!(stats.resident <= 8 << 20, "{} bytes resident", stats.resident); + assert_eq!(gpu.uploading(), 0, "nothing waits for room"); + assert!(gpu.resident() <= 8 << 20, "{} bytes resident", gpu.resident()); for x in [SIZE / 4, SIZE / 2, SIZE * 3 / 4] { assert!(close(at(&pixels, x, SIZE / 2), [255, 0, 0, 255]), "at {x}: {:?}", at(&pixels, x, SIZE / 2)); } } +#[test] +fn a_rewritten_path_uploads_only_its_group_of_rows() { + let Some(gpu) = gpu() else { return }; + let rows = 2000_u32; + let path = |r: u32, lift: f64| -> Vec<[f64; 2]> { + (0..6).map(|k| [f64::from(k) * 10.0 - 25.0, f64::from(r % 40) - 20.0 + lift]).collect() + }; + let data = Data::new("roads"); + data.apply( + Table::new("Road", Ids::count(rows)).field("geometry", (0..rows).map(|r| path(r, 0.0)).collect::>()), + ) + .expect("valid"); + let scene = + Scene::new(plane()).theme(plain_theme()).ease(std::time::Duration::ZERO).draw(Lines::of("roads", "Road")); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_camera(camera()); + view.set_sources([&data]).expect("a source"); + frame(&gpu, &mut view); + assert_eq!(errors(&mut view), []); + let before = gpu.work(); + data.apply(Table::new("Road", [700_u64]).field("geometry", vec![path(700, 0.5)])).expect("valid"); + frame(&gpu, &mut view); + let work = gpu.work() - before; + // The page's points take 96 KB and its row table 16 KB. The 64-row group holding the changed + // row takes 3 KB. + assert!(work.uploaded <= 4 << 10, "only the row's 64-row group uploads: {} bytes", work.uploaded); +} + #[test] fn a_path_longer_than_a_chunk_draws_whole() { let Some(gpu) = gpu() else { return }; let points = 100_000_u32; let path: Vec<[f64; 2]> = (0..points).map(|i| [-60.0 + 120.0 * f64::from(i) / f64::from(points - 1), 0.0]).collect(); - let mut data = Data::new("roads"); + let data = Data::new("roads"); data.apply(Table::new("Road", Ids::count(1)).field("geometry", vec![path])).expect("valid"); let scene = Scene::new(plane()) .theme(plain_theme()) .draw(Lines::of("roads", "Road").width(3.0).color(Color::rgb(1.0, 1.0, 0.0))); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); let center = SIZE / 2; for x in [center - 50, center, center + 50] { @@ -345,12 +895,13 @@ fn a_path_longer_than_a_chunk_draws_whole() { #[test] fn a_view_sleeps_while_it_waits() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0])).expect("valid"); - let scene = Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position("x", "y")); + let scene = + Scene::new(plane()).theme(plain_theme()).draw(Vertices::of("graph", "Node").position(field("x"), field("y"))); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); assert!(!view.needs_render(), "settled"); let _hits = view.pick([64.0, 64.0], 4.0); @@ -378,5 +929,5 @@ fn a_view_sleeps_while_it_waits() { gpu.queue().submit([encoder.finish()]); assert!(!view.needs_render(), "while the answer is read back, the view sleeps"); common::finish(&gpu); - assert!(view.needs_render(), "the answer landing wakes it"); + assert!(view.needs_render(), "the answer arriving wakes it"); } diff --git a/crates/latkit/tests/scenes.rs b/crates/latkit/tests/scenes.rs index d7393036..d3f67984 100644 --- a/crates/latkit/tests/scenes.rs +++ b/crates/latkit/tests/scenes.rs @@ -1,4 +1,5 @@ -//! The scenes of `npm/test/scenes.js` decode: the JSON form TypeScript checks is the form latkit reads. +//! The scenes in `npm/test/scenes.js` decode, so the JSON form TypeScript checks is the form latkit +//! reads. #![cfg(feature = "gpu")] diff --git a/crates/latkit/tests/schema.rs b/crates/latkit/tests/schema.rs index 4a78d8c3..d1caceb4 100644 --- a/crates/latkit/tests/schema.rs +++ b/crates/latkit/tests/schema.rs @@ -1,14 +1,14 @@ -//! npm's input types are written from `npm/schema.json`, so it must be what the forms say, and the -//! documentation's colors from `docs/_static/theme.json`, so it must be latkit's default themes: -//! `LATKIT_BLESS=1 cargo test -p latkit --test schema` writes both, and `node npm/build.mjs schema` -//! writes the types. +//! The npm package's input types are generated from `npm/schema.json`, so it must match latkit's +//! schemas. The documentation's colors come from `docs/_static/theme.json`, so it must match +//! latkit's default themes. `LATKIT_BLESS=1 cargo test -p latkit --test schema` writes both files, +//! and `node npm/build.mjs schema` regenerates the types. #![cfg(feature = "gpu")] use std::path::Path; #[test] -fn npm_schema_is_the_forms() { +fn npm_schema_matches_the_input_schemas() { let path = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../npm/schema.json"); let schemas = serde_json::to_string_pretty(&latkit::schemas()).expect("schemas serialize") + "\n"; if std::env::var_os("LATKIT_BLESS").is_some() { @@ -18,13 +18,13 @@ fn npm_schema_is_the_forms() { let committed = std::fs::read_to_string(&path).unwrap_or_default().replace("\r\n", "\n"); assert!( committed == schemas, - "npm/schema.json is not what the forms say: run LATKIT_BLESS=1 cargo test -p latkit --test schema, \ + "npm/schema.json does not match latkit's schemas: run LATKIT_BLESS=1 cargo test -p latkit --test schema, \ then node npm/build.mjs schema" ); } #[test] -fn docs_palette_is_the_default_themes() { +fn docs_palette_matches_the_default_themes() { let path = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../docs/_static/theme.json"); let themes = serde_json::json!({ "light": latkit::Theme::light(), "dark": latkit::Theme::dark() }); let text = serde_json::to_string_pretty(&themes).expect("themes serialize") + "\n"; @@ -40,13 +40,27 @@ fn docs_palette_is_the_default_themes() { } #[test] -fn every_input_has_a_schema_and_reads_what_it_describes() { +fn every_input_has_a_schema_matching_its_json_form() { let schemas = latkit::schemas(); - for name in ["Scene", "Data", "Patch", "Query", "Camera", "Fit", "Budget", "Shot", "Expr", "Theme", "Mark"] { + let inputs = [ + "Scene", + "Data", + "Patch", + "Query", + "EnvelopeRequest", + "Camera", + "Fit", + "Budget", + "Shot", + "Expr", + "Theme", + "Mark", + ]; + for name in inputs { assert!(schemas["$defs"][name].is_object(), "{name} has a schema"); } - let theme = &schemas["$defs"]["Theme"]["anyOf"]; - assert!(theme.to_string().contains("vertexRadius"), "options are in camelCase"); + let diagram = &schemas["$defs"]["DiagramTheme"]; + assert!(diagram.to_string().contains("groupPadding"), "options are in camelCase"); let names = &schemas["$defs"]["ColormapName"]["enum"]; - assert!(names.as_array().is_some_and(|n| n.iter().any(|n| n == "viridis")), "the catalog names its maps"); + assert!(names.as_array().is_some_and(|n| n.iter().any(|n| n == "viridis")), "the schema lists the colormap names"); } diff --git a/crates/latkit/tests/shadows.rs b/crates/latkit/tests/shadows.rs deleted file mode 100644 index b915cfb9..00000000 --- a/crates/latkit/tests/shadows.rs +++ /dev/null @@ -1,90 +0,0 @@ -//! Shadows fall onto the ground away from the light: under raised faces and vertices, sharp near -//! what casts them, and only from marks that cast. - -#![cfg(feature = "gpu")] - -mod common; - -use common::{SIZE, at, camera, errors, frame, gpu, plane}; -use latkit::{Color, Data, Faces, Ids, Patch, Scene, Shadow, Table, Theme, Vertices, View}; - -const GROUND: f32 = 0.5; - -fn theme() -> Theme { - Theme { - background: Color::rgb(GROUND, GROUND, GROUND), - shadow: Shadow { color: Color::BLACK.alpha(0.5), light: [-1.0, 0.0, 1.0], softness: 0.0, lift: 0.0 }, - ..Theme::dark() - } -} - -/// A square face 20 units across, raised 20 units, and a vertex raised as high. -fn data() -> Data { - let mut data = Data::new("mesh"); - data.apply( - Patch::new() - .set( - Table::new("Corner", Ids::count(4)) - .field("x", vec![-10.0, 10.0, 10.0, -10.0]) - .field("y", vec![-10.0, -10.0, 10.0, 10.0]) - .field("z", vec![20.0; 4]), - ) - .set( - Table::new("Face", Ids::count(2)) - .reference("a", "Corner", vec![0_u64, 0]) - .reference("b", "Corner", vec![1_u64, 2]) - .reference("c", "Corner", vec![2_u64, 3]), - ) - .set( - Table::new("Ball", Ids::count(1)) - .field("x", vec![-30.0]) - .field("y", vec![-40.0]) - .field("z", vec![20.0]), - ), - ) - .expect("valid"); - data -} - -fn scene(cast: bool) -> Scene { - Scene::new(plane()) - .theme(theme()) - .draw(Faces::of("mesh", "Face").corners("a", "b", "c").color(Color::WHITE).shadow(cast)) - .draw(Vertices::of("mesh", "Corner").position("x", "y").elevation("z").radius(0.0).pick(false)) - .draw( - Vertices::of("mesh", "Ball").position("x", "y").elevation("z").radius(8.0).color(Color::WHITE).shadow(cast), - ) -} - -fn ground() -> u8 { - (GROUND * 255.0).round() as u8 -} - -#[test] -fn raised_marks_shadow_the_ground_away_from_the_light() { - let Some(gpu) = gpu() else { return }; - let mut view = View::new(&gpu, scene(true)).expect("valid scene"); - view.set_camera(camera()); - view.set_data(data()); - let pixels = frame(&gpu, &mut view); - let c = SIZE / 2; - // The light leans toward -x, so a caster 20 units up shadows the ground 20 units toward +x. - let under_face = at(&pixels, c + 20, c); - assert!(under_face[0] < ground() - 40, "the face's shadow darkens the ground: {under_face:?}"); - let beside = at(&pixels, c - 20, c); - assert!(beside[0].abs_diff(ground()) <= 3, "the lit side stays: {beside:?}"); - let under_ball = at(&pixels, c - 10, c + 40); - assert!(under_ball[0] < ground() - 30, "the vertex's shadow: {under_ball:?}"); - assert_eq!(errors(&mut view), Vec::new()); -} - -#[test] -fn marks_that_do_not_cast_leave_the_ground_lit() { - let Some(gpu) = gpu() else { return }; - let mut view = View::new(&gpu, scene(false)).expect("valid scene"); - view.set_camera(camera()); - view.set_data(data()); - let pixels = frame(&gpu, &mut view); - let c = SIZE / 2; - assert!(at(&pixels, c + 20, c)[0].abs_diff(ground()) <= 3, "{:?}", at(&pixels, c + 20, c)); -} diff --git a/crates/latkit/tests/shots.rs b/crates/latkit/tests/shots.rs index 70b22d2c..2223e43d 100644 --- a/crates/latkit/tests/shots.rs +++ b/crates/latkit/tests/shots.rs @@ -1,6 +1,6 @@ -//! Shots: a picture draws what a view shows at a size of its own once it is complete and still, -//! panels compose side by side, a transparent picture keeps its alpha, and a frame tells whether -//! anything stood in for work under way. +//! Shots: a picture draws what a view shows, at its own size, once the view is complete and still. +//! Panels compose side by side, a transparent picture keeps its alpha, and a frame reports whether +//! it drew placeholders for pending work. Legends draw colormaps and named categories. #![cfg(feature = "gpu")] @@ -9,24 +9,27 @@ mod common; use std::time::Duration; use common::{SIZE, at, camera, close, frame, gpu, plain_theme, plane, step}; -use latkit::{Color, Data, Ids, Image, Scene, Shot, Table, Theme, Vertices, View}; +use latkit::{ + Areas, Color, Data, Edges, Faces, Fit, Ids, Image, Label, Lines, Patch, Scene, Shot, Table, Theme, Vertices, View, + field, +}; const RED: [u8; 4] = [255, 0, 0, 255]; const BLACK: [u8; 4] = [0, 0, 0, 255]; fn dot(x: f64, color: Color) -> (Scene, Data) { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![x]).field("y", vec![0.0])).expect("valid"); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(6.0).color(color)); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(6.0).color(color)); (scene, data) } -fn view(gpu: &latkit::Gpu, scene: Scene, data: Data) -> View { +fn view(gpu: &latkit::Gpu, scene: Scene, data: &Data) -> View { let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(data); + view.set_sources([data]).expect("a source"); view } @@ -36,10 +39,10 @@ fn pixel(image: &Image, x: u32, y: u32) -> [u8; 4] { } #[test] -fn a_picture_draws_what_the_view_shows() { +fn a_picture_draws_the_same_pixels_as_the_view() { let Some(gpu) = gpu() else { return }; let (scene, data) = dot(20.0, Color::rgb(1.0, 0.0, 0.0)); - let mut view = view(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let shown = frame(&gpu, &mut view); let image = pollster::block_on(Shot::new([SIZE, SIZE]).image(&mut [&mut view])).expect("a picture"); assert_eq!(image.size, [SIZE, SIZE]); @@ -50,29 +53,60 @@ fn a_picture_draws_what_the_view_shows() { } #[test] -fn pictures_make_only_what_they_draw_with() { +fn pictures_compile_only_the_pipelines_they_draw_with() { let Some(gpu) = gpu() else { return }; let (scene, data) = dot(20.0, Color::rgb(1.0, 0.0, 0.0)); - let mut view = view(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let shot = Shot::new([SIZE, SIZE]); pollster::block_on(shot.image(&mut [&mut view])).expect("a picture"); let made = gpu.work().pipelines; pollster::block_on(shot.image(&mut [&mut view])).expect("a picture"); - // A picture takes no input, so it never makes the probes a pointer would need, and what it makes - // it asked for first. - assert_eq!(gpu.work().pipelines, made, "another picture makes nothing"); - assert_eq!(gpu.work().late, 0, "nothing made where it was first used"); + // A picture takes no input, so it never creates the probe pipelines a pointer needs. Every + // pipeline it creates was requested up front. + assert_eq!(gpu.work().pipelines, made, "another picture compiles no pipeline"); + assert_eq!(gpu.work().late, 0, "no pipeline compiles at first use"); +} + +#[test] +fn every_kind_of_mark_draws_with_the_pipelines_it_requested() { + let Some(gpu) = gpu() else { return }; + let data = Data::new("graph"); + let square = vec![vec![[-60.0, -60.0], [-40.0, -60.0], [-40.0, -40.0], [-60.0, -40.0], [-60.0, -60.0]]]; + let nodes = Table::new("Node", vec![1_u64, 2, 3]) + .field("x", vec![-20.0, 20.0, 0.0]) + .field("y", vec![0.0, 0.0, 20.0]) + .field("name", vec!["a", "b", "c"]); + let link = + Table::new("Link", vec![1_u64]).reference("from", "Node", vec![1_u64]).reference("to", "Node", vec![2_u64]); + let face = Table::new("Face", vec![1_u64]) + .reference("a", "Node", vec![1_u64]) + .reference("b", "Node", vec![2_u64]) + .reference("c", "Node", vec![3_u64]); + let zone = Table::new("Zone", vec![1_u64]).field("geometry", vec![square]); + data.apply(Patch::new().set(nodes).set(link).set(face).set(zone)).expect("valid"); + let scene = Scene::new(plane()) + .theme(plain_theme()) + .draw(Areas::of("graph", "Zone")) + .draw(Lines::of("graph", "Zone")) + .draw(Faces::of("graph", "Face").corners("a", "b", "c")) + .draw(Edges::of("graph", "Link").ends("from", "to")) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).label(Label::new(field("name")))); + let mut view = view(&gpu, scene, &data); + // A fit measures every mark that frames it. + view.fit(Fit::All); + pollster::block_on(Shot::new([SIZE, SIZE]).image(&mut [&mut view])).expect("a picture"); + assert_eq!(gpu.work().late, 0, "every pipeline drawn with was requested, by the same name"); } #[test] fn a_scaled_picture_draws_the_same_view_sharper() { let Some(gpu) = gpu() else { return }; let (scene, data) = dot(20.0, Color::rgb(1.0, 0.0, 0.0)); - let mut view = view(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let image = pollster::block_on(Shot::new([2 * SIZE, 2 * SIZE]).scale(2.0).image(&mut [&mut view])).expect("a picture"); assert_eq!(image.size, [2 * SIZE, 2 * SIZE]); - // The same CSS layout at twice the pixels: the dot sits twice as far out, in twice the pixels. + // The same CSS layout at twice the pixels: the dot is twice as far out and twice as large. assert!(close(pixel(&image, SIZE + 40, SIZE), RED), "{:?}", pixel(&image, SIZE + 40, SIZE)); assert!(close(pixel(&image, SIZE + 40 + 10, SIZE), RED), "a radius of 6 CSS pixels is 12 pixels"); assert!(close(pixel(&image, SIZE, SIZE), BLACK)); @@ -83,16 +117,16 @@ fn panels_compose_side_by_side() { let Some(gpu) = gpu() else { return }; let (red_scene, red) = dot(0.0, Color::rgb(1.0, 0.0, 0.0)); let (blue_scene, blue) = dot(0.0, Color::rgb(0.0, 0.0, 1.0)); - let mut left = view(&gpu, red_scene, red); - let mut right = view(&gpu, blue_scene, blue); + let mut left = view(&gpu, red_scene, &red); + let mut right = view(&gpu, blue_scene, &blue); let shot = Shot::new([2 * SIZE, SIZE]).panels([[0.0, 0.0, 0.5, 1.0], [0.5, 0.0, 0.5, 1.0]]); let image = pollster::block_on(shot.image(&mut [&mut left, &mut right])).expect("a picture"); assert!(close(pixel(&image, SIZE / 2, SIZE / 2), RED), "{:?}", pixel(&image, SIZE / 2, SIZE / 2)); assert!(close(pixel(&image, SIZE + SIZE / 2, SIZE / 2), [0, 0, 255, 255])); - assert!(close(pixel(&image, SIZE, SIZE / 2), BLACK), "each panel's own background between"); - assert_eq!(gpu.work().late, 0, "what composes the panels was asked for first"); + assert!(close(pixel(&image, SIZE, SIZE / 2), BLACK), "each panel's own background shows between the dots"); + assert_eq!(gpu.work().late, 0, "the compose pipeline was requested ahead of use"); let wrong = pollster::block_on(shot.image(&mut [&mut left])); - assert!(wrong.is_err(), "a view per panel"); + assert!(wrong.is_err(), "a shot needs a view per panel"); } #[test] @@ -100,41 +134,39 @@ fn a_transparent_picture_keeps_its_alpha() { let Some(gpu) = gpu() else { return }; let (scene, data) = dot(0.0, Color::rgb(1.0, 0.0, 0.0)); let scene = scene.theme(Theme { background: Color::TRANSPARENT, ..plain_theme() }); - let mut view = view(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let image = pollster::block_on(Shot::new([SIZE, SIZE]).transparent().image(&mut [&mut view])).expect("a picture"); - assert_eq!(pixel(&image, 3, 3)[3], 0, "nothing drawn is clear"); + assert_eq!(pixel(&image, 3, 3)[3], 0, "undrawn pixels are clear"); assert!(close(pixel(&image, SIZE / 2, SIZE / 2), RED), "{:?}", pixel(&image, SIZE / 2, SIZE / 2)); let png = Image::from_png(&image.encode(latkit::ImageFormat::Png).expect("written")).expect("a PNG"); assert_eq!(png, image); } #[test] -fn a_picture_waits_for_what_eases_to_land() { +fn a_picture_waits_for_easing_to_finish() { let Some(gpu) = gpu() else { return }; let (scene, data) = dot(0.0, Color::rgb(1.0, 0.0, 0.0)); - let mut view = view(&gpu, scene.ease(Duration::from_secs(2)), data.clone()); + let mut view = view(&gpu, scene.ease(Duration::from_secs(2)), &data); frame(&gpu, &mut view); - let mut moved = data; - moved.apply(Table::new("Node", Ids::count(1)).field("x", vec![40.0])).expect("valid"); - view.set_data(moved); + data.apply(Table::new("Node", Ids::count(1)).field("x", vec![40.0])).expect("valid"); step(&gpu, &mut view); assert!(view.needs_render(), "the dot eases over two seconds"); let image = pollster::block_on(Shot::new([SIZE, SIZE]).image(&mut [&mut view])).expect("a picture"); - assert!(close(pixel(&image, SIZE / 2 + 40, SIZE / 2), RED), "the picture shows where it lands"); + assert!(close(pixel(&image, SIZE / 2 + 40, SIZE / 2), RED), "the picture shows the dot at its final position"); assert!(close(pixel(&image, SIZE / 2, SIZE / 2), BLACK)); } #[test] -fn a_frame_tells_whether_anything_stood_in() { +fn a_frame_reports_whether_it_drew_placeholders() { let Some(gpu) = gpu() else { return }; let (scene, data) = dot(0.0, Color::rgb(1.0, 0.0, 0.0)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); assert!(!view.complete(), "nothing drawn yet"); - view.set_data(data); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); assert!(view.complete()); - let waiting = view.stats().waiting; + let waiting = view.waiting(); assert!(waiting.is_empty(), "waiting on {waiting}"); } @@ -149,17 +181,17 @@ fn timeouts_and_font_errors_return_the_view_without_a_partial_image() { } let Some(gpu) = gpu() else { return }; let (scene, data) = dot(0.0, Color::rgb(1.0, 0.0, 0.0)); - let mut view = view(&gpu, scene, data); - let result = pollster::block_on(Shot::new([33, 35]).deadline(Duration::ZERO).image(&mut [&mut view])); + let mut view = view(&gpu, scene, &data); + let result = pollster::block_on(Shot::new([33, 35]).deadline(Some(Duration::ZERO)).image(&mut [&mut view])); assert_eq!(result.unwrap_err(), latkit::Error::Timeout); assert!(!view.lent()); gpu.set_host(BrokenText); - let mut data = Data::new("labels"); + let data = Data::new("labels"); data.apply(Table::new("Node", Ids::count(1)).field("name", vec!["Hello"])).unwrap(); - let scene = - Scene::new(latkit::Diagram::default()).draw(Vertices::of("labels", "Node").label(latkit::Label::new("name"))); - view.show(scene).unwrap(); - view.set_data(data); + let scene = Scene::new(latkit::Diagram::default()) + .draw(Vertices::of("labels", "Node").label(latkit::Label::new(field("name")))); + view.set_scene(scene).unwrap(); + view.set_sources([&data]).expect("a source"); let error = pollster::block_on(Shot::new([128, 128]).image(&mut [&mut view])).unwrap_err(); assert!(error.to_string().contains("test font failure")); assert!(!view.lent()); @@ -172,11 +204,11 @@ fn fixed_effect_time_is_repeatable_and_independent_of_capture_steps() { let (_, data) = dot(0.0, Color::rgb(1.0, 0.0, 0.0)); let scene = Scene::new(plane()).theme(plain_theme()).ease(Duration::ZERO).draw( Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .radius(latkit::elapsed() + 4.0) .color(Color::rgb(1.0, 0.0, 0.0)), ); - let mut view = view(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let shot = Shot::new([127, 129]).elapsed(Duration::ZERO); let first = pollster::block_on(shot.image(&mut [&mut view])).unwrap(); let later = pollster::block_on(shot.clone().elapsed(Duration::from_secs(12)).image(&mut [&mut view])).unwrap(); @@ -191,19 +223,19 @@ fn fixed_effect_time_is_repeatable_and_independent_of_capture_steps() { fn native_fonts_draw_visible_glyphs_into_the_export() { let Some(gpu) = gpu() else { return }; gpu.set_host(latkit::host::Native::new()); - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Table::new("Node", Ids::count(1)).field("x", vec![0.0]).field("y", vec![0.0]).field("name", vec!["Hello"]), ) .unwrap(); let scene = Scene::new(plane()).theme(plain_theme()).ease(Duration::ZERO).draw( Vertices::of("graph", "Node") - .position("x", "y") + .position(field("x"), field("y")) .radius(4.0) .color(Color::rgb(1.0, 0.0, 0.0)) - .label(latkit::Label::new("name").color(Color::WHITE).size(18.0)), + .label(latkit::Label::new(field("name")).color(Color::WHITE).size(18.0)), ); - let mut view = view(&gpu, scene, data); + let mut view = view(&gpu, scene, &data); let image = pollster::block_on(Shot::new([127, 129]).image(&mut [&mut view])).unwrap(); let white = image.rgba.as_chunks::<4>().0.iter().filter(|pixel| pixel[0] > 180 && pixel[1] > 180 && pixel[2] > 180).count(); @@ -213,7 +245,7 @@ fn native_fonts_draw_visible_glyphs_into_the_export() { #[test] fn a_legend_draws_its_marks_colormap_across_the_domain() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Table::new("Node", Ids::count(2)) .field("x", vec![-1.0, 1.0]) @@ -224,25 +256,58 @@ fn a_legend_draws_its_marks_colormap_across_the_domain() { let colors = latkit::field("v").colormap(latkit::Colormap::VIRIDIS).domain(0.0..=1.0); let scene = Scene::new(plane()) .theme(plain_theme()) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(4.0).color(colors).legend("")); - let mut view = view(&gpu, scene, data); + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0).color(colors).legend("")); + let mut view = view(&gpu, scene, &data); let image = pollster::block_on(Shot::new([400, 200]).image(&mut [&mut view])).expect("a picture"); - // The bar ends 16 px from the right; its middle row sits above the ticks' labels. + // The bar ends 16 px from the right edge. Its middle row is above the tick labels. let y = 200 - 16 - 16 - 4 - 4; let near = |a: [u8; 4], b: [u8; 3]| a.iter().zip(b).all(|(x, y)| x.abs_diff(y) <= 24); let low = pixel(&image, 400 - 16 - 218, y); let high = pixel(&image, 400 - 16 - 2, y); assert!(near(low, [68, 1, 84]), "viridis's low end: {low:?}"); assert!(near(high, [253, 231, 37]), "viridis's high end: {high:?}"); - let scene = Scene::new(plane()).draw(Vertices::of("graph", "Node").position("x", "y").legend("")); + let scene = Scene::new(plane()).draw(Vertices::of("graph", "Node").position(field("x"), field("y")).legend("")); assert!(View::new(&gpu, scene).is_err(), "a legend needs a colormap with a domain"); } +#[test] +fn a_plots_legend_stays_clear_of_its_axes() { + let Some(gpu) = gpu() else { return }; + let data = Data::new("d"); + let frames: Vec = (0..10).map(f64::from).collect(); + let values: Vec = (0..10_u8).map(f32::from).collect(); + data.apply(Patch::new().axis("time", "s").replace(Table::new("S", Ids::count(1)).field("k", vec![1.0])).append( + "S", + frames, + [("v", values)], + )) + .expect("valid"); + let colors = field("k").colormap(latkit::Colormap::VIRIDIS).domain(0.0..=1.0); + // The lowest row holding the colormap's yellow high end: the bottom of the legend's bar. + let bar_bottom = |scene: Scene| { + let mut view = view(&gpu, scene, &data); + let image = pollster::block_on(Shot::new([400, 200]).image(&mut [&mut view])).expect("a picture"); + let yellow = |p: [u8; 4]| p[0] > 200 && p[1] > 190 && p[2] < 90; + (0..200).rev().find(|&y| (300..400).any(|x| yellow(pixel(&image, x, y)))).expect("a legend") + }; + let on_plane = bar_bottom( + Scene::new(plane()) + .theme(plain_theme()) + .draw(Vertices::of("d", "S").position(0.0, 0.0).color(colors.clone()).legend("")), + ); + let on_plot = bar_bottom( + Scene::new(latkit::Plot::default()) + .theme(plain_theme()) + .draw(Lines::of("d", "S").trace("v").color(colors).legend("")), + ); + assert!(on_plot + 20 < on_plane, "above the axis labels: {on_plot} on the plot, {on_plane} on a plane"); +} + #[test] fn a_legend_lists_named_categories_by_their_colors() { - use latkit::{Colormap, ColormapKind, field}; + use latkit::{Colormap, ColormapKind}; let Some(gpu) = gpu() else { return }; - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Table::new("Node", Ids::count(2)) .field("x", vec![-1.0, 1.0]) @@ -252,15 +317,15 @@ fn a_legend_lists_named_categories_by_their_colors() { .expect("valid"); let map = Colormap::new(ColormapKind::Categorical, [Color::rgb(1.0, 0.0, 0.0), Color::rgb(0.0, 0.0, 1.0)]) .expect("two categories"); - let node = Vertices::of("graph", "Node").position("x", "y").radius(4.0); + let node = Vertices::of("graph", "Node").position(field("x"), field("y")).radius(4.0); let scene = Scene::new(plane()).theme(plain_theme()); let unnamed = scene.clone().draw(node.clone().color(field("k").colormap(map.clone())).legend("")); assert!(View::new(&gpu, unnamed).is_err(), "categories without names need a domain"); let named = field("k").colormap(map.categories(["red", "blue"])); - let mut view = view(&gpu, scene.draw(node.color(named).legend("")), data); + let mut view = view(&gpu, scene.draw(node.color(named).legend("")), &data); let image = pollster::block_on(Shot::new([400, 200]).image(&mut [&mut view])).expect("a picture"); - // Without a host no names are set, so the swatches end at the right margin, a row each above the - // bottom one: rows of 16.5 px up from 16 px above the bottom. + // Without a host no names are rasterized, so the swatches end at the right margin, one row each + // above the bottom margin: rows of 16.5 px, starting 16 px above the bottom. let x = 400 - 16 - 4; assert!(close(pixel(&image, x, 200 - 16 - 25), RED), "the first category: {:?}", pixel(&image, x, 159)); assert!(close(pixel(&image, x, 200 - 16 - 8), [0, 0, 255, 255]), "the second: {:?}", pixel(&image, x, 176)); diff --git a/crates/latkit/tests/tiles.rs b/crates/latkit/tests/tiles.rs index 26943be9..f577a6f1 100644 --- a/crates/latkit/tests/tiles.rs +++ b/crates/latkit/tests/tiles.rs @@ -1,5 +1,6 @@ -//! Tiles: the vector tiles the camera asks for and the time tiles a plot window asks for, fetched from -//! the host, decoded, and drawn as a dataset the view owns; GeoJSON read as data. +//! Tiles: the vector tiles the camera needs and the time tiles a plot window needs are fetched +//! from the host once for all views, decoded, and drawn as a dataset in each view. GeoJSON reads +//! as data. #![cfg(feature = "gpu")] @@ -8,14 +9,17 @@ mod common; use std::collections::BTreeMap; use std::sync::{Arc, Mutex}; -use common::{SIZE, at, errors, frame, gpu, plain_theme}; +use common::{SETTLE, SIZE, at, errors, frame, gpu, plain_theme, step}; use latkit::Tiles; use latkit::host::{Host, Raster, Request, TextRun}; -use latkit::{Areas, Axis, Camera, Color, Data, Earth, Ids, Lines, Patch, Pending, Plot, Scene, Table, Vertices, View}; +use latkit::{ + Areas, Axis, Camera, Color, Data, Earth, Ids, Lines, Patch, Pending, Plot, Projection, Query, Reading, Scene, + Source, Table, Vertices, View, +}; const C: u32 = SIZE / 2; -/// Protocol buffer fields, written. +/// Writes protocol buffer fields. #[derive(Default)] struct Writer(Vec); @@ -47,16 +51,37 @@ fn zig(v: i64) -> u32 { ((v << 1) ^ (v >> 63)) as u32 } -/// A tile whose layer `water` holds a square covering it and reaching into its buffer, named `bay`: -/// clipped, it covers the tile exactly. +/// A tile whose layer `water` has a square named `bay` that covers the tile and extends into its +/// buffer. Clipped, it covers the tile exactly. fn tile() -> Vec { let (x, side) = (-64_i64, 4224_i64); - let geometry = [9, zig(x), zig(x), 26, zig(side), 0, 0, zig(side), zig(-side), 0, 15]; + tile_of(&[9, zig(x), zig(x), 26, zig(side), 0, 0, zig(side), zig(-side), 0, 15]) +} + +/// A tile whose layer `water` has a feature named `bay` of 2,025 small squares: more points than a +/// frame can afford to shape. +fn dense_tile() -> Vec { + let (step, side) = (90_i64, 80_i64); + let mut geometry = Vec::new(); + let mut cursor = [0_i64, 0]; + for i in 0..45 { + for j in 0..45 { + let [x, y] = [5 + step * i, 5 + step * j]; + let to = [zig(x - cursor[0]), zig(y - cursor[1])]; + geometry.extend([9, to[0], to[1], 26, zig(side), 0, 0, zig(side), zig(-side), 0, 15]); + cursor = [x, y + side]; + } + } + tile_of(&geometry) +} + +/// A tile whose layer `water` has one polygon feature named `bay` with `geometry`. +fn tile_of(geometry: &[u32]) -> Vec { let mut feature = Writer::default(); feature.packed(2, &[0, 0]); feature.varint(3 << 3); feature.varint(3); - feature.packed(4, &geometry); + feature.packed(4, geometry); let mut value = Writer::default(); value.bytes(1, b"bay"); let mut layer = Writer::default(); @@ -69,13 +94,15 @@ fn tile() -> Vec { tile.0 } -/// A host serving every tile of `tiles/{z}/{x}/{y}` as [`tile`], counting the fetches. +/// A host serving every `tiles/{z}/{x}/{y}` tile as [`tile`], or as [`dense_tile`] with `dense`, +/// and recording the fetches. #[derive(Default)] struct Served { fetched: Mutex>, + dense: bool, } -/// What the GPU is given as its host: the served tiles, shared with the test. +/// The host given to the GPU: the served tiles, shared with the test. struct Serving(Arc); impl Host for Serving { @@ -85,48 +112,115 @@ impl Host for Serving { fn fetch(&self, request: &Request) -> Pending> { self.0.fetched.lock().expect("unpoisoned").push(request.url.to_string()); + let tile = if self.0.dense { dense_tile } else { tile }; Pending::ready(Ok(if request.url.starts_with("tiles/") { tile() } else { Vec::new() })) } } #[test] -fn tiles_the_camera_asks_for_draw_as_a_dataset() { +fn an_index_that_is_not_tile_json_fails_as_invalid_input() { + let Some(gpu) = gpu() else { return }; + // The host serves empty bytes for anything outside `tiles/`. + gpu.set_host(Serving(Arc::new(Served::default()))); + let scene = Scene::new(Earth::default()).draw(Areas::of("base", "water")); + let mut view = View::new(&gpu, scene).expect("valid scene"); + view.set_sources([&Tiles::new("base", "index.json")]).expect("a source"); + frame(&gpu, &mut view); + let errors = errors(&mut view); + let [latkit::Error::Invalid { path, .. }] = errors.as_slice() else { panic!("one invalid input: {errors:?}") }; + assert_eq!(path.to_string(), "base", "named by the tiles' dataset"); +} + +#[test] +fn tiles_the_camera_needs_draw_as_a_dataset() { let Some(gpu) = gpu() else { return }; let served = Arc::new(Served::default()); gpu.set_host(Serving(served.clone())); - let earth = Earth { graticule: false, ..Earth::mercator() }; + let earth = Earth { graticule: false, ..Earth::default() }; let scene = Scene::new(earth) .draw(Areas::of("base", "water").color(Color::hex("#ff0000").expect("hex"))) .theme(plain_theme()); - let mut view = View::new(&gpu, scene).expect("valid scene"); - view.set_camera(Camera::Map { center: [0.0, 0.0, 0.0], zoom: 2.0, pitch: 0.0, bearing: 0.0 }); - view.set_tiles(Tiles::new("base", "tiles/{z}/{x}/{y}.pbf")); + let camera = Camera::Map { center: [0.0, 0.0, 0.0], zoom: 2.0, pitch: 0.0, bearing: 0.0 }; + let tiles = Tiles::new("base", "tiles/{z}/{x}/{y}.pbf"); + let mut view = View::new(&gpu, scene.clone()).expect("valid scene"); + view.set_camera(camera); + view.set_sources([&tiles]).expect("a source"); let pixels = frame(&gpu, &mut view); let fetched = served.fetched.lock().expect("unpoisoned").clone(); assert!(!fetched.is_empty() && fetched.iter().all(|url| url.starts_with("tiles/")), "{fetched:?}"); - let before: Vec = - view.data("base").and_then(|d| d.table("water")).expect("the tiles' dataset").ids().collect(); + let ids = |view: &View| -> Vec { + let water = Query::of("water").ids_alone(); + view.snapshot("base").and_then(|d| d.query(&water).ok()).expect("the tiles' dataset").table.ids().collect() + }; + let before = ids(&view); assert!(before.len() >= 4, "a row per tile drawn: {}", before.len()); let stats = view.stats(); assert_eq!((stats.tiles, stats.tiles_loading), (before.len() as u64, 0), "stats count the tiles drawn"); assert!(at(&pixels, C + 8, C + 8)[0] > 150, "filled: {:?}", at(&pixels, C + 8, C + 8)); assert_eq!(errors(&mut view), Vec::new()); - // Zooming in asks for finer tiles, which replace their ancestors. + // A second view of the same tiles draws what the first fetched, without fetching again. + let mut second = View::new(&gpu, scene).expect("valid scene"); + second.set_camera(camera); + second.set_sources([&tiles]).expect("a source"); + let asked = served.fetched.lock().expect("unpoisoned").len(); + frame(&gpu, &mut second); + assert_eq!(served.fetched.lock().expect("unpoisoned").len(), asked, "fetched once for every view"); + assert_eq!(ids(&second), before, "the same rows"); + drop(second); + // Zooming in needs finer tiles, which replace their ancestors in the frame where the last of + // them arrives. The ancestors' rows stay in the dataset, not drawn, until they outnumber the + // rows drawn. view.set_camera(Camera::Map { center: [10.0, 10.0, 0.0], zoom: 6.0, pitch: 0.0, bearing: 0.0 }); + let mut loading = false; + for _ in 0..SETTLE { + step(&gpu, &mut view); + let now = view.stats().tiles_loading > 0; + if loading && !now { + break; + } + loading |= now; + } + frame(&gpu, &mut view); + let stats = view.stats(); + assert_eq!(stats.drawn, stats.tiles, "a row per tile drawn: the finer tiles' rows, not their ancestors'"); + let held = ids(&view); + assert!(before.iter().all(|id| held.contains(id)), "the ancestors' rows stay while they are few"); + view.set_sources::([]).expect("no sources"); + assert!(view.snapshot("base").is_none(), "a dataset no source provides is removed"); +} + +#[test] +fn tiles_arrive_shaped_by_the_jobs_that_decode_them() { + let Some(gpu) = gpu() else { return }; + let served = Arc::new(Served { dense: true, ..Served::default() }); + gpu.set_host(Serving(served.clone())); + let earth = Earth { graticule: false, ..Earth::default() }; + let scene = Scene::new(earth).draw(Areas::of("base", "water")).theme(plain_theme()); + let mut view = View::new(&gpu, scene).expect("valid scene"); + let camera = |zoom| Camera::Map { center: [10.0, 10.0, 0.0], zoom, pitch: 0.0, bearing: 0.0 }; + view.set_camera(camera(2.0)); + view.set_sources([&Tiles::new("base", "tiles/{z}/{x}/{y}.pbf")]).expect("a source"); frame(&gpu, &mut view); - let after: Vec = view.data("base").and_then(|d| d.table("water")).expect("rows").ids().collect(); - assert!(after.iter().all(|id| !before.contains(id)), "finer tiles' rows"); - view.remove_data("base"); - assert!(view.data("base").is_none()); + // As the camera zooms in and out, tiles come and go. A job decodes each tile and shapes its + // rows, and no frame shapes them again, even when it draws rows an earlier frame loaded. + let fetches = || served.fetched.lock().expect("unpoisoned").len() as u64; + let (jobs, asked) = (gpu.work().jobs, fetches()); + for zoom in [6.0, 0.0] { + view.set_camera(camera(zoom)); + frame(&gpu, &mut view); + } + assert!(fetches() > asked, "tiles were fetched"); + assert_eq!(gpu.work().jobs - jobs, fetches() - asked, "one job per tile"); + assert_eq!(errors(&mut view), Vec::new()); } -/// A host serving files by path, keeping the paths asked for. +/// A host serving files by path, recording the requested paths. struct Files { files: BTreeMap>, asked: Mutex>, } -/// What the GPU is given as its host: the files, shared with the test. +/// The host given to the GPU: the files, shared with the test. struct ServingFiles(Arc); impl Host for ServingFiles { @@ -141,19 +235,20 @@ impl Host for ServingFiles { } #[test] -fn time_tiles_draw_the_spans_a_plot_window_asks_for() { +fn time_tiles_draw_the_spans_a_plot_window_needs() { let Some(gpu) = gpu() else { return }; - let mut data = Data::new("signals"); + let data = Data::new("signals"); data.apply(Patch::new().axis("time", "s").set(Table::new("Signal", Ids::count(1)))).expect("valid"); let times: Vec = (0..20_000).map(f64::from).collect(); let values: Vec = times.iter().map(|t| 0.5 + 0.4 * (t / 50.0).sin()).collect(); - data.apply(Patch::new().append(×, "Signal", [("v", values)])).expect("frames"); + data.apply(Patch::new().append("Signal", ×, [("v", values)])).expect("frames"); let mut files = BTreeMap::new(); - data.to_tiles(256, |path, bytes| { - files.insert(format!("history/{path}"), bytes.to_vec()); - Ok(()) - }) - .expect("tiles"); + data.snapshot() + .to_tiles(256, |path, bytes| { + files.insert(format!("history/{path}"), bytes.to_vec()); + Ok(()) + }) + .expect("tiles"); let written = files.len(); let served = Arc::new(Files { files, asked: Mutex::default() }); gpu.set_host(ServingFiles(served.clone())); @@ -164,16 +259,16 @@ fn time_tiles_draw_the_spans_a_plot_window_asks_for() { .draw(Lines::of("signals", "Signal").trace("v").color(red).width(2.0)); let camera = Camera::Plot { x: [1000.0, 1400.0], y: Some([0.0, 1.0]) }; let mut whole = View::new(&gpu, scene.clone()).expect("valid scene"); - whole.set_data(data); + whole.set_sources([&data]).expect("a source"); whole.set_camera(camera); let expected = frame(&gpu, &mut whole); let mut tiled = View::new(&gpu, scene).expect("valid scene"); - tiled.set_tiles(Tiles::new("signals", "history/frames.json")); + tiled.set_sources([Tiles::new("signals", "history/frames.json")]).expect("a source"); tiled.set_camera(camera); let pixels = frame(&gpu, &mut tiled); assert_eq!(errors(&mut tiled), Vec::new()); let stats = tiled.stats(); - assert_eq!(stats.tiles_loading, 0, "every span asked for landed"); + assert_eq!(stats.tiles_loading, 0, "every requested span arrived"); let differing = pixels.iter().zip(&expected).filter(|(a, b)| !common::close(**a, **b)).count(); if differing > SIZE as usize { let ascii = |p: &[[u8; 4]]| { @@ -190,7 +285,7 @@ fn time_tiles_draw_the_spans_a_plot_window_asks_for() { assert!(differing <= SIZE as usize, "the window draws as the whole history does: {differing} pixels differ"); assert!(pixels.iter().any(|p| p[0] > 200), "the trace draws"); let asked = served.asked.lock().expect("unpoisoned").clone(); - assert!(asked.len() * 8 < written, "only the spans the window draws: {asked:?} of {written}"); + assert!(asked.len() * 8 < written, "only the spans the window draws are fetched: {asked:?} of {written}"); } #[test] @@ -201,8 +296,9 @@ fn geojson_draws_as_lines_and_vertices_on_the_globe() { "geometry": {"type": "LineString", "coordinates": [[-60, 0], [60, 0]]}}, {"type": "Feature", "properties": {"name": "center"}, "geometry": {"type": "Point", "coordinates": [0, 30]}}]}"#; - let data = Data::read(bytes, Some("map")).expect("GeoJSON"); - let earth = Earth { graticule: false, ..Earth::globe() }; + let data = + pollster::block_on(Data::read(bytes, Reading { name: Some("map"), ..Reading::default() })).expect("GeoJSON"); + let earth = Earth { graticule: false, ..Earth::default().projection(Projection::Globe) }; let red = Color::hex("#ff0000").expect("hex"); let scene = Scene::new(earth) .draw(Lines::of("map", "lines").width(3.0).color(red)) @@ -210,10 +306,10 @@ fn geojson_draws_as_lines_and_vertices_on_the_globe() { .theme(plain_theme()); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(Camera::Map { center: [0.0, 0.0, 0.0], zoom: 0.5, pitch: 0.0, bearing: 0.0 }); - view.set_data(data); + view.set_sources([&data]).expect("a source"); let pixels = frame(&gpu, &mut view); assert!(at(&pixels, C, C)[0] > 150, "the long line bends over the globe through its middle"); - let dot = view.point([0.0, 30.0, 0.0]).expect("in view"); + let dot = view.screen([0.0, 30.0, 0.0]).expect("in view"); assert!(at(&pixels, dot[0] as u32, dot[1] as u32)[0] > 150, "the point"); assert_eq!(errors(&mut view), Vec::new()); } diff --git a/crates/latkit/tests/transitions.rs b/crates/latkit/tests/transitions.rs index 99b4a95b..11b54950 100644 --- a/crates/latkit/tests/transitions.rs +++ b/crates/latkit/tests/transitions.rs @@ -1,6 +1,7 @@ -//! Changes ease: a moved vertex starts where it was, a removed one fades where it stood, and edges -//! follow their vertices; another dataset of the same name replaces rows rather than moving them; -//! sampled fields and new frames never ease. +//! Changes ease: a moved vertex starts from its old position, a removed one fades in place, and +//! edges follow their vertices. Another dataset with the same name replaces rows instead of moving +//! them. Sampled fields and new frames never ease. A transition evaluates its channels once, and a +//! change to one page evaluates only that page's rows. #![cfg(feature = "gpu")] @@ -9,21 +10,21 @@ mod common; use std::time::Duration; use common::{SIZE, at, camera, close, errors, frame, gpu, plain_theme, plane, step}; -use latkit::{Color, Data, Edges, Ids, Patch, Scene, Table, Theme, Vertices, View}; +use latkit::{Color, Data, Edges, Ids, Patch, Scene, Table, Vertices, View, field}; const WHITE: [u8; 4] = [255, 255, 255, 255]; const BLACK: [u8; 4] = [0, 0, 0, 255]; fn scene() -> Scene { Scene::new(plane()) - .theme(Theme { edge: Some(Color::rgb(0.0, 1.0, 0.0)), ..plain_theme() }) + .theme(common::green_edges()) .ease(Duration::from_secs(2)) .draw(Edges::of("graph", "Link").ends("from", "to").width(2.0)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(6.0).color(Color::WHITE)) + .draw(Vertices::of("graph", "Node").position(field("x"), field("y")).radius(6.0).color(Color::WHITE)) } fn graph() -> Data { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply( Patch::new() .set(Table::new("Node", vec![1_u64, 2]).field("x", vec![-30.0, 30.0]).field("y", vec![0.0, 0.0])) @@ -42,19 +43,18 @@ fn moved_vertices_start_where_they_were_and_their_edges_follow() { let Some(gpu) = gpu() else { return }; let mut view = View::new(&gpu, scene()).expect("valid scene"); view.set_camera(camera()); - let mut data = graph(); - view.set_data(data.clone()); + let data = graph(); + view.set_sources([&data]).expect("a source"); let c = SIZE / 2; let pixels = frame(&gpu, &mut view); assert!(close(at(&pixels, c - 30, c), WHITE), "{:?}", at(&pixels, c - 30, c)); data.apply(Table::new("Node", vec![1_u64]).field("y", vec![40.0])).expect("valid"); - view.set_data(data); let pixels = step(&gpu, &mut view); assert!(close(at(&pixels, c - 30, c), WHITE), "the moved vertex starts where it was"); assert!(close(at(&pixels, c - 30, c - 40), BLACK), "and has not jumped"); assert!(view.needs_render(), "the transition runs"); let pixels = frame(&gpu, &mut view); - assert!(close(at(&pixels, c - 30, c - 40), WHITE), "it ends where its data says"); + assert!(close(at(&pixels, c - 30, c - 40), WHITE), "it ends at its new position"); assert!(close(at(&pixels, c - 30, c), BLACK)); let middle = at(&pixels, c, c - 20); assert!(middle[1] > 200 && middle[0] < 60, "the edge joins the moved vertex: {middle:?}"); @@ -66,8 +66,8 @@ fn removed_rows_fade_where_they_stood_and_new_rows_grow() { let Some(gpu) = gpu() else { return }; let mut view = View::new(&gpu, scene()).expect("valid scene"); view.set_camera(camera()); - let mut data = graph(); - view.set_data(data.clone()); + let data = graph(); + view.set_sources([&data]).expect("a source"); let c = SIZE / 2; frame(&gpu, &mut view); data.apply( @@ -77,10 +77,9 @@ fn removed_rows_fade_where_they_stood_and_new_rows_grow() { .set(Table::new("Node", vec![3_u64]).field("x", vec![0.0]).field("y", vec![-40.0])), ) .expect("valid"); - view.set_data(data); let pixels = step(&gpu, &mut view); assert!(close(at(&pixels, c - 30, c), WHITE), "the removed vertex still shows: {:?}", at(&pixels, c - 30, c)); - assert!(close(at(&pixels, c + 30, c), WHITE), "the kept one stays"); + assert!(close(at(&pixels, c + 30, c), WHITE), "the remaining one stays"); assert!(at(&pixels, c, c + 40)[0] < 60, "the new vertex starts from nothing: {:?}", at(&pixels, c, c + 40)); let pixels = frame(&gpu, &mut view); assert!(close(at(&pixels, c - 30, c), BLACK), "the removed vertex is gone"); @@ -95,13 +94,13 @@ fn another_dataset_of_the_same_name_replaces_rows_rather_than_moving_them() { let Some(gpu) = gpu() else { return }; let mut view = View::new(&gpu, scene()).expect("valid scene"); view.set_camera(camera()); - view.set_data(graph()); + view.set_sources([graph()]).expect("a source"); let c = SIZE / 2; frame(&gpu, &mut view); - // The same ids in a dataset of its own: its rows are not the drawn ones moved. - let mut other = graph(); + // The same ids in a separate dataset: its rows are new rows, not the drawn ones moved. + let other = graph(); other.apply(Table::new("Node", vec![1_u64]).field("y", vec![40.0])).expect("valid"); - view.set_data(other); + view.set_sources([&other]).expect("a source"); let pixels = step(&gpu, &mut view); assert!(close(at(&pixels, c - 30, c), WHITE), "the replaced vertex fades where it stood"); assert!(at(&pixels, c - 30, c - 40)[0] < 60, "its replacement grows from nothing"); @@ -112,34 +111,65 @@ fn another_dataset_of_the_same_name_replaces_rows_rather_than_moving_them() { } #[test] -fn sampled_fields_and_zero_ease_change_at_once() { +fn zero_ease_moves_a_vertex_at_once() { let Some(gpu) = gpu() else { return }; let mut view = View::new(&gpu, scene().ease(Duration::ZERO)).expect("valid scene"); view.set_camera(camera()); - let mut data = graph(); - view.set_data(data.clone()); + let data = graph(); + view.set_sources([&data]).expect("a source"); let c = SIZE / 2; frame(&gpu, &mut view); data.apply(Table::new("Node", vec![1_u64]).field("y", vec![40.0])).expect("valid"); - view.set_data(data); let pixels = step(&gpu, &mut view); assert!(close(at(&pixels, c - 30, c - 40), WHITE), "without ease the vertex moves at once"); } +#[test] +fn sampled_fields_and_new_frames_change_at_once() { + let Some(gpu) = gpu() else { return }; + let mut view = View::new(&gpu, scene()).expect("valid scene"); + view.set_camera(camera()); + let data = Data::new("graph"); + data.apply( + Patch::new() + .set(Table::new("Node", vec![1_u64, 2]).field("x", vec![-30.0, 30.0])) + .set(Table::new("Link", Ids::count(1)).reference("from", "Node", vec![1_u64]).reference( + "to", + "Node", + vec![2_u64], + )) + .append("Node", [0.0], [("y", vec![0.0, 0.0])]) + .append("Node", [1.0], [("y", vec![40.0, 0.0])]), + ) + .expect("valid"); + view.set_sources([&data]).expect("a source"); + view.set_at(0.0); + let c = SIZE / 2; + frame(&gpu, &mut view); + // The scene eases over two seconds, but a sampled field shows its value at `at` at once. + view.set_at(1.0); + let pixels = step(&gpu, &mut view); + assert!(close(at(&pixels, c - 30, c - 40), WHITE), "the vertex shows the value at the new coordinate"); + data.apply(Patch::new().append("Node", [2.0], [("y", vec![-40.0, 0.0])])).expect("valid"); + view.set_at(2.0); + let pixels = step(&gpu, &mut view); + assert!(close(at(&pixels, c - 30, c + 40), WHITE), "a new frame shows at once"); + assert_eq!(errors(&mut view), Vec::new()); +} + #[test] fn a_transition_evaluates_once_and_eases_without_evaluating() { let Some(gpu) = gpu() else { return }; let mut view = View::new(&gpu, scene()).expect("valid scene"); view.set_camera(camera()); - let mut data = graph(); - view.set_data(data.clone()); + let data = graph(); + view.set_sources([&data]).expect("a source"); let c = SIZE / 2; frame(&gpu, &mut view); data.apply(Table::new("Node", vec![1_u64]).field("y", vec![40.0])).expect("valid"); - view.set_data(data); step(&gpu, &mut view); let evaluated = gpu.work().evaluated; - // A second into a transition of two. + // One second into a two-second transition. let mut pixels = Vec::new(); for _ in 0..60 { pixels = step(&gpu, &mut view); @@ -151,23 +181,22 @@ fn a_transition_evaluates_once_and_eases_without_evaluating() { } #[test] -fn a_change_to_one_page_evaluates_its_rows_alone() { +fn a_change_to_one_page_evaluates_only_its_rows() { let Some(gpu) = gpu() else { return }; let rows = (1 << 16) + 16; let scene = Scene::new(plane()) .theme(plain_theme()) .ease(Duration::ZERO) - .draw(Vertices::of("points", "P").position("x", "y").radius(1.0).color(Color::WHITE)); + .draw(Vertices::of("points", "P").position(field("x"), field("y")).radius(1.0).color(Color::WHITE)); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); let xs: Vec = (0..rows).map(|r| f64::from(r % 100) - 50.0).collect(); - let mut data = Data::new("points"); + let data = Data::new("points"); data.apply(Table::new("P", Ids::count(rows)).field("x", xs).field("y", vec![0.0; rows as usize])).expect("valid"); - view.set_data(data.clone()); + view.set_sources([&data]).expect("a source"); frame(&gpu, &mut view); let before = gpu.work().evaluated; data.apply(Table::new("P", vec![u64::from(rows - 1)]).field("y", vec![10.0])).expect("valid"); - view.set_data(data); frame(&gpu, &mut view); assert_eq!(gpu.work().evaluated - before, 16, "only the last page's rows evaluate"); } diff --git a/crates/latkit/tests/validation.rs b/crates/latkit/tests/validation.rs index ce47361d..52e84120 100644 --- a/crates/latkit/tests/validation.rs +++ b/crates/latkit/tests/validation.rs @@ -1,10 +1,13 @@ //! Batch validation uses the rendering compiler without requesting an adapter. #![cfg(feature = "gpu")] -use latkit::{Data, Diagram, Error, Ids, Label, Plane, Scene, Table, Vertices}; +use latkit::{ + Color, Colormap, Data, Diagram, Error, Expr, Ids, Label, Plane, Scene, Source, Table, Tiles, Vertices, field, when, + zoom, +}; -fn data() -> Data { - let mut data = Data::new("grid"); +fn data() -> Source { + let data = Data::new("grid"); data.apply( Table::new("Bus", Ids::count(2)) .field("x", vec![0.0, 1.0]) @@ -12,33 +15,76 @@ fn data() -> Data { .field("name", vec!["A", "B"]), ) .unwrap(); - data + data.into() } #[test] fn validation_requires_unique_datasets_and_drawn_tables() { - let scene = Scene::new(Plane::default()).draw(Vertices::of("grid", "Bus").position("x", "y")); - assert!(scene.validate(&[data()], &[]).is_ok()); - assert!(matches!(scene.validate(&[], &[]), Err(Error::Invalid { .. }))); - assert!(scene.validate(&[data(), data()], &[]).unwrap_err().to_string().contains("duplicate")); - assert!(scene.validate(&[Data::new("grid")], &[]).unwrap_err().to_string().contains("Bus")); - let tiles = [latkit::Tiles::new("grid", "history/frames.json")]; - assert!(scene.validate(&[], &tiles).is_ok(), "tiles' marks are checked once they land"); + let scene = Scene::new(Plane::default()).draw(Vertices::of("grid", "Bus").position(field("x"), field("y"))); + assert!(scene.validate(&[data()]).is_ok()); + assert!(matches!(scene.validate(&[]), Err(Error::Invalid { .. }))); + assert!(scene.validate(&[data(), data()]).unwrap_err().to_string().contains("two sources named")); + assert!(scene.validate(&[Data::new("grid").into()]).unwrap_err().to_string().contains("Bus")); + let tiles = [Tiles::new("grid", "history/frames.json").into()]; + assert!(scene.validate(&tiles).is_ok(), "tiles' marks are checked once the tiles arrive"); } #[test] fn validation_checks_expression_types_and_gives_the_mark_path() { - let scene = Scene::new(Plane::default()).draw(Vertices::of("grid", "Bus").position("name", "y")); - let Error::Invalid { path, .. } = scene.validate(&[data()], &[]).unwrap_err() else { panic!("invalid expression") }; + let scene = Scene::new(Plane::default()).draw(Vertices::of("grid", "Bus").position(field("name"), field("y"))); + let Error::Invalid { path, .. } = scene.validate(&[data()]).unwrap_err() else { panic!("invalid expression") }; assert!(path.to_string().starts_with("marks[0].")); - let scene = Scene::new(Plane::default()).draw(Vertices::of("grid", "Bus").label(Label::new("missing"))); - assert!(scene.validate(&[data()], &[]).is_err()); + let scene = Scene::new(Plane::default()).draw(Vertices::of("grid", "Bus").label(Label::new(field("missing")))); + assert!(scene.validate(&[data()]).is_err()); +} + +#[test] +fn a_label_text_chooses_between_one_text_field_and_missing() { + let on_a_plane = |text: Expr| { + let buses = Vertices::of("grid", "Bus").position(field("x"), field("y")).label(Label::new(text)); + Scene::new(Plane::default()).draw(buses).validate(&[data()]) + }; + let some = when(field("x").gt(0.5)).then(field("name")).otherwise(Expr::missing()); + assert!(on_a_plane(some).is_ok(), "the field where the condition holds, else no label"); + let two = when(field("x").gt(0.5)).then(field("name")).otherwise(field("other")); + let Error::Invalid { path, .. } = on_a_plane(two).unwrap_err() else { panic!("two fields") }; + assert_eq!(path.to_string(), "marks[0].label.text"); + let number = when(field("x").gt(0.5)).then(field("name")).otherwise(1.0); + assert!(on_a_plane(number).is_err(), "a value other than the field or missing"); + let zoomed = when(zoom().ge(3.0)).then(field("name")).otherwise(Expr::missing()); + assert!(on_a_plane(zoomed.clone()).is_ok(), "the zoom chooses on a plane"); + let blocks = Vertices::of("grid", "Bus").position(field("x"), field("y")).label(Label::new(zoomed)); + let in_a_diagram = Scene::new(Diagram::default()).draw(blocks).validate(&[data()]); + assert!(in_a_diagram.unwrap_err().to_string().contains("not the zoom"), "a diagram's text reads only data"); +} + +#[test] +fn a_color_option_computes_a_color() { + let colored = |color: Expr| { + let buses = Vertices::of("grid", "Bus").position(field("x"), field("y")).color(color); + Scene::new(Plane::default()).draw(buses).validate(&[data()]) + }; + assert!(colored(Color::rgb(1.0, 0.0, 0.0).into()).is_ok()); + assert!(colored(field("x").colormap(Colormap::VIRIDIS).into()).is_ok()); + assert!(colored(when(field("x").gt(0.5)).then(Color::WHITE).otherwise(Expr::missing())).is_ok()); + let error = colored(field("x")).unwrap_err().to_string(); + assert!(error.starts_with("marks[0].color"), "the error names the option: {error}"); + assert!(error.contains("found a number; map a number to colors with colormap"), "{error}"); + assert!(colored(field("name")).is_err(), "a text is no color"); } #[test] fn diagrams_allow_positions_to_be_derived_by_layout() { let scene = Scene::new(Diagram::default()).draw(Vertices::of("grid", "Bus")).layout(latkit::Layered::default()); - assert!(scene.validate(&[data()], &[]).is_ok()); + assert!(scene.validate(&[data()]).is_ok()); +} + +#[test] +fn shadows_are_valid_only_on_a_diagram() { + let block = Vertices::of("grid", "Bus").position(field("x"), field("y")).shadow(true); + let on_a_plane = Scene::new(Plane::default()).draw(block.clone()).validate(&[data()]); + assert!(on_a_plane.unwrap_err().to_string().contains("only a diagram has"), "refused on a plane"); + assert!(Scene::new(Diagram::default()).draw(block).validate(&[data()]).is_ok()); } #[test] diff --git a/crates/latkit/tests/videos.rs b/crates/latkit/tests/videos.rs index d1fd7e42..e71fb546 100644 --- a/crates/latkit/tests/videos.rs +++ b/crates/latkit/tests/videos.rs @@ -1,5 +1,5 @@ -//! Videos: frames compose into NV12 on the GPU, a frame where nothing moved is held rather than drawn -//! or encoded, keyframes fall at their times, and what the encoder gives is written as MP4. +//! Videos: frames compose into NV12 on the GPU, a frame where nothing moved is held instead of +//! drawn or encoded, keyframes fall at their times, and the encoder's output is written as MP4. #![cfg(feature = "gpu")] @@ -12,13 +12,13 @@ use std::time::Duration; use common::{SIZE, camera, gpu, plain_theme, plane}; use latkit::Ids; use latkit::host::{Encoder, Host, Raster, TextRun, Unit}; -use latkit::{Color, Data, Encoding, Label, Mp4, Pending, Pixels, Scene, Shot, Sink, Table, Vertices, View}; +use latkit::{Color, Data, Encoding, Label, Mp4, Pending, Pixels, Scene, Shot, Sink, Table, Vertices, View, field}; -/// The pictures an encoder was given: time, whether a keyframe, and the NV12 bytes. +/// The pictures an encoder received: time, whether it is a keyframe, and the NV12 bytes. type Pictures = Arc)>>>; -/// A host whose encoder keeps the pictures it is given and answers each with a stand-in unit: a -/// parameter set pair and a slice, which the muxer takes as H.264. +/// A host whose encoder keeps the pictures it receives and returns a placeholder unit for each: a +/// parameter set pair and a slice, which the muxer accepts as H.264. struct Keeping(Pictures); impl Host for Keeping { @@ -50,23 +50,22 @@ impl Encoder for Kept { } fn dot(x: f64) -> Data { - let mut data = Data::new("graph"); + let data = Data::new("graph"); data.apply(Table::new("Node", Ids::count(1)).field("x", vec![x]).field("y", vec![0.0])).expect("valid"); data } fn view(gpu: &latkit::Gpu) -> View { - let scene = Scene::new(plane()) - .theme(plain_theme()) - .ease(Duration::ZERO) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(20.0).color(Color::rgb(1.0, 0.0, 0.0))); + let scene = Scene::new(plane()).theme(plain_theme()).ease(Duration::ZERO).draw( + Vertices::of("graph", "Node").position(field("x"), field("y")).radius(20.0).color(Color::rgb(1.0, 0.0, 0.0)), + ); let mut view = View::new(gpu, scene).expect("valid scene"); view.set_camera(camera()); - view.set_data(dot(0.0)); + view.set_sources([dot(0.0)]).expect("a source"); view } -/// Luma and chroma of NV12 `bytes` of a square picture of `SIZE` at pixel `x`, `y`. +/// The luma and chroma at pixel (`x`, `y`) of NV12 `bytes` for a `SIZE` by `SIZE` picture. fn sample(bytes: &[u8], x: u32, y: u32) -> [u8; 3] { let luma = bytes[(y * SIZE + x) as usize]; let chroma = (SIZE * SIZE + (y / 2) * SIZE + (x / 2) * 2) as usize; @@ -88,13 +87,13 @@ fn frames_compose_into_nv12_and_hold_where_nothing_moved() { for _ in 0..6 { video.frame(&mut [&mut view]).await.expect("a frame"); } - view.set_data(dot(30.0)); + view.set_sources([dot(30.0)]).expect("a source"); for _ in 0..3 { video.frame(&mut [&mut view]).await.expect("a frame"); } }); assert_eq!(video.frames(), 9); - assert!(!view.lent() || video.frames() > 0, "the video holds the view while it lasts"); + assert!(!view.lent() || video.frames() > 0, "the view is lent to the video while it lasts"); let file = pollster::block_on(video.finish()).expect("a file").into_inner(); assert!(!view.lent(), "a finished video gives the view back"); let pictures = pictures.lock().unwrap_or_else(PoisonError::into_inner); @@ -103,7 +102,7 @@ fn frames_compose_into_nv12_and_hold_where_nothing_moved() { let [first, moved] = [&pictures[0].2, &pictures[1].2]; assert_eq!(first.len(), (SIZE * SIZE * 3 / 2) as usize); let c = SIZE / 2; - // BT.709 limited range: red is 63, 102, 240; black is 16, 128, 128. + // BT.709 limited range: red is (63, 102, 240) and black is (16, 128, 128). assert!(near(sample(first, c, c), [63, 102, 240]), "{:?}", sample(first, c, c)); assert!(near(sample(first, 2, 2), [16, 128, 128]), "{:?}", sample(first, 2, 2)); assert!(near(sample(moved, c + 30, c), [63, 102, 240]), "the moved dot"); @@ -116,24 +115,23 @@ fn frames_compose_into_nv12_and_hold_where_nothing_moved() { fn labeled_marks_easing_between_frames_complete_every_frame() { let Some(gpu) = gpu() else { return }; gpu.set_host(Keeping(Pictures::default())); - let scene = Scene::new(plane()) - .theme(plain_theme()) - .ease(Duration::from_millis(300)) - .draw(Vertices::of("graph", "Node").position("x", "y").radius(6.0).label(Label::new("name"))); + let scene = Scene::new(plane()).theme(plain_theme()).ease(Duration::from_millis(300)).draw( + Vertices::of("graph", "Node").position(field("x"), field("y")).radius(6.0).label(Label::new(field("name"))), + ); let mut view = View::new(&gpu, scene).expect("valid scene"); view.set_camera(camera()); let named = |x: f64| { - let mut data = dot(x); + let data = dot(x); data.apply(Table::new("Node", Ids::count(1)).field("name", vec!["a"])).expect("valid"); data }; - view.set_data(named(0.0)); - let shot = Shot::new([SIZE, SIZE]).deadline(Duration::from_secs(5)); + view.set_sources([named(0.0)]).expect("a source"); + let shot = Shot::new([SIZE, SIZE]).deadline(Some(Duration::from_secs(5))); let mut video = shot.video(&gpu, 30.0, Mp4::new(Vec::new())).expect("a video"); let started = std::time::Instant::now(); pollster::block_on(async { for i in 0..10 { - view.set_data(named(f64::from(i) * 3.0)); + view.set_sources([named(f64::from(i) * 3.0)]).expect("a source"); video.frame(&mut [&mut view]).await.expect("a frame"); } }); @@ -143,15 +141,15 @@ fn labeled_marks_easing_between_frames_complete_every_frame() { } #[test] -fn a_video_is_whole_blocks_and_needs_an_encoder() { +fn a_video_needs_an_encoder_and_rounds_its_size_to_whole_blocks() { let Some(gpu) = gpu() else { return }; assert!(Shot::new([SIZE, SIZE]).video(&gpu, 30.0, Mp4::new(Vec::new())).is_err(), "a host without an encoder"); gpu.set_host(Keeping(Pictures::default())); - assert!(Shot::new([SIZE, SIZE]).video(&gpu, 0.0, Mp4::new(Vec::new())).is_err(), "a frame rate"); - assert!( - Shot::new([SIZE + 2, SIZE]).video(&gpu, 30.0, Mp4::new(Vec::new())).is_err(), - "a width of whole 4-pixel blocks" - ); + assert!(Shot::new([SIZE, SIZE]).video(&gpu, 0.0, Mp4::new(Vec::new())).is_err(), "a frame rate above zero"); + // The encoder checks it is given whole 4×2 blocks: SIZE by SIZE. + assert!(Shot::new([SIZE + 3, SIZE + 1]).video(&gpu, 30.0, Mp4::new(Vec::new())).is_ok()); + let error = Shot::new([3, SIZE]).video(&gpu, 30.0, Mp4::new(Vec::new())).map(drop).unwrap_err(); + assert_eq!(error.to_string(), "size: must be at least 4×2 pixels"); let mut view = view(&gpu); let mut two = Shot::new([SIZE, SIZE]).panels([[0.0, 0.0, 0.5, 1.0], [0.5, 0.0, 0.5, 1.0]]); let mut video = two.video(&gpu, 30.0, Mp4::new(Vec::new())).expect("made"); @@ -163,7 +161,7 @@ fn a_video_is_whole_blocks_and_needs_an_encoder() { assert!(clear.check(latkit::Pixels::Rgba).is_ok() && clear.check(latkit::Pixels::Nv12).is_err()); } -/// A sink keeping what it is given: each frame's index and whether it was drawn, and the RGBA bytes. +/// A sink that records each frame's index and RGBA bytes (`None` if held), and the end index. #[derive(Default)] struct Frames { taken: Vec<(u64, Option>)>, @@ -188,13 +186,13 @@ impl Sink for Frames { fn any_sink_takes_every_frame_in_order_and_held_ones_without_pixels() { let Some(gpu) = gpu() else { return }; let mut view = view(&gpu); - // RGBA takes any width: no encoder, no 4-pixel blocks. + // RGBA allows any width: there is no encoder, so no 4-pixel blocks. let mut video = Shot::new([SIZE + 2, SIZE]).video(&gpu, 30.0, Frames::default()).expect("a video"); pollster::block_on(async { for _ in 0..4 { video.frame(&mut [&mut view]).await.expect("a frame"); } - view.set_data(dot(30.0)); + view.set_sources([dot(30.0)]).expect("a source"); video.frame(&mut [&mut view]).await.expect("a frame"); }); let frames = pollster::block_on(video.finish()).expect("finished"); diff --git a/docs/_static/custom.css b/docs/_static/custom.css index ae9e5db3..b5933c2c 100644 --- a/docs/_static/custom.css +++ b/docs/_static/custom.css @@ -19,6 +19,7 @@ body[data-theme] { --color-table-header-background: transparent; --color-table-border: var(--color-background-border); --color-api-background-hover: var(--color-background-secondary); + --color-highlight-on-target: var(--color-highlighted-background); } ::selection { @@ -148,6 +149,112 @@ body[data-theme] .highlight { padding: 0.75rem 1rem; } +/* Reference: each entry's declaration, then its members, each a signature over its definition. + Types in signatures are links; a member's anchor shows on hover, as a heading's does. A long + signature wraps with its later lines indented. */ + +:is(h1, h2, h3, h4) code.literal { + padding: 0; + border: 0; + background: none; + font-size: 0.95em; +} + +h3:has(> code.literal) { + margin-top: 2.5rem; +} + +p.signature, +dl.member > dt { + font-family: var(--font-stack--monospace); + font-size: var(--font-size--small); + line-height: 1.6; + text-indent: -2rem; +} + +p.signature { + margin: -0.35rem 0 0; + padding-left: 2rem; + color: var(--color-foreground-secondary); +} + +p.signature + p.signature { + margin-top: 0; +} + +p.signature + :not(p.signature) { + margin-top: 0.85rem; +} + +:is(p.signature, dl.member > dt) code.literal { + padding: 0; + border: 0; + background: none; + font-size: inherit; +} + +:is(p.signature, dl.member > dt) a.reference { + text-decoration: none; +} + +:is(p.signature, dl.member > dt) a.reference:hover { + text-decoration: underline; +} + +dl.member { + margin: 0 0 1.25rem; +} + +dl.member > dt { + margin-inline: -0.5rem; + padding: 0.15rem 0.5rem 0.15rem 2.5rem; + border-radius: 0.25rem; + color: var(--color-foreground-secondary); + transition: background 0.1s ease-out; +} + +dl.member > dt:hover { + background: var(--color-api-background-hover); +} + +dl.member:target > dt { + background: var(--color-highlight-on-target); +} + +dl.member > dt strong { + color: var(--color-api-name); + font-weight: 600; +} + +dl.member > dd { + margin: 0.2rem 0 0 1.5rem; +} + +dl.member > dd > :first-child { + margin-top: 0; +} + +dl.member > dd > :last-child { + margin-bottom: 0; +} + +dl.member dl.member { + margin: 0.6rem 0 0; +} + +dl.member dl.member > dd { + margin-top: 0; +} + +article p.rubric { + margin: 2rem 0 0.75rem; + font-size: var(--font-size--small--2); + font-weight: 600; + letter-spacing: 0.06em; + text-transform: uppercase; + color: var(--color-foreground-muted); +} + /* Tables */ table.docutils { diff --git a/docs/_static/performance.csv b/docs/_static/performance.csv index 517421c8..ae18321c 100644 --- a/docs/_static/performance.csv +++ b/docs/_static/performance.csv @@ -1,15 +1,15 @@ Scenario,Largest,Open (ms),Frame (ms),GPU (ms),Pick (ms) -"A first visit: latkit, the GPU, and pipelines made without a shader cache","10,000 rows",353,,, -"Vertices on a plane, colored through a colormap","2,000,000 rows","1,001",0.2,0.5,14.5 -"A lattice grid on the globe: buses, and the geodesic lines between them","2,000,000 buses","1,460",0.2,1.6,16.1 -"Every vertex labeled: placement, collisions, and the texts set","1,000,000 rows",781,1.7,1.6, -Edges by the thousand between the same sixteen pairs of vertices,"100,000 edges",816,0.2,19.8,70.8 -"Polygons of 33 points on the mercator map, triangulated in jobs","30,000 polygons",780,0.1,1.2,6.2 -"Vector tiles fetched as the camera asks for them, from a local server","4,096 squares a tile",512,2.8,0.2, -"Blocks with ports in trees of 64, laid out in layers, wires routed around them","10,000 blocks",581,0.1,0.2, -"A plot of traces 1,024 frames long, fed a frame every animation frame","10,000 traces",548,0.3,23.6,19.6 -One trace with a long history: levels of detail,"1,000,000 frames",441,0.2,0.4, -"Every row moving to a new place at once, easing over a second","1,000,000 rows",817,0.1,0.2, -"Views of one GPU drawing one dataset: pages shared, not copied",8 views,679,0.5,2.8, -"A 16 MiB budget: what draws fits, through tiles, and nothing fails","2,000,000 buses","1,516",0.2,0.6, -"Pictures and video of two views of a 100,000-bus grid","8,294,400 pixels",543,,, +"A first visit: latkit, the GPU, and pipelines made without a shader cache","10,000 rows",604,,, +"Vertices on a plane, colored through a colormap","2,000,000 rows",839,0.2,0.5,13.0 +"A lattice grid on the globe: buses, and the geodesic lines between them","2,000,000 buses","1,598",0.3,1.3,11.8 +"Every vertex labeled: placement, collisions, and text layout","1,000,000 rows",688,1.5,0.9, +Edges by the thousand between the same sixteen pairs of vertices,"100,000 edges",530,0.1,0.2,6.2 +"Polygons of 33 points on the mercator map, triangulated in jobs","30,000 polygons",668,0.1,1.1,6.5 +Vector tiles fetched from a local server as the camera needs them,"4,096 squares a tile",504,0.1,0.1, +"Blocks with ports in trees of 64, laid out in layers, wires routed around them","10,000 blocks",600,0.1,0.2, +"A plot of traces 1,024 frames long, fed a frame every animation frame","10,000 traces",590,0.3,23.3,20.9 +One trace with a long history: levels of detail,"1,000,000 frames",599,0.1,0.4, +"Every row moving to a new place at once, easing over a second","1,000,000 rows",688,0.1,0.2, +"Views of one GPU drawing one dataset: pages shared, not copied",8 views,549,0.3,2.8, +"A 16 MiB budget: paged tables fit, and nothing fails","2,000,000 buses","1,493",0.1,0.2, +"Pictures and video of two views of a 100,000-bus grid","8,294,400 pixels",579,,, diff --git a/docs/_static/theme.json b/docs/_static/theme.json index aba979fb..ec3cfa12 100644 --- a/docs/_static/theme.json +++ b/docs/_static/theme.json @@ -1,106 +1,130 @@ { "light": { "background": "rgba(247, 248, 250, 1)", - "surface": "rgba(232, 236, 240, 1)", "grid": "rgba(0, 0, 0, 0.08)", - "vertex": "rgba(31, 111, 235, 1)", - "vertexRadius": 4.0, - "edge": null, - "edgeWidth": 1.5, - "line": "rgba(31, 111, 235, 1)", - "lineWidth": 1.5, - "area": "rgba(197, 210, 223, 1)", - "shading": 0.6000000238418579, - "text": "rgba(74, 83, 97, 1)", - "font": "system-ui, -apple-system, \"Segoe UI\", Roboto, sans-serif", - "fontSize": 11.0, - "playhead": "rgba(227, 98, 9, 0.8)", "missing": "rgba(154, 163, 173, 1)", - "hover": "rgba(0, 0, 0, 0.35)", - "selected": "rgba(227, 98, 9, 0.9)", - "glow": 6.0, - "dim": 0.3499999940395355, - "hoverScale": 1.25, "dash": 8.0, + "shading": 0.6000000238418579, + "text": { + "color": "rgba(74, 83, 97, 1)", + "font": "system-ui, -apple-system, \"Segoe UI\", Roboto, sans-serif", + "size": 11.0, + "halo": 2.0 + }, + "vertex": { + "color": "rgba(31, 111, 235, 1)", + "radius": 4.0 + }, + "edge": { + "color": null, + "width": 1.5, + "spacing": 4.0, + "bow": 0.11999999731779099, + "arrow": 7.0 + }, + "line": { + "color": "rgba(31, 111, 235, 1)", + "width": 1.5 + }, + "area": { + "color": "rgba(197, 210, 223, 1)" + }, + "focus": { + "hover": "rgba(0, 0, 0, 0.35)", + "selected": "rgba(227, 98, 9, 0.9)", + "glow": 6.0, + "dim": 0.3499999940395355, + "grow": 1.25 + }, "flow": { "color": "rgba(255, 255, 255, 0.9)", "spacing": 64.0, "length": 24.0, - "width": 4.0, - "units": false - }, - "edgeSpacing": 4.0, - "laneBow": 0.11999999731779099, - "arrow": 7.0, - "halo": 2.0, - "groupPadding": 16.0, - "outline": "rgba(0, 0, 0, 0.14)", - "corner": 6.0, - "portSize": 7.0, - "bend": 6.0, - "shadow": { - "color": "rgba(0, 0, 0, 0.22)", - "light": [ - -0.3499999940395355, - 0.44999998807907104, - 1.0 - ], - "softness": 0.15000000596046448, - "lift": 3.0 - }, - "atmosphere": "rgba(108, 180, 255, 0.45)", - "night": 0.550000011920929 + "width": 4.0 + }, + "diagram": { + "group": "rgba(232, 236, 240, 1)", + "outline": "rgba(0, 0, 0, 0.14)", + "corner": 6.0, + "portSize": 7.0, + "bend": 6.0, + "groupPadding": 16.0, + "shadow": { + "color": "rgba(0, 0, 0, 0.22)", + "lift": 3.0 + } + }, + "earth": { + "ground": "rgba(232, 236, 240, 1)", + "atmosphere": "rgba(108, 180, 255, 0.45)", + "night": 0.550000011920929 + }, + "plot": { + "playhead": "rgba(227, 98, 9, 0.8)" + } }, "dark": { "background": "rgba(15, 20, 26, 1)", - "surface": "rgba(24, 33, 43, 1)", "grid": "rgba(255, 255, 255, 0.07)", - "vertex": "rgba(124, 196, 255, 1)", - "vertexRadius": 4.0, - "edge": null, - "edgeWidth": 1.5, - "line": "rgba(124, 196, 255, 1)", - "lineWidth": 1.5, - "area": "rgba(43, 61, 80, 1)", - "shading": 0.6000000238418579, - "text": "rgba(168, 178, 194, 1)", - "font": "system-ui, -apple-system, \"Segoe UI\", Roboto, sans-serif", - "fontSize": 11.0, - "playhead": "rgba(255, 229, 212, 0.8)", "missing": "rgba(107, 116, 128, 1)", - "hover": "rgba(255, 255, 255, 0.6)", - "selected": "rgba(255, 176, 0, 0.9)", - "glow": 6.0, - "dim": 0.3499999940395355, - "hoverScale": 1.25, "dash": 8.0, + "shading": 0.6000000238418579, + "text": { + "color": "rgba(168, 178, 194, 1)", + "font": "system-ui, -apple-system, \"Segoe UI\", Roboto, sans-serif", + "size": 11.0, + "halo": 2.0 + }, + "vertex": { + "color": "rgba(124, 196, 255, 1)", + "radius": 4.0 + }, + "edge": { + "color": null, + "width": 1.5, + "spacing": 4.0, + "bow": 0.11999999731779099, + "arrow": 7.0 + }, + "line": { + "color": "rgba(124, 196, 255, 1)", + "width": 1.5 + }, + "area": { + "color": "rgba(43, 61, 80, 1)" + }, + "focus": { + "hover": "rgba(255, 255, 255, 0.6)", + "selected": "rgba(255, 176, 0, 0.9)", + "glow": 6.0, + "dim": 0.3499999940395355, + "grow": 1.25 + }, "flow": { "color": "rgba(255, 255, 255, 0.85)", "spacing": 64.0, "length": 24.0, - "width": 4.0, - "units": false - }, - "edgeSpacing": 4.0, - "laneBow": 0.11999999731779099, - "arrow": 7.0, - "halo": 2.0, - "groupPadding": 16.0, - "outline": "rgba(255, 255, 255, 0.12)", - "corner": 6.0, - "portSize": 7.0, - "bend": 6.0, - "shadow": { - "color": "rgba(0, 0, 0, 0.45)", - "light": [ - -0.3499999940395355, - 0.44999998807907104, - 1.0 - ], - "softness": 0.15000000596046448, - "lift": 3.0 - }, - "atmosphere": "rgba(90, 169, 255, 0.55)", - "night": 0.30000001192092896 + "width": 4.0 + }, + "diagram": { + "group": "rgba(24, 33, 43, 1)", + "outline": "rgba(255, 255, 255, 0.12)", + "corner": 6.0, + "portSize": 7.0, + "bend": 6.0, + "groupPadding": 16.0, + "shadow": { + "color": "rgba(0, 0, 0, 0.45)", + "lift": 3.0 + } + }, + "earth": { + "ground": "rgba(24, 33, 43, 1)", + "atmosphere": "rgba(90, 169, 255, 0.55)", + "night": 0.30000001192092896 + }, + "plot": { + "playhead": "rgba(255, 229, 212, 0.8)" + } } } diff --git a/docs/cli.md b/docs/cli.md index 1700b9ec..b155b443 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,48 +1,35 @@ # Command line -The `latkit` executable draws scenes of datasets into images, image sequences, raw frames, and -video on the machine's GPU, with the browser's renderer. Its inputs are latkit's JSON, so any -language can write them. +The `latkit` executable draws [scenes](guide.md#scenes) of datasets to images, image sequences, raw +frames, and video on the machine's GPU, with the same renderer as the browser. Its inputs are +latkit's JSON, so any language can write them. A dataset can be a [JSON document](guide.md#json), +latkit's [bytes](guide.md#bytes), or GeoJSON named with `NAME=`. The +[command reference](reference/cli.md) documents every command and option. ```sh cargo install latkit-cli # or, from the repository: cargo build --locked -p latkit-cli --profile cli -latkit validate scene.json -d grid.json +latkit validate scene.json -d grid.json # check every input without a GPU latkit render scene.json -d grid.json -o grid.png --size 2400x1350 --scale 2 +latkit render scene.json -d grid.json --camera '{"center":[-97,31],"zoom":5,"pitch":0,"bearing":0}' -o tx.png latkit render map.json plot.json -d grid.json -d states=usa.geojson \ --panel 0,0,1,0.8 --panel 0,0.8,1,0.2 --frames frames.jsonl --fps 60 -o film.mp4 latkit render scene.json -d grid.json --frames - --format y4m -o - | ffmpeg -i - film.webm -latkit inspect grid.bin -latkit query grid.bin '{"of":"Branch","filter":{"op":"gt","args":["loading",90]},"limit":20}' -latkit tile history.bin -o history/ +latkit render scene.json -d grid.json --frames frames.jsonl -o frames/%04d.png +latkit render scene.json -d run.latk -o run.mp4 # plays the recording's span +latkit render scene.json -d run.latk --frames frames.jsonl --span 0.6,10 --rate 0.4 -o run.mp4 +latkit inspect grid.bin # tables, fields, and the axis, as JSON +latkit query grid.bin '{"of":"Branch","filter":{"gt":["$loading",90]},"limit":20}' +latkit tile history.bin -o history/ # time tiles, which --tiles opens latkit render plot.json --tiles grid=history/frames.json -o year.png latkit schema scene > scene.schema.json ``` -| Command | Does | -| ---------- | --------------------------------------------------------------------------------------------------------- | -| `render` | draws the scenes, one per `--panel`, and writes `--output` | -| `validate` | reads and checks everything `render` would, frames included, without a GPU | -| `inspect` | prints datasets as JSON: tables, rows, fields and their kinds, the axis and its span | -| `query` | prints the rows a query keeps, sorted and paged, as JSON Lines | -| `tile` | writes a dataset as [time tiles](guide.md#fetch-long-histories-in-spans) in `--output` | -| `schema` | prints the JSON Schema of every input, or of `scene`, `data`, `patch`, `query`, `camera`, `fit`, `frames` | - -## Inputs - -- **Scenes** are the [scene](guide.md#scenes) JSON, with a `title` and `subtitle`, and on any mark a - `legend`. `--camera FILE` gives scenes cameras, in order; a view without one frames `--fit` - (`all`, or a fit as JSON). `--insets T,R,B,L` keeps room clear. -- **Datasets** (`-d [NAME=]FILE`) are a [JSON document](guide.md#json), latkit's - [bytes](guide.md#bytes), or GeoJSON, which takes a `NAME`. A `NAME` renames any dataset. - Slices of binary files load relative to the document. -- **Tiles** (`--tiles NAME=FILE`) are time tiles' `frames.json`, as `tile` writes it, or a PMTiles - file, TileJSON, or a `{z}/{x}/{y}` template on disk. -- **Fonts** (`--font FILE`, repeated) replace system fonts; the first is the fallback. -- One input at most reads stdin (`-`). +At most one input can read stdin (`-`). ## Frames -`--frames` is JSON Lines, a line an output frame at `n / --fps` seconds. Every key is optional: +`--frames` is a JSON Lines file with one line per output frame, or with [`--span`](#spans), lines +at axis coordinates. Line `n` shows at `n / --fps` seconds. Every key is optional: ```json { @@ -65,36 +52,55 @@ latkit schema scene > scene.schema.json } ``` -| Key | Changes | -| -------- | ------------------------------------------------------------------------- | -| `at` | the axis coordinate every view shows | -| `patch` | datasets by name, in order, each a patch in latkit's JSON | -| `camera` | where views look at once; `fly` eases there over twice the scene's `ease` | -| `fit` | what views frame | -| `insets` | CSS pixels each view keeps clear | -| `orbit` | degrees a second a map or plane turns | +| Key | Changes | +| -------- | ------------------------------------------------------------------------------ | +| `at` | the axis coordinate every view shows | +| `patch` | patches by dataset name, applied in order, in latkit's JSON | +| `camera` | where views look, set at once; `fly` eases there over twice the scene's `ease` | +| `fit` | what views frame | +| `insets` | CSS pixels each view keeps clear | +| `orbit` | degrees per second a map or plane turns | -`camera`, `fly`, and `fit` take one value for every view, or a list with one per view (`null` -leaves one). Frames load a couple ahead of the one drawing. Scenes ease as they say; `"ease": 0` -shows each frame's data as it is. +`camera`, `fly`, and `fit` take one value for every view, or a list with one value per view (`null` +leaves that view unchanged). A camera or flight replaces a pending fit. The CLI loads up to two +frames ahead of the one it draws. Changes ease over each scene's `ease`; `"ease": 0` shows each +frame's data as is. + +## Spans + +`--span FROM,TO` plays an axis interval at `--rate` units per second (default 1). A video or image +sequence without `--frames` plays the datasets' span. + +With `--span`, each line needs `at`, in order, and applies when playback reaches it. Between lines +that set a view's camera, the view flies from one to the next: + +```text +{"at": 0.6, "camera": {"center": [-92, 36.5], "zoom": 5.09, "pitch": 48, "bearing": -10}} +{"at": 1, "patch": {"grid": [{"set": {"table": "Branch", "ids": ["b1"], "fields": {"open": [true]}}}]}} +{"at": 10, "camera": {"center": [-92, 36.5], "zoom": 5.09, "pitch": 48, "bearing": 20}} +``` ## Outputs -| Output | Writes | -| ------------------------ | ---------------------------------------------------------------- | -| `.png`, `.jpg`, `.webp` | one picture; JPEG at `--quality` | -| `frame-%04d.png` | an image a frame (`--frames`) | -| `.rgba`, `--format rgba` | straight-alpha RGBA, frames back to back | -| `.y4m` | YUV4MPEG2, 4:2:0 BT.709, for any encoder (`--frames`) | -| `.mp4` | H.264 in fragmented MP4 from the platform's encoder (`--frames`) | +| Output | Writes | +| ------------------------ | --------------------------------------------------- | +| `.png`, `.jpg`, `.webp` | one picture; JPEG at `--quality` | +| `frame-%04d.png` | one image per frame | +| `.rgba`, `--format rgba` | straight-alpha RGBA, frames back to back | +| `.y4m` | YUV4MPEG2, 4:2:0 BT.709, for any encoder | +| `.mp4` | H.264 in fragmented MP4 from the platform's encoder | -A still waits until every view is complete and settled; `--elapsed` fixes its effect time. A frame -where nothing moved is held. A file is replaced only once it is whole. MP4 needs Windows (Media -Foundation); elsewhere, pipe Y4M to an encoder. `--deadline` bounds each picture or frame. +A still picture waits until every view is complete and motion has settled. `--elapsed` sets the +time its animated effects show. A video frame where nothing moved repeats the frame before. An +output file is replaced only once the new file is completely written. MP4 needs Windows (Media +Foundation). On other systems, pipe Y4M to an encoder. `--deadline` limits how long each picture or +video frame may take, and defaults to 30 seconds. ## Diagnostics -Exit codes: 0 success, 1 failure, 2 usage. Errors print `latkit: SOURCE: message` with the part's -path, such as `frames[3].patch.grid[0].Bus.x`. `--diagnostics json` writes one object a line to -stderr, `{ type, detail }`: `progress` after each frame, `stats` when done, and `error` with its -`code`, `message`, `path`, and `source`. +Exit codes are 0 for success, 1 for failure, and 2 for a usage error. Errors print as +`latkit: SOURCE: message`, where `SOURCE` is the input file at fault. For invalid input, the +message starts with the path to the invalid part, such as `frames[3].patch.grid[0].Bus.x`. When an +output cannot be written, the message names it. `--diagnostics json` writes one JSON object per +line to stderr, `{ type, detail }`. The `type` is `progress` after each frame, `stats` when done, +or `error` with its `code`, `message`, `path`, and `source`. diff --git a/docs/conf.py b/docs/conf.py index 18dd9dde..1c3d047a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -11,7 +11,7 @@ extensions = ["myst_parser", "sphinx_copybutton"] exclude_patterns = ["_build", ".venv", "gallery"] -myst_enable_extensions = ["colon_fence", "deflist"] +myst_enable_extensions = ["attrs_block", "attrs_inline", "colon_fence", "deflist"] myst_heading_anchors = 3 copybutton_prompt_text = r"^\$ " @@ -30,25 +30,27 @@ def palette(name): - """Furo colors: the brand is the theme's mark color, pages its background and surface.""" + """Furo colors: the brand is the theme's vertex color, pages its background, and panels the + fill a diagram's groups take.""" theme, dark = THEMES[name], name == "dark" n = {role: pair[dark] for role, pair in NEUTRAL.items()} + brand, panel = theme["vertex"]["color"], theme["diagram"]["group"] return { - "color-brand-primary": theme["vertex"], - "color-brand-content": theme["vertex"], - "color-brand-visited": theme["vertex"], + "color-brand-primary": brand, + "color-brand-content": brand, + "color-brand-visited": brand, "color-foreground-primary": n["ink"], "color-foreground-secondary": n["secondary"], "color-foreground-muted": n["muted"], "color-foreground-border": n["border"], "color-background-primary": theme["background"], - "color-background-secondary": theme["surface"], + "color-background-secondary": panel, "color-background-border": n["rule"], "color-api-name": n["ink"], "color-api-pre-name": n["secondary"], - "color-code-background": theme["surface"], - "color-inline-code-background": theme["surface"], - "color-selected": theme["selected"], + "color-code-background": panel, + "color-inline-code-background": panel, + "color-selected": theme["focus"]["selected"], } diff --git a/docs/examples/bloom.md b/docs/examples/bloom.md index e67706a3..9c61122c 100644 --- a/docs/examples/bloom.md +++ b/docs/examples/bloom.md @@ -5,10 +5,10 @@ :class: picture ``` -Thirty-two thousand seeds a golden angle apart, sized and colored by a wave winding through them, through a colormap of their own. +Thirty-two thousand seeds placed a golden angle apart, sized and colored by a spiral wave through a custom colormap. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts bloom` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts bloom` to draw it again. ```{literalinclude} ../gallery/bloom.ts :language: typescript diff --git a/docs/examples/clifford.md b/docs/examples/clifford.md index ec3b5dcd..709a2309 100644 --- a/docs/examples/clifford.md +++ b/docs/examples/clifford.md @@ -1,14 +1,14 @@ # Clifford ```{image} ../images/clifford.png -:alt: A Clifford strange attractor of 260,000 translucent points, colored by the way each step turns +:alt: A Clifford strange attractor of 260,000 translucent points, colored by the direction of each step :class: picture ``` -A strange attractor of 260,000 translucent points, colored through a cyclic colormap by where each step heads, so density and flow both show. +A strange attractor of 260,000 translucent points, colored by the direction of each step through a cyclic colormap, so both density and flow show. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts clifford` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts clifford` to draw it again. ```{literalinclude} ../gallery/clifford.ts :language: typescript diff --git a/docs/examples/currents.md b/docs/examples/currents.md index a699d81b..6917fc9c 100644 --- a/docs/examples/currents.md +++ b/docs/examples/currents.md @@ -5,10 +5,10 @@ :class: picture ``` -Streamlines traced through a field of seven vortices, as a wind map draws them: each a path of geometry, colored by its speed. +Streamlines traced through a field of seven vortices, as on a wind map. Each streamline is a path geometry, colored by its speed. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts currents` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts currents` to draw it again. ```{literalinclude} ../gallery/currents.ts :language: typescript diff --git a/docs/examples/diagram.md b/docs/examples/diagram.md index 4b201941..81bfdf34 100644 --- a/docs/examples/diagram.md +++ b/docs/examples/diagram.md @@ -1,14 +1,14 @@ # Diagram ```{image} ../images/diagram.png -:alt: A microgrid controller as blocks in nested groups, wired port to port with routed wires and a feedback loop +:alt: A microgrid controller as blocks in nested groups, with wires routed port to port and a feedback loop :class: picture ``` -Blocks in nested groups, placed in layers by latkit's own layout, wires routed at right angles between their ports, a feedback loop, and a status ring. +Blocks in nested groups, placed by latkit's layered layout. Wires run at right angles between their ports, with a feedback loop and a status ring. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts diagram` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts diagram` to draw it again. ```{literalinclude} ../gallery/diagram.ts :language: typescript diff --git a/docs/examples/frequency.md b/docs/examples/frequency.md index b9bf3240..c0b0b6cf 100644 --- a/docs/examples/frequency.md +++ b/docs/examples/frequency.md @@ -1,14 +1,14 @@ # ACTIVSg10k ```{image} ../images/frequency.png -:alt: The speed of 1,456 machines over ten seconds after a fault, near ones bright and swinging hardest, far ones dark +:alt: The speed of 1,456 machines over ten seconds after a fault. Machines near the fault are bright and swing hardest, and far ones are dark :class: picture ``` -A simulated fault on the ACTIVSg10k case: every machine's speed traced over ten seconds, read straight from the recording's sampled field, and colored by distance from the fault. +A simulated fault on the ACTIVSg10k case: every machine's speed over ten seconds, traced directly from the recording's sampled field and colored by distance from the fault. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts frequency` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts frequency` to draw it again. ```{literalinclude} ../gallery/frequency.ts :language: typescript diff --git a/docs/examples/globe.md b/docs/examples/globe.md index bb1a350f..cef2e776 100644 --- a/docs/examples/globe.md +++ b/docs/examples/globe.md @@ -1,14 +1,14 @@ # Globe ```{image} ../images/globe.png -:alt: Routes between forty of the world's great cities, along great circles on a globe, comets flowing on them +:alt: Great-circle routes between forty of the world's great cities on a globe, with comets flowing along them :class: picture ``` -Routes between forty cities along great circles, comets flowing on them, over coastlines and borders read from GeoJSON, on a globe with its atmosphere. +Great-circle routes between forty cities, with comets flowing along them. Coastlines and borders come from GeoJSON, and the globe has an atmosphere. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts globe` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts globe` to draw it again. ```{literalinclude} ../gallery/globe.ts :language: typescript diff --git a/docs/examples/grid.md b/docs/examples/grid.md index ea862f7f..413419d7 100644 --- a/docs/examples/grid.md +++ b/docs/examples/grid.md @@ -5,10 +5,10 @@ :class: picture ``` -The example suite's ACTIVSg70k case, 70,000 buses and 88,207 branches, over the states on a Mercator map tilted toward the Great Lakes. Each line takes its color and width from its voltage level, and the backbone draws on top by priority. +The ACTIVSg70k case from the example suite: 70,000 buses and 88,207 branches over the US states, on a Mercator map tilted toward the Great Lakes. Each line's color and width follow its voltage level, and `priority` draws the high-voltage backbone on top. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts grid` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts grid` to draw it again. ```{literalinclude} ../gallery/grid.ts :language: typescript diff --git a/docs/examples/helios.md b/docs/examples/helios.md index 40f75d8b..06702d21 100644 --- a/docs/examples/helios.md +++ b/docs/examples/helios.md @@ -5,10 +5,10 @@ :class: picture ``` -A network in five rings: vertices colored through a colormap, labeled hubs, and comets flowing along every branch. +A network in five rings, with vertices colored through a colormap, labeled hubs, and comets flowing along every branch. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts helios` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts helios` to draw it again. ```{literalinclude} ../gallery/helios.ts :language: typescript diff --git a/docs/examples/index.md b/docs/examples/index.md index 3866f8c9..b406155f 100644 --- a/docs/examples/index.md +++ b/docs/examples/index.md @@ -1,22 +1,20 @@ # Examples -Every picture here is drawn by latkit from the module on its page: data written with `Patch`, a -typed `Scene`, and the `latkit` command line. The -[example suite](https://github.com/lukelowry/latkit/tree/main/npm/examples) runs the same features +The `latkit` command line draws every picture here. Each page shows the module that builds the +picture's datasets with `Data` and returns its scene. The +[example suite](https://github.com/lukelowry/latkit/tree/main/npm/examples) shows the same features live in the browser. :::{container} cards -[![The ACTIVSg70k grid on the globe as one 765 kV line opens: buses raised by their frequency, branches that pick up its power glow amber, those it leaves glow cyan](../images/opening.png) Line opening](opening.md) - [![Seventy thousand buses of a synthetic grid on the United States, its lines colored and widened by voltage](../images/grid.png) ACTIVSg70k](grid.md) [![Two thousand buses of a synthetic Texas grid: generators as amber discs sized by output, comets along its highest-voltage lines](../images/texas.png) ACTIVSg2000](texas.md) -[![The speed of 1,456 machines over ten seconds after a fault, near ones bright and swinging hardest, far ones dark](../images/frequency.png) ACTIVSg10k](frequency.md) +[![The speed of 1,456 machines over ten seconds after a fault. Machines near the fault are bright and swing hardest, and far ones are dark](../images/frequency.png) ACTIVSg10k](frequency.md) -[![Routes between forty of the world's great cities, along great circles on a globe, comets flowing on them](../images/globe.png) Globe](globe.md) +[![Great-circle routes between forty of the world's great cities on a globe, with comets flowing along them](../images/globe.png) Globe](globe.md) -[![A microgrid controller as blocks in nested groups, wired port to port with routed wires and a feedback loop](../images/diagram.png) Diagram](diagram.md) +[![A microgrid controller as blocks in nested groups, with wires routed port to port and a feedback loop](../images/diagram.png) Diagram](diagram.md) [![A synthetic network of 480 buses in five concentric rings, comets flowing along its 896 branches](../images/helios.png) Helios](helios.md) @@ -24,7 +22,7 @@ live in the browser. [![Ridged mountains of 112,000 lit triangles, from deep teal valleys to pale summits, seen from above at an angle](../images/terrain.png) Terrain](terrain.md) -[![A Clifford strange attractor of 260,000 translucent points, colored by the way each step turns](../images/clifford.png) Clifford](clifford.md) +[![A Clifford strange attractor of 260,000 translucent points, colored by the direction of each step](../images/clifford.png) Clifford](clifford.md) [![Thirty-two thousand seeds a golden angle apart, in arms of sapphire, rose, and amber](../images/bloom.png) Golden bloom](bloom.md) @@ -36,7 +34,6 @@ live in the browser. ```{toctree} :hidden: -opening grid texas frequency diff --git a/docs/examples/opening.md b/docs/examples/opening.md deleted file mode 100644 index f7ad413c..00000000 --- a/docs/examples/opening.md +++ /dev/null @@ -1,32 +0,0 @@ -# Line opening - -```{image} ../images/opening.png -:alt: The ACTIVSg70k grid on the globe as one 765 kV line opens: buses raised by their frequency, branches that pick up its power glow amber, those it leaves glow cyan -:class: picture banner -``` - -The ACTIVSg70k case on the globe as its Little Rock to Newark 765 kV line opens, in red. A DC -power flow of all 70,000 buses finds the angles before the opening and how far each one shifts -after it, and each of the 88,207 branches is colored by the flow it picks up, in amber, or sheds, in -cyan; changes under 10 MW stay grey, the distribution web under 115 kV dim beneath the backbone. -Each bus stands as high as its frequency 2.5 s after the opening, recorded from a simulation of the -same event, and poles drop from the high-voltage buses to the ground. - -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts opening` draws it again. - -```{literalinclude} ../gallery/opening.ts -:language: typescript -``` - -The power flow solves the grid's susceptance matrix by conjugate gradients, in a few seconds: - -```{literalinclude} ../gallery/flow.ts -:language: typescript -``` - -The recording is too large for the repository; `record.ts` keeps the one instant drawn: - -```{literalinclude} ../gallery/record.ts -:language: typescript -``` diff --git a/docs/examples/terrain.md b/docs/examples/terrain.md index 2006db37..12e56380 100644 --- a/docs/examples/terrain.md +++ b/docs/examples/terrain.md @@ -5,10 +5,10 @@ :class: picture ``` -Ridged mountains as faces between a grid of vertices: each face blends its corners' colors, lit by the normals the GPU works out, in three dimensions. +Ridged mountains in three dimensions, as faces over a grid of vertices. Each face blends its corners' colors and is lit by normals the GPU computes. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts terrain` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts terrain` to draw it again. ```{literalinclude} ../gallery/terrain.ts :language: typescript diff --git a/docs/examples/texas.md b/docs/examples/texas.md index 0459fe85..8195f785 100644 --- a/docs/examples/texas.md +++ b/docs/examples/texas.md @@ -5,10 +5,10 @@ :class: picture ``` -A synthetic Texas grid: generators sized by the square root of their output and labeled by priority, the most a frame can fit, with comets flowing along its backbone in map units, so they keep their place as the view zooms. +A synthetic Texas grid. Generators are sized by the square root of their output, and the ten largest are labeled. Comets flow along the lines of 345 kV and up, spaced in map units so they keep their place as the view zooms. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts texas` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts texas` to draw it again. ```{literalinclude} ../gallery/texas.ts :language: typescript diff --git a/docs/examples/tides.md b/docs/examples/tides.md index ea57ba8e..003844cc 100644 --- a/docs/examples/tides.md +++ b/docs/examples/tides.md @@ -5,10 +5,10 @@ :class: picture ``` -Closed contours swelling outward, each deformed by harmonics that drift with its level, colored from the center out. +Closed contours swelling outward, colored from the center out. Harmonics ripple each contour, and their phases shift from level to level. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts tides` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts tides` to draw it again. ```{literalinclude} ../gallery/tides.ts :language: typescript diff --git a/docs/examples/torus.md b/docs/examples/torus.md index 393ab1e3..b3a4524b 100644 --- a/docs/examples/torus.md +++ b/docs/examples/torus.md @@ -7,8 +7,8 @@ Sixty-four filaments around a (2, 3) torus knot, drawn as edges between points with elevation and seen from above at an angle. -The picture is drawn by the `latkit` command line from this module, which writes its data with -`Patch` and its scene as a typed `Scene`; `node docs/gallery/render.ts torus` draws it again. +The `latkit` command line draws this picture from the module below, which builds its datasets +with `Data` and returns its scene. Run `node docs/gallery/render.ts torus` to draw it again. ```{literalinclude} ../gallery/torus.ts :language: typescript diff --git a/docs/gallery/bloom.ts b/docs/gallery/bloom.ts index 59fb5926..d2161f43 100644 --- a/docs/gallery/bloom.ts +++ b/docs/gallery/bloom.ts @@ -1,53 +1,47 @@ -/** Golden bloom: seeds a golden angle apart, colored by a wave turning through them. */ -import { Patch } from 'latkit/core'; -import { THEME, floats, framed, type Piece } from './look.ts'; +/** Golden bloom: seeds placed a golden angle apart, colored by an eight-armed spiral wave. */ +import { Data, field } from 'latkit'; +import { framed } from './look.ts'; +import type { Picture } from './render.ts'; -export default function bloom(): Piece { +export default function (): Picture { const seeds = 32_000; const golden = Math.PI * (3 - Math.sqrt(5)); const r = (i: number) => 2.2 * Math.sqrt(i); const angle = (i: number) => i * golden; - // Where a seed lies around the center, unwound from its golden angle. + // The seed's polar angle around the center, in [0, 2π). const polar = (i: number) => angle(i) % (2 * Math.PI); - const data = Patch.to('bloom').replace( - 'Seed', - { count: seeds }, - { - x: floats(seeds, (i) => r(i) * Math.cos(angle(i))), - y: floats(seeds, (i) => r(i) * Math.sin(angle(i))), - // Eight arms winding outward through the seeds, dark between them. - wave: floats(seeds, (i) => (0.5 + 0.5 * Math.sin(r(i) * 0.05 - 8 * polar(i))) ** 1.6), - size: floats(seeds, (i) => 0.5 + 0.85 * Math.sqrt(i / seeds)), - }, - ); + const each = { length: seeds }; + const bloom = new Data('bloom').replace('Seed', { + x: Float32Array.from(each, (_, i) => r(i) * Math.cos(angle(i))), + y: Float32Array.from(each, (_, i) => r(i) * Math.sin(angle(i))), + // Eight spiral arms winding outward, with dark gaps between them. + wave: Float32Array.from( + each, + (_, i) => (0.5 + 0.5 * Math.sin(r(i) * 0.05 - 8 * polar(i))) ** 1.6, + ), + size: Float32Array.from(each, (_, i) => 0.5 + 0.85 * Math.sqrt(i / seeds)), + }); const edge = r(seeds) + 4; return { - name: 'bloom', - alt: 'Thirty-two thousand seeds a golden angle apart, in arms of sapphire, rose, and amber', - data: [data], - fit: framed([-edge, -edge, edge, edge]), scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14' }, marks: [ { - vertices: { - of: ['bloom', 'Seed'], - position: ['x', 'y'], - radius: 'size', - pick: false, - color: { - colormap: { - kind: 'sequential', - colors: ['#101830', '#1f3a8a', '#5b48d6', '#c45496', '#f58b72', '#ffd890'], - }, - value: 'wave', - domain: [0, 1], - }, - }, + vertices: bloom.table('Seed'), + position: ['$x', '$y'], + radius: '$size', + pick: false, + color: field('wave') + .colormap({ + kind: 'sequential', + colors: ['#101830', '#1f3a8a', '#5b48d6', '#c45496', '#f58b72', '#ffd890'], + }) + .domain(0, 1), }, ], }, + fit: framed([-edge, -edge, edge, edge]), }; } diff --git a/docs/gallery/clifford.ts b/docs/gallery/clifford.ts index e7b5a00b..eace9aeb 100644 --- a/docs/gallery/clifford.ts +++ b/docs/gallery/clifford.ts @@ -1,9 +1,10 @@ -/** Clifford: a strange attractor of a quarter million points, translucent, so density shows. */ -import { Patch } from 'latkit/core'; -import { THEME, framed, type Piece } from './look.ts'; +/** Clifford: a strange attractor of 260,000 translucent points, so density shows. */ +import { Data, field } from 'latkit'; +import { framed } from './look.ts'; +import type { Picture } from './render.ts'; -export default function clifford(): Piece { - // As many points as a view draws whole, past which it draws a dense table through tiles. +export default function (): Picture { + // Just under 2^18 = 262,144 rows. Past that, a view pages a table rather than drawing all of it. const n = 260_000; const [a, b, c, d] = [-1.4, 1.6, 1.0, 0.7]; const x = new Float32Array(n); @@ -16,46 +17,39 @@ export default function clifford(): Piece { if (i >= 0) { x[i] = nx * 300; y[i] = ny * 300; - // Where each step heads, around the circle: the attractor's flow, as a cyclic color. + // The step's direction as a fraction of a turn, for a cyclic colormap. turn[i] = (Math.atan2(ny - py, nx - px) / (2 * Math.PI) + 1) % 1; } [px, py] = [nx, ny]; } - const data = Patch.to('attractor').replace('Point', { count: n }, { x, y, turn }); + const attractor = new Data('attractor').replace('Point', { x, y, turn }); return { - name: 'clifford', - alt: 'A Clifford strange attractor of 260,000 translucent points, colored by the way each step turns', - data: [data], - fit: framed([-430, -395, 600, 465]), scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14' }, marks: [ { - vertices: { - of: ['attractor', 'Point'], - position: ['x', 'y'], - radius: 0.65, - pick: false, - color: { - colormap: { - kind: 'cyclic', - colors: [ - '#5a8dff8c', - '#9b6bff8c', - '#ff5fa28c', - '#ffb35c8c', - '#5ce0c28c', - '#5a8dff8c', - ], - }, - value: 'turn', - domain: [0, 1], - }, - }, + vertices: attractor.table('Point'), + position: ['$x', '$y'], + radius: 0.65, + pick: false, + color: field('turn') + .colormap({ + kind: 'cyclic', + colors: [ + '#5a8dff8c', + '#9b6bff8c', + '#ff5fa28c', + '#ffb35c8c', + '#5ce0c28c', + '#5a8dff8c', + ], + }) + .domain(0, 1), }, ], }, + fit: framed([-430, -395, 600, 465]), }; } diff --git a/docs/gallery/currents.ts b/docs/gallery/currents.ts index 9aeba93a..f76c4641 100644 --- a/docs/gallery/currents.ts +++ b/docs/gallery/currents.ts @@ -1,8 +1,9 @@ -/** Currents: streamlines through a field of vortices, as a wind map draws them, colored by speed. */ -import { Patch } from 'latkit/core'; -import { THEME, framed, type Piece } from './look.ts'; +/** Currents: streamlines through a field of vortices, as on a wind map, colored by speed. */ +import { Data, field } from 'latkit'; +import { framed } from './look.ts'; +import type { Picture } from './render.ts'; -/** Vortices: where each turns, how strongly, and which way. */ +/** Vortices as `[x, y, strength]`. The sign of the strength sets the direction of spin. */ const VORTICES = [ [-380, 120, 1.0], [-120, -170, -0.8], @@ -13,7 +14,8 @@ const VORTICES = [ [560, 250, -0.5], ] as const; -/** The flow at `x`, `y`: a steady drift east, and each vortex's swirl falling off from its center. */ +/** The flow velocity at `x`, `y`: a steady eastward drift plus each vortex's swirl, which falls off + * with distance from its center. */ function flow(x: number, y: number): [number, number] { let [u, v] = [0.35, 0.05]; for (const [cx, cy, s] of VORTICES) { @@ -25,13 +27,14 @@ function flow(x: number, y: number): [number, number] { return [u, v]; } -export default function currents(): Piece { +export default function (): Picture { const [w, h] = [1300, 720]; const geometry: [number, number][][] = []; const speed: number[] = []; - // Seeds on a grid, jittered by a fixed hash, so the field is covered evenly every time. + // Streamlines start on a grid, jittered by a fixed hash, so they cover the field evenly and the + // same way on every run. const jitter = (i: number) => (Math.sin(i * 12.9898) * 43758.5453) % 1; - // Inside an ellipse, so the field ends softly rather than at a frame. + // Streamlines stay inside an ellipse, so the field has a soft edge rather than a rectangular one. const inside = (x: number, y: number) => (x / (w / 2)) ** 2 + (y / (h / 2)) ** 2 < 1; for (let i = 0; i < 52 * 30; i++) { let x = (((i % 52) + 0.5 + 0.8 * jitter(i)) / 52 - 0.5) * w; @@ -40,7 +43,7 @@ export default function currents(): Piece { const path: [number, number][] = [[x, y]]; let travelled = 0; for (let s = 0; s < 70; s++) { - // A midpoint step, so curves stay curves. + // A midpoint step, which follows curves more closely than an Euler step. const [u0, v0] = flow(x, y); const [u, v] = flow(x + 2.5 * u0, y + 2.5 * v0); const length = Math.hypot(u, v); @@ -55,36 +58,25 @@ export default function currents(): Piece { geometry.push(path); speed.push(travelled / path.length); } - const data = Patch.to('wind').replace( - 'Streamline', - { count: geometry.length }, - { geometry, speed }, - ); + const wind = new Data('wind').replace('Streamline', { geometry, speed }); return { - name: 'currents', - alt: 'Over two thousand streamlines swirling around seven vortices, slow ones deep blue and fast ones pale', - data: [data], - fit: framed([-w / 2, -h / 2, w / 2, h / 2]), scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14' }, marks: [ { - lines: { - of: ['wind', 'Streamline'], - width: 0.75, - color: { - colormap: { - kind: 'sequential', - colors: ['#123056', '#1d5fd1', '#3f8cff', '#7cc4ff', '#cfe8ff'], - }, - value: 'speed', - domain: [0.2, 2.2], - }, - }, + lines: wind.table('Streamline'), + width: 0.75, + color: field('speed') + .colormap({ + kind: 'sequential', + colors: ['#123056', '#1d5fd1', '#3f8cff', '#7cc4ff', '#cfe8ff'], + }) + .domain(0.2, 2.2), }, ], }, + fit: framed([-w / 2, -h / 2, w / 2, h / 2]), }; } diff --git a/docs/gallery/diagram.ts b/docs/gallery/diagram.ts index 5b32ceec..852533f5 100644 --- a/docs/gallery/diagram.ts +++ b/docs/gallery/diagram.ts @@ -1,63 +1,66 @@ -/** Diagram: a microgrid's controller as blocks in groups, wired port to port and laid out by latkit. */ -import { Patch, reference, type Marker, type Mark, type Port, type Side } from 'latkit/core'; -import { THEME, type Piece } from './look.ts'; +/** Diagram: a microgrid controller as blocks in groups, wired port to port, laid out by latkit. */ +import { Data, field, reference, when, type Mark, type Marker, type Port } from 'latkit'; +import type { Picture } from './render.ts'; -/** Kinds of block: the marker each draws as, its fill, and its ports. */ +/** Block kinds: each kind's marker, fill color, and ports by name. An output drives its signal. */ const KINDS = { - Source: { marker: 'rounded', color: '#173836', ports: [['out', 'right', true]] }, + Source: { marker: 'rounded', color: '#173836', ports: { out: { side: 'right', output: true } } }, Process: { marker: 'block', color: '#1c2b3f', - ports: [ - ['in', 'left', false], - ['out', 'right', true], - ], + ports: { in: 'left', out: { side: 'right', output: true } }, }, Control: { marker: 'block', color: '#3a2e18', - ports: [ - ['in', 'left', false], - ['feedback', 'bottom', false], - ['out', 'right', true], - ], + ports: { + in: 'left', + feedback: { side: 'bottom', label: 'feedback' }, + out: { side: 'right', output: true }, + }, }, - Sink: { marker: 'block', color: '#29243f', ports: [['in', 'left', false]] }, -} as const satisfies Record< - string, - { marker: Marker; color: string; ports: readonly (readonly [string, Side, boolean])[] } ->; + Sink: { marker: 'block', color: '#29243f', ports: { in: 'left' } }, +} as const satisfies Record }>; type Kind = keyof typeof KINDS; interface Block { kind: Kind; id: number; name: string; - group?: number; - alarm?: boolean; + group: number | null; + alarm: boolean; } -export default function diagram(): Piece { +export default function (): Picture { + // Areas, groups, signals, and blocks share one sequence of ids. Each table is kept as columns. let next = 1; + const areas = { ids: [] as number[], name: [] as string[] }; + const groups = { ids: [] as number[], name: [] as string[], area: [] as (number | null)[] }; + const signals = { ids: [] as number[], name: [] as string[] }; const blocks: Block[] = []; - const groups: { id: number; name: string; area?: number }[] = []; - const areas: { id: number; name: string }[] = []; - const signals: { id: number; name: string }[] = []; const plugs = new Map(); - const area = (name: string) => (areas.push({ id: next, name }), next++); - const group = (name: string, inArea?: number) => ( - groups.push({ id: next, name, area: inArea }), - next++ - ); - const block = (kind: Kind, name: string, inGroup?: number, alarm = false): Block => { + const area = (name: string) => { + areas.ids.push(next); + areas.name.push(name); + return next++; + }; + const group = (name: string, inArea: number | null = null) => { + groups.ids.push(next); + groups.name.push(name); + groups.area.push(inArea); + return next++; + }; + const block = (kind: Kind, name: string, inGroup: number | null = null, alarm = false) => { const b = { kind, id: next++, name, group: inGroup, alarm }; blocks.push(b); return b; }; - // A signal driven by `from`'s output and read by each of `to`, through its input or the port named. + // Adds a signal driven by the `out` port of each block in `from`. Each block in `to` reads it + // through its `in` port, or through the port named with it. const wire = (name: string, from: Block | Block[], ...to: (Block | [Block, string])[]) => { const id = next++; - signals.push({ id, name }); + signals.ids.push(id); + signals.name.push(name); for (const driver of [from].flat()) plugs.set(`${driver.id} out`, id); for (const reader of to) { const [b, port] = Array.isArray(reader) ? reader : [reader, 'in']; @@ -65,10 +68,10 @@ export default function diagram(): Piece { } }; - const plant = area('Microgrid'); - const sensing = group('Sensing', plant); - const control = group('Control', plant); - const power = group('Power stage', plant); + const microgrid = area('Microgrid'); + const sensing = group('Sensing', microgrid); + const control = group('Control', microgrid); + const power = group('Power stage', microgrid); const solar = block('Source', 'Solar array'); const battery = block('Source', 'Battery'); const tie = block('Source', 'Grid tie'); @@ -77,7 +80,7 @@ export default function diagram(): Piece { const filter = block('Process', 'Filter', sensing); const estimator = block('Process', 'State estimator', sensing); const dispatch = block('Process', 'Dispatch', control); - // The controller closing the loop sits between the groups it joins. + // The controller that closes the loop is in no group, so it sits between the groups it joins. const droop = block('Control', 'Droop'); const limiter = block('Process', 'Limiter', control); const inverter = block('Process', 'Inverter', power, true); @@ -95,105 +98,67 @@ export default function diagram(): Piece { wire('Output', inverter, breaker); wire('Power', breaker, bus, [droop, 'feedback']); - const patch = Patch.to('plant') - .replace( - 'Signal', - signals.map((s) => s.id), - { name: signals.map((s) => s.name) }, - ) - .replace( - 'Area', - areas.map((a) => a.id), - { name: areas.map((a) => a.name) }, - ) - .replace( - 'Group', - groups.map((g) => g.id), - { - name: groups.map((g) => g.name), - area: reference( - 'Area', - groups.map((g) => g.area ?? null), - ), - }, - ); + const plant = new Data('plant') + .replace('Signal', signals.ids, { name: signals.name }) + .replace('Area', areas.ids, { name: areas.name }) + .replace('Group', groups.ids, { name: groups.name, area: reference('Area', groups.area) }); const kinds = Object.keys(KINDS) as Kind[]; for (const kind of kinds) { const rows = blocks.filter((b) => b.kind === kind); - const plugged = Object.fromEntries( - KINDS[kind].ports.map(([port]) => [ - port, - reference( - 'Signal', - rows.map((b) => plugs.get(`${b.id} ${port}`) ?? null), - ), - ]), - ); - patch.replace( - kind, - rows.map((b) => b.id), - { - name: rows.map((b) => b.name), - status: rows.map((b) => (b.alarm ? 1 : 0)), - group: reference( - 'Group', - rows.map((b) => b.group ?? null), - ), - ...plugged, - }, - ); + const ids = rows.map((b) => b.id); + const inGroup = rows.map((b) => b.group); + // Each port's reference field names the signal on it. + const ports = Object.keys(KINDS[kind].ports).map((port) => { + const signal = rows.map((b) => plugs.get(`${b.id} ${port}`) ?? null); + return [port, reference('Signal', signal)] as const; + }); + plant.replace(kind, ids, { + name: rows.map((b) => b.name), + status: rows.map((b) => (b.alarm ? 1 : 0)), + group: reference('Group', inGroup), + ...Object.fromEntries(ports), + }); } - const title = { text: 'name', size: 11, color: '#8ea3b8' }; - const marks: Mark[] = [ - { - vertices: { of: ['plant', 'Area'], contains: ['Group', 'area'], label: title, shadow: true }, - }, - { - vertices: { - of: ['plant', 'Group'], - contains: ['Process', 'group'], - label: title, - color: '#1d2632', - shadow: true, - }, - }, - { - edges: { - of: ['plant', 'Signal'], - route: 'orthogonal', - arrows: true, - width: 1.5, - color: '#8a9ab0', - }, - }, - ...kinds.map((kind): Mark => ({ - vertices: { - of: ['plant', kind], - marker: KINDS[kind].marker, - color: KINDS[kind].color, - shadow: true, - ring: { - when: [[{ op: 'gt', args: ['status', 0] }, { color: '#ffb000' }]], - otherwise: { missing: true }, - }, - ports: KINDS[kind].ports.map(([name, side, output]): [string, Port] => [ - name, - { side, output, ...(name === 'feedback' ? { label: 'feedback' } : {}) }, - ]), - label: { text: 'name', size: 12 }, - }, - })), - ]; + + const title = { text: '$name', size: 11, color: '#8ea3b8' }; return { - name: 'diagram', - alt: 'A microgrid controller as blocks in nested groups, wired port to port with routed wires and a feedback loop', - data: [patch], scene: { space: { diagram: { grid: { dots: { pitch: 20 } } } }, - layout: { layered: { flow: 'right', gap: 40, rankGap: 52 } }, + layout: { layered: { direction: 'right', gap: 40, rankGap: 52 } }, ease: 0, - theme: { ...THEME, surface: '#151c26', grid: '#ffffff0d' }, - marks, + // The dark theme, with a darker background and labels without halos. + theme: { + background: '#0b0f14', + text: { color: '#c3ccd6', font: 'sans-serif', size: 12, halo: 0 }, + grid: '#ffffff0d', + diagram: { group: '#151c26' }, + }, + marks: [ + { vertices: plant.table('Area'), contains: ['Group', 'area'], label: title, shadow: true }, + { + vertices: plant.table('Group'), + contains: ['Process', 'group'], + label: title, + color: '#1d2632', + shadow: true, + }, + { + edges: plant.table('Signal'), + route: 'orthogonal', + arrows: true, + width: 1.5, + color: '#8a9ab0', + }, + ...kinds.map((kind): Mark => ({ + vertices: plant.table(kind), + marker: KINDS[kind].marker, + color: KINDS[kind].color, + shadow: true, + ring: when(field('status').gt(0)).then('#ffb000').otherwise(null), + ports: Object.entries(KINDS[kind].ports), + label: { text: '$name', size: 12 }, + })), + ], }, }; } diff --git a/docs/gallery/flow.ts b/docs/gallery/flow.ts index b2f38615..99d3c838 100644 --- a/docs/gallery/flow.ts +++ b/docs/gallery/flow.ts @@ -1,20 +1,44 @@ /** - * A DC power flow: each bus's voltage angle from what it injects, through branches weighted by - * their susceptance, and how the angles shift when one branch opens. + * A DC power flow. It computes each bus's voltage angle from its injection, with branches weighted + * by their susceptance, each branch's flow from the angles, and how the angles shift when one branch + * opens. Each island, a set of buses joined by branches in service, balances on its own. */ -/** Buses and the branches between them, each branch's reactance in per unit. */ +/** The power base in MVA. Injections are in per unit of it, and flows come out in MW. */ +export const BASE = 100; + +/** A network of buses and the branches between them. Reactance is in per unit. Branches marked in + * `out` are out of service: they join nothing and carry nothing. */ export interface Network { readonly buses: number; readonly from: Uint32Array; readonly to: Uint32Array; - readonly reactance: Float32Array; + readonly reactance: Float32Array | Float64Array; + readonly out?: Uint8Array; } -/** The smallest reactance a branch is taken to have, so none is infinitely stiff. */ +/** The smallest reactance magnitude a branch is given, so no susceptance is infinite. */ const STIFFEST = 1e-4; -/** The network's susceptance matrix, row by row, and each bus's island. */ +/** A branch's reactance magnitude, so series capacitors (negative reactance) count by their size. */ +function reactance(net: Network, e: number): number { + return Math.max(Math.abs(net.reactance[e]!), STIFFEST); +} + +/** Returns each bus's island, as the row of one bus shared by every bus of the island. */ +export function islands(net: Network): Uint32Array { + const { buses, from, to, out } = net; + const root = Uint32Array.from({ length: buses }, (_, b) => b); + const find = (b: number): number => { + while (root[b] !== b) b = root[b] = root[root[b]!]!; + return b; + }; + for (let e = 0; e < from.length; e++) if (!out?.[e]) root[find(from[e]!)] = find(to[e]!); + return Uint32Array.from({ length: buses }, (_, b) => find(b)); +} + +/** The network's susceptance matrix in compressed sparse rows, with the diagonal stored separately, + * and each bus's island. */ class Susceptance { readonly start: Uint32Array; readonly column: Uint32Array; @@ -25,9 +49,10 @@ class Susceptance { constructor(net: Network) { this.net = net; - const { buses, from, to } = net; + const { buses, from, to, out } = net; const degree = new Uint32Array(buses + 1); for (let e = 0; e < from.length; e++) { + if (out?.[e]) continue; degree[from[e]! + 1]!++; degree[to[e]! + 1]!++; } @@ -38,6 +63,7 @@ class Susceptance { this.diagonal = new Float64Array(buses); const fill = degree.slice(0, buses); for (let e = 0; e < from.length; e++) { + if (out?.[e]) continue; const [a, c, w] = [from[e]!, to[e]!, this.susceptance(e)]; this.column[fill[a]!] = c; this.weight[fill[a]!++] = w; @@ -46,22 +72,15 @@ class Susceptance { this.diagonal[a]! += w; this.diagonal[c]! += w; } - // Islands, so each balances on its own. - const root = Uint32Array.from({ length: buses }, (_, b) => b); - const find = (b: number): number => { - while (root[b] !== b) b = root[b] = root[root[b]!]!; - return b; - }; - for (let e = 0; e < from.length; e++) root[find(from[e]!)] = find(to[e]!); - this.island = Uint32Array.from({ length: buses }, (_, b) => find(b)); + this.island = islands(net); } - /** A branch's susceptance: its reactance's inverse, series capacitors taken at their size. */ + /** A branch's susceptance: the inverse of its reactance. */ susceptance(e: number): number { - return 1 / Math.max(Math.abs(this.net.reactance[e]!), STIFFEST); + return 1 / reactance(this.net, e); } - /** `B·v` into `out`. */ + /** Writes `B·v` into `out`. */ times(v: Float64Array, out: Float64Array): void { for (let b = 0; b < v.length; b++) { let sum = this.diagonal[b]! * v[b]!; @@ -71,7 +90,7 @@ class Susceptance { } } - /** `p` less its island's mean, so every island's injections sum to zero. */ + /** Returns `p` minus its island's mean, so every island's injections sum to zero. */ balanced(p: Float64Array): Float64Array { const sum = new Float64Array(p.length); const count = new Float64Array(p.length); @@ -83,12 +102,13 @@ class Susceptance { return p.map((v, b) => v - mean[this.island[b]!]!); } - /** The angles `θ` with `B·θ = p`, by conjugate gradients with each bus's own scale. */ + /** Solves `B·θ = p` for the angles `θ`, after balancing `p` per island, by conjugate gradients + * with a diagonal (Jacobi) preconditioner. A bus with no branch in service keeps an angle of 0. */ solve(injection: Float64Array, tolerance = 1e-10): Float64Array { const n = injection.length; const p = this.balanced(injection); const theta = new Float64Array(n); - const scale = this.diagonal.map((v) => 1 / v); + const scale = this.diagonal.map((v) => (v > 0 ? 1 / v : 0)); const r = p.slice(); const z = r.map((v, b) => v * scale[b]!); const d = z.slice(); @@ -116,14 +136,25 @@ class Susceptance { } } -/** Each bus's angle in radians, from its injection in per unit. */ +/** Returns each bus's angle in radians, given its injection in per unit. */ export function angles(net: Network, injection: Float64Array): Float64Array { return new Susceptance(net).solve(injection); } +/** Returns each branch's flow in MW from its `from` end to its `to` end, given the angles + * `theta`. */ +export function flows(net: Network, theta: Float64Array): Float64Array { + const flow = new Float64Array(net.from.length); + for (let e = 0; e < flow.length; e++) + if (!net.out?.[e]) + flow[e] = ((theta[net.from[e]!]! - theta[net.to[e]!]!) * BASE) / reactance(net, e); + return flow; +} + /** - * How each bus's angle shifts when `branch` opens, from the angles before: the flow the branch - * carried moves onto the rest of the network as if injected at one end and drawn at the other. + * Returns how each bus's angle shifts when `branch` opens, given the angles `theta` before. The + * branch's flow moves onto the rest of the network, as if injected at one end and withdrawn at the + * other. */ export function shifts(net: Network, theta: Float64Array, branch: number): Float64Array { const b = new Susceptance(net); diff --git a/docs/gallery/frequency.ts b/docs/gallery/frequency.ts index 3b07740f..cf81b600 100644 --- a/docs/gallery/frequency.ts +++ b/docs/gallery/frequency.ts @@ -1,46 +1,41 @@ -/** Frequency: every machine of ACTIVSg10k swinging after a fault, traced over time on a plot. */ -import { THEME, type Piece } from './look.ts'; +/** Frequency: the speed of every machine in ACTIVSg10k after a fault, traced over time. */ +import { field } from 'latkit'; import { readCase } from './grids.ts'; +import type { Picture } from './render.ts'; -export default function frequency(): Piece { - const c = readCase('activsg10k'); - const fault = c.header.fault!; - const [fx, fy] = [c.positions[2 * fault.bus]!, c.positions[2 * fault.bus + 1]!]; - // Each machine's distance from the fault, in degrees: the disturbance reaches the near ones first. - const distance = Float32Array.from(c.machineBus, (b) => - Math.hypot(c.positions[2 * b]! - fx, c.positions[2 * b + 1]! - fy), - ); - c.patch.set('Machine', { count: c.header.machines }, { distance }); - const machines = c.header.machines.toLocaleString('en-US'); +export default async function (): Promise { + const { grid, header, lon, lat, machineBus, simulation } = await readCase('activsg10k'); + const fault = header.fault!; + const [fx, fy] = [lon[fault.bus]!, lat[fault.bus]!]; + // Each machine's distance from the fault in degrees. The disturbance reaches near machines first. + const distance = Float32Array.from(machineBus, (b) => Math.hypot(lon[b]! - fx, lat[b]! - fy)); + grid.set('Machine', { distance }); return { - name: 'frequency', - alt: `The speed of ${machines} machines over ten seconds after a fault, near ones bright and swinging hardest, far ones dark`, - data: [c.patch], - // The whole recording across; speeds fitted up the side. - camera: { x: [0, (c.header.frames! - 1) * c.header.dt!], y: null }, scene: { space: { plot: { x: { label: 'Time (s)' }, y: { label: 'Speed (mHz from 60 Hz)' } } }, ease: 0, - theme: THEME, + // The dark theme, with a darker background and labels without halos. + theme: { + background: '#0b0f14', + text: { color: '#c3ccd6', font: 'sans-serif', size: 12, halo: 0 }, + }, marks: [ { - lines: { - of: ['grid', 'Machine'], - trace: 'speed', - // The nearest machines, swinging hardest, on top. - priority: { op: 'neg', args: ['distance'] }, - width: 0.7, - color: { - colormap: { - kind: 'sequential', - colors: ['#fff1c9', '#ffb36b', '#e8577a', '#7a3fc4', '#24366e'], - }, - value: 'distance', - domain: [0, 9], - }, - }, + lines: grid.table('Machine'), + trace: 'speed', + // The nearest machines, which swing hardest, draw on top. + priority: field('distance').neg(), + width: 0.7, + color: field('distance') + .colormap({ + kind: 'sequential', + colors: ['#fff1c9', '#ffb36b', '#e8577a', '#7a3fc4', '#24366e'], + }) + .domain(0, 9), }, ], }, + // x spans the whole recording. With `y` null, the speed range fits what is in view. + camera: { x: [0, simulation!.duration], y: null }, }; } diff --git a/docs/gallery/globe.ts b/docs/gallery/globe.ts index 89cec973..3b4c3162 100644 --- a/docs/gallery/globe.ts +++ b/docs/gallery/globe.ts @@ -1,9 +1,13 @@ -/** Globe: routes between the world's great cities along great circles, comets flowing on them. */ -import { Patch, reference } from 'latkit/core'; -import { THEME, worldMap, type Piece } from './look.ts'; +/** Globe: routes between world cities along great circles, with comets flowing along them. */ +import fs from 'node:fs'; +import { Data, field, reference, units, when } from 'latkit'; +import type { Picture } from './render.ts'; -/** Cities: name, longitude, latitude, and people in millions, about. */ -const CITIES: readonly (readonly [string, number, number, number])[] = [ +/** A city's name, longitude, latitude, and population in millions. */ +type City = readonly [name: string, lon: number, lat: number, people: number]; + +/** The cities. Populations are approximate. */ +const CITIES: readonly City[] = [ ['Tokyo', 139.69, 35.69, 37], ['Delhi', 77.21, 28.61, 32], ['Shanghai', 121.47, 31.23, 28], @@ -46,7 +50,7 @@ const CITIES: readonly (readonly [string, number, number, number])[] = [ ['Reykjavík', -21.94, 64.15, 0.2], ]; -/** Long hauls between hubs, besides each city's nearest neighbors. */ +/** Long-haul routes between hubs, added to the routes to each city's three nearest neighbors. */ const HAULS = [ ['London', 'New York'], ['New York', 'Los Angeles'], @@ -71,11 +75,8 @@ const HAULS = [ ['Santiago', 'Auckland'], ] as const; -/** Degrees of the great circle between two cities. */ -function arc( - a: readonly [string, number, number, number], - b: readonly [string, number, number, number], -): number { +/** The great-circle distance between two cities, in degrees, by the haversine formula. */ +function arc(a: City, b: City): number { const r = Math.PI / 180; const [x1, y1, x2, y2] = [a[1] * r, a[2] * r, b[1] * r, b[2] * r]; const h = @@ -83,7 +84,8 @@ function arc( return (2 * Math.asin(Math.sqrt(h))) / r; } -export default function globe(): Piece { +export default async function (): Promise { + // Each city's routes to its three nearest neighbors, and the long hauls, each pair once. const index = new Map(CITIES.map((c, i) => [c[0], i])); const pairs = new Set(); CITIES.forEach((city, i) => { @@ -96,81 +98,65 @@ export default function globe(): Piece { for (const [a, b] of HAULS) pairs.add([index.get(a)!, index.get(b)!].sort((p, q) => p - q).join(',')); const routes = [...pairs].map((p) => p.split(',').map(Number) as [number, number]); - const data = Patch.to('world') - .replace( - 'City', - { count: CITIES.length }, - { - name: CITIES.map((c) => c[0]), - lon: CITIES.map((c) => c[1]), - lat: CITIES.map((c) => c[2]), - people: CITIES.map((c) => c[3]), - }, - ) - .replace( - 'Route', - { count: routes.length }, - { - from: reference( - 'City', - routes.map(([a]) => a), - ), - to: reference( - 'City', - routes.map(([, b]) => b), - ), - degrees: routes.map(([a, b]) => arc(CITIES[a]!, CITIES[b]!)), - }, - ); + const from = routes.map(([a]) => a); + const to = routes.map(([, b]) => b); + const world = new Data('world') + .replace('City', { + name: CITIES.map((c) => c[0]), + lon: CITIES.map((c) => c[1]), + lat: CITIES.map((c) => c[2]), + people: CITIES.map((c) => c[3]), + }) + .replace('Route', { + from: reference('City', from), + to: reference('City', to), + degrees: routes.map(([a, b]) => arc(CITIES[a]!, CITIES[b]!)), + }); + const maps = new URL('../../npm/examples/public/maps/', import.meta.url); + const coastline = fs.readFileSync(new URL('coastline.geojson.gz', maps)); + const countries = fs.readFileSync(new URL('countries.geojson.gz', maps)); + const coast = await Data.read(coastline, { name: 'coast' }); + const borders = await Data.read(countries, { name: 'borders' }); + // The most populous cities are named: those at least as populous as the fourteenth largest. + const named = CITIES.map((c) => c[3]).sort((p, q) => q - p)[13]!; return { - name: 'globe', - alt: `Routes between ${CITIES.length} of the world's great cities, along great circles on a globe, comets flowing on them`, - data: [ - data, - { name: 'coast', file: worldMap('coastline') }, - { name: 'borders', file: worldMap('countries') }, - ], - camera: { center: [16, 20], zoom: 2.3, pitch: 0, bearing: 0 }, - elapsed: 3, scene: { - space: { - earth: { projection: 'globe', surface: '#0e1824', graticule: false, atmosphere: true }, - }, + space: { earth: { projection: 'globe', graticule: false, atmosphere: true } }, ease: 0, + // The dark theme, with a darker background and labels without halos. theme: { - ...THEME, - flow: { color: '#fff6e0e0', spacing: 9, length: 3.5, width: 2.6, units: true }, + background: '#0b0f14', + text: { color: '#c3ccd6', font: 'sans-serif', size: 12, halo: 0 }, + earth: { ground: '#0e1824' }, + flow: { color: '#fff6e0e0', spacing: units(9), length: units(3.5), width: 2.6 }, }, marks: [ - { lines: { of: ['borders', 'lines'], width: 0.5, color: '#22384f' } }, - { lines: { of: ['coast', 'lines'], width: 0.8, color: '#3d6a8f' } }, + { lines: borders.table('lines'), width: 0.5, color: '#22384f' }, + { lines: coast.table('lines'), width: 0.8, color: '#3d6a8f' }, { - edges: { - of: ['world', 'Route'], - ends: ['from', 'to'], - width: 1.3, - flow: 2.5, - color: { - colormap: { - kind: 'sequential', - colors: ['#5ce0c2', '#5aa9ff', '#b48cff', '#ffb35c'], - }, - value: 'degrees', - domain: [5, 90], - }, - }, + edges: world.table('Route'), + ends: ['from', 'to'], + width: 1.3, + flow: 2.5, + color: field('degrees') + .colormap({ kind: 'sequential', colors: ['#5ce0c2', '#5aa9ff', '#b48cff', '#ffb35c'] }) + .domain(5, 90), }, { - vertices: { - of: ['world', 'City'], - position: ['lon', 'lat'], - radius: { scale: [2, 6], value: 'people', domain: [0, 37] }, - color: '#f4f8ff', - priority: 'people', - label: { text: 'name', size: 11, color: '#c9d6e6', priority: 'people', max: 14 }, + vertices: world.table('City'), + position: ['$lon', '$lat'], + radius: field('people').scale(2, 6).domain(0, 37), + color: '#f4f8ff', + priority: '$people', + label: { + text: when(field('people').ge(named)).then('$name').otherwise(null), + size: 11, + color: '#c9d6e6', }, }, ], }, + camera: { center: [16, 20], zoom: 2.3, pitch: 0, bearing: 0 }, + elapsed: 3, }; } diff --git a/docs/gallery/grid.ts b/docs/gallery/grid.ts index 8d070c99..70f1d544 100644 --- a/docs/gallery/grid.ts +++ b/docs/gallery/grid.ts @@ -1,60 +1,45 @@ -/** The grid: ACTIVSg70k's buses and branches over the states they stand in, drawn by voltage. */ -import { fileURLToPath } from 'node:url'; -import type { CustomColormap, Mark, Size } from 'latkit/core'; -import { THEME, type Piece } from './look.ts'; +/** The grid: ACTIVSg70k's buses and branches over the US states, colored and sized by voltage. */ +import fs from 'node:fs'; +import { Data, field } from 'latkit'; import { readCase } from './grids.ts'; +import type { Picture } from './render.ts'; -/** The states of the United States, which the CLI reads as GeoJSON. */ -export const STATES = fileURLToPath( - new URL('../../crates/latkit-cli/examples/branch-opening/states.geojson', import.meta.url), -); - -/** Colors of the voltage levels, from below 100 kV to 765 kV: the higher, the hotter. */ -export const LEVELS: CustomColormap = { - kind: 'sequential', - colors: ['#1c2e44', '#22507a', '#2d80b3', '#58bde6', '#ffcf73', '#fff4dc'], -}; - -/** Widths by voltage level, growing with its square: the backbone stands out of the rest. */ -export const WIDTHS: Size = { - scale: [0.3, 2.3], - value: { op: 'mul', args: ['level', 'level'] }, - domain: [0, 25], -}; - -/** The states drawn beneath a grid: filled a shade above the ground, outlined faintly. */ -export function states(outline = '#26364a'): Mark[] { - return [ - { areas: { of: ['states', 'polygons'], color: '#0f1720', pick: false } }, - { lines: { of: ['states', 'polygons'], width: 0.6, color: outline } }, - ]; -} - -export default function grid(): Piece { - const c = readCase('activsg70k'); +export default async function (): Promise { + const { grid } = await readCase('activsg70k'); + const usa = new URL( + '../../crates/latkit-cli/examples/branch-opening/states.geojson', + import.meta.url, + ); + const states = await Data.read(fs.readFileSync(usa), { name: 'states' }); + const level = field('level'); return { - name: 'grid', - alt: 'Seventy thousand buses of a synthetic grid on the United States, its lines colored and widened by voltage', - data: [c.patch, { name: 'states', file: STATES }], - // Over the grid from the south, tilted, so the backbone recedes toward the Great Lakes. - camera: { center: [-84.2, 37.2], zoom: 4.38, pitch: 42, bearing: -8 }, scene: { - space: { earth: { projection: 'mercator', graticule: false, surface: THEME.background } }, + space: { earth: { graticule: false } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14', earth: { ground: '#0b0f14' } }, marks: [ - ...states(), - { vertices: { of: ['grid', 'Bus'], position: ['lon', 'lat'], radius: 0, pick: false } }, + // The states, filled slightly lighter than the ground, with faint outlines. + { areas: states.table('polygons'), color: '#0f1720', pick: false }, + { lines: states.table('polygons'), width: 0.6, color: '#26364a' }, + // The buses draw nothing: they only place the lines' ends. + { vertices: grid.table('Bus'), position: ['$lon', '$lat'], radius: 0, pick: false }, { - edges: { - of: ['grid', 'Line'], - ends: ['from', 'to'], - priority: 'level', - width: WIDTHS, - color: { colormap: LEVELS, value: 'level', domain: [0, 5] }, - }, + edges: grid.table('Line'), + ends: ['from', 'to'], + priority: level, + // Width grows with the level's square, so the high-voltage backbone stands out. + width: level.mul(level).scale(0.3, 2.3).domain(0, 25), + // The voltage levels, from below 100 kV to 765 kV. Higher levels are hotter colors. + color: level + .colormap({ + kind: 'sequential', + colors: ['#1c2e44', '#22507a', '#2d80b3', '#58bde6', '#ffcf73', '#fff4dc'], + }) + .domain(0, 5), }, ], }, + // A tilted view from the south, so the backbone recedes toward the Great Lakes. + camera: { center: [-84.2, 37.2], zoom: 4.38, pitch: 42, bearing: -8 }, }; } diff --git a/docs/gallery/grids.ts b/docs/gallery/grids.ts index 40207670..f06336b9 100644 --- a/docs/gallery/grids.ts +++ b/docs/gallery/grids.ts @@ -1,127 +1,49 @@ /** - * The synthetic grid cases the example suite fetches from `npm/examples/public/grids`, read into one - * dataset named `grid`: buses with their voltage level, branches between them, machines, and where - * a case simulated a fault, each machine's speed frame by frame. The format is the suite's own: a - * "LKG1" magic, a JSON header, and typed arrays it places. + * Reads the synthetic grid cases that the example suite fetches from `npm/examples/public/grids` + * into one dataset named `grid`. It holds buses with their voltage level, the lines between them, + * machines, and, for a case that simulated a fault, each machine's speed per frame. The suite's + * `format.ts` describes the files and reads them. */ import fs from 'node:fs'; -import { fileURLToPath } from 'node:url'; -import { gunzipSync } from 'node:zlib'; -import { Patch, reference } from 'latkit/core'; +import { Data, reference } from 'latkit'; +import { inflate, parseCase, type CaseFile } from '../../npm/examples/src/pages/grids/format.ts'; -interface Header { - readonly case: string; - readonly buses: number; - readonly branches: number; - readonly machines: number; - readonly frames?: number; - readonly dt?: number; - readonly speedScale?: number; - readonly fault?: { readonly bus: number; readonly on: number; readonly off: number }; - readonly arrays: Readonly< - Record - >; +/** A case's arrays, and its dataset `grid`. */ +export interface Case extends CaseFile { + readonly grid: Data; } -export interface Case { - readonly name: string; - readonly header: Header; - readonly patch: Patch; - /** Each bus's longitude and latitude, back to back. */ - readonly positions: Float32Array; - /** Each machine's bus row. */ - readonly machineBus: Uint32Array; - /** One of the case's arrays by name, such as `branch.reactance`. */ - readonly array: (name: string, of: new (b: ArrayBuffer, o: number, l: number) => T) => T; -} - -/** A voltage level: 0 below 100 kV, then 115–161, 230, 345, 500, and 765 kV. */ -export function levelOf(kv: number): number { - return kv >= 700 ? 5 : kv >= 450 ? 4 : kv >= 300 ? 3 : kv >= 200 ? 2 : kv >= 100 ? 1 : 0; -} - -export function readCase(id: string): Case { - const file = fileURLToPath(new URL(`../../npm/examples/public/grids/${id}.bin`, import.meta.url)); - const raw = fs.readFileSync(file); - const bytes = raw[0] === 0x1f && raw[1] === 0x8b ? gunzipSync(raw) : raw; - const buffer = bytes.buffer.slice( - bytes.byteOffset, - bytes.byteOffset + bytes.byteLength, - ) as ArrayBuffer; - const view = new DataView(buffer); - if (view.getUint32(0, true) !== 0x31474b4c) throw new Error(`${id} is not a grid case`); - const length = view.getUint32(4, true); - const header = JSON.parse(new TextDecoder().decode(new Uint8Array(buffer, 8, length))) as Header; - const body = Math.ceil((8 + length) / 8) * 8; - const array = (name: string, of: new (b: ArrayBuffer, o: number, l: number) => T): T => { - const found = header.arrays[name]; - if (!found) throw new Error(`${id} has no ${name}`); - return new of(buffer, body + found[1], found[2]); - }; - const names = (name: string): string[] => { - const offsets = array(`${name}.offsets`, Uint32Array); - const text = array(`${name}.bytes`, Uint8Array); - const decoder = new TextDecoder(); - return Array.from({ length: offsets.length - 1 }, (_, i) => - decoder.decode(text.subarray(offsets[i]!, offsets[i + 1]!)), - ); - }; - const [n, m, g] = [header.buses, header.branches, header.machines]; - const position = array('bus.position', Float32Array); - const kv = array('bus.kv', Float32Array); - const ends = array('branch.ends', Uint32Array); - const machineBus = array('machine.bus', Uint32Array); - const generation = array('bus.generation', Float32Array); - const from = new Uint32Array(m); - const to = new Uint32Array(m); - const lineKv = new Float32Array(m); - for (let e = 0; e < m; e++) { - [from[e], to[e]] = [ends[2 * e]!, ends[2 * e + 1]!]; - lineKv[e] = Math.max(kv[from[e]!]!, kv[to[e]!]!); - } - const lon = Float64Array.from({ length: n }, (_, b) => position[2 * b]!); - const lat = Float64Array.from({ length: n }, (_, b) => position[2 * b + 1]!); - const patch = Patch.to('grid') - .replace( - 'Bus', - { count: n }, - { - lon, - lat, - name: names('bus.name'), - kv, - level: Float32Array.from(kv, levelOf), - load: array('bus.load', Float32Array), - generation, - }, - ) - .replace( - 'Line', - { count: m }, - { - from: reference('Bus', from), - to: reference('Bus', to), - kv: lineKv, - level: Float32Array.from(lineKv, levelOf), - }, - ) - .replace( - 'Machine', - { count: g }, - { - lon: Float64Array.from(machineBus, (b) => lon[b]!), - lat: Float64Array.from(machineBus, (b) => lat[b]!), - power: array('machine.power', Float32Array), - mva: array('machine.mva', Float32Array), - }, - ); - if (header.frames !== undefined && header.dt !== undefined) { - // Per unit of 60 Hz, in mHz. - const scale = (header.speedScale ?? 1) * 60_000; - const speeds = Float32Array.from(array('machine.speed', Int16Array), (v) => v * scale); - patch.axis('time', 's'); - for (let f = 0; f < header.frames; f++) - patch.append(f * header.dt, 'Machine', { speed: speeds.subarray(f * g, (f + 1) * g) }); +export async function readCase(id: string): Promise { + const file = new URL(`../../npm/examples/public/grids/${id}.bin`, import.meta.url); + const c = parseCase(id, await inflate(fs.readFileSync(file))); + const machineLon = Float64Array.from(c.machineBus, (b) => c.lon[b]!); + const machineLat = Float64Array.from(c.machineBus, (b) => c.lat[b]!); + const grid = new Data('grid') + .replace('Bus', { + lon: c.lon, + lat: c.lat, + name: c.names('bus.name'), + kv: c.kv, + level: c.level, + load: c.array('bus.load', Float32Array), + generation: c.array('bus.generation', Float32Array), + }) + .replace('Line', { + from: reference('Bus', c.from), + to: reference('Bus', c.to), + kv: c.lineKv, + level: c.lineLevel, + }) + .replace('Machine', { + lon: machineLon, + lat: machineLat, + power: c.array('machine.power', Float32Array), + mva: c.array('machine.mva', Float32Array), + }); + if (c.simulation) { + const { frames, dt, speeds } = c.simulation; + const times = Array.from({ length: frames }, (_, f) => f * dt); + grid.axis('time', 's').append('Machine', times, { speed: speeds }); } - return { name: header.case, header, patch, positions: position, machineBus, array }; + return { ...c, grid }; } diff --git a/docs/gallery/helios.ts b/docs/gallery/helios.ts index 30725561..5c94446e 100644 --- a/docs/gallery/helios.ts +++ b/docs/gallery/helios.ts @@ -1,29 +1,31 @@ -/** Helios: flow through a synthetic network of five rings, its edges carrying comets. */ -import { Patch, reference } from 'latkit/core'; -import { TAU, THEME, framed, type Piece } from './look.ts'; +/** Helios: a synthetic network of five rings, with comets flowing along its edges. */ +import { Data, field, reference } from 'latkit'; +import { framed } from './look.ts'; +import type { Picture } from './render.ts'; -export default function helios(): Piece { +export default function (): Picture { const rings = 5; const around = 96; const x: number[] = []; const y: number[] = []; const heat: number[] = []; const radius: number[] = []; - const name: string[] = []; + const name: (string | null)[] = []; const from: number[] = []; const to: number[] = []; for (let ring = 0; ring < rings; ring++) { for (let node = 0; node < around; node++) { - const angle = (node * TAU) / around + ring * 0.075; + const angle = (2 * Math.PI * node) / around + ring * 0.075; const r = 92 + ring * 55 + 8 * Math.sin(angle * 6); x.push(r * Math.cos(angle)); y.push(r * Math.sin(angle)); heat.push(0.25 + 0.7 * Math.sin(angle * 0.5 + ring * 0.3) ** 2); const hub = node % 12 === 0; radius.push(hub ? 5 : 2.3); - name.push(hub && ring === rings - 1 ? `S${String(node / 12 + 1).padStart(2, '0')}` : ''); + // Only the outer ring's hubs are named. Other buses have no name, so they have no label. + name.push(hub && ring === rings - 1 ? `S${String(node / 12 + 1).padStart(2, '0')}` : null); const here = ring * around + node; - // Around each ring, inward across rings, and chords across the core. + // Branches run around each ring, inward to the ring inside it, and as chords across the core. from.push(here); to.push(ring * around + ((node + 1) % around)); if (ring > 0) { @@ -36,35 +38,31 @@ export default function helios(): Piece { } } } - const data = Patch.to('art') - .replace('Bus', { count: x.length }, { x, y, heat, radius, name }) - .replace( - 'Branch', - { count: from.length }, - { from: reference('Bus', from), to: reference('Bus', to) }, - ); + const art = new Data('art') + .replace('Bus', { x, y, heat, radius, name }) + .replace('Branch', { from: reference('Bus', from), to: reference('Bus', to) }); return { - name: 'helios', - alt: 'A synthetic network of 480 buses in five concentric rings, comets flowing along its 896 branches', - data: [data], - elapsed: 1.25, - fit: framed([-335, -335, 335, 335]), scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: { ...THEME, flow: { color: '#d9fff0a0', spacing: 100 } }, + // The dark theme, with a darker background and labels without halos. + theme: { + background: '#0b0f14', + text: { color: '#c3ccd6', font: 'sans-serif', size: 12, halo: 0 }, + flow: { color: '#d9fff0a0', spacing: 100 }, + }, marks: [ - { edges: { of: ['art', 'Branch'], ends: ['from', 'to'], width: 0.85, flow: 18 } }, + { edges: art.table('Branch'), ends: ['from', 'to'], width: 0.85, flow: 18 }, { - vertices: { - of: ['art', 'Bus'], - position: ['x', 'y'], - radius: 'radius', - color: { colormap: 'viridis', value: 'heat', domain: [0, 1] }, - label: { text: 'name', size: 13, color: '#c0e5dd' }, - }, + vertices: art.table('Bus'), + position: ['$x', '$y'], + radius: '$radius', + color: field('heat').colormap('viridis').domain(0, 1), + label: { text: '$name', size: 13, color: '#c0e5dd' }, }, ], }, + fit: framed([-335, -335, 335, 335]), + elapsed: 1.25, }; } diff --git a/docs/gallery/look.ts b/docs/gallery/look.ts index 1a99b94a..93fea219 100644 --- a/docs/gallery/look.ts +++ b/docs/gallery/look.ts @@ -1,28 +1,12 @@ -/** - * What every piece of the gallery shares: its frame, its ground and type, and what a piece is. - */ -import fs from 'node:fs'; -import { fileURLToPath } from 'node:url'; -import { gunzipSync } from 'node:zlib'; -import type { Camera, Fit, Patch, Scene } from 'latkit/core'; +/** The gallery's picture size, and framing a box in it. */ +import type { Fit } from 'latkit'; -/** Pixels written. Views lay out half as many CSS pixels, drawn twice as sharp. */ +/** Picture size in pixels. Views lay out at half this size in CSS pixels. */ export const SIZE = [1600, 1000] as const; -export const SCALE = 2; - -/** Every piece's ground and type: latkit's dark theme, a deeper ground, no label halos. */ -export const THEME = { - base: 'dark', - background: '#0b0f14', - text: '#c3ccd6', - font: 'sans-serif', - fontSize: 12, - halo: 0, -} as const; /** - * A fit of `box`, a drawing's least and greatest x and y with y up, into the picture: centered, - * with an even margin around. + * Fits a box into the picture, centered with a margin of at least 7% on each side. The box is a + * drawing's `[minX, minY, maxX, maxY]`, with y up. */ export function framed([x0, y0, x1, y1]: readonly [number, number, number, number]): Fit { const aspect = SIZE[0] / SIZE[1]; @@ -37,55 +21,3 @@ export function framed([x0, y0, x1, y1]: readonly [number, number, number, numbe }, }; } - -/** A dataset as a file the CLI reads as it is: GeoJSON names its dataset. */ -export interface File { - readonly name: string; - readonly file: string; -} - -/** One picture of the gallery: its datasets, its scene, and how the view looks at it. */ -export interface Piece { - /** The picture's name: `docs/images/NAME.png`. */ - readonly name: string; - /** What it shows, for its alt text. */ - readonly alt: string; - readonly data: readonly (Patch | File)[]; - readonly scene: Scene; - /** Pixels written, when not `SIZE`: the hero is a banner. */ - readonly size?: readonly [number, number]; - /** The camera; without one the view frames everything drawn. */ - readonly camera?: Camera; - /** What a view without a camera frames: everything drawn unless set. */ - readonly fit?: Fit; - /** Seconds of effects shown: where flowing comets are. */ - readonly elapsed?: number; - /** The axis coordinate shown, where sampled fields are read. */ - readonly at?: number; -} - -/** `n` values of `f(i)`, as 32-bit floats. */ -export function floats(n: number, f: (i: number) => number): Float32Array { - const values = new Float32Array(n); - for (let i = 0; i < n; i++) values[i] = f(i); - return values; -} - -/** Row ids `0..n` named by a function of the row, for a reference field. */ -export function rows(n: number, f: (i: number) => number): Uint32Array { - const ids = new Uint32Array(n); - for (let i = 0; i < n; i++) ids[i] = f(i); - return ids; -} - -export const TAU = 2 * Math.PI; - -/** A map the example suite ships, inflated to `target/gallery` for the CLI to read as GeoJSON. */ -export function worldMap(name: string): string { - const out = fileURLToPath(new URL(`../../target/gallery/${name}.geojson`, import.meta.url)); - const source = fileURLToPath( - new URL(`../../npm/examples/public/maps/${name}.geojson.gz`, import.meta.url), - ); - fs.writeFileSync(out, gunzipSync(fs.readFileSync(source))); - return out; -} diff --git a/docs/gallery/opening.ts b/docs/gallery/opening.ts index 9212ab0d..4bd5d8c2 100644 --- a/docs/gallery/opening.ts +++ b/docs/gallery/opening.ts @@ -1,231 +1,178 @@ /** * A line opening: ACTIVSg70k as its Little Rock to Newark 765 kV line opens. A DC power flow of the - * whole grid finds where the line's power goes: the branches that pick it up glow amber, those it - * leaves glow blue, and the rest of the grid stays a quiet web. Each bus stands as high as its - * frequency, recorded 2.5 s into a simulation of the same opening. + * whole grid shows where the line's power goes. Branches that pick it up are amber, branches that + * lose flow are cyan, and the rest of the grid is a dim web. Each bus's height is its frequency + * 2.5 s into a simulation of the same opening. */ import fs from 'node:fs'; -import { fileURLToPath } from 'node:url'; import { gunzipSync } from 'node:zlib'; -import { Patch, reference, type ExprInput } from 'latkit/core'; -import { angles, shifts } from './flow.ts'; -import { levelOf, readCase } from './grids.ts'; -import { THEME, worldMap, type Piece } from './look.ts'; -import { STATES } from './grid.ts'; +import { Data, field, type Mark } from 'latkit'; +import { BASE, angles, flows, shifts } from './flow.ts'; +import { readCase } from './grids.ts'; +import type { Picture } from './render.ts'; -/** Each bus's frequency deviation in hertz, as `record.ts` wrote it. */ +/** Reads each bus's frequency deviation in hertz from the file `record.ts` writes. */ function frequency(): Float32Array { - const file = fileURLToPath(new URL('opening.frequency.gz', import.meta.url)); - const bytes = gunzipSync(fs.readFileSync(file)); + const bytes = gunzipSync(fs.readFileSync(new URL('opening.frequency.gz', import.meta.url))); const tenMicrohertz = new Int16Array(bytes.buffer, bytes.byteOffset, bytes.byteLength / 2); return Float32Array.from(tenMicrohertz, (v) => v * 1e-5); } -export default function opening(): Piece { - const c = readCase('activsg70k'); +export default async function (): Promise { + const c = await readCase('activsg70k'); + const { grid, lon, lat, kv, lineLevel: level } = c; const [n, m] = [c.header.buses, c.header.branches]; - const ends = c.array('branch.ends', Uint32Array); - const kv = c.array('bus.kv', Float32Array); const number = c.array('bus.number', Uint32Array); const load = c.array('bus.load', Float32Array); const generation = c.array('bus.generation', Float32Array); - const net = { - buses: n, - from: Uint32Array.from({ length: m }, (_, e) => ends[2 * e]!), - to: Uint32Array.from({ length: m }, (_, e) => ends[2 * e + 1]!), - reactance: c.array('branch.reactance', Float32Array), - }; + const reactance = c.array('branch.reactance', Float32Array); + const net = { buses: n, from: c.from, to: c.to, reactance }; const [i, j] = [number.indexOf(59757), number.indexOf(59774)]; const line = net.from.findIndex( (a, e) => (a === i && net.to[e] === j) || (a === j && net.to[e] === i), ); - // Injections on a 100 MVA base, so the angles come out in radians. - const theta = angles( - net, - Float64Array.from({ length: n }, (_, b) => (generation[b]! - load[b]!) / 100), - ); - const shift = shifts(net, theta, line); - const mw = (e: number, angle: Float64Array) => - ((angle[net.from[e]!]! - angle[net.to[e]!]!) * 100) / - Math.max(Math.abs(net.reactance[e]!), 1e-4); - // What each branch picks up once the line is open, in MW, compressed so a few MW still shows. - const gain = Float32Array.from({ length: m }, (_, e) => { - const before = mw(e, theta); - return Math.asinh((Math.abs(before + mw(e, shift)) - Math.abs(before)) / 4); - }); - const level = Float32Array.from({ length: m }, (_, e) => - levelOf(Math.max(kv[net.from[e]!]!, kv[net.to[e]!]!)), + // Injections in per unit, so the angles come out in radians. + const injection = Float64Array.from({ length: n }, (_, b) => (generation[b]! - load[b]!) / BASE); + const theta = angles(net, injection); + const moved = flows(net, shifts(net, theta, line)); + // Each branch's change in flow magnitude once the line opens, as asinh(ΔMW / 4), so a change of a + // few MW still shows. + const gain = Float32Array.from(flows(net, theta), (before, e) => + Math.asinh((Math.abs(before + moved[e]!) - Math.abs(before)) / 4), ); const hz = frequency(); - // A pole down to the ground from one bus of 345 kV and up in each cell of 3/4 degree, so the - // height reads without a curtain of them. + // Height as a fraction of the earth's radius: zero at deviations of -16 mHz and below, rising + // with frequency, with the highest peaks compressed by asinh. + const height = Float32Array.from(hz, (f) => 0.022 * Math.asinh(Math.max(0, f + 0.016) / 0.03)); + // A pole to the ground from one bus of 345 kV or more in each 0.75° cell, so the heights are + // readable without crowding the view with poles. const cells = new Set(); const pole = Float32Array.from({ length: n }, (_, b) => { - const cell = `${Math.floor(c.positions[2 * b]! / 0.75)} ${Math.floor(c.positions[2 * b + 1]! / 0.75)}`; + const cell = `${Math.floor(lon[b]! / 0.75)} ${Math.floor(lat[b]! / 0.75)}`; if (kv[b]! < 345 || cells.has(cell)) return 0; cells.add(cell); return 1; }); - const data = Patch.to('grid') - .replace( - 'Bus', - { count: n }, - { - lon: Float64Array.from({ length: n }, (_, b) => c.positions[2 * b]!), - lat: Float64Array.from({ length: n }, (_, b) => c.positions[2 * b + 1]!), - kv, - hz, - // A fraction of the earth's radius: on the ground at the lowest frequencies, rising with - // the frequency, the highest peaks compressed. - height: Float32Array.from(hz, (f) => 0.022 * Math.asinh(Math.max(0, f + 0.016) / 0.03)), - pole, - }, - ) - .replace( - 'Branch', - { count: m }, - { - from: reference('Bus', net.from), - to: reference('Bus', net.to), - gain, - change: Float32Array.from(gain, Math.abs), - level, - // Wider by voltage, and by how far past a quiet 10 MW the branch's flow changed. - width: Float32Array.from( - gain, - (g, e) => 0.45 + 0.12 * level[e]! + 0.3 * Math.max(0, Math.min(Math.abs(g), 6) - 1.7), - ), - open: Float32Array.from({ length: m }, (_, e) => (e === line ? 1 : 0)), - }, - ); - const opened = (name: string, width: number, color: string) => ({ - edges: { - of: ['grid', 'Branch'] as [string, string], - ends: ['from', 'to'] as [string, string], - name, - width, - color, - visible: { op: 'eq' as const, args: ['open', 1] as [string, number] }, - pick: false, - }, + // Width grows with voltage level, and with how far the flow change exceeds about 10 MW. + const width = Float32Array.from( + gain, + (g, e) => 0.45 + 0.12 * level[e]! + 0.3 * Math.max(0, Math.min(Math.abs(g), 6) - 1.7), + ); + const open = Float32Array.from({ length: m }, (_, e) => (e === line ? 1 : 0)); + grid + .set('Bus', { hz, height, pole }) + .set('Line', { gain, change: Float32Array.from(gain, Math.abs), width, open }); + + const usa = new URL( + '../../crates/latkit-cli/examples/branch-opening/states.geojson', + import.meta.url, + ); + const maps = new URL('../../npm/examples/public/maps/', import.meta.url); + const coastline = fs.readFileSync(new URL('coastline.geojson.gz', maps)); + const countries = fs.readFileSync(new URL('countries.geojson.gz', maps)); + const states = await Data.read(fs.readFileSync(usa), { name: 'states' }); + const coast = await Data.read(coastline, { name: 'coast' }); + const borders = await Data.read(countries, { name: 'borders' }); + + const opened = (name: string, width: number, color: string): Mark => ({ + edges: grid.table('Line'), + ends: ['from', 'to'], + name, + width, + color, + visible: field('open').eq(1), + pick: false, }); - // Map lines are a light steel at low alpha, so they lighten whatever lies under them. + // Map lines are a light steel blue at low alpha, so they lighten whatever is beneath them. const map = (alpha: string) => `#7fa6cc${alpha}`; - // The distribution web under 115 kV that the opening leaves be, dim and thin under the rest. - const quiet: ExprInput = { - op: 'and', - args: [ - { op: 'le', args: ['level', 1] }, - { op: 'lt', args: ['change', 1.7] }, - ], - }; + // Branches below 230 kV whose flow changes by less than about 10 MW, drawn dim and thin beneath + // the rest. + const quiet = field('level').le(1).and(field('change').lt(1.7)); + const closed = field('open').eq(0); return { - name: 'opening', - alt: 'The ACTIVSg70k grid on the globe as one 765 kV line opens: buses raised by their frequency, branches that pick up its power glow amber, those it leaves glow cyan', - size: [2400, 1000], - data: [ - data, - { name: 'states', file: STATES }, - { name: 'coast', file: worldMap('coastline') }, - { name: 'borders', file: worldMap('countries') }, - ], - // Low over the Gulf, looking north, the horizon a thin arc above the Great Lakes. - camera: { center: [-85.5, 38.6], zoom: 5.1, pitch: 56, bearing: -6 }, scene: { space: { earth: { projection: 'globe', - surface: '#111a24', graticule: true, atmosphere: true, - // Dusk at the equinox: the sun sets over the grid's west, night falls on the Atlantic. + // Dusk at the September equinox. The sun is setting over the western grid, and it is + // night over the Atlantic. daylight: Date.UTC(2026, 8, 22, 22, 20) / 1000, }, }, ease: 0, - // The docs page's own ground, air a muted steel rather than the event's cyan, a faint - // graticule, and a night that keeps the grid readable. + // The docs page's background, a muted steel atmosphere rather than the cyan of the flow + // changes, a faint graticule, and a night shade light enough to keep the grid readable. theme: { - ...THEME, background: '#0f141a', - atmosphere: '#6a93c052', grid: map('12'), - night: 0.35, + earth: { ground: '#111a24', atmosphere: '#6a93c052', night: 0.35 }, }, marks: [ - { lines: { of: ['states', 'polygons'], width: 0.5, color: map('1f') } }, - { lines: { of: ['borders', 'lines'], width: 0.6, color: map('33') } }, - { lines: { of: ['coast', 'lines'], width: 0.7, color: map('4d') } }, + { lines: states.table('polygons'), width: 0.5, color: map('1f') }, + { lines: borders.table('lines'), width: 0.6, color: map('33') }, + { lines: coast.table('lines'), width: 0.7, color: map('4d') }, + // The buses draw nothing: they raise the branches' ends to their heights. { - vertices: { - of: ['grid', 'Bus'], - position: ['lon', 'lat'], - elevation: 'height', - radius: 0, - color: '#00000000', - pick: false, - }, + vertices: grid.table('Bus'), + position: ['$lon', '$lat'], + elevation: '$height', + radius: 0, + color: '#00000000', + pick: false, }, { - edges: { - of: ['grid', 'Bus'], - poles: true, - visible: { op: 'eq', args: ['pole', 1] }, - width: 0.35, - color: '#a9b4bf33', - pick: false, - }, + edges: grid.table('Bus'), + poles: true, + visible: field('pole').eq(1), + width: 0.35, + color: '#a9b4bf33', + pick: false, }, { - edges: { - of: ['grid', 'Branch'], - ends: ['from', 'to'], - name: 'web', - width: 0.3, - color: '#39414b', - visible: { op: 'and', args: [{ op: 'eq', args: ['open', 0] }, quiet] }, - pick: false, - }, + edges: grid.table('Line'), + ends: ['from', 'to'], + name: 'web', + width: 0.3, + color: '#39414b', + visible: closed.and(quiet), + pick: false, }, { - edges: { - of: ['grid', 'Branch'], - ends: ['from', 'to'], - priority: 'change', - width: 'width', - color: { - // The backbone a quiet grey within 10 MW either way; past it, cyan and amber of equal brightness. - colormap: { - kind: 'diverging', - colors: [ - [0, '#a8e6ff'], - [0.14, '#4cc4f0'], - [0.27, '#3a8db5'], - [0.36, '#6c7784'], - [0.64, '#6c7784'], - [0.73, '#b07a3a'], - [0.86, '#f2a13a'], - [1, '#ffd27a'], - ], - }, - value: 'gain', - domain: [-6, 6], - }, - visible: { - op: 'and', - args: [ - { op: 'eq', args: ['open', 0] }, - { op: 'not', args: [quiet] }, + edges: grid.table('Line'), + ends: ['from', 'to'], + priority: '$change', + width: '$width', + // Grey for changes within about 10 MW either way. Beyond that, cyan for less flow and + // amber for more, at equal brightness. + color: field('gain') + .colormap({ + kind: 'diverging', + colors: [ + [0, '#a8e6ff'], + [0.14, '#4cc4f0'], + [0.27, '#3a8db5'], + [0.36, '#6c7784'], + [0.64, '#6c7784'], + [0.73, '#b07a3a'], + [0.86, '#f2a13a'], + [1, '#ffd27a'], ], - }, - pick: false, - }, + }) + .domain(-6, 6), + visible: closed.and(quiet.not()), + pick: false, }, - // The opened line: a faint red halo, red, and a bright core, the brightest thing drawn. + // The opened line as a faint red halo, a red line, and a pale core. It is the brightest + // thing drawn. opened('halo', 10, '#ff2e4d2e'), opened('opened', 4.4, '#ff2e4d'), opened('core', 1.3, '#ffd6dd'), ], }, + // Low over the Gulf, looking north, the horizon a thin arc above the Great Lakes. + camera: { center: [-85.5, 38.6], zoom: 5.1, pitch: 56, bearing: -6 }, + size: [2400, 1000], }; } diff --git a/docs/gallery/record.ts b/docs/gallery/record.ts index eb13277a..e08ec7ef 100644 --- a/docs/gallery/record.ts +++ b/docs/gallery/record.ts @@ -1,8 +1,9 @@ /** - * Writes `opening.frequency.gz`, what the line opening's picture lifts each bus by: its frequency - * deviation from 60 Hz at one instant of the branch-opening study's recording - * (`crates/latkit-cli/examples/branch-opening`), in ACTIVSg70k's bus order, as 16-bit integers of - * 10 µHz. The recording is 168 MB and stays out of the repository; this instant is 140 kB. + * Writes `opening.frequency.gz`, which sets each bus's height in the line opening picture. It holds + * each bus's frequency deviation from 60 Hz at one instant of the branch-opening study's recording + * (`crates/latkit-cli/examples/branch-opening`), in ACTIVSg70k's bus order, as 16-bit integers in + * units of 10 µHz. The recording is 168 MB, too large for the repository. One instant is 140 kB. + * The first frame at or after `seconds`, 2.48 by default, is the instant the picture shows. * * node docs/gallery/record.ts [seconds] [study] */ @@ -10,7 +11,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import { gzipSync } from 'node:zlib'; -import { readCase } from './grids.ts'; +import { inflate, parseCase } from '../../npm/examples/src/pages/grids/format.ts'; interface Metadata { readonly bus_ids: readonly number[]; @@ -18,7 +19,7 @@ interface Metadata { } const root = fileURLToPath(new URL('../..', import.meta.url)); -const [seconds = '2.5', study = path.join(root, 'target/branch-opening')] = process.argv.slice(2); +const [seconds = '2.48', study = path.join(root, 'target/branch-opening')] = process.argv.slice(2); const run = path.join(study, 'runs/little-rock-60hz'); const meta = JSON.parse( fs.readFileSync(path.join(run, 'frequency_metadata.json'), 'utf8'), @@ -31,7 +32,8 @@ fs.readSync(file, bytes, 0, bytes.length, frame * bytes.length); fs.closeSync(file); const hz = new Float32Array(bytes.buffer, bytes.byteOffset, count); -const c = readCase('activsg70k'); +const grid = path.join(root, 'npm/examples/public/grids/activsg70k.bin'); +const c = parseCase('activsg70k', await inflate(fs.readFileSync(grid))); const number = c.array('bus.number', Uint32Array); const row = new Map(meta.bus_ids.map((id, i) => [id, i])); const out = Int16Array.from(number, (id) => { diff --git a/docs/gallery/render.ts b/docs/gallery/render.ts index 025d2520..44e0d8c3 100644 --- a/docs/gallery/render.ts +++ b/docs/gallery/render.ts @@ -1,17 +1,34 @@ /** - * The documentation's gallery, drawn by latkit. Each piece writes its datasets with `Patch`, as - * latkit's bytes, and its scene as a typed `Scene`; the `latkit` command line draws it into - * `docs/images/NAME.png`, with nothing done to the picture after. + * Draws the documentation's gallery with latkit. Each piece's default export builds its datasets + * with `Data` and returns its picture, or a promise of it: a scene and how to look at it. The + * `latkit` CLI draws each piece to `docs/images/NAME.png` from the datasets the scene uses. The + * pictures are not edited afterwards. * * cargo build --locked -p latkit-cli --profile cli - * cargo xtask wasm # the package `latkit/core` comes from + * cargo xtask wasm # builds the `latkit` package the pieces import * node docs/gallery/render.ts [name...] */ import { execFileSync } from 'node:child_process'; import fs from 'node:fs'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; -import { SCALE, SIZE, type Piece } from './look.ts'; +import { Dataset, Table, load, type Camera, type Fit, type Scene, type Source } from 'latkit'; +import { SIZE } from './look.ts'; + +/** What a piece returns: its scene, and how the view looks at it. */ +export interface Picture { + readonly scene: Scene; + /** The camera. Without one, the view frames `fit`. */ + readonly camera?: Camera; + /** What a view without a camera frames. Defaults to everything drawn. */ + readonly fit?: Fit; + /** Elapsed seconds of effects, which sets where flowing comets are. */ + readonly elapsed?: number; + /** The axis coordinate shown, at which sampled fields are read. */ + readonly at?: number; + /** Picture size in pixels, if not `SIZE`. The hero is a wide banner. */ + readonly size?: readonly [number, number]; +} /** Every piece, in the gallery's order. */ export const PIECES = [ @@ -30,6 +47,9 @@ export const PIECES = [ 'tides', ] as const; +/** Pixels per CSS pixel. */ +const SCALE = 2; + const root = fileURLToPath(new URL('../..', import.meta.url)); const work = path.join(root, 'target/gallery'); const cli = process.env.LATKIT_CLI ?? path.join(root, 'target/cli/latkit'); @@ -38,16 +58,29 @@ const fonts = ['Inter-Regular.ttf', 'Inter-SemiBold.ttf'].map((f) => ); async function render(name: string): Promise { - const { default: make } = (await import(`./${name}.ts`)) as { default: () => Piece }; - const piece = make(); - const file = (suffix: string) => path.join(work, `${piece.name}.${suffix}`); - fs.writeFileSync(file('scene.json'), JSON.stringify(piece.scene)); - const data = piece.data.flatMap((d, i) => { - if ('file' in d) return ['--data', `${d.name}=${d.file}`]; - fs.writeFileSync(file(`${i}.bin`), d.toBytes()); - return ['--data', file(`${i}.bin`)]; + const { default: draw } = (await import(`./${name}.ts`)) as { + default: () => Picture | Promise; + }; + const picture = await draw(); + const file = (suffix: string) => path.join(work, `${name}.${suffix}`); + // The datasets the scene's tables belong to, collected the way a view collects them. + const datasets = new Set(); + const scene = JSON.stringify( + picture.scene, + function (this: Record, key, value: unknown) { + const held = this[key]; + if (held instanceof Table) datasets.add(held.source); + return value; + }, + ); + fs.writeFileSync(file('scene.json'), scene); + const data = [...datasets].flatMap((dataset, i) => { + if (!(dataset instanceof Dataset)) throw new Error(`${name}: the CLI draws no tiles`); + fs.writeFileSync(file(`${i}.latk`), dataset.toBytes()); + dataset.destroy(); + return ['--data', file(`${i}.latk`)]; }); - const output = path.join(root, 'docs/images', `${piece.name}.png`); + const output = path.join(root, 'docs/images', `${name}.png`); const args = [ 'render', file('scene.json'), @@ -55,18 +88,18 @@ async function render(name: string): Promise { '--output', output, '--size', - (piece.size ?? SIZE).join('x'), + (picture.size ?? SIZE).join('x'), '--scale', String(SCALE), ...fonts.flatMap((f) => ['--font', f]), ]; - if (piece.camera) { - fs.writeFileSync(file('camera.json'), JSON.stringify(piece.camera)); + if (picture.camera) { + fs.writeFileSync(file('camera.json'), JSON.stringify(picture.camera)); args.push('--camera', file('camera.json')); } - if (piece.fit) args.push('--fit', JSON.stringify(piece.fit)); - if (piece.elapsed !== undefined) args.push('--elapsed', String(piece.elapsed)); - if (piece.at !== undefined) args.push('--at', String(piece.at)); + if (picture.fit) args.push('--fit', JSON.stringify(picture.fit)); + if (picture.elapsed !== undefined) args.push('--elapsed', String(picture.elapsed)); + if (picture.at !== undefined) args.push('--at', String(picture.at)); execFileSync(cli, args, { stdio: 'inherit', timeout: 180_000 }); console.log(`${path.relative(root, output)} ${(fs.statSync(output).size / 1024).toFixed(0)} KB`); } @@ -75,5 +108,8 @@ const only = process.argv.slice(2); const unknown = only.filter((name) => !(PIECES as readonly string[]).includes(name)); if (unknown.length) throw new Error(`no piece ${unknown.join(', ')}; there are ${PIECES.join(', ')}`); +await load({ + wasm: new WebAssembly.Module(fs.readFileSync(new URL(import.meta.resolve('latkit/wasm')))), +}); fs.mkdirSync(work, { recursive: true }); for (const name of only.length ? only : PIECES) await render(name); diff --git a/docs/gallery/terrain.ts b/docs/gallery/terrain.ts index aecdb5b3..a8882f7c 100644 --- a/docs/gallery/terrain.ts +++ b/docs/gallery/terrain.ts @@ -1,8 +1,8 @@ -/** Terrain: ridged mountains as lit faces between a grid of vertices, colored by height. */ -import { Patch, reference } from 'latkit/core'; -import { THEME, floats, rows, type Piece } from './look.ts'; +/** Terrain: ridged mountains as lit triangles over a grid of vertices, colored by height. */ +import { Data, field, reference } from 'latkit'; +import type { Picture } from './render.ts'; -/** Height at `x`, `y`: ridges of several directions and scales, sharpened, then summed. */ +/** The height at `x`, `y`: sharpened ridges at six directions and scales, summed. */ function height(x: number, y: number): number { let h = 0; let amplitude = 1; @@ -16,7 +16,7 @@ function height(x: number, y: number): number { h += amplitude * ridge * ridge; amplitude *= 0.48; } - // A valley across the middle, and the land falling away toward the edges. + // Cut a valley across the middle, and lower the land toward the edges. return ( h * (0.35 + 0.65 * Math.min(1, Math.abs(y + 0.25 * x) / 260)) * @@ -24,70 +24,59 @@ function height(x: number, y: number): number { ); } -export default function terrain(): Piece { +export default function (): Picture { const [columns, lines] = [300, 190]; const [w, h] = [1400, 880]; const n = columns * lines; - const px = (i: number) => ((i % columns) / (columns - 1) - 0.5) * w; - const py = (i: number) => (Math.floor(i / columns) / (lines - 1) - 0.5) * h; - const z = floats(n, (i) => 150 * height(px(i), py(i))); + // Vertex i's place on the grid, row by row. + const xOf = (i: number) => ((i % columns) / (columns - 1) - 0.5) * w; + const yOf = (i: number) => (Math.floor(i / columns) / (lines - 1) - 0.5) * h; + const x = Float32Array.from({ length: n }, (_, i) => xOf(i)); + const y = Float32Array.from({ length: n }, (_, i) => yOf(i)); + const z = Float32Array.from({ length: n }, (_, i) => 150 * height(xOf(i), yOf(i))); const cells = (columns - 1) * (lines - 1); - // Two triangles a cell: its corners as rows of the grid. + // Two triangles per grid cell. Returns corner `k` (0 to 5) of the cell's triangles as a vertex + // row. const corner = (cell: number, k: number) => { const c = cell % (columns - 1); const r = Math.floor(cell / (columns - 1)); const at = r * columns + c; return [at, at + 1, at + columns, at + 1, at + columns + 1, at + columns][k]!; }; - const data = Patch.to('land') - .replace('Vertex', { count: n }, { x: floats(n, px), y: floats(n, py), z }) - .replace( - 'Face', - { count: 2 * cells }, - { - a: reference( - 'Vertex', - rows(2 * cells, (f) => corner(f >> 1, (f & 1) * 3)), - ), - b: reference( - 'Vertex', - rows(2 * cells, (f) => corner(f >> 1, (f & 1) * 3 + 1)), - ), - c: reference( - 'Vertex', - rows(2 * cells, (f) => corner(f >> 1, (f & 1) * 3 + 2)), - ), - }, + // Corner `k` (0 to 2) of every face. Faces 2c and 2c + 1 are cell c's two triangles. + const corners = (k: number) => { + const vertex = Uint32Array.from({ length: 2 * cells }, (_, f) => + corner(f >> 1, (f & 1) * 3 + k), ); + return reference('Vertex', vertex); + }; + const land = new Data('land') + .replace('Vertex', { x, y, z }) + .replace('Face', { a: corners(0), b: corners(1), c: corners(2) }); return { - name: 'terrain', - alt: 'Ridged mountains of 112,000 lit triangles, from deep teal valleys to pale summits, seen from above at an angle', - data: [data], - camera: { center: [0, -20, 30], zoom: -0.9, pitch: 56, bearing: 22 }, scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14' }, marks: [ + // The vertices draw nothing themselves: they place the faces' corners, and each face + // blends its corners' colors. { - vertices: { - of: ['land', 'Vertex'], - position: ['x', 'y'], - elevation: 'z', - radius: 0, - pick: false, - color: { - colormap: { - kind: 'sequential', - colors: ['#0e2a3f', '#1b5e6b', '#3f8f6e', '#a7a463', '#d9c9a3', '#f4efe6'], - }, - value: 'z', - domain: [0, 120], - }, - }, + vertices: land.table('Vertex'), + position: ['$x', '$y'], + elevation: '$z', + radius: 0, + pick: false, + color: field('z') + .colormap({ + kind: 'sequential', + colors: ['#0e2a3f', '#1b5e6b', '#3f8f6e', '#a7a463', '#d9c9a3', '#f4efe6'], + }) + .domain(0, 120), }, - { faces: { of: ['land', 'Face'], corners: ['a', 'b', 'c'], pick: false } }, + { faces: land.table('Face'), corners: ['a', 'b', 'c'], pick: false }, ], }, + camera: { center: [0, -20, 30], zoom: -0.9, pitch: 56, bearing: 22 }, }; } diff --git a/docs/gallery/texas.ts b/docs/gallery/texas.ts index b9505d82..21227149 100644 --- a/docs/gallery/texas.ts +++ b/docs/gallery/texas.ts @@ -1,51 +1,66 @@ -/** Texas: ACTIVSg2000's generators sized by output, its highest-voltage lines carrying comets. */ -import { THEME, type Piece } from './look.ts'; +/** Texas: ACTIVSg2000's generators sized by output, with comets on its lines of 345 kV and up. */ +import fs from 'node:fs'; +import { Data, field, units, when } from 'latkit'; import { readCase } from './grids.ts'; -import { LEVELS, STATES, WIDTHS, states } from './grid.ts'; +import type { Picture } from './render.ts'; -export default function texas(): Piece { - const c = readCase('activsg2000'); +export default async function (): Promise { + const { grid, array } = await readCase('activsg2000'); + const usa = new URL( + '../../crates/latkit-cli/examples/branch-opening/states.geojson', + import.meta.url, + ); + const states = await Data.read(fs.readFileSync(usa), { name: 'states' }); + // The largest generators are named: those making at least as much as the tenth largest. + const generation = Array.from(array('bus.generation', Float32Array)).sort((p, q) => q - p); + const named = generation[9] ?? 0; + const level = field('level'); return { - name: 'texas', - alt: 'Two thousand buses of a synthetic Texas grid: generators as amber discs sized by output, comets along its highest-voltage lines', - data: [c.patch, { name: 'states', file: STATES }], - camera: { center: [-98.7, 31.25], zoom: 5.6, pitch: 32, bearing: 0 }, - elapsed: 2, scene: { - space: { earth: { projection: 'mercator', graticule: false, surface: THEME.background } }, + space: { earth: { graticule: false } }, ease: 0, + // The dark theme, with a darker background and labels without halos. theme: { - ...THEME, - flow: { color: '#fff4dcd0', spacing: 0.9, length: 0.35, width: 3, units: true }, + background: '#0b0f14', + text: { color: '#c3ccd6', font: 'sans-serif', size: 12, halo: 0 }, + earth: { ground: '#0b0f14' }, + flow: { color: '#fff4dcd0', spacing: units(0.9), length: units(0.35), width: 3 }, }, marks: [ - ...states('#2c3e54'), + // The states, filled slightly lighter than the ground, with faint outlines. + { areas: states.table('polygons'), color: '#0f1720', pick: false }, + { lines: states.table('polygons'), width: 0.6, color: '#2c3e54' }, { - edges: { - of: ['grid', 'Line'], - ends: ['from', 'to'], - priority: 'level', - width: WIDTHS, - color: { colormap: LEVELS, value: 'level', domain: [0, 5] }, - flow: { when: [[{ op: 'ge', args: ['level', 3] }, 0.6]], otherwise: 0 }, - }, + edges: grid.table('Line'), + ends: ['from', 'to'], + priority: level, + // Width grows with the level's square, so the high-voltage backbone stands out. + width: level.mul(level).scale(0.3, 2.3).domain(0, 25), + // The voltage levels, from below 100 kV to 765 kV. Higher levels are hotter colors. + color: level + .colormap({ + kind: 'sequential', + colors: ['#1c2e44', '#22507a', '#2d80b3', '#58bde6', '#ffcf73', '#fff4dc'], + }) + .domain(0, 5), + flow: when(level.ge(3)).then(0.6).otherwise(0), }, { - vertices: { - of: ['grid', 'Bus'], - position: ['lon', 'lat'], - radius: { - scale: [0, 11], - value: { op: 'sqrt', args: ['generation'] }, - domain: [0, 45], - }, - color: '#ffb000', - priority: 'generation', - label: { text: 'name', size: 11, color: '#e6d7b8', priority: 'generation', max: 10 }, - visible: { op: 'gt', args: ['generation', 0] }, + vertices: grid.table('Bus'), + position: ['$lon', '$lat'], + radius: field('generation').sqrt().scale(0, 11).domain(0, 45), + color: '#ffb000', + priority: '$generation', + label: { + text: when(field('generation').ge(named)).then('$name').otherwise(null), + size: 11, + color: '#e6d7b8', }, + visible: field('generation').gt(0), }, ], }, + camera: { center: [-98.7, 31.25], zoom: 5.6, pitch: 32, bearing: 0 }, + elapsed: 2, }; } diff --git a/docs/gallery/tides.ts b/docs/gallery/tides.ts index c6d27f1c..2e36810a 100644 --- a/docs/gallery/tides.ts +++ b/docs/gallery/tides.ts @@ -1,8 +1,9 @@ -/** Tides: closed contours swelling outward, each deformed by harmonics that drift with its level. */ -import { Patch } from 'latkit/core'; -import { TAU, THEME, framed, type Piece } from './look.ts'; +/** Tides: nested closed contours, each rippled by harmonics whose phases shift with its level. */ +import { Data, field } from 'latkit'; +import { framed } from './look.ts'; +import type { Picture } from './render.ts'; -export default function tides(): Piece { +export default function (): Picture { const contours = 110; const steps = 600; const geometry: [number, number][][] = []; @@ -12,7 +13,7 @@ export default function tides(): Piece { const r = 50 + 270 * l; const path: [number, number][] = []; for (let s = 0; s <= steps; s++) { - const a = (s / steps) * TAU; + const a = (s / steps) * 2 * Math.PI; const ripple = 1 + 0.13 * Math.sin(3 * a + 5 * l) + @@ -23,40 +24,25 @@ export default function tides(): Piece { geometry.push(path); level.push(l); } - const data = Patch.to('tides').replace('Contour', { count: contours }, { geometry, level }); + const tides = new Data('tides').replace('Contour', { geometry, level }); return { - name: 'tides', - alt: 'One hundred and ten closed contours swelling outward, from deep blue at the center to sand at the rim', - data: [data], - fit: framed([-640, -320, 640, 320]), scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14' }, marks: [ { - lines: { - of: ['tides', 'Contour'], - width: 1.1, - color: { - colormap: { - kind: 'sequential', - colors: [ - '#0b2545', - '#13607a', - '#2a9d8f', - '#8fc69a', - '#e9c46a', - '#f4a261', - '#e76f51', - ], - }, - value: 'level', - domain: [0, 1], - }, - }, + lines: tides.table('Contour'), + width: 1.1, + color: field('level') + .colormap({ + kind: 'sequential', + colors: ['#0b2545', '#13607a', '#2a9d8f', '#8fc69a', '#e9c46a', '#f4a261', '#e76f51'], + }) + .domain(0, 1), }, ], }, + fit: framed([-640, -320, 640, 320]), }; } diff --git a/docs/gallery/torus.ts b/docs/gallery/torus.ts index ebd87f42..c161bb0e 100644 --- a/docs/gallery/torus.ts +++ b/docs/gallery/torus.ts @@ -1,10 +1,11 @@ /** Torus silk: filaments twisting around a (2, 3) torus knot, drawn as edges in three dimensions. */ -import { Patch, reference } from 'latkit/core'; -import { TAU, THEME, floats, rows, type Piece } from './look.ts'; +import { Data, field, reference } from 'latkit'; +import type { Picture } from './render.ts'; type Vec = [number, number, number]; -/** The knot's centerline at `t`: around the torus twice while it winds through three times. */ +/** The knot's centerline at `t`. It goes around the torus twice while winding through the hole + * three times. */ function knot(t: number): Vec { const r = 100 + 42 * Math.cos(3 * t); return [r * Math.cos(2 * t), r * Math.sin(2 * t), 42 * Math.sin(3 * t)]; @@ -21,7 +22,7 @@ const cross = ([a, b, c]: Vec, [d, e, f]: Vec): Vec => [ a * e - b * d, ]; -export default function torus(): Piece { +export default function (): Picture { const filaments = 64; const samples = 900; const tube = 15; @@ -32,8 +33,8 @@ export default function torus(): Piece { const tone = new Float32Array(n); const h = 1e-3; for (let s = 0; s < samples; s++) { - const t = (s / samples) * TAU; - // The centerline's frame: its tangent, the way it bends, and their cross. + const t = (s / samples) * 2 * Math.PI; + // The centerline's Frenet frame: tangent, normal (the direction it bends), and binormal. const tangent = unit(sub(knot(t + h), knot(t - h))); const ahead = unit(sub(knot(t + 2 * h), knot(t))); const behind = unit(sub(knot(t), knot(t - 2 * h))); @@ -42,7 +43,7 @@ export default function torus(): Piece { const c = knot(t); for (let f = 0; f < filaments; f++) { // Each filament turns around the tube three times along the knot, so it closes on itself. - const phase = (f / filaments) * TAU + 3 * t; + const phase = (f / filaments) * 2 * Math.PI + 3 * t; const i = f * samples + s; x[i] = c[0] + tube * (Math.cos(phase) * normal[0] + Math.sin(phase) * binormal[0]); y[i] = c[1] + tube * (Math.cos(phase) * normal[1] + Math.sin(phase) * binormal[1]); @@ -50,58 +51,45 @@ export default function torus(): Piece { tone[i] = 0.5 + 0.5 * Math.sin(phase + t); } } - const next = rows(n, (i) => Math.floor(i / samples) * samples + (((i % samples) + 1) % samples)); - const data = Patch.to('silk') - .replace('Point', { count: n }, { x, y, z, tone }) - .replace( - 'Thread', - { count: n }, - { - from: reference( - 'Point', - rows(n, (i) => i), - ), - to: reference('Point', next), - tone: floats(n, (i) => tone[i]!), - }, - ); + // Point i's thread runs to the next point along its filament, and the last back to the first. + const next = Uint32Array.from( + { length: n }, + (_, i) => Math.floor(i / samples) * samples + (((i % samples) + 1) % samples), + ); + const silk = new Data('silk').replace('Point', { x, y, z, tone }).replace('Thread', { + from: reference('Point', { count: n }), + to: reference('Point', next), + tone, + }); return { - name: 'torus', - alt: 'Sixty-four filaments in cyan, amber, and coral, twisting around a three-dimensional torus knot', - data: [data], - camera: { center: [0, 14, 60], zoom: 0.62, pitch: 52, bearing: -18 }, scene: { space: { plane: { grid: 'off' } }, ease: 0, - theme: THEME, + theme: { background: '#0b0f14' }, marks: [ + // The points draw nothing: they place the threads' ends in three dimensions. { - vertices: { - of: ['silk', 'Point'], - position: ['x', 'y'], - elevation: 'z', - radius: 0, - color: '#00000000', - pick: false, - }, + vertices: silk.table('Point'), + position: ['$x', '$y'], + elevation: '$z', + radius: 0, + color: '#00000000', + pick: false, }, { - edges: { - of: ['silk', 'Thread'], - ends: ['from', 'to'], - width: 0.7, - pick: false, - color: { - colormap: { - kind: 'sequential', - colors: ['#2bb6e8', '#79e3d6', '#ffd39a', '#ff8f5e', '#ff5c7a'], - }, - value: 'tone', - domain: [0, 1], - }, - }, + edges: silk.table('Thread'), + ends: ['from', 'to'], + width: 0.7, + pick: false, + color: field('tone') + .colormap({ + kind: 'sequential', + colors: ['#2bb6e8', '#79e3d6', '#ffd39a', '#ff8f5e', '#ff5c7a'], + }) + .domain(0, 1), }, ], }, + camera: { center: [0, 14, 60], zoom: 0.62, pitch: 52, bearing: -18 }, }; } diff --git a/docs/guide.md b/docs/guide.md index 33cbddee..9a7b7040 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -1,283 +1,259 @@ # Guide -How latkit works: datasets and the patches that change them, scenes, views, and pictures, and -what it takes to ship them. +How to use latkit: write and change datasets, describe scenes, show them in views, and draw +pictures and video. The [API reference](reference/api.md) documents every method and option. ## Data -A dataset is a name and its tables. A table is rows, each with an id, and fields, a value per row. -Patches change datasets all at once: every change of a patch, or none. +A dataset is a name and a set of tables. A table has rows and fields. Each row has an id, and each +field has one value per row. ### Write a dataset ```ts -import { loadLatkit, Patch, reference, field } from 'latkit'; - -const { Data } = await loadLatkit(); -const grid = new Data('grid').apply( - new Patch() - .replace('Bus', [101, 102, 103], { - lon: Float64Array.of(-97.7, -97.6, -97.5), - lat: Float64Array.of(30.2, 30.3, 30.4), - name: ['North', 'Mid', null], - }) - .replace( - 'Branch', - { count: 2 }, - { - from: reference('Bus', [101, 102]), - to: reference('Bus', [102, 103]), - name: ['North-Mid', 'Mid-South'], - loading: [95, 70], - }, - ), -); +import { Data, load, reference } from 'latkit'; + +await load(); +const grid = new Data('grid') + .set('Bus', [101, 102, 103], { + lon: Float64Array.of(-97.7, -97.6, -97.5), + lat: Float64Array.of(30.2, 30.3, 30.4), + name: ['North', 'Mid', null], + }) + .set( + 'Branch', + { count: 2 }, + { + from: reference('Bus', [101, 102]), + to: reference('Bus', [102, 103]), + name: ['North-Mid', 'Mid-South'], + loading: [95, 70], + }, + ); ``` -- **Ids** are nonnegative safe integers (`0..2^53 - 1`) or texts, listed, or `{ count: n }` for rows `0` to `n - 1`. -- **Values** are a column per field: typed arrays of numbers (a `Uint8Array` holds booleans), or - arrays of numbers, booleans, texts, or geometries. `null` is missing. -- **Geometries** are `[x, y]` points, paths of points, parts (a polygon's rings, or a path's - pieces), or polygons. -- **References** name rows of another table by id: `reference(table, ids)`. - -| Change | Does | -| --------------------------------------- | -------------------------------------------------------------------- | -| `replace(table, ids, fields)` | replaces the table, or adds it | -| `set(table, ids, fields)` | writes fields for those rows: known ids change, others are added | -| `remove(table, ids)` | removes the rows | -| `axis(name, unit)` | names the axis sampled fields are recorded over | -| `append(at, table, fields, ids?)` | records a frame at each coordinate of `at`, of every row or of `ids` | -| `forget(before)` | forgets the frames before a coordinate | -| `frames(table, field, run, at, values)` | holds a run of a sampled field's entries, as a time tile does | +- Ids are nonnegative safe integers or strings, or `{ count: n }` for rows `0` to `n - 1`. +- Values are one column per field: typed arrays, or arrays of numbers, booleans, strings, or + geometries, with `null` for missing. A `Uint8Array` holds booleans. +- Geometries are `[x, y]` points, paths of points, parts (a polygon's rings), or polygons. +- References name rows of another table by id: `reference(table, ids)`. ### Change it -`apply` changes a dataset in place. A snapshot keeps a version in O(1), sharing its pages, so an -undo history costs only what changed. A view draws the version it was given until it is given -another: +`set` writes rows and adds any new ids. `replace` replaces a whole table, and `remove` removes rows. +Each change applies immediately, all or nothing, and every view of the dataset shows it on the next +frame. A [`Patch`](reference/api.md#patch) holds several changes that apply together, all or +nothing, such as an undo step or one message from a server. `snapshot()` saves the current version +in O(1). Snapshots share unchanged pages, so an undo history costs only the pages that changed: ```ts const before = grid.snapshot(); -grid.apply(new Patch().set('Bus', [101], { name: ['Northgate'] })); -view.setData(grid); // the view draws the change -view.setData(before); // and the version before it +grid.set('Bus', [101], { name: ['Northgate'] }); // drawn the next frame +grid.restore(before); // and undone +grid.apply(new Patch().set('Bus', [101], { name: ['Northgate'] }).remove('Branch', [1])); // together ``` ### Record over time -Sampled values are stored as `f32`, including values supplied as `Float64Array`; coordinates -remain `f64`. NaN means missing in sampled values and numeric query results. Static columns retain -their original numeric storage. A field's first frame makes it sampled. A view shows the axis coordinate `view.at`: each sampled -field reads its last frame at or before it. +A field becomes sampled when you append its first frame. A view shows the axis coordinate +`view.at`. Marks interpolate each sampled field between the frames on either side of `view.at`. A +field read with `field(name).step()` shows the last frame at or before it instead. Sampled values +are stored as `f32` for drawing, and axis coordinates as `f64`. ```ts -grid.apply(new Patch().axis('time', 's')); -grid.apply(new Patch().append(0.5, 'Bus', { vm: Float32Array.of(1.01, 0.99, 1.0) })); -grid.apply(new Patch().append([1, 1.5], 'Bus', { vm: Float32Array.of(1, 1, 1, 0.98, 0.97, 0.99) })); -grid.apply(new Patch().append(2, 'Bus', { vm: [1.02] }, [101])); // one row; the rest hold -grid.apply(new Patch().forget(1)); // a rolling window +grid.axis('time', 's'); +grid.append('Bus', 0.5, { vm: Float32Array.of(1.01, 0.99, 1.0) }); +grid.append('Bus', [1, 1.5], { vm: Float32Array.of(1, 1, 1, 0.98, 0.97, 0.99) }); +grid.append('Bus', 2, { vm: [1.02] }, { ids: [101] }); // one row; the rest hold +grid.forget(1); // a rolling window view.at = 1.2; ``` -Each sampled field keeps levels of detail as it records: at level `k`, each row's least and -greatest value over buckets of `8^k` frames. A line tracing a field across a plot reads the level -that gives each pixel column a bucket or two, so a long history draws as fast as a short one. +Each sampled field keeps levels of detail as it records. Level `k` holds each row's minimum and +maximum over buckets of `8^k` frames. A line that traces a field on a plot reads the level with one +or two buckets per pixel column, so a long history draws as fast as a short one. ### Fetch long histories in spans -A history too large to load whole is written as time tiles, and a view fetches only what it draws: +Write a history too large to load at once as time tiles. A view then fetches only what it draws: ```sh -latkit tile history.bin -o history/ +latkit tile history.latk -o history/ ``` ```ts -view.setTiles('grid', 'history/frames.json'); +view.sources = [new Tiles('grid', 'history/frames.json')]; // in place of the dataset named grid ``` -`tile` writes `frames.json`, `base.latk` (the dataset with every sampled field's frames left out), -and each field's history in spans of `--chunk` entries, at every level of detail (`Data::to_tiles` -in Rust). A line tracing a field fetches the spans its plot window draws, at the detail it draws -them, and a mark showing a field fetches the frames about `view.at`. A span still loading draws the -coarser span above it, as a map tile still loading draws the tile a zoom out. +A line that traces a field fetches the spans in its plot window, at the detail it draws. Other +marks fetch the frames around `view.at`. While a span loads, the view draws the coarser span that +covers it. In Rust, `Snapshot::to_tiles` writes the same files. ### Stream -Patches are bytes: `patch.toBytes()` writes one frame, and frames back to back stream. Each frame -says its length up front. The parser holds at most one frame of `limit` bytes (256 MiB by default), -in addition to the input chunk and patches you retain. Breaking iteration cancels the reader. A -server in any language writes the same frames ([format](#bytes)). +`patch.toBytes()` encodes a patch as one frame. Frames written back to back form a stream, which a +server in any language can write ([format](#bytes)). `Patch.stream` yields each patch as soon as +its last byte arrives: ```ts -for await (const patch of Patch.stream((await fetch('/feed')).body!)) { - grid.apply(patch); - view.setData(grid); -} +for await (const patch of Patch.stream((await fetch('/feed')).body!)) grid.apply(patch); ``` -A patch made with `Patch.to('grid')` names its dataset, and `apply` refuses it for another. +`new Patch('grid')` names its dataset, and applying it to another dataset fails. -### Load files +### Read files ```ts -const grid = await Data.load(json, 'data/grid.json'); // a JSON document, slices relative to it +const grid = await Data.read('data/grid.json'); // a JSON document, its slices relative to it const saved = await Data.read(bytes); // latkit's bytes, as data.toBytes() wrote them -const states = await Data.read(geojson, 'states'); // GeoJSON, gzip-compressed or not, in a worker +const states = await Data.read(geojson, { name: 'states' }); // GeoJSON, gzip-compressed or not ``` -GeoJSON becomes tables `points`, `lines`, and `polygons`, each with its geometry as field -`geometry` and its properties as fields. The JSON document's form is in [Formats](#json). +GeoJSON becomes the tables `points`, `lines`, and `polygons`. [Formats](#json) describes the JSON +document. `Patch.read` reads one patch from the same kinds of source. ### Read back -Reading needs no GPU. Queries run the expressions marks draw with: +Reading needs no GPU, and queries use the same expressions as marks: ```ts grid.get('Bus', 101, 'name'); // 'Northgate' const hot = grid.query({ of: 'Branch', filter: field('loading').gt(90), - sort: [['loading', 'descending']], - select: { name: 'name', loading: 'loading' }, + sort: [['$loading', 'descending']], + select: { name: '$name', loading: '$loading' }, limit: 20, }); // { table, ids, fields: { name: string[], loading: Float64Array }, total } ``` -`total` counts the rows kept before `limit`. `field('name').contains('north')` matches texts in -queries. `describe()` lists tables, rows, fields and their kinds, and the axis and its span. -`envelope(table, field, range, spans)` gives each row's least and greatest value of a sampled field -over equal spans of the axis, read from levels of detail, so a span costs a few reads however many -frames it covers. - -`Patch.fromBytes(bytes)` copies one frame; `patch.extend(next)` appends its operations. -`Uint8Array` means booleans in JavaScript patches; JSON binary slices with dtype `u8` mean numbers. -Use `Uint32Array` for unsigned numeric bytes in a JavaScript patch. For all method signatures, -see the [API reference](reference/api.md). +[`Dataset`](reference/api.md#dataset) lists every read, including `describe()` and `envelope()`. A +`Snapshot` has the same reads, for its own version. ## Scenes -A scene is plain data: a `space`, `marks` drawn in order, and optionally a `layout`, `theme`, -`controls`, `ease` (seconds changes ease over), and a `title` and `subtitle`. A view shows a scene; -`view.show(next)` replaces it, keeping every row's place in the transition. +A scene is plain data: a `space`, a list of `marks` drawn in order, and optional `layout`, `theme`, +`controls`, `ease` (how many seconds a change eases over), `title`, and `subtitle`. Set +`view.scene = next` to replace the scene. Each row eases from where it was drawn. ```ts const scene: Scene = { - space: 'mercator', + space: 'earth', marks: [ - { edges: { of: ['grid', 'Branch'], ends: ['from', 'to'], width: 1.5 } }, + { edges: grid.table('Branch'), ends: ['from', 'to'], width: 1.5 }, { - vertices: { - of: ['grid', 'Bus'], - position: ['lon', 'lat'], - radius: units('kv', { atLeast: 2 }), - color: field('vm').colormap('balance').domain(0.95, 1.05), - label: { text: 'name', priority: 'kv' }, - legend: 'Voltage (pu)', - }, + vertices: grid.table('Bus'), + position: ['$lon', '$lat'], + radius: units('$kv', { atLeast: 2 }), + color: field('vm').colormap('balance').domain(0.95, 1.05), + label: when(field('kv').ge(230)).then(field('name')).otherwise(null), + legend: 'Voltage (pu)', }, ], - theme: { base: 'light', vertexRadius: 4 }, + theme: { base: 'light', vertex: { radius: 4 } }, controls: 'inspect', }; ``` +The [scene types](reference/types.md#scene) document every option and its default. + ### Spaces -| Space | Positions | -| ----------------------- | ----------------------------------------------------------------- | -| `'plane'` | plane units with elevation; the camera turns and tilts | -| `'mercator'`, `'globe'` | longitude and latitude in degrees, on a map or a globe | -| `'plot'` | the axis coordinate across, values up | -| `'diagram'` | a flat sheet, y down, every length a unit; ports, groups, layouts | +| Space | Positions | +| ----------- | --------------------------------------------------------------------- | +| `'plane'` | plane units with elevation; the camera turns and tilts | +| `'earth'` | longitude and latitude in degrees, on a map or a globe | +| `'plot'` | the axis coordinate across, values up | +| `'diagram'` | a flat sheet, y down, sizes in position units; ports, groups, layouts | -Each takes options as `{ plane: { grid } }`, `{ diagram: { grid } }`, `{ earth: { projection, -surface, graticule, daylight, atmosphere } }`, or `{ plot: { x, y } }`. A grid is `'off'`, -`{ lines: { pitch } }`, or `{ dots: { pitch, snap } }`. +Write a space as an object to give it options: `{ earth: { projection: 'globe' } }`, +`{ plane: { grid: { lines: { pitch: 10 } } } }`, or a plot's `{ plot: { x, y } }` axes. ### Marks -Every mark names its table, `of: [dataset, table]`, and takes `color`, `visible`, `priority`, -`label`, `legend`, `shadow`, `name`, and `pick`, `select`, and `drag`. Unless `pick` says, rows -that are things are hovered, clicked, and probed (vertices, edges, faces, a plot's traces) and -geometry is not (areas, and lines along paths: coasts, borders, roads), so hovering probes none of -it; `pick: true` makes a map's states or cells answer the pointer. - -| Mark | Draws each row as | Its own options | -| ---------- | ------------------------------------------------------------------ | -------------------------------------------------------------------- | -| `vertices` | a marker at `position` (or the point field `geometry`) | `elevation`, `radius`, `size`, `marker`, `ring`, `ports`, `contains` | -| `edges` | a stroke between the rows two reference fields name | `ends`, `bends`, `route`, `arrows`, `width`, `dash`, `flow` | -| `lines` | a path field's stroke; on a plot, a sampled field traced over time | `geometry`, `trace`, `interpolation`, `width`, `dash`, `flow` | -| `areas` | the area a polygon field covers | `geometry` | -| `faces` | a triangle between three vertices | `corners` | - -A `flow` sends comets along a stroke. In a diagram, vertices are blocks with `ports`, a vertex mark -with `contains` draws groups around the rows that name it, and edges of a table without `ends` are -nets joining the ports that name them. The theme's `outline`, `corner`, `portSize`, and `bend` set -how blocks, groups, ports, and wires look, and `shadow: true` lifts blocks and groups off the sheet. +A mark's kind key holds the table it draws: `{ vertices: grid.table('Bus') }`. A view loads every +dataset its marks draw from. `JSON.stringify` writes a table as `[data, table]`, the form the +command line reads, where `data` names one of the view's `sources`. The pointer hovers and clicks +vertices, edges, faces, and a plot's traces. Set `pick: true` to make areas and lines respond too. + +| Mark | Draws each row as | +| ---------- | ------------------------------------------------------------------------ | +| `vertices` | a marker at `position` (or at the point field `geometry`) | +| `edges` | a stroke between the two rows its `ends` fields reference | +| `lines` | a stroke along a path field; on a plot, a sampled field traced over time | +| `areas` | the area a polygon field covers | +| `faces` | a triangle between three vertices | + +`flow` animates comets along a stroke. In a diagram, vertices are blocks with `ports`. A vertices +mark with `contains` draws each row as a group around the rows that reference it. An edges mark +without `ends` draws each row as a net that joins the ports that reference it. + +`label` shows a text field beside each row, inside a block, or along a stroke, when the row is +visible and the text fits. Labels never overlap. Where two would, the row drawn on top keeps its +label, so `priority` decides which rows are labeled first. A dense view labels as many rows as fit. +A row whose text is missing has no label, so a `when` that picks between the field and `null` +labels only some rows, as in the scene above. ### Expressions -Options that vary by row take expressions: a number, a field name, a `#hex` color, or a builder. +Options that vary by row take an expression: a number, a boolean, a string, `null`, or a builder. +A string `'$name'` reads field `name`. Any other string is a CSS color. To color by a field, pass it +through a colormap or a conditional. `null` is a missing value, and an option that evaluates to +missing takes the theme's value. ```ts -import { field, when, selected, units } from 'latkit'; +import { field, when, selected, zoom, units, px } from 'latkit'; field('load').div(field('capacity')).clamp(0, 1); // arithmetic and comparisons -field('vm').colormap('balance').domain(0.95, 1.05); // through a colormap +field('vm').colormap('balance').domain(0.95, 1.05); // a color through a colormap field('kv').scale(2, 8); // onto a range -when(selected()).then('#ffb000').otherwise(field('kind').colormap('tableau10')); -units('radius', { atLeast: 2 }); // a size in position units, kept within pixels -field('vm').of('weather', 'Station'); // read from another dataset, joined by id +when(selected()).then('#ffb000').otherwise(field('kind').colormap('tableau10')); // colors +when(zoom().ge(6)).then(field('name')).otherwise(null); // a label's text, from zoom 6 on +units('$radius', { atLeast: 2 }); // a size in position units, kept within pixels +px(3); // a size in CSS pixels, however the view zooms +field('vm').from(weather.table('Station')); // read from another dataset, joined by id ``` -State expressions read the view: `selected()`, `hovered()`, `pointer()`, `elapsed()`, `zoom()`. -Colormaps are a catalog name or your own colors, `{ kind, colors }`; a categorical map can name its -`categories` for its legend. `colormapCss(map)` gives the same colors as a CSS gradient. +Sizes are in CSS pixels, or in position units in a diagram. Wrap a size in `px()` or `units()` to +choose its unit. State expressions read the view: `selected()`, `hovered()`, `pointer()`, +`elapsed()`, and `zoom()`. `colormaps()` lists the colormap catalog, and `colormapCss(map)` returns +a colormap as a CSS gradient, for legends drawn in HTML. ### Themes, controls, and layouts -- **Theme**: `'dark'`, `'light'`, or options over a base, `{ base: 'light', fontSize: 12 }`. `edge: -null` blends each edge's ends' colors, and `selected: null` glows in each row's own color. -- **Controls**: `'inspect'` (the default), `'none'`, or options: `pan`, `zoom`, `turn`, - `keyboard`, `hover`, `pickRadius`, `selectBox`, `connect`, `delete`, `cycle`, `press`, `motion`. -- **Layout** places vertex rows without positions: `'stress'`, `'layered'`, or their options. A - place, once given, holds; `view.arrange()` lays every row out again. +- `theme` is `'dark'`, `'light'`, or overrides of a base theme, grouped by what they style: + `{ base: 'light', text: { size: 12 }, edge: { width: 2 } }`. +- `controls` sets what the pointer and keys do. By default the pointer pans, zooms, turns, and + hovers. `'inspect'` leaves the camera to your application, and `'none'` suits a still picture. +- `layout` places vertex rows that have no position: `'stress'` or `'layered'`. A row keeps its + place once laid out. ## Views -A `Gpu` makes views and canvases. A view draws one scene from the datasets it is given by name; a -canvas shows views, one per panel, and passes each the input on its panel. +`View.create(canvas, scene)` draws a scene on a `` and fits everything in view once it +draws. Several views can share a canvas. Each takes a region, `[x, y, width, height]` in fractions +of the canvas, and pointer, wheel, and key input goes to the view under the pointer: ```ts -const gpu = await Gpu.create(); -const map = gpu.view(mapScene); -const plot = gpu.view(plotScene); -gpu - .canvas(element, { - panels: [ - [0, 0, 1, 0.7], - [0, 0.7, 1, 0.3], - ], - }) - .show([map, plot]); -for (const view of [map, plot]) view.setData(grid); +const map = await View.create(canvas, mapScene, { region: [0, 0, 1, 0.7] }); +const plot = await View.create(canvas, plotScene, { region: [0, 0.7, 1, 0.3] }); ``` -### Data and tiles +A view with a `null` canvas draws only into [pictures](#pictures-and-video). Views draw with the +first `Gpu` created, and `View.create` creates one if there is none. Create a `Gpu` first to set its +[budget](#memory), or create another for views that should not share its memory, and pass it as +`gpu`. -`setData(data)` gives a view a dataset, replacing the one of its name; `removeData(name)` takes it -away. `setTiles(name, url)` fetches a dataset in pieces as the view asks for them, kept as dataset -`name`: [time tiles](#fetch-long-histories-in-spans), a `frames.json` manifest, or vector -tiles as the camera asks for them (a PMTiles archive, a TileJSON document, or a `{z}/{x}/{y}` -template), each layer a table of the kind marks draw it as, or named `layer:polygons`, -`layer:lines`, or `layer:points`. +A view draws each `Data` at its latest version. `view.sources` adds datasets to the scene's, or +replaces one with the same name. Use a `Snapshot` to show a past version, or +[`Tiles`](reference/api.md#tiles) to fetch only the pieces the view needs from time tiles or vector +tiles. ### Camera ```ts -view.fit(); // everything drawn +view.fit(); // everything drawn, as a new view does view.fit({ rows: [{ data: 'grid', table: 'Bus', ids: [101, 102] }] }); view.fly({ center: [-97.7, 30.3], zoom: 8, pitch: 40, bearing: 0 }); // ease there view.camera = { x: [0, 60], y: null }; // a plot's camera: the axis range, values fitted @@ -285,136 +261,105 @@ view.insets = [48, 0, 0, 320]; // CSS pixels kept clear: top, right, bottom, lef view.orbit = 6; // degrees a second ``` +Camera moves stay within the camera's zoom and pan limits. A flat map shows one world. Its camera +zooms out only until the whole world fits, and pans only until the world's edges meet the view's +edges. A globe zooms out until it spans a third of the view. After a fit, the camera stays near the +area the fit framed ([`View.camera`](reference/api.md#view)). + ### Events -`view.on(type, listener)` returns a function that stops listening. Intents are told once each, in -the order the user acted: `click`, `open`, `menu`, `drag`, `connect`, `delete`. Changes tell only -their latest, and their state can always be read: `hover` (`view.hovered`), `select` -(`view.selection`), `camera`, and `waiting` (`view.stats.waiting`). A view tells only what the -application did not do: its setters change state silently. +Intent events (`click`, `open`, `menu`, `drag`, `connect`, `delete`) fire once per action, in the +order the user acted. Change events (`hovered`, `selection`, `camera`, `waiting`) carry the latest +state, which the property of the same name also holds. Setting a property fires no event. ```ts view.on('click', ({ hit }) => console.log(hit.row, hit.point)); view.on('drag', ({ hit, position }) => move(hit.row, position)); // where the row goes, as grabbed -view.on('select', ({ rows }) => inspect(rows)); +view.on('selection', ({ rows }) => inspect(rows)); view.on('error', (error) => console.error(error.code, error.path, error.message)); ``` ### Rows -A row is `{ data, table, id }`; a set of rows of one table is `{ data, table, ids }`. The selection, -`select` events, `delete` events, fits, and neighbors are sets: +A row is `{ data, table, id }`, and a set of rows in one table `{ data, table, ids }`: ```ts -view.select([{ data: 'grid', table: 'Bus', ids: [101, 102] }]); +view.selection = [{ data: 'grid', table: 'Bus', ids: [101, 102] }]; const joined = view.neighbors({ data: 'grid', table: 'Bus', id: 101 }); // across edges and nets -const hits = await view.pick([x, y], 8); // rows within 8 CSS pixels, nearest first -const at = await view.locate({ data: 'grid', table: 'Bus', id: 101 }); // [x, y] on screen, or null +const hits = await view.pick([x, y], { radius: 8 }); // rows within 8 CSS pixels, nearest first +const at = await view.screenOf({ data: 'grid', table: 'Bus', id: 101 }); // [x, y] on screen, or null ``` -Picks and locations answer once a frame draws the view, and reject with `timeout` after a deadline -(`{ deadline: 5 }` seconds unless given) and with `cancelled` when the view is destroyed. - ### Layouts into data -In a diagram, or on a plane with a layout, rows without positions are placed by it. -`view.arrange()` lays every row out again and answers a patch to each dataset writing the places -into its position fields: +`view.arrange()` lays out every vertex row that has no position again. It resolves to one patch per +dataset, which writes the new places into the mark's position fields. Apply the patches to make the +places part of the data: ```ts for (const patch of await view.arrange()) if (patch.data === grid.name) grid.apply(patch); -view.setData(grid); ``` -### Stats and memory - -`view.stats` tells rows drawn, draw calls, tiles, bytes resident and uploading, and CPU and GPU -milliseconds a frame. `gpu.work` counts exactly what the GPU did: bytes uploaded and written, rows -evaluated, readbacks, jobs, texts, and pipelines. `gpu.budget = { bytes, uploadPerFrame }` sets how -much memory data takes. The GPU's views share it: a view gets what it asks while an even split -covers it, and the rest split what is left; pages several views draw count once. A table too large -for its view's share draws through the tiles in view. - -A frame does the work of what changed: a patch evaluates only the pages it wrote, unless a fitted -domain or a join reads them; a transition evaluates once and blends each frame; and rows drawn -through tiles keep their places while the camera moves, so only arriving rows upload. - -`Data`, `Gpu`, `View`, `Canvas`, and `Image` have an idempotent `destroy()` and `Symbol.dispose`; destroying a `Gpu` destroys its -canvases, views, and frame listeners. +### Memory -Videos are asynchronous resources: finish them with `await video.finish()` or cancel with -`await video.abort(reason)`. `record` cleans up on errors automatically. +`gpu.budget = { bytes, uploadPerFrame }` caps the GPU memory that data uses and the bytes uploaded +per frame. Set it before data draws, on a `Gpu` created before any view or on `view.gpu`. All views +of one GPU share the budget. A view that needs less than an even split gets all it needs, and the +other views divide the rest. A table too large for its view's share is split into pages, and the +view draws only the pages on screen. A patch re-evaluates only the pages it wrote. -`gpu.lost` tells whether the device is gone. Create a new GPU, views, and canvases to recover, -then supply retained scenes and datasets. `view.failures` contains input/loading failures; -`view.wantsFrame` is useful for custom scheduling. For manual rendering use `gpu.step(now)`; -`gpu.done()` waits for drawing to settle, and `gpu.landed()` waits for pending work to arrive. -`view.ground([x, y])` converts a canvas point to the scene's ground plane when it intersects. +`view.stats`, `gpu.work`, `gpu.resident`, and `gpu.uploading` report what the views draw and what +work the GPU did. To drive frames yourself in a test or benchmark, call `gpu.step(now)` and wait on +`gpu.landed()` between frames. ## Pictures and video -A `Shot` is plain data: a size in pixels, and how views are drawn into it. It draws views whether or -not a canvas shows them; a canvas showing one holds its last frame meanwhile. +A [`Shot`](reference/api.md#shot) is plain data: a size in pixels and the views to draw into it, +each in its region. It draws views whether or not they are on a canvas. `Shot.of(view)` makes a +shot of exactly what a view's canvas shows. ```ts -const shot = new Shot([3840, 2160]).scale(2); // 2 pixels a CSS pixel: 1920 by 1080 CSS pixels -const image = await shot.image([view]); // once the view is complete and still +const shot = new Shot([3840, 2160], { scale: 2, views: [view] }); // 1920 by 1080 CSS pixels +const image = await shot.image(); // once the view is complete and still const png = await image.encode('png'); // or 'jpeg' with { quality }, or 'webp' image.destroy(); ``` -| Option | Does | -| --------------- | ---------------------------------------------------------------- | -| `scale(n)` | draws `n` pixels per CSS pixel | -| `panels([...])` | where each view draws, `[x, y, width, height]` fractions | -| `transparent()` | keeps alpha where the themes' backgrounds are | -| `deadline(s)` | rejects with `timeout` after `s` seconds, readback included (30) | -| `elapsed(s)` | fixes time-reading expressions and effects at `s` seconds | - -Files are encoded in a worker when workers are available; otherwise encoding runs on the calling thread. - ### Video -`shot.video(gpu, fps, output)` encodes H.264 in MP4 with WebCodecs as frames are drawn, and writes -it to a `WritableStream`: a file handle's, or a buffer's. `record` steps the views to each frame's -time; returning `false` finishes the video. +`shot.video({ fps, output })` encodes H.264 MP4 with WebCodecs as it draws frames, and writes it to +a `WritableStream`. Frame `n` draws the views at `n / fps` seconds, so motion follows the video's +clock, not the wall clock. A frame where nothing moved repeats the one before. `record(step)` calls +`step(time)` before each frame and finishes the video when `step` returns `false`: ```ts const file = await (await showSaveFilePicker()).createWritable(); -const video = new Shot([1920, 1080]).video(gpu, 60, file, { bitrate: 12e6 }); -await video.record([view], (time) => { +const video = new Shot([1920, 1080], { views: [view] }).video({ fps: 60, output: file }); +await video.record((time) => { view.at = time; return time < 10; }); ``` -Width and height are multiples of 4 and 2. A frame where nothing moved is held rather than drawn. -`{ signal }` aborts a video: its output is aborted and its views are given back. - -For manual control, call `await video.frame([view])` one frame at a time, then -`await video.finish()`. Finish and abort are idempotent; both release the output's writer lock, -encoder, and abort listener. Frame/time counters remain readable after completion. A failed -step, frame, write, or progress callback aborts `record`. Cancellation interrupts pending rendering -and encoding; an arbitrary writable sink must still settle its own in-flight write before its -lock can be released. Browser output uses Constrained Baseline H.264, with output timestamps -checked against input order. +To draw frames one at a time, call `video.frame()` for each frame and `video.finish()` at the end, +or `video.abort()` to stop. ## Formats ### Schema -Everything latkit reads and writes as JSON has a JSON Schema, derived from the same declarations -that read and write it: `latkit schema` prints it, and the npm package's types are written from it. -Its `$defs` are the inputs (`Scene`, `Data`, `Patch`, `Query`, `Camera`, `Fit`, `Budget`, `Shot`) -and its `outputs` what latkit writes back (`ViewEvent`, `Stats`, `Work`, `Description`, `Rows`, -`Camera`, `Budget`, `Value`, `Failure`). +All JSON that latkit reads and writes has a JSON Schema, generated from the same definitions that +read and write it. `latkit schema` prints it, and the npm package's types are generated from it. +Its `$defs` are the inputs (`Scene`, `Data`, `Patch`, `Query`, `Camera`, `Fit`, `Budget`, `Shot`). +Its `outputs` are what latkit returns (`ViewEvent`, `Stats`, `Work`, `Description`, `Rows`, +`Camera`, `Budget`, `Value`, `Failure`, and the types they use). (json)= ### JSON -A dataset's document is its name and the patch that builds it. A patch is a list of changes, each -named by its method with its parameters by name: +A dataset's JSON document holds its name and the patch that builds it. A patch is a list of +changes. Each change is an object keyed by its method name, holding its parameters by name: ```json { @@ -449,25 +394,23 @@ named by its method with its parameters by name: } ``` -- `replace` and `set` take `{ table, ids, fields }`, `remove` `{ table, ids }`, `append` - `{ at, table, ids?, fields }`, `forget` `{ before }`, `axis` `{ name, unit }`, and `frames` - `{ table, field, level?, first, at, values }`, a run of a sampled field's entries as a - [time tile](#fetch-long-histories-in-spans) holds them. -- `ids` are numbers or texts, or `{ "count": n }`; `null` leaves a reference unwired. -- A field is an array of one kind of value, a reference `{ reference, ids }`, or a slice of a binary - file beside the document, `{ file, dtype, offset, count }`: `count` values from byte `offset`, of - dtype `f32le`, `f64le`, `i32le`, `u32le`, `u64le`, `u8`, or `bool` (a byte each, nonzero true). +A field is an array of one kind of value, a reference `{ reference, ids }`, or a slice of a binary +file next to the document, `{ file, dtype, offset, count }`. A slice reads `count` values starting +at byte `offset`. Its `dtype` is `f32le`, `f64le`, `i32le`, `u32le`, `u64le`, or `u8` for numbers, +or `bool` for booleans (one byte each, nonzero is true). A `null` id in a reference leaves that row +unconnected. (bytes)= ### Bytes -LATK v1 encodes a patch as one frame; a dataset's bytes are the patch that builds it. A frame says its -length up front, so frames back to back split without decoding. Little endian throughout. +A patch encodes as one frame, and a dataset's bytes are the patch that builds it. A frame's header +gives its length, so a reader can split frames written back to back without decoding them. All +values are little endian. | Part | Layout | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| frame | `LATK`, `u16` version (1), `u16` flags (0), `u64` length of the whole frame, dataset name (empty for none), `u32` op count, ops | +| frame | `LATK`, `u64` length of the whole frame, dataset name (empty for none), `u32` op count, ops | | op | `u8` tag: 0 axis, 1 replace, 2 set, 3 remove, 4 append, 5 forget, 6 frames; then its payload | | axis | name, unit | | replace, set | table name, ids, `u32` field count; per field: name, `u8` 0 values, 1 reference (target table name, ids as values), or 2 samples | @@ -482,90 +425,68 @@ length up front, so frames back to back split without decoding. Little endian th | storage | 0 `f64`, 1 `f32`, 2 `i32`, 3 `u32`, 4 `u64`, 5 bits, 6 texts (`u32` offsets, then `u32` byte length and UTF-8), 7 geometry (`u8` depth, each level's `u32` count and offsets, `u32` point count, `f64` x y pairs) | | text | `u32` byte length, UTF-8 | -Bits are `u64` words, low bit first. A sampled field's entries are frames at level 0, and at level -`k` buckets of `8^k` frames; a bucket whose eight entries a level below are held is left out, as it -follows from them. A writer outside latkit writes frames as `append` changes. - -The header is 16 bytes. The version is independent of the crate/npm version. Readers reject -unknown versions, nonzero flags, unknown operation/storage tags, truncated frames, and trailing -bytes. Pre-release unversioned frames must be regenerated from source data; they are not v1. -The checked-in `latk-v1.hex` fixture is tested by Rust and JavaScript. - -Absolute frame indexes are `u64`; counts within an encoded run remain `u32`. A run's end and its -span in frames must fit in `u64`. JavaScript builders accept only safe integers, including the -run's end (`2^53 - 1` at most). Levels 0 through 21 are legal; the writer records the highest -stored level so changing the renderer's detail policy does not change the file's interpretation. -Normal appends currently derive up to six levels. Tile manifests record each field's `levels`. -Sample indexes in query/pick JSON outputs use decimal strings above JavaScript's safe-integer range. - -Sample values and extrema are `f32`, with NaN representing missing values. Axis coordinates and -geometry coordinates are `f64`. An appended `Float64Array` is normalized to sampled `f32` storage; -this is a visualization history, not a lossless float64 archive. Static float64 columns stay float64. - -`Patch.stream` limits one encoded frame (256 MiB by default), in addition to the supplied input -chunk and patches retained by the caller. It validates the header before allocating the frame body -and cancels its reader on failure or early termination. Rust decoding defaults to a 256 MiB frame -limit and a 512 MiB conservative decoded-storage budget; `Patch::decode_with_limits` accepts -`DecodeLimits` for applications with different needs. Applying patches and retaining snapshots -have their own memory costs, separate from decoding. +Bits are packed into `u64` words, low bit first. A sampled field's entries are frames at level 0 +and buckets of `8^k` frames at level `k`, for levels 0 to 21. A bucket is left out when all eight +entries one level below it are stored, since it can be computed from them. Writers outside latkit +should record frames as `append` changes. + +The header is 12 bytes: the magic `LATK` and the frame's length. Readers reject any other magic, +unknown operation or storage tags, truncated frames, and trailing bytes. The format has no version. +A release that changes it says so, and files written before that release no longer open. Rust and +JavaScript tests both check the committed `patch.hex` fixture. + +Absolute frame indexes are `u64`, and counts within an encoded run are `u32`. JavaScript builders +accept only safe integers. In Rust, `Patch::from_bytes` decodes with the default `DecodeLimits`: a +256 MiB frame and 512 MiB of decoded storage. `Patch::stream(read, limits)` takes other limits. ## Deploying -Install `latkit` from npm for JavaScript and `latkit-cli` from Cargo for the `latkit` command. -The Rust library is the `latkit` crate. The WASM binding crate is an implementation detail. +Install `latkit` from npm for JavaScript, and `latkit-cli` from Cargo for the `latkit` command. +The Rust library is the `latkit` crate. | Capability | Browser | Node | Native CLI | | ----------------------------------- | ------------------------------ | -------------------------------- | ---------------------------- | -| Scenes, expressions, patch builders | `latkit/core` or `latkit` | `latkit/core`, no DOM types | JSON inputs | +| Scenes, expressions, patch builders | `latkit`, no WebAssembly | `latkit`, no WebAssembly | JSON inputs | | Data and queries | WASM, no GPU required | WASM with a compiled module | Available without a GPU | | Rendering | WebGPU in a secure context | Use the CLI for native rendering | Vulkan, Metal, DX12 | | File/URL slices | Browser fetch and CORS | Supply data directly | Local paths and `file:` URLs | | H.264 MP4 | WebCodecs Constrained Baseline | Use the CLI | Windows Media Foundation | -Serve WASM with `Content-Type: application/wasm`. WebGPU requires HTTPS or localhost and a -supported device. Keep the JavaScript, WASM, and worker from the same package version together. -Images and raw frame output are available where native MP4 encoding is unavailable. +Serve WASM with `Content-Type: application/wasm`. Serve the JavaScript, WASM, and worker from the +same package version. WebGPU needs HTTPS or localhost, and a supported device. ### Assets and workers -Default loading resolves assets beside the installed modules. With Vite or another bundler -supporting URL imports, explicit assets also work under nested deployment paths: +By default, `load()` finds the WASM and worker files next to the installed modules. With Vite or +another bundler that supports URL imports, pass their URLs. This also works when the app is +deployed under a nested path: ```ts -import { loadLatkit } from 'latkit'; +import { load } from 'latkit'; import wasmUrl from 'latkit/wasm?url'; import workerUrl from 'latkit/worker?url'; -const kit = await loadLatkit({ - wasm: wasmUrl, - createWorker: () => new Worker(workerUrl), -}); +await load({ wasm: wasmUrl, worker: () => new Worker(workerUrl) }); ``` -`latkit/worker` is a standalone classic worker. A custom factory may use a Blob URL; retain the -URL while the factory can be called and revoke it on application teardown. Workers are created -on demand and retire when idle. Where workers are unavailable, jobs run on the calling thread. - -Your Content Security Policy must allow the WASM asset and data origins in `connect-src`, +`latkit/worker` is a standalone classic worker, so it also works from a Blob URL. Workers start when +needed and stop after 30 idle seconds. Where workers are unavailable, jobs run on the calling +thread. Your Content Security Policy must allow the WASM asset and data origins in `connect-src`, WebAssembly compilation (typically `'wasm-unsafe-eval'` in `script-src`), and the worker URL in -`worker-src`. Add `blob:` only when using Blob workers. Cross-origin resources need appropriate -CORS headers. These requirements are deployment choices; latkit does not change your policy. +`worker-src`. ### Server-side builders and data ```ts -import { Patch, field } from 'latkit/core'; +import { Patch, field } from 'latkit'; const bytes = new Patch().replace('T', [1], { value: [42] }).toBytes(); const expression = field('value').gt(0).toJSON(); ``` -For Node data/query operations, load the package's matching WASM bytes as a `WebAssembly.Module` -and pass it to `loadLatkit({ wasm })`. This runtime entry point exposes browser rendering types; -server projects that only write patches should use `latkit/core`. For worker rendering, provide -an `OffscreenCanvas` and drive `gpu.step(now)` if the environment has no animation-frame loop. - -See [resource lifetimes](#stats-and-memory) and [video cleanup](#video). +Patches and expressions need no WebAssembly. To use datasets and queries in Node, compile the +package's WASM bytes into a `WebAssembly.Module` and pass it to `load({ wasm })`. To render in a +worker, pass an `OffscreenCanvas` to `View.create`. ## In Rust @@ -573,29 +494,23 @@ For CPU data processing, opt out of the default GPU feature: ```toml [dependencies] -latkit = { version = "0.1", default-features = false } +latkit = { version = "0.3", default-features = false } ``` ```rust -use latkit::{Data, Ids, Patch, Query, Table, field}; - -let mut data = Data::new("grid"); -data.apply(Patch::new().replace( - Table::new("Bus", Ids::count(2)).field("kv", vec![115.0, 230.0]), -))?; -let (rows, total) = data.query(&Query::of("Bus").filter(field("kv").ge(200.0)))?; -let saved = data.to_bytes(); -let restored = Data::from_bytes(&saved)?; -``` +use latkit::{Data, Ids, Patch, Query, Reading, Table, field}; -`Data::load` and `Patch::load` accept a `resource::Fetch` implementation for binary slices. -Implement `fetch(&Request) -> Pending>`; `Pending::ready` wraps immediate results, and -`Pending::new` returns a resolver for asynchronous work. A rendering `Host` automatically -implements this smaller interface, so existing native and browser hosts work unchanged. +let grid = Data::new("grid"); +grid.apply(Patch::new().replace(Table::new("Bus", Ids::count(2)).field("kv", vec![115.0, 230.0])))?; +let high = grid.snapshot().query(&Query::of("Bus").filter(field("kv").ge(200.0)))?; +let saved = grid.snapshot().to_bytes(); +let restored = Data::read(&saved, Reading::default()).await?; +``` -Enable `gpu` for scenes, views, and wgpu rendering; `native` adds system fonts, jobs, files, and -Windows H.264. `web` enables browser integration for the WASM target. Native file loading does -not include an HTTP client; applications can provide one through `Fetch` or `Host`. +`Data::read` and `Patch::read` take a `Reading`: the dataset's name, the base its slices are +relative to, and the `resource::Fetch` that fetches them. Every rendering `Host` is also a `Fetch`. +Enable `gpu` for scenes, views, and wgpu rendering. `native` adds system fonts, jobs, files, and +Windows H.264, and `web` adds browser integration for the WASM target. ### Share an existing device @@ -607,10 +522,7 @@ device.set_device_lost_callback(move |_, _| loss.lost()); device.on_uncaptured_error(std::sync::Arc::new(move |error| events.error(&error))); ``` -Forward these events from your application's existing callbacks. `from_wgpu` validates device -limits and preserves callback ownership. Clone one `Gpu` for views that should share caches and -budgets. `Gpu::new` owns its device and installs its own callbacks. Device loss requires a new -device and views; datasets and scene descriptions can be retained. - -The [Rust API reference](https://docs.rs/latkit) documents each type, method, feature, and example. -The [format specification](#bytes) describes cross-language LATK storage. +`from_wgpu` checks the device's limits but leaves its callbacks to you. Forward device loss and +uncaptured errors from your own callbacks, as above. Clone one `Gpu` for views that should share +caches and a budget. The [Rust API reference](https://docs.rs/latkit) documents each type, feature, +and example. diff --git a/docs/images/frequency.png b/docs/images/frequency.png index 0cb4bb11..b9c5934d 100644 Binary files a/docs/images/frequency.png and b/docs/images/frequency.png differ diff --git a/docs/images/globe.png b/docs/images/globe.png index 756c59ac..d0e7a920 100644 Binary files a/docs/images/globe.png and b/docs/images/globe.png differ diff --git a/docs/images/grid.png b/docs/images/grid.png index 8fdb8f20..db5f1022 100644 Binary files a/docs/images/grid.png and b/docs/images/grid.png differ diff --git a/docs/images/helios.png b/docs/images/helios.png index dd70fd51..109c161c 100644 Binary files a/docs/images/helios.png and b/docs/images/helios.png differ diff --git a/docs/images/opening.png b/docs/images/opening.png index d616992d..d673a4b2 100644 Binary files a/docs/images/opening.png and b/docs/images/opening.png differ diff --git a/docs/images/texas.png b/docs/images/texas.png index 2c05966e..bfb648b4 100644 Binary files a/docs/images/texas.png and b/docs/images/texas.png differ diff --git a/docs/index.md b/docs/index.md index 30d95a0f..dcdc222e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,16 +8,16 @@ renders interactive views in the browser with WebGPU, and images and video from :::{container} plot ```{image} images/opening.png -:alt: The ACTIVSg70k grid on the globe as one 765 kV line opens: buses raised by their frequency, branches that pick up its power glow amber, those it leaves glow cyan +:alt: The ACTIVSg70k grid on a globe as one 765 kV line opens. Buses rise with their frequency, branches that pick up the line's power glow amber, and branches that lose flow glow cyan :class: banner ``` ::: :::{container} caption -The 70,000-bus ACTIVSg70k grid as one 765 kV line opens: a DC power flow finds where its power goes, -amber on the branches that pick it up and cyan on those it leaves, and each bus stands as high as its -frequency. Drawn by the `latkit` command line in the {doc}`examples/opening` example. +The 70,000-bus ACTIVSg70k grid as one 765 kV line opens. A DC power flow computes where the line's +power goes. Branches that pick it up glow amber, and branches that lose flow glow cyan. Each bus +rises with its frequency. Drawn by the `latkit` command line from `docs/gallery/opening.ts`. ::: ## Install @@ -30,17 +30,18 @@ cargo add latkit # in Rust ## Quickstart -`quickstart(canvas)` draws a dataset on a canvas; destroy the `gpu` and `data` it returns when the +`quickstart(canvas)` draws a dataset on a canvas. Destroy the `view` and `grid` it returns when the page is done with them. The package's consumer test type-checks and renders this file. ```{literalinclude} ../npm/test/consumer/quickstart.ts :language: typescript ``` -A **dataset** is named tables of columns, changed by **patches**. A **scene** says what to draw: -marks naming tables, in a space, with a theme and controls. A **view** draws a scene from the -datasets it is given, on a canvas or into pictures. Scenes, expressions, and patches are plain data -that need no WebAssembly; `loadLatkit()` loads it for data and drawing. +A **dataset** is a set of named tables of columns, and you change it in place. A **scene** +describes what to draw: marks that draw a dataset's tables, in a space, with a theme and controls. +A **view** draws a scene on a canvas and into pictures. Scenes and expressions are plain data, and +`JSON.stringify` writes the JSON the command line reads. `load()` loads the WebAssembly that +datasets and drawing need. `View.create()`, `Data.read()`, and `Patch.read()` call it for you. ## Examples @@ -49,7 +50,7 @@ that need no WebAssembly; `loadLatkit()` loads it for data and drawing. : 1,456 machines swinging after a fault, traced from a recording on a plot. {doc}`examples/globe` -: Routes along great circles on a globe, comets flowing on them. +: Routes along great circles on a globe, with comets flowing along them. {doc}`examples/diagram` : Blocks in nested groups, laid out and wired port to port. @@ -61,9 +62,9 @@ that need no WebAssembly; `loadLatkit()` loads it for data and drawing. : 260,000 translucent points of a strange attractor. ::: -Every picture is drawn by latkit; the {doc}`examples/index` hold them all with the code that draws -them. The {doc}`guide` covers data, scenes, views, and pictures, the {doc}`cli` draws them from -JSON, {doc}`performance` measures what each costs, and the reference lists every option. +latkit draws every picture on this site, and the {doc}`examples/index` show the code for each. The +{doc}`guide` covers data, scenes, views, and pictures. The {doc}`cli` page covers drawing from JSON. +{doc}`performance` reports benchmark results, and the reference lists every option. ```{toctree} :hidden: diff --git a/docs/performance.md b/docs/performance.md index aa12a854..881211d4 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -1,10 +1,11 @@ # Performance -Each benchmark scenario at the largest size it ran, in Chrome on one GPU: what opening took, and the -most a frame took on the CPU and the GPU while the view moved, and to answer a pick or the hover. A -frame's CPU time stays the same however many rows draw; the work grows on the GPU. `cargo xtask -bench` measures every scenario up its sizes and holds it to the committed baselines, and -`node benchmark/docs.ts` writes this table from them. +Each row is a benchmark scenario at the largest size it ran, in Chrome on one GPU. **Open** is the +time from creating the view until it has drawn everything. **Frame** and **GPU** are the median CPU +and GPU milliseconds per frame in the scenario's slowest phase with a moving view. **Pick** is the +time to answer a pick or hover. `cargo xtask bench` runs every scenario at increasing sizes and +checks it against the committed baselines. `node benchmark/docs.ts` writes this table from those +baselines. ```{csv-table} :file: _static/performance.csv @@ -14,19 +15,43 @@ bench` measures every scenario up its sizes and holds it to the committed baseli ## What a page is held to -The browser gate opens every example page cold and moves the pointer over it. It fails when: +The browser gate first compiles every pipeline cold in Chrome. Each must compile within 800 ms. It +then opens each example page in a fresh Chrome with no shader cache, moves the pointer over it +(hovering, zooming, panning, clicking, and stepping through rows with the keys), and exports a +picture. -- a pipeline takes over 800 ms to compile, or fails to; -- a page draws no frame within a second; -- a page misses two display frames in a row while it opens or under the pointer; -- a pipeline compiles where it is first used, holding back the frames that use it. +A page fails at once on: + +- A console or page error. +- A pipeline compiled when it is first used, which holds back the frames that use it. +- No exported picture. + +A page is late if any view has not drawn a frame 1,000 ms after the page opens, or if the page +misses more than one display frame in a row while opening or under the pointer. A late page is +opened again, cold, and fails only if it is late both times. A machine stall slows one open, but a +slow page slows both. + +On SwiftShader, which draws on the CPU, the gate only checks that every pipeline compiles. Page +timings need a GPU. ## What makes it fast -- Every row's drawing is computed on the GPU, so a frame's CPU work stays the same however many - rows draw. -- Pipelines compile off the GPU's thread before any frame uses them: what frames draw with at once, - their text as compile workers free up, and the rest a few at a time. -- A pipeline compiles only what its mark reads and is: the primitive it draws, its joins, sampled - fields, and view state. -- A table too large to draw whole draws the tiles the camera asks for, within the GPU's budget. +- The GPU computes how every row draws, so a frame's CPU time does not grow with the number of + rows. Only the GPU's work grows. +- Pipelines compile in the background, off the GPU's thread, before any frame needs them. The next + frame's pipelines compile first, then text, then the rest, a few at a time. +- Each pipeline contains only the code its mark uses: the primitive it draws, and whether it reads + joins, sampled fields, or view state. +- A table too large to draw at once is split into pages. The view draws only the pages on screen, + within the GPU memory budget. A page stays on the GPU while the camera moves, and after the + camera leaves while the budget allows, so coming back to it needs no upload. +- Data derived for drawing, such as edge lanes, triangles, and fitted domains, is recomputed when + the data changes, never when the camera moves. A page's triangles are built the first time it + draws. +- Map tiles are decoded and shaped in workers. An arriving tile appends its rows to its tables, + which are paged like any large table, so only the pages those rows fill upload and evaluate. + Tiles split by the area they cover on screen, so the globe's edge and a tilted horizon stay + coarse. When the tile budget runs out, the tiles drawn largest split first. +- A write updates only the pages that hold the rows it changed, and an append updates only the page + the new rows go into. A paged table rebuilds its spatial tiles only once the rows appended since + the last build reach an eighth of the rows that build used. diff --git a/docs/reference/api.md b/docs/reference/api.md index fddd44e9..46c016fc 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -1,867 +1,1613 @@ # JavaScript API -Generated from the shipped declarations. `loadLatkit()` returns `Data`, `Gpu`, `Shot`, and -`colormapCss`; builders and patches need no WebAssembly. Options take the [scene and result -types](types.md). +Every class, function, and type the `latkit` package exports. Scenes and their options use the +[scene and result types](types.md). These rules apply everywhere, so the entries below don't +repeat them: -## Loading +- Scenes, expressions, and patches are plain data. Datasets, views, pictures, and the colormap + functions need latkit's WebAssembly, and throw an `unloaded` error until [`load()`](#load) finishes. + [`View.create()`](#view-create), [`Gpu.create()`](#gpu-create), [`Data.read()`](#data-read), and [`Patch.read()`](#patch-read) of JSON call it + themselves. +- Everything latkit throws or rejects with is a [`LatkitError`](#latkiterror). Input it can't use fails with an + `invalid` error and changes nothing. The error's `path` names the bad part, such as + `marks[2].color`. +- Every promise waits until its [`Wait`](#wait) signal aborts, or for 30 seconds without one. +- A [`Handle`](#handle) holds memory until `destroy()` or the end of a `using` block frees it. -Load the runtime once a page. +## Loading, errors, and lifetimes -### loadLatkit +What every part of the package shares. -```ts -/** - * Loads the runtime once per page. Concurrent calls share one promise; a failed load can be retried. - * Options explicitly supplied after the first call must match that call. Importing the package alone - * neither initializes WASM nor requests a GPU or worker; loading asks for the page's GPU adapter - * while the WASM compiles. - */ -function loadLatkit(options?: LoadOptions): Promise; -``` +(load)= -### LoadOptions +### `load()` -```ts -/** Options for the page's shared WASM runtime. Omitted options reuse its existing configuration. */ -interface LoadOptions { - /** The matching package's WASM asset, or an already compiled module. Defaults to the bundled asset. */ - wasm?: string | URL | WebAssembly.Module; - /** Creates a worker running the standalone `latkit/worker` bundle. Defaults to the bundled worker. */ - createWorker?: () => Worker; -} +{.signature} +`function load(options?: { wasm?: string | URL | WebAssembly.Module; worker?: () => Worker }): Promise` + +Loads latkit's WebAssembly, once per page. Later and concurrent calls share the first call's +load. Options they pass must equal the first call's, or they reject with an `io` error. A failed +load rejects with an `io` error and can be retried. While the WebAssembly compiles, `load()` +requests a GPU adapter so the browser starts its GPU process early. Then it requests the first +device for [`Gpu.create()`](#gpu-create). + +{#load-wasm .member} +**`wasm`**`?: string | URL | WebAssembly.Module` [¶](#load-wasm){.headerlink} +: The URL of this version's `latkit/wasm` asset, or that asset compiled. Defaults to the + bundled asset. + +{#load-worker .member} +**`worker`**`?: () => Worker` [¶](#load-worker){.headerlink} +: Creates a worker that runs the standalone `latkit/worker` bundle. Defaults to the bundled + worker. + +(latkiterror)= + +### `LatkitError` + +{.signature} +`interface LatkitError extends Error` + +The error latkit throws: an `Error` with a `code` that names the kind of failure. + +{#latkiterror-code .member} +**`code`**`:` [`Failure`](#failure)`['code'] | 'unloaded' | 'destroyed'` [¶](#latkiterror-code){.headerlink} +: The kind of failure: + + - `invalid`: the input is malformed or does not fit. `path` says where. + - `unsupported`: the browser or device cannot do what was asked. + - `limit`: a budget or device limit would be exceeded. + - `io`: reading or writing external bytes failed, such as loading latkit or fetching a file. + - `device-lost`: the GPU device was lost. Create a new [`Gpu`](#gpu). + - `cancelled`: the wait's signal aborted, or the work was cancelled. + - `timeout`: the wait's signal timed out. + - `unloaded`: the call needs WebAssembly, and [`load()`](#load) has not finished. + - `destroyed`: the object was destroyed and can't be used. + +{#latkiterror-path .member} +**`path`**`?: string` [¶](#latkiterror-path){.headerlink} +: On `invalid` errors, the invalid part of the input, such as `marks[2].color`. + +(wait)= + +### `Wait` + +{.signature} +`interface Wait` + +Options that bound how long a promise waits. + +{#wait-signal .member} +**`signal`**`?: AbortSignal` [¶](#wait-signal){.headerlink} +: Aborts the wait. A timeout rejects with a `timeout` error. Any other abort rejects with a + `cancelled` error, with the abort reason as its `cause`. Either way, latkit cancels the call's + pending work where it can. Without a signal, each wait times out after 30 seconds. + +(handle)= + +### `Handle` + +{.signature} +`abstract class Handle` + +An object that holds WebAssembly memory (and GPU memory, for a [`Gpu`](#gpu)) until it is destroyed. +`destroy()` frees it, as does the end of a `using` block. After that, its other methods throw a +`destroyed` error. + +```{rubric} Methods ``` -### Latkit +{#handle-destroy .member} +**`destroy`**`(): void` [¶](#handle-destroy){.headerlink} +: Frees the object's memory. Calling it again does nothing. -```ts -/** - * The initialized data and rendering API. In a browser the page's first GPU is made as the runtime - * loads, and the first `Gpu.create()` takes it; later calls request devices of their own. - */ -interface Latkit { - Data: typeof import('./runtime.js').Data; - Gpu: typeof import('./runtime.js').Gpu; - Shot: typeof import('./runtime.js').Shot; - colormapCss: typeof import('./runtime.js').colormapCss; -} +## Data and patches + +Datasets, the tables in them that marks draw, and the patches that change them. Patches are written without WebAssembly. + +(dataset)= + +### `Dataset` + +{.signature} +`abstract class Dataset extends` [`Handle`](#handle) + +Named tables of columns. A [`Data`](#data) reads its current version and a [`Snapshot`](#snapshot) reads +the version it holds, through the same methods. + +```{rubric} Properties ``` -### LatkitError +{#dataset-name .member} +`readonly` **`name`**`: string` [¶](#dataset-name){.headerlink} +: The dataset's name: the `data` in its tables' JSON form `[data, table]`. -```ts -/** - * What latkit throws: an `Error` with a `code`, and for invalid input the `path` to the part, such - * as `marks[2].vertices.color`. - */ -interface LatkitError extends Error { - code: Failure['code']; - path?: string; -} +```{rubric} Methods ``` -## Data and patches +{#dataset-table .member} +**`table`**`(name: string):` [`Table`](#table) [¶](#dataset-table){.headerlink} +: Returns table `name`, for a mark to draw or a field to read. -Datasets of named tables, changed by patches written without WebAssembly. +{#dataset-describe .member} +**`describe`**`():` [`Description`](#description) [¶](#dataset-describe){.headerlink} +: Describes the tables and row counts, each field and its kind, and the axis and its span. -### Data +{#dataset-get .member} +**`get`**`(table: string, id:` [`Id`](#id)`, field: string, options?: { at?: number }):` [`Value`](#value) [¶](#dataset-get){.headerlink} +: Reads `field` of row `id` in `table`. A sampled field gives its last frame at or before + `options.at`, or its last frame without `at`. A reference gives the referenced row's id. + Returns `null` when the table, row, field, or value is missing. -```ts -/** - * A named collection of tables, changed in place by patches. A snapshot keeps a version, sharing - * every page with it, so keeping earlier versions, as an undo history does, costs only the pages - * they hold apart. A view draws the version it was given until it is given another. - */ -class Data { - constructor(name: string); - /** Releases this handle. Views and snapshots keep the versions they hold. */ - destroy(): void; - [Symbol.dispose](): void; - /** - * The dataset `bytes` hold: latkit's bytes, renamed `name` when given, or GeoJSON, gzip-compressed - * or not, read in a worker as dataset `name`: points, line strings, and polygons as tables - * `points`, `lines`, and `polygons`, each with its geometry as field `geometry` and its properties - * as fields. - */ - static read(bytes: Uint8Array, name?: string): Promise; - /** - * The dataset its JSON document builds, `{"name": ..., "patch": [...]}`, the patch loading as - * `Patch.load` loads one. - */ - static load(json: string, base?: string): Promise; - get name(): string; - /** The names of the tables. */ - get tables(): string[]; - /** Every row's id of `table`, in row order. */ - ids(table: string): Id[]; - /** The names of `table`'s fields. */ - fields(table: string): string[]; - /** The first and last axis coordinates the sampled fields span. */ - get span(): [number, number] | undefined; - /** Applies every change of `patch`, or none when one is invalid. */ - apply(patch: Patch | Uint8Array): this; - /** This version, kept while `apply` changes this one: O(1). */ - snapshot(): Data; - /** The value of `field` for the row with `id` in `table`, at axis coordinate `at` when given. */ - get(table: string, id: Id, field: string, at?: number): Value; - /** What the dataset holds: tables, rows, fields and their kinds, the axis and its span. */ - describe(): Description; - /** - * The rows of a table `query` keeps, filtered, sorted, and paged, sampled fields read at its - * coordinate, on the CPU through the expressions marks draw with. - */ - query(query: Query): Table; - /** - * Each row's least and greatest value of sampled `field` of `table` over `spans` equal spans of - * `range`, of every row or of rows `ids`: row after row, NaN where a span holds no frame. Reads the - * coarsest levels of detail that fit, so a span costs a few reads however many frames it covers. - */ - envelope( - table: string, - field: string, - range: [number, number], - spans: number, - ids?: readonly Id[], - ): Envelope; - /** The dataset as latkit's bytes: the patch that builds it. */ - toBytes(): Uint8Array; -} -``` - -### Table + {#dataset-get-at .member} + **`at`**`?: number` [¶](#dataset-get-at){.headerlink} + : The axis coordinate to read a sampled field at. -```ts -/** - * The rows a query keeps, in the forms a patch writes: numbers as a `Float64Array` (missing NaN), - * other values as arrays (missing `null`), a reference as its target ids. - */ -interface Table { - table: string; - ids: Id[]; - fields: Record< - string, - | Float64Array - | Value[] - | { - reference: string; - ids: (Id | null)[]; - } - >; - /** The rows the query kept before paging. */ - total: number; -} -``` - -### Envelope +{#dataset-query .member} +**`query`**`(query:` [`Query`](#query)`):` [`QueryResult`](#queryresult) [¶](#dataset-query){.headerlink} +: Reads a table's rows, filtered, sorted, and paged with the same expressions marks use. Sampled + fields are read at [`query.at`](#query-at). -```ts -/** Rows' least and greatest values over spans of the axis: `at` holds one more edge than spans. */ -interface Envelope { - at: Float64Array; - least: Float32Array; - greatest: Float32Array; -} -``` - -### Patch - -````ts -/** - * Changes to a dataset, applied in order and all at once by `Data.apply`: plain data, written as - * latkit's bytes as it is built. - * - * ```ts - * const patch = Patch.to('grid') - * .replace('Bus', [101, 102], { x: Float64Array.of(0, 10), name: ['North', 'South'] }) - * .append(0.5, 'Bus', { vm: Float32Array.of(1.01, 0.99) }); - * ``` - */ -class Patch { - /** The dataset the patch changes, when it names one; `Data.apply` refuses it for another. */ - readonly data: string | undefined; - /** An empty patch, for whichever dataset it is applied to. */ - constructor(data?: string); - /** An empty patch to dataset `data`. */ - static to(data: string): Patch; - /** The patch `bytes` hold: one frame, as `toBytes` or a stream writes it. */ - static fromBytes(bytes: Uint8Array): Patch; - /** - * The patches arriving back to back in `stream`, each as soon as its last byte does: a response - * body, a file, or a socket's messages. Each frame says its length up front, so a patch longer - * than `limit` bytes (256 MiB unless given) throws before allocating its body. The parser holds at most one frame, in addition to - * the input chunk and patches the caller retains. Breaking iteration cancels the reader. - * - * ```ts - * for await (const patch of Patch.stream((await fetch('/feed')).body!)) data.apply(patch); - * ``` - */ - static stream( - stream: ReadableBytes | AsyncIterable, - { - limit, - }?: { - limit?: number; - }, - ): AsyncGenerator; - /** - * The patch its JSON makes: a list of changes, each named by its method with its parameters by - * name, such as `{"set": {"table": "Bus", "ids": [1], "fields": {"kv": [345]}}}`. Slices of - * binary files, `{file, dtype, offset, count}`, are fetched at once relative to `base`, the - * document's own URL. Reading JSON takes latkit's WebAssembly, which this loads as - * `loadLatkit()` does. - */ - static load(json: string, base?: string): Promise; - /** Names the axis sampled fields are recorded over, such as `('time', 's')`. */ - axis(name: string, unit: string): this; - /** Replaces table `table` with these rows and fields, or adds it. */ - replace(table: string, ids: RowIds, fields?: Fields): this; - /** - * Writes `fields` for rows `ids` of `table`: rows with known ids change, others are added, and - * fields left out keep their values. Adds the table when there is none of its name. - */ - set(table: string, ids: RowIds, fields?: Fields): this; - /** Removes the rows of `table` with `ids`. */ - remove(table: string, ids: RowIds): this; - /** - * Records a frame at each axis coordinate of `at`: for each field, a value per row at every - * coordinate, frame-major; of every row of `table`, or of rows `ids`, the rest holding their last - * values. A field's first frame makes it sampled. Appends of float arrays to the same fields of - * every row, one after another, write as one change, so a frame a call costs no more than one call. - */ - append( - at: number | readonly number[], - table: string, - fields: Readonly>, - ids?: RowIds, - ): this; - /** Forgets every sampled field's frames before axis coordinate `before`. */ - forget(before: number): this; - /** - * Holds a run of sampled field `field`'s entries of level `level`, numbered from `first`: what a - * time tile holds, as `latkit tile` writes them. At level 0 the entries are frames, each a - * coordinate in `at` and a value per row of `table` in `values`. At level `k` they are buckets of - * `8^k` frames, each its first and last frame's coordinates in `at`, and in `values` each row's - * least value over each bucket, then each row's greatest. NaN is missing. No entry may be held - * already; frames past the field's widen it. - */ - frames( - table: string, - field: string, - { - level, - first, - }: { - level?: number; - first: number; - }, - at: Float64Array | readonly number[], - values: Float32Array | readonly number[], - ): this; - /** Appends `next`'s changes, to this patch's dataset. */ - extend(next: Patch): this; - /** The patch as bytes: one frame, naming its dataset when it has one. */ - toBytes(): Uint8Array; -} -```` - -### RowIds +{#dataset-envelope .member} +**`envelope`**`(request:` [`EnvelopeRequest`](#enveloperequest)`):` [`Envelope`](#envelope) [¶](#dataset-envelope){.headerlink} +: Returns the minimum and maximum of a sampled field over equal spans of an axis range, for every + row or for the rows in `request.ids`. Each span reads the coarsest stored levels of detail that + fit inside it, so it costs a few reads however many frames it covers. Throws a `limit` error + when rows times spans exceeds 2^26. + +{#dataset-tobytes .member} +**`toBytes`**`(): Uint8Array` [¶](#dataset-tobytes){.headerlink} +: Returns the dataset as latkit's bytes: a patch that builds it from an empty dataset. + +(data)= + +### `Data` + +{.signature} +`class Data extends` [`Dataset`](#dataset) + +A dataset that changes in place. Each change applies all at once or not at all. Every +[`Patch`](#patch) method is also a method here that applies that one change right away, and `apply` +applies a patch of several. Views draw the latest version from the frame after each change. A +[`Snapshot`](#snapshot) keeps one version and shares its pages with the dataset, so an undo history +costs only the pages that differ. Destroying a `Data` leaves the views that draw it, and its +snapshots, usable. ```ts -/** Row ids as a patch takes them: listed, `{ count }` for rows `0..count`, or a typed array. */ -type RowIds = Ids | readonly (Id | null)[] | Uint32Array | Float64Array; +const grid = new Data('grid').set('Bus', [1, 2], { kv: [230, 115] }); +grid.set('Bus', [2], { kv: [69] }); // drawn the next frame ``` -### Values +```{rubric} Constructor +``` -```ts -/** - * One field's values, a value per row; `null` is missing. Typed arrays copy at once: numbers, or - * in a `Uint8Array` booleans, nonzero true. Booleans, texts, and geometries also as arrays. - */ -type Values = - | Float64Array - | Float32Array - | Int32Array - | Uint32Array - | Uint8Array - | readonly (number | null)[] - | readonly (boolean | null)[] - | readonly (string | null)[] - | readonly (Geometry | null)[]; -``` - -### Geometry +{#data-new .member} +**`new Data`**`(name: string)` [¶](#data-new){.headerlink} +: Creates an empty dataset named `name`. -```ts -/** A point `[x, y]`, or a list of geometries one level shallower: a path, its parts, or polygons. */ -type Geometry = readonly number[] | readonly Geometry[]; +```{rubric} Methods ``` -### Reference +{#data-read .member} +`static` **`read`**`(source:` [`ByteSource`](#bytesource) `| object, options?: { name?: string; base?: string } &` [`Wait`](#wait)`): Promise` [¶](#data-read){.headerlink} +: Reads a dataset from `source`. Its first bytes decide the format: -```ts -/** A field naming rows of another table by id: `null` leaves a row unwired. */ -interface Reference { - reference: string; - ids: RowIds; -} + - latkit's bytes, as `toBytes()` writes them. + - GeoJSON, plain or gzip-compressed, read in a worker. Points, line strings, and polygons + become rows of tables `points`, `lines`, and `polygons`, with the geometry in field + `geometry` and each property as a field. + - A dataset document, `{"name": ..., "patch": [...]}`. Its patch is read like + [`Patch.read()`](#patch-read). + + {#data-read-name .member} + **`name`**`?: string` [¶](#data-read-name){.headerlink} + : The dataset's name, overriding the name in its bytes or document. Required for GeoJSON. + + {#data-read-base .member} + **`base`**`?: string` [¶](#data-read-base){.headerlink} + : The URL that a document's file slices are relative to. Defaults to the document's URL. + +{#data-apply .member} +**`apply`**`(patch:` [`Patch`](#patch) `| Uint8Array): this` [¶](#data-apply){.headerlink} +: Applies every change in `patch`, or none when one is invalid, and returns this dataset. An + `invalid` error's `path` names the failing change, such as `patch[1].Bus.kv`. + +{#data-axis .member} +**`axis`**`(name: string, unit: string): this` [¶](#data-axis){.headerlink} +: Applies [`Patch.axis()`](#patch-axis): names the axis that sampled fields are recorded over. + +{#data-replace .member} +**`replace`**`(table: string, fields:` [`Fields`](#fields)`): this` [¶](#data-replace){.headerlink} +: Applies [`Patch.replace()`](#patch-replace): replaces table `table` with `fields` for rows `ids` (or `0` to + `n - 1`). + +{.member} +**`replace`**`(table: string, ids:` [`RowIds`](#rowids)`, fields?:` [`Fields`](#fields)`): this` +: Applies [`Patch.replace()`](#patch-replace): replaces table `table` with `fields` for rows `ids` (or `0` to + `n - 1`). + +{#data-set .member} +**`set`**`(table: string, fields:` [`Fields`](#fields)`): this` [¶](#data-set){.headerlink} +: Applies [`Patch.set()`](#patch-set): writes `fields` for rows `ids` (or `0` to `n - 1`) of `table`, + adding new rows. + +{.member} +**`set`**`(table: string, ids:` [`RowIds`](#rowids)`, fields?:` [`Fields`](#fields)`): this` +: Applies [`Patch.set()`](#patch-set): writes `fields` for rows `ids` (or `0` to `n - 1`) of `table`, + adding new rows. + +{#data-remove .member} +**`remove`**`(table: string, ids:` [`RowIds`](#rowids)`): this` [¶](#data-remove){.headerlink} +: Applies [`Patch.remove()`](#patch-remove): removes the rows of `table` with `ids`. + +{#data-append .member} +**`append`**`(table: string, at: number | readonly number[], fields: Readonly>, options?: { ids?:` [`RowIds`](#rowids) `}): this` [¶](#data-append){.headerlink} +: Applies [`Patch.append()`](#patch-append): records frames of `table` at the axis coordinates `at`. + + {#data-append-ids .member} + **`ids`**`?:` [`RowIds`](#rowids) [¶](#data-append-ids){.headerlink} + : The rows the frames cover. Defaults to every row. Other rows keep their last values. + +{#data-forget .member} +**`forget`**`(before: number): this` [¶](#data-forget){.headerlink} +: Applies [`Patch.forget()`](#patch-forget): drops every sampled field's frames before `before`. + +{#data-frames .member} +**`frames`**`(table: string, field: string, run:` [`Run`](#run)`): this` [¶](#data-frames){.headerlink} +: Applies [`Patch.frames()`](#patch-frames): stores a run of a sampled field's entries, as in a time tile. + +{#data-snapshot .member} +**`snapshot`**`():` [`Snapshot`](#snapshot) [¶](#data-snapshot){.headerlink} +: Returns the current version as a [`Snapshot`](#snapshot). Later patches don't change it. O(1). + +{#data-restore .member} +**`restore`**`(snapshot:` [`Snapshot`](#snapshot)`): this` [¶](#data-restore){.headerlink} +: Makes `snapshot`, a version of this dataset, the current version. This is how to undo. + +(snapshot)= + +### `Snapshot` + +{.signature} +`class Snapshot extends` [`Dataset`](#dataset) + +One immutable version of a dataset, taken in O(1) by [`Data.snapshot()`](#data-snapshot). A view given a +snapshot draws that version, and keeps drawing it after the snapshot is destroyed. + +(tiles)= + +### `Tiles` + +{.signature} +`class Tiles` + +A dataset fetched in pieces as views need them. It is plain data: a dataset name and a URL. Each +view fetches the pieces its camera and plot window need, and views on one GPU share them. The +first bytes at `url` decide the kind: + +- Time tiles: a `frames.json` manifest written by `latkit tile`. A line that traces a sampled + field fetches the detail its plot window shows. Other marks fetch the frames around the view's + `at`. While a span loads, the coarser span that contains it shows. +- Vector tiles: a PMTiles archive, a TileJSON document, or a `{z}/{x}/{y}` URL template. Each + layer becomes a table of the features its marks draw: polygons for areas, lines for lines, + points for vertices. A table named `layer:polygons`, `layer:lines`, or `layer:points` holds + that kind regardless of the marks. Each tile that arrives appends its rows to the view's + dataset. Rows of tiles no longer drawn are kept until they outnumber the rows drawn. + +```{rubric} Constructor ``` -### reference +{#tiles-new .member} +**`new Tiles`**`(name: string, url: string)` [¶](#tiles-new){.headerlink} +: Tiles at `url`, drawn as the dataset named `name`. -```ts -/** Values of a field naming rows of `table` by id. */ -function reference(table: string, ids: RowIds): Reference; +```{rubric} Properties ``` -### Fields +{#tiles-name .member} +`readonly` **`name`**`: string` [¶](#tiles-name){.headerlink} +: The name of the dataset the tiles are drawn as. -```ts -/** A table's fields by name. */ -type Fields = Readonly>; +{#tiles-url .member} +`readonly` **`url`**`: string` [¶](#tiles-url){.headerlink} +: The tiles' URL. + +```{rubric} Methods ``` -### ReadableBytes +{#tiles-table .member} +**`table`**`(name: string):` [`Table`](#table) [¶](#tiles-table){.headerlink} +: Returns table `name`, such as a vector tile layer, for a mark to draw. -```ts -/** A byte stream, structurally compatible with the browser and Node ReadableStream. */ -interface ReadableBytes { - getReader(): { - read(): Promise<{ - done: boolean; - value?: Uint8Array; - }>; - cancel(reason?: unknown): Promise; - releaseLock(): void; - }; -} +(source)= + +### `Source` + +{.signature} +`type Source =` [`Data`](#data) `|` [`Snapshot`](#snapshot) `|` [`Tiles`](#tiles) + +A dataset a view draws: the one a [`Table`](#table) belongs to, or one of the view's `sources`. + +(table)= + +### `Table` + +{.signature} +`class Table` + +A table in a dataset, such as `grid.table('Bus')`. Marks draw its rows and `field().from()` reads +its fields. A view loads every dataset its scene uses. In JSON it is `[data, table]`. + +```{rubric} Constructor ``` -## Expressions +{#table-new .member} +**`new Table`**`(source:` [`Source`](#source)`, name: string)` [¶](#table-new){.headerlink} +: The table `name` in `source`. -What varies by row: builders writing the expressions scenes take. +```{rubric} Properties +``` -### Expr +{#table-source .member} +`readonly` **`source`**`:` [`Source`](#source) [¶](#table-source){.headerlink} +: The dataset the table belongs to. -```ts -/** A value for every row of a mark: a constant, a field, view state, or arithmetic of them. */ -class Expr { - readonly node: ExprNode; - /** An expression from its JSON form; use the builders for common operations. */ - constructor(node: ExprNode); - toJSON(): ExprNode; - /** Reads this expression's fields from `table` of dataset `data`, joined by row id. */ - of(data: string, table: string): Expr; - /** Reads sampled fields at their last frame without interpolating. */ - step(): Expr; - add(other: ExprInput): Expr; - sub(other: ExprInput): Expr; - mul(other: ExprInput): Expr; - div(other: ExprInput): Expr; - rem(other: ExprInput): Expr; - pow(other: ExprInput): Expr; - min(other: ExprInput): Expr; - max(other: ExprInput): Expr; - clamp(low: ExprInput, high: ExprInput): Expr; - lt(other: ExprInput): Expr; - le(other: ExprInput): Expr; - gt(other: ExprInput): Expr; - ge(other: ExprInput): Expr; - eq(other: ExprInput): Expr; - ne(other: ExprInput): Expr; - and(other: ExprInput): Expr; - or(other: ExprInput): Expr; - not(): Expr; - neg(): Expr; - abs(): Expr; - sqrt(): Expr; - sin(): Expr; - cos(): Expr; - floor(): Expr; - ln(): Expr; - exp(): Expr; - /** Whether a text field equals `text`. */ - is(text: string): Expr; - /** Whether a text field holds `text`, ignoring case. Queries read it; marks do not. */ - contains(text: string): Expr; - /** The value through `map`: a catalog name, or your own colors. */ - colormap(map: Colormap): Mapped; - /** The value mapped linearly onto `[low, high]`, clamped. */ - scale(low: number, high: number): Mapped; -} -``` - -### ExprInput +{#table-name .member} +`readonly` **`name`**`: string` [¶](#table-name){.headerlink} +: The table's name. -```ts -/** What an expression is written as: an {@link Expr}, or its JSON form. */ -type ExprInput = Expr | ExprNode; +```{rubric} Methods ``` -### Color +{#table-tojson .member} +**`toJSON`**`(): [data: string, table: string]` [¶](#table-tojson){.headerlink} +: Returns the JSON form, `[data, table]`. `JSON.stringify` calls it. + +(tableinput)= + +### `TableInput` + +{.signature} +`type TableInput =` [`Table`](#table) `| readonly [data: string, table: string]` + +A table for a mark to draw or a field to read: a [`Table`](#table), or its JSON form `[data, table]`, +where `data` names a dataset in the view's `sources`. + +(queryresult)= + +### `QueryResult` + +{.signature} +`interface QueryResult` + +The rows a query returns, as columns in the forms a patch takes. + +{#queryresult-table .member} +**`table`**`: string` [¶](#queryresult-table){.headerlink} +: The table read. + +{#queryresult-ids .member} +**`ids`**`:` [`Id`](#id)`[]` [¶](#queryresult-ids){.headerlink} +: The ids of the rows returned, in order. + +{#queryresult-fields .member} +**`fields`**`: Record` [¶](#queryresult-fields){.headerlink} +: Each selected field's column, keyed by output name. Numbers come as a `Float64Array` with NaN + for missing values. Other values come as an array with `null` for missing ones. A reference + comes as the referenced table and, for each row, the id of the row it references. + +{#queryresult-total .member} +**`total`**`: number` [¶](#queryresult-total){.headerlink} +: The number of rows that matched, before `offset` and `limit`. + +(envelope)= + +### `Envelope` + +{.signature} +`interface Envelope` + +The minimum and maximum of a sampled field for each row, over equal spans of the axis. + +{#envelope-at .member} +**`at`**`: Float64Array` [¶](#envelope-at){.headerlink} +: The span boundaries on the axis: one more than the number of spans. + +{#envelope-least .member} +**`least`**`: Float32Array` [¶](#envelope-least){.headerlink} +: Each row's minimum in each span, `spans` values per row, row after row. NaN for no value. + +{#envelope-greatest .member} +**`greatest`**`: Float32Array` [¶](#envelope-greatest){.headerlink} +: Each row's maximum in each span, laid out like `least`. + +(patch)= + +### `Patch` + +{.signature} +`class Patch` + +A list of changes to a dataset. [`Data.apply`](#data-apply) applies them in order, all or none. The patch is +written as latkit's bytes as it is built, without WebAssembly. Each method adds a change and +returns the patch. ```ts -/** A CSS color, such as `'#ffb000'` or `'rebeccapurple'`. Where an option takes an expression, a - * string starting with `#` is a hex color and any other string names a field. */ -type Color = string; +const patch = new Patch('grid') + .replace('Bus', [101, 102], { x: Float64Array.of(0, 10), name: ['North', 'South'] }) + .append('Bus', 0.5, { vm: Float32Array.of(1.01, 0.99) }); ``` -### field +```{rubric} Constructor +``` -```ts -/** The field `name` of each row. */ -const field: (name: string) => Expr; +{#patch-new .member} +**`new Patch`**`(data?: string)` [¶](#patch-new){.headerlink} +: Creates an empty patch for the dataset named `data`, or for any dataset when omitted. + +```{rubric} Properties ``` -### color +{#patch-data .member} +`readonly` **`data`**`: string | undefined` [¶](#patch-data){.headerlink} +: The name of the dataset this patch is for, if any. [`Data.apply`](#data-apply) rejects it for a dataset with + another name. -```ts -/** A color constant. */ -const color: (css: Color) => Expr; +```{rubric} Methods ``` -### missing +{#patch-read .member} +`static` **`read`**`(source:` [`ByteSource`](#bytesource) `| object, options?: { base?: string; limit?: number } &` [`Wait`](#wait)`): Promise` [¶](#patch-read){.headerlink} +: Reads a patch from `source`: latkit's bytes (one frame, as `toBytes()` writes it), or JSON that + any language can write without latkit. The JSON is a list of changes. Each change is keyed by + its method name and holds the method's parameters by name, such as + `{"set": {"table": "Bus", "ids": [1], "fields": {"kv": [345]}}}`. A field may be a slice of a + binary file, `{file, dtype, offset, count}`, which is fetched right away. Bytes are read + without WebAssembly. + + {#patch-read-base .member} + **`base`**`?: string` [¶](#patch-read-base){.headerlink} + : The URL that the JSON's file slices are relative to. Defaults to the JSON's URL. + + {#patch-read-limit .member} + **`limit`**`?: number` [¶](#patch-read-limit){.headerlink} + : The longest frame in bytes. Defaults to 256 MiB. A longer frame fails with a `limit` error. + +{#patch-stream .member} +`static` **`stream`**`(stream:` [`ReadableBytes`](#readablebytes) `| AsyncIterable, options?: { limit?: number }): AsyncGenerator` [¶](#patch-stream){.headerlink} +: Yields each patch in `stream` as soon as its last byte arrives. The stream carries frames back + to back, such as a response body, a file, or a socket's messages. At most one frame is + buffered, besides the current chunk and any patches the caller keeps. Breaking out of the loop + cancels the stream. Throws an `invalid` error if the stream ends inside a frame. + + ```ts + for await (const patch of Patch.stream((await fetch('/feed')).body!)) data.apply(patch); + ``` -```ts -/** A missing value: options that read one take the theme's. */ -const missing: () => Expr; + {#patch-stream-limit .member} + **`limit`**`?: number` [¶](#patch-stream-limit){.headerlink} + : The longest frame in bytes, at least 12. Defaults to 256 MiB. Each frame states its length + up front, so a longer frame fails with a `limit` error before its body is allocated. + +{#patch-axis .member} +**`axis`**`(name: string, unit: string): this` [¶](#patch-axis){.headerlink} +: Sets the name and unit of the axis that sampled fields are recorded over, such as + `axis('time', 's')`. + +{#patch-replace .member} +**`replace`**`(table: string, fields:` [`Fields`](#fields)`): this` [¶](#patch-replace){.headerlink} +: Replaces table `table` with rows `ids` and `fields`, adding the table if it does not exist. + Without ids, the rows are `0` to `n - 1`, where `n` is the length of each field's values. + +{.member} +**`replace`**`(table: string, ids:` [`RowIds`](#rowids)`, fields?:` [`Fields`](#fields)`): this` +: Replaces table `table` with rows `ids` and `fields`, adding the table if it does not exist. + Without ids, the rows are `0` to `n - 1`, where `n` is the length of each field's values. + +{#patch-set .member} +**`set`**`(table: string, fields:` [`Fields`](#fields)`): this` [¶](#patch-set){.headerlink} +: Writes `fields` for rows `ids` of `table`. Rows with existing ids change, new ids add rows, and + omitted fields keep their values. Adds the table if it does not exist. Without ids, the rows + are `0` to `n - 1`, where `n` is the length of each field's values. + +{.member} +**`set`**`(table: string, ids:` [`RowIds`](#rowids)`, fields?:` [`Fields`](#fields)`): this` +: Writes `fields` for rows `ids` of `table`. Rows with existing ids change, new ids add rows, and + omitted fields keep their values. Adds the table if it does not exist. Without ids, the rows + are `0` to `n - 1`, where `n` is the length of each field's values. + +{#patch-remove .member} +**`remove`**`(table: string, ids:` [`RowIds`](#rowids)`): this` [¶](#patch-remove){.headerlink} +: Removes the rows of `table` with `ids`. + +{#patch-append .member} +**`append`**`(table: string, at: number | readonly number[], fields: Readonly>, options?: { ids?:` [`RowIds`](#rowids) `}): this` [¶](#patch-append){.headerlink} +: Records a frame of `table` at each axis coordinate in `at`. Each field gives one value per row + for each frame, frame by frame. A field becomes a sampled field at its first frame. Consecutive + appends of typed float arrays to the same fields of every row are written as one change, so + appending one frame per call costs no more than one call with every frame. + + {#patch-append-ids .member} + **`ids`**`?:` [`RowIds`](#rowids) [¶](#patch-append-ids){.headerlink} + : The rows the frames cover. Defaults to every row. Other rows keep their last values. + +{#patch-forget .member} +**`forget`**`(before: number): this` [¶](#patch-forget){.headerlink} +: Drops every sampled field's frames before axis coordinate `before`. + +{#patch-frames .member} +**`frames`**`(table: string, field: string, run:` [`Run`](#run)`): this` [¶](#patch-frames){.headerlink} +: Stores a run of entries of sampled field `field` in `table`, at one level of detail. This is + what a time tile from `latkit tile` holds. At level 0 each entry is a frame. At level `k` each + entry is a bucket of `8^k` frames. None of the entries may already be stored. Entries past the + field's last frame extend it. + +{#patch-extend .member} +**`extend`**`(next: Patch): this` [¶](#patch-extend){.headerlink} +: Appends `next`'s changes to this patch, which keeps its own dataset name. + +{#patch-tobytes .member} +**`toBytes`**`(): Uint8Array` [¶](#patch-tobytes){.headerlink} +: Returns a copy of the patch as latkit's bytes: one frame, naming the dataset if it has one. + +(bytesource)= + +### `ByteSource` + +{.signature} +`type ByteSource = Uint8Array | ArrayBuffer | Blob | Response |` [`ReadableBytes`](#readablebytes) `| AsyncIterable | string` + +Bytes to read: an array or buffer, a `Blob`, a `Response`, a byte stream, or a URL string to +fetch. Reading is bounded by the call's [`Wait`](#wait). It fails with an `io` error when a fetch +fails or returns an error status, or when the bytes can't be read. + +(readablebytes)= + +### `ReadableBytes` + +{.signature} +`interface ReadableBytes` + +A byte stream: anything shaped like a browser or Node.js `ReadableStream` of `Uint8Array`. + +{#readablebytes-getreader .member} +**`getReader`**`(): { read(): Promise<{ done: boolean; value?: Uint8Array }>; cancel(reason?: unknown): Promise; releaseLock(): void }` [¶](#readablebytes-getreader){.headerlink} +: Returns a reader of the stream's chunks. latkit reads it to the end or cancels it. + +(rowids)= + +### `RowIds` + +{.signature} +`type RowIds =` [`Ids`](#ids) `| readonly (`[`Id`](#id) `| null)[] | Uint32Array | Float64Array` + +Row ids: a list of numbers or strings, `{ count }` for ids `0` to `count - 1`, or a `Uint32Array` +or `Float64Array` of numbers. + +(values)= + +### `Values` + +{.signature} +`type Values = Float64Array | Float32Array | Int32Array | Uint32Array | Uint8Array | readonly (number | null)[] | readonly (boolean | null)[] | readonly (string | null)[] | readonly (`[`Geometry`](#geometry) `| null)[]` + +One field's values, one per row, with `null` for a missing value. Numbers come as a typed array +or an array. A `Uint8Array` holds booleans, where nonzero is true. Booleans, strings, and +geometries also come as arrays. Values are copied when passed, so the arrays can be reused right +away. + +(geometry)= + +### `Geometry` + +{.signature} +`type Geometry = readonly number[] | readonly Geometry[]` + +A point `[x, y]`, or a nested list of points: a path (a list of points), parts (a list of paths, +such as a polygon's rings), or polygons (a list of parts). + +(reference)= + +### `Reference` + +{.signature} +`interface Reference` + +A reference field: for each row, a row of another table. + +{#reference-reference .member} +**`reference`**`: string` [¶](#reference-reference){.headerlink} +: The table referenced. + +{#reference-ids .member} +**`ids`**`:` [`RowIds`](#rowids) [¶](#reference-ids){.headerlink} +: For each row, the id of the row it references, or `null` to reference none. + +(reference-function)= + +### `reference()` + +{.signature} +`function reference(table: string, ids:` [`RowIds`](#rowids)`):` [`Reference`](#reference) + +Builds a [`Reference`](#reference) field: for each row, the id of the row it references in `table`. + +(fields)= + +### `Fields` + +{.signature} +`type Fields = Readonly>` + +A table's fields by name: values, or a [`Reference`](#reference). + +(run)= + +### `Run` + +{.signature} +`interface Run` + +A run of entries of a sampled field at one level of detail, for [`Patch.frames()`](#patch-frames). + +{#run-level .member} +**`level`**`?: number` [¶](#run-level){.headerlink} +: The level of detail, from 0 to 21. Defaults to 0. + +{#run-first .member} +**`first`**`: number` [¶](#run-first){.headerlink} +: The index of the first entry, a safe nonnegative integer. + +{#run-at .member} +**`at`**`: Float64Array | readonly number[]` [¶](#run-at){.headerlink} +: Each frame's coordinate. At a bucket level, the coordinates of each bucket's first and last + frames. + +{#run-values .member} +**`values`**`: Float32Array | readonly number[]` [¶](#run-values){.headerlink} +: One value per row for each entry, entry by entry. At a bucket level, the per-row minimums of + every bucket, then the per-row maximums of every bucket. NaN marks a missing value. + +## Expressions + +Builders for the per-row expressions that scene options take, including colors. + +(expr)= + +### `Expr` + +{.signature} +`class Expr` + +A value for every row of a mark: a constant, a color, a field, view state, or arithmetic on them. +Color options take one that computes a color: a CSS color, a value mapped through a colormap, or +a conditional that chooses between colors. + +```{rubric} Constructor ``` -### when +{#expr-new .member} +**`new Expr`**`(node:` [`ExprNode`](#exprnode)`)` [¶](#expr-new){.headerlink} +: Wraps an expression's JSON form. The builders cover common cases. -```ts -/** Starts a conditional: `when(c).then(a).when(d).then(b).otherwise(e)`. */ -function when(condition: ExprInput): When; +```{rubric} Properties ``` -### When +{#expr-node .member} +`readonly` **`node`**`:` [`ExprNode`](#exprnode) [¶](#expr-node){.headerlink} +: The expression's JSON form. -```ts -declare class When { - constructor(cases: [ExprNode, ExprNode][], condition: ExprNode); - then(value: ExprInput): Then; -} +```{rubric} Methods ``` -### Then +{#expr-tojson .member} +**`toJSON`**`():` [`ExprNode`](#exprnode) [¶](#expr-tojson){.headerlink} +: Returns the JSON form. `JSON.stringify` calls it. -```ts -declare class Then { - constructor(cases: [ExprNode, ExprNode][]); - when(condition: ExprInput): When; - otherwise(value: ExprInput): Expr; -} +{#expr-from .member} +**`from`**`(table:` [`TableInput`](#tableinput)`): Expr` [¶](#expr-from){.headerlink} +: Reads this expression's fields from `table`, such as `weather.table('Station')`, joined by row + id. The view also loads that table's dataset. + +{#expr-step .member} +**`step`**`(): Expr` [¶](#expr-step){.headerlink} +: Reads this expression's sampled fields at the last frame at or before the view's `at`, without + interpolating toward the next frame. + +{#expr-add .member} +**`add`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-add){.headerlink} +: The sum of this value and `other`. + +{#expr-sub .member} +**`sub`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-sub){.headerlink} +: This value minus `other`. + +{#expr-mul .member} +**`mul`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-mul){.headerlink} +: The product of this value and `other`. + +{#expr-div .member} +**`div`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-div){.headerlink} +: This value divided by `other`. Dividing by 0 gives a missing value. + +{#expr-rem .member} +**`rem`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-rem){.headerlink} +: The remainder of this value divided by `other`, with the sign of `other`: `-1 rem 3` is 2. + +{#expr-pow .member} +**`pow`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-pow){.headerlink} +: This value raised to the power `other`. + +{#expr-min .member} +**`min`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-min){.headerlink} +: The smaller of this value and `other`. + +{#expr-max .member} +**`max`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-max){.headerlink} +: The larger of this value and `other`. + +{#expr-clamp .member} +**`clamp`**`(low:` [`ExprInput`](#exprinput)`, high:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-clamp){.headerlink} +: This value limited to the range from `low` to `high`. + +{#expr-lt .member} +**`lt`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-lt){.headerlink} +: 1 where this value is less than `other`, else 0. + +{#expr-le .member} +**`le`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-le){.headerlink} +: 1 where this value is at most `other`, else 0. + +{#expr-gt .member} +**`gt`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-gt){.headerlink} +: 1 where this value is greater than `other`, else 0. + +{#expr-ge .member} +**`ge`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-ge){.headerlink} +: 1 where this value is at least `other`, else 0. + +{#expr-eq .member} +**`eq`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-eq){.headerlink} +: 1 where this value equals `other`, else 0. Colors compare too. + +{#expr-ne .member} +**`ne`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-ne){.headerlink} +: 1 where this value differs from `other`, else 0. Colors compare too. + +{#expr-and .member} +**`and`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-and){.headerlink} +: 1 where this value and `other` are both above 0.5, else 0. + +{#expr-or .member} +**`or`**`(other:` [`ExprInput`](#exprinput)`): Expr` [¶](#expr-or){.headerlink} +: 1 where this value or `other` is above 0.5, else 0. + +{#expr-not .member} +**`not`**`(): Expr` [¶](#expr-not){.headerlink} +: 1 where this value is at most 0.5, else 0. + +{#expr-neg .member} +**`neg`**`(): Expr` [¶](#expr-neg){.headerlink} +: The value negated. + +{#expr-abs .member} +**`abs`**`(): Expr` [¶](#expr-abs){.headerlink} +: The absolute value. + +{#expr-sqrt .member} +**`sqrt`**`(): Expr` [¶](#expr-sqrt){.headerlink} +: The square root. Negative values give a missing value. + +{#expr-sin .member} +**`sin`**`(): Expr` [¶](#expr-sin){.headerlink} +: The sine of the value, an angle in radians. + +{#expr-cos .member} +**`cos`**`(): Expr` [¶](#expr-cos){.headerlink} +: The cosine of the value, an angle in radians. + +{#expr-floor .member} +**`floor`**`(): Expr` [¶](#expr-floor){.headerlink} +: The largest whole number at most the value. + +{#expr-ln .member} +**`ln`**`(): Expr` [¶](#expr-ln){.headerlink} +: The natural logarithm. Values at most 0 give a missing value. + +{#expr-exp .member} +**`exp`**`(): Expr` [¶](#expr-exp){.headerlink} +: e raised to the value. + +{#expr-is .member} +**`is`**`(text: string): Expr` [¶](#expr-is){.headerlink} +: Whether a text field equals `text`, plain text read as neither a field nor a color. + +{#expr-contains .member} +**`contains`**`(text: string): Expr` [¶](#expr-contains){.headerlink} +: Whether a text field contains `text` (plain text), ignoring case. In a mark, the field must be + in the mark's own table. + +{#expr-colormap .member} +**`colormap`**`(map:` [`Colormap`](#colormap)`):` [`Mapped`](#mapped) [¶](#expr-colormap){.headerlink} +: Maps the value to a color through `map`, a catalog name or custom colors. The domain fits the + values shown unless set with [`Mapped.domain()`](#mapped-domain). + +{#expr-scale .member} +**`scale`**`(low: number, high: number):` [`Mapped`](#mapped) [¶](#expr-scale){.headerlink} +: Maps the value linearly onto `[low, high]`, clamped. The domain fits the values shown unless + set with [`Mapped.domain()`](#mapped-domain). + +(exprinput)= + +### `ExprInput` + +{.signature} +`type ExprInput =` [`Expr`](#expr) `|` [`ExprNode`](#exprnode) + +An option's value: an [`Expr`](#expr) or its JSON form. In JSON, `null` is a missing value, a string +starting with `$` reads a field (`'$kv'` reads field `kv`), and any other string is a CSS color. +Color options take a value that computes a color. + +(mapped)= + +### `Mapped` + +{.signature} +`class Mapped extends` [`Expr`](#expr) + +A value mapped through a colormap or onto a range. Its input domain can be set. + +```{rubric} Methods ``` -### Mapped +{#mapped-domain .member} +**`domain`**`(low: number, high: number): Mapped` [¶](#mapped-domain){.headerlink} +: Sets the input values that map to the ends of the colormap or range. Without it, the domain + fits the values shown. -```ts -/** A value through a colormap or onto a range, whose domain can be set. */ -class Mapped extends Expr { - /** The input values that map to the ends; it fits the values shown unless set. */ - domain(low: number, high: number): Mapped; -} +(color)= + +### `Color` + +{.signature} +`type Color = string` + +A CSS color, such as `'#ffb000'` or `'rebeccapurple'`. In an expression, a string that does not +start with `$` is a color. + +(field)= + +### `field()` + +{.signature} +`function field(name: string):` [`Expr`](#expr) + +Reads field `name` of each row. In JSON it is `'$name'`, or the `{ field }` form after `from()` +or `step()`. + +(when-function)= + +### `when()` + +{.signature} +`function when(condition:` [`ExprInput`](#exprinput)`):` [`When`](#when) + +Starts a conditional. `when(c).then(a).when(d).then(b).otherwise(e)` takes the value of the first +case whose condition holds, else `e`. The values may be numbers, colors, or a text field and +`null`, whichever the option takes. + +(when)= + +### `When` + +{.signature} +`class When` + +A conditional's first condition. `then` gives its value. + +```{rubric} Methods ``` -### units +{#when-then .member} +**`then`**`(value:` [`ExprInput`](#exprinput)`):` [`Then`](#then) [¶](#when-then){.headerlink} +: Sets the value where the condition holds. -```ts -/** - * A size in position units, so it scales as the view zooms, kept within `atLeast` and `atMost` CSS - * pixels when given: `units('r', { atLeast: 2 })` grows with the map and never vanishes. A marker - * keeps its proportions, its larger side held. - */ -const units: ( - size: ExprInput, - px?: { - atLeast?: number; - atMost?: number; - }, -) => Size; -``` - -### selected +(then)= -```ts -/** Whether the row is selected. */ -const selected: () => Expr; +### `Then` + +{.signature} +`class Then` + +A conditional's cases so far. Add a case with `when`, or finish with `otherwise`. + +```{rubric} Methods ``` -### hovered +{#then-when .member} +**`when`**`(condition:` [`ExprInput`](#exprinput)`):` [`Next`](#next) [¶](#then-when){.headerlink} +: Adds a case, tried when no earlier condition holds. -```ts -/** Whether the pointer is over the row. */ -const hovered: () => Expr; +{#then-otherwise .member} +**`otherwise`**`(value:` [`ExprInput`](#exprinput)`):` [`Expr`](#expr) [¶](#then-otherwise){.headerlink} +: Sets the value where no condition holds, and returns the finished conditional. + +(next)= + +### `Next` + +{.signature} +`class Next` + +A conditional's next condition. `then` gives its value. + +```{rubric} Methods ``` -### pointer +{#next-then .member} +**`then`**`(value:` [`ExprInput`](#exprinput)`):` [`Then`](#then) [¶](#next-then){.headerlink} +: Sets the value where this condition holds. + +(px-function)= + +### `px()` + +{.signature} +`function px(size: number): { px: number }` + +{.signature} +`function px(size:` [`ExprInput`](#exprinput)`):` [`Px`](#px) + +A size in CSS pixels in any space, so it stays the same on screen as the view zooms. `px(1)` +keeps a diagram's stroke 1 pixel wide at any zoom. Given a number, the result also works as a +theme length. + +(units-function)= + +### `units()` + +{.signature} +`function units(size: number): { units: number }` + +{.signature} +`function units(size:` [`ExprInput`](#exprinput)`, limits?: Omit<`[`Units`](#units)`, 'units'>):` [`Units`](#units) + +A size in position units (degrees on the earth) in any space, so it scales as the view zooms. +`limits` bound its size on screen in CSS pixels: `units('$r', { atLeast: 2 })` grows with the +map and never shrinks below 2 pixels. A marker keeps its proportions, and the limits apply to its +larger side. Given only a number, the result also works as a theme length. + +(selected)= + +### `selected()` + +{.signature} +`function selected():` [`Expr`](#expr) + +Whether the row is selected. + +(hovered)= + +### `hovered()` + +{.signature} +`function hovered():` [`Expr`](#expr) + +Whether the pointer is over the row. + +(pointer)= + +### `pointer()` + +{.signature} +`function pointer():` [`Expr`](#expr) + +CSS pixels from a vertex to the pointer. Vertex marks only, and not in a vertex's position. + +(zoom-function)= + +### `zoom()` + +{.signature} +`function zoom():` [`Expr`](#expr) + +The camera's zoom: the web mercator level on the earth, log₂ CSS pixels per unit elsewhere. + +(elapsed)= + +### `elapsed()` + +{.signature} +`function elapsed():` [`Expr`](#expr) + +Seconds since the view was created. A mark that reads it redraws every frame. + +(icon)= + +### `icon()` + +{.signature} +`function icon(images: readonly` [`RgbaImage`](#rgbaimage)`[], index:` [`ExprInput`](#exprinput)`):` [`Marker`](#marker) + +An icon marker: each row shows `images[index]`, fitted in the marker's box, with `index` counting +from 0. The pixels are copied into the marker. + +(rgbaimage)= + +### `RgbaImage` + +{.signature} +`interface RgbaImage` + +RGBA pixels with straight alpha, rows top to bottom. A browser `ImageData` has this shape. + +{#rgbaimage-width .member} +**`width`**`: number` [¶](#rgbaimage-width){.headerlink} +: The width in pixels. + +{#rgbaimage-height .member} +**`height`**`: number` [¶](#rgbaimage-height){.headerlink} +: The height in pixels. + +{#rgbaimage-data .member} +**`data`**`: Uint8ClampedArray` [¶](#rgbaimage-data){.headerlink} +: 4 bytes per pixel, red, green, blue, and alpha, row after row. + +(colormaps)= + +### `colormaps()` + +{.signature} +`function colormaps(): { name:` [`ColormapName`](#colormapname)`; label: string; kind:` [`ColormapKind`](#colormapkind) `}[]` + +Returns the catalog's colormaps in order, each with its name, its label such as `'Red / Blue'`, +and its kind. + +(colormapcss)= + +### `colormapCss()` + +{.signature} +`function colormapCss(map:` [`Colormap`](#colormap)`, direction?: string): string` + +Returns a CSS `linear-gradient(...)` with the colors the GPU draws for `map`, a catalog name or a +custom colormap. `direction` is the CSS gradient direction and defaults to `'to right'`. A +categorical map gives one solid band per category. + +## Views + +Views, which draw scenes on canvases, and the GPU they draw with. + +(view)= + +### `View` + +{.signature} +`class View extends` [`Handle`](#handle) + +A scene drawn from its datasets on a canvas and into pictures. Create one with +[`View.create()`](#view-create). The canvas passes it pointer, wheel, and key input, and a `Shot` draws it +into pictures. Destroying a view rejects its unanswered queries, and its canvas keeps the last +frame. ```ts -/** CSS pixels from a vertex to the pointer. */ -const pointer: () => Expr; +const grid = new Data('grid').set('Bus', [1, 2], { x: [0, 30], y: [0, 20], kv: [230, 115] }); +const view = await View.create(canvas, { + space: 'plane', + marks: [{ vertices: grid.table('Bus'), position: ['$x', '$y'], color: field('kv').colormap('viridis') }], +}); ``` -### zoom +```{rubric} Properties +``` -```ts -/** The camera's zoom: the web mercator level on the earth, log₂ CSS pixels per unit elsewhere. */ -const zoom: () => Expr; +{#view-gpu .member} +`readonly` **`gpu`**`:` [`Gpu`](#gpu) [¶](#view-gpu){.headerlink} +: The GPU that draws the view. + +{#view-region .member} +**`region`**`:` [`Rect`](#rect) [¶](#view-region){.headerlink} +: The part of its canvas, and of pictures, that the view draws in: `[x, y, width, height]` in + fractions, from the top left. Views on one canvas draw in the order they were created. + +{#view-scene .member} +**`scene`**`:` [`Scene`](#scene) [¶](#view-scene){.headerlink} +: The scene drawn. A mark that matches one in the previous scene by kind, dataset, table, and + name keeps its state and eases to its new options. An invalid scene leaves the current one in + place. + +{#view-sources .member} +**`sources`**`: readonly` [`Source`](#source)`[]` [¶](#view-sources){.headerlink} +: Datasets the view draws besides those its scene uses. A source replaces a scene dataset with + the same name. A mark whose table is written `[data, table]` draws the source named `data`. The + view draws a [`Data`](#data) at its latest version from the frame after each change, a [`Snapshot`](#snapshot) as it + is, and [`Tiles`](#tiles) as the pieces it needs. A mark whose dataset no longer fits it draws nothing. + It is listed in `failures`, and the next frame emits an `error` event. + +{#view-at .member} +**`at`**`: number` [¶](#view-at){.headerlink} +: The axis coordinate shown, such as a time. Marks read sampled fields at it, interpolating + between the frame at or before it and the next, unless an expression uses `step()`. + +{#view-camera .member} +**`camera`**`:` [`Camera`](#camera) [¶](#view-camera){.headerlink} +: Where the view looks. Setting it moves the camera at once, replacing a flight or a fit not yet + applied. Use [`View.fly()`](#view-fly) to ease. A camera of the wrong kind for the space, such as a plot + camera on a map, is ignored. + + Camera moves stay within the camera's zoom and pan limits. On the earth, the space sets the + limits inside the insets, as [`Earth`](#earth) describes. A fit narrows them to 3 zoom levels out and 8 + in from the box it framed, with the center near that box. When a resize, new insets, or a new + space moves the camera back within its limits, a `camera` event reports it. + +{#view-insets .member} +**`insets`**`:` [`Insets`](#insets) [¶](#view-insets){.headerlink} +: CSS pixels kept clear at the top, right, bottom, and left, such as where page panels cover the + view. The camera centers and fits within the rest. + +{#view-orbit .member} +**`orbit`**`: number` [¶](#view-orbit){.headerlink} +: Degrees per second the camera turns about its center on a map or plane. 0 stops it. A diagram + never turns. + +{#view-selection .member} +**`selection`**`: readonly` [`Rows`](#rows)`[]` [¶](#view-selection){.headerlink} +: The selected rows, one set per table. Setting it selects exactly those rows. + +{#view-hovered .member} +`readonly` **`hovered`**`:` [`Hit`](#hit) `| null` [¶](#view-hovered){.headerlink} +: The row under the pointer or the row keyboard navigation moved to, or `null`. + +{#view-waiting .member} +`readonly` **`waiting`**`:` [`Waiting`](#waiting) [¶](#view-waiting){.headerlink} +: The kinds of pending work that kept the last frame from being complete. Empty when it was + complete. See [`Waiting`](#waiting). + +{#view-complete .member} +`readonly` **`complete`**`: boolean` [¶](#view-complete){.headerlink} +: Whether the last frame was complete, with nothing in `waiting`. Shots wait for this. + +{#view-failures .member} +`readonly` **`failures`**`: readonly` [`LatkitError`](#latkiterror)`[]` [¶](#view-failures){.headerlink} +: An error for each mark that draws nothing because its data no longer fits it, such as a missing + table. Each error's `path` names the mark. + +{#view-stats .member} +`readonly` **`stats`**`:` [`Stats`](#stats) [¶](#view-stats){.headerlink} +: Counts and timings of the view's frames; see [`Stats`](#stats). + +{#view-wantsframe .member} +`readonly` **`wantsFrame`**`: boolean` [¶](#view-wantsframe){.headerlink} +: Whether the view needs a frame, because it changed or work it was waiting on arrived. + +```{rubric} Methods ``` -### elapsed +{#view-create .member} +`static` **`create`**`(canvas: HTMLCanvasElement | OffscreenCanvas | null, scene:` [`Scene`](#scene)`, options?: { region?:` [`Rect`](#rect)`; camera?:` [`Camera`](#camera)`; sources?: readonly` [`Source`](#source)`[]; gpu?:` [`Gpu`](#gpu)`; pixelRatio?: number }): Promise` [¶](#view-create){.headerlink} +: Creates a view that draws `scene` on `canvas`, in `options.region`, over any views already + there. With a `null` canvas, the view draws only into pictures. The view loads every dataset + its scene uses. Unless given a camera, it frames everything when it first draws. + + {#view-create-region .member} + **`region`**`?:` [`Rect`](#rect) [¶](#view-create-region){.headerlink} + : The part of the canvas, and of pictures, that the view draws in. Defaults to all of it. + + {#view-create-camera .member} + **`camera`**`?:` [`Camera`](#camera) [¶](#view-create-camera){.headerlink} + : Where the view looks first. Without one, it frames everything drawn. + + {#view-create-sources .member} + **`sources`**`?: readonly` [`Source`](#source)`[]` [¶](#view-create-sources){.headerlink} + : Datasets to draw besides those the scene uses. See [`View.sources`](#view-sources). -```ts -/** Seconds since the view was created. */ -const elapsed: () => Expr; + {#view-create-gpu .member} + **`gpu`**`?:` [`Gpu`](#gpu) [¶](#view-create-gpu){.headerlink} + : The GPU that draws the view. Defaults to the first [`Gpu`](#gpu) created, creating one if needed. + + {#view-create-pixelratio .member} + **`pixelRatio`**`?: number` [¶](#view-create-pixelratio){.headerlink} + : Pixels per CSS pixel of an `OffscreenCanvas`. Defaults to 1. The first view on the canvas + sets it. + +{#view-fit .member} +**`fit`**`(fit?:` [`Fit`](#fit)`): void` [¶](#view-fit){.headerlink} +: Frames what `fit` names, keeping pitch and bearing. Defaults to every row. It takes effect on + the next frame, or a frame or two later when the GPU must measure the extent. + +{#view-fly .member} +**`fly`**`(camera:` [`Camera`](#camera)`, seconds?: number): void` [¶](#view-fly){.headerlink} +: Eases the camera to `camera` over `seconds`, or over twice the scene's ease time without it, + replacing a fit not yet applied. When motion is reduced, it moves at once. The camera stays + within its limits, as [`View.camera`](#view-camera) describes. A camera of the wrong kind for the space is + ignored. + +{#view-screen .member} +**`screen`**`(position:` [`Position`](#position)`):` [`Point`](#point) `| null` [¶](#view-screen){.headerlink} +: Projects `position` to a point on screen with the current camera, for anchoring HTML overlays. + Returns `null` when the position is behind the camera. + +{#view-position .member} +**`position`**`(point:` [`Point`](#point)`):` [`Position`](#position) `| null` [¶](#view-position){.headerlink} +: Returns the position on the ground under screen point `point`: at elevation 0 on a plane, on + the map or globe on the earth, and the values there on a plot. Returns `null` where the point + looks past the ground. + +{#view-screenof .member} +**`screenOf`**`(row:` [`Row`](#row)`, wait?:` [`Wait`](#wait)`): Promise<`[`Point`](#point) `| null>` [¶](#view-screenof){.headerlink} +: Resolves to where `row` is on screen once a frame draws the view, or to `null` when no mark + draws it in view. A diagram's vertices resolve at once. + +{#view-pick .member} +**`pick`**`(point:` [`Point`](#point)`, options?: { radius?: number } &` [`Wait`](#wait)`): Promise<`[`Hit`](#hit)`[]>` [¶](#view-pick){.headerlink} +: Resolves, once a frame draws the view, to every row of a pickable mark near `point`, nearest + first. + + {#view-pick-radius .member} + **`radius`**`?: number` [¶](#view-pick-radius){.headerlink} + : How near, in CSS pixels. Defaults to 0. + +{#view-neighbors .member} +**`neighbors`**`(row:` [`Row`](#row)`):` [`Rows`](#rows)`[]` [¶](#view-neighbors){.headerlink} +: Returns the rows connected to `row`, one set per table: a vertex's neighbors across edges and + nets, or the vertices of an edge or net. The cost is proportional to the row's degree. + +{#view-arrange .member} +**`arrange`**`(wait?:` [`Wait`](#wait)`): Promise<`[`Patch`](#patch)`[]>` [¶](#view-arrange){.headerlink} +: Lays out again every vertex row that has no position, and resolves to one patch per dataset. + Each patch writes each row's place into its mark's position fields, so applying it makes the + places part of the data. Marks that don't read their position from fields are left out. + + ```ts + for (const patch of await view.arrange()) if (patch.data === grid.name) grid.apply(patch); + ``` + +{#view-snapshot .member} +**`snapshot`**`(name: string):` [`Snapshot`](#snapshot) `| undefined` [¶](#view-snapshot){.headerlink} +: Returns the version of dataset `name` that the view draws, or `undefined` if it draws no + dataset by that name. That is the latest version of a [`Data`](#data) the view has taken, the snapshot + it was given, or the pieces of [`Tiles`](#tiles) fetched for its camera. + +{#view-on .member} +**`on`**`(type: T, listener: (event:` [`ViewEvents`](#viewevents)`[T]) => void): () => void` [¶](#view-on){.headerlink} +: Calls `listener` with each event of `type` emitted while it is registered. Returns a function + that removes it. Past events are not replayed. Read the current state from the property that a + change event is named after. + +(rect)= + +### `Rect` + +{.signature} +`type Rect = readonly [x: number, y: number, width: number, height: number]` + +A region `[x, y, width, height]` in fractions of a canvas or picture, from its top left. + +(viewevents)= + +### `ViewEvents` + +{.signature} +`type ViewEvents = { [E in` [`ViewEvent`](#viewevent) `as E['type']]: E extends { type: 'error' } ?` [`LatkitError`](#latkiterror) `: E['detail'] }` + +The events a view emits, mapping each type to its detail. Intents (`click`, `open`, `menu`, +`drag`, `connect`, `delete`) are emitted once each, in the order the user acted. Changes +(`hovered`, `selection`, `camera`, `waiting`) are named after the property that changed and carry +only its latest value. An `error` event's detail is a [`LatkitError`](#latkiterror). Setters emit nothing, +so events report only what the application did not do itself. + +(insets)= + +### `Insets` + +{.signature} +`type Insets = [top: number, right: number, bottom: number, left: number]` + +CSS pixels kept clear at the top, right, bottom, and left. + +(point)= + +### `Point` + +{.signature} +`type Point = [x: number, y: number]` + +A point on screen, in CSS pixels from the view's top left corner. + +(position)= + +### `Position` + +{.signature} +`type Position = [x: number, y: number, elevation: number]` + +A position in the space's coordinates: x, y, and elevation. On the earth, x and y are longitude +and latitude; on a plot, the axis coordinate and the value. + +(gpu)= + +### `Gpu` + +{.signature} +`class Gpu extends` [`Handle`](#handle) + +A WebGPU device that views draw with. Views on one `Gpu` share its memory and draw together in +each animation frame, so linked views always show the same moment. Views use the first `Gpu` +created unless given another, and [`View.create`](#view-create) creates one if none exists. Destroying a `Gpu` +destroys its views and frame listeners. Other GPUs and datasets stay usable. + +```{rubric} Properties ``` -### icon +{#gpu-lost .member} +`readonly` **`lost`**`: boolean` [¶](#gpu-lost){.headerlink} +: Whether the device was lost. Drawing then does nothing, so create a new [`Gpu`](#gpu). -```ts -/** An icon marker showing one of `images` for each row, picked by `index`. */ -function icon(images: readonly RgbaImage[], index: ExprInput): Marker; +{#gpu-budget .member} +**`budget`**`:` [`Budget`](#budget) [¶](#gpu-budget){.headerlink} +: The GPU memory budget for data, shared by every view on this GPU. See [`Budget`](#budget). Defaults + to half the device's largest storage buffer, with 32 MiB uploaded per frame. + +{#gpu-work .member} +`readonly` **`work`**`:` [`Work`](#work) [¶](#gpu-work){.headerlink} +: Exact counts of the work this GPU has done since it was created. See [`Work`](#work). The same + budget, device limits, and frames give the same counts, so tests and benchmarks can assert + them. + +{#gpu-resident .member} +`readonly` **`resident`**`: number` [¶](#gpu-resident){.headerlink} +: Bytes of GPU memory the views draw from: data pages and each view's per-row data. + +{#gpu-uploading .member} +`readonly` **`uploading`**`: number` [¶](#gpu-uploading){.headerlink} +: Bytes waiting to upload to the GPU, across all its views. + +```{rubric} Methods ``` -### RgbaImage +{#gpu-create .member} +`static` **`create`**`(): Promise` [¶](#gpu-create){.headerlink} +: Requests a WebGPU device. The first call takes the device [`load()`](#load) requested. Create one before + any view to set its `budget` first, or create another for views that should not share memory + with the rest. Rejects with an `unsupported` error when the browser has no WebGPU device. + +{#gpu-on .member} +**`on`**`(type: 'frame', listener: (now: number) => void): () => void` [¶](#gpu-on){.headerlink} +: Calls `listener` every animation frame with the frame's timestamp in milliseconds. It runs + before the views draw, so its changes show in that frame. While any listener is registered, a + frame runs on every animation frame. Returns a function that removes the listener. + +{#gpu-step .member} +**`step`**`(now: number): void` [¶](#gpu-step){.headerlink} +: Draws one frame now, at `now` milliseconds on the `requestAnimationFrame` clock. It runs the + frame listeners, then draws every canvas that needs a frame. After the first call the GPU + stops requesting animation frames, so a test or benchmark that steps every frame draws the + same frames on any device. + +{#gpu-done .member} +**`done`**`(wait?:` [`Wait`](#wait)`): Promise` [¶](#gpu-done){.headerlink} +: Resolves once the GPU has finished its submitted work, every readback has arrived, and every + pipeline a view or shot requested has compiled. A frame stepped after that uses all of them. + +{#gpu-landed .member} +**`landed`**`(wait?:` [`Wait`](#wait)`): Promise` [¶](#gpu-landed){.headerlink} +: Resolves when work a view waits on arrives after this call (a readback, a pipeline compile, a + job, a label text, or a fetch), or when a view requests a frame. A harness that steps frames + awaits it to know when to step again. + +## Pictures and video + +Shots draw views into images and video. + +(shot)= + +### `Shot` + +{.signature} +`class Shot` + +A picture of views, `size` pixels in all, with each view drawn in its region. While a shot draws +a view, the view's canvas keeps its last frame. An image draws each view's data as it was when +the picture began. `Shot.of(view)` makes a shot of what a view's canvas shows. ```ts -/** Straight-alpha RGBA pixels, compatible with browser ImageData and server-produced pixels. */ -interface RgbaImage { - width: number; - height: number; - data: Uint8ClampedArray; -} +const png = await (await Shot.of(view, { scale: 2 }).image()).encode('png'); ``` -### colormapCss +```{rubric} Constructor +``` -```ts -/** - * A CSS gradient through `map`, a catalog name or colors of your own, for legends: the colors the - * GPU draws, a band each for a categorical map's categories. - */ -function colormapCss(map: Colormap, direction?: string): string; +{#shot-new .member} +**`new Shot`**`(size: readonly [number, number], options?:` [`ShotOptions`](#shotoptions) `& { views?: readonly` [`Placed`](#placed)`[] })` [¶](#shot-new){.headerlink} +: Creates a shot of `size`, `[width, height]` in pixels. + + {#shot-new-views .member} + **`views`**`?: readonly` [`Placed`](#placed)`[]` [¶](#shot-new-views){.headerlink} + : The views drawn, later views over earlier ones. Each draws in its own region, or in + `region` when given as `{ view, region }`. + +```{rubric} Properties ``` -## Views +{#shot-size .member} +`readonly` **`size`**`: readonly [number, number]` [¶](#shot-size){.headerlink} +: The picture's size, `[width, height]` in pixels. -A GPU makes views, and canvases that show them. +{#shot-options .member} +`readonly` **`options`**`:` [`ShotOptions`](#shotoptions) `& { views?: readonly` [`Placed`](#placed)`[] }` [¶](#shot-options){.headerlink} +: How the picture is taken, and in `views`, the views drawn into it. -### Gpu +```{rubric} Methods +``` -```ts -/** The GPU device views draw with. Views of one GPU draw in one animation frame. */ -class Gpu { - /** Requests a WebGPU device: the first call takes the one made as latkit loaded. */ - static create(): Promise; - /** - * A view of `scene`; its marks draw once {@link View.setData} gives their data. A {@link Canvas} - * shows it, and a {@link Shot} draws it with or without one. - */ - view(scene: Scene): View; - /** - * A canvas showing views in `panels` of `target`: a `` element, which sizes itself to its - * CSS box and passes each view the input landing on its panel, or an `OffscreenCanvas` in a - * worker, which keeps the size it is given at `pixelRatio` pixels a CSS pixel. Panels are - * `[x, y, width, height]` fractions, one per view; by default one view fills the canvas. - */ - canvas( - target: HTMLCanvasElement | OffscreenCanvas, - options?: { - panels?: readonly Panel[]; - pixelRatio?: number; - }, - ): Canvas; - /** - * Calls `listener` with the frame's timestamp every animation frame, before the views draw, so - * what it changes shows in that frame. Returns a function that stops it. - */ - on(type: 'frame', listener: (now: number) => void): () => void; - /** Whether the device was lost; a new `Gpu` is needed. */ - get lost(): boolean; - /** How much GPU memory data takes. */ - get budget(): Budget; - set budget(budget: Budget); - /** What this GPU did since it was created, counted exactly; see {@link Work}. */ - get work(): Work; - /** - * Draws one frame now at `now`, milliseconds on the clock animation frames give: the frame - * listeners, then every canvas that needs a frame. From the first step no animation frames are - * asked for: a harness, as tests and benchmarks are, steps every frame itself, so a run draws the - * same frames however fast the device is. - */ - step(now: number): void; - /** - * Resolves once the GPU finished the work submitted to it, every readback landed, and every - * pipeline a view or shot asked for compiled: a frame stepped then draws all of it. A `timeout` - * error after `deadline` seconds. - */ - done({ deadline: seconds }?: { deadline?: number | undefined }): Promise; - /** - * Resolves when work a view waits on lands after this call (a readback, a compile, a job, a text, - * a fetch), or a view asks for a frame. A `timeout` error after `deadline` seconds. - */ - landed({ deadline: seconds }?: { deadline?: number | undefined }): Promise; - /** Releases this GPU's canvases, views, and frame listeners. Other GPUs and data stay usable. */ - destroy(): void; - [Symbol.dispose](): void; -} -``` - -### View - -````ts -/** A scene drawn from data it is given by name: shown by a canvas, which passes it input, or drawn by shots. */ -class View { - /** Draws `scene` instead; an invalid scene throws and changes nothing. */ - show(scene: Scene): void; - /** - * Gives the view `data` as it is now, replacing the dataset of its name: give it again after - * applying patches. A mark it no longer fits draws nothing, and the next frame reports why as an - * error, unless a scene shown before then replaces the mark. - */ - setData(data: Data): void; - /** - * Fetches the data at `url` in pieces as the view asks for them, kept as dataset `name`, which the - * view owns; the source's first bytes say what it is. - * - * - Time tiles, a `frames.json` manifest as `latkit tile` writes it: a dataset whose sampled - * fields' histories are fetched in spans. A line tracing a field fetches the detail its plot - * window draws, a mark showing one the frames about the view's coordinate, and a span still - * loading draws the coarser span above it. - * - Vector tiles: a PMTiles archive, a TileJSON document, or a `{z}/{x}/{y}` template, fetched as - * the camera asks. Each layer is a table of its features of the kind marks draw it as (areas - * polygons, lines lines, vertices points), or named `layer:polygons`, `layer:lines`, or - * `layer:points`. - */ - setTiles(name: string, url: string): void; - /** Stops drawing the dataset named `name`, given or fetched as tiles. */ - removeData(name: string): void; - /** The dataset named `name`, as given or as the view fetched it, as a snapshot. */ - data(name: string): Data | undefined; - /** The axis coordinate shown: sampled fields read their last frame at or before it. */ - get at(): number; - set at(at: number); - get camera(): Camera; - set camera(camera: Camera); - /** Eases to `camera`. */ - fly(camera: Camera): void; - /** Frames what `fit` names on the next frame, keeping pitch and bearing. */ - fit(fit?: Fit): void; - /** Where `position` (x, y, elevation) lands on screen, CSS pixels from the top left. */ - point(position: [number, number, number]): [number, number] | undefined; - /** The position on the ground under screen point `point`, if the view shows ground there. */ - ground(point: [number, number]): [number, number, number] | undefined; - /** CSS pixels kept clear at the top, right, bottom, and left, as when panels cover the view. */ - get insets(): [number, number, number, number]; - set insets(insets: [number, number, number, number]); - /** Degrees per second a map or plane turns around its center; 0 stops it. A diagram never turns. */ - get orbit(): number; - set orbit(degreesPerSecond: number); - /** The rows selected, a set per table. */ - get selection(): Rows[]; - /** - * Selects `rows`, and no others. Like every setter it tells nothing: `select` events tell what - * the user and the data changed. - */ - select(rows: readonly Rows[]): void; - /** What the user points at: the row under the pointer, or the row the keys stepped to. */ - get hovered(): Hit | null; - /** Every mark that draws nothing because its data no longer fits it, and why. */ - get failures(): LatkitError[]; - /** - * Every row of a mark that picks within `radius` CSS pixels of `point`, nearest first, once a frame - * draws the view: a `timeout` error after `deadline` seconds without one. - */ - pick( - point: [number, number], - radius: number, - { - deadline: seconds, - }?: { - deadline?: number | undefined; - }, - ): Promise; - /** - * Where `row` lies on screen, CSS pixels from the top left, or `null` when no mark draws it in - * view, once a frame draws the view: a `timeout` error after `deadline` seconds without one. - */ - locate( - row: Row, - { - deadline: seconds, - }?: { - deadline?: number | undefined; - }, - ): Promise<[number, number] | null>; - /** The rows joined to `row`, a set per table: a vertex's neighbors across edges and nets, or an - * edge's or a net's vertices. */ - neighbors(row: Row): Rows[]; - /** - * Lays every vertex row without a position out again by the scene's layout: a diagram's blocks, or - * points on a plane or the earth. Places a layout gave otherwise hold, so rows added later are - * placed around the rest. Once the layout lands, answers a patch to each dataset writing each row - * laid out into its mark's position fields; applied, the places are the data's own. A `timeout` - * error after `deadline` seconds without it. - * - * ```ts - * for (const patch of await view.arrange()) if (patch.data === data.name) data.apply(patch); - * ``` - */ - arrange({ deadline: seconds }?: { deadline?: number | undefined }): Promise; - get stats(): Stats; - /** Whether the view wants a frame: it changed, or work it waited on landed. */ - get wantsFrame(): boolean; - /** - * Calls `listener` with each event of `type` told while it listens; returns a function that - * stops it. What lasts is state to read: `hovered`, `selection`, `camera`, `stats.waiting`, and - * `failures`. - */ - on(type: T, listener: (event: ViewEvents[T]) => void): () => void; - /** Frees the view, and rejects what it was asked and has not answered; a canvas showing it keeps - * its last frame. */ - destroy(): void; - [Symbol.dispose](): void; -} -```` - -### ViewEvents +{#shot-of .member} +`static` **`of`**`(view:` [`View`](#view)`, options?:` [`ShotOptions`](#shotoptions)`): Shot` [¶](#shot-of){.headerlink} +: Returns a shot of exactly what the canvas showing `view` shows: its views in their regions, at + the canvas's CSS size times `options.scale` in pixels. Throws an `invalid` error for a view on + no canvas. Give such a shot a size with the constructor. -```ts -/** - * What a view tells its listeners, by type. Intents (`click`, `open`, `menu`, `drag`, `connect`, - * `delete`) are told once each, in the order the user acted. Changes (`hover`, `select`, `camera`, - * `waiting`) tell only their latest, and their state can always be read. A view tells only what the - * application did not do itself: its setters change state silently. - */ -type ViewEvents = { - [E in ViewEvent as E['type']]: E extends { - type: 'error'; - } - ? LatkitError - : E['detail']; -}; -``` - -### Canvas +{#shot-image .member} +**`image`**`(wait?:` [`Wait`](#wait)`): Promise<`[`Image`](#image)`>` [¶](#shot-image){.headerlink} +: Draws the picture once every view is complete, as `capture` says, and resolves to the + [`Image`](#image). Rejects with the first error a view reports while drawing. -```ts -/** Views shown in the panels of a `` or an `OffscreenCanvas`. */ -class Canvas { - /** Shows `views`, one per panel, in place of those it showed. A view shows in one canvas at a time. */ - show(views: readonly View[]): this; - /** Stops drawing; the views remain, and the element keeps its last frame. */ - destroy(): void; - [Symbol.dispose](): void; -} +{#shot-video .member} +**`video`**`(options: { fps: number; output: WritableStream; bitrate?: number } &` [`Wait`](#wait)`):` [`Video`](#video) [¶](#shot-video){.headerlink} +: Starts a video of this shot: H.264 in MP4, written to `options.output` as frames are drawn. The + width is rounded down to a multiple of 4 and the height to an even number, as the encoder + requires. Canvases showing the views keep their last frame until the video finishes or aborts. + Aborting `options.signal` aborts the video and its output. Frames have no other time limit. + + {#shot-video-fps .member} + **`fps`**`: number` [¶](#shot-video-fps){.headerlink} + : Frames per second. + + {#shot-video-output .member} + **`output`**`: WritableStream` [¶](#shot-video-output){.headerlink} + : Where the file's bytes go, locked until the video finishes or aborts. + + {#shot-video-bitrate .member} + **`bitrate`**`?: number` [¶](#shot-video-bitrate){.headerlink} + : Bits per second. Defaults to about 0.15 bits per pixel per frame. + +(shotoptions)= + +### `ShotOptions` + +{.signature} +`interface ShotOptions` + +Options for a [`Shot`](#shot). + +{#shotoptions-scale .member} +**`scale`**`?: number` [¶](#shotoptions-scale){.headerlink} +: Pixels per CSS pixel. Defaults to 1. Views lay out at `size / scale` CSS pixels, so 2 draws the + same layout twice as sharp. + +{#shotoptions-transparent .member} +**`transparent`**`?: boolean` [¶](#shotoptions-transparent){.headerlink} +: Leaves the background transparent where the theme's background color would be. + +{#shotoptions-capture .member} +**`capture`**`?: 'still' | 'now'` [¶](#shotoptions-capture){.headerlink} +: When the picture is taken. `'still'`, the default, waits until every view is complete and + still: transitions, camera flights, and tile fades run until they finish. `'now'` waits only + until every view is complete, and freezes anything moving where it is. + +{#shotoptions-elapsed .member} +**`elapsed`**`?: number` [¶](#shotoptions-elapsed){.headerlink} +: The time in seconds that [`elapsed()`](#elapsed) expressions and flowing comets show in a still picture. + Defaults to the view's clock. + +(placed)= + +### `Placed` + +{.signature} +`type Placed =` [`View`](#view) `| { view:` [`View`](#view)`; region:` [`Rect`](#rect) `}` + +A view drawn into a picture, in its own region or in `region`. + +{#placed-view .member} +`readonly` **`view`**`:` [`View`](#view) [¶](#placed-view){.headerlink} +: The view drawn. + +{#placed-region .member} +`readonly` **`region`**`:` [`Rect`](#rect) [¶](#placed-region){.headerlink} +: Where it draws in the picture, overriding its own region. + +(image)= + +### `Image` + +{.signature} +`class Image extends` [`Handle`](#handle) + +A picture's RGBA pixels, with straight alpha, rows top to bottom. + +```{rubric} Properties ``` -### Panel +{#image-width .member} +`readonly` **`width`**`: number` [¶](#image-width){.headerlink} +: Width in pixels. -```ts -/** A view's panel of a canvas or a picture: `[x, y, width, height]` as fractions of it. */ -type Panel = readonly [number, number, number, number]; +{#image-height .member} +`readonly` **`height`**`: number` [¶](#image-height){.headerlink} +: Height in pixels. + +{#image-rgba .member} +`readonly` **`rgba`**`: Uint8Array` [¶](#image-rgba){.headerlink} +: A copy of the pixels: 4 bytes per pixel, `width * height * 4` bytes in all. + +```{rubric} Methods ``` -## Pictures and video +{#image-encode .member} +**`encode`**`(format:` [`ImageFormat`](#imageformat)`, options?: { quality?: number } &` [`Wait`](#wait)`): Promise` [¶](#image-encode){.headerlink} +: Encodes the image as a file, in a worker. PNG is lossless with alpha. JPEG drops alpha. WebP + is lossless with alpha and smaller than PNG. -Shots draw views into images and video. + {#image-encode-quality .member} + **`quality`**`?: number` [¶](#image-encode-quality){.headerlink} + : JPEG quality, from 1 (smallest) to 100 (sharpest). Defaults to 90. -### Shot - -````ts -/** - * A picture of views, each drawn into its panel of `size` pixels at `scale` pixels per CSS pixel: - * plain data, like a scene. A view a shot draws holds its canvas's last frame meanwhile. - * - * ```ts - * const png = await (await new Shot([3840, 2160]).scale(2).image([view])).encode('png'); - * ``` - */ -class Shot { - readonly size: readonly [number, number]; - /** A picture with width and height in pixels. */ - constructor(size: readonly [number, number]); - /** Draws `scale` pixels per CSS pixel: a view lays out `size / scale` CSS pixels. */ - scale(scale: number): Shot; - /** Where each view draws, `[x, y, width, height]` as fractions, one per view in order; by default one fills it. */ - panels(panels: readonly Panel[]): Shot; - /** Keeps the picture's alpha: views clear to transparent where their themes' backgrounds are. */ - transparent(): Shot; - /** The shot's deadline in seconds, including readback; rejects on timeout. Defaults to 30. */ - deadline(seconds: number): Shot; - /** Fixes elapsed-time expressions and drawing effects at `seconds` while transitions settle. */ - elapsed(seconds: number): Shot; - /** The picture of `views`, one per panel, once every one is complete and still. */ - image(views: readonly View[]): Promise; - /** - * A video of these pictures at `fps` frames a second, H.264 in MP4, written to `output` as it is - * drawn, at `bitrate` bits a second or about 0.15 bits a pixel a frame. Width and height are - * multiples of 4 and 2. `signal` aborts it: the output is aborted and the views are given back. - */ - video( - gpu: Gpu, - fps: number, - output: WritableStream, - options?: { - bitrate?: number; - signal?: AbortSignal; - }, - ): Video; -} -```` - -### Image +(imageformat)= -```ts -/** RGBA pixels, straight alpha, rows top to bottom. */ -class Image { - get width(): number; - get height(): number; - /** The pixels, copied. */ - get rgba(): Uint8Array; - /** - * The image as a file, written in a worker: PNG, lossless with alpha; JPEG at `quality` from 1 to - * 100 (90 by default), small, its alpha dropped; or WebP, lossless with alpha, smaller than PNG. - */ - encode( - format: ImageFormat, - options?: { - quality?: number; - }, - ): Promise; - destroy(): void; - [Symbol.dispose](): void; -} -``` - -### ImageFormat +### `ImageFormat` -```ts -/** How an image is written as a file. */ -type ImageFormat = 'png' | 'jpeg' | 'webp'; +{.signature} +`type ImageFormat = 'png' | 'jpeg' | 'webp'` + +An image file format for [`Image.encode()`](#image-encode). + +(video)= + +### `Video` + +{.signature} +`class Video` + +A video being drawn, encoded, and written. Create one with [`Shot.video()`](#shot-video). Draw frames with +[`Video.frame()`](#video-frame) or [`Video.record()`](#video-record), then call [`Video.finish()`](#video-finish). + +```{rubric} Properties ``` -### Video +{#video-time .member} +`readonly` **`time`**`: number` [¶](#video-time){.headerlink} +: The time in seconds at which the next frame shows. Still readable after the video ends. -```ts -/** A video being drawn, encoded, and written. */ -class Video { - /** Seconds into the video the next frame shows at; retained after completion. */ - get time(): number; - /** Frames drawn or held so far; retained after completion. */ - get frames(): number; - /** Draws and writes one frame, waiting for output backpressure. Only one frame may run at a time. */ - frame(views: readonly View[]): Promise; - /** Records until `step(time)` returns false; any step, drawing, output, or progress error aborts it. */ - record( - views: readonly View[], - step: (time: number) => boolean | Promise, - onProgress?: (progress: { frames: number; time: number }) => void, - ): Promise; - /** Finishes pending work, writes the end of the file, and closes and unlocks the output. Idempotent. */ - finish(): Promise; - /** Cancels drawing and encoding, aborts the output, and unlocks it after pending writes settle. Idempotent. */ - abort(reason?: unknown): Promise; -} +{#video-frames .member} +`readonly` **`frames`**`: number` [¶](#video-frames){.headerlink} +: Frames recorded so far, including frames repeated because nothing moved. Still readable after + the video ends. + +```{rubric} Methods ``` + +{#video-frame .member} +**`frame`**`(): Promise` [¶](#video-frame){.headerlink} +: Draws the next frame once its views are complete, then writes its bytes, waiting on the + output's backpressure. Calls must not overlap. A call made before the previous one resolves, + or after `finish()`, rejects with an `invalid` error. A call after `abort()` rejects with a + `cancelled` error. A failure aborts the video. + +{#video-record .member} +**`record`**`(step: (time: number) => boolean | Promise, onProgress?: (progress: { frames: number; time: number }) => void): Promise` [¶](#video-record){.headerlink} +: Records frames until `step(time)` returns false, then finishes the video. `step` gets the next + frame's time in seconds and can change the views before that frame is drawn. `onProgress` is + called after each frame. An error from `step`, drawing, the output, or `onProgress` aborts the + video and is rethrown. + +{#video-finish .member} +**`finish`**`(): Promise` [¶](#video-finish){.headerlink} +: Waits for the frame in progress, encodes the remaining frames, writes the end of the file, and + closes and unlocks the output. Later calls return the same promise. Rejects with a `cancelled` + error if the video was aborted. + +{#video-abort .member} +**`abort`**`(reason?: unknown): Promise` [¶](#video-abort){.headerlink} +: Cancels drawing and encoding, aborts the output with `reason`, and unlocks it once pending + writes settle. Later calls return the same promise. After `finish()` completes, it does + nothing. diff --git a/docs/reference/cli.json b/docs/reference/cli.json new file mode 100644 index 00000000..508e1b9b --- /dev/null +++ b/docs/reference/cli.json @@ -0,0 +1,888 @@ +[ + { + "name": "latkit", + "usage": "latkit [OPTIONS] ", + "about": "Render scenes of JSON or binary data to images and videos", + "args": [ + { + "id": "diagnostics", + "long": "diagnostics", + "short": null, + "value": "DIAGNOSTICS", + "positional": false, + "required": false, + "many": false, + "global": true, + "help": "Format of errors and render progress on stderr", + "defaults": [ + "human" + ], + "values": [ + { + "name": "human", + "help": "Lines of text, for people" + }, + { + "name": "json", + "help": "One JSON object per line, for programs" + } + ] + } + ] + }, + { + "name": "latkit render", + "usage": "latkit render [OPTIONS] ...", + "about": "Draw scenes of datasets into an image, image sequence, raw frames, or a video", + "args": [ + { + "id": "scenes", + "long": null, + "short": null, + "value": "SCENE", + "positional": true, + "required": true, + "many": true, + "global": false, + "help": "Scene JSON files, one per panel (`-` reads stdin)", + "defaults": [], + "values": [] + }, + { + "id": "data", + "long": "data", + "short": "d", + "value": "[NAME=]FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A dataset file: a JSON document, latkit bytes, or GeoJSON. `NAME=` sets its name, which GeoJSON requires", + "defaults": [], + "values": [] + }, + { + "id": "tiles", + "long": "tiles", + "short": null, + "value": "NAME=FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Tiles every view fetches as dataset `NAME`: a time tiles `frames.json`, or a vector tiles PMTiles file, TileJSON, or `{z}/{x}/{y}` template", + "defaults": [], + "values": [] + }, + { + "id": "output", + "long": "output", + "short": "o", + "value": "FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Output file, `%d` pattern for an image sequence, or `-` for stdout", + "defaults": [], + "values": [] + }, + { + "id": "format", + "long": "format", + "short": null, + "value": "FORMAT", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Output format. Defaults to the format of the output's extension, or PNG", + "defaults": [], + "values": [ + { + "name": "png", + "help": "PNG, lossless with alpha. One picture, or one per frame with a `%d` output pattern" + }, + { + "name": "jpeg", + "help": "JPEG at `--quality`, without alpha. One picture, or one per frame" + }, + { + "name": "webp", + "help": "Lossless WebP with alpha. One picture, or one per frame" + }, + { + "name": "rgba", + "help": "Raw RGBA with straight alpha, frames back to back" + }, + { + "name": "y4m", + "help": "YUV4MPEG2 (4:2:0, BT.709 limited range), which any encoder reads" + }, + { + "name": "mp4", + "help": "H.264 in fragmented MP4, from the platform's encoder" + } + ] + }, + { + "id": "quality", + "long": "quality", + "short": null, + "value": "QUALITY", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "JPEG quality, from 1 (smallest) to 100 (sharpest)", + "defaults": [ + "90" + ], + "values": [] + }, + { + "id": "size", + "long": "size", + "short": null, + "value": "SIZE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The picture's width and height in pixels, such as `1920x1080`. A video rounds its width down to a multiple of 4 and its height to an even number", + "defaults": [ + "1600x900" + ], + "values": [] + }, + { + "id": "scale", + "long": "scale", + "short": null, + "value": "SCALE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Pixels per CSS pixel. Views lay out size / scale CSS pixels, so 2 draws twice as sharp", + "defaults": [ + "1" + ], + "values": [] + }, + { + "id": "panel", + "long": "panel", + "short": null, + "value": "X,Y,W,H", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A scene's panel of the picture: its x, y, width, and height, as fractions. Give one per scene, in order", + "defaults": [], + "values": [] + }, + { + "id": "transparent", + "long": "transparent", + "short": null, + "value": null, + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Keep alpha where the themes' backgrounds would be", + "defaults": [], + "values": [] + }, + { + "id": "deadline", + "long": "deadline", + "short": null, + "value": "DEADLINE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The longest a picture or video frame may take, in seconds", + "defaults": [ + "30" + ], + "values": [] + }, + { + "id": "font", + "long": "font", + "short": null, + "value": "FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A TTF, OTF, or TTC font file. Repeat for fallbacks. Replaces the system fonts", + "defaults": [], + "values": [] + }, + { + "id": "camera", + "long": "camera", + "short": null, + "value": "CAMERA", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A camera per scene, in scene order: JSON, or a file of it (`-` reads stdin). Views without one use `--fit`", + "defaults": [], + "values": [] + }, + { + "id": "fit", + "long": "fit", + "short": null, + "value": "FIT", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "What views without a camera frame: `all`, or a fit as JSON", + "defaults": [ + "all" + ], + "values": [] + }, + { + "id": "insets", + "long": "insets", + "short": null, + "value": "T,R,B,L", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "CSS pixels to keep clear at the top, right, bottom, and left", + "defaults": [], + "values": [] + }, + { + "id": "at", + "long": "at", + "short": null, + "value": "AT", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The axis coordinate to show. Sampled fields show their last frame at or before it", + "defaults": [], + "values": [] + }, + { + "id": "elapsed", + "long": "elapsed", + "short": null, + "value": "ELAPSED", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A still's fixed effect time in seconds, for elapsed-time expressions and flowing comets", + "defaults": [], + "values": [] + }, + { + "id": "frames", + "long": "frames", + "short": null, + "value": "FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Frames as JSON Lines (`-` reads stdin): one per output frame, or with `--span`, at their `at`", + "defaults": [], + "values": [] + }, + { + "id": "span", + "long": "span", + "short": null, + "value": "FROM,TO", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The axis interval to play. A video or image sequence without `--frames` plays the datasets' span", + "defaults": [], + "values": [] + }, + { + "id": "rate", + "long": "rate", + "short": null, + "value": "RATE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Axis units played per second. Defaults to 1", + "defaults": [], + "values": [] + }, + { + "id": "fps", + "long": "fps", + "short": null, + "value": "FPS", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Output frames per second", + "defaults": [ + "60" + ], + "values": [] + }, + { + "id": "bitrate", + "long": "bitrate", + "short": null, + "value": "BITRATE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "MP4 bitrate in bits per second. Defaults to about 0.15 bits per pixel per frame", + "defaults": [], + "values": [] + } + ] + }, + { + "name": "latkit validate", + "usage": "latkit validate [OPTIONS] ...", + "about": "Check every input render reads, including frames, without a GPU", + "args": [ + { + "id": "scenes", + "long": null, + "short": null, + "value": "SCENE", + "positional": true, + "required": true, + "many": true, + "global": false, + "help": "Scene JSON files, one per panel (`-` reads stdin)", + "defaults": [], + "values": [] + }, + { + "id": "data", + "long": "data", + "short": "d", + "value": "[NAME=]FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A dataset file: a JSON document, latkit bytes, or GeoJSON. `NAME=` sets its name, which GeoJSON requires", + "defaults": [], + "values": [] + }, + { + "id": "tiles", + "long": "tiles", + "short": null, + "value": "NAME=FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Tiles every view fetches as dataset `NAME`: a time tiles `frames.json`, or a vector tiles PMTiles file, TileJSON, or `{z}/{x}/{y}` template", + "defaults": [], + "values": [] + }, + { + "id": "output", + "long": "output", + "short": "o", + "value": "FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Output file, `%d` pattern for an image sequence, or `-` for stdout", + "defaults": [], + "values": [] + }, + { + "id": "format", + "long": "format", + "short": null, + "value": "FORMAT", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Output format. Defaults to the format of the output's extension, or PNG", + "defaults": [], + "values": [ + { + "name": "png", + "help": "PNG, lossless with alpha. One picture, or one per frame with a `%d` output pattern" + }, + { + "name": "jpeg", + "help": "JPEG at `--quality`, without alpha. One picture, or one per frame" + }, + { + "name": "webp", + "help": "Lossless WebP with alpha. One picture, or one per frame" + }, + { + "name": "rgba", + "help": "Raw RGBA with straight alpha, frames back to back" + }, + { + "name": "y4m", + "help": "YUV4MPEG2 (4:2:0, BT.709 limited range), which any encoder reads" + }, + { + "name": "mp4", + "help": "H.264 in fragmented MP4, from the platform's encoder" + } + ] + }, + { + "id": "quality", + "long": "quality", + "short": null, + "value": "QUALITY", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "JPEG quality, from 1 (smallest) to 100 (sharpest)", + "defaults": [ + "90" + ], + "values": [] + }, + { + "id": "size", + "long": "size", + "short": null, + "value": "SIZE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The picture's width and height in pixels, such as `1920x1080`. A video rounds its width down to a multiple of 4 and its height to an even number", + "defaults": [ + "1600x900" + ], + "values": [] + }, + { + "id": "scale", + "long": "scale", + "short": null, + "value": "SCALE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Pixels per CSS pixel. Views lay out size / scale CSS pixels, so 2 draws twice as sharp", + "defaults": [ + "1" + ], + "values": [] + }, + { + "id": "panel", + "long": "panel", + "short": null, + "value": "X,Y,W,H", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A scene's panel of the picture: its x, y, width, and height, as fractions. Give one per scene, in order", + "defaults": [], + "values": [] + }, + { + "id": "transparent", + "long": "transparent", + "short": null, + "value": null, + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Keep alpha where the themes' backgrounds would be", + "defaults": [], + "values": [] + }, + { + "id": "deadline", + "long": "deadline", + "short": null, + "value": "DEADLINE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The longest a picture or video frame may take, in seconds", + "defaults": [ + "30" + ], + "values": [] + }, + { + "id": "font", + "long": "font", + "short": null, + "value": "FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A TTF, OTF, or TTC font file. Repeat for fallbacks. Replaces the system fonts", + "defaults": [], + "values": [] + }, + { + "id": "camera", + "long": "camera", + "short": null, + "value": "CAMERA", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A camera per scene, in scene order: JSON, or a file of it (`-` reads stdin). Views without one use `--fit`", + "defaults": [], + "values": [] + }, + { + "id": "fit", + "long": "fit", + "short": null, + "value": "FIT", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "What views without a camera frame: `all`, or a fit as JSON", + "defaults": [ + "all" + ], + "values": [] + }, + { + "id": "insets", + "long": "insets", + "short": null, + "value": "T,R,B,L", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "CSS pixels to keep clear at the top, right, bottom, and left", + "defaults": [], + "values": [] + }, + { + "id": "at", + "long": "at", + "short": null, + "value": "AT", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The axis coordinate to show. Sampled fields show their last frame at or before it", + "defaults": [], + "values": [] + }, + { + "id": "elapsed", + "long": "elapsed", + "short": null, + "value": "ELAPSED", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "A still's fixed effect time in seconds, for elapsed-time expressions and flowing comets", + "defaults": [], + "values": [] + }, + { + "id": "frames", + "long": "frames", + "short": null, + "value": "FILE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Frames as JSON Lines (`-` reads stdin): one per output frame, or with `--span`, at their `at`", + "defaults": [], + "values": [] + }, + { + "id": "span", + "long": "span", + "short": null, + "value": "FROM,TO", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The axis interval to play. A video or image sequence without `--frames` plays the datasets' span", + "defaults": [], + "values": [] + }, + { + "id": "rate", + "long": "rate", + "short": null, + "value": "RATE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Axis units played per second. Defaults to 1", + "defaults": [], + "values": [] + }, + { + "id": "fps", + "long": "fps", + "short": null, + "value": "FPS", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Output frames per second", + "defaults": [ + "60" + ], + "values": [] + }, + { + "id": "bitrate", + "long": "bitrate", + "short": null, + "value": "BITRATE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "MP4 bitrate in bits per second. Defaults to about 0.15 bits per pixel per frame", + "defaults": [], + "values": [] + } + ] + }, + { + "name": "latkit inspect", + "usage": "latkit inspect [OPTIONS] <[NAME=]FILE>...", + "about": "Describe datasets as JSON: tables, row counts, fields with kinds, and the axis and its span", + "args": [ + { + "id": "data", + "long": null, + "short": null, + "value": "[NAME=]FILE", + "positional": true, + "required": true, + "many": true, + "global": false, + "help": "Dataset files, each `[NAME=]FILE` (`-` reads stdin)", + "defaults": [], + "values": [] + }, + { + "id": "deadline", + "long": "deadline", + "short": null, + "value": "DEADLINE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The longest a dataset may take to load, in seconds", + "defaults": [ + "30" + ], + "values": [] + } + ] + }, + { + "name": "latkit schema", + "usage": "latkit schema [OPTIONS] [INPUT]", + "about": "Print the JSON Schema of every input latkit reads, or of one input", + "args": [ + { + "id": "input", + "long": null, + "short": null, + "value": "INPUT", + "positional": true, + "required": false, + "many": false, + "global": false, + "help": "Print only this input's schema", + "defaults": [], + "values": [ + { + "name": "scene", + "help": "A scene" + }, + { + "name": "data", + "help": "A dataset's JSON document" + }, + { + "name": "patch", + "help": "A patch" + }, + { + "name": "query", + "help": "A query for `latkit query`" + }, + { + "name": "camera", + "help": "A camera for `--camera`" + }, + { + "name": "fit", + "help": "A fit for `--fit`" + }, + { + "name": "frames", + "help": "One line of a `--frames` file" + } + ] + } + ] + }, + { + "name": "latkit query", + "usage": "latkit query [OPTIONS] <[NAME=]FILE> ", + "about": "Print the rows a query selects, sorted and paged, as JSON Lines with one object per row", + "args": [ + { + "id": "data", + "long": null, + "short": null, + "value": "[NAME=]FILE", + "positional": true, + "required": true, + "many": false, + "global": false, + "help": "The dataset, `[NAME=]FILE` (`-` reads stdin)", + "defaults": [], + "values": [] + }, + { + "id": "query", + "long": null, + "short": null, + "value": "QUERY", + "positional": true, + "required": true, + "many": false, + "global": false, + "help": "The query: JSON, such as `{\"of\":\"Branch\",\"filter\":{\"gt\":[\"$loading\",90]}}`, or a file of it (`-` reads stdin)", + "defaults": [], + "values": [] + }, + { + "id": "deadline", + "long": "deadline", + "short": null, + "value": "DEADLINE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The longest the dataset may take to load, in seconds", + "defaults": [ + "30" + ], + "values": [] + } + ] + }, + { + "name": "latkit tile", + "usage": "latkit tile [OPTIONS] --output <[NAME=]FILE>", + "about": "Write a dataset as time tiles: `frames.json` (which `--tiles` opens), the dataset without the histories of its sampled fields, and those histories in spans at every level of detail", + "args": [ + { + "id": "data", + "long": null, + "short": null, + "value": "[NAME=]FILE", + "positional": true, + "required": true, + "many": false, + "global": false, + "help": "The dataset, `[NAME=]FILE` (`-` reads stdin)", + "defaults": [], + "values": [] + }, + { + "id": "output", + "long": "output", + "short": "o", + "value": "DIR", + "positional": false, + "required": true, + "many": false, + "global": false, + "help": "Directory to write the tiles to", + "defaults": [], + "values": [] + }, + { + "id": "chunk", + "long": "chunk", + "short": null, + "value": "CHUNK", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "Entries per span: frames at full detail, buckets at coarser levels. A multiple of 8", + "defaults": [ + "256" + ], + "values": [] + }, + { + "id": "deadline", + "long": "deadline", + "short": null, + "value": "DEADLINE", + "positional": false, + "required": false, + "many": false, + "global": false, + "help": "The longest the dataset may take to load, in seconds", + "defaults": [ + "30" + ], + "values": [] + } + ] + } +] diff --git a/docs/reference/cli.md b/docs/reference/cli.md index ee0ab95d..679ffed5 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1,366 +1,317 @@ # Command reference -Generated from Clap. See the [CLI guide](../cli.md) for examples. +Every command and option of the `latkit` command line, generated from its help. The [command-line guide](../cli.md) shows them in use. `--help` prints a command's help, and `latkit --version` prints the version. -## latkit +(cli-latkit)= -```text -Render scenes of JSON or binary data to images and videos +## `latkit` -Usage: latkit [OPTIONS] +{.signature} +`latkit [OPTIONS] ` -Commands: - render Draw scenes of datasets into an image, image sequence, raw frames, or a video - validate Check everything render reads, frames included, without a GPU - inspect Describe datasets as JSON: tables, rows, fields and their kinds, the axis and its span - schema Print the JSON Schema of what latkit reads: every input, or one of them - query Read rows back as JSON Lines, an object a row: those a query keeps, sorted and paged - tile Write a dataset as time tiles: frames.json, which --tiles opens, the dataset without its sampled fields' histories, and their spans at every level of detail - help Print this message or the help of the given subcommand(s) +Render scenes of JSON or binary data to images and videos. -Options: - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line - - [default: human] - [possible values: human, json] - - -h, --help - Print help - - -V, --version - Print version +```{rubric} Commands ``` -## render - -```text -Draw scenes of datasets into an image, image sequence, raw frames, or a video - -Usage: latkit render [OPTIONS] ... - -Arguments: - ... - Scene JSON files, one per panel; - reads stdin - -Options: - -d, --data <[NAME=]FILE> - A dataset: JSON document, latkit bytes, or GeoJSON (which needs a NAME); NAME= renames it - - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line +{.member} +[`latkit render`](#cli-render) +: Draw scenes of datasets into an image, image sequence, raw frames, or a video. - [default: human] - [possible values: human, json] +{.member} +[`latkit validate`](#cli-validate) +: Check every input render reads, including frames, without a GPU. - --tiles - Tiles every view fetches as dataset NAME: time tiles' frames.json, or vector tiles' PMTiles file, TileJSON, or {z}/{x}/{y} template +{.member} +[`latkit inspect`](#cli-inspect) +: Describe datasets as JSON: tables, row counts, fields with kinds, and the axis and its span. - -o, --output - Image file, %d pattern for an image sequence, or - for stdout +{.member} +[`latkit schema`](#cli-schema) +: Print the JSON Schema of every input latkit reads, or of one input. - --format - What to write; otherwise the output's extension says, or PNG for stdout +{.member} +[`latkit query`](#cli-query) +: Print the rows a query selects, sorted and paged, as JSON Lines with one object per row. - Possible values: - - png: PNG, lossless with alpha: one picture, or one a frame where the output has a %d pattern - - jpeg: JPEG at --quality, its alpha dropped: one picture, or one a frame - - webp: WebP, lossless with alpha: one picture, or one a frame - - rgba: Raw RGBA, straight alpha, frames back to back - - y4m: YUV4MPEG2: 4:2:0 BT.709 limited range, for any encoder - - mp4: H.264 in fragmented MP4, from the platform's encoder +{.member} +[`latkit tile`](#cli-tile) +: Write a dataset as time tiles: `frames.json` (which [`--tiles`](#cli-render-tiles) opens), the dataset without the histories of its sampled fields, and those histories in spans at every level of detail. - --quality - A JPEG's quality, from 1, smallest, to 100, sharpest - - [default: 90] - - --size - The picture's width and height in pixels, such as 1920x1080 - - [default: 1600x900] - - --scale - Pixels per CSS pixel: views lay out size / scale CSS pixels, so 2 draws twice as sharp - - [default: 1] - - --panel - A scene's panel of the picture, X,Y,WIDTH,HEIGHT as fractions; one per scene, in order - - --transparent - Keep alpha where the themes' backgrounds would be +```{rubric} Options +``` - --deadline - The longest a picture, or a video frame, may take, in seconds +{#cli-latkit-diagnostics .member} +**`--diagnostics`** `` [¶](#cli-latkit-diagnostics){.headerlink} +: Format of errors and render progress on stderr. Defaults to `human`. Every command takes it. - [default: 30] + {#cli-latkit-diagnostics-human .member} + **`human`** [¶](#cli-latkit-diagnostics-human){.headerlink} + : Lines of text, for people. - --font - TTF/OTF/TTC file; repeat for fallbacks; replaces system fonts + {#cli-latkit-diagnostics-json .member} + **`json`** [¶](#cli-latkit-diagnostics-json){.headerlink} + : One JSON object per line, for programs. - --camera - Camera JSON for a scene, in scene order; a view without one fits +(cli-render)= - --fit - What a view without a camera frames: "all", or a fit as JSON +## `latkit render` - [default: all] +{.signature} +`latkit render [OPTIONS] ...` - --insets - CSS pixels kept clear at the top, right, bottom, and left, T,R,B,L +Draw scenes of datasets into an image, image sequence, raw frames, or a video. - --at - The axis coordinate shown; sampled fields read their last frame at or before it +```{rubric} Arguments +``` - --elapsed - A still's effect time in seconds, where elapsed time and flowing comets read it +{#cli-render-scenes .member} +**`...`** [¶](#cli-render-scenes){.headerlink} +: Scene JSON files, one per panel (`-` reads stdin). - --frames - Frames as JSON Lines, one per output frame, or - for stdin +```{rubric} Options +``` - --fps - Output frames a second +{#cli-render-data .member} +`-d,` **`--data`** `<[NAME=]FILE>` [¶](#cli-render-data){.headerlink} +: A dataset file: a JSON document, latkit bytes, or GeoJSON. `NAME=` sets its name, which GeoJSON requires. - [default: 60] +{#cli-render-tiles .member} +**`--tiles`** `` [¶](#cli-render-tiles){.headerlink} +: Tiles every view fetches as dataset `NAME`: a time tiles `frames.json`, or a vector tiles PMTiles file, TileJSON, or `{z}/{x}/{y}` template. - --bitrate - MP4 bits a second; about 0.15 bits a pixel a frame otherwise +{#cli-render-output .member} +`-o,` **`--output`** `` [¶](#cli-render-output){.headerlink} +: Output file, `%d` pattern for an image sequence, or `-` for stdout. - -h, --help - Print help (see a summary with '-h') -``` +{#cli-render-format .member} +**`--format`** `` [¶](#cli-render-format){.headerlink} +: Output format. Defaults to the format of the output's extension, or PNG. -## validate + {#cli-render-format-png .member} + **`png`** [¶](#cli-render-format-png){.headerlink} + : PNG, lossless with alpha. One picture, or one per frame with a `%d` output pattern. -```text -Check everything render reads, frames included, without a GPU + {#cli-render-format-jpeg .member} + **`jpeg`** [¶](#cli-render-format-jpeg){.headerlink} + : JPEG at [`--quality`](#cli-render-quality), without alpha. One picture, or one per frame. -Usage: latkit validate [OPTIONS] ... + {#cli-render-format-webp .member} + **`webp`** [¶](#cli-render-format-webp){.headerlink} + : Lossless WebP with alpha. One picture, or one per frame. -Arguments: - ... - Scene JSON files, one per panel; - reads stdin + {#cli-render-format-rgba .member} + **`rgba`** [¶](#cli-render-format-rgba){.headerlink} + : Raw RGBA with straight alpha, frames back to back. -Options: - -d, --data <[NAME=]FILE> - A dataset: JSON document, latkit bytes, or GeoJSON (which needs a NAME); NAME= renames it + {#cli-render-format-y4m .member} + **`y4m`** [¶](#cli-render-format-y4m){.headerlink} + : YUV4MPEG2 (4:2:0, BT.709 limited range), which any encoder reads. - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line + {#cli-render-format-mp4 .member} + **`mp4`** [¶](#cli-render-format-mp4){.headerlink} + : H.264 in fragmented MP4, from the platform's encoder. - [default: human] - [possible values: human, json] +{#cli-render-quality .member} +**`--quality`** `` [¶](#cli-render-quality){.headerlink} +: JPEG quality, from 1 (smallest) to 100 (sharpest). Defaults to `90`. - --tiles - Tiles every view fetches as dataset NAME: time tiles' frames.json, or vector tiles' PMTiles file, TileJSON, or {z}/{x}/{y} template +{#cli-render-size .member} +**`--size`** `` [¶](#cli-render-size){.headerlink} +: The picture's width and height in pixels, such as `1920x1080`. A video rounds its width down to a multiple of 4 and its height to an even number. Defaults to `1600x900`. - -o, --output - Image file, %d pattern for an image sequence, or - for stdout +{#cli-render-scale .member} +**`--scale`** `` [¶](#cli-render-scale){.headerlink} +: Pixels per CSS pixel. Views lay out size / scale CSS pixels, so 2 draws twice as sharp. Defaults to `1`. - --format - What to write; otherwise the output's extension says, or PNG for stdout +{#cli-render-panel .member} +**`--panel`** `` [¶](#cli-render-panel){.headerlink} +: A scene's panel of the picture: its x, y, width, and height, as fractions. Give one per scene, in order. - Possible values: - - png: PNG, lossless with alpha: one picture, or one a frame where the output has a %d pattern - - jpeg: JPEG at --quality, its alpha dropped: one picture, or one a frame - - webp: WebP, lossless with alpha: one picture, or one a frame - - rgba: Raw RGBA, straight alpha, frames back to back - - y4m: YUV4MPEG2: 4:2:0 BT.709 limited range, for any encoder - - mp4: H.264 in fragmented MP4, from the platform's encoder +{#cli-render-transparent .member} +**`--transparent`** [¶](#cli-render-transparent){.headerlink} +: Keep alpha where the themes' backgrounds would be. - --quality - A JPEG's quality, from 1, smallest, to 100, sharpest +{#cli-render-deadline .member} +**`--deadline`** `` [¶](#cli-render-deadline){.headerlink} +: The longest a picture or video frame may take, in seconds. Defaults to `30`. - [default: 90] +{#cli-render-font .member} +**`--font`** `` [¶](#cli-render-font){.headerlink} +: A TTF, OTF, or TTC font file. Repeat for fallbacks. Replaces the system fonts. - --size - The picture's width and height in pixels, such as 1920x1080 +{#cli-render-camera .member} +**`--camera`** `` [¶](#cli-render-camera){.headerlink} +: A camera per scene, in scene order: JSON, or a file of it (`-` reads stdin). Views without one use [`--fit`](#cli-render-fit). - [default: 1600x900] +{#cli-render-fit .member} +**`--fit`** `` [¶](#cli-render-fit){.headerlink} +: What views without a camera frame: `all`, or a fit as JSON. Defaults to `all`. - --scale - Pixels per CSS pixel: views lay out size / scale CSS pixels, so 2 draws twice as sharp +{#cli-render-insets .member} +**`--insets`** `` [¶](#cli-render-insets){.headerlink} +: CSS pixels to keep clear at the top, right, bottom, and left. - [default: 1] +{#cli-render-at .member} +**`--at`** `` [¶](#cli-render-at){.headerlink} +: The axis coordinate to show. Sampled fields show their last frame at or before it. - --panel - A scene's panel of the picture, X,Y,WIDTH,HEIGHT as fractions; one per scene, in order +{#cli-render-elapsed .member} +**`--elapsed`** `` [¶](#cli-render-elapsed){.headerlink} +: A still's fixed effect time in seconds, for elapsed-time expressions and flowing comets. - --transparent - Keep alpha where the themes' backgrounds would be +{#cli-render-frames .member} +**`--frames`** `` [¶](#cli-render-frames){.headerlink} +: Frames as JSON Lines (`-` reads stdin): one per output frame, or with [`--span`](#cli-render-span), at their `at`. - --deadline - The longest a picture, or a video frame, may take, in seconds +{#cli-render-span .member} +**`--span`** `` [¶](#cli-render-span){.headerlink} +: The axis interval to play. A video or image sequence without [`--frames`](#cli-render-frames) plays the datasets' span. - [default: 30] +{#cli-render-rate .member} +**`--rate`** `` [¶](#cli-render-rate){.headerlink} +: Axis units played per second. Defaults to 1. - --font - TTF/OTF/TTC file; repeat for fallbacks; replaces system fonts +{#cli-render-fps .member} +**`--fps`** `` [¶](#cli-render-fps){.headerlink} +: Output frames per second. Defaults to `60`. - --camera - Camera JSON for a scene, in scene order; a view without one fits +{#cli-render-bitrate .member} +**`--bitrate`** `` [¶](#cli-render-bitrate){.headerlink} +: MP4 bitrate in bits per second. Defaults to about 0.15 bits per pixel per frame. - --fit - What a view without a camera frames: "all", or a fit as JSON +(cli-validate)= - [default: all] +## `latkit validate` - --insets - CSS pixels kept clear at the top, right, bottom, and left, T,R,B,L +{.signature} +`latkit validate [OPTIONS] ...` - --at - The axis coordinate shown; sampled fields read their last frame at or before it +Check every input render reads, including frames, without a GPU. - --elapsed - A still's effect time in seconds, where elapsed time and flowing comets read it +Takes [`latkit render`](#cli-render)'s arguments and options. - --frames - Frames as JSON Lines, one per output frame, or - for stdin +(cli-inspect)= - --fps - Output frames a second +## `latkit inspect` - [default: 60] +{.signature} +`latkit inspect [OPTIONS] <[NAME=]FILE>...` - --bitrate - MP4 bits a second; about 0.15 bits a pixel a frame otherwise +Describe datasets as JSON: tables, row counts, fields with kinds, and the axis and its span. - -h, --help - Print help (see a summary with '-h') +```{rubric} Arguments ``` -## inspect - -```text -Describe datasets as JSON: tables, rows, fields and their kinds, the axis and its span +{#cli-inspect-data .member} +**`<[NAME=]FILE>...`** [¶](#cli-inspect-data){.headerlink} +: Dataset files, each `[NAME=]FILE` (`-` reads stdin). -Usage: latkit inspect [OPTIONS] <[NAME=]FILE>... +```{rubric} Options +``` -Arguments: - <[NAME=]FILE>... - Dataset files, each [NAME=]FILE; - reads stdin +{#cli-inspect-deadline .member} +**`--deadline`** `` [¶](#cli-inspect-deadline){.headerlink} +: The longest a dataset may take to load, in seconds. Defaults to `30`. -Options: - --deadline - The longest a dataset may take to load, in seconds +(cli-schema)= - [default: 30] +## `latkit schema` - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line +{.signature} +`latkit schema [OPTIONS] [INPUT]` - [default: human] - [possible values: human, json] +Print the JSON Schema of every input latkit reads, or of one input. - -h, --help - Print help +```{rubric} Arguments ``` -## schema - -```text -Print the JSON Schema of what latkit reads: every input, or one of them - -Usage: latkit schema [OPTIONS] [INPUT] - -Arguments: - [INPUT] - One input's schema +{#cli-schema-input .member} +**`[INPUT]`** [¶](#cli-schema-input){.headerlink} +: Print only this input's schema. - Possible values: - - scene: A scene - - data: A dataset's JSON document - - patch: A patch's JSON - - query: A query of `latkit query` - - camera: A camera of `--camera` - - fit: A fit of `--fit` - - frames: A line of `--frames` + {#cli-schema-input-scene .member} + **`scene`** [¶](#cli-schema-input-scene){.headerlink} + : A scene. -Options: - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line + {#cli-schema-input-data .member} + **`data`** [¶](#cli-schema-input-data){.headerlink} + : A dataset's JSON document. - [default: human] - [possible values: human, json] + {#cli-schema-input-patch .member} + **`patch`** [¶](#cli-schema-input-patch){.headerlink} + : A patch. - -h, --help - Print help (see a summary with '-h') -``` - -## query + {#cli-schema-input-query .member} + **`query`** [¶](#cli-schema-input-query){.headerlink} + : A query for [`latkit query`](#cli-query). -```text -Read rows back as JSON Lines, an object a row: those a query keeps, sorted and paged + {#cli-schema-input-camera .member} + **`camera`** [¶](#cli-schema-input-camera){.headerlink} + : A camera for [`--camera`](#cli-render-camera). -Usage: latkit query [OPTIONS] <[NAME=]FILE> + {#cli-schema-input-fit .member} + **`fit`** [¶](#cli-schema-input-fit){.headerlink} + : A fit for [`--fit`](#cli-render-fit). -Arguments: - <[NAME=]FILE> - The dataset, [NAME=]FILE; - reads stdin + {#cli-schema-input-frames .member} + **`frames`** [¶](#cli-schema-input-frames){.headerlink} + : One line of a [`--frames`](#cli-render-frames) file. - - The query: JSON such as {"of":"Branch","filter":{"op":"gt","args":["loading",90]}}, or a file of it +(cli-query)= -Options: - --deadline - The longest the dataset may take to load, in seconds +## `latkit query` - [default: 30] +{.signature} +`latkit query [OPTIONS] <[NAME=]FILE> ` - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line +Print the rows a query selects, sorted and paged, as JSON Lines with one object per row. - [default: human] - [possible values: human, json] - - -h, --help - Print help +```{rubric} Arguments ``` -## tile - -```text -Write a dataset as time tiles: frames.json, which --tiles opens, the dataset without its sampled fields' histories, and their spans at every level of detail +{#cli-query-data .member} +**`<[NAME=]FILE>`** [¶](#cli-query-data){.headerlink} +: The dataset, `[NAME=]FILE` (`-` reads stdin). -Usage: latkit tile [OPTIONS] --output <[NAME=]FILE> +{#cli-query-query .member} +**``** [¶](#cli-query-query){.headerlink} +: The query: JSON, such as `{"of":"Branch","filter":{"gt":["$loading",90]}}`, or a file of it (`-` reads stdin). -Arguments: - <[NAME=]FILE> - The dataset, [NAME=]FILE; - reads stdin +```{rubric} Options +``` -Options: - --diagnostics - Diagnostics on stderr: errors, and progress while rendering; JSON is one object a line +{#cli-query-deadline .member} +**`--deadline`** `` [¶](#cli-query-deadline){.headerlink} +: The longest the dataset may take to load, in seconds. Defaults to `30`. - [default: human] - [possible values: human, json] +(cli-tile)= - -o, --output - The directory the tiles go in +## `latkit tile` - --chunk - Entries a span holds: frames, or buckets of a level of detail; a multiple of 8 +{.signature} +`latkit tile [OPTIONS] --output <[NAME=]FILE>` - [default: 256] +Write a dataset as time tiles: `frames.json` (which [`--tiles`](#cli-render-tiles) opens), the dataset without the histories of its sampled fields, and those histories in spans at every level of detail. - --deadline - The longest the dataset may take to load, in seconds +```{rubric} Arguments +``` - [default: 30] +{#cli-tile-data .member} +**`<[NAME=]FILE>`** [¶](#cli-tile-data){.headerlink} +: The dataset, `[NAME=]FILE` (`-` reads stdin). - -h, --help - Print help +```{rubric} Options ``` -## help +{#cli-tile-output .member} +`-o,` **`--output`** `` [¶](#cli-tile-output){.headerlink} +: Directory to write the tiles to. -```text -Print this message or the help of the given subcommand(s) +{#cli-tile-chunk .member} +**`--chunk`** `` [¶](#cli-tile-chunk){.headerlink} +: Entries per span: frames at full detail, buckets at coarser levels. A multiple of 8. Defaults to `256`. -Usage: latkit help [COMMAND]... - -Arguments: - [COMMAND]... - Print help for the subcommand(s) -``` +{#cli-tile-deadline .member} +**`--deadline`** `` [¶](#cli-tile-deadline){.headerlink} +: The longest the dataset may take to load, in seconds. Defaults to `30`. diff --git a/docs/reference/types.md b/docs/reference/types.md index fa5d47bc..7eba473b 100644 --- a/docs/reference/types.md +++ b/docs/reference/types.md @@ -1,1802 +1,2574 @@ # Scene and result types -Generated from the same schema as the package types. Each field documents its meaning and defaults. +The types of the scenes latkit draws and the results it reports, generated from the same schema as +the package's types. Each property gives its meaning and default. ## Scenes -A scene: its space, its marks, and how it looks and answers input. - -### Scene - -```ts -/** - * What a view draws and how it answers input: a space, marks in drawing order, a layout, a theme, - * controls, how long changes ease, and the title written over it. Plain data: as JSON, each field - * by its name in camelCase, each mark `{ vertices | edges | faces | lines | areas: options }`. - */ -interface Scene { - space: Space; - /** What the view draws, in drawing order. */ - marks?: Mark[]; - /** - * How the view places vertex rows without positions: a diagram's blocks, or points on a plane or - * the earth, gaps in the space's units. - */ - layout?: Layout; - theme?: Theme; - controls?: Controls; - /** Seconds that changes ease over; 0.3 unless set. */ - ease?: number; - /** Written at the view's top left, larger than the theme's text. */ - title?: string | null; - /** Written under the title. */ - subtitle?: string | null; -} -``` - -### Space - -```ts -/** - * The coordinate space a scene's positions are in: a name, or - * `{ plane | diagram | earth | plot: options }`. A diagram is a flat sheet seen from straight - * above: y runs down, and every length is a unit, a CSS pixel at zoom 0, except glow, halo, and - * pick radius. Only a diagram has ports, groups, nets, orthogonal routes, and layouts. - */ -type Space = - | 'plane' - | 'diagram' - | 'mercator' - | 'globe' - | 'plot' - | { - /** Longitude and latitude in degrees, on a map or a globe. */ - earth: Earth; - } - | { - /** Plane units with elevation: graphs and scenes the camera turns and tilts around. */ - plane: Plane; - } - | { - /** The data's axis across, values up. */ - plot: Plot; - } - | { - /** A sheet for diagrams: x right and y down in units, every length a unit. */ - diagram: Diagram; - }; -``` - -### Plane - -```ts -/** A plane: positions are plane units. */ -interface Plane { - /** The grid under the marks. */ - grid?: Grid; -} -``` - -### Earth - -```ts -/** The earth: positions are longitude and latitude in degrees. */ -interface Earth { - /** A flat map or a globe. */ - projection?: Projection; - /** The ground's color; the theme's surface unless set. */ - surface?: Color | null; - /** Whether lines of longitude and latitude cross the ground, in the theme's grid color. */ - graticule?: boolean; - /** - * Lit by the sun as it stands this many seconds after the Unix epoch, plus the view's axis - * coordinate when it shows one: night falls on the ground and what stands on it, down to the - * theme's night. Lit evenly unless set. - */ - daylight?: number | null; - /** Whether air glows around the globe, in the theme's atmosphere color, brighter where it is day. */ - atmosphere?: boolean; -} -``` - -### Projection - -```ts -/** How the earth is drawn. */ -type Projection = 'mercator' | 'globe'; -``` - -### Diagram - -```ts -/** - * A sheet for diagrams, flat and seen from straight above: the camera pans and zooms. Positions - * are units with y down, and every length is a unit, a CSS pixel at zoom 0 (sizes, padding, ports, - * strokes, text, and the theme's), except glow, halo, and pick radius. Zooming scales one fixed - * picture. Only a diagram has ports, groups, nets, and orthogonal routes; it lays out blocks. - */ -interface Diagram { - /** The grid under the marks. */ - grid?: Grid; -} -``` - -### Plot - -```ts -/** A plot: x is the data's axis, y the values. */ -interface Plot { - /** The horizontal axis. */ - x?: Axis; - /** The vertical axis. */ - y?: Axis; -} -``` - -### Axis - -```ts -/** A plot axis. */ -interface Axis { - /** The axis title. */ - label?: string | null; - /** Whether grid lines cross the plot at the ticks. */ - grid?: boolean; - /** Whether the axis is hidden. */ - hidden?: boolean; - /** - * Ticks at these values, each a value or `[value, label]`, their values written unless labeled; - * round values unless set. - */ - ticks?: Tick[] | null; - /** How tick values are written. */ - format?: Format; - /** - * Decimals of fixed values, or of the significand of scientific and engineering ones; as many as - * the ticks' step needs unless set. - */ - precision?: number | null; -} -``` - -### Tick - -```ts -/** A tick as written: a value, or `[value, label]`. */ -type Tick = number | [number, string | null]; -``` - -### Format - -```ts -/** How an axis writes its tick values. */ -type Format = 'auto' | 'fixed' | 'scientific' | 'engineering'; -``` - -### Grid - -```ts -/** A plane's or a diagram's grid: `"off"`, `{ lines: { pitch } }`, or `{ dots: { pitch, snap } }`. */ -type Grid = - | 'off' - | { - /** Lines `pitch` units apart, thinned by tens as the view zooms out. */ - lines: { - /** Units between lines. */ - pitch: number; - }; - } - | { - /** Dots `pitch` units apart; drags snap to them when `snap`. */ - dots: { - /** Units between dots. */ - pitch: number; - /** Whether moved vertices snap to the dots. */ - snap?: boolean; - }; - }; -``` - -### Theme - -```ts -/** - * Colors and sizes marks take where their options read nothing, and the look of focus and motion: - * a base theme's name, or options over one (`base`, dark unless set). Lengths are CSS pixels; in a - * diagram they are units, except `glow` and `halo`. - */ -type Theme = - | 'dark' - | 'light' - | { - /** Behind everything. */ - background?: Color; - /** The earth's ground, and a group's container. */ - surface?: Color; - /** A plane's or a diagram's grid. */ - grid?: Color; - /** Vertex color. */ - vertex?: Color; - /** Vertex radius. */ - vertexRadius?: number; - /** - * Edge color; `null` blends the colors of each edge's two vertices, and in a diagram takes the - * text color. - */ - edge?: Color | null; - /** Edge width. */ - edgeWidth?: number; - /** Line color. */ - line?: Color; - /** Line width. */ - lineWidth?: number; - /** Area color. */ - area?: Color; - /** How much a face turned away from the camera darkens, 0 to 1. */ - shading?: number; - /** Text: labels, axis ticks, and titles. */ - text?: Color; - /** The CSS font family text is set in. */ - font?: string; - /** Text size. */ - fontSize?: number; - /** The line a plot draws at the shown coordinate. */ - playhead?: Color; - /** What a missing color reads as. */ - missing?: Color; - /** The glow around hovered rows. */ - hover?: Color; - /** The glow around selected rows; `null` glows in each row's own color. */ - selected?: Color | null; - /** How far focus glows reach, CSS pixels. */ - glow?: number; - /** The opacity of rows outside a selection. */ - dim?: number; - /** How much a hovered vertex sized in CSS pixels grows. */ - hoverScale?: number; - /** Length of a dash. */ - dash?: number; - /** How comets flowing along edges and lines look. */ - flow?: Flow; - /** - * In a diagram, distance between the lanes of edges joining the same two vertices, and between - * parallel orthogonal wires, which keep twice it apart. - */ - edgeSpacing?: number; - /** - * Elsewhere, how far edges joining the same two vertices bow apart: each lane by this fraction of - * the edge's length at its middle. A bow is part of the drawing, so zooming never moves it. - */ - laneBow?: number; - /** How far an arrowhead reaches past its edge's width. */ - arrow?: number; - /** CSS pixels of background around label text. */ - halo?: number; - /** Distance between a group's members and its outline. */ - groupPadding?: number; - /** The border of a diagram's blocks and groups, a unit wide; `null` draws none. */ - outline?: Color | null; - /** The radius of a diagram's block and group corners; a block's marker may set its own. */ - corner?: number; - /** How wide a diagram's ports are. */ - portSize?: number; - /** The radius of an orthogonal wire's bends. */ - bend?: number; - /** How marks that cast shadows cast them. */ - shadow?: Shadow; - /** The air glowing around the globe, where the earth has an atmosphere. */ - atmosphere?: Color; - /** How bright the earth's night side stays where it is lit by daylight, 0 to 1. */ - night?: number; - /** The theme these options change; dark unless set. */ - base?: 'dark' | 'light'; - }; -``` - -### Flow - -```ts -/** - * How comets flowing along edges and lines look: each a round head, brightest at its middle, - * trailing a tail that thins and fades. - */ -interface Flow { - /** A comet's color; its alpha is how strongly a comet covers its stroke. */ - color?: Color; - /** Distance between comets. */ - spacing?: number; - /** How far a comet's tail trails its head. */ - length?: number; - /** How wide a comet's head is, CSS pixels, or units in a diagram; never narrower than its stroke. */ - width?: number; - /** - * Whether comets are part of the drawing: their spacing, length, and each row's flow in position - * units, degrees on the earth, so they zoom with the view and stay where they are on their lines. - * Otherwise they keep their size and speed on screen, and slide along their lines as it zooms. - */ - units?: boolean; -} -``` - -### Shadow - -```ts -/** How marks that cast shadows cast them onto the ground and onto faces. */ -interface Shadow { - /** What a shadow darkens toward; its alpha is how dark a shadow at contact gets. */ - color?: Color; - /** The direction toward the light, fixed in the world: shadows fall away from it. */ - light?: [number, number, number]; - /** How much a shadow spreads per unit of a caster's height: the light's size. */ - softness?: number; - /** - * CSS pixels flat marks cast from, as if raised; in a diagram, how far a block's or a group's - * shadow falls below it, and as far again it softens. - */ - lift?: number; -} -``` - -### Controls - -```ts -/** - * How a view answers the pointer and keyboard, and how much it moves on its own: a preset, or - * options over the defaults. Each mark says whether its rows pick, select, and drag; these say - * what the view itself does. - */ -type Controls = - | 'none' - | 'inspect' - | { - /** Dragging the background moves the camera. */ - pan?: boolean; - /** - * What zooms: the wheel and pinching, or the wheel only with Control or Meta held (as a trackpad's - * pinch sends it), leaving the plain wheel to scroll the page; or nothing. - */ - zoom?: Zoom; - /** Dragging with the secondary button, or with Shift, turns and tilts a map or plane. */ - turn?: boolean; - /** - * Tab steps through the rows nearest the view's middle and arrows to the nearest row their way; - * Enter opens the row stepped to and Space selects it. Otherwise arrows pan, `+` and `-` zoom, - * Home fits, and Escape clears the selection. - */ - keyboard?: boolean; - /** - * The row the user points at glows. It is hovered either way: events tell it, and `hovered()` - * reads it. - */ - hover?: boolean; - /** CSS pixels around the pointer that picks reach; touch reaches 8 more. */ - pickRadius?: number; - /** - * Dragging with Shift selects the rows in a box, instead of turning; a Shift click still toggles - * its row. - */ - selectBox?: boolean; - /** - * Dragging from a port, from a vertex with Alt, or an edge off its end draws a connection, told as - * `Event::Connect`. - */ - connect?: boolean; - /** Delete and Backspace tell `Event::Delete` with the selection. */ - delete?: boolean; - /** - * Clicking again where nothing moved selects the next row under the point, nearest first, so rows - * drawn over one another can each be selected. - */ - cycle?: boolean; - /** - * Seconds a touch or a pen held still asks for a menu, as the secondary button does; zero never - * does. - */ - press?: number; - /** - * How much the view moves on its own: eases, flights, coasting, orbits, label fades, comets, and - * what reads `elapsed()`. - */ - motion?: Motion; - }; -``` - -### Motion - -```ts -/** How much a view moves on its own. */ -type Motion = 'system' | 'full' | 'reduced'; -``` - -### Zoom - -```ts -/** What zooms a view. */ -type Zoom = 'wheel' | 'modifier' | 'off'; -``` - -### Layout - -```ts -/** - * How the view places vertex rows without positions: a diagram's blocks, sized, or the points of a - * plane or the earth, among the rows linked to them, gaps in the space's units: a name, or - * `{ stress | layered: options }`. A place, once given, holds: rows added later are placed around - * the rest, and `View::arrange` lays every row out again. - */ -type Layout = - | 'none' - | 'stress' - | 'layered' - | { - /** Distances on the drawing follow distances in the graph. */ - stress: Stress; - } - | { - /** Ranks along a direction, for flows and hierarchies. */ - layered: Layered; - }; -``` - -### Layered - -```ts -/** Layered drawing: vertices in ranks, edges pointing along `flow`, and nets from their outputs. */ -interface Layered { - /** Where edges point. */ - flow?: Direction; - /** Units between vertices of a rank. */ - gap?: number; - /** Units between ranks. */ - rankGap?: number; -} -``` - -### Stress - -```ts -/** - * Stress majorization: each pair of vertices pulls toward its graph distance; then their boxes - * part, keeping a quarter of the gap clear. - */ -interface Stress { - /** Units between adjacent vertices' centers. */ - gap?: number; -} -``` - -### Direction - -```ts -/** A direction on screen. */ -type Direction = 'right' | 'left' | 'down' | 'up'; -``` - -### Order - -```ts -/** Which way a sort runs; rows a key reads nothing for come last either way. */ -type Order = 'ascending' | 'descending'; -``` - -### Budget - -```ts -/** How much GPU memory latkit uses. */ -interface Budget { - /** - * Bytes the data every view of the `Gpu` draws may take on the GPU: its pages, and each view's - * rows. A view draws a table too large for its share through the tiles the camera asks for, so - * what it draws always fits; pages no view draws stay cached in the room left, the least recently - * drawn going first. By default half the device's largest storage buffer. The pool of pages grows - * as data needs it, and never shrinks: set the budget before data draws. - */ - bytes: number; - /** Bytes uploaded per frame at most; larger data streams in over frames. */ - uploadPerFrame: number; -} -``` +A scene: its space, how it responds to input, and how it is laid out. + +(scene)= + +### `Scene` + +{.signature} +`interface Scene` + +A view's content and behavior: a space, marks drawn in order, a layout, a theme, controls, an +ease duration, and an optional title. + +A scene is plain data. In JSON, fields are camelCase, and the key naming each mark's kind holds +its dataset and table: `{ "vertices": ["grid", "Bus"], "radius": 4 }`. + +{#scene-space .member} +**`space`**`:` [`Space`](#space) [¶](#scene-space){.headerlink} +: The coordinate space positions are in. + +{#scene-marks .member} +**`marks`**`?:` [`Mark`](#mark)`[]` [¶](#scene-marks){.headerlink} +: The marks, in drawing order: each draws over the marks before it. + +{#scene-layout .member} +**`layout`**`?:` [`Layout`](#layout) [¶](#scene-layout){.headerlink} +: How the view places vertex rows that have no position, such as a diagram's blocks or points on a + plane or the earth. Defaults to `none`. + +{#scene-theme .member} +**`theme`**`?:` [`Theme`](#theme) [¶](#scene-theme){.headerlink} +: Default colors and sizes, and the look of text, focus, and each space. Defaults to `dark`. + +{#scene-controls .member} +**`controls`**`?:` [`Controls`](#controls) [¶](#scene-controls){.headerlink} +: How the view responds to the pointer and keyboard. + +{#scene-ease .member} +**`ease`**`?: number` [¶](#scene-ease){.headerlink} +: How many seconds scene and data changes take to animate. 0 turns animation off. Defaults to 0.3. + +{#scene-title .member} +**`title`**`?: string | null` [¶](#scene-title){.headerlink} +: A title drawn at the view's top left, larger than the theme's text. + +{#scene-subtitle .member} +**`subtitle`**`?: string | null` [¶](#scene-subtitle){.headerlink} +: A subtitle drawn under the title. + +(space)= + +### `Space` + +{.signature} +`type Space = 'plane' | 'diagram' | 'earth' | 'plot' | { earth:` [`Earth`](#earth) `} | { plane:` [`Plane`](#plane) `} | { plot:` [`Plot`](#plot) `} | { diagram:` [`Diagram`](#diagram) `}` + +The coordinate space of a scene's positions: a name, or +`{ plane | diagram | earth | plot: options }`. + +The space sets the unit of lengths that do not name their own. In a diagram it is the diagram +unit, one CSS pixel at zoom 0. Elsewhere it is the CSS pixel. Only a diagram has ports, groups, +nets, and orthogonal routes. + +{#space-earth .member} +`{` **`earth`**`:` [`Earth`](#earth) `}` [¶](#space-earth){.headerlink} +: Longitude and latitude in degrees, on a flat map or a globe. + +{#space-plane .member} +`{` **`plane`**`:` [`Plane`](#plane) `}` [¶](#space-plane){.headerlink} +: Plane units, with elevation. The camera can turn and tilt. + +{#space-plot .member} +`{` **`plot`**`:` [`Plot`](#plot) `}` [¶](#space-plot){.headerlink} +: The data's axis horizontally and values vertically. + +{#space-diagram .member} +`{` **`diagram`**`:` [`Diagram`](#diagram) `}` [¶](#space-diagram){.headerlink} +: A flat sheet for diagrams: x right and y down, every length in units. + +(plane)= + +### `Plane` + +{.signature} +`interface Plane` + +A plane: positions are in plane units. + +{#plane-grid .member} +**`grid`**`?:` [`Grid`](#grid) [¶](#plane-grid){.headerlink} +: The grid drawn under the marks. Defaults to lines 10 units apart. + +(earth)= + +### `Earth` + +{.signature} +`interface Earth` + +The earth: positions are longitude and latitude in degrees. The ground uses the theme's `earth` +colors. + +A flat map shows the world once. Its ground ends at the antimeridian and at Web Mercator's +latitude limit. The camera zooms out until the whole world fits the view, and pans until the +world's edges meet the view's edges. On a globe, longitude wraps, and the camera zooms out until +the globe spans a third of the view's shorter side. + +{#earth-projection .member} +**`projection`**`?:` [`Projection`](#projection) [¶](#earth-projection){.headerlink} +: A flat web mercator map (`mercator`) or a globe (`globe`). Defaults to `mercator`. + +{#earth-graticule .member} +**`graticule`**`?: boolean` [¶](#earth-graticule){.headerlink} +: Whether lines of longitude and latitude cross the ground, in the theme's `grid` color. Defaults + to `true`. + +{#earth-daylight .member} +**`daylight`**`?: number | null` [¶](#earth-daylight){.headerlink} +: Lights the earth by the sun's position this many seconds after the Unix epoch, plus the view's + axis coordinate when it has one. The night side darkens the ground and everything on it, down to + the theme's `earth.night` brightness. When unset, the earth is lit evenly. + +{#earth-atmosphere .member} +**`atmosphere`**`?: boolean` [¶](#earth-atmosphere){.headerlink} +: Whether air glows around the globe, in the theme's `earth.atmosphere` color, brighter on the day + side. Defaults to `false`. + +(projection)= + +### `Projection` + +{.signature} +`type Projection = 'mercator' | 'globe'` + +How the earth is projected. + +{#projection-mercator .member} +**`'mercator'`** [¶](#projection-mercator){.headerlink} +: Web mercator. + +{#projection-globe .member} +**`'globe'`** [¶](#projection-globe){.headerlink} +: A sphere. + +(diagram)= + +### `Diagram` + +{.signature} +`interface Diagram` + +A flat sheet for diagrams, seen from straight above. The camera pans and zooms but does not +turn. Positions are in units with y down, and a unit is a CSS pixel at zoom 0. Sizes, padding, +ports, strokes, text, and the theme's lengths are in units too. Sizes given with `px`, the text +halo, the focus glow, and the pick radius stay in CSS pixels. Zooming scales the whole picture. +Only a diagram has ports, groups, nets, and orthogonal routes, and the scene's layout places its +blocks. + +{#diagram-grid .member} +**`grid`**`?:` [`Grid`](#grid) [¶](#diagram-grid){.headerlink} +: The grid drawn under the marks. Defaults to dots 20 units apart. + +(plot)= + +### `Plot` + +{.signature} +`interface Plot` + +A plot: x is the data's axis, y the values. + +{#plot-x .member} +**`x`**`?:` [`Axis`](#axis) [¶](#plot-x){.headerlink} +: The horizontal axis. + +{#plot-y .member} +**`y`**`?:` [`Axis`](#axis) [¶](#plot-y){.headerlink} +: The vertical axis. + +(axis)= + +### `Axis` + +{.signature} +`interface Axis` + +A plot axis. + +{#axis-label .member} +**`label`**`?: string | null` [¶](#axis-label){.headerlink} +: The axis title. + +{#axis-grid .member} +**`grid`**`?: boolean` [¶](#axis-grid){.headerlink} +: Whether grid lines cross the plot at the ticks. Defaults to `true`. + +{#axis-hidden .member} +**`hidden`**`?: boolean` [¶](#axis-hidden){.headerlink} +: Whether the axis is hidden. Defaults to `false`. + +{#axis-ticks .member} +**`ticks`**`?:` [`Tick`](#tick)`[] | null` [¶](#axis-ticks){.headerlink} +: Tick positions, each a value or `[value, label]`. A tick without a label shows its value. When + unset, ticks fall at round values. + +{#axis-format .member} +**`format`**`?:` [`Format`](#format) [¶](#axis-format){.headerlink} +: How tick values are formatted. Defaults to `auto`. + +{#axis-precision .member} +**`precision`**`?: number | null` [¶](#axis-precision){.headerlink} +: Decimal places of fixed values, or of the significand in scientific and engineering formats. At + most 15. When unset, as many as the tick step needs. + +(tick)= + +### `Tick` + +{.signature} +`type Tick = number | [number, string | null]` + +A tick in JSON: a value, or `[value, label]`. + +(format)= + +### `Format` + +{.signature} +`type Format = 'auto' | 'fixed' | 'scientific' | 'engineering'` + +How an axis formats its tick values. + +{#format-auto .member} +**`'auto'`** [¶](#format-auto){.headerlink} +: Fixed decimals, scientific past a million or below a ten-thousandth. + +{#format-fixed .member} +**`'fixed'`** [¶](#format-fixed){.headerlink} +: Fixed decimals: `1500.0`. + +{#format-scientific .member} +**`'scientific'`** [¶](#format-scientific){.headerlink} +: A significand and a power of ten: `1.5e3`. + +{#format-engineering .member} +**`'engineering'`** [¶](#format-engineering){.headerlink} +: A significand and a power of ten that is a multiple of three: `1.5e3`, `150e-6`. + +(grid)= + +### `Grid` + +{.signature} +`type Grid = 'off' | { lines: { … } } | { dots: { … } }` + +A plane's or a diagram's grid: `"off"`, `{ lines: { pitch } }`, or `{ dots: { pitch, snap } }`. + +{#grid-off .member} +**`'off'`** [¶](#grid-off){.headerlink} +: No grid. + +{#grid-lines .member} +`{` **`lines`**`: { pitch: number } }` [¶](#grid-lines){.headerlink} +: Lines `pitch` units apart, thinned by factors of ten as the view zooms out. + + {#grid-lines-pitch .member} + **`pitch`**`: number` [¶](#grid-lines-pitch){.headerlink} + : Units between lines. + +{#grid-dots .member} +`{` **`dots`**`: { pitch: number; snap?: boolean } }` [¶](#grid-dots){.headerlink} +: Dots `pitch` units apart. With `snap`, drags in a diagram snap to them. + + {#grid-dots-pitch .member} + **`pitch`**`: number` [¶](#grid-dots-pitch){.headerlink} + : Units between dots. + + {#grid-dots-snap .member} + **`snap`**`?: boolean` [¶](#grid-dots-snap){.headerlink} + : Whether drag positions snap to the dots, in a diagram. Defaults to `false`. + +(controls)= + +### `Controls` + +{.signature} +`type Controls = 'none' | 'inspect' | { … }` + +How a view responds to the pointer and keyboard, and how much it animates on its own: a preset +(`"none"` or `"inspect"`), or options that change the defaults. Each mark sets whether its own +rows pick, select, and drag. These options control the view itself. + +{#controls-pan .member} +**`pan`**`?: boolean` [¶](#controls-pan){.headerlink} +: Whether dragging the background pans the camera. Defaults to `true`. + +{#controls-zoom .member} +**`zoom`**`?:` [`Zoom`](#zoom) [¶](#controls-zoom){.headerlink} +: Which input zooms the view. `wheel` zooms with the wheel and pinching. `modifier` zooms with + pinching, and with the wheel only while Control or Meta is held (as trackpad pinches send it), + so the plain wheel scrolls the page. `off` turns zooming off. Defaults to `wheel`. + +{#controls-turn .member} +**`turn`**`?: boolean` [¶](#controls-turn){.headerlink} +: Whether dragging with the secondary button, or with Shift, rotates and tilts a map or plane. + Defaults to `true`. + +{#controls-keyboard .member} +**`keyboard`**`?: boolean` [¶](#controls-keyboard){.headerlink} +: Whether the keyboard controls the view. Defaults to `true`. Tab and Shift+Tab step through the + rows nearest the view's center. Arrow keys step to the nearest row in their direction. Enter + opens the row stepped to, and Space selects it. With no row stepped to, arrow keys pan + (Shift+arrows turn), `+` and `-` zoom, Home fits everything, and Escape clears the selection. + +{#controls-hover .member} +**`hover`**`?: boolean` [¶](#controls-hover){.headerlink} +: Whether the hovered row glows and a hovered vertex sized in CSS pixels grows. Defaults to + `true`. Hover is tracked either way: a `'hovered'` event still fires and `hovered` still returns + the row. + +{#controls-pickradius .member} +**`pickRadius`**`?: number` [¶](#controls-pickradius){.headerlink} +: How far a row can be from the pointer and still be picked, in CSS pixels. Touch adds 8. Defaults + to 6. + +{#controls-selectbox .member} +**`selectBox`**`?: boolean` [¶](#controls-selectbox){.headerlink} +: Whether Shift+drag selects the rows inside a box instead of turning the camera. Shift+click + still toggles one row. Defaults to `false`. + +{#controls-connect .member} +**`connect`**`?: boolean` [¶](#controls-connect){.headerlink} +: Whether the user can draw connections by dragging from a port, from a vertex with Alt held, or + an edge's end off its vertex. Connections are reported as a `'connect'` event. Defaults to + `false`. + +{#controls-delete .member} +**`delete`**`?: boolean` [¶](#controls-delete){.headerlink} +: Whether Delete and Backspace report the selection as a `'delete'` event. Defaults to `false`. + +{#controls-cycle .member} +**`cycle`**`?: boolean` [¶](#controls-cycle){.headerlink} +: Whether clicking again at the same point, with the camera unmoved, selects the next row under + the pointer, nearest first. This lets overlapping rows each be selected. Defaults to `false`. + +{#controls-press .member} +**`press`**`?: number` [¶](#controls-press){.headerlink} +: How many seconds a touch or pen must hold still to request a menu (a `'menu'` event), like the + secondary button. 0 turns this off. Defaults to 0.5. + +{#controls-motion .member} +**`motion`**`?:` [`Motion`](#motion) [¶](#controls-motion){.headerlink} +: How much the view animates on its own: eases, camera flights, coasting, orbits, comets, and + options that read `elapsed`. `system` follows the user's reduced-motion setting, `full` animates + everything, and `reduced` animates only motion the user drives. Defaults to `system`. + +(motion)= + +### `Motion` + +{.signature} +`type Motion = 'system' | 'full' | 'reduced'` + +How much a view animates on its own. + +{#motion-system .member} +**`'system'`** [¶](#motion-system){.headerlink} +: Follows the system setting: reduced when the user has asked their system for less motion. + +{#motion-full .member} +**`'full'`** [¶](#motion-full){.headerlink} +: All motion. + +{#motion-reduced .member} +**`'reduced'`** [¶](#motion-reduced){.headerlink} +: Only motion the user drives: changes apply at once, flights jump, nothing coasts or orbits, and + comets and options that read `elapsed` stay still. + +(zoom)= + +### `Zoom` + +{.signature} +`type Zoom = 'wheel' | 'modifier' | 'off'` + +Which input zooms a view. + +{#zoom-wheel .member} +**`'wheel'`** [¶](#zoom-wheel){.headerlink} +: The wheel, pinching, and the keys. + +{#zoom-modifier .member} +**`'modifier'`** [¶](#zoom-modifier){.headerlink} +: Pinching and the keys, and the wheel only with Control or Meta held (as trackpad pinches send + it), so the plain wheel scrolls the page. + +{#zoom-off .member} +**`'off'`** [¶](#zoom-off){.headerlink} +: No input zooms. + +(layout)= + +### `Layout` + +{.signature} +`type Layout = 'none' | 'stress' | 'layered' | { stress:` [`Stress`](#stress) `} | { layered:` [`Layered`](#layered) `}` + +How the view places vertex rows that have no position: a name, or +`{ stress | layered: options }`. In a diagram it places blocks. On a plane or the earth it +places points among the rows they link to. Gaps are in position units: diagram units, plane +units, or degrees on the earth. + +A placed row keeps its place. Rows added later are placed around the others, and [`view.arrange`](#view-arrange) +lays out every row again. + +{#layout-stress .member} +`{` **`stress`**`:` [`Stress`](#stress) `}` [¶](#layout-stress){.headerlink} +: Drawn distances follow graph distances. + +{#layout-layered .member} +`{` **`layered`**`:` [`Layered`](#layered) `}` [¶](#layout-layered){.headerlink} +: Ranks along a direction, for processes and hierarchies. + +(layered)= + +### `Layered` + +{.signature} +`interface Layered` + +Layered drawing: vertices in ranks, edges pointing along `direction`, and nets ranked from their +outputs. + +{#layered-direction .member} +**`direction`**`?:` [`Direction`](#direction) [¶](#layered-direction){.headerlink} +: The direction edges point. Defaults to `right`. + +{#layered-gap .member} +**`gap`**`?: number` [¶](#layered-gap){.headerlink} +: Distance between vertices within a rank, in position units. Defaults to 30. + +{#layered-rankgap .member} +**`rankGap`**`?: number` [¶](#layered-rankgap){.headerlink} +: Distance between ranks, in position units. Defaults to 80. + +(stress)= + +### `Stress` + +{.signature} +`interface Stress` + +Stress majorization: vertices move until drawn distances match graph distances. Then overlapping +boxes separate, keeping a quarter of `gap` between them. + +{#stress-gap .member} +**`gap`**`?: number` [¶](#stress-gap){.headerlink} +: Distance between the centers of linked vertices, in position units. Defaults to 40. + +(direction)= + +### `Direction` + +{.signature} +`type Direction = 'right' | 'left' | 'down' | 'up'` + +A direction on screen. + +{#direction-right .member} +**`'right'`** [¶](#direction-right){.headerlink} +: Toward the right. + +{#direction-left .member} +**`'left'`** [¶](#direction-left){.headerlink} +: Toward the left. + +{#direction-down .member} +**`'down'`** [¶](#direction-down){.headerlink} +: Toward the bottom. + +{#direction-up .member} +**`'up'`** [¶](#direction-up){.headerlink} +: Toward the top. + +(order)= + +### `Order` + +{.signature} +`type Order = 'ascending' | 'descending'` + +Sort direction. Rows where the key has no value sort last in either direction. + +{#order-ascending .member} +**`'ascending'`** [¶](#order-ascending){.headerlink} +: Smallest first. + +{#order-descending .member} +**`'descending'`** [¶](#order-descending){.headerlink} +: Largest first. + +(budget)= + +### `Budget` + +{.signature} +`interface Budget` + +How much GPU memory latkit uses. + +{#budget-bytes .member} +**`bytes`**`: number` [¶](#budget-bytes){.headerlink} +: GPU memory in bytes for the data all views of the [`Gpu`](#gpu) draw: its pages and each view's per-row + state. A table too large for a view's share draws as a paged table, with only the pages the + camera needs, so it always fits. Pages no view draws stay cached in the rest of the budget, and + the least recently drawn are evicted first. Defaults to half the device's largest storage + buffer. The page pool grows as data needs it and never shrinks, so set the budget before drawing + data. + +{#budget-uploadperframe .member} +**`uploadPerFrame`**`: number` [¶](#budget-uploadperframe){.headerlink} +: The most bytes uploaded per frame. Larger data streams in over several frames. Defaults to 32 + MiB. + +## Themes + +How a scene looks: a group of options for each kind of thing it draws, and the lengths those options take. + +(theme)= + +### `Theme` + +{.signature} +`type Theme = 'dark' | 'light' | { … }` + +Default colors and sizes for marks, and the look of text, focus, comets, and each space, grouped +by subject: a base theme's name (`"dark"` or `"light"`), or options that change a base. `base` +names the base and defaults to `dark`. Each option replaces only the base value it sets. + +Lengths are in CSS pixels, or units in a diagram. The text halo and the focus glow are always in +CSS pixels. + +{#theme-background .member} +**`background`**`?:` [`Color`](#color) [¶](#theme-background){.headerlink} +: The background color. + +{#theme-grid .member} +**`grid`**`?:` [`Color`](#color) [¶](#theme-grid){.headerlink} +: The color of a plane's or diagram's grid, a plot's grid lines, and the earth's graticule. + +{#theme-missing .member} +**`missing`**`?:` [`Color`](#color) [¶](#theme-missing){.headerlink} +: The color a colormap gives a missing value, such as a row with no value in the field. + +{#theme-dash .member} +**`dash`**`?: number` [¶](#theme-dash){.headerlink} +: The dash period: one dash and one gap of equal length. + +{#theme-shading .member} +**`shading`**`?: number` [¶](#theme-shading){.headerlink} +: How much faces darken as they turn away from the camera, from 0 to 1. + +{#theme-text .member} +**`text`**`?:` [`TextTheme`](#texttheme) [¶](#theme-text){.headerlink} +: How labels, axis ticks, and titles look. + +{#theme-vertex .member} +**`vertex`**`?:` [`VertexTheme`](#vertextheme) [¶](#theme-vertex){.headerlink} +: Defaults for vertices. + +{#theme-edge .member} +**`edge`**`?:` [`EdgeTheme`](#edgetheme) [¶](#theme-edge){.headerlink} +: Defaults for edges, and how parallel edges and arrowheads look. + +{#theme-line .member} +**`line`**`?:` [`LineTheme`](#linetheme) [¶](#theme-line){.headerlink} +: Defaults for lines. + +{#theme-area .member} +**`area`**`?:` [`AreaTheme`](#areatheme) [¶](#theme-area){.headerlink} +: Defaults for areas. + +{#theme-focus .member} +**`focus`**`?:` [`FocusTheme`](#focustheme) [¶](#theme-focus){.headerlink} +: How hovered and selected rows glow, and how rows outside a selection dim. + +{#theme-flow .member} +**`flow`**`?:` [`FlowTheme`](#flowtheme) [¶](#theme-flow){.headerlink} +: How comets flowing along edges and lines look. + +{#theme-diagram .member} +**`diagram`**`?:` [`DiagramTheme`](#diagramtheme) [¶](#theme-diagram){.headerlink} +: How a diagram's blocks, groups, ports, and wires look. + +{#theme-earth .member} +**`earth`**`?:` [`EarthTheme`](#earththeme) [¶](#theme-earth){.headerlink} +: How the earth looks. + +{#theme-plot .member} +**`plot`**`?:` [`PlotTheme`](#plottheme) [¶](#theme-plot){.headerlink} +: How a plot looks. + +{#theme-base .member} +**`base`**`?: 'dark' | 'light'` [¶](#theme-base){.headerlink} +: The theme these options change. Defaults to `dark`. + +(texttheme)= + +### `TextTheme` + +{.signature} +`interface TextTheme` + +How labels, axis ticks, and titles look. + +{#texttheme-color .member} +**`color`**`?:` [`Color`](#color) [¶](#texttheme-color){.headerlink} +: The text color. + +{#texttheme-font .member} +**`font`**`?: string` [¶](#texttheme-font){.headerlink} +: The CSS font family list. + +{#texttheme-size .member} +**`size`**`?: number` [¶](#texttheme-size){.headerlink} +: The font size, in the space's unit. + +{#texttheme-halo .member} +**`halo`**`?: number` [¶](#texttheme-halo){.headerlink} +: Width of the background-colored halo around label text, in CSS pixels. + +(vertextheme)= + +### `VertexTheme` + +{.signature} +`interface VertexTheme` + +Defaults for vertices. + +{#vertextheme-color .member} +**`color`**`?:` [`Color`](#color) [¶](#vertextheme-color){.headerlink} +: The vertex color. + +{#vertextheme-radius .member} +**`radius`**`?: number` [¶](#vertextheme-radius){.headerlink} +: The vertex radius, in the space's unit. + +(edgetheme)= + +### `EdgeTheme` + +{.signature} +`interface EdgeTheme` + +Defaults for edges, and how parallel edges and arrowheads look. + +{#edgetheme-color .member} +**`color`**`?:` [`Color`](#color) `| null` [¶](#edgetheme-color){.headerlink} +: The edge color. `null` blends the colors of each edge's two vertices. With `null`, a diagram's + wires and edges without `ends` (nets and poles) use the text color. + +{#edgetheme-width .member} +**`width`**`?: number` [¶](#edgetheme-width){.headerlink} +: The edge width, in the space's unit. + +{#edgetheme-spacing .member} +**`spacing`**`?: number` [¶](#edgetheme-spacing){.headerlink} +: In a diagram, the distance between edges joining the same two vertices and between parallel + orthogonal wires. Wires keep twice this distance from blocks where they can. + +{#edgetheme-bow .member} +**`bow`**`?: number` [¶](#edgetheme-bow){.headerlink} +: Outside a diagram, how far edges joining the same two vertices bow apart. Each lane bows out by + this fraction of the edge's length, measured at its middle. Parallel edges spread at most eight + lanes wide, and more lanes pack closer. Bows scale with the zoom. + +{#edgetheme-arrow .member} +**`arrow`**`?: number` [¶](#edgetheme-arrow){.headerlink} +: How far an arrowhead extends beyond the edge's width. A head's length is this plus the width. + +(linetheme)= + +### `LineTheme` + +{.signature} +`interface LineTheme` + +Defaults for lines. + +{#linetheme-color .member} +**`color`**`?:` [`Color`](#color) [¶](#linetheme-color){.headerlink} +: The line color. + +{#linetheme-width .member} +**`width`**`?: number` [¶](#linetheme-width){.headerlink} +: The line width, in the space's unit. + +(areatheme)= + +### `AreaTheme` + +{.signature} +`interface AreaTheme` + +Defaults for areas. + +{#areatheme-color .member} +**`color`**`?:` [`Color`](#color) [¶](#areatheme-color){.headerlink} +: The area color. + +(flowtheme)= + +### `FlowTheme` + +{.signature} +`interface FlowTheme` + +How comets along edges and lines look. A comet has a round head, brightest at its middle, and a +tail that thins and fades. + +`spacing` and `length` must use the same unit, and each row's `flow` speed is in that unit too. +In position units (degrees on the earth), comets scale with the zoom and stay in place along +their lines. In CSS pixels, they keep their size and speed on screen. A plain number is in the +space's unit. + +{#flowtheme-color .member} +**`color`**`?:` [`Color`](#color) [¶](#flowtheme-color){.headerlink} +: The comet color. Its alpha sets how strongly a comet covers its stroke. A fully transparent + color turns comets off. + +{#flowtheme-spacing .member} +**`spacing`**`?:` [`Length`](#length) [¶](#flowtheme-spacing){.headerlink} +: The distance between consecutive comets. + +{#flowtheme-length .member} +**`length`**`?:` [`Length`](#length) [¶](#flowtheme-length){.headerlink} +: The length of a comet's tail behind its head. + +{#flowtheme-width .member} +**`width`**`?: number` [¶](#flowtheme-width){.headerlink} +: The width of a comet's head, in the space's unit. A head is never narrower than its stroke. + +(focustheme)= + +### `FocusTheme` + +{.signature} +`interface FocusTheme` + +How hovered and selected rows glow, and how rows outside a selection dim. + +{#focustheme-hover .member} +**`hover`**`?:` [`Color`](#color) [¶](#focustheme-hover){.headerlink} +: The glow color around hovered rows. + +{#focustheme-selected .member} +**`selected`**`?:` [`Color`](#color) `| null` [¶](#focustheme-selected){.headerlink} +: The glow color around selected rows. `null` uses each row's own color. + +{#focustheme-glow .member} +**`glow`**`?: number` [¶](#focustheme-glow){.headerlink} +: How far the hover and selection glows reach, in CSS pixels. + +{#focustheme-dim .member} +**`dim`**`?: number` [¶](#focustheme-dim){.headerlink} +: The opacity of rows outside the selection while any row is selected, from 0 to 1. + +{#focustheme-grow .member} +**`grow`**`?: number` [¶](#focustheme-grow){.headerlink} +: The scale factor for a hovered vertex sized in CSS pixels. Vertices sized in units only glow. + +(diagramtheme)= + +### `DiagramTheme` + +{.signature} +`interface DiagramTheme` + +How a diagram's blocks, groups, ports, and wires look. + +{#diagramtheme-group .member} +**`group`**`?:` [`Color`](#color) [¶](#diagramtheme-group){.headerlink} +: The fill color of group containers. + +{#diagramtheme-outline .member} +**`outline`**`?:` [`Color`](#color) `| null` [¶](#diagramtheme-outline){.headerlink} +: The color of the one-unit border around blocks and groups. `null` draws no border. + +{#diagramtheme-corner .member} +**`corner`**`?: number` [¶](#diagramtheme-corner){.headerlink} +: The corner radius of blocks and groups. A block marker can set its own. + +{#diagramtheme-portsize .member} +**`portSize`**`?: number` [¶](#diagramtheme-portsize){.headerlink} +: The width of ports. + +{#diagramtheme-bend .member} +**`bend`**`?: number` [¶](#diagramtheme-bend){.headerlink} +: The bend radius of orthogonal wires. + +{#diagramtheme-grouppadding .member} +**`groupPadding`**`?: number` [¶](#diagramtheme-grouppadding){.headerlink} +: The space between a group's members and its border. + +{#diagramtheme-shadow .member} +**`shadow`**`?:` [`ShadowTheme`](#shadowtheme) [¶](#diagramtheme-shadow){.headerlink} +: How blocks and groups cast shadows on the sheet. + +(shadowtheme)= + +### `ShadowTheme` + +{.signature} +`interface ShadowTheme` + +How a diagram's blocks and groups cast shadows on the sheet, for vertex marks with `shadow` on. + +{#shadowtheme-color .member} +**`color`**`?:` [`Color`](#color) [¶](#shadowtheme-color){.headerlink} +: The color a shadow darkens toward. Its alpha sets how dark the shadow is under the block. + +{#shadowtheme-lift .member} +**`lift`**`?: number` [¶](#shadowtheme-lift){.headerlink} +: How far a shadow falls below its block or group. Its edge softens over the same distance. + +(earththeme)= + +### `EarthTheme` + +{.signature} +`interface EarthTheme` + +How the earth looks. + +{#earththeme-ground .member} +**`ground`**`?:` [`Color`](#color) [¶](#earththeme-ground){.headerlink} +: The ground color. + +{#earththeme-atmosphere .member} +**`atmosphere`**`?:` [`Color`](#color) [¶](#earththeme-atmosphere){.headerlink} +: The color of the air glowing around the globe, when the earth's `atmosphere` is on. + +{#earththeme-night .member} +**`night`**`?: number` [¶](#earththeme-night){.headerlink} +: The brightness of the night side when the earth has `daylight`, from 0 to 1. + +(plottheme)= + +### `PlotTheme` + +{.signature} +`interface PlotTheme` + +How a plot looks. + +{#plottheme-playhead .member} +**`playhead`**`?:` [`Color`](#color) [¶](#plottheme-playhead){.headerlink} +: The color of the line at the view's current axis coordinate. + +(length)= + +### `Length` + +{.signature} +`type Length = number | { px: number } | { units: number }` + +A length: a number in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. +`{ units }` is position units (degrees on the earth), which grow as the view zooms in. + +{#length-px .member} +`{` **`px`**`: number }` [¶](#length-px){.headerlink} +: CSS pixels, the same at any zoom. + +{#length-units .member} +`{` **`units`**`: number }` [¶](#length-units){.headerlink} +: Position units (degrees on the earth), which grow as the view zooms in. ## Marks -What a scene draws each table as. - -### Mark - -```ts -/** One drawn table of a scene: `{ vertices | edges | faces | lines | areas: options }` as JSON. */ -type Mark = - | { - /** Rows drawn as markers. */ - vertices: Vertices; - } - | { - /** Rows drawn as strokes between two vertices. */ - edges: Edges; - } - | { - /** Rows drawn as triangles between three vertices. */ - faces: Faces; - } - | { - /** Rows drawn as lines along their paths, or on a plot through their samples. */ - lines: Lines; - } - | { - /** Rows drawn as the areas their polygons cover. */ - areas: Areas; - }; -``` - -### Vertices - -```ts -/** Rows drawn as markers: points of a map, nodes of a graph, blocks of a diagram. */ -interface Vertices { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** - * Each row's x and y. Without them, the point field `geometry` names places rows; rows without - * either the scene's layout places. - */ - position?: [ExprInput, ExprInput] | null; - /** The point field placing each row where `position` is not given; `geometry` unless set. */ - geometry?: string | null; - /** - * Height above the ground: on the earth, a fraction of its radius. Not in a diagram, which is - * flat. - */ - elevation?: ExprInput; - /** CSS pixels, or position units with `units`; units in a diagram. */ - radius?: Size | null; - /** - * Width and height, for markers that are not round: CSS pixels, or position units with `units`; - * units in a diagram, where a block without a size fits its label and ports. - */ - size?: [Size, Size] | null; - /** How each row is drawn. */ - marker?: Marker; - /** A ring around each row's marker in this color; a missing or transparent color draws none. */ - ring?: ExprInput | null; - /** - * A port on each row for each reference field naming a net, which an `edges` mark of the net table - * without ends draws. Diagrams only. - */ - ports?: [reference: string, port: Port][]; - /** - * Makes each row a group: it holds the rows of `table` whose reference field names it, and is - * drawn as a container around them, its label a title in its header. Diagrams only. - */ - contains?: [table: string, reference: string] | null; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; -} -``` - -### Edges - -```ts -/** - * Rows drawn as strokes between the vertices two reference fields name: branches, links, wires. In - * a diagram, a row without ends is a net: it joins every vertex port whose reference field names - * it. - */ -interface Edges { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** The reference fields naming each row's two vertices. */ - ends?: [from: string, to: string] | null; - /** A path field of points each row runs through between its ends. */ - bends?: string | null; - /** - * How each row runs between its ends; on the earth geodesic, elsewhere straight, unless set. - * Orthogonal routes run around blocks, bends rounded; diagrams only. - */ - route?: Route | null; - /** Whether each row ends in an arrowhead at its second end; a net's at each input it reaches. */ - arrows?: boolean; - /** - * Draws each row as a pole from its own vertex down to the ground beneath it, on a plane or the - * earth; it has no ends. - */ - poles?: boolean; - /** - * Draws each row as a net on a plane or the earth: it joins the vertex rows of `table` whose - * reference field names it, as a star from their middle to each. - */ - joins?: [table: string, reference: string] | null; - /** CSS pixels, or position units with `units`; units in a diagram. */ - width?: Size | null; - /** Whether each row is dashed. */ - dash?: ExprInput; - /** - * Comets flowing along each row at this speed a second: CSS pixels, units in a diagram, or - * position units where the theme's flow is in units; negative flows backward. A speed that changes - * carries comets on from where they are. - */ - flow?: ExprInput; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; -} -``` - -### Lines - -```ts -/** - * Rows drawn as lines: along a path field (roads, rivers, borders) or a polygon field's rings, - * outlining its areas; on a plot, through a sampled field's history across the data's axis, what a - * monitor shows. On the earth paths bend with the globe and stay whole across the antimeridian. - */ -interface Lines { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** The path or polygon field each row runs along; `geometry` unless set. Not on a plot. */ - geometry?: string | null; - /** - * The sampled field each row traces across the data's axis, on a plot alone. A field not recorded - * yet draws nothing until its first frame. - */ - trace?: string | null; - /** How each row runs between its samples, on a plot. */ - interpolation?: Interpolation; - /** CSS pixels, or position units with `units`; units in a diagram. */ - width?: Size | null; - /** Whether each row is dashed. */ - dash?: ExprInput; - /** - * Comets flowing along each row at this speed a second: CSS pixels, units in a diagram, or - * position units where the theme's flow is in units; negative flows backward. A speed that changes - * carries comets on from where they are. - */ - flow?: ExprInput; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; -} -``` - -### Areas - -```ts -/** - * Rows drawn as the areas a polygon field holds: parcels, states, lakes, the land and water of a - * map. The view cuts each row's polygons into triangles once, off the frame when they are large; - * on the earth they bend with the globe and stay whole across the antimeridian. Outline areas with - * a `Lines` mark of the same table, which draws their rings. - */ -interface Areas { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** The polygon field each row fills: one polygon's rings, or polygons. `geometry` unless set. */ - geometry?: string | null; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; -} -``` - -### Faces - -```ts -/** - * Rows drawn as triangles between the vertices three reference fields name: meshes, surfaces, and - * fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen from - * above shows its exact colors. - */ -interface Faces { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** - * The reference fields naming each row's three vertices. Without a color, a face blends the colors - * of its corners. - */ - corners?: [a: string, b: string, c: string] | null; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; -} -``` - -### Marker - -```ts -/** - * How a vertex is drawn: a shape's name, `"block"`, or - * `{ shape | block | gauge | pie | icon | wgsl: options }`. - */ -type Marker = - | 'ellipse' - | 'rectangle' - | 'rounded' - | 'diamond' - | 'block' - | { - /** A filled shape. */ - shape: Shape; - } - | { - /** A block: a rectangle with rounded corners, sized for diagrams. */ - block: { - /** Corner radius, CSS pixels, or units in a diagram; the theme's `corner` unless set. */ - corner?: number | null; - }; - } - | { - /** A shape's outline, filled to `fill` from 0 to 1: around a circle, up other shapes. */ - gauge: { - /** How full each row's gauge is. */ - fill: ExprInput; - /** The outline. */ - shape?: Shape; - }; - } - | { - /** A circle cut into slices, each as large as its value. */ - pie: { - /** Each slice's value. */ - slices: ExprInput[]; - /** Each slice's color. */ - colors: Color[]; - /** The hole in the middle, as a fraction of the radius. */ - hole?: number; - }; - } - | { - /** One of `images`, fitted in the marker's box; `index` picks which for each row. */ - icon: { - /** The images. */ - images: Bitmap[]; - /** Each row's image, from 0. */ - index: ExprInput; - }; - } - | { - /** - * A marker in WGSL: `source` defines `fn marker(f: MarkerFragment) -> MarkerColor`, reading up to - * eight `inputs` as `f.input[i]`. It compiles once, when the view first draws it. - */ - wgsl: { - /** The function's source. */ - source: string; - /** Each row's inputs. */ - inputs?: ExprInput[]; - }; - }; -``` - -### Shape - -```ts -/** A marker's outline. */ -type Shape = 'ellipse' | 'rectangle' | 'rounded' | 'diamond'; -``` - -### Bitmap - -```ts -/** RGBA texels, straight alpha, sRGB, rows top to bottom. */ -interface Bitmap { - /** Width and height in texels. */ - size: [number, number]; - /** `size[0] * size[1] * 4` bytes. */ - rgba: number[]; -} -``` - -### Port - -```ts -/** - * A vertex's port: where a net its reference field names meets it, on one side of its marker: a - * side, or options. An output drives its net: layouts rank a net from its outputs, and its arrows - * point away from them. - */ -type Port = - | 'left' - | 'right' - | 'top' - | 'bottom' - | { - /** The side it sits on. */ - side: Side; - /** - * Its place among the ports on that side, from the top or the left; in declaration order unless - * set. - */ - order?: number | null; - /** Its name, drawn beside it inside the marker. */ - label?: string | null; - /** Its marker. */ - shape?: Shape; - /** Whether it drives the net its reference field names; an input unless set. */ - output?: boolean; - }; -``` - -### Side - -```ts -/** A side of a marker. */ -type Side = 'left' | 'right' | 'top' | 'bottom'; -``` - -### End - -```ts -/** A vertex a connection ends at, and its port when it has the one the connection meets. */ -interface End { - /** The vertex. */ - row: Row; - /** The port, by the reference field naming its net. */ - port?: string; -} -``` - -### Route - -```ts -/** How an edge runs between its ends. */ -type Route = 'straight' | 'geodesic' | 'orthogonal'; -``` - -### Interpolation - -```ts -/** How a line runs between its samples. */ -type Interpolation = 'linear' | 'stepBefore' | 'stepAfter'; -``` - -### Label - -```ts -type Label = - | ExprInput - | { - /** A text field. */ - text: ExprInput; - /** CSS pixels, or units in a diagram; the theme's font size unless set. */ - size?: number | null; - /** Each row's text color; the theme's text color unless set. */ - color?: ExprInput | null; - /** Where labels would overlap, the higher shows; the mark's priority unless set. */ - priority?: ExprInput | null; - /** How far a line runs before it wraps: CSS pixels, or units in a diagram. */ - maxWidth?: number | null; - /** The CSS font family list its texts are set in; the theme's font unless set. */ - font?: string | null; - /** - * The most labels shown at once, those of highest priority among the rows in view; where they - * would overlap, fewer. - */ - max?: number | null; - }; -``` - -### Size - -```ts -/** A size: CSS pixels, or position units with `units`. */ -type Size = ExprInput | Units; -``` - -### Units - -```ts -/** A size in position units, so it scales as the view zooms, kept within CSS pixels when given. */ -interface Units { - units: ExprInput; - /** CSS pixels it keeps at least, however far the view zooms out. */ - atLeast?: number | null; - /** CSS pixels it keeps at most, however far the view zooms in. */ - atMost?: number | null; -} -``` +The marks that draw a scene's tables. + +(mark)= + +### `Mark` + +{.signature} +`type Mark =` [`Vertices`](#vertices) `|` [`Edges`](#edges) `|` [`Faces`](#faces) `|` [`Lines`](#lines) `|` [`Areas`](#areas) + +One table a scene draws. The mark's kind is the key that holds its table, next to its options: +`{ vertices: [data, table], radius: 4 }`. + +(markoptions)= + +### `MarkOptions` + +{.signature} +`interface MarkOptions` + +The options every mark has. + +{#markoptions-color .member} +**`color`**`?:` [`ExprInput`](#exprinput) `| null` [¶](#markoptions-color){.headerlink} +: Each row's color: a CSS color, a value through a colormap, or a conditional that picks between + colors. Defaults to the theme's color for the mark. Faces default to a blend of their corners' + colors. + +{#markoptions-visible .member} +**`visible`**`?:` [`ExprInput`](#exprinput) [¶](#markoptions-visible){.headerlink} +: Whether each row is drawn and can be picked. Defaults to `true`. + +{#markoptions-priority .member} +**`priority`**`?:` [`ExprInput`](#exprinput) `| null` [¶](#markoptions-priority){.headerlink} +: Each row's priority. It reads only data, not view state. Higher rows draw on top, keep their + labels where labels overlap, and are dropped last when a large table is thinned. + +{#markoptions-name .member} +**`name`**`?: string | null` [¶](#markoptions-name){.headerlink} +: Distinguishes two marks of the same table across scenes, and names the mark in hits. + +{#markoptions-pick .member} +**`pick`**`?: boolean | null` [¶](#markoptions-pick){.headerlink} +: Whether rows respond to hover, clicks, and picks. Defaults to on for vertices, edges, faces, and + plot traces, and off for geometry: areas, and lines along paths such as coasts, borders, and + roads. + +{#markoptions-select .member} +**`select`**`?: boolean` [¶](#markoptions-select){.headerlink} +: Whether clicking a row selects it. Defaults to `true`. + +{#markoptions-drag .member} +**`drag`**`?: boolean` [¶](#markoptions-drag){.headerlink} +: Whether pressing and moving a row drags it instead of panning the camera. Drags are reported as + a `'drag'` event. Defaults to `false`. + +{#markoptions-label .member} +**`label`**`?:` [`Label`](#label) `| null` [¶](#markoptions-label){.headerlink} +: Each row's label: a text field, or label options. + +{#markoptions-legend .member} +**`legend`**`?: string | null` [¶](#markoptions-legend){.headerlink} +: A legend of the mark's colors at the view's bottom right, with this title. It shows a ticked + colormap across its domain, a cell for each category, or a swatch for each name in the + colormap's `categories`. `color` must map through a colormap with a fixed domain or named + categories. In a conditional, `otherwise` is searched before the cases. + +(strokeoptions)= + +### `StrokeOptions` + +{.signature} +`interface StrokeOptions` + +The options of marks drawn as strokes: edges and lines. + +{#strokeoptions-width .member} +**`width`**`?:` [`Size`](#size) `| null` [¶](#strokeoptions-width){.headerlink} +: Each row's width in CSS pixels, or units in a diagram, unless given with `px` or `units`. + Defaults to the theme's width. + +{#strokeoptions-dash .member} +**`dash`**`?:` [`ExprInput`](#exprinput) [¶](#strokeoptions-dash){.headerlink} +: Whether each row is dashed, using the theme's `dash` period. Defaults to `false`. + +{#strokeoptions-flow .member} +**`flow`**`?:` [`ExprInput`](#exprinput) [¶](#strokeoptions-flow){.headerlink} +: The speed of comets along each row, per second, in the unit of the theme's `flow` lengths. + Negative speeds flow backward. 0 draws no comets. Defaults to 0. When the speed changes, comets + continue from where they are. + +(vertices)= + +### `Vertices` + +{.signature} +`interface Vertices extends` [`MarkOptions`](#markoptions) + +Rows drawn as markers: points of a map, nodes of a graph, blocks of a diagram. + +{#vertices-vertices .member} +**`vertices`**`:` [`TableInput`](#tableinput) [¶](#vertices-vertices){.headerlink} +: The dataset and table drawn, `[data, table]`. + +{#vertices-position .member} +**`position`**`?: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] | null` [¶](#vertices-position){.headerlink} +: Each row's x and y. Rows without a position are placed by the point field `geometry` names. The + scene's layout places rows that have neither. + +{#vertices-geometry .member} +**`geometry`**`?: string | null` [¶](#vertices-geometry){.headerlink} +: The point field that places rows when `position` is not set. Defaults to the field named + `geometry`. + +{#vertices-elevation .member} +**`elevation`**`?:` [`ExprInput`](#exprinput) [¶](#vertices-elevation){.headerlink} +: Height above the ground. On the earth, a fraction of the earth's radius. Defaults to 0. Not + allowed in a diagram, which is flat. + +{#vertices-radius .member} +**`radius`**`?:` [`Size`](#size) `| null` [¶](#vertices-radius){.headerlink} +: Each row's radius in CSS pixels, or units in a diagram, unless given with `px` or `units`. A + diagram does not accept `px`. Defaults to the theme's `vertex.radius`. + +{#vertices-size .member} +**`size`**`?: [`[`Size`](#size)`,` [`Size`](#size)`] | null` [¶](#vertices-size){.headerlink} +: Each row's width and height, for markers that are not round. In CSS pixels, or units in a + diagram, unless given with `px` or `units`. A diagram does not accept `px`. In a diagram, a + block without a size fits its label and ports. + +{#vertices-marker .member} +**`marker`**`?:` [`Marker`](#marker) [¶](#vertices-marker){.headerlink} +: How each row is drawn. Defaults to an ellipse. + +{#vertices-ring .member} +**`ring`**`?:` [`ExprInput`](#exprinput) `| null` [¶](#vertices-ring){.headerlink} +: A ring around each row's marker in this color. A missing or transparent color draws no ring. + +{#vertices-ports .member} +**`ports`**`?: [reference: string, port:` [`Port`](#port)`][]` [¶](#vertices-ports){.headerlink} +: Ports on each row, one for each reference field that names a net. An `edges` mark of the net + table, without `ends`, draws the nets. Diagrams only. + +{#vertices-contains .member} +**`contains`**`?: [table: string, reference: string] | null` [¶](#vertices-contains){.headerlink} +: Makes each row a group containing the rows of `table` whose `reference` field names it. A group + draws as a container around its members, with its label as a title in its header. Diagrams only. + +{#vertices-shadow .member} +**`shadow`**`?: boolean` [¶](#vertices-shadow){.headerlink} +: Whether each block or group casts a shadow on the sheet, styled by the theme's `diagram.shadow`. + Defaults to `false`. Diagrams only. + +(edges)= + +### `Edges` + +{.signature} +`interface Edges extends` [`MarkOptions`](#markoptions)`,` [`StrokeOptions`](#strokeoptions) + +Rows drawn as strokes between two vertices named by reference fields: branches, links, wires. In +a diagram, a row without ends is a net. It joins every vertex port whose reference field names +it. + +{#edges-edges .member} +**`edges`**`:` [`TableInput`](#tableinput) [¶](#edges-edges){.headerlink} +: The dataset and table drawn, `[data, table]`. + +{#edges-ends .member} +**`ends`**`?: [from: string, to: string] | null` [¶](#edges-ends){.headerlink} +: The reference fields naming each row's two vertices. + +{#edges-bends .member} +**`bends`**`?: string | null` [¶](#edges-bends){.headerlink} +: A path field whose points each row passes through between its ends. + +{#edges-route .member} +**`route`**`?:` [`Route`](#route) `| null` [¶](#edges-route){.headerlink} +: How each row runs between its ends. Defaults to geodesic on the earth and straight elsewhere. + `orthogonal` routes around blocks with rounded bends, in diagrams only. + +{#edges-arrows .member} +**`arrows`**`?: boolean` [¶](#edges-arrows){.headerlink} +: Whether each row has an arrowhead at its second end. A net has one at each input it reaches. + Defaults to `false`. + +{#edges-poles .member} +**`poles`**`?: boolean` [¶](#edges-poles){.headerlink} +: Draws each row as a pole from its own vertex down to the ground, on a plane or the earth. Poles + take no `ends`, `bends`, or `joins`. Defaults to `false`. + +{#edges-joins .member} +**`joins`**`?: [table: string, reference: string] | null` [¶](#edges-joins){.headerlink} +: Draws each row as a net on a plane or the earth. The net joins the vertex rows of `table` whose + `reference` field names it, as a star from their center to each member. + +(lines)= + +### `Lines` + +{.signature} +`interface Lines extends` [`MarkOptions`](#markoptions)`,` [`StrokeOptions`](#strokeoptions) + +Rows drawn as lines along a path field (roads, rivers, borders) or a polygon field's rings (area +outlines). On a plot, each row traces a sampled field's history across the data's axis, like a +monitor. On the earth, paths follow the globe's curve and stay unbroken across the antimeridian. + +{#lines-lines .member} +**`lines`**`:` [`TableInput`](#tableinput) [¶](#lines-lines){.headerlink} +: The dataset and table drawn, `[data, table]`. + +{#lines-geometry .member} +**`geometry`**`?: string | null` [¶](#lines-geometry){.headerlink} +: The path or polygon field each row runs along. Defaults to the field named `geometry`. Not + allowed on a plot. + +{#lines-trace .member} +**`trace`**`?: string | null` [¶](#lines-trace){.headerlink} +: The sampled field each row traces across the data's axis. Required on a plot, and allowed only + there. A field with no frames yet draws nothing until its first frame arrives. + +{#lines-interpolation .member} +**`interpolation`**`?:` [`Interpolation`](#interpolation) [¶](#lines-interpolation){.headerlink} +: How each row runs between its samples on a plot. Defaults to `linear`. + +(areas)= + +### `Areas` + +{.signature} +`interface Areas extends` [`MarkOptions`](#markoptions) + +Rows drawn as the areas of a polygon field: parcels, states, lakes, the land and water of a map. +The view triangulates each row's polygons once, in the background when they are large. On the +earth, areas follow the globe's curve and stay unbroken across the antimeridian. To outline +areas, add a [`Lines`](#lines) mark of the same table. It draws their rings. + +{#areas-areas .member} +**`areas`**`:` [`TableInput`](#tableinput) [¶](#areas-areas){.headerlink} +: The dataset and table drawn, `[data, table]`. + +{#areas-geometry .member} +**`geometry`**`?: string | null` [¶](#areas-geometry){.headerlink} +: The polygon field each row fills. It holds one polygon's rings or several polygons. Defaults to + the field named `geometry`. + +(faces)= + +### `Faces` + +{.signature} +`interface Faces extends` [`MarkOptions`](#markoptions) + +Rows drawn as triangles between three vertices named by reference fields: meshes, surfaces, and +fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen from +above shows its exact colors. + +{#faces-faces .member} +**`faces`**`:` [`TableInput`](#tableinput) [¶](#faces-faces){.headerlink} +: The dataset and table drawn, `[data, table]`. + +{#faces-corners .member} +**`corners`**`?: [a: string, b: string, c: string] | null` [¶](#faces-corners){.headerlink} +: The reference fields naming each row's three vertices. Required. Without a color, a face blends + the colors of its corners. + +(marker)= + +### `Marker` + +{.signature} +`type Marker = 'ellipse' | 'rectangle' | 'rounded' | 'diamond' | 'block' | { shape:` [`Shape`](#shape) `} | { block: { … } } | { gauge: { … } } | { pie: { … } } | { icon: { … } } | { wgsl: { … } }` + +How a vertex is drawn: a shape's name, `"block"`, or +`{ shape | block | gauge | pie | icon | wgsl: options }`. Defaults to an ellipse. + +{#marker-shape .member} +`{` **`shape`**`:` [`Shape`](#shape) `}` [¶](#marker-shape){.headerlink} +: A filled shape. + +{#marker-block .member} +`{` **`block`**`: { corner?: number | null } }` [¶](#marker-block){.headerlink} +: A rectangle with rounded corners that holds its label inside, like a diagram's blocks. + + {#marker-block-corner .member} + **`corner`**`?: number | null` [¶](#marker-block-corner){.headerlink} + : Corner radius in CSS pixels, or units in a diagram. Defaults to the theme's `diagram.corner`. + +{#marker-gauge .member} +`{` **`gauge`**`: { fill:` [`ExprInput`](#exprinput)`; shape?:` [`Shape`](#shape) `} }` [¶](#marker-gauge){.headerlink} +: A shape's outline, filled to `fill` (0 to 1). A circle fills around its edge, and other shapes + fill upward. + + {#marker-gauge-fill .member} + **`fill`**`:` [`ExprInput`](#exprinput) [¶](#marker-gauge-fill){.headerlink} + : How full each row's gauge is, from 0 to 1. + + {#marker-gauge-shape .member} + **`shape`**`?:` [`Shape`](#shape) [¶](#marker-gauge-shape){.headerlink} + : The outline's shape. Defaults to an ellipse. + +{#marker-pie .member} +`{` **`pie`**`: { slices:` [`ExprInput`](#exprinput)`[]; colors:` [`Color`](#color)`[]; hole?: number } }` [¶](#marker-pie){.headerlink} +: A circle divided into slices sized in proportion to their values. + + {#marker-pie-slices .member} + **`slices`**`:` [`ExprInput`](#exprinput)`[]` [¶](#marker-pie-slices){.headerlink} + : Each slice's value. + + {#marker-pie-colors .member} + **`colors`**`:` [`Color`](#color)`[]` [¶](#marker-pie-colors){.headerlink} + : Each slice's color. + + {#marker-pie-hole .member} + **`hole`**`?: number` [¶](#marker-pie-hole){.headerlink} + : The hole in the middle, as a fraction of the radius. Defaults to 0. + +{#marker-icon .member} +`{` **`icon`**`: { images:` [`Bitmap`](#bitmap)`[]; index:` [`ExprInput`](#exprinput) `} }` [¶](#marker-icon){.headerlink} +: One of `images`, fitted to the marker's box. `index` picks the image for each row. + + {#marker-icon-images .member} + **`images`**`:` [`Bitmap`](#bitmap)`[]` [¶](#marker-icon-images){.headerlink} + : The images. + + {#marker-icon-index .member} + **`index`**`:` [`ExprInput`](#exprinput) [¶](#marker-icon-index){.headerlink} + : Each row's image index, from 0. + +{#marker-wgsl .member} +`{` **`wgsl`**`: { source: string; inputs?:` [`ExprInput`](#exprinput)`[] } }` [¶](#marker-wgsl){.headerlink} +: A custom marker in WGSL. `source` defines `fn marker(f: MarkerFragment) -> MarkerColor`, which + can read up to eight `inputs` as `f.input[i]`. It compiles once, when the view first draws it. + + {#marker-wgsl-source .member} + **`source`**`: string` [¶](#marker-wgsl-source){.headerlink} + : The WGSL source of the `marker` function. + + {#marker-wgsl-inputs .member} + **`inputs`**`?:` [`ExprInput`](#exprinput)`[]` [¶](#marker-wgsl-inputs){.headerlink} + : Each row's inputs, at most eight. + +(shape)= + +### `Shape` + +{.signature} +`type Shape = 'ellipse' | 'rectangle' | 'rounded' | 'diamond'` + +A marker's outline. + +{#shape-ellipse .member} +**`'ellipse'`** [¶](#shape-ellipse){.headerlink} +: An ellipse: a circle when its width and height are equal. + +{#shape-rectangle .member} +**`'rectangle'`** [¶](#shape-rectangle){.headerlink} +: A rectangle. + +{#shape-rounded .member} +**`'rounded'`** [¶](#shape-rounded){.headerlink} +: A rectangle with rounded corners. + +{#shape-diamond .member} +**`'diamond'`** [¶](#shape-diamond){.headerlink} +: A diamond. + +(bitmap)= + +### `Bitmap` + +{.signature} +`interface Bitmap` + +An image of RGBA texels: straight alpha, sRGB, rows top to bottom. + +{#bitmap-size .member} +**`size`**`: [number, number]` [¶](#bitmap-size){.headerlink} +: Width and height in texels. + +{#bitmap-rgba .member} +**`rgba`**`: number[]` [¶](#bitmap-rgba){.headerlink} +: `size[0] * size[1] * 4` bytes. + +(port)= + +### `Port` + +{.signature} +`type Port = 'left' | 'right' | 'top' | 'bottom' | { … }` + +A port on one side of a vertex's marker, where a net connects: a side name, or options. The +vertex names the net in a reference field. An output drives its net: layouts rank a net from its +outputs, and the net's arrows point away from them. + +{#port-side .member} +**`side`**`:` [`Side`](#side) [¶](#port-side){.headerlink} +: The side of the marker it sits on. + +{#port-order .member} +**`order`**`?: number | null` [¶](#port-order){.headerlink} +: Its position among the ports on its side, counted from the top or left. Ports without an `order` + follow those with one, in declaration order. + +{#port-label .member} +**`label`**`?: string | null` [¶](#port-label){.headerlink} +: Its name, drawn beside it inside the marker. + +{#port-shape .member} +**`shape`**`?:` [`Shape`](#shape) [¶](#port-shape){.headerlink} +: Its shape. Defaults to an ellipse. + +{#port-output .member} +**`output`**`?: boolean` [¶](#port-output){.headerlink} +: Whether it is an output, which drives its net. Defaults to `false` (an input). + +(side)= + +### `Side` + +{.signature} +`type Side = 'left' | 'right' | 'top' | 'bottom'` + +A side of a marker. + +{#side-left .member} +**`'left'`** [¶](#side-left){.headerlink} +: The left side. + +{#side-right .member} +**`'right'`** [¶](#side-right){.headerlink} +: The right side. + +{#side-top .member} +**`'top'`** [¶](#side-top){.headerlink} +: The top. + +{#side-bottom .member} +**`'bottom'`** [¶](#side-bottom){.headerlink} +: The bottom. + +(route)= + +### `Route` + +{.signature} +`type Route = 'straight' | 'geodesic' | 'orthogonal'` + +How an edge runs between its ends. + +{#route-straight .member} +**`'straight'`** [¶](#route-straight){.headerlink} +: A straight line on screen. + +{#route-geodesic .member} +**`'geodesic'`** [¶](#route-geodesic){.headerlink} +: Along the great circle, on the earth. + +{#route-orthogonal .member} +**`'orthogonal'`** [¶](#route-orthogonal){.headerlink} +: Horizontal and vertical runs around vertices that leave and enter at their sides, with rounded + bends. Diagrams only. + +(interpolation)= + +### `Interpolation` + +{.signature} +`type Interpolation = 'linear' | 'stepBefore' | 'stepAfter'` + +How a line runs between its samples. + +{#interpolation-linear .member} +**`'linear'`** [¶](#interpolation-linear){.headerlink} +: Straight from each sample to the next. + +{#interpolation-stepbefore .member} +**`'stepBefore'`** [¶](#interpolation-stepbefore){.headerlink} +: Each value holds from the previous sample up to its own, so the line steps at the previous + sample. + +{#interpolation-stepafter .member} +**`'stepAfter'`** [¶](#interpolation-stepafter){.headerlink} +: Each value holds until the next sample, where the line steps. + +(label)= + +### `Label` + +{.signature} +`type Label =` [`ExprInput`](#exprinput) `| { … }` + +Text for each row, placed beside, inside, or along the row's shape and moving with it: a text +field, or options. A label shows when its row is visible and its text fits, for example inside a +block. + +Labels beside or along their rows never overlap. Where they would, the row drawn on top keeps +its label (so `priority` decides), and a dense view labels as many rows as fit. A row whose text +is missing or empty has no label, so a `when` that returns the field or missing labels only some +rows. In a diagram, a label scales with the zoom, fades out when too small to read, and shows +wherever it fits its vertex. + +{#label-text .member} +**`text`**`:` [`ExprInput`](#exprinput) [¶](#label-text){.headerlink} +: The text field to show, or a `when` that returns that field or missing. A row whose text is + missing or empty has no label. In a diagram, the text may read only data, since the sheet sizes + blocks around it. + +{#label-size .member} +**`size`**`?: number | null` [¶](#label-size){.headerlink} +: Font size in CSS pixels, or diagram units in a diagram. Defaults to the theme's `text.size`. + +{#label-color .member} +**`color`**`?:` [`ExprInput`](#exprinput) `| null` [¶](#label-color){.headerlink} +: Each row's text color. Defaults to the theme's text color. + +{#label-maxwidth .member} +**`maxWidth`**`?: number | null` [¶](#label-maxwidth){.headerlink} +: The width at which lines wrap, in CSS pixels, or diagram units in a diagram. + +{#label-font .member} +**`font`**`?: string | null` [¶](#label-font){.headerlink} +: The CSS font family list for the text. Defaults to the theme's `text.font`. + +(size)= + +### `Size` + +{.signature} +`type Size =` [`ExprInput`](#exprinput) `|` [`Px`](#px) `|` [`Units`](#units) + +A size: a value in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. `{ units }` +is position units (degrees on the earth), which grow as the view zooms in. + +(px)= + +### `Px` + +{.signature} +`interface Px` + +A size in CSS pixels, regardless of the space's unit. + +{#px-px .member} +**`px`**`:` [`ExprInput`](#exprinput) [¶](#px-px){.headerlink} +: The size, in CSS pixels. + +(units)= + +### `Units` + +{.signature} +`interface Units` + +A size in position units (degrees on the earth), regardless of the space's unit. It scales with +the zoom. `atLeast` and `atMost` bound it in CSS pixels. + +{#units-units .member} +**`units`**`:` [`ExprInput`](#exprinput) [¶](#units-units){.headerlink} +: The size, in position units. + +{#units-atleast .member} +**`atLeast`**`?: number | null` [¶](#units-atleast){.headerlink} +: The minimum on-screen size in CSS pixels, however far the view zooms out. + +{#units-atmost .member} +**`atMost`**`?: number | null` [¶](#units-atmost){.headerlink} +: The maximum on-screen size in CSS pixels, however far the view zooms in. + +## Expressions + +Expressions as JSON: a number, a boolean, `null`, a field as `"$name"`, a CSS color, or one of these nodes. + +(exprnode)= + +### `ExprNode` + +{.signature} +`type ExprNode = number | boolean | null | string |` [`FieldNode`](#fieldnode) `|` [`StateNode`](#statenode) `|` [`WhenNode`](#whennode) `|` [`ScaleNode`](#scalenode) `|` [`ColormapNode`](#colormapnode) `|` [`CallNode`](#callnode) + +A value for each row. A number or boolean is a constant and `null` is missing. A string starting +with `$` reads that field, and any other string is a CSS color. An object computes a value. + +{#exprnode-null .member} +**`null`** [¶](#exprnode-null){.headerlink} +: A missing value. An option that is missing uses the theme's value. + +(fieldnode)= + +### `FieldNode` + +{.signature} +`interface FieldNode` + +A field's value in each row, from the mark's own table. With `from`, it reads another dataset's +table, joined by row id. `"$name"` is the short form of `{ "field": "name" }`. + +{#fieldnode-field .member} +**`field`**`: string` [¶](#fieldnode-field){.headerlink} +: The field's name. + +{#fieldnode-from .member} +**`from`**`?:` [`TableInput`](#tableinput) `| null` [¶](#fieldnode-from){.headerlink} +: The dataset and table the field is read from, joined by row id. + +{#fieldnode-step .member} +**`step`**`?: boolean` [¶](#fieldnode-step){.headerlink} +: For a sampled field, reads the last frame at or before the shown coordinate instead of + interpolating. Defaults to `false`. + +(statenode)= + +### `StateNode` + +{.signature} +`interface StateNode` + +View state of each row, such as whether it is selected. + +{#statenode-state .member} +**`state`**`:` [`State`](#state) [¶](#statenode-state){.headerlink} +: The state read. + +(callnode)= + +### `CallNode` + +{.signature} +`type CallNode = { neg: [`[`ExprInput`](#exprinput)`] } | { not: [`[`ExprInput`](#exprinput)`] } | { abs: [`[`ExprInput`](#exprinput)`] } | { sqrt: [`[`ExprInput`](#exprinput)`] } | { sin: [`[`ExprInput`](#exprinput)`] } | { cos: [`[`ExprInput`](#exprinput)`] } | { floor: [`[`ExprInput`](#exprinput)`] } | { ln: [`[`ExprInput`](#exprinput)`] } | { exp: [`[`ExprInput`](#exprinput)`] } | { add: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { sub: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { mul: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { div: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { rem: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { pow: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { min: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { max: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { lt: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { le: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { gt: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { ge: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { eq: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { ne: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { and: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { or: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] } | { is: [field:` [`ExprInput`](#exprinput)`, text: string] } | { contains: [field:` [`ExprInput`](#exprinput)`, text: string] }` + +An operation, written as an object with one key: the operation's name, holding its arguments. +For example `{ "add": ["$a", 1] }`. The second argument of `is` and `contains` is plain text, +not a field or a color. + +{#callnode-neg .member} +`{` **`neg`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-neg){.headerlink} +: The value negated. + +{#callnode-not .member} +`{` **`not`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-not){.headerlink} +: 1 where the value is at most 0.5, else 0. + +{#callnode-abs .member} +`{` **`abs`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-abs){.headerlink} +: The absolute value. + +{#callnode-sqrt .member} +`{` **`sqrt`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-sqrt){.headerlink} +: The square root. Negative values give a missing value. + +{#callnode-sin .member} +`{` **`sin`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-sin){.headerlink} +: The sine of the value, an angle in radians. + +{#callnode-cos .member} +`{` **`cos`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-cos){.headerlink} +: The cosine of the value, an angle in radians. + +{#callnode-floor .member} +`{` **`floor`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-floor){.headerlink} +: The largest whole number at most the value. + +{#callnode-ln .member} +`{` **`ln`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-ln){.headerlink} +: The natural logarithm. Values at most 0 give a missing value. + +{#callnode-exp .member} +`{` **`exp`**`: [`[`ExprInput`](#exprinput)`] }` [¶](#callnode-exp){.headerlink} +: e raised to the value. + +{#callnode-add .member} +`{` **`add`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-add){.headerlink} +: The sum of the two values. + +{#callnode-sub .member} +`{` **`sub`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-sub){.headerlink} +: The first value minus the second. + +{#callnode-mul .member} +`{` **`mul`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-mul){.headerlink} +: The product of the two values. + +{#callnode-div .member} +`{` **`div`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-div){.headerlink} +: The first value divided by the second. Dividing by 0 gives a missing value. + +{#callnode-rem .member} +`{` **`rem`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-rem){.headerlink} +: The remainder of the first value divided by the second, with the sign of the second: -1 and 3 + give 2. + +{#callnode-pow .member} +`{` **`pow`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-pow){.headerlink} +: The first value raised to the power of the second. + +{#callnode-min .member} +`{` **`min`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-min){.headerlink} +: The smaller of the two values. + +{#callnode-max .member} +`{` **`max`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-max){.headerlink} +: The larger of the two values. + +{#callnode-lt .member} +`{` **`lt`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-lt){.headerlink} +: 1 where the first value is less than the second, else 0. + +{#callnode-le .member} +`{` **`le`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-le){.headerlink} +: 1 where the first value is at most the second, else 0. + +{#callnode-gt .member} +`{` **`gt`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-gt){.headerlink} +: 1 where the first value is greater than the second, else 0. + +{#callnode-ge .member} +`{` **`ge`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-ge){.headerlink} +: 1 where the first value is at least the second, else 0. + +{#callnode-eq .member} +`{` **`eq`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-eq){.headerlink} +: 1 where the two values are equal, else 0. Colors compare too. + +{#callnode-ne .member} +`{` **`ne`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-ne){.headerlink} +: 1 where the two values differ, else 0. Colors compare too. + +{#callnode-and .member} +`{` **`and`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-and){.headerlink} +: 1 where both values are above 0.5, else 0. + +{#callnode-or .member} +`{` **`or`**`: [`[`ExprInput`](#exprinput)`,` [`ExprInput`](#exprinput)`] }` [¶](#callnode-or){.headerlink} +: 1 where either value is above 0.5, else 0. + +{#callnode-is .member} +`{` **`is`**`: [field:` [`ExprInput`](#exprinput)`, text: string] }` [¶](#callnode-is){.headerlink} +: Whether a text field equals the plain text exactly. + +{#callnode-contains .member} +`{` **`contains`**`: [field:` [`ExprInput`](#exprinput)`, text: string] }` [¶](#callnode-contains){.headerlink} +: Whether a text field contains the plain text, ignoring case. In a mark, the field must belong to + the mark's own table. + +(whennode)= + +### `WhenNode` + +{.signature} +`interface WhenNode` + +The value of the first case whose condition holds, else `otherwise`. + +{#whennode-when .member} +**`when`**`: [condition:` [`ExprInput`](#exprinput)`, value:` [`ExprInput`](#exprinput)`][]` [¶](#whennode-when){.headerlink} +: The cases, as `[condition, value]` pairs tried in order. + +{#whennode-otherwise .member} +**`otherwise`**`:` [`ExprInput`](#exprinput) [¶](#whennode-otherwise){.headerlink} +: The value where no condition holds. + +(scalenode)= + +### `ScaleNode` + +{.signature} +`interface ScaleNode` + +A value mapped linearly onto a range, clamped to the range. + +{#scalenode-scale .member} +**`scale`**`: [low: number, high: number]` [¶](#scalenode-scale){.headerlink} +: The range. + +{#scalenode-value .member} +**`value`**`:` [`ExprInput`](#exprinput) [¶](#scalenode-value){.headerlink} +: The value mapped. + +{#scalenode-domain .member} +**`domain`**`?: [low: number, high: number] | null` [¶](#scalenode-domain){.headerlink} +: The values mapped to the range's ends. Defaults to the extent of the values shown. + +(colormapnode)= + +### `ColormapNode` + +{.signature} +`interface ColormapNode` + +A value mapped through a colormap. + +{#colormapnode-colormap .member} +**`colormap`**`:` [`Colormap`](#colormap) [¶](#colormapnode-colormap){.headerlink} +: The colormap. + +{#colormapnode-value .member} +**`value`**`:` [`ExprInput`](#exprinput) [¶](#colormapnode-value){.headerlink} +: The value mapped. A categorical map takes each value's integer part as its category. + +{#colormapnode-domain .member} +**`domain`**`?: [low: number, high: number] | null` [¶](#colormapnode-domain){.headerlink} +: The values mapped to the colormap's ends. Defaults to the extent of the values shown. A + categorical map ignores it. + +(state)= + +### `State` + +{.signature} +`type State = 'selected' | 'hovered' | 'pointer' | 'elapsed' | 'zoom'` + +View state of each row: `selected`, `hovered`, `pointer` (CSS pixels from a vertex to the +pointer), `elapsed` (seconds since the view was created), or `zoom` (the camera's zoom). + +(colormap)= -## Expressions +### `Colormap` + +{.signature} +`type Colormap =` [`ColormapName`](#colormapname) `|` [`CustomColormap`](#customcolormap) + +Colors for a range of values: a catalog name, or your own colors. + +(colormapname)= + +### `ColormapName` + +{.signature} +`type ColormapName = 'amber' | 'amp' | 'batlow' | 'cividis' | 'crest' | 'deep' | 'devon' | 'flare' | 'grays' | 'haline' | 'inferno' | 'lajolla' | 'magma' | 'mako' | 'oslo' | 'plasma' | 'rocket' | 'thermal' | 'turku' | 'viridis' | 'balance' | 'berlin' | 'brbg' | 'broc' | 'coolwarm' | 'cork' | 'icefire' | 'managua' | 'piyg' | 'puor' | 'rdbu' | 'spectral' | 'vanimo' | 'vik' | 'vlag' | 'phase' | 'romao' | 'twilight' | 'dark2' | 'okabeito' | 'paired' | 'set2' | 'tab10' | 'tab20' | 'tableau10' | 'turbo'` + +A colormap from the catalog. + +(colormapkind)= + +### `ColormapKind` + +{.signature} +`type ColormapKind = 'sequential' | 'diverging' | 'cyclic' | 'categorical'` + +How a colormap maps values: `sequential` and `diverging` maps clamp, `cyclic` maps wrap, and +`categorical` maps give each integer its own color. + +{#colormapkind-sequential .member} +**`'sequential'`** [¶](#colormapkind-sequential){.headerlink} +: Low to high. + +{#colormapkind-diverging .member} +**`'diverging'`** [¶](#colormapkind-diverging){.headerlink} +: Below and above a midpoint. + +{#colormapkind-cyclic .member} +**`'cyclic'`** [¶](#colormapkind-cyclic){.headerlink} +: Around a circle: values wrap. + +{#colormapkind-categorical .member} +**`'categorical'`** [¶](#colormapkind-categorical){.headerlink} +: Distinct colors for distinct integers. + +(customcolormap)= -Expressions as JSON: a constant, a field, or a node of these kinds. - -### ExprNode - -```ts -/** A value for every row of a mark: a constant, a field, view state, or arithmetic of them. */ -type ExprNode = - | number - | boolean - | string - | FieldNode - | ColorNode - | StateNode - | MissingNode - | UnaryNode - | BinaryNode - | WhenNode - | IsNode - | ContainsNode - | ScaleNode - | ColormapNode; -``` - -### FieldNode - -```ts -/** A field of each row: of the mark's own table, or of another dataset's table joined by row id. */ -interface FieldNode { - field: string; - /** The dataset and table the field is read from, joined by row id. */ - of?: [data: string, table: string] | null; - /** Reads a sampled field at its last frame without interpolating. */ - step?: boolean; -} -``` - -### ColorNode - -```ts -interface ColorNode { - /** A CSS color, such as `#ffb000` or `rebeccapurple`. */ - color: Color; -} -``` - -### StateNode - -```ts -/** What the view knows of each row. */ -interface StateNode { - state: State; -} -``` - -### MissingNode - -```ts -/** A missing value: options that read one take the theme's. */ -interface MissingNode { - missing: true; -} -``` - -### UnaryNode - -```ts -interface UnaryNode { - op: Unary; - args: [ExprInput]; -} -``` - -### BinaryNode - -```ts -interface BinaryNode { - op: Binary; - args: [ExprInput, ExprInput]; -} -``` - -### WhenNode - -```ts -/** The value of the first case whose condition holds, else `otherwise`. */ -interface WhenNode { - when: [condition: ExprInput, value: ExprInput][]; - otherwise: ExprInput; -} -``` - -### IsNode - -```ts -/** Whether a text field equals the text. */ -interface IsNode { - is: [field: ExprInput, text: string]; -} -``` - -### ContainsNode - -```ts -/** Whether a text field holds the text, ignoring case: read by queries, which hold texts. */ -interface ContainsNode { - contains: [field: ExprInput, text: string]; -} -``` - -### ScaleNode - -```ts -/** - * `value` mapped linearly onto the range `scale`, clamped: from its `domain`, else from the extent - * of the values shown. - */ -interface ScaleNode { - scale: [low: number, high: number]; - value: ExprInput; - domain?: [low: number, high: number] | null; -} -``` - -### ColormapNode - -```ts -/** `value` through `colormap`: from its `domain`, else from the extent of the values shown. */ -interface ColormapNode { - colormap: Colormap; - value: ExprInput; - domain?: [low: number, high: number] | null; -} -``` - -### Unary - -```ts -/** An operation of one value. */ -type Unary = 'neg' | 'not' | 'abs' | 'sqrt' | 'sin' | 'cos' | 'floor' | 'ln' | 'exp'; -``` - -### Binary - -```ts -/** An operation of two values. */ -type Binary = - | 'add' - | 'sub' - | 'mul' - | 'div' - | 'rem' - | 'pow' - | 'min' - | 'max' - | 'lt' - | 'le' - | 'gt' - | 'ge' - | 'eq' - | 'ne' - | 'and' - | 'or'; -``` - -### State - -```ts -/** What the view knows of each row. */ -type State = 'selected' | 'hovered' | 'pointer' | 'elapsed' | 'zoom'; -``` - -### Colormap - -```ts -/** Colors for a range of values: a catalog name, or your own colors. */ -type Colormap = ColormapName | CustomColormap; -``` - -### ColormapName - -```ts -/** A colormap of the catalog. */ -type ColormapName = - | 'amber' - | 'amp' - | 'batlow' - | 'cividis' - | 'crest' - | 'deep' - | 'devon' - | 'flare' - | 'grays' - | 'haline' - | 'inferno' - | 'lajolla' - | 'magma' - | 'mako' - | 'oslo' - | 'plasma' - | 'rocket' - | 'thermal' - | 'turku' - | 'viridis' - | 'balance' - | 'berlin' - | 'brbg' - | 'broc' - | 'coolwarm' - | 'cork' - | 'icefire' - | 'managua' - | 'piyg' - | 'puor' - | 'rdbu' - | 'spectral' - | 'vanimo' - | 'vik' - | 'vlag' - | 'phase' - | 'romao' - | 'twilight' - | 'dark2' - | 'okabeito' - | 'paired' - | 'set2' - | 'tab10' - | 'tab20' - | 'tableau10' - | 'turbo'; -``` - -### ColormapKind - -```ts -/** How a colormap spans values. */ -type ColormapKind = 'sequential' | 'diverging' | 'cyclic' | 'categorical'; -``` - -### CustomColormap - -```ts -/** Your own colormap: its kind, its colors, and for a categorical map its categories' names. */ -interface CustomColormap { - kind: ColormapKind; - /** - * Colors blended in OKLab, each at its `[place, color]` from 0 to 1 or evenly between its - * neighbors, as in a CSS gradient; a categorical map's, one per category. - */ - colors: (Color | [number, Color])[]; - /** The categories' names, in order, for a legend: a categorical map's alone. */ - categories?: string[]; -} -``` +### `CustomColormap` + +{.signature} +`interface CustomColormap` + +A custom colormap: its kind, its colors, and for a categorical map, its category names. + +{#customcolormap-kind .member} +**`kind`**`:` [`ColormapKind`](#colormapkind) [¶](#customcolormap-kind){.headerlink} +: How the colors map values. + +{#customcolormap-colors .member} +**`colors`**`: (`[`Color`](#color) `| [number,` [`Color`](#color)`])[]` [¶](#customcolormap-colors){.headerlink} +: The colors, blended in OKLab. Each is a color, spaced evenly between its neighbors like a CSS + gradient, or a `[place, color]` pair with a place from 0 to 1. Places must ascend. A categorical + map takes one color per category, without places, at most 256. + +{#customcolormap-categories .member} +**`categories`**`?: string[]` [¶](#customcolormap-categories){.headerlink} +: The category names in category order, shown in legends. Categorical maps only. ## Rows, cameras, and queries -Naming rows, framing them, and asking for them. - -### Id - -```ts -/** - * A row's id: a whole number from 0, or a text. A number above 2^53 - 1 is written as its decimal - * text, which names the same row. - */ -type Id = number | string; -``` - -### Ids - -```ts -/** Row ids: numbers or texts (null leaves a reference unwired), or `{count}` naming rows 0..count. */ -type Ids = - | (Id | null)[] - | { - count: number; - }; -``` - -### Row - -```ts -/** A row of a table in a dataset. It survives any change that keeps the names and the id. */ -interface Row { - /** The dataset's name. */ - data: string; - /** The table's name. */ - table: string; - /** The row's id in the table. */ - id: Id; -} -``` - -### Rows - -```ts -/** Rows of one table in a dataset, by id: what a view selects, fits, and tells as sets of rows. */ -interface Rows { - /** The dataset's name. */ - data: string; - /** The table's name. */ - table: string; - /** The rows' ids in the table. */ - ids: Id[]; -} -``` - -### Camera - -```ts -/** Where a view looks. */ -type Camera = - | { - /** - * Longitude, latitude, and elevation on the earth; x, y, and elevation on a plane or a diagram. In - * JSON, `[x, y]` looks at the ground. - */ - center: [number, number] | [number, number, number]; - /** On the earth the web mercator level; elsewhere, log₂ of pixels per unit. */ - zoom: number; - /** Degrees from looking straight down, up to 75; 0 in a diagram. */ - pitch: number; - /** Degrees clockwise from north, or from up on a plane; 0 in a diagram. */ - bearing: number; - } - | { - /** The axis range shown. */ - x: [number, number]; - /** The value range shown, or none to fit what is in view. */ - y?: [number, number] | null; - }; -``` - -### Fit - -```ts -/** What a view's fit frames. */ -type Fit = - | 'all' - | { - /** These rows. */ - rows: Rows[]; - } - | { - /** Brings this row into view, keeping the zoom. */ - reveal: Row; - } - | { - /** The box from `min` to `max`: x, y, and elevation. */ - bounds: { - /** The least corner. */ - min: [number, number, number]; - /** The greatest corner. */ - max: [number, number, number]; - }; - }; -``` - -### Query - -```ts -/** The rows of a table to read back, run through the expressions marks draw with. */ -interface Query { - /** The table read. */ - of: string; - /** - * The rows read, in this order unless sorted; every row unless set. Ids that name no row are - * passed over. - */ - ids?: Ids | null; - /** Keeps the rows it is true for. */ - filter?: ExprInput | null; - /** - * Keys to sort by, each after those before it; rows a key reads nothing for come last, and rows - * all keys order alike keep their order. - */ - sort?: [ExprInput, Order][]; - /** The kept rows skipped first. */ - offset?: number; - /** The most rows read; a limit picks them in linear time before sorting only those. */ - limit?: number | null; - /** Fields to read by name: a field as it is, anything else as numbers. Every field unless set. */ - select?: { - [name: string]: ExprInput; - }; - /** The axis coordinate sampled fields are read at; their last frame unless set. */ - at?: number | null; -} -``` +Identifying rows, framing them, and querying them. + +(id)= + +### `Id` + +{.signature} +`type Id = number | string` + +A row's id: a whole number from 0, or a text. A number above 2^53 - 1 is written as its decimal +text, which names the same row. + +(ids)= + +### `Ids` + +{.signature} +`type Ids = (`[`Id`](#id) `| null)[] | { count: number }` + +Row ids: a list of numbers or texts, or `{count}` for rows 0 to count - 1. In a reference, null +names no row. + +{#ids-count .member} +`{` **`count`**`: number }` [¶](#ids-count){.headerlink} +: The number of rows, whose ids are 0 to count - 1. + +(row)= + +### `Row` + +{.signature} +`interface Row` + +One row, identified by dataset name, table name, and id. It stays valid across any change that +keeps those names and the id. + +{#row-data .member} +**`data`**`: string` [¶](#row-data){.headerlink} +: The dataset's name. + +{#row-table .member} +**`table`**`: string` [¶](#row-table){.headerlink} +: The table's name. + +{#row-id .member} +**`id`**`:` [`Id`](#id) [¶](#row-id){.headerlink} +: The row's id in the table. + +(rows)= + +### `Rows` + +{.signature} +`interface Rows` + +Rows of one table in a dataset, by id. Views use it for selections, fits, and the rows events +report. + +{#rows-data .member} +**`data`**`: string` [¶](#rows-data){.headerlink} +: The dataset's name. + +{#rows-table .member} +**`table`**`: string` [¶](#rows-table){.headerlink} +: The table's name. + +{#rows-ids .member} +**`ids`**`:` [`Id`](#id)`[]` [¶](#rows-ids){.headerlink} +: The rows' ids in the table. + +(camera)= + +### `Camera` + +{.signature} +`type Camera = { … } | { x: [number, number]; y?: [number, number] | null }` + +Where a view looks. + +A camera over the earth, a plane, or a diagram: the point it looks at and turns about, its zoom, +its tilt, and its rotation. A diagram is always seen from straight above, upright. + +{#camera-center .member} +**`center`**`: [number, number] | [number, number, number]` [¶](#camera-center){.headerlink} +: On the earth, longitude, latitude, and elevation. On a plane or a diagram, x, y, and elevation. + In JSON, `[x, y]` is shorthand for elevation 0. + +{#camera-zoom .member} +**`zoom`**`: number` [¶](#camera-zoom){.headerlink} +: The web mercator zoom level on the earth. Elsewhere, log₂ of CSS pixels per unit. + +{#camera-pitch .member} +**`pitch`**`: number` [¶](#camera-pitch){.headerlink} +: Tilt in degrees from looking straight down, up to 75. Ignored in a diagram. + +{#camera-bearing .member} +**`bearing`**`: number` [¶](#camera-bearing){.headerlink} +: Degrees clockwise from north, or from up on a plane. Ignored in a diagram. + +A plot camera: the x range shown, and the value range, or `null` to fit the values in view. + +{#camera-x .member} +**`x`**`: [number, number]` [¶](#camera-x){.headerlink} +: The x-axis range shown. + +{#camera-y .member} +**`y`**`?: [number, number] | null` [¶](#camera-y){.headerlink} +: The value range shown, or `null` to fit the values in view. + +(fit)= + +### `Fit` + +{.signature} +`type Fit = 'all' | { rows:` [`Rows`](#rows)`[] } | { reveal:` [`Row`](#row) `} | { bounds: { … } }` + +The target of a view's fit. + +{#fit-all .member} +**`'all'`** [¶](#fit-all){.headerlink} +: Everything drawn. + +{#fit-rows .member} +`{` **`rows`**`:` [`Rows`](#rows)`[] }` [¶](#fit-rows){.headerlink} +: These rows. + +{#fit-reveal .member} +`{` **`reveal`**`:` [`Row`](#row) `}` [¶](#fit-reveal){.headerlink} +: Brings this row into view, keeping the zoom. + +{#fit-bounds .member} +`{` **`bounds`**`: { min: [number, number, number]; max: [number, number, number] } }` [¶](#fit-bounds){.headerlink} +: The box from `min` to `max`: x, y, and elevation. + + {#fit-bounds-min .member} + **`min`**`: [number, number, number]` [¶](#fit-bounds-min){.headerlink} + : The minimum corner. + + {#fit-bounds-max .member} + **`max`**`: [number, number, number]` [¶](#fit-bounds-max){.headerlink} + : The maximum corner. + +(query)= + +### `Query` + +{.signature} +`interface Query` + +The rows of a table to read back, run through the expressions marks draw with. + +{#query-of .member} +**`of`**`: string` [¶](#query-of){.headerlink} +: The table to read. + +{#query-ids .member} +**`ids`**`?:` [`Ids`](#ids) `| null` [¶](#query-ids){.headerlink} +: The rows to read, in this order unless sorted. Defaults to every row. Ids that name no row are + skipped. + +{#query-filter .member} +**`filter`**`?:` [`ExprInput`](#exprinput) `| null` [¶](#query-filter){.headerlink} +: Keeps only the rows for which this is true. + +{#query-sort .member} +**`sort`**`?: [`[`ExprInput`](#exprinput)`,` [`Order`](#order)`][]` [¶](#query-sort){.headerlink} +: Keys to sort by. Each key breaks ties left by the keys before it. Rows where a key has no value + sort last, and rows equal on every key keep their order. + +{#query-offset .member} +**`offset`**`?: number` [¶](#query-offset){.headerlink} +: How many of the kept rows to skip. + +{#query-limit .member} +**`limit`**`?: number | null` [¶](#query-limit){.headerlink} +: The most rows to return. When sorting with a limit, the first `offset + limit` rows are selected + in linear time and only those are sorted. + +{#query-select .member} +**`select`**`?: { [name: string]:` [`ExprInput`](#exprinput) `} | null` [¶](#query-select){.headerlink} +: Fields to return, by output name. A plain field returns that field as stored, and any other + expression returns numbers. Defaults to every field. An empty object returns the ids alone. + +{#query-at .member} +**`at`**`?: number | null` [¶](#query-at){.headerlink} +: The axis coordinate at which sampled fields are read. Defaults to their last frame. + +(enveloperequest)= + +### `EnvelopeRequest` + +{.signature} +`interface EnvelopeRequest` + +A request for each row's least and greatest value of a sampled field over equal spans of the +axis. It reads field `field` of table `table` over `spans` equal spans from `range[0]` to +`range[1]`, for every row or for the rows in `ids`, in that order. Like a query, it is plain +data that deserializes from JSON. + +{#enveloperequest-table .member} +**`table`**`: string` [¶](#enveloperequest-table){.headerlink} +: The table to read. + +{#enveloperequest-field .member} +**`field`**`: string` [¶](#enveloperequest-field){.headerlink} +: The sampled field to read. + +{#enveloperequest-range .member} +**`range`**`: [number, number]` [¶](#enveloperequest-range){.headerlink} +: The axis range the spans cover, `[first, last]`. Both are finite, with `first <= last`. + +{#enveloperequest-spans .member} +**`spans`**`: number` [¶](#enveloperequest-spans){.headerlink} +: The number of equal spans to divide the range into. At least 1. + +{#enveloperequest-ids .member} +**`ids`**`?:` [`Ids`](#ids) `| null` [¶](#enveloperequest-ids){.headerlink} +: The rows to read, in this order. Defaults to every row. ## Results -What views tell: events, hits, descriptions, and what frames cost. - -### ViewEvent - -```ts -/** What happened in a view, for the application to act on. Drain them with `View::events`. */ -type ViewEvent = - | { - type: 'click'; - detail: { - /** The row. */ - hit: Hit; - /** How the user clicked it. */ - trigger: Trigger; - }; - } - | { - type: 'open'; - detail: { - /** The row. */ - hit: Hit; - /** How the user opened it. */ - trigger: Trigger; - }; - } - | { - type: 'menu'; - detail: { - /** What lies under the point, nearest first. */ - hits: Hit[]; - /** How the user asked. */ - trigger: Trigger; - }; - } - | { - type: 'drag'; - detail: { - /** Whether the drag starts, moves, or ends. */ - phase: Phase; - /** The row dragged, as it was hit. */ - hit: Hit; - /** - * Where the row goes, in the space's coordinates: where it was hit, moved as the pointer moved - * since the press, at the depth the drag started; a vertex keeps where it was grabbed. On a - * diagram's dots when its grid snaps. - */ - position: [number, number, number]; - /** The pointer. */ - trigger: Trigger; - }; - } - | { - type: 'connect'; - detail: { - /** The end it was drawn from: for an edge, the end that stayed. */ - from: End; - /** The vertex it was released over, if any. */ - to: End | null; - /** The edge whose other end was dragged, which `to` replaces. */ - edge: Row | null; - /** Where it was released. */ - trigger: Trigger; - }; - } - | { - type: 'delete'; - detail: { - /** The rows selected. */ - rows: Rows[]; - /** The key. */ - trigger: Trigger; - }; - } - | { - type: 'hover'; - detail: { - /** The row now pointed at; `null` once it left every row. */ - hit: Hit | null; - /** The pointer, or the keys. */ - trigger: Trigger; - }; - } - | { - type: 'select'; - detail: { - /** The rows selected now. */ - rows: Rows[]; - /** How the user changed it; `null` when the data dropped rows from it. */ - trigger: Trigger | null; - }; - } - | { - type: 'camera'; - detail: Camera; - } - | { - type: 'waiting'; - detail: Waiting; - } - | { - type: 'error'; - detail: Failure; - }; -``` - -### Hit - -```ts -/** A row under a point, as the frame drew it. */ -interface Hit { - /** The row. */ - row: Row; - /** The name of the mark that drew it, when it has one: what tells two marks of one table apart. */ - mark?: string; - /** - * The point of the row nearest the probe, CSS pixels from the top left: a vertex's or a port's - * middle, the nearest point of a stroke or a triangle. - */ - point: [number, number]; - /** That point in the space's coordinates: x, y, and elevation. */ - position: [number, number, number]; - /** CSS pixels from the probe to the row. */ - distance: number; - /** The port hit, when a vertex's port was. */ - port?: string; - /** The sample hit, when a line tracing a sampled field was. */ - sample?: Sample; -} -``` - -### Failure - -```ts -/** - * What failed: its `code`, the message, and for invalid input the `path` to the part, such as - * `marks[2].vertices.color`. - */ -interface Failure { - code: 'invalid' | 'unsupported' | 'limit' | 'io' | 'device-lost' | 'cancelled' | 'timeout'; - message: string; - path?: string; -} -``` - -### Phase - -```ts -/** The phase of a `Drag`. */ -type Phase = 'start' | 'move' | 'end'; -``` - -### Trigger - -```ts -/** - * Where, with what keys, and with what the user acted. For keys, the point is the row they point - * at, else the pointer, else the view's middle. - */ -interface Trigger { - /** CSS pixels from the top left. */ - point: [number, number]; - /** The keys held. */ - modifiers: Modifiers; - /** What acted. */ - device: Device; -} -``` - -### Device - -```ts -/** What the user acts with. */ -type Device = 'mouse' | 'pen' | 'touch' | 'keyboard'; -``` - -### Modifiers - -```ts -/** Keys held during an input. */ -interface Modifiers { - /** Shift. */ - shift: boolean; - /** Control. */ - control: boolean; - /** Alt or Option. */ - alt: boolean; - /** Meta: Command or Windows. */ - meta: boolean; -} -``` - -### Description - -```ts -/** - * What a dataset holds: its tables, their rows and fields, and the axis its sampled fields are - * recorded over and span. - */ -interface Description { - /** The dataset's name. */ - name: string; - /** The axis's name and unit. */ - axis: [string, string] | null; - /** The first and last coordinates its sampled fields hold. */ - span: [number, number] | null; - /** Its tables, by name. */ - tables: TableDescription[]; -} -``` - -### TableDescription - -```ts -/** A table described: its rows, and its fields by name. */ -interface TableDescription { - /** The table's name. */ - name: string; - /** How many rows it has. */ - rows: number; - /** Its fields, by name. */ - fields: FieldDescription[]; -} -``` - -### FieldDescription - -```ts -/** A field described: what it holds, the table a reference names, the coordinates samples span. */ -interface FieldDescription { - /** The field's name. */ - name: string; - /** What it holds. */ - kind: Kind; - /** The table a reference names. */ - to?: string; - /** The first and last coordinates a sampled field holds. */ - span?: [number, number]; -} -``` - -### Kind - -```ts -/** What a field holds. */ -type Kind = - 'number' | 'bool' | 'text' | 'point' | 'path' | 'parts' | 'polygons' | 'reference' | 'sampled'; -``` - -### Value - -```ts -/** - * A value read from a dataset: a number (`null` when not finite), boolean, or text; a geometry as - * nested `[x, y]` points; a reference as the id of the row it names. - */ -type Value = number | boolean | string | null | [number, number] | unknown[][]; -``` - -### Sample - -```ts -/** A sample of a traced field: its frame, the axis coordinate it was recorded at, and its value. */ -interface Sample { - /** The frame, numbered from the first ever recorded. JSON uses a decimal string above 2^53 - 1. */ - frame: Id; - /** The axis coordinate. */ - at: number; - /** The value. */ - value: number; -} -``` - -### Stats - -```ts -/** What a view does each frame. */ -interface Stats { - /** Frames rendered. */ - frames: number; - /** Rows of every mark's table. */ - rows: number; - /** - * Rows every mark draws: every row of a table drawn whole, the rows of the tiles in view of one - * drawn through tiles. - */ - drawn: number; - /** Draw calls the last frame recorded, shadows and labels with the marks. */ - draws: number; - /** - * Instances the last frame's draws drew: points, segments, triangles, glyphs; at most, for - * segments the GPU culls. - */ - instances: number; - /** Fetched tiles the view draws, of every tile source. */ - tiles: number; - /** Tiles being fetched or decoded, of every tile source. */ - tilesLoading: number; - /** Bytes of data waiting to upload. */ - uploading: number; - /** Bytes of data pages on the GPU, for every view of its `Gpu`. */ - resident: number; - /** Bytes the GPU's `Budget` lets data take. */ - budget: number; - /** - * The work under way the last frame stood in for; nothing when it drew everything as the data, - * scene, and camera say: see `View::complete`. - */ - waiting: Waiting; - /** Milliseconds the last frame took to record on the CPU. */ - cpuMs: number; - /** - * Milliseconds the GPU takes on a frame: the median of the last frames it measured; 0 where the - * device cannot measure. - */ - gpuMs: number; -} -``` - -### Waiting - -```ts -/** - * Work under way a frame stands in for: pipelines the scene draws with compiling, until which the - * view draws nothing new (`compiling`); pages uploading or tables not yet built from them - * (`data`); layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); a label placement or - * texts not yet set (`labels`); a fit due or being measured (`fit`); row regions laid out again, - * segment counts, or a transition beginning (`marks`). Empty once a frame is complete. - */ -type Waiting = ('compiling' | 'data' | 'derived' | 'labels' | 'fit' | 'marks')[]; -``` - -### Work - -```ts -/** What a `Gpu` did since it was made; subtract two for what happened between. */ -interface Work { - /** Bytes of data pages uploaded. */ - uploaded: number; - /** Bytes written besides pages: uniforms, programs, label runs, the selection, textures. */ - written: number; - /** Rows whose channels the GPU evaluated. */ - evaluated: number; - /** - * Reads back from the GPU: probes, queries, fits, label placements, segment counts, pictures; not - * the frame timer's, which only some devices have. - */ - readbacks: number; - /** - * Jobs handed out: layouts, routes, lanes, triangulations, pyramids, tiles; to the host's workers, - * or done at once without a host. - */ - jobs: number; - /** Label texts the host was asked to set. */ - texts: number; - /** - * Pipelines compiled: in a browser, each as a view or a shot asks for it ahead, off the GPU's - * thread; natively, where it is first made. - */ - pipelines: number; - /** - * Pipelines made without being asked for ahead, each compiled where it is first used: in a browser - * on the GPU's thread, holding frames back. None, unless a scene uses what it did not ask for. - */ - late: number; -} -``` +View events and hits, dataset descriptions and values, and frame statistics. + +(viewevent)= + +### `ViewEvent` + +{.signature} +`type ViewEvent = { type: 'click'; detail: { … } } | { type: 'open'; detail: { … } } | { type: 'menu'; detail: { … } } | { type: 'drag'; detail: { … } } | { type: 'connect'; detail: { … } } | { type: 'delete'; detail: { … } } | { type: 'hovered'; detail: { … } } | { type: 'selection'; detail: { … } } | { type: 'camera'; detail:` [`Camera`](#camera) `} | { type: 'waiting'; detail:` [`Waiting`](#waiting) `} | { type: 'error'; detail:` [`Failure`](#failure) `}` + +Something that happened in a view, as [`view.on`](#view-on) reports it. + +{#viewevent-click .member} +**`'click'`** [¶](#viewevent-click){.headerlink} +: A row was clicked. + + {#viewevent-click-hit .member} + **`hit`**`:` [`Hit`](#hit) [¶](#viewevent-click-hit){.headerlink} + : The row. + + {#viewevent-click-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-click-trigger){.headerlink} + : How the user clicked it. + +{#viewevent-open .member} +**`'open'`** [¶](#viewevent-open){.headerlink} +: A row was double-clicked, or Enter was pressed on the row the keyboard points at. + + {#viewevent-open-hit .member} + **`hit`**`:` [`Hit`](#hit) [¶](#viewevent-open-hit){.headerlink} + : The row. + + {#viewevent-open-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-open-trigger){.headerlink} + : How the user opened it. + +{#viewevent-menu .member} +**`'menu'`** [¶](#viewevent-menu){.headerlink} +: A context menu was requested: by a secondary click, the menu key, or a touch or pen held still. + Lists the rows under the trigger's point, for a menu of actions on them. + + {#viewevent-menu-hits .member} + **`hits`**`:` [`Hit`](#hit)`[]` [¶](#viewevent-menu-hits){.headerlink} + : The rows under the point, nearest first. + + {#viewevent-menu-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-menu-trigger){.headerlink} + : How the user asked. + +{#viewevent-drag .member} +**`'drag'`** [¶](#viewevent-drag){.headerlink} +: A row of a draggable mark is dragged. The drag starts once the pointer moves more than 4 CSS + pixels from where it pressed the row. + + {#viewevent-drag-phase .member} + **`phase`**`:` [`Phase`](#phase) [¶](#viewevent-drag-phase){.headerlink} + : Whether the drag starts, moves, or ends. + + {#viewevent-drag-hit .member} + **`hit`**`:` [`Hit`](#hit) [¶](#viewevent-drag-hit){.headerlink} + : The row dragged, as it was hit. + + {#viewevent-drag-position .member} + **`position`**`: [number, number, number]` [¶](#viewevent-drag-position){.headerlink} + : The row's new position in the space's coordinates. It is the hit point moved with the pointer + since the press, at the depth where the drag started, so a vertex keeps its offset from the + pointer. Snapped to a diagram's grid dots when the grid snaps. + + {#viewevent-drag-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-drag-trigger){.headerlink} + : The pointer. + +{#viewevent-connect .member} +**`'connect'`** [¶](#viewevent-connect){.headerlink} +: A connection was drawn with [`controls.connect`](#controls-connect) on: from a port, from a vertex with Alt held, or + by dragging an edge's end off it. + + {#viewevent-connect-from .member} + **`from`**`:` [`End`](#end) [¶](#viewevent-connect-from){.headerlink} + : The end it was drawn from. For a dragged edge end, the edge's other end. + + {#viewevent-connect-to .member} + **`to`**`:` [`End`](#end) `| null` [¶](#viewevent-connect-to){.headerlink} + : The vertex it was released over, if any. + + {#viewevent-connect-edge .member} + **`edge`**`:` [`Row`](#row) `| null` [¶](#viewevent-connect-edge){.headerlink} + : The edge whose end was dragged off. `to` replaces that end. + + {#viewevent-connect-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-connect-trigger){.headerlink} + : Where it was released. + +{#viewevent-delete .member} +**`'delete'`** [¶](#viewevent-delete){.headerlink} +: Delete or Backspace was pressed with rows selected and [`controls.delete`](#controls-delete) on. + + {#viewevent-delete-rows .member} + **`rows`**`:` [`Rows`](#rows)`[]` [¶](#viewevent-delete-rows){.headerlink} + : The rows selected. + + {#viewevent-delete-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-delete-trigger){.headerlink} + : The key. + +{#viewevent-hovered .member} +**`'hovered'`** [¶](#viewevent-hovered){.headerlink} +: The hovered row changed: [`view.hovered`](#view-hovered). + + {#viewevent-hovered-hit .member} + **`hit`**`:` [`Hit`](#hit) `| null` [¶](#viewevent-hovered-hit){.headerlink} + : The row now hovered, or `null` if no row is. + + {#viewevent-hovered-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) [¶](#viewevent-hovered-trigger){.headerlink} + : The pointer, or the keys. + +{#viewevent-selection .member} +**`'selection'`** [¶](#viewevent-selection){.headerlink} +: The selection changed: [`view.selection`](#view-selection). + + {#viewevent-selection-rows .member} + **`rows`**`:` [`Rows`](#rows)`[]` [¶](#viewevent-selection-rows){.headerlink} + : The rows selected now. + + {#viewevent-selection-trigger .member} + **`trigger`**`:` [`Trigger`](#trigger) `| null` [¶](#viewevent-selection-trigger){.headerlink} + : How the user changed it, or `null` when the data dropped rows from it. + +{#viewevent-camera .member} +**`'camera'`**`:` [`Camera`](#camera) [¶](#viewevent-camera){.headerlink} +: The camera moved: [`view.camera`](#view-camera). A gesture, fit, flight, coast, or orbit moved it, or it moved + back within its zoom and pan limits after the view's size, insets, or space changed. + +{#viewevent-waiting .member} +**`'waiting'`**`:` [`Waiting`](#waiting) [¶](#viewevent-waiting){.headerlink} +: The work the last frame waited on changed: [`view.waiting`](#view-waiting). Empty once frames are complete. + +{#viewevent-error .member} +**`'error'`**`:` [`Failure`](#failure) [¶](#viewevent-error){.headerlink} +: Something failed. Reported once. A mark that no longer fits its data is also listed in + [`view.failures`](#view-failures). + +(hit)= + +### `Hit` + +{.signature} +`interface Hit` + +A row under a point, as the frame drew it. + +{#hit-row .member} +**`row`**`:` [`Row`](#row) [¶](#hit-row){.headerlink} +: The row. + +{#hit-mark .member} +**`mark`**`?: string` [¶](#hit-mark){.headerlink} +: The name of the mark that drew it, if the mark has one. It tells apart two marks of one table. + +{#hit-point .member} +**`point`**`: [number, number]` [¶](#hit-point){.headerlink} +: The row's point nearest the probe, in CSS pixels from the top left: a vertex's or port's middle, + or the nearest point of a stroke or triangle. + +{#hit-position .member} +**`position`**`: [number, number, number]` [¶](#hit-position){.headerlink} +: That point in the space's coordinates: x, y, and elevation. + +{#hit-distance .member} +**`distance`**`: number` [¶](#hit-distance){.headerlink} +: CSS pixels from the probe to the row. + +{#hit-port .member} +**`port`**`?: string` [¶](#hit-port){.headerlink} +: The port hit, if the hit is on a vertex's port. + +{#hit-sample .member} +**`sample`**`?:` [`Sample`](#sample) [¶](#hit-sample){.headerlink} +: The sample hit, if the hit is on a line tracing a sampled field. + +(end)= + +### `End` + +{.signature} +`interface End` + +A vertex a connection ends at, and the port the connection meets, if any. + +{#end-row .member} +**`row`**`:` [`Row`](#row) [¶](#end-row){.headerlink} +: The vertex. + +{#end-port .member} +**`port`**`?: string` [¶](#end-port){.headerlink} +: The port, by the reference field naming its net. + +(failure)= + +### `Failure` + +{.signature} +`interface Failure` + +A failure as JSON, in the same form as the npm package's [`LatkitError`](#latkiterror). + +{#failure-code .member} +**`code`**`: 'invalid' | 'unsupported' | 'limit' | 'io' | 'device-lost' | 'cancelled' | 'timeout'` [¶](#failure-code){.headerlink} +: The kind of failure. See [`LatkitError.code`](#latkiterror-code). + +{#failure-message .member} +**`message`**`: string` [¶](#failure-message){.headerlink} +: A description of what failed. + +{#failure-path .member} +**`path`**`?: string` [¶](#failure-path){.headerlink} +: For `invalid` failures, the path of the invalid input, such as `marks[2].color`. + +(phase)= + +### `Phase` + +{.signature} +`type Phase = 'start' | 'move' | 'end'` + +The phase of a `Drag`. + +{#phase-start .member} +**`'start'`** [¶](#phase-start){.headerlink} +: The pointer moved far enough to drag. + +{#phase-move .member} +**`'move'`** [¶](#phase-move){.headerlink} +: The pointer moved. + +{#phase-end .member} +**`'end'`** [¶](#phase-end){.headerlink} +: The pointer was released, or the drag was cancelled. + +(trigger)= + +### `Trigger` + +{.signature} +`interface Trigger` + +Where the user acted, with which modifier keys and device. For keyboard input, the point is the +row the keys point at, or else the pointer, or else the view's middle. + +{#trigger-point .member} +**`point`**`: [number, number]` [¶](#trigger-point){.headerlink} +: CSS pixels from the top left. + +{#trigger-modifiers .member} +**`modifiers`**`:` [`Modifiers`](#modifiers) [¶](#trigger-modifiers){.headerlink} +: The modifier keys held. + +{#trigger-device .member} +**`device`**`:` [`Device`](#device) [¶](#trigger-device){.headerlink} +: The device used. + +(device)= + +### `Device` + +{.signature} +`type Device = 'mouse' | 'pen' | 'touch' | 'keyboard'` + +The input device. + +{#device-mouse .member} +**`'mouse'`** [¶](#device-mouse){.headerlink} +: A mouse or a trackpad. + +{#device-pen .member} +**`'pen'`** [¶](#device-pen){.headerlink} +: A pen. + +{#device-touch .member} +**`'touch'`** [¶](#device-touch){.headerlink} +: A finger. Picks find rows up to 8 CSS pixels farther away. + +{#device-keyboard .member} +**`'keyboard'`** [¶](#device-keyboard){.headerlink} +: The keyboard. + +(modifiers)= + +### `Modifiers` + +{.signature} +`interface Modifiers` + +Modifier keys held during an input. + +{#modifiers-shift .member} +**`shift`**`: boolean` [¶](#modifiers-shift){.headerlink} +: Shift. + +{#modifiers-control .member} +**`control`**`: boolean` [¶](#modifiers-control){.headerlink} +: Control. + +{#modifiers-alt .member} +**`alt`**`: boolean` [¶](#modifiers-alt){.headerlink} +: Alt or Option. + +{#modifiers-meta .member} +**`meta`**`: boolean` [¶](#modifiers-meta){.headerlink} +: Meta: Command or Windows. + +(description)= + +### `Description` + +{.signature} +`interface Description` + +A summary of a dataset: its tables with their rows and fields, the axis sampled fields are +recorded over, and the part of the axis they cover. + +{#description-name .member} +**`name`**`: string` [¶](#description-name){.headerlink} +: The dataset's name. + +{#description-axis .member} +**`axis`**`: [string, string] | null` [¶](#description-axis){.headerlink} +: The axis name and unit, if set. + +{#description-span .member} +**`span`**`: [number, number] | null` [¶](#description-span){.headerlink} +: The first and last axis coordinates held by any sampled field, or `null` if no field holds any. + +{#description-tables .member} +**`tables`**`:` [`TableDescription`](#tabledescription)`[]` [¶](#description-tables){.headerlink} +: Its tables, sorted by name. + +(tabledescription)= + +### `TableDescription` + +{.signature} +`interface TableDescription` + +A summary of one table: its row count and its fields, sorted by name. + +{#tabledescription-name .member} +**`name`**`: string` [¶](#tabledescription-name){.headerlink} +: The table's name. + +{#tabledescription-rows .member} +**`rows`**`: number` [¶](#tabledescription-rows){.headerlink} +: The number of rows. + +{#tabledescription-fields .member} +**`fields`**`:` [`FieldDescription`](#fielddescription)`[]` [¶](#tabledescription-fields){.headerlink} +: Its fields, sorted by name. + +(fielddescription)= + +### `FieldDescription` + +{.signature} +`interface FieldDescription` + +A summary of one field: its kind, plus its target table or axis span where those apply. + +{#fielddescription-name .member} +**`name`**`: string` [¶](#fielddescription-name){.headerlink} +: The field's name. + +{#fielddescription-kind .member} +**`kind`**`:` [`Kind`](#kind) [¶](#fielddescription-kind){.headerlink} +: The kind of values it holds. + +{#fielddescription-to .member} +**`to`**`?: string` [¶](#fielddescription-to){.headerlink} +: For a reference, the target table. + +{#fielddescription-span .member} +**`span`**`?: [number, number]` [¶](#fielddescription-span){.headerlink} +: For a sampled field, the first and last axis coordinates it holds. + +(kind)= + +### `Kind` + +{.signature} +`type Kind = 'number' | 'bool' | 'text' | 'point' | 'path' | 'parts' | 'polygons' | 'reference' | 'sampled'` + +The kind of values a field holds. + +{#kind-number .member} +**`'number'`** [¶](#kind-number){.headerlink} +: Numbers. + +{#kind-bool .member} +**`'bool'`** [¶](#kind-bool){.headerlink} +: Booleans. + +{#kind-text .member} +**`'text'`** [¶](#kind-text){.headerlink} +: Texts. + +{#kind-point .member} +**`'point'`** [¶](#kind-point){.headerlink} +: `[x, y]` points. + +{#kind-path .member} +**`'path'`** [¶](#kind-path){.headerlink} +: Paths of points. + +{#kind-parts .member} +**`'parts'`** [¶](#kind-parts){.headerlink} +: Parts: the rings of a polygon, or the paths of a multi-path. + +{#kind-polygons .member} +**`'polygons'`** [¶](#kind-polygons){.headerlink} +: Polygons of rings. + +{#kind-reference .member} +**`'reference'`** [¶](#kind-reference){.headerlink} +: Rows of another table, by id. + +{#kind-sampled .member} +**`'sampled'`** [¶](#kind-sampled){.headerlink} +: Numbers sampled over the dataset's axis. + +(value)= + +### `Value` + +{.signature} +`type Value = number | boolean | string | null | [number, number] | unknown[][]` + +A value read from a dataset: a number (`null` when not finite), boolean, or text; a geometry as +nested `[x, y]` points; a reference as the id of the row it names. + +(sample)= + +### `Sample` + +{.signature} +`interface Sample` + +A sample of a traced field: its frame, the axis coordinate it was recorded at, and its value. + +{#sample-frame .member} +**`frame`**`:` [`Id`](#id) [¶](#sample-frame){.headerlink} +: The frame, numbered from the first ever recorded. JSON uses a decimal string above 2^53 - 1. + +{#sample-at .member} +**`at`**`: number` [¶](#sample-at){.headerlink} +: The axis coordinate. + +{#sample-value .member} +**`value`**`: number` [¶](#sample-value){.headerlink} +: The value. + +(stats)= + +### `Stats` + +{.signature} +`interface Stats` + +Counters describing a view's frames. + +{#stats-frames .member} +**`frames`**`: number` [¶](#stats-frames){.headerlink} +: Frames rendered. + +{#stats-rows .member} +**`rows`**`: number` [¶](#stats-rows){.headerlink} +: Rows of every mark's table. + +{#stats-drawn .member} +**`drawn`**`: number` [¶](#stats-drawn){.headerlink} +: Rows the marks draw: every row of a fully drawn table, and the rows in the tiles in view of a + paged table. + +{#stats-draws .member} +**`draws`**`: number` [¶](#stats-draws){.headerlink} +: Draw calls the last frame recorded, label draws included. + +{#stats-instances .member} +**`instances`**`: number` [¶](#stats-instances){.headerlink} +: Instances the last frame's draws drew: points, segments, triangles, and glyphs. An upper bound + for segments, which the GPU may cull. + +{#stats-tiles .member} +**`tiles`**`: number` [¶](#stats-tiles){.headerlink} +: Fetched tiles the view draws, across all tile sources. + +{#stats-tilesloading .member} +**`tilesLoading`**`: number` [¶](#stats-tilesloading){.headerlink} +: Tiles being fetched or decoded, across all tile sources. + +{#stats-cpums .member} +**`cpuMs`**`: number` [¶](#stats-cpums){.headerlink} +: Milliseconds the last frame took to record on the CPU. + +{#stats-gpums .member} +**`gpuMs`**`: number` [¶](#stats-gpums){.headerlink} +: Milliseconds the GPU takes per frame: the median of recently measured frames. 0 if the device + cannot measure GPU time. + +(waiting)= + +### `Waiting` + +{.signature} +`type Waiting = ('compiling' | 'data' | 'derived' | 'labels' | 'fit' | 'marks')[]` + +What a frame still waits on: pipelines the scene draws with, which compile before the view draws +anything new (`compiling`); pages uploading, or tables not yet built from them (`data`); +layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); label texts being read back, or +not yet set (`labels`); a fit due or being measured (`fit`); segment counts, or a transition +starting (`marks`). Empty once the frame is complete. + +(work)= + +### `Work` + +{.signature} +`interface Work` + +Counts of the work a [`Gpu`](#gpu) has done since it was created. Subtract two snapshots to get the work +between them. + +{#work-uploaded .member} +**`uploaded`**`: number` [¶](#work-uploaded){.headerlink} +: Bytes of data pages uploaded. + +{#work-uploads .member} +**`uploads`**`: number` [¶](#work-uploads){.headerlink} +: Data pages, or pieces of pages, uploaded. A change's uploads count pages here and bytes in + `uploaded`. + +{#work-written .member} +**`written`**`: number` [¶](#work-written){.headerlink} +: Bytes written other than pages: uniforms, programs, label runs, the selection, and textures. + +{#work-evaluated .member} +**`evaluated`**`: number` [¶](#work-evaluated){.headerlink} +: Rows whose channels the GPU evaluated. + +{#work-resolved .member} +**`resolved`**`: number` [¶](#work-resolved){.headerlink} +: Edge and face rows whose endpoints the GPU resolved to resident vertex rows. These are rows that + changed, plus every row of a page that references a vertex now resident elsewhere. + +{#work-cleared .member} +**`cleared`**`: number` [¶](#work-cleared){.headerlink} +: Bytes of per-row word pages the GPU cleared: pages read before anything writes them, and the + normals that faces sum from scratch. + +{#work-packed .member} +**`packed`**`: number` [¶](#work-packed){.headerlink} +: Rows packed into pages of a paged table's draw order. Each page is packed once, the first time a + view draws it, and again only when a write changes its rows. + +{#work-readbacks .member} +**`readbacks`**`: number` [¶](#work-readbacks){.headerlink} +: Readbacks from the GPU: probes, queries, fits, label texts, segment counts, and pictures. Frame + timer readbacks, which only some devices support, are not counted. + +{#work-jobs .member} +**`jobs`**`: number` [¶](#work-jobs){.headerlink} +: Jobs started: layouts, routes, lanes, triangulations, pyramids, and tiles. They run on the + host's workers, or immediately without a host. + +{#work-texts .member} +**`texts`**`: number` [¶](#work-texts){.headerlink} +: Label texts the host was asked to rasterize. + +{#work-pipelines .member} +**`pipelines`**`: number` [¶](#work-pipelines){.headerlink} +: Pipelines compiled. In a browser, each compiles in the background when a view or shot requests + it ahead of use. Natively, each compiles when first created. + +{#work-late .member} +**`late`**`: number` [¶](#work-late){.headerlink} +: Pipelines not requested ahead of use, compiled when first used. In a browser they compile on the + GPU's thread and delay frames. Zero unless a scene uses a pipeline it did not request. diff --git a/npm/README.md b/npm/README.md index 7db8d906..e7c8ffd3 100644 --- a/npm/README.md +++ b/npm/README.md @@ -1,71 +1,30 @@ # latkit -Columnar data drawn as vertices, edges, lines, and areas with WebGPU. -[Documentation](https://latkit.readthedocs.io). +Draw large networks, such as power grids, with WebGPU. Write datasets of tables, describe scenes of +vertices, edges, lines, and areas over them, and export pictures and video of the views. ```ts -import { loadLatkit, Patch, field, type Scene } from 'latkit'; +import { Data, View, field, load } from 'latkit'; -const { Data, Gpu } = await loadLatkit(); -const data = new Data('grid').apply( - new Patch().replace('Bus', [101, 102], { x: [0, 30], y: [0, 20], kv: [230, 115] }), -); +await load(); +const grid = new Data('grid').set('Bus', [101, 102], { x: [0, 30], y: [0, 20], kv: [230, 115] }); -const scene: Scene = { +const view = await View.create(document.querySelector('canvas')!, { space: 'plane', marks: [ { - vertices: { - of: ['grid', 'Bus'], - position: ['x', 'y'], - radius: 6, - color: field('kv').colormap('viridis'), - }, + vertices: grid.table('Bus'), + position: ['$x', '$y'], + radius: 6, + color: field('kv').colormap('viridis'), }, ], -}; -const gpu = await Gpu.create(); -const view = gpu.view(scene); -gpu.canvas(document.querySelector('canvas')!).show([view]); -view.setData(data); -view.fit(); -``` - -- **Plain data needs no WebAssembly.** Node/server code can import `latkit/core` without DOM types. Scenes, expressions, and patches are built without loading - anything, so a worker or a Node server writes patches as well as a page. `loadLatkit()` loads the - WebAssembly once per page; `Gpu.create()` requests a WebGPU device. -- **Data changes in place.** `data.apply(patch)` makes every change or none. `data.snapshot()` keeps a - version in O(1), sharing its pages, as an undo history does. A view draws the version it was - given until `setData` gives it another. -- **Patches are bytes.** A patch is a frame of latkit's byte format, written in TypeScript as it is - built; servers in any language write the same frames. `Patch.stream(body)` yields them as they - arrive, and `Data.read(bytes)` reads a dataset's bytes or GeoJSON. -- **Data reads back** without a GPU, through the expressions marks draw with: - `data.query({ of, filter, sort, select, limit })`, `data.describe()`, `data.envelope(...)`. -- **Types come from the schema.** Scene, query, camera, and fit types (`inputs.ts`) and events, - hits, and stats (`outputs.ts`) are written from latkit's JSON Schema; `latkit schema` prints it. -- **Errors** throw a `LatkitError`: a `code` (`invalid`, `unsupported`, `limit`, `io`, `device-lost`, - `cancelled`, `timeout`) and, for invalid input, the `path` to the part. -- **Handles** (`Data`, `Gpu`, `View`, `Canvas`, `Image`) have idempotent `destroy()` and - `Symbol.dispose`. Picks, locations, and layouts reject with `cancelled` when their view is - destroyed, and with `timeout` when no frame draws it in time. +}); -Workers start when a view first needs one, share the compiled module, and retire after 30 seconds -idle. For your own asset hosting, pass the matching `latkit/wasm` and `latkit/worker` exports; the -worker is a standalone script, fine as a Blob URL: - -```ts -import wasmUrl from 'latkit/wasm?url'; -import workerUrl from 'latkit/worker?url'; - -const kit = await loadLatkit({ wasm: wasmUrl, createWorker: () => new Worker(workerUrl) }); +grid.set('Bus', [102], { kv: [345] }); // drawn the next frame ``` -Serve WebAssembly as `application/wasm`, in a secure context for WebGPU. - -From the repository: `cargo xtask wasm --dev` builds the package for development, and -`cargo xtask package` builds an optimized tarball under `target/npm`. - -LATK v1 uses 64-bit frame indexes. Sample values use `f32` and axis coordinates use `f64`. -`Video.record()` cleans up on failure; manual recordings must finish or abort. -See [deployment and platform support](https://latkit.readthedocs.io/en/latest/deployment.html). +A scene is plain data, with types from latkit's schema. Scenes, expressions, and patches need no +WebAssembly, so you can build them in a worker or on a server too. The +[guide](https://latkit.readthedocs.io/en/latest/guide.html) shows each task, and the +[API reference](https://latkit.readthedocs.io/en/latest/reference/api.html) documents every export. diff --git a/npm/build.mjs b/npm/build.mjs index be55cca8..f777aeaa 100644 --- a/npm/build.mjs +++ b/npm/build.mjs @@ -1,4 +1,7 @@ -/** Builds from pinned workspace tools; all release and consumer outputs stay below target/npm. */ +/** + * Build steps for the npm package, run with the workspace's pinned tools. Release and consumer + * outputs all stay under target/npm. + */ import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; import { createHash } from 'node:crypto'; @@ -55,7 +58,7 @@ if (mode === 'prepare') { assert.equal( fs.readFileSync(path.join(stage, `src/${name}.ts`), 'utf8').replaceAll('\r\n', '\n'), source, - `src/${name}.ts is not what npm/schema.json says: run node npm/build.mjs schema`, + `src/${name}.ts is out of date with npm/schema.json: run node npm/build.mjs schema`, ); } execFileSync( @@ -85,7 +88,7 @@ if (mode === 'prepare') { format: 'iife', platform: 'browser', target: 'es2022', - // wasm-bindgen's unused async initializer mentions import.meta; initSync doesn't need it. + // wasm-bindgen's unused async initializer mentions import.meta, which initSync does not need. logOverride: { 'empty-import-meta': 'silent' }, logLevel: 'warning', }); diff --git a/npm/docs.mjs b/npm/docs.mjs index b330c6d3..3a4381ae 100644 --- a/npm/docs.mjs +++ b/npm/docs.mjs @@ -1,4 +1,17 @@ -/** Checked-in reference snapshots let Read the Docs build with Python alone. */ +/** + * Writes the reference pages in `docs/reference`: + * + * - `api.md`, the JavaScript API, from the package's shipped declarations. + * - `types.md`, the scene and result types, from the `inputs.ts` and `outputs.ts` built from the + * schema. + * - `cli.md`, the command reference, from the command tree the CLI's test writes to `cli.json`. + * + * Each entry has a heading, its declaration, and its documentation, then its members or the forms + * of its union. Each member or form is a signature over its definition, with its own anchor. Types + * in signatures link to their entries, as do code spans in the documentation that name an entry or + * member. With `--check`, asserts that the checked-in pages are current and writes nothing. The + * pages are checked in so Read the Docs can build them with Python alone. + */ import assert from 'node:assert/strict'; import fs from 'node:fs'; import path from 'node:path'; @@ -7,245 +20,979 @@ import prettier from 'prettier'; const root = path.resolve(import.meta.dirname, '..'); const check = process.argv.includes('--check'); -const printer = ts.createPrinter({ newLine: ts.NewLineKind.LineFeed }); +const prettierConfig = await prettier.resolveConfig(path.join(root, 'npm/src/index.ts')); + +// Reading declarations + +/** + * Returns `text` with its first comment blanked out. Every module opens with a comment about + * itself. Where the declarations drop the imports below it, it would read as the first + * declaration's doc. + */ +function withoutModuleDoc(text) { + const comments = ts.getLeadingCommentRanges(text, 0) ?? []; + const first = comments.find((range) => !text.startsWith('///', range.pos)); + if (!first) return text; + return text.slice(0, first.pos) + ' '.repeat(first.end - first.pos) + text.slice(first.end); +} + +/** Returns `text` with every comment blanked out and its positions kept. */ +function uncommented(text) { + const scanner = ts.createScanner( + ts.ScriptTarget.Latest, + false, + ts.LanguageVariant.Standard, + text, + ); + let out = ''; + let at = 0; + for (let kind = scanner.scan(); kind !== ts.SyntaxKind.EndOfFileToken; kind = scanner.scan()) { + if ( + kind === ts.SyntaxKind.SingleLineCommentTrivia || + kind === ts.SyntaxKind.MultiLineCommentTrivia + ) { + const [start, end] = [scanner.getTokenStart(), scanner.getTokenEnd()]; + out += text.slice(at, start) + text.slice(start, end).replace(/[^\n]/g, ' '); + at = end; + } + } + return out + text.slice(at); +} + +/** + * Returns the doc comment just before position `pos` of `text` as Markdown, or '' if there is none. + * The comment may sit on the line before, or on the same line, as before a union member: + * `| /** … *\/ { … }`. + */ +function docBefore(text, pos) { + const ranges = [ + ...(ts.getTrailingCommentRanges(text, pos) ?? []), + ...(ts.getLeadingCommentRanges(text, pos) ?? []), + ]; + const comment = ranges.filter((range) => text.startsWith('/**', range.pos)).at(-1); + if (!comment) return ''; + return text + .slice(comment.pos + 3, comment.end - 2) + .split('\n') + .map((line) => line.replace(/^\s*\* ?/, '')) + .join('\n') + .trim(); +} + +const has = (node, kind) => (ts.getModifiers?.(node) ?? []).some((m) => m.kind === kind); + +/** + * Returns the declarations of `file` by name. Each has a `kind` (function, const, class, interface, + * or type) and its doc. Depending on the kind, it also has its overloads, its members, or its + * union's branches. + */ +function read(file) { + const raw = withoutModuleDoc(fs.readFileSync(path.join(root, file), 'utf8')); + const source = ts.createSourceFile(file, raw, ts.ScriptTarget.Latest, true); + const code = uncommented(raw); + /** The node's code, without comments, on one line. */ + const text = (node) => code.slice(node.getStart(source), node.end).replace(/\s+/g, ' ').trim(); + const doc = (node) => docBefore(raw, node.pos); + /** A declaration's type parameters, in full or (`names`) by name only, as `<…>`. */ + const generics = (node, names = false) => + node.typeParameters + ? `<${node.typeParameters.map((p) => (names ? p.name.getText(source) : text(p))).join(', ')}>` + : ''; + + /** + * The fields of a function's options: the members of each parameter typed as an object type, or + * as an intersection with one, such as `{ radius?: number } & Wait`. + */ + const optionsOf = (node) => + node.parameters.flatMap((p) => { + const types = p.type && ts.isIntersectionTypeNode(p.type) ? p.type.types : [p.type]; + return types.filter((t) => t && ts.isTypeLiteralNode(t)).flatMap((t) => members(t)); + }); + + /** + * The members of a class, interface, or object type, in order. A getter and its setter form one + * property, and a method's overloads form one method. A property with an object type lists that + * type's members as its `fields`, and a method lists its options' members. `Symbol.dispose` is + * left to `Handle`'s documentation. + */ + const members = (node) => { + const out = []; + const byName = new Map(); + for (const m of node.members) { + if (has(m, ts.SyntaxKind.PrivateKeyword) || has(m, ts.SyntaxKind.ProtectedKeyword)) continue; + if (m.name && ts.isPrivateIdentifier(m.name)) continue; + if (m.name && ts.isComputedPropertyName(m.name)) continue; + const about = doc(m); + if (/^@internal\b/.test(about)) continue; + const isStatic = has(m, ts.SyntaxKind.StaticKeyword); + const name = m.name ? m.name.getText(source) : 'new'; + if (ts.isGetAccessor(m) || ts.isSetAccessor(m)) { + let entry = byName.get(name); + if (!entry) { + entry = { kind: 'property', name, doc: '', type: '', readonly: true, static: isStatic }; + byName.set(name, entry); + out.push(entry); + } + if (ts.isGetAccessor(m)) entry.type = text(m.type); + else { + entry.readonly = false; + entry.type ||= text(m.parameters[0].type); + } + entry.doc ||= about; + } else if (ts.isPropertyDeclaration(m) || ts.isPropertySignature(m)) { + out.push({ + kind: 'property', + name, + doc: about, + type: m.type ? text(m.type) : 'unknown', + optional: Boolean(m.questionToken), + readonly: has(m, ts.SyntaxKind.ReadonlyKeyword), + static: isStatic, + fields: m.type && ts.isTypeLiteralNode(m.type) ? members(m.type) : [], + }); + } else if ( + ts.isConstructorDeclaration(m) || + ts.isMethodDeclaration(m) || + ts.isMethodSignature(m) + ) { + const overload = { code: text(m), doc: about, options: optionsOf(m) }; + const known = byName.get(`${name}()`); + if (known) { + known.overloads.push(overload); + known.doc ||= about; + continue; + } + const kind = ts.isConstructorDeclaration(m) ? 'constructor' : 'method'; + const entry = { kind, name, doc: about, overloads: [overload], static: isStatic }; + byName.set(`${name}()`, entry); + out.push(entry); + } else assert.fail(`${file}: ${text(m)} is a kind of member the reference cannot show`); + } + return out; + }; + + /** + * A union's branches, each with its doc, or the single type of a type that is not a union. The + * first branch's doc may come before the union's leading `|`. + */ + const branches = (node) => + (ts.isUnionTypeNode(node) ? node.types : [node]).map((t, i) => { + const branch = { text: text(t), doc: doc(t) || (i === 0 ? doc(node) : '') }; + if (ts.isLiteralTypeNode(t)) return { ...branch, kind: 'literal' }; + if (ts.isTypeReferenceNode(t) && !t.typeArguments) + return { ...branch, kind: 'named', name: t.typeName.getText(source) }; + if (ts.isTypeLiteralNode(t)) return { ...branch, kind: 'object', members: members(t) }; + return { ...branch, kind: 'other' }; + }); -/** Each declaration of `file` by name, as its signature. */ -function declarations(file) { - const text = fs.readFileSync(path.join(root, file), 'utf8'); - const source = ts.createSourceFile(file, text, ts.ScriptTarget.Latest, true); const found = new Map(); for (const node of source.statements) { if (ts.isImportDeclaration(node) || ts.isExportDeclaration(node) || ts.isEmptyStatement(node)) continue; - const name = - node.name?.getText(source) ?? - (ts.isVariableStatement(node) - ? node.declarationList.declarations[0].name.getText(source) - : undefined); - if (!name) continue; - const signature = printer - .printNode(ts.EmitHint.Unspecified, node, source) - .replace(/\bexport declare /g, '') - .replace(/\bexport /g, '') - .replace(/^\s*(?:#private;|private .*;)\s*$/gm, ''); - found.set(name, signature.trim()); + const about = doc(node); + if (ts.isFunctionDeclaration(node)) { + const name = node.name.getText(source); + const code = text(node).replace(/^export (declare )?/, ''); + const overload = { code, doc: about, options: optionsOf(node) }; + const known = found.get(name); + if (known) { + known.overloads.push(overload); + known.doc ||= about; + } else found.set(name, { kind: 'function', name, doc: about, overloads: [overload] }); + } else if (ts.isVariableStatement(node)) { + // A constant holding a function reads as a function. + const variable = node.declarationList.declarations[0]; + const name = variable.name.getText(source); + const type = variable.type; + const callable = Boolean(type && ts.isFunctionTypeNode(type)); + const declaration = callable + ? `function ${name}${generics(type)}(${type.parameters.map(text).join(', ')}): ${text(type.type)}` + : `const ${name}: ${type ? text(type) : 'unknown'}`; + found.set(name, { + kind: callable ? 'function' : 'const', + name, + doc: about, + overloads: [{ code: declaration, doc: about }], + }); + } else if (ts.isClassDeclaration(node) || ts.isInterfaceDeclaration(node)) { + const name = node.name.getText(source); + found.set(name, { + kind: ts.isClassDeclaration(node) ? 'class' : 'interface', + abstract: has(node, ts.SyntaxKind.AbstractKeyword), + name, + generics: generics(node), + parameters: generics(node, true), + doc: about, + heritage: (node.heritageClauses ?? []).map(text), + members: members(node), + }); + } else if (ts.isTypeAliasDeclaration(node)) { + const name = node.name.getText(source); + found.set(name, { + kind: 'type', + name, + generics: generics(node), + parameters: generics(node, true), + doc: about, + type: text(node.type), + branches: branches(node.type), + }); + } } return found; } +// Anchors and links + +/** An anchor for a reference entry or one of its members: lowercase words joined by hyphens. */ +const anchor = (...parts) => + parts + .join('-') + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-|-$/g, ''); + /** - * A page of `files`' declarations under `head`, in `sections`: each a heading, what it holds, and - * its declarations in reading order. Every declaration is in one section, so none goes unlisted. + * What each documented name links to: its anchor, its link label, and its doc. Anchors are labels + * for the whole site, so a link needs no page. */ -function page(head, files, sections) { - const all = new Map(files.flatMap((file) => [...declarations(file)])); - const listed = sections.flatMap(([, , names]) => names); - const unlisted = [...all.keys()].filter((name) => !listed.includes(name)); - const missing = listed.filter((name) => !all.has(name)); - assert.deepEqual(unlisted, [], `npm/docs.mjs lists no section for ${unlisted.join(', ')}`); - assert.deepEqual(missing, [], `npm/docs.mjs lists what the package lacks: ${missing.join(', ')}`); - return ( - head + - sections - .map( - ([title, about, names]) => - `## ${title}\n\n${about}\n\n` + - names.map((name) => `### ${name}\n\n\`\`\`ts\n${all.get(name)}\n\`\`\`\n`).join('\n'), - ) - .join('\n') +const targets = new Map(); + +/** Returns the Markdown link to `name`, an entry or `Owner.member`, or undefined if it has none. */ +function link(name, label) { + const target = targets.get(name); + return target && `[${label ?? `\`${target.label}\``}](#${target.anchor})`; +} + +/** + * Returns the entry or member a code span would name: `Name`, `Name.member`, `instance.member` for + * a class's `Name.member`, or a call of any of these or of a function, `name()`. Returns undefined + * for other spans. + */ +function named(span) { + const call = span.endsWith('()'); + const bare = call ? span.slice(0, -2) : span; + if (!/^[A-Za-z]\w*(\.\w+)?$/.test(bare)) return undefined; + const [owner, member] = bare.split('.'); + if (!member) return /^[A-Z]/.test(owner) || call ? owner : undefined; + return `${owner[0].toUpperCase()}${owner.slice(1)}.${member}`; +} + +/** + * Returns `doc` with each `{@link Name}` made a link, and the first code span naming each other + * documented entry or member (not `self`) made one too. Fenced code is left alone. Throws for a + * `{@link}` to an unknown name. + */ +function linked(doc, self) { + const done = new Set([self]); + const prose = (text) => + text + .replace(/\{@link\s+([\w.$]+)(?:\s*\|\s*|\s+)?([^}]*)\}/g, (_, name, label) => { + const made = link(name, label.trim() || undefined); + assert(made, `{@link ${name}} names nothing the reference documents`); + done.add(name); + return made; + }) + .replace(/(? { + const name = named(code); + if (!name || done.has(name) || !targets.has(name)) return span; + done.add(name); + return link(name, span); + }); + return doc + .split(/(```[\s\S]*?```)/) + .map((part, i) => (i % 2 ? part : prose(part))) + .join(''); +} + +/** What lacks documentation, reported together once the pages are written. */ +const undocumented = []; + +/** Notes `what` as undocumented if `doc` is empty. */ +const documented = (doc, what) => doc || undocumented.push(what); + +// Signatures + +/** + * Formats `code` on one line with Prettier, without a final semicolon. `code` is a declaration by + * default, a class or interface member when `as` is `member`, and a type when `as` is `type`. + */ +async function oneLine(code, as = 'declaration') { + const [before, after] = { + declaration: ['declare ', ''], + member: ['declare class Members {\n', '\n}'], + type: ['type Type = ', ''], + }[as]; + const out = await prettier.format(`${before}${code}${after}`, { + ...prettierConfig, + parser: 'typescript', + printWidth: 100_000, + }); + const lines = out.trim().split('\n'); + const body = as === 'member' ? lines.slice(1, -1) : lines; + assert.equal(body.length, 1, `${code} prints on more than one line`); + const line = body[0].trim().replace(/;$/, ''); + return as === 'member' ? line : line.slice(before.length); +} + +/** + * Wraps `text` in a code span, keeping the spaces at its ends outside the span. Markdown trims a + * span that starts and ends with a space. + */ +function codeSpan(text) { + const [, lead, inner, trail] = text.match(/^( *)(.*?)( *)$/); + if (!inner) return text; + const ticks = '`'.repeat(Math.max(0, ...(inner.match(/`+/g) ?? []).map((t) => t.length)) + 1); + const pad = /^`|`$/.test(inner) ? ' ' : ''; + return `${lead}${ticks}${pad}${inner}${pad}${ticks}${trail}`; +} + +/** Returns `code` as code spans, linking each documented type other than `self`. */ +function linkedCode(code, self) { + const scanner = ts.createScanner( + ts.ScriptTarget.Latest, + false, + ts.LanguageVariant.Standard, + code, ); + const out = []; + let at = 0; + for (let kind = scanner.scan(); kind !== ts.SyntaxKind.EndOfFileToken; kind = scanner.scan()) { + const word = scanner.getTokenText(); + if (kind !== ts.SyntaxKind.Identifier || !/^[A-Z]/.test(word) || word === self) continue; + const made = link(word, `\`${word}\``); + if (!made) continue; + out.push(codeSpan(code.slice(at, scanner.getTokenStart())), made); + at = scanner.getTokenEnd(); + } + out.push(codeSpan(code.slice(at))); + return out.join(''); +} + +/** + * Returns a signature as Markdown: `code`, a one-line declaration, with each documented type other + * than `self` linked and the declared name, `code.slice(...name)`, in bold. + */ +function signature(code, self, name) { + if (!name) return linkedCode(code, self); + const [start, end] = name; + return [ + linkedCode(code.slice(0, start), self), + `**${codeSpan(code.slice(start, end))}**`, + linkedCode(code.slice(end), self), + ].join(''); +} + +/** Where the name begins in a member's one-line declaration: after its modifiers. */ +const nameAt = (code) => code.match(/^((static|readonly|abstract|override|async) )*/)[0].length; + +/** + * Returns a union's declaration on one line, leaving the details of its forms to the items below. + * An object with more than two keys reads `{ … }`, as does an object type inside a form. + */ +async function unionDeclaration(entry) { + const shown = entry.branches.map((b) => { + if (b.kind !== 'object') return b.text; + if (b.members.length > 2) return 'Hidden'; + const keys = b.members.map((m) => { + const type = m.fields?.length ? 'Hidden' : m.type; + return `${m.name}${m.optional ? '?' : ''}: ${type}`; + }); + return `{ ${keys.join('; ')} }`; + }); + const line = await oneLine(`type ${entry.name}${entry.generics} = ${shown.join(' | ')}`); + return line.replace(/\bHidden\b/g, '{ … }'); +} + +// Rendering + +/** Every anchor the pages define, to check that each is defined once. */ +const anchors = []; + +/** + * One item of a definition list: `term`, its permalink, and `definition`. An item with an anchor + * is a target for links. Lines after the first are indented to stay in the definition. + */ +function item(id, term, definition) { + if (id) anchors.push(id); + const attributes = id ? `{#${id} .member}` : '{.member}'; + const permalink = id ? ` [¶](#${id}){.headerlink}` : ''; + const lines = definition.trim().split('\n'); + const indented = [lines[0], ...lines.slice(1).map((l) => (l ? ` ${l}` : ''))].join('\n'); + return `${attributes}\n${term}${permalink}\n: ${indented}`; +} + +/** + * Returns the items for `member` of entry `owner`, at `path` below it. A method gets one item per + * overload, and the first carries the member's anchor. A property gets one item, with an item for + * each of its fields inside its definition. + */ +async function memberItems(owner, member, path = []) { + const id = anchor(owner, ...path, member.name); + const self = [owner, ...path, member.name].join('.'); + documented(member.doc, `${self} has no documentation`); + if (member.kind === 'property') { + const modifiers = `${member.static ? 'static ' : ''}${member.readonly ? 'readonly ' : ''}`; + const optional = member.optional ? '?' : ''; + const code = await oneLine(`${modifiers}${member.name}${optional}: ${member.type}`, 'member'); + const at = nameAt(code); + const fields = []; + for (const field of member.fields ?? []) + fields.push(...(await memberItems(owner, field, [...path, member.name]))); + const term = signature(code, owner, [at, at + member.name.length]); + return [item(id, term, [linked(member.doc, self), ...fields].join('\n\n'))]; + } + const items = []; + for (const [i, overload] of member.overloads.entries()) { + let code = await oneLine(overload.code, 'member'); + let name = [nameAt(code), nameAt(code) + member.name.length]; + if (member.kind === 'constructor') { + code = code.replace(/^constructor/, `new ${owner}`); + name = [0, `new ${owner}`.length]; + } + const term = signature(code, owner, name); + const options = await optionItems(owner, [...path, member.name], overload.options); + const about = linked(overload.doc || member.doc, self); + items.push(item(i === 0 ? id : undefined, term, [about, ...options].join('\n\n'))); + } + return items; +} + +/** The items of a function's or a method's options, at `path` below entry `owner`. */ +async function optionItems(owner, path, options = []) { + const items = []; + for (const option of options) items.push(...(await memberItems(owner, option, path))); + return items; } -const pages = { - 'api.md': page( - '# JavaScript API\n\nGenerated from the shipped declarations. `loadLatkit()` returns `Data`, `Gpu`, `Shot`, and\n`colormapCss`; builders and patches need no WebAssembly. Options take the [scene and result\ntypes](types.md).\n\n', - ['load', 'patch', 'scene', 'runtime'].map((name) => `npm/dist/${name}.d.ts`), +/** The items for a class's or interface's members: constructor, properties, then methods. */ +async function membersOf(entry) { + const groups = [ + ['Constructor', entry.members.filter((m) => m.kind === 'constructor')], + ['Properties', entry.members.filter((m) => m.kind === 'property')], + ['Methods', entry.members.filter((m) => m.kind === 'method')], + ].filter(([, members]) => members.length); + const out = []; + for (const [title, members] of groups) { + if (groups.length > 1 || entry.kind === 'class') out.push(`\`\`\`{rubric} ${title}\n\`\`\``); + for (const member of members) out.push(...(await memberItems(entry.name, member))); + } + return out; +} + +/** + * The key an event union's branches share, such as `type` in `{ type: 'name'; detail: … }`. Every + * branch is an object with this key as a literal and at most one other key. Undefined for other + * unions. + */ +function eventKey(entry) { + const objects = entry.branches.filter((b) => b.kind === 'object'); + if (objects.length !== entry.branches.length || objects.length < 2) return undefined; + const literal = (type) => /^(['"]).*\1$/.test(type); + return objects[0].members.find((m) => + objects.every( + (b) => b.members.length <= 2 && b.members.some((o) => o.name === m.name && literal(o.type)), + ), + )?.name; +} + +/** + * Returns the items for a union's branches: each documented value or named type, and each form. A + * named type with no doc of its own gets no item, since the declaration links it. An event, + * `{ type: 'name'; detail: … }`, is one item named by its tag, with its detail's fields inside its + * definition. A tagged form, `{ key: options }`, is one item with its fields inside its + * definition. Any other form lists its keys, after its doc when the union has several such forms. + */ +async function branchesOf(entry) { + const owner = entry.name; + const key = eventKey(entry); + const forms = entry.branches.filter((b) => b.kind === 'object' && !key && b.members.length > 1); + const out = []; + for (const branch of entry.branches) { + if (branch.kind === 'literal' && branch.doc) { + const term = signature(branch.text, owner, [0, branch.text.length]); + out.push(item(anchor(owner, branch.text), term, linked(branch.doc, owner))); + } else if (branch.kind === 'named' && branch.doc && targets.has(branch.name)) { + out.push(item(anchor(owner, branch.name), link(branch.name), linked(branch.doc, owner))); + } else if (branch.kind === 'object' && (key || branch.members.length === 1)) { + const tag = key ? branch.members.find((m) => m.name === key).type : branch.members[0].name; + const payload = key ? branch.members.find((m) => m.name !== key) : branch.members[0]; + const name = key ? tag.slice(1, -1) : tag; + const about = branch.doc || (key ? '' : payload.doc); + documented(about, `${owner} ${tag} has no documentation`); + const fields = []; + for (const field of payload?.fields ?? []) + fields.push(...(await memberItems(owner, field, [name]))); + let term; + if (key) { + // An event shows its tag, and the type of its detail when the detail has no fields. + const detail = payload && !payload.fields?.length ? `: ${payload.type}` : ''; + term = `**${codeSpan(tag)}**${detail && linkedCode(detail, owner)}`; + } else { + const code = await oneLine(branch.text, 'type'); + const at = code.indexOf(tag); + term = signature(code, owner, [at, at + tag.length]); + } + out.push(item(anchor(owner, name), term, [linked(about, owner), ...fields].join('\n\n'))); + } else if (branch.kind === 'object') { + if (forms.length > 1) { + documented(branch.doc, `a form of ${owner} has no documentation`); + out.push(linked(branch.doc, owner)); + } + for (const member of branch.members) out.push(...(await memberItems(owner, member))); + } + } + return out; +} + +/** + * An entry's heading text: a function's name with parentheses, or any other name with the names + * of its type parameters. + */ +const heading = (entry) => + entry.kind === 'function' ? `${entry.name}()` : `${entry.name}${entry.parameters ?? ''}`; + +/** An entry's declaration: a line for each overload of a function, else one line. */ +async function declaration(entry) { + if (entry.overloads) return Promise.all(entry.overloads.map((o) => oneLine(o.code))); + if (entry.kind === 'type') { + if (entry.branches.length > 1) return [await unionDeclaration(entry)]; + return [await oneLine(`type ${entry.name}${entry.generics} = ${entry.type}`)]; + } + const kind = `${entry.abstract ? 'abstract ' : ''}${entry.kind}`; + return [[`${kind} ${entry.name}${entry.generics}`, ...entry.heritage].join(' ')]; +} + +/** Renders an entry as Markdown. */ +async function render(entry) { + anchors.push(entry.anchor); + const lines = (await declaration(entry)).map((line) => signature(line, entry.name)); + documented(entry.doc, `${entry.name} has no documentation`); + const parts = [ + `(${entry.anchor})=`, + `### \`${heading(entry)}\``, + ...lines.map((line) => `{.signature}\n${line}`), + linked(entry.doc, entry.name), + ]; + const options = (entry.overloads ?? []).find((o) => o.options?.length)?.options; + parts.push(...(await optionItems(entry.name, [], options))); + if (entry.members) parts.push(...(await membersOf(entry))); + if (entry.branches && entry.branches.length > 1) parts.push(...(await branchesOf(entry))); + return parts.join('\n\n'); +} + +/** + * Renders a reference page: `head`, then each of `sections`, given as `[title, description, + * names]`, with its entries in the order listed. + */ +async function page(head, entries, sections) { + const out = [linked(head)]; + for (const [title, about, names] of sections) { + out.push(`## ${title}`, linked(about)); + for (const name of names) out.push(await render(entries.get(name))); + } + return `${out.join('\n\n')}\n`; +} + +// The command reference + +/** + * Renders the command reference from `cli.json`, the command tree the CLI's test writes. Each + * command shows its usage and description, then its arguments and options. Each argument or option + * is a term over its definition, with its own anchor. Code spans that name a command or option link + * to it. A command with the same arguments as `render` says so and does not list them again. + */ +function cliPage() { + const commands = JSON.parse(fs.readFileSync(path.join(root, 'docs/reference/cli.json'), 'utf8')); + const anchorOf = (command) => `cli-${command.name.split(' ')[1] ?? 'latkit'}`; + const idOf = (command, arg) => `${anchorOf(command)}-${arg.long ?? arg.id}`; + // What code spans link to: each command, and each option at the first command that has it. A + // command's own options override this below. + const links = new Map(commands.map((command) => [command.name, anchorOf(command)])); + for (const command of commands) + for (const arg of command.args) + if (arg.long && !links.has(`--${arg.long}`)) links.set(`--${arg.long}`, idOf(command, arg)); + const sentence = (help) => { + const text = help.trim(); + return !text || text.endsWith('.') ? text : `${text}.`; + }; + const term = (arg) => { + if (arg.positional) { + const value = arg.required ? `<${arg.value}>` : `[${arg.value}]`; + return `**\`${value}${arg.many ? '...' : ''}\`**`; + } + const short = arg.short ? `\`-${arg.short},\` ` : ''; + return `${short}**\`--${arg.long}\`**${arg.value ? ` \`<${arg.value}>\`` : ''}`; + }; + const render = commands.find((command) => command.name === 'latkit render'); + const out = [ + '# Command reference', + "Every command and option of the `latkit` command line, generated from its help. The [command-line guide](../cli.md) shows them in use. `--help` prints a command's help, and `latkit --version` prints the version.", + ]; + for (const command of commands) { + const top = command.name === 'latkit'; + const anchor = anchorOf(command); + anchors.push(anchor); + const local = new Map(links); + for (const arg of command.args) if (arg.long) local.set(`--${arg.long}`, idOf(command, arg)); + const linked = (text) => + text.replace(/`([^`\n]+)`/g, (span, code) => + local.has(code) ? `[${span}](#${local.get(code)})` : span, + ); + out.push( + `(${anchor})=`, + `## \`${command.name}\``, + `{.signature}\n\`${command.usage}\``, + linked(sentence(command.about)), + ); + if (top) { + out.push('```{rubric} Commands\n```'); + for (const sub of commands.slice(1)) + out.push( + item(undefined, `[\`${sub.name}\`](#${anchorOf(sub)})`, linked(sentence(sub.about))), + ); + } + if ( + !top && + command !== render && + JSON.stringify(command.args) === JSON.stringify(render.args) + ) { + out.push(`Takes ${linked('`latkit render`')}'s arguments and options.`); + continue; + } + for (const [title, positional] of [ + ['Arguments', true], + ['Options', false], + ]) { + const args = command.args.filter((arg) => arg.positional === positional); + if (!args.length) continue; + out.push(`\`\`\`{rubric} ${title}\n\`\`\``); + for (const arg of args) { + const id = idOf(command, arg); + let about = sentence(arg.help); + if (arg.defaults.length) + about += ` Defaults to ${arg.defaults.map((v) => `\`${v}\``).join(', ')}.`; + if (top && arg.global) about += ' Every command takes it.'; + const values = arg.values.map((value) => { + assert(value.help, `${command.name} ${id}: value ${value.name} has no help`); + return item(`${id}-${value.name}`, `**\`${value.name}\`**`, linked(sentence(value.help))); + }); + out.push(item(id, term(arg), [linked(about), ...values].join('\n\n'))); + } + } + } + return `${out.join('\n\n')}\n`; +} + +// The pages + +const API = [ + 'load', + 'internal', + 'data', + 'patch', + 'scene', + 'colormap', + 'gpu', + 'view', + 'canvas', + 'shot', +]; +const TYPES = ['inputs', 'outputs']; + +const sections = { + 'api.md': [ [ + 'Loading, errors, and lifetimes', + 'What every part of the package shares.', + ['load', 'LatkitError', 'Wait', 'Handle'], + ], + [ + 'Data and patches', + 'Datasets, the tables in them that marks draw, and the patches that change them. Patches are written without WebAssembly.', [ - 'Loading', - 'Load the runtime once a page.', - ['loadLatkit', 'LoadOptions', 'Latkit', 'LatkitError'], - ], - [ - 'Data and patches', - 'Datasets of named tables, changed by patches written without WebAssembly.', - [ - 'Data', - 'Table', - 'Envelope', - 'Patch', - 'RowIds', - 'Values', - 'Geometry', - 'Reference', - 'reference', - 'Fields', - 'ReadableBytes', - ], - ], - [ - 'Expressions', - 'What varies by row: builders writing the expressions scenes take.', - [ - 'Expr', - 'ExprInput', - 'Color', - 'field', - 'color', - 'missing', - 'when', - 'When', - 'Then', - 'Mapped', - 'units', - 'selected', - 'hovered', - 'pointer', - 'zoom', - 'elapsed', - 'icon', - 'RgbaImage', - 'colormapCss', - ], - ], - [ - 'Views', - 'A GPU makes views, and canvases that show them.', - ['Gpu', 'View', 'ViewEvents', 'Canvas', 'Panel'], + 'Dataset', + 'Data', + 'Snapshot', + 'Tiles', + 'Source', + 'Table', + 'TableInput', + 'QueryResult', + 'Envelope', + 'Patch', + 'ByteSource', + 'ReadableBytes', + 'RowIds', + 'Values', + 'Geometry', + 'Reference', + 'reference', + 'Fields', + 'Run', ], + ], + [ + 'Expressions', + 'Builders for the per-row expressions that scene options take, including colors.', [ - 'Pictures and video', - 'Shots draw views into images and video.', - ['Shot', 'Image', 'ImageFormat', 'Video'], + 'Expr', + 'ExprInput', + 'Mapped', + 'Color', + 'field', + 'when', + 'When', + 'Then', + 'Next', + 'px', + 'units', + 'selected', + 'hovered', + 'pointer', + 'zoom', + 'elapsed', + 'icon', + 'RgbaImage', + 'colormaps', + 'colormapCss', ], ], - ), - 'types.md': page( - '# Scene and result types\n\nGenerated from the same schema as the package types. Each field documents its meaning and defaults.\n\n', - ['inputs', 'outputs'].map((name) => `npm/src/${name}.ts`), [ + 'Views', + 'Views, which draw scenes on canvases, and the GPU they draw with.', + ['View', 'Rect', 'ViewEvents', 'Insets', 'Point', 'Position', 'Gpu'], + ], + [ + 'Pictures and video', + 'Shots draw views into images and video.', + ['Shot', 'ShotOptions', 'Placed', 'Image', 'ImageFormat', 'Video'], + ], + ], + 'types.md': [ + [ + 'Scenes', + 'A scene: its space, how it responds to input, and how it is laid out.', [ - 'Scenes', - 'A scene: its space, its marks, and how it looks and answers input.', - [ - 'Scene', - 'Space', - 'Plane', - 'Earth', - 'Projection', - 'Diagram', - 'Plot', - 'Axis', - 'Tick', - 'Format', - 'Grid', - 'Theme', - 'Flow', - 'Shadow', - 'Controls', - 'Motion', - 'Zoom', - 'Layout', - 'Layered', - 'Stress', - 'Direction', - 'Order', - 'Budget', - ], + 'Scene', + 'Space', + 'Plane', + 'Earth', + 'Projection', + 'Diagram', + 'Plot', + 'Axis', + 'Tick', + 'Format', + 'Grid', + 'Controls', + 'Motion', + 'Zoom', + 'Layout', + 'Layered', + 'Stress', + 'Direction', + 'Order', + 'Budget', ], + ], + [ + 'Themes', + 'How a scene looks: a group of options for each kind of thing it draws, and the lengths those options take.', [ - 'Marks', - 'What a scene draws each table as.', - [ - 'Mark', - 'Vertices', - 'Edges', - 'Lines', - 'Areas', - 'Faces', - 'Marker', - 'Shape', - 'Bitmap', - 'Port', - 'Side', - 'End', - 'Route', - 'Interpolation', - 'Label', - 'Size', - 'Units', - ], + 'Theme', + 'TextTheme', + 'VertexTheme', + 'EdgeTheme', + 'LineTheme', + 'AreaTheme', + 'FlowTheme', + 'FocusTheme', + 'DiagramTheme', + 'ShadowTheme', + 'EarthTheme', + 'PlotTheme', + 'Length', ], + ], + [ + 'Marks', + "The marks that draw a scene's tables.", [ - 'Expressions', - 'Expressions as JSON: a constant, a field, or a node of these kinds.', - [ - 'ExprNode', - 'FieldNode', - 'ColorNode', - 'StateNode', - 'MissingNode', - 'UnaryNode', - 'BinaryNode', - 'WhenNode', - 'IsNode', - 'ContainsNode', - 'ScaleNode', - 'ColormapNode', - 'Unary', - 'Binary', - 'State', - 'Colormap', - 'ColormapName', - 'ColormapKind', - 'CustomColormap', - ], + 'Mark', + 'MarkOptions', + 'StrokeOptions', + 'Vertices', + 'Edges', + 'Lines', + 'Areas', + 'Faces', + 'Marker', + 'Shape', + 'Bitmap', + 'Port', + 'Side', + 'Route', + 'Interpolation', + 'Label', + 'Size', + 'Px', + 'Units', ], + ], + [ + 'Expressions', + 'Expressions as JSON: a number, a boolean, `null`, a field as `"$name"`, a CSS color, or one of these nodes.', [ - 'Rows, cameras, and queries', - 'Naming rows, framing them, and asking for them.', - ['Id', 'Ids', 'Row', 'Rows', 'Camera', 'Fit', 'Query'], + 'ExprNode', + 'FieldNode', + 'StateNode', + 'CallNode', + 'WhenNode', + 'ScaleNode', + 'ColormapNode', + 'State', + 'Colormap', + 'ColormapName', + 'ColormapKind', + 'CustomColormap', ], + ], + [ + 'Rows, cameras, and queries', + 'Identifying rows, framing them, and querying them.', + ['Id', 'Ids', 'Row', 'Rows', 'Camera', 'Fit', 'Query', 'EnvelopeRequest'], + ], + [ + 'Results', + 'View events and hits, dataset descriptions and values, and frame statistics.', [ - 'Results', - 'What views tell: events, hits, descriptions, and what frames cost.', - [ - 'ViewEvent', - 'Hit', - 'Failure', - 'Phase', - 'Trigger', - 'Device', - 'Modifiers', - 'Description', - 'TableDescription', - 'FieldDescription', - 'Kind', - 'Value', - 'Sample', - 'Stats', - 'Waiting', - 'Work', - ], + 'ViewEvent', + 'Hit', + 'End', + 'Failure', + 'Phase', + 'Trigger', + 'Device', + 'Modifiers', + 'Description', + 'TableDescription', + 'FieldDescription', + 'Kind', + 'Value', + 'Sample', + 'Stats', + 'Waiting', + 'Work', ], ], - ), + ], +}; + +const heads = { + 'api.md': `# JavaScript API + +Every class, function, and type the \`latkit\` package exports. Scenes and their options use the +[scene and result types](types.md). These rules apply everywhere, so the entries below don't +repeat them: + +- Scenes, expressions, and patches are plain data. Datasets, views, pictures, and the colormap + functions need latkit's WebAssembly, and throw an \`unloaded\` error until \`load()\` finishes. + \`View.create()\`, \`Gpu.create()\`, \`Data.read()\`, and \`Patch.read()\` of JSON call it + themselves. +- Everything latkit throws or rejects with is a \`LatkitError\`. Input it can't use fails with an + \`invalid\` error and changes nothing. The error's \`path\` names the bad part, such as + \`marks[2].color\`. +- Every promise waits until its \`Wait\` signal aborts, or for 30 seconds without one. +- A \`Handle\` holds memory until \`destroy()\` or the end of a \`using\` block frees it.`, + 'types.md': `# Scene and result types + +The types of the scenes latkit draws and the results it reports, generated from the same schema as +the package's types. Each property gives its meaning and default.`, }; -for (const [name, content] of Object.entries(pages)) { +/** The declarations of every file in `files`, read from `dir`. */ +const readAll = (files, dir, ext) => new Map(files.flatMap((f) => [...read(`${dir}/${f}${ext}`)])); +const entries = { + 'api.md': readAll(API, 'npm/dist', '.d.ts'), + 'types.md': readAll(TYPES, 'npm/src', '.ts'), +}; + +// Anchors ignore case, so a builder named like a type on either page, such as `zoom()` and `Zoom`, +// gets `-function` appended to its anchor. +const types = new Set( + Object.values(entries) + .flatMap((all) => [...all.values()]) + .filter((e) => e.kind !== 'function') + .map((e) => anchor(e.name)), +); +// Every declaration is listed in exactly one section, and documented on exactly one page. +for (const [name, all] of Object.entries(entries)) { + const listed = sections[name].flatMap(([, , names]) => names); + const unlisted = [...all.keys()].filter((n) => !listed.includes(n)); + const missing = listed.filter((n) => !all.has(n)); + assert.deepEqual(unlisted, [], `npm/docs.mjs lists no section for ${unlisted.join(', ')}`); + assert.deepEqual(missing, [], `npm/docs.mjs lists undeclared names: ${missing.join(', ')}`); + for (const entry of all.values()) { + assert(!targets.has(entry.name), `${entry.name} is documented twice`); + const collides = entry.kind === 'function' && types.has(anchor(entry.name)); + entry.anchor = collides ? anchor(entry.name, 'function') : anchor(entry.name); + const label = heading(entry).replace(/<.*>$/, ''); + targets.set(entry.name, { anchor: entry.anchor, label, doc: entry.doc }); + for (const member of entry.members ?? []) { + const prefix = entry.kind === 'class' ? `${entry.name}.` : ''; + const call = member.kind === 'method' ? '()' : ''; + targets.set(`${entry.name}.${member.name}`, { + anchor: anchor(entry.name, member.name), + label: `${prefix}${member.name}${call}`, + doc: member.doc, + }); + } + // A union's forms list their keys like members, except an event's, which nest in the event. + const forms = entry.branches && !eventKey(entry) ? entry.branches : []; + for (const member of forms.flatMap((b) => (b.kind === 'object' ? b.members : []))) { + targets.set(`${entry.name}.${member.name}`, { + anchor: anchor(entry.name, member.name), + label: member.name, + doc: member.doc, + }); + } + } +} + +const written = new Map(); +for (const name of Object.keys(entries)) { const file = path.join(root, 'docs/reference', name); - const formatted = await prettier.format(content, { - ...(await prettier.resolveConfig(file)), - parser: 'markdown', - }); + const content = await page(heads[name], entries[name], sections[name]); + // The reference must use npm's names. No line may contain a Rust path (`::`). types.md, which the + // schema writes from Rust's documentation, may not contain a Rust call (`name()`) either. + const rust = name === 'types.md' ? /::|\w\(\)/ : /::/; + const leftover = content.split('\n').filter((line) => rust.test(line)); + assert.deepEqual(leftover, [], `${file}: Rust's names in the TypeScript reference`); + assert(!content.includes('{@link'), `${file}: a link left unresolved`); + written.set(file, content); +} +written.set(path.join(root, 'docs/reference/cli.md'), cliPage()); + +// Anchors are labels for the whole site, so each must be unique, and none may be a heading's slug. +const slugs = [ + ...Object.values(heads), + ...Object.values(sections) + .flat() + .map(([title]) => title), +].map((text) => anchor(text.split('\n')[0].replace(/^# /, ''))); +const twice = anchors.filter((a, i) => anchors.indexOf(a) !== i || slugs.includes(a)); +assert.deepEqual(twice, [], 'anchors used twice'); +assert.deepEqual(undocumented, [], 'every entry, member, and form documents itself'); +for (const [file, content] of written) { if (check) assert.equal( fs.readFileSync(file, 'utf8').replaceAll('\r\n', '\n'), - formatted, + content, `${file}: run node npm/docs.mjs after building the package`, ); else { fs.mkdirSync(path.dirname(file), { recursive: true }); - fs.writeFileSync(file, formatted); + fs.writeFileSync(file, content); } } diff --git a/npm/examples/gate/browser.ts b/npm/examples/gate/browser.ts index 19966c58..376aad20 100644 --- a/npm/examples/gate/browser.ts +++ b/npm/examples/gate/browser.ts @@ -1,16 +1,18 @@ /** - * What the gate and the benchmark share: Chrome to drive, and deadlines every wait races. + * Helpers shared by the gate and the benchmark: serving a page, launching Chrome, collecting a + * page's errors, and deadlines for every wait. */ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; -import { chromium, type Browser } from 'playwright-core'; +import { chromium, type Browser, type Page } from 'playwright-core'; +import { createServer } from 'vite'; /** Milliseconds any one step may take before giving up on it. */ export const STEP = 60_000; -/** Rejects after `ms` with what timed out. */ +/** Races `work` against a timer that rejects after `ms` with an error naming `step`. */ export function deadline(work: Promise, ms: number, step: string): Promise { let timer: NodeJS.Timeout | undefined; const late = new Promise((_, reject) => { @@ -19,7 +21,51 @@ export function deadline(work: Promise, ms: number, step: string): Promise return Promise.race([work, late]).finally(() => clearTimeout(timer)); } -/** Chrome to drive: `$LATKIT_CHROME`, or the newest Chromium in Playwright's browser cache. */ +/** A Vite dev server, and the address it serves at. */ +export interface Served { + readonly url: string; + /** Stops the server, waiting at most 10 s. */ + close(): Promise; +} + +/** Serves the Vite project in `root` with its `vite.config.ts`, at `port` or the next free port. + * `what` names it in errors. */ +export async function serve(root: string, port: number, what: string): Promise { + const server = await deadline( + createServer({ + root, + configFile: path.join(root, 'vite.config.ts'), + logLevel: 'error', + server: { port, strictPort: false }, + }), + STEP, + `configure ${what}`, + ); + const close = () => deadline(server.close(), 10_000, `stop serving ${what}`).catch(() => {}); + try { + await deadline(server.listen(), STEP, `serve ${what}`); + const url = server.resolvedUrls?.local[0]; + if (!url) throw new Error(`${what} has no address`); + return { url, close }; + } catch (error) { + await close(); + throw error; + } +} + +/** Collects `page`'s uncaught errors and console errors as they happen, each cut to 300 + * characters. */ +export function errorsOf(page: Page): string[] { + const errors: string[] = []; + page.on('pageerror', (error) => errors.push(`page: ${error.message.slice(0, 300)}`)); + page.on('console', (message) => { + if (message.type() === 'error') errors.push(`console: ${message.text().slice(0, 300)}`); + }); + return errors; +} + +/** Returns the path of the Chrome to drive: `$LATKIT_CHROME`, or else the newest Chromium in + * Playwright's browser cache. Throws if there is none. */ export function chrome(): string { const given = process.env.LATKIT_CHROME; if (given) return given; @@ -55,20 +101,20 @@ export function chrome(): string { ); } -/** A Chrome being driven, and how it ends. */ +/** A launched Chrome. */ export interface Chrome { browser: Browser; /** - * Ends Chrome: politely within seconds, else by killing its processes. A page stuck on its GPU - * never closes politely, and a Chrome left behind keeps the GPU busy under the next one. + * Closes Chrome, or kills its processes if it has not closed within 5 s. A page stuck on the GPU + * never closes normally, and a leftover Chrome would keep the GPU busy during the next one. */ close(): Promise; } /** - * WebGPU on SwiftShader, which draws on the CPU, so no shader can hang the machine's GPU. These - * override Chrome's own GPU checks, so a real GPU never gets them. ANGLE draws on SwiftShader too on - * Linux, as its GPU-less CI needs; on Windows that leaves WebGPU no adapter at all. + * Flags that run WebGPU on SwiftShader, which draws on the CPU, so no shader can hang the machine's + * GPU. They override Chrome's own GPU checks, so they are never passed with a real GPU. On Linux, + * ANGLE uses SwiftShader too, as GPU-less CI needs. On Windows that flag leaves WebGPU no adapter. */ const SOFTWARE = [ '--enable-unsafe-webgpu', @@ -81,13 +127,14 @@ const SOFTWARE = [ ]; /** - * A machine with two GPUs draws on its fast one, every launch: Chrome ignores WebGPU's - * `powerPreference` on Windows, and picks either otherwise, so times and limit-bound work would - * compare two devices. + * Makes a machine with two GPUs draw on the fast one on every launch. Chrome ignores WebGPU's + * `powerPreference` on Windows and may pick either GPU elsewhere, so times and limit-bound work + * would otherwise compare two devices. */ const HARDWARE = ['--force_high_performance_gpu']; -/** Chrome, headless, with `args` besides: on SwiftShader when `software`, as Linux always is. */ +/** Launches headless Chrome with the extra `args`. It draws on SwiftShader when `software`, which + * defaults to true on Linux. */ export async function launch( args: readonly string[] = [], software = process.platform === 'linux', diff --git a/npm/examples/gate/gate.ts b/npm/examples/gate/gate.ts index ef2f8f4e..f3a5abe3 100644 --- a/npm/examples/gate/gate.ts +++ b/npm/examples/gate/gate.ts @@ -1,43 +1,67 @@ /** - * The browser gate. Times every shader variant's pipelines in Chrome against a budget, then opens - * every page of the suite in a fresh browser and moves the pointer over it. It fails on a pipeline - * over budget or failing to compile, a console error, a page that draws no frame in time, a page - * missing frames in a row while it opens or under the pointer, or a pipeline compiled where first - * used. Every wait races a deadline. + * The browser gate. It times the pipelines of every shader variant in Chrome against a budget, then + * opens each page of the suite in a fresh browser and moves the pointer over it. * - * On SwiftShader (`--software`, as Linux always is), the CPU draws, slower than any budget, so the - * gate only checks that every pipeline compiles: pages and times are judged on a GPU. + * A pipeline fails if it does not compile or is over budget. A page fails on a console or page + * error, a pipeline compiled at first use, or no exported picture. A page is slow if it draws no + * frame in time, or misses frames in a row while it opens or under the pointer. A slow page is + * opened again, cold, and fails only if every open is slow: a machine stall slows one open, but a + * slow page slows them all. Every wait races a deadline. * - * `node gate/gate.ts [--software] [page...]`; `cargo xtask browser` runs - * it. Naming pages visits only those, without timing the shaders. Chrome is `$LATKIT_CHROME`, or the - * newest Chromium Playwright installed. + * On SwiftShader (`--software`, and always on Linux) the CPU draws, slower than any budget, so the + * gate only checks that every pipeline compiles. Pages and times are judged only on a GPU. + * + * Usage: `node gate/gate.ts [--software] [page...]`, which + * `cargo xtask browser` runs. Naming pages visits only those and skips timing the shaders. Chrome + * is `$LATKIT_CHROME`, or else the newest Chromium that Playwright installed. */ import fs from 'node:fs'; import http from 'node:http'; import path from 'node:path'; import type { Page } from 'playwright-core'; -import { createServer } from 'vite'; -import { STEP, deadline, launch, type Chrome } from './browser.ts'; +import { STEP, deadline, errorsOf, launch, serve, type Chrome } from './browser.ts'; +import { quantile } from './quantile.ts'; -/** Milliseconds Chrome may take to compile any one pipeline, cold: the slowest takes about 0.35 s, - * and a view opens no sooner than the slowest it uses, the rest compiling beside it. */ +/** Maximum milliseconds Chrome may take to compile one pipeline cold. The slowest takes about + * 0.35 s. A view opens no sooner than its slowest pipeline. The others compile in parallel. */ const COMPILE_BUDGET = 800; -/** Milliseconds from opening a page, cold, to its views' first frames, which draw everything their - * scenes use: every pipeline compiles side by side, off Chrome's GPU thread. */ +/** Maximum milliseconds from opening a page cold to its views' first frames. A first frame draws + * everything its scene uses. Every pipeline compiles in parallel, off Chrome's GPU thread. */ const FIRST_FRAME_BUDGET = 1000; -/** Display frames in a row a page may miss while it opens or the pointer moves over it. Frames come - * at the display's pace, which the machine sets: 60 Hz on a panel, 32 Hz over Remote Desktop, whose - * vblanks slip one at a time. Nothing compiles on Chrome's GPU thread, so no frame waits on latkit; - * what it compiles where first used is counted apart, as `work.late`. */ +/** Maximum consecutive display frames a page may miss while it opens or under the pointer. The + * machine sets the display rate: 60 Hz on a panel, 32 Hz over Remote Desktop, where single vblanks + * slip. Nothing compiles on Chrome's GPU thread, so no frame waits on latkit. Pipelines compiled at + * first use are counted separately, as `work.late`. */ const MISSED = 1; +/** Milliseconds between display frames beyond which the gate logs what the views were doing. */ +const LONG = 50; +/** Maximum cold opens per page to draw its first frame in time without missing frames. An error on + * any open fails the page at once. */ +const OPENS = 2; + +/** The failures of one open of a page: its errors, and the ways it was slow. */ +interface Opened { + wrong: string[]; + slow: string[]; +} + +/** A gap between display frames longer than `LONG`. It records each view's last CPU and GPU frame + * times and `waiting` list, and the GPU work done since the previous frame. */ +interface Long { + gap: number; + cpuMs: number[]; + gpuMs: number[]; + waiting: string[][]; + work: Record; +} const here = path.dirname(new URL(import.meta.url).pathname.replace(/^\/(\w:)/, '$1')); const examples = path.dirname(here); const args = process.argv.slice(2); const software = args.includes('--software') || process.platform === 'linux'; -/** A variant of shaders.json: its file, whether it draws, its compute entries, and its constants. */ +/** A shader variant listed in shaders.json. `append` is source added after the file's own. */ interface Variant { file: string; draws: boolean; @@ -61,12 +85,14 @@ interface Timing { error?: string; } -/** Chrome compiling cold on every launch, as a first visit does. */ +/** Launches Chrome without its shader disk cache, so every pipeline compiles cold, as on a first + * visit. */ function cold(): Promise { return launch(['--disable-gpu-shader-disk-cache'], software); } -/** Every pipeline the variants in `dir` make: each compute entry, or the draw, per constants. */ +/** Lists the pipelines of the variants in `dir`: one per compute entry point, or one draw, for each + * set of constants. */ function pipelines(dir: string): Pipeline[] { const variants = JSON.parse(fs.readFileSync(path.join(dir, 'shaders.json'), 'utf8')) as Variant[]; return variants.flatMap((variant) => { @@ -86,7 +112,8 @@ function pipelines(dir: string): Pipeline[] { }); } -/** In the page: makes each pipeline cold, one at a time, and times it. Runs from its source alone. */ +/** Runs in the page. Creates each pipeline cold, sequentially, and times it. Only its source + * reaches the page, so it must be self-contained. */ async function time({ pipelines, step, @@ -123,7 +150,7 @@ async function time({ if (!module) modules.set(pipeline.code, (module = device.createShaderModule({ code: pipeline.code }))); const constants = pipeline.constants; - // Draws test against depth, as latkit's do; the backdrop writes it. + // Like latkit's draw pipelines, these test against depth, which the backdrop writes. const made = pipeline.draws ? device.createRenderPipelineAsync({ layout: 'auto', @@ -165,7 +192,8 @@ async function time({ return timings; } -/** Times every pipeline against the budget; returns the failures. */ +/** Times every pipeline and returns the failures: compile errors and, on a GPU, pipelines over + * `COMPILE_BUDGET`. */ async function compileBudget(dir: string): Promise { const blank = http.createServer((_, response) => response.end('gate'), @@ -206,22 +234,22 @@ function frames(page: Page): Promise { return deadline(two, STEP, 'animation frames'); } -/** Waits, for at most `budget` ms, until every view of `page` has opened: it was complete, or it drew - * 30 frames waiting on nothing but its fit, as a plot of streaming data measuring its value range - * again does every frame. A view draws once every pipeline its scene uses compiled, so then nothing - * is left to compile. */ +/** Waits up to `budget` ms until every view of `page` has opened. A view has opened once nothing is + * waiting, or once it has drawn 30 frames waiting only on `fit`, since a plot of streaming data + * refits its value range every frame. A view draws only after every pipeline its scene uses has + * compiled, so nothing is left to compile by then. */ async function settled(page: Page, budget: number): Promise { await page.waitForFunction( () => { const w = window as unknown as { - stage?: { views: { stats: { frames: number; waiting: string[] } }[] }; + stage?: { views: { stats: { frames: number }; waiting: string[] }[] }; opened?: Set; refitting?: Map; }; const views = w.stage?.views ?? []; w.opened ??= new Set(); w.refitting ??= new Map(); - views.forEach(({ stats: { frames, waiting } }, i) => { + views.forEach(({ stats: { frames }, waiting }, i) => { if (!waiting.length) w.opened!.add(i); if (waiting.join() !== 'fit') w.refitting!.delete(i); else if (!w.refitting!.has(i)) w.refitting!.set(i, frames); @@ -234,47 +262,91 @@ async function settled(page: Page, budget: number): Promise { ); } -/** Opens page `id` in a fresh browser, waits for its first frames, then moves the pointer over - * every view: hovering a grid of points, zooming, panning, and clicking. Returns the failures. */ +/** Opens page `id` until an open is in time or has errors, at most `OPENS` times. Returns the + * failures of the last open. */ async function visit(url: string, id: string): Promise { - const failures: string[] = []; + for (let n = 1; ; n++) { + const { wrong, slow } = await open(url, id); + if (wrong.length || !slow.length || n === OPENS) return [...wrong, ...slow]; + console.log(` ${id}: ${slow.join('; ')}; opening it again`); + } +} + +/** Opens page `id` in a fresh browser and waits for its first frames. Then it works every view with + * the pointer and keys (hovering a grid of points, zooming, dragging, clicking, right-clicking, and + * stepping with Tab and the arrows) and exports a picture. */ +async function open(url: string, id: string): Promise { const chrome = await cold(); try { - const context = await chrome.browser.newContext({ - viewport: { width: 1280, height: 800 }, - deviceScaleFactor: 1, - }); - const page = await context.newPage(); - page.on('console', (message) => { - if (message.type() === 'error') failures.push(`console: ${message.text().slice(0, 300)}`); - }); - page.on('pageerror', (error) => failures.push(`page: ${error.message.slice(0, 300)}`)); - // The longest gap between animation frames since the last reset, and the display's frame - // interval: the median of the latest gaps. - await page.addInitScript(() => { - const w = window as unknown as { gap: number; resetGap(): void; interval(): number }; + const context = await deadline( + chrome.browser.newContext({ viewport: { width: 1280, height: 800 }, deviceScaleFactor: 1 }), + STEP, + 'a browser context', + ); + const page = await deadline(context.newPage(), STEP, 'a page'); + const opened: Opened = { wrong: errorsOf(page), slow: [] }; + const { wrong, slow } = opened; + // Tracks the longest gap between animation frames since the last reset, the last 120 gaps + // (their median is the display's frame interval), and each gap longer than `LONG` with what the + // views were doing. + await page.addInitScript((long: number) => { + interface Stage { + views: { stats: { cpuMs: number; gpuMs: number }; waiting: string[] }[]; + gpu: { work: Record }; + } + const w = window as unknown as { + gap: number; + gaps: number[]; + long: Long[]; + resetGap(): void; + stage?: Stage; + }; let last = 0; + let work: Record = {}; const gaps: number[] = []; w.gap = 0; + w.gaps = gaps; + w.long = []; w.resetGap = () => { w.gap = 0; + w.long = []; last = 0; }; - w.interval = () => [...gaps].sort((a, b) => a - b)[gaps.length >> 1] ?? 1000 / 60; const tick = (now: number) => { + const stage = w.stage; + const done = stage?.gpu.work ?? {}; if (last) { - w.gap = Math.max(w.gap, now - last); - gaps.push(now - last); + const gap = now - last; + w.gap = Math.max(w.gap, gap); + gaps.push(gap); if (gaps.length > 120) gaps.shift(); + if (gap > long && stage) { + const delta = Object.entries(done) + .map(([kind, n]) => [kind, n - (work[kind] ?? 0)] as const) + .filter(([, n]) => n > 0); + w.long.push({ + gap, + cpuMs: stage.views.map((view) => view.stats.cpuMs), + gpuMs: stage.views.map((view) => view.stats.gpuMs), + waiting: stage.views.map((view) => view.waiting), + work: Object.fromEntries(delta), + }); + } } + work = done; last = now; requestAnimationFrame(tick); }; requestAnimationFrame(tick); - }); - const opened = performance.now(); + }, LONG); + /** Logs each frame longer than `LONG` since the last reset, labeled with `doing`. */ + const tell = async (doing: string) => { + const long = await page.evaluate(() => (window as unknown as { long: Long[] }).long); + for (const frame of long) console.log(` ${id}: ${describe(frame)} ${doing}`); + }; + const started = performance.now(); await deadline(page.goto(`${url}#${id}`), STEP, `open ${id}`); - /** Waits until every view has drawn more than `frames` frames, for at most `budget` ms. */ + /** Waits up to `budget` ms until every view has drawn more than `frames` frames. */ const drawn = (frames: number, budget: number) => page.waitForFunction( (frames) => { @@ -293,24 +365,27 @@ async function visit(url: string, id: string): Promise { `${id}'s first frame`, ); } catch { - failures.push(`no frame within ${FIRST_FRAME_BUDGET} ms`); - return failures; + slow.push(`no frame within ${FIRST_FRAME_BUDGET} ms`); + return opened; } - const first = performance.now() - opened; - // What stalls while the views open, from their first frames: what the page did loading before - // them shows nothing to stall. + const first = performance.now() - started; + // Stalls count from the views' first frames, since the page shows nothing that could stall + // while it loads. await page.evaluate(() => (window as unknown as { resetGap(): void }).resetGap()); await deadline(settled(page, STEP), STEP + 1000, `${id}'s opening`); await deadline(page.waitForTimeout(500), STEP, 'settle'); const opening = await page.evaluate(() => (window as unknown as { gap: number }).gap); + await tell('opening'); let longest = { gap: 0, step: '' }; - /** Does `step` and waits two frames; keeps the longest gap between frames and its step. */ + /** Performs `step` and waits two frames. Records the longest gap between frames with its step, + * and logs each frame longer than `LONG`. */ const act = async (step: string, action: () => Promise) => { await page.evaluate(() => (window as unknown as { resetGap(): void }).resetGap()); await deadline(action(), STEP, step); await frames(page); const gap = await page.evaluate(() => (window as unknown as { gap: number }).gap); if (gap > longest.gap) longest = { gap, step }; + await tell(`on ${step}`); }; const canvases = await page.locator('main canvas').all(); for (const [c, canvas] of canvases.entries()) { @@ -334,15 +409,16 @@ async function visit(url: string, id: string): Promise { await page.mouse.up(); }); await act(`click canvas ${c}`, () => page.mouse.click(cx, cy)); - // The intents whose picks list what lies under a point, and the keys stepping through rows. + // Right-click raises a menu intent listing what lies under the point. Tab and the arrow keys + // step through rows. await act(`menu on canvas ${c}`, () => page.mouse.click(cx, cy, { button: 'right' })); await act(`step with Tab on canvas ${c}`, () => page.keyboard.press('Tab')); await act(`step with an arrow on canvas ${c}`, () => page.keyboard.press('ArrowRight')); await act(`stop stepping on canvas ${c}`, () => page.keyboard.press('Escape')); } - // Every page exports a picture of its views: a shot draws them and a worker writes the PNG. + // Every page exports a picture of its views. A shot draws them and a worker writes the PNG. await page.locator('main canvas').first().focus(); - await deadline(page.keyboard.press('x'), STEP, 'ask for a picture'); + await deadline(page.keyboard.press('x'), STEP, 'press x to export a picture'); const saved = page.waitForFunction( () => document.body.textContent?.includes('Saved views.png'), undefined, @@ -354,28 +430,29 @@ async function visit(url: string, id: string): Promise { try { await deadline(saved, STEP + 1000, `${id}'s picture`); } catch { - // What the views waited on, and what the page said, name why. + // Explain the failure with the views' `waiting` lists and the page's status text. const said = await page.evaluate(() => { - const views = ( - window as unknown as { stage?: { views: { stats: { waiting: string[] } }[] } } - ).stage?.views; - const waiting = (views ?? []).map((view) => view.stats.waiting.join('+') || 'nothing'); + const views = (window as unknown as { stage?: { views: { waiting: string[] }[] } }).stage + ?.views; + const waiting = (views ?? []).map((view) => view.waiting.join('+') || 'nothing'); const status = [...document.querySelectorAll('.status, [role=status], output')] .map((element) => element.textContent?.trim()) .filter(Boolean); - return `views waiting on ${waiting.join(', ')}; the page said ${status.join(' | ') || 'nothing'}`; + return `views waiting on ${waiting.join(', ')}; page status: ${status.join(' | ') || 'nothing'}`; }); - failures.push(`no picture exported: ${said}`); + wrong.push(`no picture exported: ${said}`); } const { gap, step } = longest; - const { interval, late } = await page.evaluate(() => { + const { gaps, late } = await page.evaluate(() => { const w = window as unknown as { - interval(): number; + gaps: number[]; stage?: { gpu: { work: { late: number } } }; }; - return { interval: w.interval(), late: w.stage?.gpu.work.late ?? 0 }; + return { gaps: w.gaps, late: w.stage?.gpu.work.late ?? 0 }; }); - // A gap of n intervals missed n - 1 frames; past MISSED + 1.5 intervals, more than MISSED did. + const interval = gaps.length ? quantile(gaps, 0.5) : 1000 / 60; + // A gap of n intervals missed n - 1 frames, so a gap over MISSED + 1.5 intervals missed more + // than MISSED. const budget = (MISSED + 1.5) * interval; console.log( `${id}: first frame ${first.toFixed(0)} ms, frames every ${interval.toFixed(0)} ms, longest while ` + @@ -383,35 +460,38 @@ async function visit(url: string, id: string): Promise { ); const missed = (ms: number) => Math.round(ms / interval) - 1; if (opening > budget) - failures.push(`missed ${missed(opening)} frames in a row opening (${opening.toFixed(0)} ms)`); + slow.push(`missed ${missed(opening)} frames in a row opening (${opening.toFixed(0)} ms)`); if (gap > budget) - failures.push(`missed ${missed(gap)} frames in a row on ${step} (${gap.toFixed(0)} ms)`); + slow.push(`missed ${missed(gap)} frames in a row on ${step} (${gap.toFixed(0)} ms)`); if (late > 0) - failures.push( - `${late} pipelines compiled where first used, holding back the frames using them`, - ); + wrong.push(`${late} pipelines compiled at first use, delaying the frames that use them`); + return opened; } finally { await chrome.close(); } - return failures; } -/** Serves the suite and visits `only` its pages, or every page its bar lists. */ +/** Formats a frame longer than `LONG` for the log: each view's last CPU and GPU times and `waiting` + * list, and the GPU work done, with byte counts in MB. */ +function describe({ gap, cpuMs, gpuMs, waiting, work }: Long): string { + const ms = (list: number[]) => list.map((n) => n.toFixed(1)).join('/'); + const waited = waiting.map((list) => list.join('+') || 'nothing').join(', '); + const bytes = new Set(['uploaded', 'written', 'cleared']); + const done = Object.entries(work) + .map(([kind, n]) => `${kind} ${bytes.has(kind) ? `${(n / 1e6).toFixed(2)} MB` : n}`) + .join(', '); + return `frame of ${gap.toFixed(0)} ms (cpu ${ms(cpuMs)} ms, gpu ${ms(gpuMs)} ms, waiting on ${waited}; ${done || 'no work'})`; +} + +/** Serves the suite and visits the pages named in `only`, or, if it is empty, every page in the + * example picker. */ async function suite(only: string[]): Promise { - const server = await createServer({ - root: examples, - configFile: path.join(examples, 'vite.config.ts'), - server: { port: 5210, strictPort: false }, - logLevel: 'error', - }); - await deadline(server.listen(), STEP, 'serve the suite'); + const { url, close } = await serve(examples, 5210, 'the suite'); try { - const url = server.resolvedUrls?.local[0]; - if (!url) throw new Error('the suite has no address'); const chrome = await cold(); let ids: string[]; try { - const page = await chrome.browser.newPage(); + const page = await deadline(chrome.browser.newPage(), STEP, 'a page'); await deadline(page.goto(url), STEP, 'open the suite'); const picker = page.getByRole('combobox', { name: 'Example', exact: true }); await deadline(picker.waitFor(), STEP, 'the example picker'); @@ -427,7 +507,7 @@ async function suite(only: string[]): Promise { } return failures; } finally { - await server.close(); + await close(); } } @@ -437,7 +517,7 @@ if (!dir) { process.exit(2); } if (software) - console.log('SwiftShader draws on the CPU: pipelines must compile, untimed; no pages'); + console.log('SwiftShader draws on the CPU, so the gate only checks that pipelines compile'); const shaders = only.length && !software ? [] : (await compileBudget(dir)).map((f) => `shaders: ${f}`); const failures = [...shaders, ...(software ? [] : await suite(only))]; diff --git a/npm/examples/gate/package.ts b/npm/examples/gate/package.ts index b2e88618..c4ec4a66 100644 --- a/npm/examples/gate/package.ts +++ b/npm/examples/gate/package.ts @@ -6,7 +6,7 @@ import http from 'node:http'; import os from 'node:os'; import path from 'node:path'; import { build } from 'vite'; -import { deadline, launch, STEP, type Chrome } from './browser.ts'; +import { deadline, errorsOf, launch, STEP, type Chrome } from './browser.ts'; const root = path.resolve(import.meta.dirname, '../../..'); const reportPath = path.join(root, 'target/npm/package.json'); @@ -25,15 +25,6 @@ execFileSync( [path.join(root, 'node_modules/typescript/bin/tsc'), '-p', consumer], { stdio: 'inherit' }, ); -execFileSync( - process.execPath, - [ - path.join(root, 'node_modules/typescript/bin/tsc'), - '-p', - path.join(consumer, 'tsconfig.core.json'), - ], - { stdio: 'inherit' }, -); execFileSync( process.execPath, [ @@ -59,6 +50,11 @@ await build({ const dist = path.join(consumer, 'dist'); const server = http.createServer((request, response) => { const url = new URL(request.url ?? '/', 'http://localhost'); + // Chrome asks every site for an icon. The consumer has none, and a 404 would log a console error. + if (url.pathname === '/favicon.ico') { + response.writeHead(204).end(); + return; + } const relative = decodeURIComponent(url.pathname).replace(/^\/nested\/app\//, ''); const file = path.resolve(dist, relative || 'index.html'); if ( @@ -85,10 +81,9 @@ const timings: unknown[] = []; try { for (const mode of ['default', 'custom', 'compiled']) { chrome = await launch(['--disable-gpu-shader-disk-cache']); - const page = await chrome.browser.newPage(); - const errors: string[] = []; + const page = await deadline(chrome.browser.newPage(), STEP, 'open a page'); + const errors = errorsOf(page); const requests: string[] = []; - page.on('pageerror', (error) => errors.push(error.message)); page.on('request', (request) => requests.push(request.url())); await page.addInitScript(() => { const original = window.requestAnimationFrame.bind(window); @@ -130,7 +125,7 @@ try { ); assert.deepEqual(errors, []); timings.push({ mode, ...(result as object) }); - await page.close(); + await deadline(page.close(), 5_000, 'close the page'); await chrome.close(); } const measurement = { diff --git a/npm/examples/gate/quantile.ts b/npm/examples/gate/quantile.ts new file mode 100644 index 00000000..325f8a0b --- /dev/null +++ b/npm/examples/gate/quantile.ts @@ -0,0 +1,7 @@ +/** Returns the `q` quantile of `values`, from 0 (the least) to 1 (the greatest): the value that + * fraction of the way through them in order, rounding down. Returns 0 when there are none. The + * benchmark's page and runner use it too, so it imports nothing. */ +export function quantile(values: readonly number[], q: number): number { + if (!values.length) return 0; + return [...values].sort((a, b) => a - b)[Math.floor(q * (values.length - 1))]!; +} diff --git a/npm/examples/gate/tsconfig.json b/npm/examples/gate/tsconfig.json index 9a603b88..3ecd24b8 100644 --- a/npm/examples/gate/tsconfig.json +++ b/npm/examples/gate/tsconfig.json @@ -13,5 +13,5 @@ "types": ["node"], "lib": ["ES2023", "DOM", "DOM.Iterable"] }, - "include": ["gate.ts", "browser.ts", "package.ts"] + "include": ["gate.ts", "browser.ts", "package.ts", "quantile.ts"] } diff --git a/npm/examples/package.json b/npm/examples/package.json index e2ff9bee..94b6b43a 100644 --- a/npm/examples/package.json +++ b/npm/examples/package.json @@ -6,7 +6,8 @@ "scripts": { "dev": "vite", "build": "tsc --noEmit && vite build", - "preview": "vite preview" + "preview": "vite preview", + "history": "node scripts/history.ts" }, "dependencies": { "latkit": "link:.." diff --git a/npm/examples/public/history/0/0/0.latk b/npm/examples/public/history/0/0/0.latk new file mode 100644 index 00000000..2f14e012 Binary files /dev/null and b/npm/examples/public/history/0/0/0.latk differ diff --git a/npm/examples/public/history/0/0/1.latk b/npm/examples/public/history/0/0/1.latk new file mode 100644 index 00000000..5c5b092b Binary files /dev/null and b/npm/examples/public/history/0/0/1.latk differ diff --git a/npm/examples/public/history/0/0/10.latk b/npm/examples/public/history/0/0/10.latk new file mode 100644 index 00000000..75314ca0 Binary files /dev/null and b/npm/examples/public/history/0/0/10.latk differ diff --git a/npm/examples/public/history/0/0/11.latk b/npm/examples/public/history/0/0/11.latk new file mode 100644 index 00000000..f593b0f4 Binary files /dev/null and b/npm/examples/public/history/0/0/11.latk differ diff --git a/npm/examples/public/history/0/0/12.latk b/npm/examples/public/history/0/0/12.latk new file mode 100644 index 00000000..23d7cefe Binary files /dev/null and b/npm/examples/public/history/0/0/12.latk differ diff --git a/npm/examples/public/history/0/0/13.latk b/npm/examples/public/history/0/0/13.latk new file mode 100644 index 00000000..c7c5ad99 Binary files /dev/null and b/npm/examples/public/history/0/0/13.latk differ diff --git a/npm/examples/public/history/0/0/14.latk b/npm/examples/public/history/0/0/14.latk new file mode 100644 index 00000000..5f63fdda Binary files /dev/null and b/npm/examples/public/history/0/0/14.latk differ diff --git a/npm/examples/public/history/0/0/15.latk b/npm/examples/public/history/0/0/15.latk new file mode 100644 index 00000000..5a33799c Binary files /dev/null and b/npm/examples/public/history/0/0/15.latk differ diff --git a/npm/examples/public/history/0/0/16.latk b/npm/examples/public/history/0/0/16.latk new file mode 100644 index 00000000..caae08ac Binary files /dev/null and b/npm/examples/public/history/0/0/16.latk differ diff --git a/npm/examples/public/history/0/0/17.latk b/npm/examples/public/history/0/0/17.latk new file mode 100644 index 00000000..3ec93f68 Binary files /dev/null and b/npm/examples/public/history/0/0/17.latk differ diff --git a/npm/examples/public/history/0/0/18.latk b/npm/examples/public/history/0/0/18.latk new file mode 100644 index 00000000..65a2737d Binary files /dev/null and b/npm/examples/public/history/0/0/18.latk differ diff --git a/npm/examples/public/history/0/0/19.latk b/npm/examples/public/history/0/0/19.latk new file mode 100644 index 00000000..d77381ea Binary files /dev/null and b/npm/examples/public/history/0/0/19.latk differ diff --git a/npm/examples/public/history/0/0/2.latk b/npm/examples/public/history/0/0/2.latk new file mode 100644 index 00000000..52bae85c Binary files /dev/null and b/npm/examples/public/history/0/0/2.latk differ diff --git a/npm/examples/public/history/0/0/20.latk b/npm/examples/public/history/0/0/20.latk new file mode 100644 index 00000000..bdf2ee39 Binary files /dev/null and b/npm/examples/public/history/0/0/20.latk differ diff --git a/npm/examples/public/history/0/0/21.latk b/npm/examples/public/history/0/0/21.latk new file mode 100644 index 00000000..22895c40 Binary files /dev/null and b/npm/examples/public/history/0/0/21.latk differ diff --git a/npm/examples/public/history/0/0/22.latk b/npm/examples/public/history/0/0/22.latk new file mode 100644 index 00000000..a8b33901 Binary files /dev/null and b/npm/examples/public/history/0/0/22.latk differ diff --git a/npm/examples/public/history/0/0/3.latk b/npm/examples/public/history/0/0/3.latk new file mode 100644 index 00000000..fd09fe0f Binary files /dev/null and b/npm/examples/public/history/0/0/3.latk differ diff --git a/npm/examples/public/history/0/0/4.latk b/npm/examples/public/history/0/0/4.latk new file mode 100644 index 00000000..ceee2a6d Binary files /dev/null and b/npm/examples/public/history/0/0/4.latk differ diff --git a/npm/examples/public/history/0/0/5.latk b/npm/examples/public/history/0/0/5.latk new file mode 100644 index 00000000..b9f43154 Binary files /dev/null and b/npm/examples/public/history/0/0/5.latk differ diff --git a/npm/examples/public/history/0/0/6.latk b/npm/examples/public/history/0/0/6.latk new file mode 100644 index 00000000..c317e1d5 Binary files /dev/null and b/npm/examples/public/history/0/0/6.latk differ diff --git a/npm/examples/public/history/0/0/7.latk b/npm/examples/public/history/0/0/7.latk new file mode 100644 index 00000000..05f0f973 Binary files /dev/null and b/npm/examples/public/history/0/0/7.latk differ diff --git a/npm/examples/public/history/0/0/8.latk b/npm/examples/public/history/0/0/8.latk new file mode 100644 index 00000000..e8c2aa87 Binary files /dev/null and b/npm/examples/public/history/0/0/8.latk differ diff --git a/npm/examples/public/history/0/0/9.latk b/npm/examples/public/history/0/0/9.latk new file mode 100644 index 00000000..08aada51 Binary files /dev/null and b/npm/examples/public/history/0/0/9.latk differ diff --git a/npm/examples/public/history/0/1/0.latk b/npm/examples/public/history/0/1/0.latk new file mode 100644 index 00000000..d90888b3 Binary files /dev/null and b/npm/examples/public/history/0/1/0.latk differ diff --git a/npm/examples/public/history/0/1/1.latk b/npm/examples/public/history/0/1/1.latk new file mode 100644 index 00000000..729c75fb Binary files /dev/null and b/npm/examples/public/history/0/1/1.latk differ diff --git a/npm/examples/public/history/0/1/2.latk b/npm/examples/public/history/0/1/2.latk new file mode 100644 index 00000000..f0ed070a Binary files /dev/null and b/npm/examples/public/history/0/1/2.latk differ diff --git a/npm/examples/public/history/0/2/0.latk b/npm/examples/public/history/0/2/0.latk new file mode 100644 index 00000000..b4f4e724 Binary files /dev/null and b/npm/examples/public/history/0/2/0.latk differ diff --git a/npm/examples/public/history/0/3/0.latk b/npm/examples/public/history/0/3/0.latk new file mode 100644 index 00000000..efc40614 Binary files /dev/null and b/npm/examples/public/history/0/3/0.latk differ diff --git a/npm/examples/public/history/0/4/0.latk b/npm/examples/public/history/0/4/0.latk new file mode 100644 index 00000000..344785e5 Binary files /dev/null and b/npm/examples/public/history/0/4/0.latk differ diff --git a/npm/examples/public/history/base.latk b/npm/examples/public/history/base.latk new file mode 100644 index 00000000..14ea9dc9 Binary files /dev/null and b/npm/examples/public/history/base.latk differ diff --git a/npm/examples/public/history/frames.json b/npm/examples/public/history/frames.json new file mode 100644 index 00000000..19212357 --- /dev/null +++ b/npm/examples/public/history/frames.json @@ -0,0 +1,16 @@ +{ + "data": "base.latk", + "chunk": 256, + "url": "{field}/{level}/{index}.latk", + "frames": [ + { + "table": "Signal", + "field": "value", + "levels": 4, + "frames": [ + 0, + 5760 + ] + } + ] +} \ No newline at end of file diff --git a/npm/examples/scripts/history.ts b/npm/examples/scripts/history.ts new file mode 100644 index 00000000..c23ab3f5 --- /dev/null +++ b/npm/examples/scripts/history.ts @@ -0,0 +1,81 @@ +/** + * Writes the recorded day that the Telemetry page reads as time tiles, into `public/history/`. It + * contains the voltages of 64 buses in six areas, every 15 s for a day. A load curve swings them, + * and three disturbances pull areas down before they recover. Every run writes the same bytes. Run + * it from the examples with `node scripts/history.ts`. It tiles the day with the repository's + * `latkit tile`. + */ +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { Patch } from 'latkit'; +import { random } from '../src/random.ts'; + +/** The bus count, the area names, the seconds between samples, and the seconds in a day. */ +const BUSES = 64; +const AREAS = ['North', 'South', 'East', 'West', 'Coast', 'Valley']; +const EVERY = 15; +const DAY = 24 * 60 * 60; +/** Each disturbance's start in seconds of the day, its area as an index into `AREAS`, and how far + * it pulls voltages down, in per unit. */ +const DISTURBANCES = [ + { at: 9.2 * 3600, area: 2, depth: 0.05 }, + { at: 13.8 * 3600, area: 4, depth: 0.07 }, + { at: 18.5 * 3600, area: 0, depth: 0.06 }, +]; + +/** The system load at second `t` of the day, between 0 and 1. It is low at night and peaks in the + * morning and evening. */ +function load(t: number): number { + const hour = t / 3600; + const peak = (at: number, width: number) => Math.exp(-(((hour - at) / width) ** 2)); + return ( + 0.35 + 0.35 * peak(8.5, 2.5) + 0.45 * peak(19, 3) + 0.1 * Math.sin((hour / 24) * 2 * Math.PI) + ); +} + +/** The voltage drop, in per unit, of a disturbance of `depth`, `since` seconds after it starts. */ +function dip(since: number, depth: number): number { + return since < 0 ? 0 : depth * Math.exp(-since / 240) * Math.min(1, since / 20); +} + +const next = random(17); +const area = Float64Array.from({ length: BUSES }, (_, i) => i % AREAS.length); +const name = Array.from( + { length: BUSES }, + (_, i) => `${AREAS[i % AREAS.length]} ${Math.floor(i / AREAS.length) + 1}`, +); +const offset = Float32Array.from({ length: BUSES }, () => next() - 0.5); +const at = Array.from({ length: DAY / EVERY }, (_, k) => k * EVERY); +const values = new Float32Array(at.length * BUSES); +for (const [k, t] of at.entries()) { + const swing = load(t); + for (let i = 0; i < BUSES; i++) { + const a = area[i]!; + const struck = DISTURBANCES.filter((d) => d.area === a).reduce( + (sum, d) => sum + dip(t - d.at, d.depth), + 0, + ); + const noise = (next() - 0.5) * 0.004; + values[k * BUSES + i] = + 1.03 + 0.025 * offset[i]! - 0.06 * swing - (0.01 * a * swing) / 5 - struck + noise; + } +} +const day = new Patch('scada') + .axis('time', 's') + .set('Signal', { count: BUSES }, { name, area }) + .append('Signal', at, { value: values }); + +const examples = path.resolve(import.meta.dirname, '..'); +const out = path.join(examples, 'public', 'history'); +const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'latkit-history-')), 'day.latk'); +fs.writeFileSync(file, day.toBytes()); +fs.rmSync(out, { recursive: true, force: true }); +execFileSync( + 'cargo', + ['run', '--quiet', '-p', 'latkit-cli', '--', 'tile', file, '--output', out, '--chunk', '256'], + { cwd: path.resolve(examples, '../..'), stdio: 'inherit', timeout: 10 * 60_000 }, +); +fs.rmSync(path.dirname(file), { recursive: true, force: true }); +console.log(`wrote ${out}`); diff --git a/npm/examples/src/examples/colormaps/index.ts b/npm/examples/src/examples/colormaps/index.ts deleted file mode 100644 index 54aeae72..00000000 --- a/npm/examples/src/examples/colormaps/index.ts +++ /dev/null @@ -1,314 +0,0 @@ -import { theme } from '../../ui/theme.js'; -/** - * The colormap catalog on a field of hexagons: each cell an area colored on the GPU through the - * chosen colormap, beside the same colormap as a CSS ramp, so the two can be compared. Choose a - * colormap, a field to show, and how many cells; hover a cell to read it. - */ -import { - Data, - Patch, - colormapCss, - field, - type Colormap, - type ColormapName, - type Mark, - type Scene, -} from '../../latkit.js'; -import type { Stage } from '../../ui/stage.js'; -import { h } from '../../ui/dom.js'; - -/** - * A map of our own: clear to a placed, half-clear blue, then an opaque amber. Over the plane's - * grid, its alpha shows as the lines through the cells. - */ -const CUSTOM: Colormap = { - kind: 'sequential', - colors: ['#2a6fdb00', [0.55, '#2a6fdb99'], '#ffb000'], -}; -/** A colormap of the catalog, or ours. */ -type Choice = ColormapName | 'custom'; -const mapOf = (choice: Choice): Colormap => (choice === 'custom' ? CUSTOM : choice); -/** A checkerboard under a ramp, so what is clear in it shows. */ -const CHECKS = 'repeating-conic-gradient(#888 0 25%, #ccc 0 50%) 0 0 / 8px 8px'; - -/** Every catalog colormap by kind, then ours. */ -const CATALOG: readonly (readonly [string, readonly Choice[]])[] = [ - [ - 'Sequential', - [ - 'viridis', - 'cividis', - 'plasma', - 'inferno', - 'magma', - 'mako', - 'rocket', - 'crest', - 'flare', - 'amber', - 'amp', - 'batlow', - 'deep', - 'devon', - 'grays', - 'haline', - 'lajolla', - 'oslo', - 'thermal', - 'turku', - 'turbo', - ], - ], - [ - 'Diverging', - [ - 'balance', - 'berlin', - 'brbg', - 'broc', - 'coolwarm', - 'cork', - 'icefire', - 'managua', - 'piyg', - 'puor', - 'rdbu', - 'spectral', - 'vanimo', - 'vik', - 'vlag', - ], - ], - ['Cyclic', ['phase', 'romao', 'twilight']], - ['Categorical', ['tab10', 'tab20', 'tableau10', 'set2', 'dark2', 'paired', 'okabeito']], - ['Custom', ['custom']], -]; -const CATEGORICAL = new Set(CATALOG.find(([kind]) => kind === 'Categorical')![1]); -const KINDS = ['All', ...CATALOG.map(([kind]) => kind)] as const; - -/** What the cells show, each a value from 0 to 1 at a point; categorical maps read classes. */ -const FIELDS = { - waves: (x: number, y: number) => - 0.5 + 0.3 * Math.sin(x * 0.11) * Math.cos(y * 0.09) + 0.2 * Math.sin((x + y) * 0.035), - peaks: (x: number, y: number) => { - const peak = (cx: number, cy: number, r: number) => - Math.exp(-((x - cx) ** 2 + (y - cy) ** 2) / (2 * r * r)); - return Math.min(1, 0.9 * peak(30, 25, 14) + 0.7 * peak(80, 55, 20) + 0.5 * peak(110, 15, 10)); - }, - ramp: (x: number, _y: number, width: number) => x / width, - angle: (x: number, y: number, width: number, height: number) => - (Math.atan2(y - height / 2, x - width / 2) / (2 * Math.PI) + 1) % 1, -} as const; -type FieldName = keyof typeof FIELDS; - -/** A field of `columns` by `rows` hexagons of side 1, pointy side up, as dataset `colors`. */ -function hexagons(columns: number, rows: number, value: FieldName): Data { - const count = columns * rows; - const width = columns * Math.sqrt(3); - const height = rows * 1.5; - const geometry: [number, number][][][] = []; - const values = new Float64Array(count); - const classes = new Float64Array(count); - const s = Math.sqrt(3); - for (let r = 0; r < rows; r++) - for (let c = 0; c < columns; c++) { - const x = c * s + (r % 2) * (s / 2); - const y = r * 1.5; - const ring: [number, number][] = []; - for (let k = 0; k <= 6; k++) { - const a = (Math.PI / 3) * (k % 6) + Math.PI / 6; - ring.push([x + Math.cos(a), y + Math.sin(a)]); - } - geometry.push([ring]); - const v = FIELDS[value](x, y, width, height); - values[r * columns + c] = v; - classes[r * columns + c] = Math.min(9, Math.floor(v * 10)); - } - return new Data('colors').apply( - new Patch().set('Cell', { count }, { geometry, value: values, class: classes }), - ); -} - -export default function page(stage: Stage): void { - let colormap: Choice = 'viridis'; - let value: FieldName = 'peaks'; - let size: [number, number] = [96, 64]; - let outline = false; - let reversed = false; - let light = false; - - const scene = (): Scene => { - const input = CATEGORICAL.has(colormap) ? field('class') : field('value'); - const mapped = input.colormap(mapOf(colormap)); - const background = light ? '#f4f1ea' : theme.background; - const outlines: Mark[] = outline - ? [{ lines: { of: ['colors', 'Cell'], color: background, width: 0.75 } }] - : []; - return { - // Lines under the cells show through what a map leaves clear. - space: { plane: { grid: colormap === 'custom' ? { lines: { pitch: 4 } } : 'off' } }, - theme: { - ...theme, - base: light ? 'light' : 'dark', - background, - hover: light ? '#000000aa' : '#ffffffaa', - }, - ease: 0.25, - marks: [ - { - // The cells are what the page reads: hovering one reads its value. - areas: { - of: ['colors', 'Cell'], - color: CATEGORICAL.has(colormap) - ? mapped - : reversed - ? mapped.domain(1, 0) - : mapped.domain(0, 1), - pick: true, - }, - }, - ...outlines, - ], - }; - }; - const view = stage.view(scene(), { - label: 'A field of hexagons through a colormap: hover a cell to read it', - }); - const legend = stage.legend(); - - const rebuild = () => { - const [columns, rows] = size; - // The view keeps its own version of the data. - const data = hexagons(columns, rows, value); - view.setData(data); - data.destroy(); - stage.detail(`${(columns * rows).toLocaleString()} cells`); - }; - const restyle = () => { - view.show(scene()); - legend.ramp( - mapOf(colormap), - `${colormap}${CATEGORICAL.has(colormap) ? ', ten classes' : reversed ? ', reversed' : ''}`, - ); - for (const [name, swatch] of swatches) - swatch.setAttribute('aria-pressed', String(name === colormap)); - }; - - const palette = stage.section('Colormap'); - const choose = palette.choice( - 'Colormap', - CATALOG.flatMap(([kind, names]) => - names.map( - (name) => - [ - name === 'custom' ? 'ours · alpha, a placed stop' : `${name} · ${kind.toLowerCase()}`, - name, - ] as const, - ), - ), - colormap, - (name) => { - colormap = name; - restyle(); - }, - ); - palette.toggle('Reversed', 'r', reversed, (on) => { - reversed = on; - restyle(); - }); - palette.toggle('Outlines', 'o', outline, (on) => { - outline = on; - restyle(); - }); - palette.toggle('Light', 'l', light, (on) => { - light = on; - restyle(); - }); - - const shown = stage.section('Field'); - shown.choice( - 'Value', - [ - ['Peaks', 'peaks'], - ['Waves', 'waves'], - ['A ramp across', 'ramp'], - ['The angle around', 'angle'], - ] as const, - value, - (next) => { - value = next; - rebuild(); - }, - ); - // By columns: a field three rows for every two columns wide. - shown.choice( - 'Cells', - [ - ['1,500', 48], - ['6,000', 96], - ['24,000', 192], - ['96,000', 384], - ] as const, - size[0], - (columns) => { - size = [columns, (columns * 2) / 3]; - rebuild(); - view.fit(); - }, - ); - const reading = shown.note('Hover a cell.'); - view.on('hover', ({ hit }) => { - if (!hit) return void (reading.textContent = 'Hover a cell.'); - const data = view.data('colors'); - const v = data?.get('Cell', hit.row.id, 'value') as number | null; - reading.textContent = `Cell ${hit.row.id}: ${v?.toFixed(3) ?? '–'}`; - }); - - // The catalog as CSS ramps: the colors the GPU draws, drawn by the browser over checks, so - // what a map leaves clear shows. - const catalog = stage.section('Catalog'); - const groups: [string, HTMLElement[]][] = []; - catalog.choice( - 'Kind', - KINDS.map((kind) => [kind, kind] as const), - 'All', - (shown) => { - for (const [kind, elements] of groups) - for (const element of elements) element.hidden = shown !== 'All' && shown !== kind; - }, - ); - const swatches = new Map(); - for (const [kind, names] of CATALOG) { - const title = h('p', { class: 'note' }, kind); - const grid = h('div', { class: 'swatches' }); - groups.push([kind, [title, grid]]); - catalog.content.append(title); - for (const name of names) { - const swatch = h( - 'button', - { type: 'button', class: 'swatch', title: name, 'aria-pressed': 'false' }, - h('span', { class: 'ramp' }), - name, - ); - (swatch.firstChild as HTMLElement).style.background = - `${colormapCss(mapOf(name))}, ${CHECKS}`; - swatch.addEventListener( - 'click', - () => { - colormap = name; - choose.selectedIndex = CATALOG.flatMap(([, n]) => n).indexOf(name); - restyle(); - }, - { signal: stage.signal }, - ); - swatches.set(name, swatch); - grid.append(swatch); - } - catalog.content.append(grid); - } - - stage.meter(view); - rebuild(); - restyle(); - view.fit(); -} diff --git a/npm/examples/src/examples/diagram/index.ts b/npm/examples/src/examples/diagram/index.ts deleted file mode 100644 index ae1de7df..00000000 --- a/npm/examples/src/examples/diagram/index.ts +++ /dev/null @@ -1,688 +0,0 @@ -import { theme } from '../../ui/theme.js'; -import { - Data, - Patch, - color, - field, - missing, - reference, - when, - type Id, - type Layout, - type Mark, - type Port, - type Row, - type Rows, - type Marker, - type Scene, - type Side, -} from '../../latkit.js'; -import { each, one } from '../../ui/rows.js'; -import type { Stage } from '../../ui/stage.js'; - -/** - * Kinds of block: the marker each draws as, its fill on the dark and the light sheet, and its ports, - * each naming the signal it is plugged into. - */ -const KINDS = { - Input: { marker: 'rounded', fill: ['#173836', '#e2f3ef'], ports: [['out', 'right', 'out']] }, - Process: { - marker: 'block', - fill: ['#1c2b3f', '#ffffff'], - ports: [ - ['in', 'left', 'in'], - ['out', 'right', 'out'], - ], - }, - Control: { - marker: 'block', - fill: ['#3a2e18', '#fbf0da'], - ports: [ - ['in', 'left', 'in'], - ['feedback', 'bottom', 'in'], - ['out', 'right', 'out'], - ], - }, - Output: { marker: 'block', fill: ['#29243f', '#ece8fb'], ports: [['in', 'left', 'in']] }, -} as const satisfies Record< - string, - { - marker: Marker; - fill: readonly [string, string]; - ports: readonly (readonly [string, Side, 'in' | 'out'])[]; - } ->; -type Kind = keyof typeof KINDS; -const kinds = Object.keys(KINDS) as Kind[]; - -/** A port of a block, by the reference field naming its signal. */ -interface Plug { - kind: Kind; - id: Id; - port: string; -} - -const direction = (kind: Kind, port: string) => - KINDS[kind].ports.find(([name]) => name === port)?.[2]; - -/** A kind's ports as the scene declares them: outputs drive their signals, and a feedback port is named. */ -const ports = (kind: Kind): [string, Port][] => - KINDS[kind].ports.map(([name, side, way]): [string, Port] => [ - name, - { side, output: way === 'out', ...(name === 'feedback' ? { label: 'feedback' } : {}) }, - ]); - -/** - * A diagram drawn up as rows: blocks of each kind, the signals their ports plug into, and the groups - * holding Process blocks, themselves held by stations. Positions, x right and y down, are optional: - * blocks without one are placed by the layout. - */ -class Sketch { - readonly blocks = new Map< - Kind, - { id: number; name: string; at?: [number, number]; group?: number }[] - >(); - readonly plugs = new Map(); - readonly signals: { id: number; name: string }[] = []; - readonly groups: { id: number; name: string; station?: number }[] = []; - readonly stations: { id: number; name: string }[] = []; - /** Blocks whose status ring shows. */ - readonly alarms = new Set(); - next = 1; - - block(kind: Kind, name: string, at?: [number, number], group?: number): Plug { - const id = this.next++; - const rows = this.blocks.get(kind) ?? []; - rows.push({ id, name, at, group }); - this.blocks.set(kind, rows); - return { kind, id, port: 'out' }; - } - - /** A signal driven by `from`'s output (or the port it names) and read by each of `to`'s inputs. */ - wire(name: string, from: Plug | Plug[], ...to: (Plug | [Plug, string])[]): void { - const id = this.next++; - this.signals.push({ id, name }); - for (const driver of [from].flat()) this.plugs.set(`${driver.kind} ${driver.id} out`, id); - for (const reader of to) { - const [plug, port] = Array.isArray(reader) ? reader : [reader, 'in']; - this.plugs.set(`${plug.kind} ${plug.id} ${port}`, id); - } - } - - group(name: string, station?: number): number { - const id = this.next++; - this.groups.push({ id, name, station }); - return id; - } - - station(name: string): number { - const id = this.next++; - this.stations.push({ id, name }); - return id; - } - - /** The dataset. */ - data(): Data { - const patch = new Patch().set( - 'Signal', - this.signals.map((s) => s.id), - { name: this.signals.map((s) => s.name) }, - ); - if (this.stations.length) { - patch.set( - 'Station', - this.stations.map((s) => s.id), - { name: this.stations.map((s) => s.name) }, - ); - } - if (this.groups.length) { - const stations = this.groups.map((g) => g.station ?? null); - patch.set( - 'Group', - this.groups.map((g) => g.id), - { - name: this.groups.map((g) => g.name), - ...(this.stations.length ? { station: reference('Station', stations) } : {}), - }, - ); - } - for (const [kind, rows] of this.blocks) { - const plugged = Object.fromEntries( - KINDS[kind].ports.map(([port]) => [ - port, - reference( - 'Signal', - rows.map((row) => this.plugs.get(`${kind} ${row.id} ${port}`) ?? null), - ), - ]), - ); - const grouped = kind === 'Process' && this.groups.length; - patch.set( - kind, - rows.map((row) => row.id), - { - name: rows.map((row) => row.name), - x: rows.map((row) => row.at?.[0] ?? null), - y: rows.map((row) => row.at?.[1] ?? null), - status: rows.map((row) => (this.alarms.has(row.id) ? 1 : 0)), - ...plugged, - ...(grouped - ? { - group: reference( - 'Group', - rows.map((row) => row.group ?? null), - ), - } - : {}), - }, - ); - } - return new Data('studio').apply(patch); - } -} - -/** The presets: a water plant laid out in ranks, then four drawn where they stand. */ -const PRESETS: readonly (readonly [string, () => Sketch])[] = [ - [ - 'Water plant', - () => { - const s = new Sketch(); - const [intake, treatment, output] = ['Intake', 'Treatment', 'Output'].map((name) => - s.group(name), - ); - const wellA = s.block('Input', 'Well A'); - const wellB = s.block('Input', 'Well B'); - const screen = s.block('Process', 'Screen', undefined, intake); - const dosing = s.block('Input', 'Dosing'); - const mixer = s.block('Process', 'Mixer', undefined, treatment); - const clarifier = s.block('Process', 'Clarifier', undefined, treatment); - const filters = [1, 2].map((n) => s.block('Process', `Filter ${n}`, undefined, treatment)); - const tank = s.block('Process', 'Tank', undefined, output); - const pump = s.block('Process', 'Pump', undefined, output); - const main = s.block('Output', 'Main'); - s.wire('Raw water', [wellA, wellB], screen); - s.wire('To mixer', [screen, dosing], mixer); - s.wire('Mixed', mixer, clarifier); - s.wire('Clarified', clarifier, ...filters); - s.wire('Filtered', filters, tank); - s.wire('Stored', tank, pump); - s.wire('Pumped', pump, main); - return s; - }, - ], - [ - 'Control loop', - () => { - const s = new Sketch(); - const reference = s.block('Input', 'Reference', [0, 0]); - const controller = s.block('Control', 'Controller', [240, 0]); - const actuator = s.block('Process', 'Actuator', [500, 0]); - const plant = s.block('Process', 'Plant', [500, 240]); - const sensor = s.block('Process', 'Sensor', [240, 240]); - const response = s.block('Output', 'Response', [740, 240]); - s.wire('Set point', reference, controller); - s.wire('Command', controller, actuator); - s.wire('Drive', actuator, plant); - s.wire('Measured', plant, response, sensor); - s.wire('Feedback', sensor, [controller, 'feedback']); - s.alarms.add(plant.id as number); - return s; - }, - ], - [ - 'Nested groups', - () => { - const s = new Sketch(); - const station = s.station('Generating station'); - const dispatch = s.block('Input', 'Dispatch', [0, 210]); - const plants: Plug[] = []; - const controls: Plug[] = []; - for (let i = 0; i < 3; i++) { - const unit = s.group(`Unit ${i + 1}`, station); - controls.push(s.block('Process', `Controller ${i + 1}`, [320, i * 210], unit)); - plants.push(s.block('Process', `Plant ${i + 1}`, [620, i * 210], unit)); - s.wire(`Drive ${i + 1}`, controls[i]!, plants[i]!); - } - const meter = s.block('Output', 'Metering', [950, 210]); - s.wire('Dispatch', dispatch, ...controls); - // The meter reads one signal, so the plants share it. - s.wire('Power', plants, meter); - return s; - }, - ], - [ - 'Shape atlas', - () => { - const s = new Sketch(); - const input = s.block('Input', 'Input', [0, 0]); - const process = s.block('Process', 'Process', [300, 0]); - const decision = s.block('Control', 'Decision', [300, 250]); - const result = s.block('Output', 'Result', [640, 250]); - s.wire('Source', input, process); - s.wire('Evaluate', process, decision); - s.wire('Accept', decision, result); - return s; - }, - ], - [ - 'Scale study: 1,024 blocks', - () => { - const s = new Sketch(); - let before: Plug | undefined; - for (let row = 0; row < 32; row++) { - for (let col = 0; col < 32; col++) { - const block = s.block('Process', `Block ${row * 32 + col + 1}`, [col * 240, row * 150]); - if (col && before) s.wire('', before, block); - before = block; - } - } - return s; - }, - ], -]; - -/** - * Earlier versions of the data, to step back to and forward again: snapshots, each held here or - * live, never both, and freed once forgotten. - */ -class History { - readonly #undo: Data[] = []; - #redo: Data[] = []; - - /** Remembers `before`, a snapshot of the version an edit started from. */ - commit(before: Data): void { - this.#undo.push(before); - if (this.#undo.length > 100) this.#undo.shift()!.destroy(); - for (const after of this.#redo) after.destroy(); - this.#redo = []; - } - - /** The version before `current`, which redo then steps forward to. */ - undo(current: Data): Data | undefined { - const before = this.#undo.pop(); - if (before) this.#redo.push(current); - return before; - } - - redo(current: Data): Data | undefined { - const after = this.#redo.pop(); - if (after) this.#undo.push(current); - return after; - } - - clear(): void { - for (const version of [...this.#undo, ...this.#redo]) version.destroy(); - this.#undo.length = 0; - this.#redo = []; - } -} - -export default function page(stage: Stage): void { - let preset = PRESETS[0]!; - let sketch = preset[1](); - let data = sketch.data(); - let layout: Layout = { layered: { flow: 'right', gap: 30, rankGap: 80 } }; - let route: 'orthogonal' | 'straight' = 'orthogonal'; - let comets = false; - let dots = true; - let light = false; - const history = new History(); - - const scene = (): Scene => { - const marks: Mark[] = []; - // Containers take the theme's surface, tinted by how deep they sit; their titles head them, and - // they and the blocks they hold lift off the sheet. - const title = { text: 'name', size: 11, color: light ? '#5b6b7d' : '#7f93a8' }; - if (sketch.stations.length) { - marks.push({ - vertices: { - of: ['studio', 'Station'], - contains: ['Group', 'station'], - label: title, - shadow: true, - }, - }); - } - if (sketch.groups.length) { - marks.push({ - vertices: { - of: ['studio', 'Group'], - contains: ['Process', 'group'], - label: title, - shadow: true, - }, - }); - } - // Comets run from each signal's driver to its readers. - marks.push({ - edges: { - of: ['studio', 'Signal'], - route, - arrows: true, - width: 1.5, - ...(comets ? { flow: 40 } : {}), - }, - }); - // A mark for each kind of block the data holds; a table the data lacks would fail. - for (const kind of kinds.filter((k) => data.tables.includes(k))) { - // Without a size, a block fits its label and ports. - marks.push({ - vertices: { - of: ['studio', kind], - position: ['x', 'y'], - marker: KINDS[kind].marker, - color: KINDS[kind].fill[light ? 1 : 0], - shadow: true, - ring: when(field('status').gt(0)).then(color('#ff8c42')).otherwise(missing()), - ports: ports(kind), - drag: true, - label: { text: 'name', size: 12 }, - }, - }); - } - return { - space: { diagram: { grid: dots ? { dots: { pitch: 20 } } : 'off' } }, - layout, - theme: light ? { base: 'light', background: '#f4f6f9' } : theme, - controls: { connect: true, delete: true, selectBox: true }, - marks, - }; - }; - const view = stage.view(scene(), { - label: 'A diagram: drag blocks, drag from a port to wire it, Delete removes', - }); - - /** Shows the current version: `next`, or the data as it is now, and a mark for each kind it holds. */ - let kindsShown = data.tables.join(); - const show = (next = data) => { - data = next; - view.setData(data); - if (data.tables.join() !== kindsShown) { - kindsShown = data.tables.join(); - view.show(scene()); - } - describe(); - }; - /** Applies `patch` as one edit that undo steps back over. */ - const edit = (patch: Patch, said: string) => { - history.commit(data.snapshot()); - show(data.apply(patch)); - note(said); - }; - const load = (choice: (typeof PRESETS)[number]) => { - preset = choice; - sketch = choice[1](); - history.clear(); - view.show(scene()); - data.destroy(); - show(sketch.data()); - view.fit(); - note('Drag blocks or connect ports.'); - const blocks = kinds.reduce((n, kind) => n + (sketch.blocks.get(kind)?.length ?? 0), 0); - stage.detail( - `${blocks.toLocaleString()} blocks · ${sketch.signals.length.toLocaleString()} signals`, - ); - }; - - const name = (row: Row) => - (data.get(row.table, row.id, 'name') as string | null) ?? `${row.table} ${row.id}`; - const isBlock = (table: string): table is Kind => table in KINDS; - - // Moving: a block goes where the drag says, moves at once, and lands as one edit. - let before: Data | undefined; - view.on('drag', ({ phase, hit, position }) => { - if (!isBlock(hit.row.table)) return; - if (phase === 'start') { - before = data.snapshot(); - return; - } - const at = new Patch().set(hit.row.table, [hit.row.id as number], { - x: [position[0]], - y: [position[1]], - }); - if (phase === 'end') { - if (before) history.commit(before); - before = undefined; - show(data.apply(at)); - note(`Moved ${name(hit.row)}.`); - } else show(data.apply(at)); - }); - - // Wiring: an output and an input join on the output's signal, made when it has none; an input - // wired elsewhere leaves the signal it was on. A plugged port let go over nothing is unplugged. - view.on('connect', ({ from, to }) => { - if (!to) { - const plugged = - isBlock(from.row.table) && from.port && data.get(from.row.table, from.row.id, from.port); - if (!plugged) return note('Let go over nothing.'); - const patch = new Patch().set(from.row.table, [from.row.id as number], { - [from.port!]: reference('Signal', [null]), - }); - return edit(patch, `Unplugged ${name(from.row)}'s ${from.port}.`); - } - if (!isBlock(from.row.table) || !isBlock(to.row.table)) return note('Wire blocks to blocks.'); - const a: Plug = { kind: from.row.table, id: from.row.id, port: from.port ?? 'out' }; - if (!direction(a.kind, a.port)) a.port = KINDS[a.kind].ports[0]![0]; - const wants = direction(a.kind, a.port) === 'out' ? 'in' : 'out'; - const port = to.port ?? KINDS[to.row.table].ports.find(([, , d]) => d === wants)?.[0]; - if (!port) return note(`${name(to.row)} has no ${wants === 'in' ? 'input' : 'output'}.`); - const b: Plug = { kind: to.row.table, id: to.row.id, port }; - if (direction(b.kind, b.port) !== wants) return note('Wire an output to an input.'); - const [driver, reader] = wants === 'in' ? [a, b] : [b, a]; - const plugged = data.get(driver.kind, driver.id, driver.port) as Row | null; - const patch = new Patch(); - let signal = plugged?.id; - if (signal === undefined) { - signal = sketch.next++; - patch.set('Signal', [signal], { name: [`Signal ${signal}`] }); - patch.set(driver.kind, [driver.id as number], { - [driver.port]: reference('Signal', [signal as number]), - }); - } - patch.set(reader.kind, [reader.id as number], { - [reader.port]: reference('Signal', [signal as number]), - }); - edit(patch, `Wired ${name(from.row)} and ${name(to.row)}.`); - }); - - /** Removes `rows` as one edit. */ - const remove = (rows: readonly Rows[]) => { - const patch = new Patch(); - for (const { table, ids } of rows) patch.remove(table, ids); - edit(patch, `Removed ${each(rows).map(name).join(', ')}.`); - }; - view.on('delete', ({ rows }) => remove(rows)); - - /** Selects `row` and what it joins, and frames them. */ - const reveal = (row: Row) => { - const joined = view.neighbors(row); - const near = [one(row), ...joined]; - view.select(near); - view.fit({ rows: near }); - describe(); - note(`${name(row)} and ${each(joined).length} joined to it.`); - }; - /** Selects `row` and puts its name up for editing. */ - const rename = (row: Row) => { - view.select([one(row)]); - describe(); - input.focus(); - input.select(); - }; - const named = (row: Row) => - isBlock(row.table) || row.table === 'Group' || row.table === 'Station'; - - // Opening a row renames it; the menu names what can be done with the row under it. - view.on('open', ({ hit }) => named(hit.row) && rename(hit.row)); - const menu = stage.menu(); - view.on('menu', ({ hits, trigger }) => { - const row = hits[0]?.row; - if (row && !view.selection.some((s) => s.table === row.table && s.ids.includes(row.id))) - view.select([one(row)]); - describe(); - const rows = view.selection; - const count = each(rows).length; - menu.open(view, trigger.point, [ - { label: 'Rename', run: () => row && rename(row), disabled: !row || !named(row) }, - { label: 'Reveal neighbors', run: () => row && reveal(row), disabled: !row }, - { - label: count > 1 ? `Delete ${count} rows` : 'Delete', - run: () => remove(rows), - disabled: !count, - }, - { label: 'Fit everything', run: () => view.fit() }, - ]); - }); - - const studio = stage.section('Diagram'); - studio.choice( - 'Preset', - PRESETS.map((p) => [p[0], p] as const), - preset, - load, - ); - studio.actions([ - { - label: 'Undo', - key: 'z', - run: () => { - const before = history.undo(data); - if (!before) return; - show(before); - note('Undone.'); - }, - }, - { - label: 'Redo', - key: 'y', - run: () => { - const after = history.redo(data); - if (!after) return; - show(after); - note('Redone.'); - }, - }, - { - label: 'Arrange', - key: 'a', - // Every block lets go of its position, the layout places them all again, and the places it - // answers become the blocks' positions: one edit undo steps back over. - run: () => { - const patch = new Patch(); - for (const kind of kinds) { - const ids = sketch.blocks.get(kind)?.map((b) => b.id) ?? []; - const all = ids.filter((id) => data.get(kind, id, 'name') !== null); - if (all.length) patch.set(kind, all, { x: all.map(() => null), y: all.map(() => null) }); - } - history.commit(data.snapshot()); - show(data.apply(patch)); - view.fit(); - void view.arrange().then((patches) => { - for (const placed of patches) if (placed.data === data.name) show(data.apply(placed)); - note('Arranged.'); - }); - }, - }, - { label: 'Fit', key: 'f', run: () => view.fit() }, - { label: 'Zoom in', key: '=', run: () => zoom(0.5) }, - { label: 'Zoom out', key: '-', run: () => zoom(-0.5) }, - ]); - /** Eases the camera in by `by` steps of doubling. */ - const zoom = (by: number) => { - const camera = view.camera; - if ('center' in camera) view.fly({ ...camera, zoom: camera.zoom + by }); - }; - const add = stage.section('Add'); - add.actions( - kinds.map((kind, i) => ({ - label: kind, - key: String(i + 1), - run: () => { - // New blocks land where the camera looks; nothing else moves. - const camera = view.camera; - const [x, y] = 'center' in camera ? camera.center : [0, 0]; - const id = sketch.next++; - sketch.blocks.set(kind, [ - ...(sketch.blocks.get(kind) ?? []), - { id, name: `New ${kind.toLowerCase()}` }, - ]); - const plugged = Object.fromEntries( - KINDS[kind].ports.map(([port]) => [port, reference('Signal', [null])]), - ); - edit( - new Patch().set(kind, [id], { - name: [`New ${kind.toLowerCase()}`], - x: [x], - y: [y], - status: [0], - ...plugged, - }), - `Added a ${kind.toLowerCase()} block.`, - ); - }, - })), - ); - - const look = stage.section('Display'); - const layouts: readonly (readonly [string, Layout])[] = [ - ['Left to right', layout], - ['Top to bottom', { layered: { flow: 'down', gap: 30, rankGap: 60 } }], - ['Right to left', { layered: { flow: 'left', gap: 30, rankGap: 80 } }], - ['Bottom to top', { layered: { flow: 'up', gap: 30, rankGap: 60 } }], - ['Force layout', { stress: { gap: 140 } }], - ]; - look.choice('Layout', layouts, layout, (next) => { - layout = next; - view.show(scene()); - view.fit(); - }); - look.choice( - 'Routing', - [ - ['Around blocks', 'orthogonal'], - ['Straight', 'straight'], - ] as const, - route, - (next) => { - route = next; - view.show(scene()); - }, - ); - look.toggle('Flow', 'c', comets, (on) => { - comets = on; - view.show(scene()); - }); - look.toggle('Grid', 'g', dots, (on) => { - dots = on; - view.show(scene()); - }); - look.toggle('Light', 'l', light, (on) => { - light = on; - view.show(scene()); - }); - - // The selected block's name, editable. - const selected = stage.section('Selection'); - const input = selected.text('Name', (value) => { - const [row] = each(view.selection); - if (row) - edit( - new Patch().set(row.table, [row.id as number], { name: [value] }), - `Renamed to ${value}.`, - ); - }); - input.placeholder = 'Select a block'; - input.disabled = true; - const told = selected.note(); - const note = (text: string) => (told.textContent = text); - const describe = () => { - const [row] = each(view.selection); - const editable = !!row && named(row); - input.disabled = !editable; - input.value = editable ? name(row) : ''; - }; - view.on('select', describe); - - stage.meter(view); - load(preset); -} diff --git a/npm/examples/src/examples/index.ts b/npm/examples/src/examples/index.ts deleted file mode 100644 index c10dd8a0..00000000 --- a/npm/examples/src/examples/index.ts +++ /dev/null @@ -1,13 +0,0 @@ -import type { Page } from '../ui/shell.js'; - -/** Names and modules for the example picker. IDs keep existing links working. */ -export const examples: readonly Page[] = [ - { id: 'bunny', title: 'Soft body', load: () => import('./soft-body/index.js') }, - { id: 'grid', title: 'Playback', load: () => import('./playback/index.js') }, - { id: 'network', title: 'Networks', load: () => import('./networks/index.js') }, - { id: 'operations', title: 'Power flow', load: () => import('./power-flow/index.js') }, - { id: 'telemetry', title: 'Live data', load: () => import('./live-data/index.js') }, - { id: 'signals', title: 'Linked views', load: () => import('./linked-views/index.js') }, - { id: 'diagram', title: 'Diagram', load: () => import('./diagram/index.js') }, - { id: 'colors', title: 'Colormaps', load: () => import('./colormaps/index.js') }, -]; diff --git a/npm/examples/src/examples/linked-views/index.ts b/npm/examples/src/examples/linked-views/index.ts deleted file mode 100644 index dc8b4efe..00000000 --- a/npm/examples/src/examples/linked-views/index.ts +++ /dev/null @@ -1,155 +0,0 @@ -import { theme } from '../../ui/theme.js'; -/** - * One record in two views on one canvas: a lattice of nodes on a tilted plane, each standing on a - * pole as high as its signal, and the signals traced on a plot. One playhead moves both; play at a - * speed or scrub, and read where each view stands. - */ -import { Data, Patch, field, reference, type Scene, type View } from '../../latkit.js'; -import type { Stage } from '../../ui/stage.js'; - -/** Nodes a side of the lattice, frames recorded, seconds between them, and a full signal's height. */ -const SIDE = 20; -const FRAMES = 1201; -const STEP = 0.05; -const HEIGHT = 3; -/** Every `TRACED`th node, from the middle of the first run, is traced when few are. */ -const TRACED = 25; - -/** - * The record: nodes at whole x and y, linked to their neighbors right and down, each a signal - * rippling across the lattice, `0.5 + 0.45 sin(0.37 x + 1.3 t) cos(0.31 y - 0.7 t)`. - */ -function record(): Data { - const n = SIDE * SIDE; - const x = Float64Array.from({ length: n }, (_, i) => i % SIDE); - const y = Float64Array.from({ length: n }, (_, i) => Math.floor(i / SIDE)); - const traced = Float64Array.from({ length: n }, (_, i) => Number(i % TRACED === 10)); - const [from, to]: [number[], number[]] = [[], []]; - const link = (a: number, b: number) => { - from.push(a); - to.push(b); - }; - for (let i = 0; i < n; i++) { - if (x[i]! < SIDE - 1) link(i, i + 1); - if (y[i]! < SIDE - 1) link(i, i + SIDE); - } - const patch = new Patch() - .axis('time', 's') - .set('Node', { count: n }, { x, y, traced }) - .set( - 'Link', - { count: from.length }, - { from: reference('Node', from), to: reference('Node', to) }, - ); - // Every frame in one append, frame by frame. - const times = Array.from({ length: FRAMES }, (_, f) => f * STEP); - const signal = new Float32Array(FRAMES * n); - times.forEach((t, f) => { - for (let i = 0; i < n; i++) - signal[f * n + i] = - 0.5 + 0.45 * Math.sin(0.37 * x[i]! + 1.3 * t) * Math.cos(0.31 * y[i]! - 0.7 * t); - }); - return new Data('signals').apply(patch.append(times, 'Node', { signal })); -} - -export default function page(stage: Stage): void { - const duration = (FRAMES - 1) * STEP; - let few = true; - const signal = field('signal').colormap('viridis').domain(0, 1); - const lattice: Scene = { - space: { plane: { grid: 'off' } }, - theme, - marks: [ - { edges: { of: ['signals', 'Link'], ends: ['from', 'to'], width: 1, color: '#3a4556' } }, - { edges: { of: ['signals', 'Node'], poles: true, width: 1.5, color: signal } }, - { - vertices: { - of: ['signals', 'Node'], - position: ['x', 'y'], - elevation: field('signal').mul(HEIGHT), - radius: 3.4, - color: signal, - }, - }, - ], - }; - const traces = (): Scene => ({ - space: { - plot: { x: { label: 'Time (s)' }, y: { label: 'Signal', format: 'fixed', precision: 1 } }, - }, - theme, - ease: 0, - marks: [ - { - lines: { - of: ['signals', 'Node'], - trace: 'signal', - color: signal, - width: few ? 1.5 : 0.8, - ...(few ? { visible: field('traced').gt(0.5) } : {}), - }, - }, - ], - }); - const [plane, plot] = stage.panels([lattice, traces()], { - label: 'A lattice on poles beside its signals traced: one playhead moves both', - panels: [ - [0, 0, 0.55, 1], - [0.55, 0, 0.45, 1], - ], - }) as [View, View]; - const data = record(); - plane.setData(data); - plot.setData(data); - plane.camera = { center: [(SIDE - 1) / 2, (SIDE - 1) / 2, 0], zoom: 3, pitch: 50, bearing: 20 }; - plane.fit(); - plot.camera = { x: [0, duration], y: [0, 1] }; - - const views = [ - ['Lattice', plane], - ['Plot', plot], - ] as const; - const playback = stage.playback({ - live: false, - show(at) { - for (const [, view] of views) view.at = at; - }, - }); - playback.record([0, duration]); - - // Where each view stands: both draw in the canvas's one frame, so they never part. - const where = stage.section('View stats', false); - const lines = views.map(() => where.note()); - let read = 0; - stage.frame(() => { - if (performance.now() - read < 250) return; - read = performance.now(); - views.forEach(([name, view], i) => { - const { frames, cpuMs } = view.stats; - lines[i]!.textContent = - `${name} at ${view.at.toFixed(3)} s · ${frames} frames · ${cpuMs.toFixed(1)} ms`; - }); - }); - - const play = stage.section('Playback'); - play.speed([0.25, 1, 4], 1, (rate) => (playback.rate = rate)); - play.choice( - 'Traces', - [ - [`${(SIDE * SIDE) / TRACED} nodes`, true], - [`All ${SIDE * SIDE}`, false], - ] as const, - few, - (next) => { - few = next; - plot.show(traces()); - }, - ); - const legend = stage.legend('top'); - legend.ramp('viridis', 'Signal 0–1'); - - stage.meter(plane); - stage.detail( - `${(SIDE * SIDE).toLocaleString()} nodes · ${FRAMES.toLocaleString()} frames over ${duration} s`, - ); -} diff --git a/npm/examples/src/examples/live-data/index.ts b/npm/examples/src/examples/live-data/index.ts deleted file mode 100644 index caf7e74e..00000000 --- a/npm/examples/src/examples/live-data/index.ts +++ /dev/null @@ -1,428 +0,0 @@ -import { random } from '../../random.js'; -import { theme } from '../../ui/theme.js'; -import { - Data, - Patch, - colormapCss, - field, - type ColormapName, - type Hit, - type Id, - type Scene, -} from '../../latkit.js'; -import { h } from '../../ui/dom.js'; -import { each } from '../../ui/rows.js'; -import type { Stage } from '../../ui/stage.js'; - -/** The areas signals belong to, in the colors tab10 gives them. */ -const AREAS = [ - ['North', '#1f77b4'], - ['South', '#ff7f0e'], - ['East', '#2ca02c'], - ['West', '#d62728'], - ['Coast', '#9467bd'], - ['Valley', '#8c564b'], -] as const; -/** Seconds of history kept; older frames are forgotten. */ -const HISTORY = 120; -/** Seconds between disturbances, and how long one lasts: long enough to have died away. */ -const EVERY = 25; -const LASTS = 6; -/** Signals the hot list names, and how often it reads them again. */ -const HOT = 12; -const HOT_EVERY = 250; - -/** - * What the signals measure: each its unit, the range it keeps to, the digits it reads to, and - * whether its low values are the ones to watch. - */ -const QUANTITIES = { - voltage: { label: 'Voltage', unit: 'pu', range: [0.92, 1.04], digits: 4, low: true }, - frequency: { label: 'Frequency', unit: 'Hz', range: [59.8, 60.06], digits: 3, low: true }, - loading: { label: 'Loading', unit: '%', range: [0, 130], digits: 1, low: false }, - temperature: { label: 'Temperature', unit: '°C', range: [20, 110], digits: 1, low: false }, -} as const; -type Quantity = keyof typeof QUANTITIES; -/** The colormaps a signal's value now may take. */ -const MAPS: readonly ColormapName[] = ['coolwarm', 'viridis', 'turbo', 'magma', 'icefire']; - -/** - * Synthetic SCADA signals: each sits at its own offset, its area swings slowly, every signal - * carries its own noise, and every `EVERY` seconds one area is disturbed and recovers, as is an - * area struck by hand. A disturbance pulls voltages and the system's frequency down, and pushes - * loading and temperature up. - */ -class Signals { - readonly area: Float64Array; - readonly name: string[] = []; - readonly #offset: Float32Array; - readonly #noise: Float32Array; - readonly #random = random(11); - /** The area struck by hand, and when. */ - #struck?: { area: number; at: number }; - - constructor( - readonly count: number, - readonly quantity: Quantity, - ) { - this.area = new Float64Array(count); - this.#offset = new Float32Array(count); - this.#noise = new Float32Array(count); - for (let i = 0; i < count; i++) { - this.area[i] = i % AREAS.length; - this.#offset[i] = this.#random() - 0.5; - this.#noise[i] = 0.3 + this.#random(); - this.name.push(`${AREAS[i % AREAS.length]![0]} ${Math.floor(i / AREAS.length) + 1}`); - } - } - - /** Strikes area `area` at `t`: it is disturbed and recovers as the timed disturbances do. */ - strike(area: number, t: number): void { - this.#struck = { area, at: t }; - } - - /** Every signal's value at `t`, into `out`. */ - sample(t: number, out: Float32Array): void { - const struck = Math.floor(t / EVERY) % AREAS.length; - const dip = depth(t - Math.floor(t / EVERY) * EVERY); - const byHand = this.#struck && t >= this.#struck.at ? depth(t - this.#struck.at) : 0; - // The system's frequency falls with any disturbance, wherever it strikes. - const system = 60 + 0.01 * Math.sin(t * 0.21) - 2 * Math.max(dip, byHand); - for (let i = 0; i < this.count; i++) { - const a = this.area[i]!; - const swing = Math.sin(t * (0.6 + a * 0.13) + a); - const noise = (this.#random() - 0.5) * this.#noise[i]!; - const dips = (a === struck ? dip : 0) + (a === this.#struck?.area ? byHand : 0); - const offset = this.#offset[i]!; - out[i] = - this.quantity === 'voltage' - ? 1 + 0.03 * offset + 0.012 * swing + 0.003 * noise - dips - : this.quantity === 'frequency' - ? system + 0.004 * swing + 0.001 * noise - : this.quantity === 'loading' - ? 60 + 25 * offset + 15 * swing + 2 * noise + 600 * dips - : 45 + 20 * offset + 8 * swing + 0.5 * noise + 400 * dips; - } - } -} - -/** How far a disturbance has pulled values, `since` seconds after it struck. */ -function depth(since: number): number { - return since < LASTS - ? 0.06 * Math.exp(-since * 1.4) * Math.sin(Math.min(since, 0.5) * Math.PI) - : 0; -} - -export default function page(stage: Stage): void { - let count = 2000; - let quantity: Quantity = 'voltage'; - let window = 20; - let rate = 50; - let steps = false; - let auto = true; - let colorBy: 'area' | 'value' = 'area'; - let colormap: ColormapName = 'coolwarm'; - let width = 1; - let signals = new Signals(count, quantity); - let values = new Float32Array(count); - let data = new Data('scada'); - let t = 0; - let carry = 0; - const shown = () => QUANTITIES[quantity]; - - const scene = (): Scene => { - const { label, unit, range, digits } = shown(); - return { - space: { - plot: { - x: { label: 'Time (s)' }, - y: { label: `${label} (${unit})`, format: 'fixed', precision: digits - 1 }, - }, - }, - theme: { ...theme, dim: 0.15 }, - ease: 0, - marks: [ - { - lines: { - of: ['scada', 'Signal'], - trace: 'value', - // By value, each trace takes the color of its value at the time shown. - color: - colorBy === 'area' - ? field('area').colormap('tab10') - : field('value').colormap(colormap).domain(range[0], range[1]), - width, - interpolation: steps ? 'stepAfter' : 'linear', - }, - }, - ], - }; - }; - const view = stage.view(scene(), { - label: 'Live signals: hover a trace to read it, click it to pick it out', - }); - - /** Frames what `at` shows: the window before it, or all that is kept. */ - const frame = (at: number) => { - const start = window === Infinity ? (data.span?.[0] ?? at) : at - window; - const range = shown().range; - view.camera = { x: [start, at], y: auto ? null : [range[0], range[1]] }; - }; - - /** Starts the record again: `n` signals measuring the quantity shown. */ - const restart = (n = count) => { - count = n; - signals = new Signals(n, quantity); - values = new Float32Array(n); - t = 0; - carry = 0; - // The record starts with its first frame, so the value field is there from the start. - signals.sample(0, values); - data.destroy(); - data = new Data('scada').apply( - new Patch() - .axis('time', 's') - .set('Signal', { count: n }, { name: signals.name, area: signals.area }) - .append(0, 'Signal', { value: values }), - ); - view.select([]); - view.setData(data); - frame(0); - detail(); - describe(null); - hot.replaceChildren(); - }; - - const playback = stage.playback({ - replay: { seconds: 10, rate: 0.25 }, - show(at) { - view.at = at; - frame(at); - }, - }); - - const meter = stage.meter(view); - meter.note(() => `${((count * rate) / 1000).toFixed(0)}k samples/s`); - let listed = 0; - stage.frame((dt) => { - if (performance.now() - listed > HOT_EVERY) list(); - if (playback.mode !== 'live') return; - meter.measure('append', () => { - carry += dt * rate; - const frames = Math.floor(carry); - if (!frames) return; - carry -= frames; - const patch = new Patch(); - for (let k = 0; k < frames; k++) { - t += 1 / rate; - signals.sample(t, values); - patch.append(t, 'Signal', { value: values }); - } - data.apply(patch.forget(t - HISTORY)); - view.setData(data); - playback.record(data.span!); - }); - }); - - const readout = stage.section('Reading'); - const reading = h('p', { class: 'series' }); - readout.content.append(reading); - const name = (id: Id) => (data.get('Signal', id, 'name') as string | null) ?? `Signal ${id}`; - const format = (value: number) => { - const { unit, digits } = shown(); - return `${value.toFixed(digits)} ${unit}`; - }; - const describe = (hit: Hit | null) => { - const picked = each(view.selection) - .map((row) => name(row.id)) - .join(', '); - const sample = hit?.sample; - reading.textContent = sample - ? `${name(hit.row.id)} at ${sample.at.toFixed(2)} s: ${format(sample.value)}` - : picked - ? `Picked ${picked}.` - : 'Hover a trace.'; - }; - view.on('hover', ({ hit }) => describe(hit)); - view.on('select', () => { - describe(null); - list(); - }); - - // The signals furthest out at the time shown, read by a query: a pick, of every signal, in - // linear time. - const hottest = stage.section('Largest changes', false); - const hot = h('ol', { class: 'hot' }); - hottest.content.append(hot); - const list = () => { - listed = performance.now(); - if (!data.span) return; - const { low, range } = shown(); - const rows = data.query({ - of: 'Signal', - sort: [['value', low ? 'ascending' : 'descending']], - limit: HOT, - select: { name: 'name', value: 'value' }, - at: view.at, - }); - const names = rows.fields.name as readonly string[]; - const read = rows.fields.value as Float64Array; - const picked = new Set(each(view.selection).map((row) => String(row.id))); - hot.replaceChildren( - ...rows.ids.map((id, k) => { - const value = read[k]!; - const span = (value - range[0]) / (range[1] - range[0]); - const score = Math.min(1, Math.max(0, low ? 1 - span : span)); - const button = h( - 'button', - { type: 'button', 'aria-pressed': String(picked.has(String(id))) }, - h('span', {}, String(k + 1).padStart(2, '0')), - h('span', {}, names[k] ?? `Signal ${id}`), - h('span', {}, format(value)), - h('span', { class: 'bar', style: `--score: ${(score * 100).toFixed(0)}%` }), - ); - button.addEventListener('click', () => { - view.select([{ data: 'scada', table: 'Signal', ids: [id] }]); - describe(null); - list(); - }); - return h('li', {}, button); - }), - ); - }; - - const record = stage.section('Data'); - record.note('Simulated data'); - record.choice( - 'Measure', - Object.entries(QUANTITIES).map( - ([key, q]) => [`${q.label} (${q.unit})`, key as Quantity] as const, - ), - quantity, - (next) => { - quantity = next; - restyle(); - restart(); - }, - ); - record.choice( - 'Signals', - [ - ['500', 500], - ['2,000', 2000], - ['5,000', 5000], - ], - count, - restart, - ); - record.choice( - 'Samples', - [ - ['20 per second', 20], - ['50 per second', 50], - ['100 per second', 100], - ], - rate, - (r) => (rate = r), - ); - record.actions([{ label: 'Reset', key: 'n', run: () => restart() }]); - - const show = stage.section('Display'); - show.choice( - 'Window', - [ - ['5 s', 5], - ['20 s', 20], - ['60 s', 60], - [`All ${HISTORY} s kept`, Infinity], - ], - window, - (w) => { - window = w; - frame(view.at); - }, - ); - show.choice( - 'Color by', - [ - ['Area', 'area'], - ['Value now', 'value'], - ] as const, - colorBy, - (next) => { - colorBy = next; - restyle(); - }, - ); - show.range( - 'Width', - { min: 0.5, max: 3, step: 0.5, value: width, format: (v) => `${v} px` }, - (next) => { - if (next === width) return; - width = next; - view.show(scene()); - }, - ); - show.toggle('Steps', 's', steps, (on) => { - steps = on; - view.show(scene()); - }); - show.toggle('Auto range', 'a', auto, (on) => { - auto = on; - frame(view.at); - }); - const areas = show.series(AREAS.map(([label, color]) => ({ label, color }))); - const maps = h('div', { class: 'swatches' }); - const swatches = MAPS.map((map) => { - const swatch = h( - 'button', - { type: 'button', class: 'swatch', 'aria-pressed': String(map === colormap) }, - h('span', { class: 'ramp', style: `background: ${colormapCss(map)}` }), - map, - ); - swatch.addEventListener('click', () => { - colormap = map; - restyle(); - }); - maps.append(swatch); - return [map, swatch] as const; - }); - show.content.append(maps); - const legend = stage.legend('top'); - const restyle = () => { - view.show(scene()); - const { label, unit, range } = shown(); - areas.hidden = colorBy !== 'area'; - maps.hidden = legend.hidden = colorBy !== 'value'; - legend.ramp(colormap, `${label} now, ${range[0]} – ${range[1]} ${unit}`); - for (const [map, swatch] of swatches) - swatch.setAttribute('aria-pressed', String(map === colormap)); - }; - - const disturb = stage.section('Disturbance'); - let target = 0; - disturb.choice( - 'Area', - AREAS.map(([label], i) => [label, i] as const), - target, - (next) => (target = next), - ); - disturb.actions([ - { - label: 'Strike it now', - key: 'd', - run: () => { - signals.strike(target, t); - if (playback.mode !== 'live') playback.play(); - }, - }, - ]); - disturb.note(`Auto disturbance: every ${EVERY} s`); - - const detail = () => - stage.detail( - `${count.toLocaleString()} signals · ${shown().label.toLowerCase()} · ${HISTORY} s kept`, - ); - restyle(); - restart(); -} diff --git a/npm/examples/src/examples/networks/cases.test.ts b/npm/examples/src/examples/networks/cases.test.ts deleted file mode 100644 index 4a10ac2d..00000000 --- a/npm/examples/src/examples/networks/cases.test.ts +++ /dev/null @@ -1,46 +0,0 @@ -import { readFileSync } from 'node:fs'; -import { gunzipSync } from 'node:zlib'; -import { describe, expect, test } from 'vitest'; -import { readCase } from './cases.js'; - -const CASES = ['wecc240', 'activsg2000', 'activsg10k', 'activsg25k', 'activsg70k'] as const; - -/** Case `id` as the page reads it, from the file it fetches. */ -function read(id: string) { - const raw = readFileSync(`npm/examples/public/grids/${id}.bin`); - const bytes = raw[0] === 0x1f && raw[1] === 0x8b ? gunzipSync(raw) : raw; - return readCase(id, bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength)); -} - -describe('grid cases', () => { - test.each(CASES)('%s reads into buses, lines, and machines that stand on the earth', (id) => { - const found = read(id); - const rows = Object.fromEntries( - found.data.describe().tables.map((table) => [table.name, table.rows]), - ); - expect(rows).toMatchObject({ Bus: found.buses, Line: found.lines, Machine: found.machines }); - expect(found.buses).toBeGreaterThan(200); - for (let i = 0; i < found.positions.length; i += 2) { - expect(Math.abs(found.positions[i]!)).toBeLessThanOrEqual(180); - expect(Math.abs(found.positions[i + 1]!)).toBeLessThanOrEqual(90); - } - // Every line's ends name buses of the case. - const lines = found.data.query({ of: 'Line', select: { from: 'from', to: 'to' } }); - const ends = lines.fields as Record; - expect(ends.from!.ids.every((id) => id !== null)).toBe(true); - expect(ends.to!.ids.every((id) => id !== null)).toBe(true); - expect(found.machineBus.every((bus) => bus < found.buses)).toBe(true); - }); - - test('a simulated case faults at a second, and its machines swing after', () => { - const { simulation, machines } = read('wecc240'); - expect(simulation?.fault?.on).toBeCloseTo(1, 1); - const frame = Math.round(3 / simulation!.dt); - const speeds = simulation!.speeds.subarray(frame * machines, (frame + 1) * machines); - expect(speeds.some((mhz) => Math.abs(mhz) > 0)).toBe(true); - }); - - test('bytes that are no case are refused', () => { - expect(() => readCase('nothing', new ArrayBuffer(16))).toThrow('not a grid case'); - }); -}); diff --git a/npm/examples/src/examples/networks/cases.ts b/npm/examples/src/examples/networks/cases.ts deleted file mode 100644 index 1c6aea4c..00000000 --- a/npm/examples/src/examples/networks/cases.ts +++ /dev/null @@ -1,249 +0,0 @@ -/** - * Synthetic grid cases, as the page fetches them from `public/grids`, each gzip-compressed. - * - * A case file holds a "LKG1" magic and a u32 header length, then a JSON header naming the case and - * its counts, then typed arrays on eight-byte boundaries past the header, where its `arrays` place - * them as [type, offset, length]: each bus's number, name, kV, longitude and latitude, load and - * generation in MW; each branch's two bus rows, reactance, and whether it transforms; each machine's - * bus row, name, rating, and output. Simulated cases also hold each machine's speed deviation, frame - * by frame as int16 of `speedScale` per unit, `dt` seconds apart, and the fault the simulation - * applied. - */ -import { Data, Patch, reference } from '../../latkit.js'; - -interface Header { - readonly case: string; - readonly buses: number; - readonly branches: number; - readonly machines: number; - readonly frames?: number; - readonly dt?: number; - readonly speedScale?: number; - readonly fault?: { readonly bus: number; readonly on: number; readonly off: number }; - readonly arrays: Readonly< - Record - >; -} - -/** A case's simulation: each machine's speed deviation over the seconds simulated. */ -export interface Simulation { - readonly frames: number; - readonly dt: number; - readonly duration: number; - /** Speed deviation in mHz, frame by frame, a value per machine. */ - readonly speeds: Float32Array; - /** The largest deviation, in mHz, either way. */ - readonly largest: number; - readonly fault?: { readonly bus: string; readonly on: number; readonly off: number }; -} - -export interface Case { - readonly data: Data; - readonly name: string; - readonly buses: number; - readonly lines: number; - readonly machines: number; - /** Bytes downloaded, compressed. */ - readonly bytes: number; - /** Whether its buses have names, which labels show. */ - readonly named: boolean; - /** Each bus's longitude and latitude, and its load and generation in MW. */ - readonly positions: Float32Array; - readonly load: Float32Array; - readonly generation: Float32Array; - /** Each machine's bus row, and its output in MW. */ - readonly machineBus: Uint32Array; - readonly machinePower: Float32Array; - readonly simulation?: Simulation; -} - -/** A bus's or line's voltage level: 0 below 100 kV, then 115–161, 230, 345, 500, and 765 kV. */ -export function levelOf(kv: number): number { - return kv >= 700 ? 5 : kv >= 450 ? 4 : kv >= 300 ? 3 : kv >= 200 ? 2 : kv >= 100 ? 1 : 0; -} -export const LEVELS = ['< 100 kV', '115–161 kV', '230 kV', '345 kV', '500 kV', '765 kV'] as const; - -const MAGIC = 0x31474b4c; -const text = new TextDecoder(); - -/** A case file's bytes, read into one dataset named `grid`: buses, lines, machines, and the fault. */ -export function readCase(id: string, buffer: ArrayBuffer, downloaded = buffer.byteLength): Case { - const view = new DataView(buffer); - if (buffer.byteLength < 8 || view.getUint32(0, true) !== MAGIC) - throw new Error(`${id} is not a grid case`); - const length = view.getUint32(4, true); - const header = JSON.parse(text.decode(new Uint8Array(buffer, 8, length))) as Header; - const body = Math.ceil((8 + length) / 8) * 8; - const array = (name: string, of: new (b: ArrayBuffer, o: number, l: number) => T): T => { - const found = header.arrays[name]; - if (!found) throw new Error(`${id} has no ${name}`); - return new of(buffer, body + found[1], found[2]); - }; - const names = (name: string): string[] => { - const offsets = array(`${name}.offsets`, Uint32Array); - const bytes = array(`${name}.bytes`, Uint8Array); - return Array.from({ length: offsets.length - 1 }, (_, i) => - text.decode(bytes.subarray(offsets[i]!, offsets[i + 1]!)), - ); - }; - const n = header.buses; - const m = header.branches; - const g = header.machines; - const position = array('bus.position', Float32Array); - const kv = array('bus.kv', Float32Array); - const number = array('bus.number', Uint32Array); - const ends = array('branch.ends', Uint32Array); - const machineBus = array('machine.bus', Uint32Array); - const load = array('bus.load', Float32Array); - const generation = array('bus.generation', Float32Array); - const machinePower = array('machine.power', Float32Array); - const busNames = names('bus.name'); - const lon = new Float64Array(n); - const lat = new Float64Array(n); - for (let b = 0; b < n; b++) { - lon[b] = position[b * 2]!; - lat[b] = position[b * 2 + 1]!; - } - const from = new Uint32Array(m); - const to = new Uint32Array(m); - const lineKv = new Float32Array(m); - const transformer = Array.from(array('branch.transformer', Uint8Array), (v) => v !== 0); - for (let e = 0; e < m; e++) { - const a = ends[e * 2]!; - const b = ends[e * 2 + 1]!; - from[e] = number[a]!; - to[e] = number[b]!; - lineKv[e] = Math.max(kv[a]!, kv[b]!); - } - const machineLon = new Float64Array(g); - const machineLat = new Float64Array(g); - for (let k = 0; k < g; k++) { - machineLon[k] = lon[machineBus[k]!]!; - machineLat[k] = lat[machineBus[k]!]!; - } - const simulated = header.frames !== undefined && header.dt !== undefined; - const fault = simulated ? header.fault : undefined; - const patch = new Patch() - .set('Bus', number, { - lon, - lat, - name: busNames, - number: Float64Array.from(number), - kv, - level: Float32Array.from(kv, levelOf), - load, - generation, - }) - .set( - 'Line', - { count: m }, - { - from: reference('Bus', from), - to: reference('Bus', to), - kv: lineKv, - level: Float32Array.from(lineKv, levelOf), - reactance: array('branch.reactance', Float32Array), - transformer, - }, - ) - .set( - 'Machine', - { count: g }, - { - lon: machineLon, - lat: machineLat, - name: names('machine.name'), - power: machinePower, - mva: array('machine.mva', Float32Array), - }, - ); - if (fault) patch.set('Fault', { count: 1 }, { lon: [lon[fault.bus]!], lat: [lat[fault.bus]!] }); - let simulation: Simulation | undefined; - if (simulated) { - const frames = header.frames!; - const dt = header.dt!; - // Per unit of 60 Hz, in mHz. - const scale = (header.speedScale ?? 1) * 60_000; - const speeds = Float32Array.from(array('machine.speed', Int16Array), (value) => value * scale); - patch.axis('time', 's'); - for (let f = 0; f < frames; f++) - patch.append(f * dt, 'Machine', { speed: speeds.subarray(f * g, (f + 1) * g) }); - simulation = { - frames, - dt, - duration: (frames - 1) * dt, - speeds, - largest: 32767 * scale, - ...(fault - ? { - fault: { bus: busNames[fault.bus] ?? `bus ${fault.bus}`, on: fault.on, off: fault.off }, - } - : {}), - }; - } - return { - data: new Data('grid').apply(patch), - name: header.case, - buses: n, - lines: m, - machines: g, - bytes: downloaded, - named: true, - positions: position, - load, - generation, - machineBus, - machinePower, - ...(simulation ? { simulation } : {}), - }; -} - -/** How far a download has come: bytes received, of how many when the response says. */ -export type Progress = (received: number, total: number | undefined) => void; - -/** Megabytes received, of how many when known, for a status line. */ -export const megabytes = (received: number, total: number | undefined) => - `${(received / 1e6).toFixed(1)}${total ? ` of ${(total / 1e6).toFixed(1)}` : ''} MB`; - -/** - * A file's bytes, inflated where they arrive gzip-compressed, and how many arrived; `progress` - * hears of each chunk. A total the response gives that counts fewer bytes than arrive, as a - * compressed transfer's does, goes untold. - */ -export async function download( - url: string, - signal: AbortSignal, - progress?: Progress, -): Promise<{ readonly bytes: Uint8Array; readonly received: number }> { - const response = await fetch(url, { - signal: AbortSignal.any([signal, AbortSignal.timeout(180_000)]), - }); - if (!response.ok || !response.body) throw new Error(`${url} answered ${response.status}`); - const total = Number(response.headers.get('content-length')) || undefined; - const chunks: Uint8Array[] = []; - let received = 0; - const reader = response.body.getReader(); - for (let read = await reader.read(); !read.done; read = await reader.read()) { - chunks.push(read.value); - received += read.value.length; - progress?.(received, total !== undefined && total >= received ? total : undefined); - } - const bytes = new Uint8Array(received); - chunks.reduce((at, chunk) => (bytes.set(chunk, at), at + chunk.length), 0); - if (bytes[0] !== 0x1f || bytes[1] !== 0x8b) return { bytes, received: bytes.length }; - const inflated = new Blob([bytes]).stream().pipeThrough(new DecompressionStream('gzip')); - return { - bytes: new Uint8Array(await new Response(inflated).arrayBuffer()), - received: bytes.length, - }; -} - -export async function loadCase( - id: string, - signal: AbortSignal, - progress?: Progress, -): Promise { - const url = `${import.meta.env.BASE_URL}grids/${id}.bin`; - const { bytes, received } = await download(url, signal, progress); - return readCase(id, bytes.buffer as ArrayBuffer, received); -} diff --git a/npm/examples/src/examples/networks/index.ts b/npm/examples/src/examples/networks/index.ts deleted file mode 100644 index 9ff27218..00000000 --- a/npm/examples/src/examples/networks/index.ts +++ /dev/null @@ -1,545 +0,0 @@ -import { theme } from '../../ui/theme.js'; -/** - * A grid at scale over the earth. Choose a synthetic grid, from WECC 240-bus to ACTIVSg70k, a - * continent's grid of two million buses built in the page, or a lattice of up to a million buses, - * and the map beneath it: coasts and borders, Natural Earth's states colored by the grid's figures, - * or OpenStreetMap's planet as vector tiles. Play a simulated fault and watch the machines' speeds - * swing; turn the globe at noon or now; hover or click anything to inspect it. - */ -import { - field, - type ColormapName, - type Fit, - type Mark, - type Row, - type Scene, -} from '../../latkit.js'; -import { each } from '../../ui/rows.js'; -import type { Stage } from '../../ui/stage.js'; -import { LEVELS, loadCase, megabytes, type Case } from './cases.js'; -import { continent } from './continent.js'; -import { lattice } from './lattice.js'; -import { LOOK, MAPS, type Loaded, type MapChoice } from './maps.js'; -import { STATE_FIELDS, joinStates, type Joined, type StateField } from './states.js'; - -/** The grids the page draws, synthetic cases first. */ -const GRIDS = [ - ['WECC 240', 'wecc240'], - ['ACTIVSg2000', 'activsg2000'], - ['ACTIVSg10k', 'activsg10k'], - ['ACTIVSg25k', 'activsg25k'], - ['ACTIVSg70k', 'activsg70k'], - ['North America · 2M', 'continent-2m'], - ['Lattice · 100K', 'lattice-100k'], - ['Lattice · 1M', 'lattice-1m'], - ['None', 'none'], -] as const; -type GridId = (typeof GRIDS)[number][1]; - -/** Each voltage level's color, from below 100 kV to 765 kV. */ -const LEVEL_COLORS = ['#56657a', '#2fa49b', '#7fc97f', '#e6c84f', '#f08a3c', '#e5486f']; -const BUS_COLORS = { - level: { label: 'voltage level', colormap: undefined }, - load: { label: 'load', colormap: 'mako' }, - generation: { label: 'generation', colormap: 'amber' }, -} as const satisfies Record; -type BusColor = keyof typeof BUS_COLORS; -const STATE_COLORMAPS: Readonly> = { - buses: 'viridis', - load: 'mako', - generation: 'amber', - swing: 'inferno', -}; -/** Fields the inspector leaves out: where a row is, which the view shows, and translations. */ -const HIDDEN = /^(lon|lat|geometry|position|name_.*|name:.*)$/; -/** Noon over the western US at midsummer, where the first grid lies: seconds after the epoch. */ -const NOON = Date.UTC(2026, 5, 21, 19) / 1000; -const fixed = (value: number) => (value >= 100 ? value.toFixed(0) : value.toPrecision(2)); - -/** A grid drawn: its dataset and counts, and the case it came from when it is one. */ -interface Grid { - readonly id: GridId; - readonly label: string; - readonly buses: number; - readonly lines: number; - readonly summary: string; - readonly case?: Case; - readonly peaks: { readonly load: number; readonly generation: number; readonly power: number }; - /** The box its buses stand in, which framing the grid shows; none for the map alone. */ - readonly bounds?: Fit; -} - -/** The box a case's buses stand in. */ -function extent(positions: Float32Array): Fit { - const min: [number, number, number] = [Infinity, Infinity, 0]; - const max: [number, number, number] = [-Infinity, -Infinity, 0]; - for (let i = 0; i < positions.length; i += 2) { - min[0] = Math.min(min[0], positions[i]!); - min[1] = Math.min(min[1], positions[i + 1]!); - max[0] = Math.max(max[0], positions[i]!); - max[1] = Math.max(max[1], positions[i + 1]!); - } - return { bounds: { min, max } }; -} - -export default async function page(stage: Stage): Promise { - const state = { - projection: 'globe' as 'mercator' | 'tilt' | 'globe', - daylight: 'noon' as 'off' | 'now' | 'noon', - atmosphere: true, - graticule: false, - color: 'level' as BusColor, - buses: true, - lines: true, - machines: true, - labels: false, - mapLabels: true, - statesBy: 'none' as 'none' | StateField, - flow: false, - }; - let grid: Grid | undefined; - let map: { choice: MapChoice; loaded: Loaded } | undefined; - let joined: Joined | undefined; - - const scene = (): Scene => ({ - space: { - earth: { - projection: state.projection === 'globe' ? 'globe' : 'mercator', - surface: LOOK.ground, - graticule: state.graticule, - atmosphere: state.atmosphere, - ...(state.daylight === 'off' - ? {} - : { daylight: state.daylight === 'now' ? Date.now() / 1000 : NOON }), - }, - }, - theme: { - ...theme, - area: LOOK.fill, - atmosphere: '#5aa9ff99', - night: 0.28, - glow: 5, - // Comets kept soft, as rivers carry them across the map too. - flow: { color: '#d6e6ffa6', width: 3, length: 20, spacing: 72 }, - }, - controls: { selectBox: true }, - marks: [...mapMarks(), ...gridMarks()], - }); - - /** The map's marks, its states colored as chosen. */ - const mapMarks = (): Mark[] => { - if (!map) return []; - const by = state.statesBy; - const states = - by !== 'none' && joined - ? { by, peak: joined.peaks[by], colormap: STATE_COLORMAPS[by] } - : undefined; - return map.loaded.marks({ labels: state.mapLabels, ...(states ? { states } : {}) }); - }; - - /** What the grid draws, as the panel says. */ - const gridMarks = (): Mark[] => { - if (!grid?.buses) return []; - const marks: Mark[] = []; - const big = grid.buses > 20_000; - if (!grid.case) { - if (state.lines) - marks.push({ - edges: { of: ['grid', 'Line'], ends: ['from', 'to'], width: 0.6, pick: false }, - }); - if (state.buses) - marks.push({ - vertices: { - of: ['grid', 'Bus'], - position: ['lon', 'lat'], - color: field('load').colormap('viridis').domain(0, 1), - radius: 1.2, - }, - }); - return marks; - } - const level = field('level').colormap({ kind: 'categorical', colors: LEVEL_COLORS }); - if (state.lines) - marks.push({ - edges: { - of: ['grid', 'Line'], - ends: ['from', 'to'], - // Lines of higher voltage draw on top, and stay drawn as the camera pulls back. - priority: 'level', - color: level, - width: field('level') - .scale(big ? 0.5 : 0.8, big ? 1.8 : 2.8) - .domain(0, 5), - dash: 'transformer', - ...(state.flow ? { flow: field('level').mul(6).add(8) } : {}), - }, - }); - if (state.buses) { - const color = - state.color === 'level' - ? level - : field(state.color) - .colormap(BUS_COLORS[state.color].colormap) - .domain(0, Math.max(1, grid.peaks[state.color])); - marks.push({ - vertices: { - of: ['grid', 'Bus'], - position: ['lon', 'lat'], - color, - // Where buses crowd, those of higher voltage stand out, on top and from farther out. - priority: 'level', - radius: big ? field('level').scale(0.8, 2.6).domain(0, 5) : grid.buses > 5000 ? 2 : 3, - ...(state.labels && grid.case.named - ? { label: { text: 'name', priority: 'kv', size: 10 } } - : {}), - }, - }); - } - const simulation = grid.case.simulation; - if (state.machines && grid.case.machines) - marks.push({ - vertices: { - of: ['grid', 'Machine'], - position: ['lon', 'lat'], - marker: 'diamond', - color: simulation - ? field('speed').colormap('coolwarm').domain(-simulation.largest, simulation.largest) - : '#ebf0f7', - radius: field('power') - .scale(big ? 2 : 3, big ? 5 : 8) - .domain(0, Math.max(1, grid.peaks.power)), - }, - }); - if (simulation?.fault) - marks.push({ - vertices: { - of: ['grid', 'Fault'], - position: ['lon', 'lat'], - color: '#ff3829', - radius: 9, - marker: 'rounded', - }, - }); - return marks; - }; - - const view = stage.view(scene(), { - label: 'A grid over the earth: drag to pan, Shift-drag to turn, hover or click to inspect', - }); - // Fits keep clear of the readout over the view's top and the credit and legend over its foot. - view.insets = [32, 16, 32, 16]; - const restyle = () => view.show(scene()); - /** Frames the grid, whatever the map beneath it spans; the map alone, all of it. */ - const frame = () => view.fit(grid?.bounds ?? 'all'); - const legend = stage.legend(); - const credit = stage.credit(); - - // ── Grid ── - const gridSection = stage.section('Grid'); - gridSection.choice( - 'Grid', - GRIDS.map(([label, id]) => [label, id] as const), - GRIDS[0][1] as GridId, - (id) => void loadGrid(id).catch(stage.fail), - ); - const colorBy = gridSection.choice( - 'Bus color', - (Object.keys(BUS_COLORS) as BusColor[]).map((key) => [BUS_COLORS[key].label, key] as const), - state.color, - (color) => { - state.color = color; - restyle(); - showLegend(); - }, - ); - const levels = gridSection.series(LEVELS.map((label, i) => ({ label, color: LEVEL_COLORS[i]! }))); - const gridSummary = gridSection.note(); - - // ── Simulation ── - const simulationSection = stage.section('Simulation'); - simulationSection.speed([0.1, 0.25, 1], 0.25, (rate) => (playback.rate = rate)); - const simulationSummary = simulationSection.note(); - const playback = stage.playback({ - live: false, - rate: 0.25, - show(at) { - view.at = at; - inspect(); - }, - }); - - // ── Map ── - const mapSection = stage.section('Map'); - mapSection.choice( - 'Map', - [['None', undefined] as const, ...MAPS.map((m) => [m.label, m] as const)], - MAPS[0] as MapChoice | undefined, - (choice) => void loadMap(choice).catch(stage.fail), - ); - const statesBy = mapSection.choice( - 'State color', - [ - ['nothing', 'none'] as const, - ...(Object.keys(STATE_FIELDS) as StateField[]).map( - (k) => [STATE_FIELDS[k].label, k] as const, - ), - ], - 'none' as 'none' | StateField, - (by) => { - state.statesBy = by; - join(); - }, - ); - const mapSummary = mapSection.note(); - - // ── View ── - const viewSection = stage.section('View'); - viewSection.choice( - 'Projection', - [ - ['Globe', 'globe'], - ['Map', 'mercator'], - ['Tilted map', 'tilt'], - ] as const, - state.projection, - (projection) => { - state.projection = projection; - restyle(); - const camera = view.camera; - if ('center' in camera) view.fly({ ...camera, pitch: projection === 'tilt' ? 50 : 0 }); - }, - ); - viewSection.choice( - 'Daylight', - [ - ['Noon', 'noon'], - ['Now', 'now'], - ['Off', 'off'], - ] as const, - state.daylight, - (daylight) => { - state.daylight = daylight; - restyle(); - }, - ); - viewSection.toggle('Atmosphere', 'a', state.atmosphere, (on) => { - state.atmosphere = on; - restyle(); - }); - viewSection.toggle('Grid', 'g', state.graticule, (on) => { - state.graticule = on; - restyle(); - }); - viewSection.toggle('Orbit', 't', false, (on) => (view.orbit = on ? 4 : 0)); - viewSection.actions([{ label: 'Fit', key: 'f', run: frame }]); - - // ── Layers ── - const layers = stage.section('Layers'); - const layer = ( - label: string, - key: string, - name: 'buses' | 'lines' | 'machines' | 'labels' | 'mapLabels' | 'flow', - ) => - layers.toggle(label, key, state[name], (on) => { - state[name] = on; - restyle(); - }); - layer('Buses', 'b', 'buses'); - layer('Lines', 'l', 'lines'); - layer('Machines', 'm', 'machines'); - layer('Bus labels', 'n', 'labels'); - layer('Map labels', 'p', 'mapLabels'); - layer('Flow', 'c', 'flow'); - - // ── Inspector ── - const inspector = stage.section('Inspector'); - const described = inspector.note('Select an item.'); - const list = document.createElement('dl'); - list.className = 'fields'; - inspector.content.append(list); - let hovered: Row | null = null; - const inspect = () => { - const row = hovered ?? each(view.selection)[0]; - list.replaceChildren(); - if (!row) return void (described.textContent = 'Select an item.'); - const data = view.data(row.data); - if (!data) return; - const value = (name: string) => data.get(row.table, row.id, name, view.at); - const fields = data.fields(row.table).filter((name) => !HIDDEN.test(name)); - const name = fields.find((f) => /^name$/i.test(f)); - described.textContent = `${row.table} · ${name ? format(value(name)) : `#${row.id}`}${hovered ? '' : ' · selected'}`; - // The grid's figures first, then the rest, leaving out what this row has no value for. - const first = fields.filter((f) => f in STATE_FIELDS); - let shown = 0; - for (const f of [...first, ...fields.filter((f) => f !== name && !(f in STATE_FIELDS))]) { - const v = value(f); - if (v === null || v === undefined || v === '' || Number.isNaN(v)) continue; - const dt = document.createElement('dt'); - const dd = document.createElement('dd'); - dt.textContent = f in STATE_FIELDS ? STATE_FIELDS[f as StateField].label : f; - dd.textContent = format(v) + (f in STATE_FIELDS ? STATE_FIELDS[f as StateField].unit : ''); - list.append(dt, dd); - if (++shown === 18) break; - } - }; - view.on('hover', ({ hit }) => { - hovered = hit?.row ?? null; - inspect(); - }); - view.on('select', inspect); - - const meter = stage.meter(view); - const thousands = (n: number) => `${(n / 1000).toFixed(0)}k`; - meter.note(() => `${thousands(view.stats.drawn)} of ${thousands(view.stats.rows)} rows drawn`); - meter.note( - () => - `${(view.stats.resident / 1e6).toFixed(0)} of ${(view.stats.budget / 1e6).toFixed(0)} MB on the GPU`, - ); - - function showLegend(): void { - levels.hidden = !grid?.case || state.color !== 'level'; - const simulation = grid?.case?.simulation; - if (state.statesBy !== 'none' && joined) - return legend.ramp( - STATE_COLORMAPS[state.statesBy], - `${STATE_FIELDS[state.statesBy].label} 0 – ${fixed(joined.peaks[state.statesBy])}${STATE_FIELDS[state.statesBy].unit}`, - ); - if (simulation && state.machines) - return legend.ramp('coolwarm', `machine speed ±${fixed(simulation.largest)} mHz`); - if (!grid?.case) return legend.ramp('viridis', 'distance from the middle'); - if (state.color === 'level') return legend.ramp('grays', 'buses by voltage level'); - const { colormap, label } = BUS_COLORS[state.color]; - legend.ramp(colormap, `${label} 0 – ${fixed(grid.peaks[state.color])} MW`); - } - - /** The grid's figures in the map's states, when it has states, and what they color by. */ - function join(): void { - joined?.data.destroy(); - joined = undefined; - const states = map?.loaded.states; - if (states && grid?.case && state.statesBy !== 'none') { - joined = joinStates(states, grid.case); - view.setData(joined.data); - mapSummary.textContent = `${map!.loaded.summary} · the grid reaches ${joined.states} states`; - } - restyle(); - showLegend(); - } - - let gridLoad: AbortController | undefined; - async function loadGrid(id: GridId): Promise { - gridLoad?.abort(); - const load = (gridLoad = new AbortController()); - const label = GRIDS.find((g) => g[1] === id)![0]; - stage.detail(`${label}: loading…`); - const started = performance.now(); - let next: Grid; - if (id === 'none') { - view.removeData('grid'); - next = { - id, - label, - buses: 0, - lines: 0, - peaks: { load: 0, generation: 0, power: 0 }, - summary: 'nothing to draw over the map', - }; - } else if (id.startsWith('lattice')) { - const built = lattice(id); - view.setData(built.data); - next = { - id, - label, - buses: built.buses, - lines: built.lines, - bounds: built.bounds, - peaks: { load: 1, generation: 0, power: 0 }, - summary: `built in ${(performance.now() - started).toFixed(0)} ms`, - }; - } else { - const found = id.startsWith('continent') - ? continent(id) - : await loadCase(id, load.signal, (received, total) => - stage.detail(`${label}: downloading ${megabytes(received, total)}`), - ); - if (load.signal.aborted) return; - view.setData(found.data); - const peak = (values: Float32Array) => values.reduce((top, v) => Math.max(top, v), 0); - next = { - id, - label, - buses: found.buses, - lines: found.lines, - case: found, - bounds: extent(found.positions), - peaks: { - load: peak(found.load), - generation: peak(found.generation), - power: peak(found.machinePower), - }, - summary: found.bytes - ? `${(found.bytes / 1e6).toFixed(2)} MB in ${(performance.now() - started).toFixed(0)} ms` - : `built in ${(performance.now() - started).toFixed(0)} ms`, - }; - } - grid = next; - view.select([]); - hovered = null; - const simulation = next.case?.simulation; - simulationSection.element.hidden = !simulation; - playback.hidden = !simulation; - colorBy.disabled = !next.case; - playback.pause(); - playback.record(simulation ? [0, simulation.duration] : [0, 0]); - view.at = 0; - simulationSummary.textContent = simulation - ? `${simulation.frames} frames over ${simulation.duration.toFixed(0)} s` + - (simulation.fault - ? ` · fault at ${simulation.fault.bus}, ${simulation.fault.on.toFixed(2)}–${simulation.fault.off.toFixed(2)} s` - : '') - : ''; - gridSummary.textContent = - `${next.buses.toLocaleString()} buses · ${next.lines.toLocaleString()} lines` + - (next.case ? ` · ${next.case.machines.toLocaleString()} machines` : '') + - ` · ${next.summary}`; - statesBy.disabled = !next.case || !map?.loaded.states; - join(); - frame(); - stage.detail(`${label} · ${next.summary}`); - inspect(); - } - - let mapLoad: AbortController | undefined; - async function loadMap(choice: MapChoice | undefined): Promise { - mapLoad?.abort(); - const load = (mapLoad = new AbortController()); - map?.loaded.unload(view); - map = undefined; - joined = undefined; - credit.textContent = ''; - mapSummary.textContent = choice ? 'loading…' : ''; - restyle(); - if (!choice) return; - const loaded = await choice.load(view, load.signal, (status) => { - if (!load.signal.aborted) mapSummary.textContent = status; - }); - if (load.signal.aborted) return loaded.unload(view); - map = { choice, loaded }; - credit.textContent = choice.attribution; - mapSummary.textContent = loaded.summary; - statesBy.disabled = !grid?.case || !loaded.states; - join(); - } - - showLegend(); - await Promise.all([loadGrid(GRIDS[0][1]), loadMap(MAPS[0])]); -} - -/** A value read from a dataset, written for the inspector. */ -function format(value: unknown): string { - if (value === null || value === undefined) return '–'; - if (typeof value === 'number') - return Number.isInteger(value) ? value.toLocaleString() : value.toPrecision(4); - if (typeof value === 'object' && 'id' in value) return String((value as Row).id); - return typeof value === 'object' ? JSON.stringify(value) : String(value as string | boolean); -} diff --git a/npm/examples/src/examples/networks/lattice.ts b/npm/examples/src/examples/networks/lattice.ts deleted file mode 100644 index 6da25fee..00000000 --- a/npm/examples/src/examples/networks/lattice.ts +++ /dev/null @@ -1,94 +0,0 @@ -/** - * Square lattices of buses over most of the globe: the stress cases, past any synthetic case in - * size. Each bus joins its right and lower neighbours, and a few lines span the whole box as great - * circles. Built in time linear in the bus count. - */ -import { Data, Patch, reference, type Fit } from '../../latkit.js'; - -interface Spec { - readonly cols: number; - readonly rows: number; - readonly lon: readonly [number, number]; - readonly lat: readonly [number, number]; - /** Lines across the box, as [column, row] to [column, row]. */ - readonly spans: readonly (readonly [number, number, number, number])[]; -} - -export const LATTICES: Readonly> = { - 'lattice-100k': { - cols: 400, - rows: 250, - lon: [-160, 160], - lat: [-80, 80], - spans: [ - [0, 0, 399, 249], - [399, 0, 0, 249], - [0, 125, 399, 125], - [200, 0, 200, 249], - ], - }, - 'lattice-1m': { - cols: 1000, - rows: 1000, - lon: [-170, 170], - lat: [-75, 75], - spans: [ - [0, 0, 999, 999], - [999, 0, 0, 999], - ], - }, -}; - -/** The lattice `id` as dataset `grid`, buses colored by their distance from the box's middle, and - * the box. */ -export function lattice(id: string): { data: Data; buses: number; lines: number; bounds: Fit } { - const spec = LATTICES[id]; - if (!spec) throw new Error(`No lattice ${id}`); - const { cols, spans } = spec; - const count = cols * spec.rows; - const [west, east] = spec.lon; - const [south, north] = spec.lat; - const lon = new Float64Array(count); - const lat = new Float64Array(count); - const load = new Float32Array(count); - const at = (c: number, r: number) => r * cols + c; - let farthest = 1e-6; - for (let r = 0; r < spec.rows; r++) - for (let c = 0; c < cols; c++) { - const v = at(c, r); - lon[v] = west + ((east - west) * c) / (cols - 1); - lat[v] = south + ((north - south) * r) / (spec.rows - 1); - load[v] = Math.hypot(lon[v]! - (west + east) / 2, lat[v]! - (south + north) / 2); - farthest = Math.max(farthest, load[v]!); - } - for (let v = 0; v < count; v++) load[v]! /= farthest; - const lines = spec.rows * (cols - 1) + cols * (spec.rows - 1) + spans.length; - const from = new Uint32Array(lines); - const to = new Uint32Array(lines); - let e = 0; - for (let r = 0; r < spec.rows; r++) - for (let c = 0; c + 1 < cols; c++) { - from[e] = at(c, r); - to[e++] = at(c + 1, r); - } - for (let r = 0; r + 1 < spec.rows; r++) - for (let c = 0; c < cols; c++) { - from[e] = at(c, r); - to[e++] = at(c, r + 1); - } - for (const [c0, r0, c1, r1] of spans) { - from[e] = at(c0, r0); - to[e++] = at(c1, r1); - } - const data = new Data('grid').apply( - new Patch() - .set('Bus', { count }, { lon, lat, load }) - .set('Line', { count: lines }, { from: reference('Bus', from), to: reference('Bus', to) }), - ); - return { - data, - buses: count, - lines, - bounds: { bounds: { min: [west, south, 0], max: [east, north, 0] } }, - }; -} diff --git a/npm/examples/src/examples/networks/maps.ts b/npm/examples/src/examples/networks/maps.ts deleted file mode 100644 index c831360b..00000000 --- a/npm/examples/src/examples/networks/maps.ts +++ /dev/null @@ -1,170 +0,0 @@ -/** - * What the map beneath the grid draws: Natural Earth's coasts and borders as lines, shipped with the - * page; Natural Earth's states and provinces as areas the grid's figures color, from Natural Earth's - * repository; or OpenStreetMap's planet as OpenFreeMap's vector tiles. GeoJSON downloads as bytes - * and is read in a worker; tiles are fetched as the camera asks for them. - */ -import { Data, field, type ColormapName, type Mark, type View } from '../../latkit.js'; -import { download, megabytes } from './cases.js'; - -/** The page's colors, as CSS. */ -export const LOOK = { - background: '#05070a', - ground: '#0e131b', - fill: '#1a212b', - line: '#6b80948c', - coast: '#8ca3b8d9', - landcover: '#1c2924', - park: '#1a3021', - water: '#0d1a2b', - waterway: '#2e5485e6', - road: '#5c616bb3', - place: '#c3ccd8', -} as const; - -/** How the map draws: whether its areas are labeled, and what its states color by. */ -export interface Styling { - readonly labels: boolean; - readonly states?: { readonly by: string; readonly peak: number; readonly colormap: ColormapName }; -} - -/** A map, loaded: its marks, drawn under the grid, and its states when it has them. */ -export interface Loaded { - readonly summary: string; - marks(styling: Styling): Mark[]; - /** The dataset of states the grid's figures color, its rows in table `polygons`. */ - readonly states?: Data; - /** Takes back from the view what loading gave it. */ - unload(view: View): void; -} - -export interface MapChoice { - readonly id: string; - readonly label: string; - readonly attribution: string; - /** Loads the map into `view`, telling `progress` how it goes. */ - load(view: View, signal: AbortSignal, progress: (status: string) => void): Promise; -} - -const NE = 'https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/'; -const NATURAL_EARTH = 'Natural Earth · public domain'; -const seconds = (since: number) => `${((performance.now() - since) / 1000).toFixed(1)} s`; - -/** Natural Earth's coasts, land borders, and province borders at 1:50M: datasets of lines. */ -const borders: MapChoice = { - id: 'borders', - label: 'Coasts and borders', - attribution: NATURAL_EARTH, - async load(view, signal) { - const started = performance.now(); - const files = ['coastline', 'countries', 'provinces'] as const; - const sets = await Promise.all( - files.map(async (name) => { - const { bytes } = await download( - `${import.meta.env.BASE_URL}maps/${name}.geojson.gz`, - signal, - ); - return Data.read(bytes, name); - }), - ); - for (const data of sets) view.setData(data); - return { - summary: `coasts and borders in ${seconds(started)}`, - marks: () => [ - { - lines: { of: ['provinces', 'lines'], color: LOOK.line, width: 0.6, dash: true }, - }, - { lines: { of: ['countries', 'lines'], color: LOOK.line, width: 1 } }, - { lines: { of: ['coastline', 'lines'], color: LOOK.coast, width: 1 } }, - ], - unload: (view) => files.forEach((name) => view.removeData(name)), - }; - }, -}; - -/** Natural Earth's states and provinces at `scale`: areas the grid's figures color. */ -function states(scale: '50m' | '10m', label: string): MapChoice { - return { - id: `states-${scale}`, - label, - attribution: NATURAL_EARTH, - async load(view, signal, progress) { - const started = performance.now(); - const { bytes, received } = await download( - `${NE}ne_${scale}_admin_1_states_provinces.geojson`, - signal, - (got, total) => progress(`downloading ${megabytes(got, total)}`), - ); - const fetched = performance.now(); - const data = await Data.read(bytes, 'states'); - view.setData(data); - return { - states: data, - summary: `${(received / 1e6).toFixed(1)} MB in ${seconds(started)}, read in ${seconds(fetched)}`, - marks: ({ labels, states }) => [ - { - // Each state holds the grid's figures: hovering one inspects them. - areas: { - of: ['states', 'polygons'], - color: states - ? field(states.by).colormap(states.colormap).domain(0, Math.max(states.peak, 1e-6)) - : LOOK.fill, - ...(labels ? { label: { text: 'name', size: 10, color: '#8e99aa' } } : {}), - pick: true, - }, - }, - { lines: { of: ['states', 'polygons'], color: LOOK.line, width: 0.7 } }, - ], - unload: (view) => view.removeData('states'), - }; - }, - }; -} - -/** OpenStreetMap's planet as OpenFreeMap's vector tiles. */ -const planet: MapChoice = { - id: 'planet', - label: 'Planet, vector tiles', - attribution: 'OpenFreeMap · © OpenMapTiles · © OpenStreetMap contributors', - load(view) { - view.setTiles('base', 'https://tiles.openfreemap.org/planet'); - return Promise.resolve({ - summary: 'tiles fetch as the camera asks for them', - marks: ({ labels }) => [ - { areas: { of: ['base', 'landcover'], color: LOOK.landcover } }, - { areas: { of: ['base', 'park'], color: LOOK.park } }, - { areas: { of: ['base', 'water'], color: LOOK.water } }, - // Rivers carry comets downstream, as their ways run. - { lines: { of: ['base', 'waterway'], color: LOOK.waterway, width: 0.8, flow: 10 } }, - { lines: { of: ['base', 'transportation'], color: LOOK.road, width: 0.7 } }, - { lines: { of: ['base', 'boundary'], color: LOOK.line, width: 1, dash: true } }, - // Places by rank, 1 the most important; a tile holds only the points inside it, so none repeat. - ...(labels - ? [ - { - vertices: { - of: ['base', 'place'], - color: LOOK.place, - radius: 1.5, - label: { - text: 'name', - size: 11, - color: LOOK.place, - priority: field('rank').mul(-1), - }, - }, - } satisfies Mark, - ] - : []), - ], - unload: (view) => view.removeData('base'), - }); - }, -}; - -export const MAPS: readonly MapChoice[] = [ - borders, - states('50m', 'States and provinces, 1:50M'), - states('10m', 'States and provinces, 1:10M (40 MB)'), - planet, -]; diff --git a/npm/examples/src/examples/networks/states.ts b/npm/examples/src/examples/networks/states.ts deleted file mode 100644 index 527345fb..00000000 --- a/npm/examples/src/examples/networks/states.ts +++ /dev/null @@ -1,133 +0,0 @@ -/** - * A grid's figures for each state it reaches, written into the states dataset as fields its areas - * color by: buses, load, and generation counted where each bus stands, and where the grid was - * simulated, how far its machines' speeds swing in each state, frame by frame. - */ -import { Patch, type Data, type Id } from '../../latkit.js'; -import type { Case } from './cases.js'; - -/** What each state's figures hold, and in what unit. */ -export const STATE_FIELDS = { - buses: { label: 'buses', unit: '' }, - load: { label: 'load', unit: ' MW' }, - generation: { label: 'generation', unit: ' MW' }, - swing: { label: 'largest speed swing', unit: ' mHz' }, -} as const; -export type StateField = keyof typeof STATE_FIELDS; - -export interface Joined { - readonly data: Data; - /** How many states the grid reaches. */ - readonly states: number; - /** Each figure's largest value in any state. */ - readonly peaks: Readonly>; -} - -type Ring = [number, number][]; - -interface State { - readonly row: number; - /** West, south, east, north. */ - readonly bounds: readonly [number, number, number, number]; - readonly rings: readonly Ring[]; -} - -/** Whether a point lies inside a state: crossing its rings an odd number of times, holes included. */ -function inside(state: State, x: number, y: number): boolean { - const [w, s, e, n] = state.bounds; - if (x < w || x > e || y < s || y > n) return false; - let crossings = 0; - for (const ring of state.rings) - for (let i = 0, j = ring.length - 1; i < ring.length; j = i++) { - const [xi, yi] = ring[i]!; - const [xj, yj] = ring[j]!; - if (yi > y !== yj > y && x < ((xj - xi) * (y - yi)) / (yj - yi) + xi) crossings++; - } - return crossings % 2 === 1; -} - -/** A snapshot of `states` with the grid's figures for every state it reaches. */ -export function joinStates(states: Data, grid: Case): Joined { - const { positions, simulation } = grid; - const extent = [Infinity, Infinity, -Infinity, -Infinity]; - for (let b = 0; b < grid.buses; b++) { - extent[0] = Math.min(extent[0]!, positions[b * 2]!); - extent[1] = Math.min(extent[1]!, positions[b * 2 + 1]!); - extent[2] = Math.max(extent[2]!, positions[b * 2]!); - extent[3] = Math.max(extent[3]!, positions[b * 2 + 1]!); - } - const ids: Id[] = states.ids('polygons'); - const found: State[] = []; - ids.forEach((id, row) => { - const polygons = (states.get('polygons', id, 'geometry') ?? []) as Ring[][]; - const rings = polygons.flat(); - const bounds: [number, number, number, number] = [Infinity, Infinity, -Infinity, -Infinity]; - for (const ring of rings) - for (const [x, y] of ring) { - bounds[0] = Math.min(bounds[0], x); - bounds[1] = Math.min(bounds[1], y); - bounds[2] = Math.max(bounds[2], x); - bounds[3] = Math.max(bounds[3], y); - } - const meets = - bounds[0] <= extent[2]! && - bounds[2] >= extent[0]! && - bounds[1] <= extent[3]! && - bounds[3] >= extent[1]!; - if (meets) found.push({ row, bounds, rings }); - }); - // Each degree cell's states, so a bus tests only those near it. - const cells = new Map(); - const cell = (x: number, y: number) => Math.floor(x + 180) * 180 + Math.floor(y + 90); - for (const state of found) { - const [w, s, e, n] = state.bounds; - for (let x = Math.floor(Math.max(w, extent[0]!)); x <= Math.min(e, extent[2]!); x++) - for (let y = Math.floor(Math.max(s, extent[1]!)); y <= Math.min(n, extent[3]!); y++) { - const key = cell(x, y); - let list = cells.get(key); - if (!list) cells.set(key, (list = [])); - list.push(state); - } - } - const rows = ids.length; - const of = new Int32Array(grid.buses).fill(-1); - const buses = new Float32Array(rows).fill(Number.NaN); - const load = new Float32Array(rows).fill(Number.NaN); - const generation = new Float32Array(rows).fill(Number.NaN); - const reached = new Set(); - for (let b = 0; b < grid.buses; b++) { - const x = positions[b * 2]!; - const y = positions[b * 2 + 1]!; - const state = cells.get(cell(x, y))?.find((candidate) => inside(candidate, x, y)); - if (!state) continue; - const k = state.row; - of[b] = k; - if (!reached.has(k)) { - reached.add(k); - buses[k] = load[k] = generation[k] = 0; - } - buses[k]! += 1; - load[k]! += grid.load[b]!; - generation[k]! += grid.generation[b]!; - } - const peak = (values: Float32Array) => values.reduce((top, v) => (v > top ? v : top), 0); - const patch = new Patch().set('polygons', ids as number[], { buses, load, generation }); - const peaks = { buses: peak(buses), load: peak(load), generation: peak(generation), swing: 0 }; - if (simulation) { - // Each frame, the largest swing of a state's machines either way. - const g = grid.machines; - const swing = new Float32Array(rows); - for (let f = 0; f < simulation.frames; f++) { - swing.fill(Number.NaN); - for (const k of reached) swing[k] = 0; - for (let m = 0; m < g; m++) { - const k = of[grid.machineBus[m]!]!; - if (k < 0) continue; - swing[k] = Math.max(swing[k]!, Math.abs(simulation.speeds[f * g + m]!)); - } - peaks.swing = Math.max(peaks.swing, peak(swing)); - patch.append(f * simulation.dt, 'polygons', { swing }); - } - } - return { data: states.snapshot().apply(patch), states: reached.size, peaks }; -} diff --git a/npm/examples/src/examples/playback/index.ts b/npm/examples/src/examples/playback/index.ts deleted file mode 100644 index 67cb01ff..00000000 --- a/npm/examples/src/examples/playback/index.ts +++ /dev/null @@ -1,222 +0,0 @@ -import { random } from '../../random.js'; -import { theme } from '../../ui/theme.js'; -import { - Data, - Patch, - color, - field, - missing, - reference, - when, - type ColormapName, - type Id, - type Mark, - type Scene, -} from '../../latkit.js'; -import type { Stage } from '../../ui/stage.js'; - -/** Buses on a side of the lattice. */ -const SIDE = 60; -/** Seconds of recorded voltages. */ -const SECONDS = 60; - -/** A synthetic grid over Texas: buses on a jittered lattice, lines to their neighbors, and a minute of - * voltages that ripple across it. */ -function grid(): { data: Data; buses: number; lines: number } { - const buses = SIDE * SIDE; - const rng = random(7); - const lon = new Float64Array(buses); - const lat = new Float64Array(buses); - const kv = new Float64Array(buses); - const name: string[] = []; - for (let i = 0; i < buses; i++) { - lon[i] = -106 + ((i % SIDE) + rng() * 0.6) * (12 / SIDE); - lat[i] = 26 + (Math.floor(i / SIDE) + rng() * 0.6) * (10 / SIDE); - kv[i] = [69, 138, 345][Math.floor(rng() * 3)]!; - name.push(`Bus ${i + 1}`); - } - const from: number[] = []; - const to: number[] = []; - const link = (a: number, b: number) => { - from.push(a); - to.push(b); - }; - for (let i = 0; i < buses; i++) { - if (i % SIDE < SIDE - 1) link(i, i + 1); - if (i + SIDE < buses) link(i, i + SIDE); - } - const data = new Data('grid').apply( - new Patch() - .axis('time', 's') - .set('Bus', { count: buses }, { lon, lat, kv, name }) - .set( - 'Line', - { count: from.length }, - { from: reference('Bus', from), to: reference('Bus', to) }, - ), - ); - const vm = new Float32Array(buses); - for (let t = 0; t <= SECONDS; t++) { - for (let i = 0; i < buses; i++) { - const x = (i % SIDE) / SIDE; - const y = Math.floor(i / SIDE) / SIDE; - vm[i] = 1 + 0.04 * Math.sin(6 * x + t / 6) * Math.cos(5 * y - t / 9); - } - data.apply(new Patch().append(t, 'Bus', { vm })); - } - return { data, buses, lines: from.length }; -} - -export default function page(stage: Stage): void { - const { data, buses, lines } = grid(); - const look = { - space: 'mercator' as Scene['space'], - colormap: 'balance' as ColormapName, - /** How far from 1 pu the colormap's ends lie. */ - spread: 0.04, - size: 1, - lines: true, - labels: true, - outliers: false, - }; - const scene = (): Scene => { - const marks: Mark[] = []; - if (look.lines) - marks.push({ edges: { of: ['grid', 'Line'], ends: ['from', 'to'], width: 1, pick: false } }); - marks.push({ - vertices: { - of: ['grid', 'Bus'], - position: ['lon', 'lat'], - radius: field('kv').div(100).add(1).mul(look.size), - color: field('vm') - .colormap(look.colormap) - .domain(1 - look.spread, 1 + look.spread), - // Buses past the colormap's ends ring, so they stand out at any zoom. - ...(look.outliers - ? { - ring: when(field('vm').sub(1).abs().gt(look.spread)) - .then(color('#ffd166')) - .otherwise(missing()), - } - : {}), - ...(look.labels ? { label: { text: 'name', priority: 'kv', size: 10 } } : {}), - }, - }); - return { - space: look.space, - theme, - controls: { selectBox: true }, - marks, - }; - }; - const view = stage.view(scene(), { - label: 'A synthetic grid over Texas: click a bus to select it, or Shift-drag a box', - }); - view.setData(data); - view.fit(); - const legend = stage.legend(); - const restyle = () => { - view.show(scene()); - legend.ramp( - look.colormap, - `Voltage ${(1 - look.spread).toFixed(2)} – ${(1 + look.spread).toFixed(2)} pu`, - ); - }; - - const space = stage.section('View'); - space.choice( - 'Projection', - [ - ['Map', 'mercator'], - ['Globe', 'globe'], - ['Plane', 'plane'], - ] as const, - 'mercator', - (next) => { - look.space = next; - restyle(); - view.fit(); - }, - ); - space.actions([{ label: 'Fit', key: 'f', run: () => view.fit() }]); - - const style = stage.section('Display'); - style.choice( - 'Colormap', - (['balance', 'vik', 'coolwarm', 'rdbu', 'berlin'] as const).map( - (name) => [name, name] as const, - ), - look.colormap, - (colormap) => { - look.colormap = colormap; - restyle(); - }, - ); - style.choice( - 'Range', - [ - ['±2%', 0.02], - ['±4%', 0.04], - ['±6%', 0.06], - ] as const, - look.spread, - (spread) => { - look.spread = spread; - restyle(); - }, - ); - style.range( - 'Bus size', - { min: 0.5, max: 3, step: 0.25, value: look.size, format: (v) => `${v.toFixed(2)}×` }, - (size) => { - if (size === look.size) return; - look.size = size; - restyle(); - }, - ); - style.toggle('Lines', 'l', look.lines, (on) => { - look.lines = on; - restyle(); - }); - style.toggle('Labels', 'n', look.labels, (on) => { - look.labels = on; - restyle(); - }); - style.toggle('Outliers', 'o', look.outliers, (on) => { - look.outliers = on; - restyle(); - }); - - const selected = stage.section('Selection'); - const readout = selected.note(); - const describe = () => { - const ids = view.selection.filter((rows) => rows.table === 'Bus').flatMap((rows) => rows.ids); - const vm = (id: Id) => data.get('Bus', id, 'vm', view.at) as number | null; - if (!ids.length) readout.textContent = 'Click a bus, or Shift-drag a box.'; - else if (ids.length === 1) { - const id = ids[0]!; - const kv = data.get('Bus', id, 'kv') as number | null; - readout.textContent = `Bus ${id}: ${kv ?? '–'} kV, ${vm(id)?.toFixed(3) ?? '–'} pu`; - } else { - const values = ids.map(vm).filter((v): v is number => v !== null); - readout.textContent = `${ids.length} buses: ${Math.min(...values).toFixed(3)} – ${Math.max(...values).toFixed(3)} pu`; - } - }; - view.on('select', describe); - stage.meter(view); - const playback = stage.playback({ - live: false, - rate: 4, - show(at) { - view.at = at; - describe(); - }, - }); - selected.speed([1, 4, 10], 4, (rate) => (playback.rate = rate)); - playback.record(data.span!); - restyle(); - describe(); - stage.detail( - `${buses.toLocaleString()} buses · ${lines.toLocaleString()} lines · ${SECONDS} s of voltages`, - ); -} diff --git a/npm/examples/src/examples/power-flow/grid.test.ts b/npm/examples/src/examples/power-flow/grid.test.ts deleted file mode 100644 index 9dfdd946..00000000 --- a/npm/examples/src/examples/power-flow/grid.test.ts +++ /dev/null @@ -1,60 +0,0 @@ -import { describe, expect, test } from 'vitest'; -import { Grid } from './grid.js'; -import texas from './texas.json'; - -/** MW each substation sends out along its branches in service, by the flows solved. */ -function sent(grid: Grid): Float64Array { - const out = new Float64Array(grid.n); - texas.branches.forEach((branch, j) => { - out[branch.from]! += grid.flow[j]!; - out[branch.to]! -= grid.flow[j]!; - }); - return out; -} - -/** The largest gap between what each substation injects and what its branches carry away. */ -function imbalance(grid: Grid): number { - const out = sent(grid); - return Math.max(...Array.from(out, (mw, i) => Math.abs(mw - grid.injection[i]!))); -} - -describe('the DC power flow', () => { - test('balances every island: plants make what loads are served, and each bus sends on its injection', () => { - const grid = new Grid(); - const total = grid.injection.reduce((sum, mw) => sum + mw, 0); - expect(Math.abs(total)).toBeLessThan(1e-6); - expect(imbalance(grid)).toBeLessThan(1e-6); - expect(grid.dark).toBe(0); - }); - - test('a tripped branch carries nothing, and the rest take its flow', () => { - const grid = new Grid(); - const before = Float64Array.from(grid.flow); - grid.tripWorst(); - const tripped = grid.out.findIndex((out) => out === 1); - expect(tripped).toBeGreaterThanOrEqual(0); - expect(grid.trips).toBe(1); - expect(grid.flow[tripped]).toBe(0); - expect(imbalance(grid)).toBeLessThan(1e-6); - expect(grid.flow.some((mw, j) => j !== tripped && Math.abs(mw - before[j]!) > 1e-3)).toBe(true); - }); - - test('restoring recloses every branch and solves as before', () => { - const grid = new Grid(); - const before = Float64Array.from(grid.flow); - grid.tripWorst(); - grid.restore(); - expect(grid.out.every((out) => out === 0)).toBe(true); - expect(grid.trips).toBe(0); - grid.flow.forEach((mw, j) => expect(mw).toBeCloseTo(before[j]!, 6)); - }); - - test('more demand loads branches more', () => { - const grid = new Grid(); - const loaded = () => grid.loading.reduce((sum, l) => sum + l, 0); - const light = loaded(); - grid.demand = 1.1; - grid.step(0); - expect(loaded()).toBeGreaterThan(light); - }); -}); diff --git a/npm/examples/src/examples/power-flow/grid.ts b/npm/examples/src/examples/power-flow/grid.ts deleted file mode 100644 index 260873ae..00000000 --- a/npm/examples/src/examples/power-flow/grid.ts +++ /dev/null @@ -1,237 +0,0 @@ -/** - * The grid a control room watches, as a model: plants dispatch to meet demand, a DC power flow sets - * every branch's flow, protection trips what runs past its rating, and islands balance alone. - */ -import texas from './texas.json'; - -/** What each substation is, in the order its color index counts them. */ -export const KINDS = [ - { label: 'nuclear', color: '#a181ef', fuels: ['Nuclear Steam'] }, - { - label: 'thermal', - color: '#c08a52', - fuels: ['Gas Turbine', 'Gas Combined Cycle', 'Coal-fired Steam'], - }, - { label: 'wind', color: '#00d1da', fuels: ['Wind'] }, - { label: 'solar', color: '#ffca29', fuels: ['Solar PV'] }, - { label: 'load', color: '#8a93a0', fuels: ['Load'] }, -] as const; -/** How far past its rating a branch runs before its protection trips it, and for how many ticks. */ -export const TRIP_AT = 1.15; -export const TRIP_AFTER = 3; -/** Seconds a tripped branch stays out before it recloses. */ -export const OUT_FOR = 14; -/** Per unit reactance per kilometre of line. */ -export const REACTANCE = 0.0004; - -type Substation = (typeof texas.substations)[number]; - -/** Each substation's kind, by index into `KINDS`. */ -const kindOf = (s: Substation) => - KINDS.findIndex((k) => (k.fuels as readonly string[]).includes(s.category)); - -/** - * The grid's state and its physics: plants dispatch to meet demand, a DC power flow sets every - * branch's flow from the injections, protection trips branches held past their rating, and each - * island balances on its own plants, its loads dark where it has none. - */ -export class Grid { - readonly n = texas.substations.length; - readonly m = texas.branches.length; - /** What each plant can make and each load asks for, MW. */ - readonly capacity = Float64Array.from(texas.substations, (s) => - s.units.reduce((a, b) => a + b, 0), - ); - readonly load = Uint8Array.from(texas.substations, (s) => (s.category === 'Load' ? 1 : 0)); - readonly kind = Float32Array.from(texas.substations, kindOf); - readonly rating = Float64Array.from(texas.branches, (b) => b.pmax); - readonly #x: Float64Array; - /** Each substation's output, or each load's demand served, as a share; MW injected. */ - readonly fill = new Float32Array(this.n); - readonly injection = new Float64Array(this.n); - /** Each branch's flow in MW, from its first end to its second; its loading; whether it is out. */ - readonly flow = new Float64Array(this.m); - readonly loading = new Float64Array(this.m); - readonly out = new Uint8Array(this.m); - /** Ticks each branch has run past its trip setting, and when an out branch recloses. */ - readonly #over = new Uint8Array(this.m); - readonly #reclose = new Float64Array(this.m); - /** Plants out of service until a time. */ - readonly #plantOut = new Float64Array(this.n); - /** Demand as a share of the loads' size, and the time. */ - demand = 0.78; - t = 0; - /** Branches tripped since the last restore, and loads dark now. */ - trips = 0; - dark = 0; - - constructor() { - const s = texas.substations; - this.#x = Float64Array.from(texas.branches, (b) => { - const [a, c] = [s[b.from]!, s[b.to]!]; - const km = - Math.hypot((a.lon - c.lon) * Math.cos((a.lat * Math.PI) / 180), a.lat - c.lat) * 111; - return Math.max(0.002, REACTANCE * km); - }); - this.step(0); - } - - /** Trips the branch loaded nearest its rating, as a fault would. */ - tripWorst(): void { - let worst = -1; - for (let j = 0; j < this.m; j++) - if (!this.out[j] && (worst < 0 || this.loading[j]! > this.loading[worst]!)) worst = j; - if (worst >= 0) this.#trip(worst); - this.step(0); - } - - /** Recloses every branch and plant. */ - restore(): void { - this.out.fill(0); - this.#over.fill(0); - this.#plantOut.fill(0); - this.trips = 0; - this.step(0); - } - - #trip(j: number): void { - this.out[j] = 1; - this.#reclose[j] = this.t + OUT_FOR; - this.trips++; - } - - /** Advances `dt` seconds: dispatch, flow, protection. */ - step(dt: number): void { - this.t += dt; - const { n, m, t } = this; - for (let j = 0; j < m; j++) - if (this.out[j] && t >= this.#reclose[j]! && this.#reclose[j]! > 0) this.out[j] = 0; - // What each load asks for, swinging through the day, and what each plant could give. - const want = new Float64Array(n); - const can = new Float64Array(n); - for (let i = 0; i < n; i++) { - const wave = 0.08 * Math.sin(t * 0.07 + i * 1.3) + 0.04 * Math.sin(t * 0.31 + i * 0.7); - if (this.load[i]) - want[i] = this.capacity[i]! * Math.min(1.4, Math.max(0, this.demand + wave)); - else if (t >= this.#plantOut[i]!) { - // Wind and sun come and go; the rest follow dispatch. - const k = this.kind[i]!; - const weather = - k === 2 - ? 0.55 + 0.4 * Math.sin(t * 0.11 + i) - : k === 3 - ? 0.8 + 0.15 * Math.sin(t * 0.05 + i) - : 1; - can[i] = this.capacity[i]! * weather; - } - } - // Islands of the branches in service, each balanced on its own plants. - const island = this.#islands(); - this.dark = 0; - for (const members of island) { - const asked = members.reduce((sum, i) => sum + want[i]!, 0); - const able = members.reduce((sum, i) => sum + can[i]!, 0); - const served = able > 0 ? Math.min(1, able / Math.max(asked, 1e-9)) : 0; - const dispatch = asked > 0 ? Math.min(1, (asked * served) / Math.max(able, 1e-9)) : 0; - for (const i of members) { - if (this.load[i]) { - this.fill[i] = served; - this.injection[i] = -want[i]! * served; - if (served < 0.05 && want[i]! > 0) this.dark++; - } else { - this.fill[i] = this.capacity[i]! > 0 ? (can[i]! * dispatch) / this.capacity[i]! : 0; - this.injection[i] = can[i]! * dispatch; - } - } - this.#solve(members); - } - // Protection: a branch held past its trip setting trips; a plant now and then does too. - for (let j = 0; j < m; j++) { - this.loading[j] = this.out[j] ? 0 : Math.abs(this.flow[j]!) / this.rating[j]!; - this.#over[j] = this.loading[j]! > TRIP_AT ? this.#over[j]! + 1 : 0; - if (dt > 0 && this.#over[j]! >= TRIP_AFTER) this.#trip(j); - } - if (dt > 0 && Math.random() < 0.004) { - const plants = [...Array(n).keys()].filter((i) => !this.load[i] && t >= this.#plantOut[i]!); - const plant = plants[Math.floor(Math.random() * plants.length)]; - if (plant !== undefined) this.#plantOut[plant] = t + 20; - } - } - - /** Whether plant `i` is out of service. */ - tripped(i: number): boolean { - return !this.load[i] && this.t < this.#plantOut[i]!; - } - - /** Substations joined by branches in service, island by island. */ - #islands(): number[][] { - const seen = new Uint8Array(this.n); - const near: number[][] = Array.from({ length: this.n }, () => []); - texas.branches.forEach((b, j) => { - if (!this.out[j]) { - near[b.from]!.push(b.to); - near[b.to]!.push(b.from); - } - }); - const islands: number[][] = []; - for (let start = 0; start < this.n; start++) { - if (seen[start]) continue; - const members = [start]; - seen[start] = 1; - for (let k = 0; k < members.length; k++) - for (const next of near[members[k]!]!) { - if (seen[next]) continue; - seen[next] = 1; - members.push(next); - } - islands.push(members); - } - return islands; - } - - /** The DC power flow of one island: angles from its injections, its first member the reference, - * then each branch's flow. */ - #solve(members: number[]): void { - const size = members.length - 1; - const at = new Map(members.slice(1).map((bus, k) => [bus, k])); - const b = Array.from({ length: size }, () => new Float64Array(size + 1)); - texas.branches.forEach((branch, j) => { - if (this.out[j] || !(at.has(branch.from) || branch.from === members[0])) return; - if (!(at.has(branch.to) || branch.to === members[0])) return; - const y = 1 / this.#x[j]!; - const [p, q] = [at.get(branch.from), at.get(branch.to)]; - if (p !== undefined) b[p]![p]! += y; - if (q !== undefined) b[q]![q]! += y; - if (p !== undefined && q !== undefined) { - b[p]![q]! -= y; - b[q]![p]! -= y; - } - }); - members.slice(1).forEach((bus, k) => (b[k]![size] = this.injection[bus]! / 100)); - // Gaussian elimination with partial pivoting; the island is small. - for (let c = 0; c < size; c++) { - let pivot = c; - for (let r = c + 1; r < size; r++) - if (Math.abs(b[r]![c]!) > Math.abs(b[pivot]![c]!)) pivot = r; - [b[c], b[pivot]] = [b[pivot]!, b[c]!]; - const d = b[c]![c]! || 1e-12; - for (let r = c + 1; r < size; r++) { - const f = b[r]![c]! / d; - if (f) for (let k = c; k <= size; k++) b[r]![k]! -= f * b[c]![k]!; - } - } - const theta = new Float64Array(size); - for (let r = size - 1; r >= 0; r--) { - let sum = b[r]![size]!; - for (let k = r + 1; k < size; k++) sum -= b[r]![k]! * theta[k]!; - theta[r] = sum / (b[r]![r]! || 1e-12); - } - const angle = (bus: number) => (bus === members[0] ? 0 : theta[at.get(bus)!]!); - texas.branches.forEach((branch, j) => { - if (at.has(branch.from) || branch.from === members[0]) - this.flow[j] = this.out[j] - ? 0 - : ((angle(branch.from) - angle(branch.to)) * 100) / this.#x[j]!; - }); - } -} diff --git a/npm/examples/src/examples/power-flow/index.ts b/npm/examples/src/examples/power-flow/index.ts deleted file mode 100644 index 62c54f9d..00000000 --- a/npm/examples/src/examples/power-flow/index.ts +++ /dev/null @@ -1,317 +0,0 @@ -import { theme } from '../../ui/theme.js'; -/** - * Grid operations: a state's transmission grid as a control room watches it. Each substation is a - * gauge of its output sized by its capacity, a plant's round and a load's square; branches are as - * wide as their rating, color by loading, carry comets along their flow, and bow apart where they - * run in parallel. The flows come from a DC power flow solved every tick: push demand up or trip a - * line, and overloaded branches trip in turn, flows swing onto what is left, and loads cut off from - * every plant go dark. Each tick eases into place; what trips pulses until it is restored. - */ -import { - Data, - Patch, - elapsed, - field, - reference, - when, - type Hit, - type Motion, - type Scene, -} from '../../latkit.js'; -import { each } from '../../ui/rows.js'; -import type { Stage } from '../../ui/stage.js'; -import { Grid, KINDS } from './grid.js'; -import texas from './texas.json'; - -/** The control room's dark theme: a quiet network, its alarms and flows bright. */ -const LOOK = { - background: '#07090c', - ground: '#10161d', - border: '#3b4552', - text: '#dfe6ee', - hover: '#53a3f2', - comet: '#e8fff4', - warning: '#ffc000', - critical: '#ff6200', - tripped: '#ff1915', -} as const; - -/** A branch's color by its loading, % of its rating: calm blues to 80%, then amber at 90% through - * red where its protection trips it. */ -const LOADING = field('loading') - .colormap({ - kind: 'sequential', - colors: [ - '#2b3e54', - '#2e4760', - '#31506c', - '#345a78', - '#376484', - '#3b6e8f', - '#40799a', - '#4a86a3', - '#6a9eaa', - LOOK.warning, - '#ff8f00', - LOOK.critical, - LOOK.tripped, - ], - }) - .domain(0, 120); - -/** Seconds between ticks of the simulation. */ -const TICK = 0.5; -/** The grid's state as a patch of the `grid` dataset's changing fields. */ -function patchOf(grid: Grid): Patch { - const kind = Float32Array.from(grid.kind, (k, i) => - grid.tripped(i) || (grid.load[i] && grid.fill[i]! < 0.05) ? KINDS.length : k, - ); - return new Patch() - .set( - 'Substation', - { count: grid.n }, - { - fill: grid.fill, - kind, - alarm: Uint8Array.from(kind, (k) => (k === KINDS.length ? 1 : 0)), - mw: Float64Array.from(grid.injection, Math.abs), - }, - ) - .set( - 'Branch', - { count: grid.m }, - { - out: grid.out, - // Overloaded or out. - alarm: Uint8Array.from(grid.loading, (l, j) => (grid.out[j] || l > 1 ? 1 : 0)), - flow: Float64Array.from(grid.flow, (f, j) => - grid.out[j] ? 0 : Math.max(-1, Math.min(1, f / grid.rating[j]!)), - ), - mw: Float64Array.from(grid.flow, Math.abs), - loading: Float64Array.from(grid.loading, (l) => l * 100), - }, - ); -} - -export default async function page(stage: Stage): Promise { - // Labels set in Jura once it has loaded. - const jura = new FontFace( - 'Jura', - `url(${import.meta.env.BASE_URL}fonts/jura-latin-400-normal.woff2)`, - ); - document.fonts.add(await jura.load()); - - const grid = new Grid(); - const s = texas.substations; - const data = new Data('grid').apply( - new Patch() - .set( - 'Substation', - { count: grid.n }, - { - name: s.map((x) => x.name), - lon: Float64Array.from(s, (x) => x.lon), - lat: Float64Array.from(s, (x) => x.lat), - load: grid.load, - capacity: grid.capacity, - }, - ) - .set( - 'Branch', - { count: grid.m }, - { - from: reference( - 'Substation', - Uint32Array.from(texas.branches, (b) => b.from), - ), - to: reference( - 'Substation', - Uint32Array.from(texas.branches, (b) => b.to), - ), - rating: grid.rating, - }, - ), - ); - data.apply(patchOf(grid)); - const border = new Data('state').apply( - new Patch().set( - 'Border', - { count: 1 }, - { - geometry: [[...texas.border, texas.border[0]!] as [number, number][]], - }, - ), - ); - - const state = { comets: true, labels: true, playing: true }; - let motion: Motion = 'system'; - /** What trips pulses: red to dark red and back, twice a second. */ - const pulse = elapsed() - .mul(4 * Math.PI) - .sin() - .colormap({ kind: 'sequential', colors: ['#5c0805', LOOK.tripped] }) - .domain(-1, 1); - const kinds = field('kind').colormap({ - kind: 'categorical', - colors: [...KINDS.map((k) => k.color), LOOK.tripped], - categories: [...KINDS.map((k) => k.label), 'tripped or dark'], - }); - // Comets run with the flow, in degrees a second, faster the nearer its rating, and never on a - // branch out. - const comets = when(field('out').gt(0.5)) - .then(0) - .when(field('flow').lt(0)) - .then(field('flow').mul(1.8).sub(0.45)) - .otherwise(field('flow').mul(1.8).add(0.45)); - /** A substation mark: plants as round gauges, loads as square ones, in one table. */ - const substations = (name: string, load: number, shape: 'ellipse' | 'rounded') => ({ - vertices: { - of: ['grid', 'Substation'] as [string, string], - name, - position: ['lon', 'lat'] as [string, string], - visible: field('load').eq(load), - // Both marks size every row alike, by its capacity, so branches stop at the outline that shows. - radius: field('capacity').sqrt().scale(8, 17).domain(11, 46), - marker: { gauge: { fill: field('fill'), shape } }, - color: kinds, - ring: when(field('alarm').gt(0.5)).then(pulse).otherwise('#00000000'), - shadow: true, - ...(load ? {} : { legend: 'Substations' }), - ...(state.labels ? { label: { text: 'name', size: 12, priority: 'capacity' } } : {}), - }, - }); - const scene = (): Scene => ({ - space: { earth: { projection: 'mercator', surface: LOOK.ground, graticule: false } }, - theme: { - ...theme, - text: LOOK.text, - hover: LOOK.hover, - selected: LOOK.hover, - // Comets are part of the map, in degrees: they zoom with it and stay on their branches. - flow: { color: LOOK.comet, spacing: 1.35, length: 0.6, width: 6, units: true }, - font: 'Jura, sans-serif', - fontSize: 12, - halo: 3, - glow: 6, - laneBow: 0.22, - }, - ease: 0.45, - // Easing, pulses, and comets hold still where the system asks for reduced motion, unless set. - controls: { motion }, - marks: [ - { lines: { of: ['state', 'Border'], width: 1.25, color: LOOK.border } }, - { - edges: { - of: ['grid', 'Branch'], - ends: ['from', 'to'], - route: 'straight', - width: field('rating').scale(1.6, 3.6).domain(300, 800), - color: when(field('alarm').gt(0.5)).then(pulse).otherwise(LOADING), - dash: 'out', - legend: 'Loading, % of rating', - ...(state.comets ? { flow: comets } : {}), - }, - }, - substations('plants', 0, 'ellipse'), - substations('loads', 1, 'rounded'), - ], - }); - - const view = stage.view(scene(), { - label: 'A state’s grid: hover or click a substation or branch to inspect it', - }); - view.setData(border); - view.setData(data); - const { xMin, xMax, yMin, yMax } = texas.bounds; - view.fit({ bounds: { min: [xMin, yMin, 0], max: [xMax, yMax, 0] } }); - const restyle = () => view.show(scene()); - - // ── Grid ── - const gridSection = stage.section('Grid'); - gridSection.note('Simulated data'); - gridSection.range( - 'Demand', - { - min: 0.5, - max: 1.2, - step: 0.01, - value: grid.demand, - format: (v) => `${Math.round(v * 100)}%`, - }, - (v) => (grid.demand = v), - ); - gridSection.actions([ - { label: 'Trip a line', key: 't', run: () => (grid.tripWorst(), show()) }, - { label: 'Restore', key: 'r', run: () => (grid.restore(), show()) }, - ]); - gridSection.toggle('Run', ' ', state.playing, (on) => (state.playing = on)); - const totals = gridSection.note(); - - // ── Layers ── - const layers = stage.section('Layers'); - layers.toggle('Flow', 'c', state.comets, (on) => ((state.comets = on), restyle())); - layers.toggle('Labels', 'l', state.labels, (on) => ((state.labels = on), restyle())); - layers.choice( - 'Motion', - [ - ['System', 'system'], - ['Full', 'full'], - ['Reduced', 'reduced'], - ] as const, - motion, - (next) => ((motion = next), restyle()), - ); - - // ── Inspector ── - const inspector = stage.section('Inspector'); - const described = inspector.note('Select a substation or line.'); - let hovered: Hit | null = null; - const inspect = () => { - const row = hovered?.row ?? each(view.selection)[0]; - if (!row) return void (described.textContent = 'Select a substation or line.'); - const value = (name: string) => data.get(row.table, row.id, name) as number; - const i = Number(row.id); - if (row.table === 'Substation') { - const what = grid.load[i] ? 'load' : KINDS[grid.kind[i]!]!.label; - const status = grid.tripped(i) - ? 'tripped' - : grid.load[i] && grid.fill[i]! < 0.05 - ? 'dark' - : `${Math.round(grid.fill[i]! * 100)}%`; - described.textContent = `${s[i]!.name} · ${what} · ${Math.round(value('mw'))} of ${Math.round(grid.capacity[i]!)} MW · ${status}`; - } else { - const b = texas.branches[i]!; - const [from, to] = grid.flow[i]! >= 0 ? [b.from, b.to] : [b.to, b.from]; - described.textContent = grid.out[i] - ? `${s[b.from]!.name}–${s[b.to]!.name} · out` - : `${s[from]!.name} → ${s[to]!.name} · ${Math.round(value('mw'))} MW · ${Math.round(value('loading'))}% of ${b.pmax} MW`; - } - }; - view.on('hover', ({ hit }) => ((hovered = hit), inspect())); - view.on('select', inspect); - - const meter = stage.meter(view); - meter.note(() => `${grid.trips} trips`); - - /** Shows the grid as it stands. */ - function show(): void { - data.apply(patchOf(grid)); - view.setData(data); - const served = grid.injection.reduce((sum, p, i) => sum + (grid.load[i] ? -p : 0), 0); - const out = grid.out.reduce((a, b) => a + b, 0); - totals.textContent = `${Math.round(served).toLocaleString()} MW served · ${out} branches out · ${grid.dark} loads dark`; - inspect(); - } - - let carry = 0; - stage.frame((dt) => { - if (!state.playing) return; - carry += dt; - if (carry < TICK) return; - carry -= TICK; - grid.step(TICK); - show(); - }); - show(); - stage.detail(`${grid.n} substations · ${grid.m} branches, ${texas.name}`); -} diff --git a/npm/examples/src/examples/power-flow/texas.json b/npm/examples/src/examples/power-flow/texas.json deleted file mode 100644 index f7923a2e..00000000 --- a/npm/examples/src/examples/power-flow/texas.json +++ /dev/null @@ -1,948 +0,0 @@ -{ - "name": "Texas", - "bounds": { "xMax": -93, "xMin": -107, "yMax": 37, "yMin": 25.5 }, - "substations": [ - { - "name": "Abilene", - "lon": -99.66, - "lat": 32.51, - "category": "Load", - "units": [104.33, 104.33, 104.33] - }, - { - "name": "Armstrong", - "lon": -97.6, - "lat": 27, - "category": "Wind", - "units": [108.8, 108.8, 108.8, 108.8, 108.8] - }, - { - "name": "Austin", - "lon": -97.78, - "lat": 30.29, - "category": "Load", - "units": [158.75, 158.75, 158.75, 158.75] - }, - { - "name": "Bastrop", - "lon": -97.27, - "lat": 30.15, - "category": "Solar PV", - "units": [240, 240, 240, 240] - }, - { - "name": "Brownsville", - "lon": -97.32, - "lat": 25.94, - "category": "Load", - "units": [88.67, 88.67, 88.67] - }, - { "name": "Bryan", "lon": -96.46, "lat": 30.79, "category": "Gas Turbine", "units": [85, 85] }, - { - "name": "College Station", - "lon": -96.33, - "lat": 30.61, - "category": "Load", - "units": [101.5, 101.5, 101.5, 101.5] - }, - { - "name": "Corpus Christi", - "lon": -97.43, - "lat": 27.81, - "category": "Load", - "units": [85.75, 85.75, 85.75, 85.75] - }, - { - "name": "Dallas", - "lon": -96.68, - "lat": 32.71, - "category": "Load", - "units": [144.29, 144.29, 144.29, 144.29, 144.29, 144.29, 144.29] - }, - { - "name": "El Campo", - "lon": -96.28, - "lat": 29.21, - "category": "Gas Combined Cycle", - "units": [103.75, 103.75, 103.75, 103.75, 103.75, 103.75, 103.75, 103.75] - }, - { - "name": "Fluvanna", - "lon": -101.15, - "lat": 33.05, - "category": "Wind", - "units": [147, 147, 147] - }, - { - "name": "Franklin", - "lon": -96.49, - "lat": 31.18, - "category": "Coal-fired Steam", - "units": [483.33, 483.33, 483.33] - }, - { - "name": "Ft Worth", - "lon": -97.31, - "lat": 32.53, - "category": "Load", - "units": [153, 153, 153, 153, 153, 153] - }, - { - "name": "Galveston", - "lon": -94.78, - "lat": 29.31, - "category": "Load", - "units": [47, 47, 47, 47] - }, - { - "name": "Houston", - "lon": -95.4, - "lat": 29.61, - "category": "Load", - "units": [170.12, 170.12, 170.12, 170.12, 170.12, 170.12, 170.12, 170.12] - }, - { - "name": "Katy", - "lon": -95.83, - "lat": 29.74, - "category": "Load", - "units": [138.2, 138.2, 138.2, 138.2, 138.2] - }, - { - "name": "Kerrville", - "lon": -99.16, - "lat": 30.04, - "category": "Gas Combined Cycle", - "units": [72.5, 72.5, 72.5, 72.5, 72.5, 72.5, 72.5, 72.5] - }, - { "name": "Lubbock", "lon": -101.39, "lat": 33.66, "category": "Load", "units": [88, 88, 88] }, - { - "name": "McCamey", - "lon": -102.24, - "lat": 31.21, - "category": "Solar PV", - "units": [201.67, 201.67, 201.67] - }, - { - "name": "McKinney", - "lon": -96.6, - "lat": 33.17, - "category": "Gas Combined Cycle", - "units": [206, 206, 206, 206, 206] - }, - { - "name": "Mission", - "lon": -98.4, - "lat": 26.21, - "category": "Gas Turbine", - "units": [73.75, 73.75, 73.75, 73.75] - }, - { - "name": "Monahans", - "lon": -102.96, - "lat": 31.58, - "category": "Wind", - "units": [240, 240, 240, 240] - }, - { - "name": "Odessa", - "lon": -102.33, - "lat": 31.84, - "category": "Load", - "units": [59.33, 59.33, 59.33] - }, - { - "name": "O'Donnell", - "lon": -101.65, - "lat": 32.93, - "category": "Wind", - "units": [305, 305, 305, 305, 305, 305] - }, - { - "name": "Pasadena", - "lon": -95.18, - "lat": 29.72, - "category": "Gas Turbine", - "units": [112.5, 112.5, 112.5, 112.5] - }, - { - "name": "Rockdale", - "lon": -97.06, - "lat": 30.57, - "category": "Coal-fired Steam", - "units": [290, 290] - }, - { - "name": "San Antonio", - "lon": -98.61, - "lat": 29.59, - "category": "Load", - "units": [171, 171, 171, 171, 171] - }, - { - "name": "Snyder", - "lon": -100.95, - "lat": 32.75, - "category": "Gas Turbine", - "units": [160, 160] - }, - { - "name": "Tyler", - "lon": -95.32, - "lat": 32.24, - "category": "Load", - "units": [97.33, 97.33, 97.33] - }, - { - "name": "Waco", - "lon": -97.1, - "lat": 31.62, - "category": "Load", - "units": [106, 106, 106, 106] - }, - { - "name": "Wadsworth", - "lon": -96.05, - "lat": 28.8, - "category": "Nuclear Steam", - "units": [1050, 1050] - }, - { - "name": "Wichita Falls", - "lon": -98.49, - "lat": 33.91, - "category": "Load", - "units": [43, 43, 43] - } - ], - "branches": [ - { "from": 0, "to": 18, "pmax": 500 }, - { "from": 0, "to": 22, "pmax": 500 }, - { "from": 0, "to": 27, "pmax": 500 }, - { "from": 0, "to": 27, "pmax": 500 }, - { "from": 0, "to": 29, "pmax": 500 }, - { "from": 1, "to": 7, "pmax": 500 }, - { "from": 2, "to": 3, "pmax": 800 }, - { "from": 2, "to": 11, "pmax": 500 }, - { "from": 2, "to": 25, "pmax": 500 }, - { "from": 2, "to": 25, "pmax": 500 }, - { "from": 2, "to": 29, "pmax": 500 }, - { "from": 4, "to": 1, "pmax": 500 }, - { "from": 4, "to": 7, "pmax": 500 }, - { "from": 5, "to": 28, "pmax": 500 }, - { "from": 6, "to": 5, "pmax": 500 }, - { "from": 6, "to": 5, "pmax": 500 }, - { "from": 6, "to": 11, "pmax": 500 }, - { "from": 7, "to": 9, "pmax": 500 }, - { "from": 7, "to": 30, "pmax": 500 }, - { "from": 8, "to": 12, "pmax": 500 }, - { "from": 8, "to": 12, "pmax": 500 }, - { "from": 8, "to": 19, "pmax": 500 }, - { "from": 8, "to": 29, "pmax": 500 }, - { "from": 9, "to": 15, "pmax": 500 }, - { "from": 9, "to": 26, "pmax": 500 }, - { "from": 10, "to": 17, "pmax": 300 }, - { "from": 10, "to": 23, "pmax": 500 }, - { "from": 11, "to": 5, "pmax": 500 }, - { "from": 11, "to": 8, "pmax": 500 }, - { "from": 11, "to": 8, "pmax": 500 }, - { "from": 12, "to": 0, "pmax": 500 }, - { "from": 12, "to": 0, "pmax": 500 }, - { "from": 12, "to": 27, "pmax": 500 }, - { "from": 12, "to": 29, "pmax": 500 }, - { "from": 13, "to": 15, "pmax": 500 }, - { "from": 13, "to": 24, "pmax": 800 }, - { "from": 14, "to": 6, "pmax": 500 }, - { "from": 14, "to": 9, "pmax": 500 }, - { "from": 14, "to": 9, "pmax": 500 }, - { "from": 14, "to": 24, "pmax": 500 }, - { "from": 15, "to": 3, "pmax": 800 }, - { "from": 15, "to": 14, "pmax": 500 }, - { "from": 15, "to": 14, "pmax": 500 }, - { "from": 15, "to": 25, "pmax": 500 }, - { "from": 16, "to": 2, "pmax": 500 }, - { "from": 16, "to": 18, "pmax": 500 }, - { "from": 17, "to": 23, "pmax": 300 }, - { "from": 17, "to": 31, "pmax": 300 }, - { "from": 19, "to": 12, "pmax": 500 }, - { "from": 19, "to": 31, "pmax": 300 }, - { "from": 20, "to": 4, "pmax": 500 }, - { "from": 20, "to": 7, "pmax": 500 }, - { "from": 20, "to": 26, "pmax": 300 }, - { "from": 21, "to": 18, "pmax": 500 }, - { "from": 22, "to": 18, "pmax": 500 }, - { "from": 22, "to": 21, "pmax": 500 }, - { "from": 22, "to": 23, "pmax": 500 }, - { "from": 24, "to": 6, "pmax": 500 }, - { "from": 25, "to": 5, "pmax": 500 }, - { "from": 25, "to": 11, "pmax": 500 }, - { "from": 25, "to": 29, "pmax": 500 }, - { "from": 26, "to": 2, "pmax": 500 }, - { "from": 26, "to": 2, "pmax": 500 }, - { "from": 26, "to": 16, "pmax": 500 }, - { "from": 26, "to": 16, "pmax": 500 }, - { "from": 27, "to": 10, "pmax": 800 }, - { "from": 27, "to": 23, "pmax": 800 }, - { "from": 28, "to": 8, "pmax": 500 }, - { "from": 30, "to": 9, "pmax": 500 }, - { "from": 30, "to": 9, "pmax": 500 }, - { "from": 30, "to": 13, "pmax": 800 }, - { "from": 30, "to": 26, "pmax": 500 } - ], - "border": [ - [-103.98018, 32.00012], - [-103.06451, 32.00042], - [-103.06463, 32.9999], - [-103.04365, 34.00364], - [-103.0409, 35.8961], - [-103.04171, 36.50044], - [-100.00041, 36.4997], - [-100.00028, 34.56045], - [-99.9589, 34.57127], - [-99.90349, 34.55836], - [-99.86895, 34.52762], - [-99.8329, 34.50007], - [-99.80286, 34.46567], - [-99.7656, 34.43749], - [-99.72703, 34.41055], - [-99.70041, 34.3847], - [-99.65419, 34.37652], - [-99.60003, 34.37469], - [-99.58006, 34.41665], - [-99.52979, 34.41145], - [-99.48722, 34.39796], - [-99.43827, 34.37116], - [-99.39568, 34.39769], - [-99.38923, 34.44686], - [-99.34013, 34.4301], - [-99.30926, 34.4097], - [-99.26128, 34.40351], - [-99.24354, 34.37071], - [-99.2172, 34.34151], - [-99.2097, 34.2989], - [-99.19384, 34.26031], - [-99.18978, 34.21436], - [-99.13808, 34.21923], - [-99.08892, 34.20847], - [-99.04446, 34.19811], - [-99.00224, 34.20937], - [-98.95128, 34.21219], - [-98.91833, 34.18183], - [-98.87292, 34.16658], - [-98.81295, 34.15844], - [-98.77808, 34.13949], - [-98.74197, 34.12553], - [-98.69007, 34.13316], - [-98.64807, 34.16444], - [-98.59979, 34.16057], - [-98.56661, 34.13744], - [-98.53626, 34.10735], - [-98.50415, 34.07229], - [-98.44903, 34.07346], - [-98.40972, 34.09251], - [-98.39403, 34.13191], - [-98.36562, 34.15699], - [-98.30021, 34.13458], - [-98.24795, 34.13072], - [-98.20173, 34.11706], - [-98.15435, 34.12273], - [-98.13082, 34.15053], - [-98.09237, 34.13215], - [-98.11942, 34.08447], - [-98.10189, 34.05097], - [-98.08403, 34.01437], - [-98.04112, 33.99346], - [-97.99086, 33.99887], - [-97.94757, 33.99105], - [-97.96035, 33.95193], - [-97.96875, 33.90602], - [-97.93895, 33.87947], - [-97.89674, 33.85799], - [-97.84378, 33.85681], - [-97.81764, 33.88099], - [-97.78084, 33.89445], - [-97.76277, 33.9344], - [-97.7206, 33.94395], - [-97.69918, 33.9757], - [-97.65621, 33.98949], - [-97.61607, 33.96972], - [-97.59503, 33.93736], - [-97.58108, 33.89968], - [-97.53421, 33.90573], - [-97.48772, 33.91751], - [-97.45106, 33.89168], - [-97.46149, 33.84956], - [-97.4268, 33.81864], - [-97.37469, 33.81855], - [-97.34721, 33.84511], - [-97.33188, 33.88448], - [-97.28539, 33.86824], - [-97.24505, 33.90322], - [-97.18658, 33.90155], - [-97.16663, 33.84731], - [-97.1998, 33.8274], - [-97.1904, 33.78115], - [-97.17463, 33.74504], - [-97.13392, 33.71813], - [-97.08737, 33.74114], - [-97.0951, 33.79217], - [-97.04815, 33.81746], - [-97.0872, 33.84137], - [-97.02144, 33.84638], - [-96.98384, 33.89095], - [-96.99502, 33.93203], - [-96.95178, 33.94474], - [-96.901, 33.93878], - [-96.88301, 33.86802], - [-96.85059, 33.84721], - [-96.79428, 33.86889], - [-96.77067, 33.82962], - [-96.71242, 33.83163], - [-96.68473, 33.86291], - [-96.67345, 33.91228], - [-96.62829, 33.89448], - [-96.59011, 33.88066], - [-96.62902, 33.85241], - [-96.59293, 33.83092], - [-96.55106, 33.81925], - [-96.51569, 33.79516], - [-96.48634, 33.77296], - [-96.43199, 33.7791], - [-96.40822, 33.75054], - [-96.36959, 33.71681], - [-96.34686, 33.68622], - [-96.31223, 33.7063], - [-96.29811, 33.75912], - [-96.22902, 33.74802], - [-96.1786, 33.76043], - [-96.16076, 33.79591], - [-96.14923, 33.83709], - [-96.09936, 33.83047], - [-96.04883, 33.83647], - [-96.00405, 33.84524], - [-95.94733, 33.85892], - [-95.89521, 33.86758], - [-95.85683, 33.85014], - [-95.81557, 33.85985], - [-95.77024, 33.8452], - [-95.75794, 33.89075], - [-95.71218, 33.88552], - [-95.66548, 33.909], - [-95.61637, 33.92763], - [-95.56378, 33.93294], - [-95.55122, 33.88971], - [-95.50662, 33.89672], - [-95.47047, 33.8816], - [-95.42768, 33.87046], - [-95.36257, 33.8669], - [-95.32994, 33.88633], - [-95.28893, 33.87307], - [-95.27596, 33.91128], - [-95.25, 33.93886], - [-95.21695, 33.96267], - [-95.1715, 33.94672], - [-95.13016, 33.9368], - [-95.11039, 33.90541], - [-95.08241, 33.87988], - [-95.06948, 33.91759], - [-95.05057, 33.8783], - [-95.01574, 33.85773], - [-94.9658, 33.8487], - [-94.94787, 33.80836], - [-94.91287, 33.77948], - [-94.87637, 33.75733], - [-94.82795, 33.74088], - [-94.78007, 33.74447], - [-94.74767, 33.72515], - [-94.71953, 33.68886], - [-94.67177, 33.69331], - [-94.65492, 33.65993], - [-94.62204, 33.68253], - [-94.57927, 33.67082], - [-94.54302, 33.65518], - [-94.56997, 33.62847], - [-94.52813, 33.64125], - [-94.48836, 33.62741], - [-94.44864, 33.64277], - [-94.46945, 33.60732], - [-94.43004, 33.59112], - [-94.40067, 33.5698], - [-94.36987, 33.54566], - [-94.33956, 33.56767], - [-94.29497, 33.57912], - [-94.24278, 33.58971], - [-94.21921, 33.5561], - [-94.18391, 33.59469], - [-94.15263, 33.57592], - [-94.12761, 33.55072], - [-94.09744, 33.57374], - [-94.05658, 33.56093], - [-94.04278, 32.02483], - [-94.02289, 31.99045], - [-93.99571, 31.95414], - [-93.97171, 31.92039], - [-93.92865, 31.91265], - [-93.89719, 31.89102], - [-93.88288, 31.85467], - [-93.87084, 31.81378], - [-93.83561, 31.79449], - [-93.8317, 31.74551], - [-93.81669, 31.70792], - [-93.82646, 31.66692], - [-93.81699, 31.62518], - [-93.83492, 31.58621], - [-93.81397, 31.55056], - [-93.78084, 31.52538], - [-93.73524, 31.5135], - [-93.75137, 31.47907], - [-93.72193, 31.45345], - [-93.70468, 31.4189], - [-93.67412, 31.39768], - [-93.64826, 31.37086], - [-93.66902, 31.33883], - [-93.68728, 31.30607], - [-93.65869, 31.28162], - [-93.61992, 31.26518], - [-93.59839, 31.2311], - [-93.60244, 31.18254], - [-93.54998, 31.18663], - [-93.54474, 31.13607], - [-93.55818, 31.09798], - [-93.52765, 31.07451], - [-93.50903, 31.03289], - [-93.53918, 31.00821], - [-93.57336, 30.99183], - [-93.54731, 30.96475], - [-93.52784, 30.93307], - [-93.55856, 30.91324], - [-93.5706, 30.87519], - [-93.55363, 30.83514], - [-93.57468, 30.80137], - [-93.5921, 30.76368], - [-93.61903, 30.74034], - [-93.62665, 30.68836], - [-93.65974, 30.66205], - [-93.68421, 30.63188], - [-93.69222, 30.58571], - [-93.72616, 30.56952], - [-93.72048, 30.52433], - [-93.70976, 30.48265], - [-93.69758, 30.44223], - [-93.72755, 30.41872], - [-93.75261, 30.39275], - [-93.76365, 30.34604], - [-93.74565, 30.31174], - [-93.71227, 30.29233], - [-93.70554, 30.2465], - [-93.72126, 30.20961], - [-93.70424, 30.17418], - [-93.69285, 30.13521], - [-93.71549, 30.10053], - [-93.70058, 30.06368], - [-93.73745, 30.03728], - [-93.76724, 30.00338], - [-93.79444, 29.97477], - [-93.81729, 29.91842], - [-93.83837, 29.88285], - [-93.86357, 29.85718], - [-93.91367, 29.82719], - [-93.9265, 29.78956], - [-93.89386, 29.76729], - [-93.87002, 29.73548], - [-93.83741, 29.68988], - [-93.81435, 29.59658], - [-93.85798, 29.60509], - [-93.90604, 29.625], - [-94.03357, 29.625], - [-94.12518, 29.59556], - [-94.37035, 29.50023], - [-94.41925, 29.48345], - [-94.49945, 29.45234], - [-94.62519, 29.39766], - [-94.6239, 29.34314], - [-94.64242, 29.30014], - [-94.6803, 29.2776], - [-94.74117, 29.25958], - [-94.77269, 29.23172], - [-94.84015, 29.19835], - [-94.91217, 29.15932], - [-94.96366, 29.12524], - [-95.0002, 29.10445], - [-95.0864, 29.06824], - [-95.10653, 29.01493], - [-95.13947, 28.9917], - [-95.18084, 28.94893], - [-95.22723, 28.91038], - [-95.29081, 28.87164], - [-95.31542, 28.84221], - [-95.35347, 28.81786], - [-95.40116, 28.80597], - [-95.49389, 28.76534], - [-95.53378, 28.75026], - [-95.77785, 28.62527], - [-95.87524, 28.58279], - [-96.00024, 28.53207], - [-96.07833, 28.5], - [-96.18224, 28.45204], - [-96.23546, 28.42147], - [-96.26707, 28.38823], - [-96.30353, 28.36502], - [-96.34181, 28.34238], - [-96.37525, 28.29076], - [-96.42581, 28.26869], - [-96.531, 28.22116], - [-96.64075, 28.15539], - [-96.74625, 28.07962], - [-96.8442, 28.0003], - [-96.87526, 27.96992], - [-96.90874, 27.93409], - [-96.93855, 27.9012], - [-96.98428, 27.80378], - [-97.02185, 27.78336], - [-97.05022, 27.75031], - [-97.12527, 27.6406], - [-97.1651, 27.5683], - [-97.19982, 27.50032], - [-97.25026, 27.39231], - [-97.28834, 27.27758], - [-97.30936, 27.19611], - [-97.31982, 27.11002], - [-97.32315, 27.05115], - [-97.32115, 27.00033], - [-97.31388, 26.91313], - [-97.30042, 26.83461], - [-97.28273, 26.76342], - [-97.25171, 26.66187], - [-97.22443, 26.59769], - [-97.18457, 26.47585], - [-97.1565, 26.37535], - [-97.12525, 26.24828], - [-97.1107, 26.14537], - [-97.09739, 26.09351], - [-97.09414, 26.04304], - [-97.09112, 25.97384], - [-97.14665, 25.97097], - [-97.18708, 25.95848], - [-97.23987, 25.95498], - [-97.28964, 25.95403], - [-97.32057, 25.9298], - [-97.3615, 25.91587], - [-97.35669, 25.86735], - [-97.38206, 25.83976], - [-97.41131, 25.86134], - [-97.45669, 25.86834], - [-97.49669, 25.87968], - [-97.52734, 25.90261], - [-97.54974, 25.93622], - [-97.58301, 25.95483], - [-97.61767, 25.97996], - [-97.64938, 26.01086], - [-97.68567, 26.02675], - [-97.7349, 26.03202], - [-97.77683, 26.04397], - [-97.81805, 26.0564], - [-97.85933, 26.06696], - [-97.91124, 26.05645], - [-97.96734, 26.05178], - [-98.00739, 26.06216], - [-98.03924, 26.04128], - [-98.07507, 26.05987], - [-98.12584, 26.06179], - [-98.17213, 26.07114], - [-98.22067, 26.07647], - [-98.26451, 26.08551], - [-98.2906, 26.12066], - [-98.3352, 26.13762], - [-98.35768, 26.16883], - [-98.40225, 26.17952], - [-98.44204, 26.19875], - [-98.47178, 26.22019], - [-98.51485, 26.22722], - [-98.55458, 26.24738], - [-98.59915, 26.25761], - [-98.63418, 26.24261], - [-98.67785, 26.25691], - [-98.71941, 26.2723], - [-98.75097, 26.29632], - [-98.77703, 26.32573], - [-98.79821, 26.36017], - [-98.85342, 26.36502], - [-98.89673, 26.35333], - [-98.9234, 26.38137], - [-98.96189, 26.39837], - [-99.01064, 26.39213], - [-99.04547, 26.40982], - [-99.08513, 26.39878], - [-99.11085, 26.42628], - [-99.09198, 26.47238], - [-99.12778, 26.5252], - [-99.17123, 26.54196], - [-99.17806, 26.62055], - [-99.20052, 26.65644], - [-99.20891, 26.72476], - [-99.24002, 26.74585], - [-99.26221, 26.81567], - [-99.28047, 26.85805], - [-99.32588, 26.87738], - [-99.3373, 26.92276], - [-99.37915, 26.93449], - [-99.38737, 26.9824], - [-99.41548, 27.01724], - [-99.45232, 27.06267], - [-99.42921, 27.09098], - [-99.43827, 27.14479], - [-99.43278, 27.20184], - [-99.44155, 27.24992], - [-99.48906, 27.26148], - [-99.50095, 27.30601], - [-99.50778, 27.35425], - [-99.49271, 27.39195], - [-99.4951, 27.45152], - [-99.48129, 27.48847], - [-99.52614, 27.49659], - [-99.52192, 27.54409], - [-99.5362, 27.59168], - [-99.57863, 27.60229], - [-99.59404, 27.63857], - [-99.63893, 27.62676], - [-99.66184, 27.65575], - [-99.71151, 27.65836], - [-99.75764, 27.71211], - [-99.79634, 27.73559], - [-99.81312, 27.774], - [-99.85088, 27.79397], - [-99.87789, 27.82535], - [-99.90149, 27.86416], - [-99.91746, 27.91797], - [-99.93854, 27.95406], - [-99.96277, 27.98354], - [-100.00863, 28.02396], - [-100.01791, 28.06479], - [-100.05312, 28.08473], - [-100.07547, 28.12488], - [-100.11963, 28.15559], - [-100.16059, 28.16816], - [-100.1965, 28.19022], - [-100.21756, 28.22693], - [-100.2676, 28.25027], - [-100.29411, 28.2804], - [-100.3142, 28.34586], - [-100.34187, 28.38495], - [-100.33619, 28.43018], - [-100.3575, 28.46364], - [-100.33381, 28.49925], - [-100.38826, 28.51486], - [-100.41127, 28.5507], - [-100.42986, 28.59644], - [-100.44553, 28.63714], - [-100.47964, 28.65523], - [-100.51005, 28.69072], - [-100.50761, 28.7406], - [-100.53005, 28.77042], - [-100.54217, 28.80847], - [-100.57473, 28.82879], - [-100.59104, 28.86305], - [-100.61422, 28.90172], - [-100.63886, 28.92762], - [-100.64865, 28.97742], - [-100.66021, 29.0315], - [-100.6752, 29.10074], - [-100.70997, 29.11968], - [-100.73912, 29.14166], - [-100.7726, 29.16837], - [-100.78552, 29.22814], - [-100.81628, 29.25386], - [-100.85584, 29.27555], - [-100.88684, 29.30785], - [-100.9195, 29.32532], - [-100.94897, 29.34725], - [-100.99561, 29.3634], - [-101.01926, 29.39091], - [-101.04336, 29.42988], - [-101.07718, 29.46419], - [-101.13004, 29.47842], - [-101.17166, 29.50395], - [-101.22742, 29.52235], - [-101.25038, 29.55662], - [-101.24733, 29.61848], - [-101.27788, 29.59081], - [-101.31171, 29.61537], - [-101.34602, 29.66224], - [-101.37277, 29.69943], - [-101.39629, 29.72706], - [-101.40232, 29.77119], - [-101.43815, 29.75092], - [-101.4558, 29.78805], - [-101.50322, 29.76458], - [-101.53792, 29.78806], - [-101.57581, 29.76932], - [-101.6269, 29.77076], - [-101.67456, 29.76523], - [-101.71973, 29.77574], - [-101.77383, 29.78868], - [-101.81252, 29.80979], - [-101.8665, 29.79801], - [-101.91572, 29.79521], - [-101.96338, 29.80015], - [-102.01523, 29.79749], - [-102.055, 29.78703], - [-102.10138, 29.79361], - [-102.14357, 29.80332], - [-102.17996, 29.82638], - [-102.22353, 29.84053], - [-102.2615, 29.85324], - [-102.29587, 29.87361], - [-102.34309, 29.86524], - [-102.36293, 29.83149], - [-102.3772, 29.78989], - [-102.40454, 29.76518], - [-102.44738, 29.7775], - [-102.49799, 29.78145], - [-102.53424, 29.75494], - [-102.57057, 29.771], - [-102.61243, 29.74827], - [-102.66024, 29.73563], - [-102.68904, 29.70504], - [-102.70865, 29.66699], - [-102.73657, 29.64183], - [-102.74633, 29.59265], - [-102.77342, 29.56687], - [-102.79201, 29.533], - [-102.80727, 29.49429], - [-102.82564, 29.45794], - [-102.83344, 29.4105], - [-102.84532, 29.36473], - [-102.88437, 29.34795], - [-102.88818, 29.29336], - [-102.90638, 29.26008], - [-102.87175, 29.24178], - [-102.8951, 29.20852], - [-102.93251, 29.19474], - [-102.97553, 29.1863], - [-103.00251, 29.14996], - [-103.02759, 29.11602], - [-103.06418, 29.09148], - [-103.08551, 29.05996], - [-103.1015, 29.0181], - [-103.12358, 28.9843], - [-103.16393, 28.9722], - [-103.20241, 28.9872], - [-103.25042, 28.98043], - [-103.28694, 29.00127], - [-103.31434, 29.02606], - [-103.3596, 29.01864], - [-103.40105, 29.03106], - [-103.43371, 29.0522], - [-103.47135, 29.06673], - [-103.49364, 29.10795], - [-103.5228, 29.13465], - [-103.55369, 29.158], - [-103.60618, 29.16027], - [-103.65363, 29.16342], - [-103.68857, 29.17857], - [-103.72872, 29.19288], - [-103.75287, 29.21949], - [-103.78168, 29.24128], - [-103.80917, 29.26594], - [-103.85431, 29.28147], - [-103.89929, 29.28741], - [-103.95003, 29.29535], - [-103.99166, 29.30381], - [-104.0375, 29.31971], - [-104.06675, 29.341], - [-104.09775, 29.36164], - [-104.13142, 29.37998], - [-104.16744, 29.39474], - [-104.18611, 29.4283], - [-104.2177, 29.45656], - [-104.23382, 29.49365], - [-104.26885, 29.51358], - [-104.31148, 29.5232], - [-104.3625, 29.53588], - [-104.39427, 29.55518], - [-104.45202, 29.60353], - [-104.48614, 29.62866], - [-104.5176, 29.64739], - [-104.5441, 29.68069], - [-104.55245, 29.72371], - [-104.56961, 29.75709], - [-104.589, 29.79566], - [-104.60849, 29.82795], - [-104.63012, 29.86452], - [-104.65585, 29.88921], - [-104.67992, 29.92118], - [-104.6855, 29.97084], - [-104.69325, 30.01913], - [-104.70479, 30.05832], - [-104.68553, 30.09321], - [-104.69671, 30.13446], - [-104.68798, 30.17639], - [-104.70248, 30.21278], - [-104.71895, 30.24709], - [-104.74958, 30.26697], - [-104.76709, 30.30371], - [-104.7948, 30.32696], - [-104.82258, 30.35062], - [-104.84561, 30.37921], - [-104.85663, 30.42152], - [-104.86964, 30.4592], - [-104.87196, 30.51109], - [-104.8877, 30.54673], - [-104.90955, 30.58514], - [-104.95082, 30.60405], - [-104.9795, 30.6286], - [-104.99385, 30.66852], - [-105.03196, 30.68099], - [-105.06688, 30.69926], - [-105.09962, 30.72195], - [-105.12766, 30.74785], - [-105.15619, 30.76977], - [-105.19074, 30.7857], - [-105.22744, 30.79919], - [-105.27169, 30.80938], - [-105.31909, 30.81226], - [-105.34575, 30.84061], - [-105.38732, 30.85373], - [-105.40376, 30.89211], - [-105.44116, 30.90845], - [-105.47079, 30.93124], - [-105.49681, 30.95542], - [-105.52708, 30.97818], - [-105.5701, 31.0082], - [-105.58499, 31.05696], - [-105.6271, 31.09841], - [-105.71053, 31.13697], - [-105.74193, 31.1646], - [-105.78003, 31.18274], - [-105.81838, 31.23015], - [-105.85033, 31.26497], - [-105.87639, 31.29167], - [-105.90739, 31.31221], - [-105.94775, 31.33959], - [-106.00492, 31.39246], - [-106.07531, 31.39762], - [-106.10862, 31.42237], - [-106.15693, 31.43736], - [-106.20012, 31.46241], - [-106.22476, 31.50093], - [-106.24657, 31.54166], - [-106.28079, 31.5621], - [-106.30089, 31.60962], - [-106.33444, 31.66356], - [-106.37237, 31.71257], - [-106.41568, 31.75089], - [-106.46756, 31.75921], - [-106.51369, 31.77071], - [-106.536, 31.79851], - [-106.57724, 31.81041], - [-106.60194, 31.8396], - [-106.63592, 31.86623], - [-106.62557, 31.91243], - [-106.61432, 31.95161], - [-106.63954, 31.98034], - [-106.0829, 32.00265], - [-104.84774, 32.00042], - [-104.77211, 31.99749], - [-103.98018, 32.00012] - ] -} diff --git a/npm/examples/src/examples/soft-body/index.ts b/npm/examples/src/examples/soft-body/index.ts deleted file mode 100644 index 0b453da9..00000000 --- a/npm/examples/src/examples/soft-body/index.ts +++ /dev/null @@ -1,464 +0,0 @@ -import { theme } from '../../ui/theme.js'; -import { - Data, - Patch, - field, - reference, - units, - type ColormapName, - type Scene, -} from '../../latkit.js'; -import type { Stage } from '../../ui/stage.js'; -import { Body, ROOM, STEP, type Controls } from './body.js'; -import { loadMesh } from './mesh.js'; - -/** Seconds of frames kept to replay and scrub. */ -const HISTORY = 20; -/** Seconds of simulation between recorded frames, whatever the display's refresh rate. */ -const RECORD = 1 / 60; -/** Seconds the plot shows. */ -const WINDOW = 12; -/** What the bunny can be painted by. Each domain starts below rest, so a bunny at rest draws bright. */ -const PAINTS: Record< - string, - { label: string; field: string; colormap: ColormapName; domain: [number, number]; legend: string } -> = { - solid: { - label: 'Pearl', - field: 'strain', - colormap: 'turbo', - domain: [-0.05, 0.35], - legend: '', - }, - strain: { - label: 'Strain', - field: 'strain', - colormap: 'turbo', - domain: [-0.05, 0.35], - legend: 'Strain 0 – 0.35 m', - }, - speed: { - label: 'Speed', - field: 'speed', - colormap: 'plasma', - domain: [-1, 9], - legend: 'Speed 0 – 9 m/s', - }, - height: { - label: 'Height', - field: 'z', - colormap: 'viridis', - domain: [-0.5, 4], - legend: 'Height 0 – 4 m', - }, -}; -const SERIES = [ - { field: 'height', label: 'Center height', color: '#66c7ff' }, - { field: 'speed', label: 'Mean speed', color: '#ff8f52' }, - { field: 'volume', label: 'Volume', color: '#cc9eff' }, -]; -const THEME = { - ...theme, - shading: 0.7, -} as const; - -// A soft floor footprint avoids self-shadow acne on deforming triangles in the packaged renderer. -// This is a custom marker through the public JS scene API, with no renderer changes. -const CONTACT_SHADOW = `fn marker(f: MarkerFragment) -> MarkerColor { - let p = f.p / max(f.half, vec2f(0.001)); - let d = length(p); - let alpha = (1.0 - smoothstep(0.15, 1.0, d)) * f.input[0]; - return MarkerColor(vec4f(0.0, 0.0, 0.0, alpha), (d - 1.0) * f.half.x); -}`; - -export default async function bunny(stage: Stage): Promise { - const mesh = await loadMesh(new URL('./stanford-bunny.bin', import.meta.url)); - if (stage.signal.aborted) return; - const body = new Body(mesh); - // Reduced motion hops lower, pops gentler, and never orbits. - const reducedMotion = matchMedia('(prefers-reduced-motion: reduce)'); - reducedMotion.addEventListener( - 'change', - () => { - if (reducedMotion.matches) space.orbit = 0; - }, - { signal: stage.signal }, - ); - const controls: Controls = { gravity: 9.81, firmness: 0.45, inflate: 1, wind: false }; - /** Sets a control, waking the body: a resting body answers new gravity or a new material. */ - const waking = - (set: (value: T) => void) => - (value: T) => { - set(value); - body.wake(); - }; - let paint = 'solid'; - let inspect = false; - let fragments = false; - let timeScale = 1; - let carry = 0; - let recordAt = 0; - - // The mesh once; then every step, a frame of every vertex and of the body as a whole. - const data = new Data('bunny').apply( - new Patch() - .axis('time', 's') - .set('Vertex', { count: mesh.count }, {}) - .set( - 'Spring', - { count: mesh.springs.a.length }, - { - a: reference('Vertex', mesh.springs.a), - b: reference('Vertex', mesh.springs.b), - }, - ) - .set( - 'Face', - { count: mesh.faces.a.length }, - { - a: reference('Vertex', mesh.faces.a), - b: reference('Vertex', mesh.faces.b), - c: reference('Vertex', mesh.faces.c), - }, - ) - .set('ChunkVertex', { count: body.burst.count }, {}) - .set( - 'ChunkFace', - { count: body.burst.faces.a.length }, - { - a: reference('ChunkVertex', body.burst.faces.a), - b: reference('ChunkVertex', body.burst.faces.b), - c: reference('ChunkVertex', body.burst.faces.c), - }, - ) - .set('Corner', { count: 4 }, { x: [-ROOM, ROOM, ROOM, -ROOM], y: [-ROOM, -ROOM, ROOM, ROOM] }) - .set( - 'Wall', - { count: 4 }, - { - from: reference('Corner', [0, 1, 2, 3]), - to: reference('Corner', [1, 2, 3, 0]), - }, - ) - .set('Body', { count: 1 }, {}), - ); - - data.apply(new Patch().set('Shadow', { count: body.burst.chunks.length + 1 }, {})); - - // Lit faces and soft contact shadows; the vertices remain pickable with inspection dots hidden. - const room = (live: boolean): Scene => { - const p = PAINTS[paint]!; - return { - space: { plane: { grid: { lines: { pitch: 1 } } } }, - theme: THEME, - controls: { pickRadius: 14 }, - marks: [ - { - vertices: { - of: ['bunny', 'Shadow'], - position: ['x', 'y'], - size: [units(field('radius').mul(2)), units(field('radius'))], - marker: { wgsl: { source: CONTACT_SHADOW, inputs: [field('opacity')] } }, - color: '#000000', - pick: false, - }, - }, - { - vertices: { - of: ['bunny', 'Corner'], - position: ['x', 'y'], - radius: 3, - color: '#f09478aa', - pick: false, - }, - }, - { - edges: { - of: ['bunny', 'Wall'], - ends: ['from', 'to'], - color: '#f0947866', - width: 1.5, - pick: false, - }, - }, - { - faces: { - of: ['bunny', fragments ? 'ChunkFace' : 'Face'], - corners: ['a', 'b', 'c'], - ...(paint === 'solid' ? { color: '#edccb4' } : {}), - pick: false, - shadow: false, - }, - }, - { - edges: { - of: ['bunny', 'Spring'], - ends: ['a', 'b'], - color: '#ffffff14', - width: 1, - pick: false, - visible: inspect && !fragments, - }, - }, - { - vertices: { - of: ['bunny', fragments ? 'ChunkVertex' : 'Vertex'], - position: ['x', 'y'], - elevation: 'z', - radius: inspect ? 1.6 : 0, - color: - paint === 'solid' - ? inspect - ? '#edccb4' - : '#edccb400' - : field(p.field) - .colormap(p.colormap) - .domain(...p.domain), - select: false, - drag: live && !fragments, - pick: !fragments, - }, - }, - ], - }; - }; - - const space = stage.view(room(true), { - label: 'The bunny: drag it to pull and fling it, click it to poke it', - }); - space.camera = { center: [0, 0], zoom: 6, pitch: 58, bearing: 30 }; - space.fit({ bounds: { min: [-ROOM, -ROOM, 0], max: [ROOM, ROOM, 2.4] } }); - - const act = (run: () => void) => () => { - playback.play(); - run(); - }; - const actions = stage.section('Actions'); - actions.actions([ - { label: 'Hop', key: ' ', run: act(() => body.hop(reducedMotion.matches)) }, - { label: 'Squash', key: 's', run: act(() => body.squash()) }, - { label: 'Pop', key: 'e', run: act(() => body.pop(reducedMotion.matches)) }, - { label: 'Drop', key: 'd', run: act(() => body.drop()) }, - { - label: 'Poke', - key: 'p', - run: act(() => body.poke(Math.floor(Math.random() * mesh.count), 9)), - }, - ]); - const popButton = actions.element.querySelectorAll('button')[2]!; - popButton.title = 'Pop into soft pieces; press again to gather them'; - const squashButton = actions.element.querySelectorAll('button')[1]!; - squashButton.title = 'Hold to squash; release to rebound'; - squashButton.addEventListener( - 'pointerdown', - (event) => { - if (event.button !== 0) return; - squashButton.setPointerCapture(event.pointerId); - act(() => body.squash(true))(); - }, - { signal: stage.signal }, - ); - for (const event of ['pointerup', 'pointercancel', 'lostpointercapture']) - squashButton.addEventListener(event, () => body.endSquash(), { signal: stage.signal }); - const world = stage.section('Physics'); - world.toggle( - 'Wind', - 'w', - false, - waking((on: boolean) => (controls.wind = on)), - ); - world.toggle('Orbit', 'o', false, (on) => (space.orbit = on && !reducedMotion.matches ? 12 : 0)); - world.choice( - 'Gravity', - [ - ['Space 0', 0], - ['Moon 1.6', 1.62], - ['Earth 9.8', 9.81], - ['Jupiter 24.8', 24.8], - ], - 9.81, - waking((g: number) => (controls.gravity = g)), - ); - world.speed([1, 0.5, 0.2], 1, (rate) => (timeScale = rate)); - const material = stage.section('Material'); - material.range( - 'Firmness', - { - min: 0, - max: 1, - step: 0.01, - value: 0.45, - format: (v) => (v < 0.2 ? 'Jelly' : v < 0.6 ? 'Gummy' : 'Rubber'), - }, - waking((v: number) => (controls.firmness = v)), - ); - material.range( - 'Inflate', - { min: 0.5, max: 1.8, step: 0.01, value: 1, format: (v) => `${Math.round(v * 100)}%` }, - waking((v: number) => (controls.inflate = v)), - ); - const legend = stage.legend(); - const repaint = (p: string) => { - paint = p; - space.show(room(playback.mode === 'live')); - legend.ramp(PAINTS[p]!.colormap, PAINTS[p]!.legend); - legend.hidden = p === 'solid'; - }; - material.toggle('Mesh', 'i', false, (on) => { - inspect = on; - space.show(room(playback.mode === 'live')); - }); - material.choice( - 'Color', - Object.entries(PAINTS).map(([key, p]) => [p.label, key] as const), - paint, - repaint, - ); - - const monitor = stage.section('History'); - monitor.series(SERIES); - const plot = monitor.view( - { - space: { plot: { x: { label: 'Time (s)' }, y: { label: 'm · m/s · volume' } } }, - theme: THEME, - controls: 'inspect', - marks: SERIES.map((s) => ({ - lines: { of: ['bunny', 'Body'], trace: s.field, color: s.color, width: 2 }, - })), - }, - { label: 'Center height, speed, and volume', height: 220 }, - ); - - const playback = stage.playback({ - replay: { seconds: 5, rate: 0.3 }, - show(at) { - const pieces = data.get('Body', 0, 'phase', at) === 1; - if (pieces !== fragments) { - fragments = pieces; - space.show(room(playback.mode === 'live')); - } - space.at = at; - plot.at = at; - const x0 = Math.max(playback.start, at - WINDOW * 0.85); - plot.camera = { x: [x0, x0 + WINDOW], y: null }; - }, - change(live) { - // Paused, a press pans instead of grabbing. - space.show(room(live)); - if (!live) { - body.release(true); - body.endSquash(); - } - }, - }); - - const frame = body.frame(); - const chunkFrame = body.burst.frame(); - const shadow = { - x: new Float32Array(body.burst.chunks.length + 1), - y: new Float32Array(body.burst.chunks.length + 1), - radius: new Float32Array(body.burst.chunks.length + 1), - opacity: new Float32Array(body.burst.chunks.length + 1), - }; - let recordedChunks = false; - let wasBurst = false; - const publish = () => { - body.sample(frame); - const [height, speed, volume] = body.metrics(); - shadow.opacity.fill(0); - if (body.burst.active) { - body.burst.chunks.forEach((chunk, i) => { - shadow.x[i + 1] = chunk.position[0]; - shadow.y[i + 1] = chunk.position[1]; - shadow.radius[i + 1] = chunk.radius + chunk.position[2] * 0.12; - shadow.opacity[i + 1] = 0.32 / (1 + chunk.position[2]); - }); - } else { - shadow.x[0] = frame.x.reduce((sum, v) => sum + v, 0) / mesh.count; - shadow.y[0] = frame.y.reduce((sum, v) => sum + v, 0) / mesh.count; - shadow.radius[0] = 0.85 * Math.cbrt(controls.inflate) + height * 0.15; - shadow.opacity[0] = 0.42 / (1 + height * height * 0.4); - } - const patch = new Patch() - .append(body.time, 'Vertex', { ...frame }) - .append(body.time, 'Shadow', shadow) - .append(body.time, 'Body', { - height: [height], - speed: [speed], - volume: [Number.isFinite(volume) ? volume : null], - phase: [body.burst.active ? 1 : 0], - }) - .forget(body.time - HISTORY); - if (body.burst.active || wasBurst || !recordedChunks) { - body.burst.sample(chunkFrame, STEP); - patch.append(body.time, 'ChunkVertex', { ...chunkFrame }); - recordedChunks = true; - } - wasBurst = body.burst.active; - data.apply(patch); - space.setData(data); - plot.setData(data); - playback.record(data.span!); - popButton.firstChild!.textContent = body.burst.active ? 'Gather' : 'Pop'; - }; - - space.on('drag', ({ phase, hit, position }) => { - if (phase === 'start') body.grab(hit.row.id as number, position); - else if (phase === 'move') body.drag(position); - else body.release(); - }); - space.on('click', ({ hit }) => act(() => body.poke(hit.row.id as number))()); - - const canvas = stage.regions.main.querySelector('canvas')!; - // The JS drag event intentionally combines release and cancellation. Capture cancellation - // before the view's bubbling listener, so it cannot leave a throw behind. - canvas.addEventListener('pointercancel', () => body.release(true), { - capture: true, - signal: stage.signal, - }); - addEventListener( - 'blur', - () => { - body.release(true); - body.endSquash(); - }, - { signal: stage.signal }, - ); - - const meter = stage.meter(space); - meter.note(() => { - const at = playback.at; - const height = (data.get('Body', 0, 'height', at) as number | null) ?? 0; - const speed = (data.get('Body', 0, 'speed', at) as number | null) ?? 0; - const volume = data.get('Body', 0, 'volume', at) as number | null; - const state = - playback.mode === 'live' ? body.state : playback.mode === 'playing' ? 'replay' : 'paused'; - return `${state} · center ${height.toFixed(2)} m · ${speed.toFixed(1)} m/s · ${volume !== null && Number.isFinite(volume) ? `volume ${(volume * 100).toFixed(0)}%` : '24 pieces'}`; - }); - stage.frame((dt) => { - if (playback.mode !== 'live') return; - // A resting bunny steps and records nothing, so both views sleep until something acts on it. - if (body.asleep) { - carry = 0; - return; - } - meter.measure('simulate', () => { - carry = Math.min(carry + dt * timeScale, 16 * STEP); - const steps = body.advance(carry, controls); - carry -= steps * STEP; - // Recorded at a fixed rate of simulated time, and once more as it comes to rest. - if (steps && (body.time + STEP / 2 >= recordAt || body.asleep)) { - publish(); - recordAt = body.time + RECORD; - } - }); - }); - - stage.detail( - `${mesh.count.toLocaleString()} vertices · ${mesh.springs.a.length.toLocaleString()} springs · ${mesh.faces.a.length.toLocaleString()} faces`, - ); - legend.ramp(PAINTS[paint]!.colormap, PAINTS[paint]!.legend); - legend.hidden = true; - body.drop(true); - publish(); -} diff --git a/npm/examples/src/latkit.ts b/npm/examples/src/latkit.ts deleted file mode 100644 index 6cf31602..00000000 --- a/npm/examples/src/latkit.ts +++ /dev/null @@ -1,7 +0,0 @@ -/** The example application explicitly loads its shared rendering runtime. */ -import { loadLatkit } from 'latkit'; -export * from 'latkit'; -export type Data = import('latkit').Data; -export type Gpu = import('latkit').Gpu; -export type Shot = import('latkit').Shot; -export const { Data, Gpu, Shot, colormapCss } = await loadLatkit(); diff --git a/npm/examples/src/main.ts b/npm/examples/src/main.ts index bc8b251b..1be08660 100644 --- a/npm/examples/src/main.ts +++ b/npm/examples/src/main.ts @@ -1,6 +1,10 @@ -import { Gpu } from './latkit.js'; -import { examples } from './examples/index.js'; -import { Shell } from './ui/shell.js'; -import './ui/style.css'; +import { start } from './shell/shell.js'; +import './shell/style.css'; -new Shell(examples, Gpu.create()); +start([ + { id: 'grids', title: 'Power grids', load: () => import('./pages/grids/index.js') }, + { id: 'bunny', title: 'Soft body', load: () => import('./pages/bunny/index.js') }, + { id: 'orbits', title: 'Orbits', load: () => import('./pages/orbits.js') }, + { id: 'telemetry', title: 'Telemetry', load: () => import('./pages/telemetry.js') }, + { id: 'diagram', title: 'Diagram', load: () => import('./pages/diagram.js') }, +]); diff --git a/npm/examples/src/examples/soft-body/body.test.ts b/npm/examples/src/pages/bunny/body.test.ts similarity index 97% rename from npm/examples/src/examples/soft-body/body.test.ts rename to npm/examples/src/pages/bunny/body.test.ts index c505b1b3..17ec278f 100644 --- a/npm/examples/src/examples/soft-body/body.test.ts +++ b/npm/examples/src/pages/bunny/body.test.ts @@ -1,4 +1,4 @@ -/** The body's numerical checks, run by vitest; no wasm build. */ +/** Numerical checks of the soft body, run by vitest without a wasm build. */ import assert from 'node:assert/strict'; import { readFileSync } from 'node:fs'; import { test } from 'vitest'; @@ -101,7 +101,7 @@ test('body rests where it lands, hops at once, and ignores presses in the air', }); test('stands at any firmness, stays down once toppled, and wakes when poked', () => { - // Creeping as fast as before, a body would never count as still, and never rest. + // A body that kept creeping would never count as still, so it would never rest. for (const firmness of [0, 0.45, 1]) { const body = new Body(mesh); body.drop(true); @@ -179,7 +179,7 @@ test('pop lifts every piece and visibly separates before gathering', () => { ); assert.ok(separated.length >= 20, `${separated.length}/24 pieces left the original silhouette`); run(body, 0.6); - assert.ok(!body.burst.gathering, 'the full scatter has time to read before automatic return'); + assert.ok(!body.burst.gathering, 'the pieces stay scattered long enough to see'); run(body, 4); assert.ok(!body.burst.active, 'automatic assembly still completes'); diff --git a/npm/examples/src/examples/soft-body/body.ts b/npm/examples/src/pages/bunny/body.ts similarity index 80% rename from npm/examples/src/examples/soft-body/body.ts rename to npm/examples/src/pages/bunny/body.ts index ee49508b..e83c5a75 100644 --- a/npm/examples/src/examples/soft-body/body.ts +++ b/npm/examples/src/pages/bunny/body.ts @@ -1,47 +1,48 @@ import type { Mesh } from './mesh.js'; import { Burst } from './burst.js'; -/** What the controls set; the body reads them every step. */ +/** Settings from the page's controls. The body reads them every step. */ export interface Controls { /** Meters per second squared. */ gravity: number; - /** 0 is jelly, 1 is rubber: how hard the body pulls back to its shape. */ + /** How strongly the body pulls back to its shape, from 0 (jelly) to 1 (rubber). */ firmness: number; - /** The volume the body's pressure holds, as a share of its rest volume. */ + /** The volume the body's pressure keeps, as a fraction of its rest volume. */ inflate: number; wind: boolean; } -/** One frame of every vertex, a column per field, filled in place by {@link Body.sample}. */ +/** Per-vertex state at one step, one column per field, filled in place by {@link Body.sample}. */ export interface Frame { readonly x: Float32Array; readonly y: Float32Array; readonly z: Float32Array; /** Meters per second. */ readonly speed: Float32Array; - /** Meters from where the rest shape would put the vertex. */ + /** Meters between the vertex and its place in the rest shape the body is pulled toward. */ readonly strain: Float32Array; } -/** One step, in seconds. */ +/** Length of one simulation step, in seconds. */ export const STEP = 1 / 120; -/** The walls stand this far from the center. */ +/** Distance from the center to each wall, in meters. */ export const ROOM = 3.6; const CEILING = 6.5; -/** Mean speed, in meters per second, that counts as still, and the seconds it lasts before the - * body rests. */ +/** The body counts as still below this mean speed, in meters per second, and rests after staying + * still for `REST_AFTER` seconds. */ const STILL = 0.02; const REST_AFTER = 0.5; -/** Floor friction against how far a vertex sank this step: static holds it, kinetic slows it. */ +/** Floor friction, scaled by how far a vertex sank into the floor this step. Static friction stops + * a short slide, and kinetic friction slows a longer one. */ const STATIC_FRICTION = 0.8; const KINETIC_FRICTION = 0.5; -/** Bends resist folding at this multiple of the springs' compliance. */ +/** Compliance of the bend constraints, as a multiple of the springs' compliance. */ const BEND = 12; -/** How far a full squash lowers the rest shape, which keeps its volume. */ +/** Fraction by which a full squash lowers the rest shape. The shape widens to keep its volume. */ const SQUASH = 0.29; -/** Past this tilt from upright, as the cosine of its angle, the bunny topples and stays down. */ +/** Cosine of the tilt from upright past which the bunny topples and stays down. */ const TIPPING = 0.6; -/** How fast, per second, a standing bunny's turn returns upright. */ +/** Rate, per second, at which a standing bunny's orientation returns to upright. */ const BALANCE = 18; type Vec3 = [number, number, number]; @@ -51,10 +52,10 @@ const cross = (a: Vec3, b: Vec3): Vec3 => [ a[0] * b[1] - a[1] * b[0], ]; const dot = (a: Vec3, b: Vec3) => a[0] * b[0] + a[1] * b[1] + a[2] * b[2]; -/** A vector's length: what `Math.hypot` gives, at a fraction of its cost in the inner loops. */ +/** A vector's length. Same as `Math.hypot`, but much cheaper in the inner loops. */ const norm = (x: number, y: number, z: number) => Math.sqrt(x * x + y * y + z * z); const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v)); -/** Column-major rotation of a unit quaternion [x, y, z, w]. */ +/** The rotation matrix of a unit quaternion [x, y, z, w], as its three columns. */ function rotation([x, y, z, w]: number[]): [Vec3, Vec3, Vec3] { return [ [1 - 2 * (y! * y! + z! * z!), 2 * (x! * y! + w! * z!), 2 * (x! * z! - w! * y!)], @@ -64,22 +65,23 @@ function rotation([x, y, z, w]: number[]): [Vec3, Vec3, Vec3] { } /** - * A soft bunny: Verlet integration with compliant stretch, bend, and volume constraints, pulled - * toward its rest shape. It moves only when something acts on it, and rests once still, so a - * resting bunny costs nothing until it is touched. + * A soft bunny, simulated with Verlet integration, compliant stretch, bend, and volume constraints, + * and a pull toward its rest shape. It moves only when something acts on it and rests once still, + * so a resting bunny needs no simulation steps until it is touched. */ export class Body { readonly count: number; - /** Positions now and one step ago, x, y, z per vertex. */ + /** Current positions, x, y, z per vertex. `previous` holds the positions one step earlier. */ readonly x: Float64Array; private readonly previous: Float64Array; - /** Rest positions about the rest center. */ + /** Rest positions relative to the rest shape's center. */ private readonly rest: Float64Array; - /** How closely each vertex follows the rest shape: the ears, softer, lag behind. */ + /** How closely each vertex follows the rest shape. The ears are softer and lag behind. */ private readonly softness: Float64Array; /** - * Distance constraints, a spring along every edge and then a bend across it: their two ends, - * rest lengths, compliance as a multiple of the springs', and multipliers. + * Distance constraints. Springs along every edge come first, then bends joining the two corners + * opposite each edge. Each has two end vertices, a rest length, a compliance as a multiple of the + * springs', and a Lagrange multiplier. */ private readonly ends: Uint32Array; private readonly lengths: Float64Array; @@ -107,7 +109,7 @@ export class Body { /** Seconds it has been still with nothing acting on it. */ private still = 0; private scale = 1; - /** How far squashed, 0 to 1, and until when the squash holds. */ + /** How far squashed, from 0 to 1, and the simulated time until which the squash is held. */ private squashing = 0; private squashUntil = -Infinity; private grabbed?: { @@ -125,7 +127,7 @@ export class Body { this.rest = Float64Array.from(mesh.positions); const center = this.center(this.rest); for (let i = 0; i < count * 3; i++) this.rest[i] = this.rest[i]! - center[i % 3]!; - // The ears, above 1.5 m, follow the rest shape loosely, their tips most. + // The ears, above 1.5 m, follow the rest shape loosely, and their tips most loosely. this.softness = Float64Array.from( { length: count }, (_, i) => 1 - 0.8 * clamp((mesh.positions[i * 3 + 2]! - 1.5) / 0.75, 0, 1), @@ -143,8 +145,8 @@ export class Body { this.neighbors[a[e]!]!.push({ row: b[e]!, length }); this.neighbors[b[e]!]!.push({ row: a[e]!, length }); } - // Area-weighted mass avoids making dense parts of the scan artificially heavy. Each edge's two - // opposite corners make a bend. + // Area-weighted mass avoids making dense parts of the scan artificially heavy. Each edge shared + // by two faces adds a bend between its two opposite corners. const mass = new Float64Array(count); const opposite = new Map(); for (let t = 0; t < mesh.faces.a.length; t++) { @@ -181,23 +183,24 @@ export class Body { this.burst = new Burst(mesh, ROOM, CEILING); } - /** A frame to {@link sample} into, reused every step: appending copies it. */ + /** Creates a frame for {@link sample} to fill. Reuse it every step, since appending copies it. */ frame(): Frame { const column = () => new Float32Array(this.count); return { x: column(), y: column(), z: column(), speed: column(), strain: column() }; } - /** Resting: still, with nothing acting on it. Nothing steps until something wakes it. */ + /** Whether the body is resting: still, with nothing acting on it. It takes no steps until + * woken. */ get asleep(): boolean { return this.still >= REST_AFTER; } - /** Anything that acts on the body wakes it, as do changed controls. */ + /** Wakes the body. Anything that acts on the body wakes it, and so should changed controls. */ wake(): void { this.still = 0; } - /** Advance by whole steps, none while resting; returns how many ran. */ + /** Advances by whole steps, at most 16, stopping once the body rests. Returns the steps taken. */ advance(seconds: number, controls: Controls): number { const most = Math.min(16, Math.floor(seconds / STEP)); let steps = 0; @@ -205,7 +208,7 @@ export class Body { return steps; } - /** Every vertex's position, speed, and strain now. */ + /** Writes every vertex's current position, speed, and strain into `frame`. */ sample(frame: Frame): void { const [c, r0, r1, r2] = this.goal; const [lateral, vertical] = this.shape(); @@ -222,7 +225,7 @@ export class Body { y - this.previous[i * 3 + 1]!, z - this.previous[i * 3 + 2]!, ) / STEP; - // Strain is the distance from the shape the body is pulled toward: inflated and squashed. + // Strain is the distance from the inflated, squashed rest shape the body is pulled toward. const rx = this.rest[i * 3]! * lateral; const ry = this.rest[i * 3 + 1]! * lateral; const rz = this.rest[i * 3 + 2]! * vertical; @@ -234,7 +237,8 @@ export class Body { } } - /** Center height, mean speed, and intact volume. Volume is missing while in pieces. */ + /** Returns the center height, the mean speed, and the volume as a fraction of the rest volume. + * The volume is NaN while the body is in pieces. */ metrics(): [number, number, number] { if (this.burst.active) { const chunks = this.burst.chunks; @@ -258,13 +262,14 @@ export class Body { ]; } - // Impulses change velocity, which Verlet keeps as the step from previous to now. + // Verlet keeps velocity as the step from the previous position to the current one, so an + // impulse moves the previous position. private push(row: number, v: Vec3): void { for (let a = 0; a < 3; a++) this.previous[row * 3 + a] = this.previous[row * 3 + a]! - v[a]! * STEP; } - /** A hop: an upward push, and a little forward, while on the floor; `gentle` hops lower. */ + /** Hops up and slightly forward if the body is on the floor. A `gentle` hop is lower. */ hop(gentle = false): void { if (this.contacts === 0 || this.grabbed || this.burst.active) return; const facing = this.goal[1]; @@ -273,12 +278,13 @@ export class Body { const vz = (this.x[i * 3 + 2]! - this.previous[i * 3 + 2]!) / STEP; this.push(i, [facing[0] * 0.45, facing[1] * 0.45, Math.max(0, speed - vz)]); } - // It has left the floor: presses before it lands do nothing. + // The body has left the floor, so presses before it lands do nothing. this.contacts = 0; this.wake(); } - /** Squash now: held, until {@link endSquash}; else for a moment. The material does the rebound. */ + /** Squashes the body until {@link endSquash} if `held`, else for 0.3 s. The material makes the + * rebound. */ squash(held = false): void { if (this.burst.active || this.grabbed) return; this.squashUntil = held ? Infinity : this.time + 0.3; @@ -289,7 +295,8 @@ export class Body { if (this.squashUntil === Infinity) this.squashUntil = this.time; } - /** Pop into capped pieces now, `gentle` ones flying less; pressed again, call them home. */ + /** Pops the body into capped pieces, which fly less if `gentle`. Called while the body is in + * pieces, it gathers them back. */ pop(gentle = false): void { if (this.burst.active) return this.burst.gather(); this.release(); @@ -300,7 +307,7 @@ export class Body { this.wake(); } - /** Dent the surface around a vertex. */ + /** Dents the surface around vertex `row`. */ poke(row: number, speed = 6): void { if (this.burst.active) return; const near = this.near(row, 0.7); @@ -316,7 +323,7 @@ export class Body { this.wake(); } - /** Lift it high, turned at random unless `upright`, and let go. */ + /** Lifts the body high, turned at random unless `upright`, and lets go. */ drop(upright = false): void { const yaw = upright ? 0.6 : Math.random() * Math.PI * 2; const tilt = upright ? 0 : (Math.random() - 0.5) * 0.9; @@ -341,7 +348,7 @@ export class Body { this.wake(); } - /** Hold a vertex and the surface around it toward `target` until released. */ + /** Pulls vertex `row` and the surface around it toward `target` until released. */ grab(row: number, target: readonly number[]): void { if (this.burst.active) return; this.squashUntil = -Infinity; @@ -405,7 +412,7 @@ export class Body { this.still = !acting && speed < STILL ? this.still + STEP : 0; } - /** Gravity, wind, and damping move every vertex on from where its last step took it. */ + /** Moves every vertex on by its damped velocity from the last step, plus gravity and wind. */ private integrate(controls: Controls): void { const { x, previous, count, normals } = this; const t = this.time; @@ -438,7 +445,8 @@ export class Body { } } - /** Three rounds of every distance constraint and the pressure, then the normals once. */ + /** Runs three iterations of every distance constraint and the pressure, then updates the + * normals. */ private solve(controls: Controls): void { const compliance = 1e-7 * 10 ** (3 * (1 - controls.firmness)); this.lambdas.fill(0); @@ -450,7 +458,7 @@ export class Body { this.normalize(); } - /** Every spring and bend, each pulled toward its rest length, inflated. */ + /** Pulls every spring and bend toward its rest length, scaled by the inflation. */ private distances(compliance: number): void { const { x, ends, lengths, compliances, lambdas, inverseMass, scale } = this; const alpha = compliance / (STEP * STEP); @@ -480,14 +488,14 @@ export class Body { } } - /** The enclosed volume and its gradient per vertex, with the normals they imply. */ + /** Computes the enclosed volume, its gradient per vertex, and the normals from that gradient. */ private measure(): number { const volume = this.enclose(); this.normalize(); return volume; } - /** The enclosed volume, and its gradient per vertex. */ + /** Computes the enclosed volume and its gradient per vertex. */ private enclose(): number { const { x, gradient } = this; const { a: fa, b: fb, c: fc } = this.mesh.faces; @@ -518,7 +526,7 @@ export class Body { return (this.volume = volume / 6); } - /** The unit outward normals the last volume gradient implies. */ + /** Sets the unit outward normals from the last volume gradient. */ private normalize(): void { const { gradient, normals } = this; for (let i = 0; i < gradient.length; i += 3) { @@ -529,7 +537,7 @@ export class Body { } } - /** Pull the enclosed volume toward `inflate` of the rest volume. */ + /** Pulls the enclosed volume toward `inflate` times the rest volume. */ private pressure(compliance: number, inflate: number): void { const volume = this.enclose(); const { x, gradient, inverseMass, count } = this; @@ -550,13 +558,14 @@ export class Body { } } - /** The rest shape's sideways and upward scale: inflated, and squashed keeping its volume. */ + /** The rest shape's sideways and vertical scale, with the inflation and a squash that keeps the + * volume. */ private shape(): [number, number] { const squashed = 1 - SQUASH * this.squashing; return [this.scale / Math.sqrt(squashed), this.scale * squashed]; } - /** Pull toward the rest shape, turned and moved to fit where the body is now. */ + /** Pulls the body toward its rest shape, rotated and moved to best fit the current positions. */ private match(stiffness: number): void { const { x, rest, count, softness } = this; const c = this.center(x); @@ -578,7 +587,7 @@ export class Body { column[2] += pz * r; } } - // Müller et al. 2016: the rotation nearest A, warm started from the last step. + // Find the rotation nearest A (Müller et al. 2016), warm started from the last step. let q = this.quaternion; for (let iteration = 0; iteration < 8; iteration++) { const R = rotation(q); @@ -605,9 +614,9 @@ export class Body { const n = Math.hypot(...q); q = q.map((v) => v / n); } - // Matching reads part of gravity's sag as a turn, and on its own would roll the bunny over: - // standing short of its tipping point, the turn keeps upright at the same heading. Toppled - // past it, the bunny lies where it fell. + // Shape matching reads part of gravity's sag as a rotation, which alone would roll the bunny + // over. Short of the tipping point, the rotation eases back upright at the same heading. Past + // it, the bunny lies where it fell. if (stiffness > 0 && this.contacts > 8 && !this.grabbed && rotation(q)[2][2] > TIPPING) { const heading = rotation(q)[0]; const yaw = Math.atan2(heading[1], heading[0]); @@ -652,20 +661,22 @@ export class Body { } } - /** The floor with its friction, the ceiling, then the walls; returns the mean speed this step. */ + /** Collides with the floor (with friction), the ceiling, then the walls. Returns the mean speed + * this step. */ private collide(): number { const { x, previous } = this; let contacts = 0; let travel = 0; for (let i = 0; i < x.length; i += 3) { - // The floor first: friction needs where the vertex was, before a wall's bounce rewrites it. + // The floor comes first, since friction needs the vertex's previous position before a wall + // bounce rewrites it. if (x[i + 2]! < 0) { const depth = -x[i + 2]!; const vz = x[i + 2]! - previous[i + 2]!; x[i + 2] = 0; previous[i + 2] = vz < -0.5 * STEP ? vz * 0.18 : 0; - // Friction acts on what this step moved it along the floor, so no push from the springs - // or the rest shape can ratchet the body across it. + // Friction acts on this step's movement along the floor, so pushes from the springs or + // the rest shape cannot ratchet the body across it. const tx = x[i]! - previous[i]!; const ty = x[i + 1]! - previous[i + 1]!; const slide = Math.sqrt(tx * tx + ty * ty); @@ -701,7 +712,8 @@ export class Body { } private near(row: number, radius: number) { - // Walk the rest surface: adjacent folded ears must not share a grab neighborhood. + // Search along the rest surface's edges, not through space, so ears folded against each other + // do not share a grab neighborhood. const distance = new Float64Array(this.count).fill(Infinity); distance[row] = 0; const queue = [row]; diff --git a/npm/examples/src/examples/soft-body/burst.ts b/npm/examples/src/pages/bunny/burst.ts similarity index 95% rename from npm/examples/src/examples/soft-body/burst.ts rename to npm/examples/src/pages/bunny/burst.ts index dbafc6f3..7f046031 100644 --- a/npm/examples/src/examples/soft-body/burst.ts +++ b/npm/examples/src/pages/bunny/burst.ts @@ -1,3 +1,4 @@ +import { congruential } from '../../random.js'; import type { Frame } from './body.js'; import { fracture, type Piece, type Vec3 } from './fracture.js'; import type { Mesh } from './mesh.js'; @@ -18,7 +19,8 @@ interface Chunk { const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v)); -/** Independent capped pieces. No original triangle or spring connects them during a pop. */ +/** The body popped into independent capped pieces, which no original triangle or spring + * connects. */ export class Burst { readonly chunks: Chunk[]; readonly count: number; @@ -94,12 +96,8 @@ export class Burst { this.targetBody[i + 1] = cy + rx * sin + ry * cos; this.targetBody[i + 2] = this.mesh.positions[i + 2]! * scale + 0.025; } - // Stable variation, sampled only at launch. Replay records the resulting positions. - let seed = 0x12b9; - const random = () => { - seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0; - return seed / 4294967296; - }; + // Seeded random variation, sampled only at launch. Replay records the resulting positions. + const random = congruential(0x12b9); for (const chunk of this.chunks) { chunk.position = [0, 0, 0]; chunk.target = [0, 0, 0]; @@ -157,7 +155,7 @@ export class Burst { } gather(): void { - // Queue early requests until the pieces have visibly separated, including key repeats. + // Defer early requests, including key repeats, until the pieces have visibly separated. if (this.active) this.returnAt = Math.min(this.returnAt, Math.max(this.age, this.reduced ? 0.5 : 0.9)); } @@ -170,7 +168,7 @@ export class Burst { this.active = false; } - /** Returns true on the one step that finishes assembly. */ + /** Advances one step. Returns true on the step that finishes reassembly. */ step(dt: number, gravity: number): boolean { if (!this.active) return false; this.previous.set(this.x); @@ -213,7 +211,8 @@ export class Burst { } this.rotate(chunk, dt); } - // Soft sphere proxies separate tumbling pieces; stop before seams close on return. + // Soft sphere proxies push tumbling pieces apart. They stop once the return starts, so the + // seams can close. if (this.age < this.returnAt) this.separate(); this.writeVertices(dt); if (settled && this.age > this.returnAt + 1.1) { diff --git a/npm/examples/src/examples/soft-body/fracture.ts b/npm/examples/src/pages/bunny/fracture.ts similarity index 93% rename from npm/examples/src/examples/soft-body/fracture.ts rename to npm/examples/src/pages/bunny/fracture.ts index 36d78bbd..a0002f67 100644 --- a/npm/examples/src/examples/soft-body/fracture.ts +++ b/npm/examples/src/pages/bunny/fracture.ts @@ -15,7 +15,7 @@ export interface Piece { const EPS = 1e-7; -/** Split the closed mesh into independently capped solids, once at page load. */ +/** Splits the closed mesh into `count` separately capped solids. Runs once, at page load. */ export function fracture(mesh: Mesh, count = 24): Piece[] { const first: Piece = { vertices: Array.from({ length: mesh.count }, (_, i) => ({ @@ -26,7 +26,7 @@ export function fracture(mesh: Mesh, count = 24): Piece[] { }; const pieces = [first]; while (pieces.length < count) { - // Cut the largest bounding volume, with stable ordering for repeatable pieces. + // Cut the piece with the largest bounding box. Stable ordering makes the pieces repeatable. const bounds = pieces.map((piece) => { const lo = [Infinity, Infinity, Infinity]; const hi = [-Infinity, -Infinity, -Infinity]; @@ -104,7 +104,8 @@ function split(piece: Piece, axis: number, at: number): [Piece, Piece] { if (cut.size === 2) segments.push([...cut] as [number, number]); } - // Each cut boundary is a loop; triangulate concave loops instead of a center fan. + // Each cut boundary is a loop. Cap it by ear clipping, which handles concave loops that a + // center fan would not. const neighbors = new Map(); for (const [a, b] of segments) { neighbors.set(a, [...(neighbors.get(a) ?? []), b]); @@ -136,7 +137,8 @@ function split(piece: Piece, axis: number, at: number): [Piece, Piece] { return sides.map(compact) as [Piece, Piece]; } -/** Counterclockwise triangles viewed along the positive cut axis. */ +/** Caps a cut loop by ear clipping. The triangles are counterclockwise viewed along the positive + * cut axis. */ function cap(loop: number[], vertices: CutVertex[], axis: number): [number, number, number][] { const u = (axis + 1) % 3; const v = (axis + 2) % 3; diff --git a/npm/examples/src/pages/bunny/index.ts b/npm/examples/src/pages/bunny/index.ts new file mode 100644 index 00000000..d1731eb8 --- /dev/null +++ b/npm/examples/src/pages/bunny/index.ts @@ -0,0 +1,268 @@ +/** + * A soft bunny made of springs, in a room. Hop, squash, poke, fling, or pop it, then replay or scrub + * its history: every frame of every vertex is recorded, and the views read it back at the + * timeline's time. Faces darken as they turn from the light, above soft contact shadows drawn by a + * custom WGSL marker. A plot traces the bunny's height and speed. + */ +import { Data, field, reference, units, type Scene } from 'latkit'; +import { choice, page } from '../../shell/page.js'; +import { Body, ROOM, STEP } from './body.js'; +import { loadMesh } from './mesh.js'; + +/** Simulated seconds between recorded frames, whatever the display's refresh rate, and the + * seconds the plot shows. */ +const RECORD = 1 / 60; +const WINDOW = 12; +/** The material and the world the bunny lives in. */ +const WORLD = { gravity: 9.81, firmness: 0.45, inflate: 1, wind: false }; +/** Each coloring: the field it reads, through which colormap, over which values. */ +const PAINTS = { + speed: { field: 'speed', colormap: 'plasma', domain: [-1, 9], legend: 'Speed (m/s)' }, + stress: { + field: 'strain', + colormap: 'turbo', + domain: [-0.05, 0.35], + legend: 'Stress (m from rest)', + }, +} as const; +const SERIES = [ + { field: 'height', label: 'Center height (m)', color: '#66c7ff' }, + { field: 'speed', label: 'Mean speed (m/s)', color: '#ff8f52' }, +]; +/** A soft contact shadow, darker the lower the body above it: a custom marker in WGSL that reads + * its darkness from each row's `opacity`. */ +const SHADOW = `fn marker(f: MarkerFragment) -> MarkerColor { + let d = length(f.p / max(f.half, vec2f(0.001))); + let alpha = (1.0 - smoothstep(0.15, 1.0, d)) * f.input[0]; + return MarkerColor(vec4f(0.0, 0.0, 0.0, alpha), (d - 1.0) * f.half.x); +}`; + +export default page({ + title: 'Soft body', + inspect: false, + settings: { paint: choice('Color', { Speed: 'speed', Stress: 'stress' }, { key: 'c' }) }, + + async open({ settings, signal, view, actions, timeline, frame, series, detail }) { + const mesh = await loadMesh(new URL('./stanford-bunny.bin', import.meta.url)); + const body = new Body(mesh); + const { burst } = body; + const pieces = burst.chunks.length; + // The mesh, its pieces once popped, and the room are set once. Each recorded frame then + // appends every vertex, the shadows, and the body as a whole. + const data = new Data('bunny') + .axis('time', 's') + .set('Vertex', { count: mesh.count }) + .set('Spring', { + a: reference('Vertex', mesh.springs.a), + b: reference('Vertex', mesh.springs.b), + }) + .set('Face', { + a: reference('Vertex', mesh.faces.a), + b: reference('Vertex', mesh.faces.b), + c: reference('Vertex', mesh.faces.c), + }) + .set('ChunkVertex', { count: burst.count }) + .set('ChunkFace', { + a: reference('ChunkVertex', burst.faces.a), + b: reference('ChunkVertex', burst.faces.b), + c: reference('ChunkVertex', burst.faces.c), + }) + .set('Corner', { x: [-ROOM, ROOM, ROOM, -ROOM], y: [-ROOM, -ROOM, ROOM, ROOM] }) + .set('Wall', { + from: reference('Corner', [0, 1, 2, 3]), + to: reference('Corner', [1, 2, 3, 0]), + }) + .set('Shadow', { count: pieces + 1 }) + .set('Body', { count: 1 }); + + let popped = false; + let live = true; + const room = (s: typeof settings): Scene => { + const paint = PAINTS[s.paint]; + return { + space: { plane: { grid: { lines: { pitch: 1 } } } }, + theme: { + base: 'dark', + background: '#101214', + grid: '#ffffff1c', + shading: 0.7, + vertex: { radius: 0 }, + }, + controls: { pickRadius: 14 }, + marks: [ + { + vertices: data.table('Shadow'), + position: ['$x', '$y'], + size: [units(field('radius').mul(2)), units(field('radius'))], + marker: { wgsl: { source: SHADOW, inputs: [field('opacity')] } }, + color: '#000000', + pick: false, + }, + { + vertices: data.table('Corner'), + position: ['$x', '$y'], + radius: 3, + color: '#f09478aa', + pick: false, + }, + { + edges: data.table('Wall'), + ends: ['from', 'to'], + color: '#f0947866', + width: 1.5, + pick: false, + }, + // Faces without a color blend their corners' colors. + { + faces: data.table(popped ? 'ChunkFace' : 'Face'), + corners: ['a', 'b', 'c'], + pick: false, + }, + { + vertices: data.table(popped ? 'ChunkVertex' : 'Vertex'), + position: ['$x', '$y'], + elevation: '$z', + color: field(paint.field) + .colormap(paint.colormap) + .domain(paint.domain[0], paint.domain[1]), + legend: paint.legend, + select: false, + // Outside live mode, a press pans instead of grabbing. + drag: live && !popped, + pick: !popped, + }, + ], + }; + }; + const space = await view(room, { + region: [0, 0, 1, 0.74], + camera: { center: [0, 0], zoom: 6, pitch: 58, bearing: 30 }, + label: 'The bunny: drag it to fling it, click it to poke it', + }); + // The room fills the view, seen from the camera's angle. + space.fit({ bounds: { min: [-ROOM, -ROOM, 0], max: [ROOM, ROOM, 2.4] } }); + const plot = await view( + () => ({ + space: { plot: { x: { label: 'Time (s)' } } }, + theme: { base: 'dark', background: '#101214' }, + controls: 'inspect', + ease: 0, + marks: SERIES.map((s) => ({ + lines: data.table('Body'), + trace: s.field, + color: s.color, + width: 2, + })), + }), + { region: [0, 0.74, 1, 0.26], camera: { x: [0, WINDOW] } }, + ); + series('History', SERIES); + const redraw = () => (space.scene = room(settings)); + const letGo = () => { + body.release(true); + body.endSquash(); + }; + + const recording = timeline({ + data, + keep: 20, + replay: { seconds: 5, rate: 0.3 }, + // The plot keeps the time shown near its right edge. While a replay passes a pop, the pieces + // stand in for the whole body. + show(at) { + const from = Math.max(0, at - 0.85 * WINDOW); + plot.camera = { x: [from, from + WINDOW] }; + const was = data.get('Body', 0, 'popped', { at }) === 1; + if (was === popped) return; + popped = was; + redraw(); + }, + change(now) { + live = now; + if (!live) letGo(); + redraw(); + }, + }); + + /** Runs an action live, going live first if the timeline was replaying or paused. */ + const act = (run: () => void) => () => { + recording.play(); + run(); + }; + actions('Actions', [ + { label: 'Hop', key: ' ', run: act(() => body.hop()) }, + { + label: 'Squash', + key: 's', + run: act(() => body.squash(true)), + release: () => body.endSquash(), + }, + { label: 'Pop', key: 'e', run: act(() => body.pop()) }, + { label: 'Drop', key: 'd', run: act(() => body.drop()) }, + ]); + space.on('drag', ({ phase, hit, position }) => { + if (phase === 'start') body.grab(Number(hit.row.id), position); + else if (phase === 'move') body.drag(position); + else body.release(); + }); + space.on('click', ({ hit }) => act(() => body.poke(Number(hit.row.id)))()); + addEventListener('blur', letGo, { signal }); + + // Each recorded frame: every vertex, the pieces while popped, a shadow under the body or under + // each piece, and the body's height and speed. + const vertices = body.frame(); + const chunks = burst.frame(); + const shadow = { + x: new Float32Array(pieces + 1), + y: new Float32Array(pieces + 1), + radius: new Float32Array(pieces + 1), + opacity: new Float32Array(pieces + 1), + }; + const record = () => { + body.sample(vertices); + const [height, speed] = body.metrics(); + shadow.opacity.fill(0); + if (burst.active) { + burst.chunks.forEach(({ position: [x, y, z], radius }, i) => { + [shadow.x[i + 1], shadow.y[i + 1]] = [x, y]; + shadow.radius[i + 1] = radius + z * 0.12; + shadow.opacity[i + 1] = 0.32 / (1 + z); + }); + } else { + shadow.x[0] = vertices.x.reduce((sum, v) => sum + v, 0) / mesh.count; + shadow.y[0] = vertices.y.reduce((sum, v) => sum + v, 0) / mesh.count; + shadow.radius[0] = 0.85 + height * 0.15; + shadow.opacity[0] = 0.42 / (1 + height * height * 0.4); + } + burst.sample(chunks, STEP); + data + .append('Vertex', body.time, { ...vertices }) + .append('ChunkVertex', body.time, { ...chunks }) + .append('Shadow', body.time, shadow) + .append('Body', body.time, { + height: [height], + speed: [speed], + popped: [burst.active ? 1 : 0], + }); + }; + + // A resting bunny takes no steps and records nothing, so both views sleep until something acts. + let carry = 0; + let next = 0; + frame((dt) => { + if (body.asleep) return void (carry = 0); + carry = Math.min(carry + dt, 16 * STEP); + const steps = body.advance(carry, WORLD); + carry -= steps * STEP; + if (steps && (body.time + STEP / 2 >= next || body.asleep)) { + record(); + next = body.time + RECORD; + } + }); + detail( + `${mesh.count.toLocaleString()} vertices · ${mesh.springs.a.length.toLocaleString()} springs · ${mesh.faces.a.length.toLocaleString()} faces`, + ); + body.drop(true); + record(); + }, +}); diff --git a/npm/examples/src/examples/soft-body/mesh.ts b/npm/examples/src/pages/bunny/mesh.ts similarity index 77% rename from npm/examples/src/examples/soft-body/mesh.ts rename to npm/examples/src/pages/bunny/mesh.ts index 3397be4d..37179713 100644 --- a/npm/examples/src/examples/soft-body/mesh.ts +++ b/npm/examples/src/pages/bunny/mesh.ts @@ -1,16 +1,18 @@ /** - * The Stanford bunny, from the Stanford 3D Scanning Repository (graphics.stanford.edu/data/3Dscanrep), - * as the public-domain `bunny` package ships it: a closed low-resolution reconstruction. The file - * holds its vertex and triangle counts as two uint32s, then float32 x, y, z per vertex with y up, then - * uint16 triangles wound outward. + * The Stanford bunny, from the Stanford 3D Scanning Repository + * (graphics.stanford.edu/data/3Dscanrep), as shipped by the public-domain `bunny` package: a + * closed, low-resolution reconstruction. + * + * The file holds the vertex and triangle counts as two uint32s, then float32 x, y, z per vertex + * with y up, then uint16 triangles wound outward. */ export interface Mesh { readonly count: number; /** x, y, z per vertex in meters, z up, resting on z = 0 and centered over the origin. */ readonly positions: Float64Array; - /** Each outward triangle's three corners, a column each. */ + /** The three corners of each outward-wound triangle, one column per corner. */ readonly faces: { readonly a: Uint32Array; readonly b: Uint32Array; readonly c: Uint32Array }; - /** Each triangle side once: the springs, and the wireframe. */ + /** Every triangle edge, listed once. They are the springs and the wireframe. */ readonly springs: { readonly a: Uint32Array; readonly b: Uint32Array }; } @@ -21,7 +23,7 @@ export async function loadMesh(url: URL, height = 2.4): Promise { const [count, faces] = new Uint32Array(bytes, 0, 2) as unknown as [number, number]; const scan = new Float32Array(bytes, 8, count * 3); const triangles = new Uint16Array(bytes, 8 + count * 12, faces * 3); - // The scan is y up; turn it z up, which keeps its triangles wound outward. + // The scan is y up. Rotate it to z up, which keeps its triangles wound outward. const positions = new Float64Array(count * 3); const lo = [Infinity, Infinity, Infinity]; const hi = [-Infinity, -Infinity, -Infinity]; diff --git a/npm/examples/src/examples/soft-body/stanford-bunny.bin b/npm/examples/src/pages/bunny/stanford-bunny.bin similarity index 100% rename from npm/examples/src/examples/soft-body/stanford-bunny.bin rename to npm/examples/src/pages/bunny/stanford-bunny.bin diff --git a/npm/examples/src/pages/diagram.ts b/npm/examples/src/pages/diagram.ts new file mode 100644 index 00000000..92fa6d12 --- /dev/null +++ b/npm/examples/src/pages/diagram.ts @@ -0,0 +1,267 @@ +/** + * A block diagram on a sheet: blocks with ports, wires routed around them, and groups drawn around + * their members. Each block kind is a table. Its reference fields name the signal on each port, and + * an edges mark over the signals draws them as wires. Drag blocks, draw wires from an output to an + * input, delete with the Delete key, arrange, and undo. + */ +import { + Data, + field, + reference, + type Id, + type Mark, + type Port, + type Reference, + type Row, + type Snapshot, + type Values, +} from 'latkit'; +import { choice, page, toggle } from '../shell/page.js'; + +/** The block kinds: each one's marker fill and its ports, as name, side, and whether it drives. */ +const KINDS = { + Input: { fill: '#173836', ports: [['out', 'right', true]] }, + Process: { + fill: '#1c2b3f', + ports: [ + ['in', 'left', false], + ['out', 'right', true], + ], + }, + Control: { + fill: '#3a2e18', + ports: [ + ['in', 'left', false], + ['feedback', 'bottom', false], + ['out', 'right', true], + ], + }, + Output: { fill: '#29243f', ports: [['in', 'left', false]] }, +} as const; +type Kind = keyof typeof KINDS; +const kinds = Object.keys(KINDS) as Kind[]; + +/** A block: its kind, name, group, and the signal on each of its ports. */ +interface Block { + kind: Kind; + name: string; + group?: number; + plugs: Record; +} + +/** Writes blocks, the signals that wire them, and the groups around them into a dataset. Blocks + * have no position, so the layout places them. */ +function sheet(blocks: Block[], signals: string[], groups: string[]): Data { + const data = new Data('sheet').set('Group', { name: groups }).set('Signal', { name: signals }); + for (const kind of kinds) { + const mine = blocks.filter((b) => b.kind === kind); + const fields: Record = { + name: mine.map((b) => b.name), + x: mine.map(() => null), + y: mine.map(() => null), + }; + for (const [port] of KINDS[kind].ports) + fields[port] = reference( + 'Signal', + mine.map((b) => b.plugs[port] ?? null), + ); + if (kind === 'Process') + fields.group = reference( + 'Group', + mine.map((b) => b.group ?? null), + ); + data.set(kind, fields); + } + return data; +} + +/** A control loop: a setpoint, a controller with feedback, and a plant of three stages. */ +function loop(): Data { + return sheet( + [ + { kind: 'Input', name: 'Setpoint', plugs: { out: 0 } }, + { kind: 'Control', name: 'PID', plugs: { in: 0, feedback: 4, out: 1 } }, + { kind: 'Process', name: 'Actuator', group: 0, plugs: { in: 1, out: 2 } }, + { kind: 'Process', name: 'Plant', group: 0, plugs: { in: 2, out: 3 } }, + { kind: 'Process', name: 'Sensor', group: 0, plugs: { in: 3, out: 4 } }, + { kind: 'Output', name: 'Scope', plugs: { in: 3 } }, + ], + ['setpoint', 'command', 'force', 'output', 'measured'], + ['Plant'], + ); +} + +/** Twenty lines of a production floor, each an input through five stages in its own group. */ +function floor(): Data { + const blocks: Block[] = []; + const signals: string[] = []; + const groups: string[] = []; + for (let line = 0; line < 20; line++) { + groups.push(`Line ${line + 1}`); + let signal = signals.push(`feed ${line + 1}`) - 1; + blocks.push({ kind: 'Input', name: `Feed ${line + 1}`, plugs: { out: signal } }); + for (let stage = 0; stage < 5; stage++) { + const out = signals.push(`${line + 1}.${stage + 1}`) - 1; + blocks.push({ + kind: 'Process', + name: `Stage ${stage + 1}`, + group: line, + plugs: { in: signal, out }, + }); + signal = out; + } + blocks.push({ kind: 'Output', name: `Bin ${line + 1}`, plugs: { in: signal } }); + } + return sheet(blocks, signals, groups); +} + +export default page({ + title: 'Diagram', + settings: { + preset: choice('Diagram', { 'Control loop': 'loop', 'Production floor': 'floor' }), + layout: choice('Layout', { + 'Left to right': 'right', + 'Top to bottom': 'down', + Stress: 'stress', + }), + route: choice('Wires', { Orthogonal: 'orthogonal', Straight: 'straight' }), + flow: toggle('Flow', { key: 'f' }), + }, + + async open({ settings, view, actions, detail }) { + let data = settings.preset === 'loop' ? loop() : floor(); + const shown = await view((s) => ({ + space: { diagram: { grid: { dots: { pitch: 20 } } } }, + layout: + s.layout === 'stress' + ? { stress: { gap: 50 } } + : { layered: { direction: s.layout, gap: 30, rankGap: 70 } }, + theme: { base: 'dark', background: '#101214', diagram: { outline: '#ffffff1f' } }, + controls: { connect: true, delete: true }, + marks: [ + { + vertices: data.table('Group'), + contains: ['Process', 'group'], + label: { text: '$name', size: 11, color: '#7f93a8' }, + shadow: true, + }, + { + edges: data.table('Signal'), + route: s.route, + arrows: true, + width: 1.5, + dash: field('name').is('measured'), + flow: s.flow ? 40 : 0, + }, + ...kinds.map((kind): Mark => ({ + vertices: data.table(kind), + position: ['$x', '$y'], + marker: kind === 'Input' ? 'rounded' : 'block', + color: KINDS[kind].fill, + shadow: true, + drag: true, + label: { text: '$name', size: 12 }, + ports: KINDS[kind].ports.map(([name, side, output]): [string, Port] => [ + name, + { side, output }, + ]), + })), + ], + })); + + // Every edit can be undone. Each edit keeps the version before it, which shares pages with the + // data, so the history costs only what changed. + const undo: Snapshot[] = []; + const redo: Snapshot[] = []; + const edit = (change: () => void) => { + undo.push(data.snapshot()); + redo.splice(0).forEach((s) => s.destroy()); + change(); + }; + const step = (from: Snapshot[], to: Snapshot[]) => { + const back = from.pop(); + if (!back) return; + to.push(data.snapshot()); + data.restore(back); + back.destroy(); + count(); + }; + const count = () => { + const tables = data.describe().tables; + const rows = (names: string[]) => + tables.filter((t) => names.includes(t.name)).reduce((n, t) => n + t.rows, 0); + detail( + `${rows(kinds).toLocaleString()} blocks · ${rows(['Signal']).toLocaleString()} signals`, + ); + }; + count(); + + // A drag moves a block at once and is undone as one edit. + let dragging = false; + shown.on('drag', ({ phase, hit, position: [x, y] }) => { + const move = () => data.set(hit.row.table, [hit.row.id], { x: [x], y: [y] }); + if (phase === 'start' && !dragging) { + dragging = true; + edit(move); + } else move(); + if (phase === 'end') dragging = false; + }); + + // A wire joins an output to an input on the output's signal, making one if the output has none. + shown.on('connect', ({ from, to }) => { + if (!to || !(from.row.table in KINDS) || !(to.row.table in KINDS)) return; + const port = (row: Row, drives: boolean, named?: string) => + KINDS[row.table as Kind].ports.find( + ([name, , output]) => output === drives && (!named || name === named), + )?.[0]; + const output = port(from.row, true, from.port) ? from : to; + const input = output === from ? to : from; + const out = port(output.row, true, output.port); + const into = port(input.row, false, input.port); + if (!out || !into) return; + edit(() => { + // A reference field reads as the id of the row it references. + let signal = data.get(output.row.table, output.row.id, out) as Id | null; + if (signal === null) { + signal = Math.max(-1, ...data.query({ of: 'Signal', select: {} }).ids.map(Number)) + 1; + data.set('Signal', [signal], { name: [`signal ${signal}`] }); + data.set(output.row.table, [output.row.id], { [out]: reference('Signal', [signal]) }); + } + data.set(input.row.table, [input.row.id], { [into]: reference('Signal', [signal]) }); + }); + count(); + }); + shown.on('delete', ({ rows }) => { + edit(() => rows.forEach(({ table, ids }) => data.remove(table, ids))); + count(); + }); + + actions('Edit', [ + { label: 'Undo', key: 'z', run: () => step(undo, redo) }, + { label: 'Redo', key: 'y', run: () => step(redo, undo) }, + { + label: 'Arrange', + key: 'a', + // The layout places every block again and writes the places into its x and y fields. + run: () => { + void shown.arrange().then((patches) => edit(() => patches.forEach((p) => data.apply(p)))); + }, + }, + { label: 'Fit', key: 'home', run: () => shown.fit() }, + ]); + + return { + change(s, key) { + if (key !== 'preset') return; + const old = data; + data = s.preset === 'loop' ? loop() : floor(); + [...undo.splice(0), ...redo.splice(0)].forEach((snapshot) => snapshot.destroy()); + return () => { + old.destroy(); + shown.fit(); + count(); + }; + }, + }; + }, +}); diff --git a/npm/examples/src/pages/grids/cases.test.ts b/npm/examples/src/pages/grids/cases.test.ts new file mode 100644 index 00000000..d48ea92d --- /dev/null +++ b/npm/examples/src/pages/grids/cases.test.ts @@ -0,0 +1,47 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, test } from 'vitest'; +import { readCase } from './cases.js'; +import { inflate } from './format.js'; + +const CASES = ['wecc240', 'activsg2000', 'activsg10k', 'activsg25k', 'activsg70k'] as const; + +/** Reads case `id` from the file the page fetches. */ +async function read(id: string) { + return readCase(id, await inflate(readFileSync(`npm/examples/public/grids/${id}.bin`))); +} + +describe('grid cases', () => { + test.each(CASES)( + '%s reads into buses, lines, and machines positioned on the earth', + async (id) => { + const found = await read(id); + const rows = Object.fromEntries( + found.data.describe().tables.map((table) => [table.name, table.rows]), + ); + expect(rows).toMatchObject({ Bus: found.buses, Line: found.lines, Machine: found.machines }); + expect(found.buses).toBeGreaterThan(200); + for (let i = 0; i < found.positions.length; i += 2) { + expect(Math.abs(found.positions[i]!)).toBeLessThanOrEqual(180); + expect(Math.abs(found.positions[i + 1]!)).toBeLessThanOrEqual(90); + } + // Every line's `from` and `to` reference a bus of the case, and every machine's bus row exists. + const lines = found.data.query({ of: 'Line', select: { from: '$from', to: '$to' } }); + const ends = lines.fields as Record; + expect(ends.from!.ids.every((id) => id !== null)).toBe(true); + expect(ends.to!.ids.every((id) => id !== null)).toBe(true); + expect(found.machineBus.every((bus) => bus < found.buses)).toBe(true); + }, + ); + + test('a simulated case faults at a second, and its machines swing after', async () => { + const { simulation, machines } = await read('wecc240'); + expect(simulation?.fault?.on).toBeCloseTo(1, 1); + const frame = Math.round(3 / simulation!.dt); + const speeds = simulation!.speeds.subarray(frame * machines, (frame + 1) * machines); + expect(speeds.some((mhz) => Math.abs(mhz) > 0)).toBe(true); + }); + + test('bytes that are not a grid case are rejected', () => { + expect(() => readCase('nothing', new ArrayBuffer(16))).toThrow('not a grid case'); + }); +}); diff --git a/npm/examples/src/pages/grids/cases.ts b/npm/examples/src/pages/grids/cases.ts new file mode 100644 index 00000000..a0de8542 --- /dev/null +++ b/npm/examples/src/pages/grids/cases.ts @@ -0,0 +1,167 @@ +/** + * Synthetic grid cases, which the page fetches gzip-compressed from `public/grids`. `format.ts` + * describes the files and reads them. + */ +import { Data, Patch, reference } from 'latkit'; +import { inflate, parseCase, type Simulation } from './format.js'; + +export interface Case { + readonly data: Data; + readonly name: string; + readonly buses: number; + readonly lines: number; + readonly machines: number; + /** Bytes downloaded, compressed. */ + readonly bytes: number; + /** Whether its buses have names, which labels and search use. */ + readonly named: boolean; + /** Each bus's longitude and latitude, interleaved. `load` and `generation` are per bus, in MW. */ + readonly positions: Float32Array; + readonly load: Float32Array; + readonly generation: Float32Array; + /** Each machine's bus row. `machinePower` is each machine's output in MW. */ + readonly machineBus: Uint32Array; + readonly machinePower: Float32Array; + /** The simulation, with the fault it applied at the bus named `bus`. */ + readonly simulation?: Simulation & { + readonly fault?: { readonly bus: string; readonly on: number; readonly off: number }; + }; +} + +/** The bounding box of interleaved longitudes and latitudes: west, south, east, and north. */ +export function extent(positions: Float32Array): [number, number, number, number] { + const box: [number, number, number, number] = [Infinity, Infinity, -Infinity, -Infinity]; + for (let i = 0; i < positions.length; i += 2) { + box[0] = Math.min(box[0], positions[i]!); + box[1] = Math.min(box[1], positions[i + 1]!); + box[2] = Math.max(box[2], positions[i]!); + box[3] = Math.max(box[3], positions[i + 1]!); + } + return box; +} + +/** Reads an inflated case file into a dataset named `grid` with tables `Bus`, `Line`, `Machine`, + * and, for a simulated fault, `Fault`. Throws if the bytes are not a case. */ +export function readCase(id: string, buffer: ArrayBuffer, downloaded = buffer.byteLength): Case { + const c = parseCase(id, buffer); + const { buses, branches: lines, machines } = c.header; + const number = c.array('bus.number', Uint32Array); + const load = c.array('bus.load', Float32Array); + const generation = c.array('bus.generation', Float32Array); + const machinePower = c.array('machine.power', Float32Array); + const busNames = c.names('bus.name'); + const fault = c.simulation && c.header.fault; + // Buses are keyed by number, so lines reference them by number. + const numbered = (rows: Uint32Array) => Uint32Array.from(rows, (b) => number[b]!); + const patch = new Patch() + .set('Bus', number, { + lon: c.lon, + lat: c.lat, + name: busNames, + number: Float64Array.from(number), + kv: c.kv, + level: c.level, + load, + generation, + }) + .set( + 'Line', + { count: lines }, + { + from: reference('Bus', numbered(c.from)), + to: reference('Bus', numbered(c.to)), + kv: c.lineKv, + level: c.lineLevel, + reactance: c.array('branch.reactance', Float32Array), + transformer: Array.from(c.array('branch.transformer', Uint8Array), (v) => v !== 0), + }, + ) + .set( + 'Machine', + { count: machines }, + { + lon: Float64Array.from(c.machineBus, (b) => c.lon[b]!), + lat: Float64Array.from(c.machineBus, (b) => c.lat[b]!), + kv: Float64Array.from(c.machineBus, (b) => c.kv[b]!), + name: c.names('machine.name'), + power: machinePower, + mva: c.array('machine.mva', Float32Array), + }, + ); + if (fault) + patch.set('Fault', { count: 1 }, { lon: [c.lon[fault.bus]!], lat: [c.lat[fault.bus]!] }); + let simulation: Case['simulation']; + if (c.simulation) { + const { frames, dt, speeds } = c.simulation; + patch.axis('time', 's'); + for (let f = 0; f < frames; f++) + patch.append('Machine', f * dt, { speed: speeds.subarray(f * machines, (f + 1) * machines) }); + simulation = { + ...c.simulation, + ...(fault + ? { + fault: { bus: busNames[fault.bus] ?? `bus ${fault.bus}`, on: fault.on, off: fault.off }, + } + : {}), + }; + } + return { + data: new Data('grid').apply(patch), + name: c.header.case, + buses, + lines, + machines, + bytes: downloaded, + named: true, + positions: c.positions, + load, + generation, + machineBus: c.machineBus, + machinePower, + ...(simulation ? { simulation } : {}), + }; +} + +/** Reports download progress: the bytes received so far, and the total if the response gives it. */ +export type Progress = (received: number, total: number | undefined) => void; + +/** Formats download progress in megabytes for a status line, with the total when known. */ +export const megabytes = (received: number, total: number | undefined) => + `${(received / 1e6).toFixed(1)}${total ? ` of ${(total / 1e6).toFixed(1)}` : ''} MB`; + +/** + * Downloads a file and returns its bytes. `progress` is called for each chunk. It gets no total + * once the bytes received exceed the response's length, as they do when the transfer is + * compressed. + */ +export async function download( + url: string, + signal: AbortSignal, + progress?: Progress, +): Promise> { + const response = await fetch(url, { + signal: AbortSignal.any([signal, AbortSignal.timeout(180_000)]), + }); + if (!response.ok || !response.body) throw new Error(`${url}: HTTP ${response.status}`); + const total = Number(response.headers.get('content-length')) || undefined; + const chunks: Uint8Array[] = []; + let received = 0; + const reader = response.body.getReader(); + for (let read = await reader.read(); !read.done; read = await reader.read()) { + chunks.push(read.value); + received += read.value.length; + progress?.(received, total !== undefined && total >= received ? total : undefined); + } + const bytes = new Uint8Array(received); + chunks.reduce((at, chunk) => (bytes.set(chunk, at), at + chunk.length), 0); + return bytes; +} + +export async function loadCase( + id: string, + signal: AbortSignal, + progress?: Progress, +): Promise { + const bytes = await download(`${import.meta.env.BASE_URL}grids/${id}.bin`, signal, progress); + return readCase(id, await inflate(bytes), bytes.length); +} diff --git a/npm/examples/src/examples/networks/continent.ts b/npm/examples/src/pages/grids/continent.ts similarity index 88% rename from npm/examples/src/examples/networks/continent.ts rename to npm/examples/src/pages/grids/continent.ts index b5f32490..41a50f39 100644 --- a/npm/examples/src/examples/networks/continent.ts +++ b/npm/examples/src/pages/grids/continent.ts @@ -1,13 +1,15 @@ import { random } from '../../random.js'; /** - * A synthetic grid the size of a continent's. Buses cluster around North America's cities as a real - * grid's do: every substation feeds radial distribution feeders, which hold most of the buses; each - * city's substations mesh at 115 and 230 kV; and cities join along 345 to 765 kV corridors through - * switching stations, the busiest by double or triple circuits, with rural substations and power - * plants along the way. Seeded, so every build draws the same grid, in time linear in its size. + * A synthetic grid the size of a continent's. Its buses cluster around North America's cities, as a + * real grid's do. Each substation feeds radial distribution feeders, which hold most of the buses. + * Each city's substations form a mesh at 115 and 230 kV. Cities connect along 230 to 765 kV + * corridors through switching stations, with double or triple circuits on the busiest, and rural + * substations and power plants along the way. The generator is seeded, so every build produces the + * same grid, in time linear in its size. */ -import { Data, Patch, reference } from '../../latkit.js'; -import { levelOf, type Case } from './cases.js'; +import { Data, Patch, reference } from 'latkit'; +import type { Case } from './cases.js'; +import { levelOf } from './format.js'; /** Metro areas: name, longitude, latitude, and millions of people. */ const CITIES: readonly (readonly [string, number, number, number])[] = [ @@ -186,13 +188,14 @@ const STATION_SPACING = 90; /** The share of buses fed along corridors rather than in cities. */ const RURAL = 0.12; -/** Kilometres between two points, near enough for a continent. */ +/** Approximate kilometres between two points, accurate enough at the scale of a continent. */ function km(lon0: number, lat0: number, lon1: number, lat1: number): number { const across = (lon1 - lon0) * Math.cos(((lat0 + lat1) * Math.PI) / 360); return Math.hypot(across, lat1 - lat0) * KM; } -/** Buses and lines as they are made, in typed arrays that grow by half again. */ +/** Collects buses and lines in typed arrays that grow by half when full, and machines in plain + * arrays. */ class Builder { lon = new Float64Array(1024); lat = new Float64Array(1024); @@ -268,8 +271,8 @@ class Builder { } } -/** The cheapest tree joining `n` points, by Prim's algorithm on squared distance: each point's parent - * in it, the first point's being -1. */ +/** The minimum spanning tree of `n` points under `distance`, by Prim's algorithm in O(n²). Returns + * each point's parent, with -1 for point 0, the root. */ function spanning(n: number, distance: (a: number, b: number) => number): Int32Array { const parent = new Int32Array(n).fill(-1); const best = new Float64Array(n).fill(Infinity); @@ -291,8 +294,8 @@ function spanning(n: number, distance: (a: number, b: number) => number): Int32A return parent; } -/** Feeds `count` distribution buses from substation bus `root`: radial feeders that run outward from - * it and branch, steps of `step` degrees apart. */ +/** Adds `count` 13.8 kV distribution buses fed from substation bus `root`, on radial feeders that + * run outward and branch, with buses about `step` degrees apart. */ function feed(grid: Builder, rng: () => number, root: number, count: number, step: number): void { const feeders = Math.max(3, Math.round((count / 50) * (0.8 + 0.4 * rng()))); const lat0 = grid.lat[root]!; @@ -304,13 +307,13 @@ function feed(grid: Builder, rng: () => number, root: number, count: number, ste ? left : Math.min(left, Math.round((count / feeders) * (0.6 + 0.8 * rng()))); left -= size; - // The feeder's buses, and the way each one's line ran. + // The feeder's first bus, and the heading of the line to each of its buses. const first = grid.buses; const headings = new Float32Array(size); let heading = (2 * Math.PI * (f + rng())) / feeders; let parent = root; for (let k = 0; k < size; k++) { - // Mostly the trunk runs on; now and then a lateral branches off a bus already fed. + // Usually the trunk continues. Sometimes (18%) a lateral branches off an earlier bus. if (k > 0 && rng() < 0.18) { const from = Math.floor(rng() * k); parent = first + from; @@ -329,12 +332,13 @@ function feed(grid: Builder, rng: () => number, root: number, count: number, ste } } -/** The synthetic grids, by id, and the buses each is built to. */ +/** The target bus count of each synthetic grid, by id. */ export const CONTINENTS: Readonly> = { 'continent-2m': 2_000_000, }; -/** The synthetic grid `id` as a case named `grid`, as the case files are read. */ +/** Builds synthetic grid `id` as a `Case` with a dataset named `grid`, in the form `readCase` + * returns. Throws for an unknown id. */ export function continent(id: string): Case { const target = CONTINENTS[id]; if (!target) throw new Error(`No synthetic grid ${id}`); @@ -343,8 +347,8 @@ export function continent(id: string): Case { const weights = CITIES.map(([, , , people]) => people ** 0.85); const total = weights.reduce((a, b) => a + b, 0); - // Corridors first, as a graph of cities: the cheapest tree joining them all, and each city's two - // nearest neighbours within 700 km. + // Plan the corridors first, as a graph of cities: a minimum spanning tree joining them all, plus + // links from each city to its two nearest neighbours within 700 km. const n = CITIES.length; const distance = (a: number, b: number) => km(CITIES[a]![1], CITIES[a]![2], CITIES[b]![1], CITIES[b]![2]); @@ -360,7 +364,8 @@ export function continent(id: string): Case { for (const b of near) if (distance(a, b) < 700) corridors.add(pair(a, b)); } - // Each city: substations meshing at 115 and 230 kV, each feeding its share of the city's buses. + // In each city, substations form a mesh at 115 and 230 kV, and each feeds its share of the + // city's buses. const hubs = new Map(); const substations: number[][] = []; for (let c = 0; c < n; c++) { @@ -388,7 +393,7 @@ export function continent(id: string): Case { const [a, b] = [subs[p]!, subs[s]!]; grid.line(a, b, Math.min(grid.kv[a]!, grid.kv[b]!), grid.kv[a] !== grid.kv[b]); }); - // A mesh as well as a tree: each substation also joins one more near it. + // Links beyond the tree make a mesh: each substation also joins a random other one nearby. for (let s = 1; s < count; s++) { const other = Math.floor(rng() * count); if (other !== s && other !== parent[s]) { @@ -399,13 +404,14 @@ export function continent(id: string): Case { grid.line(a, b, Math.min(grid.kv[a]!, grid.kv[b]!), grid.kv[a] !== grid.kv[b]); } } - // Feeders run shorter where people crowd closer. + // Feeder spans are shorter in bigger cities, where people live closer together. const step = 0.0012 + 0.002 / Math.sqrt(1 + people); for (let s = 0; s < count; s++) feed(grid, rng, subs[s]!, Math.max(0, Math.floor(buses / count) - 1), step); } - /** City `c`'s bus at `kv`, made on first use and joined to its central substation. */ + /** City `c`'s bus at `kv`, created on first use and connected through a transformer to the + * city's central substation. */ const hub = (c: number, kv: number) => { const key = c * 1000 + kv; const found = hubs.get(key); @@ -419,8 +425,8 @@ export function continent(id: string): Case { return bus; }; - // Corridors: each a chain of switching stations, its circuits as many as its cities need, with a - // rural substation and its feeders off every station, and a power plant at some. + // Each corridor is a chain of switching stations, with one to three circuits by the population + // of its two cities. Every station feeds a rural substation, and some have a power plant. let stations = 0; const links = [...corridors].map((p) => [Math.floor(p / n), p % n] as const); for (const [a, b] of links) stations += Math.floor(distance(a, b) / STATION_SPACING); @@ -441,7 +447,7 @@ export function continent(id: string): Case { const circuits = people > 14 ? 3 : people > 6 ? 2 : 1; const length = distance(a, b); const hops = Math.floor(length / STATION_SPACING) + 1; - // The corridor bends a little off the straight line, the same way along its whole length. + // The corridor bows slightly to one side of the straight line along its whole length. const bend = (rng() - 0.5) * 0.12; let previous = hub(a, kv); for (let h = 1; h <= hops; h++) { diff --git a/npm/examples/src/pages/grids/format.ts b/npm/examples/src/pages/grids/format.ts new file mode 100644 index 00000000..08cb5c7e --- /dev/null +++ b/npm/examples/src/pages/grids/format.ts @@ -0,0 +1,153 @@ +/** + * The synthetic grid cases in `public/grids`, which the networks page and the docs gallery read. + * + * A case file is gzip-compressed. Inflated, it starts with the magic "LKG1" and a u32 header length, + * followed by a JSON header with the case's name and counts. Typed arrays follow on eight-byte + * boundaries. The header's `arrays` gives each as [type, offset, length], with the offset in bytes + * from the first eight-byte boundary after the header and the length in elements. The arrays hold: + * - per bus: number, name, kV, longitude and latitude, and load and generation in MW; + * - per branch: its two bus rows, reactance, and whether it is a transformer; + * - per machine: bus row, name, rating, and output. + * + * Simulated cases also hold each machine's speed deviation for every frame, `dt` seconds apart, as + * int16 in steps of `speedScale` per unit, and the fault the simulation applied. + */ + +export interface Header { + readonly case: string; + readonly buses: number; + readonly branches: number; + readonly machines: number; + readonly frames?: number; + readonly dt?: number; + readonly speedScale?: number; + readonly fault?: { readonly bus: number; readonly on: number; readonly off: number }; + readonly arrays: Readonly< + Record + >; +} + +/** A case's simulation: each machine's speed deviation over the simulated time. */ +export interface Simulation { + readonly frames: number; + readonly dt: number; + readonly duration: number; + /** Speed deviation in mHz, one value per machine for each frame, frame-major. */ + readonly speeds: Float32Array; + /** The largest deviation either way, in mHz: the int16 maximum at the file's `speedScale`. */ + readonly largest: number; +} + +/** A case file's header and arrays, and the columns its readers derive from them. */ +export interface CaseFile { + readonly header: Header; + /** Returns one of the case's arrays by name, such as `branch.reactance`. Throws if it is + * missing. */ + array(name: string, of: new (b: ArrayBuffer, o: number, l: number) => T): T; + /** Returns the strings of one of the case's text arrays, such as `bus.name`. */ + names(name: string): string[]; + /** Each bus's longitude and latitude, interleaved in `positions` and apart in `lon` and `lat`. */ + readonly positions: Float32Array; + readonly lon: Float64Array; + readonly lat: Float64Array; + /** Each bus's kV and voltage level. */ + readonly kv: Float32Array; + readonly level: Float32Array; + /** Each branch's two bus rows, its kV (the higher of its buses'), and its voltage level. */ + readonly from: Uint32Array; + readonly to: Uint32Array; + readonly lineKv: Float32Array; + readonly lineLevel: Float32Array; + /** Each machine's bus row. */ + readonly machineBus: Uint32Array; + readonly simulation?: Simulation; +} + +/** The voltage level of a bus or line, an index into `LEVELS`: 0 below 100 kV, then 115–161, 230, + * 345, 500, and 765 kV. */ +export function levelOf(kv: number): number { + return kv >= 700 ? 5 : kv >= 450 ? 4 : kv >= 300 ? 3 : kv >= 200 ? 2 : kv >= 100 ? 1 : 0; +} +export const LEVELS = ['< 100 kV', '115–161 kV', '230 kV', '345 kV', '500 kV', '765 kV'] as const; + +/** Returns `bytes` as an `ArrayBuffer`, inflated if they are gzip-compressed, as case files are. */ +export function inflate(bytes: Uint8Array): Promise { + if (bytes[0] !== 0x1f || bytes[1] !== 0x8b) + return Promise.resolve( + bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength), + ); + const inflated = new Blob([bytes]).stream().pipeThrough(new DecompressionStream('gzip')); + return new Response(inflated).arrayBuffer(); +} + +const MAGIC = 0x31474b4c; +const text = new TextDecoder(); + +/** Reads an inflated case file. Throws if the bytes are not a case, naming it `id`. */ +export function parseCase(id: string, buffer: ArrayBuffer): CaseFile { + const view = new DataView(buffer); + if (buffer.byteLength < 8 || view.getUint32(0, true) !== MAGIC) + throw new Error(`${id} is not a grid case`); + const length = view.getUint32(4, true); + const header = JSON.parse(text.decode(new Uint8Array(buffer, 8, length))) as Header; + const body = Math.ceil((8 + length) / 8) * 8; + const array = (name: string, of: new (b: ArrayBuffer, o: number, l: number) => T): T => { + const found = header.arrays[name]; + if (!found) throw new Error(`${id} has no ${name}`); + return new of(buffer, body + found[1], found[2]); + }; + const names = (name: string): string[] => { + const offsets = array(`${name}.offsets`, Uint32Array); + const bytes = array(`${name}.bytes`, Uint8Array); + return Array.from({ length: offsets.length - 1 }, (_, i) => + text.decode(bytes.subarray(offsets[i]!, offsets[i + 1]!)), + ); + }; + const n = header.buses; + const m = header.branches; + const positions = array('bus.position', Float32Array); + const kv = array('bus.kv', Float32Array); + const ends = array('branch.ends', Uint32Array); + const lon = new Float64Array(n); + const lat = new Float64Array(n); + for (let b = 0; b < n; b++) { + lon[b] = positions[b * 2]!; + lat[b] = positions[b * 2 + 1]!; + } + const from = new Uint32Array(m); + const to = new Uint32Array(m); + const lineKv = new Float32Array(m); + for (let e = 0; e < m; e++) { + from[e] = ends[e * 2]!; + to[e] = ends[e * 2 + 1]!; + lineKv[e] = Math.max(kv[from[e]!]!, kv[to[e]!]!); + } + let simulation: Simulation | undefined; + if (header.frames !== undefined && header.dt !== undefined) { + // Converts int16 steps to mHz: a step is `speedScale` per unit, and 1 per unit is 60 Hz. + const scale = (header.speedScale ?? 1) * 60_000; + simulation = { + frames: header.frames, + dt: header.dt, + duration: (header.frames - 1) * header.dt, + speeds: Float32Array.from(array('machine.speed', Int16Array), (value) => value * scale), + largest: 32767 * scale, + }; + } + return { + header, + array, + names, + positions, + lon, + lat, + kv, + level: Float32Array.from(kv, levelOf), + from, + to, + lineKv, + lineLevel: Float32Array.from(lineKv, levelOf), + machineBus: array('machine.bus', Uint32Array), + ...(simulation ? { simulation } : {}), + }; +} diff --git a/npm/examples/src/pages/grids/index.ts b/npm/examples/src/pages/grids/index.ts new file mode 100644 index 00000000..8d9a7637 --- /dev/null +++ b/npm/examples/src/pages/grids/index.ts @@ -0,0 +1,217 @@ +/** + * Power grids on the earth: synthetic cases from WECC 240 buses to ACTIVSg70k, and a grid of North + * America's two million buses built in the page, over coasts and borders, states, or + * OpenStreetMap. Color buses by voltage level, load, or generation, raise them by their voltage, or + * lay the grid out by its topology instead of its geography and watch every bus ease to its place. + */ +import { field, when, zoom, type Mark, type Scene } from 'latkit'; +import { choice, page, toggle } from '../../shell/page.js'; +import { extent, loadCase, megabytes, type Case } from './cases.js'; +import { CONTINENTS, continent } from './continent.js'; +import { LEVELS } from './format.js'; +import { LOOK, MAPS, NO_MAP, type Basemap } from './maps.js'; + +/** Each voltage level's color, from below 100 kV to 765 kV. */ +const LEVEL_COLORS = ['#56657a', '#2fa49b', '#7fc97f', '#e6c84f', '#f08a3c', '#e5486f']; +/** Noon at midsummer over the western US, in seconds since the epoch. */ +const NOON = Date.UTC(2026, 5, 21, 19) / 1000; + +/** A grid's case and its largest load and generation, which their colormaps span. */ +interface Grid extends Case { + readonly peak: { readonly load: number; readonly generation: number }; +} + +/** Loads case `id`, or builds the synthetic continent, reporting progress to `detail`. */ +async function load( + id: string, + signal: AbortSignal, + detail: (text: string) => void, +): Promise { + detail(`Loading ${id}…`); + const found = + id in CONTINENTS + ? continent(id) + : await loadCase(id, signal, (got, total) => + detail(`Loading ${id}, ${megabytes(got, total)}`), + ); + const peak = (values: Float32Array) => values.reduce((most, v) => Math.max(most, v), 1); + detail( + `${found.name} · ${found.buses.toLocaleString()} buses · ${found.lines.toLocaleString()} lines`, + ); + return { ...found, peak: { load: peak(found.load), generation: peak(found.generation) } }; +} + +export default page({ + title: 'Power grids', + settings: { + grid: choice( + 'Grid', + { + 'WECC 240': 'wecc240', + ACTIVSg2000: 'activsg2000', + ACTIVSg10k: 'activsg10k', + ACTIVSg25k: 'activsg25k', + ACTIVSg70k: 'activsg70k', + 'North America · 2M': 'continent-2m', + }, + { value: 'activsg25k' }, + ), + color: choice('Color', { 'Voltage level': 'level', Load: 'load', Generation: 'generation' }), + height: choice('Height', { Flat: 'flat', 'Voltage (kV)': 'kv' }, { key: 'h' }), + layout: choice('Layout', { Geography: 'geography', Topology: 'topology' }, { key: 't' }), + labels: toggle('Labels', { value: true, key: 'l' }), + comets: toggle('Comets', { key: 'c' }), + map: choice( + 'Map', + { + 'Coasts and borders': 'borders', + 'States and provinces': 'states', + OpenStreetMap: 'planet', + None: 'none', + }, + { section: 'Earth' }, + ), + globe: toggle('Globe', { value: true, key: 'g', section: 'Earth' }), + light: choice('Light', { Noon: 'noon', Now: 'now', Even: 'even' }, { section: 'Earth' }), + }, + + async open({ settings, signal, view, detail, credit, fail }) { + let grid = await load(settings.grid, signal, detail); + // The map joins the scene once its datasets are read, so the grid draws first. + let map: Basemap = NO_MAP; + + const scene = (s: typeof settings): Scene => { + const geography = s.layout === 'geography'; + const data = grid.data; + const big = grid.buses > 20_000; + const level = field('level').colormap({ + kind: 'categorical', + colors: LEVEL_COLORS, + categories: [...LEVELS], + }); + // On the earth, height is a fraction of its radius: 765 kV stands about 140 km high. Laid out + // by topology, it is in the layout's units. + const height = s.height === 'kv' ? field('kv').mul(geography ? 2.5e-5 : 0.15) : 0; + const where = geography ? { position: ['$lon', '$lat'] as ['$lon', '$lat'] } : {}; + const marks: Mark[] = [ + ...(geography ? map.marks : []), + { + edges: data.table('Line'), + ends: ['from', 'to'], + // Higher-voltage lines draw on top, and stay visible as the camera zooms out. + priority: '$level', + color: level, + width: field('level') + .scale(big ? 0.5 : 0.8, big ? 1.8 : 2.8) + .domain(0, 5), + dash: '$transformer', + flow: s.comets ? field('level').mul(6).add(8) : 0, + }, + { + vertices: data.table('Bus'), + ...where, + elevation: height, + priority: '$level', + color: + s.color === 'level' + ? level + : field(s.color) + .colormap(s.color === 'load' ? 'mako' : 'amber') + .domain(0, grid.peak[s.color]), + radius: big ? field('level').scale(0.8, 2.6).domain(0, 5) : grid.buses > 5000 ? 2 : 3, + legend: + s.color === 'level' + ? 'Voltage level' + : s.color === 'load' + ? 'Load (MW)' + : 'Generation (MW)', + // A crowded grid names its buses only once zoomed in. + ...(s.labels && grid.named + ? { + label: { + text: when(zoom().ge(geography ? 5.5 : 3)) + .then(field('name')) + .otherwise(null), + size: 10, + }, + } + : {}), + }, + ]; + if (geography && grid.machines) + marks.push({ + vertices: data.table('Machine'), + position: ['$lon', '$lat'], + elevation: height, + marker: 'diamond', + color: '#ebf0f7', + radius: field('power') + .scale(big ? 2 : 3, big ? 5 : 8) + .domain(0, grid.peak.generation), + }); + return { + space: geography + ? { + earth: { + projection: s.globe ? 'globe' : 'mercator', + graticule: false, + atmosphere: s.globe, + ...(s.light === 'even' + ? {} + : { daylight: s.light === 'now' ? Date.now() / 1000 : NOON }), + }, + } + : { plane: { grid: 'off' } }, + layout: geography ? 'none' : 'stress', + theme: { + base: 'dark', + background: LOOK.background, + area: { color: LOOK.fill }, + earth: { ground: LOOK.ground, atmosphere: '#5aa9ff99', night: 0.28 }, + edge: { bow: 0.12 }, + flow: { color: '#d6e6ffa6', width: 3, length: 20, spacing: 72 }, + }, + marks, + }; + }; + + /** Frames the grid's buses. */ + const frame = () => { + if (settings.layout !== 'geography') return shown.fit(); + const [west, south, east, north] = extent(grid.positions); + shown.fit({ bounds: { min: [west, south, 0], max: [east, north, 0] } }); + }; + /** Swaps in map `id` once its datasets are read, unless another map was chosen meanwhile. */ + const showMap = async (id: typeof settings.map) => { + const next = await MAPS[id](signal); + if (id !== settings.map || signal.aborted) { + for (const d of next.data) d.destroy(); + return; + } + const old = map; + map = next; + credit(map.credit); + shown.scene = scene(settings); + for (const d of old.data) d.destroy(); + }; + const shown = await view(scene, { label: 'A power grid: hover or click a bus or line' }); + frame(); + showMap(settings.map).catch(fail); + + return { + async change(s, key) { + if (key === 'grid') { + const next = await load(s.grid, signal, detail); + const old = grid; + grid = next; + return () => { + frame(); + old.data.destroy(); + }; + } + if (key === 'map') showMap(s.map).catch(fail); + if (key === 'layout') return frame; + }, + }; + }, +}); diff --git a/npm/examples/src/pages/grids/maps.ts b/npm/examples/src/pages/grids/maps.ts new file mode 100644 index 00000000..551a13d9 --- /dev/null +++ b/npm/examples/src/pages/grids/maps.ts @@ -0,0 +1,110 @@ +/** + * The maps drawn beneath a grid: + * + * - Natural Earth's coasts and borders, shipped with the page. + * - Natural Earth's states and provinces, downloaded from its repository. + * - OpenStreetMap's planet as OpenFreeMap's vector tiles, fetched as the camera needs them. + * + * GeoJSON, plain or gzip-compressed, is read in a worker. + */ +import { Data, Tiles, field, when, zoom, type Mark } from 'latkit'; +import { download } from './cases.js'; + +/** The page's colors, as CSS. */ +export const LOOK = { + background: '#05070a', + ground: '#0e131b', + fill: '#1a212b', + line: '#6b80948c', + coast: '#8ca3b8d9', + landcover: '#1c2924', + park: '#1a3021', + water: '#0d1a2b', + waterway: '#2e5485e6', + road: '#5c616bb3', + place: '#c3ccd8', +} as const; + +/** A loaded map: the marks drawn beneath the grid, who to credit, and the datasets to free. */ +export interface Basemap { + readonly marks: readonly Mark[]; + readonly credit: string; + readonly data: readonly Data[]; +} + +/** No map: the earth's ground alone. */ +export const NO_MAP: Basemap = { credit: '', data: [], marks: [] }; + +const NATURAL_EARTH = 'Natural Earth · public domain'; +const NE = 'https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/'; + +export const MAPS: Record< + 'borders' | 'states' | 'planet' | 'none', + (signal: AbortSignal) => Basemap | Promise +> = { + /** Coastlines, country borders, and province borders at 1:50M, as lines. */ + async borders(signal) { + const read = (name: string) => + Data.read(`${import.meta.env.BASE_URL}maps/${name}.geojson.gz`, { name, signal }); + const [coast, countries, provinces] = await Promise.all( + ['coastline', 'countries', 'provinces'].map(read), + ); + return { + credit: NATURAL_EARTH, + data: [coast!, countries!, provinces!], + marks: [ + { lines: provinces!.table('lines'), color: LOOK.line, width: 0.6, dash: true }, + { lines: countries!.table('lines'), color: LOOK.line, width: 1 }, + { lines: coast!.table('lines'), color: LOOK.coast, width: 1 }, + ], + }; + }, + + /** States and provinces at 1:50M, as areas named by their labels. */ + async states(signal) { + const bytes = await download(`${NE}ne_50m_admin_1_states_provinces.geojson`, signal); + const states = await Data.read(bytes, { name: 'states', signal }); + return { + credit: NATURAL_EARTH, + data: [states], + marks: [ + { + areas: states.table('polygons'), + color: LOOK.fill, + label: { text: '$name', size: 10, color: '#8e99aa' }, + }, + { lines: states.table('polygons'), color: LOOK.line, width: 0.7 }, + ], + }; + }, + + /** OpenStreetMap's planet, with rivers that carry comets downstream. */ + planet() { + const base = new Tiles('base', 'https://tiles.openfreemap.org/planet'); + return { + credit: 'OpenFreeMap · © OpenMapTiles · © OpenStreetMap contributors', + data: [], + marks: [ + { areas: base.table('landcover'), color: LOOK.landcover }, + { areas: base.table('park'), color: LOOK.park }, + { areas: base.table('water'), color: LOOK.water }, + { lines: base.table('waterway'), color: LOOK.waterway, width: 0.8, flow: 10 }, + { lines: base.table('transportation'), color: LOOK.road, width: 0.7 }, + { lines: base.table('boundary'), color: LOOK.line, width: 1, dash: true }, + // A place is named once the zoom passes its rank, where 1 is the most important. + { + vertices: base.table('place'), + color: LOOK.place, + radius: 1.5, + label: { + text: when(field('rank').lt(zoom())).then(field('name')).otherwise(null), + size: 11, + color: LOOK.place, + }, + }, + ], + }; + }, + + none: () => NO_MAP, +}; diff --git a/npm/examples/src/pages/orbits.ts b/npm/examples/src/pages/orbits.ts new file mode 100644 index 00000000..5d90b21c --- /dev/null +++ b/npm/examples/src/pages/orbits.ts @@ -0,0 +1,85 @@ +/** + * A spiral galaxy whose bodies orbit its center. Each body's position is an expression of the + * elapsed time, its radius, phase, and angular speed, evaluated on the GPU: once the data is + * uploaded, no frame writes a byte, however many bodies move. Inner bodies orbit faster than outer + * ones, so the arms wind as time passes. Near the pointer, bodies swell like a lens. + */ +import { Data, elapsed, field, pointer, type CustomColormap } from 'latkit'; +import { page, choice, toggle } from '../shell/page.js'; +import { random } from '../random.js'; + +/** Glowing colors, translucent where faint, so crowded bodies brighten one another. */ +const GLOW: CustomColormap = { + kind: 'sequential', + colors: ['#2a1b5c66', '#7b2cbf99', '#e0559dbb', '#ff9e64dd', '#fff1c9'], +}; + +/** Places `n` bodies in a disk with four arms: their radius, phase, angular speed, and height. */ +function galaxy(n: number) { + const next = random(7); + const gauss = () => Math.sqrt(-2 * Math.log(next() + 1e-12)) * Math.cos(2 * Math.PI * next()); + const r = new Float64Array(n); + const phase = new Float64Array(n); + const speed = new Float64Array(n); + const z = new Float64Array(n); + for (let i = 0; i < n; i++) { + r[i] = 0.3 + 12 * next() ** 1.6; + // Arms trail along a logarithmic spiral, with bodies scattered about them. + const arm = (i % 4) * (Math.PI / 2); + phase[i] = arm + 1.9 * Math.log(r[i]!) + 0.35 * gauss(); + // A flat rotation curve: the same orbital speed at every radius, so angular speed falls as 1/r. + speed[i] = 0.9 / (r[i]! + 0.6); + z[i] = 0.25 * gauss() * Math.exp(-r[i]! / 5); + } + return { r, phase, speed, z }; +} + +export default page({ + title: 'Orbits', + inspect: false, + settings: { + bodies: choice( + 'Bodies', + { '50 thousand': 50_000, '200 thousand': 200_000, '1 million': 1_000_000 }, + { value: 200_000 }, + ), + color: choice('Color', { 'Orbital speed': 'speed', Distance: 'r' }), + lens: toggle('Lens', { value: true, key: 'l' }), + }, + async open({ settings, view, detail }) { + const data = new Data('galaxy').set('Body', galaxy(settings.bodies)); + const angle = field('phase').add(field('speed').mul(elapsed())); + const shown = await view( + (s) => ({ + space: { plane: { grid: 'off' } }, + theme: { base: 'dark', background: '#05060b' }, + controls: { hover: false }, + marks: [ + { + vertices: data.table('Body'), + position: [field('r').mul(angle.cos()), field('r').mul(angle.sin())], + elevation: '$z', + // Up to three times as large right at the pointer, normal 120 pixels away. + radius: s.lens ? pointer().div(120).neg().add(1).max(0).mul(2).add(1).mul(1.3) : 1.3, + color: + s.color === 'speed' + ? field('speed').colormap(GLOW).domain(0.07, 1.2) + : field('r').neg().colormap(GLOW).domain(-12.3, -0.3), + pick: false, + }, + ], + }), + { camera: { center: [0, 0], zoom: 5, pitch: 52, bearing: 0 } }, + ); + // The disk fills the view, seen from the camera's angle. + shown.fit({ bounds: { min: [-12.5, -12.5, 0], max: [12.5, 12.5, 0] } }); + detail(`${settings.bodies.toLocaleString()} bodies, moved by the GPU`); + return { + change(s, key) { + if (key !== 'bodies') return; + data.replace('Body', galaxy(s.bodies)); + detail(`${s.bodies.toLocaleString()} bodies, moved by the GPU`); + }, + }; + }, +}); diff --git a/npm/examples/src/pages/telemetry.ts b/npm/examples/src/pages/telemetry.ts new file mode 100644 index 00000000..2cb5d8c2 --- /dev/null +++ b/npm/examples/src/pages/telemetry.ts @@ -0,0 +1,160 @@ +/** + * Telemetry: thousands of streaming voltages traced over a scrolling window. Frames are appended as + * they arrive and the oldest are forgotten. Pause or scrub the timeline to read back what passed. + * Every 20 seconds one area dips and recovers. Strike one yourself, and query for the signals hit + * hardest. Or draw a recorded day, fetched span by span as time tiles at the detail the plot needs. + */ +import { Data, Tiles, field, selected, when, type CustomColormap } from 'latkit'; +import { choice, page } from '../shell/page.js'; +import { random } from '../random.js'; + +const AREAS = ['North', 'South', 'East', 'West', 'Coast', 'Valley']; +const BY_AREA: CustomColormap = { + kind: 'categorical', + colors: ['#4e79a7', '#f28e2b', '#e15759', '#76b7b2', '#59a14f', '#edc948'], + categories: AREAS, +}; +/** Samples per second, seconds kept, seconds shown, and seconds between automatic dips. */ +const RATE = 50; +const KEEP = 60; +const WINDOW = 20; +const EVERY = 20; + +/** How far a dip has pulled voltages down, `since` seconds after it struck. 0 after six seconds. */ +const dip = (since: number) => + since < 0 || since > 6 + ? 0 + : 0.06 * Math.exp(-since * 1.4) * Math.sin(Math.min(since, 0.5) * Math.PI); + +/** `n` signals' areas, names, offsets, and noise levels. */ +function signals(n: number) { + const next = random(11); + const area = Float64Array.from({ length: n }, (_, i) => i % AREAS.length); + const name = Array.from( + { length: n }, + (_, i) => `${AREAS[i % AREAS.length]} ${Math.floor(i / AREAS.length) + 1}`, + ); + const offset = Float32Array.from({ length: n }, () => next() - 0.5); + const noise = Float32Array.from({ length: n }, () => 0.3 + next()); + return { area, name, offset, noise, next }; +} + +export default page({ + title: 'Telemetry', + settings: { + source: choice('Source', { Live: 'live', 'Recorded day': 'day' }), + count: choice('Signals', { '500': 500, '2,000': 2000, '10,000': 10_000 }, { value: 2000 }), + color: choice('Color', { Area: 'area', Voltage: 'value' }, { key: 'c' }), + }, + + async open({ settings, view, actions, timeline, frame, detail }) { + let s = signals(settings.count); + const live = new Data('live').axis('time', 's').set('Signal', { name: s.name, area: s.area }); + const day = new Tiles('scada', 'history/frames.json'); + let t = 0; + let struck = -Infinity; + let area = 0; + + const plot = await view( + (settings) => { + const table = settings.source === 'live' ? live.table('Signal') : day.table('Signal'); + return { + space: { + plot: { + x: { label: settings.source === 'live' ? 'Time (s)' : 'Time of day (s)' }, + y: { label: 'Voltage (pu)' }, + }, + }, + theme: { base: 'dark', background: '#101214', focus: { dim: 0.15 } }, + ease: 0, + marks: [ + { + lines: table, + trace: 'value', + width: when(selected()) + .then(2.5) + .otherwise(settings.count > 2000 ? 0.6 : 1), + color: + settings.color === 'area' + ? field('area').colormap(BY_AREA) + : field('value').colormap('coolwarm').domain(0.94, 1.04), + legend: settings.color === 'area' ? 'Area' : 'Voltage (pu)', + }, + ], + }; + }, + { camera: { x: [-WINDOW, 0] } }, + ); + timeline({ data: live, keep: KEEP }); + + actions('Actions', [ + { + label: 'Strike an area', + key: 's', + run: () => { + area = (area + 1) % AREAS.length; + struck = t; + }, + }, + { + label: 'Select the lowest', + key: 'q', + // A query reads the latest frame and returns the twelve lowest voltages, which the plot + // then highlights. + run: () => { + const lowest = live.query({ + of: 'Signal', + sort: [['$value', 'ascending']], + limit: 12, + at: t, + }); + plot.selection = [{ data: 'live', table: 'Signal', ids: lowest.ids }]; + }, + }, + ]); + + // Every frame appends the samples that came due since the last, each signal swinging with its + // area, with noise, and dipping when its area is struck. + let carry = 0; + let values = new Float32Array(settings.count); + frame((dt) => { + if (settings.source !== 'live') return; + carry += dt; + while (carry >= 1 / RATE) { + carry -= 1 / RATE; + t += 1 / RATE; + const automatic = Math.floor(t / EVERY) % AREAS.length; + for (let i = 0; i < values.length; i++) { + const a = s.area[i]!; + const dips = (a === automatic ? dip(t % EVERY) : 0) + (a === area ? dip(t - struck) : 0); + const swing = Math.sin(t * (0.6 + a * 0.13) + a); + values[i] = + 1 + 0.03 * s.offset[i]! + 0.012 * swing + 0.003 * (s.next() - 0.5) * s.noise[i]! - dips; + } + live.append('Signal', t, { value: values }); + } + plot.camera = { x: [t - WINDOW, t] }; + }); + detail(`${settings.count.toLocaleString()} signals at ${RATE} Hz, ${KEEP} s kept`); + + return { + change(next, key) { + if (key === 'count') { + s = signals(next.count); + values = new Float32Array(next.count); + live.replace('Signal', { name: s.name, area: s.area }); + detail(`${next.count.toLocaleString()} signals at ${RATE} Hz, ${KEEP} s kept`); + } + if (key === 'source') + return () => { + if (next.source === 'day') plot.camera = { x: [0, 86_400] }; + detail( + next.source === 'day' + ? '64 buses every 15 s for a day, fetched as time tiles' + : `${next.count.toLocaleString()} signals at ${RATE} Hz`, + ); + }; + }, + }; + }, +}); diff --git a/npm/examples/src/random.ts b/npm/examples/src/random.ts index 452785bf..92d1bb3a 100644 --- a/npm/examples/src/random.ts +++ b/npm/examples/src/random.ts @@ -1,4 +1,5 @@ -/** Deterministic sample data, shared by the synthetic examples. */ +/** Returns a seeded random generator of numbers in [0, 1). The synthetic examples use it so their + * sample data is the same on every load. */ export function random(seed: number): () => number { return () => { seed = (seed + 0x6d2b79f5) | 0; @@ -7,3 +8,10 @@ export function random(seed: number): () => number { return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; } + +/** Returns a seeded linear congruential generator of numbers in [0, 1): a simpler sequence than + * `random`'s, which the soft body's burst and the benchmark's data are drawn from. */ +export function congruential(seed: number): () => number { + let s = seed >>> 0 || 1; + return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 2 ** 32; +} diff --git a/npm/examples/src/ui/dom.ts b/npm/examples/src/shell/dom.ts similarity index 76% rename from npm/examples/src/ui/dom.ts rename to npm/examples/src/shell/dom.ts index c80f7a76..f4ede5e0 100644 --- a/npm/examples/src/ui/dom.ts +++ b/npm/examples/src/shell/dom.ts @@ -1,7 +1,10 @@ type Child = Node | string | null | undefined | false; type Attributes = Record; -/** An element with attributes and children; `false` and `undefined` attributes are left off. */ +/** + * Creates an element with `attributes` and `children`. Attributes that are `false` or `undefined` + * are omitted, `true` sets an empty attribute, and falsy children are skipped. + */ export function h( tag: K, attributes: Attributes = {}, diff --git a/npm/examples/src/shell/page.ts b/npm/examples/src/shell/page.ts new file mode 100644 index 00000000..8c3e3e58 --- /dev/null +++ b/npm/examples/src/shell/page.ts @@ -0,0 +1,148 @@ +/** + * What an example page declares: its settings, which the shell shows as controls, and an `open` + * function that builds its data and views. Each view's scene is a function of the settings. The + * shell redraws it whenever a setting changes, so a page has no code that restyles its views. + */ +import type { Camera, Data, Rect, Scene, View } from 'latkit'; + +/** One value of a page's settings and the control that sets it. */ +export type Setting = { + readonly label: string; + readonly value: T; + /** The side panel section it shows in. Defaults to `Settings`. */ + readonly section?: string; + /** A key that changes it: a toggle flips, and a choice steps to its next option. */ + readonly key?: string; +} & ( + | { readonly kind: 'choice'; readonly options: readonly (readonly [string, T])[] } + | { readonly kind: 'toggle' } + | { + readonly kind: 'range'; + readonly min: number; + readonly max: number; + readonly step: number; + readonly format: (value: number) => string; + } +); + +type Where = { section?: string; key?: string }; + +/** A dropdown of `options`, keyed by their labels. Defaults to the first option. */ +export function choice( + label: string, + options: Readonly>, + where: Where & { value?: T } = {}, +): Setting { + const entries = Object.entries(options); + return { + kind: 'choice', + label, + options: entries, + value: where.value ?? entries[0]![1], + ...where, + }; +} + +/** A toggle button. Defaults to off. */ +export function toggle(label: string, where: Where & { value?: boolean } = {}): Setting { + return { kind: 'toggle', label, ...where, value: where.value ?? false }; +} + +/** A slider. */ +export function range( + label: string, + o: { min: number; max: number; step: number; value: number; format: (value: number) => string }, + where: Where = {}, +): Setting { + return { kind: 'range', label, ...o, ...where }; +} + +export type Settings = Readonly>>; +/** The current values of `S`. */ +export type Values = { -readonly [K in keyof S]: S[K]['value'] }; + +/** A button, also run by its key. */ +export interface Action { + readonly label: string; + readonly key?: string; + readonly run: () => void; + /** Runs `run` on press and `release` on release, for an action held down. */ + readonly release?: () => void; +} + +/** The recorded span of a dataset with an axis, played live, paused, or replayed. */ +export interface Timeline { + readonly mode: 'live' | 'playing' | 'paused'; + /** The axis coordinate every view of the page shows. */ + readonly at: number; + /** Goes live, following the end of the data. */ + play(): void; +} + +/** What a page can do while it is open. Everything made through it closes with the page. */ +export interface Context { + /** The current settings. */ + readonly settings: Readonly; + /** Aborted when the page closes. */ + readonly signal: AbortSignal; + /** + * Creates a view on the page's canvas, in `region`, that draws `scene` for the current settings + * and redraws it after each change. Without a camera, it frames everything drawn. + */ + view( + scene: (settings: V) => Scene, + options?: { region?: Rect; camera?: Camera; label?: string }, + ): Promise; + /** Adds buttons to the side panel, in section `title`. */ + actions(title: string, actions: readonly Action[]): void; + /** + * Plays `data`'s axis in a timeline under the canvas. Live, it follows the end of the data and + * forgets frames older than `keep` seconds. Paused or replaying, it shows earlier frames. Every + * view on the page shows the timeline's coordinate. + */ + timeline(options: { + data: Data; + keep?: number; + /** Plays back the last `seconds` at `rate`. */ + replay?: { seconds: number; rate: number }; + /** Called after the views move to coordinate `at`. */ + show?(at: number): void; + /** Called when the timeline goes live or stops being live. */ + change?(live: boolean): void; + }): Timeline; + /** Calls `step` every animation frame while the page is live, with the seconds since the last. */ + frame(step: (dt: number) => void): void; + /** Lists the colors of a view's series in the side panel. */ + series(title: string, items: readonly { label: string; color: string }[]): void; + /** Shows short text in the header, such as the page's counts or what is loading. */ + detail(text: string): void; + /** Credits the data's source under the canvas. */ + credit(text: string): void; + /** Shows the page as failed with `error`, such as from work it finishes after opening. */ + fail(error: unknown): void; +} + +/** Nothing, or a function to run once the views draw the new settings. */ +export type Changed = void | (() => void); + +/** What a page returns from `open`. */ +export interface Opened { + /** + * Called when setting `key` changes, before the views draw the new settings. It may return a + * function to run once they do, such as one that frames what they now draw. + */ + change?(settings: V, key: keyof V): Changed | Promise; +} + +export interface Page { + readonly title: string; + readonly settings?: S; + /** Whether hovering or clicking a row shows its fields in the side panel. Defaults to true. */ + readonly inspect?: boolean; + open(context: Context>): Promise> | void>; +} + +/** Declares a page, typing its settings' values. */ +export function page(page: Page): Page { + return page; +} diff --git a/npm/examples/src/shell/shell.ts b/npm/examples/src/shell/shell.ts new file mode 100644 index 00000000..f71ef426 --- /dev/null +++ b/npm/examples/src/shell/shell.ts @@ -0,0 +1,544 @@ +/** + * The examples' shell: the layout every page shares, and everything a page does not declare + * itself. It shows a page's settings as controls and redraws each view's scene when one changes. + * It runs the page's frames and timeline, and gives every page an inspector, a performance readout, + * and picture and video export. + */ +import { + Shot, + View, + load, + type Camera, + type Gpu, + type Rect, + type Row, + type Scene, + type Value, +} from 'latkit'; +import { h } from './dom.js'; +import type { Action, Context, Opened, Page, Setting, Settings } from './page.js'; +import { Timeline, type TimelineOptions } from './timeline.js'; + +declare global { + interface Window { + /** The open page's views and their GPU, for the browser gate. */ + stage?: { views: View[]; gpu?: Gpu }; + } +} + +/** A page in the picker: its id in the URL, and its module. */ +export interface Entry { + readonly id: string; + readonly title: string; + load(): Promise<{ default: Page }>; +} + +/** Fields the inspector leaves out: positions, which the view already shows. */ +const HIDDEN = /^(lon|lat|x|y|z|geometry|position)$/; +const MB = 1 << 20; + +/** Starts the examples: the shared layout, and the page the URL names. */ +export function start(pages: readonly Entry[]): void { + const picker = h('select', { 'aria-label': 'Example' }); + picker.append(...pages.map(({ id, title }) => h('option', { value: id }, title))); + picker.addEventListener('change', () => (location.hash = picker.value)); + const regions = { + detail: h('p', { class: 'detail' }), + error: h('div', { role: 'alert', class: 'error', hidden: true }), + main: h('main', { id: 'example' }), + side: h('aside', { id: 'controls', 'aria-label': 'Controls' }), + footer: h('footer', { 'aria-label': 'Playback' }), + }; + const toggle = h( + 'button', + { type: 'button', class: 'controls-toggle', 'aria-expanded': 'true' }, + 'Controls', + ); + toggle.addEventListener('click', () => { + regions.side.hidden = !regions.side.hidden; + toggle.setAttribute('aria-expanded', String(!regions.side.hidden)); + }); + document.body.append( + h( + 'header', + {}, + h('a', { class: 'brand', href: `#${pages[0]!.id}` }, 'latkit'), + h('label', { class: 'example-picker' }, h('span', {}, 'Examples'), picker), + regions.detail, + toggle, + ), + regions.error, + regions.main, + regions.side, + regions.footer, + ); + + let open: OpenPage | undefined; + let opening = 0; + const show = async () => { + const id = location.hash.slice(1).split('?')[0]; + const entry = pages.find((p) => p.id === id) ?? pages[0]!; + const opened = ++opening; + open?.close(); + open = undefined; + regions.error.hidden = true; + picker.value = entry.id; + document.title = `${entry.title} · latkit`; + regions.detail.textContent = 'Loading…'; + try { + const [{ default: page }] = await Promise.all([entry.load(), load()]); + if (opened !== opening) return; + open = new OpenPage(page, regions, (error) => opened === opening && fail(error)); + await open.start(); + } catch (error) { + if (opened === opening) fail(error); + } + }; + const fail = (error: unknown) => { + const unsupported = (error as { code?: string }).code === 'unsupported'; + regions.error.hidden = false; + regions.error.replaceChildren( + h( + 'p', + {}, + unsupported + ? 'WebGPU is unavailable. Open these examples in a browser with WebGPU enabled.' + : `Could not run this example. ${String(error)}`, + ), + ); + console.error(error); + }; + // Settings update the URL without firing this, so a change here comes from a link or the address + // bar. The page reopens with the settings in the URL. + addEventListener('hashchange', () => void show()); + void show(); +} + +type Regions = { + detail: HTMLElement; + error: HTMLElement; + main: HTMLElement; + side: HTMLElement; + footer: HTMLElement; +}; + +/** An open page, and the context it runs in. */ +class OpenPage implements Context> { + readonly id: string; + readonly settings: Record = {}; + readonly #abort = new AbortController(); + readonly signal = this.#abort.signal; + readonly #views: { view: View; scene: (settings: never) => Scene; region: Rect }[] = []; + /** The page's steps, run while it is live, and the shell's, run every frame. */ + readonly #steps: ((dt: number) => void)[] = []; + readonly #ticks: (() => void)[] = []; + readonly #keys = new Map void>(); + readonly #actions = h('div'); + readonly #panel = h('div'); + #canvas?: HTMLCanvasElement; + #timeline?: Timeline; + #opened?: Opened>; + #stepMs = 0; + + constructor( + private readonly page: Page, + private readonly regions: Regions, + readonly fail: (error: unknown) => void, + ) { + this.id = location.hash.slice(1).split('?')[0] ?? ''; + const given = new URLSearchParams(location.hash.split('?')[1]); + for (const [name, setting] of Object.entries(page.settings ?? {})) { + this.settings[name] = valueFrom(setting, given.get(name)) ?? setting.value; + } + regions.side.replaceChildren(this.#actions, this.#panel); + regions.side.hidden = false; + this.#controls(page.settings ?? {}); + addEventListener( + 'keydown', + (event) => { + const typing = (event.target as Element).closest?.('input, select, textarea, button'); + const action = this.#keys.get(event.key.toLowerCase()); + if (!action || typing || event.ctrlKey || event.metaKey || event.altKey) return; + event.preventDefault(); + action(); + }, + { signal: this.signal }, + ); + } + + async start(): Promise { + this.#opened = (await this.page.open(this as never)) ?? undefined; + if (this.signal.aborted) return; + const views = this.#views.map(({ view }) => view); + window.stage = { views, gpu: views[0]?.gpu }; + if (this.page.inspect !== false) this.#inspector(); + if (views[0]) { + this.#meter(views[0]); + this.#export(views[0]); + } + let last: number | undefined; + const stop = views[0]?.gpu.on('frame', (now) => { + const dt = last === undefined ? 0 : Math.min(0.1, (now - last) / 1000); + last = now; + try { + if (!this.#timeline || this.#timeline.mode === 'live') { + const start = performance.now(); + for (const step of this.#steps) step(dt); + this.#stepMs = this.#stepMs * 0.9 + (performance.now() - start) * 0.1; + } + this.#timeline?.tick(dt); + for (const tick of this.#ticks) tick(); + } catch (error) { + this.fail(error); + } + }); + if (stop) this.signal.addEventListener('abort', stop); + if (this.regions.detail.textContent === 'Loading…') this.regions.detail.textContent = ''; + } + + close(): void { + this.#abort.abort(); + for (const { view } of this.#views) view.destroy(); + for (const region of [this.regions.main, this.regions.side, this.regions.footer]) + region.replaceChildren(); + this.regions.detail.textContent = ''; + window.stage = undefined; + } + + // The context a page runs in. Its functions are bound, so a page can destructure them. + + view = async ( + scene: (settings: never) => Scene, + options: { region?: Rect; camera?: Camera; label?: string } = {}, + ): Promise => { + this.#canvas ??= this.regions.main.appendChild( + h('canvas', { tabindex: 0, 'aria-label': options.label ?? this.page.title }), + ); + const region = options.region ?? [0, 0, 1, 1]; + const view = await View.create(this.#canvas, scene(this.settings as never), { + region: this.#fitted(region), + ...(options.camera ? { camera: options.camera } : {}), + }); + view.on('error', (error) => + error.code === 'io' ? this.detail(error.message) : this.fail(error), + ); + this.#views.push({ view, scene, region }); + return view; + }; + + actions = (title: string, actions: readonly Action[]): void => { + const row = h('div', { class: 'actions' }); + for (const action of actions) { + const button = h('button', { type: 'button' }, action.label, ...kbd(action.key)); + if (action.release) { + button.addEventListener('pointerdown', () => action.run(), { signal: this.signal }); + for (const type of ['pointerup', 'pointercancel', 'pointerleave']) + button.addEventListener(type, () => action.release!(), { signal: this.signal }); + } else button.addEventListener('click', () => action.run(), { signal: this.signal }); + if (action.key) this.#keys.set(action.key.toLowerCase(), action.run); + row.append(button); + } + this.#actions.append(section(title, true, row)); + }; + + timeline = (options: TimelineOptions): Timeline => { + this.#timeline = new Timeline( + options, + this.regions.footer, + (key, run) => this.#keys.set(key, run), + (at) => this.#views.forEach(({ view }) => (view.at = at)), + this.signal, + ); + return this.#timeline; + }; + + frame = (step: (dt: number) => void): void => { + this.#steps.push(step); + }; + + series = (title: string, items: readonly { label: string; color: string }[]): void => { + const swatches = items.map(({ label, color }) => + h('span', { style: `--swatch: ${color}` }, label), + ); + this.#panel.append(section(title, true, h('p', { class: 'series' }, ...swatches))); + }; + + detail = (text: string): void => { + this.regions.detail.textContent = text; + }; + + credit = (text: string): void => { + const line = this.regions.main.querySelector('.credit') ?? h('p', { class: 'credit' }); + line.textContent = text; + this.regions.main.append(line); + }; + + // What the shell adds to every page. + + /** Shows each setting as a control in its section, and applies changes. */ + #controls(settings: Settings): void { + const sections = new Map(); + for (const [name, setting] of Object.entries(settings)) { + const title = setting.section ?? 'Settings'; + let body = sections.get(title); + if (!body) { + const group = this.#panel.appendChild(section(title, true)); + sections.set(title, (body = group.lastElementChild as HTMLElement)); + } + const set = (value: unknown) => void this.#change(name, value); + body.append(this.#control(setting, this.settings[name], set)); + } + } + + #control(setting: Setting, value: unknown, set: (value: unknown) => void): HTMLElement { + if (setting.kind === 'toggle') { + const button = h('button', { type: 'button', 'aria-pressed': String(value) }, setting.label); + button.append(...kbd(setting.key)); + const flip = () => { + const on = button.getAttribute('aria-pressed') !== 'true'; + button.setAttribute('aria-pressed', String(on)); + set(on); + }; + button.addEventListener('click', flip, { signal: this.signal }); + if (setting.key) this.#keys.set(setting.key, flip); + return button; + } + if (setting.kind === 'range') { + const input = h('input', { + type: 'range', + min: setting.min, + max: setting.max, + step: setting.step, + }); + input.value = String(value); + const output = h('output', {}, setting.format(value as number)); + input.addEventListener( + 'input', + () => { + output.value = setting.format(input.valueAsNumber); + set(input.valueAsNumber); + }, + { signal: this.signal }, + ); + return h('label', { class: 'field' }, h('span', {}, setting.label, output), input); + } + const options = setting.options; + const select = h( + 'select', + {}, + ...options.map(([label, v], i) => h('option', { value: i, selected: v === value }, label)), + ); + select.addEventListener('change', () => set(options[select.selectedIndex]![1]), { + signal: this.signal, + }); + if (setting.key) + this.#keys.set(setting.key, () => { + select.selectedIndex = (select.selectedIndex + 1) % options.length; + set(options[select.selectedIndex]![1]); + }); + return h( + 'label', + { class: 'field' }, + h('span', {}, setting.label, ...kbd(setting.key)), + select, + ); + } + + /** Sets `name` to `value`, updates the URL, and redraws every view's scene. */ + async #change(name: string, value: unknown): Promise { + this.settings[name] = value; + const query = new URLSearchParams(); + for (const [key, setting] of Object.entries(this.page.settings ?? {})) + if (this.settings[key] !== setting.value) query.set(key, String(this.settings[key])); + history.replaceState(null, '', `#${this.id}${query.size ? `?${query}` : ''}`); + try { + const after = await this.#opened?.change?.(this.settings, name); + if (this.signal.aborted) return; + for (const { view, scene } of this.#views) view.scene = scene(this.settings as never); + after?.(); + } catch (error) { + this.fail(error); + } + } + + /** A view's region, with its axes swapped on narrow screens so side-by-side views stack. */ + #fitted(region: Rect): Rect { + const [x, y, width, height] = region; + return matchMedia('(max-width: 700px)').matches ? [y, x, height, width] : region; + } + + /** The Inspect section: the fields of the row under the pointer, or of the last one clicked. */ + #inspector(): void { + const title = h('p', { class: 'note' }, 'Hover or click a row.'); + const fields = h('dl', { class: 'fields' }); + this.#panel.append(section('Inspect', true, title, fields)); + let pinned: { view: View; row: Row } | undefined; + const read = (view: View, row: Row) => { + const data = view.snapshot(row.data); + if (!data) return; + try { + const table = data.describe().tables.find((t) => t.name === row.table); + title.textContent = `${row.table} ${row.id}`; + fields.replaceChildren(); + for (const { name } of table?.fields ?? []) { + if (HIDDEN.test(name)) continue; + const value = data.get(row.table, row.id, name, { at: view.at }); + if (value === null) continue; + fields.append(h('dt', {}, name), h('dd', {}, format(value))); + } + } finally { + data.destroy(); + } + }; + for (const { view } of this.#views) { + view.on('hovered', ({ hit }) => { + if (hit) read(view, hit.row); + else if (pinned) read(pinned.view, pinned.row); + }); + view.on('click', ({ hit }) => { + pinned = { view, row: hit.row }; + read(view, hit.row); + }); + } + } + + /** The Performance section: frame rate and times, GPU work and memory, and the GPU's budget. */ + #meter(view: View): void { + const line = h('p', { class: 'meter', 'aria-live': 'off' }); + const gpu = view.gpu; + const given = gpu.budget; + this.signal.addEventListener('abort', () => (gpu.budget = given)); + const memory = h( + 'select', + {}, + ...[given.bytes, 256 * MB, 64 * MB, 16 * MB].map((bytes, i) => + h('option', { value: bytes }, i ? `${bytes / MB} MB` : 'Default'), + ), + ); + memory.addEventListener( + 'change', + () => (gpu.budget = { ...gpu.budget, bytes: Number(memory.value) }), + ); + this.#panel.append( + section( + 'Performance', + false, + line, + h('label', { class: 'field' }, h('span', {}, 'GPU memory'), memory), + ), + ); + let frames = 0; + let since = performance.now(); + this.#ticks.push(() => { + frames++; + const now = performance.now(); + if (now - since < 500) return; + const stats = view.stats; + line.textContent = [ + `${((frames * 1000) / (now - since)).toFixed(0)} fps`, + `frame ${stats.cpuMs.toFixed(1)} ms${stats.gpuMs ? ` · GPU ${stats.gpuMs.toFixed(2)} ms` : ''}`, + `${stats.drawn.toLocaleString()} rows drawn · ${stats.draws} draws`, + `${(gpu.resident / MB).toFixed(0)} of ${(gpu.budget.bytes / MB).toFixed(0)} MB on the GPU`, + ...(this.#stepMs > 0.05 ? [`page ${this.#stepMs.toFixed(1)} ms a frame`] : []), + ...(view.waiting.length ? [`waiting on ${view.waiting.join(', ')}`] : []), + ].join('\n'); + frames = 0; + since = now; + }); + } + + /** The Export section: a PNG of the canvas, or six seconds of it as MP4. */ + #export(view: View): void { + const status = h('p', { class: 'note', role: 'status' }); + const save = (name: string, file: Blob) => { + const url = URL.createObjectURL(file); + h('a', { href: url, download: name }).click(); + setTimeout(() => URL.revokeObjectURL(url), 10_000); + }; + const scale = () => Math.min(2, devicePixelRatio); + const picture = async () => { + const image = await Shot.of(view, { scale: scale(), capture: 'now' }).image(); + const png = await image.encode('png'); + save('views.png', new Blob([png.slice()], { type: 'image/png' })); + status.textContent = `Saved views.png, ${image.width}×${image.height}`; + image.destroy(); + }; + let recording = false; + const video = async () => { + if (recording || typeof VideoEncoder !== 'function') return; + recording = true; + const chunks: Uint8Array[] = []; + const output = new WritableStream({ + write: (chunk) => void chunks.push(chunk.slice()), + }); + const started = performance.now(); + try { + // Each frame waits for the page to reach its time, so the video plays at the page's speed. + await Shot.of(view, { scale: scale() }) + .video({ fps: 30, output }) + .record(async (time) => { + while (performance.now() - started < time * 1000) + await new Promise((resolve) => requestAnimationFrame(resolve)); + status.textContent = `Recording ${time.toFixed(1)} of 6 s`; + return time < 6; + }); + save('views.mp4', new Blob(chunks, { type: 'video/mp4' })); + status.textContent = 'Saved views.mp4'; + } finally { + recording = false; + } + }; + const run = (act: () => Promise) => () => + void act().catch( + (error: unknown) => (status.textContent = `Could not export: ${String(error)}`), + ); + const row = h('div', { class: 'actions' }); + for (const [label, key, act] of [ + ['Picture', 'x', picture], + ['Video', 'v', video], + ] as const) { + const button = h('button', { type: 'button' }, label, ...kbd(key)); + button.addEventListener('click', run(act), { signal: this.signal }); + this.#keys.set(key, run(act)); + row.append(button); + } + this.#panel.append(section('Export', false, row, status)); + } +} + +/** A collapsible titled group of the side panel. */ +function section(title: string, open: boolean, ...children: Node[]): HTMLElement { + return h( + 'details', + { class: 'section', open }, + h('summary', {}, title), + h('div', { class: 'section-body' }, ...children), + ); +} + +/** The key label of a button, if it has a key. */ +function kbd(key: string | undefined): HTMLElement[] { + return key ? [h('kbd', {}, key === ' ' ? 'Space' : key.toUpperCase())] : []; +} + +/** A setting's value from its URL parameter, or `undefined` when absent or not one of its values. */ +function valueFrom(setting: Setting, given: string | null): unknown { + if (given === null) return undefined; + if (setting.kind === 'toggle') return given === 'true'; + if (setting.kind === 'range') return Number.isFinite(Number(given)) ? Number(given) : undefined; + return setting.options.find(([, v]) => String(v) === given)?.[1]; +} + +/** + * A field's value for the inspector: a number, a text, a point, or the word `geometry` for a longer + * geometry. A reference shows as the id of the row it references. + */ +function format(value: Value): string { + if (typeof value === 'number') + return Number.isInteger(value) ? value.toLocaleString() : value.toPrecision(4); + if (Array.isArray(value)) + return typeof value[0] === 'number' + ? value.map((v) => format(v as number)).join(', ') + : 'geometry'; + return String(value); +} diff --git a/npm/examples/src/ui/style.css b/npm/examples/src/shell/style.css similarity index 83% rename from npm/examples/src/ui/style.css rename to npm/examples/src/shell/style.css index f55f947a..24529c3c 100644 --- a/npm/examples/src/ui/style.css +++ b/npm/examples/src/shell/style.css @@ -140,10 +140,6 @@ aside { gap: 12px; padding: 2px 16px 16px; } -.section canvas { - width: 100%; - height: 180px; -} .actions { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); @@ -254,7 +250,6 @@ input[type='text'] { font: 11px/1.8 var(--mono); white-space: pre-line; } -.legend, .credit { position: absolute; margin: 0; @@ -265,44 +260,11 @@ input[type='text'] { background: var(--bg); padding: 4px 6px; } -.legend { - right: 12px; - bottom: 10px; - display: flex; - flex-wrap: wrap; - align-items: center; - gap: 8px; - max-width: calc(100% - 24px); -} -.legend.top { - top: 10px; - bottom: auto; -} .credit { left: 12px; bottom: 10px; max-width: 48%; } -.ramp { - width: 88px; - height: 6px; -} -.menu { - position: absolute; - z-index: 2; - display: grid; - min-width: 180px; - padding: 4px; - background: var(--surface); - border: 1px solid var(--line-strong); - border-radius: var(--radius); -} -.menu button { - justify-content: start; - text-align: left; - background: none; - border-color: transparent; -} .fields { display: grid; grid-template-columns: auto minmax(0, 1fr); @@ -319,50 +281,6 @@ input[type='text'] { font-variant-numeric: tabular-nums; overflow-wrap: anywhere; } -.swatches { - display: grid; - grid-template-columns: repeat(2, minmax(0, 1fr)); - gap: 6px; -} -.swatch { - display: grid; - grid-template-columns: minmax(0, 1fr); - gap: 5px; - padding: 6px 8px; - font-size: 11px; - text-align: left; -} -.swatch .ramp { - width: 100%; -} -.hot { - display: grid; - gap: 4px; - margin: 0; - padding: 0; - list-style: none; -} -.hot button { - display: grid; - grid-template-columns: 2ch minmax(0, 1fr) auto; - gap: 3px 8px; - width: 100%; - padding: 5px 8px; - font-size: 12px; - font-variant-numeric: tabular-nums; - text-align: left; -} -.hot .bar { - grid-column: 1 / -1; - height: 2px; - background: linear-gradient(to right, var(--accent) var(--score), var(--raised) var(--score)); -} -.hot button[aria-pressed='true'], -.swatch[aria-pressed='true'] { - color: var(--text); - background: var(--raised); - border-color: var(--accent); -} footer { grid-area: footer; display: flex; @@ -497,8 +415,4 @@ body:has(aside[hidden]) :is(main, footer) { grid-template-columns: minmax(0, 1fr); grid-template-rows: 56px auto minmax(0, 1fr) auto 0; } - .legend { - right: 8px; - bottom: 8px; - } } diff --git a/npm/examples/src/shell/timeline.ts b/npm/examples/src/shell/timeline.ts new file mode 100644 index 00000000..d50437b3 --- /dev/null +++ b/npm/examples/src/shell/timeline.ts @@ -0,0 +1,128 @@ +/** + * A page's timeline: the recorded span of a dataset with an axis, played live, paused, or + * replayed, with playback controls under the canvas. When live, it follows the end of the data and + * forgets frames older than the span it keeps. + */ +import type { Data } from 'latkit'; +import { h } from './dom.js'; +import type { Timeline as Shown } from './page.js'; + +export interface TimelineOptions { + data: Data; + keep?: number; + replay?: { seconds: number; rate: number }; + show?(at: number): void; + change?(live: boolean): void; +} + +export class Timeline implements Shown { + mode: 'live' | 'playing' | 'paused' = 'live'; + #at = 0; + #span: readonly [number, number] = [0, 0]; + #rate = 1; + readonly #toggle = h('button', { type: 'button', class: 'primary' }); + readonly #slider = h('input', { + type: 'range', + step: 'any', + min: 0, + max: 1, + 'aria-label': 'Time', + }); + readonly #time = h('output', {}, '0.00 s'); + + constructor( + private readonly options: TimelineOptions, + footer: HTMLElement, + key: (key: string, run: () => void) => void, + /** Moves every view of the page to coordinate `at`. */ + private readonly move: (at: number) => void, + signal: AbortSignal, + ) { + const transport = h('div', { class: 'transport' }, this.#toggle); + const flip = () => (this.mode === 'paused' ? this.play() : this.#set('paused')); + this.#toggle.addEventListener('click', flip, { signal }); + key('k', flip); + this.#slider.addEventListener('input', () => this.#seek(this.#slider.valueAsNumber), { + signal, + }); + if (options.replay) { + const replay = h('button', { type: 'button' }, 'Replay', h('kbd', {}, 'R')); + replay.addEventListener('click', () => this.replay(), { signal }); + key('r', () => this.replay()); + transport.append(replay); + } + transport.append(this.#time); + footer.append(transport, this.#slider); + document.addEventListener('visibilitychange', () => document.hidden && this.#set('paused'), { + signal, + }); + this.#label(); + } + + get at(): number { + return this.#at; + } + + /** Goes live: shows the end of the data as it records. */ + play(): void { + this.#rate = 1; + this.#set('live'); + this.#show(this.#span[1]); + } + + /** Plays the last `replay.seconds` at `replay.rate`, then goes live. */ + replay(): void { + const { replay } = this.options; + if (!replay) return; + this.#rate = replay.rate; + this.#set('playing'); + this.#show(Math.max(this.#span[0], this.#span[1] - replay.seconds)); + } + + /** Advances by `dt` seconds of display time. Live, it takes the data's newest span. */ + tick(dt: number): void { + if (this.mode === 'live') { + const span = this.options.data.describe().span; + if (!span || span[1] === this.#span[1]) return; + const { keep } = this.options; + if (keep !== undefined && span[1] - span[0] > keep) this.options.data.forget(span[1] - keep); + this.#extent([Math.max(span[0], span[1] - (keep ?? Infinity)), span[1]]); + this.#show(span[1]); + } else if (this.mode === 'playing') { + const next = this.#at + dt * this.#rate; + if (next < this.#span[1]) this.#show(next); + else this.play(); + } + } + + #extent(span: readonly [number, number]): void { + this.#span = span; + this.#slider.min = String(span[0]); + this.#slider.max = String(Math.max(span[1], span[0] + 1e-3)); + } + + #seek(at: number): void { + this.#set('paused'); + this.#show(at); + } + + #set(mode: Timeline['mode']): void { + const was = this.mode === 'live'; + this.mode = mode; + this.#label(); + if (was !== (mode === 'live')) this.options.change?.(mode === 'live'); + } + + #label(): void { + this.#toggle.textContent = this.mode === 'paused' ? 'Live' : 'Pause'; + this.#toggle.append(h('kbd', {}, 'K')); + } + + #show(at: number): void { + this.#at = at; + this.#slider.value = String(at); + this.#time.value = `${at.toFixed(2)} s`; + this.move(at); + this.options.show?.(at); + } +} diff --git a/npm/examples/src/ui/controls.ts b/npm/examples/src/ui/controls.ts deleted file mode 100644 index 4e39da6b..00000000 --- a/npm/examples/src/ui/controls.ts +++ /dev/null @@ -1,148 +0,0 @@ -import type { Scene, View } from 'latkit'; -import { h } from './dom.js'; -import type { Stage } from './stage.js'; - -/** A button that runs `run`, and the key that runs it too. */ -export interface Action { - label: string; - key: string; - run: () => void; -} - -const keyName = (key: string) => (key === ' ' ? 'Space' : key.toUpperCase()); - -/** A titled group of controls in the side panel. */ -export class Section { - readonly element: HTMLElement; - readonly content = h('div', { class: 'section-body' }); - /** The row toggles share, made on first use. */ - #toggles?: HTMLElement; - - constructor( - private readonly stage: Stage, - title: string, - open = true, - ) { - this.element = h('details', { class: 'section', open }, h('summary', {}, title), this.content); - } - - /** Buttons in a grid, each also run by its key. */ - actions(actions: readonly Action[]): void { - const row = h('div', { class: 'actions' }); - for (const { label, key, run } of actions) { - const button = h('button', { type: 'button' }, label, h('kbd', {}, keyName(key))); - button.addEventListener('click', run, { signal: this.stage.signal }); - this.stage.key(key, run); - row.append(button); - } - this.content.append(row); - } - - /** A button that stays pressed, also flipped by its key. Returns a setter for outside changes. */ - toggle( - label: string, - key: string, - value: boolean, - change: (on: boolean) => void, - ): (on: boolean) => void { - const pressed = String(value); - const button = h( - 'button', - { type: 'button', 'aria-pressed': pressed }, - label, - h('kbd', {}, keyName(key)), - ); - const set = (on: boolean) => { - button.setAttribute('aria-pressed', String(on)); - change(on); - }; - const flip = () => set(button.getAttribute('aria-pressed') !== 'true'); - button.addEventListener('click', flip, { signal: this.stage.signal }); - this.stage.key(key, flip); - if (!this.#toggles) this.content.append((this.#toggles = h('div', { class: 'actions' }))); - this.#toggles.append(button); - return set; - } - - /** A slider with its value shown through `format`. */ - range( - label: string, - o: { min: number; max: number; step: number; value: number; format: (value: number) => string }, - change: (value: number) => void, - ): void { - const input = h('input', { - type: 'range', - min: o.min, - max: o.max, - step: o.step, - value: o.value, - }); - const output = h('output'); - const update = () => { - output.value = o.format(input.valueAsNumber); - change(input.valueAsNumber); - }; - input.addEventListener('input', update, { signal: this.stage.signal }); - update(); - this.content.append(h('label', { class: 'field' }, h('span', {}, label, output), input)); - } - - /** One of several options; returns the select, for outside changes. */ - choice( - label: string, - options: readonly (readonly [string, T])[], - value: T, - change: (value: T) => void, - ): HTMLSelectElement { - const select = h( - 'select', - {}, - ...options.map(([name, v], i) => h('option', { value: i, selected: v === value }, name)), - ); - select.addEventListener('change', () => change(options[select.selectedIndex]![1]), { - signal: this.stage.signal, - }); - this.content.append(h('label', { class: 'field' }, h('span', {}, label), select)); - return select; - } - - /** Swatches naming the series a view draws. */ - series(items: readonly { label: string; color: string }[]): HTMLElement { - const swatches = items.map(({ label, color }) => - h('span', { style: `--swatch: ${color}` }, label), - ); - const line = h('p', { class: 'series' }, ...swatches); - this.content.append(line); - return line; - } - - /** A line of text the page keeps up to date. */ - note(text = ''): HTMLParagraphElement { - const line = h('p', { class: 'note' }, text); - this.content.append(line); - return line; - } - - /** A view inside this section. */ - view(scene: Scene, options: { label: string; height?: number }): View { - return this.stage.view(scene, { ...options, into: this.content }); - } - - /** Standard playback speed control. */ - speed(rates: readonly number[], value: number, change: (value: number) => void): void { - this.choice( - 'Speed', - rates.map((rate) => [`${rate}×`, rate] as const), - value, - change, - ); - } - - /** A labeled text field, such as a selected item's name. */ - text(label: string, change: (value: string) => void): HTMLInputElement { - const input = h('input', { type: 'text' }); - input.addEventListener('change', () => change(input.value), { signal: this.stage.signal }); - this.content.append(h('label', { class: 'field' }, h('span', {}, label), input)); - return input; - } -} diff --git a/npm/examples/src/ui/export.ts b/npm/examples/src/ui/export.ts deleted file mode 100644 index e68f3f3d..00000000 --- a/npm/examples/src/ui/export.ts +++ /dev/null @@ -1,122 +0,0 @@ -import { Shot, type View } from '../latkit.js'; -import { h } from './dom.js'; -import type { Stage } from './stage.js'; - -/** Seconds of video a recording takes. */ -const SECONDS = 6; -/** Frames a second of video. */ -const FPS = 30; - -/** - * The Export section every page gets: a picture of its views laid out as the page shows them, - * written as PNG in a worker, and a video of them as MP4, recorded as the page runs, with its - * progress, stopped by pressing again. - */ -export function exportSection(stage: Stage): void { - if (!stage.shown.length) return; - const section = stage.section('Export', false); - const status = section.note(); - status.setAttribute('role', 'status'); - let recording: AbortController | undefined; - /** The views, and a shot of them laid out as the page shows them, `align` pixels a block. */ - const shot = (align: [number, number]): [Shot, View[]] => { - const shown = stage.shown - .map(({ view, element, panel: [x, y, w, h] }) => { - const canvas = element.getBoundingClientRect(); - const [left, top] = [canvas.left + x * canvas.width, canvas.top + y * canvas.height]; - const [width, height] = [w * canvas.width, h * canvas.height]; - return { - view, - box: { left, top, width, height, right: left + width, bottom: top + height }, - }; - }) - .filter(({ box }) => box.width > 0 && box.height > 0); - const left = Math.min(...shown.map((s) => s.box.left)); - const top = Math.min(...shown.map((s) => s.box.top)); - const width = Math.max(...shown.map((s) => s.box.right)) - left; - const height = Math.max(...shown.map((s) => s.box.bottom)) - top; - const scale = Math.min(2, devicePixelRatio); - const size: [number, number] = [ - Math.max(align[0], Math.floor((width * scale) / align[0]) * align[0]), - Math.max(align[1], Math.floor((height * scale) / align[1]) * align[1]), - ]; - const panels = shown.map( - ({ box }) => - [ - (box.left - left) / width, - (box.top - top) / height, - box.width / width, - box.height / height, - ] as const, - ); - return [new Shot(size).scale(size[0] / width).panels(panels), shown.map((s) => s.view)]; - }; - const picture = async () => { - status.textContent = 'Taking a picture…'; - const [taken, views] = shot([1, 1]); - // A picture waits for its views to be still; the page holds its own motion meanwhile. - const image = await stage.hold(() => taken.image(views)); - try { - const file = await image.encode('png'); - save('views.png', new Blob([file.slice()], { type: 'image/png' })); - status.textContent = `Saved views.png, ${image.width}×${image.height}, ${kilobytes(file.length)}`; - } finally { - image.destroy(); - } - }; - const video = async () => { - if (recording) return recording.abort(new Error('stopped')); - const chunks: Uint8Array[] = []; - const output = new WritableStream({ - write: (chunk) => void chunks.push(chunk.slice()), - }); - recording = new AbortController(); - const [taken, views] = shot([4, 2]); - const started = performance.now(); - try { - const recorder = taken.video(stage.gpu, FPS, output, { signal: recording.signal }); - // Each frame waits for the page to reach its time, so the video runs as the page does. - await recorder.record( - views, - async (time) => { - while (performance.now() - started < time * 1000) await nextFrame(); - return time < SECONDS; - }, - ({ time }) => - (status.textContent = `Recording ${time.toFixed(1)} of ${SECONDS} s; V stops`), - ); - const file = new Blob(chunks, { type: 'video/mp4' }); - save('views.mp4', file); - status.textContent = `Saved views.mp4, ${SECONDS} s, ${kilobytes(file.size)}`; - } catch (error) { - status.textContent = recording.signal.aborted - ? 'Stopped' - : `Could not record: ${String(error)}`; - } finally { - recording = undefined; - } - }; - const failing = (run: () => Promise) => () => - void run().catch( - (error: unknown) => (status.textContent = `Could not export: ${String(error)}`), - ); - section.actions([ - { label: 'Save PNG', key: 'x', run: failing(picture) }, - { label: 'Record', key: 'v', run: failing(video) }, - ]); -} - -/** Saves `file` as a download named `name`. */ -function save(name: string, file: Blob): void { - const url = URL.createObjectURL(file); - h('a', { href: url, download: name }).click(); - setTimeout(() => URL.revokeObjectURL(url), 10_000); -} - -function kilobytes(bytes: number): string { - return `${Math.max(1, Math.round(bytes / 1024)).toLocaleString()} kB`; -} - -function nextFrame(): Promise { - return new Promise((resolve) => requestAnimationFrame(() => resolve())); -} diff --git a/npm/examples/src/ui/legend.ts b/npm/examples/src/ui/legend.ts deleted file mode 100644 index 0b0fcf5e..00000000 --- a/npm/examples/src/ui/legend.ts +++ /dev/null @@ -1,25 +0,0 @@ -import { colormapCss, type Colormap } from '../latkit.js'; -import { h } from './dom.js'; -import type { Stage } from './stage.js'; - -/** A colormap ramp with its label, over the main view: at its foot, or its top over a plot's axes. */ -export class Legend { - readonly #ramp = h('span', { class: 'ramp', 'aria-hidden': 'true' }); - readonly #label = h('span'); - readonly #element = h('p', { class: 'legend' }, this.#ramp, this.#label); - - constructor(stage: Stage, place: 'foot' | 'top' = 'foot') { - this.#element.classList.toggle('top', place === 'top'); - stage.regions.main.append(this.#element); - } - - ramp(colormap: Colormap, label: string): void { - this.#ramp.style.background = colormapCss(colormap); - this.#label.textContent = label; - } - - /** Hides the legend, as while nothing it describes shows. */ - set hidden(hidden: boolean) { - this.#element.hidden = hidden; - } -} diff --git a/npm/examples/src/ui/menu.ts b/npm/examples/src/ui/menu.ts deleted file mode 100644 index 323e5285..00000000 --- a/npm/examples/src/ui/menu.ts +++ /dev/null @@ -1,60 +0,0 @@ -import type { View } from 'latkit'; -import { h } from './dom.js'; -import type { Stage } from './stage.js'; - -/** An item of a menu: what it says, and what it does; a disabled one shows but does nothing. */ -export interface Item { - label: string; - run: () => void; - disabled?: boolean; -} - -/** A menu over the main view where a view asked for one; gone once an item runs, on Escape, or on a - * press elsewhere. */ -export class Menu { - readonly #element = h('div', { class: 'menu', role: 'menu', hidden: true }); - - constructor(private readonly stage: Stage) { - const { signal } = stage; - stage.regions.main.append(this.#element); - addEventListener( - 'pointerdown', - (event) => { - if (!this.#element.contains(event.target as Node)) this.close(); - }, - { signal, capture: true }, - ); - addEventListener('keydown', (event) => event.key === 'Escape' && this.close(), { signal }); - } - - /** Opens `items` at `point`, CSS pixels over `view`, kept inside the main view. */ - open(view: View, point: [number, number], items: readonly Item[]): void { - const shown = this.stage.shown.find((s) => s.view === view); - if (!shown) return; - const { element: canvas, panel } = shown; - this.#element.replaceChildren( - ...items.map(({ label, run, disabled }) => { - const button = h('button', { type: 'button', role: 'menuitem', disabled }, label); - button.addEventListener('click', () => { - this.close(); - run(); - }); - return button; - }), - ); - this.#element.hidden = false; - const main = this.stage.regions.main; - const [width, height] = [this.#element.offsetWidth, this.#element.offsetHeight]; - const x = canvas.offsetLeft + panel[0] * canvas.clientWidth + point[0]; - const y = canvas.offsetTop + panel[1] * canvas.clientHeight + point[1]; - const left = Math.min(x, main.clientWidth - width - 8); - const top = Math.min(y, main.clientHeight - height - 8); - this.#element.style.left = `${Math.max(8, left)}px`; - this.#element.style.top = `${Math.max(8, top)}px`; - this.#element.querySelector('button:not([disabled])')?.focus(); - } - - close(): void { - this.#element.hidden = true; - } -} diff --git a/npm/examples/src/ui/meter.ts b/npm/examples/src/ui/meter.ts deleted file mode 100644 index 3940948c..00000000 --- a/npm/examples/src/ui/meter.ts +++ /dev/null @@ -1,55 +0,0 @@ -import type { View } from 'latkit'; -import { h } from './dom.js'; -import type { Stage } from './stage.js'; - -/** - * A readout over a view: frames per second, the view's frame time, draws, instances, and tiles, and - * the page's own measures. - */ -export class Meter { - readonly #line = h('p', { class: 'meter', 'aria-live': 'off' }); - readonly #times = new Map(); - readonly #notes: (() => string)[] = []; - #frames = 0; - #since = performance.now(); - - constructor(stage: Stage, view: View) { - const section = stage.section('Performance', false); - section.element.classList.add('performance'); - section.content.append(this.#line); - stage.frame(() => { - this.#frames++; - const now = performance.now(); - if (now - this.#since < 500) return; - const stats = view.stats; - const parts = [ - `${((this.#frames * 1000) / (now - this.#since)).toFixed(0)} fps`, - `frame ${stats.cpuMs.toFixed(1)} ms`, - `${stats.draws} draws · ${stats.instances.toLocaleString()} instances`, - ]; - if (stats.tiles || stats.tilesLoading) { - parts.push( - `${stats.tiles} tiles${stats.tilesLoading ? ` (${stats.tilesLoading} loading)` : ''}`, - ); - } - for (const [label, ms] of this.#times) parts.push(`${label} ${ms.toFixed(1)} ms`); - this.#line.textContent = [...parts, ...this.#notes.map((note) => note())].join('\n'); - this.#frames = 0; - this.#since = now; - }); - } - - /** Runs `work`, keeping a moving average of its time under `label`. */ - measure(label: string, work: () => T): T { - const start = performance.now(); - const result = work(); - const ms = performance.now() - start; - this.#times.set(label, (this.#times.get(label) ?? ms) * 0.9 + ms * 0.1); - return result; - } - - /** A readout computed at each refresh. */ - note(read: () => string): void { - this.#notes.push(read); - } -} diff --git a/npm/examples/src/ui/playback.ts b/npm/examples/src/ui/playback.ts deleted file mode 100644 index 05b69327..00000000 --- a/npm/examples/src/ui/playback.ts +++ /dev/null @@ -1,141 +0,0 @@ -import { h } from './dom.js'; -import type { Stage } from './stage.js'; - -export interface PlaybackOptions { - /** Shows coordinate `at` in every view. */ - show(at: number): void; - /** Whether the data records as it plays: live follows its end. Otherwise playing loops. */ - live?: boolean; - /** Seconds of data a second of playing shows. */ - rate?: number; - /** Instant replay: how many seconds back from the end, and at what rate. */ - replay?: { seconds: number; rate: number }; - /** Called as playback goes live or leaves it. */ - change?(live: boolean): void; -} - -/** Live, paused, or playing a recorded span, with the transport that drives it. */ -export class Playback { - mode: 'live' | 'playing' | 'paused'; - start = 0; - end = 0; - #at = 0; - #rate: number; - readonly #live: boolean; - readonly #toggle = h('button', { type: 'button', class: 'primary' }); - readonly #slider = h('input', { - type: 'range', - step: 'any', - min: 0, - max: 1, - value: 0, - 'aria-label': 'Time', - }); - readonly #time = h('output', {}, '0.00 s'); - readonly #transport = h('div', { class: 'transport' }, this.#toggle); - - constructor( - stage: Stage, - private readonly options: PlaybackOptions, - ) { - const { signal } = stage; - this.#live = options.live ?? true; - this.#rate = options.rate ?? 1; - this.mode = this.#live ? 'live' : 'playing'; - const flip = () => (this.mode === 'paused' ? this.play() : this.pause()); - this.#toggle.addEventListener('click', flip, { signal }); - this.#slider.addEventListener( - 'input', - () => { - this.#set('paused'); - this.#show(this.#slider.valueAsNumber); - }, - { signal }, - ); - stage.key('k', flip); - if (options.replay) { - const replay = h('button', { type: 'button' }, 'Replay', h('kbd', {}, 'R')); - replay.addEventListener('click', () => this.replay(), { signal }); - stage.key('r', () => this.replay()); - this.#transport.append(replay); - } - this.#transport.append(this.#time); - stage.regions.timeline.append(this.#transport, this.#slider); - stage.frame((dt) => { - if (this.mode !== 'playing') return; - const next = this.#at + dt * this.#rate; - if (next < this.end) this.#show(next); - else if (this.#live) this.play(); - else this.#show(this.start + ((next - this.start) % Math.max(this.end - this.start, 1e-9))); - }); - document.addEventListener('visibilitychange', () => document.hidden && this.pause(), { - signal, - }); - this.#label(); - } - - get at(): number { - return this.#at; - } - - /** Hides the transport, as while there is nothing to play. */ - set hidden(hidden: boolean) { - this.#transport.hidden = hidden; - this.#slider.hidden = hidden; - } - - /** Seconds of data a second of playing shows, from now on. */ - set rate(rate: number) { - this.options.rate = rate; - if (this.mode === 'playing') this.#rate = rate; - } - - /** The recorded span changed; live playback shows its end. */ - record([start, end]: readonly [number, number]): void { - this.start = start; - this.end = end; - this.#slider.min = String(start); - this.#slider.max = String(Math.max(end, start + 1e-3)); - if (this.mode === 'live') this.#show(end); - else if (this.#at < start) this.#show(start); - } - - /** Goes live, or for recorded data plays on from where it is. */ - play(): void { - this.#rate = this.options.rate ?? 1; - this.#set(this.#live ? 'live' : 'playing'); - if (this.#live) this.#show(this.end); - } - - pause(): void { - this.#set('paused'); - } - - /** Plays the last seconds again, slowly, then goes live. */ - replay(): void { - const replay = this.options.replay; - if (!replay) return; - this.#rate = replay.rate; - this.#set('playing'); - this.#show(Math.max(this.start, this.end - replay.seconds)); - } - - #set(mode: Playback['mode']): void { - const was = this.mode === 'live'; - this.mode = mode; - this.#label(); - if (was !== (mode === 'live')) this.options.change?.(mode === 'live'); - } - - #label(): void { - this.#toggle.textContent = this.mode !== 'paused' ? 'Pause' : this.#live ? 'Live' : 'Play'; - this.#toggle.append(h('kbd', {}, 'K')); - } - - #show(at: number): void { - this.#at = at; - this.#slider.value = String(at); - this.#time.value = `${at.toFixed(2)} s`; - this.options.show(at); - } -} diff --git a/npm/examples/src/ui/rows.ts b/npm/examples/src/ui/rows.ts deleted file mode 100644 index ec4e1f89..00000000 --- a/npm/examples/src/ui/rows.ts +++ /dev/null @@ -1,9 +0,0 @@ -/** Sets of rows, as a view selects and tells them, and the rows in them. */ -import type { Row, Rows } from '../latkit.js'; - -/** Each row of `sets`, in order. */ -export const each = (sets: readonly Rows[]): Row[] => - sets.flatMap(({ data, table, ids }) => ids.map((id) => ({ data, table, id }))); - -/** `row` as a set of rows. */ -export const one = ({ data, table, id }: Row): Rows => ({ data, table, ids: [id] }); diff --git a/npm/examples/src/ui/shell.ts b/npm/examples/src/ui/shell.ts deleted file mode 100644 index b6a59a4f..00000000 --- a/npm/examples/src/ui/shell.ts +++ /dev/null @@ -1,132 +0,0 @@ -import type { Gpu } from 'latkit'; -import { h } from './dom.js'; -import { exportSection } from './export.js'; -import { Stage } from './stage.js'; - -declare global { - interface Window { - /** Active example, exposed for the browser gate. */ - stage?: Stage; - } -} - -export interface Page { - readonly id: string; - readonly title: string; - load(): Promise<{ default: (stage: Stage) => Promise | void }>; -} - -/** Navigation, controls, and a full-window canvas shared by every example. */ -export class Shell { - readonly #picker = h('select', { 'aria-label': 'Example' }); - readonly #detail = h('p', { class: 'detail' }); - readonly #error = h('div', { role: 'alert', class: 'error', hidden: true }); - readonly #loading = h('p', { class: 'loading', role: 'status', hidden: true }, 'Loading…'); - readonly #regions = { - main: h('main', { id: 'example', 'aria-label': 'Example' }), - side: h('aside', { id: 'controls', 'aria-label': 'Controls' }), - timeline: h('footer', { 'aria-label': 'Playback' }), - }; - #stage?: Stage; - #showing = 0; - - constructor( - private readonly pages: readonly Page[], - private readonly gpu: Promise, - ) { - this.#picker.append(...pages.map(({ id, title }) => h('option', { value: id }, title))); - this.#picker.addEventListener('change', () => (location.hash = this.#picker.value)); - const toggle = h( - 'button', - { - type: 'button', - class: 'controls-toggle', - 'aria-expanded': 'true', - 'aria-controls': 'controls', - }, - 'Controls', - ); - toggle.addEventListener('click', () => { - const hidden = !this.#regions.side.hidden; - this.#regions.side.hidden = hidden; - toggle.setAttribute('aria-expanded', String(!hidden)); - }); - document.body.append( - h('a', { class: 'skip-link', href: '#example' }, 'Skip to example'), - h( - 'header', - {}, - h('a', { class: 'brand', href: `#${pages[0]!.id}` }, 'latkit'), - h('label', { class: 'example-picker' }, h('span', {}, 'Examples'), this.#picker), - this.#detail, - toggle, - ), - this.#error, - this.#loading, - this.#regions.main, - this.#regions.side, - this.#regions.timeline, - ); - document.querySelector('.skip-link')!.addEventListener('click', (event) => { - event.preventDefault(); - this.#regions.main.querySelector('canvas')?.focus(); - }); - addEventListener('hashchange', () => void this.#show()); - void this.#show(); - } - - async #show(): Promise { - const showing = ++this.#showing; - const page = this.pages.find((p) => `#${p.id}` === location.hash) ?? this.pages[0]!; - this.#stage?.close(); - this.#stage = window.stage = undefined; - this.#error.hidden = true; - this.#loading.hidden = false; - this.#regions.main.setAttribute('aria-busy', 'true'); - this.#picker.value = page.id; - document.title = `${page.title} · latkit`; - this.#regions.main.setAttribute('aria-label', page.title); - try { - const [gpu, module] = await Promise.all([this.gpu, page.load()]); - if (showing !== this.#showing) return; - const stage = new Stage(gpu, this.#regions, this.#detail, (error) => { - if (showing === this.#showing) this.#fail(error); - }); - this.#stage = window.stage = stage; - await module.default(stage); - if (showing === this.#showing && !stage.signal.aborted) { - for (const section of this.#regions.side.querySelectorAll('.performance')) { - this.#regions.side.append(section); - } - exportSection(stage); - } - } catch (error) { - if (showing === this.#showing) this.#fail(error); - } finally { - if (showing === this.#showing) { - this.#loading.hidden = true; - this.#regions.main.removeAttribute('aria-busy'); - } - } - } - - #fail(error: unknown): void { - this.#stage?.close(); - this.#stage = window.stage = undefined; - this.#error.hidden = false; - const unsupported = (error as { code?: string }).code === 'unsupported'; - this.#error.replaceChildren( - h( - 'p', - {}, - unsupported - ? 'WebGPU is unavailable. Open these examples in a browser with WebGPU enabled.' - : `Could not load this example. ${String(error)}`, - ), - ); - const retry = h('button', { type: 'button' }, 'Reload'); - retry.addEventListener('click', () => location.reload()); - this.#error.append(retry); - console.error(error); - } -} diff --git a/npm/examples/src/ui/stage.ts b/npm/examples/src/ui/stage.ts deleted file mode 100644 index 21a72737..00000000 --- a/npm/examples/src/ui/stage.ts +++ /dev/null @@ -1,195 +0,0 @@ -import type { Canvas, Gpu, Panel, Scene, View } from 'latkit'; -import { Section } from './controls.js'; -import { h } from './dom.js'; -import { Legend } from './legend.js'; -import { Menu } from './menu.js'; -import { Meter } from './meter.js'; -import { Playback, type PlaybackOptions } from './playback.js'; - -/** Where a page draws: the main view, the side panel, and the timeline. */ -export interface Regions { - main: HTMLElement; - side: HTMLElement; - timeline: HTMLElement; -} - -/** What a page draws with. Everything made through it goes when the page closes. */ -export class Stage { - readonly #abort = new AbortController(); - /** Aborted when the page closes: pass it to listeners the page adds itself. */ - readonly signal = this.#abort.signal; - readonly #views = new Set(); - readonly #canvases = new Set(); - /** Each view a canvas of the page shows, its element, and the panel of it the view fills, in the - * order they were made. */ - readonly shown: { view: View; element: HTMLCanvasElement; panel: Panel }[] = []; - readonly #keys = new Map void>(); - #holding = 0; - - constructor( - readonly gpu: Gpu, - readonly regions: Regions, - private readonly line: HTMLElement, - readonly fail: (error: unknown) => void, - ) { - addEventListener( - 'keydown', - (event) => { - const action = this.#keys.get(event.key.toLowerCase()); - const typing = (event.target as Element | null)?.closest?.( - 'input, select, textarea, button, summary, [contenteditable]', - ); - if (!action || typing || event.ctrlKey || event.metaKey || event.altKey) return; - event.preventDefault(); - action(); - }, - { signal: this.signal }, - ); - } - - /** Every view the page made. */ - get views(): View[] { - return [...this.#views]; - } - - /** A view in a canvas of its own, in `into`, the main region unless given. */ - view(scene: Scene, options: { label: string; into?: HTMLElement; height?: number }): View { - return this.panels([scene], options)[0]!; - } - - /** - * Views sharing one canvas in `into`, the main region unless given: each in its panel, its x, y, - * width, and height as fractions of the canvas, side by side unless given. Input goes to the view - * under it. - */ - panels( - scenes: readonly Scene[], - { - label, - into = this.regions.main, - height, - panels = scenes.map((_, i) => [i / scenes.length, 0, 1 / scenes.length, 1] as const), - }: { label: string; into?: HTMLElement; height?: number; panels?: readonly Panel[] }, - ): View[] { - const canvas = h('canvas', { tabindex: 0, 'aria-label': label }); - if (height) canvas.style.height = `${height}px`; - into.append(canvas); - const views = scenes.map((scene) => { - const view = this.gpu.view(scene); - // Leave room for the full font bounds above a plot's value-axis title. - if (typeof scene.space === 'object' && 'plot' in scene.space) view.insets = [12, 0, 0, 0]; - return view; - }); - const narrow = matchMedia('(max-width: 700px)'); - const layout = () => - narrow.matches && scenes.length > 1 - ? panels.map(([x, y, width, height]) => [y, x, height, width] as const) - : panels; - let current = layout(); - let surface = this.gpu.canvas(canvas, { panels: current }).show(views); - this.#canvases.add(surface); - const shown = views.map((view, i) => ({ view, element: canvas, panel: current[i]! })); - this.shown.push(...shown); - if (scenes.length > 1) - narrow.addEventListener( - 'change', - () => { - surface.destroy(); - this.#canvases.delete(surface); - current = layout(); - shown.forEach((item, i) => (item.panel = current[i]!)); - surface = this.gpu.canvas(canvas, { panels: current }).show(views); - this.#canvases.add(surface); - }, - { signal: this.signal }, - ); - views.forEach((view) => { - // What could not be fetched only goes undrawn; anything else stops the page. - view.on('error', (error) => - error.code === 'io' ? this.detail(error.message) : this.fail(error), - ); - this.#views.add(view); - }); - return views; - } - - /** Calls `step` every frame before the views draw, with the seconds since the last, at most 0.1; - * not while the page holds still. */ - frame(step: (dt: number) => void): void { - let last: number | undefined; - const stop = this.gpu.on('frame', (now) => { - const dt = last === undefined || this.#holding ? 0 : Math.min(0.1, (now - last) / 1000); - last = now; - if (this.#holding) return; - try { - step(dt); - } catch (error) { - this.fail(error); - } - }); - this.signal.addEventListener('abort', stop); - } - - /** Holds the page's frame steps still while `work` runs, as a picture of it waits for stillness. */ - async hold(work: () => Promise): Promise { - this.#holding++; - try { - return await work(); - } finally { - this.#holding--; - } - } - - /** Runs `action` when `key` is pressed outside a text field. */ - key(key: string, action: () => void): void { - this.#keys.set(key.toLowerCase(), action); - } - - /** A titled group of controls in the side panel. */ - section(title: string, open = true): Section { - const section = new Section(this, title, open); - this.regions.side.append(section.element); - return section; - } - - /** The transport: live, paused, or playing a recorded span. */ - playback(options: PlaybackOptions): Playback { - return new Playback(this, options); - } - - /** A readout over `view`: frames per second, its frame time, and the page's own measures. */ - meter(view: View): Meter { - return new Meter(this, view); - } - - /** A menu over the main view, opened where a view asks for one. */ - menu(): Menu { - return new Menu(this); - } - - /** A colormap ramp over the main view: at its foot, or its top over a plot's axes. */ - legend(place: 'foot' | 'top' = 'foot'): Legend { - return new Legend(this, place); - } - - /** A line under the main view crediting where its data comes from. */ - credit(): HTMLElement { - const line = h('p', { class: 'credit' }); - this.regions.main.append(line); - return line; - } - - /** Compact dataset information in the shared header. */ - detail(text: string): void { - this.line.textContent = text; - } - - close(): void { - this.#abort.abort(); - for (const canvas of this.#canvases) canvas.destroy(); - for (const view of this.#views) view.destroy(); - const { main, side, timeline } = this.regions; - for (const region of [main, side, timeline]) region.replaceChildren(); - this.line.textContent = ''; - } -} diff --git a/npm/examples/src/ui/theme.ts b/npm/examples/src/ui/theme.ts deleted file mode 100644 index 536b5fc0..00000000 --- a/npm/examples/src/ui/theme.ts +++ /dev/null @@ -1,8 +0,0 @@ -import type { Scene } from 'latkit'; - -/** Default canvas colors; examples only override what they demonstrate. */ -export const theme = { - base: 'dark', - background: '#101214', - grid: '#ffffff1c', -} as const satisfies Scene['theme']; diff --git a/npm/examples/test/setup.ts b/npm/examples/test/setup.ts index 9695a8f2..b607c0c1 100644 --- a/npm/examples/test/setup.ts +++ b/npm/examples/test/setup.ts @@ -1,8 +1,10 @@ /** - * The page models' tests run in node, where fetch reads no files: the runtime's WebAssembly, which - * it fetches from beside its module, is read from disk instead. + * Setup for the examples' tests, which run in node. Node's fetch cannot read files, so `file:` + * URLs, such as latkit's WebAssembly beside its module, are read from disk instead. latkit is + * loaded before any test makes data. */ import { readFile } from 'node:fs/promises'; +import { load } from 'latkit'; const fetch = globalThis.fetch; globalThis.fetch = async (input, init) => { @@ -11,3 +13,4 @@ globalThis.fetch = async (input, init) => { const type = url.endsWith('.wasm') ? 'application/wasm' : 'application/octet-stream'; return new Response(await readFile(new URL(url)), { headers: { 'Content-Type': type } }); }; +await load(); diff --git a/npm/examples/tsconfig.json b/npm/examples/tsconfig.json index 6a0250c8..0d60eb49 100644 --- a/npm/examples/tsconfig.json +++ b/npm/examples/tsconfig.json @@ -5,9 +5,11 @@ "moduleResolution": "bundler", "strict": true, "noUncheckedIndexedAccess": true, + "allowImportingTsExtensions": true, + "noEmit": true, "skipLibCheck": true, "types": ["vite/client", "node"], "lib": ["ES2022", "DOM", "DOM.Iterable", "ESNext.Disposable"] }, - "include": ["src", "test", "vite.config.ts"] + "include": ["src", "scripts", "test", "vite.config.ts"] } diff --git a/npm/examples/vite.config.ts b/npm/examples/vite.config.ts index 8b885965..2d3891bf 100644 --- a/npm/examples/vite.config.ts +++ b/npm/examples/vite.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from 'vite'; export default defineConfig({ server: { host: '127.0.0.1', port: 5200, strictPort: true }, preview: { host: '127.0.0.1', port: 5200 }, - // The package loads its wasm next to itself; prebundling would move it. + // latkit loads its wasm from beside its own module, which prebundling would move. optimizeDeps: { exclude: ['latkit'] }, build: { target: 'es2022' }, }); diff --git a/npm/package.json b/npm/package.json index 8a42cbfb..c2c5559c 100644 --- a/npm/package.json +++ b/npm/package.json @@ -15,11 +15,7 @@ "default": "./dist/index.js" }, "./wasm": "./pkg/latkit_bg.wasm", - "./worker": "./dist/worker.js", - "./core": { - "types": "./dist/core.d.ts", - "default": "./dist/core.js" - } + "./worker": "./dist/worker.js" }, "files": [ "dist", diff --git a/npm/release.mjs b/npm/release.mjs index b7adb9f4..af1adecc 100644 --- a/npm/release.mjs +++ b/npm/release.mjs @@ -1,4 +1,5 @@ -/** One retryable release: verify or publish each artifact, then create the Changesets tags. */ +/** Runs a release that can be retried. It verifies or publishes each artifact, then creates the + * Changesets tags. */ import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; import { createHash } from 'node:crypto'; @@ -14,7 +15,8 @@ const run = (command, args, options = {}) => const hash = (file, algorithm, encoding = 'hex') => createHash(algorithm).update(fs.readFileSync(file)).digest(encoding); -/** Only an explicit 404 means absent. Authentication, rate limits, and network failures stop release. */ +/** Fetches registry metadata from `url`. Returns undefined only on a 404. Any other failure, such + * as authentication, rate limits, or the network, throws and stops the release. */ export async function registry(url) { const response = await fetch(url, { headers: { 'User-Agent': 'latkit-release (github.com/lukelowry/latkit)' }, @@ -24,7 +26,8 @@ export async function registry(url) { return response.json(); } -/** Successful registry responses must contain a digest, never count as absent. */ +/** Returns the digest that `select` reads from the registry metadata at `url`, or undefined if the + * version is absent. A successful response without a digest throws rather than count as absent. */ async function digest(url, select) { const metadata = await registry(url); if (metadata === undefined) return undefined; @@ -33,7 +36,11 @@ async function digest(url, select) { return value; } -/** Existing versions must contain the expected artifact; retries resume after the last completed publish. */ +/** + * Publishes each artifact its registry does not have yet, waiting up to 2 minutes for it to appear, + * then checks that the registry's digest matches the verified build. A retry resumes after the last + * completed publish. + */ export async function ensureArtifacts(artifacts) { for (const artifact of artifacts) { let seen = await artifact.lookup(); @@ -58,7 +65,8 @@ async function release() { const report = read('target/npm/package.json'); const { name, version } = read('npm/package.json'); const git = (args) => run('git', args, { stdio: 'pipe', encoding: 'utf8' }).trim(); - // Tags follow a complete release, so later commits at the same version have nothing to publish. + // Tags are created only after a complete release, so later commits at the same version publish + // nothing. if (git(['tag', '--list', `${name}@${version}`]) !== '') { console.log(`${name}@${version}: released`); return; diff --git a/npm/schema.json b/npm/schema.json index 3bd58006..ae4730b9 100644 --- a/npm/schema.json +++ b/npm/schema.json @@ -6,49 +6,52 @@ "additionalProperties": false, "properties": { "space": { - "$ref": "#/$defs/Space" + "$ref": "#/$defs/Space", + "description": "The coordinate space positions are in." }, "marks": { "type": "array", "items": { "$ref": "#/$defs/Mark" }, - "description": "What the view draws, in drawing order." + "description": "The marks, in drawing order: each draws over the marks before it." }, "layout": { "$ref": "#/$defs/Layout", - "description": "How the view places vertex rows without positions: a diagram's blocks, or points on a plane\nor the earth, gaps in the space's units." + "description": "How the view places vertex rows that have no position, such as a diagram's blocks or points\non a plane or the earth. Defaults to `none`." }, "theme": { - "$ref": "#/$defs/Theme" + "$ref": "#/$defs/Theme", + "description": "Default colors and sizes, and the look of text, focus, and each space. Defaults to `dark`." }, "controls": { - "$ref": "#/$defs/Controls" + "$ref": "#/$defs/Controls", + "description": "How the view responds to the pointer and keyboard." }, "ease": { "type": "number", "format": "double", - "description": "Seconds that changes ease over; 0.3 unless set." + "description": "How many seconds scene and data changes take to animate. 0 turns animation off. Defaults to\n0.3." }, "title": { "type": [ "string", "null" ], - "description": "Written at the view's top left, larger than the theme's text." + "description": "A title drawn at the view's top left, larger than the theme's text." }, "subtitle": { "type": [ "string", "null" ], - "description": "Written under the title." + "description": "A subtitle drawn under the title." } }, "required": [ "space" ], - "description": "What a view draws and how it answers input: a space, marks in drawing order, a layout, a theme,\ncontrols, how long changes ease, and the title written over it. Plain data: as JSON, each field by\nits name in camelCase, each mark `{ vertices | edges | faces | lines | areas: options }`.\n\n```\n# use latkit::{Colormap, Edges, Earth, Scene, Vertices, field};\nlet voltage = field(\"vm\").colormap(Colormap::BALANCE).domain(0.95..=1.05);\nlet scene = Scene::new(Earth::globe())\n .draw(Edges::of(\"grid\", \"Branch\").ends(\"from\", \"to\"))\n .draw(Vertices::of(\"grid\", \"Bus\").position(\"lon\", \"lat\").color(voltage).legend(\"Bus voltage · pu\"))\n .title(\"Voltage\");\n```" + "description": "A view's content and behavior: a space, marks drawn in order, a layout, a theme, controls, an\nease duration, and an optional title.\n\nA scene is plain data. In JSON, fields are camelCase, and the key naming each mark's kind holds\nits dataset and table: `{ \"vertices\": [\"grid\", \"Bus\"], \"radius\": 4 }`.\n\n```\n# use latkit::{Colormap, Edges, Earth, Projection, Scene, Vertices, field};\nlet voltage = field(\"vm\").colormap(Colormap::BALANCE).domain(0.95..=1.05);\nlet buses = Vertices::of(\"grid\", \"Bus\").position(field(\"lon\"), field(\"lat\")).color(voltage);\nlet scene = Scene::new(Earth::default().projection(Projection::Globe))\n .draw(Edges::of(\"grid\", \"Branch\").ends(\"from\", \"to\"))\n .draw(buses.legend(\"Bus voltage · pu\"))\n .title(\"Voltage\");\n```" }, "Space": { "anyOf": [ @@ -57,15 +60,14 @@ "enum": [ "plane", "diagram", - "mercator", - "globe", + "earth", "plot" ] }, { "oneOf": [ { - "description": "Longitude and latitude in degrees, on a map or a globe.", + "description": "Longitude and latitude in degrees, on a flat map or a globe.", "type": "object", "properties": { "earth": { @@ -78,7 +80,7 @@ "additionalProperties": false }, { - "description": "Plane units with elevation: graphs and scenes the camera turns and tilts around.", + "description": "Plane units, with elevation. The camera can turn and tilt.", "type": "object", "properties": { "plane": { @@ -91,7 +93,7 @@ "additionalProperties": false }, { - "description": "The data's axis across, values up.", + "description": "The data's axis horizontally and values vertically.", "type": "object", "properties": { "plot": { @@ -104,7 +106,7 @@ "additionalProperties": false }, { - "description": "A sheet for diagrams: x right and y down in units, every length a unit.", + "description": "A flat sheet for diagrams: x right and y down, every length in units.", "type": "object", "properties": { "diagram": { @@ -119,7 +121,7 @@ ] } ], - "description": "The coordinate space a scene's positions are in: a name, or `{ plane | diagram | earth | plot:\noptions }`. A diagram is a flat sheet seen from straight above: y runs down, and every length is a\nunit, a CSS pixel at zoom 0, except glow, halo, and pick radius. Only a diagram has ports, groups,\nnets, orthogonal routes, and layouts." + "description": "The coordinate space of a scene's positions: a name, or\n`{ plane | diagram | earth | plot: options }`.\n\nThe space sets the unit of lengths that do not name their own. In a diagram it is the diagram\nunit, one CSS pixel at zoom 0. Elsewhere it is the CSS pixel. Only a diagram has ports, groups,\nnets, and orthogonal routes." }, "Earth": { "type": "object", @@ -127,19 +129,11 @@ "properties": { "projection": { "$ref": "#/$defs/Projection", - "description": "A flat map or a globe." - }, - "surface": { - "description": "The ground's color; the theme's surface unless set.", - "tsType": "Color", - "type": [ - "string", - "null" - ] + "description": "A flat web mercator map (`mercator`) or a globe (`globe`). Defaults to `mercator`." }, "graticule": { "type": "boolean", - "description": "Whether lines of longitude and latitude cross the ground, in the theme's grid color." + "description": "Whether lines of longitude and latitude cross the ground, in the theme's `grid` color.\nDefaults to `true`." }, "daylight": { "type": [ @@ -147,14 +141,14 @@ "null" ], "format": "double", - "description": "Lit by the sun as it stands this many seconds after the Unix epoch, plus the view's axis\ncoordinate when it shows one: night falls on the ground and what stands on it, down to the\ntheme's night. Lit evenly unless set." + "description": "Lights the earth by the sun's position this many seconds after the Unix epoch, plus the\nview's axis coordinate when it has one. The night side darkens the ground and everything on\nit, down to the theme's `earth.night` brightness. When unset, the earth is lit evenly." }, "atmosphere": { "type": "boolean", - "description": "Whether air glows around the globe, in the theme's atmosphere color, brighter where it is day." + "description": "Whether air glows around the globe, in the theme's `earth.atmosphere` color, brighter on the\nday side. Defaults to `false`." } }, - "description": "The earth: positions are longitude and latitude in degrees." + "description": "The earth: positions are longitude and latitude in degrees. The ground uses the theme's `earth`\ncolors.\n\nA flat map shows the world once. Its ground ends at the antimeridian and at Web Mercator's\nlatitude limit. The camera zooms out until the whole world fits the view, and pans until the\nworld's edges meet the view's edges. On a globe, longitude wraps, and the camera zooms out until\nthe globe spans a third of the view's shorter side." }, "Projection": { "oneOf": [ @@ -169,7 +163,7 @@ "description": "A sphere." } ], - "description": "How the earth is drawn." + "description": "How the earth is projected." }, "Plane": { "type": "object", @@ -177,10 +171,10 @@ "properties": { "grid": { "$ref": "#/$defs/Grid", - "description": "The grid under the marks." + "description": "The grid drawn under the marks. Defaults to lines 10 units apart." } }, - "description": "A plane: positions are plane units." + "description": "A plane: positions are in plane units." }, "Grid": { "oneOf": [ @@ -211,7 +205,7 @@ "lines" ], "additionalProperties": false, - "description": "Lines `pitch` units apart, thinned by tens as the view zooms out." + "description": "Lines `pitch` units apart, thinned by factors of ten as the view zooms out." }, { "type": "object", @@ -225,7 +219,7 @@ "format": "double" }, "snap": { - "description": "Whether moved vertices snap to the dots.", + "description": "Whether drag positions snap to the dots, in a diagram. Defaults to `false`.", "type": "boolean" } }, @@ -239,7 +233,7 @@ "dots" ], "additionalProperties": false, - "description": "Dots `pitch` units apart; drags snap to them when `snap`." + "description": "Dots `pitch` units apart. With `snap`, drags in a diagram snap to them." } ], "description": "A plane's or a diagram's grid: `\"off\"`, `{ lines: { pitch } }`, or `{ dots: { pitch, snap } }`." @@ -272,11 +266,11 @@ }, "grid": { "type": "boolean", - "description": "Whether grid lines cross the plot at the ticks." + "description": "Whether grid lines cross the plot at the ticks. Defaults to `true`." }, "hidden": { "type": "boolean", - "description": "Whether the axis is hidden." + "description": "Whether the axis is hidden. Defaults to `false`." }, "ticks": { "type": [ @@ -286,11 +280,11 @@ "items": { "$ref": "#/$defs/Tick" }, - "description": "Ticks at these values, each a value or `[value, label]`, their values written unless labeled;\nround values unless set." + "description": "Tick positions, each a value or `[value, label]`. A tick without a label shows its value.\nWhen unset, ticks fall at round values." }, "format": { "$ref": "#/$defs/Format", - "description": "How tick values are written." + "description": "How tick values are formatted. Defaults to `auto`." }, "precision": { "type": [ @@ -300,7 +294,7 @@ "format": "uint8", "minimum": 0, "maximum": 15, - "description": "Decimals of fixed values, or of the significand of scientific and engineering ones; as many\nas the ticks' step needs unless set." + "description": "Decimal places of fixed values, or of the significand in scientific and engineering formats.\nAt most 15. When unset, as many as the tick step needs." } }, "description": "A plot axis." @@ -329,7 +323,7 @@ "maxItems": 2 } ], - "description": "A tick as written: a value, or `[value, label]`." + "description": "A tick in JSON: a value, or `[value, label]`." }, "Format": { "oneOf": [ @@ -354,7 +348,7 @@ "description": "A significand and a power of ten that is a multiple of three: `1.5e3`, `150e-6`." } ], - "description": "How an axis writes its tick values." + "description": "How an axis formats its tick values." }, "Diagram": { "type": "object", @@ -362,180 +356,215 @@ "properties": { "grid": { "$ref": "#/$defs/Grid", - "description": "The grid under the marks." + "description": "The grid drawn under the marks. Defaults to dots 20 units apart." } }, - "description": "A sheet for diagrams, flat and seen from straight above: the camera pans and zooms. Positions are\nunits with y down, and every length is a unit, a CSS pixel at zoom 0 (sizes, padding, ports,\nstrokes, text, and the theme's), except glow, halo, and pick radius. Zooming scales one fixed\npicture. Only a diagram has ports, groups, nets, and orthogonal routes; it lays out blocks." + "description": "A flat sheet for diagrams, seen from straight above. The camera pans and zooms but does not\nturn. Positions are in units with y down, and a unit is a CSS pixel at zoom 0. Sizes, padding,\nports, strokes, text, and the theme's lengths are in units too. Sizes given with `px`, the text\nhalo, the focus glow, and the pick radius stay in CSS pixels. Zooming scales the whole picture.\nOnly a diagram has ports, groups, nets, and orthogonal routes, and the scene's layout places its\nblocks." }, "Mark": { + "description": "One table a scene draws. The mark's kind is the key that holds its table, next to its options: `{ vertices: [data, table], radius: 4 }`.", "oneOf": [ { - "type": "object", - "properties": { - "vertices": { - "$ref": "#/$defs/Vertices" - } - }, - "required": [ - "vertices" - ], - "additionalProperties": false, - "description": "Rows drawn as markers." + "$ref": "#/$defs/Vertices" }, { - "type": "object", - "properties": { - "edges": { - "$ref": "#/$defs/Edges" - } - }, - "required": [ - "edges" - ], - "additionalProperties": false, - "description": "Rows drawn as strokes between two vertices." + "$ref": "#/$defs/Edges" }, { - "type": "object", - "properties": { - "faces": { - "$ref": "#/$defs/Faces" - } - }, - "required": [ - "faces" - ], - "additionalProperties": false, - "description": "Rows drawn as triangles between three vertices." + "$ref": "#/$defs/Faces" }, { - "type": "object", - "properties": { - "lines": { - "$ref": "#/$defs/Lines" - } - }, - "required": [ - "lines" - ], - "additionalProperties": false, - "description": "Rows drawn as lines along their paths, or on a plot through their samples." + "$ref": "#/$defs/Lines" + }, + { + "$ref": "#/$defs/Areas" + } + ] + }, + "Vertices": { + "allOf": [ + { + "$ref": "#/$defs/MarkOptions" }, { "type": "object", + "description": "The mark's JSON form: the key naming its kind, its own options, then the stroke options\nand the options every mark has.", "properties": { - "areas": { - "$ref": "#/$defs/Areas" + "vertices": { + "type": "array", + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "The dataset and table drawn, `[data, table]`.", + "tsType": "TableInput" + }, + "position": { + "type": [ + "array", + "null" + ], + "items": { + "$ref": "#/$defs/Expr" + }, + "minItems": 2, + "maxItems": 2, + "description": "Each row's x and y. Rows without a position are placed by the point field `geometry` names.\nThe scene's layout places rows that have neither." + }, + "geometry": { + "type": [ + "string", + "null" + ], + "description": "The point field that places rows when `position` is not set. Defaults to the field named\n`geometry`." + }, + "elevation": { + "$ref": "#/$defs/Expr", + "description": "Height above the ground. On the earth, a fraction of the earth's radius. Defaults to 0. Not\nallowed in a diagram, which is flat." + }, + "radius": { + "anyOf": [ + { + "$ref": "#/$defs/Size" + }, + { + "type": "null" + } + ], + "description": "Each row's radius in CSS pixels, or units in a diagram, unless given with `px` or `units`.\nA diagram does not accept `px`. Defaults to the theme's `vertex.radius`." + }, + "size": { + "type": [ + "array", + "null" + ], + "items": { + "$ref": "#/$defs/Size" + }, + "minItems": 2, + "maxItems": 2, + "description": "Each row's width and height, for markers that are not round. In CSS pixels, or units in a\ndiagram, unless given with `px` or `units`. A diagram does not accept `px`. In a diagram, a\nblock without a size fits its label and ports." + }, + "marker": { + "$ref": "#/$defs/Marker", + "description": "How each row is drawn. Defaults to an ellipse." + }, + "ring": { + "anyOf": [ + { + "$ref": "#/$defs/Expr" + }, + { + "type": "null" + } + ], + "description": "A ring around each row's marker in this color. A missing or transparent color draws no ring." + }, + "ports": { + "type": "array", + "items": { + "type": "array", + "prefixItems": [ + { + "type": "string" + }, + { + "$ref": "#/$defs/Port" + } + ], + "minItems": 2, + "maxItems": 2 + }, + "description": "Ports on each row, one for each reference field that names a net. An `edges` mark of the net\ntable, without `ends`, draws the nets. Diagrams only.", + "tsType": "[reference: string, port: Port][]" + }, + "contains": { + "type": [ + "array", + "null" + ], + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "Makes each row a group containing the rows of `table` whose `reference` field names it. A\ngroup draws as a container around its members, with its label as a title in its header.\nDiagrams only.", + "tsType": "[table: string, reference: string]" + }, + "shadow": { + "type": "boolean", + "description": "Whether each block or group casts a shadow on the sheet, styled by the theme's\n`diagram.shadow`. Defaults to `false`. Diagrams only." } }, "required": [ - "areas" - ], - "additionalProperties": false, - "description": "Rows drawn as the areas their polygons cover." + "vertices" + ] } ], - "description": "One drawn table of a scene: `{ vertices | edges | faces | lines | areas: options }` as JSON." + "unevaluatedProperties": false, + "description": "Rows drawn as markers: points of a map, nodes of a graph, blocks of a diagram." }, - "Vertices": { - "type": "object", - "additionalProperties": false, - "properties": { - "of": { - "type": "array", - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The dataset and table drawn.", - "tsType": "[data: string, table: string]" + "Expr": { + "$ref": "#/$defs/ExprNode", + "tsType": "ExprInput" + }, + "ExprNode": { + "description": "A value for each row. A number or boolean is a constant and `null` is missing. A string starting with `$` reads that field, and any other string is a CSS color. An object computes a value.", + "anyOf": [ + { + "type": "number" }, - "position": { - "type": [ - "array", - "null" - ], - "items": { - "$ref": "#/$defs/Expr" - }, - "minItems": 2, - "maxItems": 2, - "description": "Each row's x and y. Without them, the point field `geometry` names places rows; rows without\neither the scene's layout places." + { + "type": "boolean" }, - "geometry": { - "type": [ - "string", - "null" - ], - "description": "The point field placing each row where `position` is not given; `geometry` unless set." + { + "description": "A missing value. An option that is missing uses the theme's value.", + "type": "null" }, - "elevation": { - "$ref": "#/$defs/Expr", - "description": "Height above the ground: on the earth, a fraction of its radius. Not in a diagram, which is\nflat." + { + "description": "A field or a color. `\"$kv\"` reads field `kv`. Any other string is a CSS color, such as `#ffb000` or `rebeccapurple`. Write a field named `$x` as `{ \"field\": \"$x\" }`.", + "type": "string" }, - "radius": { - "anyOf": [ - { - "$ref": "#/$defs/Size" - }, - { - "type": "null" - } - ], - "description": "CSS pixels, or position units with `units`; units in a diagram." + { + "$ref": "#/$defs/FieldNode" }, - "size": { - "type": [ - "array", - "null" - ], - "items": { - "$ref": "#/$defs/Size" - }, - "minItems": 2, - "maxItems": 2, - "description": "Width and height, for markers that are not round: CSS pixels, or position units with `units`;\nunits in a diagram, where a block without a size fits its label and ports." + { + "$ref": "#/$defs/StateNode" }, - "marker": { - "$ref": "#/$defs/Marker", - "description": "How each row is drawn." + { + "$ref": "#/$defs/WhenNode" }, - "ring": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "A ring around each row's marker in this color; a missing or transparent color draws none." + { + "$ref": "#/$defs/ScaleNode" }, - "ports": { - "type": "array", - "items": { - "type": "array", - "prefixItems": [ - { - "type": "string" - }, - { - "$ref": "#/$defs/Port" - } - ], - "minItems": 2, - "maxItems": 2 - }, - "description": "A port on each row for each reference field naming a net, which an `edges` mark of the net\ntable without ends draws. Diagrams only.", - "tsType": "[reference: string, port: Port][]" + { + "$ref": "#/$defs/ColormapNode" + }, + { + "$ref": "#/$defs/CallNode" + } + ] + }, + "FieldNode": { + "type": "object", + "additionalProperties": false, + "properties": { + "field": { + "type": "string", + "description": "The field's name." }, - "contains": { + "from": { "type": [ "array", "null" @@ -550,198 +579,32 @@ ], "minItems": 2, "maxItems": 2, - "description": "Makes each row a group: it holds the rows of `table` whose reference field names it, and is\ndrawn as a container around them, its label a title in its header. Diagrams only.", - "tsType": "[table: string, reference: string]" - }, - "color": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's color; the theme's unless set." - }, - "visible": { - "$ref": "#/$defs/Expr", - "description": "Whether each row is drawn and picked." - }, - "priority": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's priority, from its data alone: what matters more draws on top, stays drawn as\na table too large to draw whole thins to the tiles in view, and is labeled first." - }, - "name": { - "type": [ - "string", - "null" - ], - "description": "Tells two marks of one table apart across scenes, and names the mark in hits." - }, - "pick": { - "type": [ - "boolean", - "null" - ], - "description": "Whether rows are hovered, clicked, and probed. Unless set, rows that are things are\n(vertices, edges, faces, a plot's traces) and geometry is not (areas, and lines along\npaths: coasts, borders, roads)." - }, - "select": { - "type": "boolean", - "description": "Whether a click selects a row." - }, - "drag": { - "type": "boolean", - "description": "Whether pressing a row drags it instead of panning." - }, - "label": { - "anyOf": [ - { - "$ref": "#/$defs/Label" - }, - { - "type": "null" - } - ], - "description": "Each row's text: a text field, or a label's options." - }, - "shadow": { - "type": "boolean", - "description": "Whether each row casts a shadow onto the ground and onto faces, away from the theme's\nlight." - }, - "legend": { - "type": [ - "string", - "null" - ], - "description": "A legend of the colors at the view's bottom right, under this title: the colormap across\nits domain, ticked; a cell for each category; or a swatch beside each name of a map's\n`categories`. The color maps through a colormap with a domain or named categories; a\nconditional's `otherwise` is read before its cases." - } - }, - "required": [ - "of" - ], - "description": "Rows drawn as markers: points of a map, nodes of a graph, blocks of a diagram." - }, - "Expr": { - "$ref": "#/$defs/ExprNode", - "tsType": "ExprInput" - }, - "ExprNode": { - "description": "A value for every row of a mark: a constant, a field, view state, or arithmetic of them.", - "anyOf": [ - { - "type": "number" - }, - { - "type": "boolean" - }, - { - "description": "A field's name, or a hex color when it starts with `#`.", - "type": "string" - }, - { - "$ref": "#/$defs/FieldNode" - }, - { - "$ref": "#/$defs/ColorNode" - }, - { - "$ref": "#/$defs/StateNode" - }, - { - "$ref": "#/$defs/MissingNode" - }, - { - "$ref": "#/$defs/UnaryNode" - }, - { - "$ref": "#/$defs/BinaryNode" - }, - { - "$ref": "#/$defs/WhenNode" - }, - { - "$ref": "#/$defs/IsNode" - }, - { - "$ref": "#/$defs/ContainsNode" - }, - { - "$ref": "#/$defs/ScaleNode" - }, - { - "$ref": "#/$defs/ColormapNode" - } - ] - }, - "FieldNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "field": { - "type": "string" - }, - "of": { - "type": [ - "array", - "null" - ], - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The dataset and table the field is read from, joined by row id.", - "tsType": "[data: string, table: string]" + "description": "The dataset and table the field is read from, joined by row id.", + "tsType": "TableInput" }, "step": { "type": "boolean", - "description": "Reads a sampled field at its last frame without interpolating." + "description": "For a sampled field, reads the last frame at or before the shown coordinate instead of\ninterpolating. Defaults to `false`." } }, "required": [ "field" ], - "description": "A field of each row: of the mark's own table, or of another dataset's table joined by row id." - }, - "ColorNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "color": { - "description": "A CSS color, such as `#ffb000` or `rebeccapurple`.", - "tsType": "Color", - "type": "string" - } - }, - "required": [ - "color" - ] + "description": "A field's value in each row, from the mark's own table. With `from`, it reads another\ndataset's table, joined by row id. `\"$name\"` is the short form of `{ \"field\": \"name\" }`." }, "StateNode": { "type": "object", "additionalProperties": false, "properties": { "state": { - "$ref": "#/$defs/State" + "$ref": "#/$defs/State", + "description": "The state read." } }, "required": [ "state" ], - "description": "What the view knows of each row." + "description": "View state of each row, such as whether it is selected." }, "State": { "type": "string", @@ -752,100 +615,7 @@ "elapsed", "zoom" ], - "description": "What the view knows of each row." - }, - "MissingNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "missing": { - "type": "boolean", - "const": true - } - }, - "required": [ - "missing" - ], - "description": "A missing value: options that read one take the theme's." - }, - "UnaryNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "op": { - "$ref": "#/$defs/Unary" - }, - "args": { - "type": "array", - "items": { - "$ref": "#/$defs/Expr" - }, - "minItems": 1, - "maxItems": 1 - } - }, - "required": [ - "op", - "args" - ] - }, - "Unary": { - "type": "string", - "enum": [ - "neg", - "not", - "abs", - "sqrt", - "sin", - "cos", - "floor", - "ln", - "exp" - ], - "description": "An operation of one value." - }, - "BinaryNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "op": { - "$ref": "#/$defs/Binary" - }, - "args": { - "type": "array", - "items": { - "$ref": "#/$defs/Expr" - }, - "minItems": 2, - "maxItems": 2 - } - }, - "required": [ - "op", - "args" - ] - }, - "Binary": { - "type": "string", - "enum": [ - "add", - "sub", - "mul", - "div", - "rem", - "pow", - "min", - "max", - "lt", - "le", - "gt", - "ge", - "eq", - "ne", - "and", - "or" - ], - "description": "An operation of two values." + "description": "View state of each row: `selected`, `hovered`, `pointer` (CSS pixels from a vertex to the\npointer), `elapsed` (seconds since the view was created), or `zoom` (the camera's zoom)." }, "WhenNode": { "type": "object", @@ -866,10 +636,12 @@ "minItems": 2, "maxItems": 2 }, + "description": "The cases, as `[condition, value]` pairs tried in order.", "tsType": "[condition: ExprInput, value: ExprInput][]" }, "otherwise": { - "$ref": "#/$defs/Expr" + "$ref": "#/$defs/Expr", + "description": "The value where no condition holds." } }, "required": [ @@ -878,54 +650,6 @@ ], "description": "The value of the first case whose condition holds, else `otherwise`." }, - "IsNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "is": { - "type": "array", - "prefixItems": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "tsType": "[field: ExprInput, text: string]" - } - }, - "required": [ - "is" - ], - "description": "Whether a text field equals the text." - }, - "ContainsNode": { - "type": "object", - "additionalProperties": false, - "properties": { - "contains": { - "type": "array", - "prefixItems": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "tsType": "[field: ExprInput, text: string]" - } - }, - "required": [ - "contains" - ], - "description": "Whether a text field holds the text, ignoring case: read by queries, which hold texts." - }, "ScaleNode": { "type": "object", "additionalProperties": false, @@ -938,10 +662,12 @@ }, "minItems": 2, "maxItems": 2, + "description": "The range.", "tsType": "[low: number, high: number]" }, "value": { - "$ref": "#/$defs/Expr" + "$ref": "#/$defs/Expr", + "description": "The value mapped." }, "domain": { "type": [ @@ -954,6 +680,7 @@ }, "minItems": 2, "maxItems": 2, + "description": "The values mapped to the range's ends. Defaults to the extent of the values shown.", "tsType": "[low: number, high: number]" } }, @@ -961,17 +688,19 @@ "scale", "value" ], - "description": "`value` mapped linearly onto the range `scale`, clamped: from its `domain`, else from the extent of\nthe values shown." + "description": "A value mapped linearly onto a range, clamped to the range." }, "ColormapNode": { "type": "object", "additionalProperties": false, "properties": { "colormap": { - "$ref": "#/$defs/Colormap" + "$ref": "#/$defs/Colormap", + "description": "The colormap." }, "value": { - "$ref": "#/$defs/Expr" + "$ref": "#/$defs/Expr", + "description": "The value mapped. A categorical map takes each value's integer part as its category." }, "domain": { "type": [ @@ -984,6 +713,7 @@ }, "minItems": 2, "maxItems": 2, + "description": "The values mapped to the colormap's ends. Defaults to the extent of the values shown. A\ncategorical map ignores it.", "tsType": "[low: number, high: number]" } }, @@ -991,7 +721,7 @@ "colormap", "value" ], - "description": "`value` through `colormap`: from its `domain`, else from the extent of the values shown." + "description": "A value mapped through a colormap." }, "Colormap": { "description": "Colors for a range of values: a catalog name, or your own colors.", @@ -1005,7 +735,7 @@ ] }, "ColormapName": { - "description": "A colormap of the catalog.", + "description": "A colormap from the catalog.", "type": "string", "enum": [ "amber", @@ -1061,7 +791,8 @@ "additionalProperties": false, "properties": { "kind": { - "$ref": "#/$defs/ColormapKind" + "$ref": "#/$defs/ColormapKind", + "description": "How the colors map values." }, "colors": { "type": "array", @@ -1091,21 +822,21 @@ } ] }, - "description": "Colors blended in OKLab, each at its `[place, color]` from 0 to 1 or evenly between its\nneighbors, as in a CSS gradient; a categorical map's, one per category." + "description": "The colors, blended in OKLab. Each is a color, spaced evenly between its neighbors like a\nCSS gradient, or a `[place, color]` pair with a place from 0 to 1. Places must ascend. A\ncategorical map takes one color per category, without places, at most 256." }, "categories": { "type": "array", "items": { "type": "string" }, - "description": "The categories' names, in order, for a legend: a categorical map's alone." + "description": "The category names in category order, shown in legends. Categorical maps only." } }, "required": [ "kind", "colors" ], - "description": "Your own colormap: its kind, its colors, and for a categorical map its categories' names." + "description": "A custom colormap: its kind, its colors, and for a categorical map, its category names." }, "ColormapKind": { "oneOf": [ @@ -1117,7 +848,7 @@ { "type": "string", "const": "diverging", - "description": "Below and above a middle." + "description": "Below and above a midpoint." }, { "type": "string", @@ -1130,33 +861,652 @@ "description": "Distinct colors for distinct integers." } ], - "description": "How a colormap spans values." + "description": "How a colormap maps values: `sequential` and `diverging` maps clamp, `cyclic` maps wrap, and\n`categorical` maps give each integer its own color." }, - "Size": { - "description": "A size: CSS pixels, or position units with `units`.", + "CallNode": { + "description": "An operation, written as an object with one key: the operation's name, holding its arguments. For example `{ \"add\": [\"$a\", 1] }`. The second argument of `is` and `contains` is plain text, not a field or a color.", "anyOf": [ { - "$ref": "#/$defs/Expr" + "type": "object", + "properties": { + "neg": { + "description": "The value negated.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "neg" + ], + "additionalProperties": false }, { - "$ref": "#/$defs/Units" - } - ] - }, - "Units": { - "type": "object", - "additionalProperties": false, - "properties": { - "units": { - "$ref": "#/$defs/Expr" - }, - "atLeast": { - "type": [ - "number", - "null" - ], - "format": "float", - "description": "CSS pixels it keeps at least, however far the view zooms out.", + "type": "object", + "properties": { + "not": { + "description": "1 where the value is at most 0.5, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "not" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "abs": { + "description": "The absolute value.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "abs" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "sqrt": { + "description": "The square root. Negative values give a missing value.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "sqrt" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "sin": { + "description": "The sine of the value, an angle in radians.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "sin" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "cos": { + "description": "The cosine of the value, an angle in radians.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "cos" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "floor": { + "description": "The largest whole number at most the value.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "floor" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "ln": { + "description": "The natural logarithm. Values at most 0 give a missing value.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "ln" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "exp": { + "description": "e raised to the value.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 1, + "maxItems": 1 + } + }, + "required": [ + "exp" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "add": { + "description": "The sum of the two values.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "add" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "sub": { + "description": "The first value minus the second.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "sub" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "mul": { + "description": "The product of the two values.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "mul" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "div": { + "description": "The first value divided by the second. Dividing by 0 gives a missing value.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "div" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "rem": { + "description": "The remainder of the first value divided by the second, with the sign of the second: -1 and 3 give 2.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "rem" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "pow": { + "description": "The first value raised to the power of the second.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "pow" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "min": { + "description": "The smaller of the two values.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "min" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "max": { + "description": "The larger of the two values.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "max" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "lt": { + "description": "1 where the first value is less than the second, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "lt" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "le": { + "description": "1 where the first value is at most the second, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "le" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "gt": { + "description": "1 where the first value is greater than the second, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "gt" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "ge": { + "description": "1 where the first value is at least the second, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "ge" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "eq": { + "description": "1 where the two values are equal, else 0. Colors compare too.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "eq" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "ne": { + "description": "1 where the two values differ, else 0. Colors compare too.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "ne" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "and": { + "description": "1 where both values are above 0.5, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "and" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "or": { + "description": "1 where either value is above 0.5, else 0.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Expr" + } + ], + "minItems": 2, + "maxItems": 2 + } + }, + "required": [ + "or" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "is": { + "description": "Whether a text field equals the plain text exactly.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "tsType": "[field: ExprInput, text: string]" + } + }, + "required": [ + "is" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "contains": { + "description": "Whether a text field contains the plain text, ignoring case. In a mark, the field must belong to the mark's own table.", + "type": "array", + "prefixItems": [ + { + "$ref": "#/$defs/Expr" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "tsType": "[field: ExprInput, text: string]" + } + }, + "required": [ + "contains" + ], + "additionalProperties": false + } + ] + }, + "Size": { + "description": "A size: a value in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. `{ units }` is position units (degrees on the earth), which grow as the view zooms in.", + "anyOf": [ + { + "$ref": "#/$defs/Expr" + }, + { + "$ref": "#/$defs/Px" + }, + { + "$ref": "#/$defs/Units" + } + ] + }, + "Px": { + "type": "object", + "additionalProperties": false, + "properties": { + "px": { + "$ref": "#/$defs/Expr", + "description": "The size, in CSS pixels." + } + }, + "required": [ + "px" + ], + "description": "A size in CSS pixels, regardless of the space's unit." + }, + "Units": { + "type": "object", + "additionalProperties": false, + "properties": { + "units": { + "$ref": "#/$defs/Expr", + "description": "The size, in position units." + }, + "atLeast": { + "type": [ + "number", + "null" + ], + "format": "float", + "description": "The minimum on-screen size in CSS pixels, however far the view zooms out.", "minimum": 0 }, "atMost": { @@ -1165,14 +1515,14 @@ "null" ], "format": "float", - "description": "CSS pixels it keeps at most, however far the view zooms in.", + "description": "The maximum on-screen size in CSS pixels, however far the view zooms in.", "minimum": 0 } }, "required": [ "units" ], - "description": "A size in position units, so it scales as the view zooms, kept within CSS pixels when given." + "description": "A size in position units (degrees on the earth), regardless of the space's unit. It scales with\nthe zoom. `atLeast` and `atMost` bound it in CSS pixels." }, "Marker": { "anyOf": [ @@ -1202,14 +1552,14 @@ "additionalProperties": false }, { - "description": "A block: a rectangle with rounded corners, sized for diagrams.", + "description": "A rectangle with rounded corners that holds its label inside, like a diagram's blocks.", "type": "object", "properties": { "block": { "type": "object", "properties": { "corner": { - "description": "Corner radius, CSS pixels, or units in a diagram; the theme's `corner` unless set.", + "description": "Corner radius in CSS pixels, or units in a diagram. Defaults to the theme's\n`diagram.corner`.", "type": [ "number", "null" @@ -1226,18 +1576,18 @@ "additionalProperties": false }, { - "description": "A shape's outline, filled to `fill` from 0 to 1: around a circle, up other shapes.", + "description": "A shape's outline, filled to `fill` (0 to 1). A circle fills around its edge, and other\nshapes fill upward.", "type": "object", "properties": { "gauge": { "type": "object", "properties": { "fill": { - "description": "How full each row's gauge is.", + "description": "How full each row's gauge is, from 0 to 1.", "$ref": "#/$defs/Expr" }, "shape": { - "description": "The outline.", + "description": "The outline's shape. Defaults to an ellipse.", "$ref": "#/$defs/Shape" } }, @@ -1253,7 +1603,7 @@ "additionalProperties": false }, { - "description": "A circle cut into slices, each as large as its value.", + "description": "A circle divided into slices sized in proportion to their values.", "type": "object", "properties": { "pie": { @@ -1276,7 +1626,7 @@ } }, "hole": { - "description": "The hole in the middle, as a fraction of the radius.", + "description": "The hole in the middle, as a fraction of the radius. Defaults to 0.", "type": "number", "format": "float" } @@ -1294,7 +1644,7 @@ "additionalProperties": false }, { - "description": "One of `images`, fitted in the marker's box; `index` picks which for each row.", + "description": "One of `images`, fitted to the marker's box. `index` picks the image for each row.", "type": "object", "properties": { "icon": { @@ -1308,7 +1658,7 @@ } }, "index": { - "description": "Each row's image, from 0.", + "description": "Each row's image index, from 0.", "$ref": "#/$defs/Expr" } }, @@ -1325,18 +1675,18 @@ "additionalProperties": false }, { - "description": "A marker in WGSL: `source` defines `fn marker(f: MarkerFragment) -> MarkerColor`, reading up to\neight `inputs` as `f.input[i]`. It compiles once, when the view first draws it.", + "description": "A custom marker in WGSL. `source` defines `fn marker(f: MarkerFragment) -> MarkerColor`,\nwhich can read up to eight `inputs` as `f.input[i]`. It compiles once, when the view first\ndraws it.", "type": "object", "properties": { "wgsl": { "type": "object", "properties": { "source": { - "description": "The function's source.", + "description": "The WGSL source of the `marker` function.", "type": "string" }, "inputs": { - "description": "Each row's inputs.", + "description": "Each row's inputs, at most eight.", "type": "array", "items": { "$ref": "#/$defs/Expr" @@ -1357,7 +1707,7 @@ ] } ], - "description": "How a vertex is drawn: a shape's name, `\"block\"`, or `{ shape | block | gauge | pie | icon | wgsl:\noptions }`." + "description": "How a vertex is drawn: a shape's name, `\"block\"`, or\n`{ shape | block | gauge | pie | icon | wgsl: options }`. Defaults to an ellipse." }, "Shape": { "oneOf": [ @@ -1414,7 +1764,7 @@ "size", "rgba" ], - "description": "RGBA texels, straight alpha, sRGB, rows top to bottom." + "description": "An image of RGBA texels: straight alpha, sRGB, rows top to bottom." }, "Port": { "anyOf": [ @@ -1431,11 +1781,11 @@ "type": "object", "properties": { "side": { - "description": "The side it sits on.", + "description": "The side of the marker it sits on.", "$ref": "#/$defs/Side" }, "order": { - "description": "Its place among the ports on that side, from the top or the left; in declaration order unless set.", + "description": "Its position among the ports on its side, counted from the top or left. Ports without an\n`order` follow those with one, in declaration order.", "type": [ "integer", "null" @@ -1451,11 +1801,11 @@ ] }, "shape": { - "description": "Its marker.", + "description": "Its shape. Defaults to an ellipse.", "$ref": "#/$defs/Shape" }, "output": { - "description": "Whether it drives the net its reference field names; an input unless set.", + "description": "Whether it is an output, which drives its net. Defaults to `false` (an input).", "type": "boolean" } }, @@ -1465,7 +1815,7 @@ ] } ], - "description": "A vertex's port: where a net its reference field names meets it, on one side of its marker: a\nside, or options. An output drives its net: layouts rank a net from its outputs, and its arrows\npoint away from them.\n\n```\n# use latkit::{Port, Side, Vertices};\nlet pumps = Vertices::of(\"plant\", \"Pump\").port(\"in\", Side::Left).port(\"out\", Port::output(Side::Right));\n```" + "description": "A port on one side of a vertex's marker, where a net connects: a side name, or options. The\nvertex names the net in a reference field. An output drives its net: layouts rank a net from its\noutputs, and the net's arrows point away from them.\n\n```\n# use latkit::{Port, Side, Vertices};\nlet pumps = Vertices::of(\"plant\", \"Pump\").port(\"in\", Side::Left).port(\"out\", Port::output(Side::Right));\n```" }, "Side": { "oneOf": [ @@ -1498,15 +1848,14 @@ "$ref": "#/$defs/Expr" }, { - "description": "A row's text, placed beside, inside, or along what its mark draws, following it every frame: a\ntext field, or options. Labels that would overlap give way by priority; among equals, those\nalready shown keep their place. In a diagram a label is part of the drawing: it scales with the\nzoom, fades out once too small to read, and shows wherever it fits its vertex.\n\n```\n# use latkit::{field, Label, Vertices};\nlet buses = Vertices::of(\"grid\", \"Bus\").position(\"lon\", \"lat\").label(Label::new(\"name\").priority(field(\"kv\")));\n```", "type": "object", "properties": { "text": { - "description": "A text field.", + "description": "The text field to show, or a `when` that returns that field or missing. A row whose text is\nmissing or empty has no label. In a diagram, the text may read only data, since the sheet\nsizes blocks around it.", "$ref": "#/$defs/Expr" }, "size": { - "description": "CSS pixels, or units in a diagram; the theme's font size unless set.", + "description": "Font size in CSS pixels, or diagram units in a diagram. Defaults to the theme's\n`text.size`.", "type": [ "number", "null" @@ -1514,18 +1863,7 @@ "format": "float" }, "color": { - "description": "Each row's text color; the theme's text color unless set.", - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ] - }, - "priority": { - "description": "Where labels would overlap, the higher shows; the mark's priority unless set.", + "description": "Each row's text color. Defaults to the theme's text color.", "anyOf": [ { "$ref": "#/$defs/Expr" @@ -1536,265 +1874,33 @@ ] }, "maxWidth": { - "description": "How far a line runs before it wraps: CSS pixels, or units in a diagram.", + "description": "The width at which lines wrap, in CSS pixels, or diagram units in a diagram.", "type": [ "number", "null" ], "format": "float" }, - "font": { - "description": "The CSS font family list its texts are set in; the theme's font unless set.", - "type": [ - "string", - "null" - ] - }, - "max": { - "description": "The most labels shown at once, those of highest priority among the rows in view; where they\nwould overlap, fewer.", - "type": [ - "integer", - "null" - ], - "format": "uint32", - "minimum": 0 - } - }, - "additionalProperties": false, - "required": [ - "text" - ] - } - ] - }, - "Edges": { - "type": "object", - "additionalProperties": false, - "properties": { - "of": { - "type": "array", - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The dataset and table drawn.", - "tsType": "[data: string, table: string]" - }, - "ends": { - "type": [ - "array", - "null" - ], - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The reference fields naming each row's two vertices.", - "tsType": "[from: string, to: string]" - }, - "bends": { - "type": [ - "string", - "null" - ], - "description": "A path field of points each row runs through between its ends." - }, - "route": { - "anyOf": [ - { - "$ref": "#/$defs/Route" - }, - { - "type": "null" - } - ], - "description": "How each row runs between its ends; on the earth geodesic, elsewhere straight, unless set.\nOrthogonal routes run around blocks, bends rounded; diagrams only." - }, - "arrows": { - "type": "boolean", - "description": "Whether each row ends in an arrowhead at its second end; a net's at each input it reaches." - }, - "poles": { - "type": "boolean", - "description": "Draws each row as a pole from its own vertex down to the ground beneath it, on a plane or the\nearth; it has no ends." - }, - "joins": { - "type": [ - "array", - "null" - ], - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "Draws each row as a net on a plane or the earth: it joins the vertex rows of `table` whose\nreference field names it, as a star from their middle to each.", - "tsType": "[table: string, reference: string]" - }, - "width": { - "anyOf": [ - { - "$ref": "#/$defs/Size" - }, - { - "type": "null" - } - ], - "description": "CSS pixels, or position units with `units`; units in a diagram." - }, - "dash": { - "$ref": "#/$defs/Expr", - "description": "Whether each row is dashed." - }, - "flow": { - "$ref": "#/$defs/Expr", - "description": "Comets flowing along each row at this speed a second: CSS pixels, units in a diagram,\nor position units where the theme's flow is in units; negative flows backward. A speed\nthat changes carries comets on from where they are." - }, - "color": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's color; the theme's unless set." - }, - "visible": { - "$ref": "#/$defs/Expr", - "description": "Whether each row is drawn and picked." - }, - "priority": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's priority, from its data alone: what matters more draws on top, stays drawn as\na table too large to draw whole thins to the tiles in view, and is labeled first." - }, - "name": { - "type": [ - "string", - "null" - ], - "description": "Tells two marks of one table apart across scenes, and names the mark in hits." - }, - "pick": { - "type": [ - "boolean", - "null" - ], - "description": "Whether rows are hovered, clicked, and probed. Unless set, rows that are things are\n(vertices, edges, faces, a plot's traces) and geometry is not (areas, and lines along\npaths: coasts, borders, roads)." - }, - "select": { - "type": "boolean", - "description": "Whether a click selects a row." - }, - "drag": { - "type": "boolean", - "description": "Whether pressing a row drags it instead of panning." - }, - "label": { - "anyOf": [ - { - "$ref": "#/$defs/Label" - }, - { - "type": "null" + "font": { + "description": "The CSS font family list for the text. Defaults to the theme's `text.font`.", + "type": [ + "string", + "null" + ] } - ], - "description": "Each row's text: a text field, or a label's options." - }, - "shadow": { - "type": "boolean", - "description": "Whether each row casts a shadow onto the ground and onto faces, away from the theme's\nlight." - }, - "legend": { - "type": [ - "string", - "null" - ], - "description": "A legend of the colors at the view's bottom right, under this title: the colormap across\nits domain, ticked; a cell for each category; or a swatch beside each name of a map's\n`categories`. The color maps through a colormap with a domain or named categories; a\nconditional's `otherwise` is read before its cases." - } - }, - "required": [ - "of" - ], - "description": "Rows drawn as strokes between the vertices two reference fields name: branches, links, wires. In a diagram, a row without ends is a net: it joins every vertex port whose reference field names it." - }, - "Route": { - "oneOf": [ - { - "type": "string", - "const": "straight", - "description": "Straight across the screen." - }, - { - "type": "string", - "const": "geodesic", - "description": "Along the great circle, on the earth." - }, - { - "type": "string", - "const": "orthogonal", - "description": "In horizontal and vertical runs around vertices, leaving and entering at their sides, bends\nrounded. Diagrams only." + }, + "additionalProperties": false, + "required": [ + "text" + ] } ], - "description": "How an edge runs between its ends." + "description": "Text for each row, placed beside, inside, or along the row's shape and moving with it: a text\nfield, or options. A label shows when its row is visible and its text fits, for example inside\na block.\n\nLabels beside or along their rows never overlap. Where they would, the row drawn on top keeps\nits label (so `priority` decides), and a dense view labels as many rows as fit. A row whose text\nis missing or empty has no label, so a `when` that returns the field or missing labels only\nsome rows. In a diagram, a label scales with the zoom, fades out when too small to read, and\nshows wherever it fits its vertex.\n\n```\n# use latkit::{field, when, zoom, Expr, Label, Vertices};\nlet named = when(zoom().ge(6.0)).then(field(\"name\")).otherwise(Expr::missing());\nlet buses = Vertices::of(\"grid\", \"Bus\").position(field(\"lon\"), field(\"lat\"));\nlet labeled = buses.label(Label::new(named));\n```" }, - "Faces": { + "MarkOptions": { "type": "object", - "additionalProperties": false, + "description": "The options every mark has.", "properties": { - "of": { - "type": "array", - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The dataset and table drawn.", - "tsType": "[data: string, table: string]" - }, - "corners": { - "type": [ - "array", - "null" - ], - "items": { - "type": "string" - }, - "minItems": 3, - "maxItems": 3, - "description": "The reference fields naming each row's three vertices. Without a color, a face blends the\ncolors of its corners.", - "tsType": "[a: string, b: string, c: string]" - }, "color": { "anyOf": [ { @@ -1804,11 +1910,11 @@ "type": "null" } ], - "description": "Each row's color; the theme's unless set." + "description": "Each row's color: a CSS color, a value through a colormap, or a conditional that\npicks between colors. Defaults to the theme's color for the mark. Faces default to a\nblend of their corners' colors." }, "visible": { "$ref": "#/$defs/Expr", - "description": "Whether each row is drawn and picked." + "description": "Whether each row is drawn and can be picked. Defaults to `true`." }, "priority": { "anyOf": [ @@ -1819,29 +1925,29 @@ "type": "null" } ], - "description": "Each row's priority, from its data alone: what matters more draws on top, stays drawn as\na table too large to draw whole thins to the tiles in view, and is labeled first." + "description": "Each row's priority. It reads only data, not view state. Higher rows draw on top,\nkeep their labels where labels overlap, and are dropped last when a large table is\nthinned." }, "name": { "type": [ "string", "null" ], - "description": "Tells two marks of one table apart across scenes, and names the mark in hits." + "description": "Distinguishes two marks of the same table across scenes, and names the mark in hits." }, "pick": { "type": [ "boolean", "null" ], - "description": "Whether rows are hovered, clicked, and probed. Unless set, rows that are things are\n(vertices, edges, faces, a plot's traces) and geometry is not (areas, and lines along\npaths: coasts, borders, roads)." + "description": "Whether rows respond to hover, clicks, and picks. Defaults to on for vertices,\nedges, faces, and plot traces, and off for geometry: areas, and lines along paths\nsuch as coasts, borders, and roads." }, "select": { "type": "boolean", - "description": "Whether a click selects a row." + "description": "Whether clicking a row selects it. Defaults to `true`." }, "drag": { "type": "boolean", - "description": "Whether pressing a row drags it instead of panning." + "description": "Whether pressing and moving a row drags it instead of panning the camera. Drags are\nreported as [`Event::Drag`](crate::Event). Defaults to `false`." }, "label": { "anyOf": [ @@ -1852,62 +1958,139 @@ "type": "null" } ], - "description": "Each row's text: a text field, or a label's options." - }, - "shadow": { - "type": "boolean", - "description": "Whether each row casts a shadow onto the ground and onto faces, away from the theme's\nlight." + "description": "Each row's label: a text field, or label options." }, "legend": { "type": [ "string", "null" ], - "description": "A legend of the colors at the view's bottom right, under this title: the colormap across\nits domain, ticked; a cell for each category; or a swatch beside each name of a map's\n`categories`. The color maps through a colormap with a domain or named categories; a\nconditional's `otherwise` is read before its cases." + "description": "A legend of the mark's colors at the view's bottom right, with this title. It shows\na ticked colormap across its domain, a cell for each category, or a swatch for each\nname in the colormap's `categories`. `color` must map through a colormap with a\nfixed domain or named categories. In a conditional, `otherwise` is searched before\nthe cases." } - }, - "required": [ - "of" - ], - "description": "Rows drawn as triangles between the vertices three reference fields name: meshes, surfaces, and fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen from above shows its exact colors." + } }, - "Lines": { - "type": "object", - "additionalProperties": false, - "properties": { - "of": { - "type": "array", - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The dataset and table drawn.", - "tsType": "[data: string, table: string]" + "Edges": { + "allOf": [ + { + "$ref": "#/$defs/MarkOptions" }, - "geometry": { - "type": [ - "string", - "null" - ], - "description": "The path or polygon field each row runs along; `geometry` unless set. Not on a plot." + { + "$ref": "#/$defs/StrokeOptions" }, - "trace": { - "type": [ - "string", - "null" - ], - "description": "The sampled field each row traces across the data's axis, on a plot alone. A field not recorded\nyet draws nothing until its first frame." + { + "type": "object", + "description": "The mark's JSON form: the key naming its kind, its own options, then the stroke options\nand the options every mark has.", + "properties": { + "edges": { + "type": "array", + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "The dataset and table drawn, `[data, table]`.", + "tsType": "TableInput" + }, + "ends": { + "type": [ + "array", + "null" + ], + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "The reference fields naming each row's two vertices.", + "tsType": "[from: string, to: string]" + }, + "bends": { + "type": [ + "string", + "null" + ], + "description": "A path field whose points each row passes through between its ends." + }, + "route": { + "anyOf": [ + { + "$ref": "#/$defs/Route" + }, + { + "type": "null" + } + ], + "description": "How each row runs between its ends. Defaults to geodesic on the earth and straight\nelsewhere. `orthogonal` routes around blocks with rounded bends, in diagrams only." + }, + "arrows": { + "type": "boolean", + "description": "Whether each row has an arrowhead at its second end. A net has one at each input it\nreaches. Defaults to `false`." + }, + "poles": { + "type": "boolean", + "description": "Draws each row as a pole from its own vertex down to the ground, on a plane or the earth.\nPoles take no `ends`, `bends`, or `joins`. Defaults to `false`." + }, + "joins": { + "type": [ + "array", + "null" + ], + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "Draws each row as a net on a plane or the earth. The net joins the vertex rows of `table`\nwhose `reference` field names it, as a star from their center to each member.", + "tsType": "[table: string, reference: string]" + } + }, + "required": [ + "edges" + ] + } + ], + "unevaluatedProperties": false, + "description": "Rows drawn as strokes between two vertices named by reference fields: branches, links, wires. In a diagram, a row without ends is a net. It joins every vertex port whose reference field names it." + }, + "Route": { + "oneOf": [ + { + "type": "string", + "const": "straight", + "description": "A straight line on screen." }, - "interpolation": { - "$ref": "#/$defs/Interpolation", - "description": "How each row runs between its samples, on a plot." + { + "type": "string", + "const": "geodesic", + "description": "Along the great circle, on the earth." }, + { + "type": "string", + "const": "orthogonal", + "description": "Horizontal and vertical runs around vertices that leave and enter at their sides, with\nrounded bends. Diagrams only." + } + ], + "description": "How an edge runs between its ends." + }, + "StrokeOptions": { + "type": "object", + "description": "The options of marks drawn as strokes: edges and lines.", + "properties": { "width": { "anyOf": [ { @@ -1917,91 +2100,117 @@ "type": "null" } ], - "description": "CSS pixels, or position units with `units`; units in a diagram." + "description": "Each row's width in CSS pixels, or units in a diagram, unless given with `px` or\n`units`. Defaults to the theme's width." }, "dash": { "$ref": "#/$defs/Expr", - "description": "Whether each row is dashed." + "description": "Whether each row is dashed, using the theme's `dash` period. Defaults to `false`." }, "flow": { "$ref": "#/$defs/Expr", - "description": "Comets flowing along each row at this speed a second: CSS pixels, units in a diagram,\nor position units where the theme's flow is in units; negative flows backward. A speed\nthat changes carries comets on from where they are." - }, - "color": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's color; the theme's unless set." - }, - "visible": { - "$ref": "#/$defs/Expr", - "description": "Whether each row is drawn and picked." + "description": "The speed of comets along each row, per second, in the unit of the theme's `flow`\nlengths. Negative speeds flow backward. 0 draws no comets. Defaults to 0. When the\nspeed changes, comets continue from where they are." + } + } + }, + "Faces": { + "allOf": [ + { + "$ref": "#/$defs/MarkOptions" }, - "priority": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" + { + "type": "object", + "description": "The mark's JSON form: the key naming its kind, its own options, then the stroke options\nand the options every mark has.", + "properties": { + "faces": { + "type": "array", + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "The dataset and table drawn, `[data, table]`.", + "tsType": "TableInput" }, - { - "type": "null" + "corners": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "minItems": 3, + "maxItems": 3, + "description": "The reference fields naming each row's three vertices. Required. Without a color, a face\nblends the colors of its corners.", + "tsType": "[a: string, b: string, c: string]" } - ], - "description": "Each row's priority, from its data alone: what matters more draws on top, stays drawn as\na table too large to draw whole thins to the tiles in view, and is labeled first." - }, - "name": { - "type": [ - "string", - "null" - ], - "description": "Tells two marks of one table apart across scenes, and names the mark in hits." - }, - "pick": { - "type": [ - "boolean", - "null" - ], - "description": "Whether rows are hovered, clicked, and probed. Unless set, rows that are things are\n(vertices, edges, faces, a plot's traces) and geometry is not (areas, and lines along\npaths: coasts, borders, roads)." - }, - "select": { - "type": "boolean", - "description": "Whether a click selects a row." + }, + "required": [ + "faces" + ] + } + ], + "unevaluatedProperties": false, + "description": "Rows drawn as triangles between three vertices named by reference fields: meshes, surfaces, and fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen from above shows its exact colors." + }, + "Lines": { + "allOf": [ + { + "$ref": "#/$defs/MarkOptions" }, - "drag": { - "type": "boolean", - "description": "Whether pressing a row drags it instead of panning." + { + "$ref": "#/$defs/StrokeOptions" }, - "label": { - "anyOf": [ - { - "$ref": "#/$defs/Label" + { + "type": "object", + "description": "The mark's JSON form: the key naming its kind, its own options, then the stroke options\nand the options every mark has.", + "properties": { + "lines": { + "type": "array", + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "The dataset and table drawn, `[data, table]`.", + "tsType": "TableInput" }, - { - "type": "null" + "geometry": { + "type": [ + "string", + "null" + ], + "description": "The path or polygon field each row runs along. Defaults to the field named `geometry`. Not\nallowed on a plot." + }, + "trace": { + "type": [ + "string", + "null" + ], + "description": "The sampled field each row traces across the data's axis. Required on a plot, and allowed\nonly there. A field with no frames yet draws nothing until its first frame arrives." + }, + "interpolation": { + "$ref": "#/$defs/Interpolation", + "description": "How each row runs between its samples on a plot. Defaults to `linear`." } - ], - "description": "Each row's text: a text field, or a label's options." - }, - "shadow": { - "type": "boolean", - "description": "Whether each row casts a shadow onto the ground and onto faces, away from the theme's\nlight." - }, - "legend": { - "type": [ - "string", - "null" - ], - "description": "A legend of the colors at the view's bottom right, under this title: the colormap across\nits domain, ticked; a cell for each category; or a swatch beside each name of a map's\n`categories`. The color maps through a colormap with a domain or named categories; a\nconditional's `otherwise` is read before its cases." + }, + "required": [ + "lines" + ] } - }, - "required": [ - "of" ], - "description": "Rows drawn as lines: along a path field (roads, rivers, borders) or a polygon field's rings, outlining its areas; on a plot, through a sampled field's history across the data's axis, what a monitor shows. On the earth paths bend with the globe and stay whole across the antimeridian." + "unevaluatedProperties": false, + "description": "Rows drawn as lines along a path field (roads, rivers, borders) or a polygon field's rings (area outlines). On a plot, each row traces a sampled field's history across the data's axis, like a monitor. On the earth, paths follow the globe's curve and stay unbroken across the antimeridian." }, "Interpolation": { "oneOf": [ @@ -2013,117 +2222,55 @@ { "type": "string", "const": "stepBefore", - "description": "Each value holds from the sample before it: steps rise at their samples' ends." + "description": "Each value holds from the previous sample up to its own, so the line steps at the previous\nsample." }, { "type": "string", "const": "stepAfter", - "description": "Each value holds until the next sample: steps rise at the next sample." + "description": "Each value holds until the next sample, where the line steps." } ], "description": "How a line runs between its samples." }, "Areas": { - "type": "object", - "additionalProperties": false, - "properties": { - "of": { - "type": "array", - "prefixItems": [ - { - "type": "string" - }, - { - "type": "string" - } - ], - "minItems": 2, - "maxItems": 2, - "description": "The dataset and table drawn.", - "tsType": "[data: string, table: string]" - }, - "geometry": { - "type": [ - "string", - "null" - ], - "description": "The polygon field each row fills: one polygon's rings, or polygons. `geometry` unless set." - }, - "color": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's color; the theme's unless set." - }, - "visible": { - "$ref": "#/$defs/Expr", - "description": "Whether each row is drawn and picked." - }, - "priority": { - "anyOf": [ - { - "$ref": "#/$defs/Expr" - }, - { - "type": "null" - } - ], - "description": "Each row's priority, from its data alone: what matters more draws on top, stays drawn as\na table too large to draw whole thins to the tiles in view, and is labeled first." - }, - "name": { - "type": [ - "string", - "null" - ], - "description": "Tells two marks of one table apart across scenes, and names the mark in hits." - }, - "pick": { - "type": [ - "boolean", - "null" - ], - "description": "Whether rows are hovered, clicked, and probed. Unless set, rows that are things are\n(vertices, edges, faces, a plot's traces) and geometry is not (areas, and lines along\npaths: coasts, borders, roads)." - }, - "select": { - "type": "boolean", - "description": "Whether a click selects a row." - }, - "drag": { - "type": "boolean", - "description": "Whether pressing a row drags it instead of panning." + "allOf": [ + { + "$ref": "#/$defs/MarkOptions" }, - "label": { - "anyOf": [ - { - "$ref": "#/$defs/Label" + { + "type": "object", + "description": "The mark's JSON form: the key naming its kind, its own options, then the stroke options\nand the options every mark has.", + "properties": { + "areas": { + "type": "array", + "prefixItems": [ + { + "type": "string" + }, + { + "type": "string" + } + ], + "minItems": 2, + "maxItems": 2, + "description": "The dataset and table drawn, `[data, table]`.", + "tsType": "TableInput" }, - { - "type": "null" + "geometry": { + "type": [ + "string", + "null" + ], + "description": "The polygon field each row fills. It holds one polygon's rings or several polygons. Defaults\nto the field named `geometry`." } - ], - "description": "Each row's text: a text field, or a label's options." - }, - "shadow": { - "type": "boolean", - "description": "Whether each row casts a shadow onto the ground and onto faces, away from the theme's\nlight." - }, - "legend": { - "type": [ - "string", - "null" - ], - "description": "A legend of the colors at the view's bottom right, under this title: the colormap across\nits domain, ticked; a cell for each category; or a swatch beside each name of a map's\n`categories`. The color maps through a colormap with a domain or named categories; a\nconditional's `otherwise` is read before its cases." + }, + "required": [ + "areas" + ] } - }, - "required": [ - "of" ], - "description": "Rows drawn as the areas a polygon field holds: parcels, states, lakes, the land and water of a map. The view cuts each row's polygons into triangles once, off the frame when they are large; on the earth they bend with the globe and stay whole across the antimeridian. Outline areas with a [`Lines`](crate::Lines) mark of the same table, which draws their rings." + "unevaluatedProperties": false, + "description": "Rows drawn as the areas of a polygon field: parcels, states, lakes, the land and water of a map. The view triangulates each row's polygons once, in the background when they are large. On the earth, areas follow the globe's curve and stay unbroken across the antimeridian. To outline areas, add a [`Lines`](crate::Lines) mark of the same table. It draws their rings." }, "Layout": { "anyOf": [ @@ -2138,7 +2285,7 @@ { "oneOf": [ { - "description": "Distances on the drawing follow distances in the graph.", + "description": "Drawn distances follow graph distances.", "type": "object", "properties": { "stress": { @@ -2151,7 +2298,7 @@ "additionalProperties": false }, { - "description": "Ranks along a direction, for flows and hierarchies.", + "description": "Ranks along a direction, for processes and hierarchies.", "type": "object", "properties": { "layered": { @@ -2166,7 +2313,7 @@ ] } ], - "description": "How the view places vertex rows without positions: a diagram's blocks, sized, or the points of a\nplane or the earth, among the rows linked to them, gaps in the space's units: a name, or `{ stress\n| layered: options }`. A place, once given, holds: rows added later are placed around the rest,\nand [`View::arrange`](crate::View::arrange) lays every row out again." + "description": "How the view places vertex rows that have no position: a name, or\n`{ stress | layered: options }`. In a diagram it places blocks. On a plane or the earth it\nplaces points among the rows they link to. Gaps are in position units: diagram units, plane\nunits, or degrees on the earth.\n\nA placed row keeps its place. Rows added later are placed around the others, and\n[`View::arrange`](crate::View::arrange) lays out every row again." }, "Stress": { "type": "object", @@ -2175,32 +2322,32 @@ "gap": { "type": "number", "format": "double", - "description": "Units between adjacent vertices' centers.", + "description": "Distance between the centers of linked vertices, in position units. Defaults to 40.", "minimum": 0 } }, - "description": "Stress majorization: each pair of vertices pulls toward its graph distance; then their boxes\npart, keeping a quarter of the gap clear." + "description": "Stress majorization: vertices move until drawn distances match graph distances. Then\noverlapping boxes separate, keeping a quarter of `gap` between them." }, "Layered": { "type": "object", "additionalProperties": false, "properties": { - "flow": { + "direction": { "$ref": "#/$defs/Direction", - "description": "Where edges point." + "description": "The direction edges point. Defaults to `right`." }, "gap": { "type": "number", "format": "double", - "description": "Units between vertices of a rank." + "description": "Distance between vertices within a rank, in position units. Defaults to 30." }, "rankGap": { "type": "number", "format": "double", - "description": "Units between ranks." + "description": "Distance between ranks, in position units. Defaults to 80." } }, - "description": "Layered drawing: vertices in ranks, edges pointing along `flow`, and nets from their outputs." + "description": "Layered drawing: vertices in ranks, edges pointing along `direction`, and nets ranked from their\noutputs." }, "Direction": { "oneOf": [ @@ -2240,188 +2387,72 @@ "type": "object", "properties": { "background": { - "description": "Behind everything.", - "type": "string", - "tsType": "Color" - }, - "surface": { - "description": "The earth's ground, and a group's container.", + "description": "The background color.", "type": "string", "tsType": "Color" }, "grid": { - "description": "A plane's or a diagram's grid.", - "type": "string", - "tsType": "Color" - }, - "vertex": { - "description": "Vertex color.", + "description": "The color of a plane's or diagram's grid, a plot's grid lines, and the earth's graticule.", "type": "string", "tsType": "Color" }, - "vertexRadius": { - "description": "Vertex radius.", - "type": "number", - "format": "float" - }, - "edge": { - "description": "Edge color; `None` blends the colors of each edge's two vertices, and in a diagram takes the\ntext color.", - "type": [ - "string", - "null" - ], - "tsType": "Color" - }, - "edgeWidth": { - "description": "Edge width.", - "type": "number", - "format": "float" - }, - "line": { - "description": "Line color.", + "missing": { + "description": "The color a colormap gives a missing value, such as a row with no value in the field.", "type": "string", "tsType": "Color" }, - "lineWidth": { - "description": "Line width.", + "dash": { + "description": "The dash period: one dash and one gap of equal length.", "type": "number", "format": "float" }, - "area": { - "description": "Area color.", - "type": "string", - "tsType": "Color" - }, "shading": { - "description": "How much a face turned away from the camera darkens, 0 to 1.", + "description": "How much faces darken as they turn away from the camera, from 0 to 1.", "type": "number", "format": "float" }, "text": { - "description": "Text: labels, axis ticks, and titles.", - "type": "string", - "tsType": "Color" - }, - "font": { - "description": "The CSS font family text is set in.", - "type": "string" - }, - "fontSize": { - "description": "Text size.", - "type": "number", - "format": "float" - }, - "playhead": { - "description": "The line a plot draws at the shown coordinate.", - "type": "string", - "tsType": "Color" - }, - "missing": { - "description": "What a missing color reads as.", - "type": "string", - "tsType": "Color" - }, - "hover": { - "description": "The glow around hovered rows.", - "type": "string", - "tsType": "Color" + "description": "How labels, axis ticks, and titles look.", + "$ref": "#/$defs/TextTheme" }, - "selected": { - "description": "The glow around selected rows; `None` glows in each row's own color.", - "type": [ - "string", - "null" - ], - "tsType": "Color" + "vertex": { + "description": "Defaults for vertices.", + "$ref": "#/$defs/VertexTheme" }, - "glow": { - "description": "How far focus glows reach, CSS pixels.", - "type": "number", - "format": "float" + "edge": { + "description": "Defaults for edges, and how parallel edges and arrowheads look.", + "$ref": "#/$defs/EdgeTheme" }, - "dim": { - "description": "The opacity of rows outside a selection.", - "type": "number", - "format": "float" + "line": { + "description": "Defaults for lines.", + "$ref": "#/$defs/LineTheme" }, - "hoverScale": { - "description": "How much a hovered vertex sized in CSS pixels grows.", - "type": "number", - "format": "float" + "area": { + "description": "Defaults for areas.", + "$ref": "#/$defs/AreaTheme" }, - "dash": { - "description": "Length of a dash.", - "type": "number", - "format": "float" + "focus": { + "description": "How hovered and selected rows glow, and how rows outside a selection dim.", + "$ref": "#/$defs/FocusTheme" }, "flow": { "description": "How comets flowing along edges and lines look.", - "$ref": "#/$defs/Flow" - }, - "edgeSpacing": { - "description": "In a diagram, distance between the lanes of edges joining the same two vertices, and between\nparallel orthogonal wires, which keep twice it apart.", - "type": "number", - "format": "float" - }, - "laneBow": { - "description": "Elsewhere, how far edges joining the same two vertices bow apart: each lane by this fraction\nof the edge's length at its middle. A bow is part of the drawing, so zooming never moves it.", - "type": "number", - "format": "float" - }, - "arrow": { - "description": "How far an arrowhead reaches past its edge's width.", - "type": "number", - "format": "float" - }, - "halo": { - "description": "CSS pixels of background around label text.", - "type": "number", - "format": "float" - }, - "groupPadding": { - "description": "Distance between a group's members and its outline.", - "type": "number", - "format": "float" - }, - "outline": { - "description": "The border of a diagram's blocks and groups, a unit wide; `None` draws none.", - "type": [ - "string", - "null" - ], - "tsType": "Color" - }, - "corner": { - "description": "The radius of a diagram's block and group corners; a block's marker may set its own.", - "type": "number", - "format": "float" + "$ref": "#/$defs/FlowTheme" }, - "portSize": { - "description": "How wide a diagram's ports are.", - "type": "number", - "format": "float" - }, - "bend": { - "description": "The radius of an orthogonal wire's bends.", - "type": "number", - "format": "float" - }, - "shadow": { - "description": "How marks that cast shadows cast them.", - "$ref": "#/$defs/Shadow" + "diagram": { + "description": "How a diagram's blocks, groups, ports, and wires look.", + "$ref": "#/$defs/DiagramTheme" }, - "atmosphere": { - "description": "The air glowing around the globe, where the earth has an atmosphere.", - "type": "string", - "tsType": "Color" + "earth": { + "description": "How the earth looks.", + "$ref": "#/$defs/EarthTheme" }, - "night": { - "description": "How bright the earth's night side stays where it is lit by daylight, 0 to 1.", - "type": "number", - "format": "float" + "plot": { + "description": "How a plot looks.", + "$ref": "#/$defs/PlotTheme" }, "base": { - "description": "The theme these options change; dark unless set.", + "description": "The theme these options change. Defaults to `dark`.", "enum": [ "dark", "light" @@ -2430,71 +2461,304 @@ }, "additionalProperties": false } - ], - "description": "Colors and sizes marks take where their options read nothing, and the look of focus and motion: a\nbase theme's name, or options over one (`base`, dark unless set). Lengths are CSS pixels; in a\ndiagram they are units, except `glow` and `halo`." + ], + "description": "Default colors and sizes for marks, and the look of text, focus, comets, and each space, grouped\nby subject: a base theme's name (`\"dark\"` or `\"light\"`), or options that change a base. `base`\nnames the base and defaults to `dark`. Each option replaces only the base value it sets.\n\nLengths are in CSS pixels, or units in a diagram. The text halo and the focus glow are always in\nCSS pixels." + }, + "TextTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "color": { + "description": "The text color.", + "tsType": "Color", + "type": "string" + }, + "font": { + "type": "string", + "description": "The CSS font family list." + }, + "size": { + "type": "number", + "format": "float", + "description": "The font size, in the space's unit." + }, + "halo": { + "type": "number", + "format": "float", + "description": "Width of the background-colored halo around label text, in CSS pixels." + } + }, + "description": "How labels, axis ticks, and titles look." }, - "Flow": { + "VertexTheme": { "type": "object", "additionalProperties": false, "properties": { "color": { - "description": "A comet's color; its alpha is how strongly a comet covers its stroke.", + "description": "The vertex color.", "tsType": "Color", "type": "string" }, + "radius": { + "type": "number", + "format": "float", + "description": "The vertex radius, in the space's unit." + } + }, + "description": "Defaults for vertices." + }, + "EdgeTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "color": { + "description": "The edge color. `None` blends the colors of each edge's two vertices. With `None`, a\ndiagram's wires and edges without `ends` (nets and poles) use the text color.", + "tsType": "Color", + "type": [ + "string", + "null" + ] + }, + "width": { + "type": "number", + "format": "float", + "description": "The edge width, in the space's unit." + }, "spacing": { "type": "number", "format": "float", - "description": "Distance between comets." + "description": "In a diagram, the distance between edges joining the same two vertices and between parallel\northogonal wires. Wires keep twice this distance from blocks where they can." }, - "length": { + "bow": { "type": "number", "format": "float", - "description": "How far a comet's tail trails its head." + "description": "Outside a diagram, how far edges joining the same two vertices bow apart. Each lane bows out\nby this fraction of the edge's length, measured at its middle. Parallel edges spread at most\neight lanes wide, and more lanes pack closer. Bows scale with the zoom." + }, + "arrow": { + "type": "number", + "format": "float", + "description": "How far an arrowhead extends beyond the edge's width. A head's length is this plus the\nwidth." + } + }, + "description": "Defaults for edges, and how parallel edges and arrowheads look." + }, + "LineTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "color": { + "description": "The line color.", + "tsType": "Color", + "type": "string" }, "width": { "type": "number", "format": "float", - "description": "How wide a comet's head is, CSS pixels, or units in a diagram; never narrower than its stroke." + "description": "The line width, in the space's unit." + } + }, + "description": "Defaults for lines." + }, + "AreaTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "color": { + "description": "The area color.", + "tsType": "Color", + "type": "string" + } + }, + "description": "Defaults for areas." + }, + "FocusTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "hover": { + "description": "The glow color around hovered rows.", + "tsType": "Color", + "type": "string" + }, + "selected": { + "description": "The glow color around selected rows. `None` uses each row's own color.", + "tsType": "Color", + "type": [ + "string", + "null" + ] + }, + "glow": { + "type": "number", + "format": "float", + "description": "How far the hover and selection glows reach, in CSS pixels." }, - "units": { - "type": "boolean", - "description": "Whether comets are part of the drawing: their spacing, length, and each row's flow in position\nunits, degrees on the earth, so they zoom with the view and stay where they are on their lines.\nOtherwise they keep their size and speed on screen, and slide along their lines as it zooms." + "dim": { + "type": "number", + "format": "float", + "description": "The opacity of rows outside the selection while any row is selected, from 0 to 1." + }, + "grow": { + "type": "number", + "format": "float", + "description": "The scale factor for a hovered vertex sized in CSS pixels. Vertices sized in units only\nglow." } }, - "description": "How comets flowing along edges and lines look: each a round head, brightest at its middle, trailing\na tail that thins and fades." + "description": "How hovered and selected rows glow, and how rows outside a selection dim." }, - "Shadow": { + "FlowTheme": { "type": "object", "additionalProperties": false, "properties": { "color": { - "description": "What a shadow darkens toward; its alpha is how dark a shadow at contact gets.", + "description": "The comet color. Its alpha sets how strongly a comet covers its stroke. A fully transparent\ncolor turns comets off.", "tsType": "Color", "type": "string" }, - "light": { - "type": "array", - "items": { - "type": "number", - "format": "float" + "spacing": { + "$ref": "#/$defs/Length", + "description": "The distance between consecutive comets." + }, + "length": { + "$ref": "#/$defs/Length", + "description": "The length of a comet's tail behind its head." + }, + "width": { + "type": "number", + "format": "float", + "description": "The width of a comet's head, in the space's unit. A head is never narrower than its stroke." + } + }, + "description": "How comets along edges and lines look. A comet has a round head, brightest at its middle, and a\ntail that thins and fades.\n\n`spacing` and `length` must use the same unit, and each row's `flow` speed is in that unit too.\nIn position units (degrees on the earth), comets scale with the zoom and stay in place along\ntheir lines. In CSS pixels, they keep their size and speed on screen. A plain number is in the\nspace's unit." + }, + "Length": { + "description": "A length: a number in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. `{ units }` is position units (degrees on the earth), which grow as the view zooms in.", + "anyOf": [ + { + "type": "number" + }, + { + "type": "object", + "properties": { + "px": { + "type": "number", + "description": "CSS pixels, the same at any zoom." + } }, - "minItems": 3, - "maxItems": 3, - "description": "The direction toward the light, fixed in the world: shadows fall away from it." + "required": [ + "px" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "units": { + "type": "number", + "description": "Position units (degrees on the earth), which grow as the view zooms in." + } + }, + "required": [ + "units" + ], + "additionalProperties": false + } + ] + }, + "DiagramTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "group": { + "description": "The fill color of group containers.", + "tsType": "Color", + "type": "string" + }, + "outline": { + "description": "The color of the one-unit border around blocks and groups. `None` draws no border.", + "tsType": "Color", + "type": [ + "string", + "null" + ] + }, + "corner": { + "type": "number", + "format": "float", + "description": "The corner radius of blocks and groups. A block marker can set its own." }, - "softness": { + "portSize": { "type": "number", "format": "float", - "description": "How much a shadow spreads per unit of a caster's height: the light's size." + "description": "The width of ports." + }, + "bend": { + "type": "number", + "format": "float", + "description": "The bend radius of orthogonal wires." + }, + "groupPadding": { + "type": "number", + "format": "float", + "description": "The space between a group's members and its border." + }, + "shadow": { + "$ref": "#/$defs/ShadowTheme", + "description": "How blocks and groups cast shadows on the sheet." + } + }, + "description": "How a diagram's blocks, groups, ports, and wires look." + }, + "ShadowTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "color": { + "description": "The color a shadow darkens toward. Its alpha sets how dark the shadow is under the block.", + "tsType": "Color", + "type": "string" }, "lift": { "type": "number", "format": "float", - "description": "CSS pixels flat marks cast from, as if raised; in a diagram, how far a block's or a group's\nshadow falls below it, and as far again it softens." + "description": "How far a shadow falls below its block or group. Its edge softens over the same distance." + } + }, + "description": "How a diagram's blocks and groups cast shadows on the sheet, for vertex marks with `shadow` on." + }, + "EarthTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "ground": { + "description": "The ground color.", + "tsType": "Color", + "type": "string" + }, + "atmosphere": { + "description": "The color of the air glowing around the globe, when the earth's `atmosphere` is on.", + "tsType": "Color", + "type": "string" + }, + "night": { + "type": "number", + "format": "float", + "description": "The brightness of the night side when the earth has `daylight`, from 0 to 1." + } + }, + "description": "How the earth looks." + }, + "PlotTheme": { + "type": "object", + "additionalProperties": false, + "properties": { + "playhead": { + "description": "The color of the line at the view's current axis coordinate.", + "tsType": "Color", + "type": "string" } }, - "description": "How marks that cast shadows cast them onto the ground and onto faces." + "description": "How a plot looks." }, "Controls": { "anyOf": [ @@ -2509,60 +2773,60 @@ "type": "object", "properties": { "pan": { - "description": "Dragging the background moves the camera.", + "description": "Whether dragging the background pans the camera. Defaults to `true`.", "type": "boolean" }, "zoom": { - "description": "What zooms: the wheel and pinching, or the wheel only with Control or Meta held (as a\ntrackpad's pinch sends it), leaving the plain wheel to scroll the page; or nothing.", + "description": "Which input zooms the view. `wheel` zooms with the wheel and pinching. `modifier` zooms with\npinching, and with the wheel only while Control or Meta is held (as trackpad pinches send\nit), so the plain wheel scrolls the page. `off` turns zooming off. Defaults to `wheel`.", "$ref": "#/$defs/Zoom" }, "turn": { - "description": "Dragging with the secondary button, or with Shift, turns and tilts a map or plane.", + "description": "Whether dragging with the secondary button, or with Shift, rotates and tilts a map or plane.\nDefaults to `true`.", "type": "boolean" }, "keyboard": { - "description": "Tab steps through the rows nearest the view's middle and arrows to the nearest row their way;\nEnter opens the row stepped to and Space selects it. Otherwise arrows pan, `+` and `-` zoom,\nHome fits, and Escape clears the selection.", + "description": "Whether the keyboard controls the view. Defaults to `true`. Tab and Shift+Tab step through\nthe rows nearest the view's center. Arrow keys step to the nearest row in their direction.\nEnter opens the row stepped to, and Space selects it. With no row stepped to, arrow keys pan\n(Shift+arrows turn), `+` and `-` zoom, Home fits everything, and Escape clears the\nselection.", "type": "boolean" }, "hover": { - "description": "The row the user points at glows. It is hovered either way: events tell it, and `hovered()`\nreads it.", + "description": "Whether the hovered row glows and a hovered vertex sized in CSS pixels grows. Defaults to\n`true`. Hover is tracked either way: [`Event::Hovered`](crate::Event) still fires and\n`hovered()` still returns the row.", "type": "boolean" }, "pickRadius": { - "description": "CSS pixels around the pointer that picks reach; touch reaches 8 more.", + "description": "How far a row can be from the pointer and still be picked, in CSS pixels. Touch adds 8.\nDefaults to 6.", "type": "number", "format": "float" }, "selectBox": { - "description": "Dragging with Shift selects the rows in a box, instead of turning; a Shift click still toggles\nits row.", + "description": "Whether Shift+drag selects the rows inside a box instead of turning the camera. Shift+click\nstill toggles one row. Defaults to `false`.", "type": "boolean" }, "connect": { - "description": "Dragging from a port, from a vertex with Alt, or an edge off its end draws a connection,\ntold as [`Event::Connect`](crate::Event).", + "description": "Whether the user can draw connections by dragging from a port, from a vertex with Alt held,\nor an edge's end off its vertex. Connections are reported as\n[`Event::Connect`](crate::Event). Defaults to `false`.", "type": "boolean" }, "delete": { - "description": "Delete and Backspace tell [`Event::Delete`](crate::Event) with the selection.", + "description": "Whether Delete and Backspace report the selection as [`Event::Delete`](crate::Event).\nDefaults to `false`.", "type": "boolean" }, "cycle": { - "description": "Clicking again where nothing moved selects the next row under the point, nearest first, so\nrows drawn over one another can each be selected.", + "description": "Whether clicking again at the same point, with the camera unmoved, selects the next row\nunder the pointer, nearest first. This lets overlapping rows each be selected. Defaults to\n`false`.", "type": "boolean" }, "press": { - "description": "Seconds a touch or a pen held still asks for a menu, as the secondary button does; zero never\ndoes.", + "description": "How many seconds a touch or pen must hold still to request a menu\n([`Event::Menu`](crate::Event)), like the secondary button. 0 turns this off. Defaults to\n0.5.", "type": "number", "format": "double" }, "motion": { - "description": "How much the view moves on its own: eases, flights, coasting, orbits, label fades, comets, and\nwhat reads `elapsed()`.", + "description": "How much the view animates on its own: eases, camera flights, coasting, orbits, comets, and\noptions that read `elapsed()`. `system` follows the user's reduced-motion setting, `full`\nanimates everything, and `reduced` animates only motion the user drives. Defaults to\n`system`.", "$ref": "#/$defs/Motion" } }, "additionalProperties": false } ], - "description": "How a view answers the pointer and keyboard, and how much it moves on its own: a preset, or options\nover the defaults. Each mark says whether its rows pick, select, and drag; these say what the view\nitself does." + "description": "How a view responds to the pointer and keyboard, and how much it animates on its own: a preset\n(`\"none\"` or `\"inspect\"`), or options that change the defaults. Each mark sets whether its own\nrows pick, select, and drag. These options control the view itself." }, "Zoom": { "oneOf": [ @@ -2574,35 +2838,35 @@ { "type": "string", "const": "modifier", - "description": "Pinching and the keys, and the wheel only with Control or Meta held, as a trackpad's pinch\nsends it: the plain wheel scrolls the page." + "description": "Pinching and the keys, and the wheel only with Control or Meta held (as trackpad pinches\nsend it), so the plain wheel scrolls the page." }, { "type": "string", "const": "off", - "description": "Nothing." + "description": "No input zooms." } ], - "description": "What zooms a view." + "description": "Which input zooms a view." }, "Motion": { "oneOf": [ { "type": "string", "const": "system", - "description": "As the platform asks: reduced where the user asked their system for less motion." + "description": "Follows the system setting: reduced when the user has asked their system for less motion." }, { "type": "string", "const": "full", - "description": "Every motion." + "description": "All motion." }, { "type": "string", "const": "reduced", - "description": "Only what the user moves: changes apply at once, flights jump, nothing coasts or orbits,\nlabels show without fading, and comets and what reads `elapsed()` hold still." + "description": "Only motion the user drives: changes apply at once, flights jump, nothing coasts or orbits,\nand comets and options that read `elapsed()` stay still." } ], - "description": "How much a view moves on its own." + "description": "How much a view animates on its own." }, "Data": { "type": "object", @@ -2621,7 +2885,7 @@ "required": [ "name" ], - "description": "A dataset: its name and the patch that builds it." + "description": "A dataset's JSON document: its name and the patch that builds it." }, "Change": { "oneOf": [ @@ -2649,7 +2913,7 @@ "axis" ], "additionalProperties": false, - "description": "Names the axis sampled fields are recorded over, such as `time` in `s`." + "description": "Sets the axis that sampled fields are recorded over, such as `time` in `s`." }, { "type": "object", @@ -2813,7 +3077,7 @@ "set" ], "additionalProperties": false, - "description": "Writes fields for these rows: rows with known ids change, others are added, and fields left\nout keep their values." + "description": "Writes fields for these rows. Rows with existing ids are updated, new ids add rows, and\nfields left out keep their values. A missing table is added." }, { "type": "object", @@ -2839,7 +3103,7 @@ "remove" ], "additionalProperties": false, - "description": "Removes the rows with these ids." + "description": "Removes the rows with these ids. Ids that match no row are ignored." }, { "type": "object", @@ -2945,7 +3209,7 @@ "append" ], "additionalProperties": false, - "description": "Records a frame at each axis coordinate of `at`: for each field, a value per row at every\ncoordinate, frame-major; of every row, or of the rows `ids` names, the rest holding their last\nvalues." + "description": "Records a frame at each axis coordinate in `at`. Each field gives one value per row per\ncoordinate, frame-major. Covers every row, or only the rows in `ids`. Other rows keep their\nlast values." }, { "type": "object", @@ -3024,13 +3288,13 @@ "frames" ], "additionalProperties": false, - "description": "Holds a run of sampled field `field`'s entries of level `level`, numbered from `first`: at\nlevel 0 frames, each a coordinate in `at` and a value per row in `values`; at level `k` buckets\nof `8^k` frames, each its first and last frame's coordinates, and each row's least value over\neach bucket, then each row's greatest. Null is missing." + "description": "Inserts a run of entries of sampled field `field` at level `level`, numbered from `first`,\nin the form time tiles store them. At level 0 each entry is a frame, with one coordinate in\n`at` and one value per row in `values`. At level `k` each entry is a bucket of `8^k` frames.\n`at` holds the coordinates of each bucket's first and last frame. `values` holds every\nbucket's least value per row, followed by every bucket's greatest. Null is missing." } ], - "description": "One change of a patch, named by the method that makes it." + "description": "One change in a patch, keyed by the name of the `Patch` method that makes it." }, "Ids": { - "description": "Row ids: numbers or texts (null leaves a reference unwired), or `{count}` naming rows 0..count.", + "description": "Row ids: a list of numbers or texts, or `{count}` for rows 0 to count - 1. In a reference, null names no row.", "anyOf": [ { "type": "array", @@ -3049,6 +3313,7 @@ "type": "object", "properties": { "count": { + "description": "The number of rows, whose ids are 0 to count - 1.", "type": "integer", "minimum": 0, "maximum": 4294967295 @@ -3094,7 +3359,7 @@ ] }, "Geometry": { - "description": "A point `[x, y]` (a third coordinate is read past), or a list of geometries one level shallower: a path, its parts, or polygons.", + "description": "A point `[x, y]` (a third coordinate is ignored), or a list of geometries one level down: a path is a list of points, parts a list of paths, and polygons a list of parts.", "anyOf": [ { "type": "array", @@ -3131,7 +3396,7 @@ "u8", "bool" ], - "description": "How a slice's bytes hold values: little endian; `bool` a byte each, nonzero true." + "description": "The value type of a slice's bytes. Numbers are little endian. `bool` is one byte each, nonzero\nmeaning true." }, "Patch": { "type": "array", @@ -3145,7 +3410,7 @@ "properties": { "of": { "type": "string", - "description": "The table read." + "description": "The table to read." }, "ids": { "anyOf": [ @@ -3156,7 +3421,7 @@ "type": "null" } ], - "description": "The rows read, in this order unless sorted; every row unless set. Ids that name no row are\npassed over." + "description": "The rows to read, in this order unless sorted. Defaults to every row. Ids that name no row\nare skipped." }, "filter": { "anyOf": [ @@ -3167,7 +3432,7 @@ "type": "null" } ], - "description": "Keeps the rows it is true for." + "description": "Keeps only the rows for which this is true." }, "sort": { "type": "array", @@ -3184,13 +3449,13 @@ "minItems": 2, "maxItems": 2 }, - "description": "Keys to sort by, each after those before it; rows a key reads nothing for come last, and rows\nall keys order alike keep their order." + "description": "Keys to sort by. Each key breaks ties left by the keys before it. Rows where a key has no\nvalue sort last, and rows equal on every key keep their order." }, "offset": { "type": "integer", "format": "uint", "minimum": 0, - "description": "The kept rows skipped first." + "description": "How many of the kept rows to skip." }, "limit": { "type": [ @@ -3199,14 +3464,17 @@ ], "format": "uint", "minimum": 0, - "description": "The most rows read; a limit picks them in linear time before sorting only those." + "description": "The most rows to return. When sorting with a limit, the first `offset + limit` rows are\nselected in linear time and only those are sorted." }, "select": { - "type": "object", + "type": [ + "object", + "null" + ], "additionalProperties": { "$ref": "#/$defs/Expr" }, - "description": "Fields to read by name: a field as it is, anything else as numbers. Every field unless set." + "description": "Fields to return, by output name. A plain field returns that field as stored, and any other\nexpression returns numbers. Defaults to every field. An empty object returns the ids alone." }, "at": { "type": [ @@ -3214,7 +3482,7 @@ "null" ], "format": "double", - "description": "The axis coordinate sampled fields are read at; their last frame unless set." + "description": "The axis coordinate at which sampled fields are read. Defaults to their last frame." } }, "required": [ @@ -3227,15 +3495,63 @@ { "type": "string", "const": "ascending", - "description": "Least first." + "description": "Smallest first." }, { "type": "string", "const": "descending", - "description": "Greatest first." + "description": "Largest first." + } + ], + "description": "Sort direction. Rows where the key has no value sort last in either direction." + }, + "EnvelopeRequest": { + "type": "object", + "additionalProperties": false, + "properties": { + "table": { + "type": "string", + "description": "The table to read." + }, + "field": { + "type": "string", + "description": "The sampled field to read." + }, + "range": { + "type": "array", + "items": { + "type": "number", + "format": "double" + }, + "minItems": 2, + "maxItems": 2, + "description": "The axis range the spans cover, `[first, last]`. Both are finite, with `first <= last`." + }, + "spans": { + "type": "integer", + "format": "uint", + "minimum": 0, + "description": "The number of equal spans to divide the range into. At least 1." + }, + "ids": { + "anyOf": [ + { + "$ref": "#/$defs/Ids" + }, + { + "type": "null" + } + ], + "description": "The rows to read, in this order. Defaults to every row." } + }, + "required": [ + "table", + "field", + "range", + "spans" ], - "description": "Which way a sort runs; rows a key reads nothing for come last either way." + "description": "A request for each row's least and greatest value of a sampled field over equal spans of the\naxis. It reads field `field` of table `table` over `spans` equal spans from `range[0]` to\n`range[1]`, for every row or for the rows in `ids`, in that order. Like a query, it is plain\ndata that deserializes from JSON." }, "Camera": { "anyOf": [ @@ -3264,22 +3580,22 @@ "maxItems": 3 } ], - "description": "Longitude, latitude, and elevation on the earth; x, y, and elevation on a plane or a\ndiagram. In JSON, `[x, y]` looks at the ground." + "description": "On the earth, longitude, latitude, and elevation. On a plane or a diagram, x, y, and\nelevation. In JSON, `[x, y]` is shorthand for elevation 0." }, "zoom": { "type": "number", "format": "double", - "description": "On the earth the web mercator level; elsewhere, log₂ of pixels per unit." + "description": "The web mercator zoom level on the earth. Elsewhere, log₂ of CSS pixels per unit." }, "pitch": { "type": "number", "format": "double", - "description": "Degrees from looking straight down, up to 75; 0 in a diagram." + "description": "Tilt in degrees from looking straight down, up to 75. Ignored in a diagram." }, "bearing": { "type": "number", "format": "double", - "description": "Degrees clockwise from north, or from up on a plane; 0 in a diagram." + "description": "Degrees clockwise from north, or from up on a plane. Ignored in a diagram." } }, "required": [ @@ -3288,7 +3604,7 @@ "pitch", "bearing" ], - "description": "On the earth, a plane, or a diagram: the point looked at and turned about, how far in, and how\ntilted and turned. A diagram is always seen from straight above, upright." + "description": "A camera over the earth, a plane, or a diagram: the point it looks at and turns about, its\nzoom, its tilt, and its rotation. A diagram is always seen from straight above, upright." }, { "type": "object", @@ -3302,7 +3618,7 @@ }, "minItems": 2, "maxItems": 2, - "description": "The axis range shown." + "description": "The x-axis range shown." }, "y": { "type": [ @@ -3315,13 +3631,13 @@ }, "minItems": 2, "maxItems": 2, - "description": "The value range shown, or none to fit what is in view." + "description": "The value range shown, or `None` to fit the values in view." } }, "required": [ "x" ], - "description": "On a plot: the axis range across, and the value range up, or none to fit the values in view." + "description": "A plot camera: the x range shown, and the value range, or `None` to fit the values in view." } ], "description": "Where a view looks." @@ -3369,7 +3685,7 @@ "type": "object", "properties": { "min": { - "description": "The least corner.", + "description": "The minimum corner.", "type": "array", "items": { "type": "number", @@ -3379,7 +3695,7 @@ "maxItems": 3 }, "max": { - "description": "The greatest corner.", + "description": "The maximum corner.", "type": "array", "items": { "type": "number", @@ -3402,7 +3718,7 @@ "description": "The box from `min` to `max`: x, y, and elevation." } ], - "description": "What a view's fit frames." + "description": "The target of a view's fit." }, "Rows": { "type": "object", @@ -3428,7 +3744,7 @@ "table", "ids" ], - "description": "Rows of one table in a dataset, by id: what a view selects, fits, and tells as sets of rows." + "description": "Rows of one table in a dataset, by id. Views use it for selections, fits, and the rows events\nreport." }, "Row": { "type": "object", @@ -3451,7 +3767,7 @@ "table", "id" ], - "description": "A row of a table in a dataset. It survives any change that keeps the names and the id." + "description": "One row, identified by dataset name, table name, and id. It stays valid across any change that\nkeeps those names and the id." }, "Budget": { "type": "object", @@ -3461,13 +3777,13 @@ "type": "integer", "format": "uint64", "minimum": 0, - "description": "Bytes the data every view of the `Gpu` draws may take on the GPU: its pages, and each view's\nrows. A view draws a table too large for its share through the tiles the camera asks for, so\nwhat it draws always fits; pages no view draws stay cached in the room left, the least\nrecently drawn going first. By default half the device's largest storage buffer. The pool of\npages grows as data needs it, and never shrinks: set the budget before data draws." + "description": "GPU memory in bytes for the data all views of the `Gpu` draw: its pages and each view's\nper-row state. A table too large for a view's share draws as a paged table, with only the\npages the camera needs, so it always fits. Pages no view draws stay cached in the rest of\nthe budget, and the least recently drawn are evicted first. Defaults to half the device's\nlargest storage buffer. The page pool grows as data needs it and never shrinks, so set the\nbudget before drawing data." }, "uploadPerFrame": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Bytes uploaded per frame at most; larger data streams in over frames." + "description": "The most bytes uploaded per frame. Larger data streams in over several frames. Defaults to\n32 MiB." } }, "required": [ @@ -3494,20 +3810,19 @@ "scale": { "type": "number", "format": "float", - "description": "Pixels per CSS pixel: views lay out `size / scale` CSS pixels; 1 unless set." + "description": "Pixels per CSS pixel. Views lay out `size / scale` CSS pixels. Defaults to 1." }, "panels": { "$ref": "#/$defs/Panels", - "description": "Each view's panel of the picture, `[x, y, width, height]` as fractions; the whole by default." + "description": "Each view's panel, `[x, y, width, height]` as fractions of the picture. Defaults to one view\nfilling it." }, "transparent": { "type": "boolean", - "description": "Whether alpha is kept where the themes' backgrounds would be." + "description": "Whether the picture keeps alpha where the themes' backgrounds would be." }, - "deadline": { - "type": "number", - "format": "double", - "description": "Seconds a picture, or a video frame, may take; 30 unless set." + "capture": { + "$ref": "#/$defs/Capture", + "description": "When the picture is taken: once its views are complete and still, or as they are now." }, "elapsed": { "type": [ @@ -3515,13 +3830,13 @@ "null" ], "format": "double", - "description": "A still's effect time in seconds, where elapsed time and flowing comets read it." + "description": "Fixed effect time in seconds for elapsed-time expressions and flowing comets. `None` follows\nthe view's clock." } }, "required": [ "size" ], - "description": "A picture of views, each drawn into its panel of `size` pixels at `scale` pixels per CSS pixel:\nplain data, like a scene.\n\n```no_run\n# async fn example(view: &mut latkit::View) -> latkit::Result<()> {\nuse latkit::Shot;\nlet image = Shot::new([3840, 2160]).scale(2.0).image(&mut [view]).await?;\nstd::fs::write(\"view.png\", image.encode(latkit::ImageFormat::Png)?).expect(\"written\");\n# Ok(()) }\n```" + "description": "A picture of views, taken when they are complete, as its [`Capture`] says. Each view draws into\nits panel of a `size`-pixel picture, at `scale` pixels per CSS pixel. A shot is plain data, like\na scene.\n\n```no_run\n# async fn example(view: &mut latkit::View) -> latkit::Result<()> {\nuse latkit::Shot;\nlet image = Shot::new([3840, 2160]).scale(2.0).image(&mut [view]).await?;\nstd::fs::write(\"view.png\", image.encode(latkit::ImageFormat::Png)?).expect(\"written\");\n# Ok(()) }\n```" }, "Panels": { "type": "array", @@ -3534,7 +3849,22 @@ "minItems": 4, "maxItems": 4 }, - "description": "Where each of several views draws in a target: x, y, width, and height as fractions of it, a\npanel per view, in order; a later panel draws over an earlier one. None given is one view filling\nthe target.\n\n```\nuse latkit::Panels;\nlet map_over_plot = Panels::from([[0.0, 0.0, 1.0, 0.8], [0.0, 0.8, 1.0, 0.2]]);\nassert_eq!(map_over_plot.len(), 2);\nassert_eq!(Panels::default().len(), 1);\n```" + "description": "The region of a shared target each view draws in, one panel per view, in order. A panel is\n`[x, y, width, height]` as fractions of the target, from its top left. Later panels draw over\nearlier ones. With no panels, one view fills the target.\n\n```\nuse latkit::Panels;\nlet map_over_plot = Panels::from([[0.0, 0.0, 1.0, 0.8], [0.0, 0.8, 1.0, 0.2]]);\nassert_eq!(map_over_plot.len(), 2);\nassert_eq!(Panels::default().len(), 1);\n```" + }, + "Capture": { + "oneOf": [ + { + "type": "string", + "const": "still", + "description": "Once every view is complete and still. Transitions, camera flights, and tile fades advance\n1/60 s per step until they finish." + }, + { + "type": "string", + "const": "now", + "description": "Once every view is complete, as it is now. Anything in motion is captured where it is." + } + ], + "description": "When a picture is taken." } }, "outputs": { @@ -3600,7 +3930,7 @@ "type", "detail" ], - "description": "A row was double clicked, or Enter pressed on the row the keys point at." + "description": "A row was double-clicked, or Enter was pressed on the row the keyboard points at." }, { "type": "object", @@ -3613,7 +3943,7 @@ "type": "object", "properties": { "hits": { - "description": "What lies under the point, nearest first.", + "description": "The rows under the point, nearest first.", "type": "array", "items": { "$ref": "#/outputs/Hit" @@ -3634,7 +3964,7 @@ "type", "detail" ], - "description": "The secondary button clicked, or the menu key pressed: what lies under the trigger's point,\nfor a menu of what to do with it." + "description": "A context menu was requested: by a secondary click, the menu key, or a touch or pen held\nstill. Lists the rows under the trigger's point, for a menu of actions on them." }, { "type": "object", @@ -3655,7 +3985,7 @@ "$ref": "#/outputs/Hit" }, "position": { - "description": "Where the row goes, in the space's coordinates: where it was hit, moved as the pointer moved\nsince the press, at the depth the drag started; a vertex keeps where it was grabbed. On a\ndiagram's dots when its grid snaps.", + "description": "The row's new position in the space's coordinates. It is the hit point moved with the\npointer since the press, at the depth where the drag started, so a vertex keeps its\noffset from the pointer. Snapped to a diagram's grid dots when the grid snaps.", "type": "array", "items": { "type": "number", @@ -3681,7 +4011,7 @@ "type", "detail" ], - "description": "A row of a draggable mark follows the pointer: the drag starts once the pointer moves a few\npixels from where it pressed the row." + "description": "A row of a draggable mark is dragged. The drag starts once the pointer moves more than 4 CSS\npixels from where it pressed the row." }, { "type": "object", @@ -3694,7 +4024,7 @@ "type": "object", "properties": { "from": { - "description": "The end it was drawn from: for an edge, the end that stayed.", + "description": "The end it was drawn from. For a dragged edge end, the edge's other end.", "$ref": "#/outputs/End" }, "to": { @@ -3709,7 +4039,7 @@ ] }, "edge": { - "description": "The edge whose other end was dragged, which `to` replaces.", + "description": "The edge whose end was dragged off. `to` replaces that end.", "anyOf": [ { "$ref": "#/outputs/Row" @@ -3736,7 +4066,7 @@ "type", "detail" ], - "description": "A connection was drawn with [`Controls::connect`](crate::Controls): from a port, a vertex\nwith Alt held, or an edge whose end was dragged off it." + "description": "A connection was drawn with [`Controls::connect`](crate::Controls) on: from a port, from a\nvertex with Alt held, or by dragging an edge's end off it." }, { "type": "object", @@ -3770,20 +4100,20 @@ "type", "detail" ], - "description": "Delete or Backspace with rows selected and [`Controls::delete`](crate::Controls) on." + "description": "Delete or Backspace was pressed with rows selected and [`Controls::delete`](crate::Controls)\non." }, { "type": "object", "properties": { "type": { "type": "string", - "const": "hover" + "const": "hovered" }, "detail": { "type": "object", "properties": { "hit": { - "description": "The row now pointed at; `None` once it left every row.", + "description": "The row now hovered, or `None` if no row is.", "anyOf": [ { "$ref": "#/outputs/Hit" @@ -3808,14 +4138,14 @@ "type", "detail" ], - "description": "What the user points at changed: [`View::hovered`](crate::View::hovered)." + "description": "The hovered row changed: [`View::hovered`](crate::View::hovered)." }, { "type": "object", "properties": { "type": { "type": "string", - "const": "select" + "const": "selection" }, "detail": { "type": "object", @@ -3828,7 +4158,7 @@ } }, "trigger": { - "description": "How the user changed it; `None` when the data dropped rows from it.", + "description": "How the user changed it, or `None` when the data dropped rows from it.", "anyOf": [ { "$ref": "#/outputs/Trigger" @@ -3866,7 +4196,7 @@ "type", "detail" ], - "description": "The camera moved by a gesture, a fit, a flight, coasting, or an orbit:\n[`View::camera`](crate::View::camera)." + "description": "The camera moved: [`View::camera`](crate::View::camera). A gesture, fit, flight, coast, or\norbit moved it, or it moved back within its zoom and pan limits after the view's size,\ninsets, or space changed." }, { "type": "object", @@ -3883,7 +4213,7 @@ "type", "detail" ], - "description": "What the last frame stood in for changed; nothing once it is complete:\n[`View::complete`](crate::View::complete)." + "description": "The work the last frame waited on changed: [`View::waiting`](crate::View::waiting). Empty\nonce frames are complete." }, { "type": "object", @@ -3900,10 +4230,10 @@ "type", "detail" ], - "description": "Something failed, told once. A mark its data no longer fits also lists in\n[`View::failures`](crate::View::failures)." + "description": "Something failed. Reported once. A mark that no longer fits its data is also listed in\n[`View::failures`](crate::View::failures)." } ], - "description": "What happened in a view, for the application to act on. Drain them with\n[`View::events`](crate::View::events)." + "description": "Something that happened in a view, as [`View::events`](crate::View::events) reports it." }, "Hit": { "type": "object", @@ -3917,7 +4247,7 @@ "string", "null" ], - "description": "The name of the mark that drew it, when it has one: what tells two marks of one table apart." + "description": "The name of the mark that drew it, if the mark has one. It tells apart two marks of one\ntable." }, "point": { "type": "array", @@ -3927,7 +4257,7 @@ }, "minItems": 2, "maxItems": 2, - "description": "The point of the row nearest the probe, CSS pixels from the top left: a vertex's or a port's\nmiddle, the nearest point of a stroke or a triangle." + "description": "The row's point nearest the probe, in CSS pixels from the top left: a vertex's or port's\nmiddle, or the nearest point of a stroke or triangle." }, "position": { "type": "array", @@ -3949,7 +4279,7 @@ "string", "null" ], - "description": "The port hit, when a vertex's port was." + "description": "The port hit, if the hit is on a vertex's port." }, "sample": { "anyOf": [ @@ -3960,7 +4290,7 @@ "type": "null" } ], - "description": "The sample hit, when a line tracing a sampled field was." + "description": "The sample hit, if the hit is on a line tracing a sampled field." } }, "required": [ @@ -3992,7 +4322,7 @@ "table", "id" ], - "description": "A row of a table in a dataset. It survives any change that keeps the names and the id." + "description": "One row, identified by dataset name, table name, and id. It stays valid across any change that\nkeeps those names and the id." }, "Id": { "description": "A row's id: a whole number from 0, or a text. A number above 2^53 - 1 is written as its decimal text, which names the same row.", @@ -4046,11 +4376,11 @@ }, "modifiers": { "$ref": "#/outputs/Modifiers", - "description": "The keys held." + "description": "The modifier keys held." }, "device": { "$ref": "#/outputs/Device", - "description": "What acted." + "description": "The device used." } }, "required": [ @@ -4058,7 +4388,7 @@ "modifiers", "device" ], - "description": "Where, with what keys, and with what the user acted. For keys, the point is the row they point\nat, else the pointer, else the view's middle." + "description": "Where the user acted, with which modifier keys and device. For keyboard input, the point is the\nrow the keys point at, or else the pointer, or else the view's middle." }, "Modifiers": { "type": "object", @@ -4086,7 +4416,7 @@ "alt", "meta" ], - "description": "Keys held during an input." + "description": "Modifier keys held during an input." }, "Device": { "oneOf": [ @@ -4103,15 +4433,15 @@ { "type": "string", "const": "touch", - "description": "A finger: picks reach 8 CSS pixels farther." + "description": "A finger. Picks find rows up to 8 CSS pixels farther away." }, { "type": "string", "const": "keyboard", - "description": "The keys." + "description": "The keyboard." } ], - "description": "What the user acts with." + "description": "The input device." }, "Phase": { "oneOf": [ @@ -4128,7 +4458,7 @@ { "type": "string", "const": "end", - "description": "The pointer let go, or the drag was cancelled." + "description": "The pointer was released, or the drag was cancelled." } ], "description": "The phase of a [`Drag`](Event::Drag)." @@ -4151,7 +4481,7 @@ "required": [ "row" ], - "description": "A vertex a connection ends at, and its port when it has the one the connection meets." + "description": "A vertex a connection ends at, and the port the connection meets, if any." }, "Rows": { "type": "object", @@ -4177,7 +4507,7 @@ "table", "ids" ], - "description": "Rows of one table in a dataset, by id: what a view selects, fits, and tells as sets of rows." + "description": "Rows of one table in a dataset, by id. Views use it for selections, fits, and the rows events\nreport." }, "Camera": { "anyOf": [ @@ -4206,22 +4536,22 @@ "maxItems": 3 } ], - "description": "Longitude, latitude, and elevation on the earth; x, y, and elevation on a plane or a\ndiagram. In JSON, `[x, y]` looks at the ground." + "description": "On the earth, longitude, latitude, and elevation. On a plane or a diagram, x, y, and\nelevation. In JSON, `[x, y]` is shorthand for elevation 0." }, "zoom": { "type": "number", "format": "double", - "description": "On the earth the web mercator level; elsewhere, log₂ of pixels per unit." + "description": "The web mercator zoom level on the earth. Elsewhere, log₂ of CSS pixels per unit." }, "pitch": { "type": "number", "format": "double", - "description": "Degrees from looking straight down, up to 75; 0 in a diagram." + "description": "Tilt in degrees from looking straight down, up to 75. Ignored in a diagram." }, "bearing": { "type": "number", "format": "double", - "description": "Degrees clockwise from north, or from up on a plane; 0 in a diagram." + "description": "Degrees clockwise from north, or from up on a plane. Ignored in a diagram." } }, "required": [ @@ -4230,7 +4560,7 @@ "pitch", "bearing" ], - "description": "On the earth, a plane, or a diagram: the point looked at and turned about, how far in, and how\ntilted and turned. A diagram is always seen from straight above, upright." + "description": "A camera over the earth, a plane, or a diagram: the point it looks at and turns about, its\nzoom, its tilt, and its rotation. A diagram is always seen from straight above, upright." }, { "type": "object", @@ -4244,7 +4574,7 @@ }, "minItems": 2, "maxItems": 2, - "description": "The axis range shown." + "description": "The x-axis range shown." }, "y": { "type": [ @@ -4257,20 +4587,20 @@ }, "minItems": 2, "maxItems": 2, - "description": "The value range shown, or none to fit what is in view." + "description": "The value range shown, or `None` to fit the values in view." } }, "required": [ "x", "y" ], - "description": "On a plot: the axis range across, and the value range up, or none to fit the values in view." + "description": "A plot camera: the x range shown, and the value range, or `None` to fit the values in view." } ], "description": "Where a view looks." }, "Waiting": { - "description": "Work under way a frame stands in for: pipelines the scene draws with compiling, until which the view draws nothing new (`compiling`); pages uploading or tables not yet built from them (`data`); layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); a label placement or texts not yet set (`labels`); a fit due or being measured (`fit`); row regions laid out again, segment counts, or a transition beginning (`marks`). Empty once a frame is complete.", + "description": "What a frame still waits on: pipelines the scene draws with, which compile before the view draws anything new (`compiling`); pages uploading, or tables not yet built from them (`data`); layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); label texts being read back, or not yet set (`labels`); a fit due or being measured (`fit`); segment counts, or a transition starting (`marks`). Empty once the frame is complete.", "type": "array", "items": { "enum": [ @@ -4284,10 +4614,11 @@ } }, "Failure": { - "description": "What failed: its `code`, the message, and for invalid input the `path` to the part, such as `marks[2].vertices.color`.", + "description": "A failure as JSON, in the same form as the npm package's `LatkitError`.", "type": "object", "properties": { "code": { + "description": "The kind of failure. See `LatkitError.code`.", "enum": [ "invalid", "unsupported", @@ -4299,9 +4630,11 @@ ] }, "message": { + "description": "A description of what failed.", "type": "string" }, "path": { + "description": "For `invalid` failures, the path of the invalid input, such as `marks[2].color`.", "type": "string" } }, @@ -4329,53 +4662,31 @@ "type": "integer", "format": "uint64", "minimum": 0, - "description": "Rows every mark draws: every row of a table drawn whole, the rows of the tiles in view of one\ndrawn through tiles." + "description": "Rows the marks draw: every row of a fully drawn table, and the rows in the tiles in view of\na paged table." }, "draws": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Draw calls the last frame recorded, shadows and labels with the marks." + "description": "Draw calls the last frame recorded, label draws included." }, "instances": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Instances the last frame's draws drew: points, segments, triangles, glyphs; at most, for\nsegments the GPU culls." + "description": "Instances the last frame's draws drew: points, segments, triangles, and glyphs. An upper\nbound for segments, which the GPU may cull." }, "tiles": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Fetched tiles the view draws, of every tile source." + "description": "Fetched tiles the view draws, across all tile sources." }, "tilesLoading": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Tiles being fetched or decoded, of every tile source." - }, - "uploading": { - "type": "integer", - "format": "uint64", - "minimum": 0, - "description": "Bytes of data waiting to upload." - }, - "resident": { - "type": "integer", - "format": "uint64", - "minimum": 0, - "description": "Bytes of data pages on the GPU, for every view of its `Gpu`." - }, - "budget": { - "type": "integer", - "format": "uint64", - "minimum": 0, - "description": "Bytes the GPU's [`Budget`](crate::Budget) lets data take." - }, - "waiting": { - "$ref": "#/outputs/Waiting", - "description": "The work under way the last frame stood in for; nothing when it drew everything as the data,\nscene, and camera say: see [`View::complete`](crate::View::complete)." + "description": "Tiles being fetched or decoded, across all tile sources." }, "cpuMs": { "type": "number", @@ -4385,7 +4696,7 @@ "gpuMs": { "type": "number", "format": "float", - "description": "Milliseconds the GPU takes on a frame: the median of the last frames it measured; 0 where the\ndevice cannot measure." + "description": "Milliseconds the GPU takes per frame: the median of recently measured frames. 0 if the\ndevice cannot measure GPU time." } }, "required": [ @@ -4396,14 +4707,10 @@ "instances", "tiles", "tilesLoading", - "uploading", - "resident", - "budget", - "waiting", "cpuMs", "gpuMs" ], - "description": "What a view does each frame." + "description": "Counters describing a view's frames." }, "Work": { "type": "object", @@ -4414,11 +4721,17 @@ "minimum": 0, "description": "Bytes of data pages uploaded." }, + "uploads": { + "type": "integer", + "format": "uint64", + "minimum": 0, + "description": "Data pages, or pieces of pages, uploaded. A change's uploads count pages here and bytes in\n`uploaded`." + }, "written": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Bytes written besides pages: uniforms, programs, label runs, the selection, textures." + "description": "Bytes written other than pages: uniforms, programs, label runs, the selection, and textures." }, "evaluated": { "type": "integer", @@ -4426,48 +4739,70 @@ "minimum": 0, "description": "Rows whose channels the GPU evaluated." }, + "resolved": { + "type": "integer", + "format": "uint64", + "minimum": 0, + "description": "Edge and face rows whose endpoints the GPU resolved to resident vertex rows. These are rows\nthat changed, plus every row of a page that references a vertex now resident elsewhere." + }, + "cleared": { + "type": "integer", + "format": "uint64", + "minimum": 0, + "description": "Bytes of per-row word pages the GPU cleared: pages read before anything writes them, and the\nnormals that faces sum from scratch." + }, + "packed": { + "type": "integer", + "format": "uint64", + "minimum": 0, + "description": "Rows packed into pages of a paged table's draw order. Each page is packed once, the first\ntime a view draws it, and again only when a write changes its rows." + }, "readbacks": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Reads back from the GPU: probes, queries, fits, label placements, segment counts, pictures;\nnot the frame timer's, which only some devices have." + "description": "Readbacks from the GPU: probes, queries, fits, label texts, segment counts, and pictures.\nFrame timer readbacks, which only some devices support, are not counted." }, "jobs": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Jobs handed out: layouts, routes, lanes, triangulations, pyramids, tiles; to the host's\nworkers, or done at once without a host." + "description": "Jobs started: layouts, routes, lanes, triangulations, pyramids, and tiles. They run on the\nhost's workers, or immediately without a host." }, "texts": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Label texts the host was asked to set." + "description": "Label texts the host was asked to rasterize." }, "pipelines": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Pipelines compiled: in a browser, each as a view or a shot asks for it ahead, off the GPU's\nthread; natively, where it is first made." + "description": "Pipelines compiled. In a browser, each compiles in the background when a view or shot\nrequests it ahead of use. Natively, each compiles when first created." }, "late": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Pipelines made without being asked for ahead, each compiled where it is first used: in a\nbrowser on the GPU's thread, holding frames back. None, unless a scene uses what it did not\nask for." + "description": "Pipelines not requested ahead of use, compiled when first used. In a browser they compile\non the GPU's thread and delay frames. Zero unless a scene uses a pipeline it did not\nrequest." } }, "required": [ "uploaded", + "uploads", "written", "evaluated", + "resolved", + "cleared", + "packed", "readbacks", "jobs", "texts", "pipelines", "late" ], - "description": "What a [`Gpu`](crate::Gpu) did since it was made; subtract two for what happened between." + "description": "Counts of the work a [`Gpu`](crate::Gpu) has done since it was created. Subtract two\nsnapshots to get the work between them." }, "Description": { "type": "object", @@ -4486,7 +4821,7 @@ }, "minItems": 2, "maxItems": 2, - "description": "The axis's name and unit." + "description": "The axis name and unit, if set." }, "span": { "type": [ @@ -4499,14 +4834,14 @@ }, "minItems": 2, "maxItems": 2, - "description": "The first and last coordinates its sampled fields hold." + "description": "The first and last axis coordinates held by any sampled field, or `None` if no field holds\nany." }, "tables": { "type": "array", "items": { "$ref": "#/outputs/TableDescription" }, - "description": "Its tables, by name." + "description": "Its tables, sorted by name." } }, "required": [ @@ -4515,7 +4850,7 @@ "span", "tables" ], - "description": "What a dataset holds: its tables, their rows and fields, and the axis its sampled fields are\nrecorded over and span." + "description": "A summary of a dataset: its tables with their rows and fields, the axis sampled fields are\nrecorded over, and the part of the axis they cover." }, "TableDescription": { "type": "object", @@ -4528,14 +4863,14 @@ "type": "integer", "format": "uint", "minimum": 0, - "description": "How many rows it has." + "description": "The number of rows." }, "fields": { "type": "array", "items": { "$ref": "#/outputs/FieldDescription" }, - "description": "Its fields, by name." + "description": "Its fields, sorted by name." } }, "required": [ @@ -4543,7 +4878,7 @@ "rows", "fields" ], - "description": "A table described: its rows, and its fields by name." + "description": "A summary of one table: its row count and its fields, sorted by name." }, "FieldDescription": { "type": "object", @@ -4554,14 +4889,14 @@ }, "kind": { "$ref": "#/outputs/Kind", - "description": "What it holds." + "description": "The kind of values it holds." }, "to": { "type": [ "string", "null" ], - "description": "The table a reference names." + "description": "For a reference, the target table." }, "span": { "type": [ @@ -4574,14 +4909,14 @@ }, "minItems": 2, "maxItems": 2, - "description": "The first and last coordinates a sampled field holds." + "description": "For a sampled field, the first and last axis coordinates it holds." } }, "required": [ "name", "kind" ], - "description": "A field described: what it holds, the table a reference names, the coordinates samples span." + "description": "A summary of one field: its kind, plus its target table or axis span where those apply." }, "Kind": { "oneOf": [ @@ -4628,10 +4963,10 @@ { "type": "string", "const": "sampled", - "description": "Numbers sampled over the data's axis." + "description": "Numbers sampled over the dataset's axis." } ], - "description": "What a field holds." + "description": "The kind of values a field holds." }, "Budget": { "type": "object", @@ -4641,13 +4976,13 @@ "type": "integer", "format": "uint64", "minimum": 0, - "description": "Bytes the data every view of the `Gpu` draws may take on the GPU: its pages, and each view's\nrows. A view draws a table too large for its share through the tiles the camera asks for, so\nwhat it draws always fits; pages no view draws stay cached in the room left, the least\nrecently drawn going first. By default half the device's largest storage buffer. The pool of\npages grows as data needs it, and never shrinks: set the budget before data draws." + "description": "GPU memory in bytes for the data all views of the `Gpu` draw: its pages and each view's\nper-row state. A table too large for a view's share draws as a paged table, with only the\npages the camera needs, so it always fits. Pages no view draws stay cached in the rest of\nthe budget, and the least recently drawn are evicted first. Defaults to half the device's\nlargest storage buffer. The page pool grows as data needs it and never shrinks, so set the\nbudget before drawing data." }, "uploadPerFrame": { "type": "integer", "format": "uint64", "minimum": 0, - "description": "Bytes uploaded per frame at most; larger data streams in over frames." + "description": "The most bytes uploaded per frame. Larger data streams in over several frames. Defaults to\n32 MiB." } }, "required": [ diff --git a/npm/schema.mjs b/npm/schema.mjs index 06138c9c..6ca4a887 100644 --- a/npm/schema.mjs +++ b/npm/schema.mjs @@ -1,17 +1,19 @@ /** - * npm's types, written from `schema.json`: what latkit reads (`$defs`) becomes `inputs.ts`, and what - * it writes (`outputs`) becomes `outputs.ts`. Each definition reachable from the roots becomes a type, - * documented as its form is; an output an input already declares is that input's type. A schema's - * `tsType` names the hand-written type that stands for it, such as `ExprInput`. + * Generates npm's types from `schema.json`. The types latkit reads (`$defs`) become `inputs.ts`, + * and the types it writes (`outputs`) become `outputs.ts`. Each definition reachable from the roots + * becomes a type, documented with its schema description. A definition that extends shared options + * becomes an interface that extends them. An output that an input already declares uses the + * input's type. A schema's `tsType` names a hand-written type to use in its place, such as + * `ExprInput`. */ import fs from 'node:fs'; import path from 'node:path'; import prettier from 'prettier'; -/** The types npm takes as plain data; everything they are built of comes with them. */ -const INPUTS = ['Scene', 'Query', 'Camera', 'Fit', 'Budget', 'ColormapKind']; +/** Root types npm takes as plain data. The types they are built from are generated too. */ +const INPUTS = ['Scene', 'Query', 'EnvelopeRequest', 'Camera', 'Fit', 'Budget', 'ColormapKind']; -/** The types npm reads back. */ +/** Root types npm reads back from latkit. */ const OUTPUTS = [ 'ViewEvent', 'Stats', @@ -24,29 +26,32 @@ const OUTPUTS = [ 'Failure', ]; -/** The hand-written types a `tsType` names, and the module that declares them. */ -const IMPORTS = { './scene.js': ['Color', 'ExprInput'] }; +/** Hand-written types that a `tsType` names, keyed by the module that declares them. */ +const IMPORTS = { './data.js': ['TableInput'], './scene.js': ['Color', 'ExprInput'] }; -/** `inputs.ts` and `outputs.ts` of `schema.json` at `file`, formatted as the repository's sources are. */ +/** + * Returns the sources of `inputs.ts` and `outputs.ts` for the schema at `file`, formatted with the + * repository's Prettier config. + */ export async function typescript(file) { const schema = JSON.parse(fs.readFileSync(file, 'utf8')); const inputs = declare(schema.$defs, '#/$defs/', INPUTS, new Set()); const outputs = declare(schema.outputs, '#/outputs/', OUTPUTS, new Set(inputs.names)); const config = await prettier.resolveConfig(path.join(path.dirname(file), 'src/inputs.ts')); const format = (source) => prettier.format(source, { ...config, parser: 'typescript' }); - const header = (from) => - `/** Written by \`node npm/build.mjs schema\` from \`npm/schema.json\`, ${from}: do not edit. */`; + const header = (types) => + `/** ${types}, written by \`node npm/build.mjs schema\` from \`npm/schema.json\`. Do not edit. */`; const imports = (modules) => Object.entries(modules) .filter(([, names]) => names.length) .map(([module, names]) => `import type { ${names.sort().join(', ')} } from '${module}';`); return { inputs: await format( - [header('what latkit reads'), '', ...imports(IMPORTS), '', inputs.source, ''].join('\n'), + [header('The types latkit reads'), '', ...imports(IMPORTS), '', inputs.source, ''].join('\n'), ), outputs: await format( [ - header('what latkit writes'), + header('The types latkit writes'), '', ...imports({ './inputs.js': outputs.borrowed }), '', @@ -58,8 +63,9 @@ export async function typescript(file) { } /** - * The declarations of the definitions `defs` that `roots` reach, refs to them starting `prefix`; - * `declared` names those another module declares, which these borrow. + * Declares the definitions in `defs` that `roots` reach. Refs to them start with `prefix`. Names + * in `declared` belong to another module. They are not declared again, and the ones used are + * returned as `borrowed` for import. */ function declare(defs, prefix, roots, declared) { const output = prefix === '#/outputs/'; @@ -80,13 +86,14 @@ function declare(defs, prefix, roots, declared) { const type = (schema) => { if (schema === true || schema === undefined || Object.keys(schema).length === 0) return 'unknown'; - // An optional value that `null` also sets takes the null its schema allows. + // A `tsType` whose schema also allows null accepts null too. if (schema.tsType) return [schema.type].flat().includes('null') ? `${schema.tsType} | null` : schema.tsType; const name = ref(schema); if (name !== undefined) return defs[name]?.tsType ?? name; if ('const' in schema) return JSON.stringify(schema.const); if (schema.enum) return schema.enum.map((v) => JSON.stringify(v)).join(' | '); + if (schema.allOf) return schema.allOf.map(type).join(' & '); const union = schema.anyOf ?? schema.oneOf; if (union) return members(union); if (Array.isArray(schema.type)) { @@ -110,16 +117,21 @@ function declare(defs, prefix, roots, declared) { return 'unknown'; } }; - // A member's documentation moves onto the one key of a tagged variant; elsewhere TypeScript would - // show it nowhere. + // A union member's description moves onto the single key of a tagged variant, where editors show + // it. Any other member keeps its description as a doc comment before it, for the reference pages + // to read. That covers what each value of an enumeration means, and each form of a type with + // several. const members = (schemas) => { const parts = schemas.map((s) => { const keys = Object.keys(s.properties ?? {}); - if (!s.description || keys.length !== 1 || s.properties[keys[0]].description) return type(s); - return type({ - ...s, - properties: { [keys[0]]: { ...s.properties[keys[0]], description: s.description } }, - }); + if (!s.description) return type(s); + if (keys.length === 1 && !s.properties[keys[0]].description) { + return type({ + ...s, + properties: { [keys[0]]: { ...s.properties[keys[0]], description: s.description } }, + }); + } + return `${doc(s.description).trimEnd()} ${type({ ...s, description: undefined })}`; }); return [...new Set(parts)].join(' | '); }; @@ -140,7 +152,8 @@ function declare(defs, prefix, roots, declared) { } return /[|&]/.test(item) ? `(${item})[]` : `${item}[]`; }; - // What latkit writes leaves an absent value out rather than writing null. + // latkit leaves absent values out of its output and never writes null for them, so optional + // output fields drop their null. const absent = (schema) => { if (Array.isArray(schema.type)) return { ...schema, type: schema.type.filter((t) => t !== 'null') }; @@ -165,12 +178,24 @@ function declare(defs, prefix, roots, declared) { return `{ ${lines.join('\n')} }`; }; + // An object that extends shared options (`allOf` of references and its own object) becomes an + // interface that extends theirs. + const extending = (schema) => { + const parts = schema.allOf ?? []; + const own = parts.filter((p) => ref(p) === undefined); + return own.length === 1 && own[0].type === 'object' + ? { bases: parts.map(ref).filter((n) => n !== undefined), own: own[0] } + : undefined; + }; const names = [...reached].filter((name) => !declared.has(name) && !defs[name].tsType).sort(); const source = names .map((name) => { const schema = defs[name]; - const body = type({ ...schema, description: undefined, tsType: undefined }); const comment = schema.description ? doc(schema.description) : ''; + const extended = extending(schema); + if (extended) + return `${comment}export interface ${name} extends ${extended.bases.join(', ')} ${object(extended.own)}`; + const body = type({ ...schema, description: undefined, tsType: undefined }); return schema.type === 'object' && schema.properties ? `${comment}export interface ${name} ${body}` : `${comment}export type ${name} = ${body};`; @@ -181,10 +206,33 @@ function declare(defs, prefix, roots, declared) { return { names, source, borrowed: [...reached].filter((n) => declared.has(n) && used(n)) }; } +/** Rust names that npm spells differently. */ +const RENAMED = { 'View::events': 'view.on' }; + +/** + * Rewrites a code span from Rust's documentation with npm's names. An event becomes the event npm + * emits (`Event::Connect` becomes a `'connect'` event). A lowercase member becomes an instance + * member in camelCase (`View::complete` becomes `view.complete`), and any other member stays on its + * type (`Space::Diagram` becomes `Space.Diagram`). A call drops its parentheses (`hovered()` + * becomes `hovered`). + */ +function named(code) { + const camel = (name) => name.replace(/_([a-z])/g, (_, c) => c.toUpperCase()); + const lower = (name) => name[0].toLowerCase() + name.slice(1); + if (RENAMED[code]) return `\`${RENAMED[code]}\``; + const event = code.match(/^Event::(\w+)$/); + if (event) return `a \`'${lower(event[1])}'\` event`; + const span = code + .replace(/\b([A-Z]\w*)::([a-z]\w*)/g, (_, owner, member) => `${lower(owner)}.${camel(member)}`) + .replace(/\b([A-Z]\w*)::(\w+)/g, '$1.$2') + .replace(/\(\)/g, ''); + return `\`${span}\``; +} + /** - * Documentation as the forms write it, wrapped anew: paragraphs kept, lines at most 96 wide, Rust's - * links to its own items as plain code, its `None` as the `null` JSON writes, and its examples, - * which are Rust, left out. + * Converts a schema description into a doc comment. It keeps paragraphs, rewraps lines to at most + * 96 columns, turns Rust's intra-doc links into plain code spans, renames Rust names to npm's, and + * writes `None` as `null` (as JSON does). It drops code blocks, since they are Rust examples. */ function doc(text) { const lines = []; @@ -192,11 +240,12 @@ function doc(text) { .replace(/```[\s\S]*?```/g, '') .replace(/\*\//g, '*\\/') .replace(/\[(`[^`]*`)\]\([^)]*\)/g, '$1') - .replaceAll('`None`', '`null`'); + .replaceAll('`None`', '`null`') + .replace(/`([^`]*)`/g, (_, code) => named(code)); for (const paragraph of plain.split('\n\n').filter((p) => p.trim())) { if (lines.length) lines.push(''); let line = ''; - // A code span stays on one line, as TSDoc reads them. + // Each code span stays on one line, as TSDoc requires. const words = (paragraph.match(/(?:`[^`]*`|[^\s`])+/g) ?? []).map((w) => w.replace(/\s+/g, ' '), ); diff --git a/npm/src/canvas.ts b/npm/src/canvas.ts new file mode 100644 index 00000000..2a5b3129 --- /dev/null +++ b/npm/src/canvas.ts @@ -0,0 +1,99 @@ +/** Canvases. A canvas shows views in one `` or `OffscreenCanvas`. Views create them. */ + +import type * as bindings from '../pkg/latkit.js'; +import { Handle, inner } from './internal.js'; +import { handleOf, type View } from './view.js'; + +/** A region `[x, y, width, height]` in fractions of a canvas or picture, from its top left. */ +export type Rect = readonly [x: number, y: number, width: number, height: number]; + +/** @internal A `` element or an `OffscreenCanvas` that views draw in. */ +export type Target = HTMLCanvasElement | OffscreenCanvas; + +/** + * @internal Shows views in a `` element or an `OffscreenCanvas`, each in its region, with + * later views over earlier ones. A `` element sizes itself to its CSS box at the device + * pixel ratio and passes pointer, wheel, and key input to the view under the pointer. A frame + * redraws only the views that need it, or every view after a resize or a change of views. The + * canvas is destroyed with its last view, and the element keeps its last frame. + */ +export class Canvas extends Handle { + #views: View[] = []; + #released: () => void; + + private constructor( + device: bindings.Gpu, + target: Target, + pixelRatio: number | undefined, + released: () => void, + ) { + super(device.canvas(target, ...encoded([]), pixelRatio), 'Canvas'); + this.#released = released; + } + + /** Creates a canvas on `device` in `target`. `released` runs when the canvas is destroyed. */ + static made( + device: bindings.Gpu, + target: Target, + pixelRatio: number | undefined, + released: () => void, + ): Canvas { + return new Canvas(device, target, pixelRatio, released); + } + + /** The views shown, in drawing order. */ + get views(): readonly View[] { + return this.#views; + } + + /** The canvas's size in CSS pixels. */ + get cssSize(): [width: number, height: number] { + const [width = 1, height = 1] = inner(this).cssSize; + return [width, height]; + } + + /** Shows `view` over the others. */ + add(view: View): void { + this.#views.push(view); + this.relayout(); + } + + /** Stops showing `view`. Destroys the canvas when no view is left. */ + remove(view: View): void { + this.#views = this.#views.filter((shown) => shown !== view); + if (this.#views.length) this.relayout(); + else this.destroy(); + } + + /** Tells the bindings where each view draws, after a view or its region changed. */ + relayout(): void { + inner(this).setLayout(...encoded(this.#views)); + } + + protected override freeing(): void { + this.#views = []; + this.#released(); + } +} + +/** A view drawn into a picture, in its own region or in `region`. */ +export type Placed = + | View + | { + /** The view drawn. */ + readonly view: View; + /** Where it draws in the picture, overriding its own region. */ + readonly region: Rect; + }; + +/** + * @internal Encodes views for the bindings: their handles, and their regions as four fractions + * each in one array. + */ +export function encoded(views: readonly Placed[]): [bindings.Handle[], Float32Array] { + const placed = views.map((v) => ('view' in v ? v : { view: v, region: v.region })); + return [ + placed.map(({ view }) => handleOf(view)), + Float32Array.from(placed.flatMap(({ region }) => region)), + ]; +} diff --git a/npm/src/colormap.ts b/npm/src/colormap.ts new file mode 100644 index 00000000..1f7eebb9 --- /dev/null +++ b/npm/src/colormap.ts @@ -0,0 +1,21 @@ +/** The colormap catalog, and colormaps as CSS gradients for legends drawn in HTML. */ + +import type { Colormap, ColormapKind, ColormapName } from './inputs.js'; +import { wasm } from './internal.js'; + +/** + * Returns a CSS `linear-gradient(...)` with the colors the GPU draws for `map`, a catalog name or a + * custom colormap. `direction` is the CSS gradient direction and defaults to `'to right'`. A + * categorical map gives one solid band per category. + */ +export function colormapCss(map: Colormap, direction = 'to right'): string { + return wasm().colormapCss(JSON.stringify(map), direction); +} + +/** + * Returns the catalog's colormaps in order, each with its name, its label such as `'Red / Blue'`, + * and its kind. + */ +export function colormaps(): { name: ColormapName; label: string; kind: ColormapKind }[] { + return JSON.parse(wasm().colormaps()) as ReturnType; +} diff --git a/npm/src/core.ts b/npm/src/core.ts deleted file mode 100644 index a24e2721..00000000 --- a/npm/src/core.ts +++ /dev/null @@ -1,5 +0,0 @@ -/** Plain data and builders for Node, browsers, and workers; no DOM or WebAssembly types required. */ -export type * from './inputs.js'; -export type * from './outputs.js'; -export * from './patch.js'; -export * from './scene.js'; diff --git a/npm/src/data.ts b/npm/src/data.ts new file mode 100644 index 00000000..c7515975 --- /dev/null +++ b/npm/src/data.ts @@ -0,0 +1,304 @@ +/** Datasets: `Data` changes in place, a `Snapshot` is one version, and `Tiles` load in pieces. */ + +import type * as bindings from '../pkg/latkit.js'; +import type { EnvelopeRequest, Id, Query } from './inputs.js'; +import { bytesOf, Handle, inner, wasm, waited, type ByteSource, type Wait } from './internal.js'; +import { load } from './load.js'; +import type { Description, Value } from './outputs.js'; +import { framed, Patch, type Fields, type RowIds, type Run, type Values } from './patch.js'; + +/** + * A table in a dataset, such as `grid.table('Bus')`. Marks draw its rows and `field().from()` reads + * its fields. A view loads every dataset its scene uses. In JSON it is `[data, table]`. + */ +export class Table { + /** The dataset the table belongs to. */ + readonly source: Source; + /** The table's name. */ + readonly name: string; + + /** The table `name` in `source`. */ + constructor(source: Source, name: string) { + this.source = source; + this.name = name; + } + + /** Returns the JSON form, `[data, table]`. `JSON.stringify` calls it. */ + toJSON(): [data: string, table: string] { + return [this.source.name, this.name]; + } +} + +/** + * A table for a mark to draw or a field to read: a {@link Table}, or its JSON form `[data, table]`, + * where `data` names a dataset in the view's `sources`. + */ +export type TableInput = Table | readonly [data: string, table: string]; + +/** The rows a query returns, as columns in the forms a patch takes. */ +export interface QueryResult { + /** The table read. */ + table: string; + /** The ids of the rows returned, in order. */ + ids: Id[]; + /** + * Each selected field's column, keyed by output name. Numbers come as a `Float64Array` with NaN + * for missing values. Other values come as an array with `null` for missing ones. A reference + * comes as the referenced table and, for each row, the id of the row it references. + */ + fields: Record; + /** The number of rows that matched, before `offset` and `limit`. */ + total: number; +} + +/** The minimum and maximum of a sampled field for each row, over equal spans of the axis. */ +export interface Envelope { + /** The span boundaries on the axis: one more than the number of spans. */ + at: Float64Array; + /** Each row's minimum in each span, `spans` values per row, row after row. NaN for no value. */ + least: Float32Array; + /** Each row's maximum in each span, laid out like `least`. */ + greatest: Float32Array; +} + +/** A bindings object that reads one version of a dataset: a `Data` or a `Snapshot`. */ +type Reads = bindings.Data | bindings.Snapshot; + +/** + * Named tables of columns. A {@link Data} reads its current version and a {@link Snapshot} reads + * the version it holds, through the same methods. + */ +export abstract class Dataset extends Handle { + /** The dataset's name: the `data` in its tables' JSON form `[data, table]`. */ + get name(): string { + return inner(this).name; + } + + /** Returns table `name`, for a mark to draw or a field to read. */ + table(name: string): Table { + return new Table(this as Dataset as Data | Snapshot, name); + } + + /** Describes the tables and row counts, each field and its kind, and the axis and its span. */ + describe(): Description { + return JSON.parse(inner(this).describe()) as Description; + } + + /** + * Reads `field` of row `id` in `table`. A sampled field gives its last frame at or before + * `options.at`, or its last frame without `at`. A reference gives the referenced row's id. + * Returns `null` when the table, row, field, or value is missing. + */ + get( + table: string, + id: Id, + field: string, + options: { + /** The axis coordinate to read a sampled field at. */ + at?: number; + } = {}, + ): Value { + return JSON.parse(inner(this).get(table, id, field, options.at)) as Value; + } + + /** + * Reads a table's rows, filtered, sorted, and paged with the same expressions marks use. Sampled + * fields are read at `query.at`. + */ + query(query: Query): QueryResult { + return inner(this).query(JSON.stringify(query)) as QueryResult; + } + + /** + * Returns the minimum and maximum of a sampled field over equal spans of an axis range, for every + * row or for the rows in `request.ids`. Each span reads the coarsest stored levels of detail that + * fit inside it, so it costs a few reads however many frames it covers. Throws a `limit` error + * when rows times spans exceeds 2^26. + */ + envelope(request: EnvelopeRequest): Envelope { + return inner(this).envelope(JSON.stringify(request)) as Envelope; + } + + /** Returns the dataset as latkit's bytes: a patch that builds it from an empty dataset. */ + toBytes(): Uint8Array { + return inner(this).toBytes(); + } +} + +/** + * A dataset that changes in place. Each change applies all at once or not at all. Every + * {@link Patch} method is also a method here that applies that one change right away, and `apply` + * applies a patch of several. Views draw the latest version from the frame after each change. A + * {@link Snapshot} keeps one version and shares its pages with the dataset, so an undo history + * costs only the pages that differ. Destroying a `Data` leaves the views that draw it, and its + * snapshots, usable. + * + * ```ts + * const grid = new Data('grid').set('Bus', [1, 2], { kv: [230, 115] }); + * grid.set('Bus', [2], { kv: [69] }); // drawn the next frame + * ``` + */ +export class Data extends Dataset { + /** Creates an empty dataset named `name`. */ + constructor(name: string); + /** @internal Wraps the bindings dataset `bound`. */ + constructor(bound: bindings.Data); + constructor(from: string | bindings.Data) { + super(typeof from === 'string' ? new (wasm().Data)(from) : from, 'Data'); + } + + /** + * Reads a dataset from `source`. Its first bytes decide the format: + * + * - latkit's bytes, as `toBytes()` writes them. + * - GeoJSON, plain or gzip-compressed, read in a worker. Points, line strings, and polygons + * become rows of tables `points`, `lines`, and `polygons`, with the geometry in field + * `geometry` and each property as a field. + * - A dataset document, `{"name": ..., "patch": [...]}`. Its patch is read like + * {@link Patch.read}. + */ + static async read( + source: ByteSource | object, + options: { + /** The dataset's name, overriding the name in its bytes or document. Required for GeoJSON. */ + name?: string; + /** The URL that a document's file slices are relative to. Defaults to the document's URL. */ + base?: string; + } & Wait = {}, + ): Promise { + await load(); + const { bytes, base } = await bytesOf(source, options); + const read = wasm().Data.read(bytes, options.name, options.base ?? base); + return new Data(await waited(read, options, 'reading a dataset')); + } + + /** + * Applies every change in `patch`, or none when one is invalid, and returns this dataset. An + * `invalid` error's `path` names the failing change, such as `patch[1].Bus.kv`. + */ + apply(patch: Patch | Uint8Array): this { + inner(this).apply(patch instanceof Uint8Array ? patch : framed(patch)); + return this; + } + + /** Applies {@link Patch.axis}: names the axis that sampled fields are recorded over. */ + axis(name: string, unit: string): this { + return this.apply(new Patch().axis(name, unit)); + } + + /** + * Applies {@link Patch.replace}: replaces table `table` with `fields` for rows `ids` (or `0` to + * `n - 1`). + */ + replace(table: string, fields: Fields): this; + replace(table: string, ids: RowIds, fields?: Fields): this; + replace(table: string, ids: RowIds | Fields, fields?: Fields): this { + // `Patch.replace` tells ids from fields itself. + return this.apply(new Patch().replace(table, ids as RowIds, fields)); + } + + /** + * Applies {@link Patch.set}: writes `fields` for rows `ids` (or `0` to `n - 1`) of `table`, + * adding new rows. + */ + set(table: string, fields: Fields): this; + set(table: string, ids: RowIds, fields?: Fields): this; + set(table: string, ids: RowIds | Fields, fields?: Fields): this { + return this.apply(new Patch().set(table, ids as RowIds, fields)); + } + + /** Applies {@link Patch.remove}: removes the rows of `table` with `ids`. */ + remove(table: string, ids: RowIds): this { + return this.apply(new Patch().remove(table, ids)); + } + + /** Applies {@link Patch.append}: records frames of `table` at the axis coordinates `at`. */ + append( + table: string, + at: number | readonly number[], + fields: Readonly>, + options?: { + /** The rows the frames cover. Defaults to every row. Other rows keep their last values. */ + ids?: RowIds; + }, + ): this { + return this.apply(new Patch().append(table, at, fields, options)); + } + + /** Applies {@link Patch.forget}: drops every sampled field's frames before `before`. */ + forget(before: number): this { + return this.apply(new Patch().forget(before)); + } + + /** Applies {@link Patch.frames}: stores a run of a sampled field's entries, as in a time tile. */ + frames(table: string, field: string, run: Run): this { + return this.apply(new Patch().frames(table, field, run)); + } + + /** Returns the current version as a {@link Snapshot}. Later patches don't change it. O(1). */ + snapshot(): Snapshot { + return Snapshot.made(inner(this).snapshot()); + } + + /** Makes `snapshot`, a version of this dataset, the current version. This is how to undo. */ + restore(snapshot: Snapshot): this { + inner(this).restore(inner(snapshot)); + return this; + } +} + +/** + * One immutable version of a dataset, taken in O(1) by {@link Data.snapshot}. A view given a + * snapshot draws that version, and keeps drawing it after the snapshot is destroyed. + */ +export class Snapshot extends Dataset { + private constructor(bound: bindings.Snapshot) { + super(bound, 'Snapshot'); + } + + /** @internal Wraps the bindings snapshot `bound`. */ + static made(bound: bindings.Snapshot): Snapshot { + return new Snapshot(bound); + } +} + +/** + * A dataset fetched in pieces as views need them. It is plain data: a dataset name and a URL. Each + * view fetches the pieces its camera and plot window need, and views on one GPU share them. The + * first bytes at `url` decide the kind: + * + * - Time tiles: a `frames.json` manifest written by `latkit tile`. A line that traces a sampled + * field fetches the detail its plot window shows. Other marks fetch the frames around the view's + * `at`. While a span loads, the coarser span that contains it shows. + * - Vector tiles: a PMTiles archive, a TileJSON document, or a `{z}/{x}/{y}` URL template. Each + * layer becomes a table of the features its marks draw: polygons for areas, lines for lines, + * points for vertices. A table named `layer:polygons`, `layer:lines`, or `layer:points` holds + * that kind regardless of the marks. Each tile that arrives appends its rows to the view's + * dataset. Rows of tiles no longer drawn are kept until they outnumber the rows drawn. + */ +export class Tiles { + /** The name of the dataset the tiles are drawn as. */ + readonly name: string; + /** The tiles' URL. */ + readonly url: string; + + /** Tiles at `url`, drawn as the dataset named `name`. */ + constructor(name: string, url: string) { + this.name = name; + this.url = url; + } + + /** Returns table `name`, such as a vector tile layer, for a mark to draw. */ + table(name: string): Table { + return new Table(this, name); + } +} + +/** A dataset a view draws: the one a {@link Table} belongs to, or one of the view's `sources`. */ +export type Source = Data | Snapshot | Tiles; + +/** @internal Converts `source` to a new bindings `Source` on every call. */ +export function sourceOf(source: Source): bindings.Source { + if (source instanceof Tiles) return wasm().Source.tiles(source.name, source.url); + return inner(source).source(); +} diff --git a/npm/src/encoder.ts b/npm/src/encoder.ts index 5b295d72..c3ef8a9d 100644 --- a/npm/src/encoder.ts +++ b/npm/src/encoder.ts @@ -1,12 +1,15 @@ /** - * The browser's H.264 encoder, as latkit's video asks for one: NV12 pictures (BT.709, limited range) - * in, Annex B units out, each `[time, key, bytes]` in order. WebCodecs encodes, on the GPU's video - * engine where the device has one; the level is the least that holds the size and frame rate. + * The H.264 encoder for latkit's videos in a browser, built on WebCodecs. It takes NV12 pictures + * (BT.709, limited range) and returns Annex B units in order, each `[time, key, bytes]`, with + * `time` in 90 kHz ticks. It prefers hardware encoding and uses the lowest H.264 level that fits + * the size and frame rate. */ +import { thrown, type LatkitError } from './internal.js'; + type Unit = [time: number, key: boolean, bytes: Uint8Array]; -/** H.264 levels by the macroblocks a second and a frame they hold, as their codec byte. */ +/** H.264 levels as `[codec byte, max macroblocks per second, max macroblocks per frame]`. */ const LEVELS: readonly [level: number, perSecond: number, perFrame: number][] = [ [0x1f, 108_000, 3_600], [0x28, 245_760, 8_192], @@ -19,10 +22,16 @@ const LEVELS: readonly [level: number, perSecond: number, perFrame: number][] = [0x3e, 16_711_680, 139_264], ]; -/** Constrained Baseline forbids B pictures; the MP4 writer uses equal decode and presentation times. */ +/** + * Constrained Baseline only. It has no B-frames, so decode and presentation times are equal, as + * the MP4 writer assumes. + */ const PROFILES = ['42e0']; -/** The codec configurations to try for `width`×`height` at `fps`, best first. */ +/** + * The encoder configurations to try for `width`×`height` at `fps`, hardware first. The level is + * the lowest that fits, or 6.2 when none does. + */ function configurations( width: number, height: number, @@ -49,13 +58,18 @@ function configurations( ); } -/** An encoder for `width`×`height` pictures at `fps` and `bitrate` bits a second. */ +/** + * Creates an encoder for `width`×`height` pictures at `fps` frames per second and `bitrate` bits + * per second. Throws an `unsupported` error when the browser has no `VideoEncoder`. Its methods + * reject with an `unsupported` error when the browser cannot encode H.264 at that size and rate, + * and with an `io` error when encoding fails. + */ export function encoder(width: number, height: number, fps: number, bitrate: number) { if (typeof VideoEncoder !== 'function') - throw new Error('this browser has no WebCodecs VideoEncoder'); + throw thrown({ code: 'unsupported', message: 'this browser has no WebCodecs VideoEncoder' }); const units: Unit[] = []; const times: { timestamp: number; time: number }[] = []; - let failure: unknown; + let failure: LatkitError | undefined; const waiters = new Set<() => void>(); const wake = () => { for (const resolve of waiters) resolve(); @@ -65,7 +79,7 @@ export function encoder(width: number, height: number, fps: number, bitrate: num output(chunk) { const expected = times.shift(); if (!expected || chunk.timestamp !== expected.timestamp) { - failure = new Error('the encoder reordered or omitted a frame'); + failure = thrown({ code: 'io', message: 'the encoder reordered or omitted a frame' }); close(); return; } @@ -74,7 +88,7 @@ export function encoder(width: number, height: number, fps: number, bitrate: num units.push([expected.time, chunk.type === 'key', bytes]); }, error(error) { - failure = error; + failure = thrown({ code: 'io', message: 'the video encoder failed', cause: error }); wake(); }, }); @@ -89,13 +103,14 @@ export function encoder(width: number, height: number, fps: number, bitrate: num for (const config of configurations(width, height, fps, bitrate)) { if ((await VideoEncoder.isConfigSupported(config)).supported) return video.configure(config); } - throw new Error(`this browser encodes no H.264 at ${width}×${height}, ${fps} frames a second`); + throw thrown({ + code: 'unsupported', + message: `this browser cannot encode H.264 at ${width}×${height}, ${fps} frames a second`, + }); }; const failed = () => { - if (failure || video.state === 'closed') - throw failure instanceof Error - ? failure - : new Error('the video encoder failed', { cause: failure }); + if (failure) throw failure; + if (video.state === 'closed') throw thrown({ code: 'io', message: 'the video encoder closed' }); }; return { async encode(nv12: Uint8Array, time: number, key: boolean): Promise { @@ -115,7 +130,7 @@ export function encoder(width: number, height: number, fps: number, bitrate: num } finally { frame.close(); } - // The encoder sets the pace: a frame waits while two others are still being encoded. + // Backpressure: wait while more than two frames are queued in the encoder. while (video.encodeQueueSize > 2) { await new Promise((resolve) => waiters.add(resolve)); failed(); @@ -129,7 +144,7 @@ export function encoder(width: number, height: number, fps: number, bitrate: num failed(); await video.flush(); failed(); - if (times.length) throw new Error('the encoder omitted a frame'); + if (times.length) throw thrown({ code: 'io', message: 'the encoder omitted a frame' }); return units.splice(0); } finally { close(); diff --git a/npm/src/gpu.ts b/npm/src/gpu.ts new file mode 100644 index 00000000..671aeadc --- /dev/null +++ b/npm/src/gpu.ts @@ -0,0 +1,159 @@ +/** The GPU device. It draws views on their canvases, holds their shared memory, and runs frames. */ + +import type * as bindings from '../pkg/latkit.js'; +import { Canvas, type Target } from './canvas.js'; +import type { Budget } from './inputs.js'; +import { Handle, inner, wasm, waited, type Wait } from './internal.js'; +import { firstDevice, load } from './load.js'; +import type { Work } from './outputs.js'; +import type { View } from './view.js'; + +/** + * A WebGPU device that views draw with. Views on one `Gpu` share its memory and draw together in + * each animation frame, so linked views always show the same moment. Views use the first `Gpu` + * created unless given another, and `View.create` creates one if none exists. Destroying a `Gpu` + * destroys its views and frame listeners. Other GPUs and datasets stay usable. + */ +export class Gpu extends Handle { + /** The default GPU for views, and the pending request while it is being created. */ + static #shared: Gpu | undefined; + static #creating: Promise | undefined; + + #views = new Set(); + #canvases = new Map(); + #listeners = new Set<() => void>(); + + private constructor(device: bindings.Gpu) { + super(device, 'Gpu'); + } + + /** + * Requests a WebGPU device. The first call takes the device `load()` requested. Create one before + * any view to set its `budget` first, or create another for views that should not share memory + * with the rest. Rejects with an `unsupported` error when the browser has no WebGPU device. + */ + static async create(): Promise { + await load(); + const gpu = new Gpu(await (firstDevice() ?? wasm().Gpu.create())); + if (!Gpu.#shared || Gpu.#shared.lost) Gpu.#shared = gpu; + return gpu; + } + + /** @internal Returns the default GPU for views, creating it if there is none. */ + static shared(): Promise { + const shared = Gpu.#shared; + if (shared && !shared.lost) return Promise.resolve(shared); + return (Gpu.#creating ??= Gpu.create().finally(() => (Gpu.#creating = undefined))); + } + + /** + * Calls `listener` every animation frame with the frame's timestamp in milliseconds. It runs + * before the views draw, so its changes show in that frame. While any listener is registered, a + * frame runs on every animation frame. Returns a function that removes the listener. + */ + on(type: 'frame', listener: (now: number) => void): () => void { + const handle = device(this).onFrame(listener); + const stop = (): void => { + if (this.#listeners.delete(stop)) handle.free(); + }; + this.#listeners.add(stop); + return stop; + } + + /** Whether the device was lost. Drawing then does nothing, so create a new `Gpu`. */ + get lost(): boolean { + return device(this).lost; + } + + /** + * The GPU memory budget for data, shared by every view on this GPU. See {@link Budget}. Defaults + * to half the device's largest storage buffer, with 32 MiB uploaded per frame. + */ + get budget(): Budget { + return JSON.parse(device(this).budget) as Budget; + } + + set budget(budget: Budget) { + device(this).setBudget(JSON.stringify(budget)); + } + + /** + * Exact counts of the work this GPU has done since it was created. See {@link Work}. The same + * budget, device limits, and frames give the same counts, so tests and benchmarks can assert + * them. + */ + get work(): Work { + return JSON.parse(device(this).work) as Work; + } + + /** Bytes of GPU memory the views draw from: data pages and each view's per-row data. */ + get resident(): number { + return device(this).resident; + } + + /** Bytes waiting to upload to the GPU, across all its views. */ + get uploading(): number { + return device(this).uploading; + } + + /** + * Draws one frame now, at `now` milliseconds on the `requestAnimationFrame` clock. It runs the + * frame listeners, then draws every canvas that needs a frame. After the first call the GPU + * stops requesting animation frames, so a test or benchmark that steps every frame draws the + * same frames on any device. + */ + step(now: number): void { + device(this).step(now); + } + + /** + * Resolves once the GPU has finished its submitted work, every readback has arrived, and every + * pipeline a view or shot requested has compiled. A frame stepped after that uses all of them. + */ + async done(wait: Wait = {}): Promise { + await waited(device(this).done(), wait, 'waiting for the GPU'); + } + + /** + * Resolves when work a view waits on arrives after this call (a readback, a pipeline compile, a + * job, a label text, or a fetch), or when a view requests a frame. A harness that steps frames + * awaits it to know when to step again. + */ + async landed(wait: Wait = {}): Promise { + await waited(device(this).landed(), wait, 'waiting for work'); + } + + /** + * @internal Keeps `view` until it or the GPU is destroyed, and shows it on `target`, over the + * views already there. Returns the canvas showing it. + */ + adopt(view: View, target: Target | null, pixelRatio?: number): Canvas | undefined { + this.#views.add(view); + if (!target) return undefined; + let canvas = this.#canvases.get(target); + if (!canvas) { + canvas = Canvas.made(device(this), target, pixelRatio, () => this.#canvases.delete(target)); + this.#canvases.set(target, canvas); + } + canvas.add(view); + return canvas; + } + + /** @internal Forgets `view`, once it is destroyed. */ + release(view: View): void { + this.#views.delete(view); + } + + /** @internal Destroys the GPU's views, canvases, and frame listeners before its device. */ + protected override freeing(): void { + if (Gpu.#shared === this) Gpu.#shared = undefined; + for (const view of this.#views) view.destroy(); + for (const canvas of this.#canvases.values()) canvas.destroy(); + for (const stop of this.#listeners) stop(); + } +} + +/** @internal Returns `owner`'s bindings device. */ +export function device(owner: Gpu): bindings.Gpu { + return inner(owner); +} diff --git a/npm/src/index.ts b/npm/src/index.ts index fe3683d9..f41a45f3 100644 --- a/npm/src/index.ts +++ b/npm/src/index.ts @@ -1,24 +1,47 @@ +/// +/// /** - * Scenes, expressions, and patches are plain data, usable without WebAssembly. Call loadLatkit() to - * read data and draw. + * latkit: columnar data drawn with WebGPU. + * + * Scenes, expressions, and patches are plain data and need no WebAssembly. Datasets, views, + * pictures, and the colormap functions do, so call `await load()` first. Every `Handle` works with + * `using`. The references above let the browser types latkit uses (canvases, `Blob`, `Response`, + * `WebAssembly.Module`) and `Symbol.dispose` typecheck under any `lib` setting, in Node.js too. */ export type * from './inputs.js'; export type * from './outputs.js'; -export * from './patch.js'; -export * from './scene.js'; -export { loadLatkit, type LoadOptions, type Latkit } from './load.js'; -export type { - Canvas, +export { load } from './load.js'; +export { + Handle, + type ByteSource, + type LatkitError, + type ReadableBytes, + type Wait, +} from './internal.js'; +export { Data, - Envelope, - Gpu, - Image, - ImageFormat, - LatkitError, - Panel, - Shot, + Dataset, + Snapshot, Table, - Video, - View, - ViewEvents, -} from './runtime.js'; + Tiles, + type Envelope, + type QueryResult, + type Source, + type TableInput, +} from './data.js'; +export { + Patch, + reference, + type Fields, + type Geometry, + type Reference, + type RowIds, + type Run, + type Values, +} from './patch.js'; +export * from './scene.js'; +export { colormapCss, colormaps } from './colormap.js'; +export { Gpu } from './gpu.js'; +export { View, type Insets, type Point, type Position, type ViewEvents } from './view.js'; +export type { Placed, Rect } from './canvas.js'; +export { Image, Shot, Video, type ImageFormat, type ShotOptions } from './shot.js'; diff --git a/npm/src/inputs.ts b/npm/src/inputs.ts index 32847d5c..30a99d23 100644 --- a/npm/src/inputs.ts +++ b/npm/src/inputs.ts @@ -1,99 +1,53 @@ -/** Written by `node npm/build.mjs schema` from `npm/schema.json`, what latkit reads: do not edit. */ +/** The types latkit reads, written by `node npm/build.mjs schema` from `npm/schema.json`. Do not edit. */ +import type { TableInput } from './data.js'; import type { Color, ExprInput } from './scene.js'; +/** Defaults for areas. */ +export interface AreaTheme { + /** The area color. */ + color?: Color; +} + /** - * Rows drawn as the areas a polygon field holds: parcels, states, lakes, the land and water of a - * map. The view cuts each row's polygons into triangles once, off the frame when they are large; - * on the earth they bend with the globe and stay whole across the antimeridian. Outline areas with - * a `Lines` mark of the same table, which draws their rings. + * Rows drawn as the areas of a polygon field: parcels, states, lakes, the land and water of a map. + * The view triangulates each row's polygons once, in the background when they are large. On the + * earth, areas follow the globe's curve and stay unbroken across the antimeridian. To outline + * areas, add a `Lines` mark of the same table. It draws their rings. */ -export interface Areas { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** The polygon field each row fills: one polygon's rings, or polygons. `geometry` unless set. */ - geometry?: string | null; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; +export interface Areas extends MarkOptions { + /** The dataset and table drawn, `[data, table]`. */ + areas: TableInput; /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). + * The polygon field each row fills. It holds one polygon's rings or several polygons. Defaults to + * the field named `geometry`. */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; + geometry?: string | null; } /** A plot axis. */ export interface Axis { /** The axis title. */ label?: string | null; - /** Whether grid lines cross the plot at the ticks. */ + /** Whether grid lines cross the plot at the ticks. Defaults to `true`. */ grid?: boolean; - /** Whether the axis is hidden. */ + /** Whether the axis is hidden. Defaults to `false`. */ hidden?: boolean; /** - * Ticks at these values, each a value or `[value, label]`, their values written unless labeled; - * round values unless set. + * Tick positions, each a value or `[value, label]`. A tick without a label shows its value. When + * unset, ticks fall at round values. */ ticks?: Tick[] | null; - /** How tick values are written. */ + /** How tick values are formatted. Defaults to `auto`. */ format?: Format; /** - * Decimals of fixed values, or of the significand of scientific and engineering ones; as many as - * the ticks' step needs unless set. + * Decimal places of fixed values, or of the significand in scientific and engineering formats. At + * most 15. When unset, as many as the tick step needs. */ precision?: number | null; } -/** An operation of two values. */ -export type Binary = - | 'add' - | 'sub' - | 'mul' - | 'div' - | 'rem' - | 'pow' - | 'min' - | 'max' - | 'lt' - | 'le' - | 'gt' - | 'ge' - | 'eq' - | 'ne' - | 'and' - | 'or'; - -export interface BinaryNode { - op: Binary; - args: [ExprInput, ExprInput]; -} - -/** RGBA texels, straight alpha, sRGB, rows top to bottom. */ +/** An image of RGBA texels: straight alpha, sRGB, rows top to bottom. */ export interface Bitmap { /** Width and height in texels. */ size: [number, number]; @@ -104,51 +58,181 @@ export interface Bitmap { /** How much GPU memory latkit uses. */ export interface Budget { /** - * Bytes the data every view of the `Gpu` draws may take on the GPU: its pages, and each view's - * rows. A view draws a table too large for its share through the tiles the camera asks for, so - * what it draws always fits; pages no view draws stay cached in the room left, the least recently - * drawn going first. By default half the device's largest storage buffer. The pool of pages grows - * as data needs it, and never shrinks: set the budget before data draws. + * GPU memory in bytes for the data all views of the `Gpu` draw: its pages and each view's per-row + * state. A table too large for a view's share draws as a paged table, with only the pages the + * camera needs, so it always fits. Pages no view draws stay cached in the rest of the budget, and + * the least recently drawn are evicted first. Defaults to half the device's largest storage + * buffer. The page pool grows as data needs it and never shrinks, so set the budget before drawing + * data. */ bytes: number; - /** Bytes uploaded per frame at most; larger data streams in over frames. */ + /** + * The most bytes uploaded per frame. Larger data streams in over several frames. Defaults to 32 + * MiB. + */ uploadPerFrame: number; } +/** + * An operation, written as an object with one key: the operation's name, holding its arguments. + * For example `{ "add": ["$a", 1] }`. The second argument of `is` and `contains` is plain text, + * not a field or a color. + */ +export type CallNode = + | { + /** The value negated. */ + neg: [ExprInput]; + } + | { + /** 1 where the value is at most 0.5, else 0. */ + not: [ExprInput]; + } + | { + /** The absolute value. */ + abs: [ExprInput]; + } + | { + /** The square root. Negative values give a missing value. */ + sqrt: [ExprInput]; + } + | { + /** The sine of the value, an angle in radians. */ + sin: [ExprInput]; + } + | { + /** The cosine of the value, an angle in radians. */ + cos: [ExprInput]; + } + | { + /** The largest whole number at most the value. */ + floor: [ExprInput]; + } + | { + /** The natural logarithm. Values at most 0 give a missing value. */ + ln: [ExprInput]; + } + | { + /** e raised to the value. */ + exp: [ExprInput]; + } + | { + /** The sum of the two values. */ + add: [ExprInput, ExprInput]; + } + | { + /** The first value minus the second. */ + sub: [ExprInput, ExprInput]; + } + | { + /** The product of the two values. */ + mul: [ExprInput, ExprInput]; + } + | { + /** The first value divided by the second. Dividing by 0 gives a missing value. */ + div: [ExprInput, ExprInput]; + } + | { + /** + * The remainder of the first value divided by the second, with the sign of the second: -1 and 3 + * give 2. + */ + rem: [ExprInput, ExprInput]; + } + | { + /** The first value raised to the power of the second. */ + pow: [ExprInput, ExprInput]; + } + | { + /** The smaller of the two values. */ + min: [ExprInput, ExprInput]; + } + | { + /** The larger of the two values. */ + max: [ExprInput, ExprInput]; + } + | { + /** 1 where the first value is less than the second, else 0. */ + lt: [ExprInput, ExprInput]; + } + | { + /** 1 where the first value is at most the second, else 0. */ + le: [ExprInput, ExprInput]; + } + | { + /** 1 where the first value is greater than the second, else 0. */ + gt: [ExprInput, ExprInput]; + } + | { + /** 1 where the first value is at least the second, else 0. */ + ge: [ExprInput, ExprInput]; + } + | { + /** 1 where the two values are equal, else 0. Colors compare too. */ + eq: [ExprInput, ExprInput]; + } + | { + /** 1 where the two values differ, else 0. Colors compare too. */ + ne: [ExprInput, ExprInput]; + } + | { + /** 1 where both values are above 0.5, else 0. */ + and: [ExprInput, ExprInput]; + } + | { + /** 1 where either value is above 0.5, else 0. */ + or: [ExprInput, ExprInput]; + } + | { + /** Whether a text field equals the plain text exactly. */ + is: [field: ExprInput, text: string]; + } + | { + /** + * Whether a text field contains the plain text, ignoring case. In a mark, the field must belong to + * the mark's own table. + */ + contains: [field: ExprInput, text: string]; + }; + /** Where a view looks. */ export type Camera = - | { + /** + * A camera over the earth, a plane, or a diagram: the point it looks at and turns about, its zoom, + * its tilt, and its rotation. A diagram is always seen from straight above, upright. + */ | { /** - * Longitude, latitude, and elevation on the earth; x, y, and elevation on a plane or a diagram. In - * JSON, `[x, y]` looks at the ground. + * On the earth, longitude, latitude, and elevation. On a plane or a diagram, x, y, and elevation. + * In JSON, `[x, y]` is shorthand for elevation 0. */ center: [number, number] | [number, number, number]; - /** On the earth the web mercator level; elsewhere, log₂ of pixels per unit. */ + /** The web mercator zoom level on the earth. Elsewhere, log₂ of CSS pixels per unit. */ zoom: number; - /** Degrees from looking straight down, up to 75; 0 in a diagram. */ + /** Tilt in degrees from looking straight down, up to 75. Ignored in a diagram. */ pitch: number; - /** Degrees clockwise from north, or from up on a plane; 0 in a diagram. */ + /** Degrees clockwise from north, or from up on a plane. Ignored in a diagram. */ bearing: number; } - | { - /** The axis range shown. */ + | /** A plot camera: the x range shown, and the value range, or `null` to fit the values in view. */ { + /** The x-axis range shown. */ x: [number, number]; - /** The value range shown, or none to fit what is in view. */ + /** The value range shown, or `null` to fit the values in view. */ y?: [number, number] | null; }; -export interface ColorNode { - /** A CSS color, such as `#ffb000` or `rebeccapurple`. */ - color: Color; -} - /** Colors for a range of values: a catalog name, or your own colors. */ export type Colormap = ColormapName | CustomColormap; -/** How a colormap spans values. */ -export type ColormapKind = 'sequential' | 'diverging' | 'cyclic' | 'categorical'; +/** + * How a colormap maps values: `sequential` and `diverging` maps clamp, `cyclic` maps wrap, and + * `categorical` maps give each integer its own color. + */ +export type ColormapKind = + | /** Low to high. */ 'sequential' + | /** Below and above a midpoint. */ 'diverging' + | /** Around a circle: values wrap. */ 'cyclic' + | /** Distinct colors for distinct integers. */ 'categorical'; -/** A colormap of the catalog. */ +/** A colormap from the catalog. */ export type ColormapName = | 'amber' | 'amp' @@ -197,271 +281,312 @@ export type ColormapName = | 'tableau10' | 'turbo'; -/** `value` through `colormap`: from its `domain`, else from the extent of the values shown. */ +/** A value mapped through a colormap. */ export interface ColormapNode { + /** The colormap. */ colormap: Colormap; + /** The value mapped. A categorical map takes each value's integer part as its category. */ value: ExprInput; + /** + * The values mapped to the colormap's ends. Defaults to the extent of the values shown. A + * categorical map ignores it. + */ domain?: [low: number, high: number] | null; } -/** Whether a text field holds the text, ignoring case: read by queries, which hold texts. */ -export interface ContainsNode { - contains: [field: ExprInput, text: string]; -} - /** - * How a view answers the pointer and keyboard, and how much it moves on its own: a preset, or - * options over the defaults. Each mark says whether its rows pick, select, and drag; these say - * what the view itself does. + * How a view responds to the pointer and keyboard, and how much it animates on its own: a preset + * (`"none"` or `"inspect"`), or options that change the defaults. Each mark sets whether its own + * rows pick, select, and drag. These options control the view itself. */ export type Controls = | 'none' | 'inspect' | { - /** Dragging the background moves the camera. */ + /** Whether dragging the background pans the camera. Defaults to `true`. */ pan?: boolean; /** - * What zooms: the wheel and pinching, or the wheel only with Control or Meta held (as a trackpad's - * pinch sends it), leaving the plain wheel to scroll the page; or nothing. + * Which input zooms the view. `wheel` zooms with the wheel and pinching. `modifier` zooms with + * pinching, and with the wheel only while Control or Meta is held (as trackpad pinches send it), + * so the plain wheel scrolls the page. `off` turns zooming off. Defaults to `wheel`. */ zoom?: Zoom; - /** Dragging with the secondary button, or with Shift, turns and tilts a map or plane. */ + /** + * Whether dragging with the secondary button, or with Shift, rotates and tilts a map or plane. + * Defaults to `true`. + */ turn?: boolean; /** - * Tab steps through the rows nearest the view's middle and arrows to the nearest row their way; - * Enter opens the row stepped to and Space selects it. Otherwise arrows pan, `+` and `-` zoom, - * Home fits, and Escape clears the selection. + * Whether the keyboard controls the view. Defaults to `true`. Tab and Shift+Tab step through the + * rows nearest the view's center. Arrow keys step to the nearest row in their direction. Enter + * opens the row stepped to, and Space selects it. With no row stepped to, arrow keys pan + * (Shift+arrows turn), `+` and `-` zoom, Home fits everything, and Escape clears the selection. */ keyboard?: boolean; /** - * The row the user points at glows. It is hovered either way: events tell it, and `hovered()` - * reads it. + * Whether the hovered row glows and a hovered vertex sized in CSS pixels grows. Defaults to + * `true`. Hover is tracked either way: a `'hovered'` event still fires and `hovered` still returns + * the row. */ hover?: boolean; - /** CSS pixels around the pointer that picks reach; touch reaches 8 more. */ + /** + * How far a row can be from the pointer and still be picked, in CSS pixels. Touch adds 8. Defaults + * to 6. + */ pickRadius?: number; /** - * Dragging with Shift selects the rows in a box, instead of turning; a Shift click still toggles - * its row. + * Whether Shift+drag selects the rows inside a box instead of turning the camera. Shift+click + * still toggles one row. Defaults to `false`. */ selectBox?: boolean; /** - * Dragging from a port, from a vertex with Alt, or an edge off its end draws a connection, told as - * `Event::Connect`. + * Whether the user can draw connections by dragging from a port, from a vertex with Alt held, or + * an edge's end off its vertex. Connections are reported as a `'connect'` event. Defaults to + * `false`. */ connect?: boolean; - /** Delete and Backspace tell `Event::Delete` with the selection. */ + /** Whether Delete and Backspace report the selection as a `'delete'` event. Defaults to `false`. */ delete?: boolean; /** - * Clicking again where nothing moved selects the next row under the point, nearest first, so rows - * drawn over one another can each be selected. + * Whether clicking again at the same point, with the camera unmoved, selects the next row under + * the pointer, nearest first. This lets overlapping rows each be selected. Defaults to `false`. */ cycle?: boolean; /** - * Seconds a touch or a pen held still asks for a menu, as the secondary button does; zero never - * does. + * How many seconds a touch or pen must hold still to request a menu (a `'menu'` event), like the + * secondary button. 0 turns this off. Defaults to 0.5. */ press?: number; /** - * How much the view moves on its own: eases, flights, coasting, orbits, label fades, comets, and - * what reads `elapsed()`. + * How much the view animates on its own: eases, camera flights, coasting, orbits, comets, and + * options that read `elapsed`. `system` follows the user's reduced-motion setting, `full` animates + * everything, and `reduced` animates only motion the user drives. Defaults to `system`. */ motion?: Motion; }; -/** Your own colormap: its kind, its colors, and for a categorical map its categories' names. */ +/** A custom colormap: its kind, its colors, and for a categorical map, its category names. */ export interface CustomColormap { + /** How the colors map values. */ kind: ColormapKind; /** - * Colors blended in OKLab, each at its `[place, color]` from 0 to 1 or evenly between its - * neighbors, as in a CSS gradient; a categorical map's, one per category. + * The colors, blended in OKLab. Each is a color, spaced evenly between its neighbors like a CSS + * gradient, or a `[place, color]` pair with a place from 0 to 1. Places must ascend. A categorical + * map takes one color per category, without places, at most 256. */ - colors: (Color | [number, Color])[]; - /** The categories' names, in order, for a legend: a categorical map's alone. */ + colors: (/** A CSS color, such as `#ffb000` or `rebeccapurple`. */ Color | [number, Color])[]; + /** The category names in category order, shown in legends. Categorical maps only. */ categories?: string[]; } /** - * A sheet for diagrams, flat and seen from straight above: the camera pans and zooms. Positions - * are units with y down, and every length is a unit, a CSS pixel at zoom 0 (sizes, padding, ports, - * strokes, text, and the theme's), except glow, halo, and pick radius. Zooming scales one fixed - * picture. Only a diagram has ports, groups, nets, and orthogonal routes; it lays out blocks. + * A flat sheet for diagrams, seen from straight above. The camera pans and zooms but does not + * turn. Positions are in units with y down, and a unit is a CSS pixel at zoom 0. Sizes, padding, + * ports, strokes, text, and the theme's lengths are in units too. Sizes given with `px`, the text + * halo, the focus glow, and the pick radius stay in CSS pixels. Zooming scales the whole picture. + * Only a diagram has ports, groups, nets, and orthogonal routes, and the scene's layout places its + * blocks. */ export interface Diagram { - /** The grid under the marks. */ + /** The grid drawn under the marks. Defaults to dots 20 units apart. */ grid?: Grid; } +/** How a diagram's blocks, groups, ports, and wires look. */ +export interface DiagramTheme { + /** The fill color of group containers. */ + group?: Color; + /** The color of the one-unit border around blocks and groups. `null` draws no border. */ + outline?: Color | null; + /** The corner radius of blocks and groups. A block marker can set its own. */ + corner?: number; + /** The width of ports. */ + portSize?: number; + /** The bend radius of orthogonal wires. */ + bend?: number; + /** The space between a group's members and its border. */ + groupPadding?: number; + /** How blocks and groups cast shadows on the sheet. */ + shadow?: ShadowTheme; +} + /** A direction on screen. */ -export type Direction = 'right' | 'left' | 'down' | 'up'; +export type Direction = + | /** Toward the right. */ 'right' + | /** Toward the left. */ 'left' + | /** Toward the bottom. */ 'down' + | /** Toward the top. */ 'up'; -/** The earth: positions are longitude and latitude in degrees. */ +/** + * The earth: positions are longitude and latitude in degrees. The ground uses the theme's `earth` + * colors. + * + * A flat map shows the world once. Its ground ends at the antimeridian and at Web Mercator's + * latitude limit. The camera zooms out until the whole world fits the view, and pans until the + * world's edges meet the view's edges. On a globe, longitude wraps, and the camera zooms out until + * the globe spans a third of the view's shorter side. + */ export interface Earth { - /** A flat map or a globe. */ + /** A flat web mercator map (`mercator`) or a globe (`globe`). Defaults to `mercator`. */ projection?: Projection; - /** The ground's color; the theme's surface unless set. */ - surface?: Color | null; - /** Whether lines of longitude and latitude cross the ground, in the theme's grid color. */ + /** + * Whether lines of longitude and latitude cross the ground, in the theme's `grid` color. Defaults + * to `true`. + */ graticule?: boolean; /** - * Lit by the sun as it stands this many seconds after the Unix epoch, plus the view's axis - * coordinate when it shows one: night falls on the ground and what stands on it, down to the - * theme's night. Lit evenly unless set. + * Lights the earth by the sun's position this many seconds after the Unix epoch, plus the view's + * axis coordinate when it has one. The night side darkens the ground and everything on it, down to + * the theme's `earth.night` brightness. When unset, the earth is lit evenly. */ daylight?: number | null; - /** Whether air glows around the globe, in the theme's atmosphere color, brighter where it is day. */ + /** + * Whether air glows around the globe, in the theme's `earth.atmosphere` color, brighter on the day + * side. Defaults to `false`. + */ atmosphere?: boolean; } +/** How the earth looks. */ +export interface EarthTheme { + /** The ground color. */ + ground?: Color; + /** The color of the air glowing around the globe, when the earth's `atmosphere` is on. */ + atmosphere?: Color; + /** The brightness of the night side when the earth has `daylight`, from 0 to 1. */ + night?: number; +} + +/** Defaults for edges, and how parallel edges and arrowheads look. */ +export interface EdgeTheme { + /** + * The edge color. `null` blends the colors of each edge's two vertices. With `null`, a diagram's + * wires and edges without `ends` (nets and poles) use the text color. + */ + color?: Color | null; + /** The edge width, in the space's unit. */ + width?: number; + /** + * In a diagram, the distance between edges joining the same two vertices and between parallel + * orthogonal wires. Wires keep twice this distance from blocks where they can. + */ + spacing?: number; + /** + * Outside a diagram, how far edges joining the same two vertices bow apart. Each lane bows out by + * this fraction of the edge's length, measured at its middle. Parallel edges spread at most eight + * lanes wide, and more lanes pack closer. Bows scale with the zoom. + */ + bow?: number; + /** How far an arrowhead extends beyond the edge's width. A head's length is this plus the width. */ + arrow?: number; +} + /** - * Rows drawn as strokes between the vertices two reference fields name: branches, links, wires. In - * a diagram, a row without ends is a net: it joins every vertex port whose reference field names + * Rows drawn as strokes between two vertices named by reference fields: branches, links, wires. In + * a diagram, a row without ends is a net. It joins every vertex port whose reference field names * it. */ -export interface Edges { - /** The dataset and table drawn. */ - of: [data: string, table: string]; +export interface Edges extends MarkOptions, StrokeOptions { + /** The dataset and table drawn, `[data, table]`. */ + edges: TableInput; /** The reference fields naming each row's two vertices. */ ends?: [from: string, to: string] | null; - /** A path field of points each row runs through between its ends. */ + /** A path field whose points each row passes through between its ends. */ bends?: string | null; /** - * How each row runs between its ends; on the earth geodesic, elsewhere straight, unless set. - * Orthogonal routes run around blocks, bends rounded; diagrams only. + * How each row runs between its ends. Defaults to geodesic on the earth and straight elsewhere. + * `orthogonal` routes around blocks with rounded bends, in diagrams only. */ route?: Route | null; - /** Whether each row ends in an arrowhead at its second end; a net's at each input it reaches. */ + /** + * Whether each row has an arrowhead at its second end. A net has one at each input it reaches. + * Defaults to `false`. + */ arrows?: boolean; /** - * Draws each row as a pole from its own vertex down to the ground beneath it, on a plane or the - * earth; it has no ends. + * Draws each row as a pole from its own vertex down to the ground, on a plane or the earth. Poles + * take no `ends`, `bends`, or `joins`. Defaults to `false`. */ poles?: boolean; /** - * Draws each row as a net on a plane or the earth: it joins the vertex rows of `table` whose - * reference field names it, as a star from their middle to each. + * Draws each row as a net on a plane or the earth. The net joins the vertex rows of `table` whose + * `reference` field names it, as a star from their center to each member. */ joins?: [table: string, reference: string] | null; - /** CSS pixels, or position units with `units`; units in a diagram. */ - width?: Size | null; - /** Whether each row is dashed. */ - dash?: ExprInput; - /** - * Comets flowing along each row at this speed a second: CSS pixels, units in a diagram, or - * position units where the theme's flow is in units; negative flows backward. A speed that changes - * carries comets on from where they are. - */ - flow?: ExprInput; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; } -/** A value for every row of a mark: a constant, a field, view state, or arithmetic of them. */ +/** + * A request for each row's least and greatest value of a sampled field over equal spans of the + * axis. It reads field `field` of table `table` over `spans` equal spans from `range[0]` to + * `range[1]`, for every row or for the rows in `ids`, in that order. Like a query, it is plain + * data that deserializes from JSON. + */ +export interface EnvelopeRequest { + /** The table to read. */ + table: string; + /** The sampled field to read. */ + field: string; + /** The axis range the spans cover, `[first, last]`. Both are finite, with `first <= last`. */ + range: [number, number]; + /** The number of equal spans to divide the range into. At least 1. */ + spans: number; + /** The rows to read, in this order. Defaults to every row. */ + ids?: Ids | null; +} + +/** + * A value for each row. A number or boolean is a constant and `null` is missing. A string starting + * with `$` reads that field, and any other string is a CSS color. An object computes a value. + */ export type ExprNode = | number | boolean - | string + | /** A missing value. An option that is missing uses the theme's value. */ null + | /** + * A field or a color. `"$kv"` reads field `kv`. Any other string is a CSS color, such as `#ffb000` + * or `rebeccapurple`. Write a field named `$x` as `{ "field": "$x" }`. + */ string | FieldNode - | ColorNode | StateNode - | MissingNode - | UnaryNode - | BinaryNode | WhenNode - | IsNode - | ContainsNode | ScaleNode - | ColormapNode; + | ColormapNode + | CallNode; /** - * Rows drawn as triangles between the vertices three reference fields name: meshes, surfaces, and + * Rows drawn as triangles between three vertices named by reference fields: meshes, surfaces, and * fields interpolated across a network. Faces are lit from the camera, so a flat mesh seen from * above shows its exact colors. */ -export interface Faces { - /** The dataset and table drawn. */ - of: [data: string, table: string]; +export interface Faces extends MarkOptions { + /** The dataset and table drawn, `[data, table]`. */ + faces: TableInput; /** - * The reference fields naming each row's three vertices. Without a color, a face blends the colors - * of its corners. + * The reference fields naming each row's three vertices. Required. Without a color, a face blends + * the colors of its corners. */ corners?: [a: string, b: string, c: string] | null; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; - /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). - */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; } -/** A field of each row: of the mark's own table, or of another dataset's table joined by row id. */ +/** + * A field's value in each row, from the mark's own table. With `from`, it reads another dataset's + * table, joined by row id. `"$name"` is the short form of `{ "field": "name" }`. + */ export interface FieldNode { + /** The field's name. */ field: string; /** The dataset and table the field is read from, joined by row id. */ - of?: [data: string, table: string] | null; - /** Reads a sampled field at its last frame without interpolating. */ + from?: TableInput | null; + /** + * For a sampled field, reads the last frame at or before the shown coordinate instead of + * interpolating. Defaults to `false`. + */ step?: boolean; } -/** What a view's fit frames. */ +/** The target of a view's fit. */ export type Fit = - | 'all' + | /** Everything drawn. */ 'all' | { /** These rows. */ rows: Rows[]; @@ -473,53 +598,73 @@ export type Fit = | { /** The box from `min` to `max`: x, y, and elevation. */ bounds: { - /** The least corner. */ + /** The minimum corner. */ min: [number, number, number]; - /** The greatest corner. */ + /** The maximum corner. */ max: [number, number, number]; }; }; /** - * How comets flowing along edges and lines look: each a round head, brightest at its middle, - * trailing a tail that thins and fades. + * How comets along edges and lines look. A comet has a round head, brightest at its middle, and a + * tail that thins and fades. + * + * `spacing` and `length` must use the same unit, and each row's `flow` speed is in that unit too. + * In position units (degrees on the earth), comets scale with the zoom and stay in place along + * their lines. In CSS pixels, they keep their size and speed on screen. A plain number is in the + * space's unit. */ -export interface Flow { - /** A comet's color; its alpha is how strongly a comet covers its stroke. */ - color?: Color; - /** Distance between comets. */ - spacing?: number; - /** How far a comet's tail trails its head. */ - length?: number; - /** How wide a comet's head is, CSS pixels, or units in a diagram; never narrower than its stroke. */ - width?: number; +export interface FlowTheme { /** - * Whether comets are part of the drawing: their spacing, length, and each row's flow in position - * units, degrees on the earth, so they zoom with the view and stay where they are on their lines. - * Otherwise they keep their size and speed on screen, and slide along their lines as it zooms. + * The comet color. Its alpha sets how strongly a comet covers its stroke. A fully transparent + * color turns comets off. */ - units?: boolean; + color?: Color; + /** The distance between consecutive comets. */ + spacing?: Length; + /** The length of a comet's tail behind its head. */ + length?: Length; + /** The width of a comet's head, in the space's unit. A head is never narrower than its stroke. */ + width?: number; +} + +/** How hovered and selected rows glow, and how rows outside a selection dim. */ +export interface FocusTheme { + /** The glow color around hovered rows. */ + hover?: Color; + /** The glow color around selected rows. `null` uses each row's own color. */ + selected?: Color | null; + /** How far the hover and selection glows reach, in CSS pixels. */ + glow?: number; + /** The opacity of rows outside the selection while any row is selected, from 0 to 1. */ + dim?: number; + /** The scale factor for a hovered vertex sized in CSS pixels. Vertices sized in units only glow. */ + grow?: number; } -/** How an axis writes its tick values. */ -export type Format = 'auto' | 'fixed' | 'scientific' | 'engineering'; +/** How an axis formats its tick values. */ +export type Format = + | /** Fixed decimals, scientific past a million or below a ten-thousandth. */ 'auto' + | /** Fixed decimals: `1500.0`. */ 'fixed' + | /** A significand and a power of ten: `1.5e3`. */ 'scientific' + | /** A significand and a power of ten that is a multiple of three: `1.5e3`, `150e-6`. */ 'engineering'; /** A plane's or a diagram's grid: `"off"`, `{ lines: { pitch } }`, or `{ dots: { pitch, snap } }`. */ export type Grid = - | 'off' + | /** No grid. */ 'off' | { - /** Lines `pitch` units apart, thinned by tens as the view zooms out. */ + /** Lines `pitch` units apart, thinned by factors of ten as the view zooms out. */ lines: { /** Units between lines. */ pitch: number; }; } | { - /** Dots `pitch` units apart; drags snap to them when `snap`. */ + /** Dots `pitch` units apart. With `snap`, drags in a diagram snap to them. */ dots: { /** Units between dots. */ pitch: number; - /** Whether moved vertices snap to the dots. */ + /** Whether drag positions snap to the dots, in a diagram. Defaults to `false`. */ snap?: boolean; }; }; @@ -530,155 +675,186 @@ export type Grid = */ export type Id = number | string; -/** Row ids: numbers or texts (null leaves a reference unwired), or `{count}` naming rows 0..count. */ -export type Ids = (Id | null)[] | { count: number }; +/** + * Row ids: a list of numbers or texts, or `{count}` for rows 0 to count - 1. In a reference, null + * names no row. + */ +export type Ids = + | (Id | null)[] + | { + /** The number of rows, whose ids are 0 to count - 1. */ + count: number; + }; /** How a line runs between its samples. */ -export type Interpolation = 'linear' | 'stepBefore' | 'stepAfter'; - -/** Whether a text field equals the text. */ -export interface IsNode { - is: [field: ExprInput, text: string]; -} +export type Interpolation = + | /** Straight from each sample to the next. */ 'linear' + | /** + * Each value holds from the previous sample up to its own, so the line steps at the previous + * sample. + */ 'stepBefore' + | /** Each value holds until the next sample, where the line steps. */ 'stepAfter'; +/** + * Text for each row, placed beside, inside, or along the row's shape and moving with it: a text + * field, or options. A label shows when its row is visible and its text fits, for example inside a + * block. + * + * Labels beside or along their rows never overlap. Where they would, the row drawn on top keeps + * its label (so `priority` decides), and a dense view labels as many rows as fit. A row whose text + * is missing or empty has no label, so a `when` that returns the field or missing labels only some + * rows. In a diagram, a label scales with the zoom, fades out when too small to read, and shows + * wherever it fits its vertex. + */ export type Label = | ExprInput | { - /** A text field. */ + /** + * The text field to show, or a `when` that returns that field or missing. A row whose text is + * missing or empty has no label. In a diagram, the text may read only data, since the sheet sizes + * blocks around it. + */ text: ExprInput; - /** CSS pixels, or units in a diagram; the theme's font size unless set. */ + /** Font size in CSS pixels, or diagram units in a diagram. Defaults to the theme's `text.size`. */ size?: number | null; - /** Each row's text color; the theme's text color unless set. */ + /** Each row's text color. Defaults to the theme's text color. */ color?: ExprInput | null; - /** Where labels would overlap, the higher shows; the mark's priority unless set. */ - priority?: ExprInput | null; - /** How far a line runs before it wraps: CSS pixels, or units in a diagram. */ + /** The width at which lines wrap, in CSS pixels, or diagram units in a diagram. */ maxWidth?: number | null; - /** The CSS font family list its texts are set in; the theme's font unless set. */ + /** The CSS font family list for the text. Defaults to the theme's `text.font`. */ font?: string | null; - /** - * The most labels shown at once, those of highest priority among the rows in view; where they - * would overlap, fewer. - */ - max?: number | null; }; -/** Layered drawing: vertices in ranks, edges pointing along `flow`, and nets from their outputs. */ +/** + * Layered drawing: vertices in ranks, edges pointing along `direction`, and nets ranked from their + * outputs. + */ export interface Layered { - /** Where edges point. */ - flow?: Direction; - /** Units between vertices of a rank. */ + /** The direction edges point. Defaults to `right`. */ + direction?: Direction; + /** Distance between vertices within a rank, in position units. Defaults to 30. */ gap?: number; - /** Units between ranks. */ + /** Distance between ranks, in position units. Defaults to 80. */ rankGap?: number; } /** - * How the view places vertex rows without positions: a diagram's blocks, sized, or the points of a - * plane or the earth, among the rows linked to them, gaps in the space's units: a name, or - * `{ stress | layered: options }`. A place, once given, holds: rows added later are placed around - * the rest, and `View::arrange` lays every row out again. + * How the view places vertex rows that have no position: a name, or + * `{ stress | layered: options }`. In a diagram it places blocks. On a plane or the earth it + * places points among the rows they link to. Gaps are in position units: diagram units, plane + * units, or degrees on the earth. + * + * A placed row keeps its place. Rows added later are placed around the others, and `view.arrange` + * lays out every row again. */ export type Layout = | 'none' | 'stress' | 'layered' | { - /** Distances on the drawing follow distances in the graph. */ + /** Drawn distances follow graph distances. */ stress: Stress; } | { - /** Ranks along a direction, for flows and hierarchies. */ + /** Ranks along a direction, for processes and hierarchies. */ layered: Layered; }; /** - * Rows drawn as lines: along a path field (roads, rivers, borders) or a polygon field's rings, - * outlining its areas; on a plot, through a sampled field's history across the data's axis, what a - * monitor shows. On the earth paths bend with the globe and stay whole across the antimeridian. + * A length: a number in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. + * `{ units }` is position units (degrees on the earth), which grow as the view zooms in. */ -export interface Lines { - /** The dataset and table drawn. */ - of: [data: string, table: string]; - /** The path or polygon field each row runs along; `geometry` unless set. Not on a plot. */ +export type Length = + | number + | { + /** CSS pixels, the same at any zoom. */ + px: number; + } + | { + /** Position units (degrees on the earth), which grow as the view zooms in. */ + units: number; + }; + +/** Defaults for lines. */ +export interface LineTheme { + /** The line color. */ + color?: Color; + /** The line width, in the space's unit. */ + width?: number; +} + +/** + * Rows drawn as lines along a path field (roads, rivers, borders) or a polygon field's rings (area + * outlines). On a plot, each row traces a sampled field's history across the data's axis, like a + * monitor. On the earth, paths follow the globe's curve and stay unbroken across the antimeridian. + */ +export interface Lines extends MarkOptions, StrokeOptions { + /** The dataset and table drawn, `[data, table]`. */ + lines: TableInput; + /** + * The path or polygon field each row runs along. Defaults to the field named `geometry`. Not + * allowed on a plot. + */ geometry?: string | null; /** - * The sampled field each row traces across the data's axis, on a plot alone. A field not recorded - * yet draws nothing until its first frame. + * The sampled field each row traces across the data's axis. Required on a plot, and allowed only + * there. A field with no frames yet draws nothing until its first frame arrives. */ trace?: string | null; - /** How each row runs between its samples, on a plot. */ + /** How each row runs between its samples on a plot. Defaults to `linear`. */ interpolation?: Interpolation; - /** CSS pixels, or position units with `units`; units in a diagram. */ - width?: Size | null; - /** Whether each row is dashed. */ - dash?: ExprInput; +} + +/** + * One table a scene draws. The mark's kind is the key that holds its table, next to its options: + * `{ vertices: [data, table], radius: 4 }`. + */ +export type Mark = Vertices | Edges | Faces | Lines | Areas; + +/** The options every mark has. */ +export interface MarkOptions { /** - * Comets flowing along each row at this speed a second: CSS pixels, units in a diagram, or - * position units where the theme's flow is in units; negative flows backward. A speed that changes - * carries comets on from where they are. + * Each row's color: a CSS color, a value through a colormap, or a conditional that picks between + * colors. Defaults to the theme's color for the mark. Faces default to a blend of their corners' + * colors. */ - flow?: ExprInput; - /** Each row's color; the theme's unless set. */ color?: ExprInput | null; - /** Whether each row is drawn and picked. */ + /** Whether each row is drawn and can be picked. Defaults to `true`. */ visible?: ExprInput; /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. + * Each row's priority. It reads only data, not view state. Higher rows draw on top, keep their + * labels where labels overlap, and are dropped last when a large table is thinned. */ priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ + /** Distinguishes two marks of the same table across scenes, and names the mark in hits. */ name?: string | null; /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). + * Whether rows respond to hover, clicks, and picks. Defaults to on for vertices, edges, faces, and + * plot traces, and off for geometry: areas, and lines along paths such as coasts, borders, and + * roads. */ pick?: boolean | null; - /** Whether a click selects a row. */ + /** Whether clicking a row selects it. Defaults to `true`. */ select?: boolean; - /** Whether pressing a row drags it instead of panning. */ + /** + * Whether pressing and moving a row drags it instead of panning the camera. Drags are reported as + * a `'drag'` event. Defaults to `false`. + */ drag?: boolean; - /** Each row's text: a text field, or a label's options. */ + /** Each row's label: a text field, or label options. */ label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ - shadow?: boolean; /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. + * A legend of the mark's colors at the view's bottom right, with this title. It shows a ticked + * colormap across its domain, a cell for each category, or a swatch for each name in the + * colormap's `categories`. `color` must map through a colormap with a fixed domain or named + * categories. In a conditional, `otherwise` is searched before the cases. */ legend?: string | null; } -/** One drawn table of a scene: `{ vertices | edges | faces | lines | areas: options }` as JSON. */ -export type Mark = - | { - /** Rows drawn as markers. */ - vertices: Vertices; - } - | { - /** Rows drawn as strokes between two vertices. */ - edges: Edges; - } - | { - /** Rows drawn as triangles between three vertices. */ - faces: Faces; - } - | { - /** Rows drawn as lines along their paths, or on a plot through their samples. */ - lines: Lines; - } - | { - /** Rows drawn as the areas their polygons cover. */ - areas: Areas; - }; - /** * How a vertex is drawn: a shape's name, `"block"`, or - * `{ shape | block | gauge | pie | icon | wgsl: options }`. + * `{ shape | block | gauge | pie | icon | wgsl: options }`. Defaults to an ellipse. */ export type Marker = | 'ellipse' @@ -691,68 +867,72 @@ export type Marker = shape: Shape; } | { - /** A block: a rectangle with rounded corners, sized for diagrams. */ + /** A rectangle with rounded corners that holds its label inside, like a diagram's blocks. */ block: { - /** Corner radius, CSS pixels, or units in a diagram; the theme's `corner` unless set. */ + /** Corner radius in CSS pixels, or units in a diagram. Defaults to the theme's `diagram.corner`. */ corner?: number | null; }; } | { - /** A shape's outline, filled to `fill` from 0 to 1: around a circle, up other shapes. */ + /** + * A shape's outline, filled to `fill` (0 to 1). A circle fills around its edge, and other shapes + * fill upward. + */ gauge: { - /** How full each row's gauge is. */ + /** How full each row's gauge is, from 0 to 1. */ fill: ExprInput; - /** The outline. */ + /** The outline's shape. Defaults to an ellipse. */ shape?: Shape; }; } | { - /** A circle cut into slices, each as large as its value. */ + /** A circle divided into slices sized in proportion to their values. */ pie: { /** Each slice's value. */ slices: ExprInput[]; /** Each slice's color. */ colors: Color[]; - /** The hole in the middle, as a fraction of the radius. */ + /** The hole in the middle, as a fraction of the radius. Defaults to 0. */ hole?: number; }; } | { - /** One of `images`, fitted in the marker's box; `index` picks which for each row. */ + /** One of `images`, fitted to the marker's box. `index` picks the image for each row. */ icon: { /** The images. */ images: Bitmap[]; - /** Each row's image, from 0. */ + /** Each row's image index, from 0. */ index: ExprInput; }; } | { /** - * A marker in WGSL: `source` defines `fn marker(f: MarkerFragment) -> MarkerColor`, reading up to - * eight `inputs` as `f.input[i]`. It compiles once, when the view first draws it. + * A custom marker in WGSL. `source` defines `fn marker(f: MarkerFragment) -> MarkerColor`, which + * can read up to eight `inputs` as `f.input[i]`. It compiles once, when the view first draws it. */ wgsl: { - /** The function's source. */ + /** The WGSL source of the `marker` function. */ source: string; - /** Each row's inputs. */ + /** Each row's inputs, at most eight. */ inputs?: ExprInput[]; }; }; -/** A missing value: options that read one take the theme's. */ -export interface MissingNode { - missing: true; -} +/** How much a view animates on its own. */ +export type Motion = + | /** Follows the system setting: reduced when the user has asked their system for less motion. */ 'system' + | /** All motion. */ 'full' + | /** + * Only motion the user drives: changes apply at once, flights jump, nothing coasts or orbits, and + * comets and options that read `elapsed` stay still. + */ 'reduced'; -/** How much a view moves on its own. */ -export type Motion = 'system' | 'full' | 'reduced'; +/** Sort direction. Rows where the key has no value sort last in either direction. */ +export type Order = /** Smallest first. */ 'ascending' | /** Largest first. */ 'descending'; -/** Which way a sort runs; rows a key reads nothing for come last either way. */ -export type Order = 'ascending' | 'descending'; - -/** A plane: positions are plane units. */ +/** A plane: positions are in plane units. */ export interface Plane { - /** The grid under the marks. */ + /** The grid drawn under the marks. Defaults to lines 10 units apart. */ grid?: Grid; } @@ -764,10 +944,16 @@ export interface Plot { y?: Axis; } +/** How a plot looks. */ +export interface PlotTheme { + /** The color of the line at the view's current axis coordinate. */ + playhead?: Color; +} + /** - * A vertex's port: where a net its reference field names meets it, on one side of its marker: a - * side, or options. An output drives its net: layouts rank a net from its outputs, and its arrows - * point away from them. + * A port on one side of a vertex's marker, where a net connects: a side name, or options. The + * vertex names the net in a reference field. An output drives its net: layouts rank a net from its + * outputs, and the net's arrows point away from them. */ export type Port = | 'left' @@ -775,54 +961,75 @@ export type Port = | 'top' | 'bottom' | { - /** The side it sits on. */ + /** The side of the marker it sits on. */ side: Side; /** - * Its place among the ports on that side, from the top or the left; in declaration order unless - * set. + * Its position among the ports on its side, counted from the top or left. Ports without an `order` + * follow those with one, in declaration order. */ order?: number | null; /** Its name, drawn beside it inside the marker. */ label?: string | null; - /** Its marker. */ + /** Its shape. Defaults to an ellipse. */ shape?: Shape; - /** Whether it drives the net its reference field names; an input unless set. */ + /** Whether it is an output, which drives its net. Defaults to `false` (an input). */ output?: boolean; }; -/** How the earth is drawn. */ -export type Projection = 'mercator' | 'globe'; +/** How the earth is projected. */ +export type Projection = /** Web mercator. */ 'mercator' | /** A sphere. */ 'globe'; + +/** A size in CSS pixels, regardless of the space's unit. */ +export interface Px { + /** The size, in CSS pixels. */ + px: ExprInput; +} /** The rows of a table to read back, run through the expressions marks draw with. */ export interface Query { - /** The table read. */ + /** The table to read. */ of: string; /** - * The rows read, in this order unless sorted; every row unless set. Ids that name no row are - * passed over. + * The rows to read, in this order unless sorted. Defaults to every row. Ids that name no row are + * skipped. */ ids?: Ids | null; - /** Keeps the rows it is true for. */ + /** Keeps only the rows for which this is true. */ filter?: ExprInput | null; /** - * Keys to sort by, each after those before it; rows a key reads nothing for come last, and rows - * all keys order alike keep their order. + * Keys to sort by. Each key breaks ties left by the keys before it. Rows where a key has no value + * sort last, and rows equal on every key keep their order. */ sort?: [ExprInput, Order][]; - /** The kept rows skipped first. */ + /** How many of the kept rows to skip. */ offset?: number; - /** The most rows read; a limit picks them in linear time before sorting only those. */ + /** + * The most rows to return. When sorting with a limit, the first `offset + limit` rows are selected + * in linear time and only those are sorted. + */ limit?: number | null; - /** Fields to read by name: a field as it is, anything else as numbers. Every field unless set. */ - select?: { [name: string]: ExprInput }; - /** The axis coordinate sampled fields are read at; their last frame unless set. */ + /** + * Fields to return, by output name. A plain field returns that field as stored, and any other + * expression returns numbers. Defaults to every field. An empty object returns the ids alone. + */ + select?: { [name: string]: ExprInput } | null; + /** The axis coordinate at which sampled fields are read. Defaults to their last frame. */ at?: number | null; } /** How an edge runs between its ends. */ -export type Route = 'straight' | 'geodesic' | 'orthogonal'; +export type Route = + | /** A straight line on screen. */ 'straight' + | /** Along the great circle, on the earth. */ 'geodesic' + | /** + * Horizontal and vertical runs around vertices that leave and enter at their sides, with rounded + * bends. Diagrams only. + */ 'orthogonal'; -/** A row of a table in a dataset. It survives any change that keeps the names and the id. */ +/** + * One row, identified by dataset name, table name, and id. It stays valid across any change that + * keeps those names and the id. + */ export interface Row { /** The dataset's name. */ data: string; @@ -832,7 +1039,10 @@ export interface Row { id: Id; } -/** Rows of one table in a dataset, by id: what a view selects, fits, and tells as sets of rows. */ +/** + * Rows of one table in a dataset, by id. Views use it for selections, fits, and the rows events + * report. + */ export interface Rows { /** The dataset's name. */ data: string; @@ -842,298 +1052,287 @@ export interface Rows { ids: Id[]; } -/** - * `value` mapped linearly onto the range `scale`, clamped: from its `domain`, else from the extent - * of the values shown. - */ +/** A value mapped linearly onto a range, clamped to the range. */ export interface ScaleNode { + /** The range. */ scale: [low: number, high: number]; + /** The value mapped. */ value: ExprInput; + /** The values mapped to the range's ends. Defaults to the extent of the values shown. */ domain?: [low: number, high: number] | null; } /** - * What a view draws and how it answers input: a space, marks in drawing order, a layout, a theme, - * controls, how long changes ease, and the title written over it. Plain data: as JSON, each field - * by its name in camelCase, each mark `{ vertices | edges | faces | lines | areas: options }`. + * A view's content and behavior: a space, marks drawn in order, a layout, a theme, controls, an + * ease duration, and an optional title. + * + * A scene is plain data. In JSON, fields are camelCase, and the key naming each mark's kind holds + * its dataset and table: `{ "vertices": ["grid", "Bus"], "radius": 4 }`. */ export interface Scene { + /** The coordinate space positions are in. */ space: Space; - /** What the view draws, in drawing order. */ + /** The marks, in drawing order: each draws over the marks before it. */ marks?: Mark[]; /** - * How the view places vertex rows without positions: a diagram's blocks, or points on a plane or - * the earth, gaps in the space's units. + * How the view places vertex rows that have no position, such as a diagram's blocks or points on a + * plane or the earth. Defaults to `none`. */ layout?: Layout; + /** Default colors and sizes, and the look of text, focus, and each space. Defaults to `dark`. */ theme?: Theme; + /** How the view responds to the pointer and keyboard. */ controls?: Controls; - /** Seconds that changes ease over; 0.3 unless set. */ + /** How many seconds scene and data changes take to animate. 0 turns animation off. Defaults to 0.3. */ ease?: number; - /** Written at the view's top left, larger than the theme's text. */ + /** A title drawn at the view's top left, larger than the theme's text. */ title?: string | null; - /** Written under the title. */ + /** A subtitle drawn under the title. */ subtitle?: string | null; } -/** How marks that cast shadows cast them onto the ground and onto faces. */ -export interface Shadow { - /** What a shadow darkens toward; its alpha is how dark a shadow at contact gets. */ +/** How a diagram's blocks and groups cast shadows on the sheet, for vertex marks with `shadow` on. */ +export interface ShadowTheme { + /** The color a shadow darkens toward. Its alpha sets how dark the shadow is under the block. */ color?: Color; - /** The direction toward the light, fixed in the world: shadows fall away from it. */ - light?: [number, number, number]; - /** How much a shadow spreads per unit of a caster's height: the light's size. */ - softness?: number; - /** - * CSS pixels flat marks cast from, as if raised; in a diagram, how far a block's or a group's - * shadow falls below it, and as far again it softens. - */ + /** How far a shadow falls below its block or group. Its edge softens over the same distance. */ lift?: number; } /** A marker's outline. */ -export type Shape = 'ellipse' | 'rectangle' | 'rounded' | 'diamond'; +export type Shape = + | /** An ellipse: a circle when its width and height are equal. */ 'ellipse' + | /** A rectangle. */ 'rectangle' + | /** A rectangle with rounded corners. */ 'rounded' + | /** A diamond. */ 'diamond'; /** A side of a marker. */ -export type Side = 'left' | 'right' | 'top' | 'bottom'; +export type Side = + | /** The left side. */ 'left' + | /** The right side. */ 'right' + | /** The top. */ 'top' + | /** The bottom. */ 'bottom'; -/** A size: CSS pixels, or position units with `units`. */ -export type Size = ExprInput | Units; +/** + * A size: a value in CSS pixels, or units in a diagram. `{ px }` is always CSS pixels. `{ units }` + * is position units (degrees on the earth), which grow as the view zooms in. + */ +export type Size = ExprInput | Px | Units; /** - * The coordinate space a scene's positions are in: a name, or - * `{ plane | diagram | earth | plot: options }`. A diagram is a flat sheet seen from straight - * above: y runs down, and every length is a unit, a CSS pixel at zoom 0, except glow, halo, and - * pick radius. Only a diagram has ports, groups, nets, orthogonal routes, and layouts. + * The coordinate space of a scene's positions: a name, or + * `{ plane | diagram | earth | plot: options }`. + * + * The space sets the unit of lengths that do not name their own. In a diagram it is the diagram + * unit, one CSS pixel at zoom 0. Elsewhere it is the CSS pixel. Only a diagram has ports, groups, + * nets, and orthogonal routes. */ export type Space = | 'plane' | 'diagram' - | 'mercator' - | 'globe' + | 'earth' | 'plot' | { - /** Longitude and latitude in degrees, on a map or a globe. */ + /** Longitude and latitude in degrees, on a flat map or a globe. */ earth: Earth; } | { - /** Plane units with elevation: graphs and scenes the camera turns and tilts around. */ + /** Plane units, with elevation. The camera can turn and tilt. */ plane: Plane; } | { - /** The data's axis across, values up. */ + /** The data's axis horizontally and values vertically. */ plot: Plot; } | { - /** A sheet for diagrams: x right and y down in units, every length a unit. */ + /** A flat sheet for diagrams: x right and y down, every length in units. */ diagram: Diagram; }; -/** What the view knows of each row. */ +/** + * View state of each row: `selected`, `hovered`, `pointer` (CSS pixels from a vertex to the + * pointer), `elapsed` (seconds since the view was created), or `zoom` (the camera's zoom). + */ export type State = 'selected' | 'hovered' | 'pointer' | 'elapsed' | 'zoom'; -/** What the view knows of each row. */ +/** View state of each row, such as whether it is selected. */ export interface StateNode { + /** The state read. */ state: State; } /** - * Stress majorization: each pair of vertices pulls toward its graph distance; then their boxes - * part, keeping a quarter of the gap clear. + * Stress majorization: vertices move until drawn distances match graph distances. Then overlapping + * boxes separate, keeping a quarter of `gap` between them. */ export interface Stress { - /** Units between adjacent vertices' centers. */ + /** Distance between the centers of linked vertices, in position units. Defaults to 40. */ gap?: number; } +/** The options of marks drawn as strokes: edges and lines. */ +export interface StrokeOptions { + /** + * Each row's width in CSS pixels, or units in a diagram, unless given with `px` or `units`. + * Defaults to the theme's width. + */ + width?: Size | null; + /** Whether each row is dashed, using the theme's `dash` period. Defaults to `false`. */ + dash?: ExprInput; + /** + * The speed of comets along each row, per second, in the unit of the theme's `flow` lengths. + * Negative speeds flow backward. 0 draws no comets. Defaults to 0. When the speed changes, comets + * continue from where they are. + */ + flow?: ExprInput; +} + +/** How labels, axis ticks, and titles look. */ +export interface TextTheme { + /** The text color. */ + color?: Color; + /** The CSS font family list. */ + font?: string; + /** The font size, in the space's unit. */ + size?: number; + /** Width of the background-colored halo around label text, in CSS pixels. */ + halo?: number; +} + /** - * Colors and sizes marks take where their options read nothing, and the look of focus and motion: - * a base theme's name, or options over one (`base`, dark unless set). Lengths are CSS pixels; in a - * diagram they are units, except `glow` and `halo`. + * Default colors and sizes for marks, and the look of text, focus, comets, and each space, grouped + * by subject: a base theme's name (`"dark"` or `"light"`), or options that change a base. `base` + * names the base and defaults to `dark`. Each option replaces only the base value it sets. + * + * Lengths are in CSS pixels, or units in a diagram. The text halo and the focus glow are always in + * CSS pixels. */ export type Theme = | 'dark' | 'light' | { - /** Behind everything. */ + /** The background color. */ background?: Color; - /** The earth's ground, and a group's container. */ - surface?: Color; - /** A plane's or a diagram's grid. */ + /** The color of a plane's or diagram's grid, a plot's grid lines, and the earth's graticule. */ grid?: Color; - /** Vertex color. */ - vertex?: Color; - /** Vertex radius. */ - vertexRadius?: number; - /** - * Edge color; `null` blends the colors of each edge's two vertices, and in a diagram takes the - * text color. - */ - edge?: Color | null; - /** Edge width. */ - edgeWidth?: number; - /** Line color. */ - line?: Color; - /** Line width. */ - lineWidth?: number; - /** Area color. */ - area?: Color; - /** How much a face turned away from the camera darkens, 0 to 1. */ - shading?: number; - /** Text: labels, axis ticks, and titles. */ - text?: Color; - /** The CSS font family text is set in. */ - font?: string; - /** Text size. */ - fontSize?: number; - /** The line a plot draws at the shown coordinate. */ - playhead?: Color; - /** What a missing color reads as. */ + /** The color a colormap gives a missing value, such as a row with no value in the field. */ missing?: Color; - /** The glow around hovered rows. */ - hover?: Color; - /** The glow around selected rows; `null` glows in each row's own color. */ - selected?: Color | null; - /** How far focus glows reach, CSS pixels. */ - glow?: number; - /** The opacity of rows outside a selection. */ - dim?: number; - /** How much a hovered vertex sized in CSS pixels grows. */ - hoverScale?: number; - /** Length of a dash. */ + /** The dash period: one dash and one gap of equal length. */ dash?: number; + /** How much faces darken as they turn away from the camera, from 0 to 1. */ + shading?: number; + /** How labels, axis ticks, and titles look. */ + text?: TextTheme; + /** Defaults for vertices. */ + vertex?: VertexTheme; + /** Defaults for edges, and how parallel edges and arrowheads look. */ + edge?: EdgeTheme; + /** Defaults for lines. */ + line?: LineTheme; + /** Defaults for areas. */ + area?: AreaTheme; + /** How hovered and selected rows glow, and how rows outside a selection dim. */ + focus?: FocusTheme; /** How comets flowing along edges and lines look. */ - flow?: Flow; - /** - * In a diagram, distance between the lanes of edges joining the same two vertices, and between - * parallel orthogonal wires, which keep twice it apart. - */ - edgeSpacing?: number; - /** - * Elsewhere, how far edges joining the same two vertices bow apart: each lane by this fraction of - * the edge's length at its middle. A bow is part of the drawing, so zooming never moves it. - */ - laneBow?: number; - /** How far an arrowhead reaches past its edge's width. */ - arrow?: number; - /** CSS pixels of background around label text. */ - halo?: number; - /** Distance between a group's members and its outline. */ - groupPadding?: number; - /** The border of a diagram's blocks and groups, a unit wide; `null` draws none. */ - outline?: Color | null; - /** The radius of a diagram's block and group corners; a block's marker may set its own. */ - corner?: number; - /** How wide a diagram's ports are. */ - portSize?: number; - /** The radius of an orthogonal wire's bends. */ - bend?: number; - /** How marks that cast shadows cast them. */ - shadow?: Shadow; - /** The air glowing around the globe, where the earth has an atmosphere. */ - atmosphere?: Color; - /** How bright the earth's night side stays where it is lit by daylight, 0 to 1. */ - night?: number; - /** The theme these options change; dark unless set. */ + flow?: FlowTheme; + /** How a diagram's blocks, groups, ports, and wires look. */ + diagram?: DiagramTheme; + /** How the earth looks. */ + earth?: EarthTheme; + /** How a plot looks. */ + plot?: PlotTheme; + /** The theme these options change. Defaults to `dark`. */ base?: 'dark' | 'light'; }; -/** A tick as written: a value, or `[value, label]`. */ +/** A tick in JSON: a value, or `[value, label]`. */ export type Tick = number | [number, string | null]; -/** An operation of one value. */ -export type Unary = 'neg' | 'not' | 'abs' | 'sqrt' | 'sin' | 'cos' | 'floor' | 'ln' | 'exp'; - -export interface UnaryNode { - op: Unary; - args: [ExprInput]; -} - -/** A size in position units, so it scales as the view zooms, kept within CSS pixels when given. */ +/** + * A size in position units (degrees on the earth), regardless of the space's unit. It scales with + * the zoom. `atLeast` and `atMost` bound it in CSS pixels. + */ export interface Units { + /** The size, in position units. */ units: ExprInput; - /** CSS pixels it keeps at least, however far the view zooms out. */ + /** The minimum on-screen size in CSS pixels, however far the view zooms out. */ atLeast?: number | null; - /** CSS pixels it keeps at most, however far the view zooms in. */ + /** The maximum on-screen size in CSS pixels, however far the view zooms in. */ atMost?: number | null; } +/** Defaults for vertices. */ +export interface VertexTheme { + /** The vertex color. */ + color?: Color; + /** The vertex radius, in the space's unit. */ + radius?: number; +} + /** Rows drawn as markers: points of a map, nodes of a graph, blocks of a diagram. */ -export interface Vertices { - /** The dataset and table drawn. */ - of: [data: string, table: string]; +export interface Vertices extends MarkOptions { + /** The dataset and table drawn, `[data, table]`. */ + vertices: TableInput; /** - * Each row's x and y. Without them, the point field `geometry` names places rows; rows without - * either the scene's layout places. + * Each row's x and y. Rows without a position are placed by the point field `geometry` names. The + * scene's layout places rows that have neither. */ position?: [ExprInput, ExprInput] | null; - /** The point field placing each row where `position` is not given; `geometry` unless set. */ + /** + * The point field that places rows when `position` is not set. Defaults to the field named + * `geometry`. + */ geometry?: string | null; /** - * Height above the ground: on the earth, a fraction of its radius. Not in a diagram, which is - * flat. + * Height above the ground. On the earth, a fraction of the earth's radius. Defaults to 0. Not + * allowed in a diagram, which is flat. */ elevation?: ExprInput; - /** CSS pixels, or position units with `units`; units in a diagram. */ + /** + * Each row's radius in CSS pixels, or units in a diagram, unless given with `px` or `units`. A + * diagram does not accept `px`. Defaults to the theme's `vertex.radius`. + */ radius?: Size | null; /** - * Width and height, for markers that are not round: CSS pixels, or position units with `units`; - * units in a diagram, where a block without a size fits its label and ports. + * Each row's width and height, for markers that are not round. In CSS pixels, or units in a + * diagram, unless given with `px` or `units`. A diagram does not accept `px`. In a diagram, a + * block without a size fits its label and ports. */ size?: [Size, Size] | null; - /** How each row is drawn. */ + /** How each row is drawn. Defaults to an ellipse. */ marker?: Marker; - /** A ring around each row's marker in this color; a missing or transparent color draws none. */ + /** A ring around each row's marker in this color. A missing or transparent color draws no ring. */ ring?: ExprInput | null; /** - * A port on each row for each reference field naming a net, which an `edges` mark of the net table - * without ends draws. Diagrams only. + * Ports on each row, one for each reference field that names a net. An `edges` mark of the net + * table, without `ends`, draws the nets. Diagrams only. */ ports?: [reference: string, port: Port][]; /** - * Makes each row a group: it holds the rows of `table` whose reference field names it, and is - * drawn as a container around them, its label a title in its header. Diagrams only. + * Makes each row a group containing the rows of `table` whose `reference` field names it. A group + * draws as a container around its members, with its label as a title in its header. Diagrams only. */ contains?: [table: string, reference: string] | null; - /** Each row's color; the theme's unless set. */ - color?: ExprInput | null; - /** Whether each row is drawn and picked. */ - visible?: ExprInput; - /** - * Each row's priority, from its data alone: what matters more draws on top, stays drawn as a table - * too large to draw whole thins to the tiles in view, and is labeled first. - */ - priority?: ExprInput | null; - /** Tells two marks of one table apart across scenes, and names the mark in hits. */ - name?: string | null; /** - * Whether rows are hovered, clicked, and probed. Unless set, rows that are things are (vertices, - * edges, faces, a plot's traces) and geometry is not (areas, and lines along paths: coasts, - * borders, roads). + * Whether each block or group casts a shadow on the sheet, styled by the theme's `diagram.shadow`. + * Defaults to `false`. Diagrams only. */ - pick?: boolean | null; - /** Whether a click selects a row. */ - select?: boolean; - /** Whether pressing a row drags it instead of panning. */ - drag?: boolean; - /** Each row's text: a text field, or a label's options. */ - label?: Label | null; - /** Whether each row casts a shadow onto the ground and onto faces, away from the theme's light. */ shadow?: boolean; - /** - * A legend of the colors at the view's bottom right, under this title: the colormap across its - * domain, ticked; a cell for each category; or a swatch beside each name of a map's `categories`. - * The color maps through a colormap with a domain or named categories; a conditional's `otherwise` - * is read before its cases. - */ - legend?: string | null; } /** The value of the first case whose condition holds, else `otherwise`. */ export interface WhenNode { + /** The cases, as `[condition, value]` pairs tried in order. */ when: [condition: ExprInput, value: ExprInput][]; + /** The value where no condition holds. */ otherwise: ExprInput; } -/** What zooms a view. */ -export type Zoom = 'wheel' | 'modifier' | 'off'; +/** Which input zooms a view. */ +export type Zoom = + | /** The wheel, pinching, and the keys. */ 'wheel' + | /** + * Pinching and the keys, and the wheel only with Control or Meta held (as trackpad pinches send + * it), so the plain wheel scrolls the page. + */ 'modifier' + | /** No input zooms. */ 'off'; diff --git a/npm/src/internal.ts b/npm/src/internal.ts new file mode 100644 index 00000000..3555ad87 --- /dev/null +++ b/npm/src/internal.ts @@ -0,0 +1,264 @@ +/** + * Code the package's classes share: access to the WebAssembly bindings, the `Handle` base class, + * errors, bounded waits, and byte reading. `Handle`, `LatkitError`, `Wait`, `ByteSource`, and + * `ReadableBytes` are public. + */ + +import * as bindings from '../pkg/latkit.js'; +import type { Failure } from './outputs.js'; + +/** The error latkit throws: an `Error` with a `code` that names the kind of failure. */ +export interface LatkitError extends Error { + /** + * The kind of failure: + * + * - `invalid`: the input is malformed or does not fit. `path` says where. + * - `unsupported`: the browser or device cannot do what was asked. + * - `limit`: a budget or device limit would be exceeded. + * - `io`: reading or writing external bytes failed, such as loading latkit or fetching a file. + * - `device-lost`: the GPU device was lost. Create a new `Gpu`. + * - `cancelled`: the wait's signal aborted, or the work was cancelled. + * - `timeout`: the wait's signal timed out. + * - `unloaded`: the call needs WebAssembly, and `load()` has not finished. + * - `destroyed`: the object was destroyed and can't be used. + */ + code: Failure['code'] | 'unloaded' | 'destroyed'; + /** On `invalid` errors, the invalid part of the input, such as `marks[2].color`. */ + path?: string; +} + +/** Options that bound how long a promise waits. */ +export interface Wait { + /** + * Aborts the wait. A timeout rejects with a `timeout` error. Any other abort rejects with a + * `cancelled` error, with the abort reason as its `cause`. Either way, latkit cancels the call's + * pending work where it can. Without a signal, each wait times out after 30 seconds. + */ + signal?: AbortSignal; +} + +let ready = false; + +/** @internal Marks the bindings as loaded, so `wasm()` returns them. */ +export function loaded(): void { + ready = true; +} + +/** @internal Returns the bindings. Throws an `unloaded` error until `load()` has finished. */ +export function wasm(): typeof bindings { + if (!ready) + throw thrown({ code: 'unloaded', message: 'latkit is not loaded: call `await load()` first' }); + return bindings; +} + +/** @internal Builds a {@link LatkitError}: a code, a message, and optionally a path and a cause. */ +export function thrown({ + code, + message, + path, + cause, +}: Pick & { + code: LatkitError['code']; + cause?: unknown; +}): LatkitError { + const error = new Error(message, cause === undefined ? undefined : { cause }); + return Object.assign(error, path === undefined ? { code } : { code, path }); +} + +/** Each handle's bindings object, until the handle is destroyed, and the name its errors use. */ +const held = new WeakMap(); + +/** + * An object that holds WebAssembly memory (and GPU memory, for a `Gpu`) until it is destroyed. + * `destroy()` frees it, as does the end of a `using` block. After that, its other methods throw a + * `destroyed` error. + */ +export abstract class Handle { + /** @internal Holds `bound`, the bindings object to free. Errors call the object `what`. */ + constructor(bound: { free(): void }, what: string) { + held.set(this, { bound, what }); + } + + /** Frees the object's memory. Calling it again does nothing. */ + destroy(): void { + const entry = held.get(this); + const bound = entry?.bound; + if (!bound) return; + entry.bound = undefined; + this.freeing(); + bound.free(); + } + + /** @internal Runs once when the object is destroyed, before its memory is freed. */ + protected freeing(): void {} + + /** Calls `destroy()`, so a `using` declaration destroys the object when its block ends. */ + [Symbol.dispose](): void { + this.destroy(); + } +} + +/** @internal Returns `owner`'s bindings object. Throws a `destroyed` error once it is destroyed. */ +export function inner(owner: Handle): T { + const entry = held.get(owner)!; + if (!entry.bound) + throw thrown({ code: 'destroyed', message: `this ${entry.what} has been destroyed` }); + return entry.bound as T; +} + +/** A byte stream: anything shaped like a browser or Node.js `ReadableStream` of `Uint8Array`. */ +export interface ReadableBytes { + /** Returns a reader of the stream's chunks. latkit reads it to the end or cancels it. */ + getReader(): { + read(): Promise<{ done: boolean; value?: Uint8Array }>; + cancel(reason?: unknown): Promise; + releaseLock(): void; + }; +} + +/** + * Bytes to read: an array or buffer, a `Blob`, a `Response`, a byte stream, or a URL string to + * fetch. Reading is bounded by the call's {@link Wait}. It fails with an `io` error when a fetch + * fails or returns an error status, or when the bytes can't be read. + */ +export type ByteSource = + Uint8Array | ArrayBuffer | Blob | Response | ReadableBytes | AsyncIterable | string; + +/** + * @internal Reads `source` into bytes, with the URL they came from as `base` (empty if none). An + * object that is not a {@link ByteSource} is encoded as JSON text. Waits within `wait`, and + * cancels a fetch or stream it gives up on. + */ +export async function bytesOf( + source: ByteSource | object, + wait: Wait, +): Promise<{ bytes: Uint8Array; base: string }> { + const failed = (message: string) => (cause: unknown) => + Promise.reject(thrown({ code: 'io', message, cause })); + const reading = (what: string, bytes: Promise) => + waited( + bytes.then((read) => new Uint8Array(read), failed(`${what} could not be read`)), + wait, + `reading ${what}`, + ); + if (source instanceof Uint8Array) return { bytes: source, base: '' }; + if (source instanceof ArrayBuffer) return { bytes: new Uint8Array(source), base: '' }; + if (typeof source === 'string') { + const fetched = fetch(source, { signal: wait.signal }).catch( + failed(`${source} could not be fetched`), + ); + return bytesOf(await waited(fetched, wait, `fetching ${source}`), wait); + } + if (source instanceof Response) { + const { url, status, statusText } = source; + if (!source.ok) throw thrown({ code: 'io', message: `${url}: ${status} ${statusText}` }); + return { bytes: await reading(url || 'the response', source.arrayBuffer()), base: url }; + } + if (source instanceof Blob) + return { bytes: await reading('the blob', source.arrayBuffer()), base: '' }; + if ('getReader' in source || Symbol.asyncIterator in source) { + const stream = source as ReadableBytes | AsyncIterable; + return { + bytes: await waited(collect(stream, wait.signal), wait, 'reading the stream'), + base: '', + }; + } + return { bytes: new TextEncoder().encode(JSON.stringify(source)), base: '' }; +} + +/** The bytes of `stream` in one array; see {@link chunks}. */ +async function collect( + stream: ReadableBytes | AsyncIterable, + signal?: AbortSignal, +): Promise { + const parts: Uint8Array[] = []; + for await (const chunk of chunks(stream, signal)) parts.push(chunk); + const bytes = new Uint8Array(parts.reduce((n, part) => n + part.length, 0)); + parts.reduce((at, part) => (bytes.set(part, at), at + part.length), 0); + return bytes; +} + +/** + * @internal Yields the chunks of `stream` until it ends or `signal` aborts. Reads through + * `getReader()` when the stream has one, else by async iteration. Stopping early cancels the + * stream with the abort reason. A failed read throws an `io` error. + */ +export async function* chunks( + stream: ReadableBytes | AsyncIterable, + signal?: AbortSignal, +): AsyncGenerator { + const failed = (cause: unknown) => + thrown({ code: 'io', message: 'the stream could not be read', cause }); + if (!('getReader' in stream)) { + try { + for await (const chunk of stream) { + if (signal?.aborted) return; + yield chunk; + } + } catch (cause) { + throw failed(cause); + } + return; + } + const reader = stream.getReader(); + const cancel = () => void reader.cancel(signal?.reason).catch(() => {}); + signal?.addEventListener('abort', cancel, { once: true }); + let ended = false; + try { + while (!signal?.aborted) { + const read = await reader.read().catch((cause: unknown) => Promise.reject(failed(cause))); + if ((ended = read.done)) return; + if (!read.value) throw thrown({ code: 'io', message: 'a stream chunk is missing' }); + yield read.value; + } + } finally { + signal?.removeEventListener('abort', cancel); + if (!ended) await reader.cancel(signal?.reason).catch(() => {}); + reader.releaseLock(); + } +} + +/** + * @internal Awaits `pending`, a promise or a bindings task, until `signal` aborts. The signal + * defaults to a 30-second timeout. If the signal aborts first, a task is cancelled so its work is + * skipped, and the call rejects with an error naming `what`: `timeout` for a timeout, else + * `cancelled`. Work that fails after the signal aborted, such as a fetch given the same signal, + * rejects the same way. A task is always freed. + */ +export async function waited( + pending: Promise | bindings.Task, + { signal = AbortSignal.timeout(30_000) }: Wait, + what: string, +): Promise { + const task = pending instanceof Promise ? undefined : pending; + const promise = (task ? task.promise : pending) as Promise; + let stop = (): void => {}; + const aborted = new Promise((_, reject) => { + stop = () => { + task?.cancel(); + reject(abortError(signal, what)); + }; + }); + if (signal.aborted) stop(); + else signal.addEventListener('abort', stop, { once: true }); + try { + return await Promise.race([promise, aborted]); + } catch (error) { + // A fetch given the signal rejects with the abort's reason before the abort event fires. + throw signal.aborted ? abortError(signal, what) : error; + } finally { + signal.removeEventListener('abort', stop); + task?.free(); + } +} + +/** + * The error for an aborted wait on `what`: `timeout` when `signal` timed out, else `cancelled` + * with the abort reason as its `cause`. + */ +function abortError(signal: AbortSignal, what: string): LatkitError { + const reason = signal.reason as { name?: string } | undefined; + return reason?.name === 'TimeoutError' + ? thrown({ code: 'timeout', message: `${what} timed out` }) + : thrown({ code: 'cancelled', message: `${what} was cancelled`, cause: reason }); +} diff --git a/npm/src/jobs.ts b/npm/src/jobs.ts index 46717f1d..d0bc6595 100644 --- a/npm/src/jobs.ts +++ b/npm/src/jobs.ts @@ -1,4 +1,11 @@ -/** Shared on-demand workers. Each receives the compiled module once and owns its WASM memory. */ +/** + * The pool of workers that run latkit's jobs, started on demand. Each worker receives the compiled + * module once and has its own WebAssembly memory. + */ + +import { thrown, type LatkitError } from './internal.js'; + +/** A worker's message: that it is ready, a job's bytes, or why it failed. */ interface Answer { ready?: boolean; bytes?: Uint8Array; @@ -8,7 +15,7 @@ interface Answer { interface Job { bytes: Uint8Array; resolve(bytes: Uint8Array): void; - reject(error: Error): void; + reject(error: LatkitError): void; } interface Slot { @@ -18,17 +25,25 @@ interface Slot { timer?: ReturnType; } +/** The error for a failed job: `cause` if it is a latkit error, else an `io` error from it. */ +function failure(cause: unknown): LatkitError { + if (cause instanceof Error && 'code' in cause) return cause as LatkitError; + const message = cause instanceof Error ? cause.message : String(cause); + return thrown({ code: 'io', message, cause: cause instanceof Error ? cause : undefined }); +} + /** - * Runs one job per worker. Idle workers retire after 30 seconds; startup and jobs have a one-minute - * deadline. Failure rejects the affected job and retires the worker; later jobs get a fresh one. + * Returns the function latkit hands its jobs to. It runs each job on a pooled worker, one job per + * worker at a time. latkit queues jobs itself and hands one over only when it can run, so the pool + * grows only to the number of jobs running at once and nothing queues here. Idle workers are + * terminated after 30 seconds. A job times out after one minute, counting a new worker's startup. + * A failure rejects that job and terminates its worker. Later jobs start a new one. */ export function jobs( module: WebAssembly.Module, - createWorker = () => new Worker(new URL('./worker.js', import.meta.url)), + worker = () => new Worker(new URL('./worker.js', import.meta.url)), ): (bytes: Uint8Array) => Promise { - const size = Math.max(1, Math.min(4, (globalThis.navigator?.hardwareConcurrency || 2) - 1)); - const queue: Job[] = []; - const slots = new Set(); + const idle = new Set(); const retire = (slot: Slot): void => { clearTimeout(slot.timer); @@ -36,14 +51,13 @@ export function jobs( slot.worker.onerror = null; slot.worker.onmessageerror = null; slot.worker.terminate(); - slots.delete(slot); + idle.delete(slot); }; const fail = (slot: Slot, cause: unknown): void => { const job = slot.job; retire(slot); - job?.reject(cause instanceof Error ? cause : new Error(String(cause))); - pump(); + job?.reject(failure(cause)); }; const send = (slot: Slot): void => { @@ -57,10 +71,8 @@ export function jobs( }; const start = (): Slot => { - const worker = createWorker(); - const slot: Slot = { worker, ready: false }; - slots.add(slot); - worker.onmessage = ({ data }: MessageEvent) => { + const slot: Slot = { worker: worker(), ready: false }; + slot.worker.onmessage = ({ data }: MessageEvent) => { if (!slot.ready) { if (!data.ready) return fail(slot, data.error ?? 'worker initialization failed'); slot.ready = true; @@ -72,41 +84,37 @@ export function jobs( slot.job = undefined; clearTimeout(slot.timer); slot.timer = setTimeout(() => retire(slot), 30_000); + idle.add(slot); job.resolve(data.bytes); - pump(); } }; - worker.onerror = (event) => { + slot.worker.onerror = (event) => { event.preventDefault(); fail(slot, event.message || 'worker failed'); }; - worker.onmessageerror = () => fail(slot, 'could not read worker message'); + slot.worker.onmessageerror = () => fail(slot, 'could not read worker message'); return slot; }; - function pump(): void { - while (queue.length) { - let slot = [...slots].find((candidate) => !candidate.job); - if (!slot && slots.size >= size) return; - const job = queue.shift()!; + return (bytes) => + new Promise((resolve, reject) => { + const job = { bytes, resolve, reject }; + let slot = idle.values().next().value; try { - slot ??= start(); + if (slot) idle.delete(slot); + else { + slot = start(); + slot.worker.postMessage(module); + } clearTimeout(slot.timer); slot.job = job; const active = slot; - slot.timer = setTimeout(() => fail(active, 'worker job timed out'), 60_000); - if (slot.ready) send(slot); - else slot.worker.postMessage(module); + const timedOut = () => thrown({ code: 'timeout', message: 'a worker job timed out' }); + slot.timer = setTimeout(() => fail(active, timedOut()), 60_000); + send(slot); } catch (error) { if (slot) retire(slot); - job.reject(error instanceof Error ? error : new Error(String(error))); + reject(failure(error)); } - } - } - - return (bytes) => - new Promise((resolve, reject) => { - queue.push({ bytes, resolve, reject }); - pump(); }); } diff --git a/npm/src/load.ts b/npm/src/load.ts index b5477c06..20575f94 100644 --- a/npm/src/load.ts +++ b/npm/src/load.ts @@ -1,63 +1,91 @@ -/** Options for the page's shared WASM runtime. Omitted options reuse its existing configuration. */ -export interface LoadOptions { - /** The matching package's WASM asset, or an already compiled module. Defaults to the bundled asset. */ - wasm?: string | URL | WebAssembly.Module; - /** Creates a worker running the standalone `latkit/worker` bundle. Defaults to the bundled worker. */ - createWorker?: () => Worker; -} +/** Loading latkit's WebAssembly. */ -/** - * The initialized data and rendering API. In a browser the page's first GPU is made as the runtime - * loads, and the first `Gpu.create()` takes it; later calls request devices of their own. - */ -export interface Latkit { - Data: typeof import('./runtime.js').Data; - Gpu: typeof import('./runtime.js').Gpu; - Shot: typeof import('./runtime.js').Shot; - colormapCss: typeof import('./runtime.js').colormapCss; -} +import init, * as bindings from '../pkg/latkit.js'; +import { encoder } from './encoder.js'; +import { loaded, thrown } from './internal.js'; +import { jobs } from './jobs.js'; -let configured: LoadOptions | undefined; -let loading: Promise | undefined; +let configured: { wasm?: string | WebAssembly.Module; worker?: () => Worker } | undefined; +let loading: Promise | undefined; + +/** The GPU device requested while loading, kept until the first `Gpu.create()` takes it. */ +let first: Promise | undefined; /** - * Loads the runtime once per page. Concurrent calls share one promise; a failed load can be retried. - * Options explicitly supplied after the first call must match that call. Importing the package alone - * neither initializes WASM nor requests a GPU or worker; loading asks for the page's GPU adapter - * while the WASM compiles. + * Loads latkit's WebAssembly, once per page. Later and concurrent calls share the first call's + * load. Options they pass must equal the first call's, or they reject with an `io` error. A failed + * load rejects with an `io` error and can be retried. While the WebAssembly compiles, `load()` + * requests a GPU adapter so the browser starts its GPU process early. Then it requests the first + * device for `Gpu.create()`. */ -export function loadLatkit(options: LoadOptions = {}): Promise { - // Snapshot URLs so a caller cannot change the configuration while it loads. +export function load( + options: { + /** + * The URL of this version's `latkit/wasm` asset, or that asset compiled. Defaults to the + * bundled asset. + */ + wasm?: string | URL | WebAssembly.Module; + /** + * Creates a worker that runs the standalone `latkit/worker` bundle. Defaults to the bundled + * worker. + */ + worker?: () => Worker; + } = {}, +): Promise { + // Store a URL as text, so the caller cannot change the configuration while it loads. const wasm = options.wasm instanceof URL ? options.wasm.href : options.wasm; if (loading) { if ( (wasm !== undefined && wasm !== configured?.wasm) || - (options.createWorker !== undefined && options.createWorker !== configured?.createWorker) - ) { + (options.worker !== undefined && options.worker !== configured?.worker) + ) return Promise.reject( - Object.assign(new Error('latkit is already loading or loaded with different options'), { - code: 'io', - }), + thrown({ code: 'io', message: 'latkit is already loading or loaded with other options' }), ); - } return loading; } - configured = { wasm, createWorker: options.createWorker }; + configured = { wasm, worker: options.worker }; startGpuProcess(); - loading = import('./runtime.js') - .then((runtime) => runtime.initialize(configured!)) - .catch((cause: unknown) => { - configured = undefined; - loading = undefined; - throw Object.assign(new Error('could not load latkit', { cause }), { code: 'io' }); - }); + loading = initialize(configured).catch((cause: unknown) => { + configured = undefined; + loading = undefined; + throw thrown({ code: 'io', message: 'could not load latkit', cause }); + }); return loading; } /** - * Asks for a GPU adapter as latkit's own request does, while the runtime loads: the browser starts - * its GPU process meanwhile, which takes most of a cold device's 280 ms, and latkit's request finds - * it running. + * Instantiates the bindings, installs the job workers and the video encoder where the browser has + * them, and requests the first GPU device. + */ +async function initialize(options: { wasm?: string | WebAssembly.Module; worker?: () => Worker }) { + await init(options.wasm === undefined ? undefined : { module_or_path: options.wasm }); + if (options.worker || typeof Worker === 'function') + bindings.setJobs(jobs(bindings.module() as WebAssembly.Module, options.worker)); + if (typeof VideoEncoder === 'function') bindings.setEncoder(encoder); + loaded(); + // Request the first device now, so drawing does not wait for it later. The device compiles + // nothing until a view is created. + if ((globalThis.navigator as { gpu?: unknown } | undefined)?.gpu) { + first = bindings.Gpu.create(); + first.catch(() => {}); + } +} + +/** + * @internal Returns the device requested while loading, once. Later calls return `undefined`, and + * `Gpu.create()` requests a new device. + */ +export function firstDevice(): Promise | undefined { + const device = first; + first = undefined; + return device; +} + +/** + * Requests a high-performance GPU adapter, like latkit's own request, while the WebAssembly loads. + * This starts the browser's GPU process early. Starting it takes most of a cold device's 280 ms, so + * latkit's later request finds the process running. */ function startGpuProcess(): void { const gpu = ( diff --git a/npm/src/outputs.ts b/npm/src/outputs.ts index 9741df92..0fa44eb1 100644 --- a/npm/src/outputs.ts +++ b/npm/src/outputs.ts @@ -1,26 +1,30 @@ -/** Written by `node npm/build.mjs schema` from `npm/schema.json`, what latkit writes: do not edit. */ +/** The types latkit writes, written by `node npm/build.mjs schema` from `npm/schema.json`. Do not edit. */ import type { Camera, Id, Row, Rows } from './inputs.js'; /** - * What a dataset holds: its tables, their rows and fields, and the axis its sampled fields are - * recorded over and span. + * A summary of a dataset: its tables with their rows and fields, the axis sampled fields are + * recorded over, and the part of the axis they cover. */ export interface Description { /** The dataset's name. */ name: string; - /** The axis's name and unit. */ + /** The axis name and unit, if set. */ axis: [string, string] | null; - /** The first and last coordinates its sampled fields hold. */ + /** The first and last axis coordinates held by any sampled field, or `null` if no field holds any. */ span: [number, number] | null; - /** Its tables, by name. */ + /** Its tables, sorted by name. */ tables: TableDescription[]; } -/** What the user acts with. */ -export type Device = 'mouse' | 'pen' | 'touch' | 'keyboard'; +/** The input device. */ +export type Device = + | /** A mouse or a trackpad. */ 'mouse' + | /** A pen. */ 'pen' + | /** A finger. Picks find rows up to 8 CSS pixels farther away. */ 'touch' + | /** The keyboard. */ 'keyboard'; -/** A vertex a connection ends at, and its port when it has the one the connection meets. */ +/** A vertex a connection ends at, and the port the connection meets, if any. */ export interface End { /** The vertex. */ row: Row; @@ -28,25 +32,25 @@ export interface End { port?: string; } -/** - * What failed: its `code`, the message, and for invalid input the `path` to the part, such as - * `marks[2].vertices.color`. - */ +/** A failure as JSON, in the same form as the npm package's `LatkitError`. */ export interface Failure { + /** The kind of failure. See `LatkitError.code`. */ code: 'invalid' | 'unsupported' | 'limit' | 'io' | 'device-lost' | 'cancelled' | 'timeout'; + /** A description of what failed. */ message: string; + /** For `invalid` failures, the path of the invalid input, such as `marks[2].color`. */ path?: string; } -/** A field described: what it holds, the table a reference names, the coordinates samples span. */ +/** A summary of one field: its kind, plus its target table or axis span where those apply. */ export interface FieldDescription { /** The field's name. */ name: string; - /** What it holds. */ + /** The kind of values it holds. */ kind: Kind; - /** The table a reference names. */ + /** For a reference, the target table. */ to?: string; - /** The first and last coordinates a sampled field holds. */ + /** For a sampled field, the first and last axis coordinates it holds. */ span?: [number, number]; } @@ -54,28 +58,36 @@ export interface FieldDescription { export interface Hit { /** The row. */ row: Row; - /** The name of the mark that drew it, when it has one: what tells two marks of one table apart. */ + /** The name of the mark that drew it, if the mark has one. It tells apart two marks of one table. */ mark?: string; /** - * The point of the row nearest the probe, CSS pixels from the top left: a vertex's or a port's - * middle, the nearest point of a stroke or a triangle. + * The row's point nearest the probe, in CSS pixels from the top left: a vertex's or port's middle, + * or the nearest point of a stroke or triangle. */ point: [number, number]; /** That point in the space's coordinates: x, y, and elevation. */ position: [number, number, number]; /** CSS pixels from the probe to the row. */ distance: number; - /** The port hit, when a vertex's port was. */ + /** The port hit, if the hit is on a vertex's port. */ port?: string; - /** The sample hit, when a line tracing a sampled field was. */ + /** The sample hit, if the hit is on a line tracing a sampled field. */ sample?: Sample; } -/** What a field holds. */ +/** The kind of values a field holds. */ export type Kind = - 'number' | 'bool' | 'text' | 'point' | 'path' | 'parts' | 'polygons' | 'reference' | 'sampled'; + | /** Numbers. */ 'number' + | /** Booleans. */ 'bool' + | /** Texts. */ 'text' + | /** `[x, y]` points. */ 'point' + | /** Paths of points. */ 'path' + | /** Parts: the rings of a polygon, or the paths of a multi-path. */ 'parts' + | /** Polygons of rings. */ 'polygons' + | /** Rows of another table, by id. */ 'reference' + | /** Numbers sampled over the dataset's axis. */ 'sampled'; -/** Keys held during an input. */ +/** Modifier keys held during an input. */ export interface Modifiers { /** Shift. */ shift: boolean; @@ -88,7 +100,10 @@ export interface Modifiers { } /** The phase of a `Drag`. */ -export type Phase = 'start' | 'move' | 'end'; +export type Phase = + | /** The pointer moved far enough to drag. */ 'start' + | /** The pointer moved. */ 'move' + | /** The pointer was released, or the drag was cancelled. */ 'end'; /** A sample of a traced field: its frame, the axis coordinate it was recorded at, and its value. */ export interface Sample { @@ -100,68 +115,57 @@ export interface Sample { value: number; } -/** What a view does each frame. */ +/** Counters describing a view's frames. */ export interface Stats { /** Frames rendered. */ frames: number; /** Rows of every mark's table. */ rows: number; /** - * Rows every mark draws: every row of a table drawn whole, the rows of the tiles in view of one - * drawn through tiles. + * Rows the marks draw: every row of a fully drawn table, and the rows in the tiles in view of a + * paged table. */ drawn: number; - /** Draw calls the last frame recorded, shadows and labels with the marks. */ + /** Draw calls the last frame recorded, label draws included. */ draws: number; /** - * Instances the last frame's draws drew: points, segments, triangles, glyphs; at most, for - * segments the GPU culls. + * Instances the last frame's draws drew: points, segments, triangles, and glyphs. An upper bound + * for segments, which the GPU may cull. */ instances: number; - /** Fetched tiles the view draws, of every tile source. */ + /** Fetched tiles the view draws, across all tile sources. */ tiles: number; - /** Tiles being fetched or decoded, of every tile source. */ + /** Tiles being fetched or decoded, across all tile sources. */ tilesLoading: number; - /** Bytes of data waiting to upload. */ - uploading: number; - /** Bytes of data pages on the GPU, for every view of its `Gpu`. */ - resident: number; - /** Bytes the GPU's `Budget` lets data take. */ - budget: number; - /** - * The work under way the last frame stood in for; nothing when it drew everything as the data, - * scene, and camera say: see `View::complete`. - */ - waiting: Waiting; /** Milliseconds the last frame took to record on the CPU. */ cpuMs: number; /** - * Milliseconds the GPU takes on a frame: the median of the last frames it measured; 0 where the - * device cannot measure. + * Milliseconds the GPU takes per frame: the median of recently measured frames. 0 if the device + * cannot measure GPU time. */ gpuMs: number; } -/** A table described: its rows, and its fields by name. */ +/** A summary of one table: its row count and its fields, sorted by name. */ export interface TableDescription { /** The table's name. */ name: string; - /** How many rows it has. */ + /** The number of rows. */ rows: number; - /** Its fields, by name. */ + /** Its fields, sorted by name. */ fields: FieldDescription[]; } /** - * Where, with what keys, and with what the user acted. For keys, the point is the row they point - * at, else the pointer, else the view's middle. + * Where the user acted, with which modifier keys and device. For keyboard input, the point is the + * row the keys point at, or else the pointer, or else the view's middle. */ export interface Trigger { /** CSS pixels from the top left. */ point: [number, number]; - /** The keys held. */ + /** The modifier keys held. */ modifiers: Modifiers; - /** What acted. */ + /** The device used. */ device: Device; } @@ -171,9 +175,9 @@ export interface Trigger { */ export type Value = number | boolean | string | null | [number, number] | unknown[][]; -/** What happened in a view, for the application to act on. Drain them with `View::events`. */ +/** Something that happened in a view, as `view.on` reports it. */ export type ViewEvent = - | { + | /** A row was clicked. */ { type: 'click'; detail: { /** The row. */ @@ -182,7 +186,7 @@ export type ViewEvent = trigger: Trigger; }; } - | { + | /** A row was double-clicked, or Enter was pressed on the row the keyboard points at. */ { type: 'open'; detail: { /** The row. */ @@ -191,16 +195,22 @@ export type ViewEvent = trigger: Trigger; }; } - | { + | /** + * A context menu was requested: by a secondary click, the menu key, or a touch or pen held still. + * Lists the rows under the trigger's point, for a menu of actions on them. + */ { type: 'menu'; detail: { - /** What lies under the point, nearest first. */ + /** The rows under the point, nearest first. */ hits: Hit[]; /** How the user asked. */ trigger: Trigger; }; } - | { + | /** + * A row of a draggable mark is dragged. The drag starts once the pointer moves more than 4 CSS + * pixels from where it pressed the row. + */ { type: 'drag'; detail: { /** Whether the drag starts, moves, or ends. */ @@ -208,29 +218,32 @@ export type ViewEvent = /** The row dragged, as it was hit. */ hit: Hit; /** - * Where the row goes, in the space's coordinates: where it was hit, moved as the pointer moved - * since the press, at the depth the drag started; a vertex keeps where it was grabbed. On a - * diagram's dots when its grid snaps. + * The row's new position in the space's coordinates. It is the hit point moved with the pointer + * since the press, at the depth where the drag started, so a vertex keeps its offset from the + * pointer. Snapped to a diagram's grid dots when the grid snaps. */ position: [number, number, number]; /** The pointer. */ trigger: Trigger; }; } - | { + | /** + * A connection was drawn with `controls.connect` on: from a port, from a vertex with Alt held, or + * by dragging an edge's end off it. + */ { type: 'connect'; detail: { - /** The end it was drawn from: for an edge, the end that stayed. */ + /** The end it was drawn from. For a dragged edge end, the edge's other end. */ from: End; /** The vertex it was released over, if any. */ to: End | null; - /** The edge whose other end was dragged, which `to` replaces. */ + /** The edge whose end was dragged off. `to` replaces that end. */ edge: Row | null; /** Where it was released. */ trigger: Trigger; }; } - | { + | /** Delete or Backspace was pressed with rows selected and `controls.delete` on. */ { type: 'delete'; detail: { /** The rows selected. */ @@ -239,65 +252,97 @@ export type ViewEvent = trigger: Trigger; }; } - | { - type: 'hover'; + | /** The hovered row changed: `view.hovered`. */ { + type: 'hovered'; detail: { - /** The row now pointed at; `null` once it left every row. */ + /** The row now hovered, or `null` if no row is. */ hit: Hit | null; /** The pointer, or the keys. */ trigger: Trigger; }; } - | { - type: 'select'; + | /** The selection changed: `view.selection`. */ { + type: 'selection'; detail: { /** The rows selected now. */ rows: Rows[]; - /** How the user changed it; `null` when the data dropped rows from it. */ + /** How the user changed it, or `null` when the data dropped rows from it. */ trigger: Trigger | null; }; } - | { type: 'camera'; detail: Camera } - | { type: 'waiting'; detail: Waiting } - | { type: 'error'; detail: Failure }; + | /** + * The camera moved: `view.camera`. A gesture, fit, flight, coast, or orbit moved it, or it moved + * back within its zoom and pan limits after the view's size, insets, or space changed. + */ { type: 'camera'; detail: Camera } + | /** The work the last frame waited on changed: `view.waiting`. Empty once frames are complete. */ { + type: 'waiting'; + detail: Waiting; + } + | /** + * Something failed. Reported once. A mark that no longer fits its data is also listed in + * `view.failures`. + */ { type: 'error'; detail: Failure }; /** - * Work under way a frame stands in for: pipelines the scene draws with compiling, until which the - * view draws nothing new (`compiling`); pages uploading or tables not yet built from them - * (`data`); layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); a label placement or - * texts not yet set (`labels`); a fit due or being measured (`fit`); row regions laid out again, - * segment counts, or a transition beginning (`marks`). Empty once a frame is complete. + * What a frame still waits on: pipelines the scene draws with, which compile before the view draws + * anything new (`compiling`); pages uploading, or tables not yet built from them (`data`); + * layouts, routes, lanes, shapes, tiles, or pyramids (`derived`); label texts being read back, or + * not yet set (`labels`); a fit due or being measured (`fit`); segment counts, or a transition + * starting (`marks`). Empty once the frame is complete. */ export type Waiting = ('compiling' | 'data' | 'derived' | 'labels' | 'fit' | 'marks')[]; -/** What a `Gpu` did since it was made; subtract two for what happened between. */ +/** + * Counts of the work a `Gpu` has done since it was created. Subtract two snapshots to get the work + * between them. + */ export interface Work { /** Bytes of data pages uploaded. */ uploaded: number; - /** Bytes written besides pages: uniforms, programs, label runs, the selection, textures. */ + /** + * Data pages, or pieces of pages, uploaded. A change's uploads count pages here and bytes in + * `uploaded`. + */ + uploads: number; + /** Bytes written other than pages: uniforms, programs, label runs, the selection, and textures. */ written: number; /** Rows whose channels the GPU evaluated. */ evaluated: number; /** - * Reads back from the GPU: probes, queries, fits, label placements, segment counts, pictures; not - * the frame timer's, which only some devices have. + * Edge and face rows whose endpoints the GPU resolved to resident vertex rows. These are rows that + * changed, plus every row of a page that references a vertex now resident elsewhere. + */ + resolved: number; + /** + * Bytes of per-row word pages the GPU cleared: pages read before anything writes them, and the + * normals that faces sum from scratch. + */ + cleared: number; + /** + * Rows packed into pages of a paged table's draw order. Each page is packed once, the first time a + * view draws it, and again only when a write changes its rows. + */ + packed: number; + /** + * Readbacks from the GPU: probes, queries, fits, label texts, segment counts, and pictures. Frame + * timer readbacks, which only some devices support, are not counted. */ readbacks: number; /** - * Jobs handed out: layouts, routes, lanes, triangulations, pyramids, tiles; to the host's workers, - * or done at once without a host. + * Jobs started: layouts, routes, lanes, triangulations, pyramids, and tiles. They run on the + * host's workers, or immediately without a host. */ jobs: number; - /** Label texts the host was asked to set. */ + /** Label texts the host was asked to rasterize. */ texts: number; /** - * Pipelines compiled: in a browser, each as a view or a shot asks for it ahead, off the GPU's - * thread; natively, where it is first made. + * Pipelines compiled. In a browser, each compiles in the background when a view or shot requests + * it ahead of use. Natively, each compiles when first created. */ pipelines: number; /** - * Pipelines made without being asked for ahead, each compiled where it is first used: in a browser - * on the GPU's thread, holding frames back. None, unless a scene uses what it did not ask for. + * Pipelines not requested ahead of use, compiled when first used. In a browser they compile on the + * GPU's thread and delay frames. Zero unless a scene uses a pipeline it did not request. */ late: number; } diff --git a/npm/src/patch.ts b/npm/src/patch.ts index 6b571a71..284a3c65 100644 --- a/npm/src/patch.ts +++ b/npm/src/patch.ts @@ -1,20 +1,40 @@ /** - * Patches as latkit's bytes, written here without WebAssembly: a page, a worker, or a server builds - * them alike. The layout is `crates/latkit/src/data/bytes.rs`'s: a frame of `LATK`, its length, the - * dataset it changes, and its operations, little endian throughout. + * Patches, written directly as latkit's bytes without WebAssembly so a page, worker, or server can + * build them. The format matches `crates/latkit/src/data/bytes.rs`: a frame with a + * 12-byte header (the magic `LATK` and the frame's length), the name of the dataset it changes, an + * operation count, and the operations, all little endian. */ import type { Id, Ids } from './inputs.js'; +import { + bytesOf, + chunks, + thrown, + waited, + wasm, + type ByteSource, + type ReadableBytes, + type Wait, +} from './internal.js'; +import { load } from './load.js'; -/** Row ids as a patch takes them: listed, `{ count }` for rows `0..count`, or a typed array. */ +/** + * Row ids: a list of numbers or strings, `{ count }` for ids `0` to `count - 1`, or a `Uint32Array` + * or `Float64Array` of numbers. + */ export type RowIds = Ids | readonly (Id | null)[] | Uint32Array | Float64Array; -/** A point `[x, y]`, or a list of geometries one level shallower: a path, its parts, or polygons. */ +/** + * A point `[x, y]`, or a nested list of points: a path (a list of points), parts (a list of paths, + * such as a polygon's rings), or polygons (a list of parts). + */ export type Geometry = readonly number[] | readonly Geometry[]; /** - * One field's values, a value per row; `null` is missing. Typed arrays copy at once: numbers, or - * in a `Uint8Array` booleans, nonzero true. Booleans, texts, and geometries also as arrays. + * One field's values, one per row, with `null` for a missing value. Numbers come as a typed array + * or an array. A `Uint8Array` holds booleans, where nonzero is true. Booleans, strings, and + * geometries also come as arrays. Values are copied when passed, so the arrays can be reused right + * away. */ export type Values = | Float64Array @@ -27,97 +47,138 @@ export type Values = | readonly (string | null)[] | readonly (Geometry | null)[]; -/** A field naming rows of another table by id: `null` leaves a row unwired. */ +/** A reference field: for each row, a row of another table. */ export interface Reference { + /** The table referenced. */ reference: string; + /** For each row, the id of the row it references, or `null` to reference none. */ ids: RowIds; } -/** Values of a field naming rows of `table` by id. */ +/** Builds a {@link Reference} field: for each row, the id of the row it references in `table`. */ export function reference(table: string, ids: RowIds): Reference { return { reference: table, ids }; } -/** A table's fields by name. */ +/** A table's fields by name: values, or a {@link Reference}. */ export type Fields = Readonly>; -/** A byte stream, structurally compatible with the browser and Node ReadableStream. */ -export interface ReadableBytes { - getReader(): { - read(): Promise<{ done: boolean; value?: Uint8Array }>; - cancel(reason?: unknown): Promise; - releaseLock(): void; - }; +/** A run of entries of a sampled field at one level of detail, for {@link Patch.frames}. */ +export interface Run { + /** The level of detail, from 0 to 21. Defaults to 0. */ + level?: number; + /** The index of the first entry, a safe nonnegative integer. */ + first: number; + /** + * Each frame's coordinate. At a bucket level, the coordinates of each bucket's first and last + * frames. + */ + at: Float64Array | readonly number[]; + /** + * One value per row for each entry, entry by entry. At a bucket level, the per-row minimums of + * every bucket, then the per-row maximums of every bucket. NaN marks a missing value. + */ + values: Float32Array | readonly number[]; } const MAGIC = [0x4c, 0x41, 0x54, 0x4b]; // LATK -/** The magic and the frame's length. */ -const HEADER = 16; -const VERSION = 1; +/** Header bytes: the magic and the frame's length. */ +const HEADER = 12; +/** The longest frame read by default, in bytes: 256 MiB. */ +const LIMIT = 1 << 28; const OP = { axis: 0, replace: 1, set: 2, remove: 3, append: 4, forget: 5, frames: 6 } as const; -/** Levels of detail kept above frames. */ +/** The highest level of detail; level 0 is frames. */ const LEVELS = 21; const FIELD_VALUES = 0; const FIELD_REFERENCE = 1; const STORE = { f64: 0, f32: 1, i32: 2, u32: 3, u64: 4, bool: 5, text: 6, geometry: 7 } as const; -/** The largest whole number a JavaScript number holds exactly. */ +/** The largest integer a JavaScript number represents exactly, 2^53 - 1. */ const SAFE = Number.MAX_SAFE_INTEGER; const utf8 = new TextEncoder(); +/** @internal Returns `patch`'s frame without copying it. Read it before the patch changes again. */ +export let framed: (patch: Patch) => Uint8Array; + /** - * Changes to a dataset, applied in order and all at once by `Data.apply`: plain data, written as - * latkit's bytes as it is built. + * A list of changes to a dataset. `Data.apply` applies them in order, all or none. The patch is + * written as latkit's bytes as it is built, without WebAssembly. Each method adds a change and + * returns the patch. * * ```ts - * const patch = Patch.to('grid') + * const patch = new Patch('grid') * .replace('Bus', [101, 102], { x: Float64Array.of(0, 10), name: ['North', 'South'] }) - * .append(0.5, 'Bus', { vm: Float32Array.of(1.01, 0.99) }); + * .append('Bus', 0.5, { vm: Float32Array.of(1.01, 0.99) }); * ``` */ export class Patch { - /** The dataset the patch changes, when it names one; `Data.apply` refuses it for another. */ - readonly data: string | undefined; + static { + framed = (patch) => patch.#frame(); + } + + #data: string | undefined; #writer: Writer; #count = 0; - /** Where the operations start: after the header, the name, and their count. */ + /** Byte offset of the first operation: after the header, dataset name, and operation count. */ #ops: number; - /** Appends of float frames to the same fields of every row of one table, waiting to be written as - * one: a field records its frames in one write, however many calls made them. */ + /** + * Unwritten consecutive appends of float frames to the same fields of every row of one table. + * They are written as one append, so each field records all their frames in one write. + */ #run: Appends | undefined; - /** An empty patch, for whichever dataset it is applied to. */ - constructor(data?: string); - /** @internal */ - constructor(data: string, owned: Uint8Array); - constructor(data?: string, owned?: Uint8Array) { - this.data = data || undefined; - this.#writer = new Writer(owned); - if (owned) { - this.#ops = 0; - return; - } + /** Creates an empty patch for the dataset named `data`, or for any dataset when omitted. */ + constructor(data?: string) { + this.#data = data || undefined; + this.#writer = new Writer(); this.#writer.bytes(MAGIC); - this.#writer.u16(VERSION); - this.#writer.u16(0); this.#writer.zeros(8); - this.#writer.str(this.data ?? ''); + this.#writer.str(this.#data ?? ''); this.#writer.u32(0); this.#ops = this.#writer.length; } - /** An empty patch to dataset `data`. */ - static to(data: string): Patch { - return new Patch(data); + /** + * The name of the dataset this patch is for, if any. `Data.apply` rejects it for a dataset with + * another name. + */ + get data(): string | undefined { + return this.#data; } - /** The patch `bytes` hold: one frame, as `toBytes` or a stream writes it. */ - static fromBytes(bytes: Uint8Array): Patch { - return Patch.#fromOwnedFrame(bytes.slice()); + /** + * Reads a patch from `source`: latkit's bytes (one frame, as `toBytes()` writes it), or JSON that + * any language can write without latkit. The JSON is a list of changes. Each change is keyed by + * its method name and holds the method's parameters by name, such as + * `{"set": {"table": "Bus", "ids": [1], "fields": {"kv": [345]}}}`. A field may be a slice of a + * binary file, `{file, dtype, offset, count}`, which is fetched right away. Bytes are read + * without WebAssembly. + */ + static async read( + source: ByteSource | object, + options: { + /** The URL that the JSON's file slices are relative to. Defaults to the JSON's URL. */ + base?: string; + /** + * The longest frame in bytes. Defaults to 256 MiB. A longer frame fails with a `limit` error. + */ + limit?: number; + } & Wait = {}, + ): Promise { + const { bytes, base } = await bytesOf(source, options); + if (MAGIC.every((byte, i) => bytes[i] === byte)) + return Patch.#owning(bytes.slice(), options.limit); + await load(); + const read = wasm().readPatch(bytes, options.base ?? base); + return Patch.#owning(await waited(read, options, 'reading a patch'), options.limit); } - static #fromOwnedFrame(bytes: Uint8Array): Patch { + /** + * Wraps frame `bytes` as a patch without copying them. Throws unless they are exactly one valid + * frame of at most `limit` bytes. + */ + static #owning(bytes: Uint8Array, limit = LIMIT): Patch { const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - const length = frameLength(bytes, Infinity); + const length = frameLength(bytes, limit); if (length === undefined || length !== bytes.length) throw malformed('not one whole frame'); const nameLength = bytes.length >= HEADER + 4 ? view.getUint32(HEADER, true) : -1; const start = HEADER + 4 + nameLength + 4; @@ -128,19 +189,21 @@ export class Patch { bytes.subarray(HEADER + 4, start - 4), ); } catch { - throw malformed('text is not UTF-8'); + throw malformed('the dataset name is not UTF-8'); } - const patch = new Patch(name, bytes); + const patch = new Patch(); + patch.#data = name || undefined; + patch.#writer = new Writer(bytes); patch.#count = view.getUint32(start - 4, true); patch.#ops = start; return patch; } /** - * The patches arriving back to back in `stream`, each as soon as its last byte does: a response - * body, a file, or a socket's messages. Each frame says its length up front, so a patch longer - * than `limit` bytes (256 MiB unless given) throws before allocating its body. The parser holds at most one frame, in addition to - * the input chunk and patches the caller retains. Breaking iteration cancels the reader. + * Yields each patch in `stream` as soon as its last byte arrives. The stream carries frames back + * to back, such as a response body, a file, or a socket's messages. At most one frame is + * buffered, besides the current chunk and any patches the caller keeps. Breaking out of the loop + * cancels the stream. Throws an `invalid` error if the stream ends inside a frame. * * ```ts * for await (const patch of Patch.stream((await fetch('/feed')).body!)) data.apply(patch); @@ -148,10 +211,17 @@ export class Patch { */ static async *stream( stream: ReadableBytes | AsyncIterable, - { limit = 1 << 28 }: { limit?: number } = {}, + options: { + /** + * The longest frame in bytes, at least 12. Defaults to 256 MiB. Each frame states its length + * up front, so a longer frame fails with a `limit` error before its body is allocated. + */ + limit?: number; + } = {}, ): AsyncGenerator { + const { limit = LIMIT } = options; if (!Number.isSafeInteger(limit) || limit < HEADER) - throw new RangeError(`limit must be a safe integer of at least ${HEADER} bytes`); + throw invalid('limit', `a whole number of at least ${HEADER} bytes, not ${limit}`); let buffer = new Uint8Array(HEADER); let used = 0; let length: number | undefined; @@ -169,7 +239,7 @@ export class Patch { buffer = frame; } if (used === length) { - const patch = Patch.#fromOwnedFrame(buffer); + const patch = Patch.#owning(buffer, limit); buffer = new Uint8Array(HEADER); used = 0; length = undefined; @@ -181,20 +251,9 @@ export class Patch { } /** - * The patch its JSON makes: a list of changes, each named by its method with its parameters by - * name, such as `{"set": {"table": "Bus", "ids": [1], "fields": {"kv": [345]}}}`. Slices of - * binary files, `{file, dtype, offset, count}`, are fetched at once relative to `base`, the - * document's own URL. Reading JSON takes latkit's WebAssembly, which this loads as - * `loadLatkit()` does. + * Sets the name and unit of the axis that sampled fields are recorded over, such as + * `axis('time', 's')`. */ - static async load(json: string, base = ''): Promise { - const { loadLatkit } = await import('./load.js'); - await loadLatkit(); - const { readPatch } = await import('./runtime.js'); - return Patch.fromBytes(await readPatch(json, base)); - } - - /** Names the axis sampled fields are recorded over, such as `('time', 's')`. */ axis(name: string, unit: string): this { return this.#op(OP.axis, (w) => { w.str(name); @@ -202,17 +261,25 @@ export class Patch { }); } - /** Replaces table `table` with these rows and fields, or adds it. */ - replace(table: string, ids: RowIds, fields: Fields = {}): this { - return this.#op(OP.replace, (w) => writeTable(w, table, ids, fields)); + /** + * Replaces table `table` with rows `ids` and `fields`, adding the table if it does not exist. + * Without ids, the rows are `0` to `n - 1`, where `n` is the length of each field's values. + */ + replace(table: string, fields: Fields): this; + replace(table: string, ids: RowIds, fields?: Fields): this; + replace(table: string, ids: RowIds | Fields, fields?: Fields): this { + return this.#op(OP.replace, (w) => writeTable(w, table, ...rows(ids, fields))); } /** - * Writes `fields` for rows `ids` of `table`: rows with known ids change, others are added, and - * fields left out keep their values. Adds the table when there is none of its name. + * Writes `fields` for rows `ids` of `table`. Rows with existing ids change, new ids add rows, and + * omitted fields keep their values. Adds the table if it does not exist. Without ids, the rows + * are `0` to `n - 1`, where `n` is the length of each field's values. */ - set(table: string, ids: RowIds, fields: Fields = {}): this { - return this.#op(OP.set, (w) => writeTable(w, table, ids, fields)); + set(table: string, fields: Fields): this; + set(table: string, ids: RowIds, fields?: Fields): this; + set(table: string, ids: RowIds | Fields, fields?: Fields): this { + return this.#op(OP.set, (w) => writeTable(w, table, ...rows(ids, fields))); } /** Removes the rows of `table` with `ids`. */ @@ -224,17 +291,21 @@ export class Patch { } /** - * Records a frame at each axis coordinate of `at`: for each field, a value per row at every - * coordinate, frame-major; of every row of `table`, or of rows `ids`, the rest holding their last - * values. A field's first frame makes it sampled. Appends of float arrays to the same fields of - * every row, one after another, write as one change, so a frame a call costs no more than one call. + * Records a frame of `table` at each axis coordinate in `at`. Each field gives one value per row + * for each frame, frame by frame. A field becomes a sampled field at its first frame. Consecutive + * appends of typed float arrays to the same fields of every row are written as one change, so + * appending one frame per call costs no more than one call with every frame. */ append( - at: number | readonly number[], table: string, + at: number | readonly number[], fields: Readonly>, - ids?: RowIds, + options: { + /** The rows the frames cover. Defaults to every row. Other rows keep their last values. */ + ids?: RowIds; + } = {}, ): this { + const { ids } = options; const frames = typeof at === 'number' ? [at] : at; const entries = Object.entries(fields); const floats = @@ -266,29 +337,22 @@ export class Patch { return this.#op(OP.append, (w) => writeAppend(w, table, ids, frames, entries)); } - /** Forgets every sampled field's frames before axis coordinate `before`. */ + /** Drops every sampled field's frames before axis coordinate `before`. */ forget(before: number): this { return this.#op(OP.forget, (w) => w.f64(before)); } /** - * Holds a run of sampled field `field`'s entries of level `level`, numbered from `first`: what a - * time tile holds, as `latkit tile` writes them. At level 0 the entries are frames, each a - * coordinate in `at` and a value per row of `table` in `values`. At level `k` they are buckets of - * `8^k` frames, each its first and last frame's coordinates in `at`, and in `values` each row's - * least value over each bucket, then each row's greatest. NaN is missing. No entry may be held - * already; frames past the field's widen it. + * Stores a run of entries of sampled field `field` in `table`, at one level of detail. This is + * what a time tile from `latkit tile` holds. At level 0 each entry is a frame. At level `k` each + * entry is a bucket of `8^k` frames. None of the entries may already be stored. Entries past the + * field's last frame extend it. */ - frames( - table: string, - field: string, - { level = 0, first }: { level?: number; first: number }, - at: Float64Array | readonly number[], - values: Float32Array | readonly number[], - ): this { + frames(table: string, field: string, run: Run): this { + const { level = 0, first, at, values } = run; const path = `${table}.${field}`; if (!Number.isInteger(level) || level < 0 || level > LEVELS) - throw invalid(path, `levels run from 0 to ${LEVELS}`); + throw invalid(path, `the level must be a whole number from 0 to ${LEVELS}`); const per = level === 0 ? 1 : 2; const coordinates = Float64Array.from(at); const numbers = values instanceof Float32Array ? values : Float32Array.from(values); @@ -297,7 +361,7 @@ export class Patch { if (!Number.isInteger(entries) || !Number.isInteger(rows)) throw invalid( path, - `${coordinates.length} coordinates and ${numbers.length} values make no run`, + `${coordinates.length} coordinates and ${numbers.length} values do not divide evenly`, ); if ( !Number.isSafeInteger(first) || @@ -317,7 +381,7 @@ export class Patch { }); } - /** Appends `next`'s changes, to this patch's dataset. */ + /** Appends `next`'s changes to this patch, which keeps its own dataset name. */ extend(next: Patch): this { this.#flush(); next.#flush(); @@ -326,22 +390,22 @@ export class Patch { return this; } - /** The patch as bytes: one frame, naming its dataset when it has one. */ + /** Returns a copy of the patch as latkit's bytes: one frame, naming the dataset if it has one. */ toBytes(): Uint8Array { - return this.frame().slice(); + return this.#frame().slice(); } - /** @internal The frame, without copying it: read at once, before the patch changes. */ - frame(): Uint8Array { + /** Completes the header and returns the frame uncopied. Read it before the patch changes. */ + #frame(): Uint8Array { this.#flush(); const view = this.#writer.view(); const data = new DataView(view.buffer, view.byteOffset, view.byteLength); - data.setBigUint64(8, BigInt(view.length), true); + data.setBigUint64(MAGIC.length, BigInt(view.length), true); data.setUint32(this.#ops - 4, this.#count, true); return view; } - /** Writes the appends waiting as one, before anything after them. */ + /** Writes the pending merged appends as one operation, before any later operation. */ #flush(): void { const run = this.#run; if (!run) return; @@ -350,7 +414,7 @@ export class Patch { this.#op(OP.append, (w) => writeAppend(w, run.table, undefined, run.at, entries)); } - /** Writes an operation, or nothing when `body` throws. */ + /** Writes an operation tagged `tag`. If `body` throws, removes what it wrote and rethrows. */ #op(tag: number, body: (w: Writer) => void): this { this.#flush(); const mark = this.#writer.length; @@ -368,14 +432,14 @@ export class Patch { type Floats = Float64Array | Float32Array; -/** Frames of the same fields of every row of a table: coordinates, and each field's values. */ +/** Pending merged appends to one table: the frames' coordinates and each field's value chunks. */ interface Appends { table: string; at: number[]; fields: { name: string; parts: Floats[] }[]; } -/** `parts` one after another, in the first's storage. */ +/** Concatenates `parts` into one array of the first part's type. */ function joined(parts: readonly Floats[]): Floats { if (parts.length === 1) return parts[0]!; const length = parts.reduce((n, p) => n + p.length, 0); @@ -389,7 +453,10 @@ function joined(parts: readonly Floats[]): Floats { return out; } -/** An append: its table, its rows or every row, its frames' coordinates, and its fields' values. */ +/** + * Writes an append's body: the table, its rows or a flag for every row, the frames' coordinates, + * and each field's values. + */ function writeAppend( w: Writer, table: string, @@ -412,10 +479,10 @@ function writeAppend( } } -/** A table: its name, its ids, and its fields, each values or a reference. */ +/** Writes a table's body: its name, its ids, and its fields, each values or a reference. */ function writeTable(w: Writer, table: string, ids: RowIds, fields: Fields): void { const entries = Object.entries(fields); - // Room for the typed columns at once, rather than doubling past them. + // Reserve room for every typed-array column up front, instead of doubling the buffer repeatedly. const typed = (v: unknown) => (ArrayBuffer.isView(v) ? v.byteLength : 0); w.room(entries.reduce((n, [, f]) => n + typed(isReference(f) ? f.ids : f), typed(ids))); w.str(table); @@ -439,37 +506,62 @@ function isReference(field: Values | Reference): field is Reference { return !ArrayBuffer.isView(field) && !Array.isArray(field) && 'reference' in field; } -/** The length a frame's header says, once `bytes` hold it; an error for what is no frame. */ +/** + * Returns the ids and fields of `replace` or `set`, given as `(ids, fields)` or as `(fields)`. With + * `(fields)` the rows are `0` to `n - 1`, where `n` is the length of the first field's values. + */ +function rows(ids: RowIds | Fields, fields: Fields | undefined): [RowIds, Fields] { + if (fields !== undefined || isIds(ids)) return [ids as RowIds, fields ?? {}]; + const first = Object.values(ids)[0]; + return [{ count: first ? length(isReference(first) ? first.ids : first) : 0 }, ids]; +} + +/** Whether `x` is row ids rather than fields: a list, or `{ count }`, since no field is a number. */ +function isIds(x: RowIds | Fields): x is RowIds { + return ( + Array.isArray(x) || + ArrayBuffer.isView(x) || + typeof (x as { count?: unknown }).count === 'number' + ); +} + +/** How many values or ids `x` holds. */ +function length(x: Values | RowIds): number { + return 'length' in x ? x.length : x.count; +} + +/** + * Returns the frame length from the header, or `undefined` until `bytes` hold a whole header. + * Throws for bytes that are not a frame, or a frame longer than `limit`. + */ function frameLength(bytes: Uint8Array, limit: number): number | undefined { for (let i = 0; i < Math.min(bytes.length, 4); i++) if (bytes[i] !== MAGIC[i]) throw malformed('not latkit bytes'); if (bytes.length < HEADER) return undefined; const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - if (view.getUint16(4, true) !== VERSION) throw malformed('unsupported LATK version'); - if (view.getUint16(6, true) !== 0) throw malformed('unsupported LATK flags'); - const encoded = view.getBigUint64(8, true); + const encoded = view.getBigUint64(MAGIC.length, true); if (encoded > BigInt(Number.MAX_SAFE_INTEGER)) throw malformed('frame length is not a safe integer'); const length = Number(encoded); if (length > limit) - throw Object.assign(new Error(`a patch longer than ${limit} bytes`), { code: 'limit' }); - if (length < HEADER) throw malformed('a frame shorter than its header'); + throw thrown({ code: 'limit', message: `the patch is longer than ${limit} bytes` }); + if (length < HEADER) throw malformed('the frame is shorter than its header'); return length; } function malformed(reason: string): Error { - return Object.assign(new Error(`bytes: ${reason}`), { code: 'invalid', path: 'bytes' }); + return invalid('bytes', reason); } function invalid(path: string, reason: string): Error { - return Object.assign(new Error(`${path}: ${reason}`), { code: 'invalid', path }); + return thrown({ code: 'invalid', message: `${path}: ${reason}`, path }); } -/** Rows' ids: `u8 0` and a count, or `u8 1` and their values. */ +/** Writes row ids: `u8 0` and a count, or `u8 1` and the ids as values. */ function writeIds(w: Writer, ids: RowIds, path: string): void { if (!ArrayBuffer.isView(ids) && !Array.isArray(ids) && 'count' in ids) { if (!Number.isInteger(ids.count) || ids.count < 0 || ids.count > 0xffffffff) - throw invalid(path, 'a count of ids is a whole number from 0'); + throw invalid(path, 'a count of ids must be a whole number from 0 to 2^32 - 1'); w.u8(0); w.u32(ids.count); return; @@ -479,8 +571,8 @@ function writeIds(w: Writer, ids: RowIds, path: string): void { } /** - * Ids as values: numbers as `u32` when every one fits, which latkit reads as the `u64` it keeps, - * else as `u64`; texts when any is one; `null` missing. + * Writes ids as values: as text if any id is a string, else as `u32` when every id fits (latkit + * widens them to the `u64` it stores), else as `u64`. `null` marks a missing id. */ function writeIdValues(w: Writer, ids: RowIds, path: string): void { if (ids instanceof Uint32Array) { @@ -503,7 +595,7 @@ function writeIdValues(w: Writer, ids: RowIds, path: string): void { if (id === null || id === undefined) missing = true; else if (typeof id === 'string') texts = true; else if (!Number.isInteger(id) || id < 0 || id > SAFE) - throw invalid(path, `id ${id}: a numeric id is a whole number from 0 to 2^53 - 1`); + throw invalid(path, `id ${id} is not a whole number from 0 to 2^53 - 1`); else if (id > 0xffffffff) wide = true; } if (texts) { @@ -512,7 +604,7 @@ function writeIdValues(w: Writer, ids: RowIds, path: string): void { Array.from(list, (id) => (id === null || id === undefined ? null : String(id))), ); } else { - // Each id as one little-endian word, or two, low then high. + // One little-endian u32 per id, or two when wide: the low word, then the high. const words = new Uint32Array(wide ? n * 2 : n); for (let i = 0; i < n; i++) { const id = (list[i] as number | null) ?? 0; @@ -526,11 +618,11 @@ function writeIdValues(w: Writer, ids: RowIds, path: string): void { writeValidity(w, n, missing ? (i) => list[i] !== null && list[i] !== undefined : null); } -/** Whether an array value is present: `null`, `undefined`, and an empty geometry are missing. */ +/** Whether a value is present. `null`, `undefined`, and an empty geometry count as missing. */ const present = (v: unknown): boolean => v !== null && v !== undefined && !(Array.isArray(v) && v.length === 0); -/** Values as one storage, a validity mask after them when any is missing. */ +/** Writes values in one storage type, followed by a validity mask when any value is missing. */ function writeValues(w: Writer, values: Values, path: string): void { if (ArrayBuffer.isView(values)) { if (values instanceof Uint8Array) { @@ -549,7 +641,7 @@ function writeValues(w: Writer, values: Values, path: string): void { ? STORE.u32 : undefined; if (store === undefined) - throw invalid(path, `${values.constructor.name} is not a column's typed array`); + throw invalid(path, `${values.constructor.name} is not a supported typed array`); w.u8(store); w.u32(values.length); w.bytes(new Uint8Array(values.buffer, values.byteOffset, values.byteLength)); @@ -595,12 +687,12 @@ function writeValues(w: Writer, values: Values, path: string): void { writeGeometry(w, list as readonly (Geometry | null)[], path); break; default: - throw invalid(path, `a field holds numbers, booleans, texts, or geometries, not ${kind}`); + throw invalid(path, `a field holds numbers, booleans, strings, or geometries, not ${kind}`); } writeValidity(w, n, missing ? (i) => present(list[i]) : null); } -/** Texts: their byte offsets, then their UTF-8; a missing text is empty. */ +/** Writes texts as `n + 1` byte offsets, then their UTF-8 bytes. A missing text is empty. */ function writeTexts(w: Writer, texts: readonly (string | null)[]): void { const n = texts.length; let chars = 0; @@ -622,24 +714,28 @@ function writeTexts(w: Writer, texts: readonly (string | null)[]): void { w.bytes(bytes.subarray(0, at)); } -/** Geometries: their depth, each level's offsets into the next, then their `[x, y]` points. */ +/** + * Writes geometries: their nesting depth, each level's offsets into the next level, then the + * `[x, y]` points as `f64`. + */ function writeGeometry(w: Writer, rows: readonly (Geometry | null)[], path: string): void { const depthOf = (g: Geometry): number => typeof g[0] === 'number' ? 0 : Array.isArray(g[0]) ? 1 + depthOf(g[0] as Geometry) : 1; const first = rows.find((g) => present(g)); const depth = first ? depthOf(first) : 0; - if (depth > 3) throw invalid(path, 'geometries nest at most polygons of rings of [x, y] points'); + if (depth > 3) + throw invalid(path, 'geometry nests deeper than polygons of rings of [x, y] points'); const levels: number[][] = Array.from({ length: depth }, () => [0]); const coords: number[] = []; const push = (g: Geometry, level: number): void => { if (level === depth) { if (typeof g[0] !== 'number' || typeof g[1] !== 'number') - throw invalid(path, `geometries nest alike: ${depth} levels deep`); + throw invalid(path, `every geometry must nest ${depth} levels deep, like the first`); coords.push(g[0], g[1]); return; } if (typeof g[0] === 'number') - throw invalid(path, `geometries nest alike: ${depth} levels deep`); + throw invalid(path, `every geometry must nest ${depth} levels deep, like the first`); for (const item of g as readonly Geometry[]) push(item, level + 1); const next = level + 1 < depth ? levels[level + 1]!.length - 1 : coords.length / 2; levels[level]!.push(next); @@ -660,20 +756,20 @@ function writeGeometry(w: Writer, rows: readonly (Geometry | null)[], path: stri w.bytes(new Uint8Array(Float64Array.from(coords).buffer)); } -/** `n` bits, `bit(i)` each, in `u64` words, low bit first. */ +/** Writes `n` bits, bit `i` being `bit(i)`, packed into `u64` words, lowest bit first. */ function writeBits(w: Writer, n: number, bit: (i: number) => boolean): void { const words = new Uint8Array(Math.ceil(n / 64) * 8); for (let i = 0; i < n; i++) if (bit(i)) words[i >> 3]! |= 1 << (i & 7); w.bytes(words); } -/** `u8 0`, or `u8 1` and the bits of which of `n` values are present. */ +/** Writes `u8 0` when every value is present, else `u8 1` and a bit per value, set if present. */ function writeValidity(w: Writer, n: number, valid: ((i: number) => boolean) | null): void { w.u8(valid ? 1 : 0); if (valid) writeBits(w, n, valid); } -/** Bytes written little endian into a buffer that doubles as it fills. */ +/** A little-endian byte writer over a buffer that doubles in size as it fills. */ class Writer { #buffer: Uint8Array; #data: DataView; @@ -689,7 +785,7 @@ class Writer { this.length = owned?.length ?? 0; } - /** Room for `n` more bytes past the end. */ + /** Ensures capacity for `n` more bytes past the end. */ room(n: number): void { if (this.length + n <= this.#buffer.length) return; const grown = new Uint8Array(Math.max(2 * this.#buffer.length, this.length + n)); @@ -721,20 +817,16 @@ class Writer { } u32(v: number): void { - if (!Number.isInteger(v) || v < 0 || v > 0xffffffff) throw new RangeError(`${v} is no u32`); + if (!Number.isInteger(v) || v < 0 || v > 0xffffffff) + throw thrown({ code: 'limit', message: `${v} does not fit in a 32-bit count` }); this.room(4); this.#data.setUint32(this.length, v, true); this.length += 4; } - u16(v: number): void { - this.room(2); - this.#data.setUint16(this.length, v, true); - this.length += 2; - } - u64(v: number): void { - if (!Number.isSafeInteger(v) || v < 0) throw new RangeError(`${v} is no safe unsigned integer`); + if (!Number.isSafeInteger(v) || v < 0) + throw thrown({ code: 'invalid', message: `${v} is not a safe nonnegative integer` }); this.room(8); this.#data.setBigUint64(this.length, BigInt(v), true); this.length += 8; @@ -752,29 +844,3 @@ class Writer { this.bytes(bytes); } } - -/** The chunks of a stream, read with a reader where async iteration is missing. */ -async function* chunks( - stream: ReadableBytes | AsyncIterable, -): AsyncGenerator { - if (!('getReader' in stream)) { - yield* stream; - return; - } - const reader = stream.getReader(); - let completed = false; - try { - for (;;) { - const { done, value } = await reader.read(); - if (done) { - completed = true; - return; - } - if (!value) throw malformed('a stream chunk is missing'); - yield value; - } - } finally { - if (!completed) await reader.cancel().catch(() => {}); - reader.releaseLock(); - } -} diff --git a/npm/src/runtime.ts b/npm/src/runtime.ts deleted file mode 100644 index ed53198a..00000000 --- a/npm/src/runtime.ts +++ /dev/null @@ -1,925 +0,0 @@ -/// -/** WebAssembly-backed data and drawing, loaded only by loadLatkit(). */ - -import init, * as wasm from '../pkg/latkit.js'; -import { encoder } from './encoder.js'; -import type { Budget, Camera, Colormap, Fit, Id, Query, Row, Rows, Scene } from './inputs.js'; -import { jobs } from './jobs.js'; -import type { LoadOptions } from './load.js'; -import type { Description, Failure, Hit, Stats, Value, ViewEvent, Work } from './outputs.js'; -import { Patch } from './patch.js'; - -let run: ((bytes: Uint8Array) => Promise) | undefined; -/** The page's first GPU, made as the runtime loads, until `Gpu.create()` takes it. */ -let first: Promise | undefined; - -/** @internal Initializes the bindings before returning their public API. */ -export async function initialize(options: LoadOptions) { - await init(options.wasm === undefined ? undefined : { module_or_path: options.wasm }); - if (options.createWorker || typeof Worker === 'function') { - run = jobs(wasm.module() as WebAssembly.Module, options.createWorker); - wasm.setJobs(run); - } - if (typeof VideoEncoder === 'function') wasm.setEncoder(encoder); - // The page's first GPU is made now, so the page does not wait for its device when it draws; it - // compiles nothing until a view opens. - if ((globalThis.navigator as { gpu?: unknown } | undefined)?.gpu) { - first = wasm.Gpu.create(); - first.catch(() => {}); - } - return Object.freeze({ Data, Gpu, Shot, colormapCss }); -} - -/** @internal The bytes of the patch its JSON makes, for `Patch.load`. */ -export async function readPatch(json: string, base: string): Promise { - return (await wasm.loadPatch(json, base)) as Uint8Array; -} - -/** - * What latkit throws: an `Error` with a `code`, and for invalid input the `path` to the part, such - * as `marks[2].vertices.color`. - */ -export interface LatkitError extends Error { - code: Failure['code']; - path?: string; -} - -/** - * What a view tells its listeners, by type. Intents (`click`, `open`, `menu`, `drag`, `connect`, - * `delete`) are told once each, in the order the user acted. Changes (`hover`, `select`, `camera`, - * `waiting`) tell only their latest, and their state can always be read. A view tells only what the - * application did not do itself: its setters change state silently. - */ -export type ViewEvents = { - [E in ViewEvent as E['type']]: E extends { type: 'error' } ? LatkitError : E['detail']; -}; - -/** The error `failure` describes, as latkit throws them. */ -function thrown({ code, message, path }: Failure): LatkitError { - return Object.assign(new Error(message), path === undefined ? { code } : { code, path }); -} - -/** `promise`, or a `timeout` error once `seconds` pass first. */ -function deadline(promise: Promise, seconds: number, what: string): Promise { - let timer: ReturnType | undefined; - const late = new Promise((_, reject) => { - timer = setTimeout( - () => - reject( - thrown({ - code: 'timeout', - message: `${what} took over ${seconds} s: is the view drawn?`, - }), - ), - seconds * 1000, - ); - }); - return Promise.race([promise, late]).finally(() => clearTimeout(timer)); -} - -/** - * The rows a query keeps, in the forms a patch writes: numbers as a `Float64Array` (missing NaN), - * other values as arrays (missing `null`), a reference as its target ids. - */ -export interface Table { - table: string; - ids: Id[]; - fields: Record; - /** The rows the query kept before paging. */ - total: number; -} - -/** Rows' least and greatest values over spans of the axis: `at` holds one more edge than spans. */ -export interface Envelope { - at: Float64Array; - least: Float32Array; - greatest: Float32Array; -} - -/** - * A named collection of tables, changed in place by patches. A snapshot keeps a version, sharing - * every page with it, so keeping earlier versions, as an undo history does, costs only the pages - * they hold apart. A view draws the version it was given until it is given another. - */ -export class Data { - #inner?: wasm.Data; - - /** @internal */ - get inner(): wasm.Data { - if (!this.#inner) throw new Error('this Data has been destroyed'); - return this.#inner; - } - - constructor(name: string); - /** @internal */ - constructor(name: wasm.Data); - constructor(name: string | wasm.Data) { - this.#inner = typeof name === 'string' ? new wasm.Data(name) : name; - } - - /** Releases this handle. Views and snapshots keep the versions they hold. */ - destroy(): void { - this.#inner?.free(); - this.#inner = undefined; - } - - [Symbol.dispose](): void { - this.destroy(); - } - - /** - * The dataset `bytes` hold: latkit's bytes, renamed `name` when given, or GeoJSON, gzip-compressed - * or not, read in a worker as dataset `name`: points, line strings, and polygons as tables - * `points`, `lines`, and `polygons`, each with its geometry as field `geometry` and its properties - * as fields. - */ - static async read(bytes: Uint8Array, name?: string): Promise { - if ([0x4c, 0x41, 0x54, 0x4b].every((b, i) => bytes[i] === b)) - return new Data(wasm.Data.read(bytes, name)); - if (name === undefined) - throw thrown({ code: 'invalid', message: 'GeoJSON is read as a named dataset' }); - const job = wasm.geojsonJob(name, bytes); - const answer = run ? await run(job) : wasm.job(job); - if (answer.length === 0) - throw thrown({ - code: 'invalid', - message: `${name}: the bytes are neither latkit's nor GeoJSON`, - }); - return new Data(wasm.Data.read(answer, name)); - } - - /** - * The dataset its JSON document builds, `{"name": ..., "patch": [...]}`, the patch loading as - * `Patch.load` loads one. - */ - static async load(json: string, base = ''): Promise { - return new Data((await wasm.Data.load(json, base)) as wasm.Data); - } - - get name(): string { - return this.inner.name; - } - - /** The names of the tables. */ - get tables(): string[] { - return this.inner.tables(); - } - - /** Every row's id of `table`, in row order. */ - ids(table: string): Id[] { - return this.inner.ids(table) as Id[]; - } - - /** The names of `table`'s fields. */ - fields(table: string): string[] { - return this.inner.fields(table); - } - - /** The first and last axis coordinates the sampled fields span. */ - get span(): [number, number] | undefined { - const span = this.inner.span; - return span ? [span[0]!, span[1]!] : undefined; - } - - /** Applies every change of `patch`, or none when one is invalid. */ - apply(patch: Patch | Uint8Array): this { - this.inner.apply(patch instanceof Patch ? patch.frame() : patch); - return this; - } - - /** This version, kept while `apply` changes this one: O(1). */ - snapshot(): Data { - return new Data(this.inner.snapshot()); - } - - /** The value of `field` for the row with `id` in `table`, at axis coordinate `at` when given. */ - get(table: string, id: Id, field: string, at?: number): Value { - return JSON.parse(this.inner.get(table, id, field, at)) as Value; - } - - /** What the dataset holds: tables, rows, fields and their kinds, the axis and its span. */ - describe(): Description { - return JSON.parse(this.inner.describe()) as Description; - } - - /** - * The rows of a table `query` keeps, filtered, sorted, and paged, sampled fields read at its - * coordinate, on the CPU through the expressions marks draw with. - */ - query(query: Query): Table { - return this.inner.query(JSON.stringify(query)) as Table; - } - - /** - * Each row's least and greatest value of sampled `field` of `table` over `spans` equal spans of - * `range`, of every row or of rows `ids`: row after row, NaN where a span holds no frame. Reads the - * coarsest levels of detail that fit, so a span costs a few reads however many frames it covers. - */ - envelope( - table: string, - field: string, - range: [number, number], - spans: number, - ids?: readonly Id[], - ): Envelope { - const listed = ids === undefined ? undefined : JSON.stringify(ids); - return this.inner.envelope(table, field, Float64Array.from(range), spans, listed) as Envelope; - } - - /** The dataset as latkit's bytes: the patch that builds it. */ - toBytes(): Uint8Array { - return this.inner.toBytes(); - } -} - -/** The GPU device views draw with. Views of one GPU draw in one animation frame. */ -export class Gpu { - #inner?: wasm.Gpu; - #views = new Set(); - #canvases = new Set(); - #listeners = new Set<() => void>(); - - private constructor(inner: wasm.Gpu) { - this.#inner = inner; - } - - /** @internal */ - get inner(): wasm.Gpu { - if (!this.#inner) throw new Error('this Gpu has been destroyed'); - return this.#inner; - } - - /** Requests a WebGPU device: the first call takes the one made as latkit loaded. */ - static async create(): Promise { - const inner = first ?? wasm.Gpu.create(); - first = undefined; - return new Gpu(await inner); - } - - /** - * A view of `scene`; its marks draw once {@link View.setData} gives their data. A {@link Canvas} - * shows it, and a {@link Shot} draws it with or without one. - */ - view(scene: Scene): View { - const view: View = new View( - (tell) => this.inner.view(JSON.stringify(scene), tell), - () => this.#views.delete(view), - ); - this.#views.add(view); - return view; - } - - /** - * A canvas showing views in `panels` of `target`: a `` element, which sizes itself to its - * CSS box and passes each view the input landing on its panel, or an `OffscreenCanvas` in a - * worker, which keeps the size it is given at `pixelRatio` pixels a CSS pixel. Panels are - * `[x, y, width, height]` fractions, one per view; by default one view fills the canvas. - */ - canvas( - target: HTMLCanvasElement | OffscreenCanvas, - options: { panels?: readonly Panel[]; pixelRatio?: number } = {}, - ): Canvas { - const canvas: Canvas = new Canvas( - this.inner.canvas(target, JSON.stringify(options.panels ?? []), options.pixelRatio), - () => this.#canvases.delete(canvas), - ); - this.#canvases.add(canvas); - return canvas; - } - - /** - * Calls `listener` with the frame's timestamp every animation frame, before the views draw, so - * what it changes shows in that frame. Returns a function that stops it. - */ - on(type: 'frame', listener: (now: number) => void): () => void { - const handle = this.inner.onFrame(listener); - const stop = (): void => { - if (this.#listeners.delete(stop)) handle.free(); - }; - this.#listeners.add(stop); - return stop; - } - - /** Whether the device was lost; a new `Gpu` is needed. */ - get lost(): boolean { - return this.inner.lost; - } - - /** How much GPU memory data takes. */ - get budget(): Budget { - return JSON.parse(this.inner.budget) as Budget; - } - - set budget(budget: Budget) { - this.inner.setBudget(JSON.stringify(budget)); - } - - /** What this GPU did since it was created, counted exactly; see {@link Work}. */ - get work(): Work { - return JSON.parse(this.inner.work) as Work; - } - - /** - * Draws one frame now at `now`, milliseconds on the clock animation frames give: the frame - * listeners, then every canvas that needs a frame. From the first step no animation frames are - * asked for: a harness, as tests and benchmarks are, steps every frame itself, so a run draws the - * same frames however fast the device is. - */ - step(now: number): void { - this.inner.step(now); - } - - /** - * Resolves once the GPU finished the work submitted to it, every readback landed, and every - * pipeline a view or shot asked for compiled: a frame stepped then draws all of it. A `timeout` - * error after `deadline` seconds. - */ - async done({ deadline: seconds = 30 } = {}): Promise { - await deadline(this.inner.done() as Promise, seconds, 'the GPU'); - } - - /** - * Resolves when work a view waits on lands after this call (a readback, a compile, a job, a text, - * a fetch), or a view asks for a frame. A `timeout` error after `deadline` seconds. - */ - async landed({ deadline: seconds = 30 } = {}): Promise { - await deadline(this.inner.landed() as Promise, seconds, 'work landing'); - } - - /** Releases this GPU's canvases, views, and frame listeners. Other GPUs and data stay usable. */ - destroy(): void { - if (!this.#inner) return; - for (const canvas of this.#canvases) canvas.destroy(); - for (const view of this.#views) view.destroy(); - for (const stop of this.#listeners) stop(); - this.#inner.free(); - this.#inner = undefined; - } - - [Symbol.dispose](): void { - this.destroy(); - } -} - -/** A view's panel of a canvas or a picture: `[x, y, width, height]` as fractions of it. */ -export type Panel = readonly [number, number, number, number]; - -/** Views shown in the panels of a `` or an `OffscreenCanvas`. */ -export class Canvas { - #inner?: wasm.Canvas; - - /** @internal */ - constructor( - inner: wasm.Canvas, - private readonly released: () => void, - ) { - this.#inner = inner; - } - - /** Shows `views`, one per panel, in place of those it showed. A view shows in one canvas at a time. */ - show(views: readonly View[]): this { - if (!this.#inner) throw new Error('this Canvas has been destroyed'); - this.#inner.show(views.map((v) => v.handle())); - return this; - } - - /** Stops drawing; the views remain, and the element keeps its last frame. */ - destroy(): void { - if (!this.#inner) return; - this.#inner.free(); - this.#inner = undefined; - this.released(); - } - - [Symbol.dispose](): void { - this.destroy(); - } -} - -/** A scene drawn from data it is given by name: shown by a canvas, which passes it input, or drawn by shots. */ -export class View { - #listeners = new Map void>>(); - #inner?: wasm.View; - - /** @internal Makes the view with what takes its events, a JSON batch at a time, from the start. */ - constructor( - make: (tell: (json: string) => void) => wasm.View, - private readonly released: () => void, - ) { - this.#inner = make((json) => this.#tell(json)); - } - - private get inner(): wasm.View { - if (!this.#inner) throw new Error('this View has been destroyed'); - return this.#inner; - } - - /** @internal The view, for a canvas to show or a shot to draw. */ - handle(): wasm.Handle { - return this.inner.handle(); - } - - /** Draws `scene` instead; an invalid scene throws and changes nothing. */ - show(scene: Scene): void { - this.inner.show(JSON.stringify(scene)); - } - - /** - * Gives the view `data` as it is now, replacing the dataset of its name: give it again after - * applying patches. A mark it no longer fits draws nothing, and the next frame reports why as an - * error, unless a scene shown before then replaces the mark. - */ - setData(data: Data): void { - this.inner.setData(data.inner); - } - - /** - * Fetches the data at `url` in pieces as the view asks for them, kept as dataset `name`, which the - * view owns; the source's first bytes say what it is. - * - * - Time tiles, a `frames.json` manifest as `latkit tile` writes it: a dataset whose sampled - * fields' histories are fetched in spans. A line tracing a field fetches the detail its plot - * window draws, a mark showing one the frames about the view's coordinate, and a span still - * loading draws the coarser span above it. - * - Vector tiles: a PMTiles archive, a TileJSON document, or a `{z}/{x}/{y}` template, fetched as - * the camera asks. Each layer is a table of its features of the kind marks draw it as (areas - * polygons, lines lines, vertices points), or named `layer:polygons`, `layer:lines`, or - * `layer:points`. - */ - setTiles(name: string, url: string): void { - this.inner.setTiles(name, url); - } - - /** Stops drawing the dataset named `name`, given or fetched as tiles. */ - removeData(name: string): void { - this.inner.removeData(name); - } - - /** The dataset named `name`, as given or as the view fetched it, as a snapshot. */ - data(name: string): Data | undefined { - const data = this.inner.data(name); - return data ? new Data(data) : undefined; - } - - /** The axis coordinate shown: sampled fields read their last frame at or before it. */ - get at(): number { - return this.inner.at; - } - - set at(at: number) { - this.inner.at = at; - } - - get camera(): Camera { - return JSON.parse(this.inner.camera) as Camera; - } - - set camera(camera: Camera) { - this.inner.setCamera(JSON.stringify(camera)); - } - - /** Eases to `camera`. */ - fly(camera: Camera): void { - this.inner.fly(JSON.stringify(camera)); - } - - /** Frames what `fit` names on the next frame, keeping pitch and bearing. */ - fit(fit: Fit = 'all'): void { - this.inner.fit(JSON.stringify(fit)); - } - - /** Where `position` (x, y, elevation) lands on screen, CSS pixels from the top left. */ - point(position: [number, number, number]): [number, number] | undefined { - const point = this.inner.point(Float64Array.from(position)); - return point ? [point[0]!, point[1]!] : undefined; - } - - /** The position on the ground under screen point `point`, if the view shows ground there. */ - ground(point: [number, number]): [number, number, number] | undefined { - const position = this.inner.ground(Float32Array.from(point)); - return position ? [position[0]!, position[1]!, position[2]!] : undefined; - } - - /** CSS pixels kept clear at the top, right, bottom, and left, as when panels cover the view. */ - get insets(): [number, number, number, number] { - const [top, right, bottom, left] = this.inner.insets; - return [top!, right!, bottom!, left!]; - } - - set insets(insets: [number, number, number, number]) { - this.inner.insets = Float32Array.from(insets); - } - - /** Degrees per second a map or plane turns around its center; 0 stops it. A diagram never turns. */ - get orbit(): number { - return this.inner.orbit; - } - - set orbit(degreesPerSecond: number) { - this.inner.orbit = degreesPerSecond; - } - - /** The rows selected, a set per table. */ - get selection(): Rows[] { - return JSON.parse(this.inner.selection) as Rows[]; - } - - /** - * Selects `rows`, and no others. Like every setter it tells nothing: `select` events tell what - * the user and the data changed. - */ - select(rows: readonly Rows[]): void { - this.inner.select(JSON.stringify(rows)); - } - - /** What the user points at: the row under the pointer, or the row the keys stepped to. */ - get hovered(): Hit | null { - return JSON.parse(this.inner.hovered) as Hit | null; - } - - /** Every mark that draws nothing because its data no longer fits it, and why. */ - get failures(): LatkitError[] { - return (JSON.parse(this.inner.failures) as Failure[]).map(thrown); - } - - /** - * Every row of a mark that picks within `radius` CSS pixels of `point`, nearest first, once a frame - * draws the view: a `timeout` error after `deadline` seconds without one. - */ - async pick( - point: [number, number], - radius: number, - { deadline: seconds = 5 } = {}, - ): Promise { - const hits = this.inner.pick(Float32Array.from(point), radius) as Promise; - return JSON.parse(await deadline(hits, seconds, 'a pick')) as Hit[]; - } - - /** - * Where `row` lies on screen, CSS pixels from the top left, or `null` when no mark draws it in - * view, once a frame draws the view: a `timeout` error after `deadline` seconds without one. - */ - locate(row: Row, { deadline: seconds = 5 } = {}): Promise<[number, number] | null> { - const point = this.inner.locate(JSON.stringify(row)) as Promise<[number, number] | null>; - return deadline(point, seconds, 'locating a row'); - } - - /** The rows joined to `row`, a set per table: a vertex's neighbors across edges and nets, or an - * edge's or a net's vertices. */ - neighbors(row: Row): Rows[] { - return JSON.parse(this.inner.neighbors(JSON.stringify(row))) as Rows[]; - } - - /** - * Lays every vertex row without a position out again by the scene's layout: a diagram's blocks, or - * points on a plane or the earth. Places a layout gave otherwise hold, so rows added later are - * placed around the rest. Once the layout lands, answers a patch to each dataset writing each row - * laid out into its mark's position fields; applied, the places are the data's own. A `timeout` - * error after `deadline` seconds without it. - * - * ```ts - * for (const patch of await view.arrange()) if (patch.data === data.name) data.apply(patch); - * ``` - */ - async arrange({ deadline: seconds = 60 } = {}): Promise { - const frames = (await deadline(this.inner.arrange(), seconds, 'arranging')) as Uint8Array[]; - return frames.map((frame) => Patch.fromBytes(frame)); - } - - get stats(): Stats { - return JSON.parse(this.inner.stats) as Stats; - } - - /** Whether the view wants a frame: it changed, or work it waited on landed. */ - get wantsFrame(): boolean { - return this.inner.wantsFrame; - } - - /** - * Calls `listener` with each event of `type` told while it listens; returns a function that - * stops it. What lasts is state to read: `hovered`, `selection`, `camera`, `stats.waiting`, and - * `failures`. - */ - on(type: T, listener: (event: ViewEvents[T]) => void): () => void { - let set = this.#listeners.get(type); - if (!set) this.#listeners.set(type, (set = new Set())); - set.add(listener); - return () => set.delete(listener); - } - - /** Hands a batch of the view's events to the listeners of each one's type. */ - #tell(json: string): void { - if (this.#listeners.size === 0) return; - for (const told of JSON.parse(json) as ViewEvent[]) { - const listeners = this.#listeners.get(told.type) as Set<(event: unknown) => void> | undefined; - if (!listeners?.size) continue; - const event = told.type === 'error' ? thrown(told.detail) : told.detail; - for (const listener of listeners) listener(event); - } - } - - /** Frees the view, and rejects what it was asked and has not answered; a canvas showing it keeps - * its last frame. */ - destroy(): void { - if (!this.#inner) return; - for (const listeners of this.#listeners.values()) listeners.clear(); - this.#listeners.clear(); - this.#inner.free(); - this.#inner = undefined; - this.released(); - } - - [Symbol.dispose](): void { - this.destroy(); - } -} - -type ShotSettings = Readonly<{ - scale: number; - panels: readonly Panel[]; - transparent: boolean; - deadline: number; - elapsed?: number; -}>; - -/** - * A picture of views, each drawn into its panel of `size` pixels at `scale` pixels per CSS pixel: - * plain data, like a scene. A view a shot draws holds its canvas's last frame meanwhile. - * - * ```ts - * const png = await (await new Shot([3840, 2160]).scale(2).image([view])).encode('png'); - * ``` - */ -export class Shot { - /** A picture with width and height in pixels. */ - constructor(size: readonly [number, number]); - /** @internal */ - constructor(size: readonly [number, number], settings: ShotSettings); - constructor( - readonly size: readonly [number, number], - /** @internal */ - readonly settings: ShotSettings = { scale: 1, panels: [], transparent: false, deadline: 30 }, - ) {} - - /** Draws `scale` pixels per CSS pixel: a view lays out `size / scale` CSS pixels. */ - scale(scale: number): Shot { - return new Shot(this.size, { ...this.settings, scale }); - } - - /** Where each view draws, `[x, y, width, height]` as fractions, one per view in order; by default one fills it. */ - panels(panels: readonly Panel[]): Shot { - return new Shot(this.size, { ...this.settings, panels }); - } - - /** Keeps the picture's alpha: views clear to transparent where their themes' backgrounds are. */ - transparent(): Shot { - return new Shot(this.size, { ...this.settings, transparent: true }); - } - - /** The shot's deadline in seconds, including readback; rejects on timeout. Defaults to 30. */ - deadline(seconds: number): Shot { - return new Shot(this.size, { ...this.settings, deadline: seconds }); - } - - /** Fixes elapsed-time expressions and drawing effects at `seconds` while transitions settle. */ - elapsed(seconds: number): Shot { - return new Shot(this.size, { ...this.settings, elapsed: seconds }); - } - - /** The picture of `views`, one per panel, once every one is complete and still. */ - async image(views: readonly View[]): Promise { - return new Image( - (await wasm.image( - this.#config(), - views.map((v) => v.handle()), - )) as wasm.Image, - ); - } - - /** - * A video of these pictures at `fps` frames a second, H.264 in MP4, written to `output` as it is - * drawn, at `bitrate` bits a second or about 0.15 bits a pixel a frame. Width and height are - * multiples of 4 and 2. `signal` aborts it: the output is aborted and the views are given back. - */ - video( - gpu: Gpu, - fps: number, - output: WritableStream, - options: { bitrate?: number; signal?: AbortSignal } = {}, - ): Video { - options.signal?.throwIfAborted(); - const writer = output.getWriter(); - try { - return new Video( - gpu.inner.video(this.#config(), fps, options.bitrate), - writer, - fps, - options.signal, - ); - } catch (error) { - writer.releaseLock(); - throw error; - } - } - - /** The shot as the crate reads it: JSON, times in seconds. */ - #config(): string { - return JSON.stringify({ size: [...this.size], ...this.settings }); - } -} - -/** RGBA pixels, straight alpha, rows top to bottom. */ -export class Image { - #inner?: wasm.Image; - - /** @internal */ - constructor(inner: wasm.Image) { - this.#inner = inner; - } - - private get inner(): wasm.Image { - if (!this.#inner) throw new Error('this Image has been destroyed'); - return this.#inner; - } - - get width(): number { - return this.inner.width; - } - - get height(): number { - return this.inner.height; - } - - /** The pixels, copied. */ - get rgba(): Uint8Array { - return this.inner.rgba; - } - - /** - * The image as a file, written in a worker: PNG, lossless with alpha; JPEG at `quality` from 1 to - * 100 (90 by default), small, its alpha dropped; or WebP, lossless with alpha, smaller than PNG. - */ - async encode(format: ImageFormat, options: { quality?: number } = {}): Promise { - const job = this.inner.encodeJob(format, options.quality); - const file = run ? await run(job) : wasm.job(job); - if (file.length === 0) - throw thrown({ code: 'invalid', message: `the image could not be written as ${format}` }); - return file; - } - - destroy(): void { - this.#inner?.free(); - this.#inner = undefined; - } - - [Symbol.dispose](): void { - this.destroy(); - } -} - -/** How an image is written as a file. */ -export type ImageFormat = 'png' | 'jpeg' | 'webp'; - -/** A video being drawn, encoded, and written. */ -export class Video { - #inner?: wasm.Video; - #state: 'open' | 'finishing' | 'aborting' | 'closed' = 'open'; - #operation?: Promise; - #finished?: Promise; - #aborted?: Promise; - #reason: unknown; - #frames = 0; - #removeAbort?: () => void; - - /** @internal */ - constructor( - inner: wasm.Video, - private readonly writer: WritableStreamDefaultWriter, - private readonly fps: number, - signal?: AbortSignal, - ) { - this.#inner = inner; - if (signal) { - const abort = () => { - void this.abort(signal.reason); - }; - signal.addEventListener('abort', abort, { once: true }); - this.#removeAbort = () => signal.removeEventListener('abort', abort); - if (signal.aborted) abort(); - } - } - - #check(): wasm.Video { - if (!this.#inner || this.#state === 'aborting' || this.#state === 'closed') - throw Object.assign( - thrown({ code: 'cancelled', message: 'this Video was finished or aborted' }), - { cause: this.#reason }, - ); - return this.#inner; - } - - /** Seconds into the video the next frame shows at; retained after completion. */ - get time(): number { - return this.frames / this.fps; - } - - /** Frames drawn or held so far; retained after completion. */ - get frames(): number { - return this.#inner?.frames ?? this.#frames; - } - - /** Draws and writes one frame, waiting for output backpressure. Only one frame may run at a time. */ - frame(views: readonly View[]): Promise { - if (this.#state !== 'open' || this.#operation) - return Promise.reject(new Error('a Video draws one frame at a time, before finish')); - const operation = (async () => { - const bytes = (await this.#check().frame(views.map((v) => v.handle()))) as Uint8Array; - this.#frames = this.#check().frames; - if (bytes.length) await this.writer.write(bytes); - })(); - this.#operation = operation; - return operation - .catch(async (error: unknown) => { - await this.abort(error); - throw error; - }) - .finally(() => { - if (this.#operation === operation) this.#operation = undefined; - }); - } - - /** Records until `step(time)` returns false; any step, drawing, output, or progress error aborts it. */ - async record( - views: readonly View[], - step: (time: number) => boolean | Promise, - onProgress?: (progress: { frames: number; time: number }) => void, - ): Promise { - try { - while (await step(this.time)) { - await this.frame(views); - onProgress?.({ frames: this.frames, time: this.time }); - } - await this.finish(); - } catch (error) { - await this.abort(error); - throw error; - } - } - - /** Finishes pending work, writes the end of the file, and closes and unlocks the output. Idempotent. */ - finish(): Promise { - if (this.#finished) return this.#finished; - if (this.#state !== 'open') return Promise.reject(new Error('this Video was aborted')); - this.#state = 'finishing'; - const frame = this.#operation; - const operation = (async () => { - await frame; - const bytes = (await this.#check().finish()) as Uint8Array; - this.#check(); - if (bytes.length) await this.writer.write(bytes); - this.#check(); - await this.writer.close(); - })(); - this.#operation = operation; - this.#finished = operation - .then(() => { - if (this.#state === 'finishing') this.#release(); - }) - .catch(async (error: unknown) => { - await this.abort(error); - throw error; - }); - return this.#finished; - } - - /** Cancels drawing and encoding, aborts the output, and unlocks it after pending writes settle. Idempotent. */ - abort(reason?: unknown): Promise { - if (this.#aborted) return this.#aborted; - if (this.#state === 'closed') return Promise.resolve(); - this.#state = 'aborting'; - this.#reason = reason; - this.#frames = this.frames; - this.#inner?.cancel(); - this.#aborted = Promise.allSettled([this.#operation, this.writer.abort(reason)]).then(() => - this.#release(), - ); - return this.#aborted; - } - - #release(): void { - this.#frames = this.frames; - this.#inner?.free(); - this.#inner = undefined; - this.#removeAbort?.(); - this.#removeAbort = undefined; - this.writer.releaseLock(); - this.#state = 'closed'; - } -} -/** - * A CSS gradient through `map`, a catalog name or colors of your own, for legends: the colors the - * GPU draws, a band each for a categorical map's categories. - */ -export function colormapCss(map: Colormap, direction = 'to right'): string { - return wasm.colormapCss(JSON.stringify(map), direction); -} diff --git a/npm/src/scene.ts b/npm/src/scene.ts index 535876ea..19e97dd1 100644 --- a/npm/src/scene.ts +++ b/npm/src/scene.ts @@ -1,240 +1,344 @@ /** - * The expression builders that write scenes' options. A scene is plain data the view reads, its - * types written from the forms latkit reads (`inputs.ts`); nothing here needs WebAssembly. + * Builders for the expressions that scene options take. A scene is plain data, typed in + * `inputs.ts`. Nothing here needs WebAssembly. */ +import type { TableInput } from './data.js'; import type { - Binary, Colormap, + ColormapNode, ExprNode, FieldNode, Marker, - Size, + Px, + ScaleNode, State, - Unary, + Units, } from './inputs.js'; -/** A CSS color, such as `'#ffb000'` or `'rebeccapurple'`. Where an option takes an expression, a - * string starting with `#` is a hex color and any other string names a field. */ +/** + * A CSS color, such as `'#ffb000'` or `'rebeccapurple'`. In an expression, a string that does not + * start with `$` is a color. + */ export type Color = string; -/** What an expression is written as: an {@link Expr}, or its JSON form. */ +/** + * An option's value: an {@link Expr} or its JSON form. In JSON, `null` is a missing value, a string + * starting with `$` reads a field (`'$kv'` reads field `kv`), and any other string is a CSS color. + * Color options take a value that computes a color. + */ export type ExprInput = Expr | ExprNode; const nodeOf = (input: ExprInput): ExprNode => (input instanceof Expr ? input.node : input); -/** A value for every row of a mark: a constant, a field, view state, or arithmetic of them. */ +/** + * A value for every row of a mark: a constant, a color, a field, view state, or arithmetic on them. + * Color options take one that computes a color: a CSS color, a value mapped through a colormap, or + * a conditional that chooses between colors. + */ export class Expr { - /** An expression from its JSON form; use the builders for common operations. */ - constructor(readonly node: ExprNode) {} + /** The expression's JSON form. */ + readonly node: ExprNode; + + /** Wraps an expression's JSON form. The builders cover common cases. */ + constructor(node: ExprNode) { + this.node = node; + } + /** Returns the JSON form. `JSON.stringify` calls it. */ toJSON(): ExprNode { return this.node; } - /** Reads this expression's fields from `table` of dataset `data`, joined by row id. */ - of(data: string, table: string): Expr { - return new Expr(mapFields(this.node, (f) => ({ ...f, of: [data, table] }))); + /** + * Reads this expression's fields from `table`, such as `weather.table('Station')`, joined by row + * id. The view also loads that table's dataset. + */ + from(table: TableInput): Expr { + return new Expr(mapFields(this.node, (f) => ({ ...f, from: table }))); } - /** Reads sampled fields at their last frame without interpolating. */ + /** + * Reads this expression's sampled fields at the last frame at or before the view's `at`, without + * interpolating toward the next frame. + */ step(): Expr { return new Expr(mapFields(this.node, (f) => ({ ...f, step: true }))); } + /** The sum of this value and `other`. */ add(other: ExprInput): Expr { - return binary('add', this, other); + return new Expr({ add: [this.node, nodeOf(other)] }); } + /** This value minus `other`. */ sub(other: ExprInput): Expr { - return binary('sub', this, other); + return new Expr({ sub: [this.node, nodeOf(other)] }); } + /** The product of this value and `other`. */ mul(other: ExprInput): Expr { - return binary('mul', this, other); + return new Expr({ mul: [this.node, nodeOf(other)] }); } + /** This value divided by `other`. Dividing by 0 gives a missing value. */ div(other: ExprInput): Expr { - return binary('div', this, other); + return new Expr({ div: [this.node, nodeOf(other)] }); } + /** The remainder of this value divided by `other`, with the sign of `other`: `-1 rem 3` is 2. */ rem(other: ExprInput): Expr { - return binary('rem', this, other); + return new Expr({ rem: [this.node, nodeOf(other)] }); } + /** This value raised to the power `other`. */ pow(other: ExprInput): Expr { - return binary('pow', this, other); + return new Expr({ pow: [this.node, nodeOf(other)] }); } + /** The smaller of this value and `other`. */ min(other: ExprInput): Expr { - return binary('min', this, other); + return new Expr({ min: [this.node, nodeOf(other)] }); } + /** The larger of this value and `other`. */ max(other: ExprInput): Expr { - return binary('max', this, other); + return new Expr({ max: [this.node, nodeOf(other)] }); } + /** This value limited to the range from `low` to `high`. */ clamp(low: ExprInput, high: ExprInput): Expr { return this.max(low).min(high); } + /** 1 where this value is less than `other`, else 0. */ lt(other: ExprInput): Expr { - return binary('lt', this, other); + return new Expr({ lt: [this.node, nodeOf(other)] }); } + /** 1 where this value is at most `other`, else 0. */ le(other: ExprInput): Expr { - return binary('le', this, other); + return new Expr({ le: [this.node, nodeOf(other)] }); } + /** 1 where this value is greater than `other`, else 0. */ gt(other: ExprInput): Expr { - return binary('gt', this, other); + return new Expr({ gt: [this.node, nodeOf(other)] }); } + /** 1 where this value is at least `other`, else 0. */ ge(other: ExprInput): Expr { - return binary('ge', this, other); + return new Expr({ ge: [this.node, nodeOf(other)] }); } + /** 1 where this value equals `other`, else 0. Colors compare too. */ eq(other: ExprInput): Expr { - return binary('eq', this, other); + return new Expr({ eq: [this.node, nodeOf(other)] }); } + /** 1 where this value differs from `other`, else 0. Colors compare too. */ ne(other: ExprInput): Expr { - return binary('ne', this, other); + return new Expr({ ne: [this.node, nodeOf(other)] }); } + /** 1 where this value and `other` are both above 0.5, else 0. */ and(other: ExprInput): Expr { - return binary('and', this, other); + return new Expr({ and: [this.node, nodeOf(other)] }); } + /** 1 where this value or `other` is above 0.5, else 0. */ or(other: ExprInput): Expr { - return binary('or', this, other); + return new Expr({ or: [this.node, nodeOf(other)] }); } + /** 1 where this value is at most 0.5, else 0. */ not(): Expr { - return unary('not', this); + return new Expr({ not: [this.node] }); } + /** The value negated. */ neg(): Expr { - return unary('neg', this); + return new Expr({ neg: [this.node] }); } + /** The absolute value. */ abs(): Expr { - return unary('abs', this); + return new Expr({ abs: [this.node] }); } + /** The square root. Negative values give a missing value. */ sqrt(): Expr { - return unary('sqrt', this); + return new Expr({ sqrt: [this.node] }); } + /** The sine of the value, an angle in radians. */ sin(): Expr { - return unary('sin', this); + return new Expr({ sin: [this.node] }); } + /** The cosine of the value, an angle in radians. */ cos(): Expr { - return unary('cos', this); + return new Expr({ cos: [this.node] }); } + /** The largest whole number at most the value. */ floor(): Expr { - return unary('floor', this); + return new Expr({ floor: [this.node] }); } + /** The natural logarithm. Values at most 0 give a missing value. */ ln(): Expr { - return unary('ln', this); + return new Expr({ ln: [this.node] }); } + /** e raised to the value. */ exp(): Expr { - return unary('exp', this); + return new Expr({ exp: [this.node] }); } - /** Whether a text field equals `text`. */ + /** Whether a text field equals `text`, plain text read as neither a field nor a color. */ is(text: string): Expr { return new Expr({ is: [this.node, text] }); } - /** Whether a text field holds `text`, ignoring case. Queries read it; marks do not. */ + /** + * Whether a text field contains `text` (plain text), ignoring case. In a mark, the field must be + * in the mark's own table. + */ contains(text: string): Expr { return new Expr({ contains: [this.node, text] }); } - /** The value through `map`: a catalog name, or your own colors. */ + /** + * Maps the value to a color through `map`, a catalog name or custom colors. The domain fits the + * values shown unless set with {@link Mapped.domain}. + */ colormap(map: Colormap): Mapped { return new Mapped({ colormap: map, value: this.node }); } - /** The value mapped linearly onto `[low, high]`, clamped. */ + /** + * Maps the value linearly onto `[low, high]`, clamped. The domain fits the values shown unless + * set with {@link Mapped.domain}. + */ scale(low: number, high: number): Mapped { return new Mapped({ scale: [low, high], value: this.node }); } } -/** A value through a colormap or onto a range, whose domain can be set. */ +/** A value mapped through a colormap or onto a range. Its input domain can be set. */ export class Mapped extends Expr { - /** The input values that map to the ends; it fits the values shown unless set. */ + /** + * Sets the input values that map to the ends of the colormap or range. Without it, the domain + * fits the values shown. + */ domain(low: number, high: number): Mapped { - return new Mapped({ ...(this.node as MappedNode), domain: [low, high] }); + return new Mapped({ ...(this.node as ScaleNode | ColormapNode), domain: [low, high] }); } } -type MappedNode = Extract; - -const unary = (op: Unary, a: ExprInput): Expr => new Expr({ op, args: [nodeOf(a)] }); -const binary = (op: Binary, a: ExprInput, b: ExprInput): Expr => - new Expr({ op, args: [nodeOf(a), nodeOf(b)] }); - -/** `node` with every field read through `f`; a string starting with `#` is a color, not a field. */ +/** Returns `input`'s JSON form with every field replaced by `f(field)`, in its `{ field }` form. */ function mapFields(input: ExprInput, f: (field: FieldNode) => FieldNode): ExprNode { const node = nodeOf(input); - if (typeof node === 'string') return node.startsWith('#') ? node : f({ field: node }); - if (typeof node !== 'object') return node; + if (typeof node === 'string') return node.startsWith('$') ? f({ field: node.slice(1) }) : node; + if (node === null || typeof node !== 'object') return node; const map = (child: ExprInput) => mapFields(child, f); if ('field' in node) return f(node); - if ('op' in node) return { ...node, args: node.args.map(map) } as ExprNode; + if ('state' in node) return node; if ('when' in node) { const cases = node.when.map(([c, v]): [ExprNode, ExprNode] => [map(c), map(v)]); return { when: cases, otherwise: map(node.otherwise) }; } - if ('is' in node) return { is: [map(node.is[0]), node.is[1]] }; - if ('contains' in node) return { contains: [map(node.contains[0]), node.contains[1]] }; if ('value' in node) return { ...node, value: map(node.value) }; - return node; + // A call: its one key names the operation. `is` and `contains` take a plain text after the value. + const [op, args] = Object.entries(node)[0] as [string, ExprInput[]]; + const mapped = op === 'is' || op === 'contains' ? [map(args[0]), args[1]] : args.map(map); + return { [op]: mapped } as ExprNode; } const state = (name: State): Expr => new Expr({ state: name }); -/** The field `name` of each row. */ -export const field = (name: string): Expr => new Expr({ field: name }); -/** A color constant. */ -export const color = (css: Color): Expr => new Expr({ color: css }); +/** + * Reads field `name` of each row. In JSON it is `'$name'`, or the `{ field }` form after `from()` + * or `step()`. + */ +export const field = (name: string): Expr => new Expr(`$${name}`); /** Whether the row is selected. */ export const selected = (): Expr => state('selected'); /** Whether the pointer is over the row. */ export const hovered = (): Expr => state('hovered'); -/** CSS pixels from a vertex to the pointer. */ +/** CSS pixels from a vertex to the pointer. Vertex marks only, and not in a vertex's position. */ export const pointer = (): Expr => state('pointer'); -/** Seconds since the view was created. */ +/** Seconds since the view was created. A mark that reads it redraws every frame. */ export const elapsed = (): Expr => state('elapsed'); /** The camera's zoom: the web mercator level on the earth, log₂ CSS pixels per unit elsewhere. */ export const zoom = (): Expr => state('zoom'); -/** A missing value: options that read one take the theme's. */ -export const missing = (): Expr => new Expr({ missing: true }); + +/** + * A size in CSS pixels in any space, so it stays the same on screen as the view zooms. `px(1)` + * keeps a diagram's stroke 1 pixel wide at any zoom. Given a number, the result also works as a + * theme length. + */ +export function px(size: number): { px: number }; +export function px(size: ExprInput): Px; +export function px(size: ExprInput): Px { + return { px: nodeOf(size) }; +} + /** - * A size in position units, so it scales as the view zooms, kept within `atLeast` and `atMost` CSS - * pixels when given: `units('r', { atLeast: 2 })` grows with the map and never vanishes. A marker - * keeps its proportions, its larger side held. + * A size in position units (degrees on the earth) in any space, so it scales as the view zooms. + * `limits` bound its size on screen in CSS pixels: `units('$r', { atLeast: 2 })` grows with the + * map and never shrinks below 2 pixels. A marker keeps its proportions, and the limits apply to its + * larger side. Given only a number, the result also works as a theme length. */ -export const units = (size: ExprInput, px: { atLeast?: number; atMost?: number } = {}): Size => ({ - units: nodeOf(size), - ...px, -}); +export function units(size: number): { units: number }; +export function units(size: ExprInput, limits?: Omit): Units; +export function units(size: ExprInput, limits: Omit = {}): Units { + return { units: nodeOf(size), ...limits }; +} -/** Starts a conditional: `when(c).then(a).when(d).then(b).otherwise(e)`. */ +/** One case of a conditional: a condition, and the value it selects. */ +type Case = [condition: ExprNode, value: ExprNode]; + +/** + * Starts a conditional. `when(c).then(a).when(d).then(b).otherwise(e)` takes the value of the first + * case whose condition holds, else `e`. The values may be numbers, colors, or a text field and + * `null`, whichever the option takes. + */ export function when(condition: ExprInput): When { - return new When([], nodeOf(condition)); + return new When(nodeOf(condition)); } +/** A conditional's first condition. `then` gives its value. */ class When { - constructor( - private readonly cases: [ExprNode, ExprNode][], - private readonly condition: ExprNode, - ) {} + /** @internal */ + constructor(private readonly condition: ExprNode) {} + + /** Sets the value where the condition holds. */ then(value: ExprInput): Then { - return new Then([...this.cases, [this.condition, nodeOf(value)]]); + return new Then([[this.condition, nodeOf(value)]]); } } +/** A conditional's cases so far. Add a case with `when`, or finish with `otherwise`. */ class Then { - constructor(private readonly cases: [ExprNode, ExprNode][]) {} - when(condition: ExprInput): When { - return new When(this.cases, nodeOf(condition)); + /** @internal */ + constructor(private readonly cases: readonly Case[]) {} + + /** Adds a case, tried when no earlier condition holds. */ + when(condition: ExprInput): Next { + return new Next(this.cases, nodeOf(condition)); } + + /** Sets the value where no condition holds, and returns the finished conditional. */ otherwise(value: ExprInput): Expr { - return new Expr({ when: this.cases, otherwise: nodeOf(value) }); + return new Expr({ when: [...this.cases], otherwise: nodeOf(value) }); } } -/** Straight-alpha RGBA pixels, compatible with browser ImageData and server-produced pixels. */ +/** A conditional's next condition. `then` gives its value. */ +class Next { + /** @internal */ + constructor( + private readonly cases: readonly Case[], + private readonly condition: ExprNode, + ) {} + + /** Sets the value where this condition holds. */ + then(value: ExprInput): Then { + return new Then([...this.cases, [this.condition, nodeOf(value)]]); + } +} + +/** RGBA pixels with straight alpha, rows top to bottom. A browser `ImageData` has this shape. */ export interface RgbaImage { + /** The width in pixels. */ width: number; + /** The height in pixels. */ height: number; + /** 4 bytes per pixel, red, green, blue, and alpha, row after row. */ data: Uint8ClampedArray; } -/** An icon marker showing one of `images` for each row, picked by `index`. */ +/** + * An icon marker: each row shows `images[index]`, fitted in the marker's box, with `index` counting + * from 0. The pixels are copied into the marker. + */ export function icon(images: readonly RgbaImage[], index: ExprInput): Marker { return { icon: { diff --git a/npm/src/shot.ts b/npm/src/shot.ts new file mode 100644 index 00000000..0a31854d --- /dev/null +++ b/npm/src/shot.ts @@ -0,0 +1,358 @@ +/** Pictures of views: images to encode as files, and H.264 video written as frames are drawn. */ + +import type * as bindings from '../pkg/latkit.js'; +import { encoded, type Placed } from './canvas.js'; +import { Handle, inner, thrown, waited, wasm, type LatkitError, type Wait } from './internal.js'; +import { canvasOf, type View } from './view.js'; + +/** Options for a {@link Shot}. */ +export interface ShotOptions { + /** + * Pixels per CSS pixel. Defaults to 1. Views lay out at `size / scale` CSS pixels, so 2 draws the + * same layout twice as sharp. + */ + scale?: number; + /** Leaves the background transparent where the theme's background color would be. */ + transparent?: boolean; + /** + * When the picture is taken. `'still'`, the default, waits until every view is complete and + * still: transitions, camera flights, and tile fades run until they finish. `'now'` waits only + * until every view is complete, and freezes anything moving where it is. + */ + capture?: 'still' | 'now'; + /** + * The time in seconds that `elapsed()` expressions and flowing comets show in a still picture. + * Defaults to the view's clock. + */ + elapsed?: number; +} + +/** + * A picture of views, `size` pixels in all, with each view drawn in its region. While a shot draws + * a view, the view's canvas keeps its last frame. An image draws each view's data as it was when + * the picture began. `Shot.of(view)` makes a shot of what a view's canvas shows. + * + * ```ts + * const png = await (await Shot.of(view, { scale: 2 }).image()).encode('png'); + * ``` + */ +export class Shot { + /** The picture's size, `[width, height]` in pixels. */ + readonly size: readonly [number, number]; + /** How the picture is taken, and in `views`, the views drawn into it. */ + readonly options: ShotOptions & { views?: readonly Placed[] }; + + /** Creates a shot of `size`, `[width, height]` in pixels. */ + constructor( + size: readonly [number, number], + options: ShotOptions & { + /** + * The views drawn, later views over earlier ones. Each draws in its own region, or in + * `region` when given as `{ view, region }`. + */ + views?: readonly Placed[]; + } = {}, + ) { + this.size = size; + this.options = options; + } + + /** + * Returns a shot of exactly what the canvas showing `view` shows: its views in their regions, at + * the canvas's CSS size times `options.scale` in pixels. Throws an `invalid` error for a view on + * no canvas. Give such a shot a size with the constructor. + */ + static of(view: View, options: ShotOptions = {}): Shot { + const canvas = canvasOf(view); + if (!canvas) + throw thrown({ + code: 'invalid', + message: 'the view is on no canvas: create a Shot of a size', + }); + const scale = options.scale ?? 1; + const [width, height] = canvas.cssSize.map((side) => Math.max(1, Math.round(side * scale))); + return new Shot([width!, height!], { ...options, views: canvas.views }); + } + + /** + * Draws the picture once every view is complete, as `capture` says, and resolves to the + * {@link Image}. Rejects with the first error a view reports while drawing. + */ + async image(wait: Wait = {}): Promise { + const taken = wasm().image(this.#json(), ...encoded(this.options.views ?? [])); + return Image.made(await waited(taken, wait, 'taking a picture')); + } + + /** + * Starts a video of this shot: H.264 in MP4, written to `options.output` as frames are drawn. The + * width is rounded down to a multiple of 4 and the height to an even number, as the encoder + * requires. Canvases showing the views keep their last frame until the video finishes or aborts. + * Aborting `options.signal` aborts the video and its output. Frames have no other time limit. + */ + video( + options: { + /** Frames per second. */ + fps: number; + /** Where the file's bytes go, locked until the video finishes or aborts. */ + output: WritableStream; + /** Bits per second. Defaults to about 0.15 bits per pixel per frame. */ + bitrate?: number; + } & Wait, + ): Video { + options.signal?.throwIfAborted(); + const writer = options.output.getWriter(); + try { + const views = encoded(this.options.views ?? []); + const video = wasm().video(this.#json(), ...views, options.fps, options.bitrate); + return Video.made(video, writer, options.fps, options.signal); + } catch (error) { + writer.releaseLock(); + throw error; + } + } + + /** The shot as JSON for the bindings, with times in seconds. The views are passed separately. */ + #json(): string { + const { scale = 1, transparent = false, capture = 'still', elapsed } = this.options; + return JSON.stringify({ size: [...this.size], scale, transparent, capture, elapsed }); + } +} + +/** A picture's RGBA pixels, with straight alpha, rows top to bottom. */ +export class Image extends Handle { + private constructor(image: bindings.Image) { + super(image, 'Image'); + } + + /** @internal Wraps a bindings image. */ + static made(image: bindings.Image): Image { + return new Image(image); + } + + /** Width in pixels. */ + get width(): number { + return inner(this).width; + } + + /** Height in pixels. */ + get height(): number { + return inner(this).height; + } + + /** A copy of the pixels: 4 bytes per pixel, `width * height * 4` bytes in all. */ + get rgba(): Uint8Array { + return inner(this).rgba; + } + + /** + * Encodes the image as a file, in a worker. PNG is lossless with alpha. JPEG drops alpha. WebP + * is lossless with alpha and smaller than PNG. + */ + encode( + format: ImageFormat, + options: { + /** JPEG quality, from 1 (smallest) to 100 (sharpest). Defaults to 90. */ + quality?: number; + } & Wait = {}, + ): Promise { + const encoding = inner(this).encode(format, options.quality); + return waited(encoding, options, 'encoding an image'); + } +} + +/** An image file format for {@link Image.encode}. */ +export type ImageFormat = 'png' | 'jpeg' | 'webp'; + +/** + * A video being drawn, encoded, and written. Create one with {@link Shot.video}. Draw frames with + * {@link Video.frame} or {@link Video.record}, then call {@link Video.finish}. + */ +export class Video { + #video: bindings.Video | undefined; + #state: 'open' | 'finishing' | 'aborting' | 'closed' = 'open'; + /** The operation in progress, and the bindings task it waits on. */ + #operation?: Promise; + #task?: bindings.Task; + #finished?: Promise; + #aborted?: Promise; + #reason: unknown; + #frames = 0; + #removeAbort?: () => void; + #writer: WritableStreamDefaultWriter; + #fps: number; + + private constructor( + video: bindings.Video, + writer: WritableStreamDefaultWriter, + fps: number, + signal?: AbortSignal, + ) { + this.#video = video; + this.#writer = writer; + this.#fps = fps; + if (signal) { + const abort = () => void this.abort(signal.reason); + signal.addEventListener('abort', abort, { once: true }); + this.#removeAbort = () => signal.removeEventListener('abort', abort); + if (signal.aborted) abort(); + } + } + + /** @internal Wraps a bindings video whose bytes go to `writer`. */ + static made( + video: bindings.Video, + writer: WritableStreamDefaultWriter, + fps: number, + signal?: AbortSignal, + ): Video { + return new Video(video, writer, fps, signal); + } + + /** The time in seconds at which the next frame shows. Still readable after the video ends. */ + get time(): number { + return this.frames / this.#fps; + } + + /** + * Frames recorded so far, including frames repeated because nothing moved. Still readable after + * the video ends. + */ + get frames(): number { + return this.#video?.frames ?? this.#frames; + } + + /** + * Draws the next frame once its views are complete, then writes its bytes, waiting on the + * output's backpressure. Calls must not overlap. A call made before the previous one resolves, + * or after `finish()`, rejects with an `invalid` error. A call after `abort()` rejects with a + * `cancelled` error. A failure aborts the video. + */ + frame(): Promise { + if (this.#state !== 'open') return Promise.reject(this.#ended()); + if (this.#operation) + return Promise.reject( + thrown({ code: 'invalid', message: 'a Video draws one frame at a time' }), + ); + return this.#run(async () => { + const bytes = await this.#await(this.#open().frame()); + // An abort while the frame drew rejects it, even if the frame finished first. + this.#open(); + if (bytes.length) await this.#writer.write(bytes); + }); + } + + /** + * Records frames until `step(time)` returns false, then finishes the video. `step` gets the next + * frame's time in seconds and can change the views before that frame is drawn. `onProgress` is + * called after each frame. An error from `step`, drawing, the output, or `onProgress` aborts the + * video and is rethrown. + */ + async record( + step: (time: number) => boolean | Promise, + onProgress?: (progress: { frames: number; time: number }) => void, + ): Promise { + try { + while (await step(this.time)) { + await this.frame(); + onProgress?.({ frames: this.frames, time: this.time }); + } + await this.finish(); + } catch (error) { + await this.abort(error); + throw error; + } + } + + /** + * Waits for the frame in progress, encodes the remaining frames, writes the end of the file, and + * closes and unlocks the output. Later calls return the same promise. Rejects with a `cancelled` + * error if the video was aborted. + */ + finish(): Promise { + if (this.#finished) return this.#finished; + if (this.#state !== 'open') return Promise.reject(this.#ended()); + this.#state = 'finishing'; + const frame = this.#operation; + this.#finished = this.#run(async () => { + await frame; + const bytes = await this.#await(this.#open().finish()); + if (bytes.length) await this.#writer.write(bytes); + this.#open(); + await this.#writer.close(); + }).then(() => { + if (this.#state === 'finishing') this.#release(); + }); + return this.#finished; + } + + /** + * Cancels drawing and encoding, aborts the output with `reason`, and unlocks it once pending + * writes settle. Later calls return the same promise. After `finish()` completes, it does + * nothing. + */ + abort(reason?: unknown): Promise { + if (this.#aborted) return this.#aborted; + if (this.#state === 'closed') return Promise.resolve(); + this.#state = 'aborting'; + this.#reason = reason; + this.#frames = this.frames; + this.#task?.cancel(); + this.#video?.cancel(); + this.#aborted = Promise.allSettled([this.#operation, this.#writer.abort(reason)]).then(() => + this.#release(), + ); + return this.#aborted; + } + + /** Runs `operation` as the one in progress. A failure aborts the video and is rethrown. */ + #run(operation: () => Promise): Promise { + const running = operation(); + this.#operation = running; + return running + .catch(async (error: unknown) => { + await this.abort(error); + throw error; + }) + .finally(() => { + if (this.#operation === running) this.#operation = undefined; + }); + } + + /** Awaits bindings task `task`, which `abort()` cancels, and frees it. */ + async #await(task: bindings.Task): Promise { + this.#task = task; + try { + return (await task.promise) as Uint8Array; + } finally { + this.#task = undefined; + task.free(); + } + } + + /** Returns the bindings video while it is open or finishing. Otherwise throws why it ended. */ + #open(): bindings.Video { + const open = this.#state === 'open' || this.#state === 'finishing'; + if (!open || !this.#video) throw this.#ended(); + return this.#video; + } + + /** + * The error for a call after the video ended: `cancelled` with the abort reason as its `cause` + * once aborted, else `invalid`. + */ + #ended(): LatkitError { + return this.#aborted + ? thrown({ code: 'cancelled', message: 'this Video was aborted', cause: this.#reason }) + : thrown({ code: 'invalid', message: 'a Video draws no frames after finish()' }); + } + + #release(): void { + this.#frames = this.frames; + this.#video?.free(); + this.#video = undefined; + this.#removeAbort?.(); + this.#removeAbort = undefined; + this.#writer.releaseLock(); + this.#state = 'closed'; + } +} diff --git a/npm/src/view.ts b/npm/src/view.ts new file mode 100644 index 00000000..1da4891b --- /dev/null +++ b/npm/src/view.ts @@ -0,0 +1,443 @@ +/** + * Views: a scene drawn from its datasets on a canvas. A view's state is a set of properties. + * Setting one emits no event. When the user changes one, the view emits an event named after the + * property. Queries are methods, answered by the frames that draw the view. + */ + +import type * as bindings from '../pkg/latkit.js'; +import type { Canvas, Rect } from './canvas.js'; +import { Snapshot, sourceOf, Table, type Source } from './data.js'; +import { device, Gpu } from './gpu.js'; +import type { Camera, Fit, Row, Rows, Scene } from './inputs.js'; +import { Handle, inner, thrown, waited, type LatkitError, type Wait } from './internal.js'; +import type { Failure, Hit, Stats, ViewEvent, Waiting } from './outputs.js'; +import { Patch } from './patch.js'; + +/** The region covering a whole canvas or picture. */ +const WHOLE: Rect = [0, 0, 1, 1]; + +/** @internal Returns the canvas showing `view`, if any. */ +export let canvasOf: (view: View) => Canvas | undefined; + +/** + * The events a view emits, mapping each type to its detail. Intents (`click`, `open`, `menu`, + * `drag`, `connect`, `delete`) are emitted once each, in the order the user acted. Changes + * (`hovered`, `selection`, `camera`, `waiting`) are named after the property that changed and carry + * only its latest value. An `error` event's detail is a {@link LatkitError}. Setters emit nothing, + * so events report only what the application did not do itself. + */ +export type ViewEvents = { + [E in ViewEvent as E['type']]: E extends { type: 'error' } ? LatkitError : E['detail']; +}; + +/** CSS pixels kept clear at the top, right, bottom, and left. */ +export type Insets = [top: number, right: number, bottom: number, left: number]; + +/** A point on screen, in CSS pixels from the view's top left corner. */ +export type Point = [x: number, y: number]; + +/** + * A position in the space's coordinates: x, y, and elevation. On the earth, x and y are longitude + * and latitude; on a plot, the axis coordinate and the value. + */ +export type Position = [x: number, y: number, elevation: number]; + +/** + * A scene drawn from its datasets on a canvas and into pictures. Create one with + * {@link View.create}. The canvas passes it pointer, wheel, and key input, and a `Shot` draws it + * into pictures. Destroying a view rejects its unanswered queries, and its canvas keeps the last + * frame. + * + * ```ts + * const grid = new Data('grid').set('Bus', [1, 2], { x: [0, 30], y: [0, 20], kv: [230, 115] }); + * const view = await View.create(canvas, { + * space: 'plane', + * marks: [{ vertices: grid.table('Bus'), position: ['$x', '$y'], color: field('kv').colormap('viridis') }], + * }); + * ``` + */ +export class View extends Handle { + static { + canvasOf = (view) => view.#canvas; + } + + #listeners = new Map void>>(); + #gpu: Gpu; + #canvas: Canvas | undefined; + #region: Rect; + #scene: Scene; + /** The datasets of the tables the scene names. */ + #tables: readonly Source[]; + #sources: readonly Source[] = []; + + private constructor(gpu: Gpu, scene: Scene, region: Rect) { + // The view sends events once it exists, so the callback finds it through `told`. + const told: { view?: View } = {}; + const tell = (json: string) => { + if (told.view) told.view.#tell(json); + }; + const { json, tables } = encoded(scene); + super(device(gpu).view(json, tell), 'View'); + told.view = this; + this.#gpu = gpu; + this.#region = region; + this.#scene = scene; + this.#tables = tables; + } + + /** + * Creates a view that draws `scene` on `canvas`, in `options.region`, over any views already + * there. With a `null` canvas, the view draws only into pictures. The view loads every dataset + * its scene uses. Unless given a camera, it frames everything when it first draws. + */ + static async create( + canvas: HTMLCanvasElement | OffscreenCanvas | null, + scene: Scene, + options: { + /** The part of the canvas, and of pictures, that the view draws in. Defaults to all of it. */ + region?: Rect; + /** Where the view looks first. Without one, it frames everything drawn. */ + camera?: Camera; + /** Datasets to draw besides those the scene uses. See {@link View.sources}. */ + sources?: readonly Source[]; + /** + * The GPU that draws the view. Defaults to the first `Gpu` created, creating one if needed. + */ + gpu?: Gpu; + /** + * Pixels per CSS pixel of an `OffscreenCanvas`. Defaults to 1. The first view on the canvas + * sets it. + */ + pixelRatio?: number; + } = {}, + ): Promise { + const gpu = options.gpu ?? (await Gpu.shared()); + const view = new View(gpu, scene, options.region ?? WHOLE); + try { + view.sources = options.sources ?? []; + if (options.camera) view.camera = options.camera; + else view.fit(); + view.#canvas = gpu.adopt(view, canvas, options.pixelRatio); + } catch (error) { + view.destroy(); + throw error; + } + return view; + } + + /** The GPU that draws the view. */ + get gpu(): Gpu { + return this.#gpu; + } + + /** + * The part of its canvas, and of pictures, that the view draws in: `[x, y, width, height]` in + * fractions, from the top left. Views on one canvas draw in the order they were created. + */ + get region(): Rect { + return this.#region; + } + + set region(region: Rect) { + this.#region = region; + this.#canvas?.relayout(); + } + + /** + * The scene drawn. A mark that matches one in the previous scene by kind, dataset, table, and + * name keeps its state and eases to its new options. An invalid scene leaves the current one in + * place. + */ + get scene(): Scene { + return this.#scene; + } + + set scene(scene: Scene) { + const { json, tables } = encoded(scene); + const changed = + tables.length !== this.#tables.length || tables.some((t, i) => t !== this.#tables[i]); + const drawn = changed ? this.#drawn(tables, this.#sources) : undefined; + view(this).setScene(json); + this.#scene = scene; + this.#tables = tables; + if (drawn) view(this).setSources(drawn); + } + + /** + * Datasets the view draws besides those its scene uses. A source replaces a scene dataset with + * the same name. A mark whose table is written `[data, table]` draws the source named `data`. The + * view draws a `Data` at its latest version from the frame after each change, a `Snapshot` as it + * is, and `Tiles` as the pieces it needs. A mark whose dataset no longer fits it draws nothing. + * It is listed in `failures`, and the next frame emits an `error` event. + */ + get sources(): readonly Source[] { + return this.#sources; + } + + set sources(sources: readonly Source[]) { + view(this).setSources(this.#drawn(this.#tables, sources)); + this.#sources = [...sources]; + } + + /** + * The datasets drawn, as the bindings take them: `sources`, then those of the scene's `tables` + * that no source replaces by name. Throws an `invalid` error when two datasets share a name. + */ + #drawn(tables: readonly Source[], sources: readonly Source[]): bindings.Source[] { + const named = new Set(sources.map((source) => source.name)); + const drawn = [...sources, ...tables.filter((table) => !named.has(table.name))]; + const twice = drawn.find((source, i) => drawn.findIndex((s) => s.name === source.name) !== i); + if (twice) + throw thrown({ + code: 'invalid', + message: `two datasets are named '${twice.name}'`, + path: 'sources', + }); + return drawn.map(sourceOf); + } + + /** + * The axis coordinate shown, such as a time. Marks read sampled fields at it, interpolating + * between the frame at or before it and the next, unless an expression uses `step()`. + */ + get at(): number { + return view(this).at; + } + + set at(at: number) { + view(this).at = at; + } + + /** + * Where the view looks. Setting it moves the camera at once, replacing a flight or a fit not yet + * applied. Use {@link View.fly} to ease. A camera of the wrong kind for the space, such as a plot + * camera on a map, is ignored. + * + * Camera moves stay within the camera's zoom and pan limits. On the earth, the space sets the + * limits inside the insets, as `Earth` describes. A fit narrows them to 3 zoom levels out and 8 + * in from the box it framed, with the center near that box. When a resize, new insets, or a new + * space moves the camera back within its limits, a `camera` event reports it. + */ + get camera(): Camera { + return JSON.parse(view(this).camera) as Camera; + } + + set camera(camera: Camera) { + view(this).setCamera(JSON.stringify(camera)); + } + + /** + * CSS pixels kept clear at the top, right, bottom, and left, such as where page panels cover the + * view. The camera centers and fits within the rest. + */ + get insets(): Insets { + const [top, right, bottom, left] = view(this).insets; + return [top!, right!, bottom!, left!]; + } + + set insets(insets: Insets) { + view(this).insets = Float32Array.from(insets); + } + + /** + * Degrees per second the camera turns about its center on a map or plane. 0 stops it. A diagram + * never turns. + */ + get orbit(): number { + return view(this).orbit; + } + + set orbit(degreesPerSecond: number) { + view(this).orbit = degreesPerSecond; + } + + /** The selected rows, one set per table. Setting it selects exactly those rows. */ + get selection(): readonly Rows[] { + return JSON.parse(view(this).selection) as Rows[]; + } + + set selection(rows: readonly Rows[]) { + view(this).setSelection(JSON.stringify(rows)); + } + + /** The row under the pointer or the row keyboard navigation moved to, or `null`. */ + get hovered(): Hit | null { + return JSON.parse(view(this).hovered) as Hit | null; + } + + /** + * The kinds of pending work that kept the last frame from being complete. Empty when it was + * complete. See {@link Waiting}. + */ + get waiting(): Waiting { + return JSON.parse(view(this).waiting) as Waiting; + } + + /** Whether the last frame was complete, with nothing in `waiting`. Shots wait for this. */ + get complete(): boolean { + return view(this).complete; + } + + /** + * An error for each mark that draws nothing because its data no longer fits it, such as a missing + * table. Each error's `path` names the mark. + */ + get failures(): readonly LatkitError[] { + return (JSON.parse(view(this).failures) as Failure[]).map(thrown); + } + + /** Counts and timings of the view's frames; see {@link Stats}. */ + get stats(): Stats { + return JSON.parse(view(this).stats) as Stats; + } + + /** Whether the view needs a frame, because it changed or work it was waiting on arrived. */ + get wantsFrame(): boolean { + return view(this).wantsFrame; + } + + /** + * Frames what `fit` names, keeping pitch and bearing. Defaults to every row. It takes effect on + * the next frame, or a frame or two later when the GPU must measure the extent. + */ + fit(fit: Fit = 'all'): void { + view(this).fit(JSON.stringify(fit)); + } + + /** + * Eases the camera to `camera` over `seconds`, or over twice the scene's ease time without it, + * replacing a fit not yet applied. When motion is reduced, it moves at once. The camera stays + * within its limits, as {@link View.camera} describes. A camera of the wrong kind for the space is + * ignored. + */ + fly(camera: Camera, seconds?: number): void { + view(this).fly(JSON.stringify(camera), seconds); + } + + /** + * Projects `position` to a point on screen with the current camera, for anchoring HTML overlays. + * Returns `null` when the position is behind the camera. + */ + screen(position: Position): Point | null { + const point = view(this).screen(Float64Array.from(position)); + return point ? [point[0]!, point[1]!] : null; + } + + /** + * Returns the position on the ground under screen point `point`: at elevation 0 on a plane, on + * the map or globe on the earth, and the values there on a plot. Returns `null` where the point + * looks past the ground. + */ + position(point: Point): Position | null { + const position = view(this).position(Float32Array.from(point)); + return position ? [position[0]!, position[1]!, position[2]!] : null; + } + + /** + * Resolves to where `row` is on screen once a frame draws the view, or to `null` when no mark + * draws it in view. A diagram's vertices resolve at once. + */ + screenOf(row: Row, wait: Wait = {}): Promise { + return waited(view(this).screenOf(JSON.stringify(row)), wait, 'locating a row'); + } + + /** + * Resolves, once a frame draws the view, to every row of a pickable mark near `point`, nearest + * first. + */ + async pick( + point: Point, + options: { + /** How near, in CSS pixels. Defaults to 0. */ + radius?: number; + } & Wait = {}, + ): Promise { + const hits = view(this).pick(Float32Array.from(point), options.radius ?? 0); + return JSON.parse(await waited(hits, options, 'picking')) as Hit[]; + } + + /** + * Returns the rows connected to `row`, one set per table: a vertex's neighbors across edges and + * nets, or the vertices of an edge or net. The cost is proportional to the row's degree. + */ + neighbors(row: Row): Rows[] { + return JSON.parse(view(this).neighbors(JSON.stringify(row))) as Rows[]; + } + + /** + * Lays out again every vertex row that has no position, and resolves to one patch per dataset. + * Each patch writes each row's place into its mark's position fields, so applying it makes the + * places part of the data. Marks that don't read their position from fields are left out. + * + * ```ts + * for (const patch of await view.arrange()) if (patch.data === grid.name) grid.apply(patch); + * ``` + */ + async arrange(wait: Wait = {}): Promise { + const frames = await waited(view(this).arrange(), wait, 'arranging'); + return Promise.all(frames.map((frame) => Patch.read(frame))); + } + + /** + * Returns the version of dataset `name` that the view draws, or `undefined` if it draws no + * dataset by that name. That is the latest version of a `Data` the view has taken, the snapshot + * it was given, or the pieces of `Tiles` fetched for its camera. + */ + snapshot(name: string): Snapshot | undefined { + const snapshot = view(this).snapshot(name); + return snapshot ? Snapshot.made(snapshot) : undefined; + } + + /** + * Calls `listener` with each event of `type` emitted while it is registered. Returns a function + * that removes it. Past events are not replayed. Read the current state from the property that a + * change event is named after. + */ + on(type: T, listener: (event: ViewEvents[T]) => void): () => void { + let set = this.#listeners.get(type); + if (!set) this.#listeners.set(type, (set = new Set())); + set.add(listener); + return () => set.delete(listener); + } + + /** Dispatches a JSON batch of the view's events to the listeners of each event's type. */ + #tell(json: string): void { + if (this.#listeners.size === 0) return; + for (const told of JSON.parse(json) as ViewEvent[]) { + const listeners = this.#listeners.get(told.type) as Set<(event: unknown) => void> | undefined; + if (!listeners?.size) continue; + const event = told.type === 'error' ? thrown(told.detail) : told.detail; + for (const listener of listeners) listener(event); + } + } + + /** @internal Drops the listeners, and takes the view off its canvas and its GPU. */ + protected override freeing(): void { + for (const listeners of this.#listeners.values()) listeners.clear(); + this.#listeners.clear(); + this.#canvas?.remove(this); + this.#canvas = undefined; + this.#gpu.release(this); + } +} + +function view(owner: View): bindings.View { + return inner(owner); +} + +/** `scene` as JSON, and the datasets of the tables it names. */ +function encoded(scene: Scene): { json: string; tables: Source[] } { + const tables = new Set(); + const json = JSON.stringify(scene, function (this: Record, key, value: unknown) { + // `value` is what `toJSON` returned. The holder still has the table itself. + const held = this[key]; + if (held instanceof Table) tables.add(held.source); + return value; + }); + return { json, tables: [...tables] }; +} + +/** @internal Returns a new bindings handle to the view, for a canvas to show or a shot to draw. */ +export function handleOf(owner: View): bindings.Handle { + return view(owner).handle(); +} diff --git a/npm/src/worker.ts b/npm/src/worker.ts index 96ea85de..ce4a71ce 100644 --- a/npm/src/worker.ts +++ b/npm/src/worker.ts @@ -1,6 +1,7 @@ /** - * A worker doing views' jobs: it instantiates the module the page posts first, then answers each - * job's bytes with its answer's. + * The worker that runs views' jobs. The page's first message is the compiled WebAssembly module, + * which the worker instantiates and acknowledges with `{ ready: true }`. Each later message is a + * job's bytes, answered with `{ bytes }` holding the result. Failures post `{ error }`. */ import { initSync, job } from '../pkg/latkit.js'; diff --git a/npm/test/consumer/core.ts b/npm/test/consumer/core.ts deleted file mode 100644 index 46ec0a5f..00000000 --- a/npm/test/consumer/core.ts +++ /dev/null @@ -1,11 +0,0 @@ -import { Patch, field, icon, type Scene } from 'latkit/core'; -const scene: Scene = { - space: 'plane', - marks: [{ vertices: { of: ['grid', 'T'], position: [field('x'), field('y')] } }], -}; -const patch = new Patch().replace('T', [1], { x: [1], y: [2] }); -const marker = icon( - [{ width: 1, height: 1, data: new Uint8ClampedArray([255, 0, 0, 255]) }], - field('i'), -); -void [scene, patch.toBytes(), marker]; diff --git a/npm/test/consumer/main.ts b/npm/test/consumer/main.ts index 15b04f45..9ba2ff80 100644 --- a/npm/test/consumer/main.ts +++ b/npm/test/consumer/main.ts @@ -1,4 +1,4 @@ -import { loadLatkit, field, Patch, type Scene } from 'latkit'; +import { load, field, Data, Gpu, Patch, Shot, View, type Scene } from 'latkit'; import { quickstart } from './quickstart'; import wasmUrl from 'latkit/wasm?url'; import workerUrl from 'latkit/worker?url'; @@ -12,8 +12,8 @@ function destroyed(action: () => unknown): void { action(); } catch (error) { check( - String(error).includes('destroyed'), - `unexpected disposed-handle error: ${String(error)}`, + error instanceof Error && 'code' in error && error.code === 'destroyed', + `unexpected error from a destroyed handle: ${String(error)}`, ); return; } @@ -28,18 +28,17 @@ async function until(condition: () => boolean): Promise { } } +// The scene names its dataset, so each view draws the `grid` in its own sources. const scene: Scene = { space: { plane: { grid: 'off' } }, theme: 'dark', ease: 0, marks: [ { - vertices: { - of: ['grid', 'Bus'], - position: ['x', 'y'], - radius: field('radius'), - color: '#ff0000', - }, + vertices: ['grid', 'Bus'], + position: ['$x', '$y'], + radius: field('radius'), + color: '#ff0000', }, ], }; @@ -52,49 +51,47 @@ async function run(custom: boolean, compiled: boolean) { new Blob([await (await fetch(workerUrl)).text()], { type: 'text/javascript' }), ); } - const createWorker = () => { + const worker = () => { createdWorkers++; return new Worker(blob!); }; const started = performance.now(); const source = compiled ? await WebAssembly.compileStreaming(fetch(wasmUrl)) : wasmUrl; - const loading = loadLatkit( - custom ? { wasm: source, createWorker } : compiled ? { wasm: source } : undefined, - ); - check(loading === loadLatkit(), 'concurrent loads must share their promise'); - const kit = await loading; + const loading = load(custom ? { wasm: source, worker } : compiled ? { wasm: source } : undefined); + check(loading === load(), 'concurrent loads must share their promise'); + await loading; const initialized = performance.now(); - check(kit === (await loadLatkit()), 'subsequent loads must reuse the API'); + check(load() === loading, 'later loads must reuse the first'); let conflict = false; try { - await loadLatkit({ wasm: '/a-different-module.wasm' }); + await load({ wasm: '/a-different-module.wasm' }); } catch { conflict = true; } check(conflict, 'conflicting configuration must reject'); check(createdWorkers === 0, 'loading must not create workers'); - const { Data, Gpu, Shot } = kit; const data = new Data('grid').apply( - Patch.to('grid').replace('Bus', { count: 1 }, { x: [0], y: [0], radius: [12] }), + new Patch('grid').replace('Bus', { count: 1 }, { x: [0], y: [0], radius: [12] }), ); const empty = new Data('grid'); empty.destroy(); empty.destroy(); destroyed(() => empty.name); const decoded = await Data.read(data.toBytes()); + const kept = decoded.snapshot(); - // Data reads back; patches stream back to back. + // Queries read data back, and patches apply from a stream split into small chunks. const grid = new Data('grid').apply( new Patch() .axis('time', 's') .replace('Branch', [7, 8, 9], { loading: [40, 120, 95] }) - .append([0, 1, 2], 'Branch', { flow: Float32Array.of(1, 2, 3, 4, 5, 6, 7, 8, 9) }), + .append('Branch', [0, 1, 2], { flow: Float32Array.of(1, 2, 3, 4, 5, 6, 7, 8, 9) }), ); const hot = grid.query({ of: 'Branch', filter: field('loading').gt(90), - select: { flow: 'flow' }, + select: { flow: '$flow' }, at: 1, }); check(hot.ids.join() === '8,9' && hot.total === 2, 'a query filters'); @@ -112,7 +109,7 @@ async function run(custom: boolean, compiled: boolean) { }); for await (const next of Patch.stream(pieces)) grid.apply(next); check( - grid.ids('Branch').join() === '7,9' && grid.get('Branch', 7, 'loading') === 1, + grid.query({ of: 'Branch' }).ids.join() === '7,9' && grid.get('Branch', 7, 'loading') === 1, 'patches stream', ); grid.destroy(); @@ -120,27 +117,32 @@ async function run(custom: boolean, compiled: boolean) { const secondGpu = await Gpu.create(); const deviceReady = performance.now(); const canvas = (id: string) => document.querySelector(`#${id}`)!; - const first = gpu.view(scene); - const second = gpu.view(scene); - const third = secondGpu.view(scene); - gpu.canvas(canvas('first')).show([first]); - gpu.canvas(canvas('second')).show([second]); - secondGpu.canvas(canvas('third')).show([third]); - // A view no canvas shows still draws in shots. - const headless = gpu.view(scene); + const sources = [decoded]; + const first = await View.create(canvas('first'), scene, { gpu, sources }); + const second = await View.create(canvas('second'), scene, { gpu, sources }); + const third = await View.create(canvas('third'), scene, { gpu: secondGpu, sources: [kept] }); + // A view on no canvas still draws in shots. + const headless = await View.create(null, scene, { gpu, sources }); + check(first.gpu === gpu && third.gpu === secondGpu, 'views draw with the GPU they are given'); for (const view of [first, second, third, headless]) { view.on('error', (error) => { throw error; }); - view.setData(decoded); } - decoded.destroy(); // views retain the data + decoded.destroy(); // the views keep their own reference to the dataset + kept.destroy(); await until(() => [first, second, third].every((view) => view.stats.frames > 0 && view.stats.drawn === 1), ); const firstFrame = performance.now(); - const picture = await new Shot([64, 64]).image([headless]); + const picture = await new Shot([64, 64], { views: [headless] }).image(); check(picture.width === 64 && picture.height === 64, 'a view without a canvas draws in a shot'); + const shown = Shot.of(first, { scale: 2 }); + const css = canvas('first').getBoundingClientRect(); + check( + shown.size.join() === [css.width * 2, css.height * 2].map(Math.round).join(), + "a view's shot is its canvas's size", + ); const png = await picture.encode('png'); const jpeg = await picture.encode('jpeg', { quality: 80 }); const webp = await picture.encode('webp'); @@ -170,37 +172,29 @@ async function run(custom: boolean, compiled: boolean) { chunks.push(bytes); }, }); - const video = new Shot([64, 64]).video(gpu, 30, output); - await video.record([headless], (time) => time < 0.1); + const video = new Shot([64, 64], { views: [headless] }).video({ fps: 30, output }); + await video.record((time) => time < 0.1); check(video.frames === 3 && !output.locked, 'video retains counters and releases its output'); check( chunks.some((bytes) => new TextDecoder().decode(bytes).includes('ftyp')), 'video writes an MP4 header', ); const cancelledOutput = new WritableStream(); - const cancelled = new Shot([64, 64]).video(gpu, 30, cancelledOutput); - const pending = cancelled.frame([headless]); + const cancelled = new Shot([64, 64], { views: [headless] }).video({ + fps: 30, + output: cancelledOutput, + }); + const pending = cancelled.frame(); const settled = Promise.allSettled([pending]); await Promise.resolve(); await cancelled.abort('consumer cancellation'); check((await settled)[0].status === 'rejected', 'cancellation interrupts the WASM frame'); check(!cancelledOutput.locked, 'cancelled video releases the writer'); - const restored = await new Shot([64, 64]).image([headless]); + const restored = await new Shot([64, 64], { views: [headless] }).image(); restored.destroy(); } headless.destroy(); - check( - (() => { - try { - gpu.canvas(document.createElement('canvas')).show([second]); - return false; - } catch { - return true; - } - })(), - 'a view shows in one canvas at a time', - ); - const hits = await first.pick([128, 128], 8); + const hits = await first.pick([128, 128], { radius: 8 }); check( hits.some((hit) => hit.row.table === 'Bus'), 'GPU picking must see the rendered vertex', @@ -214,15 +208,15 @@ async function run(custom: boolean, compiled: boolean) { destroyed(() => first.stats); check(second.stats.drawn === 1, 'destroying one view must preserve its sibling'); const pending = Promise.allSettled([ - second.pick([128, 128], 8), - second.locate({ data: 'grid', table: 'Bus', id: 0 }), + second.pick([128, 128], { radius: 8 }), + second.screenOf({ data: 'grid', table: 'Bus', id: 0 }), ]); gpu.destroy(); gpu.destroy(); stop(); stop(); destroyed(() => second.stats); - destroyed(() => gpu.view(scene)); + destroyed(() => gpu.budget); for (const answer of await pending) { check(answer.status === 'rejected', 'destroying the view must cancel pending queries'); const error: unknown = answer.reason; @@ -231,7 +225,7 @@ async function run(custom: boolean, compiled: boolean) { 'query must reject with cancellation', ); } - check(data.ids('Bus').length === 1, 'GPU cleanup must preserve independent data'); + check(data.query({ of: 'Bus' }).ids.length === 1, 'GPU cleanup must preserve independent data'); const stoppedTicks = ticks; let otherTicks = 0; const stopOther = secondGpu.on('frame', () => otherTicks++); @@ -250,8 +244,8 @@ async function run(custom: boolean, compiled: boolean) { ], }), ); - const places = await Data.read(bytes, 'places'); - check(places.ids('points').length === 1, 'worker must decode GeoJSON'); + const places = await Data.read(bytes, { name: 'places' }); + check(places.query({ of: 'points' }).ids.length === 1, 'worker must decode GeoJSON'); if (custom) check(createdWorkers > 0, 'custom worker factory must be used'); places.destroy(); secondGpu.destroy(); @@ -261,8 +255,12 @@ async function run(custom: boolean, compiled: boolean) { document.body.append(exampleCanvas); const example = await quickstart(exampleCanvas); await until(() => example.view.stats.drawn === 2); - example.gpu.destroy(); - example.data.destroy(); + check( + example.view.gpu !== gpu, + 'a view given no GPU draws with a new one once the first is gone', + ); + example.view.gpu.destroy(); + example.grid.destroy(); exampleCanvas.remove(); const transient = await Gpu.create(); await new Promise((resolve) => { diff --git a/npm/test/consumer/quickstart.ts b/npm/test/consumer/quickstart.ts index 237b64de..2408c244 100644 --- a/npm/test/consumer/quickstart.ts +++ b/npm/test/consumer/quickstart.ts @@ -1,28 +1,19 @@ -import { loadLatkit, Patch, field, type Scene } from 'latkit'; +import { Data, View, field, load } from 'latkit'; -/** A complete, type-checked example, also rendered by the installed-package test. */ +/** Draws two buses on `canvas`, colored by their voltage. */ export async function quickstart(canvas: HTMLCanvasElement) { - const { Data, Gpu } = await loadLatkit(); - const data = new Data('grid').apply( - new Patch().replace('Bus', [101, 102], { x: [0, 30], y: [0, 20], kv: [230, 115] }), - ); - const scene: Scene = { + await load(); + const grid = new Data('grid').set('Bus', [101, 102], { x: [0, 30], y: [0, 20], kv: [230, 115] }); + const view = await View.create(canvas, { space: 'plane', marks: [ { - vertices: { - of: ['grid', 'Bus'], - position: ['x', 'y'], - radius: 6, - color: field('kv').colormap('viridis'), - }, + vertices: grid.table('Bus'), + position: ['$x', '$y'], + radius: 6, + color: field('kv').colormap('viridis'), }, ], - }; - const gpu = await Gpu.create(); - const view = gpu.view(scene); - gpu.canvas(canvas).show([view]); - view.setData(data); - view.fit(); - return { gpu, view, data }; + }); + return { view, grid }; } diff --git a/npm/test/consumer/tsconfig.core.json b/npm/test/consumer/tsconfig.core.json deleted file mode 100644 index 19f0bcdc..00000000 --- a/npm/test/consumer/tsconfig.core.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "compilerOptions": { - "target": "ES2022", - "module": "NodeNext", - "moduleResolution": "NodeNext", - "lib": ["ES2022"], - "types": [], - "strict": true, - "skipLibCheck": false, - "noEmit": true - }, - "files": ["core.ts"] -} diff --git a/npm/test/data.test.mjs b/npm/test/data.test.mjs index 578a0adf..93b77eae 100644 --- a/npm/test/data.test.mjs +++ b/npm/test/data.test.mjs @@ -1,5 +1,5 @@ -// Patches written in TypeScript and read by latkit's WebAssembly: the two halves of the byte -// format agree on every kind of value, and data reads back as it was written. Runs against the +// Tests patches written in TypeScript and read by latkit's WebAssembly. Both sides of the binary +// format must agree on every kind of value, and data must read back unchanged. Runs against the // built package (`node npm/build.mjs compile npm`) or the installed one (LATKIT_TEST_PACKAGE). import assert from 'node:assert/strict'; import fs from 'node:fs'; @@ -8,20 +8,23 @@ import { test } from 'node:test'; import { pathToFileURL } from 'node:url'; const directory = process.env.LATKIT_TEST_PACKAGE ?? path.resolve(import.meta.dirname, '..'); -const { loadLatkit, Patch, reference, field } = await import( +const { load, Data, Patch, reference, field } = await import( pathToFileURL(path.join(directory, 'dist/index.js')).href ); const wasm = new WebAssembly.Module(fs.readFileSync(path.join(directory, 'pkg/latkit_bg.wasm'))); -const { Data } = await loadLatkit({ wasm }); +await load({ wasm }); -/** Every chunk of `bytes`, `size` at a time, as a socket might hand them over. */ +/** Yields `bytes` in chunks of `size`, as a socket might deliver them. */ function* pieces(bytes, size) { for (let at = 0; at < bytes.length; at += size) yield bytes.subarray(at, at + size); } -test('patches carry every kind of value through the bytes', () => { +/** The ids of `table` in `data`, in row order. */ +const ids = (data, table) => data.query({ of: table }).ids; + +test('patches carry every kind of value through the bytes', async () => { const grid = new Data('grid').apply( - Patch.to('grid') + new Patch('grid') .axis('time', 's') .replace('Bus', [101, 102, 103], { kv: Float64Array.of(230, 115, 69), @@ -106,8 +109,8 @@ test('patches carry every kind of value through the bytes', () => { ]); assert.equal(get(102, 'path'), null, 'an empty geometry is missing'); assert.equal(get(103, 'area').length, 2, 'two polygons'); - assert.equal(grid.get('Branch', 0, 'from'), 101, 'a reference reads as the id it names'); - assert.equal(grid.get('Branch', 1, 'to'), null, 'null leaves a reference unwired'); + assert.equal(grid.get('Branch', 0, 'from'), 101, 'a reference reads as the referenced id'); + assert.equal(grid.get('Branch', 1, 'to'), null, 'a null id is a missing reference'); assert.equal(grid.get('Named', 'b', 'x'), 2); const kinds = Object.fromEntries( grid @@ -119,8 +122,8 @@ test('patches carry every kind of value through the bytes', () => { [kinds.kv, kinds.open, kinds.name, kinds.point, kinds.path, kinds.area], ['number', 'bool', 'text', 'point', 'path', 'polygons'], ); - // A dataset's bytes are the patch that builds it, read back alike. - const again = new Data('grid').apply(Patch.fromBytes(grid.toBytes())); + // A dataset's bytes are the patch that builds it, so reading them back gives an equal dataset. + const again = new Data('grid').apply(await Patch.read(grid.toBytes())); assert.deepEqual(again.describe(), grid.describe()); assert.deepEqual(again.get('Bus', 103, 'area'), get(103, 'area')); grid.destroy(); @@ -134,64 +137,100 @@ test('changes apply in place, a snapshot keeps a version, and a patch names its new Patch() .set('Bus', [3, 4], { x: [30, 40] }) .remove('Bus', [1]) - .append([0, 1], 'Bus', { v: Float32Array.of(1, 2, 3, 4, 5, 6) }) - .append(2, 'Bus', { v: [9] }, [4]), + .append('Bus', [0, 1], { v: Float32Array.of(1, 2, 3, 4, 5, 6) }) + .append('Bus', 2, { v: [9] }, { ids: [4] }), ); - assert.deepEqual(data.ids('Bus'), [2, 3, 4]); - assert.deepEqual(before.ids('Bus'), [1, 2, 3], 'the snapshot holds'); + assert.deepEqual(ids(data, 'Bus'), [2, 3, 4]); + assert.deepEqual(ids(before, 'Bus'), [1, 2, 3], 'the snapshot is unchanged'); assert.equal(data.get('Bus', 4, 'x'), 40); - assert.equal(data.get('Bus', 4, 'v', 2), 9, 'rows named take the frame'); - assert.equal(data.get('Bus', 2, 'v', 2), 4, 'the rest hold their last values'); - assert.deepEqual(data.span, [0, 2]); + assert.equal(data.get('Bus', 4, 'v', { at: 2 }), 9, 'the listed rows take the frame'); + assert.equal(data.get('Bus', 2, 'v', { at: 2 }), 4, 'other rows keep their last values'); + assert.deepEqual(data.describe().span, [0, 2]); data.apply(new Patch().forget(1)); - assert.deepEqual(data.span, [1, 2]); - assert.throws(() => data.apply(Patch.to('weather')), { code: 'invalid' }); + assert.deepEqual(data.describe().span, [1, 2]); + assert.throws(() => data.apply(new Patch('weather')), { code: 'invalid' }); assert.throws(() => data.apply(new Patch().set('Bus', [1], { x: ['a'] })), { code: 'invalid' }); assert.equal(data.get('Bus', 2, 'x'), 2, 'a failed patch changes nothing'); + data.restore(before); + assert.deepEqual(ids(data, 'Bus'), [1, 2, 3], 'restored to the snapshot'); + assert.throws(() => new Data('other').restore(before), { code: 'invalid' }); data.destroy(); before.destroy(); }); -test('appends one after another write as one change, each frame copied at its call', () => { +test('each change applies on its own, and tables write as their JSON form', () => { + const data = new Data('grid') + .axis('time', 's') + .replace('Bus', [1, 2, 3], { x: [1, 2, 3] }) + .set('Bus', [4], { x: [4] }) + .remove('Bus', [1]) + .append('Bus', [0, 1], { v: Float32Array.of(1, 2, 3, 4, 5, 6) }) + .forget(1); + assert.deepEqual(ids(data, 'Bus'), [2, 3, 4]); + assert.deepEqual(data.describe().span, [1, 1]); + assert.throws(() => data.set('Bus', [1], { x: ['a'] }), { code: 'invalid' }); + assert.equal(data.get('Bus', 2, 'x'), 2, 'a failed change changes nothing'); + const bus = data.table('Bus'); + assert.equal(bus.source, data); + assert.equal( + JSON.stringify({ vertices: bus, color: field('kv').from(data.table('Station')) }), + '{"vertices":["grid","Bus"],"color":{"field":"kv","from":["grid","Station"]}}', + ); + data.destroy(); +}); + +test('consecutive appends write one change and copy each frame when called', () => { const frame = new Float32Array(2); const patch = new Patch().replace('Bus', [1, 2], { x: [1, 2] }); for (let f = 0; f < 4; f++) { frame.set([f, 10 * f]); - patch.append(f, 'Bus', { v: frame }); + patch.append('Bus', f, { v: frame }); } const bytes = patch.toBytes(); const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - // After the header and the empty name: the count of changes. - assert.equal(view.getUint32(20, true), 2, 'the replace, and one append of every frame'); + // The count of changes follows the 12-byte header and the empty dataset name's length. + assert.equal(view.getUint32(16, true), 2, 'the replace, and one append for all frames'); const data = new Data('grid').apply(patch); - const v = (t) => data.get('Bus', 2, 'v', t); - assert.deepEqual([v(0), v(1), v(2), v(3)], [0, 10, 20, 30], 'each frame as it was at its call'); + const v = (at) => data.get('Bus', 2, 'v', { at }); + assert.deepEqual([v(0), v(1), v(2), v(3)], [0, 10, 20, 30], 'each frame copied when appended'); + const envelope = data.envelope({ table: 'Bus', field: 'v', range: [0, 4], spans: 2, ids: [2] }); + assert.deepEqual([...envelope.at], [0, 2, 4]); + assert.deepEqual( + [...envelope.least, ...envelope.greatest], + [0, 20, 10, 30], + 'row 2, span by span', + ); + assert.throws(() => data.envelope({ table: 'Nope', field: 'v', range: [0, 1], spans: 1 }), { + code: 'invalid', + path: 'Nope', + }); data.destroy(); }); -test('time tiles hold runs of a sampled field, holes between them', () => { +test('time tiles store runs of a sampled field with gaps between them', async () => { const data = new Data('grid').apply(new Patch().replace('Bus', [1, 2], { x: [1, 2] })); data.apply( new Patch() - .frames('Bus', 'v', { first: 100 }, [100, 101], Float32Array.of(1, 2, 3, 4)) - .frames('Bus', 'v', { level: 1, first: 0 }, [0, 7], [0.5, 0.25, 1.5, 1.25]), + .frames('Bus', 'v', { first: 100, at: [100, 101], values: Float32Array.of(1, 2, 3, 4) }) + .frames('Bus', 'v', { level: 1, first: 0, at: [0, 7], values: [0.5, 0.25, 1.5, 1.25] }), ); - assert.deepEqual(data.span, [0, 101]); - assert.equal(data.get('Bus', 2, 'v', 101), 4); - assert.equal(data.get('Bus', 1, 'v', 50), null, 'a hole holds no frame'); - const again = new Patch().frames('Bus', 'v', { first: 100 }, [100], [1, 2]); - assert.throws(() => data.apply(again), { code: 'invalid' }, 'frames held already'); - assert.throws(() => new Patch().frames('Bus', 'v', { level: 1, first: 0 }, [0, 7, 9], [1]), { - code: 'invalid', - }); - const back = new Data('grid').apply(Patch.fromBytes(data.toBytes())); + assert.deepEqual(data.describe().span, [0, 101]); + assert.equal(data.get('Bus', 2, 'v', { at: 101 }), 4); + assert.equal(data.get('Bus', 1, 'v', { at: 50 }), null, 'a gap has no frame'); + const again = new Patch().frames('Bus', 'v', { first: 100, at: [100], values: [1, 2] }); + assert.throws(() => data.apply(again), { code: 'invalid' }, 'frames already stored'); + assert.throws( + () => new Patch().frames('Bus', 'v', { level: 1, first: 0, at: [0, 7, 9], values: [1] }), + { code: 'invalid' }, + ); + const back = new Data('grid').apply(await Patch.read(data.toBytes())); assert.deepEqual(back.describe(), data.describe()); - assert.equal(back.get('Bus', 1, 'v', 100.5), 1); + assert.equal(back.get('Bus', 1, 'v', { at: 100.5 }), 1); data.destroy(); back.destroy(); }); -test('a failed change leaves the patch as it was', () => { +test('a failed change leaves the patch unchanged', () => { const patch = new Patch().set('Bus', [1], { x: [1] }); const bytes = patch.toBytes(); assert.throws(() => patch.set('Bus', [2], { x: [1, 'two'] }), { code: 'invalid', path: 'Bus.x' }); @@ -199,13 +238,13 @@ test('a failed change leaves the patch as it was', () => { assert.deepEqual(patch.toBytes(), bytes); const extended = new Patch().set('Bus', [2], { x: [2] }).extend(patch); const data = new Data('grid').apply(extended); - assert.deepEqual(data.ids('Bus'), [2, 1]); + assert.deepEqual(ids(data, 'Bus'), [2, 1]); data.destroy(); }); -test('patches stream back to back, split anywhere, and refuse what is too long', async () => { +test('patches stream back to back, split anywhere, and reject oversized frames', async () => { const sent = [ - Patch.to('grid').replace('Bus', [7, 8, 9], { x: [1, 2, 3] }), + new Patch('grid').replace('Bus', [7, 8, 9], { x: [1, 2, 3] }), new Patch().set('Bus', [7], { x: [10] }), new Patch().remove('Bus', [8]), ]; @@ -218,30 +257,88 @@ test('patches stream back to back, split anywhere, and refuse what is too long', data.apply(patch); } assert.deepEqual(read, ['grid', undefined, undefined]); - assert.deepEqual(data.ids('Bus'), [7, 9]); + assert.deepEqual(ids(data, 'Bus'), [7, 9]); assert.equal(data.get('Bus', 7, 'x'), 10); data.destroy(); } - const long = Patch.to('grid') - .replace('Bus', [1], { name: ['x'.repeat(100)] }) - .toBytes(); + const long = new Patch('grid').replace('Bus', [1], { name: ['x'.repeat(100)] }).toBytes(); const read = (stream, options) => Array.fromAsync(Patch.stream(stream, options)); await assert.rejects(read(pieces(long, 16), { limit: 64 }), { code: 'limit' }); + await assert.rejects(read(pieces(long, 16), { limit: 4 }), { code: 'invalid', path: 'limit' }); await assert.rejects(read(pieces(long.subarray(0, 40), 16)), /ended inside a patch/); const json = new TextEncoder().encode('{"not": "latkit"}'); await assert.rejects(read([json]), { code: 'invalid' }); }); -test('JSON, latkit bytes, and GeoJSON read as datasets', async () => { - const json = JSON.stringify({ +test('failed fetches are io errors, and aborting one rejects as cancelled', async (t) => { + const missing = new Response('gone', { status: 404 }); + await assert.rejects(Patch.read(missing), { code: 'io' }); + await assert.rejects(Data.read(new Response('gone', { status: 404 })), { code: 'io' }); + t.mock.method(globalThis, 'fetch', async (url, { signal }) => { + signal?.throwIfAborted(); + throw new TypeError(`${url}: no network`); + }); + await assert.rejects(Patch.read('https://example.invalid/p.latk'), { code: 'io' }); + await assert.rejects(Data.read('https://example.invalid/d.latk'), { code: 'io' }); + // Like a real fetch, this one rejects with the abort's reason before latkit hears of the abort. + globalThis.fetch.mock.mockImplementation( + (url, { signal }) => + new Promise((_, reject) => signal.addEventListener('abort', () => reject(signal.reason))), + ); + const controller = new AbortController(); + const reading = Patch.read('https://example.invalid/p.latk', { signal: controller.signal }); + controller.abort(); + await assert.rejects(reading, { code: 'cancelled' }); +}); + +test('blobs and streams read within their signal, fail as io, and cancel when stopped', async () => { + const bytes = new Patch().axis('t', 's').toBytes(); + const length = async (source) => (await Patch.read(source)).toBytes().length; + assert.equal(await length(new Blob([bytes])), bytes.length); + assert.equal( + await length( + (async function* () { + yield bytes; + })(), + ), + bytes.length, + ); + const stopped = AbortSignal.abort('stop'); + await assert.rejects(Patch.read(new Blob([bytes]), { signal: stopped }), { code: 'cancelled' }); + const failing = () => + new ReadableStream({ + pull() { + throw new Error('the socket closed'); + }, + }); + const closed = (error) => error.code === 'io' && error.cause.message === 'the socket closed'; + await assert.rejects(Patch.read(failing()), closed); + await assert.rejects(async () => { + for await (const patch of Patch.stream(failing())) void patch; + }, closed); + const cancelled = []; + const stalled = () => + new ReadableStream({ pull: () => new Promise(() => {}), cancel: (why) => cancelled.push(why) }); + const controller = new AbortController(); + const reading = Patch.read(stalled(), { signal: controller.signal }); + controller.abort('stop'); + await assert.rejects(reading, { code: 'cancelled', cause: 'stop' }); + await assert.rejects(Data.read(stalled(), { signal: stopped }), { code: 'cancelled' }); + assert.deepEqual(cancelled, ['stop', 'stop'], 'a stream stopped mid-read or before is cancelled'); +}); + +test('Data.read detects JSON, latkit bytes, and GeoJSON by their first bytes', async () => { + const document = { name: 'grid', patch: [{ replace: { table: 'Bus', ids: { count: 2 }, fields: { x: [1, 2] } } }], - }); - const loaded = await Data.load(json); + }; + const loaded = await Data.read(document); assert.equal(loaded.get('Bus', 1, 'x'), 2); - const read = await Data.read(loaded.toBytes(), 'renamed'); + const text = await Data.read(new TextEncoder().encode(JSON.stringify(document))); + assert.deepEqual(text.describe(), loaded.describe(), 'a document read from its bytes'); + const read = await Data.read(loaded.toBytes(), { name: 'renamed' }); assert.equal(read.name, 'renamed'); - const patch = await Patch.load('[{"set": {"table": "Bus", "ids": [1], "fields": {"x": [20]}}}]'); + const patch = await Patch.read([{ set: { table: 'Bus', ids: [1], fields: { x: [20] } } }]); assert.equal(read.apply(patch).get('Bus', 1, 'x'), 20); const geojson = new TextEncoder().encode( JSON.stringify({ @@ -251,10 +348,14 @@ test('JSON, latkit bytes, and GeoJSON read as datasets', async () => { ], }), ); - const places = await Data.read(geojson, 'places'); + const places = await Data.read(geojson, { name: 'places' }); assert.deepEqual(places.get('points', 0, 'geometry'), [1, 2]); - await assert.rejects(Data.read(geojson), { code: 'invalid' }, 'GeoJSON takes a name'); - for (const data of [loaded, read, places]) data.destroy(); + await assert.rejects(Data.read(geojson), { code: 'invalid' }, 'GeoJSON needs a name'); + const aborted = AbortSignal.abort(); + await assert.rejects(Data.read(geojson, { name: 'places', signal: aborted }), { + code: 'cancelled', + }); + for (const data of [loaded, text, read, places]) data.destroy(); }); test('queries filter, sort, page, and count', () => { @@ -268,11 +369,11 @@ test('queries filter, sort, page, and count', () => { of: 'Branch', filter: field('loading').gt(90), sort: [[field('loading'), 'descending']], - select: { name: 'name', loading: 'loading' }, + select: { name: '$name', loading: '$loading' }, limit: 1, }); assert.deepEqual(hot.ids, [8]); - assert.equal(hot.total, 2, 'rows kept before paging'); + assert.equal(hot.total, 2, 'rows matched before paging'); assert.ok(hot.fields.loading instanceof Float64Array); const named = data.query({ of: 'Branch', filter: field('name').contains('ED') }); assert.deepEqual(named.ids, [9], 'contains ignores case'); @@ -280,28 +381,25 @@ test('queries filter, sort, page, and count', () => { data.destroy(); }); -test('LATK v1 fixture and long-running sample indexes agree with Rust', () => { +test('the patch fixture and frame indexes past 2^32 agree with Rust', () => { const fixture = Buffer.from( fs .readFileSync( - new URL('../../crates/latkit/tests/fixtures/latk-v1.hex', import.meta.url), + new URL('../../crates/latkit/tests/fixtures/patch.hex', import.meta.url), 'utf8', ) .trim(), 'hex', ); - const patch = new Patch().frames('T', 'v', { first: 2 ** 32 + 3 }, [0, 1], [1, 2]); + const patch = new Patch().frames('T', 'v', { first: 2 ** 32 + 3, at: [0, 1], values: [1, 2] }); assert.deepEqual(Buffer.from(patch.toBytes()), fixture); const data = new Data('test').apply(new Patch().replace('T', { count: 1 }).extend(patch)); - data.apply(new Patch().append(2, 'T', { v: [3] })); + data.apply(new Patch().append('T', 2, { v: [3] })); assert.equal(data.get('T', 0, 'v'), 3); - assert.deepEqual(data.span, [0, 2]); + assert.deepEqual(data.describe().span, [0, 2]); data.destroy(); - const version = Uint8Array.from(fixture); - version[4] = 2; - assert.throws(() => Patch.fromBytes(version), /version/); assert.throws( - () => new Patch().frames('T', 'v', { first: Number.MAX_SAFE_INTEGER }, [0], [1]), + () => new Patch().frames('T', 'v', { first: Number.MAX_SAFE_INTEGER, at: [0], values: [1] }), /safe/, ); }); @@ -322,7 +420,8 @@ test('stream rejects oversized headers and cancels owned readers on early termin const frame = new Patch().axis('t', 's').toBytes(); const oversized = new Uint8Array(2 << 20); oversized.set(frame); - new DataView(oversized.buffer).setBigUint64(8, BigInt(oversized.length), true); + // The frame's length follows the 4-byte magic. + new DataView(oversized.buffer).setBigUint64(4, BigInt(oversized.length), true); const stream = new ReadableStream({ start(c) { c.enqueue(oversized); diff --git a/npm/test/jobs.test.mjs b/npm/test/jobs.test.mjs index ebdf78d1..8523d6a5 100644 --- a/npm/test/jobs.test.mjs +++ b/npm/test/jobs.test.mjs @@ -3,11 +3,8 @@ import path from 'node:path'; import { test } from 'node:test'; import { pathToFileURL } from 'node:url'; -const { jobs } = await import( - process.env.LATKIT_TEST_PACKAGE - ? pathToFileURL(path.join(process.env.LATKIT_TEST_PACKAGE, 'dist/jobs.js')).href - : new URL('../src/jobs.ts', import.meta.url).href -); +const directory = process.env.LATKIT_TEST_PACKAGE ?? path.resolve(import.meta.dirname, '..'); +const { jobs } = await import(pathToFileURL(path.join(directory, 'dist/jobs.js')).href); const module = new WebAssembly.Module(new Uint8Array([0, 97, 115, 109, 1, 0, 0, 0])); @@ -69,7 +66,7 @@ test('failed startup, malformed messages, and timeouts reject instead of hanging const message = assert.rejects(run(new Uint8Array()), /read worker/); workers[1].onmessageerror(); await message; - const timeout = assert.rejects(run(new Uint8Array()), /timed out/); + const timeout = assert.rejects(run(new Uint8Array()), { code: 'timeout' }); t.mock.timers.tick(60_000); await timeout; assert(workers.every((worker) => worker.terminated)); @@ -94,7 +91,7 @@ test('worker construction failure rejects, and later calls can recover', async ( t.mock.timers.tick(30_000); }); -test('queued jobs continue after a busy worker fails', async (t) => { +test('concurrent jobs run on separate workers, fail alone, and reuse idle ones', async (t) => { t.mock.timers.enable({ apis: ['setTimeout'] }); const workers = []; const run = jobs(module, () => { @@ -102,18 +99,19 @@ test('queued jobs continue after a busy worker fails', async (t) => { workers.push(worker); return worker; }); - const answers = Array.from({ length: 8 }, (_, i) => run(new Uint8Array([i]))); - const settled = Promise.allSettled(answers); - const limit = workers.length; - assert(limit >= 1 && limit <= 4); + const lost = assert.rejects(run(new Uint8Array([0])), /lost worker/); + const kept = run(new Uint8Array([1])); + assert.equal(workers.length, 2, 'latkit hands over only jobs that can run, so none queue here'); workers[0].onerror({ preventDefault() {}, message: 'lost worker' }); - for (const worker of workers) { - if (worker.terminated) continue; - worker.answer({ ready: true }); - for (let i = 0; i < 8; i++) worker.answer({ bytes: new Uint8Array([1]) }); - } - const results = await settled; - assert.equal(results.filter((result) => result.status === 'rejected').length, 1); + await lost; + workers[1].answer({ ready: true }); + workers[1].answer({ bytes: new Uint8Array([1]) }); + assert.deepEqual(await kept, new Uint8Array([1])); + const again = run(new Uint8Array([2])); + assert.equal(workers.length, 2, 'the idle worker takes the next job'); + assert.deepEqual(workers[1].sent.at(-1), new Uint8Array([2])); + workers[1].answer({ bytes: new Uint8Array([3]) }); + assert.deepEqual(await again, new Uint8Array([3])); t.mock.timers.tick(30_000); assert(workers.every((worker) => worker.terminated)); }); diff --git a/npm/test/load.test.mjs b/npm/test/load.test.mjs index fddae27c..2c4a5723 100644 --- a/npm/test/load.test.mjs +++ b/npm/test/load.test.mjs @@ -15,40 +15,39 @@ test('installed package imports without WASM, retries failures, and shares initi headers: { 'Content-Type': 'application/wasm' }, }); }); - const { loadLatkit, field, Patch } = await import( + const { load, field, Data, Patch } = await import( pathToFileURL(path.join(directory, 'dist/index.js')).href ); assert.equal(requests, 0); - assert.deepEqual(field('x').toJSON(), { field: 'x' }); - // Patches are written without WASM. + assert.equal(field('x').toJSON(), '$x'); + // Writing a patch needs no WASM. Creating a dataset throws until WASM is loaded. const patch = new Patch().replace('Node', { count: 1 }, { x: [42] }); + assert.throws(() => new Data('early'), { code: 'unloaded' }); assert.equal(requests, 0); - const first = loadLatkit({ wasm: 'https://example.invalid/latkit.wasm' }); - assert.equal(first, loadLatkit()); + const first = load({ wasm: 'https://example.invalid/latkit.wasm' }); + assert.equal(first, load()); await assert.rejects(first, { code: 'io' }); - const retry = loadLatkit({ wasm: new URL('https://example.invalid/latkit.wasm') }); - assert.equal(retry, loadLatkit({ wasm: 'https://example.invalid/latkit.wasm' })); - const kit = await retry; - assert.equal(kit, await loadLatkit()); + const retry = load({ wasm: new URL('https://example.invalid/latkit.wasm') }); + assert.equal(retry, load({ wasm: 'https://example.invalid/latkit.wasm' })); + await retry; + assert.equal(load(), retry, 'loaded once'); assert.equal(requests, 2); - await assert.rejects(loadLatkit({ wasm: 'https://example.invalid/other.wasm' }), { - code: 'io', - }); + await assert.rejects(load({ wasm: 'https://example.invalid/other.wasm' }), { code: 'io' }); await assert.rejects( - loadLatkit({ - createWorker: () => { + load({ + worker: () => { throw new Error('unused'); }, }), { code: 'io' }, ); - const data = new kit.Data('test').apply(patch); + const data = new Data('test').apply(patch); const kept = data.snapshot(); data.apply(new Patch().set('Node', [0], { x: [7] })); assert.equal(data.get('Node', 0, 'x'), 7); data.destroy(); data.destroy(); - assert.throws(() => data.name, /destroyed/); + assert.throws(() => data.name, { code: 'destroyed' }); assert.equal(kept.get('Node', 0, 'x'), 42, 'a snapshot outlives its source'); kept.destroy(); kept.destroy(); diff --git a/npm/test/release.test.mjs b/npm/test/release.test.mjs index 33c29675..751735a7 100644 --- a/npm/test/release.test.mjs +++ b/npm/test/release.test.mjs @@ -27,7 +27,7 @@ test('a partial release resumes only missing artifacts', async () => { await assert.rejects(ensureArtifacts([{ ...artifacts[0], digest: 'different' }]), /must match/); }); -test('registry failures never masquerade as unpublished versions', async (t) => { +test('registry failures are not mistaken for unpublished versions', async (t) => { t.mock.method(globalThis, 'fetch', async () => new Response(null, { status: 503 })); await assert.rejects(registry('https://example.test'), /503/); t.mock.method(globalThis, 'fetch', async () => new Response(null, { status: 404 })); diff --git a/npm/test/scenes.js b/npm/test/scenes.js index 27a2362a..27e028a6 100644 --- a/npm/test/scenes.js +++ b/npm/test/scenes.js @@ -5,127 +5,126 @@ /** @type {import('../src/inputs.js').Scene[]} */ export default [ { "space": "plane" }, - { "space": "mercator", "ease": 0 }, - { "space": "globe", "theme": "light" }, + { "space": "earth", "ease": 0 }, + { "space": { "earth": { "projection": "globe" } }, "theme": "light" }, { "space": "plot" }, { "space": "diagram", "layout": "stress" }, { "space": { "plane": { "grid": "off" } } }, - { "space": { "diagram": { "grid": { "dots": { "pitch": 10, "snap": true } } } }, "layout": { "layered": { "flow": "down", "rankGap": 60 } } }, - { "space": { "earth": { "projection": "globe", "surface": "#203040", "graticule": false } } }, + { "space": { "diagram": { "grid": { "dots": { "pitch": 10, "snap": true } } } }, "layout": { "layered": { "direction": "down", "rankGap": 60 } } }, + { "space": { "earth": { "projection": "globe", "graticule": false } }, "theme": { "earth": { "ground": "#203040" } } }, { "space": { "earth": { "projection": "globe", "daylight": 1782043200, "atmosphere": true } }, "marks": [ - { "areas": { "of": ["base", "water"], "color": "#1d3557" } }, - { "areas": { "of": ["city", "polygons"], "geometry": "geometry", "color": { "colormap": "tab10", "value": "use" }, "label": "name" } }, - { "lines": { "of": ["city", "polygons"], "width": 1, "color": "#ffffff40" } }, - { "lines": { "of": ["base", "transportation"], "width": { "op": "div", "args": ["lanes", 2] } } } + { "areas": ["base", "water"], "color": "#1d3557" }, + { "areas": ["city", "polygons"], "geometry": "geometry", "color": { "colormap": "tab10", "value": "$use" }, "label": "$name" }, + { "lines": ["city", "polygons"], "width": 1, "color": "rgba(255, 255, 255, 0.25)" }, + { "lines": ["base", "transportation"], "width": { "div": ["$lanes", 2] } } ], - "theme": { "area": "#223344", "atmosphere": "#5aa9ff88", "night": 0.35 } + "theme": { "area": { "color": "#223344" }, "earth": { "atmosphere": "#5aa9ff88", "night": 0.35 } } }, { "space": { "plot": { "x": { "label": "Time (s)" }, "y": { "grid": false } } } }, { - "space": "globe", + "space": { "earth": { "projection": "globe" } }, "marks": [ { - "edges": { - "of": ["grid", "Branch"], - "ends": ["from", "to"], - "width": { "units": 2 }, - "dash": { "op": "gt", "args": ["loading", 0.9] }, - "flow": { "op": "div", "args": ["p", 20] }, - "color": { "colormap": "amber", "value": "loading", "domain": [0, 1] }, - "name": "branches" - } + "edges": ["grid", "Branch"], + "ends": ["from", "to"], + "width": { "units": 2, "atLeast": 1 }, + "dash": { "gt": ["$loading", 0.9] }, + "flow": { "div": ["$p", 20] }, + "priority": { "abs": ["$p"] }, + "color": { "colormap": "amber", "value": "$loading", "domain": [0, 1] }, + "name": "branches" }, { - "vertices": { - "of": ["grid", "Bus"], - "position": ["lon", "lat"], - "elevation": 0, - "radius": { "op": "add", "args": [{ "op": "div", "args": ["kv", 100] }, 1] }, - "marker": "diamond", - "color": { - "when": [[{ "state": "selected" }, { "color": "#ffb000" }]], - "otherwise": { "colormap": "balance", "value": { "field": "vm", "step": true }, "domain": [0.95, 1.05] } - }, - "visible": { "op": "and", "args": [{ "op": "ge", "args": ["kv", 115] }, { "is": ["kind", "substation"] }] } - } + "vertices": ["grid", "Bus"], + "position": ["$lon", "$lat"], + "elevation": 0, + "radius": { "add": [{ "div": ["$kv", 100] }, 1] }, + "marker": "diamond", + "color": { + "when": [[{ "state": "selected" }, "#ffb000"]], + "otherwise": { "colormap": "balance", "value": { "field": "vm", "step": true }, "domain": [0.95, 1.05] } + }, + "visible": { "and": [{ "ge": ["$kv", 115] }, { "contains": ["$name", "north"] }] } }, { - "vertices": { - "of": ["grid", "Bus"], - "name": "load", - "size": [12, { "units": 4 }], - "marker": { "block": { "corner": 2 } }, - "shadow": true, - "color": { "scale": [0, 1], "value": { "field": "load", "of": ["weather", "Station"] } } - } + "vertices": ["grid", "Bus"], + "name": "load", + "size": [12, { "units": 4 }], + "marker": { "block": { "corner": 2 } }, + "color": { "colormap": "viridis", "value": { "field": "load", "from": ["weather", "Station"] } } } ], - "theme": { "base": "dark", "vertexRadius": 5, "edge": null, "selected": "#ff8800", "shadow": { "color": "#00000055", "lift": 4 } } + "theme": { + "base": "dark", + "vertex": { "radius": 5 }, + "edge": { "color": null }, + "focus": { "selected": "#ff8800" }, + "diagram": { "shadow": { "color": "#00000055", "lift": 4 } } + } }, { "space": { "plane": { "grid": { "lines": { "pitch": 1 } } } }, "controls": { "turn": false, "pickRadius": 14 }, "marks": [ - { "faces": { "of": ["mesh", "Face"], "corners": ["a", "b", "c"], "pick": false, "shadow": true } }, - { "edges": { "of": ["mesh", "Spring"], "ends": ["a", "b"], "color": "#ffffff14" } }, - { "vertices": { "of": ["mesh", "Vertex"], "position": ["x", "y"], "elevation": "z", "select": false, "drag": true } } + { "faces": ["mesh", "Face"], "corners": ["a", "b", "c"], "pick": false }, + { "edges": ["mesh", "Spring"], "ends": ["a", "b"], "color": "#ffffff14" }, + { "vertices": ["mesh", "Vertex"], "position": ["$x", "$y"], "elevation": "$z", "select": false, "drag": true } ], - "theme": { "shading": 0.7, "font": "monospace", "fontSize": 12, "playhead": "#ffe5d4" } + "theme": { "shading": 0.7, "text": { "font": "monospace", "size": 12 }, "plot": { "playhead": "#ffe5d4" } } }, { "space": { "plot": { "x": { "label": "Time (s)" } } }, - "marks": [{ "lines": { "of": ["mesh", "Body"], "trace": "height", "color": "#66c7ff", "width": 2, "flow": 12, "shadow": true } }] + "marks": [{ "lines": ["mesh", "Body"], "trace": "height", "color": "#66c7ff", "width": 2, "flow": 12 }] }, { "space": { "plot": { "y": { "format": "engineering", "precision": 1, "ticks": [0, [0.5, "half"], 1] } } }, "controls": { "selectBox": true }, "marks": [ { - "lines": { - "of": ["mesh", "Body"], - "trace": "speed", - "interpolation": "stepAfter", - "label": { "text": "name", "size": 11, "priority": "rank", "maxWidth": 120 } + "lines": ["mesh", "Body"], + "trace": "speed", + "interpolation": "stepAfter", + "label": { + "text": { "when": [[{ "ge": ["$rank", 2] }, "$name"]], "otherwise": null }, + "size": 11, + "maxWidth": 120, + "color": "white" } }, { - "vertices": { - "of": ["grid", "Bus"], - "marker": { "gauge": { "fill": "load", "shape": "rounded" } }, - "ring": { "color": "#ff0000" }, - "label": "name", - "radius": { "op": "mul", "args": [{ "state": "zoom" }, 2] } - } + "vertices": ["grid", "Bus"], + "marker": { "gauge": { "fill": "$load", "shape": "rounded" } }, + "ring": "red", + "label": "$name", + "radius": { "mul": [{ "state": "zoom" }, 2] } }, - { "vertices": { "of": ["grid", "Plant"], "marker": { "pie": { "slices": ["coal", "gas"], "colors": ["#333333", "#88aaff"], "hole": 0.4 } } } }, - { "vertices": { "of": ["grid", "Icon"], "marker": { "icon": { "images": [{ "size": [1, 1], "rgba": [255, 0, 0, 255] }], "index": 0 } } } }, - { "vertices": { "of": ["grid", "Custom"], "marker": { "wgsl": { "source": "fn marker(f: MarkerFragment) -> MarkerColor { return MarkerColor(f.color, 0.0); }", "inputs": ["load"] } } } }, - { "edges": { "of": ["grid", "Branch"], "ends": ["from", "to"], "shadow": true } } + { "vertices": ["grid", "Plant"], "marker": { "pie": { "slices": ["$coal", "$gas"], "colors": ["#333333", "#88aaff"], "hole": 0.4 } } }, + { "vertices": ["grid", "Icon"], "marker": { "icon": { "images": [{ "size": [1, 1], "rgba": [255, 0, 0, 255] }], "index": 0 } } }, + { "vertices": ["grid", "Custom"], "marker": { "wgsl": { "source": "fn marker(f: MarkerFragment) -> MarkerColor { return MarkerColor(f.color, 0.0); }", "inputs": ["$load"] } } }, + { "edges": ["grid", "Branch"], "ends": ["from", "to"] } ], - "theme": { "edgeSpacing": 5, "arrow": 8, "halo": 2, "shadow": { "light": [-1, 1, 2], "softness": 0.2 } } + "theme": { "edge": { "spacing": 5, "arrow": 8 }, "text": { "halo": 2 }, "flow": { "spacing": { "units": 40 }, "length": { "units": 12 } } } }, { "space": "diagram", "layout": { "stress": { "gap": 50 } }, "controls": { "connect": true, "delete": true }, "marks": [ - { "vertices": { "of": ["plant", "Area"], "contains": ["Device", "area"], "label": "name" } }, - { "edges": { "of": ["plant", "Wire"], "ends": ["from", "to"], "bends": "path", "arrows": true } }, - { "edges": { "of": ["plant", "Net"], "route": "orthogonal", "arrows": true } }, + { "vertices": ["plant", "Area"], "contains": ["Device", "area"], "label": "$name", "shadow": true }, + { "edges": ["plant", "Wire"], "ends": ["from", "to"], "bends": "path", "arrows": true, "width": { "px": 1.5 } }, + { "edges": ["plant", "Net"], "route": "orthogonal", "arrows": true }, { - "vertices": { - "of": ["plant", "Device"], - "size": [40, 24], - "marker": "block", - "ports": [ - ["input", "left"], - ["output", { "side": "right", "order": 0, "label": "out", "shape": "diamond", "output": true }] - ] - } + "vertices": ["plant", "Device"], + "size": [40, 24], + "marker": "block", + "ports": [ + ["input", "left"], + ["output", { "side": "right", "order": 0, "label": "out", "shape": "diamond", "output": true }] + ] } ], - "theme": { "groupPadding": 16 } + "theme": { "diagram": { "groupPadding": 16, "portSize": 9, "group": "#1b2530", "shadow": { "color": "#00000040", "lift": 4 } } } } ]; diff --git a/npm/test/video.test.mjs b/npm/test/video.test.mjs index 84404ce7..6568ecaa 100644 --- a/npm/test/video.test.mjs +++ b/npm/test/video.test.mjs @@ -4,9 +4,19 @@ import { pathToFileURL } from 'node:url'; import { test } from 'node:test'; const directory = process.env.LATKIT_TEST_PACKAGE ?? path.resolve(import.meta.dirname, '..'); -const { Video, Shot } = await import(pathToFileURL(path.join(directory, 'dist/runtime.js')).href); +const { Video, Shot } = await import(pathToFileURL(path.join(directory, 'dist/shot.js')).href); -function recording(sink = {}, options = {}) { +/** A bindings task of `promise`. Cancelling it rejects the promise, as the bindings' tasks do. */ +function task(promise) { + let cancel; + const cancelled = new Promise((_, reject) => { + cancel = () => reject(new Error('cancelled')); + }); + return { promise: Promise.race([promise, cancelled]), cancel, free() {} }; +} + +/** A video over fake bindings, whose `frame` and `finish` promises can be replaced. */ +function recording(sink = {}, { frame, finish } = {}) { const calls = { free: 0, cancel: 0, abort: 0, close: 0 }; const stream = new WritableStream({ abort() { @@ -19,12 +29,12 @@ function recording(sink = {}, options = {}) { }); const inner = { frames: 0, - async frame() { + frame() { this.frames++; - return Uint8Array.of(1); + return task(frame?.() ?? Promise.resolve(Uint8Array.of(1))); }, - async finish() { - return Uint8Array.of(2); + finish() { + return task(finish?.() ?? Promise.resolve(Uint8Array.of(2))); }, free() { calls.free++; @@ -32,14 +42,13 @@ function recording(sink = {}, options = {}) { cancel() { calls.cancel++; }, - ...options, }; - return { video: new Video(inner, stream.getWriter(), 30), stream, calls, inner }; + return { video: Video.made(inner, stream.getWriter(), 30), stream, calls, inner }; } test('finish retains counters and releases the output exactly once', async () => { const { video, stream, calls } = recording(); - await video.frame([]); + await video.frame(); const finished = video.finish(); assert.equal(video.finish(), finished); await finished; @@ -56,7 +65,6 @@ test('record aborts and unlocks on step and progress failures', async () => { const error = new Error('record failed'); await assert.rejects( video.record( - [], () => { if (!progress) throw error; return true; @@ -79,27 +87,14 @@ test('write failures close the encoder and release the output lock', async () => throw new Error('disk full'); }, }); - await assert.rejects(video.frame([]), /disk full/); + await assert.rejects(video.frame(), /disk full/); assert.equal(stream.locked, false); assert.equal(calls.free, 1); assert.equal(calls.cancel, 1); }); test('abort interrupts a pending finish and is idempotent', async () => { - let rejectFinish; - const { video, stream, calls } = recording( - {}, - { - finish: () => - new Promise((_, reject) => { - rejectFinish = reject; - }), - cancel() { - calls.cancel++; - rejectFinish(new Error('cancelled')); - }, - }, - ); + const { video, stream, calls } = recording({}, { finish: () => new Promise(() => {}) }); const finished = video.finish(); await Promise.resolve(); const rejected = assert.rejects(finished, /cancelled/); @@ -108,6 +103,16 @@ test('abort interrupts a pending finish and is idempotent', async () => { await Promise.all([aborted, rejected]); assert.equal(stream.locked, false); assert.equal(calls.free, 1); + await assert.rejects(video.frame(), { code: 'cancelled', cause: 'stop' }); +}); + +test('an abort while a frame draws rejects the frame, even one that finished first', async () => { + const { video, stream } = recording({}, { frame: () => Promise.resolve(new Uint8Array()) }); + const frame = video.frame(); + const aborted = video.abort('stop'); + await assert.rejects(frame, { code: 'cancelled', cause: 'stop' }); + await aborted; + assert.equal(stream.locked, false); }); test('concurrent frames reject without disrupting the frame already drawing', async () => { @@ -121,29 +126,20 @@ test('concurrent frames reject without disrupting the frame already drawing', as }), }, ); - const frame = video.frame([]); - await assert.rejects(video.frame([]), /one frame/); + const frame = video.frame(); + await assert.rejects(video.frame(), { code: 'invalid', message: /one frame/ }); resolveFrame(new Uint8Array()); await frame; await video.finish(); + await assert.rejects(video.frame(), { code: 'invalid' }, 'no frames after finish()'); }); -test('locked outputs do not create encoders, and setup failures unlock outputs', () => { +test('locked outputs make no encoder, and failing to make one unlocks the output', () => { const shot = new Shot([64, 64]); const stream = new WritableStream(); const writer = stream.getWriter(); - let created = 0; - const gpu = { - inner: { - video() { - created++; - throw new Error('no encoder'); - }, - }, - }; - assert.throws(() => shot.video(gpu, 30, stream)); - assert.equal(created, 0); + assert.throws(() => shot.video({ fps: 30, output: stream }), /locked/); writer.releaseLock(); - assert.throws(() => shot.video(gpu, 30, stream), /no encoder/); + assert.throws(() => shot.video({ fps: 30, output: stream }), { code: 'unloaded' }); assert.equal(stream.locked, false); }); diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 51209aec..e34c24f7 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -1,24 +1,37 @@ //! Repository tasks, run as `cargo xtask `. //! -//! - `check`: CI's checks in its order, GPU tests required: format, lint, test, document, and package -//! the crates, check that `native` stays out of browser builds, then format, lint, type-check, and -//! test the npm workspace over a fresh `wasm --dev`; then the browser gate. -//! - `browser`: the browser gate (`npm/examples/gate/gate.ts`) over a fresh `wasm --dev`: every -//! shader variant's pipelines compile in Chrome within budget, and every example page draws and -//! answers the pointer without errors or stalls. On `SwiftShader`, as Linux always is, only that -//! every pipeline compiles: pages and times need a GPU. -//! - `bench [--quick] [--only a,b] [--repeats n] [--update] [--software]`: the -//! benchmark (`benchmark/`) over a release `wasm`: every scenario up its sizes in Chrome while the -//! last one fits, each phase's time, GPU time, stalls, and exact work, held to how it may grow with -//! size, to the committed work, and to the times taken on this machine; a lost device stops the -//! sweep. `--quick` runs each scenario once at its smallest size. See `benchmark/run.ts`. -//! - `wasm [--dev]`: build `latkit-js`, bind it with the `wasm-bindgen` CLI matching `Cargo.lock`, -//! and compile the npm wrapper. `--dev` builds faster, without link-time optimization. -//! - `version`: give the crates the npm package's version, as changesets set it. -//! - `package`: build an isolated, optimized npm tarball, the one `pnpm release` publishes, and -//! report its sizes and hashes. -//! - `package-test`: install the latest tarball into a consumer and verify it in Chrome. -//! - `examples [--open]`: `wasm --dev`, then serve the example suite at ; +//! - `check`: every CI job's steps (`ci `), in CI's order. Locally, the crates package with +//! uncommitted work, and the browser gate judges frame budgets on the GPU. +//! - `ci `: the steps of one job of `.github/workflows/ci.yml`, which runs each job this way: +//! - `format`: Prettier over every file. CI's `format` job has no Rust toolchain, so it runs +//! `pnpm format:check` itself. +//! - `rust`: formats, lints, tests (with GPU tests required), documents, and packages the crates, +//! and checks that `native` stays out of browser builds. +//! - `native-cli`: tests the CLI's native code, which CI runs on macOS and Windows. +//! - `npm`: builds the npm package with `wasm --dev`, then lints, documents, type-checks, and +//! tests it. Runs the browser gate over that build, then `package` and `package-test`. +//! - `browser`: the browser gate (`npm/examples/gate/gate.ts`) over a fresh `wasm --dev`. Checks +//! that every shader variant's pipelines compile in Chrome within budget, and that every example +//! page draws and responds to the pointer without errors or stalls. On `SwiftShader`, which Linux +//! always uses, checks only that every pipeline compiles, since pages and times need a GPU. +//! - `bench [--quick] [--only a,b] [--repeats n] [--update] [--software]`: the benchmark +//! (`benchmark/`) over a release `wasm`. Runs every scenario at increasing sizes in Chrome while +//! the last size stays within budget. Measures each phase's time, GPU time, stalls, and exact +//! work, and judges them against their allowed growth with size, the committed work, and this +//! machine's times. A lost device stops the sweep. `--quick` runs each scenario once at its +//! smallest size. See `benchmark/run.ts`. +//! - `wasm [--dev]`: builds `latkit-js`, binds it with the `wasm-bindgen` CLI version in +//! `Cargo.lock`, and compiles the npm wrapper. `--dev` builds faster, without link-time +//! optimization. +//! - `bindgen-version`: prints the `wasm-bindgen` version in `Cargo.lock`. CI installs that version +//! of the CLI. +//! - `version`: sets the crates' version to the npm package's, as changesets set it. +//! - `package`: builds an isolated, optimized npm tarball, the one `pnpm release` publishes, and +//! reports its sizes and hashes. +//! - `package-test`: installs the latest tarball into a consumer and verifies it in Chrome. +//! - `release`: runs `npm/release.mjs`, which verifies or publishes each artifact, then tags the +//! release. +//! - `examples [--open]`: `wasm --dev`, then serves the example suite at . //! `--open` opens it in the browser once it is ready. use std::path::{Path, PathBuf}; @@ -29,19 +42,21 @@ fn main() -> ExitCode { let dev = std::env::args().any(|a| a == "--dev"); let result = match task.as_str() { "check" => check(), + "ci" => ci(&std::env::args().nth(2).unwrap_or_default()), "browser" => browser(), - "bench" => wasm(false).and_then(|()| { + "bench" => wasm(false).and_then(|_| { let flags: Vec = std::env::args().skip(2).collect(); let mut args = vec!["benchmark/run.ts"]; args.extend(flags.iter().map(String::as_str)); run("node", &args) }), - "wasm" => wasm(dev), + "wasm" => wasm(dev).map(drop), + "bindgen-version" => bindgen_version().map(|version| println!("{version}")), "version" => version(), "package" => package(), "package-test" => package_test(), "release" => run("node", &["npm/release.mjs"]), - "examples" => wasm(true).and_then(|()| { + "examples" => wasm(true).and_then(|_| { let mut vite = vec!["--dir", "npm/examples", "exec", "vite"]; if std::env::args().any(|a| a == "--open") { vite.push("--open"); @@ -50,7 +65,7 @@ fn main() -> ExitCode { }), _ => { eprintln!( - "usage: cargo xtask " + "usage: cargo xtask |browser|bench [flags]|wasm [--dev]|bindgen-version|version|package|package-test|release|examples [--open]>" ); return ExitCode::FAILURE; } @@ -67,7 +82,7 @@ fn main() -> ExitCode { type Result = std::result::Result; fn root() -> PathBuf { - Path::new(env!("CARGO_MANIFEST_DIR")).parent().expect("xtask sits in the workspace root").to_path_buf() + Path::new(env!("CARGO_MANIFEST_DIR")).parent().expect("xtask is in the workspace root").to_path_buf() } fn run(program: &str, args: &[&str]) -> Result { @@ -90,68 +105,113 @@ fn pnpm(args: &[&str]) -> Result { run(if cfg!(windows) { "pnpm.cmd" } else { "pnpm" }, args) } -/// What `.github/workflows/ci.yml` checks, in its order, then the browser gate. +/// The jobs of `.github/workflows/ci.yml` that `ci` runs the steps of, in its order. +const JOBS: [&str; 4] = ["format", "rust", "native-cli", "npm"]; + +/// Runs every CI job's steps in CI's order. fn check() -> Result { + for job in JOBS { + ci(job)?; + } + Ok(()) +} + +/// Runs the steps of CI job `job`. +fn ci(job: &str) -> Result { + match job { + "format" => pnpm(&["format:check"]), + "rust" => rust(), + "native-cli" => native_cli(), + "npm" => npm(), + _ => Err(format!("no CI job {job:?}; the jobs are {}", JOBS.join(", "))), + } +} + +fn rust() -> Result { let gpu = [("LATKIT_REQUIRE_GPU", "1")]; cargo(&["fmt", "--all", "--check"])?; cargo(&["clippy", "--locked", "--workspace", "--all-targets", "--all-features", "--", "-D", "warnings"])?; cargo(&["clippy", "--locked", "-p", "latkit", "--no-default-features", "--all-targets", "--", "-D", "warnings"])?; cargo(&["clippy", "--locked", "-p", "latkit-js", "--target", "wasm32-unknown-unknown", "--", "-D", "warnings"])?; run_with(env!("CARGO"), &["test", "--locked", "--workspace"], &gpu)?; - run_with(env!("CARGO"), &["test", "--locked", "-p", "latkit", "--no-default-features"], &gpu)?; run_with( env!("CARGO"), &["doc", "--locked", "-p", "latkit", "--no-deps", "--all-features"], &[("RUSTDOCFLAGS", "-D warnings")], )?; - // CI's tree is clean; here uncommitted work is packaged as it stands. - cargo(&["package", "--locked", "--allow-dirty", "-p", "latkit"])?; + // The published package builds from its own files. The wasm clippy above builds the browser + // target. + let mut package = vec!["package", "--locked", "-p", "latkit"]; + // CI's tree is clean. Locally, package uncommitted work as is. + if std::env::var_os("CI").is_none() { + package.push("--allow-dirty"); + } + cargo(&package) +} + +fn native_cli() -> Result { + // One run, so latkit builds once per platform. cargo(&[ - "package", + "test", "--locked", - "--allow-dirty", "-p", "latkit", - "--target", - "wasm32-unknown-unknown", + "-p", + "latkit-cli", "--features", - "web", - ])?; - cargo(&["check", "--locked", "-p", "latkit", "--target", "wasm32-unknown-unknown", "--features", "native,web"])?; - wasm(true)?; - pnpm(&["format:check"])?; + "latkit/native", + "--lib", + "--bins", + "--test", + "validation", + "--test", + "cli", + ]) +} + +fn npm() -> Result { + // The release build, which `package` then packs without compiling again. + let shaders = wasm(false)?; pnpm(&["lint"])?; run("node", &["npm/docs.mjs", "--check"])?; pnpm(&["typecheck"])?; pnpm(&["test"])?; - browser_gate() + browser_gate(&shaders)?; + package()?; + package_test() } -/// Builds the package for the browser, then runs the gate over the shaders the build linked. +/// Builds the wasm package, then runs the browser gate over the shaders the build linked. fn browser() -> Result { - wasm(true)?; - browser_gate() + browser_gate(&wasm(true)?) } -fn browser_gate() -> Result { - let shaders = shaders_dir()?; - run("node", &["npm/examples/gate/gate.ts", &shaders]) +/// Runs the browser gate over the linked shaders and `shaders.json` in `shaders`. +fn browser_gate(shaders: &str) -> Result { + run("node", &["npm/examples/gate/gate.ts", shaders]) } -/// The directory `latkit`'s build script wrote the linked shaders and `shaders.json` to. -fn shaders_dir() -> Result { - eprintln!("$ cargo check --locked -p latkit --message-format=json-render-diagnostics"); - let output = Command::new(env!("CARGO")) - .args(["check", "--locked", "-p", "latkit", "--message-format=json-render-diagnostics"]) +fn wasm(dev: bool) -> Result { + build_wasm(dev, "npm") +} + +/// Builds the wasm package into `output`. Returns where latkit's build script wrote the linked +/// shaders and `shaders.json`, the same for every target, which the browser gate reads. +fn build_wasm(dev: bool, output: &str) -> Result { + let profile = if dev { "wasm-dev" } else { "wasm" }; + let args = ["build", "--locked", "-p", "latkit-js", "--profile", profile, "--target", "wasm32-unknown-unknown"]; + eprintln!("$ cargo {}", args.join(" ")); + let built = Command::new(env!("CARGO")) + .args(args) + .arg("--message-format=json-render-diagnostics") .current_dir(root()) .stderr(Stdio::inherit()) .output() .map_err(|e| format!("cargo: {e}"))?; - if !output.status.success() { - return Err(format!("cargo check exited with {}", output.status)); + if !built.status.success() { + return Err(format!("cargo build exited with {}", built.status)); } - let stdout = String::from_utf8_lossy(&output.stdout); - stdout + let shaders = String::from_utf8_lossy(&built.stdout) .lines() .filter_map(|line| serde_json::from_str::(line).ok()) .find(|message| { @@ -159,18 +219,8 @@ fn shaders_dir() -> Result { && message["package_id"].as_str().is_some_and(|id| id.contains("/crates/latkit#")) }) .and_then(|message| message["out_dir"].as_str().map(str::to_owned)) - .ok_or_else(|| "cargo did not report latkit's build output".to_owned()) -} - -fn wasm(dev: bool) -> Result { - build_wasm(dev, "npm") -} - -fn build_wasm(dev: bool, output: &str) -> Result { - let profile = if dev { "wasm-dev" } else { "wasm" }; - cargo(&["build", "--locked", "-p", "latkit-js", "--profile", profile, "--target", "wasm32-unknown-unknown"])?; - let lock = std::fs::read_to_string(root().join("Cargo.lock")).map_err(|e| format!("Cargo.lock: {e}"))?; - let version = locked_version(&lock, "wasm-bindgen").ok_or("wasm-bindgen is not in Cargo.lock")?; + .ok_or("cargo did not report latkit's build output")?; + let version = bindgen_version()?; let installed = Command::new("wasm-bindgen").arg("--version").output().ok(); let installed = installed.and_then(|o| String::from_utf8(o.stdout).ok()).unwrap_or_default(); if installed.split_whitespace().nth(1) != Some(version.as_str()) { @@ -183,7 +233,8 @@ fn build_wasm(dev: bool, output: &str) -> Result { let pkg = format!("{output}/pkg"); let bind = [wasm.as_str(), "--target", "web", "--weak-refs", "--out-dir", &pkg, "--out-name", "latkit"]; run("wasm-bindgen", &bind)?; - run("node", &["npm/build.mjs", "compile", output]) + run("node", &["npm/build.mjs", "compile", output])?; + Ok(shaders) } fn package() -> Result { @@ -200,7 +251,7 @@ fn package() -> Result { } fn package_test() -> Result { - // Preparation prints the isolated consumer path into a stable manifest for the browser runner. + // Creates an isolated consumer and adds its path to the package report for the browser runner. run("node", &["npm/build.mjs", "consumer"])?; let report = std::fs::read_to_string(root().join("target/npm/package.json")).map_err(|e| e.to_string())?; let report: serde_json::Value = serde_json::from_str(&report).map_err(|e| e.to_string())?; @@ -224,8 +275,8 @@ fn package_test() -> Result { run("node", &["npm/build.mjs", "verified"]) } -/// Gives the crates the npm package's version, which changesets sets: one version for everything -/// released together. +/// Sets the crates' version to the npm package's, which changesets sets, so everything released +/// together shares one version. fn version() -> Result { let manifest = std::fs::read_to_string(root().join("npm/package.json")).map_err(|e| format!("npm: {e}"))?; let manifest: serde_json::Value = serde_json::from_str(&manifest).map_err(|e| format!("npm: {e}"))?; @@ -239,6 +290,12 @@ fn version() -> Result { cargo(&["update", "--workspace", "--offline"]) } +/// The `wasm-bindgen` version in `Cargo.lock`, which the `wasm-bindgen` CLI must match. +fn bindgen_version() -> Result { + let lock = std::fs::read_to_string(root().join("Cargo.lock")).map_err(|e| format!("Cargo.lock: {e}"))?; + locked_version(&lock, "wasm-bindgen").ok_or_else(|| "wasm-bindgen is not in Cargo.lock".to_owned()) +} + /// The version of `package` in a `Cargo.lock`. fn locked_version(lock: &str, package: &str) -> Option { let mut lines = lock.lines();