[christianhuening]

Supersedes #42 by @szEvEz. #42 found the list export, page order and sidebar drag-and-drop bugs, asked for Duplicate and proposed the first fixes. This branch rebuilds the fixes from scratch, and goes further where the root cause turned out to be wider. Thank you, @szEvEz.

Summary

The guarantee for existing pages: a healthy page (one whose current export is a fixed point) exports byte for byte as before, unless it falls in a named bug class; no export loses text. Two harnesses check this: one over a corpus in CI, one opt-in against a real database.

knot-markdown   list items with line breaks, nested checklists, multi-paragraph items,
                images/drawings in tight lists and table cells, code blocks in lists,
                lone '>'                         → round-trip instead of breaking
                damaged or unknown content       → exported as text instead of failing the page
                nesting past 128 block levels    → one paragraph of its words; no text lost
knot-crdt       My Tasks follows the page's checkboxes (a walk of the document, not its Markdown)
                rooms refuse updates that nest a page past 512 types (MAX_NESTING = 4 × 128)
knot-storage    lexorank: a valid key strictly between its neighbours (an error, never a panic,
                on bad ones); appends grow keys slowly; placement respreads a group before a
                key would pass 64 characters
                heal migration: duplicate/invalid/overlong keys repaired in the order the
                sidebar showed; one unique constraint now covers the top level too
                concurrent moves can't close a parent loop (per-workspace move lock);
                recursive walks end on a loop already in the database
knot-server     Duplicate: POST /api/docs/{id}/duplicate
                one-time background re-render of cached Markdown and task listings
                workspace import: fresh keys in the export's order (it can run twice);
                parent loops in a crafted zip import as top-level pages, without recursion
knot-export     DOCX/ODT walk joins the old importer's split lines like the Markdown export
web             drag and drop lands where the drop line shows; Alt+arrows and menu items
                move pages; screen-reader text; Duplicate… dialog

How the sidebar decides a drop:

on drag end
  if press-to-release distance < one INDENT (12 px, in dnd-kit's scroll-adjusted frame)
    move nothing
    if the pointer's path < one INDENT and it was a plain press on the row's link
      open the page                                  // a click that slipped
    return
  row  = the row under the pointer (none outside the tree → cancel)
  zone = top quarter: before · middle half: into · bottom quarter: after
         (on the dragged page's own row: top or bottom half, no into)
  into          → last child of that row
  before/after  → depth = page's depth + trunc(Δx / INDENT),
                  clamped between the next row's depth and the previous row's depth + 1
  send the move the drop line showed; nothing if it changes nothing

How a room (page or board) treats an inbound update:

knot's reader or yrs can't decode it          → close the sender (4422); nothing applied
  (the reader matches yrs's decoder on ~175k corpus updates in a differential test,
   and also refuses block clocks past u32::MAX)
the prefilter predicts nesting past 512 types → close the sender (4422); nothing applied
apply succeeds, measured depth past 512       → close the sender (4422), evict the room:
                                                nothing stored or sent on, peers reconnect (1012)
apply fails part-way
  past 512                                    → as the line above
  within it                                   → as main: logged, not stored or sent on,
                                                room and sender kept
a refused connection                          → nothing more it sends is applied

Duplicate, POST /api/docs/{id}/duplicate:

• copies a page right after itself as "Title (copy)", optionally with its live subpages ({"include_subpages": true});
• copies the drawings and the files the user can read, so the copy keeps working if the original is archived;
• remaps links between the copied pages, drawings and files;
• writes every row in one transaction. On S3 the objects are copied first; the hourly job removes copies whose transaction never committed (blob_copy_pending);
• error answers:
  • 413 doc.duplicate_too_large over 200 pages or 200 MiB;
  • 503 doc.duplicate_busy with Retry-After: 5 (two copies at a time per server, five-second wait);
  • 403 acl.editor_required for viewers, 422 doc.is_template, 409 doc.archived;
  • 409 doc.duplicate_conflict if the source changes mid-copy, or a file it shows is missing from the store (kept on purpose; to revisit if it bites);
• answers before the copy's Markdown and tasks are refreshed;
• does not copy comments, history, share links or grants.

Evidence

Compatibility harness: crates/knot-markdown/tests/{compat,tasks_compat}.rs run against knot_markdown::legacy, a frozen copy of main's serializer, importer and task extractor (behind the test-only legacy-compat feature).

• Before: only 14 hand-written fixtures were pinned byte for byte. Nothing compared a serializer change with the previous serializer over a corpus or a real database.
• After: 31 tests over all fixtures, the fix: fix sidebar drag and drop, sort keys and list-item export #42 probe cases, a generated corpus and a seeded random corpus. The generated corpus covers every node and mark, nesting up to 4, and editor- and import-shaped attributes.
  • Results: 23,705 documents, 7,315 of them healthy. The 6,228 healthy documents outside the named classes are byte-identical, and no export loses text. Same report on pulldown-cmark 0.13.5.
  • CI enforces zero differences and floors on corpus size; the counts come from a KNOT_COMPAT_REPORT=1 run.
  • Healthy documents that change on purpose, by class:
    • ordered_checkbox: 998 (checkboxes in numbered lists are now exported);
    • multi_block_item: 49;
    • nested_tasklist: 15;
    • quote_block_sequence: 25;
    • depth_limit: nesting past 128, flattened; the only class that isn't a bug legacy had.
  • Users with such pages will see different Markdown.

Real-data check: opt-in and read-only (default_transaction_read_only=on, no migrations). It runs from a source checkout of this commit:

KNOT_COMPAT_DATABASE_URL=postgres://… \
  cargo test -p knot-server --test markdown_compat -- --ignored --nocapture

It reports:

• pages whose export or /tasks listing would change, by class, with diffs;
• the sort-key groups and rows the heal would rewrite;
• how many pages the sweep would re-render;
• pages nested past the limits, and pages whose parent is in another workspace.

It fails if:

• a healthy page or correctly listed task outside the classes would change;
• a changed export loses text, or a new export fails;
• /tasks would stop listing one task per checkbox;
• a page can't be read.

On the final head it passes on the three databases its own tests seed (current schema with a cross-workspace link, deep pages, a database from before the upgrade). No database with real pages has been checked yet: see the manual checks below.

Regression tests: each behaviour fix comes with tests that the task reports show failing on the branch before the fix. Some rows:

Case	Before (main)	After
Create the 8th page under one parent	409 doc.sort_key_conflict ("Couldn't create document"); that level is blocked	201, distinct keys in order
Two pages created at one level at once	Top level: same key, order can flip. Under a parent: 409	Distinct keys
Shift+Enter in a list item, then history restore	Export fails (search, shares, zip, My Tasks, template)	Round-trips
Nested checklist in My Tasks	One merged task	One task per checkbox
Two concurrent moves into each other's subtree	Parent loop; archiving it hangs	Moves take turns; walks end
Crafted zip with parent loops or a deep chain	Server crash (recursion)	Imports: loop members become top-level pages, walked iteratively
Markdown upload 100,000 > deep, then replace it	Server abort	Capped at 128, replace works
Drag a child out from under the last top-level page	Impossible	Lands where the line shows

Suites on the final head:

Suite	Result
Rust (cargo nextest run --workspace --all-features)	940 passed, 0 failed, 3 skipped (ignored)
Web (pnpm test)	731 passed
e2e (full)	106 passed, 2 failed: both ws-reconnect tests need toxiproxy and fail identically on main
cargo fmt, clippy -D warnings, pnpm tsc, pnpm lint	clean
cargo deny check	ok (main currently fails it on RUSTSEC-2026-0414; this branch bumps pulldown-cmark to 0.13.5)
Reader vs yrs differential	94,544 corpus updates and ~80,500 prefixes, 0 disagreements

Existing tests changed (each mechanical or encoding a bug; the plan's P6 and the rulings list them; none checks less)

• lexorank.rs in-module tests: .unwrap() on between.
• move_with_unknown_anchor_falls_to_end → move_with_unknown_anchor_is_422 (422 doc.bad_anchor).
• Tests that inserted duplicate top-level keys now insert distinct ones: documents.rs, comments_integration.rs, shares_integration.rs, and search_integration.rs's make_doc.
• e2e/flows/tree-reorder.spec.ts is rewritten:
  • it teleported the pointer; now it moves step by step and checks the stored order through the API;
  • its un-nest case now drags one level left, as the drop-depth rule needs;
  • one-INDENT drags became INDENT + 2, because of Math.trunc rounding.
• e2e/flows/tree-dnd.spec.ts is removed: it checked only "no error toast".
• sidebar-resize.spec.ts: selectors follow the row markup (role="button" misuse removed).
• migrations_apply.rs: the table list gains the two new tables.
• ws_writer.rs and board_fanout.rs harnesses: connections carry the new refusal tokens.
• tree.test.ts:
  • the dropIntent cases are removed with dropIntent;
  • descendantIds/moveArgs tests run against a test-only copy, tree.legacy.ts.
• Comment-only edits in ContextMenu.test.tsx and DocTree.archive.test.tsx.

Behaviour changes to know about

• Drop depth. A before/after drop next to a row at another depth keeps the dragged page's depth as far as the gap allows. Moving the pointer sideways changes it by one level per INDENT, and the drop line shows the result. Main took the hovered row's depth.

  • Same-depth reorders and "into" reach the same final position as on main, except one case: a drop on the gap right below an expanded parent now makes the page its first child. Main put it after the parent.
  • The pointer, not the dragged row's centre, picks the row and zone.

• Small drags are clicks. A release less than one INDENT from the press moves nothing. If the pointer barely moved and it was a plain press on the link, the page opens.

• Keyboard and touch.

  • The keyboard drag is gone (Space or Enter, arrows, Space or Enter). Alt+↑/↓/←/→ and the menu's Move up, Move down, Indent and Outdent move pages instead.
  • On touch screens a drag starts after a 250 ms hold, so the sidebar scrolls.
  • A drop outside the tree cancels.
  • Releasing on the page you started from no longer reloads it.

• Orphan rows. A template's child, or a live child of an archived page, is drawn at the top level. Moving it there by menu, keys or drag makes it a real top-level page.

• API.

  • An unknown or self anchor → 422 doc.bad_anchor (was: put last).
  • Restore of a page restored, moved and archived again meanwhile → 409 doc.restore_conflict (was 500).
  • A top-level key taken mid-restore by an old pod during the rollout → 409 doc.sort_key_conflict.
  • A parent_id that is unknown or in another workspace → 404 doc.not_found on create, move, from-template and import (import used to answer 403; the others used to accept another workspace's parent).

• Workspace import ignores the manifest's keys. Pages get fresh keys in the export's order, after the target's existing pages.

• My Tasks lists these now:

  • checklists in table cells;
  • checkboxes in numbered lists;
  • one task per checkbox in nested checklists.

  A bullet that merely starts with [ ] is no longer listed.

• Depth cap.

  • Markdown/zip import, history restore and templates store content nested past 128 levels as one paragraph of its words. Marks, links and images there are dropped; the text is kept.
  • Export writes an already-deeper part the same way.
  • Live updates nesting past 512 types are refused (4422).

• Live connections. When a room ends for any reason, including a panic, every socket now closes with 1012, so the client reconnects. A refused sender gets 4422, and nothing more it sends (edits, presence, a re-join) is applied. The page and board clients treat 4422 as final: they stop reconnecting and show "Changes refused" until a reload, since a reconnect would hand the refused update over again. A page room now shares presence only from connections it still holds, so a revoked or slow-consumer-dropped connection no longer shows a cursor.

• S3.

  • Purging needs s3:DeleteObject. A refused delete keeps the page and the hourly job retries, instead of deleting the page and orphaning its files.
  • Until a retry gets through, "Delete permanently" answers 500 for that page and it can't be restored.
  • A refused GET is now an error instead of serving the bucket's error body.
  • Deleting a single attachment still ignores a refused delete (unchanged).

• Metrics (new):

  • knot_room_updates_refused_total
  • knot_room_evicted_total
  • knot_markdown_unknown_nodes_total
  • knot_markdown_depth_limited_total

Merge Danger

Door: one-way for data, two-way for the UI.

Four new migrations:

• heal_sort_keys:
  • rewrites broken keys;
  • replaces the per-parent unique constraint with one that also covers the top level (NULLS NOT DISTINCT, deferrable);
  • leaves groups with no bad key byte for byte unchanged, except a workspace's root list (top-level pages plus template children, which the sidebar shows together), which is rewritten as one list when any part of it is bad;
  • keeps distinct keys in their order and puts pages that shared a key in creation order;
  • a second run changes nothing;
  • holds a lock on documents until it commits, so page requests wait during the pre-upgrade job on a large database.
• markdown_renderer_version: marks every cached render stale. A background sweep then re-renders every live page, about ten pages every two seconds on one replica (at most about 18,000 an hour). Until a page is reached, search, share links, zip export and My Tasks show its old render.
• blob_copy_pending and markdown_sweep_attempts: bookkeeping tables.

Rolling back to 0.8.1 needs a database restore. Run the real-data check on a backup first. The gesture changes are two-way doors: easy to tune once they've been used.

Blast Radius: wide.

Every page's export path, My Tasks, page create/move/restore/import, the sidebar, live editing rooms, and S3 storage. During a rolling upgrade, old pods can answer 409 doc.sort_key_conflict on create or move until they're replaced, and new pods can too when an old pod's unlocked write takes a key. Don't run workspace imports mid-rollout.

Two notes for whoever merges:

• The four migrations were edited in place during development. A database that ran an earlier build of this branch, such as a preview, must be recreated.
• The commits between Task 8 and Task 10 panic on legacy invalid keys, so a merge commit keeps the signed history but leaves that range unsafe to bisect through.

Known limitations

Tracked in #53; other follow-ups in #54. All of these are pre-existing on main unless marked new.

• A single crafted update whose own apply recurses past the worker's stack can still abort the server. So can removing the deep part of a page stored deeper than 512 types before this release (by an edit, a history restore or a Markdown replace). The real-data check lists such pages. Fixing it needs an apply off the worker stack, or an iterative delete in yrs.
• yrs can spend CPU superlinear in the width of a crafted update, holding a tokio worker while it applies.
• An editor's update that fails only on a block a crafted writer parked earlier stays in the live doc, as on main:
  • it is not stored until the next snapshot (a process exit before then loses it);
  • it isn't sent to connected peers (they get it on rejoin);
  • after a reload, the re-parked block fails the re-sent edit again.
• A room fans out an update even when yrs parked one of its blocks, so a Yjs peer that has that block's parent may fail on it.
• Server-originated room paths (ApplyUpdate, ReplaceWithMarkdown) and bus replay keep a partly applied live doc on an apply error.
• A connection the room drops as a slow consumer stays open but gets no more updates. Its edits are dropped silently until it reconnects.
• New: a crafted writer that keeps sending updates that pass the limit only after apply evicts the room each time, so every editor of the page reconnects each time. knot's own client stops after a 4422; a crafted one can ignore that. Mitigation: a per-sender backoff.
• New: the board room can only close (4422) a refused connection it still holds; one it dropped as a slow consumer keeps its socket, though each bad update it sends is still refused.
• New: in a Duplicate copy, a text holding hidden non-text content (only a crafted update puts it there) keeps its links unremapped.
• A runtime respread rewrites one parent group, so it can move template children shown at the root relative to top-level pages.
• The move lock is per workspace. With legacy cross-workspace parent links, racing moves in two workspaces could still close a loop; the walks end on it.
• yrs adds a delete set's clock and length unchecked (a debug-build panic, a release-build wrap).
• knot-export's DOCX/ODT walk counts a depth cap as an unknown node in its fallback tally.

Follow-ups

• Markdown escaping.
• Full ARIA tree pattern for the sidebar.
• documents_tree_idx cleanup.
• e2e [vite] ws proxy error noise.
• A heal for cross-workspace parent links, if the real-data check finds any.
• S3/MinIO tests for blob copy and delete.
• Shared test harnesses in knot-test-support.
• The limitations above.

Manual checks before merge

• Real-data check on a restored production backup.
• S3 or MinIO, live: Duplicate a page with images; purge with and without s3:DeleteObject.
• Android: long-press opens the row menu; press, hold and move drags.
• iOS Safari: long-press on a row link vs. the 250 ms drag; touch scrolling.
• Windows Chrome/Edge: the Menu key opens only knot's row menu.
• A screen reader (VoiceOver or NVDA) reads levels, subpage counts and move announcements.
• Word and LibreOffice open a DOCX/ODT export of a page in the old import's split-line shape and of a depth-capped page.
• Close fix: fix sidebar drag and drop, sort keys and list-item export #42 by hand with a thank-you and a link here.

🤖 Generated with Claude Code