Skip to content

Prototype table-driven tokenizer bulk scanning - #40

Closed
isomorphisms wants to merge 3 commits into
mainfrom
issue-38-tokenizer-bulk-scan-experiment
Closed

isomorphisms wants to merge 3 commits into
mainfrom
issue-38-tokenizer-bulk-scan-experiment

Conversation

@isomorphisms

Copy link
Copy Markdown
Owner

Refs #38.

This adds a deliberately narrow, test-only byte tokenizer experiment under the existing Android-prepaint JVM harness.

What it compares:

  • a scalar oracle that makes the lexical decision one byte at a time;
  • a candidate with a small byte-class table plus state × class -> action dispatch;
  • a bulk path that consumes maximal APPEND_RUN spans before returning to exceptional-byte dispatch.

The event contract is pinned explicitly and the candidate must match it across every single two-chunk split and one-byte chunking. Fixtures cover quoted attributes containing </>, entities, malformed entities, NUL, unclosed quotes, and raw UTF-8 handoff including splits inside multibyte sequences.

The long-run test also proves the candidate is actually taking the bulk path: 4096 ordinary bytes are consumed as bulk work while exceptional dispatch stays below 32 operations.

Deliberate boundaries:

  • this is a lexical experiment, not an HTML conformance tokenizer;
  • it does not integrate with renderer/prepaint production code yet;
  • UTF-8 is handed off losslessly as raw bytes rather than decoded here;
  • SIMD/word-at-a-time scanning is intentionally absent. It should be a later implementation comparison against the same oracle, not a prerequisite for validating the architecture.

Copy link
Copy Markdown
Owner Author

Retiring the isolated, nonintegrated JVM tokenizer experiment. It answered an exploratory comparison question but never became the production tokenizer or a required current dependency. Current Idriç extraction/recovery uses the shared InformationToken boundary introduced by merged #44, and the current semantic model work is #47. Keeping a second Java-only candidate/oracle maintenance line solely for a possible future optimization is not justified. This closure does not claim that #44 implements the bulk-scanning optimization; it abandons the unused prototype scope rather than inventing a successor task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant