[enaboapps]

Closes #25

The model, tokenizer, model-bundle.json, search policy (width 8, 8 tokens, 64 evaluations, 1,600 ms cutoff) and worker protocol 2 are unchanged. No release is prepared.

Changes

• Batched search (neural/worker/src/generation.rs, llama.rs, model.rs). Every extension at a depth has the same length, so they run as one forward pass. The rows of the attention cache are selected by parent beam. llama.rs adapts candle-transformers 0.11.0 quantized_llama (MIT OR Apache-2.0) and keeps its operation order. Each extension still counts as one evaluation. The 1,600 ms cutoff is checked before each depth and reserves the previous batch's duration.
• Exact early stop. A piece that begins with a word boundary never continues a word, so later completions through a beam are bounded by its probability times its non-boundary probability. Search stops once each returned word leads the next, or any unseen word, by more than the combined bound. The ranked words are the same as the full search.
• Natural context (neural/src/lib.rs). generation_context keeps the current sentence as typed. It collapses whitespace and drops trailing opening quotes and brackets. effective_context (reranking) is unchanged.
• --natural-context for scripts/generation_evaluate.py, and docs/generation-performance.md.

Results

Frozen workload, Windows 11 on a Ryzen AI 9 HX 370, 4 threads, fresh context per query:

Worker	Median	p95	Max	Identical words
AVX2 before → after (1,760)	581 → 121 ms	881 → 220 ms	1,658 → 517 ms	1,760 / 1,760
Portable before → after (440)	1,257 → 424 ms	1,637 → 767 ms	1,671 → 1,294 ms	440 / 440

With natural context, six-slot top six goes from 72.4% to 73.2% (28 queries gained, 15 lost), and neural fill goes from 93.7% to 94.8%. Peak RSS is unchanged at about 660 MiB. A 16-wide beam added only 0.5 points for 65% more latency, so width stays at 8. macOS ARM was not measured here.

Validation

• cargo fmt --check, cargo clippy --workspace --all-targets --features test-support --locked -D warnings and cargo test --workspace --features test-support --locked all pass.
• The ignored real-model tests pass in release against the pinned bundle. pinned_model_order_and_session_cache shows the adapted forward pass reproduces the frozen ranking, and batched_rows_match_single_row_forwards checks every batched row against a single-row forward pass (within 1e-3).
• A new randomized test checks early-stop exactness across 400 synthetic models: same words, never more evaluations, and some searches stop early.
• python -m unittest test_neural passes.

Rolling this out to Switchify PC needs a release with new worker binaries and a library pin bump. The model bundle is unchanged.

🤖 Generated with Claude Code

[enaboapps]

Independent review of b730cd1 found no correctness issues. It instrumented the new real-model test, and batched rows were bit-identical to single-row forwards (max difference 0). In the property test, 384 of 400 seeds stopped early, with identical results. Two optional follow-ups were noted and not done here:

• Reserve an estimated first batch at depth 0 (batch starts at zero). The measured worst case is 1,294 ms on the portable worker, which leaves room under the 1,600 ms cutoff.
• Make batched_rows_match_single_row_forwards require a depth ≥ 1 batch with reordered parents. This is covered today because two of the three contexts don't settle at depth 0.

[enaboapps]

CI generation-qualification on b730cd1 (portable worker, 1,000 queries) compared with the same job on #24's final run (c25f068):

Runner	Median	p95	Max	Six-slot top six	Failures
macOS ARM, before	690 ms	1,466 ms	1,868 ms	72.7%	rerank 1
macOS ARM, after	151 ms	425 ms	1,070 ms	72.6%	generation 1
Windows Server, before	1,644 ms	1,677 ms	1,712 ms	72.6%	rerank 1
Windows Server, after	1,267 ms	1,584 ms	1,709 ms	72.4%	rerank 2

On the Windows runner the old search hit the 1,600 ms cutoff on most queries (median above 1,600 ms), so results there depend on timing. The new search still hits the cutoff often on that runner. On a local Ryzen, the portable worker went from 1,257 ms to 424 ms with identical words. The single macOS generation failure isn't explained by the report. The previous run had a comparable single rerank failure. A real-device macOS check would rule out a deadline problem.