Make learned words count, boundedly, without a second baseline copy

README.md

@@ -128,9 +128,12 @@ fn example() -> Result<()> {
 Fallible operations return typed `Error` values. Reuse a long-lived predictor;
 keep loading, imports, learning, refresh and resets on a worker thread. Prediction
 uses memory only. The model uses unigram/bigram/trigram weights **0.1/0.3/0.6**,
-renormalized over available contexts, and combines baseline counts with **5×**
-personal counts before probability calculation. Prefix matches rank by score,
-then alphabetically. Schema v1 and existing personal databases remain compatible.
+renormalized over available contexts. Learned counts are kept apart from the
+baseline and added to it when predicting, one learned occurrence counting as up to
+**500** baseline ones, less as more is learned. A word the baseline lacks, or has
+seen fewer than three times, counts once learned twice. See
+[personal learning](docs/personal-learning.md). Prefix matches rank by score, then
+alphabetically. Schema v1 and existing personal databases remain compatible.
 
 ## Validation and model selection
 

docs/personal-learning.md

@@ -0,0 +1,97 @@
+# Personal learning weight
+
+Issue #29 makes learned words count. Learned counts used to be added to a copy
+of the baseline at 5×. Learned text is tiny beside the baseline, so a name or
+care term typed a few times hardly ever reached the suggestions.
+
+## Design
+
+Learned counts are kept apart from the baseline and combined when predicting:
+
+- **Weight.** One learned occurrence counts as 500 baseline occurrences while
+  little has been learned. The weight is `500 / (1 + learned words / 2000)`, so
+  it has halved after 2,000 learned words. However much is learned, learned
+  unigram mass approaches 500 × 2,000 = 1,000,000 occurrences, about the size
+  of the production baseline's 1,076,617, so learned words approach half the
+  unigram distribution rather than outgrowing the baseline. These constants are
+  tuned to this baseline; a much larger or smaller one would change that share.
+- **New and rare words.** Learned counts of a word are used once it has been
+  learned twice, unless the baseline has it at least three times. A single
+  misspelling is not suggested, including one the baseline itself contains
+  once or twice: the production baseline has over 9,000 such words, many of
+  them misspellings. Words the baseline knows well count from their first use.
+  A word learned once still counts as context for the word after it.
+- **Memory.** The baseline is no longer copied. Peak memory of the predictor
+  falls from 362 MiB to 134 MiB, with or without a personal database. A second,
+  filtered copy of the usable learned counts is kept for prediction; it is
+  small beside the baseline.
+- **Speed.** What prediction uses of the learned counts, with their totals per
+  context, is updated in place when a sentence is learned and rebuilt only on
+  reload. Measured by the ignored test `prediction_cost_on_production_baseline`
+  in a release build on the production baseline, costs match #28 within noise:
+  about 5 ms for an empty sentence and 1 ms after a typed letter without
+  learning, 9–15 ms with 2,000 new words learned, and 30–40 ms with 40,000, an
+  extreme case. One learn takes 17–23 ms at 2,000 new words and about 190 ms
+  at 40,000, including one that makes a new word usable. Most of that is the
+  existing database check.
+- **No learning.** With no personal database, or an empty one, predictions are
+  bit-identical to before.
+
+Scores are computed in floating point, so stored counts cannot overflow when
+weighted. The schema and API are unchanged, and existing personal databases
+keep working. The `personal_weight: 5` in the v1 model manifest records how that
+model was described at build time; this document describes the runtime
+weighting.
+
+## Measurement
+
+`scripts/personal_evaluate.py` uses `neural/fixtures/personal.json`. The fixture
+is synthetic text imitating one person's typing; it is not user data.
+
+- It has 44 learned sentences and 25 held-out test sentences that reuse their
+  names, places, care words and interests. Four of the learned sentences each
+  contain one misspelling.
+- Each test word is queried with 0–2 typed letters: 445 queries.
+- 78 queries are personal words: learned, and with fewer than 10 baseline
+  occurrences. They cover only 18 distinct words, each learned 1–3 times.
+- A query is a hit when the word is in any of the six slots: 3 statistical and
+  3 generated.
+
+Results from #30's final code, with v0.2.1 generation:
+
+| Weighting | Top six | Personal words | Misspelling suggested |
+| --- | ---: | ---: | ---: |
+| no learning | 50.3% | 1.3% | 0% |
+| 500×, new words at once | 61.8% | 53.8% | 0.2% |
+| **500× bounded, new words twice** | **60.4%** | **46.2%** | **0%** |
+
+Some personal words in the fixture are learned only once, so requiring two uses
+costs part of the gain here. In real use, names that matter recur.
+
+An earlier sweep of a plain multiplier, measured on the faster generation build
+from #26 and before the misspellings were added, showed how much the weight
+matters. Personal words reached the six slots in 16.7% of queries at 5×, 28.2%
+at 50×, 53.8% at 500×, 60.3% at 2,000× and 65.4% at 5,000×.
+
+General English must not suffer for someone who has learned text. 600 queries
+from the frozen general workload, with natural context, were run with that
+person's learning present. "Light" is the 40 original sentences learned once;
+"heavy" is the same text learned ten times. With #30's code and v0.2.1
+generation: no learning 70.3%, light 70.7%, heavy 70.2%. These differences are
+within noise for 600 queries. The general-workload runs used a separate local
+harness, not this script.
+
+Offering learned words to the neural generator as extra candidates was also
+tried. It added under half a point and nearly doubled generation latency, so it
+is not included.
+
+Measured on Windows 11 with an AMD Ryzen AI 9 HX 370. These are synthetic
+regression measurements, not evidence of real users' accuracy.
+
+## Reproduce
+
+Build the root CLI for learning and the neural workspace, then run:
+
+```text
+python scripts/personal_evaluate.py --cli <neural-cli> --learner <switchify-prediction> --baseline <english.sqlite> --bundle <model-directory> --worker <portable-worker> [--accelerated-worker <avx2-worker>] --output <report.json>
+```

docs/production.md

@@ -49,8 +49,11 @@ a noisy shared-runner gate. Dependency audit warnings also fail CI.
 Open one long-lived `Predictor` on a worker thread. Reusing it keeps prediction
 entirely in memory; spawning a CLI process or reopening the model for each
 keystroke incurs cold validation/loading. Loading the promoted model takes about
-1.6 seconds on the development M2 Max, and learning validates personal state and
-copies model counts. Keep open/import/learn/reset/refresh off the UI thread.
+1.6 seconds on the development M2 Max. Learning writes only learned counts and
+validates the personal database: with a few thousand learned words a sentence
+takes tens of milliseconds, and about 200 ms at tens of thousands. It reloads
+the personal database when another connection has changed it. Keep
+open/import/learn/reset/refresh off the UI thread.
 
 `Predictor` is an owned, mutable learning session. Serialize access in the host
 application; do not share an unsynchronized connection across threads. Multiple

neural/fixtures/personal.json

@@ -0,0 +1,82 @@
+{
+  "note": "Synthetic text written for this evaluation, imitating one person's typing with their own names, places, care words and interests. It is not real user data. `learn` is learned into a personal database; `test` is held out and reuses the same vocabulary in new sentences. Four learn sentences each contain one misspelling, listed in `misspellings`, to check that a word typed wrongly once is not suggested.",
+  "learn": [
+    "Siobhan is coming at ten to help with the hoist.",
+    "Can you ask Siobhan to bring my blue cardigan?",
+    "Aoife phoned last night about the physio appointment.",
+    "My physio session with Declan is on Thursday.",
+    "Declan says my shoulder is getting stronger.",
+    "Please check the suction machine is charged.",
+    "The suction tubing needs changing today.",
+    "Can someone set up my PEG feed at six?",
+    "The PEG site looks a bit red this morning.",
+    "I want to watch Liverpool tonight.",
+    "Salah scored again for Liverpool!",
+    "Did you see the Liverpool match on Saturday?",
+    "Cathal is driving us to Portrush on Sunday.",
+    "I love the beach at Portrush when it is sunny.",
+    "We stopped at Ballymena for lunch.",
+    "Mum lives in Ballymena near the park.",
+    "Niamh sent photos of Biscuit chasing the ball.",
+    "Biscuit needs his walk before tea.",
+    "Can you feed Biscuit please?",
+    "I am working on my crochet blanket.",
+    "The crochet hook fell under the chair.",
+    "Coronation Street is on at half seven.",
+    "Turn Coronation Street up a bit please.",
+    "Dr Mulholland wants to see me next week.",
+    "Ask Dr Mulholland about my new tablets.",
+    "My wheelchair battery is low again.",
+    "The wheelchair footplate is loose.",
+    "Can you tilt my wheelchair back a little?",
+    "Siobhan and Aoife are both on shift tomorrow.",
+    "I need my glasses and the iPad please.",
+    "Please charge the iPad before bed.",
+    "Cathal made a lovely stew for dinner.",
+    "Tell Niamh I said happy birthday.",
+    "I would like a cup of tea with two sugars.",
+    "My carer is running late this morning.",
+    "The carer from the agency was really kind.",
+    "Can you open the curtains please?",
+    "I am too warm, can you turn the heating down?",
+    "Thank you, that is much better.",
+    "I love you all very much.",
+    "I hav a sore throat today.",
+    "Can you open teh window please?",
+    "I am tired becuase I slept badly.",
+    "Siobahn said she will be late."
+  ],
+  "test": [
+    "Is Siobhan working this weekend?",
+    "Please ring Aoife and say I am fine.",
+    "Declan wants me to do my stretches.",
+    "Can you clean the suction machine?",
+    "Please flush my PEG after the feed.",
+    "Liverpool are playing at home tomorrow.",
+    "Put the Liverpool game on please.",
+    "Can Cathal take me to Portrush next week?",
+    "We should visit Mum in Ballymena soon.",
+    "Has anyone walked Biscuit today?",
+    "Niamh is bringing Biscuit over later.",
+    "Pass me my crochet bag please.",
+    "Record Coronation Street for me.",
+    "When is my appointment with Dr Mulholland?",
+    "My wheelchair cushion needs pumping up.",
+    "The hoist sling is in the cupboard.",
+    "My physio exercises are hard today.",
+    "Can you charge my wheelchair tonight?",
+    "Ask Siobhan if she can stay for tea.",
+    "I would like to talk to Cathal.",
+    "The iPad volume is too low.",
+    "My carer helped me with the hoist.",
+    "Can you get the suction ready please?",
+    "Aoife says the physio is cancelled.",
+    "I am tired, can I go to bed?"
+  ],
+  "misspellings": [
+    "hav",
+    "teh",
+    "becuase",
+    "siobahn"
+  ]
+}

scripts/personal_evaluate.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""Measure what learning adds to six-slot prediction, on a synthetic person.
+
+A temporary personal database learns neural/fixtures/personal.json `learn` text
+through the CLI. Held-out `test` sentences are then queried with and without
+learning. The fixture is synthetic, written for this evaluation; results show
+the mechanism, not real users' accuracy.
+Reports contain counts and timing, never words. Requires psutil and regex.
+"""
+import argparse
+import json
+from pathlib import Path
+import subprocess
+import tempfile
+import time
+
+from neural_evaluate import Client, ROOT, timing, words
+
+FIXTURE = ROOT / 'neural/fixtures/personal.json'
+
+
+def queries(baseline_counts):
+    import regex
+    fixture = json.loads(FIXTURE.read_bytes())
+    learned = {w for sentence in fixture['learn'] for w in words(sentence)}
+    result = []
+    for sentence in fixture['test']:
+        text = sentence.replace('’', "'")
+        for match in regex.finditer(r"\p{L}[\p{L}\p{M}]*(?:'\p{L}[\p{L}\p{M}]*)*", text):
+            target = words(match.group())[0]
+            graphemes = regex.findall(r'\X', target)
+            for n in range(min(3, len(graphemes))):
+                # Personal: learned, and rare or unknown in the baseline.
+                own = target in learned and baseline_counts.get(target, 0) < 10
+                result.append((text[:match.start()], ''.join(graphemes[:n]), target, own))
+    return result
+
+
+def run(args, personal, items):
+    command = [str(args.cli), 'stream', '--baseline', str(args.baseline), '--bundle', str(args.bundle),
+               '--worker', str(args.worker)]
+    if args.accelerated_worker:
+        command += ['--accelerated-worker', str(args.accelerated_worker)]
+    if personal:
+        command += ['--personal', str(personal)]
+    client = Client(command)
+    hits = dict(all=0, personal=0, generated=0, misspelled=0)
+    misspellings = set(json.loads(FIXTURE.read_bytes())['misspellings'])
+    times = []
+    failures = 0
+    try:
+        for before, prefix, target, own in items:
+            client.process.stdin.write('{"command":"reset"}\n')
+            client.process.stdin.flush()
+            assert client.receive()['type'] == 'reset'
+            start = time.perf_counter()
+            client.process.stdin.write(json.dumps(dict(command='generate', before=before,
+                                                       prefix=prefix, session=1)) + '\n')
+            client.process.stdin.flush()
+            immediate = client.receive()
+            assert immediate['type'] == 'immediate', immediate['type']
+            instant = immediate['result']['words']
+            generated = []
+            if immediate['result']['refinement_requested']:
+                reply = client.receive()
+                if reply['type'] == 'refined':
+                    assert reply['result']['request_id'] == immediate['result']['request_id']
+                    generated = reply['result']['words']
+                    times.append((time.perf_counter() - start) * 1000)
+                else:
+                    failures += 1
+                    client.retry()
+            hit = target in instant + generated
+            hits['all'] += hit
+            hits['personal'] += hit and own
+            hits['generated'] += target in generated
+            hits['misspelled'] += bool(misspellings.intersection(instant + generated))
+    finally:
+        client.close()
+    count = len(items)
+    own = sum(item[3] for item in items)
+    return dict(top6=hits['all'] / count, personal_top6=hits['personal'] / own,
+                generated_hits=hits['generated'] / count,
+                misspelling_in_slots=hits['misspelled'] / count,
+                generation_failures=failures, latency=timing(times))
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    for name in ['cli', 'learner', 'baseline', 'bundle', 'worker', 'output']:
+        parser.add_argument('--' + name, type=Path, required=True)
+    parser.add_argument('--accelerated-worker', type=Path)
+    args = parser.parse_args()
+    import sqlite3
+    db = sqlite3.connect(args.baseline.resolve().as_uri() + '?mode=ro', uri=True)
+    baseline_counts = dict(db.execute("SELECT word, count FROM counts WHERE context='[]'"))
+    db.close()
+    items = queries(baseline_counts)
+    with tempfile.TemporaryDirectory() as directory:
+        personal = Path(directory) / 'personal.sqlite'
+        # Newlines separate context exactly as separate learn calls would.
+        text = '\n'.join(json.loads(FIXTURE.read_bytes())['learn'])
+        subprocess.run([str(args.learner), 'learn', '--baseline', str(args.baseline),
+                        '--personal', str(personal)], input=text, text=True, encoding='utf-8',
+                       check=True, capture_output=True)
+        report = dict(
+            queries=len(items), personal_queries=sum(item[3] for item in items),
+            without_learning=run(args, None, items),
+            with_learning=run(args, personal, items),
+            note='Synthetic fixture; shows the mechanism, not real-user accuracy.')
+    args.output.write_text(json.dumps(report, indent=2) + '\n', encoding='utf-8')
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == '__main__':
+    main()

src/evaluation.rs

@@ -179,7 +179,7 @@ pub fn score_database(path: &Path, text: &str, hardware: String) -> Result<Score
     let predictor = Predictor::open(path, None)?;
     let cold_load_ms = start.elapsed().as_secs_f64() * 1000.0;
     let vocabulary = predictor
-        .combined
+        .baseline
         .0
         .get(&Vec::new())
         .expect("validated baseline has unigrams");