diff --git a/README.md b/README.md index 14dce0e..18fcf15 100644 --- a/README.md +++ b/README.md @@ -128,9 +128,12 @@ fn example() -> Result<()> { Fallible operations return typed `Error` values. Reuse a long-lived predictor; keep loading, imports, learning, refresh and resets on a worker thread. Prediction uses memory only. The model uses unigram/bigram/trigram weights **0.1/0.3/0.6**, -renormalized over available contexts, and combines baseline counts with **5×** -personal counts before probability calculation. Prefix matches rank by score, -then alphabetically. Schema v1 and existing personal databases remain compatible. +renormalized over available contexts. Learned counts are kept apart from the +baseline and added to it when predicting, one learned occurrence counting as up to +**500** baseline ones, less as more is learned. A word the baseline lacks, or has +seen fewer than three times, counts once learned twice. See +[personal learning](docs/personal-learning.md). Prefix matches rank by score, then +alphabetically. Schema v1 and existing personal databases remain compatible. ## Validation and model selection diff --git a/docs/personal-learning.md b/docs/personal-learning.md new file mode 100644 index 0000000..64decd0 --- /dev/null +++ b/docs/personal-learning.md @@ -0,0 +1,97 @@ +# Personal learning weight + +Issue #29 makes learned words count. Learned counts used to be added to a copy +of the baseline at 5×. Learned text is tiny beside the baseline, so a name or +care term typed a few times hardly ever reached the suggestions. + +## Design + +Learned counts are kept apart from the baseline and combined when predicting: + +- **Weight.** One learned occurrence counts as 500 baseline occurrences while + little has been learned. The weight is `500 / (1 + learned words / 2000)`, so + it has halved after 2,000 learned words. However much is learned, learned + unigram mass approaches 500 × 2,000 = 1,000,000 occurrences, about the size + of the production baseline's 1,076,617, so learned words approach half the + unigram distribution rather than outgrowing the baseline. These constants are + tuned to this baseline; a much larger or smaller one would change that share. +- **New and rare words.** Learned counts of a word are used once it has been + learned twice, unless the baseline has it at least three times. A single + misspelling is not suggested, including one the baseline itself contains + once or twice: the production baseline has over 9,000 such words, many of + them misspellings. Words the baseline knows well count from their first use. + A word learned once still counts as context for the word after it. +- **Memory.** The baseline is no longer copied. Peak memory of the predictor + falls from 362 MiB to 134 MiB, with or without a personal database. A second, + filtered copy of the usable learned counts is kept for prediction; it is + small beside the baseline. +- **Speed.** What prediction uses of the learned counts, with their totals per + context, is updated in place when a sentence is learned and rebuilt only on + reload. Measured by the ignored test `prediction_cost_on_production_baseline` + in a release build on the production baseline, costs match #28 within noise: + about 5 ms for an empty sentence and 1 ms after a typed letter without + learning, 9–15 ms with 2,000 new words learned, and 30–40 ms with 40,000, an + extreme case. One learn takes 17–23 ms at 2,000 new words and about 190 ms + at 40,000, including one that makes a new word usable. Most of that is the + existing database check. +- **No learning.** With no personal database, or an empty one, predictions are + bit-identical to before. + +Scores are computed in floating point, so stored counts cannot overflow when +weighted. The schema and API are unchanged, and existing personal databases +keep working. The `personal_weight: 5` in the v1 model manifest records how that +model was described at build time; this document describes the runtime +weighting. + +## Measurement + +`scripts/personal_evaluate.py` uses `neural/fixtures/personal.json`. The fixture +is synthetic text imitating one person's typing; it is not user data. + +- It has 44 learned sentences and 25 held-out test sentences that reuse their + names, places, care words and interests. Four of the learned sentences each + contain one misspelling. +- Each test word is queried with 0–2 typed letters: 445 queries. +- 78 queries are personal words: learned, and with fewer than 10 baseline + occurrences. They cover only 18 distinct words, each learned 1–3 times. +- A query is a hit when the word is in any of the six slots: 3 statistical and + 3 generated. + +Results from #30's final code, with v0.2.1 generation: + +| Weighting | Top six | Personal words | Misspelling suggested | +| --- | ---: | ---: | ---: | +| no learning | 50.3% | 1.3% | 0% | +| 500×, new words at once | 61.8% | 53.8% | 0.2% | +| **500× bounded, new words twice** | **60.4%** | **46.2%** | **0%** | + +Some personal words in the fixture are learned only once, so requiring two uses +costs part of the gain here. In real use, names that matter recur. + +An earlier sweep of a plain multiplier, measured on the faster generation build +from #26 and before the misspellings were added, showed how much the weight +matters. Personal words reached the six slots in 16.7% of queries at 5×, 28.2% +at 50×, 53.8% at 500×, 60.3% at 2,000× and 65.4% at 5,000×. + +General English must not suffer for someone who has learned text. 600 queries +from the frozen general workload, with natural context, were run with that +person's learning present. "Light" is the 40 original sentences learned once; +"heavy" is the same text learned ten times. With #30's code and v0.2.1 +generation: no learning 70.3%, light 70.7%, heavy 70.2%. These differences are +within noise for 600 queries. The general-workload runs used a separate local +harness, not this script. + +Offering learned words to the neural generator as extra candidates was also +tried. It added under half a point and nearly doubled generation latency, so it +is not included. + +Measured on Windows 11 with an AMD Ryzen AI 9 HX 370. These are synthetic +regression measurements, not evidence of real users' accuracy. + +## Reproduce + +Build the root CLI for learning and the neural workspace, then run: + +```text +python scripts/personal_evaluate.py --cli --learner --baseline --bundle --worker [--accelerated-worker ] --output +``` diff --git a/docs/production.md b/docs/production.md index bcce03d..3654cd0 100644 --- a/docs/production.md +++ b/docs/production.md @@ -49,8 +49,11 @@ a noisy shared-runner gate. Dependency audit warnings also fail CI. Open one long-lived `Predictor` on a worker thread. Reusing it keeps prediction entirely in memory; spawning a CLI process or reopening the model for each keystroke incurs cold validation/loading. Loading the promoted model takes about -1.6 seconds on the development M2 Max, and learning validates personal state and -copies model counts. Keep open/import/learn/reset/refresh off the UI thread. +1.6 seconds on the development M2 Max. Learning writes only learned counts and +validates the personal database: with a few thousand learned words a sentence +takes tens of milliseconds, and about 200 ms at tens of thousands. It reloads +the personal database when another connection has changed it. Keep +open/import/learn/reset/refresh off the UI thread. `Predictor` is an owned, mutable learning session. Serialize access in the host application; do not share an unsynchronized connection across threads. Multiple diff --git a/neural/fixtures/personal.json b/neural/fixtures/personal.json new file mode 100644 index 0000000..6eaec95 --- /dev/null +++ b/neural/fixtures/personal.json @@ -0,0 +1,82 @@ +{ + "note": "Synthetic text written for this evaluation, imitating one person's typing with their own names, places, care words and interests. It is not real user data. `learn` is learned into a personal database; `test` is held out and reuses the same vocabulary in new sentences. Four learn sentences each contain one misspelling, listed in `misspellings`, to check that a word typed wrongly once is not suggested.", + "learn": [ + "Siobhan is coming at ten to help with the hoist.", + "Can you ask Siobhan to bring my blue cardigan?", + "Aoife phoned last night about the physio appointment.", + "My physio session with Declan is on Thursday.", + "Declan says my shoulder is getting stronger.", + "Please check the suction machine is charged.", + "The suction tubing needs changing today.", + "Can someone set up my PEG feed at six?", + "The PEG site looks a bit red this morning.", + "I want to watch Liverpool tonight.", + "Salah scored again for Liverpool!", + "Did you see the Liverpool match on Saturday?", + "Cathal is driving us to Portrush on Sunday.", + "I love the beach at Portrush when it is sunny.", + "We stopped at Ballymena for lunch.", + "Mum lives in Ballymena near the park.", + "Niamh sent photos of Biscuit chasing the ball.", + "Biscuit needs his walk before tea.", + "Can you feed Biscuit please?", + "I am working on my crochet blanket.", + "The crochet hook fell under the chair.", + "Coronation Street is on at half seven.", + "Turn Coronation Street up a bit please.", + "Dr Mulholland wants to see me next week.", + "Ask Dr Mulholland about my new tablets.", + "My wheelchair battery is low again.", + "The wheelchair footplate is loose.", + "Can you tilt my wheelchair back a little?", + "Siobhan and Aoife are both on shift tomorrow.", + "I need my glasses and the iPad please.", + "Please charge the iPad before bed.", + "Cathal made a lovely stew for dinner.", + "Tell Niamh I said happy birthday.", + "I would like a cup of tea with two sugars.", + "My carer is running late this morning.", + "The carer from the agency was really kind.", + "Can you open the curtains please?", + "I am too warm, can you turn the heating down?", + "Thank you, that is much better.", + "I love you all very much.", + "I hav a sore throat today.", + "Can you open teh window please?", + "I am tired becuase I slept badly.", + "Siobahn said she will be late." + ], + "test": [ + "Is Siobhan working this weekend?", + "Please ring Aoife and say I am fine.", + "Declan wants me to do my stretches.", + "Can you clean the suction machine?", + "Please flush my PEG after the feed.", + "Liverpool are playing at home tomorrow.", + "Put the Liverpool game on please.", + "Can Cathal take me to Portrush next week?", + "We should visit Mum in Ballymena soon.", + "Has anyone walked Biscuit today?", + "Niamh is bringing Biscuit over later.", + "Pass me my crochet bag please.", + "Record Coronation Street for me.", + "When is my appointment with Dr Mulholland?", + "My wheelchair cushion needs pumping up.", + "The hoist sling is in the cupboard.", + "My physio exercises are hard today.", + "Can you charge my wheelchair tonight?", + "Ask Siobhan if she can stay for tea.", + "I would like to talk to Cathal.", + "The iPad volume is too low.", + "My carer helped me with the hoist.", + "Can you get the suction ready please?", + "Aoife says the physio is cancelled.", + "I am tired, can I go to bed?" + ], + "misspellings": [ + "hav", + "teh", + "becuase", + "siobahn" + ] +} diff --git a/scripts/personal_evaluate.py b/scripts/personal_evaluate.py new file mode 100644 index 0000000..beeaca3 --- /dev/null +++ b/scripts/personal_evaluate.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +"""Measure what learning adds to six-slot prediction, on a synthetic person. + +A temporary personal database learns neural/fixtures/personal.json `learn` text +through the CLI. Held-out `test` sentences are then queried with and without +learning. The fixture is synthetic, written for this evaluation; results show +the mechanism, not real users' accuracy. +Reports contain counts and timing, never words. Requires psutil and regex. +""" +import argparse +import json +from pathlib import Path +import subprocess +import tempfile +import time + +from neural_evaluate import Client, ROOT, timing, words + +FIXTURE = ROOT / 'neural/fixtures/personal.json' + + +def queries(baseline_counts): + import regex + fixture = json.loads(FIXTURE.read_bytes()) + learned = {w for sentence in fixture['learn'] for w in words(sentence)} + result = [] + for sentence in fixture['test']: + text = sentence.replace('’', "'") + for match in regex.finditer(r"\p{L}[\p{L}\p{M}]*(?:'\p{L}[\p{L}\p{M}]*)*", text): + target = words(match.group())[0] + graphemes = regex.findall(r'\X', target) + for n in range(min(3, len(graphemes))): + # Personal: learned, and rare or unknown in the baseline. + own = target in learned and baseline_counts.get(target, 0) < 10 + result.append((text[:match.start()], ''.join(graphemes[:n]), target, own)) + return result + + +def run(args, personal, items): + command = [str(args.cli), 'stream', '--baseline', str(args.baseline), '--bundle', str(args.bundle), + '--worker', str(args.worker)] + if args.accelerated_worker: + command += ['--accelerated-worker', str(args.accelerated_worker)] + if personal: + command += ['--personal', str(personal)] + client = Client(command) + hits = dict(all=0, personal=0, generated=0, misspelled=0) + misspellings = set(json.loads(FIXTURE.read_bytes())['misspellings']) + times = [] + failures = 0 + try: + for before, prefix, target, own in items: + client.process.stdin.write('{"command":"reset"}\n') + client.process.stdin.flush() + assert client.receive()['type'] == 'reset' + start = time.perf_counter() + client.process.stdin.write(json.dumps(dict(command='generate', before=before, + prefix=prefix, session=1)) + '\n') + client.process.stdin.flush() + immediate = client.receive() + assert immediate['type'] == 'immediate', immediate['type'] + instant = immediate['result']['words'] + generated = [] + if immediate['result']['refinement_requested']: + reply = client.receive() + if reply['type'] == 'refined': + assert reply['result']['request_id'] == immediate['result']['request_id'] + generated = reply['result']['words'] + times.append((time.perf_counter() - start) * 1000) + else: + failures += 1 + client.retry() + hit = target in instant + generated + hits['all'] += hit + hits['personal'] += hit and own + hits['generated'] += target in generated + hits['misspelled'] += bool(misspellings.intersection(instant + generated)) + finally: + client.close() + count = len(items) + own = sum(item[3] for item in items) + return dict(top6=hits['all'] / count, personal_top6=hits['personal'] / own, + generated_hits=hits['generated'] / count, + misspelling_in_slots=hits['misspelled'] / count, + generation_failures=failures, latency=timing(times)) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + for name in ['cli', 'learner', 'baseline', 'bundle', 'worker', 'output']: + parser.add_argument('--' + name, type=Path, required=True) + parser.add_argument('--accelerated-worker', type=Path) + args = parser.parse_args() + import sqlite3 + db = sqlite3.connect(args.baseline.resolve().as_uri() + '?mode=ro', uri=True) + baseline_counts = dict(db.execute("SELECT word, count FROM counts WHERE context='[]'")) + db.close() + items = queries(baseline_counts) + with tempfile.TemporaryDirectory() as directory: + personal = Path(directory) / 'personal.sqlite' + # Newlines separate context exactly as separate learn calls would. + text = '\n'.join(json.loads(FIXTURE.read_bytes())['learn']) + subprocess.run([str(args.learner), 'learn', '--baseline', str(args.baseline), + '--personal', str(personal)], input=text, text=True, encoding='utf-8', + check=True, capture_output=True) + report = dict( + queries=len(items), personal_queries=sum(item[3] for item in items), + without_learning=run(args, None, items), + with_learning=run(args, personal, items), + note='Synthetic fixture; shows the mechanism, not real-user accuracy.') + args.output.write_text(json.dumps(report, indent=2) + '\n', encoding='utf-8') + print(json.dumps(report, indent=2)) + + +if __name__ == '__main__': + main() diff --git a/src/evaluation.rs b/src/evaluation.rs index 46216f0..57aea88 100644 --- a/src/evaluation.rs +++ b/src/evaluation.rs @@ -179,7 +179,7 @@ pub fn score_database(path: &Path, text: &str, hardware: String) -> Result, - /// The personal database's `data_version` when `combined` last matched it. + /// The personal database's `data_version` when `learned` last matched it. /// It changes only when another connection commits. seen: Option, } +/// The learned counts whose words may count, their total per context, and +/// the weight of one learned occurrence. +#[derive(Default)] +struct Usable { + counts: Counts, + totals: BTreeMap, f64>, + /// All learned unigram occurrences, which set the weight. + learned: f64, + weight: f64, +} + fn data_version(db: &Connection) -> Result { Ok(db.query_row("PRAGMA data_version", [], |r| r.get(0))?) } @@ -417,8 +444,9 @@ impl Predictor { None }; let mut result = Self { - baseline: counts.clone(), - combined: counts, + baseline: counts, + learned: Counts::default(), + usable: Usable::default(), personal, seen: None, }; @@ -428,13 +456,13 @@ impl Predictor { /// Refresh the in-memory snapshot after another process changes personal learning. /// On failure the previous prediction snapshot is retained. pub fn refresh_personal(&mut self) -> Result<()> { - let mut combined = self.baseline.clone(); + let mut learned = Counts::default(); if let Some(db) = &mut self.personal { let tx = db.transaction()?; if check(&tx)? != "personal" { return Err(Error::Invalid("expected personal database".into())); } - combined.merge(&load_checked(&tx, "personal")?, 5); + learned = load_checked(&tx, "personal")?; let version = data_version(&tx)?; if let Err(error) = tx.commit() { // The outcome is unknown, so the next write reloads. @@ -443,9 +471,103 @@ impl Predictor { } self.seen = Some(version); } - self.combined = combined; + self.learned = learned; + self.rebuild(); Ok(()) } + /// Rebuilds what predicting uses after `learned` changes. Summed as + /// floats: stored counts are bounded one by one, not in total. + fn rebuild(&mut self) { + let mut usable = Usable::default(); + for (context, words) in &self.learned.0 { + let kept: BTreeMap = words + .iter() + .filter(|(w, _)| self.usable_word(w)) + .map(|(w, c)| (w.clone(), *c)) + .collect(); + if !kept.is_empty() { + usable + .totals + .insert(context.clone(), kept.values().map(|c| *c as f64).sum()); + usable.counts.0.insert(context.clone(), kept); + } + } + self.usable = usable; + self.usable.learned = self + .learned + .0 + .get(&Vec::new()) + .map_or(0., |w| w.values().map(|c| *c as f64).sum()); + self.reweigh(); + } + /// Updates what prediction uses after `delta` was added to `learned`: + /// its usable counts are added, and a word that has just become usable + /// gets all its learned counts. Counts are whole numbers, so while totals + /// stay below 2^53, which learning cannot pass, adding them + /// in place gives the same totals as rebuilding. + fn update(&mut self, delta: &Counts) { + let empty = BTreeMap::new(); + let added = delta.0.get(&Vec::new()).unwrap_or(&empty); + let learned = self.learned.0.get(&Vec::new()).unwrap_or(&empty); + let newly: BTreeSet<&String> = added + .iter() + .filter(|(word, count)| { + let now = learned.get(*word).copied().unwrap_or(0); + !self.trusted(word) && now >= NEW_WORD_REPEATS && now - *count < NEW_WORD_REPEATS + }) + .map(|(word, _)| word) + .collect(); + let mut changes: Vec<(Vec, String, i64)> = Vec::new(); + if !newly.is_empty() { + for (context, words) in &self.learned.0 { + for (word, count) in words { + if newly.contains(word) { + changes.push((context.clone(), word.clone(), *count)); + } + } + } + } + for (context, words) in &delta.0 { + for (word, count) in words { + if !newly.contains(word) && self.usable_word(word) { + changes.push((context.clone(), word.clone(), *count)); + } + } + } + for (context, word, count) in changes { + *self.usable.totals.entry(context.clone()).or_default() += count as f64; + *self + .usable + .counts + .0 + .entry(context) + .or_default() + .entry(word) + .or_default() += count; + } + self.usable.learned += added.values().map(|c| *c as f64).sum::(); + self.reweigh(); + } + fn trusted(&self, word: &str) -> bool { + self.baseline + .0 + .get(&Vec::new()) + .and_then(|w| w.get(word)) + .is_some_and(|c| *c >= TRUSTED_BASELINE) + } + /// Whether learned counts of `word` are used. + fn usable_word(&self, word: &str) -> bool { + self.trusted(word) + || self + .learned + .0 + .get(&Vec::new()) + .and_then(|w| w.get(word)) + .is_some_and(|c| *c >= NEW_WORD_REPEATS) + } + fn reweigh(&mut self) { + self.usable.weight = PERSONAL_WEIGHT / (1. + self.usable.learned / PERSONAL_SATURATION); + } pub fn predict(&self, before: &str, prefix: &str, options: Options) -> Vec { let prefix = normalize(prefix); if prefix.graphemes(true).count() < options.min_chars || options.limit == 0 { @@ -456,35 +578,60 @@ impl Predictor { .next() .unwrap_or(""); let context = sentences(tail).pop().unwrap_or_default(); + let personal = self.usable.weight; + let empty = BTreeMap::new(); let mut distributions = Vec::new(); for (n, weight) in [(0, 0.1), (1, 0.3), (2, 0.6)] { if n > context.len() || (options.unigram_only && n != 0) { continue; } - if let Some(words) = self.combined.0.get(&context[context.len() - n..]) { - let total: f64 = words.values().map(|c| *c as f64).sum(); - distributions.push((words, total, weight)); + let key = &context[context.len() - n..]; + let base = self.baseline.0.get(key); + let learned = self.usable.counts.0.get(key); + if base.is_none() && learned.is_none() { + continue; + } + let base = base.unwrap_or(&empty); + let mut total: f64 = base.values().map(|c| *c as f64).sum(); + if learned.is_some() { + total += personal * self.usable.totals[key]; } + distributions.push((base, learned, total, weight)); } - let weight_sum: f64 = distributions.iter().map(|(_, _, w)| w).sum(); - let mut result = Vec::new(); - if let Some(vocabulary) = self.combined.0.get(&Vec::new()) { - for (word, _) in vocabulary.range(prefix.clone()..) { - if !word.starts_with(&prefix) { - break; - } + let weight_sum: f64 = distributions.iter().map(|(_, _, _, w)| w).sum(); + // Baseline words, then learned words the baseline lacks, starting with + // the prefix. + fn range<'a>(counts: &'a Counts, prefix: &str) -> impl Iterator { + counts + .0 + .get(&Vec::new()) + .into_iter() + .flat_map(move |v| v.range(prefix.to_owned()..)) + .take_while(move |(w, _)| w.starts_with(prefix)) + .map(|(w, _)| w) + } + let baseline = self.baseline.0.get(&Vec::new()).unwrap_or(&empty); + let mut words: Vec<&String> = range(&self.baseline, &prefix).collect(); + words.extend(range(&self.usable.counts, &prefix).filter(|w| !baseline.contains_key(*w))); + let mut result: Vec = words + .into_iter() + .map(|word| { let score = distributions .iter() - .map(|(words, total, w)| { - words.get(word).copied().unwrap_or(0) as f64 / total * w / weight_sum + .map(|(base, learned, total, w)| { + let own = learned.and_then(|l| l.get(word)).copied().unwrap_or(0); + (base.get(word).copied().unwrap_or(0) as f64 + personal * own as f64) + / total + * w + / weight_sum }) .sum(); - result.push(Suggestion { + Suggestion { word: word.clone(), score, - }); - } - } + } + }) + .collect(); result.sort_unstable_by(|a, b| { b.score .total_cmp(&a.score) @@ -520,12 +667,10 @@ impl Predictor { // A connection's own commits leave its data_version unchanged. let version = data_version(&tx)?; let current = self.seen == Some(version) && standard_schema(&tx)?; - let combined = if current { + let reloaded = if current { None } else { - let mut combined = self.baseline.clone(); - combined.merge(&load_checked(&tx, "personal")?, 5); - Some(combined) + Some(load_checked(&tx, "personal")?) }; if let Err(error) = tx.commit() { // The outcome is unknown, so the next write reloads. @@ -533,9 +678,15 @@ impl Predictor { return Err(error.into()); } self.seen = Some(version); - match combined { - Some(combined) => self.combined = combined, - None if imported => self.combined.merge(&delta, 5), + match reloaded { + Some(learned) => { + self.learned = learned; + self.rebuild(); + } + None if imported => { + self.learned.merge(&delta, 1); + self.update(&delta); + } None => {} } Ok(imported) @@ -552,7 +703,6 @@ impl Predictor { .personal .as_mut() .ok_or_else(|| Error::Invalid("personal database required".into()))?; - let combined = self.baseline.clone(); let tx = db.transaction_with_behavior(TransactionBehavior::Immediate)?; if check(&tx)? != "personal" { return Err(Error::Invalid("expected personal database".into())); @@ -575,12 +725,13 @@ impl Predictor { return Err(error.into()); } self.seen = Some(version); - self.combined = combined; + self.learned = Counts::default(); + self.rebuild(); Ok(()) } /// Approximate bytes of stored UTF-8 keys and counts, excluding allocator/tree overhead. pub fn model_payload_bytes(&self) -> usize { - [&self.baseline, &self.combined] + [&self.baseline, &self.learned] .iter() .map(|counts| { counts @@ -600,17 +751,82 @@ impl Predictor { mod tests { use super::*; - /// Adds a word to the baseline that only a full reload copies into the - /// snapshot. + /// Adds a learned count only memory has, which a reload from the + /// database removes and adding counts in place keeps. fn mark(p: &mut Predictor, word: &str) { - p.baseline + p.learned .0 .entry(vec![]) .or_default() .insert(word.into(), 1); } fn reloaded(p: &Predictor, word: &str) -> bool { - p.combined.0[&Vec::new()].contains_key(word) + !p.learned.0[&Vec::new()].contains_key(word) + } + + #[test] + fn updating_in_place_matches_rebuilding() { + let dir = tempfile::tempdir().unwrap(); + let base = dir.path().join("base.sqlite"); + let personal = dir.path().join("personal.sqlite"); + // "like" and "tea" are trusted: the baseline has them three times. + build( + &base, + "hello water. i like tea. we like tea. you like tea.", + "fixture", + ) + .unwrap(); + let imported = dir.path().join("imported.txt"); + fs::write(&imported, "zed likes tea. Aoife likes tea. Aoife said zed.").unwrap(); + let mut p = Predictor::open(&base, Some(&personal)).unwrap(); + let check = |p: &mut Predictor, label: &str| { + let (counts, totals, weight) = ( + p.usable.counts.0.clone(), + p.usable.totals.clone(), + p.usable.weight, + ); + p.rebuild(); + assert_eq!(counts, p.usable.counts.0, "{label}"); + assert_eq!(totals, p.usable.totals, "{label}"); + assert_eq!(weight.to_bits(), p.usable.weight.to_bits(), "{label}"); + }; + assert!(p.import(&imported).unwrap()); + check(&mut p, "import"); + assert!(!p.import(&imported).unwrap()); + check(&mut p, "duplicate import"); + for text in [ + "Siobhan likes zed water.", + "zed Siobhan likes tea.", + "hello zed. Siobhan said hello.", + "", + "we like Siobhan and zed", + "you like tea.", + ] { + p.learn(text).unwrap(); + check(&mut p, text); + if text == "we like Siobhan and zed" { + // "like" is trusted, so its first learned use counts. + assert!(p.usable.counts.0[&vec!["we".to_owned()]].contains_key("like")); + } + } + assert!(p.usable.counts.0[&Vec::new()].contains_key("siobhan")); + } + + #[test] + fn the_learned_weight_halves_as_much_is_learned() { + let dir = tempfile::tempdir().unwrap(); + let base = dir.path().join("base.sqlite"); + let personal = dir.path().join("personal.sqlite"); + build(&base, "hello water", "fixture").unwrap(); + let mut p = Predictor::open(&base, Some(&personal)).unwrap(); + assert_eq!(p.usable.weight, PERSONAL_WEIGHT); + p.learned + .0 + .entry(vec![]) + .or_default() + .insert("hello".into(), PERSONAL_SATURATION as i64); + p.rebuild(); + assert_eq!(p.usable.weight, PERSONAL_WEIGHT / 2.); } #[test] diff --git a/tests/prediction.rs b/tests/prediction.rs index 8832528..035ffaf 100644 --- a/tests/prediction.rs +++ b/tests/prediction.rs @@ -85,12 +85,12 @@ fn learning_persistence_import_dedup_reset_and_baseline_immutability() { let original = fs::read(&path).unwrap(); let personal = tmp.path().join("personal.sqlite"); let text = tmp.path().join("training.txt"); - fs::write(&text, "I drink watermelon.").unwrap(); + fs::write(&text, "I drink watermelon. I like watermelon.").unwrap(); let mut p = Predictor::open(&path, Some(&personal)).unwrap(); assert!(p.import(&text).unwrap()); assert!(!p.import(&text).unwrap()); assert_eq!(words(&p, "I drink ", "wa")[0], "watermelon"); - p.learn("I drink walnut.").unwrap(); + p.learn("I drink walnut. I drink walnut.").unwrap(); drop(p); let mut p = Predictor::open(&path, Some(&personal)).unwrap(); assert!(words(&p, "I drink ", "wa").contains(&"walnut".to_string())); @@ -197,7 +197,7 @@ fn cli_build_predict_learn_validate() { .stdin .take() .unwrap() - .write_all(b"hello walnut") + .write_all(b"hello walnut. hello walnut") .unwrap(); assert!(child.wait().unwrap().success()); let output = Command::new(exe) @@ -238,9 +238,17 @@ fn interpolation_renormalizes_and_personal_counts_are_weighted_before_probabilit assert!((tea - (0.6 * 0.5 + 0.3 * 0.5 + 0.1 / 9.0)).abs() < 1e-12); let backed_off = p.predict("unknown like ", "tea", opts).remove(0).score; assert!((backed_off - (0.3 * 0.5 + 0.1 / 9.0) / 0.4).abs() < 1e-12); + // Words this rare in the baseline count once learned twice. + p.learn("i like tea.").unwrap(); + assert_eq!(p.predict("i like ", "tea", opts).remove(0).score, tea); p.learn("i like tea.").unwrap(); let learned = p.predict("i like ", "tea", opts).remove(0).score; - assert!((learned - (0.9 * 6.0 / 7.0 + 0.1 * 6.0 / 24.0)).abs() < 1e-12); + // One learned occurrence counts as 500 baseline ones, less as more is + // learned: here six words. + let w = 500.0 / (1.0 + 6.0 / 2000.0); + let expected = + 0.9 * (1.0 + 2.0 * w) / (2.0 + 2.0 * w) + 0.1 * (1.0 + 2.0 * w) / (9.0 + 6.0 * w); + assert!((learned - expected).abs() < 1e-12); let total: f64 = p.predict("i like ", "", opts).iter().map(|s| s.score).sum(); assert!((total - 1.0).abs() < 1e-12); } @@ -345,7 +353,7 @@ fn validation_failure_rolls_back_disk_and_keeps_prediction_snapshot() { assert_eq!(predictor.predict("hello", "wa", Options::default()), before); assert_eq!(validate(&personal).unwrap().ngrams, 0); db.execute_batch("DROP TRIGGER poison").unwrap(); - predictor.learn("hello walnut").unwrap(); + predictor.learn("hello walnut. hello walnut").unwrap(); assert_eq!( predictor.predict("hello", "wa", Options::default())[0].word, "walnut" @@ -360,7 +368,7 @@ fn multiple_predictors_refresh_duplicate_imports_and_reset() { let personal = dir.path().join("personal.sqlite"); let training = dir.path().join("fixture.txt"); build(&base, "hello water", "fixture").unwrap(); - fs::write(&training, "hello walnut").unwrap(); + fs::write(&training, "hello walnut. hello walnut").unwrap(); let mut first = Predictor::open(&base, Some(&personal)).unwrap(); let mut second = Predictor::open(&base, Some(&personal)).unwrap(); assert!(first.import(&training).unwrap()); @@ -486,7 +494,9 @@ fn learning_after_another_writer_includes_its_counts() { first .learn("I drink walnut milk. I drink walnut milk.") .unwrap(); - second.learn("I drink walrus milk.").unwrap(); + second + .learn("I drink walrus milk. I drink walrus milk.") + .unwrap(); assert!(words(&second, "I drink", "wal").starts_with(&["walnut".into(), "walrus".into()])); assert_matches_reload(&second, &base, &personal); second.learn("We walk home.").unwrap(); @@ -570,3 +580,134 @@ fn a_replacing_constraint_is_still_validated_before_commit() { assert!(p.learn("say hello").is_err()); assert!(Predictor::open(&base, Some(&personal)).is_ok()); } + +#[test] +fn a_misspelling_learned_once_is_not_suggested_but_a_new_word_used_twice_is() { + let (tmp, base) = fixture(); + let personal = tmp.path().join("personal.sqlite"); + let mut p = Predictor::open(&base, Some(&personal)).unwrap(); + // A word the baseline knows well counts from its first learned use; one + // it has rarely seen, which may itself be a misspelling, needs a second. + assert_eq!(words(&p, "We walk ", "")[0], "home"); + p.learn("We walk wine.").unwrap(); + assert_eq!(words(&p, "We walk ", "")[0], "home"); + p.learn("We walk drink.").unwrap(); + assert_eq!(words(&p, "We walk ", "")[0], "drink"); + p.learn("I drink teh water.").unwrap(); + assert!(!words(&p, "I drink ", "t").contains(&"teh".to_string())); + assert!(!words(&p, "", "te").contains(&"teh".to_string())); + p.learn("I drink teh tea.").unwrap(); + assert!(words(&p, "I drink ", "t").contains(&"teh".to_string())); +} + +#[test] +fn an_empty_personal_database_predicts_exactly_like_none() { + let (tmp, base) = fixture(); + let personal = tmp.path().join("personal.sqlite"); + let with = Predictor::open(&base, Some(&personal)).unwrap(); + let without = Predictor::open(&base, None).unwrap(); + for (context, prefix) in [("", ""), ("I drink", ""), ("I drink", "wa"), ("we", "w")] { + assert_eq!( + ranked(&with, context, prefix), + ranked(&without, context, prefix) + ); + } +} + +#[test] +fn huge_stored_personal_counts_do_not_overflow() { + let (tmp, base) = fixture(); + let personal = tmp.path().join("personal.sqlite"); + let mut p = Predictor::open(&base, Some(&personal)).unwrap(); + p.learn("I drink water. I drink tea.").unwrap(); + drop(p); + let db = rusqlite::Connection::open(&personal).unwrap(); + db.execute("UPDATE counts SET count = 922337203685477580", []) + .unwrap(); + validate(&personal).unwrap(); + let p = Predictor::open(&base, Some(&personal)).unwrap(); + let ranked = p.predict( + "I drink ", + "", + Options { + limit: 20, + min_chars: 0, + unigram_only: false, + }, + ); + assert!(!ranked.is_empty()); + assert!(ranked.iter().all(|s| s.score.is_finite())); +} + +#[test] +fn without_learning_scores_are_exactly_the_interpolated_counts() { + let (_tmp, base) = fixture(); + let p = Predictor::open(&base, None).unwrap(); + let water = p.predict("I drink ", "wa", Options::default()).remove(0); + assert_eq!(water.word, "water"); + // Seventeen baseline words; "i drink" and "drink" are followed by water + // twice and wine once. Summed in the order the predictor sums them. + let weights: f64 = 0.1 + 0.3 + 0.6; + let expected = + 0.0 + 2.0 / 17.0 * 0.1 / weights + 2.0 / 3.0 * 0.3 / weights + 2.0 / 3.0 * 0.6 / weights; + assert_eq!(water.score.to_bits(), expected.to_bits()); +} + +/// Run with SWITCHIFY_BASELINE pointing at the production english.sqlite. +#[test] +#[ignore = "requires explicit SWITCHIFY_BASELINE"] +fn prediction_cost_on_production_baseline() { + let base = std::path::PathBuf::from(std::env::var_os("SWITCHIFY_BASELINE").unwrap()); + let tmp = tempfile::tempdir().unwrap(); + let personal = tmp.path().join("personal.sqlite"); + let mut p = Predictor::open(&base, Some(&personal)).unwrap(); + let options = Options { + limit: 6, + min_chars: 0, + unigram_only: false, + }; + let time = |p: &Predictor, label: &str| { + for (before, prefix) in [("", ""), ("i want to", ""), ("the", "s")] { + let started = std::time::Instant::now(); + for _ in 0..50 { + p.predict(before, prefix, options); + } + println!( + "{label} {before:?} {prefix:?}: {:?}", + started.elapsed() / 50 + ); + } + }; + time(&p, "none"); + // Digits split words, so each new word is spelled in letters. + let letters = |mut i: usize| { + let mut word = String::from("zq"); + loop { + word.push((b'a' + (i % 26) as u8) as char); + i /= 26; + if i == 0 { + return word; + } + } + }; + for (count, label) in [(2_000, "2k new words"), (40_000, "40k new words")] { + let text: String = (0..count) + .map(|i| format!("I need to call {} about the physio. ", letters(i))) + .collect(); + p.learn(&text).unwrap(); + p.learn(&text).unwrap(); + let started = std::time::Instant::now(); + p.learn("I need to call the physio about my zzhoist.") + .unwrap(); + println!("{label}: one learn {:?}", started.elapsed()); + // A second use makes the new word usable, adding its counts in one pass. + let started = std::time::Instant::now(); + p.learn("I need to call the physio about my zzhoist.") + .unwrap(); + println!( + "{label}: learn making a word usable {:?}", + started.elapsed() + ); + time(&p, label); + } +}