diff --git a/docs/neural-prediction.md b/docs/neural-prediction.md index 3efc262d..83b76417 100644 --- a/docs/neural-prediction.md +++ b/docs/neural-prediction.md @@ -2,7 +2,7 @@ Word prediction shows the statistical model's first three words immediately, then asks SmolLM2-135M Q8 for up to three different words for the remaining slots; see [six prediction slots](six-predictions.md). The statistical words are never reordered or replaced. This runs whenever word prediction is enabled. Existing disabled preferences remain disabled; the compatibility-only enhanced setting remains inert. -Only the isolated prediction worker holds Switchify's tracked typing context. This does not read arbitrary text from focused fields or learn personal text. The neural worker starts on the first prediction, loads once and uses four threads. Windows selects AVX2 only after checking AVX2, FMA and F16C; macOS uses the portable ARM worker. The display keeps its current words selectable while their row is scanned, then shows the queued generated words after leaving the row. Accepting a suggestion uses its exact batch token, never an index into a replacement list. +Only the isolated prediction worker holds Switchify's tracked typing context. This does not read arbitrary text from focused fields. The model itself does not learn; learning from typing changes only the statistical words it excludes. The neural worker starts on the first prediction, loads once and uses four threads. Windows selects AVX2 only after checking AVX2, FMA and F16C; macOS uses the portable ARM worker. The display keeps its current words selectable while their row is scanned, then shows the queued generated words after leaving the row. Accepting a suggestion uses its exact batch token, never an index into a replacement list. Neural startup, crashes and the 2 second reply deadline leave statistical predictions available. There is no automatic neural retry loop; reopening the keyboard or explicitly retrying prediction starts a fresh session. Context invalidation cancels pending generation. Windows job containment and macOS process groups cover the prediction process tree. Keyboard closure kills the context-bearing worker; its bounded spare loads only the statistical model until the next prediction. @@ -36,4 +36,4 @@ The benchmark now requires 1,000 successful warmed refinements across at most fi ## Reply timeout update -The companion inference and reset reply deadline is now 2 seconds, increased from 500 ms after repeated macOS CI timeouts. Startup remains bounded at 30 seconds, statistical suggestions remain immediate, and late or stale results remain rejected. That change pinned both Rust libraries to `171210ee89e3944d1c606d50ad779a5dfb1651b8`, the timeout fix in switchify-prediction PR #22, and kept the v0.2.0 worker assets. Both libraries are now pinned to `2a2ed0c2b0013663ffa9317dba3389ec96b6f14b` with v0.2.1 worker assets, which add six-slot generation; the deadlines are unchanged. The earlier 500 ms measurements above remain historical evidence, not measurements of the increased deadline. +The companion inference and reset reply deadline is now 2 seconds, increased from 500 ms after repeated macOS CI timeouts. Startup remains bounded at 30 seconds, statistical suggestions remain immediate, and late or stale results remain rejected. That change pinned both Rust libraries to `171210ee89e3944d1c606d50ad779a5dfb1651b8`, the timeout fix in switchify-prediction PR #22, and kept the v0.2.0 worker assets. The v0.2.1 worker assets and their converter come from `2a2ed0c2b0013663ffa9317dba3389ec96b6f14b`, and they add six-slot generation. Both Rust libraries are now pinned to `e5223aa8af58d982278db110b70293546db1cdd7`, which adds incremental, bounded personal learning without changing the worker protocol. The deadlines are unchanged. The earlier 500 ms measurements above remain historical evidence, not measurements of the increased deadline. diff --git a/docs/qwerty-keyboard.md b/docs/qwerty-keyboard.md index 7129bcb1..338997c1 100644 --- a/docs/qwerty-keyboard.md +++ b/docs/qwerty-keyboard.md @@ -16,7 +16,7 @@ The bottom control row starts with Close keyboard, followed by Letters, Navigati Modifier keys are pressed only around each emitted shortcut and immediately released. Selecting a locked modifier does not hold that operating-system key while the scanner runs. Ordinary characters use text injection; command combinations and navigation use native key events. Native shortcuts retain the operating system's layout semantics. -The keyboard and the mouse panel do not depend on other windows. They open and keep working when no window is in front. When the foreground application changes the keyboard stays open and keeps its modifiers, Caps and highlight, and sends subsequent keys to the new foreground application. Only what belongs to the text of one window is let go: the suggestions and prediction context, a space or capital the keyboard owns, and any input still held. Suggestions are unavailable while no window is in front. When its display or work area changes, for example when a taskbar hides, the keyboard moves to fit. On macOS the work area leaves out the menu bar and Dock only while they show. In a full-screen app, or with either set to hide automatically, that is only while the pointer is at the screen edge. So on macOS the keyboard and mouse panel read the work area again only when the Space changes (entering or leaving a full-screen app, or choosing another desktop), following it for a second while it settles. They keep that area on the same display and Space, so they do not jump as the bars come and go. They stay above both and never take clicks. Moving the Dock or changing its size on the same Space takes effect when the Space next changes or scanning starts again. Moving to another display, or this display changing size, still fits them again; if its display is removed it moves to the display under the pointer, and while the displays cannot be read it stays where it is. The mouse panel does the same on the display under the pointer. It closes when its scan session ends. Failed input clears keyboard modifiers and requires Select to resume. Stop, disconnect and application exit use the shared deterministic input cleanup. Word prediction is enabled by default in Scanning settings. Six suggestion slots appear on the Letters page. Selecting a suggestion inserts its missing suffix and a space. Prediction uses local, read-only data; no personal vocabulary is saved. See [word prediction](word-prediction.md) for context availability and validation. +The keyboard and the mouse panel do not depend on other windows. They open and keep working when no window is in front. When the foreground application changes the keyboard stays open and keeps its modifiers, Caps and highlight, and sends subsequent keys to the new foreground application. Only what belongs to the text of one window is let go: the suggestions and prediction context, a space or capital the keyboard owns, and any input still held. Suggestions are unavailable while no window is in front. When its display or work area changes, for example when a taskbar hides, the keyboard moves to fit. On macOS the work area leaves out the menu bar and Dock only while they show. In a full-screen app, or with either set to hide automatically, that is only while the pointer is at the screen edge. So on macOS the keyboard and mouse panel read the work area again only when the Space changes (entering or leaving a full-screen app, or choosing another desktop), following it for a second while it settles. They keep that area on the same display and Space, so they do not jump as the bars come and go. They stay above both and never take clicks. Moving the Dock or changing its size on the same Space takes effect when the Space next changes or scanning starts again. Moving to another display, or this display changing size, still fits them again; if its display is removed it moves to the display under the pointer, and while the displays cannot be read it stays where it is. The mouse panel does the same on the display under the pointer. It closes when its scan session ends. Failed input clears keyboard modifiers and requires Select to resume. Stop, disconnect and application exit use the shared deterministic input cleanup. Word prediction is enabled by default in Scanning settings. Six suggestion slots appear on the Letters page. Selecting a suggestion inserts its missing suffix and a space. Prediction runs on this computer and learns from what the keyboard types unless Learn from my typing is off. See [word prediction](word-prediction.md) for context availability and validation. ## Manual validation diff --git a/docs/six-predictions.md b/docs/six-predictions.md index d895c76a..03aeaa5f 100644 --- a/docs/six-predictions.md +++ b/docs/six-predictions.md @@ -7,7 +7,7 @@ scan direction. The entire displayed row stays stable until scanning leaves it. The worker retains exact displayed/latest batch tokens and validates acceptance against the edit revision, generation, activity epoch and foreground target. Capitalization, prefix replacement and trailing spaces use the existing path. -No focused-field capture, personal learning or new settings are introduced. +No focused-field capture or new settings are introduced. Learning from typing, described in word prediction, changes only slots 1–3. The companion uses protocol 2. Model and tokenizer bytes are unchanged. Startup remains bounded to 30 seconds and each reply to two seconds. Failure leaves the @@ -16,7 +16,7 @@ instant slots available, with explicit retry or keyboard reopening required. ## Release assets The build downloads checksum-pinned v0.2.1 companion release archives from -GitHub. Both libraries and worker provenance are pinned to the release commit. +GitHub. Worker provenance is pinned to the release commit. The Rust libraries are pinned to a later commit that adds personal learning; see word prediction. Verified local caches work offline, and installed applications need no network access or model setup. Model, tokenizer and statistical database bytes remain unchanged. diff --git a/docs/word-prediction.md b/docs/word-prediction.md index d0ad5af0..fdd4205e 100644 --- a/docs/word-prediction.md +++ b/docs/word-prediction.md @@ -23,7 +23,7 @@ A passive observer records only an activity counter and timestamp, never externa ## On-device model -One Word prediction setting controls the English `en-aac-oanc-v1` SQLite database and the SmolLM2 model through the pinned `switchify-prediction` Rust libraries. The saved `enhancedWordPrediction` field remains compatible but does not choose an engine. Only the isolated worker opens the read-only baseline; there is no personal database, saved learning, import, or runtime download. +One Word prediction setting controls the English `en-aac-oanc-v1` SQLite database and the SmolLM2 model through the pinned `switchify-prediction` Rust libraries. The saved `enhancedWordPrediction` field remains compatible but does not choose an engine. Only the isolated worker opens the read-only baseline and the personal file described in [Learning from typing](#learning-from-typing); there is no import or runtime download. The worker ranks up to three words using interpolated trigram/bigram/unigram probabilities (0.6/0.3/0.1, renormalized when context is unavailable) and shows them immediately in slots 1–3. The SmolLM2 model then fills slots 4–6 with up to three different words; see [default neural-assisted prediction](neural-prediction.md). Suggestions begin with the first typed character and continue after a space as next-word suggestions. An untouched keyboard remains empty. The prefix and preceding text are supplied separately, and sentence punctuation resets language context. There are no generated phrases, only single words. A prefix the database does not know leaves slots 1–3 empty; the model may still offer words for slots 4–6. The vocabulary can contain names, disfluencies and inappropriate words; it is English-only and is not a clinical recommendation system. @@ -31,6 +31,22 @@ Loading is asynchronous and typing remains available. The existing badge disting The loader checks the pinned database SHA-256 before opening it. Missing, corrupt or incompatible resources fail closed. `npm run prediction-model` obtains the versioned upstream release, verifies its archive and individual file hashes, and installs only allowlisted resources after verification. Verified local files work without network access. Bundles carry the source manifests, attribution and separate corpus notices; see [model provenance](../src-tauri/resources/PROVENANCE.md). +## Learning from typing + +**Learn from my typing**, under Suggestions in the keyboard's scanning settings, is on by default. It belongs to this computer: it is saved with this computer's app settings, is not part of synced settings, and changing it does not change the synced settings schema. Words and phrases learned this way are suggested sooner in slots 1–3. The AI slots do not learn, but they exclude the instant words, so they offer different words as learned ones fill the first three. + +Learning uses only the prediction buffer, which is successful Switchify keyboard input, never existing, pasted or hardware-keyboard text. The worker gives each piece of text to the library once: + +- A sentence is learned once the next sentence begins, so a correction typed just after its punctuation is learned corrected. Backspacing into learned text learns again from the start of the corrected word, so the part of that word kept before the correction counts twice. +- Enter or Tab finishes the text, and everything typed so far is learned. +- When context is lost any other way, the completed words are learned but a word still in progress is not, because it may be unfinished. +- A stretch longer than the 512-character buffer is learned in long pieces before it is clipped. The latest 256 characters are held back so they can still be corrected. +- An unfinished sentence is not learned when the keyboard closes, since closing kills the worker that held it. + +The personal file is `prediction/personal.sqlite` in this computer's app data. It holds counts of words and of the one or two words before them. When predicting, one learned use counts as up to 500 uses in the built-in English data, less as more is learned, so learned words can never outgrow it. A word the built-in data lacks, or has seen fewer than three times, is suggested once it has been learned twice, so a word typed wrongly once is not. It is not a copy of the text, but it is unencrypted and can reveal what was typed, including names or passwords. Nothing learned is logged or synced. Learning a sentence takes tens of milliseconds in the worker, and up to about 200 ms once tens of thousands of words have been learned, well within its two-second reply deadline. A personal file that cannot be opened leaves prediction working without learning. + +Turning learning on or off replaces the worker at the next keyboard update, so a worker never learns against the setting. A suggestion being accepted at that moment fails, as it does when a worker misses its deadline. **Forget learned words** asks for confirmation, then records the request on disk, kills any spare worker and replaces the keyboard's worker in the same way. The file and its SQLite journals are deleted as soon as no worker holds them. Until then no worker opens the file. A deletion left unfinished when Switchify closed finishes when it next starts. The settings page says whether the words are already gone. + ## Casing The label on a suggestion always reads as the text will after it is accepted. Casing is decided in three steps. diff --git a/scripts/prediction-neural.json b/scripts/prediction-neural.json index fc73199f..19ff7a45 100644 --- a/scripts/prediction-neural.json +++ b/scripts/prediction-neural.json @@ -303,5 +303,5 @@ "url": "https://github.com/switchifyapp/switchify-prediction/releases/download/v0.2.1/switchify-prediction-neural-0.2.1-x86_64-unknown-linux-gnu.zip" } }, - "library_revision": "2a2ed0c2b0013663ffa9317dba3389ec96b6f14b" + "library_revision": "e5223aa8af58d982278db110b70293546db1cdd7" } diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 20f2e742..ea5163a7 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -4217,7 +4217,7 @@ dependencies = [ [[package]] name = "switchify-prediction" version = "0.2.1" -source = "git+https://github.com/switchifyapp/switchify-prediction?rev=2a2ed0c2b0013663ffa9317dba3389ec96b6f14b#2a2ed0c2b0013663ffa9317dba3389ec96b6f14b" +source = "git+https://github.com/switchifyapp/switchify-prediction?rev=e5223aa8af58d982278db110b70293546db1cdd7#e5223aa8af58d982278db110b70293546db1cdd7" dependencies = [ "clap", "rusqlite", @@ -4233,7 +4233,7 @@ dependencies = [ [[package]] name = "switchify-prediction-neural" version = "0.2.1" -source = "git+https://github.com/switchifyapp/switchify-prediction?rev=2a2ed0c2b0013663ffa9317dba3389ec96b6f14b#2a2ed0c2b0013663ffa9317dba3389ec96b6f14b" +source = "git+https://github.com/switchifyapp/switchify-prediction?rev=e5223aa8af58d982278db110b70293546db1cdd7#e5223aa8af58d982278db110b70293546db1cdd7" dependencies = [ "clap", "serde", diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index b5d0f6a7..921906cc 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -14,8 +14,8 @@ crate-type = ["staticlib", "cdylib", "rlib"] tauri-build = { version = "2", features = [] } [dependencies] -switchify-prediction = { git = "https://github.com/switchifyapp/switchify-prediction", rev = "2a2ed0c2b0013663ffa9317dba3389ec96b6f14b" } -switchify-prediction-neural = { git = "https://github.com/switchifyapp/switchify-prediction", rev = "2a2ed0c2b0013663ffa9317dba3389ec96b6f14b" } +switchify-prediction = { git = "https://github.com/switchifyapp/switchify-prediction", rev = "e5223aa8af58d982278db110b70293546db1cdd7" } +switchify-prediction-neural = { git = "https://github.com/switchifyapp/switchify-prediction", rev = "e5223aa8af58d982278db110b70293546db1cdd7" } anyhow = "1" base64 = "0.22" directories = "6.0.0" diff --git a/src-tauri/build.rs b/src-tauri/build.rs index 876a13e4..16ab2618 100644 --- a/src-tauri/build.rs +++ b/src-tauri/build.rs @@ -25,6 +25,7 @@ fn main() { "modifier_overlay_ready", "modifier_overlay_present", "forget_device", + "forget_learned_words", "save_settings", "set_telemetry_consent", "mark_setup_shown", diff --git a/src-tauri/capabilities/main.json b/src-tauri/capabilities/main.json index 47fca436..1599c07b 100644 --- a/src-tauri/capabilities/main.json +++ b/src-tauri/capabilities/main.json @@ -25,6 +25,7 @@ "allow-reject-pairing", "allow-disconnect-all", "allow-forget-device", + "allow-forget-learned-words", "allow-save-settings", "allow-set-telemetry-consent", "allow-mark-setup-shown", diff --git a/src-tauri/permissions/autogenerated/forget_learned_words.toml b/src-tauri/permissions/autogenerated/forget_learned_words.toml new file mode 100644 index 00000000..e40f3623 --- /dev/null +++ b/src-tauri/permissions/autogenerated/forget_learned_words.toml @@ -0,0 +1,11 @@ +# Automatically generated - DO NOT EDIT! + +[[permission]] +identifier = "allow-forget-learned-words" +description = "Enables the forget_learned_words command without any pre-configured scope." +commands.allow = ["forget_learned_words"] + +[[permission]] +identifier = "deny-forget-learned-words" +description = "Denies the forget_learned_words command without any pre-configured scope." +commands.deny = ["forget_learned_words"] diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index c56bb7a6..83b1a437 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -564,6 +564,7 @@ fn apply_app_settings(app: &AppHandle, settings: AppSettings) -> Result<(), Stri previous_dwell_delay, previous_mouse_repeat_enabled, previous_key_repeat_enabled, + previous_learn_word_prediction, ) = { let state = &model .shared @@ -578,6 +579,7 @@ fn apply_app_settings(app: &AppHandle, settings: AppSettings) -> Result<(), Stri state.settings.dwell_click_delay_ms, state.settings.mouse_repeat_enabled, state.settings.key_repeat_enabled, + state.settings.learn_word_prediction, ) }; if settings.start_with_system != previous_start_with_system { @@ -607,10 +609,20 @@ fn apply_app_settings(app: &AppHandle, settings: AppSettings) -> Result<(), Stri { platform_stop_mouse_repeat(app); } + if settings.learn_word_prediction != previous_learn_word_prediction { + prediction::learning_changed(); + } overlay.apply_settings(settings); Ok(()) } +/// Deletes the words prediction learned on this computer. Returns whether +/// they are gone already, rather than once prediction stops using them. +#[tauri::command] +fn forget_learned_words(app: AppHandle) -> Result { + prediction::forget_learned(&app) +} + #[tauri::command] fn set_telemetry_consent(model: State<'_, AppModel>, enabled: bool) -> Result { model.apply_telemetry_choice(enabled)?; @@ -1739,6 +1751,7 @@ pub fn run() { app.manage(switch_practice::Controller::default()); switch_runtime::install(app.handle()); remote_scan::install(app.handle()); + prediction::resume_forgetting(app.handle()); point_scan_runtime::install(app.handle()); settings_sync::install(app.handle()); install_tray(app)?; @@ -1831,6 +1844,7 @@ pub fn run() { } }) .invoke_handler(tauri::generate_handler![ + forget_learned_words, get_switches, begin_switch_practice, get_switch_practice, diff --git a/src-tauri/src/portable_settings.rs b/src-tauri/src/portable_settings.rs index cb80968c..ef359148 100644 --- a/src-tauri/src/portable_settings.rs +++ b/src-tauri/src/portable_settings.rs @@ -31,8 +31,9 @@ pub struct Document { } /// `AppSettings` without `start_with_system` (registered with this OS), -/// `share_diagnostics` (consent must be given on each install) and -/// `auto_approve_account_devices` (pairing is per install). +/// `share_diagnostics` (consent must be given on each install), +/// `auto_approve_account_devices` (pairing is per install) and +/// `learn_word_prediction` (learned words stay on this computer). #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(rename_all = "camelCase", deny_unknown_fields)] pub struct PortableAppSettings { @@ -85,6 +86,7 @@ impl PortableAppSettings { start_with_system: local.start_with_system, share_diagnostics: local.share_diagnostics, auto_approve_account_devices: local.auto_approve_account_devices, + learn_word_prediction: local.learn_word_prediction, pointer_scale_percent: self.pointer_scale_percent, mouse_repeat_enabled: self.mouse_repeat_enabled, move_repeat_interval_ms: self.move_repeat_interval_ms, @@ -521,6 +523,7 @@ mod tests { start_with_system: true, share_diagnostics: true, auto_approve_account_devices: false, + learn_word_prediction: false, ..AppSettings::default() }, profiles, @@ -552,6 +555,7 @@ mod tests { "startWithSystem", "shareDiagnostics", "autoApproveAccountDevices", + "learnWordPrediction", "telemetry", "desktopId", "pairedDevices", @@ -745,6 +749,7 @@ mod tests { assert!(settings.start_with_system); assert!(settings.share_diagnostics); assert!(!settings.auto_approve_account_devices); + assert!(!settings.learn_word_prediction); assert!(plan.switches.is_none() && plan.point_scan.is_none() && plan.remote.is_none()); } diff --git a/src-tauri/src/prediction/database.rs b/src-tauri/src/prediction/database.rs index af46eb68..3f7d413b 100644 --- a/src-tauri/src/prediction/database.rs +++ b/src-tauri/src/prediction/database.rs @@ -42,11 +42,12 @@ impl Database { _ => None, } } - pub fn open(model: &Path) -> Self { + pub fn open(model: &Path, personal: Option<&Path>) -> Self { let model = model.to_path_buf(); + let personal = personal.map(Path::to_path_buf); let (tx, rx) = mpsc::sync_channel(1); std::thread::spawn(move || { - let _ = tx.send(Model::open(&model)); + let _ = tx.send(Model::open(&model, personal.as_deref())); }); Self { state: State::Loading(rx), @@ -83,6 +84,12 @@ impl Database { model.reset(); } } + /// Text completed before the model is ready is not learned. + pub fn learn(&mut self, text: &str) { + if let State::Ready(model) = &mut self.state { + model.learn(text); + } + } pub fn predict(&mut self, context: &Context) -> (Status, Prediction) { let status = self.status(); if status != Status::Ready || context.partial { @@ -189,7 +196,7 @@ mod tests { ); let missing = std::env::temp_dir().join(format!("switchify-missing-model-{}", std::process::id())); - let mut db = Database::open(&missing); + let mut db = Database::open(&missing, None); let end = Instant::now() + Duration::from_secs(5); while db.status() == Status::Loading && Instant::now() < end { std::thread::sleep(Duration::from_millis(10)); diff --git a/src-tauri/src/prediction/mod.rs b/src-tauri/src/prediction/mod.rs index 164fdf95..9d8690bd 100644 --- a/src-tauri/src/prediction/mod.rs +++ b/src-tauri/src/prediction/mod.rs @@ -65,12 +65,13 @@ struct Client { _job: Job, } impl Client { - fn start(path: &Path, ignored: Vec) -> Result { + fn start(path: &Path, personal: Option<&Path>, ignored: Vec) -> Result { let mut command = Command::new(std::env::current_exe().map_err(|_| ())?); command .arg(worker::ARG) .arg(path) .arg(serde_json::to_string(&ignored).map_err(|_| ())?) + .arg(personal.unwrap_or(Path::new(""))) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::null()); @@ -161,20 +162,129 @@ const SPARE_IDLE: Duration = Duration::from_secs(120); struct Spare { client: Client, ignored: Vec, + personal: Option, + /// `LEARNING` when it started. + learning: u64, since: Instant, } impl Spare { - fn usable(&mut self, ignored: &[u32], now: Instant) -> bool { + fn usable(&mut self, ignored: &[u32], personal: Option<&Path>, now: Instant) -> bool { self.ignored == ignored + && self.personal.as_deref() == personal && now.duration_since(self.since) < SPARE_IDLE && matches!(self.client.child.try_wait(), Ok(None)) } } /// The spare worker for this keyboard open, if one is still usable. Any /// other spare is killed and reaped. -fn adopt(spare: &mut Option, ignored: &[u32], now: Instant) -> Option { +fn adopt( + spare: &mut Option, + ignored: &[u32], + personal: Option<&Path>, + now: Instant, +) -> Option { let mut spare = spare.take()?; - spare.usable(ignored, now).then_some(spare.client) + spare.usable(ignored, personal, now).then_some(spare.client) +} + +thread_local! { + /// Bumped when learning is switched on or off or learned words are + /// forgotten. Workers started before then are replaced. Settings are + /// saved, and workers run, on the main thread. + static LEARNING: std::cell::Cell = const { std::cell::Cell::new(0) }; +} +fn learning_generation() -> u64 { + LEARNING.with(std::cell::Cell::get) +} +thread_local! { + /// Learned words waiting to be deleted once no worker holds them. Like + /// the workers it waits for, it belongs to the main thread. + static FORGETTING: RefCell> = const { RefCell::new(None) }; +} +fn forgetting() -> bool { + FORGETTING.with(|f| f.borrow().is_some()) +} + +fn personal_file(app: &AppHandle) -> Option { + Some( + app.path() + .app_config_dir() + .ok()? + .join("prediction/personal.sqlite"), + ) +} +/// Present while a request to forget is unfinished, so a restart finishes it +/// before any worker opens the file. +fn forget_marker(path: &Path) -> PathBuf { + let mut marker = path.as_os_str().to_owned(); + marker.push(".forget"); + PathBuf::from(marker) +} +/// Removes the personal file and its SQLite journals. Missing files are +/// already gone. +fn delete_personal(path: &Path) -> bool { + ["", "-journal", "-wal", "-shm"].iter().all(|suffix| { + let mut file = path.as_os_str().to_owned(); + file.push(suffix); + match std::fs::remove_file(PathBuf::from(file)) { + Ok(()) => true, + Err(error) => error.kind() == std::io::ErrorKind::NotFound, + } + }) +} +/// Learning was switched on or off: the next keyboard starts a worker that +/// matches. +pub fn learning_changed() { + LEARNING.with(|l| l.set(l.get().wrapping_add(1))); +} +/// Forget everything learned. The request is recorded on disk first. A spare +/// worker is killed now; a keyboard's worker is replaced at its next update, +/// so neither writes again. Returns whether the words are already deleted; +/// otherwise they are as soon as no worker holds them, which a restart also +/// finishes before any worker opens the file. +/// +/// It must run on the main thread, where prediction updates run, so it must +/// stay a synchronous command. +pub fn forget_learned(app: &AppHandle) -> Result { + forget_at(personal_file(app).ok_or("The learned words could not be found.")?) +} +fn forget_at(path: PathBuf) -> Result { + let unrecorded = "The request to forget learned words could not be saved."; + std::fs::create_dir_all(path.parent().ok_or(unrecorded)?).map_err(|_| unrecorded)?; + std::fs::write(forget_marker(&path), b"").map_err(|_| unrecorded)?; + pend_forgetting(path); + learning_changed(); + SPARE.with(|s| *s.borrow_mut() = None); + // The keyboard's worker may hold the file even where deleting an open + // file succeeds, and could still write to it; it goes at its next update. + let held = SERVICE.with(|s| { + let s = s.borrow(); + s.client.is_some() && s.personal.is_some() + }); + Ok(!held && finish_forgetting()) +} +/// Finishes a request to forget left by an earlier run. Nothing holds the +/// file at startup. +pub fn resume_forgetting(app: &AppHandle) { + if let Some(file) = personal_file(app).filter(|f| forget_marker(f).exists()) { + pend_forgetting(file); + finish_forgetting(); + } +} +fn pend_forgetting(path: PathBuf) { + FORGETTING.with(|f| *f.borrow_mut() = Some(path)); +} +/// Deletes forgotten learned words, and then the record of the request. +/// Called whenever no worker may hold them. Returns whether none are pending. +fn finish_forgetting() -> bool { + let Some(path) = FORGETTING.with(|f| f.borrow().clone()) else { + return true; + }; + let done = delete_personal(&path) && delete_personal(&forget_marker(&path)); + if done { + FORGETTING.with(|f| *f.borrow_mut() = None); + } + done } fn expire(spare: &mut Option, now: Instant) { if spare @@ -208,6 +318,8 @@ struct Service { known: usize, /// What the worker was started with, to start its replacement. model: Option, + personal: Option, + learning: u64, ignored: Vec, failed: bool, generation: u64, @@ -284,7 +396,7 @@ impl Service { self.beyond = true; } } - Edit::Reset => {} + Edit::Reset | Edit::End => {} } } fn forget(&mut self) { @@ -315,6 +427,15 @@ impl Service { self.held = false; self.forget(); } + /// Enter or Tab: the text so far is finished, so its last word is + /// complete, and typing continues on a new line or field. + fn finish(&mut self, scope: InputScope) { + self.shape.clear(); + self.beyond = false; + self.held = false; + self.known = 0; + self.queue_edit(Edit::End, scope); + } /// The context is lost and the caret has not moved. Inside a word, /// suggestions wait for the next one. fn interrupt(&mut self) { @@ -345,7 +466,7 @@ impl Service { match &edit { Edit::Append(text) => self.known = (self.known + text.chars().count()).min(512), Edit::Backspace => self.known -= 1, - Edit::Reset => {} + Edit::Reset | Edit::End => {} } self.queue_edit(edit, scope); } @@ -382,7 +503,7 @@ impl Service { match edit { Some(edit) => self.typed(edit, scope), // Enter and Tab start a new line or field. - None if matches!(stroke.key, Key::Named("Enter" | "Tab")) => self.moved(), + None if matches!(stroke.key, Key::Named("Enter" | "Tab")) => self.finish(scope), None => self.displaced(true), } } @@ -537,6 +658,7 @@ pub fn keyboard_input_context() -> Option { pub fn stop() { SERVICE.with(|s| *s.borrow_mut() = Service::default()); SPARE.with(|s| *s.borrow_mut() = None); + finish_forgetting(); } /// The keyboard closed. The worker and its private text context are killed /// and reaped as in `stop`, then a fresh worker loads the model for the next @@ -544,27 +666,51 @@ pub fn stop() { pub fn close() { close_with(Client::start); } -fn close_with(start: impl FnOnce(&Path, Vec) -> Result) { +fn close_with(start: impl FnOnce(&Path, Option<&Path>, Vec) -> Result) { let service = SERVICE.with(|s| std::mem::take(&mut *s.borrow_mut())); let Service { client, model, + personal, + learning, ignored, .. } = service; let worked = client.is_some(); drop(client); + // A spare never reopens words being forgotten, or learns against a + // changed setting. + let current = learning_generation(); + let personal = personal.filter(|_| learning == current && !forgetting()); + let learning = current; SPARE.with(|s| { let mut spare = s.borrow_mut(); if let (true, Some(model)) = (worked, model) { - *spare = start(&model, ignored.clone()).ok().map(|client| Spare { - client, - ignored, - since: Instant::now(), - }); + *spare = start(&model, personal.as_deref(), ignored.clone()) + .ok() + .map(|client| Spare { + client, + ignored, + personal, + learning, + since: Instant::now(), + }); } expire(&mut spare, Instant::now()); }); + finish_forgetting(); +} +/// Replaces a worker started before learning changed. A suggestion being +/// accepted fails, as when a worker misses its deadline, so the keyboard +/// does not wait on the old worker. +fn replace_stale(keyboard: &mut Keyboard) { + SERVICE.with(|s| { + let s = s.borrow(); + if s.accepting || s.accept.is_some() { + keyboard.failed(); + } + }); + retry(); } /// Restart only prediction. The keyboard and its scan session remain open, /// while the old worker and its private text context are discarded. @@ -666,6 +812,24 @@ pub fn poll(app: &AppHandle, keyboard: Option<&mut Keyboard>, enabled: bool, ign close(); return; }; + // A worker started before learning changed may hold the personal file. + let learning = learning_generation(); + let stale = SERVICE.with(|s| { + let s = s.borrow(); + s.client.is_some() && s.learning != learning + }); + if stale { + replace_stale(keyboard); + } + SPARE.with(|s| { + let mut spare = s.borrow_mut(); + if spare + .as_ref() + .is_some_and(|spare| spare.learning != learning) + { + *spare = None; + } + }); SERVICE.with(|slot| { let mut s = slot.borrow_mut(); s.observe(activity::snapshot()); @@ -694,10 +858,44 @@ pub fn poll(app: &AppHandle, keyboard: Option<&mut Keyboard>, enabled: bool, ign ignored.sort_unstable(); ignored.dedup(); let model = resource(app).ok(); + let file = personal_file(app); + // A request to forget left unfinished, even before a restart. + if let Some(file) = file.as_ref().filter(|f| forget_marker(f).exists()) { + pend_forgetting(file.clone()); + } + // No worker runs here: the spare is adopted or replaced below. + SPARE.with(|spare| { + let mut spare = spare.borrow_mut(); + if forgetting() { + *spare = None; + } + }); + finish_forgetting(); + let learn = app + .state::() + .snapshot() + .settings + .learn_word_prediction; + // Learned words still waiting to be deleted are not reopened. + let personal = file.filter(|_| learn && !forgetting()).filter(|file| { + file.parent() + .is_some_and(|dir| std::fs::create_dir_all(dir).is_ok()) + }); s.client = SPARE - .with(|spare| adopt(&mut spare.borrow_mut(), &ignored, Instant::now())) - .or_else(|| Client::start(model.as_deref()?, ignored.clone()).ok()); + .with(|spare| { + adopt( + &mut spare.borrow_mut(), + &ignored, + personal.as_deref(), + Instant::now(), + ) + }) + .or_else(|| { + Client::start(model.as_deref()?, personal.as_deref(), ignored.clone()).ok() + }); s.model = model; + s.personal = personal; + s.learning = learning; s.ignored = ignored; if s.client.is_none() { s.failed = true; @@ -1061,6 +1259,7 @@ mod tests { Edit::Append(text) => text, Edit::Backspace => "<".into(), Edit::Reset => "|".into(), + Edit::End => "$".into(), }) .collect() } @@ -1185,10 +1384,12 @@ mod tests { let mut s = service("hel"); s.stroke(key('l'), true, false, scope); assert!(!s.held && s.reset && !s.in_word()); + // A new line or field finishes the text, so its last word is complete. for name in ["Enter", "Tab"] { let mut s = service("hel"); s.stroke(named(name), true, true, scope); - assert!(!s.held && s.reset); + assert!(!s.held && !s.in_word() && s.known == 0); + assert!(matches!(s.edit.last().map(|e| &e.edit), Some(Edit::End))); } let mut s = service("hel"); s.observe((4, true)); @@ -1343,6 +1544,8 @@ mod tests { Some(Spare { client: sleeper(), ignored: ignored.to_vec(), + personal: None, + learning: 0, since, }) } @@ -1351,7 +1554,7 @@ mod tests { let now = Instant::now(); let mut slot = spare(&[32], now); let id = slot.as_ref().unwrap().client.child.id(); - let mut client = adopt(&mut slot, &[32], now + Duration::from_secs(5)).unwrap(); + let mut client = adopt(&mut slot, &[32], None, now + Duration::from_secs(5)).unwrap(); assert!(slot.is_none()); assert_eq!(client.child.id(), id); assert!(client.child.try_wait().unwrap().is_none()); @@ -1361,18 +1564,21 @@ mod tests { let now = Instant::now(); // Started for other switch keys. let mut slot = spare(&[32], now); - assert!(adopt(&mut slot, &[13], now).is_none()); + assert!(adopt(&mut slot, &[13], None, now).is_none()); assert!(slot.is_none()); // Waited too long. let mut slot = spare(&[32], now); - assert!(adopt(&mut slot, &[32], now + SPARE_IDLE).is_none()); + assert!(adopt(&mut slot, &[32], None, now + SPARE_IDLE).is_none()); assert!(slot.is_none()); // Already exited. let mut slot = spare(&[32], now); let child = &mut slot.as_mut().unwrap().client.child; child.kill().unwrap(); child.wait().unwrap(); - assert!(adopt(&mut slot, &[32], now).is_none()); + assert!(adopt(&mut slot, &[32], None, now).is_none()); + // Started without learning, or learning into another file. + let mut slot = spare(&[32], now); + assert!(adopt(&mut slot, &[32], Some(Path::new("personal")), now).is_none()); } #[test] fn an_idle_spare_worker_expires() { @@ -1413,12 +1619,14 @@ mod tests { *slot.borrow_mut() = Service { client: Some(worker), model: Some(PathBuf::from("model")), + personal: Some(PathBuf::from("personal")), ignored: vec![13, 32], edit: vec![InputScope::capture().record(Edit::Append("test".into()))], ..Default::default() }; }); - close_with(|model, ignored| { + close_with(|model, personal, ignored| { + assert_eq!(personal, Some(Path::new("personal"))); assert!( !alive(held), "the old worker is gone before the spare starts" @@ -1437,6 +1645,7 @@ mod tests { let spare = slot.as_mut().unwrap(); assert_ne!(spare.client.child.id(), held); assert_eq!(spare.ignored, [13, 32]); + assert_eq!(spare.personal.as_deref(), Some(Path::new("personal"))); assert!(spare.client.child.try_wait().unwrap().is_none()); }); // A spare that could not start is not retried. @@ -1445,9 +1654,9 @@ mod tests { s.client = Some(sleeper()); s.model = Some(PathBuf::from("model")); }); - close_with(|_, _| Err(())); + close_with(|_, _, _| Err(())); SPARE.with(|slot| assert!(slot.borrow().is_none())); - close_with(|_, _| panic!("no worker closed, so none is started")); + close_with(|_, _, _| panic!("no worker closed, so none is started")); stop(); } #[test] @@ -1478,4 +1687,114 @@ mod tests { ); assert!(worker::receive::(&mut std::io::Cursor::new([0u8; 4])).is_err()); } + + #[test] + fn replacing_a_stale_worker_fails_a_suggestion_being_accepted() { + for accepting in [false, true] { + stop(); + SERVICE.with(|slot| { + *slot.borrow_mut() = Service { + client: None, + accept: accepting.then_some((1, 0)), + ..Default::default() + }; + }); + let mut keyboard = Keyboard::new(false); + replace_stale(&mut keyboard); + assert_eq!(keyboard.error, accepting); + // An acceptance whose reply is awaited fails too. + SERVICE.with(|slot| { + *slot.borrow_mut() = Service { + client: None, + accepting, + ..Default::default() + }; + }); + let mut keyboard = Keyboard::new(false); + replace_stale(&mut keyboard); + assert_eq!(keyboard.error, accepting); + SERVICE.with(|slot| { + let s = slot.borrow(); + assert!(s.client.is_none() && s.accept.is_none() && !s.accepting); + }); + } + stop(); + } + #[test] + fn a_spare_started_while_forgetting_does_not_open_the_personal_file() { + stop(); + let dir = std::env::temp_dir().join(format!("switchify-spare-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("personal.sqlite"); + std::fs::write(forget_marker(&path), b"").unwrap(); + pend_forgetting(path.clone()); + SERVICE.with(|slot| { + *slot.borrow_mut() = Service { + client: Some(sleeper()), + model: Some(PathBuf::from("model")), + personal: Some(path.clone()), + learning: learning_generation(), + ..Default::default() + }; + }); + close_with(|_, personal, _| { + assert_eq!(personal, None); + Err(()) + }); + // With the worker gone, the forgetting finished, marker included. + assert!(!forgetting()); + assert!(!forget_marker(&path).exists()); + stop(); + std::fs::remove_dir_all(&dir).unwrap(); + } + #[test] + fn forgetting_waits_for_the_keyboard_worker_that_holds_the_file() { + stop(); + let dir = std::env::temp_dir().join(format!("switchify-held-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("personal.sqlite"); + std::fs::write(&path, "counts").unwrap(); + SERVICE.with(|slot| { + *slot.borrow_mut() = Service { + client: Some(sleeper()), + personal: Some(path.clone()), + ..Default::default() + }; + }); + SPARE.with(|slot| *slot.borrow_mut() = spare(&[], Instant::now())); + // Not deleted while the keyboard's worker could still write to it, + // even where deleting an open file would succeed. + assert!(!forget_at(path.clone()).unwrap()); + assert!(path.exists() && forget_marker(&path).exists()); + SPARE.with(|slot| assert!(slot.borrow().is_none(), "the spare is killed at once")); + // The worker is replaced at the next update; then the words go. + stop(); + assert!(!path.exists() && !forget_marker(&path).exists()); + // With no worker holding the file, it goes at once. + std::fs::write(&path, "counts").unwrap(); + assert!(forget_at(path.clone()).unwrap()); + assert!(!path.exists()); + std::fs::remove_dir_all(&dir).unwrap(); + } + #[test] + fn forgetting_deletes_the_personal_file_and_journals_once_nothing_holds_them() { + let dir = std::env::temp_dir().join(format!("switchify-forget-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("personal.sqlite"); + let files = ["", "-journal", "-wal", ".forget"].map(|suffix| { + let file = PathBuf::from(format!("{}{suffix}", path.display())); + std::fs::write(&file, "counts").unwrap(); + file + }); + // Nothing is deleted unless forgetting was asked for. + assert!(finish_forgetting()); + assert!(files.iter().all(|f| f.exists())); + pend_forgetting(path.clone()); + stop(); + assert!(files.iter().all(|f| !f.exists())); + assert!(!forgetting()); + // Already gone is done. + assert!(delete_personal(&path)); + std::fs::remove_dir_all(&dir).unwrap(); + } } diff --git a/src-tauri/src/prediction/model.rs b/src-tauri/src/prediction/model.rs index 1bb1fbc7..fd92e78f 100644 --- a/src-tauri/src/prediction/model.rs +++ b/src-tauri/src/prediction/model.rs @@ -1,4 +1,5 @@ -//! Read-only English word prediction. Only the isolated worker holds context. +//! English word prediction, with optional personal learning. Only the +//! isolated worker holds context. use sha2::{Digest, Sha256}; use std::{fs::File, io::Read, path::Path}; use switchify_prediction::{Options, Predictor}; @@ -29,6 +30,8 @@ pub trait Predict: Send { false } fn reset(&mut self) {} + /// Learn one completed, non-overlapping piece of typed text. + fn learn(&mut self, _text: &str) {} } pub struct Model { @@ -37,10 +40,14 @@ pub struct Model { request: Option, instant: [Option; 3], config: Option, + learning: bool, } impl Model { - pub fn open(path: &Path) -> Result { + /// Opens the verified baseline, and the personal file when given. A + /// personal file that cannot be opened leaves prediction working without + /// learning. + pub fn open(path: &Path, personal: Option<&Path>) -> Result { let mut file = File::open(path).map_err(|_| ())?; let mut hash = Sha256::new(); let mut buffer = [0; 65536]; @@ -54,7 +61,10 @@ impl Model { if format!("{:x}", hash.finalize()) != DATABASE_SHA256 { return Err(()); } - let predictor = Predictor::open(path, None).map_err(|_| ())?; + let (predictor, learning) = match personal.map(|p| Predictor::open(path, Some(p))) { + Some(Ok(predictor)) => (predictor, true), + _ => (Predictor::open(path, None).map_err(|_| ())?, false), + }; let config = (|| { let bundle = path.parent()?.parent()?.join("prediction-neural"); let executable = std::env::current_exe().ok()?; @@ -103,6 +113,7 @@ impl Model { config, request: None, instant: Default::default(), + learning, }) } } @@ -177,6 +188,12 @@ impl Predict for Model { n.reset(); } } + fn learn(&mut self, text: &str) { + // A failed write keeps the previous snapshot; prediction carries on. + if self.learning { + let _ = self.predictor.learn(text); + } + } } #[cfg(test)] @@ -198,6 +215,41 @@ mod tests { let _ = fs::remove_dir_all(&self.0); } } + #[test] + fn learning_writes_only_the_personal_file_and_ranks_learned_words() { + let dir = Fixture::new(); + let path = dir.0.join("english.sqlite"); + let personal = dir.0.join("personal.sqlite"); + switchify_prediction::build( + &path, + "I need help. I need help. I drink water.", + "synthetic test", + ) + .unwrap(); + let original = fs::read(&path).unwrap(); + let open = |learning| Model { + predictor: Predictor::open(&path, Some(&personal)).unwrap(), + neural: None, + config: None, + request: None, + instant: Default::default(), + learning, + }; + let first = |model: &mut Model| model.predict("I need ", "h").unwrap().words[0].clone(); + let mut model = open(false); + model.learn("I need hummus. I need hummus."); + assert_eq!(first(&mut model).as_deref(), Some("help")); + let mut model = open(true); + // A word the baseline lacks counts once learned twice. + model.learn("I need hummus."); + assert_eq!(first(&mut model).as_deref(), Some("help")); + model.learn("I need hummus."); + assert_eq!(first(&mut model).as_deref(), Some("hummus")); + // Learning persists for the next worker. + assert_eq!(first(&mut open(true)).as_deref(), Some("hummus")); + assert_eq!(fs::read(&path).unwrap(), original); + } + #[test] fn sqlite_adapter_ranks_context_and_preserves_read_only_baseline() { let dir = Fixture::new(); @@ -211,6 +263,7 @@ mod tests { config: None, request: None, instant: Default::default(), + learning: false, }; model.config = Some(switchify_prediction_neural::Config { bundle: dir.0.join("missing"), @@ -269,10 +322,10 @@ mod tests { assert_eq!(fs::read(&path).unwrap(), original); assert_eq!(fs::read_dir(&dir.0).unwrap().count(), 1); // A structurally valid custom database is not the pinned production model. - assert!(Model::open(&path).is_err()); + assert!(Model::open(&path, None).is_err()); fs::write(&path, "corrupt").unwrap(); - assert!(Model::open(&path).is_err()); - assert!(Model::open(&dir.0.join("missing")).is_err()); + assert!(Model::open(&path, None).is_err()); + assert!(Model::open(&dir.0.join("missing"), None).is_err()); } fn rss_bytes() -> u64 { @@ -315,7 +368,7 @@ mod tests { }); let before = rss_bytes(); let start = Instant::now(); - let mut model = Model::open(&path).unwrap(); + let mut model = Model::open(&path, None).unwrap(); let cold_ms = start.elapsed().as_secs_f64() * 1000.0; let loaded = rss_bytes(); let mut samples = Vec::new(); diff --git a/src-tauri/src/prediction/worker.rs b/src-tauri/src/prediction/worker.rs index 755b6a46..94d80194 100644 --- a/src-tauri/src/prediction/worker.rs +++ b/src-tauri/src/prediction/worker.rs @@ -25,8 +25,32 @@ pub enum Edit { Append(String), Backspace, Reset, + /// Like `Reset`, after text that is finished, such as before Enter or + /// Tab: its last word is complete. + End, } +fn sentence_end(c: char) -> bool { + matches!(c, '.' | '!' | '?' | '\n' | '\r') +} +/// Letters, digits and apostrophes continue a word; anything else ends it. +/// Whole graphemes are tested, so a combining accent stays with its letter. +fn word_grapheme(g: &str) -> bool { + g.chars() + .next() + .is_some_and(|c| c.is_alphanumeric() || c == '\'' || c == '\u{2019}') +} +/// The end of the last grapheme in `text` that ends a word, if any. +fn last_boundary(text: &str) -> Option { + text.grapheme_indices(true) + .rev() + .find(|(_, g)| !word_grapheme(g)) + .map(|(i, g)| i + g.len()) +} +/// Unlearned text is kept back this many characters from the end of a full +/// buffer, so the latest words can still be corrected before they are learned. +const RECENT: usize = 256; + // Each edit is scoped before injection begins. Queued edits from before an // external activity or window change cannot become context for the new target. #[derive(Clone, Serialize, Deserialize)] @@ -117,6 +141,9 @@ pub struct Engine { activity: u64, tracked: bool, buffer: String, + /// Bytes at the start of `buffer` already learned. Learning sees each + /// piece of typed text once. + learned: usize, clipped: bool, revision: u64, snapshot: Option, @@ -140,6 +167,7 @@ impl Engine { target: None, activity: 0, buffer: String::new(), + learned: 0, clipped: false, revision: 0, snapshot: None, @@ -151,7 +179,63 @@ impl Engine { generation: 0, } } + /// Learn `buffer[learned..end]`. + fn learn_to(&mut self, end: usize) { + if end > self.learned { + let text = &self.buffer[self.learned..end]; + if text.chars().any(char::is_alphanumeric) { + self.database.learn(text); + } + self.learned = end; + } + } + /// Learn the sentences that ended before the latest word began. Waiting + /// for the next word leaves a correction just after the punctuation + /// unlearned. + fn learn_sentences(&mut self) { + let rest = &self.buffer[self.learned..]; + let Some(word) = rest.rfind(char::is_alphanumeric) else { + return; + }; + if let Some(end) = rest[..word].rfind(sentence_end) { + self.learn_to(self.learned + end + 1); + } + } + /// Learn what is typed before the context is lost. Unless the text is + /// finished, a word still in progress may be incomplete and is left out. + fn learn_words(&mut self, finished: bool) { + let end = if finished { + self.buffer.len() + } else { + match last_boundary(&self.buffer[self.learned..]) { + Some(end) => self.learned + end, + None => return, + } + }; + self.learn_to(end); + } + /// Before clipping drops text that is not learned yet, learn through the + /// last word boundary at least `RECENT` characters from the end. A long + /// stretch without punctuation is learned in long pieces that keep their + /// phrases, and the latest words can still be corrected. + fn learn_before_clipping(&mut self) { + let total = self.buffer.chars().count(); + let dropped = total.saturating_sub(512); + if self.buffer[..self.learned].chars().count() >= dropped { + return; + } + let keep = self + .buffer + .char_indices() + .nth(total.saturating_sub(RECENT)) + .map_or(self.buffer.len(), |(i, _)| i); + if let Some(end) = last_boundary(&self.buffer[self.learned..keep]) { + self.learn_to(self.learned + end); + } + } fn clear(&mut self) { + self.learn_words(false); + self.learned = 0; self.database.reset(); self.displayed = None; self.buffer.clear(); @@ -166,21 +250,35 @@ impl Engine { fn apply_edit(&mut self, edit: Edit) { match edit { Edit::Reset => self.clear(), + Edit::End => { + self.learn_words(true); + self.clear(); + } Edit::Append(text) => { if text.chars().count() > 512 { self.clear(); return; } self.buffer.push_str(&text); + if self.buffer.chars().count() > 512 { + self.learn_before_clipping(); + } while self.buffer.chars().count() > 512 { let n = self.buffer.graphemes(true).next().unwrap().len(); self.buffer.drain(..n); + self.learned = self.learned.saturating_sub(n); self.clipped = true; } + self.learn_sentences(); } Edit::Backspace => { if let Some((start, _)) = self.buffer.grapheme_indices(true).next_back() { self.buffer.truncate(start); + if self.learned > start { + // A learned word is being corrected: learn it again + // whole rather than its corrected ending alone. + self.learned = last_boundary(&self.buffer).unwrap_or(0); + } } else { self.clear(); } @@ -495,10 +593,12 @@ pub fn run_from_args() -> bool { return false; } let work = || -> Result<(), ()> { - if args.len() != 4 { + if args.len() != 5 { return Err(()); } let path = PathBuf::from(&args[2]); + // Empty when learning is off. + let personal = Some(PathBuf::from(&args[4])).filter(|p| !p.as_os_str().is_empty()); let ignored: Vec = serde_json::from_str(&args[3].to_string_lossy()).map_err(|_| ())?; if ignored.len() > 129 { return Err(()); @@ -522,7 +622,7 @@ pub fn run_from_args() -> bool { activity::set_ignored(ignored); #[cfg(any(target_os = "windows", target_os = "macos"))] { - let database = Database::open(&path); + let database = Database::open(&path, personal.as_deref()); let mut engine = Engine::new(database, false); serve( &mut engine, @@ -640,7 +740,7 @@ mod tests { // reporting every failed attempt, including failures during warmup. for session in 0..5 { let mut e = engine(); - e.database = Database::open(&path); + e.database = Database::open(&path, None); let cold = Instant::now(); while e.database.status() == Status::Loading { assert!(cold.elapsed() < Duration::from_secs(30)); @@ -1338,4 +1438,121 @@ mod tests { }; assert!(send(&mut Vec::new(), &request).is_err()); } + + struct Learner(std::sync::Arc>>); + impl super::super::model::Predict for Learner { + fn predict(&mut self, _: &str, _: &str) -> Result { + Ok(Default::default()) + } + fn learn(&mut self, text: &str) { + self.0.lock().unwrap().push(text.to_owned()); + } + } + /// An engine whose model records what it is asked to learn. + fn learner() -> (Engine, std::sync::Arc>>) { + let learned = std::sync::Arc::default(); + let mut e = engine(); + e.database = Database::with_predictor(Box::new(Learner(std::sync::Arc::clone(&learned)))); + (e, learned) + } + /// Types each character, then the edits, one query at a time. + fn typing(e: &mut Engine, revision: &mut u64, text: &str, then: &[Edit]) { + let edits = text + .chars() + .map(|c| Edit::Append(c.to_string())) + .chain(then.iter().cloned()); + for edit in edits { + *revision += 1; + e.query( + vec![super::tests::edit(edit)], + *revision, + Shift::Off, + false, + false, + ); + } + } + fn learned(log: &std::sync::Arc>>) -> Vec { + std::mem::take(&mut *log.lock().unwrap()) + } + + #[test] + fn a_sentence_is_learned_once_when_the_next_one_begins() { + let (mut e, log) = learner(); + let mut revision = 0; + typing(&mut e, &mut revision, "I need hummus. ", &[]); + assert!(learned(&log).is_empty(), "a correction is still possible"); + typing(&mut e, &mut revision, "See you", &[]); + assert_eq!(learned(&log), ["I need hummus."]); + typing(&mut e, &mut revision, " soon", &[Edit::End]); + assert_eq!( + learned(&log), + [" See you soon"], + "Enter finishes the last word" + ); + typing(&mut e, &mut revision, "ok", &[Edit::End]); + assert_eq!(learned(&log), ["ok"]); + } + + #[test] + fn a_correction_after_the_punctuation_is_learned_corrected() { + let (mut e, log) = learner(); + let mut revision = 0; + typing( + &mut e, + &mut revision, + "Helo.", + &[Edit::Backspace, Edit::Backspace], + ); + typing(&mut e, &mut revision, "lo. Ok", &[]); + assert_eq!(learned(&log), ["Hello."]); + // Deleting into learned text learns the corrected word whole. + typing(&mut e, &mut revision, "", &vec![Edit::Backspace; 5]); + assert_eq!(e.buffer, "Hell"); + typing(&mut e, &mut revision, "o! Bye", &[]); + assert_eq!(learned(&log), ["Hello!"]); + } + + #[test] + fn lost_context_learns_completed_words_but_not_the_one_in_progress() { + let (mut e, log) = learner(); + let mut revision = 0; + typing(&mut e, &mut revision, "call Siob", &[Edit::Reset]); + assert_eq!(learned(&log), ["call "]); + typing(&mut e, &mut revision, "...", &[Edit::Reset]); + assert!(learned(&log).is_empty(), "no words, nothing to learn"); + } + + #[test] + fn text_clipped_from_the_buffer_is_learned_in_long_whole_pieces() { + let (mut e, log) = learner(); + let mut revision = 0; + for _ in 0..240 { + typing(&mut e, &mut revision, "abcd ", &[]); + } + // Long pieces keep their phrases; the latest words are not learned + // yet, so they can still be corrected. + let pieces = learned(&log); + assert!(!pieces.is_empty()); + assert!( + pieces.iter().all(|p| p.chars().count() >= RECENT - 5), + "{pieces:?}" + ); + assert!(pieces.concat().chars().count() <= 1200 - RECENT + 5); + typing(&mut e, &mut revision, "", &[Edit::End]); + let all = [pieces, learned(&log)].concat().concat(); + let words: Vec<_> = all.split_whitespace().collect(); + assert_eq!(words.len(), 240); + assert!(words.iter().all(|w| *w == "abcd")); + } + + #[test] + fn a_combining_accent_stays_with_its_word() { + let (mut e, log) = learner(); + let mut revision = 0; + typing(&mut e, &mut revision, "cafe\u{301}", &[Edit::Reset]); + assert!(learned(&log).is_empty(), "the word was in progress"); + typing(&mut e, &mut revision, "cafe\u{301} ", &[Edit::Reset]); + assert_eq!(learned(&log), ["cafe\u{301} "]); + } } diff --git a/src-tauri/src/state.rs b/src-tauri/src/state.rs index fefec901..2f70e369 100644 --- a/src-tauri/src/state.rs +++ b/src-tauri/src/state.rs @@ -89,6 +89,10 @@ pub struct AppSettings { /// into the same account. Local to this install, like pairing itself. #[serde(default = "default_auto_approve_account_devices")] pub auto_approve_account_devices: bool, + /// Word prediction learns from what is typed with the scanning keyboard. + /// What it learns stays on this computer, like the setting. + #[serde(default = "default_learn_word_prediction")] + pub learn_word_prediction: bool, } impl Default for AppSettings { @@ -113,6 +117,7 @@ impl Default for AppSettings { cursor_crosshairs: false, share_diagnostics: false, auto_approve_account_devices: default_auto_approve_account_devices(), + learn_word_prediction: default_learn_word_prediction(), } } } @@ -175,6 +180,10 @@ fn default_auto_approve_account_devices() -> bool { true } +fn default_learn_word_prediction() -> bool { + true +} + fn default_key_repeat_enabled() -> bool { true } @@ -1395,6 +1404,7 @@ mod tests { assert_eq!(settings.key_repeat_interval_ms, 250); assert_eq!(settings.key_repeat_initial_delay_ms, 500); assert!(settings.auto_approve_account_devices); + assert!(settings.learn_word_prediction); // Everything that was stored survives untouched. assert_eq!(settings.pointer_scale_percent, 150); diff --git a/src/App.tsx b/src/App.tsx index 1906f103..ac32c854 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -821,7 +821,7 @@ export function App() {

{updateNotice}

{view === "home" && void perform(api.disconnectAll)} onAccessibility={() => void perform(() => api.checkAccessibility(true))} />} {view === "switches" &&

Switches

The switches you use to control this computer.

} - {view === "scanning" &&

Scanning

Choose how your switches control the pointer.

} + {view === "scanning" &&

Scanning

Choose how your switches control the pointer.

} {view === "mouse" && } {(view === "mobile" || view === "devices" || view === "profiles") &&

Mobile

Control this computer from Switchify on a phone or tablet.

{view === "mobile" && void perform(api.disconnectAll)} />} diff --git a/src/PointScan.test.tsx b/src/PointScan.test.tsx index d38ef00f..b7623b50 100644 --- a/src/PointScan.test.tsx +++ b/src/PointScan.test.tsx @@ -665,3 +665,30 @@ it("saves row groups beside the pattern older versions read and lets an area cho await waitFor(() => expect(mocks.invoke.mock.lastCall?.[1].config.scanPreferences.menu).toStrictEqual({ pattern: undefined, rowGroups: undefined })); expect(screen.getByRole("button", { name: "Rows, groups, then items" })).toHaveAttribute("aria-pressed", "true"); }); + +it("learns from typing by default, keeps the choice local and forgets only after confirming", async () => { + const onSettingsChange = vi.fn(); + function WithSettings() { + const controller = useScanning(); + return ; + } + render(); + await screen.findByText(initial.message); + more(); + fireEvent.click(screen.getByRole("button", { name: "Customise keyboard" })); + const learn = screen.getByRole("checkbox", { name: "Learn from my typing" }); + expect(learn).toBeChecked(); + fireEvent.click(learn); + expect(onSettingsChange).toHaveBeenCalledWith({ ...browserState.settings, learnWordPrediction: false }); + expect(mocks.invoke).not.toHaveBeenCalledWith("configure_point_scan", expect.anything()); + fireEvent.click(screen.getByRole("button", { name: "Forget learned words…" })); + expect(mocks.invoke).not.toHaveBeenCalledWith("forget_learned_words"); + fireEvent.click(screen.getByRole("button", { name: "Keep them" })); + expect(mocks.invoke).not.toHaveBeenCalledWith("forget_learned_words"); + fireEvent.click(screen.getByRole("button", { name: "Forget learned words…" })); + mocks.invoke.mockImplementation(async (command, args) => + command === "forget_learned_words" ? false : command === "get_point_scan" ? initial : { ...initial, ...args }); + fireEvent.click(screen.getByRole("button", { name: "Forget learned words" })); + expect(mocks.invoke).toHaveBeenCalledWith("forget_learned_words"); + expect(await screen.findByText(/will be forgotten in a moment/)).toBeInTheDocument(); +}); diff --git a/src/api.ts b/src/api.ts index 67c55340..4e046595 100644 --- a/src/api.ts +++ b/src/api.ts @@ -23,6 +23,7 @@ export const browserState: AppState = { cursorOverlayEnabled: true, cursorOverlaySize: "medium", cursorOverlayColor: "red", cursorOverlayVisibility: "whileControlling", cursorCrosshairs: false, shareDiagnostics: false, autoApproveAccountDevices: true, + learnWordPrediction: true, }, capabilities: { platform: navigator.userAgent.includes("Mac") ? "macos" : "windows", @@ -62,6 +63,7 @@ async function call(command: string, args?: Record): Promise export const api = { state: () => call("get_app_state"), + forgetLearnedWords: () => "__TAURI_INTERNALS__" in window ? invoke("forget_learned_words") : Promise.resolve(true), approvePairing: (requestId: string) => call("approve_pairing", { requestId }), rejectPairing: (requestId: string) => call("reject_pairing", { requestId }), checkAccessibility: (prompt: boolean) => call("check_accessibility", { prompt }), diff --git a/src/settings/ScannerPreferences.tsx b/src/settings/ScannerPreferences.tsx index 8c761c65..419ccbdd 100644 --- a/src/settings/ScannerPreferences.tsx +++ b/src/settings/ScannerPreferences.tsx @@ -4,8 +4,10 @@ import { ChevronRight } from 'lucide-react'; import { lastModeName, mouseRepeatStopInstruction, selectOpensHome, type ScanningController, type ScannerColor, type PointScanConfig } from '../scanning/useScanning'; import { areaOptions, sharedOptions, defaultScanPreferences, usualAfterSelection, savedKeyboardWait, itemScanning, itemScanningLabels, type ItemScanning, type PanelSize, type ScanArea, type ScanOptions } from '../scanning/preferences'; import { SettingGroup, Toggle, OptionGroup } from './controls'; +import { api } from '../api'; +import type { AppSettings } from '../types'; -export function ScannerPreferences({ controller }: { controller: ScanningController }) { +export function ScannerPreferences({ controller, settings: appSettings, onSettingsChange }: { controller: ScanningController; settings?: AppSettings; onSettingsChange?: (next: AppSettings) => void }) { const { config, state, update, chooseKeyboardLayout, pending } = controller; const [area, setArea] = useState<'shared' | ScanArea>('shared'); const heading = useRef(null); @@ -183,7 +185,8 @@ export function ScannerPreferences({ controller }: { controller: ScanningControl } {area === 'keyboard' && update('wordPrediction', value)} /> -

Uses an offline English word database and an on-device language model. Suggestions appear as you type and after a space. The keyboard shows when suggestions are loading, unavailable, or have no match. If prediction fails, choose Retry predictions on the keyboard; typing still works. Your typing never leaves this computer and is not used for learning.

+

Uses an offline English word database and an on-device language model. Suggestions appear as you type and after a space. The keyboard shows when suggestions are loading, unavailable, or have no match. If prediction fails, choose Retry predictions on the keyboard; typing still works. Your typing never leaves this computer.

+ {appSettings && onSettingsChange && } } {field('automatic', 'Automatic scanning', locked => change('automatic', value)} />)} @@ -249,3 +252,28 @@ export function ScannerPreferences({ controller }: { controller: ScanningControl

Press Select to scan again.

; } + +function LearnedWords({ settings, onSettingsChange, disabled }: { settings: AppSettings; onSettingsChange: (next: AppSettings) => void; disabled: boolean }) { + const [confirming, setConfirming] = useState(false); + const [status, setStatus] = useState(null); + const forget = async () => { + setConfirming(false); + try { + const done = await api.forgetLearnedWords(); + setStatus(done ? 'Learned words forgotten.' : 'Learned words will be forgotten in a moment, as soon as the keyboard stops using them.'); + } catch (error) { + setStatus(String(error)); + } + }; + return <> + onSettingsChange({ ...settings, learnWordPrediction: value })} /> +

Words and phrases you type with the scanning keyboard are suggested sooner. What is learned stays on this computer and is not synced. It is word counts, not a copy of your text, but it can include anything you type, such as names or passwords.

+ {confirming + ?
+

Forget every word learned on this computer? Suggestions return to the built-in English words. This cannot be undone.

+
+
+ : } + {status &&

{status}

} + ; +} diff --git a/src/settings/ScanningSection.tsx b/src/settings/ScanningSection.tsx index 87c35f8f..f18b58ab 100644 --- a/src/settings/ScanningSection.tsx +++ b/src/settings/ScanningSection.tsx @@ -2,6 +2,7 @@ import { Button } from "../ui/controls"; import { Demonstration } from "../help/Demonstration"; import { type ScanningController, validSwitches, mouseRepeatStopInstruction } from "../scanning/useScanning"; import { ScannerPreferences } from "./ScannerPreferences"; +import type { AppSettings } from "../types"; const phases = { keyboard: "Choose a keyboard row, then a key", @@ -27,7 +28,7 @@ const phases = { y: "Choose the vertical position", }; -export function ScanningSection({ controller }: { controller: ScanningController }) { +export function ScanningSection({ controller, settings, onSettingsChange }: { controller: ScanningController; settings?: AppSettings; onSettingsChange?: (next: AppSettings) => void }) { const { state, config, pending, error, retry, unsaved } = controller; return
@@ -37,7 +38,7 @@ export function ScanningSection({ controller }: { controller: ScanningController {error && unsaved && } {!validSwitches(config) &&

Each switch action needs a different key. Escape is reserved for cancel.

}
- +
How scanning works

Point scanning chooses a screen location with a moving line or grid, then opens actions for clicks, scrolling and dragging. Mouse scanning moves a visible pointer ring and offers directions, clicks, dragging, scrolling and Keyboard in a persistent panel.

Select opens Home, unless you chose Last mode used. Home offers Point, Mouse and Keyboard, plus Apps and windows, Editing, Browser and Media commands that act on the app in front, so you can bring the right app forward with switches instead of focusing it first. After a click or other action, Select carries on in the same mode; Close menu, Stop scanning or Escape means the next Select opens Home. Home is also a tile in the Point action menu and Mouse Actions.

diff --git a/src/types.ts b/src/types.ts index 526543fa..7a4bf930 100644 --- a/src/types.ts +++ b/src/types.ts @@ -39,6 +39,7 @@ export type AppSettings = { cursorCrosshairs: boolean; shareDiagnostics: boolean; autoApproveAccountDevices: boolean; + learnWordPrediction: boolean; }; export type PlatformCapabilities = {