Repository navigation
Fix Han full-text matching and preserve tokenizer-budgeted text tails - #111
Closed
shimon-ai-dev wants to merge 1 commit into
Closed
shimon-ai-dev wants to merge 1 commit into
shimon-ai-dev wants to merge 1 commit into
Conversation
Tokenize Han bodies into contiguous FTS phrases and preserve original bounded snippets. Split complete prompts using actual runtime token counts, preserve source locations and continuation, and reject oversized embedding requests instead of silently truncating them. Schema 5 to 6 rebuilds stored bodies without rereading source files or removing vectors. Previously truncated vectors need explicit reindex; older binaries cannot open schema 6. Add deterministic, worker-contract and actual-model regressions. Preserve upstream identity, filename-prefix behavior and dependency lockfile.
AmirHossein-84
added a commit
to AmirHossein-84/DigUp
that referenced
this pull request
Oct 18, 2026
… localization - SQLite & Search Layer (PRs ARahim3#16, ARahim3#38, ARahim3#82, ARahim3#85, ARahim3#94, ARahim3#111): * StatementCache with sqlite3_reset statement recycling and reentrant transactions * PRAGMA mmap_size = 1GB, synchronous = NORMAL, cache_size = -64000 * Zero-copy row.copyBlob direct vector loading into searcher matrix halves * Unicode NFC normalization for cross-platform and CJK/accented file paths * Chunked IN queries with UNION ALL to eliminate N+1 roundtrips * CJK Han character detection, Han text budgeting, and Han-aware snippet generation * SIMD8<Float> acceleration in Math+Portable.swift without heap allocations * Float16Shim for Intel Mac x86_64 IEEE 754 binary16 compatibility - Crawler, Rules, Path Normalization & Extractors (PRs ARahim3#33, ARahim3#42, ARahim3#56, ARahim3#60, ARahim3#79, ARahim3#80, ARahim3#86, ARahim3#88, ARahim3#91, ARahim3#101, ARahim3#107, ARahim3#109, Issues ARahim3#67, ARahim3#97, ARahim3#105, ARahim3#127): * Fix Windows path slash/drive normalization in DirectoryWalk preventing 0-file crawl * Fix case-insensitive folder matching on Windows in FolderSelection * Fix Crawler to never treat user-selected search roots (e.g. ~) as project roots * Exclude secrets/keys (.env*, .pem, .kdbx, SSH keys, passwd) in Rules * Honor standard CACHEDIR.TAG 43-byte signature when crawling * Skip build/intermediate directories (.venv, target, .build, .next, dist, bin, obj) * Drop FileProvider download status lookups for 10-50x faster cloud folder crawls * OfficeText extractor for .xlsx, .pptx, .ods, .odp XML documents * Headless LibreOffice .doc extractor on Linux and local Notion SQLite workspace extractor * Safe typed CGPDFDictionaryGetString for PDF metadata preventing crashes * Preserve OCR keywords on image-only PDFs * Outlook PST reader utility crate in pst/ - Indexing Pipeline & Runtime (PRs ARahim3#49, ARahim3#53, ARahim3#62, ARahim3#64, ARahim3#71, ARahim3#72, ARahim3#114, Issue ARahim3#3): * Batched document text embeddings (up to 16 docs per pass) * Cumulative IndexProgress tracking across multi-pass indexing * DocumentCache for extracted chunks to prevent redundant file disk reads * Linux URLSession SIGABRT fix in FileFetcher * ImageLoader parallel prefetching and batched image embeddings * Apple Neural Engine CoreML video frame encoding support * Scoped incremental subtree sync when changed paths are known * Autoreleasepool wrapping in background worker extraction to eliminate memory leaks - Daemon, Lifecycle & Policies (PRs ARahim3#24, ARahim3#27, ARahim3#31, ARahim3#36, ARahim3#51, ARahim3#82, Issues ARahim3#22, ARahim3#68): * HelperRestartPolicy for clean stop/resume transitions * Backlog starvation recovery and --resume-pending in IndexService * Configurable indexing hours window policy * Opt-in Low Power Mode indexing to protect laptop thermals and battery * Win32 OpenProcess(SYNCHRONIZE) parent monitor to prevent zombie helpers on Windows * Correct 0-hit search behavior when all search roots are removed - UI, Cross-Platform Shell, Localization & Build (PRs ARahim3#15, ARahim3#20, ARahim3#37, ARahim3#39, ARahim3#58, ARahim3#65, ARahim3#74, ARahim3#81, ARahim3#84, ARahim3#112): * DigUpLocalization framework with en, zh-Hans, zh-Hant string catalogs * AppSearch engine and launcher for macOS App and Tauri desktop shells * KeyCommands and QuickLookController shared handling * Preserved search field text during IME composition * Search panel fade transitions and @observable code editor selection * Hardened build scripts with pinned SHA-256 vendor checksum verification
This was referenced Oct 20, 2026
Owner
|
Thanks @shimon-ai-dev. Two things are in here, and I'd rather take them one at a time with an issue each. Chinese keyword search I'll do in the tokenizer (see #85), with a Chinese eval, once. Whether chunks can overrun the model's token budget and lose their tail is a fair question: if you have a file where that happens, open an issue with it and the measurement, and I'll look at splitting by the real token count. Closing the PR as a bundle. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Clean port onto main ec28973: Han document-body FTS, original bounded snippets, actual-tokenizer budget splitting that preserves long-document tails, and worker overflow rejection. Keeps unspaced filenames and existing prefix behavior; complements #85. No UI, media, identity or unrelated fork changes.
Migration
Schema 5 to 6 rebuilds stored FTS bodies while preserving cached vectors, original text and file states, including offline files. Previously truncated vectors need explicit reindexing. Older binaries require a pre-upgrade index backup for rollback.
Validation
Clean commit: 21870a1.
CI uses workflow-only child commits; those workflows are not in this PR.
Full DigUpTestbed semantic evals, interactive GUI/live prompts and notarization are not claimed.
No prior maintainer approval is implied; I understand the issue-first contribution policy may require discussion before review.