Repository navigation
Show folder discovery progress before indexing begins - #91
Conversation
… localization - SQLite & Search Layer (PRs ARahim3#16, ARahim3#38, ARahim3#82, ARahim3#85, ARahim3#94, ARahim3#111): * StatementCache with sqlite3_reset statement recycling and reentrant transactions * PRAGMA mmap_size = 1GB, synchronous = NORMAL, cache_size = -64000 * Zero-copy row.copyBlob direct vector loading into searcher matrix halves * Unicode NFC normalization for cross-platform and CJK/accented file paths * Chunked IN queries with UNION ALL to eliminate N+1 roundtrips * CJK Han character detection, Han text budgeting, and Han-aware snippet generation * SIMD8<Float> acceleration in Math+Portable.swift without heap allocations * Float16Shim for Intel Mac x86_64 IEEE 754 binary16 compatibility - Crawler, Rules, Path Normalization & Extractors (PRs ARahim3#33, ARahim3#42, ARahim3#56, ARahim3#60, ARahim3#79, ARahim3#80, ARahim3#86, ARahim3#88, ARahim3#91, ARahim3#101, ARahim3#107, ARahim3#109, Issues ARahim3#67, ARahim3#97, ARahim3#105, ARahim3#127): * Fix Windows path slash/drive normalization in DirectoryWalk preventing 0-file crawl * Fix case-insensitive folder matching on Windows in FolderSelection * Fix Crawler to never treat user-selected search roots (e.g. ~) as project roots * Exclude secrets/keys (.env*, .pem, .kdbx, SSH keys, passwd) in Rules * Honor standard CACHEDIR.TAG 43-byte signature when crawling * Skip build/intermediate directories (.venv, target, .build, .next, dist, bin, obj) * Drop FileProvider download status lookups for 10-50x faster cloud folder crawls * OfficeText extractor for .xlsx, .pptx, .ods, .odp XML documents * Headless LibreOffice .doc extractor on Linux and local Notion SQLite workspace extractor * Safe typed CGPDFDictionaryGetString for PDF metadata preventing crashes * Preserve OCR keywords on image-only PDFs * Outlook PST reader utility crate in pst/ - Indexing Pipeline & Runtime (PRs ARahim3#49, ARahim3#53, ARahim3#62, ARahim3#64, ARahim3#71, ARahim3#72, ARahim3#114, Issue ARahim3#3): * Batched document text embeddings (up to 16 docs per pass) * Cumulative IndexProgress tracking across multi-pass indexing * DocumentCache for extracted chunks to prevent redundant file disk reads * Linux URLSession SIGABRT fix in FileFetcher * ImageLoader parallel prefetching and batched image embeddings * Apple Neural Engine CoreML video frame encoding support * Scoped incremental subtree sync when changed paths are known * Autoreleasepool wrapping in background worker extraction to eliminate memory leaks - Daemon, Lifecycle & Policies (PRs ARahim3#24, ARahim3#27, ARahim3#31, ARahim3#36, ARahim3#51, ARahim3#82, Issues ARahim3#22, ARahim3#68): * HelperRestartPolicy for clean stop/resume transitions * Backlog starvation recovery and --resume-pending in IndexService * Configurable indexing hours window policy * Opt-in Low Power Mode indexing to protect laptop thermals and battery * Win32 OpenProcess(SYNCHRONIZE) parent monitor to prevent zombie helpers on Windows * Correct 0-hit search behavior when all search roots are removed - UI, Cross-Platform Shell, Localization & Build (PRs ARahim3#15, ARahim3#20, ARahim3#37, ARahim3#39, ARahim3#58, ARahim3#65, ARahim3#74, ARahim3#81, ARahim3#84, ARahim3#112): * DigUpLocalization framework with en, zh-Hans, zh-Hant string catalogs * AppSearch engine and launcher for macOS App and Tauri desktop shells * KeyCommands and QuickLookController shared handling * Preserved search field text during IME composition * Search panel fade transitions and @observable code editor selection * Hardened build scripts with pinned SHA-256 vendor checksum verification
|
Thanks @danieladr, this is a real gap: people can't tell a scan from a broken index (#67). I'd take it in a narrower shape: the helper's scan event and the "Checking folders…" line in the Mac app, without the new test target (the crawler test is good to keep). I'll add the same line to the Windows/Linux app, which shows the same status. If you'd rather I take it from here, say so and I'll do it with credit to you. |
|
Thanks for taking a look! I've narrowed this in f68b222 to the helper scan events and the existing “Checking folders…” line on Mac. Removed the app test target and extra UI changes, kept the crawler tests, and resolved the conflicts with main. All 116 tests and the Mac build pass. You're welcome to take it from here and adjust it to fit the project. I appreciate the offer to keep the credit, and I'm happy to help with any follow-up testing. |
|
Thanks @danieladr, that was quick. Merged (bea70ab): the helper now says when it's walking the folders, the Mac shows "Checking folders…" from your change, and I've added the same line to the Windows and Linux app on top of it. It's in the next Mac release and the next beta. |
ARahim3
added
enhancement
|
It's out on Windows and Linux in 0.7.0-beta.3; the Mac gets it with 0.6.3. |
|
Thanks for merging this and for the opportunity to contribute! I appreciate your guidance on keeping the change focused, and for bringing it to Windows and Linux too. I'd love to keep helping with DigUp. Looking forward to contributing more! 🙏 |
Situation
On macOS, DigUp could show 0 files indexed / 0 of 0 files searchable while discovering files, leaving no visible distinction between a scan and a stalled index. The Mac app already had a Checking folders… status line, but never entered its
syncingstate.Task
Expose folder discovery through the helper and activate that existing Mac status line, keeping the change focused on the scope requested in review.
Action
scanningevents before and during discovery, with cumulative entries checked and the current root. Updates during a walk are throttled to roughly 250 ms; these counts are not an indexing total.syncingstate on a scan event and clear it when sync completes, an error occurs, or the helper exits. Existing idle and model-wait events also end that state.Result
Validated locally on Apple Silicon:
swift test: 116 tests passed in 15 suites on the final full run. An existing scanned-PDF keyword test failed on the first run, then passed both in isolation and in the full rerun.swift build --product DigUpApp: passed.scanningevents precededsync, one eligible file was discovered, and the helper exited with code 0.git diff --check: passed.Closes #90. Related to #67. Search ranking and embedding behavior are unchanged, so search evals were not run. This PR changes the helper protocol and native Mac status handling; it does not change the Windows/Linux app.