Repository navigation
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
paul43210
added a commit
to paul43210/ocis
that referenced
this pull request
Nov 18, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR Signed-off-by: Paul Faure <paul@faure.ca>
Contributor
|
The |
…ange Every IndexSpace walk sent every unindexed file to the content extractor again, including files whose extraction had already failed. A file that times Tika out (a large scanned PDF through OCR) cost a full timeout and a restart of Tika's forked worker on every walk, and every upload, move or restore in a space starts one. With SEARCH_EXTRACTOR_FAILED_RETRY_INTERVAL set, a file whose extraction fails is indexed by its name and metadata only, with the time of the failure, and later walks skip it while it is unchanged and the interval has not passed. Skipped files neither count towards nor reset the abort threshold. The default, 0, keeps the current behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR Signed-off-by: Paul Faure <paul@faure.ca>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR Signed-off-by: Paul Faure <paul@faure.ca>
paul43210
force-pushed
the
fix/search-failed-retry-interval
branch
from
November 19, 2026 01:05
0c2518a to
e329342
Compare
Contributor
Author
Contributor
|
@paul43210 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
IndexSpacesends every file that is not indexed, or not markedExtracted, to the content extractor on every walk, including files whose extraction already failed. Nothing records the failure:upsertItemreturns beforeengine.Upsert, so the file is not even findable by name, and the next walk tries it again. With Tika, a file that hits the watchdog (a large scanned PDF through OCR) costs a full timeout and a restart of Tika's forked JVM on every walk, and every upload, move, restore or tag change in the space starts one.This PR adds
SEARCH_EXTRACTOR_FAILED_RETRY_INTERVAL(Extractor.FailedRetryInterval, a duration, default 0 = unchanged behaviour). When it is set:Extracted=falseand a newExtractionFailedAttimestamp. It becomes findable by name;SEARCH_EXTRACTOR_MAX_CONSECUTIVE_FAILURES, and the walk summary logs them asfailed_earlier;ExtractionFailedAtis mapped in Bleve as stored but not indexed, so it is never searched, andgetResourcereads it back, soUpdate(tags) andMovekeep it. The success path is unchanged; theengine.Resourceconstruction moved intonewResourceso both paths share it. A file that was indexed with content, then changes and fails extraction, gets the name-only entry in place of its (stale) content. Negative intervals are rejected at startup.Related Issue
Motivation and Context
On a production single-binary instance (oCIS 8.2.0, Tika 3.2.3, 2 CPUs), four scanned PDFs in one space accounted for almost all of ~700
extraction failed after retrieswarnings in a week and caused 122 Tika worker restarts in two days (every onestatus: TIMEOUTat the 150 s watchdog), keeping Tika and Tesseract busy for about a third of the hours the space was in use. With an interval of e.g.24h, each of those files costs one timeout per day instead of one per walk.Follows #13034, which made the walk's abort threshold configurable in the same way (default unchanged, opt in).
How Has This Been Tested?
go.mod), Linux amd64, branch rebased on currentmaster(aebd047).go test -count=1 ./services/search/...: all packages pass.IndexSpace abort threshold›with a retry interval for failed extractions:Extracted=false, dated with the service clock;enginespecs:ExtractionFailedAtsurvivesUpsert→Update→Lookup, and a match query for its value finds nothing.parser: default 0; 24h valid; negative rejected.golangci-lint run --new-from-rev=origin/master ./...(v1.64.6) inservices/search: no findings.Types of changes
Checklist:
desc,introductionVersion: 9.0.0); changelog entry included🤖 Generated with Claude Code
https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR