Skip to content

feat(search): leave files whose extraction failed alone until they change - #13506

Open
paul43210 wants to merge 2 commits into
owncloud:masterfrom
paul43210:fix/search-failed-retry-interval
Open

paul43210 wants to merge 2 commits into
owncloud:masterfrom
paul43210:fix/search-failed-retry-interval

Conversation

@paul43210

Copy link
Copy Markdown
Contributor

Description

IndexSpace sends every file that is not indexed, or not marked Extracted, to the content extractor on every walk, including files whose extraction already failed. Nothing records the failure: upsertItem returns before engine.Upsert, so the file is not even findable by name, and the next walk tries it again. With Tika, a file that hits the watchdog (a large scanned PDF through OCR) costs a full timeout and a restart of Tika's forked JVM on every walk, and every upload, move, restore or tag change in the space starts one.

This PR adds SEARCH_EXTRACTOR_FAILED_RETRY_INTERVAL (Extractor.FailedRetryInterval, a duration, default 0 = unchanged behaviour). When it is set:

  • a file whose extraction fails is upserted by its name and metadata only (basic extraction from the stat info), with Extracted=false and a new ExtractionFailedAt timestamp. It becomes findable by name;
  • later walks skip such a file while it is unchanged (index mtime not older than the file's) and the interval has not passed. A changed file is extracted again straight away;
  • skipped files neither count towards nor reset SEARCH_EXTRACTOR_MAX_CONSECUTIVE_FAILURES, and the walk summary logs them as failed_earlier;
  • an extraction cut short because the walk's context ended records nothing, since it says nothing about the file.

ExtractionFailedAt is mapped in Bleve as stored but not indexed, so it is never searched, and getResource reads it back, so Update (tags) and Move keep it. The success path is unchanged; the engine.Resource construction moved into newResource so both paths share it. A file that was indexed with content, then changes and fails extraction, gets the name-only entry in place of its (stale) content. Negative intervals are rejected at startup.

Related Issue

Motivation and Context

On a production single-binary instance (oCIS 8.2.0, Tika 3.2.3, 2 CPUs), four scanned PDFs in one space accounted for almost all of ~700 extraction failed after retries warnings in a week and caused 122 Tika worker restarts in two days (every one status: TIMEOUT at the 150 s watchdog), keeping Tika and Tesseract busy for about a third of the hours the space was in use. With an interval of e.g. 24h, each of those files costs one timeout per day instead of one per walk.

Follows #13034, which made the walk's abort threshold configurable in the same way (default unchanged, opt in).

How Has This Been Tested?

  • test environment: Go 1.26.10 (toolchain per go.mod), Linux amd64, branch rebased on current master (aebd047).
  • go test -count=1 ./services/search/...: all packages pass.
  • New specs under IndexSpace abort threshold › with a retry interval for failed extractions:
    • a failed file is upserted name-only, Extracted=false, dated with the service clock;
    • an unchanged file with a recent failure is not sent to the extractor;
    • it is tried again once the interval has passed, and as soon as its mtime is newer than the index entry;
    • with the interval at 0 it is tried on every walk and nothing is recorded;
    • skipped files do not count towards a threshold of 1;
    • a walk whose context ends during extraction records nothing.
    • With the skip disabled, the two skip specs fail (checked).
  • engine specs: ExtractionFailedAt survives Upsert → Update → Lookup, and a match query for its value finds nothing.
  • parser: default 0; 24h valid; negative rejected.
  • golangci-lint run --new-from-rev=origin/master ./... (v1.64.6) in services/search: no findings.
  • Existing specs unchanged and passing: with the default 0 nothing new happens on failure.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)

Checklist:

  • Code changes
  • Unit tests added
  • Acceptance tests added: not applicable, the behaviour is internal to the indexer and covered by unit tests
  • Documentation ticket raised: not needed, the env var is documented through its struct tag (desc, introductionVersion: 9.0.0); changelog entry included

🤖 Generated with Claude Code

https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR

@paul43210
paul43210 requested a review from a team as a code owner November 17, 2026 19:43
@kw-security

kw-security commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues
✅ Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

paul43210 added a commit to paul43210/ocis that referenced this pull request Nov 18, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR
Signed-off-by: Paul Faure <paul@faure.ca>
@mmattel

mmattel commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

The build-and-test CI job is failing because a security-related fix that was applied to master is missing on this branch. A rebase onto the current master will pick it up and unblock CI.

git fetch origin
git rebase origin/master
git push --force-with-lease

paul43210 and others added 2 commits November 18, 2026 19:32
…ange

Every IndexSpace walk sent every unindexed file to the content extractor
again, including files whose extraction had already failed. A file that
times Tika out (a large scanned PDF through OCR) cost a full timeout and a
restart of Tika's forked worker on every walk, and every upload, move or
restore in a space starts one.

With SEARCH_EXTRACTOR_FAILED_RETRY_INTERVAL set, a file whose extraction
fails is indexed by its name and metadata only, with the time of the
failure, and later walks skip it while it is unchanged and the interval has
not passed. Skipped files neither count towards nor reset the abort
threshold. The default, 0, keeps the current behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR
Signed-off-by: Paul Faure <paul@faure.ca>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159L4KrFcxprCvYF2CYreAR
Signed-off-by: Paul Faure <paul@faure.ca>
@paul43210
paul43210 force-pushed the fix/search-failed-retry-interval branch from 0c2518a to e329342 Compare November 19, 2026 01:05
@paul43210

Copy link
Copy Markdown
Contributor Author

Rebased onto master (37a8944) as asked, @mmattel; search tests pass. Head is now e329342.

@mmattel

mmattel commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@paul43210
The changelog lint check found an overlong title in changelog/unreleased/enhancement-search-skip-failed-extraction.md.

The title is 84 characters; the limit is 80.
This caused make changelog-lint to exit with error 1 in the Acceptance Tests workflow.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

search: a file whose content extraction failed is sent to the extractor again on every walk

3 participants