Handle crawler status and preserve seeds on sitemap failures

[JpMaxMan]

The crawler could exit before its existing Nutch empty-fetch-list handler, or inject seeds after a failed sitemap download. Capture generate status before errexit; preserve the intended normal-empty and unexpected-error handlers. Retrieve sitemap seeds into a temporary file with HTTP failure detection, connection/overall time limits and pipeline failure propagation, reject empty or unsupported lines, then atomically replace the accepted seed file. Failures stop before injection and retain previous seeds.

Seven offline tests execute the real shell against temporary Nutch/curl stubs. Coverage includes two normal batches, empty fetch lists, unexpected generate errors, failures at every later pipeline stage, accepted sitemap replacement, partial/error downloads, empty/error bodies and unsupported compact XML. All pass locally; Bash syntax/diff checks pass. Ubuntu push/PR CI runs the same checks. The initial source reproduces both bypassed generate handlers.

The existing sitemap extractor still requires standalone loc lines and does not decode XML entities or traverse sitemap indexes. This change rejects unsupported line formats; it is not a general XML parser replacement.

No crawler/index has been deployed or invoked. Live deployment revision, freshness, robots/draft exclusion, update/deletion/redirect behavior and runtime modernization remain separate production-readiness gates. The VEXXHOST VM currently has no search deployment; its current production operator/access has been requested.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}