Skip to content

fix: OpenSearch creates no logs index before its template and retention policy exist - #453

Merged
HardMax71 merged 1 commit into
mainfrom
max/fix-opensearch-template-race
Oct 22, 2026
Merged

HardMax71 merged 1 commit into
mainfrom
max/fix-opensearch-template-race

Conversation

@HardMax71

@HardMax71 HardMax71 commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

Small bug found while checking release 2026.10.34 (#452) on prod. The collector's first write created logs-2026.10.22 at 19:57:10.224, 0.3 s before start.sh put the index template in (OpenSearch logged "templates []"). That index got default mappings: attributes and resource as plain objects, so mixed key types can reject lines, plus one replica that keeps it yellow on a single node.

Changes

Compose starts OpenSearch with action.auto_create_index=-logs-*,+*, so a write can't create a logs- index yet; it gets index_not_found_exception and is dropped, and Loki has the line. start.sh allows auto-creation with a persistent cluster setting once the template and the retention policy exist, so every logs- index gets both when it's created, and the step that attached the policy afterwards is gone. A restart reads the setting from the data volume.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: fresh OpenSearch 3.9 from this branch with a loop writing to today's index every 200 ms. Writes before setup got index_not_found_exception, the index was then created with templates [logs], flat_object mappings, 0 replicas and the logs-30d policy. After a restart a write created tomorrow's index the same way.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

…on policy exist

On prod the collector's first write created logs-2026.10.22 0.3 s before start.sh put the template in, so the index got default mappings and a replica. OpenSearch now starts refusing to auto-create logs- indices, and start.sh allows it once the template and the policy exist, so the step that attached the policy afterwards is gone.
@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8f747b43-e1d1-4ffe-a504-f2c88675a3f8
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@HardMax71
HardMax71 enabled auto-merge (squash) October 22, 2026 20:03
@github-actions github-actions Bot added documentation Improvements or additions to documentation area: deploy Compose, images, release workflow, production host labels Oct 22, 2026
@sonarqubecloud

Copy link
Copy Markdown

@codecov-commenter

codecov-commenter commented Oct 22, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag Coverage Δ
backend-e2e 82.04% <ø> (ø)
backend-unit 77.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Backend 94.73% <ø> (ø)
Frontend 92.25% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HardMax71
HardMax71 merged commit ddccfed into main Oct 22, 2026
22 checks passed
@HardMax71
HardMax71 deleted the max/fix-opensearch-template-race branch October 22, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: deploy Compose, images, release workflow, production host documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants