[HardMax71]

Summary

Closes #373. Log search in Loki is slow: on prod, one execution id over the 10 days Loki holds takes 3.4 s, and a field search with | json 8.2 s, since Loki reads every chunk. Mongo's connection and login lines are 85% of those bytes.

Changes

The collector drops Mongo's info lines and hands every log line to OpenSearch as well as Loki. structlog's JSON keys become attributes, stored as flat_object, so a key that's a number in one line and a string in another can't get a line rejected. One index a day, deleted after 30 days by a policy that start.sh applies together with the index template. Grafana gets the OpenSearch plugin and datasource. Loki's limit goes to 512m, since those searches took it to 207 MiB.

Warning

Prod gets a new container: OpenSearch 3.9 with a 512m heap, about 1.1 GiB resident at 30 days (limit 1536m), against 3.8 GB available today. Lines sent while it restarts are dropped from OpenSearch, because the exporter doesn't retry a refused connection; Loki keeps them.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• CI/CD or tooling changes

Test Plan

• Backend unit tests pass (cd backend && uv run pytest tests/unit)
• Frontend unit tests pass (cd frontend && npm run test)
• Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
• Frontend E2E tests pass (cd frontend && npm run test:e2e)
• API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
• Manual testing performed

Validation: local stack from this branch. With 1.94 M synthetic lines over 31 daily indices, a search by execution id took 6 to 8 ms in OpenSearch and 25 to 75 ms through Grafana. A first start from empty volumes applies the template and policy and installs the plugin, and a restart leaves every index green. The contract test and strict mkdocs pass.

Checklist

• My code follows the project's style guidelines
• I have run pre-commit run --all-files and all checks pass
• I have added tests that prove my fix/feature works
• I have updated documentation if needed
• My changes don't introduce new warnings

---

Summary by cubic

Adds OpenSearch log search alongside Loki to address #373: searches can query indexed fields instead of scanning Loki chunks.

• Parses structured backend and worker logs into searchable attributes, with daily indices retained for 30 days and a provisioned Grafana datasource.
• Drops Mongo info-level logs from both stores while keeping warnings and errors, and raises Loki’s memory limit from 256 MiB to 512 MiB.
• Adds OpenSearch to the observability stack with a 512 MiB heap and 1,536 MiB container limit; logs sent during OpenSearch startup or restarts may be missing there, but remain in Loki.
• Documents OpenSearch setup, queries, and operational details.

^{Written for commit 2db2d6e. Summary will update on new commits.}

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 5b868ea6-1d5a-45f2-93ef-7eaa65b0e09c

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag	Coverage Δ
backend-e2e	82.04% <ø> (ø)
backend-unit	77.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Backend	94.73% <ø> (ø)
Frontend	92.25% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.