From ad99f62ca41b146b054b672c0310c2b0c0757779 Mon Sep 17 00:00:00 2001 From: HardMax71 Date: Thu, 22 Oct 2026 22:02:34 +0200 Subject: [PATCH] fix: OpenSearch creates no logs index before its template and retention policy exist On prod the collector's first write created logs-2026.10.22 0.3 s before start.sh put the template in, so the index got default mappings and a replica. OpenSearch now starts refusing to auto-create logs- indices, and start.sh allows it once the template and the policy exist, so the step that attached the policy afterwards is gone. --- backend/opensearch/start.sh | 4 ++-- docker-compose.yaml | 2 ++ docs/operations/opensearch.md | 4 +++- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/backend/opensearch/start.sh b/backend/opensearch/start.sh index f8bee869..9fd88b94 100755 --- a/backend/opensearch/start.sh +++ b/backend/opensearch/start.sh @@ -13,8 +13,8 @@ curl -sf -X PUT "$api/_index_template/logs" -H 'Content-Type: application/json' curl -sf -o /dev/null "$api/_plugins/_ism/policies/logs-30d" \ || curl -sf -X PUT "$api/_plugins/_ism/policies/logs-30d" -H 'Content-Type: application/json' -d @/etc/opensearch-setup/logs-retention-policy.json > /dev/null \ || exit 1 -# An index the collector created before the policy existed gets it too; one that has it already is skipped -curl -s -X POST "$api/_plugins/_ism/add/logs-*" -H 'Content-Type: application/json' -d '{"policy_id": "logs-30d"}' > /dev/null +# Compose starts OpenSearch refusing to create logs- indices, so none exists without the template and the policy +curl -sf -X PUT "$api/_cluster/settings" -H 'Content-Type: application/json' -d '{"persistent": {"action.auto_create_index": "true"}}' > /dev/null || exit 1 # The first wait returns when the trap runs; the second waits for OpenSearch to finish shutting down wait $pid wait $pid diff --git a/docker-compose.yaml b/docker-compose.yaml index 60b42da3..0220d21e 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -365,6 +365,8 @@ services: - discovery.type=single-node # mmap would need vm.max_map_count raised on the host - node.store.allow_mmap=false + # No write can create a logs- index before start.sh has set up its template and retention policy; start.sh then allows it + - action.auto_create_index=-logs-*,+* - DISABLE_SECURITY_PLUGIN=true - DISABLE_INSTALL_DEMO_CONFIG=true - OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m diff --git a/docs/operations/opensearch.md b/docs/operations/opensearch.md index 5b322b4a..1235b3e3 100644 --- a/docs/operations/opensearch.md +++ b/docs/operations/opensearch.md @@ -43,7 +43,9 @@ The container's entrypoint is [`backend/opensearch/start.sh`](https://github.com --8<-- "backend/opensearch/logs-retention-policy.json" ``` -The policy deletes each daily index 30 days after it was created, the same retention as Loki. A stored policy is never overwritten, so a changed policy needs a new id. An index created before the policy existed, on the very first start, gets it attached by the same script. +The policy deletes each daily index 30 days after it was created, the same retention as Loki. A stored policy is never overwritten, so a changed policy needs a new id. + +The collector starts writing as soon as OpenSearch answers, a moment before start.sh has put the template in. So compose starts OpenSearch with `action.auto_create_index=-logs-*,+*`: until then a write gets `index_not_found_exception` and is dropped (Loki has it), instead of creating a `logs-` index with default mappings. Once the template and the policy exist, start.sh allows auto-creation with a persistent cluster setting, so a restart needs no setup before writes go through. The cluster reports yellow even so: the retention plugin's own `.opendistro-ism-config` index asks for a replica that a single node can't place. ## Grafana