[reckless-sherixx]

Summary

Logs, traces and metrics share the ClickHouse volume, and its TTL drops data by age, not by size. A noisy namespace can therefore fill the disk and stop every write, with no warning beforehand.

• Alerts (3 rules):

  • ClickHouseInstanceDown: scrape failing or target absent for 2 minutes, critical.
  • ClickHouseDiskFilling: any disk over 80% for 10 minutes, warning.
  • ClickHouseDiskAlmostFull: any disk over 90% for 5 minutes, critical.

  The disk rules use ClickHouse's own DiskAvailable_<disk>/DiskTotal_<disk> metrics. label_replace moves the disk name into a disk label, so every disk is covered.

• Insert guard: merge_tree.min_free_disk_ratio_to_perform_insert = 0.05. Inserts are refused (NOT_ENOUGH_SPACE) while less than 5% is free. That leaves room for merges, TTL part drops and queries, instead of a full disk.

• New template templates/clickhouse-prometheusrule.yaml renders clickhouse.metrics.prometheusRule.rules. The Bitnami subchart has no PrometheusRule template, so these rules were never applied before. It renders only when a ClickHouse ServiceMonitor does and the PrometheusRule CRD is present (or enablePrometheusStack is set).

• Fixed ClickHouseInstanceDown: it matched up{job="clickhouse"}, but the operator sets job to the Service name, so it never matched. Each expr now goes through tpl and selects this release's namespace and service, so it also never matches another release's ClickHouse.

• Removed ClickHouseHighLatency: ClickHouse does not export clickhouse_query_duration_seconds, so it could never fire.

For reviewers:

• Below 5% free, trace and metric inserts are refused too. They are dropped after the exporter's 300 s retry. Container logs wait on the nodes, with feat(logs): deliver container logs at least once across restarts and outages node-agent#407.

Type of change

• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change (chart upgrade requires user action)
• Documentation only
• CI / tooling

Chart version

• I bumped version in charts/nudgebee-agent/Chart.yaml (required for user-visible changes): 0.1.31
• No version bump needed (docs/CI only)

Test plan

• helm lint charts/nudgebee-agent passes.
• ct lint --config .github/configs/ct.yaml: run by CI.
• helm template reviewed, including fullnameOverride, nameOverride and namespaceOverride: the rule's namespace/service match the scraped Service. promtool check rules accepts the 3 rendered rules, and promtool test rules passes (two disks, up == 0, absent target, no cross-release match).
• Live: Prometheus scraping ClickHouse 24.12 with a second disk returns one disk series per disk; ClickHouseInstanceDown stays inactive once scraped.
• ClickHouse 24.12 loads the setting. With a forced threshold, an insert is refused with Code 243 NOT_ENOUGH_SPACE while reads keep working.

Related issues

None in this repo.

[gemini-code-assist]

Code Review

This pull request updates the nudgebee-agent Helm chart to version 0.1.31. It adds a new template to render PrometheusRule alerts for ClickHouse, configures ClickHouse to stop inserts when free disk space falls below 5%, and introduces two new Prometheus alerts (ClickHouseDiskFilling and ClickHouseDiskAlmostFull) to monitor disk usage. There are no review comments, so I have no feedback to provide.

[blue4209211]

The insert guard looks good: min_free_disk_ratio_to_perform_insert = 0.05 with the two disk alerts in front of it. Gating the new PrometheusRule on the CRD (or enablePrometheusStack) keeps installs without the operator safe.

Two checks before merge, because these rules render for the first time on every cluster with the Prometheus operator:

1. ClickHouseHighLatency: clickhouse_query_duration_seconds isn't exported by ClickHouse's built-in Prometheus endpoint, so this rule can never fire. On your question, I'd drop it rather than ship a dead alert. A latency alert can come later from a metric ClickHouse does export.
2. ClickHouseInstanceDown: up{job="clickhouse"} only fires if the scrape's job label really is clickhouse. It comes from this chart's ServiceMonitor, so check the rendered target's job with helm template, or match on the service label instead.

Minor:

• The disk expressions cover only the disk named default (…_DiskAvailable_default). That's fine for the bundled setup; worth a values comment for custom storage policies.
• At 95% full, logs held back upstream will wait, while traces and metrics are dropped after the exporter's retry window. That's worth one line in the chart docs next to the alert text.
• Follow-up idea: these alerts only reach clusters with Prometheus plus Alertmanager. Reporting ClickHouse disk usage on the runner's heartbeat would let the backend surface it everywhere.

[reckless-sherixx]

@blue4209211 thanks. All four points are handled in 405fbdf (rebased on main, still 0.1.31).

1. ClickHouseHighLatency: dropped. ClickHouse 24.12's /metrics endpoint exports no clickhouse_query_duration_seconds, and no histograms or summaries at all, so it could never fire. A later latency alert can use rate(ClickHouseProfileEvents_QueryTimeMicroseconds[5m]) / rate(ClickHouseProfileEvents_Query[5m]).
2. ClickHouseInstanceDown: fixed. The chart's ServiceMonitor has jobLabel: "", so the operator sets job to the Service name (<release>-clickhouse), and job="clickhouse" never matched. The rule now selects this release's namespace and service, and also fires on absent() when the target is gone. It renders only when a ClickHouse ServiceMonitor does. Rendered: up{namespace="ns",service="rel-clickhouse"} == 0 or absent(up{namespace="ns",service="rel-clickhouse"}).
3. Disks: every disk is covered. label_replace moves the disk name from DiskAvailable_<disk> into a disk label, and the summary names the disk. Checked live against ClickHouse 24.12.6.70 with a second disk backups: the expr returned two series, disk="backups" and disk="default", matching system.disks.
4. Heartbeat disk usage: agreed, a good follow-up. Not in this PR.

On the 95% point: the values comment next to the insert guard says refused inserts are retried for the exporter's retry window, then dropped. Once the logs-backpressure PR (#678) lands, logs are held rather than dropped when inserts are refused. Whichever of the two merges second will reword that comment.

Checks:

• helm lint charts/nudgebee-agent: 1 chart linted, 0 failed.
• promtool check rules (Prometheus v3.5.0) on the rendered rules: SUCCESS, 3 rules.
• promtool test rules with operator labels: with two disks only the full one alerts, up == 0 fires after for:, a missing target fires absent() with namespace and service, and another release's ClickHouse in another namespace does not match.
• No PrometheusRule renders with enableServiceMonitors=false, nudgebeeServiceMonitor.enabled=false or enablePrometheusStack=false without the CRD.