Repository navigation
Fix Grafana dashboard creation timing issue by adding wait and retry mechanisms - #750
Matt Tucker (matucker-msft) with Copilot wants to merge 3 commits into
Conversation
Co-authored-by: matucker-msft <77026928+matucker-msft@users.noreply.github.com>
Probe the Grafana data-plane API before configuration, retry data-source and dashboard creation, and fail explicitly on exhaustion. Preserve current dashboard defaults, add mocked regression coverage, and regenerate the deployment bundle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Matt Tucker (matucker-msft)
left a comment
There was a problem hiding this comment.
Reviewed against #749 and corrected the original retry-loop and readiness gaps in commit 666946f. The setup now probes the Grafana data-plane API before configuration, retries data-source/dashboard creation with bounded delays, preserves dashboard defaults from main, surfaces exhaustion as failure, and includes automated regression coverage. All six mocked scenarios passed against both source and extracted bundle; bundle generation and bash syntax checks passed. Approved on that basis; no live Azure deployment was performed.
There was a problem hiding this comment.
🟢 Approval recommended
No blocking issues were identified in the retry logic, failure handling, cleanup, regression coverage, or bundle consistency.
No open issues 🎉
What changed in this PR
Addresses #749 by waiting for Azure Managed Grafana’s API to become available before configuring ADX-Mon data sources and dashboards.
Changes:
- Adds bounded readiness checks and retries for data-source creation and dashboard imports.
- Handles retry exhaustion, uses the selected Grafana subscription, and cleans up temporary dashboard files.
- Adds mocked shell regression tests to CI, documents retry behavior, and regenerates the deployment bundle.
| File | Description |
|---|---|
docs/quick-start.md |
Documents readiness checks, retries, and failure behavior. |
build/scripts/test-grafana-setup.sh |
Adds regression scenarios for retries, failures, defaults, and cleanup. |
build/k8s/setup.sh |
Integrates readiness checks, retries, subscription selection, and cleanup. |
build/k8s/grafana.sh |
Adds a shared bounded retry helper. |
build/k8s/bundle.sh |
Includes updated setup logic and the retry helper. |
.github/workflows/build-pr.yaml |
Runs Grafana setup regression tests in CI. |
Review effort: Balanced
Summary
Fixes #749.
The original implementation needed correction: postfix increments caused the retry loops to exit immediately under
set -e, and ARM provisioning success did not establish Grafana API readiness. Data-source setup could also fail before dashboard retries ran.az grafana data-source listbefore configuring data sources or dashboards: 30 attempts, 30 seconds between attempts.Validation
make k8s-bundlecompleted successfully; bundled setup/helper match their sources.