Skip to content

Fix Grafana dashboard creation timing issue by adding wait and retry mechanisms - #750

Open
Matt Tucker (matucker-msft) with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-749
Open

Matt Tucker (matucker-msft) with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-749

Conversation

Copilot AI commented May 30, 2025 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #749.

The original implementation needed correction: postfix increments caused the retry loops to exit immediately under set -e, and ARM provisioning success did not establish Grafana API readiness. Data-source setup could also fail before dashboard retries ran.

  • Merge current main to resolve conflicts and preserve dashboard datasource/cluster defaults.
  • Probe the Grafana data-plane API using az grafana data-source list before configuring data sources or dashboards: 30 attempts, 30 seconds between attempts.
  • Retry data-source creation and dashboard imports three times with 10-second delays.
  • Preserve CLI errors and exit unsuccessfully when retries are exhausted; do not report partial setup as success.
  • Use the selected Grafana subscription consistently and clean up temporary dashboard definitions on success or failure.
  • Regenerate the deployment bundle and add mocked shell regression tests to CI.

Validation

  • Bash syntax checks passed.
  • Six regression scenarios passed against both source setup and the extracted deployment bundle: existing instance, newly provisioned instance, readiness exhaustion, data-source exhaustion, dashboard exhaustion, and immediate success.
  • Tests verify exact retry counts/delays, visible CLI errors, failure exit statuses, dashboard defaults, all five imports, and temporary-file cleanup.
  • make k8s-bundle completed successfully; bundled setup/helper match their sources.
  • Changes pass whitespace validation against current main.
  • No live Azure deployment was performed; full Go CI remains separate from the targeted shell validation.

Co-authored-by: matucker-msft <77026928+matucker-msft@users.noreply.github.com>
Copilot AI changed the title [WIP] Grafana dashboards are attempted to be created before managed grafana instance is available Fix Grafana dashboard creation timing issue by adding wait and retry mechanisms May 30, 2025
Probe the Grafana data-plane API before configuration, retry data-source and dashboard creation, and fail explicitly on exhaustion. Preserve current dashboard defaults, add mocked regression coverage, and regenerate the deployment bundle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@matucker-msft
Matt Tucker (matucker-msft) marked this pull request as ready for review October 22, 2026 20:47

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against #749 and corrected the original retry-loop and readiness gaps in commit 666946f. The setup now probes the Grafana data-plane API before configuration, retries data-source/dashboard creation with bounded delays, preserves dashboard defaults from main, surfaces exhaustion as failure, and includes automated regression coverage. All six mocked scenarios passed against both source and extracted bundle; bundle generation and bash syntax checks passed. Approved on that basis; no live Azure deployment was performed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No blocking issues were identified in the retry logic, failure handling, cleanup, regression coverage, or bundle consistency.

No open issues 🎉

What changed in this PR

Addresses #749 by waiting for Azure Managed Grafana’s API to become available before configuring ADX-Mon data sources and dashboards.

Changes:

  • Adds bounded readiness checks and retries for data-source creation and dashboard imports.
  • Handles retry exhaustion, uses the selected Grafana subscription, and cleans up temporary dashboard files.
  • Adds mocked shell regression tests to CI, documents retry behavior, and regenerates the deployment bundle.
File Description
docs/​quick-start.md Documents readiness checks, retries, and failure behavior.
build/​scripts/​test-grafana-setup.sh Adds regression scenarios for retries, failures, defaults, and cleanup.
build/​k8s/​setup.sh Integrates readiness checks, retries, subscription selection, and cleanup.
build/​k8s/​grafana.sh Adds a shared bounded retry helper.
build/​k8s/​bundle.sh Includes updated setup logic and the retry helper.
.github/​workflows/​build-pr.yaml Runs Grafana setup regression tests in CI.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grafana dashboards are attempted to be created before managed grafana instance is available

3 participants