Skip to content

feat(health): record application health history - #425

Open
alex289 wants to merge 3 commits into
feat/health-service-statusfrom
feat/health-history
Open

alex289 wants to merge 3 commits into
feat/health-service-statusfrom
feat/health-history

Conversation

@alex289

@alex289 alex289 commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Type of change

  • 🐛 Bug fix
  • 🚀 New feature
  • ❓ Other (please specify)

Description

Part 2/4 of #268. Stores every change of an application's health so availability can be computed over time.

  • New application_health_changes table (migration 000036): status, optional reason, the failing services (JSON) and changed_at. It is deleted along with its application.
  • A row is only written when the status differs from the application's latest row, using the reports agents already send. Reports for another agent's application are still ignored and not recorded.
  • Agent offline: a disconnect records unknown with reason agent_offline. At hub startup each agent is marked offline from its last_seen time, so hub downtime shows up as a gap instead of the last status silently continuing.
  • Deploys and image updates don't write history. The previous status holds until the agent reports the result, as agreed in the issue.
  • Retention: a daily pruner deletes changes older than 30 days. It keeps the latest change before the cutoff per application, so the 30d range still starts in the right state.
  • Timestamps are stored in UTC, because SQLite compares them as text.

@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.19178% with 26 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../internal/hub/models/application_health_changes.go 0.00% 10 Missing ⚠️
...ackend/internal/hub/healthhistory/healthhistory.go 90.12% 8 Missing ⚠️
backend/internal/hub/websocket/handler.go 84.61% 6 Missing ⚠️
backend/internal/hub/server.go 81.81% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@alex289
alex289 force-pushed the feat/health-history branch from 7762942 to b77bc06 Compare October 28, 2026 13:16
Comment thread backend/internal/hub/websocket/handler.go Outdated
Comment thread backend/internal/hub/healthhistory/healthhistory.go
Comment thread backend/internal/hub/healthhistory/healthhistory.go Outdated
Comment thread backend/internal/hub/websocket/handler.go Outdated
FROM application_health_changes
WHERE changed_at < ?
) WHERE position = 1
)`, cutoff, cutoff).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting query :)
Not sure if this is really easier or which one is better:

Where(`changed_at < ? AND EXISTS (
    SELECT 1 FROM application_health_changes newer
    WHERE newer.application_id = application_health_changes.application_id
      AND newer.changed_at < ?
      AND (newer.changed_at, newer.id) > (application_health_changes.changed_at, application_health_changes.id)
)`, cutoff, cutoff)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both delete the same rows. I kept the window function because it says directly "keep the latest row before the cutoff per app". Speed doesn't matter much either way: after the first prune, only about a day of rows plus one row per app are older than the cutoff. Happy to switch to the EXISTS version if you prefer it.

Store every application health change with the failing services for
context. Agent disconnects and hub restarts are recorded as unknown with
an agent offline reason, using the agent's last seen time after a
restart. Changes older than 30 days are pruned daily, keeping the last
change before the cutoff per application.
@alex289
alex289 force-pushed the feat/health-history branch from b77bc06 to f5a9983 Compare October 28, 2026 19:45
Address review: skip the history lookup for reports that change nothing,
use proto getters for services, trim service names after truncating and
test the pruner.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend enhancement New feature or request hub

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants