Repository navigation
telemetry: turnaways and retried conflicts are not errors - #54
Merged
Merged
Conversation
A create or group turned away until capacity exists now counts as status "deferred", and a routing transaction rolled back by a serialization conflict (and retried) as "conflict". Both were counted and traced as errors: a cold start logged thousands of each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cold start on prod logged ~2,100 `gateway.sandbox_create` errors and ~27,000 `routing.transaction` errors in 15 minutes, while clients saw 2 503s against ~12,700 successful creates. They were:
These now count as `status="deferred"` and `status="conflict"` in `ucloud_platform_operation_count_total`. Their spans are not marked as errors; they carry a `ucloud.outcome` attribute instead. A later 5xx response or the raised conflict does not override the outcome. A transaction that runs out of retries still fails its caller as `error`. The dashboard's failure panels filter on `error`, so they now show only real failures.
Test: `tests/test_telemetry.py::test_turnaways_and_conflicts_are_counted_apart_from_errors` (fails without the change). The full suite with Postgres passes.
🤖 Generated with Claude Code