Repository navigation
fix(asap-tools): keep 10 s more engine history for late query sends - #893
Merged
Merged
Conversation
The query engine's retention slack covered the client's query_time_offset plus one ingestion interval, with no room for the client sending late. When a 60s query group fires alongside a 1s group, the 1s queries go out ~2-3 s behind schedule, and a 1s-window instant query then found its window already evicted (7 of 300 evaluations in a classic sketch-bench smoke run). Add CLIENT_SEND_LAG_SLACK_MS (10 s) to get_retention_slack_ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
CloudLab verification (nodes 2–3, classic sketch-bench table, 1 sample/s per series, 20 min starting delay, 300 s of queries, ASAPQuery only):
|
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this PR
get_retention_slack_msgave the query engine extra history equal to the client's largestquery_time_offsetplus one ingestion interval: 11 s for the sketch-bench e2e configs (offset 10 s, 1 s interval). That leaves no room for the client sending late.In a classic sketch-bench smoke run (#824), every minute the 60s query group (46 queries, several returning 10,000 series) fires alongside the 1s group. The 1s queries then go out 2–3 s behind schedule, about 12.1–12.5 s after their evaluation time. With 1 planned window + 11 s slack, the 1s DDSketch kept 12 windows (
time_map_len=12), so the window was already evicted. The store lookup returned1 windows requested, 0 matched, and 7 of 300 instant-quantile evaluations came back empty (destination=none_unsupported).After this PR
CLIENT_SEND_LAG_SLACK_MS = 10_000is added to the slack, so a query sent up to ~10 s behind schedule still finds its window. It applies to every runner that usesget_retention_slack_ms(e2e, ClickHouse, Grafana demo).Validation
tests/test_retention_slack.py: updated expectations, plus a regression test for a query 12.4 s past its evaluation time with a 10 s offset. It fails on the old slack.🤖 Generated with Claude Code