Repository navigation
Conversation
…n (t, t+1s] Remote write sent one sub-second slot at a time, so at most one request was in flight per slot. At 200 samples/s per series ASAPQuery's ingest took about 45 ms per 1e4-row request and the feeder reached only 0.2-0.27M rows/s. Each tick now goes out concurrently. Its samples sit in (t, t+1s], the same left-open interval as ASAPQuery's one-second windows, so the reordering within a tick never crosses a window boundary and stays within the allowed lateness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
milindsrivastava1997
self-requested a review
October 13, 2026 01:44
milindsrivastava1997
previously approved these changes
Oct 13, 2026
This was referenced Oct 13, 2026
…equest Concurrent requests split a tick by time slot, so a series' later sample could arrive first and ASAPQuery's increase accumulator, which reads samples in arrival order, took the lower counter value for a reset. A tick's rows are now series-major and requests are cut only between series (the batch size rounded down to whole series, at least one), so each series' samples of a tick arrive in one request in timestamp order; requests stay concurrent. Prometheus therefore needs no out-of-order window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The paced feeder (#793, #835) sent remote write one sub-second slot at a time, so a group's samples reached ASAPQuery in timestamp order. With few rows per slot that leaves one request in flight. At 200 samples/s per series (the #777 e2e workload) ASAPQuery's ingest took about 45 ms per 1e4-row request, and the feeder reached only 0.2–0.27M of the 2M rows/s target.
This PR sends each tick concurrently again, and moves its samples to
(t, t+1s]instead of[t, t+1s):(start, end], and a window closes as soon as the group's watermark passes its end. A tick starting at a whole secondtput slot 0 exactly att, in the previous window. If a later slot of the tick arrived first, that window closed and slot 0 was dropped as late.IncreaseAccumulatorread a later sample arriving first as a counter reset andratecame out 150–520× too high. It also means Prometheus' remote-write receiver needs no out-of-order window. What still reorders within a tick is samples of different series in a group, which is the safe case above.Every sample's timestamp moves one slot later (
+1000/samples_per_secms); values are unchanged.Testing
On CloudLab (clnode109), with
precompute_enginefrommain:400 out of order sample; series-major (second commit): accepted without one, see below|
rate(data_total[w]), ASAPQuery (--planner savedconfigs) vs ClickHouse, 1e4 series × 10 samples/s, concurrency 16, 19 min | slot-ordered batches: 150–520× too high | series-major: worst relative difference 1.06e-10 (40,000 keys, 4 windows) || Prometheus 2.53 receiver with no out-of-order window, 200,000 rows/s, 4 min | — | 48M samples appended, 0 out-of-order/out-of-bound/too-old rejections, every series complete |
The "before" drop count comes from the pre-#793-review feeder, which sent ticks concurrently with slot 0 on the whole second.
cargo clippy --all-targets -- -D warnings,cargo fmt --checkand the unit tests pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae