Skip to content

perf: speed up dense count agg with array counting - #559

Open
cheb0 wants to merge 2 commits into
0-materialized-column-field-iterfrom
0-single-source-count-agg-batched
Open

cheb0 wants to merge 2 commits into
0-materialized-column-field-iterfrom
0-single-source-count-agg-batched

Conversation

@cheb0

@cheb0 cheb0 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • Aggregator now accepts batches of lids, SourceNodeIterator rerturns a batch of sources
  • MIDs are extracted for a batch of lids
  • SingleSourceCountAggregator now delegates aggregation logic to sourceCounter interface, which has two implementations: plainSourceCounter handles ordinary aggs, uses a chunked array for counting. tsSourceCounter handles timeseries aggs, uses map

Array is better than map when we have many many lids processed in agg, but can be slower when the number of lids is low and agg is high cardinality (we need to allocate an array). This is partially handled by "chunking" the array.

Query env Total cold, ms   hot, ms   cold (branch), ms   hot (branch), ms   cold diff hot diff
service:large | by k8s_pod prod 112107 57.6 ±3.80 18 ±1.33 52.57 ±2.72 13.36 ±0.78 -8.7% -25.8%
service:small | by k8s_pod prod 3211 48.1 ±2.76 7.3 ±1.66 47.89 ±1.66 6.15 ±0.37 -0.4% -15.8%
service:micro | by k8s_pod prod 99 67.64 ±1.60 6.41 ±0.45 66.73 ±1.04 6.66 ±0.53 -1.3% 3.9%
service:large | by (k8s_pod, 60s) prod 112107 122.21 ±2.95 23.04 ±1.86 119.04 ±2.93 21.99 ±0.61 -2.6% -4.6%
service:large | by (level, 60s) prod 112107 106.44 ±1.95 22.83 ±0.20 103.82 ±1.30 21.42 ±0.28 -2.5% -6.2%
request_host:huge | by response_status lb 1065220 84.05 ±3.89 53.89 ±1.09 56.52 ±2.55 29.26 ±0.94 -32.8% -45.7%
request_host:huge | by request_method lb 1065220 78.11 ±2.80 47.77 ±0.72 55.14 ±1.53 28.18 ±0.72 -29.4% -41%
response_status:500 | by remote_addr lb 172 92.18 ±2.95 16.51 ±0.72 90.06 ±3.16 16.53 ±1.02 -2.3% 0.1%
exists:remote_addr | by remote_addr lb 2844073 2027.1 ±47.77 1860.21 ±25.41 1745.57 ±85.87 1538.06 ±90.97 -13.9% -17.3%
request_host:huge | request_uri lb 1065220 100.97 ±3.68 59.22 ±1.42 78.79 ±10.05 34.73 ±3.04 -22% -41.4%
request_host:huge | k8s_service_name lb 1065220 84.42 ±2.12 49.34 ±0.90 64.24 ±1.97 28.53 ±0.63 -23.9% -42.2%
request_host:huge | by x_o3_app_name lb 1065220 88.51 ±1.84 57.45 ±0.72 63.36 ±1.57 31.02 ±0.74 -28.4% -46%

  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

Stack created with GitHub Stacks CLI • Give Feedback 💬

@cheb0
cheb0 added this pull request to stack #515 October 2, 2026 11:01
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.36111% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.10%. Comparing base (02f1355) to head (8a910ee).

Files with missing lines Patch % Lines
frac/processor/aggregator.go 92.90% 10 Missing ⚠️
frac/processor/search.go 50.00% 1 Missing ⚠️
Additional details and impacted files
@@                         Coverage Diff                          @@
##           0-materialized-column-field-iter     #559      +/-   ##
====================================================================
+ Coverage                             77.03%   77.10%   +0.06%     
====================================================================
  Files                                   251      251              
  Lines                                 19104    19162      +58     
====================================================================
+ Hits                                  14716    14774      +58     
  Misses                                 4388     4388              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggDeep/size=10000-4 70af15 1faa84
0.00 B/op 3.00 B/op NaN 🔴
AggDeep/size=1000000-4 70af15 1faa84
0.00 B/op 30902.00 B/op NaN 🔴
AggWide/size=10000-4 70af15 1faa84
0.00 B/op 3.00 B/op NaN 🔴
AggWide/size=1000000-4 70af15 1faa84
842.00 B/op 34521.00 B/op 41.00 🔴

@cheb0 cheb0 changed the title perf: make aggregators batched perf: speed up dense count agg with array counting Oct 2, 2026
@eguguchkin eguguchkin added this to the v0.81.0 milestone Oct 5, 2026
@eguguchkin
eguguchkin requested review from eguguchkin and forshev and removed request for forshev October 5, 2026 10:28
@dkharms
dkharms self-requested a review October 5, 2026 10:38
@eguguchkin
eguguchkin removed their request for review October 5, 2026 10:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants