[vkarak]

Analytics: a new results backend

This adds analytics, a results backend that records every session into a
columnar database and queries it back with SQL, and deprecates the SQLite
result storage it supersedes. The two are mutually exclusive and analytics is
off by default, so nothing changes for existing users until they opt in.

# enable it, then record as usual
export RFM_ANALYTICS_ENABLE=1
reframe -c checks/ -r

# query it back
reframe analyze --list-queries
reframe analyze --query perf-stats agg=p95
reframe analyze --query perf-compare by=basename lhs=... rhs=...
reframe analyze --import run-report.json

What it changes for users

1. Results live in one flat table meant to be queried directly, instead of
   being reachable only through a fixed set of command-line options.
2. Queries are query templates -- parameterised SQL files that ship with
   ReFrame, that users can extend with their own and that resolve by name or
   by path.
3. Queries are considerably faster; see the benchmark below.

Five templates ship in share/reframe/queries/duckdb/: perf,
perf-stats, perf-compare, perf-report and ref-suggestion.
--performance-report now takes a show/compare[:<agg>[:<N>d]] spec that
maps onto the perf-report template.

Queries run read-only on a connection that cannot modify the database or
reach the file system, and template parameters are passed as bind parameters,
so their values never become part of the SQL text.

Benchmark

1,017k rows / 64k sessions of synthetic results, written into both backends in
the same pass so they hold identical data: 1.35 GB as analytics, 2.29 GB as
result storage.

uv run python tools/gen_analytics_data.py -n 64k \
    --duckdb duckdb:///path/to/bench.duckdb --sqlite /path/to/bench.db

Six queries, each expressed for both backends where the deprecated query
syntax can express it at all. The window is a full month (45,788 rows); Q5
takes no time period by design and runs over everything.

Timings wrap only the query call, so interpreter and ReFrame start-up are
excluded. cold purges the page cache before every measurement and runs
each in a fresh process; hot is the best of 3 after a warm-up. Anything
unfinished after 60 s was cut off.

Query	cold: analytics	cold: storage		hot: analytics	hot: storage
Q1 average performance per test case over a period	223 ms	5,541 ms	25x	24 ms	3,181 ms	133x
Q2 as Q1, for one session attribute, listing the results seen	249 ms	4,827 ms	19x	55 ms	3,080 ms	56x
Q3 one test family, broken down by its parameters	327 ms	5,082 ms	16x	118 ms	2,529 ms	21x
Q4 as Q3, plus a second performance variable of the same test case	332 ms	not expressible	--	117 ms	not expressible	--
Q5 compare two sets of results over the whole data set	617 ms	>60 s	>97x	376 ms	>60 s	>160x
Q6 compare two subsets of one test family	332 ms	1,220 ms	3.7x	132 ms	299 ms	2.3x

Row counts agree in every case except Q3, where the deprecated syntax cannot
restrict to a single performance variable and so returns 12 rows where the
template returns 6, and Q5, which was cut off.

Why the spread is so wide, since a single number would be misleading:

• The result storage keeps each session as one JSON document and does the
  filtering and aggregating in Python after decoding it. Analytics stores typed
  columns and lets the database do that work.
• Q6 is the floor at 2.3x because its deprecated form can prune by test
  name in SQL before any JSON is decoded, and that family runs in few sessions.
  Q1 and Q2 have no such filter, so every session of the month is decoded.
• Analytics loses relatively more to a cold cache (9x on Q1) than the result
  storage does (1.7x), which is CPU-bound on JSON decoding either way. The
  absolute gap stays an order of magnitude.

The six queries

Ad-hoc template files written for the benchmark rather than the shipped
templates, so that each one has a counterpart to compare against. The
analytics window is June 2026 shifted by the local UTC offset, since the
template bounds are cast to a naive timestamp that the database reads as UTC
whereas the deprecated store parses %Y%m%d as local time; both backends then
select exactly the same 45,788 rows.

Elided below: analytics takes the database as --database=duckdb://<path>,
while the deprecated store reads it from the configuration and so needs
RFM_ENABLE_RESULTS_STORAGE=1 and RFM_SQLITE_DB_FILE=<path>.

Q1 -- average performance per test case over a period

reframe analyze --query q1.sql \
    start=2026-05-31T22:00:00 end=2026-06-30T22:00:00

reframe --list-stored-testcases='20260601:20260701/mean:/'

select name, sysenv, pvar, avg(pval) as "pval (mean)", punit
from {{ table }}
where job_completion_time_us >= epoch_us(CAST($start AS TIMESTAMP))
  and job_completion_time_us <  epoch_us(CAST($end AS TIMESTAMP))
  and pval is not null
group by name, sysenv, pvar, punit
order by name, pvar

Q2 -- as Q1, for one session attribute, listing the results seen

reframe analyze --query q2.sql osver=1.0 \
    start=2026-05-31T22:00:00 end=2026-06-30T22:00:00

reframe --list-stored-testcases='20260601:20260701?os_version=="1.0"/mean:/+presult'

select name, sysenv, pvar, avg(pval) as pval_avg, punit,
       string_agg(distinct presult, '|') as results
from {{ table }}
where job_completion_time_us >= epoch_us(CAST($start AS TIMESTAMP))
  and job_completion_time_us <  epoch_us(CAST($end AS TIMESTAMP))
  and sess_data['os_version'] = $osver
  and pval is not null
group by name, sysenv, pvar, punit
order by name, pvar

Q3 -- one test family, broken down by its parameters

reframe analyze --query q3.sql test=GEMMTest pvar=tflops \
    start=2026-05-31T22:00:00 end=2026-06-30T22:00:00

reframe --name=GEMMTest \
    --list-stored-testcases='20260601:20260701/mean:basename,dtype,mode,pvar,punit/'

select basename, tc_data['dtype'] as dtype, tc_data['mode'] as mode,
       pvar, avg(pval) as pval_avg, punit
from {{ table }}
where basename = $test and pvar = $pvar
  and job_completion_time_us >= epoch_us(CAST($start AS TIMESTAMP))
  and job_completion_time_us <  epoch_us(CAST($end AS TIMESTAMP))
group by basename, dtype, mode, pvar, punit
order by dtype, mode

The deprecated form cannot restrict to one performance variable, since
--filter-expr runs before a test case is expanded into one row per
performance variable. It therefore aggregates all of them and returns 12 rows
where the template returns 6; the 6 tflops rows agree exactly.

Q4 -- as Q3, plus a second performance variable of the same test case

reframe analyze --query q4.sql test=GEMMTest pvar=tflops \
    start=2026-05-31T22:00:00 end=2026-06-30T22:00:00

select basename, tc_data['dtype'] as dtype, tc_data['mode'] as mode,
       pvar, avg(pval) as pval_avg, punit,
       avg(tc_data['$power']::DOUBLE) as power_w
from {{ table }}
where basename = $test and pvar = $pvar
  and job_completion_time_us >= epoch_us(CAST($start AS TIMESTAMP))
  and job_completion_time_us <  epoch_us(CAST($end AS TIMESTAMP))
group by basename, dtype, mode, pvar, punit
order by dtype, mode

Not expressible with the deprecated syntax: a row carries only its own
performance variable, so a tflops row has no way to reach the power of the
same test case.

Q5 -- compare two sets of results over the whole data set

reframe analyze --query q5.sql osver_lhs=1.0 osver_rhs=3.0

reframe --term-lhs=3.0 --term-rhs=1.0 \
    --performance-compare='?os_version=="3.0"/?os_version=="1.0"/mean:basename,sysenv,pvar,punit/'

with lhs as (
  select basename, sysenv, pvar, punit, avg(pval) as pval_avg
  from {{ table }}
  where sess_data['os_version'] = $osver_lhs and pval is not null
  group by basename, sysenv, pvar, punit
), rhs as (
  select basename, sysenv, pvar, punit, avg(pval) as pval_avg
  from {{ table }}
  where sess_data['os_version'] = $osver_rhs and pval is not null
  group by basename, sysenv, pvar, punit
)
select lhs.basename, lhs.sysenv, lhs.pvar, lhs.punit,
       lhs.pval_avg as {{ ident(osver_lhs) }},
       rhs.pval_avg as {{ ident(osver_rhs) }},
       round(100.0*(rhs.pval_avg - lhs.pval_avg)/lhs.pval_avg, 2) as "pdiff (%)"
from lhs join rhs using (basename, sysenv, pvar, punit)
order by lhs.basename, lhs.pvar

The two backends define the difference in opposite directions, hence the
reversed order of the two comparison subspecs; with that order the percentages
agree exactly.

Q6 -- compare two subsets of one test family

reframe analyze --query q6.sql test=AllreduceTest \
    block_lhs=block01 block_rhs=block02 \
    start=2026-05-31T22:00:00 end=2026-06-30T22:00:00

reframe --name=AllreduceTest \
    -E 'topology_block=="block02"' -E 'topology_block=="block01"' \
    --term-lhs=block02 --term-rhs=block01 \
    --performance-compare='20260601:20260701/20260601:20260701/mean:basename,nodes,pvar,punit/'

with lhs as (
  select basename, tc_data['nodes'] as nodes, pvar, punit,
         avg(pval) as pval_avg
  from {{ table }}
  where basename = $test and tc_data['topology_block'] = $block_lhs
    and pval is not null
    and job_completion_time_us >= epoch_us(CAST($start AS TIMESTAMP))
    and job_completion_time_us <  epoch_us(CAST($end AS TIMESTAMP))
  group by basename, nodes, pvar, punit
), rhs as (
  select basename, tc_data['nodes'] as nodes, pvar, punit,
         avg(pval) as pval_avg
  from {{ table }}
  where basename = $test and tc_data['topology_block'] = $block_rhs
    and pval is not null
    and job_completion_time_us >= epoch_us(CAST($start AS TIMESTAMP))
    and job_completion_time_us <  epoch_us(CAST($end AS TIMESTAMP))
  group by basename, nodes, pvar, punit
)
select lhs.basename, lhs.nodes, lhs.pvar, lhs.punit,
       lhs.pval_avg as {{ ident(block_lhs) }},
       rhs.pval_avg as {{ ident(block_rhs) }},
       round(100.0*(rhs.pval_avg - lhs.pval_avg)/lhs.pval_avg, 2) as "pdiff (%)"
from lhs join rhs using (basename, nodes, pvar, punit)
order by lhs.pvar, lhs.nodes

Test system

Machine	MacBook Pro (MacBookPro15,2)
CPU	Intel Core i5-8259U @ 2.30 GHz, 4 cores / 8 threads, 6 MB L3
Memory	16 GB
Storage	internal SSD, APFS
OS	macOS 14.8.9 (23J631), x86_64
Python	3.14.7
DuckDB	1.5.6
SQLite	3.53.4
polars / pyarrow	1.41.2 / 25.0.1
Measured at	1c1ddf4e

Single-laptop numbers on a synthetic data set: enough to show the shape of the
difference, not a claim about any particular production database.

Installation

Analytics needs duckdb and pyarrow, which come with a new extra:

pip install 'reframe-hpc[analytics]'

polars is no longer a default dependency, since it serves only the
deprecated result storage, which now has an opt-in extra of its own:

pip install 'reframe-hpc[analytics-legacy]'

The no-analytics extra is removed. It existed to exclude polars, which
pip cannot do for a default dependency, so it came with a documented
work-around; making polars opt-in removes both. The error message that the
deprecated options already print when polars is missing names the new extra.

Configuration

A new analytics configuration section with enable, database,
query_search_path, target_systems and the write_retries_max /
write_retry_interval_base / write_retry_interval_max trio, plus matching
RFM_ANALYTICS_* environment variables. The general.perf_report_spec option
is documented for the first time.

Exit codes

Rebased onto the exit-code convention of #3745, which analyze follows: 0
on success, 1 on a generic error and 2 on command-line misuse, such as a
malformed query parameter or no action at all. The statuses that report test
failures, 3 and 4, cannot occur there, since analyze runs no tests. Two
cases of the main command move from 1 to 2, as incompatible option
combinations: enabling both backends at once, and a malformed
--performance-report spec.

Deprecations

The SQLite result storage and everything around it are deprecated, not
removed: the storage.* configuration options, the RFM_SQLITE_* and
RFM_ENABLE_RESULTS_STORAGE environment variables, and the
--list-stored-sessions, --list-stored-testcases,
--describe-stored-sessions, --describe-stored-testcases,
--delete-stored-sessions and --performance-compare options. Enabling both
backends at once is an error.

Existing JSON run reports of a supported data version can be loaded with
reframe analyze --import.

Documentation

A new reframe-analyze(1) man page, generated from docs/analytics.rst and
shipped in the wheel, covering the command, the templates, the table schema
and the configuration. The tutorial gains a parallel section, and the
deprecated material is marked in place rather than deleted. CI now asserts
that the man page and the query templates are present in the built wheel.

Two fixes in the installation section, found while documenting the extras: an
extra cannot be attached to a plain git+https URL, so the dev-release
examples now show the direct-reference form that can carry one; and
uv tool install reframe-hpc --extra graylog, which that section documented,
has never worked -- uv tool install has no --extra flag, only the brackets
notation.

Tests

unittests/test_analytics.py adds 120 tests covering the schema, the
flattening, the template sandbox, the read-only query gate and the backend;
unittests/test_cli.py adds coverage for the new command. Note that
duckdb and pyarrow had to be added to the dev dependency group: the test
module guards on them at module scope, so without them the whole file was
skipped silently and the suite had no coverage at all.

Closes #3638.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.96067% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.02%. Comparing base (2355c3d) to head (93d21d9).

Files with missing lines	Patch %	Lines
reframe/frontend/analyze.py	82.45%	30 Missing ⚠️
reframe/analytics/backends/duckdb.py	92.00%	6 Missing ⚠️
reframe/analytics/queries.py	96.44%	6 Missing ⚠️
reframe/analytics/__init__.py	98.52%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3747      +/-   ##
===========================================
+ Coverage    91.90%   92.02%   +0.11%     
===========================================
  Files           63       71       +8     
  Lines        13943    14649     +706     
===========================================
+ Hits         12815    13481     +666     
- Misses        1128     1168      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[victorusu]

lgtm
You are going to kill me with these long PRs! 🤣🤣🤣🤣

[vkarak]

lgtm You are going to kill me with these long PRs! 🤣🤣🤣🤣

It is admittedly quite long but I don't think it can be split in standalone parts. The only preliminary work that could stand by its own was the update of the report schema in #3722 which was a prerequisite for this one.