Type of problem
CI OOM — with GC at the product default of 60s, ASan CAS-over-S3 stateless shards finish under 6h but the shared server hits the ~16.5 GiB memory cap. OvercommitTracker then kills unrelated tests.
Summary
Follow-up to the 5s-interval timeout (#2298). A throwaway PR that only changed the test disk from 5s to 60s still failed:
PR #2296 (cas_gc_interval_sec 60, cas_gc_enabled 1)
| Shard |
Wall time |
FAIL |
BROKEN |
amd_asan_ubsan, cas s3 storage, parallel, 1/2 |
3h14m |
14 |
8 |
amd_asan_ubsan, cas s3 storage, parallel, 2/2 |
2h59m |
10 |
1 |
Most FAILs are Code: 241 MEMORY_LIMIT_EXCEEDED with RSS ~15–16 GiB, clustered in time (shard 1 ~05:59–06:00 UTC+5, shard 2 ~05:14). Examples: 01073_bad_alter_partition, 01031_mutations_interpreter_and_context, 01455_opentelemetry_distributed, 05025_cas_attach_partition_cross_disk, 02841_parallel_replicas_summary.
This is the product default (ContentAddressedSettings: gc_interval_sec 60). The test policy is in tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml. The disk is uncached cas_s3; system logs sit on that policy.
Control
PR #2297 (cas_gc_enabled 0, same ASan CAS-S3 shards):
| Shard |
Wall time |
FAIL |
MEMORY_LIMIT_EXCEEDED |
| 1/2 |
2h07m |
2 |
0 |
| 2/2 |
2h51m |
0 (pass) |
0 |
The two remaining FAILs on GC-off 1/2 are not GC: 00172_early_constant_folding (random-settings EXPLAIN PIPELINE width vs reference) and 01010_partial_merge_join (RustFS Connection refused on cas_s3/cas/ref_catalog).
Likely fix direction
GC at 60s is still too expensive under ASan + uncached CAS-S3 + this runner memory cap. Options: lengthen or disable background GC for the full sanitizer stateless lane (keep it in a dedicated GC test), put a cache in front of cas_s3, or pin system logs off CAS. Do not paper over it with sleep.
Links
Related: #2298
Related: #2296
Related: #2297
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793
Project Antalya Build Version: 26.6.2.20001.altinityantalya
Type of problem
CI OOM — with GC at the product default of 60s, ASan CAS-over-S3 stateless shards finish under 6h but the shared server hits the ~16.5 GiB memory cap. OvercommitTracker then kills unrelated tests.
Summary
Follow-up to the 5s-interval timeout (#2298). A throwaway PR that only changed the test disk from 5s to 60s still failed:
PR #2296 (
cas_gc_interval_sec60,cas_gc_enabled1)amd_asan_ubsan, cas s3 storage, parallel, 1/2amd_asan_ubsan, cas s3 storage, parallel, 2/2Most FAILs are
Code: 241 MEMORY_LIMIT_EXCEEDEDwith RSS ~15–16 GiB, clustered in time (shard 1 ~05:59–06:00 UTC+5, shard 2 ~05:14). Examples:01073_bad_alter_partition,01031_mutations_interpreter_and_context,01455_opentelemetry_distributed,05025_cas_attach_partition_cross_disk,02841_parallel_replicas_summary.This is the product default (
ContentAddressedSettings:gc_interval_sec60). The test policy is intests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml. The disk is uncachedcas_s3; system logs sit on that policy.Control
PR #2297 (
cas_gc_enabled0, same ASan CAS-S3 shards):MEMORY_LIMIT_EXCEEDEDThe two remaining FAILs on GC-off 1/2 are not GC:
00172_early_constant_folding(random-settingsEXPLAIN PIPELINEwidth vs reference) and01010_partial_merge_join(RustFSConnection refusedoncas_s3/cas/ref_catalog).Likely fix direction
GC at 60s is still too expensive under ASan + uncached CAS-S3 + this runner memory cap. Options: lengthen or disable background GC for the full sanitizer stateless lane (keep it in a dedicated GC test), put a cache in front of
cas_s3, or pin system logs off CAS. Do not paper over it withsleep.Links
Related: #2298
Related: #2296
Related: #2297
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793
Project Antalya Build Version: 26.6.2.20001.altinityantalya