feat(memtrack): capture allocation stacks in eBPF by not-matthias · Pull Request #522 · CodSpeedHQ/codspeed

not-matthias · 2026-08-28T14:17:13Z

Adds raw allocation-stack capture to memtrack and ships everything needed for off-box unwinding.

At every allocator entry the eBPF program copies the caller's user stack (chunked, budget-capped, FNV-1a-hashed) and walks frame pointers in-kernel. Definitions are deduplicated by hash in-kernel: one StackDefinition event (raw bytes, registers, FP chain) per distinct stack, referenced by a stack_hash on allocation events. Everything flows through the ordinary MemtrackArtifact event stream; unwinding and symbolication happen server-side later.

After a capture-enabled run, the runner decodes the artifact and reuses the perf walltime machinery to dump per-module symbols, unwind data, and debug info into the profile folder, plus a memory_metadata.json with per-pid process names.

Refs COD-3222

codspeed-hq · 2026-08-28T14:22:14Z

Merging this PR will degrade performance by 27.26%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 4 regressed benchmarks
✅ 9 untouched benchmarks
🆕 14 new benchmarks
⏩ 4 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`encode_events_realistic[16]`	124.3 ms	203.3 ms	-38.85%
❌	WallTime	`encode_events_realistic[8]`	145.6 ms	216.2 ms	-32.67%
❌	WallTime	`encode_events_realistic[4]`	217.5 ms	286.6 ms	-24.13%
❌	Simulation	`write_events[100000]`	137.5 ms	153.4 ms	-10.37%
🆕	WallTime	`encode_events_realistic[1]`	N/A	742.9 ms	N/A
🆕	WallTime	`encode_events_realistic[2]`	N/A	434.3 ms	N/A
🆕	WallTime	`write_stack_events`	N/A	183.2 ms	N/A
🆕	Simulation	`encode_events_realistic[1]`	N/A	1.3 s	N/A
🆕	Simulation	`encode_events_realistic[2]`	N/A	1.3 s	N/A
🆕	Simulation	`write_stack_events`	N/A	405.5 ms	N/A
🆕	Memory	`encode_events_realistic[1]`	N/A	121.5 MB	N/A
🆕	Memory	`encode_events_realistic[16]`	N/A	125 MB	N/A
🆕	Memory	`encode_events_realistic[2]`	N/A	121.5 MB	N/A
🆕	Memory	`encode_events_realistic[4]`	N/A	121.5 MB	N/A
🆕	Memory	`encode_events_realistic[8]`	N/A	125.4 MB	N/A
🆕	Memory	`write_events[10000]`	N/A	2.1 MB	N/A
🆕	Memory	`write_events[100000]`	N/A	9.6 MB	N/A
🆕	Memory	`write_stack_events`	N/A	65.6 MB	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing cod-3222-add-ebpf-based-dwarffp-unwinding (ed10235) with main (03a0288)}

4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Copy the caller's user stack in chunks at allocator entry and fold an FNV-1a digest over it in the kernel. The digest rides on the allocation event as stack_hash; the copied bytes, a DWARF-numbered register snapshot and a frame-pointer walk are emitted once per distinct digest on a dedicated ring buffer, so unwinding and symbolication can happen offline. Capture stays off until userspace sets the rodata toggle, so allocator probes are unchanged by default. Refs COD-3222

Add the userspace half of allocation stack capture: env-driven configuration, stack-definition ring parsing, loss counters, per-pid module mapping tracking, a folding recorder that deduplicates definitions and counts occurrences, and the report it produces. Nothing constructs these yet; the tracker wiring follows. Refs COD-3222

Wire the capture rodata and map sizing into skeleton load, poll the stack-definition ring alongside the event ring, and expose the loss counters and frame-pointer chains. The attach worker snapshots module mappings while it holds a process stopped, which is the only point they are guaranteed readable. Guard the lifecycle: finishing with a live session would block forever on the recorder, and a second spawn would leave the capture rings undrained, so both now fail with a descriptive error. With capture disabled the ring buffer and frame-pointer map shrink to the allocator minimum rather than reserving tens of MiB. Refs COD-3222

Add a fixture with two non-inlinable malloc call paths and privileged tests over it: distinct call paths get distinct identities with module mappings for the binary and libc, repeated calls deduplicate, and the default-off path still reports allocations. Two cases guard failure modes the default budget cannot reach. The maximum copy budget is the only configuration that exercises the verifier's instruction limit, since the frozen rodata makes the copy and hash loops scale with the configured size. Shrinking the frame-pointer map to one slot proves exhaustion costs only the fallback chain, never an allocation event. Refs COD-3222

The symbol, unwind-data and debug-info extraction is not perf-specific: it turns a set of mapped ELF modules into the deduplicated keyed artifacts a profile references, whatever discovered the mappings. Memory mode needs the same pipeline, so it moves out of wall_time/profiler/perf into executor/shared/module_artifacts.

Memory mode needs the same per-pid module references walltime writes, so the five artifact fields move into a flattened `ModuleArtifacts` shared by both formats; walltime's JSON is unchanged, asserted against output captured from the flat struct. Flattening buffers those fields through serde's `Content`, which unlike serde_json's direct deserializer cannot parse a string JSON key into a pid, so pid-keyed maps get an explicit key-parsing helper.

Allocation stacks are raw addresses, so resolving them off-box needs the module geometry perf gets from PERF_RECORD_MMAP2. No single hook provides it: security_mmap_file has the file but runs before the VMA exists, and perf_event_mmap has the addresses but cannot resolve a path. So an LSM program caches the path once per inode and a perf_event_mmap fentry emits inode-keyed address records, joined in userspace while the maps are still live. Path resolution is only reachable from an LSM program at all, and only above 5.11 (bpf_d_path on the sleepable hook) or 6.12 (the bpf_path_d_path kfunc), with the bpf LSM active. MappingSupport probes both, and when neither holds stack capture is turned off rather than shipping stacks nothing can attribute.

Memory mode now turns the mappings memtrack recorded into the same keyed unwind_data/symbols.map files walltime writes, plus a memtrack.metadata referencing them per pid, so allocation stacks can be unwound off-box. Each mapping's inode is rechecked against the path before its ELF is read: BPF cannot produce a build id, so the recorded (dev, ino) is what proves the file on disk is still the one that was mapped rather than a rebuilt binary whose eh_frame would be bound to the wrong addresses.

Replace the BPF LSM path cache and mapping ring with inherited per-CPU PERF_RECORD_MMAP2 collectors. Store executable mappings as a terminal suffix in the main memtrack stream so existing timeline consumers remain compatible, then extract and order them in the runner before generating module artifacts.

Every frame's output buffer started empty and doubled its way to the compressed size, which for a 64k event frame is 8 realloc-and-copy steps over roughly 8 MB. Not measurable in wall clock at current frame sizes; it removes the copy traffic.

The memory instrument reports allocation counts and bytes per benchmark, which is what the encode path is bound by. It runs on a hosted runner like simulation does; the runner grants memtrack its capabilities during setup.

glibc exports cfree at the same file offset as free, so attaching both instrumented one function twice: every free() produced two Free events and two stack captures. Track (library, offset) pairs and skip symbols already covered by an alias.

…tation mimalloc lowers memtrack's own memory usage, fragmentation and allocation overhead compared to glibc's allocator. As a side effect, it also doesn't route through the exported malloc/free/calloc/realloc symbols, so it skips the allocator uprobes (attached system-wide with pid -1) that would otherwise fire for memtrack's own bookkeeping allocations. Pulled in via the ebpf feature, which the binary already requires.

Each stack record needs its frame-pointer chain looked up in the stack_traces map, which is a syscall per record. Doing that inside the ring-buffer parse callback made the poll thread pay it, so a burst of stacks could push it behind the producer and records were dropped. ResolvingPoller wraps a RingBufferPoller with a dedicated resolver thread: the poll thread only parses (event, stackid) and hands it over an internal channel, and the resolver does the map lookup and forwards the completed event. Drop order keeps the existing shutdown contract, the ring is dropped first so its poll thread joins and closes the internal sender, which lets the resolver drain what it already has before its join returns.

greptile-apps · 2026-09-04T12:33:08Z

Greptile Summary

This PR adds eBPF allocation-stack capture, embeds stack definitions and mapping records in the memtrack artifact stream, collects executable mappings through perf events, and reuses wall-time module extraction to produce offline symbol and unwind metadata.

Captures raw stack bytes, registers, frame-pointer chains, and hashes at allocator probes.
Extends the shared memtrack event and metadata contracts with stack and module-placement data.
Adds per-CPU perf mapping collection and memory-mode module artifact generation.
Refactors wall-time ELF artifact extraction into shared infrastructure.
Adds stack-capture integration tests and memory encoder benchmarks.

Confidence Score: 2/5

The PR is not safe to merge until token-only memory runs remain startable, stack events have a compatible rollout strategy, and repeated module placements are retained for offline unwinding.

Three concrete failures remain: default mapping collection can reject an otherwise supported delegated-token environment, interleaved stack records silently truncate older artifact readers, and repeated mappings of one module overwrite the placement data needed to unwind earlier stacks.

Files Needing Attention: crates/memtrack/src/ebpf/tracker.rs, crates/runner-shared/src/artifacts/memtrack/mod.rs, src/executor/memory/module_artifacts.rs

Important Files Changed

Filename	Overview
crates/memtrack/src/ebpf/c/stack_capture.bpf.h	Implements bounded raw-stack copying, hashing, deduplication, register capture, and frame-pointer fallback in eBPF.
crates/memtrack/src/ebpf/tracker.rs	Wires stack and perf mapping pollers into tracked-process startup, but makes perf-event permission mandatory for a privilege mode that does not guarantee it.
crates/memtrack/src/perf_mappings.rs	Adds native per-CPU perf-ring collection and parsing of executable `MMAP2` records.
crates/runner-shared/src/artifacts/memtrack/mod.rs	Extends the serialized event contract with stack hashes, definitions, and mappings without preserving forward compatibility for older streamed readers.
src/executor/memory/module_artifacts.rs	Builds offline module artifacts from mapping events but collapses repeated placements of one path in a process.
crates/runner-shared/src/metadata.rs	Extracts common module metadata and introduces the versioned memtrack metadata envelope.
src/executor/memory/executor.rs	Runs module extraction after successful memory capture while retaining the existing privilege acceptance paths.
src/executor/shared/module_artifacts/save_artifacts.rs	Refactors wall-time symbol, debug, and unwind persistence into shared module-artifact output.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Allocator entry uprobe] --> B[Copy and hash user stack]
    B --> C[Stack definition ring]
    A --> D[Allocator return event]
    D --> E[Allocation event ring]
    C --> F[Stack resolver thread]
    E --> G[Shared event channel]
    F --> G
    H[PERF_RECORD_MMAP2 rings] --> I[Buffered mapping records]
    G --> J[Ordered MessagePack and zstd artifact]
    I -->|terminal suffix| J
    J --> K[Memory module extraction]
    K --> L[Symbols and unwind files]
    K --> M[memtrack.metadata]

Prompt To Fix All With AI

### Issue 1
crates/memtrack/src/ebpf/tracker.rs:153-155
**Token privileges cannot start mapping**

Default stack capture now makes `PerfMappingPoller::start` mandatory. If a host relies only on a delegated BPF token, that token authorizes BPF operations but does not grant permission for `perf_event_open`. With restrictive `perf_event_paranoid` and no `CAP_PERFMON`, the accepted privilege check succeeds but spawning the memory benchmark fails before the target resumes. Please validate or provide perf-event permission for this supported privilege mode, or avoid making mapping collection fatal there.

### Issue 2
crates/runner-shared/src/artifacts/memtrack/mod.rs:115-119
**Stack events truncate older readers**

`Stack` records are emitted throughout the normal event stream, but older `MemtrackEventKind` readers cannot deserialize this new tag. Their streamed iterator treats the first decode error as EOF, so during a rolling runner/backend deployment the first captured stack silently truncates the remaining allocation timeline. Unlike terminal `Mapping` records, these interleaved events do not preserve compatibility. This representation needs an explicit compatibility or versioning strategy so unknown stack definitions do not terminate older readers.

### Issue 3
src/executor/memory/module_artifacts.rs:223-230
**Repeated mappings lose earlier placement**

Module placement is stored in one `ProcessLoadedModule` per `(path, pid)`, so a second mapping of the same ELF overwrites the first mapping's load bias, address range, and timestamp. If a process unloads and later reloads a shared object, possibly at another address, metadata retains only the final placement. Stacks captured against the earlier placement then cannot be unwound or symbolicated correctly. Preserve every time- and range-specific placement rather than replacing the previous one here.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "perf(memtrack): resolve stack fp chains ..." | Re-trigger Greptile}

greptile-apps · 2026-09-04T12:33:11Z

+            let perf_mapping_poller = capture_stacks
+                .then(|| PerfMappingPoller::start(pid, tx, self.mapping_lost.clone()))
+                .transpose()?;


Token privileges cannot start mapping

Default stack capture now makes PerfMappingPoller::start mandatory. If a host relies only on a delegated BPF token, that token authorizes BPF operations but does not grant permission for perf_event_open. With restrictive perf_event_paranoid and no CAP_PERFMON, the accepted privilege check succeeds but spawning the memory benchmark fails before the target resumes. Please validate or provide perf-event permission for this supported privilege mode, or avoid making mapping collection fatal there.

Knowledge Base Used: Memory benchmarking

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/memtrack/src/ebpf/tracker.rs Line: 153-155 Comment: **Token privileges cannot start mapping** Default stack capture now makes `PerfMappingPoller::start` mandatory. If a host relies only on a delegated BPF token, that token authorizes BPF operations but does not grant permission for `perf_event_open`. With restrictive `perf_event_paranoid` and no `CAP_PERFMON`, the accepted privilege check succeeds but spawning the memory benchmark fails before the target resumes. Please validate or provide perf-event permission for this supported privilege mode, or avoid making mapping collection fatal there. **Knowledge Base Used:** [Memory benchmarking](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/memory-benchmarking.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-04T12:33:11Z

+    Stack {
+        // Box keeps the MemtrackEventKind enum small across millions of events.
+        #[serde(flatten)]
+        record: Box<StackRecord>,
+    },


Stack events truncate older readers

Stack records are emitted throughout the normal event stream, but older MemtrackEventKind readers cannot deserialize this new tag. Their streamed iterator treats the first decode error as EOF, so during a rolling runner/backend deployment the first captured stack silently truncates the remaining allocation timeline. Unlike terminal Mapping records, these interleaved events do not preserve compatibility. This representation needs an explicit compatibility or versioning strategy so unknown stack definitions do not terminate older readers.

Knowledge Base Used:

eBPF memory tracker

Memory benchmarking

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/runner-shared/src/artifacts/memtrack/mod.rs Line: 115-119 Comment: **Stack events truncate older readers** `Stack` records are emitted throughout the normal event stream, but older `MemtrackEventKind` readers cannot deserialize this new tag. Their streamed iterator treats the first decode error as EOF, so during a rolling runner/backend deployment the first captured stack silently truncates the remaining allocation timeline. Unlike terminal `Mapping` records, these interleaved events do not preserve compatibility. This representation needs an explicit compatibility or versioning strategy so unknown stack definitions do not terminate older readers. **Knowledge Base Used:** - [eBPF memory tracker](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/ebpf-memory-tracker.md) - [Memory benchmarking](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/memory-benchmarking.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-04T12:33:11Z

+        let process_loaded_module = loaded_module
+            .process_loaded_modules
+            .entry(mapping.pid)
+            .or_default();
+        process_loaded_module.symbols_load_bias = Some(load_bias);
+
+        if let Some(process_unwind_data) = process_unwind_data {
+            process_loaded_module.process_unwind_data = Some(process_unwind_data);


Repeated mappings lose earlier placement

Module placement is stored in one ProcessLoadedModule per (path, pid), so a second mapping of the same ELF overwrites the first mapping's load bias, address range, and timestamp. If a process unloads and later reloads a shared object, possibly at another address, metadata retains only the final placement. Stacks captured against the earlier placement then cannot be unwound or symbolicated correctly. Preserve every time- and range-specific placement rather than replacing the previous one here.

Knowledge Base Used: Wall-time profiling pipeline

Prompt To Fix With AI

This is a comment left during a code review. Path: src/executor/memory/module_artifacts.rs Line: 223-230 Comment: **Repeated mappings lose earlier placement** Module placement is stored in one `ProcessLoadedModule` per `(path, pid)`, so a second mapping of the same ELF overwrites the first mapping's load bias, address range, and timestamp. If a process unloads and later reloads a shared object, possibly at another address, metadata retains only the final placement. Stacks captured against the earlier placement then cannot be unwound or symbolicated correctly. Preserve every time- and range-specific placement rather than replacing the previous one here. **Knowledge Base Used:** [Wall-time profiling pipeline](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/wall-time-profiling.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

not-matthias force-pushed the cod-3222-add-ebpf-based-dwarffp-unwinding branch 8 times, most recently from 9576fd5 to bcf9cc1 Compare September 4, 2026 10:21

not-matthias added 14 commits September 4, 2026 13:08

docs(memtrack): describe the mapping recorder

3c5b5a2

test(memtrack): add nested allocation fixtures

1ef91af

test(memtrack): assert nested stack identities

3f30d71

ci: benchmark runner-shared in memory mode

5edcca0

The memory instrument reports allocation counts and bytes per benchmark, which is what the encode path is bound by. It runs on a hosted runner like simulation does; the runner grants memtrack its capabilities during setup.

not-matthias force-pushed the cod-3222-add-ebpf-based-dwarffp-unwinding branch from bcf9cc1 to 0f7b27c Compare September 4, 2026 11:09

not-matthias added 2 commits September 4, 2026 13:21

refactor(runner-shared): inline pid-key deserializer into metadata

9086ded

not-matthias force-pushed the cod-3222-add-ebpf-based-dwarffp-unwinding branch from 0f7b27c to d99e3bf Compare September 4, 2026 11:21

not-matthias added 2 commits September 4, 2026 13:31

not-matthias force-pushed the cod-3222-add-ebpf-based-dwarffp-unwinding branch from d99e3bf to ed10235 Compare September 4, 2026 11:31

not-matthias marked this pull request as ready for review September 4, 2026 12:27

not-matthias requested a review from GuillaumeLagrange September 4, 2026 12:27

greptile-apps Bot reviewed Sep 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(memtrack): capture allocation stacks in eBPF - #522

feat(memtrack): capture allocation stacks in eBPF#522
not-matthias wants to merge 18 commits into
mainfrom
cod-3222-add-ebpf-based-dwarffp-unwinding

not-matthias commented Aug 28, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 4, 2026

Important Files Changed

Flowchart

Uh oh!

greptile-apps Bot Sep 4, 2026

Uh oh!

greptile-apps Bot Sep 4, 2026

Uh oh!

greptile-apps Bot Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

not-matthias commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 27.26%

Performance Changes

Footnotes

Uh oh!

greptile-apps Bot commented Sep 4, 2026

Greptile Summary

Confidence Score: 2/5

Important Files Changed

Flowchart

Uh oh!

greptile-apps Bot Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

not-matthias commented Aug 28, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 28, 2026 •

edited

Loading