Repository navigation
[APMSVLS-469] Rate-limit error trace rescue with DD_APM_ERROR_TPS - #1490
lucaspimentel wants to merge 15 commits into
Conversation
lucaspimentel
changed the title
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@DataDog review |
31b98de to
e6e7837
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31b98deb11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![datadog-prod-us1-3[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
Configuration precedence, disabling rules, and startup wiring are consistent with the documented rescue-budget behavior. Runtime sampler behavior remains unverified.
🤖 Bits Code Review · Commit 31b98de
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent batches crossing a 5-second boundary can reset live sampler counts and distort rescue rates, while the freeze/thaw regression test does not verify that rescue resumes after a gap.
2 open issues
What changed in this PR
This follow-up to #1320 replaces keep-all error trace rescue with a configurable, per-execution-environment rate limit while preserving the existing compute-stats gate.
Changes:
- Adds
DD_APM_ERROR_TPSwith a default of 10 TPS, invalid-value fallback, and legacy-flag compatibility. - Switches rescue to
RateLimitedmode and updates startup warnings. - Expands configuration and sampler regression coverage, including shared budgets and freeze/thaw behavior.
| File | Description |
|---|---|
PLAN-error-sampler-tps.md |
Records implementation plans, caveats, and follow-up scope. |
bottlecap/tests/apm_integration_test.rs |
Uses the default TPS budget in integration fixtures. |
bottlecap/src/traces/trace_processor.rs |
Enables rate-limited sampling and adds regression tests. |
bottlecap/src/traces/trace_agent.rs |
Updates the disabled-sampler test fixture. |
bottlecap/src/startup.rs |
Wires the resolved budget and configuration warnings. |
bottlecap/src/lifecycle/invocation/processor.rs |
Updates tests to use the renamed sampler helper. |
bottlecap/src/lifecycle/invocation/processor_service.rs |
Uses a zero budget to disable rescue in tests. |
bottlecap/src/config/mod.rs |
Adds TPS parsing, precedence, legacy warnings, and configuration tests. |
Review effort: Balanced
lucaspimentel
changed the title
d395f60 to
ad85245
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad85245249
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
f1c205e to
296e853
Compare
Planning notes for rate-limited error trace rescue and the deprecation of DD_SERVERLESS_ERROR_SAMPLER_ENABLED. To be deleted before the PR merges.
Error trace rescue now uses a rate-limited budget instead of an on/off switch: it is on by default at 10 errored traces per second per execution environment when the extension computes stats. Set DD_APM_ERROR_TPS<=0 to disable rescue, or a positive value to change the budget. Exponent values without a decimal point, such as 1e3, are rejected as invalid. DD_SERVERLESS_ERROR_SAMPLER_ENABLED is deprecated and now logs a startup warning; DD_APM_ERROR_TPS wins when both are set.
The post-gap check only asserted on a Keep, so it also passed when rescue returned Drop and would not notice rescue stopping after a long freeze/thaw gap. Scan 400 distinct trace IDs at one post-gap timestamp, require at least one Keep, and assert its errors_sr stays below 1.0 at the capped recovery rate.
193b761 to
bd26a88
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd26a886e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
lambda_extension_compute_stats does not affect the error sampler resolution or the deprecation warning, so the duplicate table rows asserted identical outcomes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0fe5f63141
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Problem
_dd.errors_srmarker needed to exempt them from backend P0 dropping.Cause
AlwaysKeepmode behind an opt-in boolean.This PR
DD_APM_ERROR_TPS(YAML:apm_error_tps) defaults to 10 errored traces/s per execution environment, shared across signatures and invocations. Values<= 0disable rescue._dd.errors_srwithout changing sampling priority. Preserve existing chunk forwarding/filtering, eligibility rules, and the pre-rescue stats snapshot; remove the obsolete stats-dependent startup warning.DD_SERVERLESS_ERROR_SAMPLER_ENABLEDand warn whenever it is set. A valid TPS setting takes precedence; otherwise legacyfalsedisables rescue and legacytrueor an unset flag uses the default budget.DD_APM_ERROR_TPS=0to disable it.Testing
Test 1: Configuration and rescue regressions
Steps
DD_APM_FEATURESsupport.Result
Test 2: Formatting, lint, and workspace validation
Steps
cargo fmt --all -- --check, workspace clippy withdefaultanddefault,test-mode, andcargo nextest run --workspace --no-fail-fast.Result