Repository navigation
ft: spam log filter to reduce Kafka bloat on cheap-gas chains - #133
Merged
Merged
Conversation
On chains like Berachain where gas is dirt cheap, a single block can contain thousands of identical events from one contract. Add SPAM_LOG_THRESHOLD env var (int, 0 = disabled): when set, any (address, topic0) pair with more than that many logs per block is deduplicated to its last occurrence before publishing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an optional “spam log” deduplication feature to reduce oversized per-block log payloads on chains that emit many identical events, controlled via SPAM_LOG_THRESHOLD.
Changes:
- Introduces
filterSpamLogsto drop all but the last log for over-threshold(address, topic0)pairs. - Wires a new
SPAM_LOG_THRESHOLD/--spam-log-thresholdflag into the app and passes it into the listener handler. - Applies the filter during block handling before publishing messages.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/listener/utils.go | Adds filterSpamLogs helper implementing the deduplication logic. |
| pkg/listener/handler.go | Applies spam log filtering to blocks before publishing. |
| pkg/listener/filter_option.go | Extends listener options with spamLogThreshold and WithSpamLogThreshold. |
| internal/app/flags.go | Adds spam-log-threshold CLI flag + SPAM_LOG_THRESHOLD env var wiring. |
| internal/app/app.go | Reads the new flag and configures the handler options accordingly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+24
to
+28
| func filterSpamLogs(logs []types.Log, threshold int) []types.Log { | ||
| type key struct{ addr, topic0 string } | ||
|
|
||
| counts := make(map[key]int, len(logs)) | ||
| for _, l := range logs { |
Comment on lines
+22
to
+24
| // filterSpamLogs drops all but the last log for any (address, topic0) pair that appears more than | ||
| // threshold times in the slice. Order of surviving logs is preserved. | ||
| func filterSpamLogs(logs []types.Log, threshold int) []types.Log { |
…el events Anonymous events (no topics) key on address only, catching contracts like 0xbd78... that emit 100+ topicless events per block across many txs. Events with topics key on (address, topic0, topic1): mass-airdrop spam has a fixed sender in topic1 so all transfers share the same key, while Uniswap V4 / Balancer vault events carry the pool ID in topic1 and therefore get distinct keys per pool — preserving every pool's events. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NgoKimPhu
changed the title
Member
|
LGTM |
lehainam-dev
approved these changes
Jun 29, 2026
lehainam-dev
approved these changes
Jun 29, 2026
NgoKimPhu
force-pushed
the
ft/spam-log-filter
branch
from
June 30, 2026 09:58
4674bb3 to
763290c
Compare
- Early return in filterSpamLogs when len(logs) <= threshold (no key can possibly exceed it); drop pre-sized map hint to avoid over-allocation on blocks with many identical events but few unique keys - Apply filter to revertedBlocks in addition to newBlocks so both sides of the Kafka message are deduplicated - Add unit tests: below-threshold no-op, above-threshold keeps last, different topic1 (pool ID) preserves all, anonymous events keyed by address, order + non-spam preservation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NgoKimPhu
force-pushed
the
ft/spam-log-filter
branch
from
June 30, 2026 09:59
763290c to
5132bfd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SPAM_LOG_THRESHOLDenv var (int,0= disabled). When set, any spam key with more than that many logs per block is deduplicated down to its last occurrence before publishing to Kafka / storing to Redis.Spam key definition
The dedup key is chosen to let legitimate high-volume contracts (Uniswap V4, Balancer vault) through:
address(address, topic0, topic1)Why this works:
Transfer× 1000): samefromin topic1 → all share one key → deduplicated ✓0xbd78...× 176/block across many txs): keyed on address alone → deduplicated ✓Usage
Set to
0(or omit) to leave existing behaviour unchanged.Test plan
go test ./pkg/listener/...passesSPAM_LOG_THRESHOLD=128and verify bloated blocks are trimmed in Kafka messages while swap events from known DEX vaults are intact🤖 Generated with Claude Code