Skip to content

Use the zen-internals IP matcher - #757

Open
iacobdaniel wants to merge 24 commits into
mainfrom
feat/zen-internals-ip-matcher
Open

iacobdaniel wants to merge 24 commits into
mainfrom
feat/zen-internals-ip-matcher

Conversation

@iacobdaniel

Copy link
Copy Markdown
Contributor

Replaces pytricia and the Python fallback with the Rust matcher from zen-internals.

Comment on lines +24 to +29
def has(self, network):
if self._handle is None:
return False
if not isinstance(network, str) or not network.isascii():
return False
return zen_internals.ip_matcher_has(self._handle, network.encode())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High - Missing native matcher disables SSRF and IP-list enforcement

When the bundled zen-internals library is missing or cannot be loaded, IPMatcher has no handle and its has() method returns false for every address. The static IMDS/private-IP matchers and configured global or route IP lists then produce no matches, allowing DNS resolutions to metadata/private addresses and bypassing configured IP blocklists and allowlists.

Show fix

Retain a functional pure-Python matcher fallback for load failures, or fail closed and make the application refuse to enable these security controls unless all required native matcher symbols are available. Ensure every supported deployment target ships and loads the matching artifact before accepting IP-based security configuration.

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On platforms that have no zen-internals build (for example 32-bit Linux), Zen fails open and logs a warning, the same way SQL injection detection already behaves there on main.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 We were not able to ignore this issue because of the following reason:

You do not have the permission to ignore issues.

@iacobdaniel

Copy link
Copy Markdown
Contributor Author

Reminder: this points at the v0.1.75-beta.2 pre-release of zen-internals. Switch BASE_URL to the first stable release that contains the IP matcher.


endpoint["allowedIPAddresses"] = IPMatcher(endpoint["allowedIPAddresses"])
allowed_ips = IPMatcher(endpoint["allowedIPAddresses"])
if not allowed_ips.is_built():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any way we can avoid this? e.g. use a function create_ip_matcher that returns None if no IPMatcher could be created?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it. create_ip_matcher() returning None removes the if for route allowlists, but every other place that uses a matcher then needs a None check: the blocklist, bypass list and both SSRF lists. That's 5 checks instead of 2, and any new caller has to remember it. So I would keep it as it is.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do those call sites need to check whether the IP matcher was built?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. A matcher that isn't built matches no IP. Only an allowlist would then block everyone, so only the two allowlists check is_built(). For the other lists, matching nothing just means that check is skipped, the same as when the library isn't loaded.

Comment thread aikido_zen/helpers/zen_internals.py Resolved
[network.encode() for network in self._networks]
)
if self._handle is not None:
weakref.finalize(self, zen_internals.ip_matcher_free, self._handle)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The finalizer may receive None as its callback when the library lacks ip_matcher_free; registering it can cause a runtime exception during finalization.

Details

✨ AI Reasoning
​The create binding can be present while the free binding is absent. In that case, construction succeeds and registers a finalizer with a nullable callback, which can fail when the object is finalized.

🔧 How do I fix it?
Check values for null before use with guards, early returns, or null-coalescing. Declare nullable return types for methods that can return null, and avoid silencing nullable warnings with the null-forgiving operator (!) unless the value's non-nullness is guaranteed.

Reply on this comment to give feedback or ignore the issue.
More info

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Without ip_matcher_free, ip_matcher_create returns None.

Comment thread aikido_zen/helpers/zen_internals.py Resolved
Comment thread aikido_zen/helpers/zen_internals.py Resolved
Comment thread aikido_zen/storage/firewall_lists.py Outdated
Comment on lines +25 to +27
if not all(entry["iplist"].is_built() for entry in allowed_ips):
allowed_ips = []
self.allowed_ips = allowed_ips

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the allowed IPs are currently set? Should a currently failing attempt to set the allowed IPs override a previously successful attempt to set the allowed IPs by clearing the allowed IPs entirely?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, good point. Pushed a fix to keep previous allowlist.

Comment thread aikido_zen/storage/firewall_lists.py Outdated
def set_allowed_ips(self, allowed_ip_entries):
self.allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))
allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))
# An allowlist that failed to build would block the visitors it should allow.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An IP list that fails to build clears the allowed list. Does an empty allowed IPs list not block visitors that it should allow?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. An empty allowed list means allow mode is off, so all visitors are allowed (is_allowed_ip returns True when the list is empty).

Comment thread aikido_zen/helpers/zen_internals.py Resolved
Comment thread aikido_zen/helpers/zen_internals.py Resolved
Comment thread aikido_zen/storage/firewall_lists.py Outdated
Comment on lines +17 to +18
blocked_ips = list(map(parse_ip_entry, blocked_ip_entries))
allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Malformed allowlist entry suppresses valid blocklist refresh

A successful firewall-list response with valid blocked IP entries but a malformed allowed-list entry causes FirewallLists.update to fail while parsing the allowlist (Trigger). The new atomic update builds both lists before assigning either, and the caller swallows the exception, so the valid blocklist is not installed (Mechanism). Requests from IPs that the response intended to block then pass the global blocklist until a later successful refresh, or indefinitely when this occurs during initial setup (Consequence).

Show fix
Suggested change
blocked_ips = list(map(parse_ip_entry, blocked_ip_entries))
allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))
blocked_ips = list(map(parse_ip_entry, blocked_ip_entries))
try:
allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))
except (KeyError, TypeError, ValueError) as e:
logger.debug("Exception parsing allowed IP lists: %s", e)
self.blocked_ips = blocked_ips
self.set_blocked_user_agents(blocked_user_agents)
return

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, fixed differently: a missing ips or description no longer raises, so the update still applies everything together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue remains because FirewallLists.update still maps parse_ip_entry at aikido_zen/storage/firewall_lists.py:33 without validating each entry: a non-mapping entry (or non-iterable ips) raises before assignments at lines 58–63, and update_firewall_lists.py:40 swallows it, so the valid blocklist is not installed; the newer .get/IPMatcher filtering only handles some malformed shapes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: ips that isn't a list is now treated as empty, so a malformed allowlist entry is skipped and the rest of the update still applies.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue remains because FirewallLists.update still maps every allowlist entry through parse_ip_entry at aikido_zen/storage/firewall_lists.py:28, while parse_ip_entry calls entry.get("ips") at line 159 without validating that entry is a mapping; an entry such as None still aborts the atomic update, and update_firewall_lists swallows that exception. The new non-list ips handling is only a partial fix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 8017344.

This issue remains because aikido_zen/storage/firewall_lists.py:29-36 returns from FirewallLists.update when any allowed entry has a non-list ips value, before assigning self.blocked_ips at line 69, so a valid blocklist is still not installed; the new validation avoids the parse exception but preserves the atomic suppression.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional: null (and a missing or empty ips) means the list has no IPs, which is how realtime sends an emptied list, so the blocklist is correctly cleared.

Comment thread aikido_zen/storage/firewall_lists.py Resolved
Comment thread aikido_zen/storage/firewall_lists.py Outdated
Comment on lines +17 to +18
blocked_ips = list(map(parse_ip_entry, blocked_ip_entries))
allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Missing IP field installs an empty global allowlist

A firewall-list response with an allowlist entry whose ips field is missing or null is accepted by parse_ip_entry. The resulting empty IPMatcher is still treated as a built, non-empty allowlist entry, so is_allowed_ip rejects every public client address; the same normalization on a malformed blocked entry can also replace a previously effective blocklist with a matcher that matches nothing.

Show fix
Suggested change
blocked_ips = list(map(parse_ip_entry, blocked_ip_entries))
allowed_ips = list(map(parse_ip_entry, allowed_ip_entries))
blocked_ips = (
self.blocked_ips
if any(
not isinstance(entry, dict)
or not isinstance(entry.get("ips"), list)
for entry in blocked_ip_entries
)
else list(map(parse_ip_entry, blocked_ip_entries))
)
allowed_ips = [
parse_ip_entry(entry)
for entry in allowed_ip_entries
if isinstance(entry, dict) and isinstance(entry.get("ips"), list)
]

More info - Reply on this comment to give feedback or ignore the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid for the allowlist, fixed: allowlist entries without IPs are skipped, so they can't block every visitor.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue remains because FirewallLists.update still maps every blocked entry through parse_ip_entry at aikido_zen/storage/firewall_lists.py:17, whose IPMatcher(entry.get("ips")) at line 97 turns missing/null ips into a built empty matcher and installs it, replacing an effective prior block; the new empty-entry filter only protects allowlists at lines 19–23.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional: realtime sends "ips": null when a list has no IPs, so an empty blocklist is the correct update. Empty allowlist entries are skipped instead, so they never restrict access.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 We were not able to ignore this issue because of the following reason:

You do not have the permission to ignore issues.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue remains because update maps blocked entries at firewall_lists.py:28 and installs them at line 58; parse_ip_entry turns missing/null ips into [] at lines 159-165, and IPMatcher([]) is built when its handle exists, so a malformed response can replace an effective blocklist with a matcher matching nothing. Lines 30-35 fix only the empty-allowlist case.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed in commit 8017344.

This issue remains because FirewallLists.update only rejects non-None non-list ips at aikido_zen/storage/firewall_lists.py:30; a blocked entry with missing or null ips still reaches parse_ip_entry at line 37, which constructs an empty matcher at line 176, and that replacement is committed to self.blocked_ips at line 69. The allowlist symptom is mitigated by the empty-entry filter at lines 40–44, but malformed blocked entries can still erase effective blocks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional: null (and a missing or empty ips) means the list has no IPs, which is how realtime sends an emptied list, so the blocklist is correctly cleared.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 We were not able to ignore this issue because of the following reason:

You do not have the permission to ignore issues.

Comment on lines +11 to +17
networks = list(networks or [])
self._networks = [
network
for network in networks
if isinstance(network, str) and network.isascii() and "\n" not in network
]
self._skipped_count = len(networks) - len(self._networks)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

networks is rebound to a new list, obscuring the caller-provided value. Use a separate local for the filtered input.

Show fix
Suggested change
networks = list(networks or [])
self._networks = [
network
for network in networks
if isinstance(network, str) and network.isascii() and "\n" not in network
]
self._skipped_count = len(networks) - len(self._networks)
networks_list = list(networks or [])
self._networks = [
network
for network in networks_list
if isinstance(network, str) and network.isascii() and "\n" not in network
]
self._skipped_count = len(networks_list) - len(self._networks)
Details

✨ AI Reasoning
​The constructor replaces the caller-provided networks value with a newly constructed list, so the parameter no longer represents the original argument. Keeping the copy in a separate local would preserve that distinction and make the code easier to follow.

Reply on this comment to give feedback or ignore the issue.
More info

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional, as before: the original value isn't used after this line, so the name is reused for its list form.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛔ You don't have permission to ignore issues.

Comment thread aikido_zen/storage/firewall_lists.py Resolved
Comment thread aikido_zen/storage/firewall_lists.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants