Repository navigation
fix: read the tenant from _org_id alone - #268
Merged
Merged
Conversation
The thin chart now renders on scalo-service 2.14.19 as well. The library pin had stayed at 2.14.15, which hashes the whole config into the pod and ignores the contract's hot_paths, so every config edit rolled the pods.
The first hop now stamps _org_id and _source, so the loader only reads them. - _org_id reads routing.org_id_field and nothing else. A column directive can no longer point it at a field the sender writes, so a sender's org_id never sets the tenant. - An absent, null or empty _org_id lands in routing.default_org_id, default 'default', the same string in every component. It used to land as '', which is in no org. - A null or empty _source counts as absent: the next routing field, then the topic, then default_table. An empty one used to go to the DLQ. - org_routes keys on the tenant the record resolves to. - routing.rules (CEL), routing.db_fields, routing.compat_v2_source and routing.mapping_file are gone. Nothing set them, and mapping_file had no reader. - On the legacy_flatten path the routing fields are stripped before the header is written, so routing on _source no longer strips the _source header it just wrote.
A config reload now rebuilds the router, the header extractor and the default table from the new routing and metadata, pre-warms any uncached table the new routing names, and stops holding the new default table absent. The router reads the configured metadata instead of the default. hot_reload.enabled defaults to true and the deployment contract lists routing in hot_paths, so a routing edit reaches running pods without rolling them. routing.dlq, field_mapping and enrichment are built once at startup and now log a restart warning. tests/integration/hot_reload.rs edits the default table and default org mid-stream and proves no record is lost or duplicated and the commit passes the last offset.
The header pass took _timestamp_receiver, timestamp_received and received_at as receive stamps when the column carried no directive. The first hop now stamps _timestamp_received alone, so that is the only key read ahead of the directive, and any other spelling is read only when the column's directive names it.
The loader's own downloader is replaced by scalo's geoip_download, which caps each download at max_database_bytes, refuses a file that is not an MMDB and keeps the IPinfo token out of the logged URL. The maxminddb lookup is unchanged. geoip.auto_download gains max_database_bytes (default 512 MiB), and maxmind_account_id and maxmind_license_key become secrets, redacted in logs and config dumps. data_dir keeps its /var/lib/dfe/geoip default. The ip_locate provider now downloads IPLocate's own ASN database, and sapics downloads DB-IP's ASN data to sapics-dbip-asn.mmdb. The direct reqwest, flate2 and tar dependencies go with the old module.
The gRPC backend tests picked a port from the process id and bound it twice, so a port taken between the probe and the bind failed the test. The listener now binds port 0 and the client dials the address it reports. apply_results_memory_released_on_error compared two current_bytes readings, which follow the host's own memory use. It now asserts the lease ledger drops by the failed message's 50 bytes.
The capability catalogue still named routing rules, which no longer exist, and mounted MMDB readers, which startup now downloads. Both descriptions now say what the services do.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The GeoIP redaction test printed its fixture credentials in its failure messages, which CodeQL reads as cleartext logging. The messages now name the field and never the value or the dump.
A record whose top-level object carries _org_id or _source more than once, an escaped spelling included, is taken out at intake and dead-lettered with a reason naming the key, before any pipeline mode reads it. JSON readers disagree on which value wins and ClickHouse refuses the row under full capture, so neither value is trusted. A record with no backslash and at most one quoted spelling of each key is passed without a scan. Counted on dfe_loader_duplicate_header_total.
catinspace-au
marked this pull request as draft
October 21, 2026 07:49
catinspace-au
marked this pull request as ready for review
October 21, 2026 21:54
|
Released in v1.18.57 -- https://github.com/hyperi-io/dfe-loader/releases/tag/v1.18.57 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The loader now takes the tenant from
_org_idand nothing else. A record with no org, or no source, lands in the deployment's default org and default table instead of in no org or the DLQ. A routing edit reaches a running loader without a restart.What changes:
_org_idreadsrouting.org_id_field(default_org_id) alone. No column directive and no sender field such asorg_idcan set it.routing.default_org_id(defaultdefault, the same string every DFE component uses). An absent, null or""org lands there.routing.org_routesis keyed on that tenant.""_sourcecounts as absent, so the record falls through to the topic's source, thendefault_table. It never reaches the DLQ._org_idor_sourcemore than once, an escaped spelling included, is dead-lettered at intake on every pipeline mode with a reason naming the key, and counted ondfe_loader_duplicate_header_total. JSON readers disagree on which value wins, so the loader trusts neither. No\byte and at most one quoted spelling of each key skips the scan.routing.rules(CEL),routing.db_fields,routing.compat_v2_sourceandrouting.mapping_file, which nothing read.hot_reload.enableddefaults to true.metadata.routinginhot_paths, so a routing edit no longer rolls the pods.routing.dlq,field_mappingandenrichmentare built at startup and log a restart warning._timestamp_receivedreads only the first hop's_timestamp_receivedahead of its directive._timestamp_receiver,timestamp_receivedandreceived_atare read only when the column's directive names them.geoip_download. The maxminddb lookup is unchanged.geoip.auto_download.max_database_bytes(default 512 MiB). A file that is not an MMDB is refused.maxmind_account_idandmaxmind_license_keyare secrets now, redacted in logs and config dumps. Same YAML shape.ip_locatedownloads IPLocate's own ASN database.sapicsdownloads DB-IP's ASN data tosapics-dbip-asn.mmdb.reqwest,flate2andtardependencies are gone.apply_results_memory_released_on_errorasserts the lease ledger rather than host memory.Ack and commit timing are unchanged.
tests/integration/hot_reload.rsstreams 600 records, edits the default table and default org mid-stream, and checks nothing is lost or written twice and the group commits past the last offset.Consumers need lockstep edits: dfe-engine's loader model and default configs still render the removed keys, dfe-infra's
apps.yamlsays the watcher ships off, and dfe-docker's detection e2e sets the tenant through a payloadorg_id, which no longer works.Evidence:
hyperi-ci check: 1849 tests run, 1849 passed, 19 skipped, testcontainers included. fmt, clippy per feature, audit, deny and the feature matrix pass._source: ""record goes to the DLQ, after the edit a record for an absent table falls back to the old default table, and all three repeated-key records land. All three pass here.org_id, null and empty org, empty_source,_timestamp_receiver, thehot_reloaddefault,hot_pathsand the MaxMind key in a config dump.Done when PR CI is green and the engine, dfe-infra and dfe-docker edits land with it.