feat(plugin-mongodb): filter on nested object and array-element fields

[datlechin]

Fixes #2315.
Fixes #2322.

What was wrong

The filter column picker is fed one flat list: the loaded grid result's columns (MainEditorContentView.swift:757). For MongoDB that list is BsonDocumentFlattener.unionColumns, top-level keys only, with a nested object or an array of objects collapsed into a single JSON-typed column. So there was no way to filter on customer.country or items.sku.

The driver already computed dotted paths. BsonDocumentFlattener.fieldPaths walks nested objects and the objects an array holds, and it is exposed on the driver protocol as sampleFieldPaths. Its only consumer was mongo-shell autocomplete. Two things genuinely did not exist:

• Nothing downstream could tell an array prefix from an object prefix. typeName maps .document and .array both to "JSON", and PluginFieldPath carried only {path, typeName, depth}.
• There was no representation for "these conditions bind to the same array element". PluginQueryFilter carried (column, op, value, isCaseSensitive), and the condition list was flat and 1:1 with filter rows.

The fix

Dot notation is wiring. TableFilter.columnName is an unvalidated free String with no membership check anywhere in the path, and buildCondition already emitted "customer.country": … correctly for a dotted column. The picker now merges the driver's sampled paths into the column list, grouped under their top-level parent, capped at depth 2 and 60 items. A button beside it opens a searchable list of every path, which is also the route to a field the sample missed: typing a path there uses it.

$elemMatch is a narrow refactor. It is inherently a grouping (N rows collapse to one clause), which cannot be an operator: FilterOperator is one global enum shared by every database type, and a MongoDB-only case would hard-code one driver into it. So PluginQueryFilter gained elementScope, and buildFilterDocument became partition-then-emit. A row on an array-crossing path gets an any element / same element control; rows sharing a prefix and set to same element collapse into one $elemMatch.

$elemMatch is never substituted silently. With a single condition it is still a different query from dot notation (it also excludes documents whose field is not an array), so it is emitted only when asked for.

Type coercion was not optional. MongoDB compares only within a BSON type. MongoDBPluginDriver.buildFilteredQuery accepted a columnKinds argument and discarded it, so a Date, ObjectId or Decimal128 range filter was compared as a string and returned nothing, with no error. Nested customer.registeredAt and items.shippedAt leaves were previously unreachable as filter targets, so this new surface would have shipped broken on arrival. The driver now keeps the sampled kind per path and the builder emits $date, $oid and $numberDecimal.

Fields dot notation cannot address are excluded. A key holding a literal . or opening with $ is skipped along with everything beneath it. The server reads the dot as a path separator, so offering such a field would match a different one. The docs point at $getField inside $expr as the escape hatch.

Also fixed, in the same files and on the same control

Each of these is a silent wrong result on the filter panel a user reaches this feature through.

• A filter that could not be translated matched the whole collection. buildFilterDocument returned {} when every condition dropped. Because asPluginQueryFilter joined value + "," + secondValue, any comma in a between bound made the split fail, so the filter silently widened to every document while the panel reported it applied. PluginQueryFilter now carries secondValue intact, and the all-dropped guard fails closed with {"_id": {"$in": []}} instead of match-all. The joined value is unchanged for the other plugins that parse it.
• The raw filter row reached MongoDB as a field named __RAW__. It is the default first row, and it is exactly the workaround a user hits for this issue. It now takes a filter document, the way ElasticsearchQueryBuilder.rawColumn already does, and is labelled Raw Filter on a database that does not speak SQL. {} is left to stand, since that is MongoDB's own spelling for match-everything.
• Ignore-case matching broke non-text fields. = with Match Case off became an anchored $regex, which only ever matches a string, so age = 28 returned nothing and age != 28 returned everything. Case now applies to text kinds only.

Verified

Every semantic proven against a live MongoDB 7 (mongo:7 in Docker) seeded with the reporter's exact documents, parsing each filter with EJSON.parse so it goes through the same Extended JSON path as bson_new_from_json:

Filter	Rows
{"customer.country": "US"}	ORD-001, ORD-003
{"items.sku": "A100"}	ORD-001, ORD-003
items.price>500 AND items.name=Laptop, any element	ORD-001, ORD-002
same two conditions, same element ($elemMatch)	ORD-001
{"_id": {"$in": []}} (fail-closed)	none
createdAt >= 2024 as a string (before)	none
createdAt >= 2024 as $date (after)	ORD-001, ORD-003
_id > <oid> as a string (before)	none
_id > <oid> as $oid (after)	ORD-002, ORD-003

The two before/after pairs are the type-bracketing bug reproducing and then fixed.

scripts/check-mongodb-filter-shapes.sh is committed: it compiles a C probe against the real Libs/libbson_*.a and parses all 24 filter shapes the builder can emit, including $and nested inside $elemMatch. The builder assembles Extended JSON by hand and the Swift tests only compare strings, so nothing else would catch a shape that stops parsing after a libbson bump. All 24 parse under 1.28.1.

Also run: generate, build (PASS), 163 unit tests across six suites (PASS), lint on TablePro, Plugins/MongoDBDriverPlugin, Plugins/TableProPluginKit and TableProTests (0 violations, docs references clean).

Two existing tests changed on purpose. filterDocumentBetweenInvalid and filterDocumentUnknownOp asserted doc == "{}", pinning the fail-open behaviour as correct. They now assert the fail-closed filter. This is a deliberate spec change, not a test bent to match new output.

ABI

PluginFieldPath gained arrayPrefixes and PluginQueryFilter gained secondValue and elementScope. Both shipped initializers keep their exact signatures and are marked @_disfavoredOverload, with the new fields on new overloads, per the rule in CLAUDE.md that a defaulted parameter on an existing public init broke every shipped plugin in 0.49.0. This is additive, so no currentPluginKitVersion bump.

MongoDB is registry-only, so this needs a plugin release to reach users.

Not done

• Nested objects as expandable grid columns (the reporter's "optionally"). That is a grid feature, separately sized.
• Postgres JSONB sub-paths. The picker half would be shared, but ->> / #>> generation and the numeric-cast decision live in FilterSQLGenerator. The sampleFieldPaths hook stays open for it.
• Sorting on a nested path. Sort comes from grid header clicks and SortColumnResolver resolves names positionally against the flat column list, so a nested path is unreachable as a sort key. Nothing silently wrong is introduced; it is documented as a limitation.
• same element for a path with two or more array ancestors. That needs nested $elemMatch operators whose semantics I did not verify, so those paths filter with dot notation only rather than emitting a query I could not prove. Documented as a limitation.

Testing gaps

No TableProUITests coverage. The flow needs a live MongoDB and the registry-only MongoDB plugin installed, so it does not run deterministically in CI.

No screenshots. I got as far as a sandboxed Debug build with the MongoDB plugin installed and the seeded collection reachable, but the seeded connection did not decode and driving the window to the right state reliably needed more UI automation than the change warrants. The docs prose in docs/features/filtering.mdx describes the control and its two array modes with a table rather than pointing at an image that does not exist. A screenshot of the column picker and the any element / same element control is worth adding for docs/features/filtering.mdx#nested-fields when someone has the plugin installed against a nested collection.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
TablePro	🟢 Ready	View Preview	Aug 21, 2026, 5:23 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[datlechin]

Self-review found six defects in the first commit, fixed in 126b4d1ef. Two were severe enough that the feature would have shipped broken.

The type coercion was inert in the running app. sampleFieldPaths is routed through DatabaseManager.withBrowseMetadataDriver, and MongoDB does not set supportsConnectionPooling: false, so it runs on a MetadataConnectionPool driver instance. buildFilteredQuery runs on the session driver. The nested-path kind cache was written to one instance and read from the other, so every nested path resolved to kind == nil and the $date / $oid / $numberDecimal coercion never fired outside the unit tests. The kinds are now recorded in buildPluginResult, from the documents a browse already fetched, on the driver that builds the filter. No extra query.

Every MongoDB BETWEEN sent the joined string as its lower bound. asPluginQueryFilter still joins value as lower,upper for plugins built before secondValue existed, so 18/65 arrived as value == "18,65", secondValue == "65" and emitted {"$gte": "18,65", "$lte": 65} — a string against a number, which type bracketing matches to nothing. betweenBounds now strips the ,upper suffix rather than trusting value. My original test passed only because its fixture hand-built the un-joined form the app never sends; the replacement builds the filter from TableFilter(...).asPluginQueryFilter so it exercises the real encoding.

Four more:

• $elemMatch ignored "Match any". Rows sharing an array were always ANDed inside the group even with the panel set to OR. The logic mode now applies inside the $elemMatch body as well as between rows.
• relativePath mixed UTF-16 length with a Character-based drop, so an array field whose name contains a non-BMP character or a combining accent produced a garbage inner key. Both sides are Character-based now.
• A known string field got numeric-looking values unquoted, so filtering a zip code or SKU of 12345 emitted {"customer.zip": 12345} and matched nothing. .string now always quotes.
• A doc comment claimed fieldPaths and fieldPathKinds shared one traversal. They derive from the same walk but each call it, so the comment says that instead.

All three behaviour changes verified against the live MongoDB 7, with each bug reproducing as an empty result first:

Filter	Rows
$elemMatch with $or (same element, match any)	ORD-001, ORD-002
$elemMatch plain body (same element, match all)	ORD-001
{"$gte": "18,65", "$lte": 65} (the BETWEEN bug)	none
{"$gte": 18, "$lte": 65} (fixed)	ORD-001, ORD-003
{"sku": 12345} unquoted (the string bug)	none
{"sku": "12345"} quoted (fixed)	12345

181 tests pass, build and MongoDB plugin build clean, lint 0 violations. scripts/check-mongodb-filter-shapes.sh gained the $or-inside-$elemMatch, non-BMP key and quoted-string shapes and still parses all 27 under libbson 1.28.1.

main was merged into the branch remotely while I was working; I merged it in rather than rewriting, and re-ran everything after.

[datlechin]

Also fixes #2322 (folded in at your request), commit 025aa7ce0.

Root cause is one line. NumberText.json is a hand-written JSONWriter, not JSONSerialization, so the issue's suggested .withoutEscapingSlashes did not apply. The escaping was explicit:

case "/": output += "\\/"   // NumberText.swift:212

Escaping / is optional in JSON, so that case is now gone.

One fix covers both places the issue names. The JSON result view (ResultsJsonView.swift:266) and the right sidebar (FieldMenuView.swift:31) both call prettyPrintedAsJson() → JsonReindenter, which reindents text and preserves whatever escaping its input already carries. The \/ was baked in earlier, by BsonDocumentFlattener.serializeToJson → NumberText.json. Fixing the writer removes it before the reindenter ever sees it. The reindenter still preserves \/ in JSON it did not produce, which is correct and stays covered by JsonReindenterTests.

Blast radius, since NumberText.json is shared. Seven callers: MongoDB (cells and pretty-print), ClickHouse, Elasticsearch, DynamoDB, JSON import and Beancount. All render JSON for display or import, and \/ and / are the same value, so this is a display improvement everywhere with no semantic change.

The guard that caught this. JSONWriterEquivalenceTests asserts the writer matches JSONEncoder byte for byte over 4000 generated values, and its alphabet includes "a/b", so removing the escape would have failed it. Rather than weaken the test, the reference encoder now uses .withoutEscapingSlashes too and the doc comment records the divergence as deliberate. The invariant stays real.

Tests: three new cases in NumberTextTests pinning an unescaped slash at top level and nested, plus one confirming backslashes are still escaped.

Verified: 250/250 in TableProCore, 226/226 across the app-side JSON and MongoDB suites (JsonReindenterTests, StringJsonTests, MongoDBNestedFilterTests, MongoDBQueryBuilderTests, BsonFieldPathArrayTests, BsonDocumentFlattenerTests), lint 0 violations across the app, both plugin paths and TableProCore.