Conversation
24cabfb to
3a2c0a1
Compare
3a2c0a1 to
47f0cc1
Compare
Fields marked `river:"unique"` are read and written as JSON paths, so a name like `user.id` can omit its value and deduplicate distinct jobs. Unnamed JSON tags also omit their fields instead of using the Go field name. Escape each path component, including leading colons, and fall back to the Go field name for unnamed tags. Keep the original field ordering so unaffected jobs retain their existing unique keys. Cover escaped components, literal and nested path collisions, and unnamed tags with explicit encoding fixtures and cross-driver insertion cases.
47f0cc1 to
b5ceebd
Compare
brandur
left a comment
There was a problem hiding this comment.
Wanna ask Claude for a quick benchmark just to make sure that there isn't a serious perf regression on the unique path here?
| - Fixed `UniqueOpts.ByArgs` skipping distinct jobs when `river:"unique"` fields have JSON names containing path syntax (like `user.id` or `alice@example.com`), or an unnamed JSON tag like `json:",omitempty"`. Field names are now addressed literally, and unnamed tags use the Go field name. Unique keys change for affected jobs, so old and new clients may each insert a job with the same args during a rolling upgrade. [PR #1387](https://github.com/riverqueue/river/pull/1387). | ||
| - Fixed `UniqueOpts.ByArgs` skipping distinct jobs or failing inserts when all args participate in uniqueness and an object's keys contain JSON path syntax or are empty. These keys are now included literally, while unique keys for unaffected args remain unchanged. Affected jobs may be inserted again after upgrading or by old and new clients during a rolling upgrade. [PR #1387](https://github.com/riverqueue/river/pull/1387). |
There was a problem hiding this comment.
There is a ton of identical verbiage in these two entries. Maybe there's two separate issues in a technical sense, but I don't think the technical correctness is worth the added verbosity. Do you want to condense them down to one?
There was a problem hiding this comment.
Thanks for catching, I cleaned this up a lot! 🙏
The all-args unique key builder interprets object keys as JSON paths. Keys containing path syntax can lose their values or collide with other keys, while an empty key causes an insertion error. Walk the object directly and rebuild it in key order with raw values. Preserve the previous key encoding and first-value handling for repeated keys so unaffected hashes remain stable. Continue rejecting scalar and nonempty array args instead of silently hashing them as empty objects. Cover literal keys and duplicate detection across drivers. Pin the key encoding with explicit compatibility fixtures, including Unicode, control characters, and raw nested values, and verify rejection of non-object args.
b5ceebd to
a8b2624
Compare
|
Awesome. Thoughts on the benchmark? |
Bah, sorry, I thought I wrote this in my comment to you but must have gotten distracted and forgotten to. The PR description was updated with the benchmark results previously, it turns out this new version is actually slightly faster or a wash.
|
This fixes
UniqueOpts.ByArgsincorrectly skipping distinct jobs when their JSON field names contain dots or other characters that River interprets as path syntax.A dot is allowed in a JSON key. This object has one top-level key named
user.id:{"user.id": "u1"}River's uniqueness code mistakenly reads that key name as a path to an
idfield inside auserobject, as though the args had this different shape:{"user": {"id": "u1"}}The lookup finds nothing in the actual args, so the user ID is left out of the unique key. Jobs for different users then look like duplicates.
An application can explicitly choose the dotted JSON name with a struct tag:
encoding/jsonwrites the first JSON shape above. It does not turn the dot into nesting, and River does not automatically renameUserIDtouser.id. The example uses an unusual but valid JSON name; a conventionaljson:"user_id"tag is unaffected.With
UniqueOpts{ByArgs: true}, these successive insertions now behave correctly. Assume no matching jobs exist initially and the inserted jobs remain available:SyncUserArgs{UserID: "u1"}SyncUserArgs{UserID: "u2"}u1SyncUserArgs{UserID: "u1"}againu1The fix also covers uniqueness based on all args, without
river:"unique"tags. For example,{"file.name":"a.txt"}and{"file.name":"b.txt"}previously collided; now both jobs are inserted. An empty key such as{"":"a"}previously failed withpath cannot be empty; now it inserts normally, and changing its value produces a distinct job. Keys containing other path syntax, such asalice@example.comor a leading:, are also treated literally.A related fix handles unnamed JSON tags, such as this field:
Go uses the JSON key
Recipient, but River previously looked for an empty key, so different nonempty recipients collided. River now uses the Go field name too. An omitted recipient remains omitted.Internally, tagged fields use escaped gjson/sjson path components. When all args participate in uniqueness, River walks the top-level object directly and rebuilds it in key order using its raw values.
Unaffected args keep byte-for-byte identical unique keys. Affected jobs receive corrected keys, so a job inserted before upgrading may no longer deduplicate an identical insertion after upgrading; during a rolling upgrade, old and new clients may each insert it.
A before/after
UniqueKeymicrobenchmark on Go 1.27.1 / Apple M4 Pro found no statistically significant time change for ordinary flat or nested tagged fields, with unchanged allocations. All-args cases with 0, 1, 8, and 64 keys were 25–89% faster and allocated less. An unusual two-field case using$amountandYwas 3.8% slower (about 20 ns), with 32 extra bytes and three extra allocations per call. These measurements use ten alternating before/after samples with the field cache warmed and exclude JSON marshaling and database work.