Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion Changelog.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# NEventStore.Persistence.MongoDB

## Unreleased
## Unreleased

- Harden duplicate-write classification using driver error codes/categories and structured key patterns, with a narrowly matched legacy index-message fallback. Preserve checkpoint retries, duplicate commit detection and concurrency handling without matching index names inside user payloads. No stored-schema change; obsolete public constants are removed as noted below. [#92](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/92)

- Run the same acceptance and regression suite with pinned MongoDB.Driver 3.0.0 and 3.13.0 test dependencies in isolated CI jobs on MongoDB 7.0.45, retaining a net10.0/current-driver lane on MongoDB 9.0.4. Verify restored Driver/BSON versions and the packed dependency contract across supported frameworks. Production dependencies and the published `[3.0.0,4.0.0)` range are unchanged; no API, schema or migration change. [#93](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/93)

Expand All @@ -27,6 +29,8 @@
- Propagate cancellation from async operations, including snapshot writes. Observer reads signal request cancellation through `OnCompletedAsync` with the original token; `OperationCanceledException` while the request token is active is reported through `OnErrorAsync`. Observer reads use `ObserveMongoAsync` to complete successfully outside the guarded operation, retaining the protected virtual `TryMongoAsync` guard and its existing success behavior for derived engines. Terminal callbacks are awaited, and callback outcomes reach the returned task. [#86](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/86)

### Breaking Changes

- Removed the unused public constants `MongoCommitIndexes.CheckpointNumberMMApV1` and `MongoCommitIndexes.CheckpointNumberWiredTiger`. Source consumers and reflection-based code referencing these members must be updated. Duplicate-key classification and legacy message support are unchanged. [#92](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/92)

- Async observer reads now use `EnumerateMongoAsync` to translate only cursor acquisition and advancement errors, matching synchronous `EnumerateMongo`. Mongo exceptions from application deserializers, `OnNextAsync`, and cursor disposal reach `OnErrorAsync` unchanged instead of being wrapped as storage failures. Driver errors still retain the original exception as `InnerException`. No public observer signatures change. [#108](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/108)

Expand All @@ -46,6 +50,8 @@
- Non-observer async callers must now handle `OperationCanceledException` (including `TaskCanceledException`) from operations that previously swallowed cancellation or returned success/default values; for example, `AddSnapshotAsync` now propagates cancellation instead of returning `false`. Observer reads invoke `OnCompletedAsync` with the original token only when the request token is canceled; cancellation exceptions with an active request token go to `OnErrorAsync`. Cancellation or failure from a terminal callback propagates after it is awaited, without a second terminal notification. [#86](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/86)

### Migration notes

- Remove references to the deleted checkpoint error-message constants. They represented legacy server-message fragments, not a supported classification API. If an application needs those literal fragments for its own legacy parser, define them locally (`"$_id_"` and `"index: _id_"` respectively); do not use substring matching to classify arbitrary duplicate-write errors. No database or index migration is required. [#92](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/92)

- For async observer reads, handle application Mongo exceptions directly in `OnErrorAsync`. Set `MongoPersistenceOptions.WrapAsyncApplicationExceptions = true` to select the legacy whole-read guard once per operation, restoring legacy application/cleanup wrapping and protected guard scope. The option is read once per read and does not alter other async operations. Subclasses relying on whole-read timing, retries or interception should opt out or adapt to the new driver-operation scope. Cancellation, early-stop completion and terminal callback failures retain their observer contract. [#108](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/108)
- Keep the engine alive until all synchronous enumerations and async observer reads finish. For shutdown, cancel and await active async reads before disposing the shared engine; disposal is a batch-boundary guard, not immediate cancellation or atomic exclusion of concurrent reads. No data migration is required. [#109](https://github.com/NEventStore/NEventStore.Persistence.MongoDB/issues/109)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
using System.Reflection;
using FluentAssertions;
using MongoDB.Bson;
using MongoDB.Driver;
using NEventStore.Persistence.AcceptanceTests.BDD;
#if NUNIT
using NUnit.Framework;
namespace NEventStore.Persistence.MongoDB.Tests.AcceptanceTests.Issues;
[TestFixture(11000, "_id", "Checkpoint")]
[TestFixture(11000, "CommitId", "Commit")]
[TestFixture(11000, "StreamRevision", "Other")]
[TestFixture(121, "_id", "None")]
public class when_Issue92_structured_duplicate_details_are_available(int code, string key, string expected) : SpecificationBase
{
private string? _actual;
protected override void Because()
{
var error = new MongoCommandException(new global::MongoDB.Driver.Core.Connections.ConnectionId(new global::MongoDB.Driver.Core.Servers.ServerId(new global::MongoDB.Driver.Core.Clusters.ClusterId(), new System.Net.DnsEndPoint("localhost", 27017))), "message without index information", new BsonDocument(),
new BsonDocument { { "code", code }, { "keyPattern", key == "CommitId" ? new BsonDocument { { "BucketId", 1 }, { "StreamId", 1 }, { "CommitId", 1 } } : new BsonDocument(key, 1) } });
var classifier = typeof(MongoPersistenceEngine).Assembly.GetType("NEventStore.Persistence.MongoDB.DuplicateKeyClassifier");
_actual = classifier?.GetMethod("Classify", BindingFlags.Static | BindingFlags.NonPublic)!.Invoke(null, new object[] { error })!.ToString();
}
[Fact] public void Should_use_structured_error_identity_instead_of_message_text() => _actual.Should().Be(expected);
}
#endif


#if NUNIT
[TestFixture("E11000 duplicate key error collection: db.c index: _id_ dup key: { _id: 1 }", "Checkpoint")]
[TestFixture("E11000 duplicate key error index: db.c.$_id_ dup key: { : 1 }", "Checkpoint")]
[TestFixture("E11001 duplicate key error index: CommitId_Index dup key: { : 1 }", "Commit")]
[TestFixture("E11000 duplicate key error index: db.c.$CommitId_Index dup key: { : 1 }", "Commit")]
[TestFixture("E11000 duplicate key error index: StreamRevision_Index dup key: { value: '_id_ CommitId_Index' }", "Other")]
[TestFixture("E11000 duplicate key error index: custom_id_ dup key: { : 1 }", "Other")]
[TestFixture("E11000 duplicate key error index: CommitId_Index_extra dup key: { : 1 }", "Other")]
[TestFixture("E11000 duplicate key error without index", "Other")]
[TestFixture("validation rejected user text E11000 duplicate key error index: _id_ dup key: {}", "None")]
[TestFixture("E11002 duplicate key error index: _id_ dup key: {}", "None")]
[TestFixture("E11000 duplicate key error index: custom index dup key: { value: \"index: _id_ dup key:\" }", "Other")]
[TestFixture("E11000 duplicate key error index: custom index dup key: { value: \"index: CommitId_Index dup key:\" }", "Other")]
[TestFixture("E11000 duplicate key error dup key: { value: \"index: _id_ dup key:\" }", "Other")]
[TestFixture("E11000 duplicate key error index: custom index dup key : { value: \"index: _id_ dup key:\" }", "Other")]
public class when_Issue92_legacy_duplicate_messages_are_classified(string message, string expected) : SpecificationBase
{
private string? _actual;
protected override void Because()
{
var classifier = typeof(MongoPersistenceEngine).Assembly.GetType("NEventStore.Persistence.MongoDB.DuplicateKeyClassifier");
_actual = classifier?.GetMethod("Classify", BindingFlags.Static | BindingFlags.NonPublic)!.Invoke(null, new object[] { new MongoException(message) })!.ToString();
}
[Fact] public void Should_only_use_the_duplicate_prefix_and_complete_index_token() => _actual.Should().Be(expected);
}
#endif

#if NUNIT
[TestFixture("waiting for replication timed out")]
[TestFixture("E11000 duplicate key error index: _id_ dup key: {}")]
public class when_Issue92_a_write_concern_failure_has_no_write_error(string message) : SpecificationBase
{
private string? _actual;
protected override void Because()
{
var concern = (WriteConcernError)Activator.CreateInstance(typeof(WriteConcernError),
BindingFlags.Instance | BindingFlags.NonPublic, null,
new object?[] { 64, "WriteConcernFailed", message, new BsonDocument("wtimeout", true), Array.Empty<string>() }, null)!;
var connection = new global::MongoDB.Driver.Core.Connections.ConnectionId(
new global::MongoDB.Driver.Core.Servers.ServerId(new global::MongoDB.Driver.Core.Clusters.ClusterId(),
new System.Net.DnsEndPoint("localhost", 27017)));
var error = new MongoWriteException(connection, null, concern, null);
var classifier = typeof(MongoPersistenceEngine).Assembly.GetType("NEventStore.Persistence.MongoDB.DuplicateKeyClassifier")!;
_actual = classifier.GetMethod("Classify", BindingFlags.Static | BindingFlags.NonPublic)!
.Invoke(null, new object[] { error })!.ToString();
}
[Fact]
public void Should_leave_write_concern_failures_to_normal_storage_error_translation()
=> _actual.Should().Be("None");
}
#endif

#if NUNIT
[TestFixture("{ '_id': 1 }", "CommitId_Index", "Checkpoint")]
[TestFixture("{ 'BucketId': 1, 'StreamId': 1, 'CommitId': 1 }", "_id_", "Commit")]
[TestFixture("{ '_id': 1, 'Other': 1 }", "_id_", "Other")]
[TestFixture("{ 'Other': 1 }", "CommitId_Index", "Other")]
public class when_Issue92_metadata_conflicts_with_message(string pattern, string index, string expected) : SpecificationBase
{
private string? _actual;
protected override void Because()
{
var connection = new global::MongoDB.Driver.Core.Connections.ConnectionId(
new global::MongoDB.Driver.Core.Servers.ServerId(new global::MongoDB.Driver.Core.Clusters.ClusterId(),
new System.Net.DnsEndPoint("localhost", 27017)));
var error = new MongoCommandException(connection,
$"E11000 duplicate key error index: {index} dup key: {{ : 1 }}", new BsonDocument(),
new BsonDocument { { "code", 11000 }, { "keyPattern", BsonDocument.Parse(pattern) } });
var classifier = typeof(MongoPersistenceEngine).Assembly.GetType("NEventStore.Persistence.MongoDB.DuplicateKeyClassifier")!;
_actual = classifier.GetMethod("Classify", BindingFlags.Static | BindingFlags.NonPublic)!
.Invoke(null, new object[] { error })!.ToString();
}
[Fact]
public void Should_treat_the_complete_key_pattern_as_authoritative()
=> _actual.Should().Be(expected);
}
#endif
75 changes: 75 additions & 0 deletions src/NEventStore.Persistence.MongoDB/DuplicateKeyClassifier.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
using System.Text.RegularExpressions;
using MongoDB.Bson;
using MongoDB.Driver;

namespace NEventStore.Persistence.MongoDB
{
// None means this is not a recognized duplicate-key error; Other means it is a
// duplicate, but not one of the two index identities that need special handling.
internal enum DuplicateKeyKind { None, Checkpoint, Commit, Other }

// Shared by sync and async commit error handling, not the successful write path.
// Checkpoint collisions can be retried; commit collisions need duplicate-commit
// handling. Do not infer either from field names appearing in a document's values.
internal static class DuplicateKeyClassifier
{
internal static DuplicateKeyKind Classify(MongoException exception)
{
BsonDocument? details = null;
string message = exception.Message;
bool duplicate;
// Prefer the driver's category or server code over text. A typed non-duplicate
// error must not become a duplicate merely because its message mentions one.
if (exception is MongoWriteException write)
{
// Write-concern-only failures have no WriteError. Preserve them for
// normal storage-error translation, even if their text resembles a duplicate.
if (write.WriteError == null) return DuplicateKeyKind.None;
duplicate = write.WriteError.Category == ServerErrorCategory.DuplicateKey;
details = write.WriteError.Details;
message = write.WriteError.Message;
}
else if (exception is MongoCommandException command)
{
// Include historical duplicate-key codes returned by command responses.
duplicate = command.Code == 11000 || command.Code == 11001 || command.Code == 12582;
details = command.Result;
}
else
{
// Without typed metadata, accept only the anchored E11000/E11001 duplicate-key prefix.
duplicate = Regex.IsMatch(message, @"^E1100[01]\s+duplicate key\b", RegexOptions.CultureInvariant);
}
if (!duplicate) return DuplicateKeyKind.None;

// An available key pattern is authoritative, even when the message names a
// different index. Match the complete field set, not a subset of a compound
// index. Unknown patterns remain Other rather than falling back to text.
if (details != null && details.TryGetValue("keyPattern", out var pattern) && pattern.IsBsonDocument)
{
var keys = pattern.AsBsonDocument;
if (keys.ElementCount == 1 && keys.Contains(MongoCommitFields.CheckpointNumber)) return DuplicateKeyKind.Checkpoint;
if (keys.ElementCount == 3 && keys.Contains(MongoCommitFields.BucketId)
&& keys.Contains(MongoCommitFields.StreamId) && keys.Contains(MongoCommitFields.CommitId)) return DuplicateKeyKind.Commit;
return DuplicateKeyKind.Other;
}

// Without a usable key pattern, older responses identify the index in errmsg.
// Stop at the first value delimiter: a malformed or unknown header must not
// let an index-like clause inside a duplicate value select a retry strategy.
var payload = Regex.Match(message, @"\bdup key\s*:", RegexOptions.CultureInvariant);
if (!payload.Success) return DuplicateKeyKind.Other;
var header = message.Substring(0, payload.Index + payload.Length);
// Extract the complete index token from that header. Unrecognized text stays Other.
var index = Regex.Match(header, @"\bindex:\s+(?<name>\S+)\s+dup key\s*:", RegexOptions.CultureInvariant);
if (!index.Success) return DuplicateKeyKind.Other;
var name = index.Groups["name"].Value;
// CheckpointNumber is stored as MongoDB's _id. Accept the bare index name
// and the historical namespace.$index form, using ordinal exact comparisons.
// This excludes lookalike names such as _id_custom or CommitId_extra.
if (name == "_id_" || name.EndsWith(".$_id_", System.StringComparison.Ordinal)) return DuplicateKeyKind.Checkpoint;
if (name == MongoCommitIndexes.CommitId || name.EndsWith(".$" + MongoCommitIndexes.CommitId, System.StringComparison.Ordinal)) return DuplicateKeyKind.Commit;
return DuplicateKeyKind.Other;
}
}
}
11 changes: 0 additions & 11 deletions src/NEventStore.Persistence.MongoDB/MongoFields.cs
Original file line number Diff line number Diff line change
Expand Up @@ -166,17 +166,6 @@ public static class MongoCommitFields
/// </summary>
public static class MongoCommitIndexes
{
/// <summary>
/// the following value is used to determine the index
/// that throws exception when a duplicate is found.
/// </summary>
public const string CheckpointNumberMMApV1 = "$_id_";
/// <summary>
/// the following value is used to determine the index
/// that throws exception when a duplicate is found.
/// </summary>
public const string CheckpointNumberWiredTiger = "index: _id_";

/// <summary>
/// The commit stamp index.
/// </summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -207,15 +207,15 @@ public Task GetFromToAsync(long fromCheckpointToken, long toCheckpointToken, IAs
}
catch (MongoException e)
{
if (!e.Message.Contains(ConcurrencyException))
var duplicate = DuplicateKeyClassifier.Classify(e);
if (duplicate == DuplicateKeyKind.None)
{
MongoPersistenceEngineLogMessages.GenericPersistingError(Logger, attempt.CommitId, checkpointId, attempt.BucketId, attempt.StreamId, e.ToString(), e);
throw;
}

// checkpoint index?
if (e.Message.Contains(MongoCommitIndexes.CheckpointNumberMMApV1)
|| e.Message.Contains(MongoCommitIndexes.CheckpointNumberWiredTiger))
if (duplicate == DuplicateKeyKind.Checkpoint)
{
MongoPersistenceEngineLogMessages.DuplicatedCheckpointTokenError(Logger, attempt.CommitId, checkpointId, attempt.BucketId, attempt.StreamId, e);
await _checkpointGenerator.SignalDuplicateIdAsync(checkpointId, cancellationToken);
Expand All @@ -229,7 +229,7 @@ public Task GetFromToAsync(long fromCheckpointToken, long toCheckpointToken, IAs
await FillHoleAsync(attempt, checkpointId, cancellationToken).ConfigureAwait(false);
}

if (e.Message.Contains(MongoCommitIndexes.CommitId))
if (duplicate == DuplicateKeyKind.Commit)
{
var msg = string.Format(CultureInfo.InvariantCulture, MongoPersistenceEngineLogMessages.DuplicatedCommitErrorTemplate, attempt.CommitId, checkpointId, attempt.BucketId, attempt.StreamId);
MongoPersistenceEngineLogMessages.Information(Logger, msg);
Expand Down
Loading
Loading