[#942] Warn once per interval that a change is being retried, not once per delivery

[vharseko]

Fixes #942.

recoverFromReplayFailure() logged WARN_REPLAY_RETRYING_CHANGE for every delivery of a change
whose replay failed. The session is left down for ten seconds at the longest between two deliveries,
so a change which keeps failing had the same line logged every ten seconds - roughly 8600 lines a day
per stuck change - for as long as it was retried. Until #901 the five-minute constant bounded that;
now how long a change is retried belongs to the administrator, unlimited included.

The throttle

The shape of the alert next to it (UNREPLAYED_CHANGE_ALERT_INTERVAL_IN_MS): a timestamp and a CAS,
one line per domain and per minute.

Per domain rather than per change, unlike what the issue sketched. The cause which makes one
change unreplayable makes every change in flight unreplayable, and the replica whose ServerState is
held back by the barrier change is sent every change which follows it again over every restarted
session: a per-change throttle would still leave one line per change and per interval, that is one
line per change accumulated since the outage began, growing with the outage. A domain-wide throttle
bounds the log whatever the number of failing changes and whatever the budget.

The deliveries which are not logged are counted rather than dropped, so the line which is logged says
how many of them it stands for and how long the change has been failing:

severity=WARNING msgID=307 msg=Could not replay change 000001a0... in domain "ou=People,o=test"
(delivery 1, each attempted several times in place, failing for 0 ms). The change has not been
recorded as replayed: restarting the session to the replication server so that it is sent again.
0 further deliveries failed in this domain without being logged since the previous warning

The folded ones are traced, so replication debug logging still has one line per delivery.

What comes with it

• The count goes back to zero where the session restart backoff is reset - the moment nothing is
  failing anymore - or a line logged over another failure a day later would read as counting that
  failure's deliveries. How long the warning is not logged again is deliberately left alone: a
  backend which fails and recovers in turn would otherwise be one warning per failure again.
• resetSessionRestartBackoff() is resetReplayFailureTracking() now that it clears both, and
  UNREPLAYED_CHANGE_ALERT_NEVER_SENT is REPLAY_FAILURE_NEVER_REPORTED, the one origin the alert
  and the warning are both measured from.
• ERR_REPLAY_SKIPPING_CHANGE is left alone: it is logged once per change given up on, which is one
  line per divergence rather than one per delivery.

Tests

aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval: a change which can never be replayed is
delivered again and again; three failed deliveries carry one warning, and the change is warned about
again once the interval has passed, with that line saying how many deliveries it stands for. The
warnings are read from the error log of the test server, counting the records which differ - it
registers two error log publishers over one writer, so every record is kept twice.

Both halves were watched failing before the fix went in: without the throttle the first assertion
sees three warnings (expected [1] but found [6], the records doubled), and with a throttle which
logs once and never releases the second sees one (expected [2] but found [1]).

UpdateOperationTest 32/32

Rebased on master

Rebased onto 776339a8c6. The diff against master is 256 added and 14 removed lines in the same
three files, and git log origin/master..HEAD shows the single commit. Three lines more than the four
rebases before it, all in one comment: the call this branch makes is now inside a guard master put
around the line it replaces, see below.

Onto 776339a8c6 last, with two conflicts, both with #922:

• LDAPReplicationDomain: Replication: a change whose replay throws is left owned by a thread which is gone #922 wrapped the logger.warn(WARN_REPLAY_RETRYING_CHANGE, ...) this
  branch turns into logReplayRetryWarning(csn, failure) in if (!outOfMemory) - on the road out of a
  JVM which has run out of memory the line is not built, since building it asks for the memory the JVM
  has just refused. Resolved as master's guard around this branch's call: on that road neither the
  warning nor the trace of a folded delivery is built, and the delivery is not folded into the next
  warning either - the error ends the replay thread, and the uncaught exception handler of
  DirectoryThread writes the line and raises the alert for it, so it is not one which went unlogged.
  Master's comment says so, extended by that one sentence. The other hunk there was the constant each
  side adds after UNREPLAYED_CHANGE_ALERT_INTERVAL_IN_MS - REPLAY_RETRY_WARNING_INTERVAL_IN_MS
  here, REPLAY_RAN_OUT_OF_MEMORY there - both kept, this branch's first, next to the alert interval
  its javadoc refers to as "the alert above".
• UpdateOperationTest: Replication: a change whose replay throws is left owned by a thread which is gone #922 appended its tests and the message classes
  ModifyMsgWhoseAckRunsOutOfMemory, AddMsgWhoseAckThrows and AddMsgWhoseReplayIsUnwoundAfterItsAck
  at the point where this branch appends aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval and
  its replayRetryWarnings() helper, and the import each side adds next to Collections - HashSet
  there, LinkedHashSet here - collided with it. Resolved as master's file with this branch's test
  and helper inserted verbatim after master's classes, ahead of the [Issue 908] tests, plus the two
  imports this branch adds. java.util.Set is master's now, which is the one line fewer in that file.

replication.properties merged on its own: #922 appended 315-317 and no ordinal is claimed twice.
Nothing in the tree references resetSessionRestartBackoff() or UNREPLAYED_CHANGE_ALERT_NEVER_SENT,
and #922 added no call site of either. None of master's new tests reads WARN_REPLAY_RETRYING_CHANGE
from the error log, and none opens a broker of its own, so the note on server 19 below is unchanged.
The catch (Error e) #922 put inside the replay takes the road of a failed replay, so a change whose
replay keeps throwing an Error is warned about through the throttle like any other, and the
catch (Throwable) around replay() reaches the throttle through the same
recoverFromReplayFailure(). #976 (issue #949) is in the trace line of publishReplicaOfflineMsg(),
in PendingChanges and in the broker, away from the throttle.

Onto f559b0907a before that, with one conflict, of the same shape as the previous one: #928 appended
aModifyWhoseEntryDNDoesNotParseIsReportedRatherThanThrownOn to UpdateOperationTest at the point
where this branch appends aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval and its
replayRetryWarnings() helper, and the import each side adds next to ArrayList - Collections
there, LinkedHashSet here - collided with it. Resolved as master's test followed by this branch's
test and helper verbatim, both imports kept.

LDAPReplicationDomain merged on its own. Of the four master commits which touched it in this window
(#943, #967, #928, #927) only #928 is in the replay path, and it is the one comparison turned round
into SET_PERMISSIVE_MODIFY_FOR_DN.equals(...) inside replay(); the others are in the
configuration handling and the naming conflict resolution, away from the throttle.
replication.properties merged with #911 and #943 without a conflict and has no ordinal claimed
twice. #928's test takes server 19 for its CSNGenerator only and opens no broker, so the note
below on server 19 is unchanged.

Onto 21d03d579b before that, with two conflicts, both with what master added next to the lines
this branch edits rather than with the throttle itself:

• LDAPReplicationDomain: [#908] Wait for the changes being applied before a domain going down saves its ServerState #945 (issue Disabling a replication domain can drop a change a replay thread is applying #908) put the replayLock and the replay drain timeout exactly
  where this branch rewrites the javadoc of UNREPLAYED_CHANGE_ALERT_NEVER_SENT into that of
  REPLAY_FAILURE_NEVER_REPORTED. Master's block is kept whole and this branch's javadoc follows it,
  on the renamed constant. The rename and the two resetReplayFailureTracking() call sites merged on
  their own; nothing in the tree references either old name anymore.
• UpdateOperationTest: [#908] Wait for the changes being applied before a domain going down saves its ServerState #945 and [#916] Keep an update that lands during a ServerState save out of the saved flag #948 appended aChangeBeingAppliedIsRecordedBeforeTheDomainIsDisabled
  and theDomainStopsWaitingForAReplayWhichDoesNotFinish at the point where this branch appends
  aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval, and git interleaved the three. Resolved as
  master's file with this branch's test and its replayRetryWarnings() helper inserted verbatim ahead
  of master's two tests, plus the three imports this branch adds.

#944 landed in that window, not before 2a7bb9d7ed as an earlier version of this section said. Its
edit to ERR_REPLAY_SKIPPING_CHANGE_308 merged next to the WARN_REPLAY_RETRYING_CHANGE_307 line
edited here on its own, config.getReplayGiveUpDelay() is decided in recoverFromReplayFailure()
above the call to logReplayRetryWarning() and does not touch it, and the "since #901" this branch
refers to now points at master rather than ahead of it.

This branch's test and master's aChangeBeingAppliedIsRecordedBeforeTheDomainIsDisabled both open
their broker as server 19. Each stops it in a finally and the class already runs three tests as
server 2 in turn, so it is left as it is.

Onto 2a7bb9d7ed and 36d4af9bd7 before that. The one conflict was in the imports of
UpdateOperationTest, where #941 brought org.mockito.Mockito.*,
java.util.concurrent.TimeoutException and AtomicReference alongside the
org.opends.messages.ReplicationMessages.* and java.util.Set this branch adds - both sides kept.
#971 rewrote enable(), elsewhere in LDAPReplicationDomain than the throttle.

opendj-server-legacy test-compiles on the current head, and the UpdateOperationTest run recorded
above is on it rather than on an older one: 32 tests, no failures, the case this branch adds among
them.

[vharseko]

Rebased onto master (2a7bb9d7ed) to clear the conflict. Same single commit, now 70575415fc.

The conflict the description promised never came: #944 landed first, and the
ERR_REPLAY_SKIPPING_CHANGE_308 line it edits merged next to the WARN_REPLAY_RETRYING_CHANGE_307
line edited here on its own. The one which did come is in the imports of UpdateOperationTest,
where #941 brought org.mockito.Mockito.*, java.util.concurrent.TimeoutException and
AtomicReference alongside the org.opends.messages.ReplicationMessages.* and java.util.Set this
branch adds - both sides kept, all four still used, nothing else in the file conflicted.

What master changed in LDAPReplicationDomain in the meantime is elsewhere in the file and does not
touch the throttle: publishReplicaOfflineMsg() (#946) and the visibility of isServerFailure()
(#960), with the test additions of #941 above the test added here. No code of this branch moved in
the rebase - the diff against master is the same 253 added lines in the same three files as before.

opendj-server-legacy test-compiles on the rebased branch. The UpdateOperationTest 16/16 in the
description is the run from before the rebase, not a re-run on it.

[vharseko]

Rebased onto master (36d4af9bd7) to pick up the fix for #924. Same single commit, now 393c43befd,
and git log origin/master..HEAD shows only it. No code of this branch moved: the diff against
master is the same 253 added and 13 removed lines in the same three files.

The only file both sides touched is LDAPReplicationDomain.java, and in different places - #971
rewrote enable(), this branch changes the throttle around the replay-retry warning and the fields
next to it. replication.properties merged with nothing claimed twice; the ordinals in the file are
still unique after the rebase.

The red run on the pre-rebase head is not this branch. ReSyncTest.testResyncAfterImport failed in
build-maven (ubuntu-latest, 25),
one failure in 32383 tests, and it is the failure tracked in #963 - the same import sibling, the same
shape: the replica reconnects after the import announcing a state behind the entry added in between,
and the 30 s the test waits go by with nothing on the wire. Nothing there fails a replay, so the
throttle this branch adds is never reached.

Verified on the rebased branch rather than on the old head:

• opendj-server-legacy test-compiles.
• UpdateOperationTest - 17 tests, no failures, including the case this branch adds,
  aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval. The 16/16 still quoted in the
  description is the run from before the first rebase; the count above is what the class reports
  now.

[vharseko]

Rebased onto master (21d03d579b) to clear the conflict. Same single commit, now afd918cb53,
and git log origin/master..HEAD shows only it. No code of this branch moved: the diff against
master is the same 253 added and 13 removed lines in the same three files.

Two conflicts this time, both with what master added next to the lines this branch edits rather
than with the throttle itself:

• LDAPReplicationDomain - [#908] Wait for the changes being applied before a domain going down saves its ServerState #945 (issue Disabling a replication domain can drop a change a replay thread is applying #908) put the replayLock and the replay drain timeout
  exactly where this branch rewrites the javadoc of UNREPLAYED_CHANGE_ALERT_NEVER_SENT into that
  of REPLAY_FAILURE_NEVER_REPORTED. Master's block is kept whole and this branch's javadoc follows
  it, on the renamed constant. The rename and the two resetReplayFailureTracking() call sites
  merged on their own; nothing in the tree references either old name anymore.
• UpdateOperationTest - [#908] Wait for the changes being applied before a domain going down saves its ServerState #945 and [#916] Keep an update that lands during a ServerState save out of the saved flag #948 appended aChangeBeingAppliedIsRecordedBeforeTheDomainIsDisabled
  and theDomainStopsWaitingForAReplayWhichDoesNotFinish at the point where this branch appends
  aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval, and git interleaved the three. Resolved as
  master's file with this branch's test and its replayRetryWarnings() helper inserted verbatim
  ahead of master's two tests, plus the three imports this branch adds.

#944 landed in this window. The note on the first rebase said it had landed before 2a7bb9d7ed;
that was wrong, it merged on 2026-09-10 after 36d4af9bd7. Its edit to ERR_REPLAY_SKIPPING_CHANGE_308
merged next to the WARN_REPLAY_RETRYING_CHANGE_307 line edited here on its own,
config.getReplayGiveUpDelay() is decided in recoverFromReplayFailure() above the call to
logReplayRetryWarning() and does not touch it, and the "since #901" this branch refers to now
points at master rather than ahead of it. replication.properties has no ordinal claimed twice.

One thing worth a look rather than a conflict: this branch's test and master's
aChangeBeingAppliedIsRecordedBeforeTheDomainIsDisabled both open their broker as server 19. Each
stops it in a finally and the class already runs three tests as server 2 in turn, so it is left
as it is.

Verified on the rebased head: opendj-server-legacy test-compiles, and UpdateOperationTest -
20 tests, no failures, aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval among them. The
description is updated to match.

[vharseko]

Rebased onto master (f559b0907a) to clear the conflict. Same single commit, now 8461531018,
and git log origin/master..HEAD shows only it. No code of this branch moved: the diff against
master is the same 253 added and 13 removed lines in the same three files.

One conflict this time, in UpdateOperationTest, of the same shape as the previous one: #928
appended aModifyWhoseEntryDNDoesNotParseIsReportedRatherThanThrownOn at the point where this
branch appends aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval and its
replayRetryWarnings() helper, and the import each side adds next to ArrayList - Collections
there, LinkedHashSet here - collided with it. Resolved as master's test followed by this branch's
test and helper verbatim, both imports kept.

LDAPReplicationDomain merged on its own. Of the four master commits which touched it in this
window (#943, #967, #928, #927) only #928 is in the replay path, and it is the one comparison
turned round into SET_PERMISSIVE_MODIFY_FOR_DN.equals(...) inside replay(); the others are in
the configuration handling and the naming conflict resolution, away from the throttle. Nothing in
the tree references resetSessionRestartBackoff() or UNREPLAYED_CHANGE_ALERT_NEVER_SENT anymore.
replication.properties merged with #911 and #943 without a conflict; no ordinal is claimed twice.

#928's test takes server 19 for its CSNGenerator only and opens no broker, so the note on
server 19 from the previous rebase is unchanged.

Verified on the rebased head: opendj-server-legacy test-compiles, and UpdateOperationTest -
21 tests, no failures, aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval and #928's
aModifyWhoseEntryDNDoesNotParseIsReportedRatherThanThrownOn among them. The description is
updated to match.

[vharseko]

Rebased onto master (776339a8c6) to clear the conflict. Same single commit, now 4730ee2233,
and git log origin/master..HEAD shows only it. The diff against master is 256 added and 14 removed
lines in the same three files - three lines more than before, all of them in one comment, for the
reason below.

Two conflicts this time, both with #922:

• LDAPReplicationDomain - Replication: a change whose replay throws is left owned by a thread which is gone #922 wrapped the logger.warn(WARN_REPLAY_RETRYING_CHANGE, ...) this
  branch replaces with logReplayRetryWarning(csn, failure) in if (!outOfMemory): on the road out
  of a JVM which has run out of memory the line is not built, since building it asks for the memory
  the JVM has just refused. Resolved as master's guard around this branch's call. On that road neither
  the warning nor the trace of a folded delivery is built, and the delivery is not folded into the
  next warning either: the error ends the replay thread, and the uncaught exception handler of
  DirectoryThread writes the line and raises the alert for it, so it is not one which went
  unlogged. Master's comment says so, extended by that one sentence. The other hunk was the constant
  each side adds after UNREPLAYED_CHANGE_ALERT_INTERVAL_IN_MS - both kept, this branch's
  REPLAY_RETRY_WARNING_INTERVAL_IN_MS first, next to the alert interval its javadoc refers to.
• UpdateOperationTest - Replication: a change whose replay throws is left owned by a thread which is gone #922 appended its tests and three message classes at the point where this
  branch appends aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval and its
  replayRetryWarnings() helper, and the import each side adds next to Collections - HashSet
  there, LinkedHashSet here - collided with it. Resolved as master's file with this branch's test
  and helper inserted verbatim after master's classes, ahead of the [Issue 908] tests, plus the two
  imports this branch adds; java.util.Set is master's now.

replication.properties merged on its own, #922 appended 315-317 and no ordinal is claimed twice.
Nothing in the tree references resetSessionRestartBackoff() or UNREPLAYED_CHANGE_ALERT_NEVER_SENT,
and #922 added no call site of either. None of master's new tests reads the warning from the error
log, and none opens a broker of its own, so the note on server 19 is unchanged. The catch (Error e)
#922 put inside the replay and the catch (Throwable) it put around replay() both reach the
throttle through recoverFromReplayFailure(), so a change whose replay keeps throwing is warned
about like any other. #976 (issue #949) is in publishReplicaOfflineMsg(), PendingChanges and the
broker, away from the throttle.

Verified on the rebased head, in one reactor built from the worktree: opendj-server-legacy
compiles with its tests, and UpdateOperationTest - 32 tests, no failures,
aChangeWhichKeepsFailingIsWarnedAboutOncePerInterval and #922's tests among them. The description
is updated to match.