Replication replay records a failed operation as applied: the ServerState advances past the change and the assured ack reports success

[vharseko]

A change whose replay fails with anything other than NO_OPERATION, BUSY or UNAVAILABLE is recorded as replayed: the ServerState advances past it, the replication server never sends it again, and an assured (SAFE_READ) ack goes back to the originating master as if the change had been applied. The replica silently diverges while reporting itself fully caught up, with unresolved-naming-conflicts at 0 and a single line in the error log.

This is storage-agnostic — JE, PersistIt and JDBC all reach it on any StorageRuntimeException.

The path

1. A backend failure surfaces as StorageRuntimeException and becomes a DirectoryException carrying server-error-result-code, 80 (OTHER) by default — BackendImpl.createDirectoryException(), opendj-server-legacy/src/main/java/org/opends/server/backends/pluggable/BackendImpl.java:963-980; the default is in opendj-maven-plugin/src/main/resources/config/xml/org/forgerock/opendj/server/config/GlobalConfiguration.xml:171-179.
2. LDAPReplicationDomain.replay() retries only NO_OPERATION, BUSY and UNAVAILABLE — opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:2340-2360. Every other result code goes to solveNamingConflict().
3. solveNamingConflict() ends its else branch with a log line and return true — "The other type of errors can not be caused by naming conflicts. Log a message for the repair tool." (LDAPReplicationDomain.java:2660-2668, and the same shape in the Delete, Add and ModifyDN overloads; the ModifyDN one even increments numResolvedNamingConflicts on the way out).
4. true means replayDone, so replay() calls updateError(csn) — LDAPReplicationDomain.java:2399-2404.
5. updateError() is remotePendingChanges.commit(csn), which marks the change committed and calls state.update(csn) unconditionally — opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/RemotePendingChanges.java:169-199.
6. The resume cursor of the replication server is positioned AFTER_MATCHING_KEY — opendj-server-legacy/src/main/java/org/opends/server/replication/server/ReplicationServerDomain.java:1346 — so the change is never resent.
7. replayErrorMsg is left null on this path, so processUpdateDone(msg, null) publishes an AckMsg with no replay error — opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationDomain.java:3451-3480. For SAFE_READ the originating master is told the write is durable on this replica.

Net effect: the change is lost on this replica, both servers believe they are in sync, and recovery is a manual dsreplication initialize.

Not fixable by configuration

Setting server-error-result-code to 52 (UNAVAILABLE) does not help. The retry budget is 10 attempts, and once it is exhausted replay() still ends in updateError(csn): the ServerState advance is unconditional on every terminal path of the method.

What a fix has to separate

The else branch of the four solveNamingConflict() overloads currently collapses two different outcomes into one return true:

• the operation became a no-op after conflict resolution — the change is genuinely done, and committing it to the ServerState is right;
• the operation failed — the change is not applied, and it must not be committed to the ServerState, nor acked clean. It has to stay pending, or the domain has to be taken to a broken/degraded state, and the ack has to carry the replay error so an assured write is never reported durable.

How it was found

While reviewing #883, which shortens the validation of a pooled JDBC connection: a connection the database dropped raises a StorageRuntimeException there, and the review traced where such a failure ends up. #883 answers it on its own side — a dropped connection is replayed on a fresh one and the pool is marked distrusted — but this path stays as it is for every other storage failure.