A replayed WriteOperation registers an entry container twice: RootContainer.open fails with ERR_ENTRY_CONTAINER_ALREADY_REGISTERED and masks the failure that caused the replay

[vharseko]

RootContainer.open opens and registers the entry containers of every base DN inside a single storage.write, and the WriteOperation it passes is not idempotent: it registers each entry container in entryContainers as it goes, and nothing takes those registrations back when an attempt fails. A storage that replays the operation — which Storage.write requires it to do — therefore re-runs openAndRegisterEntryContainers from the first base DN and fails with ERR_ENTRY_CONTAINER_ALREADY_REGISTERED. That failure is not the one that caused the replay: the backend fails to open, and the real cause is nowhere in the error the operator sees.

Storage-agnostic and pre-existing: it lives in the pluggable layer, not in any one backend.

The contract

• Storage.write — "In case of a write operation rollback, implementations must ensure the write operation is retried until it succeeds" — opendj-server-legacy/src/main/java/org/opends/server/backends/pluggable/spi/Storage.java:76-77.
• WriteOperation.run — "Implementation must be idempotent since operation might be retried" — opendj-server-legacy/src/main/java/org/opends/server/backends/pluggable/spi/WriteOperation.java:24-26.

The operation RootContainer.open passes is idempotent in the database and not in Java.

The path

1. RootContainer.open wraps PersistentCompressedSchema and openAndRegisterEntryContainers in one storage.write — opendj-server-legacy/src/main/java/org/opends/server/backends/pluggable/RootContainer.java:135-143.
2. openAndRegisterEntryContainers loops over the base DNs and registers each container as soon as it has opened it — RootContainer.java:218-231 (openEntryContainer at :224, registerEntryContainer at :226).
3. An attempt that fails while opening a later base DN leaves the earlier ones in entryContainers; nothing unregisters them.
4. The replay re-runs the whole operation from base DN Automate the assembly based on the Travis project #1, and registerEntryContainer throws — RootContainer.java:192-199:

EntryContainer ec = this.entryContainers.get(baseDN);
if (ec != null)
{
  throw new InitializationException(ERR_ENTRY_CONTAINER_ALREADY_REGISTERED.get(ec.getTreePrefix(), baseDN));
}

5. That is neither a conflict nor a connection failure, so no retry loop absorbs it: it propagates out of storage.write, the backend does not open, and the failure that triggered the replay is masked by it.

Two or more base DNs are needed to reach it — the registration of the last base DN is followed by little more than the nextEntryID computation at RootContainer.java:234.

What each failed attempt leaves behind

Not just the map entry. Every attempt builds fresh objects that register themselves with the configuration:

• EntryContainer's constructor registers five listeners — opendj-server-legacy/src/main/java/org/opends/server/backends/pluggable/EntryContainer.java:470-478 (addPluggableChangeListener, the index and VLV-index add/delete listeners);
• every AttributeIndex registers one — AttributeIndex.java:447 — and every VLVIndex one — VLVIndex.java:144; they are constructed and opened at EntryContainer.java:535-536 and :549-550, once per configured index per attempt;
• nothing calls close() on the containers or the indexes of the attempt that failed.

So a retried open leaks a listener set per attempt per base DN, on top of failing.

Which storages replay

• PersistIt — PDBStorage.write retries on RollbackException, without a bound — opendj-server-legacy/src/main/java/org/opends/server/backends/pdb/PDBStorage.java:629-652.
• JDBC — JDBCStorage.write retries a transaction conflict, bounded by attempts and by a wall-clock window — JDBCStorage.java:851-883 (added in Seek the primary key in the SQL Server upsert and retry a transaction conflict #867; a deadlock on the DDL of openTree reaches it).
• JE — JEStorage.write does not retry at all — JEStorage.java:885-905 — so JE never reaches this.

The database side is not the problem

The work of the operation is idempotent where it touches the database: isExistsTable guards the create, postgres uses create index if not exists, the data writes are upserts, and nextEntryID is recomputed from getHighestEntryID on every attempt. Only the Java-side state — the entryContainers map and the objects registered with the configuration — survives an attempt and breaks the next one.

What a fix has to do

Either of these, but one of them:

• make the operation idempotent, which is what the contract asks for: before an attempt runs, unregister and close() whatever the previous attempt registered — the entry containers and the indexes with them, so the listeners go with them;
• or take the registration out of the write: open the containers inside the transaction, register them after it has committed. Then a replay repeats only the work the database can undo.

How it was found

While reviewing #883, which shortens the validation of a pooled JDBC connection and replays a write whose connection the database dropped. #883 answers it on its own side — an attempt that has committed part of its work is not replayed at all, which covers the conflict replay of #867 as well as the drop replay it adds — but that only stops the JDBC backend from widening the trigger. The contract violation itself stays, and PersistIt reaches it through a plain rollback.