[#888] Name the trees of a JDBC backend from a catalog in the database

[vharseko]

Fixes #888

Problem

listTrees() answered from tree2table, a cache a tree enters the first time this process names a table for it. Nothing seeds it — open() takes a connection and sets the storage status — so a process which has opened nothing names no trees:

// JDBCStorage.java, before
public Set<TreeName> listTrees() {
	return tree2table.asMap().keySet();
}

removeStorageFiles() is the one caller running before the root container is open, and it drops exactly what listTrees() names. In the offline import-ldif the backend is configured and never opened (BackendToolUtils.getBackends() calls configureBackend() alone), so the set was empty, the drop loop was skipped, and import-ldif --clearBackend cleared a JDBC backend of nothing, without a word in the log.

Online the same command does drop the tables: ImportTask disables the backend and calls importLDIF on the Backend object it already holds, whose JDBCStorage has been serving traffic with a fully populated cache — close() does not invalidate it. JE and PDB enumerate the environment itself, so both honour the contract whatever the process did earlier.

What survived was not only leftovers

The importer clears an entry container when its first entry arrives (OnDiskMergeImporter.doImport() → beforePhaseOne(container)), so a base DN configured in the backend but absent from the imported LDIF was cleared by nothing at all: it kept its entries and went on serving them, where the same command on JE removes the whole backend directory. The option is documented as "Remove all entries for all base DNs in the backend before importing".

The rest of what stayed behind: the table of a base DN or of an index no longer configured, and the compressed-schema tables, all of them surviving an operation documented to clear everything, with the same command behaving differently offline and online.

Fix

The trees of a backend are recorded in the database, in a tree of their own: one row per tree, keyed by the tree name, with the table holding it as its value.

The catalog is per backend and named /opendj_catalog/<backendId>, so its table name follows from the configuration without asking the database anything — which is exactly what a process that has opened nothing needs — and so that backends sharing one database URL (#873) never name each other's trees. Being an ordinary tree it needs no dialect of its own: the same create table, the same upsert switch and the same statements serve it on all four engines. It is created without the index openTree() gives a tree, which serves the where k>? order by k cursor batches the catalog never runs, and it is stamped with its tree name like any other table (#866): a clear reports what it did not drop, and on a database several backends address the catalog of a neighbouring backend is otherwise the one table such a report can attribute to nobody. The stamp costs one attempt per open of the storage and not one per tree, being issued behind the very flag that keeps the catalog from being opened twice.

• openTree(createOnDemand) enrols, and nothing else does. Naming a tree in order to read it must never put it up for removal. The row is written on every open rather than only when a table is created, so that a backend upgraded from a version without a catalog fills it in at its first read-write open instead of waiting for its trees to be created again.
• A row is written before its table is created and taken out after its table is dropped, which is deliberately not a mirror image. Of the two ways a half-done change can end, a row naming a table that is not there is the one the removal is ready for — it skips such a row and logs it — while a table nothing names is adopted with its stale rows by the next open of that tree and is dropped by no clear ever after. openTree() therefore enrols first: on postgres and sql server the commit of the create table carries the row with it, and on mysql and oracle, where DDL commits the transaction it finds open before it executes, the create table commits the row before it creates anything. deleteTree() drops first for the same reason read the other way round: an unenrolment left pending before the drop is committed by the drop itself on those two engines, and would stand even where the drop then failed — ORA-00054 on a tree another session holds, say, which write() rethrows unreplayed, it being neither a class 40 state nor ORA-00060. The delete carries a commit of its own instead of being left to the enclosing transaction, that transaction being the last thing which could still roll it back over a table already gone — and nothing would ever put that right, a deleted tree not being opened again.
• The compressed schema trees are the exception. Named from a literal, they are the same pair for every backend of a database (JDBC backends sharing a database URL share one pair of compressed-schema tables #873), so they are never enrolled: a backend must not offer for removal a tree another one may be the only owner of, and that pair is deliberately left where it lies ([#873] Give each backend its own compressed schema trees #881). The trees the fix of JDBC backends sharing a database URL share one pair of compressed-schema tables #873 names after the backend id are enrolled like any other.
• What a tool is shown is not what a clear may drop. catalogTables() answers the removal — the catalog's rows and the catalog itself, the table taken from the row rather than recomputed. listTrees() answers dbtest, and adds the shared compressed schema trees whose tables are there, so that list-raw-dbs counts them and dump-raw-db --dbName compressed_schema/… goes on resolving their names, as it did before the catalog.
• removeStorageFiles() drops the catalog last and skips a table that is not there. Dropping a table is DDL, which mysql and oracle commit as they go, so an attempt which fails halfway is finished by the next one instead of leaving behind tables nothing names any more. A skipped row is logged, and a clear which dropped nothing at all while there was something to drop is logged on top of that — that silence was the whole of JDBC backend: removeStorageFiles() drops only the tables this process has touched, so an offline import-ldif --clearBackend clears nothing #888. The lookup deciding between the skip and the drop is narrowed to the database and the schema of the connection: asked with a null catalog, Connector/J answers for every database of the server, and a table of the same name next door would turn a skip that keeps the clear going into an unqualified drop table of a table that is not here, failing the whole clear on this attempt and on every one after it.
• What a clear did not remove is reported, and reported as what it is. A table is named after the hash of its tree name, so its name says nothing about whose it is — but the comment Stamp JDBC backend tables with their tree name and refresh optimizer statistics after import #866 stamps it with says exactly that, in plain text. Once everything the catalog named is gone, the opendj tables still standing in the catalog and the schema of the connection are read for that stamp: one naming a tree of a base DN this backend does not serve belongs to a backend sharing the database (JDBC backends sharing a database URL share one pair of compressed-schema tables #873) and is passed over in silence; one naming a tree of this backend — an index taken out of the configuration while the backend was disabled, say — is reported as this backend's own, and so as removable by hand; a table carrying no stamp at all can be attributed to nobody and is reported as exactly that. The scan is scoped to getCatalog()/getSchema() because Connector/J 8 answers a null catalog for every database of the server.

isExistsTable() moved to the storage itself, unchanged, since the removal needs it outside a transaction now; the catalog lookup of #885 is untouched.

Not fixed by this

Enrolment covers the trees the current configuration opens read-write, and those alone. A table left by a tree that is no longer configured — an attribute index removed while the backend was disabled, or anything at all predating the upgrade — is named by no catalog, cannot be attributed to a backend, and is therefore reported after every clear rather than dropped. Re-adding such an index adopts the surviving table with its pre-clear rows, exactly as it did before this change; only the report is new.

A backend upgraded in place has to be started once before its first offline --clearBackend. BackendImpl.importLDIF clears the storage before it opens the root container, and nothing enrols a tree earlier, so on an installation whose tables predate this change that first offline clear finds no catalog and drops nothing — as on master, except that it now says so in the log rather than going by in silence. Every clear after the first read-write open is complete.

The compressed-schema tables of a backend are cleared once #881 gives each backend its own pair — this should land after #881 for that reason, and because the two touch the same methods.

Tests

Eight cases in jdbc/TestCase, inherited by all four engine suites:

case	asserts
testABackendIsClearedByAProcessThatNeverOpenedIt	a storage which never opened the backend names its tree and drops its tables — while the tables of another backend of the same database stay where they are, and are named in nothing this clear reports
testADeletedTreeIsNoLongerNamedByTheCatalog	a dropped tree leaves the catalog with its table, and the clear that follows does not stumble over it
testADeletedTreeStaysOutOfTheCatalogWhenItsTransactionFails	a write() throwing after deleteTree leaves the table dropped and the row gone: the delete is owed to no transaction that could still roll it back
testAClearSkipsACatalogRowWhoseTableIsGone	a row whose table was dropped behind the catalog's back neither fails the clear nor stops it dropping the rest
testAClearDropsTheTableTheCatalogRecords	the row records the table holding its tree, and a row recording none falls back to the name derived from the tree
testAClearReportsTheTablesItCanAttributeToThisBackend	an uncatalogued table of a base DN this backend serves is reported as its own, and nothing of a neighbouring backend's — its trees or its catalog — is reported at all
testReadingATreeDoesNotPutItUpForRemoval	openTree(tree, false) enrols nothing: a tree of another backend, read but not owned, survives this one's clear
testTheSharedCompressedSchemaTreesAreNamedButNeverCleared	both trees of the shared pair are named by listTrees() and left standing by a clear which drops the backend's own trees

The first fails on master: the offline storage names no tree at all and the clear drops nothing.

PluggableBackendImplTestCase.testImportLDIF already finalizes the backend and imports with setClearBackend(true) on master, but it keeps the same JDBCStorage instance with tree2table intact, so the clear it runs was never the offline one. It is unchanged here, and the cases above cover that path alone.

suite	result
PgSqlTestCase	61/61, no skips
MySqlTestCase	61/61, no skips
MsSqlTestCase	61/61, no skips
OracleTestCase	61/61, no skips

[maximthomas]

The fix for #888 is real and well scoped: the catalog makes an offline --clearBackend name the trees it has to drop, the catalog-last ordering is right, and the two new cases cover the path that was broken. Two things to settle before merge — one consistency hole in the catalog itself, one sentence in the description that is not true as written.

Stale catalog row outlives its table (major)

opendj-server-legacy/src/main/java/org/opends/server/backends/jdbc/JDBCStorage.java:1349-1356

deleteTree() commits the drop table immediately, then leaves the matching catalog row to the caller's transaction:

try (final PreparedStatement statement=con.prepareStatement("drop table "+getTableName(treeName))) {
    execute(statement);
    con.commit();          // the table is gone here
}
...
unenrolFromCatalog(treeName);   // -> delete(catalog, ...): rolls back with the enclosing write()

Delete an index (dsconfig delete-backend-index, a base DN removal, EntryContainer.clear()) and let anything later in the same transaction fail terminally — write() replays only class-40 conflicts and rethrows the rest unreplayed — and the drop stands while the row rolls back.

The row is then permanent: a deleted tree is never opened again, so nothing re-enrols or re-deletes it. listTrees() returns a TreeName whose table does not exist, and BackendStat.listRawDBs opens a cursor per listed tree, so dbtest list-raw-dbs fails with a StorageRuntimeException for good. removeStorageFiles() survives only because it has the isExistsTable skip at :862; listTrees() has no such guard.

Two lines: commit the unenrol with the drop, the way :1214 and :1307 already commit their DDL.

An uncatalogued table is never dropped, and after the first open never reported either (major)

opendj-server-legacy/src/main/java/org/opends/server/backends/jdbc/JDBCStorage.java:856-857

final Set<TreeName> trees=listTrees(con);
if (trees.isEmpty()) {
    reportUncataloguedTables(con);
} else { ...drop loop... }

listTrees(Connection) returns an empty set only while the catalog table is absent — the moment it exists, :1679 unconditionally does trees.add(catalog). So the report branch is dead from the first read-write open on, and enrolment only ever covers the trees the current configuration opens (enrolInCatalog is called from openTree(createOnDemand) alone).

An attribute index removed from the configuration while the backend was disabled, or before the upgrade, leaves an opendj_<hash> table the catalog never learns of. Every later import-ldif --clearBackend drops the catalogued trees, takes the else branch, and leaves that one untouched and unmentioned. If the index is re-added, openTree only creates a table when !isExistsTable (:1210), so it adopts the survivor with its pre-clear rows — the index then returns entry IDs the reimported id2entry does not have.

This makes the description's "the first read-write open after the upgrade makes the next clear complete" false for any tree the current configuration does not open. Correct the sentence at least; better, fire the report whenever the connection can see opendj tables the catalog does not name, not only when the catalog table is missing.

The compressed-schema trees disappear from listTrees() (minor)

opendj-server-legacy/src/main/java/org/opends/server/backends/jdbc/JDBCStorage.java:1278

Not dropping their tables is deliberate (#881), but the enrolment skip also removes them from the tool-facing listing. On master they were in tree2table after any open, read-only included, because PersistentCompressedSchema.load cursors them. Now dbtest dump-raw-db --dbName compressed_schema/compressed_attributes fails name resolution in BackendStat.getStorageTreeName and list-raw-dbs silently undercounts. They are computable from the constant — add them to the listTrees() result without enrolling them.

A clear that drops nothing still says nothing (minor)

opendj-server-legacy/src/main/java/org/opends/server/backends/jdbc/JDBCStorage.java:862

if (!isExistsTable(con, tableName)) { // a row of the catalog outliving its table
    continue;
}

No log line, and removeStorageFiles() returns normally. If every derived table name misses, --clearBackend again clears nothing without a word in the log — the exact failure mode of #888 — and BackendImpl.importLDIF goes on to import into surviving data. Log the skip, or count the skips and warn once when nothing at all was dropped.

Nits

• openTree has the same asymmetry as deleteTree (JDBCStorage.java:1214 vs :1286): the create table commits, the upsert(catalog, ...) does not. A terminal failure mid-open leaves committed tables and an empty catalog table — and since listTrees adds the catalog itself, the clear takes the else branch and drops only the catalog. Self-heals at the next successful open, since enrolment runs on every open, which is why this is a nit and deleteTree is not.
• The catalog's value column is written and never read (JDBCStorage.java:1286 vs :1680): listTrees does select k only and the drop loop recomputes getTableName(treeName). The one fact that would make the catalog robust to a change of the naming function is recorded and ignored. Either read v, or stop writing it and drop "the table holding that tree as its value" from the javadoc.
• The uncatalogued count is unscoped (JDBCStorage.java:928): getTables(null, null, "opendj%", ...) — null catalog and null schema counts other backends' tables on a shared URL (#873), their catalogs, and on Connector/J 8 (nullCatalogMeansCurrent=false) every database on the server. Pass con.getCatalog()/con.getSchema(), and say in the message that the count spans the connection.
• The description credits itself with a test change that is not in the diff: "PluggableBackendImplTestCase now really exercises the drop" — that file is untouched, and finalizeBackend() + setClearBackend(true) already exist on master at :1005-1016. That case also keeps the same JDBCStorage instance with tree2table intact, so it never covered the offline path. The two new jdbc cases cover it alone.
• The three guards the fix rests on are covered by no test: the stale-row skip at :862 is unreachable from testADeletedTreeIsNoLongerNamedByTheCatalog because deleteTree unenrols first — the test passes with the skip deleted; nothing asserts that openTree(tree, false) does not enrol; nothing asserts the compressed-schema skip. The last two are what stop one backend offering another's trees for removal.
• The new tests leak tables on failure (opendj-server-legacy/src/test/java/org/opends/server/backends/jdbc/TestCase.java:1389, :1425): neighbour.removeStorageFiles() is the last statement of the body, after five assertions, outside any try/finally. dropStaleTrees runs only in @BeforeClass and cleanUp() drops nothing, so one failure leaves opendj_ tables alive for the rest of the class. Move both clears into finally.
• Two stale artefacts (JDBCStorage.java:1357, :910): the comment "forget the mapping so listTrees() consumers (updateTableStatistics) skip the dropped table" is no longer true — listTrees() does not read tree2table and updateTableStatistics is only called with writtenTrees; and if (trees.contains(catalog)) is always true at its only call site.

Not covered here: nothing was run against an engine. MsSql and Oracle are unexercised on both sides, and the suites skip rather than fail when a container does not start — worth confirming the new create table <catalog> and select k from <catalog> on those two before this lands.

[vharseko]

Thank you — every point is addressed in e49fce2, and all four engines are now run. Two of the fixes are not the ones suggested; the reasoning is below, together with one consequence I think was overstated.

Stale catalog row outliving its table (major) — fixed, by reordering

Confirmed exactly as described: con.commit() after the drop table commits the whole enclosing transaction, so the delete of the row was the only part of deleteTree still owed to it, and write() replays a class 40 conflict and rethrows everything else unreplayed.

Rather than committing the unenrol after the drop, unenrolFromCatalog now runs before it, so the commit of the DDL carries the row with it:

unenrolFromCatalog(treeName);
if (isExistsTable(treeName)) {
    try (final PreparedStatement statement = con.prepareStatement("drop table " + getTableName(treeName))) {
        execute(statement);
        con.commit();
    }
    ...
}

One statement moved instead of one added, and it behaves better in the case the suggestion does not cover: if the drop itself fails, the pending row delete goes back with the transaction, leaving the tree named and its table standing — both still there — where a commit placed after the drop would have had nothing to commit anyway.

One correction to the consequence. dbtest list-raw-dbs does not fail: listRawDBs calls appendStorageTreeStats, which catches Exception and fills the row with appendStatsNoData (BackendStat.java:806-809), so a stale row shows as a tree with dashes rather than an error. dump-raw-db --dbName <that tree> does fail and return 1. The row is permanent as described, and the fix stands on that alone.

An uncatalogued table is never dropped, nor reported past the first open (major) — report fixed, the drop deliberately not

Both halves confirmed: catalogTables() (was listTrees(con)) adds the catalog unconditionally, so the old trees.isEmpty() branch was dead from the first read-write open, and openTree adopts an existing table (if (!isExistsTable)), stale rows included.

reportUncataloguedTables is now reportClearOutcome, and it runs after every clear, once everything the catalog named is gone: whatever opendj table is still standing is by definition named by no catalog of this backend. It is scoped to con.getCatalog()/con.getSchema() — your Connector/J 8 point — and the shared compressed schema pair is excluded from the count, since reporting it after every clear would be asking an administrator to remove the one thing this code goes out of its way to spare.

The table itself is still not dropped, and I do not think it can be: nothing about opendj_<hash> says whose it is, and on a shared database (#873) guessing would drop another backend's data. That is now stated plainly in Not fixed by this rather than implied away, and the false sentence you flagged is gone.

The compressed schema trees disappear from listTrees() (minor) — fixed, but not by adding them to listTrees()

The regression is real. The suggested fix is not safe as written, though: removeStorageFiles() iterates exactly what listTrees() returns, so adding the pair there would make --clearBackend drop them — precisely what the enrolment skip exists to prevent.

Split instead:

• catalogTables(Connection) — the rows of the catalog and the catalog itself, nothing else. This is what the removal reads.
• listTrees() — the same, plus the shared pair, each added only when isExistsTable says its table is there (so that once [#873] Give each backend its own compressed schema trees #881 gives each backend its own pair, an installation holding neither table does not start naming two trees that do not exist).

list-raw-dbs counts them again and dump-raw-db --dbName compressed_schema/compressed_attributes resolves again, with the clear leaving them where they lie.

A clear that drops nothing still says nothing (minor) — fixed

The skip logs the tree and the table it could not find, and a clear which dropped nothing at all while there was something to drop warns on top of that. The second condition matters: a first-ever --clearBackend on an empty backend legitimately drops nothing, and warning there would be noise.

Nits

• openTree asymmetry — fixed the same way as deleteTree, mirrored: enrolInCatalog now runs before the create table, so the DDL commit carries the row. Of the two ways a half-done open can end, a row naming a missing table is the one the removal is ready for; a table nothing names is the one adopted with stale rows and dropped by no clear ever after.
• The value column written and never read — now read. catalogTables does select k,v and the drop loop uses the recorded table name, falling back to getTableName(treeName) for an empty v.
• The uncatalogued count unscoped — scoped, as above. Note that isExistsTable still passes null, null: that is master's code moved verbatim, it runs per tree per open, and narrowing it changes table detection on four engines for a hazard that predates this PR. Worth its own change; I did not want it hidden in this one.
• The description crediting a test change not in the diff — removed. The section now says what is true: PluggableBackendImplTestCase.testImportLDIF already does this on master, keeps the same JDBCStorage with tree2table intact, and therefore never covered the offline path.
• The three uncovered guards — three cases added: testAClearSkipsACatalogRowWhoseTableIsGone (the table is dropped behind the catalog's back, which no code path of the backend does), testReadingATreeDoesNotPutItUpForRemoval (a tree of another backend, read but not owned, survives this one's clear), testTheSharedCompressedSchemaTreesAreNamedButNeverCleared.
• Tests leaking tables on failure — both existing cases clear in a finally now, through a clearQuietly helper that never lets the failure of a clear replace the failure being reported.
• The two stale artefacts — both gone. catalogDroppedLast went with them: the ordering now comes from the LinkedHashMap catalogTables builds, so there is no always-true contains left to be always true. The deleteTree comment is rewritten; since tree2table is a pure SHA-224 of the tree name, invalidating it was never more than dropping a memo, and the comment now says so instead of claiming listTrees() consumers depend on it.

Engines

All four run locally, none skipped:

suite	result
PgSqlTestCase	58/58, no skips
MySqlTestCase	58/58, no skips
MsSqlTestCase	58/58, no skips
OracleTestCase	58/58, no skips

53 on master, 58 here — the two cases of the first push plus the three above, on every engine.

[maximthomas]

Re-traced all eleven of the previous round against e49fce2 rather than taking the reply's word: they are genuinely addressed, and your correction about list-raw-dbs is right — appendStorageTreeStats catches and prints dashes, only dump-raw-db exits 1. Three things the new commit introduces or leaves.

A successful clear tells the operator to hand-remove another live backend's tables (major)

reportClearOutcome now runs after every clear, not only when the catalog table was missing, and counts every opendj% table in con.getCatalog()/con.getSchema() minus only the shared compressed-schema pair.

Table names are a bare SHA-224 of the tree name with no backend id (JDBCStorage.java:186-192), and the catalog table /opendj_catalog/<backendId> is itself hashed (:238-240). So on the shared-database layout of #873 the count covers backend B's ~25 live tables plus B's own catalog, and the warning ends these have to be removed by hand.

The PR's own test reproduces it. createBackendCfg gives every backend the same URL (opendj-server-legacy/src/test/java/org/opends/server/backends/jdbc/TestCase.java:128-134), and testABackendIsClearedByAProcessThatNeverOpenedIt asserts at :1408 that the neighbour's table is still standing after the clear — so leftBehind >= 2 and the warning fires inside a green test, with nothing asserting on the log.

The provenance you need already exists: #866 stamps every table with its tree name in the table comment, which getTables() returns as REMARKS. Skip a table whose stamp names a tree of a base DN this backend does not serve, and the count becomes true. Failing that, drop the removed by hand clause.

The reordering's guarantee holds on two engines of four (major)

if the drop itself fails, the pending row delete goes back with the transaction

That is PostgreSQL and SQL Server. MySQL and Oracle implicitly commit the pending transaction before executing DDL — the file says so itself at JDBCStorage.java:1796-1799, "DDL, which mysql and oracle commit as they go". There the DELETE lands first and the drop runs as a separate transaction:

unenrolFromCatalog(treeName);                 // DELETE, pending
if (isExistsTable(treeName)) {
    try (final PreparedStatement statement = con.prepareStatement("drop table " + getTableName(treeName))) {
        execute(statement);                   // mysql/oracle: commits the DELETE, THEN drops
        con.commit();
    }
}

On Oracle drop table needs an exclusive lock and ddl_lock_timeout defaults to 0, so a concurrent transaction on the tree gives ORA-00054 at once. That is SQLState 61000, and isConflict (:1121-1136) replays only SQLState 40* plus ORA-00060 — so write() rethrows it unreplayed, the row is committed-deleted and the table stands. By your own comment at :1272-1275 that is the worse half: a table nothing names, adopted with its stale rows by the next open and dropped by no clear ever after. Reachable from AttributeIndex.deleteIndex (AttributeIndex.java:1019-1024) and per-tree inside EntryContainer.clear()'s loop (EntryContainer.java:2560-2568). MySQL's equivalent is 1205 / SQLState 40001, so it replays and self-heals; Oracle is the live case.

To be clear, the bug this PR was reopened for — a terminal failure later in the same write() rolling the row back over a dropped table — is fixed on all four engines. What is wrong is the invariant asserted around it. Either give the unenrol its own con.commit() after the drop's, which is correct everywhere and converges on retry, or say in the comment at :1412-1420 and at TestCase.java:1458 that the guarantee is PostgreSQL/SQL Server only.

The first offline clear after an upgrade still drops nothing (major)

BackendImpl.java:670-673 clears the storage before it opens the root container at :681. catalogTables() returns Collections.emptyMap() when the catalog table is absent (JDBCStorage.java:1774), and nothing enrols earlier — enrolInCatalog is reached only from openTree's createOnDemand branch (:1275), which the read-write open after the clear takes.

So on an installation whose tables predate this PR — the state #888 is filed about — an offline import-ldif --clearBackend iterates zero rows and drops zero tables, exactly as on master. The residue is every tree the import does not rewrite: another base DN in the same backend, an index not rebuilt. It self-heals from the second clear on, and it does now warn.

This is a documentation change, not a code one. "Not fixed by this" currently names only trees the config no longer serves; it should also say that a backend upgraded in place must be started once before its first --clearBackend import.

The skip that keeps a clear going is unreliable on MySQL (minor)

The drop loop's guard passes a null catalog while the report ten lines below is scoped:

if (!isExistsTable(con, tableName)) { missing++; continue; }   // :874 -> getTables(null, null, ...)
...
metaData.getTables(catalog, schema, storedIdentifier(metaData, "opendj%"), ...);   // :947

opendj-server-legacy/pom.xml:294-303 pins mysql-connector-j 9.2.0, whose nullDatabaseMeansCurrent defaults to false and databaseTerm to CATALOG — a null catalog searches every database on the server and schemaPattern is ignored. Two OpenDJ databases on one MySQL server with the same backend id and suffix produce identical opendj_<hash> names, so a stale row whose local table is gone matches the twin next door, the skip is not taken, the unqualified drop table throws 1051, and the catch at :887 aborts the whole --clearBackend — deterministically, on every retry. That is the designed skip-and-warn turning into a hard failure, and testAClearSkipsACatalogRowWhoseTableIsGone cannot see it in a single-database container.

Agreed that narrowing isExistsTable in general belongs in its own change; this one call site now decides between "skip" and "drop", which the others do not.

Nits

• The fix is pinned by no test: testADeletedTreeIsNoLongerNamedByTheCatalog (TestCase.java:1424-1456) commits its write() cleanly and injects no failure after deleteTree, so it passes with the reordering reverted; the mirrored openTree reorder is asserted by nothing at all. A case that throws after deleteTree inside the same WriteOperation would pin it.
• Two branches still commit nothing: deleteTree's skip branch (:1421, table already gone) leaves the DELETE to the enclosing write() — pre-existing, but the fix passes right by it, and AttributeIndex.deleteIndex is not masked by a later tree in a loop. In openTree, enrolInCatalog is committed on every open only on postgres, where create index if not exists + commit sits outside the !isExistsTable block (:1288-1294); mysql/oracle commit only when the index is actually created (:1295-1318) and mssql has no index branch (:1319), so a reopen leaves ~25 upserts pending on three engines.
• The "dropped nothing" warning prints the wrong number: if (dropped==0 && (leftBehind>0 || missing>0)) logs only missing, so the #888 state — empty catalog, tables standing — reads the clear dropped no table at all, 0 of the trees its catalog names having lost their table already. The condition that fired was leftBehind, which is never printed. Also, the noise suppression you describe holds only on a database that holds no other opendj% table.
• clearQuietly swallows the code under test: TestCase.java:157-164 catches every Exception from removeStorageFiles(), and in testAClearSkipsACatalogRowWhoseTableIsGone and testTheSharedCompressedSchemaTrees... it is the only clear in the case. The neighbour's clear also lost the assertion it had before. Worth catching in the cleanup-only position and asserting where the clear is the subject.
• Half the shared pair is untested: testTheSharedCompressedSchemaTreesAreNamedButNeverCleared uses SHARED_COMPRESSED_SCHEMA_TREES.get(0) only. Element 1 is a hand-copy of a PersistentCompressedSchema private, and a wrong literal there would silently un-name and un-spare that tree.
• The v fallback is unexercised: every catalog row a test writes has v == getTableName(k), so neither the recorded-name path nor the empty-v fallback at :1790-1793 is covered — a swapped k/v would pass the suite. Same for the LinkedHashMap catalog-last ordering at :1800-1801, which matters only on a half-failed removal no test induces.
• unenrolFromCatalog lacks the guard enrolInCatalog has: :1342 returns early for SHARED_COMPRESSED_SCHEMA_BASE_DN, :1378 does not. No live caller today, but the asymmetry is what would let a deleteTree drop the shared pair.
• dbtest on an upgraded, never-started installation: listTrees() now names only the shared pair until the first read-write open, so dump-raw-db --dbName /dc=example,dc=com/id2entry exits 1 where the base version resolved it via the tree2table memo. Narrow, and it disappears after one start.
• Test setup still leaks: the setup half of testABackendIsClearedByAProcessThatNeverOpenedIt (:1390-1391) has a finally that only closes, so a throw there exits before the second try/finally. And the new test helper isExistsTable (:136-146) repeats getTables(null, null, null, ...), walking every accessible schema once per assertion.

[vharseko]

Thank you — all three majors are addressed in 87d3bb5, and so are the nits bar two I disagree with, with the reasoning below. All four engines run again.

A successful clear tells the operator to hand-remove another live backend's tables (major) — fixed

Confirmed, and the case in this very PR reproduces it exactly as you describe: the neighbour keeps two tables (its tree and its own catalog), so the old leftBehind counted 2 and the line ended these have to be removed by hand inside a green test.

The provenance you point at is the right one, but not through getTables(): the MS SQL Server driver returns no REMARKS for a table at all, and the Oracle driver returns it only with remarksReporting=true, so the readback would have been true on two engines and blind on the other two. readStoredComment() — the per-dialect readback #866 already carries — is used instead, once per table still standing, which on a clear that left nothing behind is no query at all.

What the report says now, per table:

• a stamp naming a tree of a base DN this backend does not serve → a table of a backend sharing the database (JDBC backends sharing a database URL share one pair of compressed-schema tables #873): passed over in silence;
• a stamp naming a tree of this backend → reported as this backend's own, and only there is removing it by hand suggested;
• no stamp at all → reported as attributable to nobody, and left exactly where it is.

One deliberate reversal. The catalog table is now stamped like any other table. A neighbouring backend's catalog is otherwise the one table such a report can attribute to no one, and it would then be reported as "unattributable" after every clear on a shared database — the same false alarm in a quieter voice. It costs one stamp attempt per open of the storage rather than one per tree, being issued behind the very flag that keeps the catalog from being opened twice. The description of the catalog in the PR text is corrected accordingly.

testAClearReportsTheTablesItCanAttributeToThisBackend asserts both halves: an uncatalogued table of a base DN this backend serves is reported as its own, and testABackendIsClearedByAProcessThatNeverOpenedIt now asserts that neither the neighbour's tree nor the neighbour's catalog is named in what the clear reports.

The reordering's guarantee holds on two engines of four (major) — fixed, by the order you named

Confirmed, and the invariant I wrote around it was wrong for mysql and oracle for exactly the reason you give. deleteTree() now drops, commits, unenrols and commits again:

if (isExistsTable(treeName)) {
    try (final PreparedStatement statement = con.prepareStatement("drop table " + getTableName(treeName))) {
        execute(statement);
        con.commit();
    } ...
}
if (unenrolFromCatalog(treeName)) {
    con.commit();
}

Which is what the rule always was, once stated without the word "mirror": a row is written before its table is created and taken out after its table is dropped. openTree() enrolling first and deleteTree() unenrolling last are the same rule read in the two directions — the state to avoid is a table nothing names, and the state the removal is ready for is a row whose table is gone. The comments at both sites say that now, and no longer claim a symmetry that would be wrong.

The unenrol carries its own commit for the bug this was reopened for, so both halves hold on all four engines: a failed drop leaves tree and table both there (nothing is pending before it), and a failure after the drop leaves at worst a row the clear skips and logs. deleteTree's "table already gone" branch commits its delete too, which it did not before.

The first offline clear after an upgrade still drops nothing (major) — documented, in two places

Confirmed: BackendImpl.java:669-673 clears before newRootContainer at :681, and nothing enrols earlier. Not fixed by this now says that a backend upgraded in place has to be started once before its first offline --clearBackend, and the log line for a clear that dropped nothing says it too — that is where an administrator meets the situation.

The skip that keeps a clear going is unreliable on MySQL (minor) — fixed at the call sites that decide a drop

isExistsTable got an overload taking a catalog and a schema, and the removal passes con.getCatalog()/con.getSchema() — in the drop loop, in catalogTables() (a twin catalog next door would have sent the select k,v at a table that is not here) and in the shared-pair lookup of listTrees(). openTree's per-tree call is untouched, as agreed: there a false "it is there" only skips a create table, which is the hazard that predates this PR.

Nits

• The fix pinned by no test — testADeletedTreeStaysOutOfTheCatalogWhenItsTransactionFails added: a WriteOperation that deletes a tree and then throws. It pins the half that a refactor could undo, the delete being owed to no transaction that could still roll it back; the mysql/oracle half is now pinned by construction, nothing being pending before the drop at all.
• Two branches committing nothing — deleteTree's skip branch fixed as above. In openTree the enrolment still rides the enclosing transaction where no table and no index are created, which is the reopen of an unchanged backend: the rows are already there, so what a rollback loses is a rewrite of what it rewrote, and the next open writes them again.
• The "dropped nothing" warning printing the wrong number — it prints all three now: rows whose table was already gone, tables of this backend named by no catalog, and tables attributable to nobody.
• clearQuietly swallowing the code under test — I do not think it does. testAClearSkipsACatalogRowWhoseTableIsGone calls storage.removeStorageFiles() in the body at TestCase.java:1487, and testTheSharedCompressedSchemaTrees… at :1564; the clearQuietly in the finally of each is the cleanup the previous round asked to be moved there ("Move both clears into finally"). The neighbour's clear is cleanup in the same sense — the case asserts on the neighbour's table before it, not on the clear.
• Half the shared pair untested — both trees now, in the open, in the listTrees() assertion and in the assertion that a clear left them standing.
• The v fallback unexercised — testAClearDropsTheTableTheCatalogRecords asserts that a row records the table holding its tree (a swapped k/v fails it), then empties every v behind the catalog's back and asserts the fallback answers with the derived name and the clear still drops the table.
• unenrolFromCatalog lacking the guard — added, with a comment saying what it is: symmetry, not protection. It could not have stopped a deleteTree from dropping the shared pair, the drop happening whatever the catalog says, and there is no row of that pair for the delete to find.
• dbtest on an upgraded, never-started installation — I checked this one and do not think it is a regression. BackendStat opens through getReadOnlyRootContainer() (:917), i.e. AccessMode.READ_ONLY, and this backend's openTree(treeName, false) does nothing whatsoever — it does not so much as call getTableName. So on master tree2table is empty at that point too and listTrees() returns nothing there either. Under this PR the same run returns the shared compressed schema pair, which is one tree more than before, not one less.
• Test setup leaking — the setup half of the offline clear case clears both backends when it throws, before the case's own finally is ever reached. The test helper isExistsTable still lists what the connection can see, deliberately: it is there so that no folding rule of the backend under test is trusted by the assertions.

Engines

suite	result
PgSqlTestCase	61/61, no skips
MySqlTestCase	61/61, no skips
MsSqlTestCase	61/61, no skips
OracleTestCase	61/61, no skips

58 on the previous push, 61 here — the three cases above, on every engine.