branch-4.1: [feat](cache-memory) add external metadata cache memory governance - #66717
branch-4.1: [feat](cache-memory) add external metadata cache memory governance#66717CalvinKirs wants to merge 1 commit into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
e87de3d to
903b7bc
Compare
|
/review |
|
run buildall |
CalvinKirs
changed the title
903b7bc to
85a09c6
Compare
CalvinKirs
changed the title
|
run buildall |
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE UT Coverage ReportIncrement line coverage |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Request changes. Full-scope review completed across all 52 changed files and three convergence rounds. Two correctness blockers remain: weighted Iceberg schema/spec DDL fails on Hadoop and BaseMetastore-backed catalogs because a detached metadata base is forwarded to an identity-checking delegate, and a refreshed Paimon table can reuse a snapshot projection from an older same-ID table generation. Six additional P2 cache-efficiency/performance issues are inline: engine-wide serialization on initialized lookups, privilege-wrapper rejection, a class-wide estimator circuit, unused history-wide manifest materialization, a full metadata JSON clone on every weighted table hit, and rejection of NULL partition publication. Reservation ownership/ABA/close, Hive event copy-on-write/fencing, CREATE/ALTER/replay validation, and routing/compatibility were traced without another defect. No required AGENTS.md files, existing review threads, or additional user-provided focus were present. No builds were run because the review prompt prohibits them. Review status: converged after Round 3.
|
|
||
| @Override | ||
| public void commit(TableMetadata base, TableMetadata newMetadata) { | ||
| delegate.commit(base, newMetadata); |
There was a problem hiding this comment.
[P1] Rebind detached metadata updates before delegating the commit
When this table entry is weight-managed, publication JSON-detaches its TableMetadata, and getIcebergTable() seeds ServiceBackedTableOperations with another detached object. Iceberg 1.10.1 SchemaUpdate and BaseUpdatePartitionSpec capture ops.current() and call ops.commit(base, update) without refreshing, while HadoopTableOperations and BaseMetastoreTableOperations require base to be their current object by identity. Forwarding this clone therefore makes ALTER TABLE schema/reorder and partition evolution fail as stale for Hadoop, Hive, JDBC, Glue, and DLF catalogs whenever a table, catalog, or global weight limit enables this path. Please rebind a verified retained generation to the delegate's actual current object and cover both update kinds under weighted caching.
| return false; | ||
| } | ||
| PaimonSnapshotEntryKey that = (PaimonSnapshotEntryKey) object; | ||
| return snapshotId == that.snapshotId |
There was a problem hiding this comment.
[P1] Include the table generation in the snapshot-cache identity
The contextual value retains the fenced Paimon Table and its partition projection, but equality uses only the table name plus snapshot/schema IDs. The table entry refreshes independently, while this contextual entry cannot auto-refresh and has its own TTL. After a drop/recreate (where IDs restart) or another same-ID physical table generation, getSnapshotCache() can read the new table fence and still hit the old value, returning the old table handle and partition map. Explicit invalidation clears both entries, but ordinary table refresh/replacement does not. Please add a stable table-generation/options identity to this key or couple every table-entry replacement to snapshot invalidation, with a same-ID replacement regression test.
| } | ||
|
|
||
| @Override | ||
| public synchronized void initCatalog(long catalogId, Map<String, String> catalogProperties) { |
There was a problem hiding this comment.
[P2] Keep initialized cache lookups off the engine-wide monitor
Every ExternalMetaCacheMgr typed accessor unconditionally calls prepareCatalogByEngine, which copies and validates the properties, and then reaches this synchronized method. Even when the catalog group already exists, the lookup therefore serializes with every other catalog using this engine and repeats compatibility mapping plus hierarchy validation before computeIfAbsent discovers there is no work. This is on normal planning paths such as Iceberg table and Paimon snapshot/schema lookup, so parallel queries across unrelated catalogs acquire one global engine lock. Please add a lock-free initialized fast path and reserve synchronization/validation for the first build after create or invalidation.
| return false; | ||
| } | ||
| String className = table.getClass().getName(); | ||
| if ("org.apache.paimon.table.AppendOnlyFileStoreTable".equals(className) |
There was a problem hiding this comment.
[P2] Support the privilege wrapper before rejecting the table
A production Paimon table can be a PrivilegedFileStoreTable: Doris explicitly accepts that delegate in PaimonReaderOptions, and its schema/time-travel copies preserve the wrapper. Such a table reaches snapshot publication still wrapped, but this exact-class allowlist rejects it as unsupported_paimon_table without examining the supported underlying file-store table. With snapshot weight governance enabled the projection is then returned once but never cached, so every request reloads and re-enumerates all partitions. Please handle the approved privilege delegate chain (and account for its owned wrapper state) and cover it with a weighted-cache test.
| } | ||
| long now = System.nanoTime(); | ||
| for (Class<?> rootType : rootTypes) { | ||
| FailureCircuit circuit = FAILURE_CIRCUITS.get(rootType); |
There was a problem hiding this comment.
[P2] Do not circuit-break data-dependent failures by root class
This circuit is static and keyed only by the root Java class. After three equal incomplete results, every graph with that class is rejected for a minute before inspection. Reasons such as object_budget_exceeded and time_budget_exceeded are graph-dependent, so three large HivePartitionValues entries can make small, fully supported tables in unrelated catalogs miss the weighted cache as estimator_circuit_open; a valid sibling cannot record success because it is short-circuited here. Please scope suppression to the actual cache owner/key or only class-break deterministic unsupported-type failures, and test an invalid root followed by a valid root of the same class.
| if (sizeEstimate == null) { | ||
| sizeEstimate = MetaCacheSizeEstimator.estimateSafely("iceberg_table_preparation_failed", () -> { | ||
| icebergTable = IcebergSnapshotCacheValue.detachTableGeneration(icebergTable); | ||
| IcebergSnapshotCacheValue.materializeAllSnapshotManifests(icebergTable); |
There was a problem hiding this comment.
[P2] Avoid materializing all historical manifests for an unexposed graph
Weighted table admission now calls both manifest accessors for every historical snapshot. For v2 snapshots this lazily reads one manifest-list file per snapshot. The lists are retained only in this hidden frozen table for weighing: every public getIcebergTable() JSON-clones the metadata into a new service-backed table, dropping Iceberg's transient memoized lists, so scans cannot reuse the work. Long-lived tables therefore pay history-linear work and v2 remote I/O on each admission/refresh solely for an unused graph. Please weigh only metadata that callers can reuse, and add a many-snapshot v2 FileIO-read regression.
| } | ||
| TableOperations retainedOperations = ((HasTableOperations) retainedTable).operations(); | ||
| TableMetadata retainedMetadata = retainedOperations.current(); | ||
| TableMetadata callerMetadata = TableMetadataParser.fromJson( |
There was a problem hiding this comment.
[P2] Avoid rebuilding full metadata on every weighted cache hit
Once weighted publication freezes this table entry, every getIcebergTable() reaches these calls and serializes then reparses the complete TableMetadata: schemas, specs, sort orders, snapshots, refs, statistics, and history logs. This runs on ordinary table/scan planning and even before getSnapshotCache() can hit its contextual entry, so a successful cache hit still performs history-linear CPU and allocation and builds a transient graph outside the retained budget. Count-only entries take the early return. Please reuse a safe read-only frozen projection and construct an isolated service-backed/writable wrapper only for mutation paths, with a long-history cache-hit benchmark.
|
|
||
| IcebergPartition immutableCopy() { | ||
| return new IcebergPartition(partitionName, specId, recordCount, fileSizeInBytes, fileCount, | ||
| lastUpdateTime, lastSnapshotId, ImmutableList.copyOf(partitionValues), |
There was a problem hiding this comment.
[P2] Preserve supported NULL partition values in the immutable copy
generateIcebergPartition() intentionally appends Java null for a NULL partition field, and getPartitionRange() has a dedicated null branch, but Guava ImmutableList.copyOf rejects null elements here. With direct or inherited snapshot weight governance, publication therefore produces an incomplete estimate and serves the projection only once without caching it; every later lookup re-enumerates the partitions metadata table. Please use an ownership-isolated, null-tolerant unmodifiable copy and add a weighted snapshot-cache test with a NULL partition value.
85a09c6 to
6004a16
Compare
FE Regression Coverage ReportIncrement line coverage |
6004a16 to
0e1923f
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes. Full-scope review completed across all 70 authoritative changed files and two convergence rounds. Five new nonduplicate findings remain: three P1 correctness/lifecycle blockers (cross-FE quota replay rejection, max-weight ALTER versus first-init race, and Iceberg HadoopCatalog drop/recreate generation reuse) and two P2 cache-availability/performance issues (Iceberg Kerberos publication outside the authenticator and Paimon remote fence discovery on every cache hit). Existing eight inline discussions were deduplicated and not repeated. Reservation ownership/ABA/close, Hive event copy-on-write/fencing, strict property routing/compatibility, estimator coverage, connector wrapper chains, and Iceberg DDL/DML/action invalidation were traced without another defect. No required AGENTS.md files or additional user-provided review focus were present. No builds were run because the review prompt prohibits them. Review status: converged after Round 2.
| if (parsed <= 0) { | ||
| throw new IllegalArgumentException(CATALOG_MAX_WEIGHT_PROPERTY + " must be positive"); | ||
| } | ||
| if (globalMaxWeight.isPresent() && parsed > globalMaxWeight.getAsLong()) { |
There was a problem hiding this comment.
[P1] Do not reject replayed catalogs against this FE's local global cap. external_meta_cache_max_weight is per-FE and may be a percentage of local heap, while meta.cache.max-weight is persisted after validation only on the master. For example, a 4 GB catalog cap accepted with global=20% on a 32 GB master will fail every lazy cache initialization on an 8 GB observer, because replay skips DDL validation and this check runs on access. Please let the local global bucket clamp the effective admission limit (while keeping DDL hierarchy validation), and cover heterogeneous-heap replay.
| if (sizeEstimate == null) { | ||
| sizeEstimate = MetaCacheSizeEstimator.estimateSafely("iceberg_table_preparation_failed", () -> { | ||
| icebergTable = IcebergSnapshotCacheValue.detachTableGeneration(icebergTable); | ||
| IcebergSnapshotCacheValue.materializeCurrentSnapshotManifests(icebergTable); |
There was a problem hiding this comment.
[P2] Keep manifest materialization inside the catalog authentication scope. The loader's getExecutionAuthenticator().execute(...) ends after ops.loadTable(), but weighted preparation later calls dataManifests(table.io()) / deleteManifests(table.io()) here. For the Kerberized Hadoop catalog, credentials are supplied only inside HadoopExecutionAuthenticator.execute, so this manifest-list read can fail; estimateSafely then marks the value incomplete and every weighted table lookup is returned uncached (the snapshot estimator has the same problem). Please run remote-I/O preparation under the owning catalog authenticator, with a credential-scoped admission/hit regression.
| } | ||
| Snapshot snapshot = metadata.currentSnapshot(); | ||
| long snapshotId = snapshot == null ? IcebergUtils.UNKNOWN_SNAPSHOT_ID : snapshot.snapshotId(); | ||
| return Optional.of(new IcebergSnapshotEntryKey(nameMapping, metadata.metadataFileLocation(), |
There was a problem hiding this comment.
[P1] Include the physical table generation in this key. HadoopCatalog reuses the deterministic metadata/v1.metadata.json path after a purged same-name drop/recreate, and an empty replacement also resets snapshot/schema/spec IDs to -1/0/0; its UUID is new, but every field here collides. After the table entry refreshes, this contextual entry can therefore return the old retained table. The same collision also passes isSameGeneration(), which accepts equal locations without checking UUID. Please key/fence on UUID or a table-entry generation and cover an empty HadoopCatalog drop/recreate.
| ExternalMetaCacheMgr extMetaCacheMgr = Env.getCurrentEnv().getExtMetaCacheMgr(); | ||
| ExternalMetaCacheMgr extMetaCacheMgr = Env.getCurrentEnv().getExtMetaCacheMgr(); | ||
| if (java.util.Objects.nonNull(schemaCacheTtl) | ||
| || updatedProps.containsKey(ExternalMetaCacheBudgetManager.CATALOG_MAX_WEIGHT_PROPERTY)) { |
There was a problem hiding this comment.
[P1] Fence this new quota invalidation against an in-flight first initialization. prepareCatalogByEngine() can copy the old properties while no group exists; if ALTER commits this setting next, removeCatalog() skips the absent group, and the delayed initializer then publishes the old count-only policy indefinitely. That silently defeats the configured memory bound. Please version/serialize the property snapshot with removal and publication, and add a paused ALTER-vs-init test that verifies the new weighted policy wins.
| return tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getLatestSnapshotCacheValue(); | ||
| PaimonTableCacheValue tableValue = tableEntry.get(nameMapping.getCtlId()).get(nameMapping); | ||
| Table table = tableValue.getPaimonTable(); | ||
| PaimonSnapshot fence = latestSnapshotProjectionLoader.loadFence(nameMapping, table).getSnapshot(); |
There was a problem hiding this comment.
[P2] Avoid resolving the remote fence before every snapshot-cache lookup. loadFence() runs before snapshotEntry.get(), and its path calls copyWithLatestSchema(), latestSnapshot(), and schemaManager().latest(), so even a hit on an admitted snapshot still performs latest-metadata discovery. Before this change, PaimonTableCacheValue memoized the projection, so stable repeated reads avoided that work. Please retain or refresh the fence under the table generation (or otherwise put discovery behind a cache) and add a repeated-hit call-count test.
FE Regression Coverage ReportIncrement line coverage |
0e1923f to
57a8d5d
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE UT Coverage ReportIncrement line coverage |
|
Design suggestion after reviewing the current head
For these goals we do not need a precise retained-object-graph size. I suggest using long weight = BASE_WEIGHT
+ partitionCount * PARTITION_WEIGHT
+ fileCount * FILE_WEIGHT;The unit can be approximate KiB. The constants can be calibrated offline with the existing benchmarks/full estimator and rounded up to simple powers of two. Production weighing should only read O(1) collection sizes or counters already produced by the normal loader. It must not reflect over object fields, build an identity set, materialize lazy state, read manifests remotely, or serialize/clone the value. Suggested formulas for the currently managed and adjacent unbounded entries: For Paimon, The formula should include only collections actually retained by that cache value. Hudi, MaxCompute, Doris, Hive single-partition, and other small/bounded entries can remain count-based until one of their values retains an unbounded collection; then the same
There is one important lifecycle requirement in the current implementation: I would keep the existing global/catalog/entry budget hierarchy, admission-before-publication protocol, generation fencing, and rejection behavior. I would replace the production The key tests should be:
This gives a substantially cheaper and more maintainable implementation while still improving by orders of magnitude over pure entry-count limits. As with the current PR scope, it controls retained MetaCache memory after load; it does not bound temporary memory used by the query that constructs the value before admission. |
FE Regression Coverage ReportIncrement line coverage |
57a8d5d to
b89e044
Compare
|
/review |
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
There was a problem hiding this comment.
Requesting changes for nine independently verified issues on this head.
- Catalog lifecycle: a failed legacy ALTER can leave the rejected cache policy initialized.
- Iceberg correctness and availability: weighted query isolation truncates metadata-log rows, historical system tables bypass stale-file retry, exact metadata reload lacks UUID validation, and recreated tables can reuse generation-blind schema entries.
- Paimon correctness and availability: recreated tables can reuse generation-blind schema entries, obsolete synthetic generations are not retired, and eager fence discovery expands remote unauthenticated work to table-only callers.
- Cache lifecycle: prior Iceberg snapshot projections are not retired after observed metadata-generation advances.
Review checkpoints: the full 71-file authoritative diff was covered through three normal-review rounds plus targeted risk passes; all existing inline threads were treated as duplicate fences; Hive event CAS, generic admission/removal ownership, writable Iceberg mutation routes, estimator coverage, configuration routing, tests, and benchmark-only changes were rechecked without another substantiated issue. No additional user focus was provided. The repo-local code-review skill was not present, so the bundle prompt and required review artifacts governed the review. No builds or source edits were performed. Review status: converged after Round 3.
| // This fallback is only for isolated construction tests before Env is initialized. | ||
| ExternalMetaCacheBudgetManager.fromConfig().validateCatalogMaxWeight(properties); | ||
| } else { | ||
| extMetaCacheMgr.validateCatalogCacheProperties(this, properties); |
There was a problem hiding this comment.
[P1] Retire cache groups when tentative validation rolls back
Legacy validators publish these candidate properties before this call, so a concurrent first lookup can initialize the managed cache group from them. If a later connector check rejects the ALTER, CatalogMgr restores CatalogProperty but does not remove that group; subsequent lookups take the initialized fast path and keep the rejected max-weight policy. Please make this validation detached or retire the group under the same lifecycle fence after rollback, and cover failed ALTER racing first initialization.
| } | ||
| TableMetadata retainedMetadata = builder.discardChanges() | ||
| .withMetadataLocation(source.metadataFileLocation()).build(); | ||
| return tableWithOperations(table, new FrozenTableOperations( |
There was a problem hiding this comment.
[P1] Restore previous metadata files for metadata_log_entries
This retained TableMetadata intentionally omits previousFiles(), but QueryScopedTable still exposes these stripped operations. Iceberg 1.10.1's MetadataLogEntriesTable reads operations().current().previousFiles() directly, so enabling a table/snapshot weight bound makes $metadata_log_entries return only the synthetic current row. Please provide that system table an exact query-local operations view and add a multi-generation weighted-cache regression.
| } | ||
|
|
||
| public Table getIcebergTable() { | ||
| return queryIsolationPrepared |
There was a problem hiding this comment.
[P1] Route system tables through the stale-metadata retry fence
After weighted publication this accessor returns a lazy query view, and IcebergSysExternalTable uses it directly. If metadata cleanup removes the retained file, $history/$snapshots/$refs and ALL_* tables throw later from their lazy accessors without invalidating the table entry, while getQueryScopedIcebergTable() already has the required retry. Please use that fenced accessor here (or wrap the lazy load) and cover a deleted pinned file through a system-table query.
| ExecutionAuthenticator authenticator = retainedOperations instanceof FrozenTableOperations | ||
| ? ((FrozenTableOperations) retainedOperations).authenticator : null; | ||
| try { | ||
| return authenticator == null |
There was a problem hiding this comment.
[P1] Validate the parsed table generation before accepting this metadata file
A purged Hadoop table recreation can reuse the same metadata/v1.metadata.json path; this read then succeeds even though its UUID differs from the retained table. QueryScopedTable subsequently mixes the old retained schema/current snapshot with replacement refs and history, so the new snapshot-key UUID does not help. Please compare the parsed UUID/generation and throw StaleMetadataException on mismatch so the existing invalidation/reload path runs.
| icebergPartitionInfo = IcebergPartitionInfo.empty(); | ||
| } else { | ||
| icebergPartitionInfo = IcebergUtils.loadPartitionInfo(dorisTable, retainedTable, | ||
| icebergPartitionInfo = IcebergUtils.loadPartitionInfo(dorisTable, projectionTable, |
There was a problem hiding this comment.
[P1] Fence this schema dependency with the physical table generation
The new snapshot key correctly misses after a same-name recreation with a different UUID, but loadPartitionInfo still resolves schema through (NameMapping, schemaId). Recreated Iceberg tables restart schema IDs, and refreshing the table entry does not retire schemaEntry, so the new snapshot can be built with the old table's partition-column types and full schema. Please include UUID/generation in the schema key or retire it with table-generation replacement.
| nameMapping, fence, tableValue.getGeneration()); | ||
| MetaCacheEntry<PaimonSnapshotEntryKey, PaimonSnapshotCacheValue> entry = | ||
| snapshotEntry.get(nameMapping.getCtlId()); | ||
| return entry.get(key, ignored -> latestSnapshotProjectionLoader.loadAtFence(nameMapping, fence)); |
There was a problem hiding this comment.
[P1] Carry the table generation into the schema lookup used by this miss
The snapshot key now separates reloaded tables, but loadAtFence immediately looks up schema by only (NameMapping, schemaId). After same-name drop/recreate with a restarted ID, that returns the old schema/partition columns and the result is then cached under the new generation key. Please generation-fence schemaEntry as well, or atomically retire schema and old snapshot keys when tableEntry publishes a new generation.
| return tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getLatestSnapshotCacheValue(); | ||
| PaimonTableCacheValue tableValue = tableEntry.get(nameMapping.getCtlId()).get(nameMapping); | ||
| PaimonSnapshot fence = tableValue.getLatestSnapshotFence().getSnapshot(); | ||
| PaimonSnapshotEntryKey key = PaimonSnapshotEntryKey.of( |
There was a problem hiding this comment.
[P2] Retire the prior snapshot generation when tableEntry refreshes
Every successful ten-minute table reload allocates a fresh generation, so this key correctly misses, but the contextual snapshot entry has no replacement hook to remove older keys. A continuously queried table can retain roughly 144 unreachable partition projections for the 24-hour access TTL, consuming quota/capacity and displacing live generations. Please couple table replacement to race-safe invalidation of its prior snapshot keys and test repeated refreshes.
| Table paimonTable = tableLoader.load(nameMapping); | ||
| return new PaimonTableCacheValue(paimonTable, | ||
| () -> latestSnapshotProjectionLoader.load(nameMapping, paimonTable)); | ||
| PaimonSnapshotCacheValue fence = latestSnapshotProjectionLoader.loadFence(nameMapping, paimonTable); |
There was a problem hiding this comment.
[P1] Do not make table-only cache publication depend on unauthenticated snapshot discovery
PaimonTableLoader returns after getPaimonTable's executionAuthenticator scope closes, then this eager fence calls copyWithLatestSchema/latestSnapshot/schemaManager.latest. That moves remote work into every cold load/refresh, so even comment/property/isPartitionedTable callers can now block or fail outside the catalog credential scope. Please capture the fence within authentication while preserving a memoized lazy path for table-only uses, and cover a table-only load with guarded snapshot metadata.
| tableValue.getRetainedIcebergTable(), | ||
| tableValue.getRetainedCurrentSnapshotJson(), isolateForQueries)); | ||
| } | ||
| IcebergSnapshotEntryKey key = optionalKey.get(); |
There was a problem hiding this comment.
[P2] Retire snapshot projections from the previous table generation
Each table auto-refresh that observes a newer Iceberg metadata file creates a distinct key here, but snapshotEntry has no replacement hook to remove older keys. Those partition projections are no longer addressable by later latest or time-travel lookups, yet remain for the 24-hour TTL and consume capacity/catalog-global reservations, displacing or rejecting current metadata. Please couple table replacement to race-safe cleanup of prior keys and cover repeated metadata advances.
3 participants
DRAFT Docs
https://github.com/CalvinKirs/doris-website/blob/2125f053594b821a6ab7556f035b9cb1e5b43a0f/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/external-meta-cache-memory-management.md
apache/doris-website#4061 (comment)
Summary
Add retained-memory governance for selected external metadata caches. Existing count-based capacity remains the default. Weighted admission is enabled only for an estimator-backed entry when at least one applicable global, catalog, or entry memory limit is configured.
Why
Managed scope
partition_values.table,snapshot, andmanifest(manifestremains disabled by default).snapshot.Other external metadata entries continue to use their existing count-based behavior.
Accounting and ownership strategy
Iceberg table/snapshot cache values use a detached, non-growing metadata generation. Historical refs/snapshots/statistics are not retained by the cache entry; a statement that needs them reads the exact pinned metadata file into a query-local table under the catalog authenticator. The statement keeps one generation even if the cache concurrently refreshes. A stale unbound cache generation is invalidated and retried once; an already-bound statement fails instead of silently switching generations. Snapshot identity includes
metadataFileLocation + snapshotId + schemaId + defaultSpecId.Paimon partition payload bytes are accumulated in the existing partition-construction loop, including every retained typed value and display name. This avoids sampling misses without a second full traversal.
Limit behavior
Configuration
external_meta_cache_max_weight=10GBor20%;0disables the FE-global quota.meta.cache.max-weight=4GB.meta.cache.<engine>.<entry>.max-weight=1GB.Not every entry needs an explicit limit. Estimator-backed entries inherit the nearest configured parent. Catalog/entry limits also work when the FE-global limit is disabled. The hierarchy is validated as
entry <= catalog <= globalwhen the corresponding parents exist. Unknown engines, entries, options, aliases, and max-weight on entries without an estimator are rejected during catalog validation.Optimizer and query-path impact
No optimizer rule, literal representation, partition-item implementation, or system-table exposure is added. The only scan-node edit stores an existing
Optionalresult once before use; it does not change scan planning semantics.Validation
git diff --checkpasses.520964 <= 524288; budget rejection did not fail queries; no incomplete estimate, accounting underflow, deadlock, or OOM was observed. The latest source behavior is covered by the focused unit regression above.Performance results
In-repo benchmark harness, Java 17,
-Xms1g -Xmx4g, 500 ms warmup and 3 x 500 ms measurement. Results are per operation.The Iceberg comparison includes
DataFile.copy()in both paths, matching the production manifest reader. Even in the dense-metrics stress cases, copying/parsing remains the larger component than the incremental counter. Iceberg table publication is 4.401 us (10 fields) / 9.991 us (100 fields); 1k versus 10k retained snapshot history is 2.931 us / 3.006 us, showing no history-length traversal. Prepared weight lookup is approximately 30-40 ns for Iceberg/Paimon.