Skip to content

[doc](lakehouse) Add external metadata cache memory management for 4.1.5 - #4061

Draft
CalvinKirs wants to merge 1 commit into
apache:masterfrom
CalvinKirs:agent/external-meta-cache-memory-doc-4.1.5
Draft

[doc](lakehouse) Add external metadata cache memory management for 4.1.5#4061
CalvinKirs wants to merge 1 commit into
apache:masterfrom
CalvinKirs:agent/external-meta-cache-memory-doc-4.1.5

Conversation

@CalvinKirs

@CalvinKirs CalvinKirs commented Aug 14, 2026

Copy link
Copy Markdown
Member

What

  • Add Chinese and English usage guides for external metadata cache memory management, supported from Doris 4.1.5.
  • Document FE-, Catalog-, and cache-entry-level memory limits and their configuration hierarchy.
  • Cover the supported Hive, Iceberg, and Paimon cache entries, configuration examples, eviction behavior, and operational guidance.
  • Document the final accounting model: type-specific conservative formulas plus loader-time payload counters, with no private-field reflection or container sampling.
  • Explain Iceberg's non-growing cached metadata generation and statement-local reads from the key-pinned metadataFileLocation.
  • Explain soft-value collection, reservation records without strong value ownership, O(1) prepared-weight lookup, and the post-load admission/OOM boundary.
  • Add the guide to the 4.x sidebar and link it from the metadata cache page.
  • Clarify that Hive cache properties are reloaded from Doris 4.1.5 by rebuilding the affected Catalog cache.

Why

This documents the user-facing configuration and runtime behavior introduced by apache/doris#66717, especially the cases discussed during review:

  • A Catalog or cache-entry limit can be configured without an FE-wide limit.
  • Oversized or safely unmeasurable objects are returned to the query but are not cached; cache admission does not fail the query.
  • The quota is checked after object construction and therefore is not pre-load OOM protection.
  • Shared infrastructure such as Catalog, FileIO, and thread pools is excluded from a single cache entry's ownership.
  • Weight-bounded values may be collected under heap pressure without a reservation record keeping the value alive.
  • Iceberg statements remain bound to one metadata generation during concurrent cache refresh.
  • Only explicitly supported cache entries accept max-weight; unsupported keys are rejected instead of being silently ignored.

Checks

  • git diff --check
  • Parsed both Chinese and English 4.x document frontmatters as JSON.
  • Verified balanced and matching Markdown code fences in both documents.
  • Verified the existing single 4.x sidebar entry remains unchanged.

A full Docusaurus build was not run because dependencies are not installed in the clean worktree.

@CalvinKirs
CalvinKirs force-pushed the agent/external-meta-cache-memory-doc-4.1.5 branch from 2125f05 to 03960ee Compare August 15, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant