[Enhancement](compaction) add information_schema.be_compaction_tasks system table#61428
Draft
Yukang-Lian wants to merge 4 commits intoapache:masterfrom
Draft
[Enhancement](compaction) add information_schema.be_compaction_tasks system table#61428Yukang-Lian wants to merge 4 commits intoapache:masterfrom
Yukang-Lian wants to merge 4 commits intoapache:masterfrom
Conversation
…system table (apache#48893) Add a new system table `be_compaction_tasks` in `information_schema` that exposes compaction task metadata across all BEs, covering PENDING, RUNNING, FINISHED, and FAILED states. Key components: - CompactionTaskTracker: singleton that tracks compaction tasks across their full lifecycle (PENDING -> RUNNING -> FINISHED/FAILED) - SchemaCompactionTasksScanner: BE scanner that fills 30 columns including identification, timing, input/output stats, IO stats, and resource usage - FE schema registration with BackendPartitionedSchemaScanNode for multi-BE fan-out Tracker is integrated at all compaction entry points: - Local: base/cumu/full, single-replica, cold-data, manual HTTP trigger - Cloud: base/cumu/full, manual HTTP trigger, index-change Closes apache#48893
Yukang-Lian
requested review from
dataroaring,
gavinchou and
w41ter
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Yukang-Lian
force-pushed
the
feature/be-compaction-tasks-system-table
branch
from
8717ce1 to
5b67650
Compare
…on_tasks regression test Add SELECT * and explicit 30-column named SELECT to verify all columns are queryable with reasonable values. Log each column for visual inspection. Also add DESC test to verify schema has exactly 30 columns.
Yukang-Lian
force-pushed
the
feature/be-compaction-tasks-system-table
branch
from
5b67650 to
e66a5ff
Compare
SCH_AUTHENTICATION_INTEGRATIONS took 67, bump SCH_BE_COMPACTION_TASKS to 68.
Collaborator
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 27068 ms |
TPC-DS: Total hot run time: 167854 ms |
luwei16
reviewed
Mar 17, 2026
| Status res = Status::OK(); | ||
| auto do_compact = [](Compaction& compaction) { | ||
| // Helper to register a compaction task as RUNNING in the tracker (direct execution, MANUAL trigger) | ||
| auto register_running_task = [&tablet](Compaction& compaction) { |
Contributor
There was a problem hiding this comment.
The progress of compaction is not reflected.
The start time of compaction execution is not recorded.
In BE, can the container used to record compaction tasks be shared with the one used to record compaction profiles?
luwei16
requested changes
Mar 17, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
information_schema.be_compaction_tasksthat exposes compaction task metadata across all BEs, covering PENDING, RUNNING, FINISHED, and FAILED states with 30 columns including identification, timing, input/output stats, IO stats, and resource usage.CompactionTaskTrackersingleton to track compaction tasks across their full lifecycle, integrated at all 7 compaction entry points (local: base/cumu/full, single-replica, cold-data, manual HTTP; cloud: base/cumu/full, manual HTTP, index-change).BackendPartitionedSchemaScanNode, fallback records for missed registrations, and proper cleanup on early-return paths.Usage Examples
Full Schema (30 columns)
Test plan
closes #48893