Skip to content

[fix](be) Fall back from expensive Hyperscan bounded repeats - #66788

Open
HappenLee wants to merge 6 commits into
apache:masterfrom
HappenLee:fix-hyperscan-bounded-repeat-fallback
Open

[fix](be) Fall back from expensive Hyperscan bounded repeats#66788
HappenLee wants to merge 6 commits into
apache:masterfrom
HappenLee:fix-hyperscan-bounded-repeat-fallback

Conversation

@HappenLee

@HappenLee HappenLee commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Hyperscan can spend excessive time and memory compiling bounded repetitions with large limits. Doris: regex quantifiers above 50 are detected before Hyperscan compilation and processed by RE2 instead.

The new enable_hyperscan_fallback session variable controls strictness. It defaults to true, preserving fallback behavior. When set to false, Doris returns the Hyperscan/interception error instead of falling back. Escaped braces and braces inside character classes are masked before detection so regex literals are not misclassified.

Release note

Add enable_hyperscan_fallback. It defaults to true; setting it to false returns an error when Hyperscan cannot compile or safely process the regular expression.

Check List (For Author)

  • Test:
    • Unit Test: GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh -j 48 --run --filter=FunctionLikeTest.* (36 passed)
    • Unit Test: DORIS_THIRDPARTY=$PWD/thirdparty ./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest (17 passed)
  • Behavior changed: Yes. Expensive bounded repeats are intercepted before Hyperscan compilation, and sessions may disable the RE2 fallback. The default remains fallback enabled.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Hyperscan compilation can become extremely expensive for regular expressions with large bounded repetitions, such as `prompt_rewrite\.h03.{0,1000}429`. Doris previously sent every compatible pattern to Hyperscan before considering RE2, so compiling such expressions could consume excessive CPU and delay query execution. Detect bounded repetitions above 50 before calling Hyperscan and reuse the existing RE2 fallback path for both constant and non-constant patterns. Keep the detector local to the LIKE/REGEXP implementation and cover its threshold behavior and end-to-end matching results.

### Release note

Fall back to RE2 for regular expressions whose bounded repetition exceeds 50 to avoid expensive Hyperscan compilation.

### Check List (For Author)

- Test: Unit Test
    - `./run-be-ut.sh -j 48 --run --filter=FunctionLikeTest.hyperscan_bounded_repeat_fallback:FunctionLikeTest.hyperscan_bounded_repeat_threshold`
- Behavior changed: Yes. Large bounded repetitions use RE2 instead of Hyperscan while preserving REGEXP results.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Hyperscan compilation can fail or be intentionally intercepted for regular expressions with expensive bounded repetitions. Doris always fell back to RE2, so users could not choose strict failure behavior. Add the enable_hyperscan_fallback session variable, propagate it through TQueryOptions, and return the Hyperscan status when fallback is disabled. Mask escaped characters and character classes before bounded-repeat detection so literal braces are not intercepted.

### Release note

Add the enable_hyperscan_fallback session variable. It defaults to true; setting it to false returns an error instead of falling back to RE2 when Hyperscan compilation is unavailable.

### Check List (For Author)

- Test:
    - Unit Test: FunctionLikeTest.* and org.apache.doris.qe.SessionVariablesTest
- Behavior changed: Yes. Hyperscan fallback can now be disabled per session; the default behavior is unchanged.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: seven substantiated correctness and lifecycle gaps remain.

  • Goal and proof: The patch aims to keep expensive bounded repeats away from Hyperscan while preserving normal REGEXP behavior and offering a session-controlled strict mode. It does not yet accomplish that end to end: valid Hyperscan patterns can regress, expensive patterns can bypass the detector, and the policy is lost or stale in alternate execution paths.
  • Scope and clarity: The patch is focused, but the hand-written lexical recognizer is not precise enough for Hyperscan grammar and the cross-FE/BE session-policy integration is incomplete.
  • Concurrency: Function-local checker initialization and const RE2 matching are thread-safe; no new lock-order or shared-mutation defect was found.
  • Lifecycle: Hyperscan database/scratch cleanup is sound. Prepared point-query reuse retains old query options and once-opened REGEXP state, and ALTER replay reconstructs the new option with the wrong value.
  • Configuration: The session variable is dynamic, forwarded, and execution-affecting on the ordinary path, but strict-mode changes are not honored by BE constant folding, reusable point queries, or replayed synchronous-MV jobs.
  • Compatibility: The optional Thrift field and default true are safe for ordinary mixed-version decoding; old BEs necessarily cannot enforce the new setting. The persisted ALTER-job path is not replay-equivalent after failover.
  • Parallel paths: Constant open, execute-time ColumnConst, genuinely row-varying RE2, LIKE conversion, storage pushdown, state cloning, BE folding, point-query reuse, and schema-change/MV execution were traced. The seven inline comments cover every substantiated divergence.
  • Conditional checks and errors: Numeric parsing, the flat > 50 comparisons, status propagation, and allocation cleanup are sound. The detector nevertheless misreads supported lexical contexts, mishandles [^^], misses multiplicative nesting, and can route valid patterns to an incapable fallback engine.
  • Test coverage: Added tests cover flat thresholds, basic escapes/classes, direct default/strict paths, LIKE brace escaping, and direct SessionVariable.toThrift(). They do not cover supported non-repeat syntax, [^^], nested repeat products, Hyperscan-only fallback inputs, BE-fold options, prepared-context invalidation, or ALTER replay.
  • Test results: The modified expectations match the implementation, but the whitespace-bearing brace case encodes the same lexical false positive described inline. No builds or tests were run in this review environment, as required by the review bundle.
  • Observability: No new metric or logging requirement was identified; existing status paths are sufficient once the behavior is corrected.
  • Persistence and transactions: No transaction protocol changed. The synchronous-MV edit-log/replay path fails to preserve the captured execution option across restart/failover.
  • Data writes: No direct write atomicity or visibility issue was found; the materialized-view build issue is an execution-policy replay mismatch.
  • FE/BE variable passing: Ordinary SessionVariable.toThrift(), forwarding, cloning, and mixed-version defaults are sound. Three distinct alternate sender/cache/persistence boundaries remain incorrect.
  • Performance: The linear precheck itself is reasonable, but [^^] and nested {50} products can still expose Hyperscan to the expensive compilation the patch is intended to prevent.
  • Additional focus: No extra user-provided review focus was present. No independent nullable, cloud/shared-nothing, MoW, or storage-format issue applies here.

Review completeness: the third and final allowed round produced a new valuable finding. This review is therefore capped/incomplete rather than converged; all seven currently substantiated findings are included.

Comment thread be/src/exprs/function/like.cpp Outdated
Comment thread be/src/exprs/function/like.cpp Outdated
Comment thread be/src/exprs/function/like.cpp
Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Comment thread gensrc/thrift/PaloInternalService.thrift
Comment thread be/src/exprs/function/like.cpp Outdated
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66788

Problem Summary: The Hyperscan bounded-repeat masker treated every leading caret in a character class as the negation marker. For `[^^]`, the second caret is class content, but Doris kept the class open and masked the real bounded repeat that followed it. Track whether the one optional leading negation marker is still allowed so the closing bracket is recognized and the expensive repeat is intercepted.

### Release note

Correct expensive Hyperscan repeat detection after negated character classes.

### Check List (For Author)

- Test: Unit Test
    - `GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh -j 48 --run --filter=FunctionLikeTest.*`
- Behavior changed: Yes. Large bounded repeats following character classes such as `[^^]` are now intercepted instead of reaching Hyperscan.
- Does this need documentation: No
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: two distinct gaps remain on head 17e768572e5cc2d7bd95771f988b63e2dc04c99d.

  1. P1 — parallel public Hyperscan compilers bypass the guard. The new checker is private to FunctionLikeBase::hs_prepare(). No-index/indexed MATCH_REGEXP, the indexed search() DSL REGEXP leaf, and multi_match_any still pass the same expensive repeat directly to hs_compile()/hs_compile_multi().
  2. P2 — load planners lose the new execution policy. Broker, routine, and ordinary stream-load paths construct fresh TQueryOptions and retain field 231's default true; broker/routine jobs also do not persist the creator's value. A load regexp therefore falls back instead of returning the requested strict error, including when a new stream-load context inherited a persisted global false.

Critical-checkpoint conclusions:

  • Goal and proof: the changed constant-pattern REGEXP/LIKE paths do intercept direct bounds above 50 and the added BE tests cover default fallback, strict open/execute, threshold, escapes, classes, and the [^^] fix. The end-to-end goal is incomplete because the public compiler paths above remain exposed, and the FE test proves only direct SessionVariable.toThrift() propagation.
  • Scope, clarity, and reuse: the local change is small and understandable, but placing the checker inside the LIKE implementation prevents the required reuse at other Hyperscan boundaries.
  • Concurrency and lifecycle: the function-local checker has thread-safe initialization and const matching; THREAD_LOCAL function state, predicate clones, databases, and scratch allocations remain independently owned. New error exits propagate status and release/null owned resources. No new lock-order, race, static-initialization, nullable/const-shape, or cleanup defect was found. The material lifecycle omission is the uncaptured/fresh load state in P2; cached point-query and ALTER replay issues are already covered by live threads.
  • Dynamic variable propagation: normal legacy/Nereids coordination, forwarded execution, fragment copies, and execution-result hashing preserve the value. Fresh load senders do not, even for the persistent global setting. BE constant folding is already covered by an existing thread.
  • Compatibility: Thrift field 231 is unique, optional, and defaults to true, preserving old-FE/new-BE behavior in Doris's documented BE-before-FE upgrade order. No storage-format or function-symbol incompatibility was introduced.
  • Parallel paths and conditions: repository-wide enumeration found four direct hs_compile() sites plus the multi-pattern compiler template; only the LIKE/REGEXP site is guarded. Class/comment lexical approximations, nested multiplication, and fallback-engine capability are already covered by existing discussions, and the reported [^^] false negative is fixed on this head.
  • Tests and observability: no regression or end-to-end load/MATCH/search/multi test covers the two remaining gaps, and no result file changed. Existing statuses identify an interception or compile failure when the option arrives, but a dropped load option is silent. Per the review-only constraint, I did not run builds or tests; the PR reports 36 BE and 17 FE unit tests, while formatting/checkstyle and other lightweight checks are green and macOS BE UT is still pending at review time.
  • Persistence, writes, and other invariants: no transaction visibility, version, delete-bitmap, or metadata-write mechanism changed. P2 can nevertheless allow a load with a regexp filter to proceed where strict mode requested failure, so the policy must be captured before dispatch and replay. The checker adds only linear pre-compilation work and no per-row regression was found.

There was no additional user-provided focus. In the second complete normal/risk review round, all reviewers returned NO_NEW_VALUABLE_FINDINGS; the two inline findings below are the final deduplicated set.

Comment thread be/src/exprs/function/like.cpp
Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66788

Problem Summary: The expensive bounded-repeat check was local to LIKE/REGEXP, so MATCH_REGEXP, inverted-index regexp queries, and multi-match could still compile pathological expressions directly with Hyperscan. Load planners also created fresh query options and lost the enable_hyperscan_fallback value selected when broker, routine, or stream load work was created. Move the checker into a shared utility, reject expensive patterns on Hyperscan-only paths, and persist and propagate the fallback option through load jobs, task descriptors, and both legacy and Nereids coordinators. Old replayed jobs without the persisted value retain the default enabled behavior.

### Release note

Apply bounded-repeat safeguards to all Hyperscan compilation paths and preserve enable_hyperscan_fallback for load execution.

### Check List (For Author)

- Test: Unit Test
    - BE: 40 targeted FunctionLike, MATCH_REGEXP, multi-match, and inverted-index regexp tests
    - FE: 32 targeted load task, coordinator, and routine-load replay tests
- Behavior changed: Yes. Hyperscan-only paths reject expensive bounded repeats, while load execution now honors the fallback setting captured when the work was created.
- Does this need documentation: No
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review

gavinchou
gavinchou previously approved these changes Aug 14, 2026

@gavinchou gavinchou left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 14, 2026
@HappenLee

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66788

Problem Summary: The load fallback propagation change placed SessionVariable imports out of lexicographical order and omitted the required blank line between two test methods, causing FE checkstyle to fail. Restore the expected import ordering and method separation.

### Release note

None

### Check List (For Author)

- Test: No need to test (code-style-only change; per request, compilation and tests were not rerun)
- Behavior changed: No
- Does this need documentation: No
@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Aug 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/31826833944

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@HappenLee

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: three distinct P2 execution-policy gaps remain on head ed8c4c190c1cd61508bdc1d27766e5187feb52e2.

  1. A server-prepared group-commit insert reuses fragment bytes containing the first execution's enable_hyperscan_fallback value after the session setting changes.
  2. RPC stream-load planning bypasses its locked request snapshot and reads the mutable global default without VariableMgr's protecting lock, racing with a concurrent global update.
  3. During Doris's supported non-master-FE-first/master-last rolling upgrade, an upgraded follower forwards the new session key to an old master that silently drops it; the already-new BEs then restore the historical default.

Critical-checkpoint conclusions:

  • Goal and proof: The PR aims to keep expensive bounded repeats away from Hyperscan and make fallback behavior dynamically controllable. The shared BE guard now covers all six direct compiler calls, and the normal current-version query/load paths propagate the option, but the three inline lifecycle/publication/compatibility gaps prevent end-to-end correctness.
  • Scope and user focus: The implementation remains focused despite the necessary load-path propagation. All 45 authoritative changed files were reviewed. The user supplied no additional focus, so the full PR was reviewed without extra narrowing.
  • Concurrency: MF-2 is the one new shared-memory defect: MySQL SET GLOBAL writes the ordinary field under VariableMgr.wlock, while a FrontendService RPC thread reads it unlocked after establishing a read-locked request snapshot. Other new task fields are creator-before-dispatch copies; no nested lock order, deadlock, or heavy-under-lock issue was found.
  • Lifecycle and static initialization: Function-local immutable checker initialization is thread-safe, LikeSearchState::clone() carries the option, and Hyperscan database/scratch ownership is unchanged. MF-1 is the remaining non-intuitive lifecycle: serialized prepared group-commit state outlives the session value that created it.
  • Dynamic configuration: Same-version normal execution observes session changes through SessionVariable.toThrift(), forwarding metadata, and execution-result hashing. Prepared group commit (MF-1), concurrent RPC global capture (MF-2), and mixed-FE forwarding (MF-4) do not honor that contract.
  • Compatibility: Old-FE/new-BE decoding is safe because optional field 231 defaults to historical true; new-FE/old-BE dispatch is outside Doris's documented BE-first upgrade order. The supported upgraded-follower/old-master interval is not safe because the old receiver drops the unknown forwarded map key (MF-4).
  • Parallel paths and conditions: LIKE/REGEXP, no-index MATCH, inverted-index v1, search-DSL v2, and ordinary/edit-distance multi-pattern compilers are all guarded before compilation/allocation. Legacy/Nereids query coordination, broker/cloud broker, Kafka/Kinesis routine load, ordinary/cloud RPC stream, insert-stream, initial group commit, and multi-table copying were traced. Lexical masking, nested repeats, incapable RE2 fallback, point-query reuse, BE constant folding, and ALTER replay are already covered by existing live threads and were not duplicated.
  • Errors and cleanup: New guard failures reach established Status/exception conversion boundaries; database and scratch cleanup remain intact. No ignored status, speculative recovery, nullable-shape, ownership, or static-initialization issue was found.
  • Test coverage: Added tests exercise each compiler family and selected option serialization/load/default paths. Missing behavior tests map directly to the three findings: prepared group-commit invalidation, latch/barrier-controlled global capture, and new-follower/old-master forwarding.
  • Test results: This review runner forbids local builds/tests, so none were run here. At submission time, macOS BE UT, CheckStyle, Clang Formatter, license, dependency, and secret checks pass; TeamCity compile and regression contexts report failure. Their logs require authentication, so this review does not attribute those failures to a cause.
  • Observability: Existing query/load identifiers and error boundaries are adequate once the policy is preserved; no new metric or logging requirement was identified.
  • Persistence and transactions: Broker/routine job maps persist creator-time values and old records default to historical true. No new EditLog, failover, transaction visibility, version, or delete-bitmap defect was found; the ALTER replay concern remains an existing live thread.
  • Data writes: Insert/load paths are affected only through whether regexp evaluation falls back or errors. No separate commit atomicity, row visibility, storage-format, MoW, or crash-cleanup issue was found.
  • FE-BE variable passing: Every current-version TQueryOptions construction/copy and specialized load sender was inventoried. The remaining distinct boundaries are the three inline findings plus the already-threaded constant-fold, point-query, and ALTER paths.
  • Performance and other concerns: The checker adds linear masking/search work before compilation and outside locks; scalar option copies and map lookups are negligible. No additional CPU, allocation, cloud/shared-nothing, nullability, or storage concern survived the final sweep.

Review completeness: converged. In the third and final round, the normal BE, normal FE, and risk-focused reviewers all returned NO_NEW_VALUABLE_FINDINGS; all accepted, dismissed, and duplicate candidates have explicit conclusions.

.setStrictMode(ConnectContext.get().getSessionVariable().enableInsertStrict);
NereidsStreamLoadTask streamLoadTask = NereidsStreamLoadTask.fromTStreamLoadPutRequest(streamLoadPutRequest);
NereidsStreamLoadTask streamLoadTask = NereidsStreamLoadTask.fromTStreamLoadPutRequest(
streamLoadPutRequest, ConnectContext.get().getSessionVariable().enableHyperscanFallback);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Invalidate prepared group-commit plans for this option

This value is serialized into execPlanFragmentParamsBytes when the planner is created, but executeGroupCommitInsert() reuses that planner whenever only the table id and base schema version match. After a session executes a prepared group-commit insert, changes enable_hyperscan_fallback, and executes it again, BE-evaluated expressions (for example a generated column using REGEXP 'a{51}') still receive the old policy. Store this option or the execution-variable digest on the planner and include it in the reuse fence, with a prepared group-commit regression that changes the setting between executions. This is distinct from the existing point-query thread because it is the separate PreparedStatementContext.groupCommitPlanner cache.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. We agree that prepared Group Commit planner reuse should be invalidated through a unified session-variable fingerprint, instead of adding another option-specific reuse check here. Since that requires a broader change to the common cache invalidation mechanism, we will address it in a follow-up PR and track it with a Jira TODO, rather than expanding the scope of this PR. We will keep this thread open until the follow-up is tracked.

Comment thread fe/fe-core/src/main/java/org/apache/doris/load/StreamLoadHandler.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66788

Problem Summary: RPC stream load planning read enable_hyperscan_fallback directly from the mutable global default session variable without holding VariableMgr's read lock. Use the request-scoped ConnectContext session snapshot instead so planning observes a consistent value, and add a unit test that distinguishes the snapshot value from the global default.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.load.StreamLoadHandlerTest
- Behavior changed: No
- Does this need documentation: No
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one additional, distinct P2 execution-policy gap on head 68f72081dccb35377e5f4793c5f7841425b88e74.

Asynchronous MTMV refreshes persist and restore only plan-affecting session variables. Because enable_hyperscan_fallback is marked only as execution-affecting, a creator's strict false value is lost before the first background refresh and after replay; the refresh uses the fresh context's default true and falls back instead of returning the requested error. The inline comment contains the full trace and requested coverage.

Critical-checkpoint conclusions:

  • Goal and proof: the head guards all six direct Hyperscan compiler sites and threads the option through ordinary query and load execution. The MTMV context trace demonstrates one remaining reachable loss of the strict policy.
  • Scope and clarity: the helper, Thrift field, FE propagation, load snapshots, and tests are otherwise internally consistent; no unrelated implementation change was introduced in the reviewed patch.
  • Concurrency and lifecycle: the previously reported RPC stream-load snapshot race is fixed on this head. Broker, routine, SQL, transaction, cloud/local, legacy/Nereids, and multi-table task lifecycles preserve their captured value; MTMV's disconnected refresh context does not.
  • Static/global state and configuration dynamics: old load journals default compatibly to true, and request/job snapshots avoid mutable-global reads. Dynamic point-query/group-commit reuse and mixed-FE forwarding concerns already have live threads and are not duplicated here.
  • Compatibility: the optional Thrift field and missing old-record keys retain the historical true default. Existing mixed-version concerns remain fenced by their current discussion; the new finding is an independent same-version MTMV persistence gap.
  • Parallel paths and error handling: constant/dynamic LIKE/REGEXP, no-index MATCH, v1/v2 index, multi-pattern/edit-distance, coordinator variants, and load planners were traced. Strict interception reports an error where the initiating policy reaches BE; MTMV incorrectly changes that condition by sending true.
  • Tests and results: changed BE/FE tests cover the ordinary threshold, strict/default, compiler, load, and replay paths, but not initial or replayed MTMV refresh. No local build or test was run because the review runner prohibits it. Current CI has CheckStyle, Clang Formatter, dependency review, license, secrets, and large-file checks passing; macOS BE UT remains pending.
  • Observability, persistence, and data writes: no separate observability defect was found. MTMV's durable plan-only session map is the persistence boundary that loses the value, and a regexp filter can therefore materialize rows under fallback semantics instead of failing strictly.
  • FE/BE interface and performance: normal coordinators serialize the option correctly, but MTMV supplies the wrong fresh-context value before that boundary. All direct compiler sites invoke the cost guard before compilation; no distinct unthreaded compiler or additional performance defect was found beyond issues already discussed.
  • Additional focus: no extra user-provided review focus was supplied.

The current head was checked against the authoritative 46-file bundle. Two full rounds converged: the second normal BE, normal FE, and separate risk-focused reviews all returned no new valuable findings, and the final comment set remained unchanged.

Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@HappenLee

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17770 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68f72081dccb35377e5f4793c5f7841425b88e74, data reload: false

------ Round 1 ----------------------------------
============================================
q1	16829	3144	3161	3144
q2	1899	226	150	150
q3	10098	902	524	524
q4	4653	247	196	196
q5	7781	577	386	386
q6	147	115	95	95
q7	529	523	385	385
q8	9377	949	906	906
q9	3449	2349	2351	2349
q10	6498	853	720	720
q11	441	258	241	241
q12	729	398	337	337
q13	17868	1903	1548	1548
q14	167	154	141	141
q15	q16	457	412	369	369
q17	832	755	842	755
q18	3115	2284	2247	2247
q19	2075	905	831	831
q20	648	541	470	470
q21	5686	1743	1980	1743
q22	335	263	233	233
Total cold run time: 93613 ms
Total hot run time: 17770 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3549	3476	3463	3463
q2	208	219	160	160
q3	2279	2318	2169	2169
q4	1188	1177	899	899
q5	2196	2113	2087	2087
q6	173	118	89	89
q7	1043	955	872	872
q8	1620	1443	1438	1438
q9	3112	3071	3075	3071
q10	1863	1803	1636	1636
q11	356	269	259	259
q12	463	449	344	344
q13	1851	1852	1533	1533
q14	171	180	171	171
q15	q16	391	400	357	357
q17	1054	1032	1043	1032
q18	4952	4422	4806	4422
q19	873	855	851	851
q20	965	941	826	826
q21	3944	3189	3376	3189
q22	424	350	330	330
Total cold run time: 32675 ms
Total hot run time: 29198 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 85511 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68f72081dccb35377e5f4793c5f7841425b88e74, data reload: false

query5	4249	417	334	334
query6	406	165	152	152
query7	4880	440	272	272
query8	286	122	110	110
query9	8660	2931	2924	2924
query10	380	256	205	205
query11	5397	1037	921	921
query12	118	72	69	69
query13	1201	460	319	319
query14	5924	2021	1866	1866
query14_1	1809	1780	1746	1746
query15	170	109	108	108
query16	916	391	344	344
query17	775	433	346	346
query18	2319	325	227	227
query19	165	145	108	108
query20	70	68	66	66
query21	217	117	101	101
query22	5453	5379	5336	5336
query23	7353	6718	6645	6645
query23_1	6586	6630	6382	6382
query24	7081	1064	775	775
query24_1	769	772	783	772
query25	412	273	235	235
query26	1268	255	159	159
query27	2736	453	297	297
query28	4496	1493	1506	1493
query29	910	428	341	341
query30	277	176	156	156
query31	970	674	600	600
query32	93	47	45	45
query33	447	203	163	163
query34	1008	838	477	477
query35	405	390	327	327
query36	556	571	539	539
query37	121	83	74	74
query38	998	864	820	820
query39	521	529	541	529
query39_1	537	503	522	503
query40	238	121	109	109
query41	58	52	51	51
query42	76	75	74	74
query43	250	255	223	223
query44	1033	568	578	568
query45	113	104	110	104
query46	774	825	516	516
query47	954	985	943	943
query48	297	321	233	233
query49	539	242	191	191
query50	843	329	256	256
query51	8341	8404	8354	8354
query52	71	67	61	61
query53	204	212	159	159
query54	227	174	198	174
query55	75	65	53	53
query56	238	213	232	213
query57	684	605	636	605
query58	224	218	170	170
query59	1102	1141	1014	1014
query60	253	211	186	186
query61	128	112	113	112
query62	389	200	178	178
query63	185	157	150	150
query64	2744	650	546	546
query65	1622	1595	1510	1510
query66	1781	289	239	239
query67	9742	9650	9784	9650
query68	3008	1237	789	789
query69	346	224	187	187
query70	666	584	578	578
query71	296	253	260	253
query72	2331	1716	1588	1588
query73	647	567	328	328
query74	1992	1246	1152	1152
query75	1232	1155	1025	1025
query76	2217	743	555	555
query77	244	249	206	206
query78	5007	4801	4458	4458
query79	1008	880	586	586
query80	502	376	339	339
query81	414	194	182	182
query82	615	133	106	106
query83	341	250	232	232
query84	306	122	100	100
query85	804	460	397	397
query86	314	181	173	173
query87	983	989	895	895
query88	2779	2136	2130	2130
query89	291	227	228	227
query90	1795	142	146	142
query91	168	143	120	120
query92	62	45	47	45
query93	1254	1115	760	760
query94	418	250	232	232
query95	640	420	327	327
query96	832	592	264	264
query97	1088	1039	1003	1003
query98	141	141	128	128
query99	431	337	307	307
Total cold run time: 176713 ms
Total hot run time: 85511 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68f72081dccb35377e5f4793c5f7841425b88e74, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.03	0.04
query3	0.26	0.12	0.11
query4	1.60	0.10	0.11
query5	0.17	0.15	0.16
query6	1.24	0.68	0.68
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.20	0.21
query10	0.34	0.34	0.35
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.30	0.31	0.32
query14	0.45	0.46	0.46
query15	0.37	0.36	0.36
query16	0.24	0.23	0.23
query17	0.68	0.71	0.66
query18	0.17	0.17	0.17
query19	1.13	1.19	1.14
query20	0.02	0.01	0.01
query21	15.43	0.15	0.11
query22	5.08	0.04	0.05
query23	16.38	0.26	0.10
query24	2.73	0.30	0.27
query25	0.10	0.05	0.03
query26	0.74	0.17	0.13
query27	0.04	0.03	0.04
query28	3.60	0.52	0.30
query29	12.47	3.19	2.58
query30	0.25	0.11	0.13
query31	2.76	0.37	0.16
query32	3.53	0.33	0.22
query33	1.53	1.43	1.44
query34	15.36	2.26	1.74
query35	1.72	1.73	1.69
query36	0.45	0.29	0.28
query37	0.07	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.07	0.07
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 90.31 s
Total hot run time: 14.65 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 46.94% (23/49) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 60.22% (26607/44183)
Line Coverage 44.71% (271427/607122)
Region Coverage 40.40% (216539/535996)
Branch Coverage 41.95% (99633/237526)

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.23% (2001/2558)
Line Coverage 65.29% (36500/55907)
Region Coverage 65.48% (18562/28347)
Branch Coverage 55.09% (10096/18328)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.78% (32065/42879)
Line Coverage 59.23% (355796/600677)
Region Coverage 55.53% (296659/534269)
Branch Coverage 56.33% (133480/236946)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 83.67% (41/49) 🎉
Increment coverage report
Complete coverage report

@HappenLee
HappenLee dismissed github-actions[bot]’s stale review August 16, 2026 13:01

no need fix in this pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants