[tool](fe) add fast-compile-fe.sh for incremental FE compilation by englefly · Pull Request #61429 · apache/doris

englefly · 2026-03-17T07:44:20Z

Add tools/fast-compile-fe.sh, a developer utility that incrementally compiles only changed Java source files and patches them directly into doris-fe.jar, avoiding a full Maven rebuild during development.

Key features:

Auto-detects stale .java files by comparing mtime with .class files
Supports explicit file arguments (by path or filename search under src/)
Caches the classpath derived from target/lib to skip repeated mvn calls
Updates both target/doris-fe.jar and output/fe/lib/doris-fe.jar
Handles inner classes and anonymous classes (e.g. Foo$Bar.class)

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Add `tools/fast-compile-fe.sh`, a developer utility that incrementally compiles only changed Java source files and patches them directly into `doris-fe.jar`, avoiding a full Maven rebuild during development. Key features: - Auto-detects stale `.java` files by comparing mtime with `.class` files - Supports explicit file arguments (by path or filename search under src/) - Caches the classpath derived from `target/lib` to skip repeated mvn calls - Updates both `target/doris-fe.jar` and `output/fe/lib/doris-fe.jar` - Handles inner classes and anonymous classes (e.g. `Foo$Bar.class`) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

hello-stephen · 2026-03-17T07:44:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2026-03-17T07:44:39Z

run buildall

github-actions · 2026-03-17T13:07:57Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-03-17T13:08:00Z

PR approved by anyone and no changes requested.

englefly · 2026-03-18T00:16:07Z

run buildall

englefly · 2026-03-18T01:56:32Z

run buildall

morrySnow · 2026-03-18T02:18:17Z

/review

morrySnow · 2026-03-18T02:19:38Z

tools/fast-compile-fe.sh

+FE_CORE="$DORIS_HOME/fe/fe-core"
+SRC_ROOT="$FE_CORE/src/main/java"
+TARGET_CLASSES="$FE_CORE/target/classes"
+TARGET_LIB="$FE_CORE/target/lib"
+OUTPUT_JAR="$DORIS_HOME/output/fe/lib/doris-fe.jar"
+TARGET_JAR="$FE_CORE/target/doris-fe.jar"
+CP_CACHE="$FE_CORE/target/fast-compile-cp.txt"
+
+# Generated source directories (protobuf/thrift/annotation processor generated java files)
+GEN_SOURCES=(
+    "$FE_CORE/target/generated-sources/doris"
+    "$FE_CORE/target/generated-sources/org"
+    "$FE_CORE/target/generated-sources/java"
+    "$FE_CORE/target/generated-sources/annotations"
+    "$FE_CORE/target/generated-sources/antlr4"


could we get these paths from mvn pom.xml?

github-actions

Code Review Summary: tools/fast-compile-fe.sh

This PR adds a developer utility script for incremental FE compilation. The script is well-structured and useful. A few issues and suggestions below.

Critical Checkpoint Conclusions

Goal/correctness: The script achieves its stated goal of incremental compilation for fe-core. However, it only covers fe/fe-core while the FE has many sub-modules (fe-common, fe-catalog, fe-foundation, fe-type, etc.). Changes in those modules won't be picked up. This scope limitation should be clearly documented.
Modification minimal/focused: Yes, single new file, well-scoped.
Concurrency: N/A (shell script, single-threaded).
Lifecycle/static init: N/A.
Configuration items: N/A.
Incompatible changes: N/A — dev tool only, no production impact.
Parallel code paths: N/A.
Special conditional checks: The stale-file detection only checks mtime of the main .class file, not inner classes. If only an inner class is stale (e.g., due to a dependency change), it won't be detected. Acceptable trade-off for a dev tool.
Test coverage: No automated tests, which is fine for a developer utility script.
Observability: Colored info/warn/error output is adequate.
Performance: The find_stale_java_files function spawns one shell loop over all .java files in SRC_ROOT. For large source trees this is slow; a find ... -newer approach or rsync --dry-run would be faster, but acceptable for a dev tool.

Issues Found

(Medium) javac -source 8 -target 8 differs from the Maven build's <release>8</release>. The --release flag restricts API usage to the target JDK's boot classpath; -source/-target does not. This means the fast-compile path could successfully compile code that uses JDK 17+ APIs, which would then fail in the real Maven build. Consider using --release 8 instead.
(Medium) collect_updated_classes uses ${java_file#$SRC_ROOT/} to compute class file paths. If a user passes a file that is not under SRC_ROOT (e.g., a generated-sources file, or a file resolved from a different module), the prefix stripping is a no-op and the resulting class path will be completely wrong, silently producing no matches.
(Low) The trap "rm -f $tmpfile" EXIT on line 200 replaces any prior EXIT trap. Currently there is no prior trap so this is benign, but it's fragile for future edits. Consider using a single top-level cleanup function.
(Low) xargs jar uf on lines 211/217 splits on whitespace by default. If any class file path contains spaces, this will break. Use xargs -d '\n' (GNU xargs) for safety.
(Info) The script only supports fe/fe-core. The usage header and help text should mention this explicitly, since developers editing fe-common or fe-catalog might expect this tool to handle those modules too.

github-actions · 2026-03-18T02:21:20Z

tools/fast-compile-fe.sh

+
+    # Compile with javac
+    javac \
+        -source 8 -target 8 \


javac flags mismatch with Maven build: The Maven compiler plugin uses <release>8</release> (equivalent to --release 8), which restricts available APIs to the JDK 8 boot classpath. Using -source 8 -target 8 here only sets syntax/bytecode levels but allows using newer JDK APIs, meaning this fast-compile could succeed on code that would fail in the real Maven build.

Suggestion: replace with --release 8 (requires JDK 9+, which the project already mandates):

javac \ --release 8 \ -encoding UTF-8 \

github-actions · 2026-03-18T02:21:20Z

tools/fast-compile-fe.sh

+    local class_files=()
+
+    for java_file in "${java_files[@]}"; do
+        local rel_path="${java_file#$SRC_ROOT/}"


collect_updated_classes breaks for files outside SRC_ROOT: The prefix stripping ${java_file#$SRC_ROOT/} is a no-op if the file is not under SRC_ROOT (e.g., a generated-sources file or a file from another module). The resulting class_prefix will point to a nonsensical path and silently produce zero matches.

This can happen when a user passes an absolute path to a file outside fe-core/src/main/java/. Consider adding a validation or falling back to basename-based lookup.

github-actions · 2026-03-18T02:21:20Z

tools/fast-compile-fe.sh

+    # Convert class file paths to relative paths for jar command
+    local tmpfile
+    tmpfile="$(mktemp)"
+    trap "rm -f $tmpfile" EXIT


Minor: trap replaces prior EXIT handler: Each trap ... EXIT replaces the previous one. This is fine today but fragile if anyone adds cleanup elsewhere. A common pattern is to define a single cleanup function at the top and append to it:

_cleanup_files=() cleanup() { rm -f "${_cleanup_files[@]}"; } trap cleanup EXIT # ... later: _cleanup_files+=("$tmpfile")

github-actions · 2026-03-18T02:21:20Z

tools/fast-compile-fe.sh

+
+    # Update target/doris-fe.jar
+    if [[ -f "$TARGET_JAR" ]]; then
+        xargs jar uf "$TARGET_JAR" < "$tmpfile"


xargs splits on whitespace by default: If any class-file relative path ever contains spaces (unlikely but possible with generated code), this will break. Consider using xargs -d '\n' for robustness:

xargs -d '\n' jar uf "$TARGET_JAR" < "$tmpfile"

github-actions · 2026-03-18T02:21:20Z

tools/fast-compile-fe.sh

+set -euo pipefail
+
+DORIS_HOME="$(cd "$(dirname "$0")/.." && pwd)"
+FE_CORE="$DORIS_HOME/fe/fe-core"


Scope limitation not documented: This script only handles fe/fe-core, but the FE workspace contains many sub-modules (fe-common, fe-catalog, fe-foundation, fe-type, etc.). Developers editing those modules might expect this tool to handle them.

Consider adding a note to the usage comment, e.g.:

# Note: Only supports fe/fe-core source files. Changes in other FE # sub-modules (fe-common, fe-catalog, etc.) require a full Maven build.

morrySnow · 2026-03-18T02:25:38Z

tools/fast-compile-fe.sh

+
+    # Compile with javac
+    javac \
+        -source 8 -target 8 \


source and target read from pom.xml is better?

englefly changed the title ~~[dev-tool] add fast-compile-fe.sh for incremental FE compilation~~ [tool](fe) add fast-compile-fe.sh for incremental FE compilation Mar 17, 2026

starocean999 previously approved these changes Mar 17, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 17, 2026

github-actions bot added the reviewed label Mar 17, 2026

feiniaofeiafei approved these changes Mar 17, 2026

View reviewed changes

lang zh->en

777bafd

englefly dismissed starocean999’s stale review via 777bafd March 18, 2026 00:15

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 18, 2026

support provided jar

30235e9

morrySnow reviewed Mar 18, 2026

View reviewed changes

github-actions bot reviewed Mar 18, 2026

View reviewed changes

morrySnow reviewed Mar 18, 2026

View reviewed changes

tools/fast-compile-fe.sh

# Compile with javac

javac \

-source 8 -target 8 \

Copy link

Contributor

morrySnow Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source and target read from pom.xml is better?

Conversation

englefly commented Mar 17, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 17, 2026

Uh oh!

englefly commented Mar 17, 2026

Uh oh!

github-actions bot commented Mar 17, 2026

Uh oh!

github-actions bot commented Mar 17, 2026

Uh oh!

englefly commented Mar 18, 2026

Uh oh!

englefly commented Mar 18, 2026

Uh oh!

morrySnow commented Mar 18, 2026

Uh oh!

morrySnow Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Code Review Summary: tools/fast-compile-fe.sh

Critical Checkpoint Conclusions

Issues Found

Uh oh!

github-actions bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

morrySnow Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants