Skip to content

fix(dist): diagnose jvm thread creation failures during startup - #3360

Merged
imbajin merged 5 commits into
apache:masterfrom
pallavithegod:fix/issue-3072-docker-seccomp-diagnostics
Oct 21, 2026
Merged

imbajin merged 5 commits into
apache:masterfrom
pallavithegod:fix/issue-3072-docker-seccomp-diagnostics

Conversation

@pallavithegod

@pallavithegod pallavithegod commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Main Changes (主要改动)

Close #3072.

  • Root cause: diagnose_jvm_startup_failure() in bin/util.sh could let generic crash headers hide EPERM details from redirected JVM stdout. print_jvm_thread_hint() also matched unrelated filesystem errors containing a thread name.
  • Fix details: Record startup log offsets and inspect newly written stdout bytes before crash headers. Give separate advice for confirmed pthread_create EPERM and generic GC worker failures, preserve launcher exit status, and ignore stale diagnostics. Remove the broad thread-name/Operation-not-permitted pattern so filesystem failures do not receive thread-creation advice.
  • Tests: test-docker-entrypoint-process.sh covers redirected output, missing/stale/truncated diagnostics, process lifecycle and signal cleanup. New early42-file-eperm-stdout and early42-file-eperm-log fixtures reject the seccomp hint for unrelated filesystem failures. HugeSecurityManagerTest and the entrypoint configuration fixture remain passing.

Verifying these changes (验证改动)

  • Added/updated unit tests in test-docker-entrypoint-process.sh; ran HugeSecurityManagerTest
  • Verified repository mvn checkstyle:check and mvn apache-rat:check pass locally
  • Ran editorconfig:format, clean compilation and all 32 entrypoint process scenarios and four partition diagnostic controls
  • Doc - Done: container startup troubleshooting documents host/runtime diagnosis and the narrower diagnostic guarantee.

Log scanning covers appends and observed shrink; a replaced or truncated-and-regrown file reaching the old offset is outside that guarantee. Current-head upstream CI requires maintainer approval; no live seccomp-host or packaged-runtime acceptance is claimed.

Visual overview

Conceptual overview of PR #3360

Conceptual startup-diagnostic flow: inspect fresh stdout before generic crash headers, distinguish confirmed pthread_create EPERM from generic GC-worker failures, and avoid thread-denial advice for unrelated filesystem EPERM. Preserve the original exit status and ignore stale output. Log scanning covers appends and observed shrink, not every file-replacement scenario.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The seccomp hint this PR adds for #3072 is never printed for a real JVM, because HotSpot writes the pthread_create failed (EPERM) line to its stdout, not to the crash file or hugegraph-server.log, so the real failure gets the memory hint instead; separately, the EPERM hint suggests raising nproc, which cannot cause EPERM. Evidence: reproduced the #3072 failure with docker run --security-opt seccomp=<profile returning EPERM for clone3> eclipse-temurin:17-jre-jammy java -XX:ErrorFile=/out/hs_err_pid%p.log -version (EPERM warning on stdout, crash file matches only Cannot create worker GC thread); ran test-docker-entrypoint-process.sh at fe5adb2 on Linux (passes), then a copy whose early42-resource fixture emits the real JVM output and asserts the seccomp hint (fails, prints the GC worker/memory hint); static reading of util.sh, docker-entrypoint.sh, init-store.sh, hugegraph-server.sh and both Dockerfiles (STDOUT_MODE=true) at this head.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh
Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
@pallavithegod

Copy link
Copy Markdown
Contributor Author

Thanks for reproducing this. I will use realistic JVM stdout/crash-file fixtures, make the generic crash hint cover the possible seccomp cause, and remove process-limit advice from the EPERM-specific hint.

Cover possible seccomp denial in generic crash-file hints.
Keep process-limit advice separate from EPERM denials.
Exercise stdout warnings with realistic crash-file fixtures.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh independent cross/adversarial review of the reviewed revision: 8/10 overall. The previous crash-advice/stdout and EPERM-specific process-limit issues are addressed; no new confirmed source blocker was found in the independently cross-reviewed change.

Independent execution passed the complete Docker entrypoint configuration fixture, 26 process-lifecycle scenarios and four partition-diagnostic controls. This checks the corrected generic-crash hint, realistic stdout/crash-file fixtures and partition diagnostics. It does not establish a real seccomp/JVM crash reproduction or an image build.

Current CI is action_required, with no check runs reported for this head. Wait for actual CI and packaged/runtime qualification; this is not yet merge-ready or an approval.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 10/10. Blocking: no.

a24c32d addresses both of my earlier comments:

  • Important (util.sh, the EPERM line never reaches the crash file, so the seccomp hint never showed): fixed in a24c32d. The generic GC-thread hint now names seccomp denial and points at the pthread_create failed (EPERM) line in the container log. I ran a real Temurin 17 JVM under a seccomp profile that returns EPERM for clone3. The crash file held only the generic header, and diagnose_jvm_startup_failure printed the new hint with the seccomp step.
  • Minor (nproc advice in the EPERM branch): fixed in a24c32d. The denied branch now covers only seccomp and the security policy.

I also checked these and found no new issues:

  • A JVM that reuses a PID across docker restart overwrites hs_err_pid.log (HotSpot opens it with O_TRUNC), so the -newermt filter still picks up the new file and skips old ones.
  • set -euo pipefail: every command that can fail inside the new functions is guarded, and run_startup_step returns init-store's status, including through the PASSWORD pipe.
  • stat -c, find -newermt and date +%N are only called from the Docker entrypoint on the jammy image. util.sh only gains function definitions, so the start scripts on macOS are not affected.
  • docker-entrypoint-test.sh passes at a24c32d (31 PASS).

What I could not check: CI has not run on this head (the run is still action_required), and I did not build the image or start a container end to end under seccomp.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The startup diagnostics work as described for the default STDOUT_MODE=true image, and the earlier stdout/crash-file and nproc threads are fixed at this head. One minor gap remains: with STDOUT_MODE=false the JVM's EPERM line goes to logs/hugegraph-server-stdout.log, which nothing scans, so the hint points to the container log instead. Evidence: read the full diff at a24c32d plus docker-entrypoint.sh, hugegraph-server.sh (crash-file and stdout redirection), start-hugegraph.sh, init-store.sh, log4j2.xml and both Dockerfiles; ran docker-entrypoint-test.sh at this head on Linux (configuration fixture and all process scenarios pass); simulated the STDOUT_MODE=false file layout against this head's util.sh and got the generic container-log hint; with the crash file removed, nothing was printed.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.23%. Comparing base (c2ec570) to head (bfe0ab5).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3360      +/-   ##
============================================
+ Coverage     55.93%   56.23%   +0.29%     
+ Complexity     2277        3    -2274     
============================================
  Files           774      508     -266     
  Lines         68534    49220   -19314     
  Branches       9333     7341    -1992     
============================================
- Hits          38334    27678   -10656     
+ Misses        26634    18899    -7735     
+ Partials       3566     2643     -923     
Flag Coverage Δ
commons ?
hstore ?
pd ?
server_hbase ?
server_memory 54.61% <ø> (+0.28%) ⬆️
server_rocksdb 42.80% <ø> (+0.46%) ⬆️
store ?

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Scan new stdout-file bytes before generic crash headers.
Ignore stale output and handle truncation between starts.
Cover file-output startup failures with and without crash files.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh independent cross/adversarial review of this revision: 8.5/10.

The previously reported STDOUT_MODE=false gap is repaired: newly appended JVM output is scanned before generic crash headers, including when no crash file exists. Independent Linux execution passed the shell configuration fixture, all 30 process-lifecycle scenarios and four partition-diagnostic controls. Substituting the previous util.sh into the focused new redirected-output case fails the EPERM-specific assertion; the current implementation passes. A second independent reviewer ran 20 old/current function probes covering append, stale output, observed shrink and generic/EAGAIN controls.

One nonblocking boundary at util.sh:50–54: an externally replaced or truncated-and-regrown log whose size reaches the old offset can still hide a new prefix warning. The normal launcher appends, so this does not block the intended repair; keep the guarantee scoped to append/observed shrink, or optionally detect file replacement without adding a monitoring subsystem.

Current-head CI is still action_required. Native Java configuration readback was skipped without its classpath. No independent image build, real JVM/seccomp reproduction, performance or same-RC acceptance is claimed. Await current CI and packaged-runtime qualification; this is a COMMENT, not approval.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The earlier threads are fixed at this head: the generic GC-thread hint names seccomp, the EPERM hint no longer gives process-limit advice, and the STDOUT_MODE=false stdout file is scanned before crash files. One minor gap remains: the [Tt]hread.*Operation not permitted rule in print_jvm_thread_hint matches startup errors that have nothing to do with thread creation, so those failures get the seccomp hint. Evidence: read the full diff against merge base be68aa2, plus docker-entrypoint.sh, init-store.sh, hugegraph-server.sh (crash-file options and stdout redirection), HugeGraphServer/HugeGraphServerBootstrap.main, conf/log4j2.xml, both Dockerfiles, test-docker-entrypoint.sh and the CI steps that run the shell fixtures. Fed sample lines to this head's print_jvm_thread_hint: an uncaught FileSystemException ... Operation not permitted line and a log4j line from a pool-3-thread-1 thread both print JVM thread creation was denied (EPERM). Checked that moving source ./bin/util.sh up causes no function-name collisions and that run_startup_step keeps init-store's exit status, including through the PASSWORD pipe. CI has not run on this head (action_required).

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh Outdated
Recognize pthread_create failures without matching unrelated thread names.

Cover filesystem EPERM in fresh stdout and server log bytes.

Preserve existing startup hints and lifecycle fixture coverage.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh independent cross/adversarial review of 67d1a37: 9/10. No new source blocker found.

The previously reported filesystem-EPERM false positive is fixed by removing the broad thread-name rule at util.sh:25–31. Independent Linux execution passed all 32 real shell/process scenarios and four partition-diagnostic controls. The two new negative cases retain exit 42 while suppressing misleading thread-denial/seccomp advice; substituting the previous helper into the focused stdout case fails that negative assertion. A second independent reviewer exercised 32 helper/input-path combinations: both pthread EPERM spellings and generic GC-worker diagnostics still work, while unrelated filesystem EPERM, EAGAIN and stale output remain silent.

The existing append/observed-shrink scope is unchanged and is now explicitly acknowledged in the PR description. This evidence closes the diagnostic-classification follow-up; it does not establish image, live JVM/seccomp, performance or same-RC acceptance. Exact-head CI is still action_required; no native Maven/full-reactor validation was independently run here. Await that CI and packaged-runtime qualification. This is a COMMENT, not approval.

@imbajin
imbajin merged commit ebb8f4e into apache:master Oct 21, 2026
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes:

3 participants