Skip to content

DO NOT MERGE: CI debug for segfault - #612

Closed
teunbrand wants to merge 22 commits into
posit-dev:mainfrom
teunbrand:ci-debug-segfault
Closed

teunbrand wants to merge 22 commits into
posit-dev:mainfrom
teunbrand:ci-debug-segfault

Conversation

@teunbrand

Copy link
Copy Markdown
Collaborator

Temporary draft PR to debug a SIGSEGV in the Build and Test workflow on the arrows branch (#PR). Runs tests single-threaded to identify the crashing test. Will be closed.

@teunbrand

Copy link
Copy Markdown
Collaborator Author

Closing: the segfault reproduces on unmodified main (run 42655753480), so this is a runner environment issue, not branch-specific.

@teunbrand teunbrand closed this Nov 10, 2026
@teunbrand
teunbrand deleted the ci-debug-segfault branch November 10, 2026 10:39
@teunbrand

Copy link
Copy Markdown
Collaborator Author

Posting summary of investigation by agent here:

SIGSEGV in ggsql CI test suite after ubuntu-latest rolled to 26.04

Symptom. cargo test --lib --bins on ubuntu-latest dies mid-suite with SIGSEGV (signal 11, invalid memory reference). No test is named — the whole ggsql test binary crashes. Always during the writer::hephaestus tests, i.e. the ones that render plots through hephaestus. First seen 2026-11-09; the same workflow was green 2026-11-04.

Environment change is the trigger. The crash reproduces on unmodified main (run 42655753480). Last green run used image ubuntu-24.04; failing runs use ubuntu-26.04 — ubuntu-latest rolled in between. Nothing in ggsql or hephaestus changed.

It's concurrency-dependent. Deterministic across reruns in parallel mode (3/3 crashed, at different points in the suite — expected, since the last "ok" isn't the crasher under parallel scheduling). With --test-threads=1 the entire ~2177-test suite passes cleanly, twice, including the full hephaestus writer tests. So: parallel-only, Linux-only, text-rendering-tests-only. All tests pass in parallel on macOS.

Suspect: fontconfig via the parley/fontique text stack. Hephaestus itself contains no unsafe (checked 0.4.1 sources). The native code on the crashing path is the text stack: parley → fontique → yeslogic-fontconfig-sys (dlopen mode, loads libfontconfig.so.1 at runtime) → system fontconfig. Fontconfig has a history of thread-safety problems, and ubuntu-26.04 ships a much newer libfontconfig than 24.04. This is inference from the crash's shape, not yet confirmed with a backtrace.

Not yet done: a native backtrace (run the test binary under gdb -batch -ex run -ex "thread apply all bt" on the runner) to name the crashing library and thread. That would confirm or refute fontconfig. Possible stopgap meanwhile: pin CI to ubuntu-24.04.

Repro on a 26.04 machine/container should just be: cargo test --lib writer:: in the ggsql repo (parallel, default features).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant