[grandixximo]

@hdiethelm, this is the coredumpctl bug you found in #4648.

A bare name given to coredumpctl matches the command name, and the GUIs start from a #!/usr/bin/python3.13 line, so systemd-coredump records a crashed gmoccapy as gmoccapy, with /usr/bin/python3.13 as the executable. The python3 match in crashdump.sh found nothing and the report always fell back to "no readable core dump". It now matches readlink -f "$(command -v python3)", which coredumpctl compares with the executable and which finds the right interpreter whatever its version, and gdb gets that path too.

Tested with systemd-coredump on Debian 13: a script with a #!/usr/bin/python3.13 line that segfaults after crashdump_arm. On master the report says there is no core; with this change it prints the native backtrace. The plain-file core_pattern path is unchanged.

[hdiethelm]

Thanks for fixing this. Tested, it works.

Now looking at the code: coredumpctl could race in parallel tests. You take the newest coredump since test start, right? So if multiple tests crash, you could get twice the same coredump or otherwise mixed up results.

There is pretty much no doc about this, so i tested it. Inside bwrap, you see coredumps created in an other bwrap the way it is set up right now with coredumpctl due to the host handles coredumps but read access inside the bwrap to coredumps works fine. However, the PID is not the one inside but the one from the host.

File coredumps work well due to there are handled respecting the mount namespace. With sysctl -w kernel.core_pattern=/tmp/linuxcnc-ui-smoke-cores/core.%e.%p, the coredumps land inside the bwrap, not visible from the host, so no risk of a race.

Do you have a good idea to fix this?

• Using the PID won't work. It is a different PID inside bwrap and the coredump shows always the host PID.
• We could check for races and show a warning. If there are more than one python3 coredump since @$crashdump_arm_time, probably something went wrong, so better ignore it than show the wrong one.
• Just write in the doc that for parallel tests, you need to set sysctl -w kernel.core_pattern=/tmp/linuxcnc-ui-smoke-cores/core.%e.%p

[grandixximo]

@hdiethelm, thanks for testing, and yes: it took the newest python core since arming, so in a parallel run one test can get the core of another. Counting the cores would not catch the worst case: a test that fails without crashing, while another test crashes, sees exactly one core and prints a backtrace that is not its own.

I pushed a fix. crashdump_arm now exports UI_SMOKE_CRASH_TAG, a fresh UUID, which the GUI inherits through linuxcnc. systemd-coredump keeps the environment of the crashed process with each core, so crashdump_report reads the python cores since arming from the journal, keeps the one whose environment carries our tag, and dumps it by the host PID recorded in that same entry. No PID from inside the sandbox is needed, and the plain-file core_pattern path is unchanged.

Tested with systemd-coredump on Debian 13, with the gmoccapy and axis ui-smoke tests run at the same time in two bwrap sandboxes and each GUI killed with SIGSEGV: before, both reports showed the axis core; now each shows its own. With a small crashing script in three sandboxes, two crashing and one not, the one that did not crash now reports no core instead of another's.

For parallel runs the file core_pattern is still the simpler setup, since the cores then stay inside each sandbox. I can add a line about it to the ui-smoke README if you think it helps.

[hdiethelm]

This is a neat trick, using the environment to tag a process family. I tested it, works nicely.

For the sandbox mode, I would not say using core_pattern is always better now that coredumpctl works reliably. Yes, it stays inside the sandbox but then you also have no way to debug it future from outside the sandbox, it is gone after sandbox exit.

There is one thing that I found while running the parallel tests: If a process segfaults in a test but does not alter the result, like rtapi_app on unload, some tests just pass. Using a similar tag + journalctl to scan for segfaults after each test should make it possible to detect this. However not every OS uses systemd. I will just create an issue for now to tackle this in the future, it should be a new PR after #4648 was merged.

[grandixximo]

@hdiethelm, thanks for testing. Fair point on the file core_pattern: the cores go away with the sandbox, so I will leave the README as it is. Catching a segfault that does not fail the test sounds useful, and a follow-up after #4648 makes sense to me.