Conversation
- LimitCORE=infinity, scoped to postgresql.service only (not machine-wide) so an unrelated crash still produces no core - CoredumpFilter excludes shared_buffers (an anonymous shared mapping) from every core, since it can be many GB - this is what actually bounds core size - systemd-coredump gets its own conservative storage limits (compression, size caps), and is installed explicitly (not assumed to already be present) - confirm postgres_prestart.sh never resets ulimit -c (regression guard) - testinfra: LimitCORE, coredump_filter value, storage limits, coredump dir permissions, and a real end-to-end crash test (crash produces a core, an unrelated process's crash doesn't)
- postgres/orioledb debug+source packages already ship on every AMI (postgres-env bundle) - nothing new to package here, verification only - check build-ID match: shipped postgres binary and orioledb.so vs the installed debug package - check GDB can read real source content through the source package (not just that a filename is known - info sources lists names regardless of whether the file is actually reachable on disk)
- orioledb-coredump.path (triggers on new cores) + .timer (10-min fallback sweep) both run a flock-guarded oneshot service - gdb and binutils are installed explicitly for this (scoped to OrioleDB images via is_psql_oriole, since only this processor needs them) - processor script: keeps only postgres-owned cores, extracts a bundle via GDB (cmds.gdb - matches OrioleDB's own CI debugging script: full backtrace, argv, shared libraries, registers, lock state), deletes the raw core, quarantines metadata-only after repeated failures, and enforces its own age/size retention independent of systemd-coredump's own limits - handles /usr/lib/postgresql/bin/postgres being a Nix wrapper script (not the real ELF) throughout: the executable filter, readelf, and gdb all use the real per-crash path reported by coredumpctl instead of a fixed guess - matches/deletes cores by PID, not raw file path (coredumpctl has no "rm" verb on this systemd version, and path-based matching was unreliable) - reads the active postgresql log path from current_logfiles at runtime, since this AMI logs via csvlog with no fixed filename - rolled out to OrioleDB images only for now (is_psql_oriole) - testinfra: end-to-end check that a real crash produces a diagnostic bundle and the raw core gets deleted
|
Found 27 test failures on Blacksmith runners: Failures
...and 17 more test failures. View all on Blacksmith
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light.svg)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
feature: collect and process coredumps for orioledb instances.
What is the current behavior?
coredumpts arent' collected
What is the new behavior?
upon crash a coredump is written, then it is processed into a bundle including:
Additional context
this work is critical for orioledb team as we're about to start beta and debugging any future crashes without coredump is going to be very problematic.
One important caveat: currently only
postgresdebug symbols are shipped. Shippingorioledbdebug symbols is outside of the scope of this PR.