Summary
The amd64 public.ecr.aws/supabase/postgres:17.6.1.105 image deterministically SIGSEGVs a backend process (signal 11) while running one large single-transaction pgTAP test file — but only on GitHub Actions ubuntu-latest runners. The byte-identical image on a local amd64 workstation (Intel i7-8850H, AVX2) runs the same file cleanly every time, and our aarch64 production build is unaffected. This points to a CPU-microarchitecture-dependent code path in the amd64 17.6.1.105 build (the runner CPU is AVX-512-capable; our local CPU is not — AVX-512 is our leading hypothesis but is not core-dump-confirmed).
Environment
|
Image / build |
CPU |
Result |
GitHub Actions ubuntu-latest |
public.ecr.aws/supabase/postgres:17.6.1.105 (amd64) |
AVX-512-capable x86_64 (Xeon/EPYC-class) |
SIGSEGV, 6/6 runs |
| Local workstation |
same image (amd64) |
Intel i7-8850H (AVX2, no AVX-512) |
clean, every run |
| Supabase production |
aarch64 build — PostgreSQL 17.6 on aarch64-unknown-linux-gnu |
ARM64 |
clean |
- Local
version(): PostgreSQL 17.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 15.2.0, 64-bit
- Invocation:
docker exec -i supabase_db_<project> psql -U postgres -d postgres -v ON_ERROR_STOP=1 < <file>.sql (Supabase CLI local stack; the file runs inside one BEGIN … ROLLBACK).
Observed behavior
A single pgTAP file (plan(363), ~4300 lines, one long transaction in one backend) crashes the backend mid-run:
LOG: server process (PID …) was terminated by signal 11: Segmentation fault
LOG: terminating any other active server connections
… database system is in recovery mode
The cluster then enters recovery and takes down the sibling test files running after it.
What it is NOT (ruled out with evidence)
- Not OOM — signal 11 (not 9);
dmesg is empty; adding swap did not help.
- Not JIT —
jit=off set per-connection via PGOPTIONS (verified active via SHOW jit); still crashed.
- Not the outer plan shape — forcing plans with GUCs (
enable_bitmapscan/hashagg/hashjoin=off, max_parallel_workers_per_gather=0) did not change the crash.
- Not the SQL of the crashing statement — the crash lands on a statement (
SELECT * FROM a SECURITY DEFINER SETOF-uuid helper) invoked by a role whose EXECUTE on that function is revoked, so it errors 42501 at the ACL check before the function body runs. The identical call by a privileged role earlier in the same file (and the same query shape ~17× before) runs clean. So the fault is not inside that function's execution — it is a delayed manifestation of state accumulated earlier in the backend.
The accumulation that appears to trigger it
The file, in one long-lived backend, builds up an unusually heavy load before the crash: hundreds of set_eq / EXCEPT-based pgTAP assertions (each materializing temp result sets), a BYPASSRLS role, and repeated trigger DDL / RLS-policy evaluation, all in one transaction. The corruption seems laid down by that accumulation and surfaces one statement later. On a fresh backend (e.g. splitting the file so the tail runs in its own psql/backend) the crash locus changes — consistent with private-backend-memory corruption rather than a specific query.
Reproduction
The full test file is self-contained pgTAP and reproduces 6/6 on an AVX-512-capable amd64 runner. We do not yet have a reduced repro (bisecting requires iterating on the AVX-512 hardware, which we only have via CI). We can share the full file privately, or work with you on a minimal case. To attempt repro:
supabase start (or run public.ecr.aws/supabase/postgres:17.6.1.105 directly) on an AVX-512-capable amd64 host.
- Apply a schema with RLS + a
SECURITY DEFINER set-returning helper, then run a long single-transaction pgTAP file that accumulates several hundred set_eq/EXCEPT assertions + a BYPASSRLS role + trigger DDL.
- Observe the backend SIGSEGV partway through.
A build compiled -O2 with CPU-dispatched SIMD (or a gcc 15.2.0 codegen issue on AVX-512) is the working hypothesis; a backtrace/core from the crashing backend on the runner would confirm the faulting routine.
Impact & current workaround
- We have quarantined the one file from CI and gated it locally on AVX2; all other tests pass. No production impact — production is aarch64 (a different build with no x86 SIMD path), and the helper executes correctly there.
- Asking here because it is your image build: a fix in the amd64 17.6.1.x build would let everyone re-enable such tests on standard CI runners.
Questions
- Is a SIGSEGV in the
17.6.1.105 amd64 build on AVX-512 hardware a known issue?
- Any recommended mitigation short of pinning an older/newer 17.x image (a build flag, a disabled dispatch path)?
- Would a full repro file + a runner
lscpu be useful, or do you want a reduced case first?
Summary
The amd64
public.ecr.aws/supabase/postgres:17.6.1.105image deterministically SIGSEGVs a backend process (signal 11) while running one large single-transaction pgTAP test file — but only on GitHub Actionsubuntu-latestrunners. The byte-identical image on a local amd64 workstation (Inteli7-8850H, AVX2) runs the same file cleanly every time, and our aarch64 production build is unaffected. This points to a CPU-microarchitecture-dependent code path in the amd64 17.6.1.105 build (the runner CPU is AVX-512-capable; our local CPU is not — AVX-512 is our leading hypothesis but is not core-dump-confirmed).Environment
ubuntu-latestpublic.ecr.aws/supabase/postgres:17.6.1.105(amd64)PostgreSQL 17.6 on aarch64-unknown-linux-gnuversion():PostgreSQL 17.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 15.2.0, 64-bitdocker exec -i supabase_db_<project> psql -U postgres -d postgres -v ON_ERROR_STOP=1 < <file>.sql(Supabase CLI local stack; the file runs inside oneBEGIN … ROLLBACK).Observed behavior
A single pgTAP file (
plan(363), ~4300 lines, one long transaction in one backend) crashes the backend mid-run:The cluster then enters recovery and takes down the sibling test files running after it.
What it is NOT (ruled out with evidence)
dmesgis empty; adding swap did not help.jit=offset per-connection viaPGOPTIONS(verified active viaSHOW jit); still crashed.enable_bitmapscan/hashagg/hashjoin=off,max_parallel_workers_per_gather=0) did not change the crash.SELECT * FROM a SECURITY DEFINER SETOF-uuid helper) invoked by a role whoseEXECUTEon that function is revoked, so it errors42501at the ACL check before the function body runs. The identical call by a privileged role earlier in the same file (and the same query shape ~17× before) runs clean. So the fault is not inside that function's execution — it is a delayed manifestation of state accumulated earlier in the backend.The accumulation that appears to trigger it
The file, in one long-lived backend, builds up an unusually heavy load before the crash: hundreds of
set_eq/EXCEPT-based pgTAP assertions (each materializing temp result sets), aBYPASSRLSrole, and repeated trigger DDL / RLS-policy evaluation, all in one transaction. The corruption seems laid down by that accumulation and surfaces one statement later. On a fresh backend (e.g. splitting the file so the tail runs in its ownpsql/backend) the crash locus changes — consistent with private-backend-memory corruption rather than a specific query.Reproduction
The full test file is self-contained pgTAP and reproduces 6/6 on an AVX-512-capable amd64 runner. We do not yet have a reduced repro (bisecting requires iterating on the AVX-512 hardware, which we only have via CI). We can share the full file privately, or work with you on a minimal case. To attempt repro:
supabase start(or runpublic.ecr.aws/supabase/postgres:17.6.1.105directly) on an AVX-512-capable amd64 host.SECURITY DEFINERset-returning helper, then run a long single-transaction pgTAP file that accumulates several hundredset_eq/EXCEPTassertions + aBYPASSRLSrole + trigger DDL.A build compiled
-O2with CPU-dispatched SIMD (or agcc 15.2.0codegen issue on AVX-512) is the working hypothesis; a backtrace/core from the crashing backend on the runner would confirm the faulting routine.Impact & current workaround
Questions
17.6.1.105amd64 build on AVX-512 hardware a known issue?lscpube useful, or do you want a reduced case first?