Skip to content

supabase/postgres:17.6.1.105 (amd64) — backend SIGSEGV during a heavy pgTAP file, CI-CPU-specific #2467

Description

@ghosgit

Summary

The amd64 public.ecr.aws/supabase/postgres:17.6.1.105 image deterministically SIGSEGVs a backend process (signal 11) while running one large single-transaction pgTAP test file — but only on GitHub Actions ubuntu-latest runners. The byte-identical image on a local amd64 workstation (Intel i7-8850H, AVX2) runs the same file cleanly every time, and our aarch64 production build is unaffected. This points to a CPU-microarchitecture-dependent code path in the amd64 17.6.1.105 build (the runner CPU is AVX-512-capable; our local CPU is not — AVX-512 is our leading hypothesis but is not core-dump-confirmed).

Environment

Image / build CPU Result
GitHub Actions ubuntu-latest public.ecr.aws/supabase/postgres:17.6.1.105 (amd64) AVX-512-capable x86_64 (Xeon/EPYC-class) SIGSEGV, 6/6 runs
Local workstation same image (amd64) Intel i7-8850H (AVX2, no AVX-512) clean, every run
Supabase production aarch64 build — PostgreSQL 17.6 on aarch64-unknown-linux-gnu ARM64 clean
  • Local version(): PostgreSQL 17.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 15.2.0, 64-bit
  • Invocation: docker exec -i supabase_db_<project> psql -U postgres -d postgres -v ON_ERROR_STOP=1 < <file>.sql (Supabase CLI local stack; the file runs inside one BEGIN … ROLLBACK).

Observed behavior

A single pgTAP file (plan(363), ~4300 lines, one long transaction in one backend) crashes the backend mid-run:

LOG:  server process (PID …) was terminated by signal 11: Segmentation fault
LOG:  terminating any other active server connections
… database system is in recovery mode

The cluster then enters recovery and takes down the sibling test files running after it.

What it is NOT (ruled out with evidence)

  • Not OOM — signal 11 (not 9); dmesg is empty; adding swap did not help.
  • Not JITjit=off set per-connection via PGOPTIONS (verified active via SHOW jit); still crashed.
  • Not the outer plan shape — forcing plans with GUCs (enable_bitmapscan/hashagg/hashjoin=off, max_parallel_workers_per_gather=0) did not change the crash.
  • Not the SQL of the crashing statement — the crash lands on a statement (SELECT * FROM a SECURITY DEFINER SETOF-uuid helper) invoked by a role whose EXECUTE on that function is revoked, so it errors 42501 at the ACL check before the function body runs. The identical call by a privileged role earlier in the same file (and the same query shape ~17× before) runs clean. So the fault is not inside that function's execution — it is a delayed manifestation of state accumulated earlier in the backend.

The accumulation that appears to trigger it

The file, in one long-lived backend, builds up an unusually heavy load before the crash: hundreds of set_eq / EXCEPT-based pgTAP assertions (each materializing temp result sets), a BYPASSRLS role, and repeated trigger DDL / RLS-policy evaluation, all in one transaction. The corruption seems laid down by that accumulation and surfaces one statement later. On a fresh backend (e.g. splitting the file so the tail runs in its own psql/backend) the crash locus changes — consistent with private-backend-memory corruption rather than a specific query.

Reproduction

The full test file is self-contained pgTAP and reproduces 6/6 on an AVX-512-capable amd64 runner. We do not yet have a reduced repro (bisecting requires iterating on the AVX-512 hardware, which we only have via CI). We can share the full file privately, or work with you on a minimal case. To attempt repro:

  1. supabase start (or run public.ecr.aws/supabase/postgres:17.6.1.105 directly) on an AVX-512-capable amd64 host.
  2. Apply a schema with RLS + a SECURITY DEFINER set-returning helper, then run a long single-transaction pgTAP file that accumulates several hundred set_eq/EXCEPT assertions + a BYPASSRLS role + trigger DDL.
  3. Observe the backend SIGSEGV partway through.

A build compiled -O2 with CPU-dispatched SIMD (or a gcc 15.2.0 codegen issue on AVX-512) is the working hypothesis; a backtrace/core from the crashing backend on the runner would confirm the faulting routine.

Impact & current workaround

  • We have quarantined the one file from CI and gated it locally on AVX2; all other tests pass. No production impact — production is aarch64 (a different build with no x86 SIMD path), and the helper executes correctly there.
  • Asking here because it is your image build: a fix in the amd64 17.6.1.x build would let everyone re-enable such tests on standard CI runners.

Questions

  1. Is a SIGSEGV in the 17.6.1.105 amd64 build on AVX-512 hardware a known issue?
  2. Any recommended mitigation short of pinning an older/newer 17.x image (a build flag, a disabled dispatch path)?
  3. Would a full repro file + a runner lscpu be useful, or do you want a reduced case first?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions