Phase 0: PostgreSQL 16 + Apache AGE 1.6.0 + pgvector 0.8.2 compatibility spike

.github/workflows/database-image.yml

@@ -0,0 +1,33 @@
+name: Phase 0 database image
+
+on:
+  pull_request:
+    paths:
+      - "docker/postgres/**"
+      - "compose.test.yaml"
+      - "scripts/verify-extensions"
+      - "tests/test_database_smoke.py"
+      - "pyproject.toml"
+      - ".github/workflows/database-image.yml"
+  push:
+    branches: [main]
+    paths:
+      - "docker/postgres/**"
+      - "compose.test.yaml"
+      - "scripts/verify-extensions"
+      - "tests/test_database_smoke.py"
+      - "pyproject.toml"
+
+jobs:
+  compatibility:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - run: docker compose -f compose.test.yaml build postgres
+      - run: docker compose -f compose.test.yaml up -d postgres
+      - run: ./scripts/verify-extensions
+      - run: uv run pytest -m "database or integration"
+      - if: always()
+        run: docker compose -f compose.test.yaml logs postgres

.gitignore

@@ -1,2 +1,8 @@
 PROJECT/PDDA-ACTIVITY.jsonl
 .pdda-gh-state.tsv
+
+# Python (first added by the Phase 0 compatibility spike)
+__pycache__/
+*.pyc
+.venv/
+.pytest_cache/

CHANGELOG.md

@@ -3,6 +3,31 @@
 Newest-first, dated end-of-iteration record. One entry per substantive iteration: what changed,
 why, and the verification. See `PROJECT/PDDA.md` for the full contract.
 
+## 2026-07-21
+
+### Phase 0 PostgreSQL extension compatibility implementation
+
+- Added a two-stage PostgreSQL 16.14 Bookworm image that builds and commit-verifies pgvector `v0.8.2`
+  and Apache AGE `PG16/v1.6.0-rc0`; the runtime is non-root and intentionally contains no compiler,
+  Git, or build source.
+- Added database initialization, relational/vector/AGE/combined SQL smoke fixtures, a minimal
+  Postgres-only compose surface, and a verification script. The combined proof binds a vector
+  candidate into a fixed Cypher query rather than accepting Cypher text.
+- Added real psycopg and SQLAlchemy smoke tests. The one-slot SQLAlchemy pool deliberately receives
+  a corrupted `search_path`; its checkout handler runs `LOAD 'age'` and resets the required search
+  path before the reused connection can execute AGE work.
+- Added architecture/upgrade documentation and a GitHub Actions workflow that runs the complete Phase
+  0 compatibility gate on Ubuntu amd64.
+- The first scoped image build exposed the missing PostgreSQL server-header package; the builder now
+  installs `postgresql-server-dev-16` while the runtime remains compiler-free.
+- Bet: the pinned AGE branch and pgvector coexist on PostgreSQL 16.14 and the checkout hook makes AGE
+  session initialization reliable. Reversibility: **Costly** after application code depends on it;
+  a failed build or smoke test is a stop condition, not an invitation to scaffold later phases.
+
+Verification: not run locally in this builder pass; the scoped Docker/pytest gate remains the required
+runtime evidence, and `docs/upgrade-matrix.md` deliberately marks every compatibility row as planned
+or unverified until it passes.
+
 ## 2026-07-20
 
 ### Build plan onto the PDDA contract; auth simplified to `reflex-local-auth`

PROJECT/2-WORKING/PROJECT-marathon/MARATHON-PLAN.md

@@ -0,0 +1,128 @@
+---
+title: Reflex Hybrid Analytics Starter — Marathon Plan & Swarm Analysis
+status: Draft — not fired
+created: 2026-07-20
+owner: noel
+source_doc: PROJECT/2-WORKING/PROJECT.md
+plan: PROJECT/2-WORKING/PROJECT-marathon/MARATHON.yaml
+doc_type: marathon-plan
+---
+
+# Marathon Plan & Swarm Analysis — PROJECT.md
+
+## Bottom line
+
+- **A marathon here is a *serial, self-gating chain*, not a swarm.** `marathon.sh` runs phases
+  strictly one at a time (GH-241); the seven-phase chain in `MARATHON.yaml` encodes PROJECT.md's
+  hard Phase-0 gate and each phase's exit gate. It "completes end-to-end" **iff every gate passes** —
+  the first failure halts it and drops an `ESCALATION.md`. That is the correct behaviour for this
+  plan, which forbids scaffolding past an unproven database.
+- **True wall-clock swarm (parallel agents) is only safe on disjoint write-sets** — and in this plan
+  that means the **scaffolding/docs lanes**, never the phase spine. Those belong to the `tick`/`xyz`
+  concurrent-lane system, mapped below.
+- **"End-to-end autonomous completion of all 7 phases" is not advisable and is not what I built a
+  fire button for.** Phase 0 is a genuine research gate with real stop conditions (AGE pooling is the
+  live unknown). The honest, fireable unit is **Phase 0 alone**, then re-evaluate. The chain exists so
+  the *order and gates* are encoded and machine-checked — not so you set it running and walk away.
+
+## Deliverables in this folder
+
+| File | Purpose |
+|---|---|
+| `MARATHON.yaml` | The 7-phase serial chain (p0→p6), self-gating. Builder = `codex` (default), reviewer = `agy` on every phase — the two must differ, so with the default codex builder every reviewer is agy. |
+| `briefs/p0…p6-*.md` | Thin per-phase briefs (deliverables + exit gate + anti-goals) pointing back to PROJECT.md. |
+| `MARATHON-PLAN.md` | This report. |
+| (in PROJECT.md) `## Swarm Preflight Contract` | Machine-readable contract scoping the Phase-0 lane. |
+
+## The two mechanisms, and why they don't mix here
+
+| | Marathon (`marathon.sh`) | Swarm (`tick` / `xyz`) |
+|---|---|---|
+| Concurrency | **None** — strictly one phase at a time | Real — N agents at once |
+| Safety rule | `depends_on` order + per-phase gate | Disjoint path-scoped write-sets, kernel ≤ 1/wave |
+| Good for | Ordered, gated builds where each step must pass before the next | Independent lanes with no shared files |
+| Used here for | The **phase spine** (Phase 0→6) | Optional: decomposing a single **post-P0** build phase into disjoint sub-lanes, run **in place of** that phase (not beside it) |
+
+The phase spine cannot be swarmed: each phase's write-set overlaps the next (shared `packages/`,
+`pyproject.toml`, migrations, ledgers) **and** each is a hard gate on the next. So the spine is the
+marathon's job. Concurrency here is only an optional per-phase decomposition — see the next section.
+
+## Swarm possibilities (an honest accounting)
+
+Short version: **within this marathon there is no free concurrent swarm.** The chain is strictly
+serial, and the "lanes" one might imagine parallelizing are not separate work — they are the *contents
+of the build phases themselves*. The table below is not a list of things to run beside the marathon; it
+is a decomposition of what P0/P1 build, shown to make one point precisely:
+
+| Candidate lane | Write-set | What it actually is |
+|---|---|---|
+| S1 ADR skeletons | `docs/architecture/decisions/**` | part of P0/P1 |
+| S2 Package skeletons | `packages/**` | **is P1** |
+| S3 Docker image | `docker/**` | **is P0** |
+| S4 App scripts | `scripts/{bootstrap,dev,…}` | part of P0/P1 |
+| S5 CI workflows | `.github/**` | part of P0/P1 |
+
+Because S1/S2/S4/S5 all live inside P1's write-set (`MARATHON.yaml` p1 `artifact:`), they **cannot run
+as a separate swarm beside P1** — they would collide with it. Nor can they run *before* P1: the
+Phase-0 gate withholds all downstream scaffolding until P0 is proven.
+
+**The one genuine parallelism option** is to *replace* a single build phase with a fan-out of its own
+disjoint sub-components — e.g. after P0 approves, build P1 not as one marathon phase but as a `tick`
+swarm of {packages, scripts, CI, ADRs} lanes run **in place of** marathon `p1` (never in addition to
+it). Then `p1`'s serial slot *is* that swarm; you don't run both. Two rules still bind it:
+
+- **Ledgers serialize.** `README.md`, `AGENTS.md`, `ROADMAP.md`, `CHANGELOG.md` — route every edit
+  through one writer; concurrent ledger edits collide.
+- **Kernel/orchestrator zone is off-limits** in every lane: `.xyz/`, `bin/`, `utils/pdda/`
+  (`orchestrator_only` in the preflight contract).
+
+Bottom line for the original question ("analyze for swarm possibilities"): the marathon answers *order
+and gating*; concurrency is available only as an optional **per-phase decomposition after P0**, run in
+place of that phase — never as a background swarm alongside the serial chain.
+
+## Recommended execution (honest path)
+
+1. **Dry-run everything first** (zero cost):
+   ```bash
+   .xyz/relay-automation/marathon.sh --plan PROJECT/2-WORKING/PROJECT-marathon/MARATHON.yaml \
+     --pre-advance-cmd "uv run pytest -m 'database or integration'" --dry-run
+   .xyz/utils/swarm-preflight.sh --project-doc PROJECT/2-WORKING/PROJECT.md --dry-run
+   ```
+2. **Fire Phase 0 only.** It carries every real risk (AGE pooled-connection init). Retry a single
+   phase with `--retry p0`. Do not queue p1–p6 to run unattended behind it.
+3. **At the Phase 0 gate:** if it passes, re-evaluate and release p1. If a stop condition hits, capture
+   logs, write a blocking task + ADR, recommend the smallest next experiment — do not scaffold onward.
+4. **Only after P0 is approved** is there any concurrency option — and only by *decomposing* a single
+   build phase (e.g. P1) into a `tick` swarm run **in place of** that marathon phase, per the section
+   above. There is no swarm to run *alongside* the serial chain, and nothing downstream runs during P0.
+
+## Validation results (dry-run, zero cost)
+
+| Check | Result |
+|---|---|
+| `bin/marathon-yaml … --format tsv` | **PASS** — parses; order resolves p0→p1→…→p6. |
+| `marathon.sh --plan … --dry-run` | **PASS** — 7 phases render in order; each phase's RELAY.md + tick seed produced. (First attempt caught a real bug: reviewer `codex` collided with the default `codex` builder — builder ≠ reviewer is required. Fixed: all reviewers → `agy`.) |
+| `swarm-preflight.sh --project-doc PROJECT.md --dry-run` | **NOT-READY (exit 5), expected.** Contract is well-formed — parsed, freshness OK, `fix_probes` (path_absent Dockerfile) satisfied, branch computed. It fails only because it checks that `artifacts[]` **already exist** at the ref. |
+
+### Why preflight says NOT-READY, and why that's correct
+
+`swarm-preflight.sh` is a **remediation** planner: it proves *an existing thing still needs fixing*
+before spending a lane, so it requires the declared `artifacts[]` to be present at the ref. This is a
+**greenfield build** — `docker/postgres/` doesn't exist yet; creating it is the point. So preflight is
+structurally the wrong gate for Phase 0 of a new build, and reporting NOT-READY is honest, not a
+defect in the contract. **Do not fabricate empty artifacts to force a `ready` verdict** — that would
+fake the exact evidence preflight exists to provide.
+
+- **Use `marathon.sh --dry-run` (which passes) as the readiness gate for this greenfield chain.**
+- The contract in PROJECT.md is **Phase-0-scoped and single-use**: its lone probe is
+  `path_absent docker/postgres/Dockerfile`, which goes **stale** the moment P0 lands. It is *not* a
+  reusable Phase-1+ gate. Each eligible next lane should get its **own** freshly-authored contract
+  (its own `fix_probes`/`artifacts`), not this one — treat the P0 contract as archival after P0.
+
+## Guardrails honored
+
+- **No branch auto-cut, no auto-fire** (GUIDING-PRINCIPLES.md §8). `risk: 3` → `skip_branch_prompt`
+  stays false; the orchestrator must prompt before cutting `marathon/reflex-hybrid-analytics-starter-<date>`.
+- **No gate skipped.** No root `validate.sh` exists, so a real `--pre-advance-cmd` must be supplied at
+  launch; the plan fails fast rather than advancing ungated.
+- **Nothing fired by this document.** This is a plan handed to the operator.

PROJECT/2-WORKING/PROJECT-marathon/MARATHON-p0.yaml

@@ -0,0 +1,20 @@
+# MARATHON-p0.yaml — Phase 0 ONLY, derived from the QA-approved MARATHON.yaml.
+#
+# The approved plan (MARATHON-PLAN.md) directs: "Fire Phase 0 only ... Do not queue p1-p6 to run
+# unattended behind it." marathon.sh runs a full chain and auto-advances on a passing gate, so firing
+# the full MARATHON.yaml could carry into p1. This single-phase plan makes "P0 only" unambiguous:
+# the chain has exactly one phase and stops when it approves or halts.
+#
+# Fire:
+#   .xyz/relay-automation/marathon.sh --plan PROJECT/2-WORKING/PROJECT-marathon/MARATHON-p0.yaml \
+#     --pre-advance-cmd "uv run pytest -m 'database or integration'"
+
+name: reflex-hybrid-analytics-starter-p0
+phases:
+  - id: p0
+    name: Compatibility spike (PG16 + AGE + pgvector, pooled)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p0-compat-spike.md
+    artifact: docker/,compose.test.yaml,scripts/,tests/,pyproject.toml,docs/,.github/,AGENTS.md,ROADMAP.md,CHANGELOG.md
+    turn_timeout_s: 2400
+    max_review_rounds: 3

PROJECT/2-WORKING/PROJECT-marathon/MARATHON.yaml

@@ -0,0 +1,82 @@
+# MARATHON.yaml — Reflex Hybrid Analytics Starter, full build plan as a self-gating chain.
+#
+# Run (ALWAYS dry-run first):
+#   .xyz/relay-automation/marathon.sh --plan PROJECT/2-WORKING/PROJECT-marathon/MARATHON.yaml \
+#     --pre-advance-cmd "uv run pytest -m 'database or integration'" --dry-run
+#
+# SEQUENCING (GH-241): marathon.sh runs phases STRICTLY ONE AT A TIME. This chain is intentionally
+# serial — the `depends_on` links encode PROJECT.md's hard Phase-0 gate and each phase's exit gate.
+# It is NOT a swarm; real wall-clock parallelism lives in the tick-lane map in MARATHON-PLAN.md.
+#
+# SELF-GATING: the chain HALTS on the first phase that fails its review or the --pre-advance-cmd gate,
+# leaving that phase's ESCALATION.md behind. So "end-to-end" here means "runs to completion IFF every
+# gate passes" — Phase 0 failing stops everything, exactly as the build plan demands.
+#
+# GATE: this repo has no root validate.sh. The canonical launch gate is the Phase-0 pytest markers,
+# which p0 itself produces:  --pre-advance-cmd "uv run pytest -m 'database or integration'".
+# (A repo-local `scripts/test` wrapper is optional and, if used, is a p0 deliverable — not assumed.)
+# A missing default fails fast rather than being silently skipped.
+
+name: reflex-hybrid-analytics-starter
+phases:
+  - id: p0
+    name: Compatibility spike (PG16 + AGE + pgvector, pooled)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p0-compat-spike.md
+    artifact: docker/,compose.test.yaml,scripts/,tests/,pyproject.toml,docs/,.github/,AGENTS.md,ROADMAP.md,CHANGELOG.md
+    turn_timeout_s: 2400
+    max_review_rounds: 3
+
+  - id: p1
+    name: Application spine (uv, SQLAlchemy, Reflex shell, auth, worker, mock engine)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p1-application-spine.md
+    artifact: apps/,packages/,migrations/,compose.yaml,compose.dev.yaml,pyproject.toml,.github/,docs/architecture/,scripts/
+    turn_timeout_s: 2400
+    depends_on: p0
+    max_review_rounds: 3
+
+  - id: p2
+    name: Safe CSV analytics (PandasAI adapter, sandbox ADR, artifact service)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p2-safe-csv-analytics.md
+    artifact: apps/,packages/,docs/threat-model.md,docs/architecture/decisions/,scripts/
+    turn_timeout_s: 2100
+    depends_on: p1
+    max_review_rounds: 3
+
+  - id: p3
+    name: Vector retrieval (embeddings protocol, pgvector search, owner filters)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p3-vector-retrieval.md
+    artifact: apps/,packages/,tests/
+    turn_timeout_s: 2100
+    depends_on: p2
+    max_review_rounds: 2
+
+  - id: p4
+    name: Graph capability (AGE gateway, curated commands, transaction tests)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p4-graph-capability.md
+    artifact: apps/,packages/,tests/
+    turn_timeout_s: 2100
+    depends_on: p3
+    max_review_rounds: 3
+
+  - id: p5
+    name: Hybrid retrieval example (query-time CTE, observable stages, provenance)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p5-hybrid-retrieval.md
+    artifact: apps/,packages/,examples/,tests/
+    turn_timeout_s: 2100
+    depends_on: p4
+    max_review_rounds: 2
+
+  - id: p6
+    name: OSS hardening (README, threat model, SBOM, release, generator ADR)
+    reviewer: agy
+    brief: PROJECT/2-WORKING/PROJECT-marathon/briefs/p6-oss-hardening.md
+    artifact: README.md,SECURITY.md,CONTRIBUTING.md,docs/,.github/
+    turn_timeout_s: 1800
+    depends_on: p5
+    max_review_rounds: 2

PROJECT/2-WORKING/PROJECT-marathon/briefs/p0-compat-spike.md

@@ -0,0 +1,48 @@
+# Phase 0 — Compatibility spike
+
+> Source of record: [`PROJECT/2-WORKING/PROJECT.md` → Phase 0](../../PROJECT.md#phase-0--compatibility-spike).
+> This brief is a pointer + gate, not a re-spec. Read the full section before building.
+
+## Objective
+
+Prove the riskiest technical assumption with the smallest possible implementation:
+**Apache AGE + pgvector coexist on a PostgreSQL 16 image *we* build, and AGE session
+initialization survives connection pooling.** Nothing downstream is scaffolded until this passes.
+
+## Deliverables (must exist before the gate)
+
+- `docker/postgres/Dockerfile` (multi-stage, non-root, no compilers in final image)
+- `docker/postgres/versions.env` — pinned PG16 tag, pgvector version, AGE version/SHA (verified upstream, not from the survey table)
+- `docker/postgres/init/{001-extensions,002-roles,003-health}.sql`
+- `docker/postgres/test/{relational,vector,age,combined}-smoke.sql`
+- `compose.test.yaml` (postgres-only / minimal test profile)
+- `scripts/verify-extensions`
+- Python psycopg/SQLAlchemy smoke tests incl. **AGE pooled-connection re-init** (`LOAD 'age'` + `search_path` on every checkout)
+- `docs/architecture/database.md` compatibility notes + `docs/upgrade-matrix.md` with a verification date
+- Per PROJECT.md's initial-task acceptance criteria, P0 also lands: `AGENTS.md`, a **minimal database-image CI workflow** (`.github/workflows/database-image.yml`), and updated `ROADMAP.md`/`CHANGELOG.md`
+- **Failure-evidence paths are in scope** (see stop conditions): a blocking task doc under `docs/tasks/` + an ADR under `docs/architecture/decisions/` — so a failed gate is documented, not containment-blocked
+
+## Exit gate (all must pass)
+
+```
+docker compose -f compose.test.yaml build postgres
+docker compose -f compose.test.yaml up -d postgres
+./scripts/verify-extensions
+uv run pytest -m "database or integration"
+```
+
+Required proofs: relational CRUD+rollback · pgvector nearest-neighbour ordering ·
+AGE create/insert/query/rollback · **pooled AGE connection reused safely with init intact** ·
+combined fixture (shared external IDs across relational+vector+graph, vector candidates feed a
+curated Cypher expansion).
+
+## Stop conditions (halt the marathon, do not scaffold onward)
+
+Pooling init unreliable (**most likely**) · pgvector/AGE conflict in the image ·
+AGE won't build reproducibly on PG16 · host arch can't run image with no CI route ·
+licensing blocks redistribution. On any stop: capture logs, write a blocking task doc + ADR,
+recommend the smallest next experiment. **Do not paper over a failed gate.**
+
+## Anti-goals
+
+No Reflex UI. No application packages. No auth. No worker. This phase is the database image and its proofs only.