Skip to content

feat: PostgreSQL backend - #611

Open
YpNo wants to merge 6 commits into
tale:mainfrom
YpNo:feat/postgres-backend
Open

feat: PostgreSQL backend#611
YpNo wants to merge 6 commits into
tale:mainfrom
YpNo:feat/postgres-backend

Conversation

@YpNo

@YpNo YpNo commented Aug 19, 2026

Copy link
Copy Markdown

Closes #605.

Adds PostgreSQL as an alternative to the embedded SQLite database. SQLite stays
the default and the zero-config path. Omitting server.database changes
nothing for an existing install.

Six commits, each reviewable on its own:

  1. fix(auth) : two bugs found while scoping this, unrelated to PostgreSQL.
    Concurrent first logins could leave the instance with no owner (the count
    ran as a statement separate from the insert, so both racers saw two rows and
    neither promoted itself), and nothing ever closed the database handle. The
    regression test fails on the old code.
  2. refactor(db) : createDbClient returns the connection bundled with its
    schema and dialect. auth.ts and hp-agent.ts unpack it once, so every query
    body is unchanged apart from users becoming tables.users.
  3. docs(changelog)
  4. feat(config) : server.database, which also lets the SQLite file live
    somewhere other than data_path.
  5. feat(db) : the PostgreSQL backend, migrations, and the contract tests.
  6. feat(db) : a SQLite -> PostgreSQL copy helper.

One thing to look at first

The plan was for the query layer to be typed against a union of both dialects'
clients. That does not compile: TypeScript cannot call a method on a union of
generic signatures, and every .select() fails with "none of those signatures
are compatible with each other"
. The only union that compiles contains any,
which would erase type checking from every query in the codebase.

So queries stay typed against the SQLite client and the PostgreSQL client is
cast to it once, in createPostgresClient, with the reasoning written at the
definition. The alternative was a repository port with one implementation per
dialect. The large refactor you have said you do not want, for a three-table
schema.

What makes the cast safe is tests/integration/db, which runs the same query
code against both engines: SQLite in memory, and real PostgreSQL 17 under
testcontainers (already a devDependency, already used by the docker and dex
suites). 31 tests, covering the divergences types would not have caught anyway.
Timestamps surfacing as Date from both an integer epoch and a timestamptz,
JSON round-tripping as an object from both text and jsonb, nullable columns,
upserts, ownership transfer.

I also checked that no SQLite-only method (.get(), .all(), .run()) appears
in shared query code, and said so in a comment where the cast lives.

If you would rather have compiler-proven safety than tested safety, say so. But
that means the repository port, and I did not want to spring that on you.

Owner bootstrap under READ COMMITTED

The single-statement guard from commit 1 is atomic under SQLite's serialized
writes. It is not under PostgreSQL: two concurrent first logins update
different rows, take no conflicting row locks, both see no owner, and both
commit. The PostgreSQL schema carries a partial unique index on role = 'owner'
as the real guarantee, with a test asserting it actually rejects a second owner.

It is created unconditionally because PostgreSQL support is new. No existing
database could already hold two owners. SQLite deliberately does not get it:
that migration would fail to apply on any install where the race already fired.

Notes

  • Existing drizzle/* migrations are untouched; PostgreSQL lives in
    drizzle/postgres/.
  • The migrators differ by dialect (SQLite synchronous, PostgreSQL returning a
    promise). A failed migration drains the pool before rethrowing so a failed
    start does not hang the process.
  • Connection strings are redacted wherever they are logged.
  • server.database.password_path follows the existing _path secret convention.
  • Driver is pg.
  • docs/configuration/database.md is new. The database was undocumented.

Testing

  • pnpm run typecheck, pnpm run lint clean
  • pnpm test:unit -> 210 passing
  • pnpm vitest run --project integration:db -> 31 passing against PostgreSQL 17

Not included

docs/NixOS-options.md needs regenerating from nix/options.nix (I added the
server.database submodule there). It is built by nixosOptionsDoc and I do not
have nix available, so I left the generated file alone rather than hand-editing
build output. Happy for you to regenerate, or I can if you tell me the command
you use.

YpNo and others added 6 commits August 19, 2026 21:20
…down

Two independent fixes, neither dependent on the PostgreSQL work in tale#605.

**Concurrent first logins leave the instance with no owner**

`findOrCreateUser` inserted the new user, then counted rows in a separate
statement, and promoted to owner only when the count was exactly 1. Two people
signing in at the same moment interleave at the `await` boundaries, so both can
observe a table that already holds two rows. Neither promotes itself, and the
instance ends up with no owner and nobody able to administer it.

The guard now lives inside the UPDATE, so the check and the write are one
statement. This also removes the `count(*)` projection entirely.

Note the condition tests for the absence of an owner rather than for a single
user row. The two agree on a fresh install, and only the former is decidable in
a single statement.

**The database handle was never closed**

Nothing disposed the database, so a graceful shutdown left the SQLite handle
open with its journal un-finalized. `closeDbClient` is registered first in the
disposer list so it runs last, after everything that might still query during
shutdown.

Adds a regression test that fails on the previous implementation (it produced
zero owners, not two).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for tale#605. No behaviour change, no new dependency, still SQLite only.

`createDbClient` now returns a `HeadplaneDb` — the drizzle client, the tables it
can execute against, its dialect, and a `dispose()`. Callers reach tables through
the bundle instead of importing the schema module directly, so the schema travels
with the connection that can execute it rather than being fixed at import time.
That is the single thing standing between here and a second dialect.

`auth.ts` and `hp-agent.ts` unpack it once (`const { client: db, tables } =
opts.db`), which keeps every query body identical apart from `users` becoming
`tables.users`. No new layer, no indirection at the call sites.

`NodeSQLiteDatabase` no longer appears outside `db/client.server.ts`, so adding
Postgres later touches the client module and the schema, not the query code.

`wrapSqliteClient` is exported so the unit test helper builds in-memory databases
through the same path the application uses, rather than a parallel one.

The dispose logic moves inside the bundle, replacing the standalone
`closeDbClient` added in the previous commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No issue link: both were found while scoping tale#605 rather than reported, and
this branch does not close it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for tale#605, and a small feature on its own: the database can now live
somewhere other than `server.data_path`, which is useful when data_path sits on
a volume chosen for cache-like data.

Omitting the block reproduces the historical location exactly, so upgrading
changes nothing and no existing database moves.

`createDbClient` now takes a resolved `DatabaseConfig` discriminated union
rather than a bare path, so adding a dialect adds a member instead of changing
the signature.

Scoped to `type: "sqlite"` deliberately. Accepting `"postgres"` here while the
client cannot open one would ship config that validates and then fails at
runtime; the union widens in the PR that can actually honour it.

Note `server.database.path` is a plain `string`, not `string.lower` like the
older path fields. That keyword rewrites the value it validates, so a path
containing a capital letter silently resolves somewhere else on a
case-sensitive filesystem. There is a test pinning this. The same issue affects
`data_path` and the other `string.lower` paths today — happy to open a separate
issue for that if you'd like it fixed.

Adds docs/configuration/database.md, covering the move procedure and what is
actually lost if the database is (the answer is roles and sessions, recoverable
but tedious).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the implementation half of tale#605. SQLite stays the default; omitting
`server.database` changes nothing for an existing install.

**A design concession I want to flag before you read the diff.**

The plan was for the query layer to be typed against a union of the two
dialects' clients, so both would be checked by the compiler. That does not
work here. TypeScript cannot call a method on a union of generic signatures —
every `.select()` and `.insert()` fails with "none of those signatures are
compatible with each other". The only union that compiles is one containing
`any`, which would erase type checking from every query in the codebase.

So queries stay typed against the SQLite client and the PostgreSQL client is
cast to it once, in `createPostgresClient`. The alternative was a repository
port with a separate implementation per dialect, which is the large refactor
you have said you do not want, for a three-table schema.

What makes the cast safe is not the type system but `tests/integration/db`,
which runs the same query code against both engines: SQLite in memory and a
real PostgreSQL 17 under testcontainers. 25 tests, covering the divergences
that types would not have caught anyway — timestamps surfacing as `Date` from
both an integer epoch and a timestamptz, JSON round-tripping as an object from
both text and jsonb, nullable columns, upserts, and the owner bootstrap.

The API surface those queries use is identical on both dialects, and I checked
that none of the SQLite-only methods (`.get()`, `.all()`, `.run()`) appear in
shared query code. The cast is documented where it happens.

**Owner bootstrap under READ COMMITTED.** The single-statement guard added
earlier is atomic under SQLite's serialized writes but not under PostgreSQL:
two concurrent first logins update different rows, take no conflicting row
locks, both see no owner, and both commit. The PostgreSQL schema carries a
partial unique index on `role = 'owner'` as the real guarantee, with a test
asserting it bites. It is created unconditionally because PostgreSQL support
is new — no existing database could already hold two owners. The SQLite schema
deliberately does not get it: that migration would fail to apply on any install
where the race has already fired.

Also: the migrator differs by dialect (SQLite synchronous, PostgreSQL returning
a promise), and a failed migration drains the pool before rethrowing so a
failed start does not hang the process. Connection strings are redacted before
they reach the log.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finishes tale#605. Without this, switching `server.database.type` silently starts
you over: every user is recreated on next sign-in with the default role, and
whoever signs in first becomes the owner. The docs previously said as much,
which is a poor answer for anyone with roles worth keeping.

  pnpm exec tsx scripts/db-copy.ts \
    --from /var/lib/headplane/hp_persist.db \
    --to postgres://headplane@10.0.0.5:5432/headplane

Carries users, roles, Headscale links, live sessions and cached node info.
Signed-in users stay signed in, since sessions come across and the cookie
secret is unchanged. Expired sessions are dropped rather than copied.

Three things it refuses to do quietly:

- Copying into a target that already holds users, which would otherwise fail
  partway through on a primary key collision and leave it half-written.
  `--allow-nonempty` opts in.
- Copying a source with more than one owner. PostgreSQL enforces a single owner
  via the partial unique index, and a constraint violation midway is a much
  worse message than naming the offending accounts up front. Reachable on a
  database hand-repaired around the owner-bootstrap race.
- Printing the connection string with its password in it.

`--dry-run` reports what would move without writing.

Kept as a standalone script rather than a subcommand so it adds nothing to the
server bundle or the startup path. It exports `copyDatabase` so the tests drive
it directly; six integration tests run it against real PostgreSQL, and I also
ran the documented command end to end to check the CLI wrapper, the refusal
paths and their exit codes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@YpNo
YpNo requested a review from tale as a code owner August 19, 2026 19:40
@github-actions github-actions Bot added Docs Improvements or additions to documentation Authentication Authentication & Permissions Config Related to Headplane specific configuration labels Aug 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — this is a clean, well-documented feature with strong cross-dialect test coverage. One inline nit and a couple of minor observations below.

Reviewed changes

  • Atomic owner bootstrap (fix(auth))findOrCreateUser now promotes the first user with a single conditional UPDATE guarded by not exists, closing the concurrent-first-login race that could leave an instance ownerless, and HeadplaneDb.dispose() releases the SQLite handle (or drains the pg pool) on shutdown.
  • HeadplaneDb bundle (refactor(db))createDbClient returns { client, tables, dialect, dispose }; auth.ts and hp-agent.ts unpack it once, so query bodies only change userstables.users.
  • server.database config (feat(config)) — new block for sqlite path / postgres url or host/name/user, ssl_mode, max_connections; omitting it preserves the historical <data_path>/hp_persist.db.
  • PostgreSQL backend (feat(db))schema.postgres.ts, migrations under drizzle/postgres/, and a partial unique index enforcing a single owner.
  • SQLite→Postgres copy helper (feat(db))scripts/db-copy.ts with --dry-run / --allow-nonempty, plus contract and copy test suites running against real PostgreSQL 17.

ℹ️ Concurrent first login can surface a 500 on PostgreSQL

The single-owner guarantee is correct: the partial unique index on role = 'owner' makes two owners impossible, and the conditional UPDATE handles the SQLite case. The remaining rough edge is that under a genuinely simultaneous first pair of logins on PostgreSQL, the NOT EXISTS guard evaluates before the other racer has committed, so both UPDATEs attempt to promote — and the index makes the loser throw 23505, which findOrCreateUser does not catch. The result is a one-off failed login for one of the two users, not data corruption; they retry and become a member. This is a once-per-install-lifetime race, so leaving it as-is is defensible, but it is worth a conscious decision.

Technical details
# Losing owner-bootstrap racer errors on PostgreSQL

## Affected sites
- app/server/web/auth.ts — the conditional owner-promotion UPDATE in findOrCreateUser does not tolerate a unique_violation losing the race

## Required outcome
- Decide whether a concurrent-first-login should surface as a 500 (current) or resolve cleanly (e.g. catch 23505 and treat it as "someone else became owner")

## Open questions for the human
- Is failing the losing racer acceptable given the race requires two simultaneous first logins on a brand-new instance?

ℹ️ Nitpicks

  • scripts/db-copy.ts --dry-run still runs pgMigrate on the target and creates the tables before short-circuiting, so the "without writing anything" wording in docs/configuration/database.md is slightly inaccurate — it writes schema, just no rows.

Pullfrog  | Fix all ➔Fix 👍s ➔View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

* be discovered by running against that dialect.
*/
type Exact<A extends B, B extends C, C = A> = A;
export type AssertUserRowsMatch = Exact<typeof postgresSchema.users.$inferSelect, HeadplaneUser>;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AssertUserRowsMatch only pins users. The HeadplaneTables doc comment (lines 33–36) says it pins "the two schemas" to identical row shapes, but authSessions and hostInfo are only guarded by the contract tests — a drift in either (e.g. authSessions.expires_at inferring string instead of Date) compiles clean and fails only at runtime. Consider adding equivalent Exact assertions for AuthSessionRecord and HostInfoRecord (both already exported from schema.ts), or narrowing the doc comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Authentication Authentication & Permissions Config Related to Headplane specific configuration Docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PostgreSQL as an alternative storage backend

1 participant