perf(server): stop the device heartbeat from blocking HOT updates

[otavio]

Merge order. #6888 now lands ahead of this PR and takes migration 019, so the three
migrations here are 020–022. #6888 also carries migrations_test.go, byte-identical to the
copy in this branch, so it drops out of this diff on rebase and no conflict arises either way
round.

Every device presence heartbeat writes last_seen, and PostgreSQL disqualifies HOT whenever an
indexed column changes. devices_last_seen was a btree on exactly that column, so no heartbeat
could ever be a HOT update — each one rewrote the heap tuple and inserted into every index on
the table.

Measured on a 58,670-device deployment: ~2,244 row updates/second, 107.7 GB of WAL per day, a
heap bloated ~12× past the width of its rows, and autovacuum running continuously without keeping
up.

What this does

Migration	Change
020	drops devices_last_seen and devices_disconnected_at
021	ALTER TABLE devices SET (fillfactor = 85)
022	VACUUM (FULL, ANALYZE) devices

Migrations only — this PR no longer touches docker-compose.postgres.yml. The
wal_compression=lz4 change it used to carry is gone: #6886 adds a
SHELLHUB_POSTGRES_EXTRA_ARGS seam, so a deployment that wants it appends
-c wal_compression=lz4 there rather than every deployment inheriting it. The measurements
below still include it, since that is how it was measured.

Order is load-bearing and follows from the numbering: the drops run before the rewrite so
VACUUM FULL never rebuilds indexes that are about to disappear, and 021 runs before 022
because VACUUM FULL honours fillfactor as it rewrites (the same 58,670 rows rebuild into
2,257 pages at the default and 2,667 at 85).

Verified in production

Each half was applied and measured independently, then reverted:

	baseline	+ wal_compression=lz4 (not in this PR)	+ indexes dropped
HOT ratio	0.0000%	0.0000%	97.2521%
WAL	107.7 GB/day	97.0 GB/day	31.7 GB/day
wal_records/s	12,203	12,204	3,044
postgres CPU (core-s/s)	0.326	0.264	0.205

The win shows up in wal_records, not wal_fpi: a non-HOT update emitted a heap record plus
index-tuple inserts into two btrees, where a HOT update emits one. wal_compression is separate
and additive at ~10% — full-page images are only ~36% of WAL volume here — and it lowered
postgres CPU by 19%, because compressing a page costs less than writing the extra bytes. That
column is context for the deployment that appends the flag, not something this PR delivers: the
HOT win in the third column is independent of it.

The trade

devices_last_seen did serve the device list's default ORDER BY last_seen DESC, which now sorts
over a sequential scan: ~121 ms on the bloated heap, ~24 ms once compacted — which is why
022 is part of this PR rather than a follow-up. No index on last_seen can coexist with HOT
(composite and partial alike), so this is structural: either the write side pays or the read side
does, and one endpoint at ~24 ms is much cheaper than 76 GB/day of WAL.

devices_disconnected_at goes along for free. Nothing filters or orders on it alone, only inside
the unselective online predicate; 0 scans in 66 days of counters.

Why 021 is needed

Production hit 97.25% HOT with fillfactor at the default, because a 12×-bloated heap already
holds all the free space HOT needs — so fillfactor is not required for HOT to work, contrary
to the original issue. It is required to keep HOT working once 022 compacts that space away.
Measured locally from a freshly compacted heap, six full passes over a 58,670-row table:

	HOT ratio	heap growth
default fillfactor	40.4%	4.7×
fillfactor = 85	85.1%	3.0×

Better HOT and a smaller table — reserving 15% up front costs less than letting the heap
rediscover the same slack by bloating. The autovacuum scale-factor knobs proposed alongside it
showed no material effect once fillfactor is set and are not included.

022 is the repo's first non-transactional migration

VACUUM cannot run inside a transaction block, so the file omits the .tx. suffix — and because
the pool runs in pgx QueryExecModeSimpleProtocol, where a multi-statement Exec is itself an
implicit transaction, every statement must sit alone between --bun:split markers. Neither
requirement is visible in the SQL, which is what TestNonTransactionalMigrations checks — the
guard exists for exactly this migration, and #6888 puts it in place ahead of it.

Two further edges are handled rather than ignored:

• Timeouts bound the boot. Migrations run inline in Server.Setup before the listener binds.
  lock_timeout makes 022 fail fast rather than queue behind a long snapshot (a nightly logical
  backup, say). bun marks a migration applied before running it, so a failure there costs one
  crash-restart and leaves the table merely still bloated — degraded, not broken. Recover with
  psql -c 'VACUUM (FULL, ANALYZE) devices;'.
• The GUCs are reset explicitly. A non-transactional migration runs on one pooled bun.Conn
  that returns to the pool without a session reset, so SET lock_timeout would otherwise leak
  into application queries.

VACUUM FULL needs free space for a full copy of the table (~320 MB at the reference scale) and
holds ACCESS EXCLUSIVE for its duration, which is acceptable inside the upgrade's own restart
window. Worth a line in the release notes.

Testing

• TestNonTransactionalMigrations — scans every embedded migration for a statement PostgreSQL
  refuses inside a transaction and asserts it neither carries .tx. nor shares its
  --bun:split chunk. TestNonTransactionalDetection covers the guard itself, including prose
  that merely names a VACUUM.
• Full pg store suite green against a schema built from 001 through 022.
• End-to-end through bun's runner on the dev stack: 021 applies, reloptions becomes
  {fillfactor=85}, 022 compacts, three indexes remain, clean boot.
• Re-verified after the renumber and the rebase onto master: go build clean,
  ./api/store/... green, golangci-lint run ./... reports 0 issues. The same suite is green on
  the combined tree (feat(server): prune sessions past a configurable retention window #6888 merged, then this rebased on top), where migrations run 019–022
  with no gap.

Fixes shellhub-io/team#197.

[github-actions]

Code Review

• Gathered PR context
• Reviewing with specialized agents
• Posting feedback

[otavio]

Follow-up #6884 is stacked on this branch — it drops three unused sessions indexes as migration 020, and is based here rather than on master so the migration numbering cannot collide. Merge this one first.

[github-actions]

Claude finished @otavio's task in 2m 22s —— View job

---

Code Review Complete

Reviewed 8 files across code quality, security, testing, language patterns, and architecture — no new issues found. The code looks good as-is.

A few things I checked closely and found correct:

• TestNonTransactionalMigrations scans every migration, including 001–016 and 018. stripSQLComments runs first, so the VACUUM FULL prose in the comment blocks of 018 and 019 is removed before keyword detection — no false-positive failures. Only the real VACUUM (FULL, ANALYZE) devices; statement in the non-.tx. file is matched.
• Session-GUC lifetime in 019. SET (not SET LOCAL) is the right choice for the non-transactional file, and the explicit RESET chunks prevent the timeouts leaking back into the pooled connection — consistent with the SET LOCAL used in the transactional 017.
• Test compiles — io/fs, strings, and the sqlMigrations embed var are all in scope.
• countStatements / findNonTransactionalStatement handle the split-across-lines and shared-chunk cases correctly, matching the table-driven expectations in TestNonTransactionalDetection.

The migration ordering (drops → fillfactor → vacuum), the lock_timeout/statement_timeout bounding of boot, and the wal_compression=lz4 pin are all well-reasoned and documented inline. Production + local measurements are included in the PR body.

To request another review round, comment /review.

--- · perf/device-heartbeat-hot-updates