FE-1569: Add durable Brunch storage and telemetry by lunelson · Pull Request #9487 · hashintel/hash

lunelson · 2026-09-01T15:26:47Z

🌟 What is the purpose of this PR?

Add the runtime behavior needed to take the containerized Brunch service to the application-to-infrastructure deployment boundary: durable Flue Postgres state, content-free operational telemetry, and explicit diagnostics for restricted deployment and recovery.

This PR does not claim that Brunch is deployed. The downstack container PR publishes to ECR and GHCR, but no Brunch ECS service, RDS target, hosted collector, restricted ingress, or deployment owner is confirmed, so Mission 8 remains open at its required handoff boundary.

🔗 Related links

FE-1569: Containerize and safely deploy Brunch on HASH infrastructure (internal)
Container build and publication PR
FE-1441: originating deployment request, now a duplicate of FE-1569 (internal)

🚫 Blocked by

Infrastructure owner confirms and provisions the Brunch ECS, RDS/IAM, collector, restricted ingress, and deployment-owner targets.
The remote Mission 8 proof matrix passes against one immutable image digest.

🔍 What does this change?

Replaces deployed SQLite with fail-closed @flue/postgres persistence using dedicated fields, verified TLS, RDS IAM tokens per physical connection, and a runtime-password fallback.
Adds content-free Flue and Node OpenTelemetry export over OTLP/gRPC with graceful lifecycle flush and operational database-failure spans.
Adds RDS IAM and restricted deployment probes plus a disposable TLS Postgres/OTel container smoke.
Documents runtime configuration, ingress restrictions, local proof, GHCR publication, and the outstanding infrastructure handoff.

🏗️ Agent notes

The branch authority is libs/@hashintel/brunch-agent/MISSION.md. It records Mission 8 as live but stopped at the explicit application-to-infrastructure handoff boundary.

Locally established:

fail-closed production configuration and Flue Postgres migration before listen;
IAM-token callback refresh, transaction pinning/rollback, TLS verification, and static-password fallback;
content-free OTLP traces, logs, and metrics with graceful shutdown;
disposable TLS Postgres and OTel collector integration smoke.

Still open:

real task-role/RDS IAM connectivity;
restricted streamed Anthropic/tool execution;
in-place and cross-host replacement recovery;
bounded provider/database/stream failure inspection in the hosted telemetry backend;
singleton replacement, rollback, and owner acceptance.

The downstack PR registers ECR and GHCR publication with no ECS target. Infrastructure provisioning and remote proof remain outside this local application artifact.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

affected the execution graph, and the turbo.json files have been updated to reflect this

⚠️ Known issues

Brunch is not deployed; the deploy catalog publishes to ECR and GHCR but has no ECS target.
The local Docker smoke uses the runtime-password path; the real task-role RDS IAM probe remains unrun.
/health proves process liveness only. Database connection and migration gate startup before the server listens.
Public identity, authorization, rate/spend controls, retention/deletion policy, backup restoration, and multi-replica ownership remain outside this restricted singleton threshold.

🐾 Next steps

Provision and record the ECS, RDS/IAM, secret, collector, ingress, health, timeout, drain, and ownership contract from the deployment handoff.
Run the remote proof matrix with one immutable digest and update this PR, Mission 8, and FE-1569 with observed results.

🛡 What tests cover this?

80 Vitest unit tests covering configuration, IAM token refresh, Postgres transactions, health, telemetry lifecycle, artifact wiring, and existing Brunch behavior.
TypeScript typecheck, application build, and Oxlint with eight pre-existing warnings and no errors.
The previously completed Docker integration smoke covers TLS Postgres migration/startup, fail-closed configuration, OTel export, non-root identity, packaged resources, repository immutability, health, and graceful shutdown.

❓ How to test this?

Run yarn workspace @apps/brunch-agent lint:tsc.
Run yarn workspace @apps/brunch-agent test:unit.
Run yarn workspace @apps/brunch-agent build.
Run turbo run test:docker --filter @apps/brunch-agent with Docker available.
After infrastructure exists, follow apps/brunch-agent/README.md and the Mission 8 deployment handoff for the IAM and remote smoke commands.

📹 Demo

No visual UI change. The proof artifact is the production container and its deployment diagnostics.

vercel · 2026-09-01T15:26:54Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 7, 2026 4:52pm UTC
hashdotdesign-tokens	Ready	Preview	Sep 7, 2026 4:52pm UTC
petrinaut	Ready	Preview	Sep 7, 2026 4:52pm UTC
petrinaut-docs	Ready	Preview	Sep 7, 2026 4:52pm UTC

codspeed-hq · 2026-09-01T15:36:40Z

Merging this PR will not alter performance

⚠️

6 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

✅ 98 untouched benchmarks

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚠️	`as_constant`	< 1 ns	< 1 ns	N/A
⚠️	`constant_equal`	< 1 ns	< 1 ns	N/A
⚠️	`constant_not_equal`	< 1 ns	< 1 ns	N/A
⚠️	`access`	< 1 ns	< 1 ns	N/A
⚠️	`runtime_equal`	< 1 ns	< 1 ns	N/A
⚠️	`runtime_not_equal`	< 1 ns	< 1 ns	N/A

_{Comparing ln/fe-1569-brunch-agent-deployment (16054f8) with main (76497be)}

lunelson · 2026-09-01T15:41:44Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

TimDiekmann

It would be good if we could isolate the Dockerfile changes into an own PR. The changes to .github/workflows/deploy.yml will make sure that the build is covered in CI and will upload it on merge to main. I will make sure that a ECR is set up where we can upload the image to and extend the definition with that.

Copilot

🟡 Changes recommended

Moderate issues remain in recovery validation, database timeouts, and Flue telemetry export.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds durable PostgreSQL persistence, RDS IAM support, operational telemetry, and deployment diagnostics for the Brunch service.

Changes:

Adds fail-closed PostgreSQL configuration with TLS and IAM/password authentication.
Adds OTLP telemetry lifecycle and database-failure reporting.
Adds deployment probes, integration tests, and infrastructure handoff documentation.

File summaries

File	Review
`yarn.lock`	Locks new database, AWS, and telemetry dependencies.
`libs/@hashintel/brunch-agent/MISSION.next.md`	Updates deployment backlog and planning context.
`libs/@hashintel/brunch-agent/MISSION.md`	Defines the restricted-deployment mission.
`libs/@hashintel/brunch-agent/docs/evidence/implementations/mission-8-deployment-handoff.md`	Records proof status and infrastructure handoff.
`apps/brunch-agent/turbo.json`	Adds telemetry environment and integration-test wiring.
`apps/brunch-agent/test/telemetry.test.ts`	Tests telemetry configuration and disposal.
`apps/brunch-agent/test/postgres.test.ts`	Tests pooling, IAM tokens, transactions, and shutdown.
`apps/brunch-agent/test/database-config.test.ts`	Tests production database validation.
`apps/brunch-agent/test/container-smoke.integration.ts`	Exercises the production container with PostgreSQL and OTel.
`apps/brunch-agent/test/build-artifact.test.ts`	Verifies persistence and telemetry bundle wiring.
`apps/brunch-agent/test/architecture/boundaries.integration.ts`	Registers the telemetry architecture boundary.
`apps/brunch-agent/src/telemetry.ts`	Configures OTLP providers and failure spans. Moderate (1 vote): Flue metrics use the unregistered global meter and are not exported. Moderate (1 vote): Flue also receives no logger, dropping its logs as well as metrics.
`apps/brunch-agent/src/telemetry-bootstrap.ts`	Installs telemetry before application initialization.
`apps/brunch-agent/src/rds-iam-probe.ts`	Adds the two-connection IAM diagnostic.
`apps/brunch-agent/src/postgres.ts`	Implements the Flue PostgreSQL runner. Moderate (1 vote): No connection timeout is configured, allowing database operations and diagnostics to wait indefinitely.
`apps/brunch-agent/src/deployment-smoke.ts`	Adds remote turn and history diagnostics. Moderate (2 votes): Recovery can report success for missing/malformed history, and streamed turns do not require a successful finish.
`apps/brunch-agent/src/db.ts`	Selects PostgreSQL in production and SQLite locally.
`apps/brunch-agent/src/database-config.ts`	Defines fail-closed production database configuration.
`apps/brunch-agent/src/app.ts`	Bootstraps telemetry before route setup.
`apps/brunch-agent/README.md`	Documents runtime and deployment requirements.
`apps/brunch-agent/package.json`	Adds dependencies and operational scripts.
`apps/brunch-agent/docs/task-dependencies.json`	Documents the new task dependencies.

Review details

Suppressed comments (4)

apps/brunch-agent/src/postgres.ts:102

A pg pool emits an error event when an idle connection is broken; without a listener, Node treats that event as unhandled and terminates the process. A routine RDS restart or network partition can therefore crash Brunch instead of producing the operational database-failure telemetry added by this PR. Attach a pool error listener and report it through reportDatabaseFailure.

export const createPostgresPool = (
  config: PostgresDatabaseConfig,
  options?: ConnectionOptions,
): Pool => new Pool(createPostgresPoolConfig(config, options));

apps/brunch-agent/src/postgres.ts:200

If one pool.connect() resolves and the other rejects, the Promise.all assignment never completes, so clients remains empty and the successful checked-out client is omitted from the finally cleanup. The failed IAM probe then leaks that client and can leave pool.end() waiting indefinitely. Acquire while appending each client (the first remains checked out, so the second is still guaranteed to be a distinct physical connection).

  let clients: PoolClient[] = [];
  try {
    clients = await Promise.all([pool.connect(), pool.connect()]);

apps/brunch-agent/src/postgres.ts:128

A transaction that fails while acquiring a pooled connection bypasses the try block, so no database_operation failure span is recorded. This is the exact path taken when Postgres is unavailable during Flue startup/migration, contradicting the operational database-failure telemetry contract. Report and rethrow acquisition failures before entering the transaction body.

      const client = await pool.connect();

apps/brunch-agent/src/postgres.ts:156

Both expressions passed to Promise.allSettled are invoked immediately, so telemetry shutdown begins before pool.end() has completed. The unit test only observes synchronous mock side effects and therefore does not enforce the documented “close Postgres, then flush OTel” lifecycle; with a real pool, final database activity can race provider shutdown. Await the Postgres result first, then invoke the telemetry closer while still collecting both failures.

      const results = await Promise.allSettled([pool.end(), afterClose?.()]);

Files reviewed: 21/22 changed files
Comments generated: 4
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

lunelson · 2026-09-04T13:53:53Z

@TimDiekmann according to my agent, CI is currently blocked here by an unrelated pruning issue:

the workflow initially prunes out @rust/hash-repo-chores
the Brunch integration test subsequently performs a nested Docker build
that build's turbo prune evaluates turbo.json and encounters the missing @rust/hash-repo-chores#build reference

Failing job: https://github.com/hashintel/hash/actions/runs/33872870612/job/101023099113

FWIW, agent thinks the integration workflow likely needs to retain @rust/hash-repo-chores, as the unit workflow already does. Rerunning the job alone will not fix it... Thoughts?

cursor