Skip to content

FE-1625: Make the merged Brunch image deployable on ECS - #9573

Merged
TimDiekmann merged 2 commits into
mainfrom
t/fe-1625-make-the-merged-brunch-image-deployable-on-ecs
Sep 7, 2026
Merged

TimDiekmann merged 2 commits into
mainfrom
t/fe-1625-make-the-merged-brunch-image-deployable-on-ecs

Conversation

@TimDiekmann

Copy link
Copy Markdown
Member

🌟 What is the purpose of this PR?

#9487 landed the Postgres and telemetry code for Brunch, but the image it produces cannot start on ECS: it requires an RDS CA bundle that nothing provides, and its telemetry setup duplicates and diverges from the one hash-api and the workers share. This PR makes the image deployable for the staging rollout and aligns it with the rest of the backend.

🔗 Related links

🚫 Blocked by

  • None

🔍 What does this change?

  • Dockerfile: adds the AWS RDS global CA bundle to the image and sets BRUNCH_POSTGRES_TLS_CA_PATH to it, the same way the Kratos and Hydra images do. Infrastructure only overrides the path for another CA.
  • Telemetry: src/telemetry.ts now uses registerOpenTelemetry, createHttpInstrumentation and createUndiciInstrumentation from @local/hash-backend-utils/opentelemetry and keeps only the Flue wrapper. Brunch gains the peer.service mapping (Anthropic shows up in the service graph), METHOD /path span names and bounded provider shutdown; twelve direct @opentelemetry/* dependencies go away. Failure spans carry the error code (ECONNREFUSED, a SQLSTATE) as error.type instead of the constructor name.
  • Shared setup: registerOpenTelemetry installs a DiagConsoleLogger at WARN, so exporter failures reach stderr in every Node service instead of the SDK's no-op handler.
  • Postgres: close() drains the pool before telemetry shuts down; query_timeout and statement_timeout bound queries on dead connections; idle pool errors are logged to stderr by code; the CA read error keeps its cause and code; a failed rollback releases the client before the reference is cleared.
  • Startup: db.ts wraps configuration and runner creation in one failure path that records the span, flushes telemetry and rethrows.
  • Probe and smoke: the IAM probe prints only measured values; the deployment smoke times out its requests and requires streamed assistant text; the container smoke's refusal run has a timeout instead of hanging when the image starts without database configuration.
  • Docs: the README describes the actual shutdown sequence and startup requirements and marks the mutually exclusive variables; the handoff document states the real publication targets; the stale turn-timing.test.ts allowlist entry is gone.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • affected the execution graph, and the turbo.json's have been updated to reflect this
    • no turbo.json change was needed; the new workspace dependency is reflected in the regenerated docs/task-dependencies.json

⚠️ Known issues

  • Store selection is still keyed on NODE_ENV: any value other than production silently selects SQLite.
  • @local/hash-backend-utils brings its full dependency set into the image (Temporal, googleapis, Linear SDK). Accepted for now.
  • test:docker still does not run in CI (SRE-1032), so the real Postgres, TLS and OTLP paths have no automated coverage.

🐾 Next steps

  • Extract the OpenTelemetry module out of @local/hash-backend-utils into a lean package shared by hash-api, the workers and Brunch.
  • Infrastructure (SRE-1013): set HASH_OTLP_ENDPOINT and the BRUNCH_POSTGRES_* variables, and give the task a stop timeout above 60 seconds.

🛡 What tests cover this?

  • Unit tests: telemetry registration order, environment trimming and the error-code attribute; Postgres pool timeouts, default pool-error logging, sequential close and aggregated close failures; smoke validation's text requirement.
  • Not covered automatically: the image build with the CA bundle (the Build brunch-agent jobs on this PR exercise it) and any real Postgres or OTLP path (see SRE-1032).

Bake the AWS RDS global CA bundle into the image and point
BRUNCH_POSTGRES_TLS_CA_PATH at it, replace the app-owned OpenTelemetry
setup with @local/hash-backend-utils/opentelemetry, drain the pool before
telemetry shuts down, bound queries, keep error causes and log idle pool
errors, add an OpenTelemetry diagnostic logger to the shared setup, and
correct the README, handoff, probe and smoke checks.
@TimDiekmann TimDiekmann self-assigned this Sep 7, 2026
@vercel

vercel Bot commented Sep 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

4 Skipped Deployments
Project Deployment Actions Updated
hash Ignored Ignored Preview Sep 7, 2026 7:21pm UTC
hashdotdesign-tokens Ignored Ignored Preview Sep 7, 2026 7:21pm UTC
petrinaut Skipped Skipped Sep 7, 2026 7:21pm UTC
petrinaut-docs Skipped Skipped Sep 7, 2026 7:21pm UTC

Request Review

@github-actions github-actions Bot added area/deps Relates to third-party dependencies (area) area/apps > hash* Affects HASH (a `hash-*` app) area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team type/eng > backend Owned by the @backend team area/tests New or updated tests area/apps labels Sep 7, 2026
@TimDiekmann
TimDiekmann marked this pull request as ready for review September 7, 2026 19:06
@TimDiekmann
TimDiekmann requested a review from a team as a code owner September 7, 2026 19:06
Copilot AI balanced review requested due to automatic review settings September 7, 2026 19:06
@cursor

cursor Bot commented Sep 7, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches production container build, RDS TLS, Postgres pool lifecycle, and shared OpenTelemetry registration used across backend services; misconfiguration could affect staging rollout observability or shutdown.

Overview
Makes the Brunch production image start on ECS by bundling the AWS RDS global CA and defaulting BRUNCH_POSTGRES_TLS_CA_PATH, and expands the Docker build to match API/worker toolchains (Rust/Cargo stubs, wasm/openapi deps) so @local/hash-backend-utils can build in the image.

Telemetry drops a dozen direct @opentelemetry/* packages in favor of registerOpenTelemetry from shared backend utils, with Flue still content-free; shutdown drains Postgres before exporter teardown, and operational failure spans use error.type codes instead of messages. Shared registerOpenTelemetry now logs export problems at WARN on stderr via DiagConsoleLogger.

Postgres adds query/statement timeouts, sequential pool-then-telemetry close(), stderr logging for idle pool errors by code, and safer rollback client release. Startup in db.ts flushes telemetry after config failures. Smokes/probes add 2-minute fetch timeouts, require streamed assistant text, trim IAM probe output, and timeout container runs that would hang without DB config. Docs and handoff notes reflect OTLP/collector behavior, env mutual exclusion, and ECR+GHCR publication.

Reviewed by Cursor Bugbot for commit 3303de9. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 65.93%. Comparing base (fced1b8) to head (81c8568).

Files with missing lines Patch % Lines
...ibs/@local/hash-backend-utils/src/opentelemetry.ts 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9573      +/-   ##
==========================================
- Coverage   65.93%   65.93%   -0.01%     
==========================================
  Files        1885     1885              
  Lines      198175   198176       +1     
  Branches     8230     8230              
==========================================
  Hits       130673   130673              
- Misses      65972    65973       +1     
  Partials     1530     1530              
Flag Coverage Δ
apps.hash-ai-worker-ts 1.99% <ø> (ø)
apps.hash-api 15.41% <ø> (ø)
local.hash-backend-utils 3.27% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The deployment smoke drops bearer authentication and can falsely accept an empty text-delta event.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Makes the Brunch image deployable on ECS by adding RDS trust configuration, shared telemetry, bounded database behavior, and stronger deployment probes.

Changes:

  • Bundles the AWS RDS CA and hardens PostgreSQL startup/shutdown.
  • Migrates Brunch to shared OpenTelemetry infrastructure.
  • Adds request timeouts, stream validation, documentation, and tests.
File summaries
File Description
yarn.lock Updates workspace dependency resolution.
libs/@local/hash-backend-utils/src/opentelemetry.ts Enables warning-level OTel diagnostics.
libs/@hashintel/brunch-agent/docs/evidence/implementations/mission-8-deployment-handoff.md Corrects publication targets.
apps/brunch-agent/test/telemetry.test.ts Tests shared telemetry integration.
apps/brunch-agent/test/postgres.test.ts Tests timeouts and shutdown behavior.
apps/brunch-agent/test/deployment-smoke-validation.test.ts Tests assistant-text validation.
apps/brunch-agent/test/container-smoke.ts Bounds the refusal smoke runtime.
apps/brunch-agent/test/architecture/boundaries.integration.ts Updates the test allowlist.
apps/brunch-agent/src/telemetry.ts Adopts shared OTel utilities.
apps/brunch-agent/src/telemetry-bootstrap.ts Documents telemetry lifecycle.
apps/brunch-agent/src/rds-iam-probe.ts Limits probe output.
apps/brunch-agent/src/postgres.ts Adds query bounds and ordered shutdown.
apps/brunch-agent/src/deployment-smoke.ts Adds request timeouts.
apps/brunch-agent/src/deployment-smoke-validation.ts Requires a text-delta event.
apps/brunch-agent/src/db.ts Unifies startup failure cleanup.
apps/brunch-agent/README.md Documents deployment requirements.
apps/brunch-agent/package.json Replaces direct OTel dependencies.
apps/brunch-agent/docs/task-dependencies.json Records the new workspace dependency.
apps/brunch-agent/docker/Dockerfile Bundles and configures the RDS CA.
Review details
  • Files reviewed: 18/19 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/brunch-agent/src/deployment-smoke-validation.ts Outdated
Comment thread apps/brunch-agent/src/deployment-smoke.ts
The shared backend utilities pull the type-system (wasm-pack) and graph
client (openapi-generator) builds into Brunch's build graph, which the
Node-only builder could not run. Mirror the API and worker images: prune
with Cargo workspace stubs, install the Rust toolchain, and build with
the same tool set and cache mounts.

The deployment smoke now requires a text delta with non-whitespace
content, so an empty text event cannot pass it.
@vercel
vercel Bot temporarily deployed to Preview – petrinaut-docs September 7, 2026 19:20 Inactive
@vercel
vercel Bot temporarily deployed to Preview – petrinaut September 7, 2026 19:20 Inactive
@TimDiekmann
TimDiekmann added this pull request to the merge queue Sep 7, 2026
Merged via the queue into main with commit 117c2da Sep 7, 2026
71 checks passed
@TimDiekmann
TimDiekmann deleted the t/fe-1625-make-the-merged-brunch-image-deployable-on-ecs branch September 7, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash* Affects HASH (a `hash-*` app) area/apps area/deps Relates to third-party dependencies (area) area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

3 participants