fix(ingest): stop rolling the EC2 fleet on every merge and right-size hosts by Makisuo · Pull Request #1716 · MapleTechLabs/maple

Makisuo · 2026-10-22T23:13:00Z

Why

AWS spend roughly doubled in October (about $17/day to $33-38/day). Two causes in this PR:

Every merge to main rolled the ingest fleet. Consecutive ingest task definitions (rev 320 vs 321) differ only in COMMIT_SHA; the image tag is identical. A new env value still registers a revision and forces a rollout, and with host networking plus distinctInstance each rollout launches two fresh c7gd hosts, drains the old ones, then scales in. On 2026-10-22 that happened ~15 times. The churn also inflates AWS Config item counts.
US prd ingest is oversized. Over 14 days the two 2 vCPU / 3 GiB tasks peaked at 25% CPU (p99 21%) and 6% memory (~180 MB). The 60% CPU scaling target never fires.

What

deploy-prd-instance.yml resolves INGEST_COMMIT_SHA = last commit touching apps/ingest (the image context), using a blobless full-history checkout. apps/ingest/alchemy.run.ts stamps that as the gateway's COMMIT_SHA, falling back to COMMIT_SHA / GITHUB_SHA (previews, local). A merge that does not touch apps/ingest leaves the task definition, and the hosts, alone.
Ingest leaves PRD_LOCKSTEP_REVISION_SERVICES: it no longer reports the deployed commit. The "Prod revision skew" rule (2a6e9529…, currently disabled) already had its SQL updated to drop ingest.
US prd ingest: c7gd.large + 2048/3072 → c7gd.medium + 1024/1536, the size EU runs. Scaling stays 2-6 at 60% CPU, so a burst adds hosts. The 48 GiB WAL cap already fits the medium's 59 GB NVMe.
Preview ingest (SMALL_INGEST) runs on a t4g.small, with a 4 GiB WAL in a directory on the 30 GB gp3 root volume. c7gd was the floor only because the WAL needs local NVMe. WAL placement and cap now come from the instance type (WAL_STORAGE in apps/ingest/alchemy.run.ts). A c7gd host still mounts its instance store before it joins the cluster, so a host whose NVMe fails never gets a task.
AMI patching rides ingest rollouts, which still happen on most days (apps/ingest sees 1-4 commits a day).

Expected effect

Roughly 1.1 fewer extra host-hours per merge and fewer Config items.
US ingest hosts at about half the hourly price: ~$65/month.
Each live full preview: ingest host ~$12/month instead of ~$66/month on c7gd.large (approximate list prices).

Verification

vitest run src/env.test.ts src/profile.test.ts in packages/infra: 31 passed.
tsrs -p tsconfig.alchemy.json and packages/infra typecheck: 0 errors. oxfmt clean.
Both boot scripts rendered and pass bash -n; the current ECS AL2023 arm64 AMI ships a 30 GB gp3 root.
Not yet run on a real t4g host: the preview label on this PR would boot one (adds ~$3/day while labelled).
After merge, check that the deploy rolls ingest once onto c7gd.medium hosts. After that, a merge that does not touch apps/ingest should leave the task definition revision unchanged (aws ecs describe-services --cluster maple-ingest --services maple-ingest-ec2).

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

… hosts Every prd deploy stamped the merged commit into ingest's task definition as COMMIT_SHA. The image is content-hashed and usually unchanged, but the new env value still registered a revision and forced a rollout, and with host networking each rollout launches two fresh c7gd hosts and drains the old ones. That was ~15 rollouts a day. - Ingest now reports INGEST_COMMIT_SHA, the last commit touching apps/ingest, resolved in deploy-prd-instance.yml (blobless full-history checkout). It leaves PRD_LOCKSTEP_REVISION_SERVICES; the skew rule's SQL is updated. - US prd ingest moves to c7gd.medium with 1 vCPU / 1.5 GiB tasks, as EU runs. Over 14 days the c7gd.large tasks peaked at 25% CPU and 6% memory. - Preview ingest uses the same smaller size.

maple-review-bot · 2026-10-22T23:13:10Z

Maple review

🟢 Confidence 9/10 · safe to merge
Deploy-config change with no new defects found; the ingest size now equals EU's already-running configuration, but the revision stamping and sizing have no test of their own.
_{no findings · tests partial −1 · risk medium}

Ingest now stamps the last commit touching apps/ingest instead of the deployed commit, so a merge elsewhere no longer rolls the EC2 fleet, and US/preview ingest drops to the EU host and task size. Safe to merge.

deploy-prd-instance.yml resolves and exports INGEST_COMMIT_SHA from git log -- apps/ingest
createMapleIngest stamps that as the gateway's COMMIT_SHA, falling back to COMMIT_SHA/GITHUB_SHA
US_PRD and SMALL_INGEST move to c7gd.medium with a 1024/1536 task
ingest leaves PRD_LOCKSTEP_REVISION_SERVICES

Before merge

Env var · Set INGEST_COMMIT_SHA in every environment, or confirm its default · apps/ingest/alchemy.run.ts:336
Infra · Check every environment has what this config now expects · apps/ingest/alchemy.run.ts
Manual · Confirm the "Prod revision skew" rule (2a6e9529-5f73-4478-9fa0-432904ff15c8) no longer selects ingest before it is re-enabled; its SQL lives in the Maple database, not this repo

What was checked

optionalPlain trims and omits blank keys (packages/infra/src/env.ts:55), so a failed git log falls through to COMMIT_SHA/GITHUB_SHA instead of stamping an empty revision
EU already runs c7gd.medium with a 1024/1536 task (packages/infra/src/profile.ts:130), so the new US and preview size is a configuration already serving production traffic
The deploy image context is apps/ingest alone (apps/ingest/alchemy.run.ts:530, Dockerfile.prebuilt), which is the path the workflow hashes, so the stamp tracks everything the image is built from

_{11ef86c · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

coderabbitai · 2026-10-22T23:13:11Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 5eda7a91-0f35-4fd5-aa3f-f9893edfe73f

📥 Commits

Reviewing files that changed from the base of the PR and between b196da1 and 7dae4b9.

📒 Files selected for processing: 6

.github/workflows/deploy-prd-instance.yml
apps/ingest/alchemy.run.ts
docs/infra.md
packages/infra/src/env.test.ts
packages/infra/src/env.ts
packages/infra/src/profile.ts

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…oot volume c7gd.medium was the floor only because the WAL needs local NVMe. A preview carries almost no traffic, so it now runs a burstable t4g.small with a 4 GiB WAL in a root-volume directory. WAL placement and cap come from the instance type (WAL_STORAGE), so a c7gd host still refuses to join the cluster when its instance store does not mount.

maple-review-bot · 2026-10-22T23:17:57Z

Maple review

🟢 Confidence 10/10 · safe to merge
Declarative infra change; the stamped revision moves with the apps/ingest image context, and env.test.ts pins the new lockstep list.
_{no findings · tests covered · risk medium}

Stops the ingest EC2 fleet from rolling on merges that leave apps/ingest alone by stamping that commit instead of the deployed one, drops ingest from the revision-lockstep list, and halves the prd ingest hosts (c7gd.medium, previews on a t4g.small). Safe to merge.

INGEST_COMMIT_SHA (last commit touching apps/ingest) becomes the gateway's COMMIT_SHA
Ingest leaves PRD_LOCKSTEP_REVISION_SERVICES
US prd ingest moves to c7gd.medium with 1024/1536 tasks
WAL_STORAGE gives a t4g.small a 4 GiB WAL on the root volume

Before merge

Env var · Set INGEST_COMMIT_SHA in every environment, or confirm its default · apps/ingest/alchemy.run.ts:350
Infra · Check every environment has what this config now expects · apps/ingest/alchemy.run.ts
Manual · Confirm the 2a6e9529-5f73-4478-9fa0-432904ff15c8 "Prod revision skew" rule's SQL no longer expects ingest before that rule is re-enabled

What was checked

git log -1 -- apps/ingest needs only trees, so it works in the filter: blob:none full-history checkout
The image context is apps/ingest (alchemy.run.ts:544), so the stamped revision changes exactly when the gateway image does
c7gd.medium (1 vCPU / 2 GiB) holds the 1024/1536 task alone; on c7gd.large the collector already needed its own host

_{7dae4b9 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

maple-review-bot · 2026-10-22T23:24:27Z

Merged with 3 steps from "Before merge" still open. If they are done, tick them on the review comment; if not, now is the time.

Env var · Set INGEST_COMMIT_SHA in every environment, or confirm its default
Infra · Check every environment has what this config now expects
Manual · Confirm the 2a6e9529-5f73-4478-9fa0-432904ff15c8 "Prod revision skew" rule's SQL no longer expects ingest before that rule is re-enabled

Makisuo merged commit f47b110 into main Oct 22, 2026
31 checks passed

Makisuo deleted the fix/aws-ingest-cost branch October 22, 2026 23:24

Navigation Menu

Repository navigation

fix(ingest): stop rolling the EC2 fleet on every merge and right-size hosts - #1716

fix(ingest): stop rolling the EC2 fleet on every merge and right-size hosts#1716
Makisuo merged 2 commits into
mainfrom
fix/aws-ingest-cost

Makisuo commented Oct 22, 2026 •

edited

Loading

Uh oh!

maple-review-bot Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

Review limit reached

Uh oh!

maple-review-bot Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

maple-review-bot Bot commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Makisuo commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Expected effect

Verification

Uh oh!

maple-review-bot Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Maple review

Before merge

Uh oh!

coderabbitai Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

maple-review-bot Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Maple review

Before merge

Uh oh!

Uh oh!

maple-review-bot Bot commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Makisuo commented Oct 22, 2026 •

edited

Loading

maple-review-bot Bot commented Oct 22, 2026 •

edited

Loading

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

maple-review-bot Bot commented Oct 22, 2026 •

edited

Loading