fix(operator): strip stage prefix on private API GW integrations

[chaodu-agent]

Summary

Fixes a real 404 bug in oabctl's API Gateway ingress (#1275) found via live
E2E testing against a real deployed Telegram bot.

Private (VPC_LINK) HTTP API integrations forward the stage-prefixed
request path to the backend by default (e.g. /prod/webhook/telegram instead
of /webhook/telegram) — this is documented AWS
behavior
for private integrations. OpenAB's webhook router (axum) matches the exact
configured path, so every request 404s at the container even though the VPC
Link, Cloud Map instance, route, and integration are all wired correctly.

Fix: set overwrite:path=$request.path on the integration's request
parameters when creating it, so the backend receives the stage-stripped path.
Existing integrations created before this fix self-heal on the next
oabctl apply — the code detects the missing parameter and patches it in via
UpdateIntegration, no manual intervention or service recreate needed.

The bug, visually

BEFORE FIX
──────────
┌────────┐  POST /prod/webhook/telegram   ┌───────────────┐
│ Client │───────────────────────────────►│ API Gateway   │
└────────┘                                │ route matches:│
                                           │ "/webhook/    │
                                           │  telegram" ✓  │
                                           └──────┬────────┘
                                                  │ forwards RAW path
                                                  │ "/prod/webhook/telegram"
                                                  │ (stage NOT stripped)
                                                  ▼
                                           ┌───────────────┐
                                           │  VPC Link     │  AVAILABLE ✓
                                           └──────┬────────┘
                                                  ▼
                                           ┌───────────────┐
                                           │  Cloud Map    │  SRV resolved,
                                           │  (SRV lookup) │  instance HEALTHY ✓
                                           └──────┬────────┘
                                                  ▼
                                           ┌───────────────┐
                                           │ openab (axum) │  router only knows
                                           │  container    │  "/webhook/telegram"
                                           └──────┬────────┘  exact-match — MISS
                                                  ▼
┌────────┐        404 (integrationStatus=404)    │
│ Client │◄───────────────────────────────────────┘
└────────┘

AFTER FIX
─────────
┌────────┐  POST /prod/webhook/telegram   ┌───────────────┐
│ Client │───────────────────────────────►│ API Gateway   │
└────────┘                                │ route matches ✓│
                                           │ + integration  │
                                           │ request param: │
                                           │ overwrite:path │
                                           │ =$request.path │
                                           └──────┬────────┘
                                                  │ forwards STRIPPED path
                                                  │ "/webhook/telegram"
                                                  ▼
                                           ┌───────────────┐
                                           │  VPC Link     │──►┌───────────────┐
                                           └───────────────┘   │  Cloud Map    │
                                                                └──────┬────────┘
                                                                       ▼
                                                                ┌───────────────┐
                                                                │ openab (axum) │
                                                                │  MATCH ✓      │
                                                                └──────┬────────┘
┌────────┐              200 OK                                        │
│ Client │◄──────────────────────────────────────────────────────────┘
└────────┘

Everything except the final hop — VPC Link status, Cloud Map SRV resolution,
route/integration config — was already correct and reported healthy by every
aws apigatewayv2/servicediscovery get-* call. Only the one request
parameter mapping on the integration was missing.

How this was found

Ran a full from-scratch oabctl apply against a real Telegram bot deployment.
The webhook URL returned a persistent 404 for 20+ minutes — not a brief
warm-up delay like previously-observed 503s. Isolated the cause by:

1. Confirming the VPC Link was genuinely AVAILABLE, the Cloud Map instance
   was HEALTHY/discoverable, the route/integration/stage were all configured
   exactly as expected via aws apigatewayv2 get-* — everything checked out.
2. Confirming the container itself was healthy and listening on /webhook/telegram
   via CloudWatch logs — no crash, no error.
3. Temporarily enabling API Gateway access logs on the stage. The log line
   showed routeKey=POST /webhook/telegram status=404 integrationStatus=404
   — proving API Gateway matched the route and invoked the integration, but
   the backend returned 404.
4. Cross-referencing openab's router code (src/main.rs) confirmed exact-path
   axum routing, and AWS's private-integration docs confirmed the
   stage-prefix forwarding behavior.
5. Manually applying overwrite:path=$request.path via the AWS CLI fixed it
   immediately (200 OK).

An earlier round of scratch testing with an echo-image backend
(mendhak/http-https-echo) never caught this, because that image echoes back
whatever path it receives — it doesn't do path-based routing, so it can't
expose a path-mismatch bug.

Verification

• cargo build --release, cargo clippy --all-targets -- -D warnings,
  cargo test --release — all clean on macmini (19/19 tests pass, 4 new
  regression tests for the extracted has_stage_path_override helper).
• Live E2E: ran 2 full oabctl apply → test webhook → oabctl delete cycles
  with the fixed binary against a real deployed bot
  (ghcr.io/openabdev/openab:pre-beta-kiro + real Telegram config/secret).
  Both cycles created the integration with the path override baked in
  automatically (no manual patching) and both returned 200 OK end-to-end
  (curl → API Gateway → VPC Link → Cloud Map → Fargate → openab), after the
  expected brief 503 VPC Link warm-up window.
• All test AWS resources (APIs, VPC Links, ECS services, SG rules) were
  cleaned up after each cycle; confirmed via a final sweep.

Testing done

cargo build --release   # clean
cargo clippy --all-targets -- -D warnings   # clean
cargo test --release    # 19 passed; 0 failed

Plus live E2E against real AWS resources in a scratch account/VPC (see
commit message for the reproduction sequence).

[chaodu-agent]

LGTM ✅ — Well-diagnosed fix for a real 404 bug in private VPC_LINK integrations, with self-healing for existing deployments and thorough regression tests. No actionable blockers.

What This PR Does

Fixes a persistent 404 on webhook endpoints caused by API Gateway forwarding the stage-prefixed path (e.g. /prod/webhook/telegram) to the backend on private VPC_LINK integrations. The openab axum router expects the raw path (/webhook/telegram), so every request fails despite VPC Link, Cloud Map, route, and integration all being correctly wired.

How It Works

Sets overwrite:path=$request.path as a request parameter on the integration at creation time, stripping the stage prefix before it reaches the container. Existing integrations created before this fix self-heal on the next oabctl apply — the code detects the missing parameter and patches it via UpdateIntegration (PATCH semantics, merge-not-replace), no manual intervention or recreate needed.

Findings

#	Severity	Finding	Location
1	🟢	Self-heal pattern (detect-and-patch) is architecturally sound — avoids downtime, preserves integration_id stability, respects AWS API rate limits	ingress.rs:750-763
2	🟢	UpdateIntegration confirmed PATCH semantics — only requestParameters is sent, all other fields (connectionId, integrationUri, etc.) are preserved. No clobber risk	AWS API Reference
3	🟢	Idempotent — once patched, subsequent reconciles detect the override and skip the update call	ingress.rs
4	🟢	has_stage_path_override handles all SDK return shapes (None, wrong key, wrong value) without panic	ingress.rs:147-156
5	🟢	overwrite:path=$request.path is the AWS-documented standard parameter mapping for stage-prefix stripping on private integrations	ingress.rs
6	🟢	payload_format_version("1.0") paired with overwrite:path is spec-compliant, no known limitations	ingress.rs:create_integration
7	🟢	4 unit tests cover all meaningful edge cases for the helper function	ingress.rs (tests module)
8	🟢	Comprehensive README documentation explaining the "why" with AWS doc links and self-heal behavior	operator/README.md

Team Review — Concerns Raised & Dispositions

The review team raised 9 potential concerns (all 🟡). After cross-referencing AWS documentation and analyzing the codebase, none are actionable blockers for this PR:

Concern	Disposition
has_stage_path_override doesn't handle trailing whitespace	AWS SDK returns exact values from the service — whitespace in the value would indicate external tampering, and re-patching is the correct self-heal response
Concurrent apply could race on self-heal	oabctl is a CLI tool (not a daemon); concurrent apply of the same bot is not a supported usage pattern. Even if it occurs, both patches are identical (idempotent)
Fail-fast on throttle — suggest retry/backoff	Intentional design: continuing apply with a broken integration would leave inconsistent state. Retry is a valid follow-up enhancement but not a correctness bug
UpdateIntegration merge vs replace	Confirmed: AWS uses PATCH semantics. Only fields provided are updated; connectionId/integrationUri are preserved. Not a blocker
Three exit paths in ensure_integration	~30 lines total, clearly separated. Extracting a sub-function is taste, not correctness
Self-heal may break integrations using stage-prefixed paths	oabctl exclusively manages these integrations; openab's router always does exact-path matching. No legitimate use case for stage-prefixed paths exists
$request.path traversal risk	Pre-existing AWS behavior, not introduced by this PR
No automated integration test	Live E2E was performed manually (2 full cycles). LocalStack/CI integration tests are a valid follow-up but out of scope for a bug fix
No cargo audit/cargo deny in CI	Infrastructure improvement, not related to this fix

What's Good (🟢)

• Zero-downtime self-heal: uses UpdateIntegration (in-place PATCH) instead of delete+recreate, preserving the integration ID and avoiding any service interruption window.
• Thorough root-cause analysis: 5-step debugging process from VPC Link status through API Gateway access logs to AWS documentation.
• Fail-fast error handling: if the patch call fails, the error propagates immediately — no broken state is left running.
• Idempotent reconciliation: the detect→patch→skip cycle ensures repeated apply calls don't generate redundant AWS API calls.
• Type-safe SDK usage: request_parameters() returns Option<&HashMap<String, String>> from the AWS Rust SDK, and the helper handles all states correctly.
• Scope isolation: the override only activates for private VPC_LINK integrations — public HTTP/HTTPS integrations are unaffected.
• Verified end-to-end: 2 full apply→test→delete cycles against real AWS resources with a live Telegram bot deployment.

Baseline Check

• PR opened: 2026-07-03
• Main already has: Full API Gateway ingress pipeline (VPC Link, Cloud Map, integration, route, stage) but no request_parameters or path override logic
• Net-new value: Stage-prefix stripping on private integrations (both create and self-heal paths), 4 regression tests, and documentation