Instrument and provision API v1 observability

[anth-volk]

Fixes #3847

Summary

• configure one API-owned policyengine-observability runtime for Flask and ASGI request handling
• sample 100% of traces in API v1
• assign request_id to every HTTP request while assigning observability_id only to accepted household calculations and society reports
• validate an incoming X-PolicyEngine-Observability-Id as a candidate, then bind it only when a calculation starts
• acquire annual simulation submission ownership before creating the report identifier
• persist the identifier with annual and budget-window report state, and restore that stored value during later polling
• keep health, metadata, invalid, and legacy null-state requests free of fabricated calculation identifiers
• forward the bound identifier to the simulation entry service in the HTTP header
• keep simulation JSON payloads and API v1 execution data classes free of observability identifiers
• keep submission_claim_id as separate cache-ownership metadata
• define household, annual economy, and budget-window span names in one stage registry
• accept explicitly supplied safe scalar attributes in local logs and spans without maintaining an exhaustive API-owned list
• transport only observability_id across asynchronous boundaries and retain separate bounded metric labels
• emit structured JSON to standard output for Cloud Run collection while configuring logs and OpenTelemetry independently
• keep runtime telemetry failures from altering HTTP responses or calculation results
• document the identifier registry, request lifecycle, simulation transport, persistence, trace boundaries, attribute handling, and failure behavior for AI tools and operators
• keep the active API v1 collector image configuration in this repository and document the separately provisioned GCP resources

Deployment dependencies

1. Preserve asynchronous observability context policyengine-observability#32 is merged, version 3.0.1 is published, and this PR requires that version.
2. Add Stage 12 observability identifier schema #3852 is merged; its nullable Stage 12 observability_id migration must be applied.
3. Deploy Configure and instrument simulation observability policyengine-sim-api#686 and stop its older Modal workers.
4. Deploy this PR after the simulation services are running the new function signatures. Roll back this PR before rolling back Configure and instrument simulation observability policyengine-sim-api#686.

Verification

• Ruff formatting and lint checks passed for the changed Python files.
• 315 focused tests passed across request context, ASGI and Flask middleware, the simulation client, annual and budget-window economy workflows, cache behavior, and household calculation routes.
• The focused runtime configuration test passed with the published policyengine-observability 3.0.1 package.
• The client lifecycle test verifies that API v1 selects one identifier, sends it to the simulation service, and does not replace it from the response.
• The branch rebased cleanly on the current master branch.

The Household API and UK Chat are outside this change.

[codecov]

Codecov Report

❌ Patch coverage is 91.14332% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.65%. Comparing base (aac95d3) to head (2054c27).

Files with missing lines	Patch %	Lines
policyengine_api/routes/household_routes.py	70.45%	9 Missing and 4 partials ⚠️
policyengine_api/runtime_cache/reform_impacts.py	75.00%	7 Missing and 4 partials ⚠️
policyengine_api/asgi_factory.py	88.50%	4 Missing and 6 partials ⚠️
policyengine_api/services/budget_window_cache.py	86.66%	7 Missing and 3 partials ⚠️
policyengine_api/services/economy_service.py	95.57%	2 Missing and 3 partials ⚠️
policyengine_api/observability/identifiers.py	84.61%	2 Missing ⚠️
policyengine_api/observability/runtime.py	92.30%	2 Missing ⚠️
policyengine_api/asgi.py	50.00%	1 Missing ⚠️
policyengine_api/gcp_logging.py	92.85%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3850       +/-   ##
===========================================
+ Coverage   46.37%   87.65%   +41.28%     
===========================================
  Files         178      197       +19     
  Lines       10421    11952     +1531     
  Branches     1759     2092      +333     
===========================================
+ Hits         4833    10477     +5644     
+ Misses       5199      901     -4298     
- Partials      389      574      +185     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[anth-volk]

This PR now documents its rollout dependencies explicitly:

1. Merge Add Stage 12 observability identifier schema #3852 and apply its nullable Stage 12 observability_id migration.
2. Deploy Configure and instrument simulation observability policyengine-sim-api#686, which now accepts both the currently deployed API identifiers and the new observability identifiers.
3. Deploy this PR.

The schema changes currently present in this branch overlap with #3852. After #3852 merges, this branch should be updated from master; the overlapping schema diff will then disappear while this PR retains the API instrumentation that uses the new nullable field.

[anth-volk]

Rebased onto master at 3069d37a after #3852 merged. The duplicate Stage 12 migration, ORM field, and shared revision edits are now supplied only by master; this PR consumes the merged nullable observability_id field while retaining identifier propagation, persistence, the stage registry, and telemetry behavior. Local checks passed: repository formatting (473 files unchanged), Ruff lint, migration contract validation, and 433 focused tests.

[anth-volk]

Commit f5884d6e makes X-PolicyEngine-Observability-Id the sole transport for observability_id between API v1 and the simulation API.

• API v1 no longer puts observability_id in _telemetry.
• It reads the identifier from simulation response headers for submission and polling.
• submission_claim_id remains separate execution metadata.
• Focused simulation client and economy service tests pass: 180 tests.

Deployment order: database precursor #3852, simulation API #686, then this PR. Rollback order is the reverse for the two application PRs.

[anth-volk]

Implemented the latest review fixes in da8d7511:

• Native FastAPI routes now start and finish the shared observability request lifecycle, attach request_id and observability_id, emit response trace headers, and retain exception isolation around every observability call.
• Flask fallback routes remain under the existing Flask adapter and do not receive a duplicate ASGI lifecycle.
• The Modal Workload Identity condition, workload inventory, and _Default log exclusion now include exact policyengine-simulation-v2-py<major>-<minor>-<patch> Stage 12 application names.
• Applied the corresponding provider condition and _Default exclusion to the live policyengine-observability project, then verified the exact resources and complete provider/sink lists.

Verification:

• 2229 passed, 45 skipped across the API v1 test selection used by make test.
• Focused Ruff checks passed.
• API v2 mypy checks passed for all 97 configured source files.
• Migration contract quality checks passed.

[anth-volk]

Implemented the follow-up identifier durability fix in 9d2a1869.

• Budget-window cache state is now one versioned document for starting, submitted, completed, or failed status.
• The document stores the observability_id with the ownership claim, batch job identifier, completed result, or failure. Success and failure records therefore retain the identifier for the same lifetime as the cached outcome.
• Requests that encounter existing work adopt the identifier from that state before returning or polling. A request that loses the initial submission claim also rereads the state and adopts the winning request's identifier.
• Deterministic submission failures and worker failures are retained with the same identifier instead of deleting the batch identifier and losing correlation state.

Focused verification:

• 144 budget-window cache, economy service, and typed worker polling tests passed.
• Ruff passed for all changed modules and tests.
• The new cache module passed mypy independently.

I stopped the broader API test run at the user's request and did not use it as verification for this update.