From cf164189665d2af6cbd2b2e35e012806026e729e Mon Sep 17 00:00:00 2001 From: Jason E Date: Mon, 24 Aug 2026 19:02:18 -0500 Subject: [PATCH 001/104] Add request phase timing design spec and implementation plan (#1069) * Add request phase timing design spec (Server-Timing subtimings + access telemetry) * Address review round 1: freeze point, template-cache naming, snapshot semantics, KV scope, geo carry, route template, sink confirmation, sampling and query model, config rollback * Address review round 2: auction-wait placement modes, conservative private-only header emission, non-null sorting key with service identity, coarse publisher route template, telemetry snapshot and outage behavior, tinybird flag decoupling, adapter phase semantics * Add request phase timing implementation plan * Address engineer review: KV timing decorator, try_lock sampling, route metadata extension, adapter-derived env, typed template-cache state, adapter-owned emission context, per-mode delivery semantics, Axum outer wrapper --- .../plans/2026-08-24-request-phase-timing.md | 1038 +++++++++++++++++ .../2026-08-24-request-phase-timing-design.md | 553 +++++++++ 2 files changed, 1591 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-24-request-phase-timing.md create mode 100644 docs/superpowers/specs/2026-08-24-request-phase-timing-design.md diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md new file mode 100644 index 000000000..9ff3905f4 --- /dev/null +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -0,0 +1,1038 @@ +# Request Phase Timing Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every application response attributes its own server time by phase via a +Server-Timing header and a sampled Tinybird access-telemetry row. + +**Architecture:** A core `RequestTimings` handle (Arc-shared, infallible recording) +collects phase spans always-on; the Fastly adapter freezes and emits at +`send_edgezero_response` immediately before `into_parts()`; a post-send emitter ships +one NDJSON row to the Tinybird Events API with a bounded, 2xx-validated await. + +**Tech Stack:** Rust 2024, `edgezero` HTTP types, Fastly Compute (wasm32-wasip1, +Viceroy tests), Axum (native tests), Tinybird Events API. + +**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`: the +plan argues from the spec; executors read both. Spec section numbers are cited per +task. + +## Global Constraints + +- Errors use `error-stack` (`Report`); errors defined with + `derive_more::Display`; never thiserror, never anyhow (except the Spin entry point). +- No `unwrap()` in production code; `expect("should ...")` only. Assertion messages + `"should ..."`. Tests use Arrange-Act-Assert. +- No inline comments; comments on their own line above the code. +- Functions never exceed 7 arguments; use a struct instead (this bit + `ec_finalize_response` in review; the timings handle travels inside existing state). +- No local imports inside functions; `use super::*` only in `#[cfg(test)]`. +- Only example/fictional data in tests and docs (`example.com` domains). +- Recording is infallible: saturating math, lock failure drops the sample, no panics + (spec 5, 13). +- Vendor identity never appears in emitted surfaces: the filter span is `ts-filter` + (spec 3). +- Test commands: `cargo test-axum` (native, fast inner loop), `cargo test-fastly` + (Viceroy) for adapter tasks. Before PR handoff: the full CI gate list in + `CLAUDE.md`. +- Commit style: sentence case, imperative, no prefixes, no trailers. + +## File Structure + +| File | Responsibility | +| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `crates/trusted-server-core/src/request_timing.rs` (new) | `Phase`, `AuctionWaitPlacement`, `RequestTimings`, `PhaseSpan`, `TimingSnapshot`, header rendering | +| `crates/trusted-server-core/src/access_telemetry.rs` (new) | `RouteClass`, `publisher_route_template`, `AccessTelemetrySnapshot`, `AccessEventRow` NDJSON | +| `crates/trusted-server-core/src/geo.rs` (modify) | `GeoLookupState` response-extension type | +| `crates/trusted-server-core/src/settings.rs` (modify) | `ObservabilitySettings`, tinybird flag decoupling, access validation | +| `crates/trusted-server-core/src/publisher.rs` (modify) | `ts-origin`, `ts-template-cache`, auction-wait spans | +| `crates/trusted-server-core/src/ec/kv.rs` (modify) | `ts-kv` at the graph abstraction | +| `crates/trusted-server-adapter-fastly/src/main.rs` (modify) | T0, appbuild span, freeze point, `DeliveryOutcome`, post-send emission ordering | +| `crates/trusted-server-adapter-fastly/src/app.rs` (modify) | filter span, geo span + `GeoLookupState` attach, route class assignment | +| `crates/trusted-server-adapter-fastly/src/middleware.rs` (modify) | finalize consumes `GeoLookupState` | +| `crates/trusted-server-adapter-fastly/src/tinybird.rs` (modify) | access sink with confirmed delivery | +| `crates/trusted-server-adapter-axum/src/` (modify) | terminal freeze layer, header emission | +| `tinybird/datasources/access_logs_raw.datasource` (modify) | phase-column schema, non-null sorting key | +| `trusted-server.example.toml` (modify) | `[observability]`, tinybird keys | + +Out of scope for this plan: the Grafana dashboard JSON (separate telemetry repo, +spec 11) and Cloudflare/Spin emission wiring (spec non-goal). + +--- + +### Task 1: Core `RequestTimings` + +**Files:** + +- Create: `crates/trusted-server-core/src/request_timing.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` (add `pub mod request_timing;`) +- Test: same file, `#[cfg(test)]` + +**Interfaces:** + +- Consumes: nothing (leaf module; `std::time`, `std::sync`). +- Produces (later tasks rely on these exact names): + - `pub enum Phase { AppBuild, Filter, Geo, EcKv, Origin, TemplateCacheLookup, AuctionWait, Stream }` + - `pub enum AuctionWaitPlacement { PreHeader, InStream }` + - `#[derive(Clone)] pub struct RequestTimings` with: + - `pub fn new() -> Self` + - `pub fn record(&self, phase: Phase, dur: Duration)` (saturating accumulate) + - `pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration)` + - `pub fn span(&self, phase: Phase) -> PhaseSpan` (records on drop) + - `pub fn mark_headers_ready(&self)` (first call wins) + - `pub fn mark_request_elapsed(&self)` (first call wins) + - `pub fn set_resp_bytes(&self, bytes: u64)` + - `pub fn server_timing_value(&self) -> Option` + - `pub fn snapshot(&self) -> TimingSnapshot` + - `pub struct TimingSnapshot { pub time_elapsed_ms: Option, pub request_elapsed_ms: Option, pub appbuild_ms: Option, pub filter_ms: Option, pub geo_ms: Option, pub kv_ms: Option, pub origin_ms: Option, pub template_cache_ms: Option, pub auction_wait_ms: Option, pub stream_ms: Option, pub auction_wait_placement: Option, pub resp_bytes: Option }` + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn render_omits_unrecorded_phases_and_orders_total_first() { + let timings = RequestTimings::new(); + timings.record(Phase::Filter, Duration::from_micros(9_100)); + timings.mark_headers_ready(); + let value = timings + .server_timing_value() + .expect("should render after mark_headers_ready"); + assert!( + value.starts_with("ts-total;dur="), + "should lead with ts-total: {value}" + ); + assert!(value.contains("ts-filter;dur=9.1"), "should render one decimal: {value}"); + assert!(!value.contains("ts-geo"), "should omit unrecorded phases: {value}"); + } + + #[test] + fn render_returns_none_before_headers_ready() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(1)); + assert!(timings.server_timing_value().is_none(), "should require the snapshot"); + } + + #[test] + fn repeated_phases_accumulate_saturating() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(2)); + timings.record(Phase::Geo, Duration::from_millis(3)); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.geo_ms, Some(5), "should accumulate repeats"); + } + + #[test] + fn mark_headers_ready_is_first_call_wins() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let first = timings.snapshot().time_elapsed_ms; + std::thread::sleep(Duration::from_millis(5)); + timings.mark_headers_ready(); + assert_eq!(timings.snapshot().time_elapsed_ms, first, "should not restamp"); + } + + #[test] + fn span_guard_records_on_drop() { + let timings = RequestTimings::new(); + { + let _span = timings.span(Phase::Origin); + std::thread::sleep(Duration::from_millis(2)); + } + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.expect("should record on drop") >= 1, + "should measure elapsed span time" + ); + } + + #[test] + fn auction_wait_records_placement() { + let timings = RequestTimings::new(); + timings.record_auction_wait(AuctionWaitPlacement::PreHeader, Duration::from_millis(40)); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.auction_wait_ms, Some(40), "should record wait"); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "should record placement" + ); + } + + #[test] + fn rendered_names_never_include_vendor_terms() { + let timings = RequestTimings::new(); + for phase in [Phase::AppBuild, Phase::Filter, Phase::Geo, Phase::EcKv, Phase::Origin, Phase::TemplateCacheLookup] { + timings.record(phase, Duration::from_millis(1)); + } + timings.mark_headers_ready(); + let value = timings.server_timing_value().expect("should render"); + assert!(!value.to_ascii_lowercase().contains("datadome"), "should mask vendors"); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core request_timing` +Expected: compile FAIL, module does not exist. + +- [ ] **Step 3: Implement** + +```rust +//! Per-request phase timing collection and Server-Timing rendering. +//! +//! Collection is always-on and infallible: saturating math, lock failure +//! drops the sample, no panics. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +const PHASE_COUNT: usize = 8; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Phase { + AppBuild, + Filter, + Geo, + EcKv, + Origin, + TemplateCacheLookup, + AuctionWait, + Stream, +} + +impl Phase { + fn index(self) -> usize { /* match self -> 0..=7 */ } + + /// Header entry name; row-only phases return None. + fn header_name(self) -> Option<&'static str> { + match self { + Self::AppBuild => Some("ts-appbuild"), + Self::Filter => Some("ts-filter"), + Self::Geo => Some("ts-geo"), + Self::EcKv => Some("ts-kv"), + Self::Origin => Some("ts-origin"), + Self::TemplateCacheLookup => Some("ts-template-cache"), + Self::AuctionWait | Self::Stream => None, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuctionWaitPlacement { + PreHeader, + InStream, +} + +struct Inner { + t0: Instant, + phases: [Option; PHASE_COUNT], + headers_ready_total: Option, + request_elapsed: Option, + auction_wait_placement: Option, + resp_bytes: Option, +} + +#[derive(Clone)] +pub struct RequestTimings(Arc>); +``` + +Implementation notes (all bodies in this task, none deferred): + +- Every method takes `if let Ok(mut inner) = self.0.try_lock()` and silently + returns otherwise: contention and poison both drop the sample instead of waiting, + per the infallibility constraint. +- `record` accumulates with `saturating_add` semantics + (`Some(existing.saturating_add(dur))`). +- `mark_headers_ready` and `mark_request_elapsed` write `t0.elapsed()` only when the + slot is `None`. +- `server_timing_value` returns `None` unless `headers_ready_total` is set; renders + `ts-total` first from the stored snapshot, then the six header phases in enum order + with `{:.1}` millisecond formatting (`dur.as_secs_f64() * 1000.0`). +- `PhaseSpan { timings: RequestTimings, phase: Phase, started: Instant }`; `Drop` + calls `record(self.phase, self.started.elapsed())`. +- `TimingSnapshot` converts each `Duration` with + `u32::try_from(dur.as_millis()).unwrap_or(u32::MAX)`. +- `impl Default for RequestTimings` delegates to `new()`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum -p trusted-server-core request_timing` +Expected: all 7 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/request_timing.rs crates/trusted-server-core/src/lib.rs +git commit -m "Add RequestTimings phase collection and Server-Timing rendering" +``` + +--- + +### Task 2: Settings: `[observability]`, tinybird decoupling, access validation + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs` +- Modify: `trusted-server.example.toml` + +**Interfaces:** + +- Produces: + - `pub struct ObservabilitySettings { pub server_timing_enabled: bool }` as + `settings.observability`, `#[serde(default)]` on the field and + `#[serde(skip_serializing_if = "ObservabilitySettings::is_default")]`. + - `TinybirdSettings.auction_enabled: bool` (`#[serde(default = "default_true")]`). + - `prepare_runtime` validation: `access_enabled` requires `enabled`, non-empty + `api_host`, `secret_store`, `access_dataset`, `access_token_secret`, + `max_body_bytes > 0`, and `access_sample_rate > 0.0`. + +- [ ] **Step 1: Write the failing tests** (in `settings.rs` tests module) + +```rust +#[test] +fn observability_defaults_off_and_serializes_away() { + let settings = create_test_settings(); + assert!(!settings.observability.server_timing_enabled, "should default off"); + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("[observability]"), + "should omit the default table so a prior binary can parse the config" + ); +} + +#[test] +fn access_enabled_requires_positive_sample_rate() { + // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. + let err = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 0.0\n", + ) + .expect_err("should reject armed-but-silent access telemetry"); + assert!(format!("{err:?}").contains("access_sample_rate"), "should name the field"); +} + +#[test] +fn access_and_auction_emission_are_independent() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept access without auction"); + assert!(!settings.tinybird.auction_enabled, "should disable auction emission"); + assert!(settings.tinybird.access_enabled, "should enable access emission"); +} + +#[test] +fn auction_enabled_defaults_true_for_existing_configs() { + let settings = settings_from_toml_with("[tinybird]\nenabled = true\napi_host = \"api.example.com\"\n") + .expect("should parse a pre-decoupling config"); + assert!(settings.tinybird.auction_enabled, "should preserve current behavior"); +} +``` + +Also REPLACE the existing rejection test +(`tinybird_access_enabled_is_rejected_until_emitter_is_wired`, `settings.rs:4123`) +with a wiring test asserting a fully-specified access config is accepted. + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core observability access_enabled auction_enabled` +Expected: compile FAIL (`observability` field missing). + +- [ ] **Step 3: Implement** + +- Add `ObservabilitySettings` (derive `Debug, Clone, Default, PartialEq, Deserialize, +Serialize`, `#[serde(deny_unknown_fields)]`), with + `fn is_default(&self) -> bool { *self == Self::default() }`. +- Add the `observability` field to `Settings` with the serde attributes above. +- Add `auction_enabled` to `TinybirdSettings` with `default_true()`; update + `Default for TinybirdSettings`. +- Extend `TinybirdSettings::prepare_runtime` with the access validation matrix; error + messages name the failing field (`"tinybird.access_sample_rate must be > 0 when +access_enabled"` and so on). +- `trusted-server.example.toml`: add a commented `[observability]` block with + `server_timing_enabled = false` present-but-false and the env-override note (the + overlay cannot create a missing leaf), plus `auction_enabled`/access keys in the + tinybird section comments. +- Gate the auction sink: in `crates/trusted-server-adapter-fastly/src/app.rs`, + `auction_sink_from_settings` condition becomes + `settings.tinybird.enabled && settings.tinybird.auction_enabled`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum -p trusted-server-core` then `cargo test-fastly` (the sink gate +touches the Fastly adapter). +Expected: PASS, including the replaced wiring test. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/settings.rs trusted-server.example.toml crates/trusted-server-adapter-fastly/src/tinybird.rs crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Add observability settings and decouple tinybird access and auction emission" +``` + +--- + +### Task 3: Fastly freeze point, header emission, `DeliveryOutcome` + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (entry T0, appbuild span, + `send_edgezero_response`) +- Test: `crates/trusted-server-adapter-fastly/src/app.rs` tests module (route-level + tests run under Viceroy) + +**Interfaces:** + +- Consumes: `RequestTimings`, `Phase` (Task 1); + `trusted_server_core::cache_policy::cache_control_headers_are_private_or_no_store`. +- Produces: + - `RequestTimings` inserted into request extensions at dispatch + (`core_req.extensions_mut().insert(timings.clone())`), alongside the existing + `config_store`/`device_signals`/`client_info` inserts. + - `send_edgezero_response(response, effects, timings) -> DeliveryOutcome` where + `pub(crate) struct DeliveryOutcome { pub bytes: u64, pub result: DeliveryResult }` + and `pub(crate) enum DeliveryResult { Complete, Error }` (streaming partial + detection lands in Task 6). + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn server_timing_emitted_on_private_response_when_enabled() { + // Arrange: settings with observability.server_timing_enabled = true; publisher + // route fixture whose response is Cache-Control: private, no-store. + // Act: dispatch through the full adapter path. + // Assert: + let header = response_header(&response, "server-timing").expect("should emit header"); + assert!(header.contains("ts-total;dur="), "should carry the stored total"); + assert_eq!( + header.matches("ts-total").count(), 1, + "should emit exactly one TS-owned metric set" + ); +} + +#[test] +fn server_timing_absent_when_flag_off() { /* same fixture, flag false: no ts-total */ } + +#[test] +fn server_timing_absent_on_cacheable_responses() { + // tsjs route (public, max-age=31536000, immutable) and a bare max-age=60 response: + // both must carry no ts-total even with the flag on. +} + +#[test] +fn preexisting_server_timing_values_survive() { + // Fixture response already carrying Server-Timing: upstream;dur=1 stays present + // alongside the appended TS set. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly server_timing` +Expected: FAIL, no header emitted. + +- [ ] **Step 3: Implement** + +In `edgezero_main` (`main.rs`): + +```rust +let timings = RequestTimings::new(); +{ + let _appbuild = timings.span(Phase::AppBuild); + // existing: open_trusted_server_config_store() + build_app_with_state() +} +``` + +Move the config-store open inside the span scope. Insert `timings.clone()` into +request extensions before dispatch. Thread the handle into both send sites and the +error paths by value (it is a cheap clone). + +In `send_edgezero_response`, immediately before `response.into_parts()`: + +```rust +timings.mark_headers_ready(); +let conclusively_private = + cache_control_headers_are_private_or_no_store(response.headers()); +if settings_enabled_server_timing && conclusively_private { + if let Some(value) = timings.server_timing_value() { + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response.headers_mut().append(header::SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } + } +} +``` + +`settings_enabled_server_timing` arrives inside a small +`SendContext { timings: RequestTimings, server_timing_enabled: bool }` so the +function stays at or under seven parameters. Return `DeliveryOutcome` with per-mode +semantics: buffered bodies capture the byte count from the body length before +`send_to_client()` (which returns no delivery result) and report complete-on-return; +the streaming branch gains a counting writer in Task 6. Existing callers ignore the +outcome in this task (Task 8 consumes it). + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/main.rs crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Emit Server-Timing at the send freeze point on conclusively private responses" +``` + +--- + +### Task 4: Filter span and geo span with `GeoLookupState` dedupe + +**Files:** + +- Modify: `crates/trusted-server-core/src/geo.rs` (add `GeoLookupState`) +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` + (`run_pre_route_filters` wrapper, `build_ec_request_state` geo span + state attach) +- Modify: `crates/trusted-server-adapter-fastly/src/middleware.rs` and `main.rs` + (`resolve_geo_for_response` consumes carried state) + +**Interfaces:** + +- Consumes: `RequestTimings` from request extensions (Task 3). +- Produces: + - `pub enum GeoLookupState { NotAttempted, Attempted, Resolved(GeoInfo) }` in + `trusted_server_core::geo`, attached as a response extension on every exit path + that attempted a lookup (including the asset fallback). + - `resolve_geo_for_response` gains the carried state as input: live lookup only on + `NotAttempted`; `Attempted` is never retried; fallback lookups are wrapped in + `timings.span(Phase::Geo)`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn finalize_reuses_request_phase_geo_without_second_lookup() { + // Counting geo stub: dispatch a publisher route; assert lookup count == 1 and + // x-geo-country still set on the response. +} + +#[test] +fn failed_lookup_is_not_retried() { + // Stub returns None once; assert GeoLookupState::Attempted carried and the + // finalize path performs zero further lookups. +} + +#[test] +fn asset_fallback_carries_geo_state_without_ec_finalize_state() { + // Asset route: response extension holds GeoLookupState, EcFinalizeState absent. +} + +#[test] +fn filter_span_recorded_when_request_filter_runs() { + // Registry fixture with a test request filter; assert snapshot().filter_ms is Some. +} + +#[test] +fn geo_lookup_skipped_for_unauthorized_responses() { + // Existing 401 rule preserved: no lookup, state NotAttempted. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly geo_ filter_span` +Expected: FAIL (lookup count 2; no `GeoLookupState`). + +- [ ] **Step 3: Implement** + +- `GeoLookupState` derives `Debug, Clone`; store in response extensions from the + dispatch layer right after `build_ec_request_state` resolves (or fails) its lookup. +- Wrap the `build_ec_request_state` lookup and any finalize fallback lookup in + `timings.span(Phase::Geo)` (accumulating slot handles the repeat case). +- Wrap `run_pre_route_filters` (`app.rs:751`) in `timings.span(Phase::Filter)`, + recording only when at least one filter is registered (skip the span when the + registry has no request filters, so the header omits `ts-filter` on unconfigured + deployments). +- `resolve_geo_for_response(response, carried: &GeoLookupState, client_ip, lookup)` + keeps the 401 short-circuit first, then matches the carried state. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo test-axum` +Expected: PASS including untouched existing geo header tests. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/geo.rs crates/trusted-server-adapter-fastly/src/app.rs crates/trusted-server-adapter-fastly/src/middleware.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Record filter and geo spans and dedupe the per-request geo lookup" +``` + +--- + +### Task 5: Core spans: origin, template cache, KV abstraction + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (origin send ~4496, template + cache lookup ~4391 on current main) +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (graph-level `ts-kv`) +- Test: `publisher.rs` and `ec/kv.rs` test modules + +**Interfaces:** + +- Consumes: `RequestTimings` read from request extensions inside + `handle_publisher_request`; `KvIdentityGraph` gains + `pub fn with_timings(self, timings: RequestTimings) -> Self` (builder-style, + optional field), set where the graph is constructed in `main.rs`. +- Produces: `origin_ms`, `template_cache_ms`, `kv_ms` populated in snapshots. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn origin_span_covers_the_publisher_fetch() { + // Stubbed origin with a small injected delay; assert snapshot().origin_ms is Some. +} + +#[test] +fn template_cache_span_recorded_only_when_lookup_runs() { + // Inline mode fixture: template_cache_ms None. Shared-mode eligible fixture: + // template_cache_ms Some. +} + +#[test] +fn kv_span_accumulates_across_graph_operations() { + // Stub KV recording two operations through a TimedKvStore-wrapped graph; assert + // kv_ms Some and covers both (accumulated, not last-write). +} + +#[test] +fn consent_store_reads_are_timed_and_pull_sync_is_not() { + // Consent read through the decorated RuntimeServices store: kv_ms Some. + // Pull-sync graph built from the untimed store: records nothing. +} + +#[test] +fn ec_finalize_kv_lands_before_freeze() { + // Adapter-level (test-fastly): EC-enabled fixture with eids cookies; assert the + // emitted header contains ts-kv, proving the freeze point sits after finalize. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core origin_span template_cache_span kv_span` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- `handle_publisher_request` reads the handle: + `let timings = req.extensions().get::().cloned().unwrap_or_default();` + (a defaulted handle records into nothing that ever renders, keeping non-adapter + tests unchanged). +- Origin: `let origin_span = timings.span(Phase::Origin);` immediately before + `services.http_client().send(platform_request).await`; `drop(origin_span)` when the + response headers are available (directly after the `match` arm binds the response). +- Template cache: same guard pattern around + `services.template_cache().lookup_or_reserve(key).await`. +- KV: add `TimedKvStore` (new type in `crates/trusted-server-core/src/platform/`), + a decorator implementing `PlatformKvStore` that wraps `Arc` + plus a `RequestTimings` handle and records `Phase::EcKv` around every trait + method. Every request-path `KvIdentityGraph` construction site (request setup, + identify, admin lookup, batch sync, finalization) receives the timed store; + consent-store access through `RuntimeServices` uses the same decorator; pull-sync + constructs its graph from the untimed store explicitly (add a test asserting the + pull-sync store records nothing). `ec_finalize_response` keeps seven arguments: + the handle rides inside the store the graph already receives. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` then `cargo test-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs crates/trusted-server-core/src/ec/kv.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Record origin, template cache, and KV phase spans in core" +``` + +--- + +### Task 6: Body-phase capture: stream, auction wait placement, bytes + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (seam wait + buffered wait) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (stream drive timing, + `DeliveryOutcome.bytes`) +- Test: `publisher.rs` tests + adapter tests + +**Interfaces:** + +- Consumes: `record_auction_wait` (Task 1), `DeliveryOutcome` (Task 3). +- Produces: `stream_ms`, `auction_wait_ms` + placement, `resp_bytes`, + `mark_request_elapsed()` called by the adapter immediately after the stream drive + returns. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn streaming_seam_wait_records_in_stream_placement() { + // Streaming fixture with a delayed auction: placement InStream, and + // stream_ms >= auction_wait_ms. +} + +#[test] +fn buffered_template_miss_records_pre_header_placement() { + // Shared-template authorized miss (buffered finalizer): placement PreHeader; the + // wait is recorded even though headers had not committed. +} + +#[test] +fn delivery_outcome_reports_bytes_and_request_elapsed_set() { + // Adapter: after send, snapshot has resp_bytes Some(body_len) and + // request_elapsed_ms Some; request_elapsed excludes post-send emitter time by + // construction (asserted by ordering test in Task 8). +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly seam_wait buffered_template delivery_outcome` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- Streaming path: around the `collect_stream_auction(...)` await inside the body + stream, measure with `Instant::now()` and call + `timings.record_auction_wait(AuctionWaitPlacement::InStream, waited)`. The handle + reaches the stream closure through `OwnedProcessResponseParams`/assembly params (it + is `Clone`; add a field). +- Buffered path (`buffer_publisher_response_async` and the shared-template miss + finalizer): same measurement with `AuctionWaitPlacement::PreHeader`. +- Adapter stream drive: wrap the `block_on(stream_asset_body(...))` region with a + counting writer that tallies bytes and observes truncation/error, record + `Phase::Stream` with the elapsed drive time, populate `DeliveryOutcome` with + bytes and Complete/Partial/Error, call `timings.set_resp_bytes(bytes)` and + `timings.mark_request_elapsed()` immediately after the drive returns, before + anything else post-send. Buffered responses keep the Task 3 complete-on-return + semantics; `body_mode` distinguishes the regimes in the row. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo test-axum` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Capture stream duration, auction wait placement, and response bytes" +``` + +--- + +### Task 7: `AccessTelemetrySnapshot`, route class, route template + +**Files:** + +- Create: `crates/trusted-server-core/src/access_telemetry.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` (RouteMetadata attach at + handler wrappers), `main.rs` (snapshot build at freeze point), + `crates/trusted-server-core/src/publisher.rs` (typed template-cache state + extension) + +**Interfaces:** + +- Consumes: `TimingSnapshot` (Task 1), `GeoLookupState` (Task 4). +- Produces: + - `pub enum RouteClass { PublisherHtml, Tsjs, IntegrationProxy, Ec, AuctionApi, Other }` + with `pub fn as_str(&self) -> &'static str` (snake_case values from the spec). + - `pub fn publisher_route_template(path: &str) -> String`: `/` plus first segment + filtered to `[a-z0-9_-]`, truncated to 32 chars, plus `/*` when deeper; empty or + disallowed first segments render `/other/*`. + - `pub struct AccessTelemetrySnapshot { pub method: String, pub status: u16, pub route_class: RouteClass, pub route_template: String, pub publisher_domain: String, pub env: String, pub service_id: String, pub pop: String, pub ts_version: String, pub country: String, pub template_cache_state: String, pub body_mode: &'static str, pub sample_rate: f64 }` + - `pub fn access_event_row(snapshot: &AccessTelemetrySnapshot, timings: &TimingSnapshot, event_ts_epoch_ms: u64) -> String` (one NDJSON line). + +- [ ] **Step 1: Write the failing tests** (adversarial, per spec 9) + +```rust +#[test] +fn admin_ec_route_template_never_contains_the_identifier() { + // Named-route template comes from the route table: "/_ts/admin/ec/{id}". + // Assert a row built for that route never contains a 64-hex EC id fixture. +} + +#[test] +fn publisher_paths_normalize_to_coarse_templates() { + assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); + assert_eq!(publisher_route_template("/"), "/"); + assert_eq!( + publisher_route_template("/user@example.com/profile"), + "/other/*", + "should reject non-allowlisted characters" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a".repeat(500))), + format!("/{}", "a".repeat(32)), + "should bound segment length" + ); + assert_eq!(publisher_route_template("/search terms here"), "/other/*"); +} + +#[test] +fn row_serializes_nulls_for_missing_phases() { + // Sparse TimingSnapshot: absent phases serialize as JSON null, dimension fields + // never null (unknown sentinel). +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core access_telemetry route_template` +Expected: compile FAIL. + +- [ ] **Step 3: Implement** + +Row serialization via `serde_json::json!` mapping spec section 9 column names exactly +(`time_elapsed_ms`, `appbuild_ms`, ..., `auction_wait_placement` as +`pre_header|in_stream|none`). `pop`/`service_id` read from Fastly env +(`FASTLY_SERVICE_ID`, `FASTLY_POP`), defaulting `"unknown"`; `env` derived by the +adapter from `FASTLY_IS_STAGING` (the `x-ts-env` input), never from `Settings`. +Route identity travels as a typed `RouteMetadata` response extension +(`pub struct RouteMetadata { pub route_class: RouteClass, pub route_template: String }` +in `access_telemetry.rs`): each named-route handler wrapper attaches its matched +route-table pattern verbatim, and the fallback and tsjs handlers attach their class +plus the coarse template; the freeze point consumes the extension (no `RouteClass` +column in `NAMED_ROUTES`, no reconstruction from a handler enum). Also in this task: +make `TemplateCacheResponseState` a typed response extension in `publisher.rs`, set +at every point that writes `x-ts-template-cache` so header and extension cannot +drift; the row reads the extension. The snapshot is built unconditionally in +`send_edgezero_response` right after `mark_headers_ready()` and returned inside +`DeliveryOutcome` (add field `pub snapshot: AccessTelemetrySnapshot`). + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` and `cargo test-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/access_telemetry.rs crates/trusted-server-core/src/lib.rs crates/trusted-server-adapter-fastly/src/app.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Add access telemetry snapshot, route classes, and coarse route templates" +``` + +--- + +### Task 8: Access sink with confirmed delivery + post-send ordering + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/tinybird.rs` (access sink) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (post-send ordering) + +**Interfaces:** + +- Consumes: `AccessTelemetrySnapshot` + `access_event_row` (Task 7), settings flags + (Task 2), `DeliveryOutcome` (Tasks 3/6). +- Produces: `pub(crate) async fn emit_access_event(client: &FastlyPlatformHttpClient, target: &TinybirdEventsTarget, row: String) -> Result<(), Report>`, + sending via the adapter's stateless platform client (the blocking variant, + post-delivery), checking `response.status().is_success()`, warning with status + otherwise. The transport context is adapter-owned and route-independent (target + derived from settings once at entry), so asset, admin, and error responses emit + without `RuntimeServices` or `EcFinalizeState`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn access_emitter_posts_ndjson_and_validates_2xx() { + // RecordingHttpClient returning 202: assert URI is /v0/events?name=access_logs_raw, + // body is the row, Authorization bearer from the secret stub. +} + +#[test] +fn access_emitter_warns_and_drops_on_non_2xx() { + // RecordingHttpClient returning 422: emit returns Err naming the status; no retry + // request recorded (exactly one request seen). +} + +#[test] +fn sampled_out_requests_emit_nothing() { + // access_sample_rate stub decision false: RecordingHttpClient sees zero requests. +} + +#[test] +fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // Instrumented stubs record call order; assert request_elapsed snapshot precedes + // pull-sync dispatch which precedes the telemetry send. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly access_emitter post_send_order` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- Reuse `TinybirdEventsTarget` with a second constructor + `from_access_config(config: TinybirdSettings)` using `access_dataset` and + `access_token_secret`. +- Sampling decision: `fn sampled_in(rate: f64, entropy: u64) -> bool` where entropy is + derived from the event timestamp nanos XOR a per-request counter (no `rand` + dependency; document that uniformity is approximate and sufficient). +- `main.rs` post-send, in order: `timings.mark_request_elapsed()` (already placed in + Task 6), existing pull-sync dispatch unchanged, then when + `settings.tinybird.enabled && settings.tinybird.access_enabled` and sampled in: + build the row from `outcome.snapshot` + `timings.snapshot()`, call + `emit_access_event`, log one warning on `Err`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/tinybird.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Emit confirmed access telemetry rows after pull-sync post-send" +``` + +--- + +### Task 9: Tinybird datasource schema + +**Files:** + +- Modify: `tinybird/datasources/access_logs_raw.datasource` + +**Interfaces:** + +- Consumes: column names exactly as serialized by `access_event_row` (Task 7). +- Produces: the deployed schema contract for the dashboard (separate repo). + +- [ ] **Step 1: Rewrite the schema** per spec section 9: keep + `event_ts DateTime64(3)`, `method`, `status UInt16`, `time_elapsed_ms UInt32`, + `sample_rate Float64`, `event_date` + 30-day TTL; add the columns from spec 9 with + dimension columns non-nullable `LowCardinality(String)` and phase columns + `Nullable(UInt32)`; drop `path` and `cache_state`; set + `ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status"`. + +- [ ] **Step 2: Validate** with the tinybird toolchain if available locally + (`tb check` / project tests under `tinybird/tests`); otherwise assert the file + parses by review and rely on rollout step 4's remote verification. Add a fixture row + in `tinybird/fixtures` matching `access_event_row` output. + +- [ ] **Step 3: Commit** + +```bash +git add tinybird/datasources/access_logs_raw.datasource tinybird/fixtures +git commit -m "Extend access_logs_raw with phase columns and a non-null sorting key" +``` + +--- + +### Task 10: Axum adapter emission + +**Files:** + +- Modify: `crates/trusted-server-adapter-axum/src/` (terminal layer at the response + serialization boundary; locate the equivalent of the Fastly send path) +- Test: axum adapter tests (`cargo test-axum`) + +**Interfaces:** + +- Consumes: `RequestTimings`, header emission helper. Extract the emission block from + Task 3 into a shared core helper so both adapters call one function: + `pub fn append_server_timing_if_private(response: &mut Response, timings: &RequestTimings, enabled: bool)` + in `request_timing.rs` (move the Fastly inline logic here and re-point Task 3's call + site). +- Produces: Axum responses carry the header under the same conservative predicate; + `ts-appbuild` absent by construction (state built at startup); router-generated + 404/405 covered by the terminal layer; `/health` excluded by route match. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn axum_emits_header_on_private_response() { /* flag on, private response: ts-total present, ts-appbuild absent */ } + +#[test] +fn axum_404_carries_header_when_private() { /* router-generated 404 passes through the terminal layer */ } + +#[test] +fn axum_health_is_excluded() { /* /health: no ts-total */ } +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum axum_emits axum_404 axum_health` +Expected: FAIL. + +- [ ] **Step 3: Implement** an outer service wrapper around the `RouterService` + inside `AxumDevServer` (not router middleware, which router-generated 404/405 + responses bypass and which returns before body serialization): create + `RequestTimings::new()` per request in the wrapper, insert into request + extensions, and on the wrapper's response side call `mark_headers_ready()` + + `append_server_timing_if_private(...)`, skipping the `/health` path by match. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-axum/src crates/trusted-server-core/src/request_timing.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Emit Server-Timing from the Axum terminal layer with adapter-specific semantics" +``` + +--- + +### Task 11: Full gate, docs, and PR + +- [ ] **Step 1: Docs.** Add a short operator section to `docs/guide/configuration.md`: + the `[observability]` flag, the tinybird access keys, the deploy/rollback ordering + from spec section 12 (binary first, config second; config first on rollback), and + the conservative emission rule. Run `cd docs && npm run format`. + +- [ ] **Step 2: Full CI gate list** from `CLAUDE.md`: + `cargo fmt --all -- --check`; all six clippy aliases; `test-fastly`, `test-axum`, + `test-cloudflare`, `test-spin`; the integration-tests parity suite; JS build/test + and formats. Cloudflare/Spin compile the new core modules (collection only), which + is exactly what the non-goal requires. + +- [ ] **Step 3: Commit docs, push the branch, open the implementation PR** referencing + the spec PR #1069 and issue #1068, with the rollout section of the spec quoted as + the deployment checklist (staging pass-through + MISS/HIT replay before production + flag-on). + +--- + +## Self-Review + +- Spec coverage: sections 5 (Task 1), 12 (Task 2), 7 (Tasks 3, 10), 8/8a (Tasks 4, + 10), 6 (Tasks 4-6), 9 (Tasks 7, 9), 10 (Task 8), 13 (Tasks 1, 3, 8), 14 (test + steps throughout), 15 steps 1-4 (Task 11 + deployment checklist). Section 11 + (dashboard) is explicitly out of scope for this repo's plan. +- Type consistency: `RequestTimings`/`TimingSnapshot`/`RouteClass`/ + `AccessTelemetrySnapshot`/`DeliveryOutcome` names and signatures match across + Tasks 1, 3, 6, 7, 8, 10. +- Known intentional deferral: `DeliveryResult::Partial` detection is named in Task 3 + and wired when the stream drive reports bytes in Task 6; no other deferrals. diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md new file mode 100644 index 000000000..06f27c1c2 --- /dev/null +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -0,0 +1,553 @@ +# Request phase timing: Server-Timing subtimings and access telemetry + +**Date:** 2026-08-24 +**Status:** Approved design, revised for review rounds 1 and 2, pending implementation +plan. +**Scope:** `trusted-server-core`, Fastly and Axum adapters, `tinybird/` schema, +performance dashboard (separate repo). + +--- + +## 1. Problem + +On 2026-08-21 a production deployment (publisher redacted, `prospect-a.example`) showed +an episodic stall: for a window of roughly 40 minutes, every request that reached the +application path carried a uniform extra ~600 ms of Fastly `time-elapsed`, and then +recovered to 20-50 ms with no deploy or config change we could observe. `/health` +(2-4 ms, short-circuits before app construction) and `/_ts/debug/ja4` (6-9 ms, settings +load only) stayed fast throughout, so the stall lived between app construction and +response send. + +Attributing that window required a live probing session: route-by-route bisection, +cookie-deletion experiments, and an eight-agent code trace. The trace found no +unconditional await on the path that could cost 570 ms, and exactly two +config-conditional candidates (the pre-route request filter's synchronous verification +POST, and EC identity KV writes before send), plus one dependency shared by every +application route (two geo hostcalls per request). We could not tell which one stalled, +because nothing in the response says where server time went. + +The Compute CPU budget is ~50 ms per request, so a large `time-elapsed` strongly +suggests wall-clock time outside active guest CPU: dependency awaits are the leading +explanation, with platform scheduling and hostcall queueing as the residual ones. The +comparison figure here is the fronting delivery layer's `time-elapsed` Server-Timing +entry, observed at its deliver phase. Either way, these are exactly the numbers a +response can carry about itself. + +## 2. Goals + +1. Every normal application response attributes its own server time by phase in a + standard header. Browsers expose the values to same-origin JavaScript via + `PerformanceResourceTiming.serverTiming`, so RUM tooling that reads that API can + surface the breakdown. Whether a given vendor or the publisher's own monitoring + extension actually collects it is verified separately in rollout; the publisher + extension needs a small change to render it. +2. The same numbers flow to Tinybird so we hold p50/p95/p99 per phase, per route class, + per PoP, per deployed version, and a future stall window self-diagnoses in one query. +3. No additional awaited I/O before first byte. The pre-send cost is a handful of + monotonic clock reads, one small allocation at entry, and rendering one header; + telemetry emission happens strictly after the last body byte. + +Scope note: phases cover the application lifecycle after T0. The `/health` and +`/_ts/debug/ja4` short-circuits, config-store open failures, and request-conversion +failures bypass the lifecycle and emit nothing. Requests served entirely by the +fronting cache never reach the guest and produce neither header entries nor rows. + +## 3. Non-goals + +- No trailer-based Server-Timing for body-phase spans (browsers do not expose trailer + values to JavaScript). +- No per-filter naming in any emitted surface. The request-filter span is `ts-filter` + regardless of which filter runs; vendor identity stays out of headers and telemetry. +- No Cloudflare or Spin emission wiring in v1. Core collection is adapter-neutral; those + adapters can wire emission later without core changes. +- No Tinybird endpoint pipe and no rollup materialized views in v1. Grafana queries the + datasource through the ClickHouse connector, matching the auction dashboards; rollups + only if panel latency demands them. +- No sampling of the header. The header is all-traffic when enabled; only Tinybird rows + sample. +- No cross-request circuit breaker for telemetry emission. Compute runs one isolate per + request; there is no shared mutable state to hold breaker state. The controls are the + bounded per-request cost and the `access_sample_rate` lever (section 10). + +## 4. Design overview + +``` +adapter entry (T0) + | RequestTimings::new() -> shared handle + v +app construction ................ ts-appbuild (adapter) +pre-route request filters ....... ts-filter (adapter wrapper) +geo lookup (single, deduped) .... ts-geo (adapter; result carried forward) +template cache lookup ........... ts-template-cache (core: publisher.rs) +origin fetch to resp headers .... ts-origin (core: publisher.rs) +EC identity KV, pre-send ........ ts-kv (core: KV abstraction) +auction wait, buffered mode ..... auction_wait_ms (row only; pre-header in this mode) + | +send_edgezero_response, immediately before into_parts(): + mark_headers_ready() snapshot (unconditional) + build AccessTelemetrySnapshot (unconditional) + append Server-Timing header (flag-gated, only on conclusively private responses) + | +headers committed; body streams + auction hold at seam .......... auction_wait_ms (row only; in-stream in this mode) + stream duration, bytes ........ stream_ms, resp_bytes (row only) + | +post-send (adapter main): + request_elapsed snapshot, then existing pull-sync, then: + sample gate -> one NDJSON row -> Tinybird Events API + bounded response await, 2xx validated +``` + +Collection is always-on and flag-free, including the `mark_headers_ready()` snapshot. +Two independent flags gate emission: the header (`observability.server_timing_enabled`) +and the telemetry row (`tinybird.access_enabled`). + +## 5. `RequestTimings` (core) + +New module `crates/trusted-server-core/src/request_timing.rs`. + +- `Phase`: a closed enum: `AppBuild`, `Filter`, `Geo`, `EcKv`, `Origin`, + `TemplateCacheLookup`, `AuctionWait`, `Stream`. Header rendering covers the first six + plus the stored total; the last two are row-only. +- Inner state: one fixed-size array of `Option` slots indexed by phase, + `t0: Instant`, `headers_ready_total: Option`, + `auction_wait_placement: Option` (`PreHeader` or `InStream`), + and `resp_bytes: Option`. Phases that repeat within a request (geo, KV) + accumulate by saturating addition into the same slot. +- `mark_headers_ready()`: stores `t0.elapsed()` once at the response-commit boundary, + unconditionally, before either emission flag is consulted. The header renders this + stored value as `ts-total`; the telemetry row reads the same stored value as + `time_elapsed_ms`. The two surfaces cannot disagree, and the row stays correct when + the header flag is off. Full request duration is captured separately as + `request_elapsed_ms`, snapshotted immediately after the body-stream drive returns and + before any other post-send work, so pull-sync and telemetry emission are never + included in it. +- Sharing: `RequestTimings` is a cheap-clone handle, `Arc>`. It crosses + three boundaries: adapter entry to core handlers, the streaming body closure (records + body-phase spans after the response object has been handed off), and the adapter's + post-send emission read. Access is exclusively `try_lock()`: a contended or + poisoned lock drops the sample immediately rather than waiting, so recording can + never delay a request. +- Recording API: `timings.record(Phase::Geo, dur)` and a scope guard + `timings.span(Phase::Origin)` that records on drop. Guards use saturating duration + math; a non-monotonic reading records zero rather than panicking. The auction-wait + recorder takes the placement explicitly so the two modes cannot be conflated. +- Rendering: `server_timing_value(&self) -> Option` produces + `ts-total;dur=41.2, ts-appbuild;dur=18.4, ts-filter;dur=9.1` with durations in + milliseconds at one decimal. Phases never recorded are omitted. Returns `None` when + `mark_headers_ready()` has not run. + +`Instant` is already used freely in the guest (`publisher.rs`, `auction/telemetry.rs`), +so no new clock abstraction is needed. + +## 6. Span taxonomy and recording sites + +| Entry | Measures | Site | +| ------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| `ts-total` | T0 to `mark_headers_ready()` at the response-commit boundary | stored snapshot | +| `ts-appbuild` | config-store open, Settings parse, orchestrator + registry + router build | Fastly `main.rs` around `open_trusted_server_config_store()` + `build_app_with_state()` | +| `ts-filter` | pre-route request filters, end to end (backend ensure, secret read, POST) | around `run_pre_route_filters` (`app.rs:751`) | +| `ts-geo` | geo hostcall (single after dedupe; accumulates if any path still repeats) | `build_ec_request_state` (`app.rs:410`) and timed finalizer fallback lookups | +| `ts-kv` | EC identity KV operations before response send (see enumeration below) | the shared KV abstraction | +| `ts-origin` | publisher backend send to response headers available (read-through cache hit or miss) | `publisher.rs` around the origin `send` | +| `ts-template-cache` | template cache `lookup_or_reserve`, including the hit-path full-body read | `publisher.rs` around the lookup | + +Naming follows the completed template-cache terminology migration (`x-ts-template-cache` +is the emitted header on `main`; `c2` naming is retired). + +`ts-kv` is instrumented by a timing decorator implementing `PlatformKvStore` that +wraps the store handed to request-scoped consumers, because no single existing +abstraction covers the taxonomy: EC graph operations go through `KvIdentityGraph` +while consent persistence uses `PlatformKvStore` directly, and graphs are constructed +independently in request setup, identify, admin lookup, batch sync, and finalization. +Every request-path graph construction receives the timed store; pull-sync explicitly +constructs its graph from an untimed store. Consent-store reads pass through the same +decorator and are timed like any other store call. Included pre-send operations: EC +generation `create_or_revive`, identify-path graph reads and evaluation, finalize-path +`ingest_eid_cookies`/`upsert_partner_ids` and withdrawal tombstones, consent-store +reads on consent routes, and batch-sync graph access when it runs before send. +Explicitly excluded: pull-sync work, which runs strictly after `send_to_client` and is +invisible to both surfaces. The decorator measures store-call latency only: no value +passing through it is read, parsed, or recorded, and the emitted surfaces carry no +consent or identity payloads. This feature therefore needs no consent gate; it is the +site measuring its own infrastructure, not processing user data. The timings handle +reaches `ec_finalize_response` inside the graph it already receives; that function +keeps the repository maximum of seven arguments and does not gain an eighth. + +Row-only fields: + +| Field | Measures | Site | +| -------------------- | --------------------------------------------------------- | ------------------------------------------------ | +| `auction_wait_ms` | wait on the dispatched auction (placement varies by mode) | seam hold (streaming) or buffered finalizer wait | +| `body_mode` | `streamed` or `buffered` response assembly | set where the response body is built | +| `stream_ms` | headers committed to last body byte | adapter around the body-stream drive | +| `resp_bytes` | bytes written to the client body | same | +| `request_elapsed_ms` | T0 to immediately after the body-stream drive returns | post-send snapshot, before pull-sync | + +Auction-wait placement is not universal. On the ordinary streaming path the wait +happens at the `` seam inside the body stream and nests inside `stream_ms`. On +buffered paths (the Fastly shared-template authorized miss, which buffers the full +transform and auction before returning a response, and every Axum response) the wait +completes before headers commit. The row therefore carries `body_mode` plus +`auction_wait_placement` (`pre_header` or `in_stream`), and derivations are +conditional: + +- `in_stream`: `stream_other_ms = greatest(coalesce(stream_ms, 0) - coalesce(auction_wait_ms, 0), 0)`. +- `pre_header`: `auction_wait_ms` joins the pre-header phase set, and `stream_other_ms = coalesce(stream_ms, 0)`. + +`unattributed_ms = greatest(coalesce(time_elapsed_ms, 0) - (coalesce(appbuild_ms, 0) + +coalesce(filter_ms, 0) + coalesce(geo_ms, 0) + coalesce(kv_ms, 0) + +coalesce(origin_ms, 0) + coalesce(template_cache_ms, 0) + pre-header auction wait), 0)`. +Every phase column is nullable, so every query-time formula wraps each term in +`coalesce(column, 0)` and every subtraction in `greatest(..., 0)`; query tests cover +sparse phase combinations. + +## 7. Freeze point and header emission + +The freeze-and-emit point is `send_edgezero_response` (Fastly `main.rs`), immediately +before `response.into_parts()`. This is the single choke point every send path shares, +and it runs after everything that can still mutate the response: the router middleware, +entry-point finalize (`apply_finalize_headers`, asset-policy reapplication), EC +finalization and its KV work, and terminal filter/privacy effects. +`apply_finalize_headers` itself does not emit; the `HEADER_X_TS_FINALIZED` sentinel +marks middleware finalization, not header commitment, and must not be treated as the +timing boundary. + +At the freeze point, in order: `mark_headers_ready()` (unconditional), the +`AccessTelemetrySnapshot` build (unconditional, section 10), then, gated on +`observability.server_timing_enabled`, append one `Server-Timing` header from +`server_timing_value()`. Append semantics, never insert: an origin-supplied +Server-Timing survives, and the fronting delivery layer's own entries (`time-elapsed`, +`hit-state`) are additive per the header's list semantics. + +Header emission is conservative: it happens only when the response is conclusively +non-storable by any shared cache, meaning `Cache-Control` contains `private` or +`no-store` (the existing `cache_control_headers_are_private_or_no_store` predicate). +Anything else, including bare `max-age`, `s-maxage` without `private`, +heuristically-cacheable responses with no cache header at all, and anything a fronting +cache override might store, emits no header, because a stored object would replay one +request's timings for its full lifetime. The long-lived immutable `tsjs` asset route +is the concrete excluded case. The snapshot and the telemetry row are unaffected by +this skip, so excluded routes still report through Tinybird. + +The Axum adapter applies the same emission rule at its terminal point before response +serialization, with adapter-specific phase semantics (section 8a). + +## 8. Geo lookup dedupe (rider) + +Today every dispatched request pays two geo hostcalls for one answer: request-phase in +`build_ec_request_state` (`app.rs:410`) and response-phase in +`FinalizeResponseMiddleware` (`middleware.rs:83`, alternate site `main.rs:285`). + +Plain request extensions cannot carry the result out: the middleware moves the request +context into `next.run(ctx)` and holds only the response afterward. The resolved geo +travels on a dedicated `GeoLookupState` response extension, attached on every exit +path that attempted a lookup, including the asset fallback, which runs +`build_ec_request_state` and then returns without `EcFinalizeState` (which is why +`EcFinalizeState` is not an acceptable carrier). States: `NotAttempted`, +`Attempted(None)` (lookup ran and failed, do not retry), and `Resolved(GeoInfo)`. The +finalize path consumes the carried value and performs a live lookup only in the +`NotAttempted` state; those legitimate fallback lookups (admin, batch, error paths) +are themselves timed into `ts-geo` so degraded geo cannot hide inside +`unattributed_ms`. The 401 rule (`resolve_geo_for_response` skips lookup for +unauthorized responses) is preserved. + +## 8a. Adapter phase semantics + +The Fastly adapter is the reference implementation of the taxonomy. Axum differs +structurally and its emissions are defined accordingly rather than pretending parity: + +- `ts-appbuild` is absent: Axum builds application state once at startup. +- `body_mode` is always `buffered`: the Axum HTTP client buffers upstream bodies, so + `stream_ms` measures buffered-body write-out and `auction_wait_placement` is always + `pre_header`. +- The freeze point is an outer service wrapper around the `RouterService` inside + `AxumDevServer`, not router middleware: router-generated 404/405 responses bypass + router middleware, and middleware returns before Axum serializes the body. The + wrapper sees every response including router-generated ones; `/health` is excluded + by path match inside the wrapper. +- Axum emits the header only; no Tinybird rows in v1 (unchanged). + +Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). + +## 9. Access telemetry row + +Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. + +Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the +`mark_headers_ready()` snapshot), `sample_rate`, `event_date`, 30-day TTL. + +Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put +EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality +and user-generated content (search terms, usernames, emails in slugs). Replaced by +`route_template`: + +- Named routes: the matched route-table pattern verbatim, parameters left as + placeholders. +- Publisher fallback: a coarse fixed template, `/` plus the first path segment + restricted to a bounded allowlisted charset, plus `/*` when deeper (for example + `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it + redacts long tokens but preserves short identifiers and arbitrary slugs. +- Tests are adversarial, not just the happy path: a literal EC identifier on the admin + route, an email address in a path segment, search-term-shaped segments, and + overlong segments must all normalize to bounded, content-free templates. + +Added columns (all dimension columns non-nullable with an `unknown` sentinel, because +ClickHouse sorting keys cannot contain nullable columns): + +``` +`service_id` LowCardinality(String), -- FASTLY_SERVICE_ID; immutable deployment identity +`publisher_domain` LowCardinality(String), -- matches auction schema +`env` LowCardinality(String), -- adapter-derived: production | staging | unknown +`route_class` LowCardinality(String), -- publisher_html | tsjs | integration_proxy | ec | auction_api | other +`route_template` String, -- bounded, normalized; replaces path +`body_mode` LowCardinality(String), -- streamed | buffered +`auction_wait_placement` LowCardinality(String), -- pre_header | in_stream | none +`appbuild_ms` Nullable(UInt32), +`filter_ms` Nullable(UInt32), +`geo_ms` Nullable(UInt32), +`kv_ms` Nullable(UInt32), +`origin_ms` Nullable(UInt32), +`template_cache_ms` Nullable(UInt32), +`auction_wait_ms` Nullable(UInt32), +`stream_ms` Nullable(UInt32), +`request_elapsed_ms` Nullable(UInt32), +`resp_bytes` Nullable(UInt64), +`template_cache_state` LowCardinality(String), -- from the typed response extension, not the public header +`country` LowCardinality(String), +`ts_version` LowCardinality(String), +`pop` LowCardinality(String) -- FASTLY_POP, 'unknown' when absent +``` + +The matched route pattern does not survive dispatch today, so a typed +`RouteMetadata` response extension carries `route_class` and `route_template`: each +named-route handler wrapper attaches its route-table pattern verbatim (handlers +serving multiple patterns attach the one that matched), and the fallback and tsjs +handlers attach their class plus the coarse template. The freeze point consumes the +extension; nothing reconstructs routes from a handler enum or path regex. + +Typed sources only: `env` is adapter-owned, derived from the same Fastly +`FASTLY_IS_STAGING` input that drives `x-ts-env` (`Settings` has no environment +field and does not gain one). `template_cache_state` comes from a typed response +extension, not the `x-ts-template-cache` header (operator-configured response +headers can override managed headers): the currently private +`TemplateCacheResponseState` in `publisher.rs` becomes a typed response extension, +and every state transition sets the managed header and the extension together so +the two can never drift. `service_id` and `pop` come from the Fastly environment. `cache_state` +from the reserved schema is dropped: the guest cannot observe the fronting cache, and +guest-visible cache behavior is already carried by `template_cache_state` and +`origin_ms`. Rows exist only for guest-handled requests; fronting-cache hits are +invisible by construction and the dashboard documentation says so. + +Sorting key: `(event_date, service_id, publisher_domain, env, route_class, pop, +status)`. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query +also carries an `event_date` predicate so the primary index prunes; rollout validates +the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the +reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the +reserved datasource was ever deployed to the remote workspace; if it was, this schema +ships as a versioned replacement datasource with a cutover, not an in-place edit. + +## 10. Emission mechanics + +- `AccessTelemetrySnapshot`: built unconditionally at the freeze point, before + `into_parts()` consumes the response. It captures method, status, route metadata + (from the `RouteMetadata` extension), and typed dimension states (`env`, + `template_cache_state`, geo country). It exists because nothing else survives to + post-send on every path: the request is consumed by dispatch, the response by + `into_parts()`, and `EcFinalizeState` is absent on asset, admin, and error paths. +- The emitter's transport context is adapter-owned and route-independent: the + Events API target (backend spec, secret store name, dataset, token secret, sample + rate) derives from settings once at entry in `main.rs`, and the HTTP client is the + adapter's stateless platform client. Asset, admin, and error responses therefore + emit without `RuntimeServices` or `EcFinalizeState`. +- `send_edgezero_response` returns a delivery outcome instead of `()`, with + per-mode semantics because the two body paths observe different things. Streamed + bodies: a counting writer reports bytes written and distinguishes complete, + partial (truncated), and error outcomes. Buffered bodies: `send_to_client()` + returns no delivery result, so the byte count is captured from the body length + before the send and the outcome is complete-on-return with no partial detection; + `body_mode` in the row keeps the two regimes distinguishable in analysis. +- Ordering after the body-stream drive returns: snapshot `request_elapsed_ms` first, + run the existing pull-sync dispatch unchanged, then telemetry emission last, so + pull-sync is never delayed behind the ingest await and never included in + `request_elapsed_ms`. +- Sampling: uniform per-request decision against `tinybird.access_sample_rate`. No + client stickiness. Sampled-out requests are silent; every other drop (row build + failure, send failure, non-2xx) logs one warning naming the reason. There is no + cross-request warning suppression (per-request isolates hold no shared state); the + overload controls are the 2 s bounded await, the single-warning-per-request cap, and + `access_sample_rate` pushed down by config as the operational abort lever. Ingest + health is monitored from the Tinybird side via ingestion freshness on the + datasource, which catches quarantine and schema rejection that per-request warnings + cannot. +- Transport: one NDJSON row to the Tinybird Events API: same `api_host`, reserved + `access_dataset` and `access_token_secret`, 2 s first-byte and between-bytes + timeouts, `max_body_bytes` guard, no retry. +- Delivery confirmation: unlike the auction sink, which starts `send_async` and drops + the pending response (it runs before delivery completes and cannot afford to wait), + the access emitter runs after the client has the full response and therefore awaits + the bounded ingest response and validates 2xx. A non-2xx or timeout logs a warning + with the status. +- Budget: at `access_sample_rate = 1.0` this adds one backend request per request to + the service, after delivery; during a Tinybird outage each such request holds its + sandbox for up to the bounded timeout. The sample rate is the budget control; 1.0 is + a diagnosis setting, not a steady state, and rollout treats sustained emission + warnings as the signal to dial it down. +- Axum adapter: emits the header only; no Tinybird rows in v1. + +## 11. Dashboard and query model + +No endpoint pipe in v1. Grafana queries `access_logs_raw` directly through the +ClickHouse connector with `$__timeFilter(event_ts)` plus an `event_date` predicate, +matching the auction dashboards. + +Dashboard: a new standalone `grafana/dashboards/edge-performance.json` in the +telemetry repo (`trusted-server-tinybird`), performance only, no panels shared with +the revenue and auction dashboards. Panels: + +- Phase percentiles (p50/p95/p99) by `route_class`, per phase column. +- Stacked phase breakdown over time using the non-overlapping set: `appbuild_ms`, + `filter_ms`, `geo_ms`, `kv_ms`, `origin_ms`, `template_cache_ms`, pre-header + auction wait (where `auction_wait_placement = 'pre_header'`), and derived + `unattributed_ms`. In-stream auction wait and derived `stream_other_ms` chart in a + separate body-phase panel and never stack with pre-header phases. +- PoP split, `ts_version` overlay, template-cache state rates. +- Stall panel: rows with `request_elapsed_ms > 500` (post-body total, so body-only + stalls are caught) grouped by dominant phase, where `unattributed_ms` competes as a + phase so the panel cannot confidently blame a small measured span while most time is + uninstrumented. + +All derivations use the `coalesce`/`greatest` forms from section 6; query tests cover +sparse phase combinations and both `auction_wait_placement` modes. + +Sampling semantics for every aggregate: `sample_rate` must be operationally stable +within any queried window. Quantile panels filter strictly to a single `sample_rate` +value. Volume panels weight each row by `1.0 / sample_rate` (the inverse-probability +estimator is `sum(1.0 / sample_rate)` over emitted rows; `count() / rate` is valid +only when the query is already filtered to one rate). Pooled unweighted quantiles +across a rate change are documented as invalid. + +## 12. Config surface + +```toml +[observability] +# Append TS phase timings to the Server-Timing response header. +server_timing_enabled = false # example default +``` + +New `ObservabilitySettings` struct with the single boolean, default off, standard +environment override (`TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED`). +Collection has no flag: the flags gate the two emission surfaces independently. + +Tinybird flag structure: `tinybird.enabled` today arms the auction sink by itself, so +"enable Tinybird for access telemetry" would silently enable auction emission too. The +master flag is demoted to transport-only (host, store, credentials), and each emitter +gets its own switch: a new `tinybird.auction_enabled` defaulting to `true` (preserving +current behavior for existing configs) and the reserved `tinybird.access_enabled` +defaulting to `false`. A settings test locks the decoupling in both directions. + +Validation when `access_enabled = true`: `tinybird.enabled`, non-empty `api_host`, +non-empty `secret_store`, `access_dataset`, and `access_token_secret`, a positive +`max_body_bytes`, and `access_sample_rate > 0`. An armed-but-silent configuration +(`access_enabled = true`, `access_sample_rate = 0`) is a configuration error, not a +valid state; disabling is done with the flag, not the rate. + +Rollback and compatibility, because `Settings` is `deny_unknown_fields`: + +- Deployment order is binary first, config second. Rollback order is config first + (remove the `[observability]` table and any new tinybird keys), binary second. A + config containing the new fields must never be pushed while a pre-observability + binary can still run. +- Config serialization omits the table when it equals the default, so round-tripping a + config through tooling does not inject a field an older binary rejects. A + compatibility test asserts the serialized default config parses under the previous + schema. +- The environment-variable overlay cannot create a missing leaf, so the key ships + present-but-false in the base operator TOML (the same pattern the GPT integration + documents in `trusted-server.example.toml`) and is flipped by config push. + +## 13. Error handling + +- Recording is infallible: saturating math, lock-failure drops the sample, no panics. +- Header rendering failure (defensive `HeaderValue::from_str` error) logs and skips + the header. +- Row emission failure logs one warning naming the reason and drops the row. The + response has already been delivered; there is nothing to degrade. + +## 14. Testing + +- Core unit tests: phase accumulation, saturating math, `mark_headers_ready()` + idempotence and both-surface consistency, render format (one decimal, omission of + unrecorded phases), row serialization shape, auction-wait placement recording. +- Adapter tests (Fastly via Viceroy, Axum native): header present and well-formed on a + conclusively-private publisher route with the flag on; absent with the flag off; + absent on the shared-cacheable tsjs route and on a bare `max-age` response with the + flag on; exactly one TS-owned metric set (a single `ts-total`) with every + pre-existing Server-Timing value preserved, across all send paths; `ts-kv` captures + EC finalize work (proving the freeze point sits after it). +- Body-mode tests: ordinary streaming (in-stream wait nested in `stream_ms`), + Fastly shared-template authorized miss (buffered, pre-header wait), and Axum + (always buffered), each asserting placement and non-negative derivations. +- Geo dedupe: finalize consumes `Resolved`; no retry on `Attempted(None)`; live + lookup only on `NotAttempted`; fallback lookups timed into `ts-geo`; asset-fallback + path carries `GeoLookupState` without `EcFinalizeState`; 401 skip preserved. +- Route template: adversarial normalization tests (literal EC identifier on the admin + route, email address in a segment, search-term segments, overlong segments) all + producing bounded content-free templates. +- Settings: the access validation matrix including the armed-but-silent rejection; + auction/access flag decoupling in both directions; the former rejection test becomes + the wiring test; the serialized-default-config compatibility test against the + previous schema. +- Sink tests: `RecordingHttpClient` pattern; assert URI, NDJSON body shape, token + header, 2xx validation and warning on non-2xx, skip when sampled out, ordering after + pull-sync. +- Query tests: derivation formulas against sparse rows and both placements. + +## 15. Rollout and verification + +1. Land collection + freeze point + header emission behind the flag, off everywhere. + Full CI gate. +2. Staging deploy with the flag on. Delivery-layer verification is two-sided: a + pass-through request confirming the appended Server-Timing survives the fronting + VCL, and a MISS-then-HIT replay against a cacheable route confirming no stale + timing header is ever served from cache. Fallback if the VCL clobbers the header: a + one-line VCL change on the delivery service, or mirroring the value to + `x-ts-timing` while that lands. +3. Production flag on. Confirm + `performance.getEntriesByType('navigation')[0].serverTiming` shows `ts-*` entries + in a real browser session, and separately confirm what the publisher's RUM tooling + actually collects; the publisher monitoring extension renders it only after a small + change on their side. +4. Verify whether `access_logs_raw` exists in the remote Tinybird workspace. If yes, + ship the schema as a versioned replacement with cutover; if no, edit in place. + Validate the dashboard panel queries with `EXPLAIN` against the sorting key. Then + land the row schema, sink, and settings changes; sample at 1.0 during stall + diagnosis with ingestion-freshness monitoring on the datasource; then the + dashboard. +5. Success criterion: the next stall window is attributable from one response header + or one dashboard query, with no live probing session. + +## 16. Overhead + +Roughly ten monotonic clock reads, two stored snapshots, and one ~130-byte header per +request; one sampled HTTP POST with a bounded await after the response has fully +streamed. No allocation in the hot path beyond the one `Arc` at entry, the +`AccessTelemetrySnapshot` at the freeze point, and the rendered header string. + +## 17. Decisions and open questions + +- **Public exposure is a decision, not an open question.** The header is all-traffic + when enabled. Rationale: values are durations only; the delivery layer already + exposes `hit-state` and `time-elapsed` publicly on every response; filter vendor + identity is masked; emission is restricted to conclusively-private responses so no + cache can replay stale timings. Revisit (quantization or gating) only if a concrete + abuse surfaces. +- The fronting delivery layer's Server-Timing pass-through is unverified until the + first staging deploy (step 2). This is the only known external dependency. +- Body-phase capture threads the timings handle into the streaming closure in + `publisher.rs`; the exact seam is an implementation-plan detail, with the constraint + that a dropped handle (error paths, early client disconnect) must still yield a + valid row with null body-phase fields and a recorded delivery outcome. +- The stall window itself remains unattributed until this ships. If it recurs first, + the bisection runbook from 2026-08-21 (cookie-free curl UA request, static-asset + path versus HTML path) is the fallback. From 052eadad04af7964933178d4482881302de0c1b5 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 19:31:18 -0500 Subject: [PATCH 002/104] Add RequestTimings phase collection and Server-Timing rendering --- crates/trusted-server-core/src/lib.rs | 1 + .../trusted-server-core/src/request_timing.rs | 437 ++++++++++++++++++ 2 files changed, 438 insertions(+) create mode 100644 crates/trusted-server-core/src/request_timing.rs diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 48e92faed..b8a1a5718 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -61,6 +61,7 @@ pub mod proxy; pub mod publisher; pub mod redacted; pub mod request_signing; +pub mod request_timing; pub mod response_privacy; pub mod rsc_flight; pub(crate) mod s3_sigv4; diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs new file mode 100644 index 000000000..b0cb7b0dd --- /dev/null +++ b/crates/trusted-server-core/src/request_timing.rs @@ -0,0 +1,437 @@ +//! Per-request phase timing collection and Server-Timing rendering. +//! +//! Collection is always-on and infallible: saturating math, lock failure +//! drops the sample, no panics. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +/// Number of [`Phase`] variants; sizes the fixed-slot duration array in +/// [`Inner`]. +const PHASE_COUNT: usize = 8; + +/// A distinct stage of request handling that duration can be attributed to. +/// +/// Variants map to fixed slots in [`RequestTimings`], in declaration order. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Phase { + /// Time spent constructing the app/handler before request processing + /// begins. + AppBuild, + /// Time spent in request/response filtering (e.g. HTML rewriting). + Filter, + /// Time spent resolving geographic signals for the request. + Geo, + /// Time spent reading or writing the Edge Cookie key-value store. + EcKv, + /// Time spent waiting on the origin fetch. + Origin, + /// Time spent looking up a cached template. + TemplateCacheLookup, + /// Time spent waiting on the auction. Row-only: never rendered as a + /// `Server-Timing` header entry. + AuctionWait, + /// Time spent streaming the response body. Row-only: never rendered as + /// a `Server-Timing` header entry. + Stream, +} + +impl Phase { + /// Maps this variant to its fixed slot in the [`Inner::phases`] array. + fn index(self) -> usize { + match self { + Self::AppBuild => 0, + Self::Filter => 1, + Self::Geo => 2, + Self::EcKv => 3, + Self::Origin => 4, + Self::TemplateCacheLookup => 5, + Self::AuctionWait => 6, + Self::Stream => 7, + } + } + + /// `Server-Timing` header entry name; row-only phases return `None`. + fn header_name(self) -> Option<&'static str> { + match self { + Self::AppBuild => Some("ts-appbuild"), + Self::Filter => Some("ts-filter"), + Self::Geo => Some("ts-geo"), + Self::EcKv => Some("ts-kv"), + Self::Origin => Some("ts-origin"), + Self::TemplateCacheLookup => Some("ts-template-cache"), + Self::AuctionWait | Self::Stream => None, + } + } +} + +/// Where in the response the auction wait occurred. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuctionWaitPlacement { + /// The auction was awaited before response headers were sent. + PreHeader, + /// The auction was awaited while streaming the response body. + InStream, +} + +/// Mutable state behind [`RequestTimings`], guarded by a [`Mutex`]. +struct Inner { + /// Instant the request started; the reference point for elapsed marks. + t0: Instant, + /// Accumulated duration per [`Phase`], indexed by [`Phase::index`]. + phases: [Option; PHASE_COUNT], + /// Elapsed time at the first [`RequestTimings::mark_headers_ready`] call. + headers_ready_total: Option, + /// Elapsed time at the first [`RequestTimings::mark_request_elapsed`] + /// call. + request_elapsed: Option, + /// Placement recorded by the most recent + /// [`RequestTimings::record_auction_wait`] call. + auction_wait_placement: Option, + /// Response body size in bytes, set via + /// [`RequestTimings::set_resp_bytes`]. + resp_bytes: Option, +} + +/// Per-request phase timing collector. +/// +/// Cheap to clone (an [`Arc`] handle) and safe to share across threads and +/// async tasks handling the same request. Every method is infallible: lock +/// contention or poisoning silently drops the sample rather than blocking or +/// panicking. +#[derive(Clone)] +pub struct RequestTimings(Arc>); + +impl RequestTimings { + /// Starts a new collector with its clock reference (`t0`) set to now. + #[must_use] + pub fn new() -> Self { + Self(Arc::new(Mutex::new(Inner { + t0: Instant::now(), + phases: [None; PHASE_COUNT], + headers_ready_total: None, + request_elapsed: None, + auction_wait_placement: None, + resp_bytes: None, + }))) + } + + /// Accumulates `dur` into `phase`'s running total. + /// + /// Repeated calls for the same phase saturate-add rather than overwrite. + /// Drops the sample silently on lock contention or poisoning. + pub fn record(&self, phase: Phase, dur: Duration) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + let index = phase.index(); + let accumulated = inner.phases[index] + .unwrap_or(Duration::ZERO) + .saturating_add(dur); + inner.phases[index] = Some(accumulated); + } + + /// Records an auction wait duration under [`Phase::AuctionWait`] and + /// stores its placement. + /// + /// Drops the sample silently on lock contention or poisoning. + pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + let index = Phase::AuctionWait.index(); + let accumulated = inner.phases[index] + .unwrap_or(Duration::ZERO) + .saturating_add(dur); + inner.phases[index] = Some(accumulated); + inner.auction_wait_placement = Some(placement); + } + + /// Starts a guard that records elapsed time into `phase` when dropped. + #[must_use] + pub fn span(&self, phase: Phase) -> PhaseSpan { + PhaseSpan { + timings: self.clone(), + phase, + started: Instant::now(), + } + } + + /// Stamps the elapsed time since `t0` as `headers_ready_total`, the + /// first time this is called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_headers_ready(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.headers_ready_total.is_none() { + inner.headers_ready_total = Some(inner.t0.elapsed()); + } + } + + /// Stamps the elapsed time since `t0` as `request_elapsed`, the first + /// time this is called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_request_elapsed(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.request_elapsed.is_none() { + inner.request_elapsed = Some(inner.t0.elapsed()); + } + } + + /// Records the response body size in bytes. + /// + /// Drops the sample silently on lock contention or poisoning. + pub fn set_resp_bytes(&self, bytes: u64) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + inner.resp_bytes = Some(bytes); + } + + /// Renders a `Server-Timing` header value, or `None` before + /// [`RequestTimings::mark_headers_ready`] has been called. + /// + /// `ts-total` is rendered first from `headers_ready_total`, followed by + /// the recorded header-bearing phases (`ts-appbuild`, `ts-filter`, + /// `ts-geo`, `ts-kv`, `ts-origin`, `ts-template-cache`) in enum + /// declaration order. Unrecorded phases are omitted. Durations are + /// rendered as milliseconds with one decimal place. Drops the sample + /// silently (returning `None`) on lock contention or poisoning. + #[must_use] + pub fn server_timing_value(&self) -> Option { + let inner = self.0.try_lock().ok()?; + let total = inner.headers_ready_total?; + let mut entries = vec![format_entry("ts-total", total)]; + for phase in HEADER_PHASES { + let Some(name) = phase.header_name() else { + continue; + }; + if let Some(dur) = inner.phases[phase.index()] { + entries.push(format_entry(name, dur)); + } + } + Some(entries.join(", ")) + } + + /// Captures the current state as a [`TimingSnapshot`]. + /// + /// Returns an all-`None` snapshot on lock contention or poisoning, + /// consistent with the infallibility of every other method. + #[must_use] + pub fn snapshot(&self) -> TimingSnapshot { + let Ok(inner) = self.0.try_lock() else { + return TimingSnapshot::default(); + }; + TimingSnapshot { + time_elapsed_ms: duration_ms(inner.headers_ready_total), + request_elapsed_ms: duration_ms(inner.request_elapsed), + appbuild_ms: duration_ms(inner.phases[Phase::AppBuild.index()]), + filter_ms: duration_ms(inner.phases[Phase::Filter.index()]), + geo_ms: duration_ms(inner.phases[Phase::Geo.index()]), + kv_ms: duration_ms(inner.phases[Phase::EcKv.index()]), + origin_ms: duration_ms(inner.phases[Phase::Origin.index()]), + template_cache_ms: duration_ms(inner.phases[Phase::TemplateCacheLookup.index()]), + auction_wait_ms: duration_ms(inner.phases[Phase::AuctionWait.index()]), + stream_ms: duration_ms(inner.phases[Phase::Stream.index()]), + auction_wait_placement: inner.auction_wait_placement, + resp_bytes: inner.resp_bytes, + } + } +} + +impl Default for RequestTimings { + fn default() -> Self { + Self::new() + } +} + +/// The header-bearing phases (see [`Phase::header_name`]), in the enum +/// declaration order [`RequestTimings::server_timing_value`] renders them in. +const HEADER_PHASES: [Phase; 6] = [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, +]; + +/// Formats one `Server-Timing` entry as `name;dur=`. +fn format_entry(name: &str, dur: Duration) -> String { + format!("{name};dur={:.1}", dur.as_secs_f64() * 1000.0) +} + +/// Converts a recorded [`Duration`] to whole milliseconds, saturating to +/// [`u32::MAX`] instead of overflowing. +fn duration_ms(dur: Option) -> Option { + dur.map(|dur| u32::try_from(dur.as_millis()).unwrap_or(u32::MAX)) +} + +/// RAII guard returned by [`RequestTimings::span`] that records its own +/// elapsed lifetime into the originating phase when dropped. +pub struct PhaseSpan { + /// The collector this span reports into on drop. + timings: RequestTimings, + /// The phase this span's elapsed time is recorded under. + phase: Phase, + /// The instant the span was created. + started: Instant, +} + +impl Drop for PhaseSpan { + fn drop(&mut self) { + self.timings.record(self.phase, self.started.elapsed()); + } +} + +/// A point-in-time, plain-data view of a [`RequestTimings`] collector. +/// +/// All durations are whole milliseconds; unrecorded phases are `None`. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct TimingSnapshot { + /// Elapsed time from request start to + /// [`RequestTimings::mark_headers_ready`], in milliseconds. + pub time_elapsed_ms: Option, + /// Elapsed time from request start to + /// [`RequestTimings::mark_request_elapsed`], in milliseconds. + pub request_elapsed_ms: Option, + /// Accumulated [`Phase::AppBuild`] duration, in milliseconds. + pub appbuild_ms: Option, + /// Accumulated [`Phase::Filter`] duration, in milliseconds. + pub filter_ms: Option, + /// Accumulated [`Phase::Geo`] duration, in milliseconds. + pub geo_ms: Option, + /// Accumulated [`Phase::EcKv`] duration, in milliseconds. + pub kv_ms: Option, + /// Accumulated [`Phase::Origin`] duration, in milliseconds. + pub origin_ms: Option, + /// Accumulated [`Phase::TemplateCacheLookup`] duration, in milliseconds. + pub template_cache_ms: Option, + /// Accumulated [`Phase::AuctionWait`] duration, in milliseconds. + pub auction_wait_ms: Option, + /// Accumulated [`Phase::Stream`] duration, in milliseconds. + pub stream_ms: Option, + /// Placement recorded by the most recent + /// [`RequestTimings::record_auction_wait`] call. + pub auction_wait_placement: Option, + /// Response body size in bytes, set via + /// [`RequestTimings::set_resp_bytes`]. + pub resp_bytes: Option, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn render_omits_unrecorded_phases_and_orders_total_first() { + let timings = RequestTimings::new(); + timings.record(Phase::Filter, Duration::from_micros(9_100)); + timings.mark_headers_ready(); + let value = timings + .server_timing_value() + .expect("should render after mark_headers_ready"); + assert!( + value.starts_with("ts-total;dur="), + "should lead with ts-total: {value}" + ); + assert!( + value.contains("ts-filter;dur=9.1"), + "should render one decimal: {value}" + ); + assert!( + !value.contains("ts-geo"), + "should omit unrecorded phases: {value}" + ); + } + + #[test] + fn render_returns_none_before_headers_ready() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(1)); + assert!( + timings.server_timing_value().is_none(), + "should require the snapshot" + ); + } + + #[test] + fn repeated_phases_accumulate_saturating() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(2)); + timings.record(Phase::Geo, Duration::from_millis(3)); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.geo_ms, Some(5), "should accumulate repeats"); + } + + #[test] + fn mark_headers_ready_is_first_call_wins() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let first = timings.snapshot().time_elapsed_ms; + std::thread::sleep(Duration::from_millis(5)); + timings.mark_headers_ready(); + assert_eq!( + timings.snapshot().time_elapsed_ms, + first, + "should not restamp" + ); + } + + #[test] + fn span_guard_records_on_drop() { + let timings = RequestTimings::new(); + { + let _span = timings.span(Phase::Origin); + std::thread::sleep(Duration::from_millis(2)); + } + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.expect("should record on drop") >= 1, + "should measure elapsed span time" + ); + } + + #[test] + fn auction_wait_records_placement() { + let timings = RequestTimings::new(); + timings.record_auction_wait(AuctionWaitPlacement::PreHeader, Duration::from_millis(40)); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.auction_wait_ms, Some(40), "should record wait"); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "should record placement" + ); + } + + #[test] + fn rendered_names_never_include_vendor_terms() { + let timings = RequestTimings::new(); + for phase in [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, + ] { + timings.record(phase, Duration::from_millis(1)); + } + timings.mark_headers_ready(); + let value = timings.server_timing_value().expect("should render"); + assert!( + !value.to_ascii_lowercase().contains("datadome"), + "should mask vendors" + ); + } +} From 7505fb888e4bacf86504cf3a73a34c3502cf5d17 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 20:27:41 -0500 Subject: [PATCH 003/104] Add observability settings and decouple tinybird access and auction emission --- .../src/tinybird.rs | 41 ++++- crates/trusted-server-core/src/settings.rs | 154 ++++++++++++++++-- trusted-server.example.toml | 29 ++++ 3 files changed, 205 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index f2df61744..08b5811fc 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -22,9 +22,14 @@ const TINYBIRD_BETWEEN_BYTES_TIMEOUT: Duration = Duration::from_secs(2); const TINYBIRD_MAX_ROWS_PER_AUCTION_BATCH: usize = 512; /// Build the configured auction telemetry sink. +/// +/// Auction emission requires both the Tinybird master toggle +/// (`tinybird.enabled`) and the auction-specific toggle +/// (`tinybird.auction_enabled`), so access-log telemetry can be enabled +/// independently without also emitting auction events. #[must_use] pub(crate) fn auction_sink_from_settings(settings: &Settings) -> Arc { - if settings.tinybird.enabled { + if settings.tinybird.enabled && settings.tinybird.auction_enabled { Arc::new(FastlyTinybirdAuctionTelemetrySink::new( settings.tinybird.clone(), )) @@ -443,6 +448,7 @@ mod tests { fn enabled_config() -> TinybirdSettings { TinybirdSettings { enabled: true, + auction_enabled: true, api_host: "api.us-east.aws.tinybird.co".to_owned(), secret_store: "ts_secrets".to_owned(), auction_dataset: "auction_events_raw".to_owned(), @@ -455,6 +461,39 @@ mod tests { } } + #[test] + fn sink_from_settings_disables_when_auction_enabled_is_false() { + let settings = Settings { + tinybird: TinybirdSettings { + auction_enabled: false, + ..enabled_config() + }, + ..Settings::default() + }; + + let sink = auction_sink_from_settings(&settings); + + assert!( + !sink.is_enabled(), + "auction telemetry should stay off when auction_enabled is false, even if tinybird.enabled is true" + ); + } + + #[test] + fn sink_from_settings_enables_when_both_toggles_are_true() { + let settings = Settings { + tinybird: enabled_config(), + ..Settings::default() + }; + + let sink = auction_sink_from_settings(&settings); + + assert!( + sink.is_enabled(), + "auction telemetry should be on when both tinybird.enabled and tinybird.auction_enabled are true" + ); + } + #[test] fn events_uri_targets_dataset_on_region_host() { assert_eq!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index ddc8ac612..bcb467fb1 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1704,9 +1704,16 @@ impl Proxy { /// Direct Tinybird Events API telemetry configuration. #[derive(Debug, Clone, Deserialize, Serialize)] pub struct TinybirdSettings { - /// Master enablement for auction telemetry ingestion. + /// Master enablement for Tinybird telemetry. Required by both auction and + /// access-log emission; each is independently toggled below. #[serde(default)] pub enabled: bool, + /// Emit auction telemetry when `enabled`. Defaults to `true` so existing + /// configs preserve their current auction-emission behavior after + /// upgrading; set `false` to silence auction events while keeping + /// `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). + #[serde(default = "default_true")] + pub auction_enabled: bool, /// Regional Tinybird API host, without scheme or path. #[serde(default)] pub api_host: String, @@ -1719,19 +1726,26 @@ pub struct TinybirdSettings { /// Secret key containing the auction datasource APPEND token. #[serde(default = "default_tinybird_auction_token_secret")] pub auction_token_secret: String, - /// Reserved for future access-log telemetry. + /// Emit access-log telemetry when `enabled`, independent of + /// `auction_enabled`. /// - /// `true` is rejected until an access-log emitter is wired, so operators - /// cannot enable a setting that silently emits nothing. + /// `true` requires `enabled`, non-empty `api_host`/`secret_store`/ + /// `access_dataset`/`access_token_secret`, `max_body_bytes > 0`, and + /// `access_sample_rate > 0.0`. This prevents an armed-but-silent sampler + /// that enables the flag but emits nothing. #[serde(default)] pub access_enabled: bool, - /// Future access-log Events API datasource name. + /// Access-log Events API datasource name. Required non-empty when + /// `access_enabled`. #[serde(default = "default_tinybird_access_dataset")] pub access_dataset: String, - /// Future Secret Store key containing the access-log datasource APPEND token. + /// Secret Store key containing the access-log datasource APPEND token. + /// Required non-empty when `access_enabled`. #[serde(default = "default_tinybird_access_token_secret")] pub access_token_secret: String, - /// Future fraction of requests to emit for optional access telemetry. + /// Fraction of requests to emit for access telemetry. Must be greater + /// than `0.0` when `access_enabled`, so an operator cannot enable access + /// telemetry while sampling it away entirely. #[serde(default)] pub access_sample_rate: f64, /// Defensive maximum NDJSON body size for one Events API request. @@ -1767,6 +1781,7 @@ impl Default for TinybirdSettings { fn default() -> Self { Self { enabled: false, + auction_enabled: default_true(), api_host: String::new(), secret_store: default_tinybird_secret_store(), auction_dataset: default_tinybird_auction_dataset(), @@ -1790,6 +1805,12 @@ impl TinybirdSettings { self.access_token_secret = self.access_token_secret.trim().to_owned(); } + /// Validate this settings block, including the access-telemetry matrix: + /// `access_enabled` requires `enabled`, a non-empty `api_host`, + /// `secret_store`, `access_dataset`, and `access_token_secret`, a + /// `max_body_bytes` above the defensive floor enforced below, and an + /// `access_sample_rate` greater than `0.0`. Auction emission is + /// independently gated by `auction_enabled` and validated the same way. fn prepare_runtime(&mut self) -> Result<(), Report> { self.normalize(); if !(0.0..=1.0).contains(&self.access_sample_rate) { @@ -1802,9 +1823,9 @@ impl TinybirdSettings { message: "tinybird.max_body_bytes must be at least 1024".to_owned(), })); } - if self.access_enabled { + if self.access_enabled && !self.enabled { return Err(Report::new(TrustedServerError::Configuration { - message: "tinybird.access_enabled is reserved for future access-log telemetry; no emitter is currently wired".to_owned(), + message: "tinybird.access_enabled requires tinybird.enabled".to_owned(), })); } if !self.enabled { @@ -1818,10 +1839,19 @@ impl TinybirdSettings { .to_owned(), })); } - if self.enabled { + if self.auction_enabled { validate_tinybird_dataset(&self.auction_dataset, "tinybird.auction_dataset")?; validate_tinybird_secret(&self.auction_token_secret, "tinybird.auction_token_secret")?; } + if self.access_enabled { + validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; + validate_tinybird_secret(&self.access_token_secret, "tinybird.access_token_secret")?; + if self.access_sample_rate <= 0.0 { + return Err(Report::new(TrustedServerError::Configuration { + message: "tinybird.access_sample_rate must be > 0 when tinybird.access_enabled is true".to_owned(), + })); + } + } Ok(()) } } @@ -2588,6 +2618,29 @@ pub enum AuctionDebugCommentFormat { Pretty, } +/// Request-observability toggles exposed to operators. +/// +/// The default table must stay omitted from serialized config blobs: this +/// struct denies unknown fields, so an older binary loading a config blob +/// carrying an `[observability]` table it does not know would reject it, +/// breaking rollback. See [`Settings::observability`]. +#[derive(Debug, Clone, Default, PartialEq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ObservabilitySettings { + /// Emit the `Server-Timing` response header with per-phase request + /// timing. Defaults to `false` (off). + #[serde(default)] + pub server_timing_enabled: bool, +} + +impl ObservabilitySettings { + /// True when every field is at its default, i.e. observability is fully + /// disabled and the table can be omitted from serialized output. + fn is_default(&self) -> bool { + *self == Self::default() + } +} + /// Tester-cookie endpoint configuration. #[derive(Debug, Default, Clone, Deserialize, Serialize)] pub struct TesterCookieConfig { @@ -2633,6 +2686,12 @@ pub struct Settings { pub tinybird: TinybirdSettings, #[serde(default)] pub debug: DebugConfig, + /// Request-observability toggles. The default table is omitted from + /// serialized config blobs so a config round-tripped without change + /// still parses under a prior binary's schema; see + /// [`ObservabilitySettings`]. + #[serde(default, skip_serializing_if = "ObservabilitySettings::is_default")] + pub observability: ObservabilitySettings, } impl Settings { @@ -3354,6 +3413,14 @@ mod tests { use crate::redacted::Redacted; use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + /// Parses `extra` appended to the shared test fixture TOML, mirroring the + /// `format!("{}\n...", crate_test_settings_str())` pattern used throughout + /// this module's other tests. + fn settings_from_toml_with(extra: &str) -> Result> { + let toml = format!("{}\n{extra}", crate_test_settings_str()); + Settings::from_toml(&toml) + } + #[test] fn auction_debug_comment_options_default_matches_serde_defaults() { let opts = AuctionDebugCommentOptions::default(); @@ -3565,17 +3632,68 @@ mod tests { } #[test] - fn tinybird_access_enabled_is_rejected_until_emitter_is_wired() { - let toml = format!( - "{}\n[tinybird]\naccess_enabled = true\n", - crate_test_settings_str() + fn tinybird_access_enabled_with_full_config_is_accepted() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept a fully-specified access telemetry config"); + assert!( + settings.tinybird.access_enabled, + "should enable access emission" ); + } - let err = Settings::from_toml(&toml) - .expect_err("should reject access telemetry before emitter exists"); + #[test] + fn access_enabled_requires_positive_sample_rate() { + // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. + let err = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 0.0\n", + ) + .expect_err("should reject armed-but-silent access telemetry"); + assert!( + format!("{err:?}").contains("access_sample_rate"), + "should name the field" + ); + } + + #[test] + fn access_and_auction_emission_are_independent() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept access without auction"); + assert!( + !settings.tinybird.auction_enabled, + "should disable auction emission" + ); + assert!( + settings.tinybird.access_enabled, + "should enable access emission" + ); + } + + #[test] + fn auction_enabled_defaults_true_for_existing_configs() { + let settings = + settings_from_toml_with("[tinybird]\nenabled = true\napi_host = \"api.example.com\"\n") + .expect("should parse a pre-decoupling config"); + assert!( + settings.tinybird.auction_enabled, + "should preserve current behavior" + ); + } + + #[test] + fn observability_defaults_off_and_serializes_away() { + let settings = create_test_settings(); + assert!( + !settings.observability.server_timing_enabled, + "should default off" + ); + let toml = toml::to_string(&settings).expect("should serialize settings"); assert!( - format!("{err:?}").contains("tinybird.access_enabled"), - "should report unsupported tinybird.access_enabled setting: {err:?}" + !toml.contains("[observability]"), + "should omit the default table so a prior binary can parse the config" ); } diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 71f0f8f78..c52a299b3 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -222,6 +222,35 @@ verbosity = "redacted" # JSON request/response bodies remain strings exactly as captured. format = "compact" +[tinybird] +enabled = false +# Regional Tinybird API host, no scheme/port/path, e.g. "api.us-east.aws.tinybird.co". +api_host = "" +secret_store = "ts_secrets" +auction_dataset = "auction_events_raw" +auction_token_secret = "tinybird_auction_append_token" +# Defaults to true: existing configs keep emitting auction telemetry once +# `enabled` is turned on. Set false to silence auction events while keeping +# `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). +auction_enabled = true +# Access-log telemetry, decoupled from auction emission. `true` requires +# `enabled`, non-empty api_host/secret_store/access_dataset/access_token_secret, +# max_body_bytes > 0, and access_sample_rate > 0.0; an armed-but-silent sampler +# (access_enabled = true with access_sample_rate = 0) fails config load. +access_enabled = false +access_dataset = "access_logs_raw" +access_token_secret = "tinybird_access_append_token" +# Fraction (0.0-1.0) of requests to emit for access telemetry when access_enabled. +access_sample_rate = 0.0 +# Defensive maximum NDJSON body size, in bytes, for one Events API request. +max_body_bytes = 1048576 + +[observability] +# Keep this leaf present so the environment override can apply; the overlay +# cannot create a missing configuration leaf. The Server-Timing header stays +# off until enabled. +server_timing_enabled = false + [creative_opportunities] # Set to false to disable server-side ad templates while retaining slot definitions # and direct POST /auction callers. Structurally inactive templates use the From 4ce413ff87621429f731fc3a62a5267e43579c18 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:01:00 -0500 Subject: [PATCH 004/104] Add test coverage for the access_enabled without tinybird.enabled rejection --- crates/trusted-server-core/src/settings.rs | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index bcb467fb1..1aecf80a9 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3643,6 +3643,21 @@ mod tests { ); } + #[test] + fn access_enabled_requires_tinybird_enabled() { + // access_enabled = true with tinybird.enabled omitted (defaults + // false) must be rejected: access telemetry cannot run without the + // master toggle on. + let err = settings_from_toml_with( + "[tinybird]\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect_err("should reject access telemetry without tinybird.enabled"); + assert!( + format!("{err:?}").contains("tinybird.access_enabled"), + "should name the field: {err:?}" + ); + } + #[test] fn access_enabled_requires_positive_sample_rate() { // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. From f83092ab2c7acf81ae3bdf0b2c81d97b3c5941e4 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:20:34 -0500 Subject: [PATCH 005/104] Emit Server-Timing at the send freeze point on conclusively private responses --- .../trusted-server-adapter-fastly/src/app.rs | 109 +++++++++++- .../trusted-server-adapter-fastly/src/main.rs | 157 ++++++++++++++++-- 2 files changed, 248 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 41e5e65ee..8f21d89ac 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -1311,7 +1311,9 @@ mod tests { use bytes::Bytes; use edgezero_core::body::Body; use edgezero_core::context::RequestContext; - use edgezero_core::http::{Method, Response, StatusCode, header, request_builder}; + use edgezero_core::http::{ + Method, Response, StatusCode, header, request_builder, response_builder, + }; use edgezero_core::key_value_store::NoopKvStore; use edgezero_core::params::PathParams; use edgezero_core::router::RouterService; @@ -1333,6 +1335,7 @@ mod tests { PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, PlatformResponse, PlatformSelectResult, RuntimeServices, }; + use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; fn settings_with_missing_consent_store() -> Settings { @@ -2750,4 +2753,108 @@ mod tests { "the filter's response-header effect must be threaded out" ); } + + /// Joins every instance of a response header into one comma-separated + /// string (mirroring how a client sees repeated header fields), or + /// `None` if the header is absent. + fn response_header(response: &Response, name: &str) -> Option { + let values: Vec<&str> = response + .headers() + .get_all(name) + .iter() + .filter_map(|value| value.to_str().ok()) + .collect(); + if values.is_empty() { + None + } else { + Some(values.join(", ")) + } + } + + #[test] + fn server_timing_emitted_on_private_response_when_enabled() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(Body::empty()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, true); + + let header = response_header(&response, "server-timing").expect("should emit header"); + assert!( + header.contains("ts-total;dur="), + "should carry the stored total: {header}" + ); + assert_eq!( + header.matches("ts-total").count(), + 1, + "should emit exactly one TS-owned metric set" + ); + } + + #[test] + fn server_timing_absent_when_flag_off() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(Body::empty()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, false); + + assert!( + response_header(&response, "server-timing").is_none(), + "should not emit server-timing when the flag is off" + ); + } + + #[test] + fn server_timing_absent_on_cacheable_responses() { + // tsjs route policy: public, long max-age, immutable. + let mut tsjs_response = response_builder() + .header("cache-control", "public, max-age=31536000, immutable") + .body(Body::empty()) + .expect("should build a tsjs-style response fixture"); + // A bare shared-cacheable response with no private/no-store directive. + let mut public_response = response_builder() + .header("cache-control", "max-age=60") + .body(Body::empty()) + .expect("should build a bare max-age response fixture"); + + crate::apply_server_timing_header(&mut tsjs_response, &RequestTimings::new(), true); + crate::apply_server_timing_header(&mut public_response, &RequestTimings::new(), true); + + assert!( + response_header(&tsjs_response, "server-timing").is_none(), + "should not emit on the public immutable tsjs cache policy" + ); + assert!( + response_header(&public_response, "server-timing").is_none(), + "should not emit on a bare shared-cacheable max-age response" + ); + } + + #[test] + fn preexisting_server_timing_values_survive() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .header("server-timing", "upstream;dur=1") + .body(Body::empty()) + .expect("should build a private response fixture carrying an upstream Server-Timing"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, true); + + let header = + response_header(&response, "server-timing").expect("should still carry a header"); + assert!( + header.contains("upstream;dur=1"), + "should preserve the pre-existing entry: {header}" + ); + assert!( + header.contains("ts-total"), + "should append the TS-owned set: {header}" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index a19d0485d..b04b84fe0 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -5,13 +5,17 @@ use edgezero_adapter_fastly::request::into_core_request; use edgezero_core::body::Body as EdgeBody; use edgezero_core::config_store::ConfigStoreHandle; use edgezero_core::error::EdgeError; -use edgezero_core::http::{Request as HttpRequest, Response as HttpResponse}; +use edgezero_core::http::{ + HeaderName, HeaderValue, Request as HttpRequest, Response as HttpResponse, +}; use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; -use trusted_server_core::cache_policy::EdgeCacheHeader; +use trusted_server_core::cache_policy::{ + EdgeCacheHeader, cache_control_headers_are_private_or_no_store, +}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -24,6 +28,7 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -48,6 +53,12 @@ use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; const TRUSTED_SERVER_CONFIG_STORE: &str = "trusted_server_config"; +/// `Server-Timing` header name. Not present in the `http` crate's `header` +/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the +/// same `HeaderName::from_static` pattern used in +/// `trusted_server_core::constants`. +const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); + /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. /// /// # Errors @@ -111,19 +122,27 @@ fn edgezero_main(mut req: FastlyRequest) { return; } - let config_store = match open_trusted_server_config_store() { - Ok(cs) => cs, - Err(e) => { - log::error!("failed to open config store: {e}"); - FastlyResponse::from_status(fastly::http::StatusCode::INTERNAL_SERVER_ERROR) - .with_body_text_plain("Internal Server Error") - .send_to_client(); - return; - } - }; + let timings = RequestTimings::new(); - let (app, app_state) = TrustedServerApp::build_app_with_state(); + let (config_store, app, app_state) = { + let _appbuild = timings.span(Phase::AppBuild); + let config_store = match open_trusted_server_config_store() { + Ok(cs) => cs, + Err(e) => { + log::error!("failed to open config store: {e}"); + FastlyResponse::from_status(fastly::http::StatusCode::INTERNAL_SERVER_ERROR) + .with_body_text_plain("Internal Server Error") + .send_to_client(); + return; + } + }; + let (app, app_state) = TrustedServerApp::build_app_with_state(); + (config_store, app, app_state) + }; let settings_snapshot = app_state.as_ref().map(|state| Arc::clone(&state.settings)); + let server_timing_enabled = settings_snapshot + .as_deref() + .is_some_and(|settings| settings.observability.server_timing_enabled); // Strip client-spoofable forwarded headers before dispatch. compat::sanitize_fastly_forwarded_headers(&mut req); @@ -171,6 +190,7 @@ fn edgezero_main(mut req: FastlyRequest) { core_req.extensions_mut().insert(config_store); core_req.extensions_mut().insert(device_signals); core_req.extensions_mut().insert(client_info); + core_req.extensions_mut().insert(timings.clone()); match futures::executor::block_on(app.router().oneshot(core_req)) { Ok(response) => response, Err(error) => edge_error_response(error), @@ -213,7 +233,14 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings: timings.clone(), + server_timing_enabled, + }, + ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); return; } @@ -228,7 +255,14 @@ fn edgezero_main(mut req: FastlyRequest) { Ok(settings) => { match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings: timings.clone(), + server_timing_enabled, + }, + ); run_edgezero_pull_sync_after_send( &settings, &partner_registry, @@ -250,7 +284,14 @@ fn edgezero_main(mut req: FastlyRequest) { } } - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings, + server_timing_enabled, + }, + ); } fn edge_error_response(error: EdgeError) -> HttpResponse { @@ -323,6 +364,68 @@ fn run_edgezero_pull_sync_after_send( } } +/// Per-response context threaded into [`send_edgezero_response`] so the +/// function stays at or under seven parameters. +struct SendContext { + /// The request's phase-timing collector. + timings: RequestTimings, + /// Whether `observability.server_timing_enabled` is set. + server_timing_enabled: bool, +} + +/// Outcome of handing a finalized response to the client. +#[allow(dead_code)] +pub(crate) struct DeliveryOutcome { + /// Response body size in bytes. + pub bytes: u64, + /// Whether delivery completed or failed partway. + pub result: DeliveryResult, +} + +/// Whether [`send_edgezero_response`] completed delivery or failed partway. +pub(crate) enum DeliveryResult { + /// The response was handed to the client in full. + Complete, + /// Delivery failed partway through. + Error, +} + +/// Stamps [`RequestTimings::mark_headers_ready`] and, when observability is +/// enabled and the response is conclusively private, appends the rendered +/// `Server-Timing` header. +/// +/// Always stamps `mark_headers_ready` regardless of whether the header is +/// rendered, so the collector's `ts-total` reflects the moment headers +/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` +/// value set upstream survives alongside the TS-owned set. A response is +/// never promoted to shared-cacheable just because the header would +/// otherwise be omitted: this only gates emission, it does not touch +/// `Cache-Control`. +pub(crate) fn apply_server_timing_header( + response: &mut HttpResponse, + timings: &RequestTimings, + server_timing_enabled: bool, +) { + timings.mark_headers_ready(); + + let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); + if !server_timing_enabled || !conclusively_private { + return; + } + + let Some(value) = timings.server_timing_value() else { + return; + }; + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response + .headers_mut() + .append(HEADER_SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } +} + /// Sends a finalized `EdgeZero` response to the client. /// /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's @@ -331,8 +434,14 @@ fn run_edgezero_pull_sync_after_send( fn send_edgezero_response( mut response: HttpResponse, request_filter_effects: Option<&RequestFilterEffects>, -) { + context: &SendContext, +) -> DeliveryOutcome { apply_terminal_response_effects(&mut response, request_filter_effects); + apply_server_timing_header( + &mut response, + &context.timings, + context.server_timing_enabled, + ); let (parts, body) = response.into_parts(); @@ -348,15 +457,29 @@ fn send_edgezero_response( if let Err(e) = streaming_body.finish() { log::error!("failed to finish EdgeZero streaming body: {e}"); } + DeliveryOutcome { + bytes: 0, + result: DeliveryResult::Complete, + } } Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); + DeliveryOutcome { + bytes: 0, + result: DeliveryResult::Error, + } } } } once => { + let bytes = + u64::try_from(once.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); compat::to_fastly_response(HttpResponse::from_parts(parts, once)).send_to_client(); + DeliveryOutcome { + bytes, + result: DeliveryResult::Complete, + } } } } From 4b19c44a3cbb9abda692fd52054293fb0fcb8d56 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:56:02 -0500 Subject: [PATCH 006/104] Record filter and geo spans and dedupe the per-request geo lookup --- .../trusted-server-adapter-fastly/src/app.rs | 342 ++++++++++++++++-- .../trusted-server-adapter-fastly/src/main.rs | 33 +- .../src/middleware.rs | 80 +++- crates/trusted-server-core/src/geo.rs | 18 + .../src/integrations/registry.rs | 11 + 5 files changed, 447 insertions(+), 37 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 8f21d89ac..0cf80fb74 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -114,6 +114,7 @@ use trusted_server_core::ec::identify::{cors_preflight_identify, handle_identify use trusted_server_core::ec::kv::KvIdentityGraph; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; +use trusted_server_core::geo::GeoLookupState; use trusted_server_core::http_util::is_navigation_request; use trusted_server_core::integrations::{ IntegrationRegistry, ProxyDispatchInput, RequestFilterEffects, RequestFilterRegistryInput, @@ -133,6 +134,7 @@ use trusted_server_core::request_signing::{ handle_deactivate_key, handle_rotate_key, handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::settings::{ProxyAssetRoute, Settings}; use trusted_server_core::settings_data::{ default_config_key, default_config_store_name, get_settings_from_config_store, @@ -355,6 +357,17 @@ impl EcRequestState { services: self.services, } } + + /// Derives the carried [`GeoLookupState`] from this request's geo lookup + /// outcome, so response-phase finalize can reuse it instead of repeating + /// the lookup. `build_ec_request_state` always attempts the lookup, so + /// `None` here means the lookup ran and failed, not that it was skipped. + fn geo_lookup_state(&self) -> GeoLookupState { + match &self.geo_info { + Some(info) => GeoLookupState::Resolved(info.clone()), + None => GeoLookupState::Attempted, + } + } } /// Derives device signals from the request's `User-Agent` header. @@ -410,13 +423,21 @@ fn build_ec_request_state( let eids_cookie = crate::extract_cookie_value(req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(req, COOKIE_SHAREDID); - let geo_info = services - .geo() - .lookup(services.client_info().client_ip) - .unwrap_or_else(|e| { - log::warn!("geo lookup failed during EC setup: {e}"); - None - }); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let geo_info = { + let _span = timings.span(Phase::Geo); + services + .geo() + .lookup(services.client_info().client_ip) + .unwrap_or_else(|e| { + log::warn!("geo lookup failed during EC setup: {e}"); + None + }) + }; let (ec_context, setup_error) = match EcContext::read_from_request_with_geo(settings, req, services, geo_info.as_ref()) { @@ -482,6 +503,18 @@ async fn run_pre_route_filters( req: &mut Request, geo_info: Option<&GeoInfo>, ) -> PreRoute { + // Only recorded when a filter is actually registered, so unconfigured + // deployments omit ts-filter from the Server-Timing header entirely. + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let _span = state + .registry + .has_request_filters() + .then(|| timings.span(Phase::Filter)); + match state .registry .filter_request(RequestFilterRegistryInput { @@ -515,6 +548,7 @@ fn attach_dispatch_extensions( ec: EcRequestState, effects: RequestFilterEffects, ) -> Response { + response.extensions_mut().insert(ec.geo_lookup_state()); response.extensions_mut().insert(ec.into_finalize_state()); if !effects.response_headers.is_empty() { response.extensions_mut().insert(effects); @@ -810,7 +844,15 @@ async fn dispatch_fallback( .then(|| state.settings.asset_route_for_path(&path)) .flatten(); if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback(state, services, req, asset_route, &effects).await; + return dispatch_asset_fallback( + state, + services, + req, + asset_route, + &effects, + ec.geo_lookup_state(), + ) + .await; } // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -900,7 +942,10 @@ fn asset_response_carries_body(method: &Method, status: StatusCode) -> bool { /// [`AssetProxyCachePolicy`] out via response extensions so `edgezero_main` /// can reapply protected cache directives after finalization. EC finalization /// is intentionally skipped: no [`EcFinalizeState`] is attached, matching the -/// legacy `should_finalize_ec = false` behavior for asset responses. +/// legacy `should_finalize_ec = false` behavior for asset responses. The +/// caller's [`GeoLookupState`] is still attached, since `build_ec_request_state` +/// already attempted the lookup before the asset route was matched — this is +/// the one exit path that carries geo state without an `EcFinalizeState`. /// /// Like legacy `route_request`, asset bodies are streamed straight to the client /// with no cap: the origin stream is attached to the response and `edgezero_main` @@ -915,6 +960,7 @@ async fn dispatch_asset_fallback( req: Request, asset_route: &ProxyAssetRoute, effects: &RequestFilterEffects, + geo_state: GeoLookupState, ) -> Response { log::info!("No explicit route matched; proxying via configured asset route"); @@ -936,6 +982,7 @@ async fn dispatch_asset_fallback( } response.extensions_mut().insert(cache_policy); + response.extensions_mut().insert(geo_state); attach_request_filter_effects(&mut response, effects); response } @@ -944,6 +991,7 @@ async fn dispatch_asset_fallback( response .extensions_mut() .insert(AssetProxyCachePolicy::NoStorePrivate); + response.extensions_mut().insert(geo_state); attach_request_filter_effects(&mut response, effects); response } @@ -1319,21 +1367,23 @@ mod tests { use edgezero_core::router::RouterService; use std::net::{IpAddr, Ipv4Addr}; use std::sync::Mutex; + use std::sync::atomic::{AtomicUsize, Ordering}; use error_stack::Report; use futures::executor::block_on; use serde_json::json; - use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; + use trusted_server_core::constants::{HEADER_X_GEO_COUNTRY, HEADER_X_GEO_INFO_AVAILABLE}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::error::TrustedServerError; + use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::{ HeaderMutation, IntegrationRegistry, IntegrationRequestFilter, RequestFilterDecision, RequestFilterEffects, RequestFilterInput, }; use trusted_server_core::platform::{ - ClientInfo, PlatformBackend, PlatformBackendSpec, PlatformError, PlatformHttpClient, - PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, PlatformResponse, - PlatformSelectResult, RuntimeServices, + ClientInfo, GeoInfo, PlatformBackend, PlatformBackendSpec, PlatformError, PlatformGeo, + PlatformHttpClient, PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, + PlatformResponse, PlatformSelectResult, RuntimeServices, }; use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; @@ -1481,19 +1531,19 @@ mod tests { ); } - /// Builds a router whose `AppState` uses a registry containing the given - /// request filters (and no routes), so dispatch-level request-filter - /// behavior can be exercised without a real integration. - fn router_with_request_filters( + /// Builds an `AppState` whose registry contains the given request + /// filters (and no routes), so dispatch-level request-filter behavior can + /// be exercised without a real integration. + fn state_with_request_filters( filters: Vec>, - ) -> RouterService { + ) -> Arc { let settings = test_settings(); let orchestrator = trusted_server_core::auction::build_orchestrator(&settings) .expect("should build orchestrator"); let registry = IntegrationRegistry::from_request_filters(filters); let default_kv_store = Arc::new(crate::platform::UnavailableKvStore) as Arc; - let state = Arc::new(super::AppState { + Arc::new(super::AppState { auction_telemetry_sink: Arc::new( trusted_server_core::auction::NoopAuctionTelemetrySink, ), @@ -1501,8 +1551,15 @@ mod tests { orchestrator: Arc::new(orchestrator), registry: Arc::new(registry), default_kv_store, - }); - TrustedServerApp::routes_for_state(&state) + }) + } + + /// Builds a router on top of [`state_with_request_filters`] so + /// dispatch-level request-filter behavior can be exercised end-to-end. + fn router_with_request_filters( + filters: Vec>, + ) -> RouterService { + TrustedServerApp::routes_for_state(&state_with_request_filters(filters)) } /// Continues routing while mutating the request and emitting a response @@ -2532,6 +2589,61 @@ mod tests { ); } + #[test] + fn asset_fallback_carries_geo_state_without_ec_finalize_state() { + // The asset-route fallback is the one exit path that skips + // EcFinalizeState but must still carry GeoLookupState, since + // build_ec_request_state (and its geo lookup) already ran before the + // asset route was matched. Without this, the finalize step would + // silently repeat the lookup for every asset request. + let settings = Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + + [proxy] + + [[proxy.asset_routes]] + prefix = "/.image/" + origin_url = "https://assets.example.com" + "#, + ) + .expect("should parse asset-route settings"); + let state = build_state_from_settings(settings).expect("should build state"); + let router = TrustedServerApp::routes_for_state(&state); + + let response = route(&router, empty_request(Method::GET, "/.image/banner.png")); + + assert!( + response.extensions().get::().is_some(), + "asset-route responses should still carry GeoLookupState even though \ + EC finalization is skipped" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "asset-route responses must skip EC finalization (no EcFinalizeState)" + ); + } + struct FixedBackend; impl PlatformBackend for FixedBackend { @@ -2652,6 +2764,7 @@ mod tests { req, asset_route, &effects, + trusted_server_core::geo::GeoLookupState::NotAttempted, )); assert_eq!( @@ -2695,6 +2808,193 @@ mod tests { ); } + /// A [`PlatformGeo`] stub that counts every `lookup` call and always + /// returns the same canned result, used to prove the request-phase geo + /// lookup is never repeated during finalize. + struct CountingGeo { + calls: Arc, + result: Option, + } + + impl PlatformGeo for CountingGeo { + fn lookup(&self, _: Option) -> Result, Report> { + self.calls.fetch_add(1, Ordering::SeqCst); + Ok(self.result.clone()) + } + } + + fn sample_geo_info() -> GeoInfo { + GeoInfo { + city: "Testville".to_string(), + country: "US".to_string(), + continent: "NorthAmerica".to_string(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + } + } + + fn runtime_services_with_geo(geo: Arc) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(geo) + .client_info(ClientInfo::default()) + .build() + } + + #[test] + fn finalize_reuses_request_phase_geo_without_second_lookup() { + // Dispatching a publisher route runs build_ec_request_state, which + // attempts the geo lookup once and carries the result via + // GeoLookupState. The finalize step (resolve_geo_for_response) must + // reuse that carried value instead of calling the geo backend again. + let calls = Arc::new(AtomicUsize::new(0)); + let geo = Arc::new(CountingGeo { + calls: Arc::clone(&calls), + result: Some(sample_geo_info()), + }); + let state = app_state_for_settings(test_settings()); + let services = runtime_services_with_geo(geo); + let req = empty_request(Method::GET, "/some-page"); + + let response = block_on(super::dispatch_fallback(&state, &services, req)); + + let carried = response + .extensions() + .get::() + .cloned() + .expect("dispatch should attach GeoLookupState"); + assert!( + matches!(carried, GeoLookupState::Resolved(_)), + "a successful lookup should carry Resolved" + ); + + let geo_info = + crate::middleware::resolve_geo_for_response(&response, &carried, None, |_| { + panic!("finalize must not repeat a resolved geo lookup"); + }); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "only the request-phase lookup should have run" + ); + + let mut response = response; + geo_info + .expect("geo info should have resolved") + .set_response_headers(&mut response); + assert!( + response.headers().get(HEADER_X_GEO_COUNTRY).is_some(), + "x-geo-country should still be set on the response after reusing the carried geo" + ); + } + + #[test] + fn failed_lookup_is_not_retried() { + // When the request-phase lookup fails (returns None), dispatch must + // carry GeoLookupState::Attempted rather than NotAttempted, and + // finalize must not retry it. + let calls = Arc::new(AtomicUsize::new(0)); + let geo = Arc::new(CountingGeo { + calls: Arc::clone(&calls), + result: None, + }); + let state = app_state_for_settings(test_settings()); + let services = runtime_services_with_geo(geo); + let req = empty_request(Method::GET, "/some-page"); + + let response = block_on(super::dispatch_fallback(&state, &services, req)); + + let carried = response + .extensions() + .get::() + .cloned() + .expect("dispatch should attach GeoLookupState even for a failed lookup"); + assert!( + matches!(carried, GeoLookupState::Attempted), + "a failed lookup should carry Attempted, not Resolved or NotAttempted" + ); + + let geo_info = + crate::middleware::resolve_geo_for_response(&response, &carried, None, |_| { + panic!("finalize must not retry a failed geo lookup"); + }); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "only the request-phase lookup should have run" + ); + assert!( + geo_info.is_none(), + "no geo info should be available after a failed lookup" + ); + } + + #[test] + fn filter_span_recorded_when_request_filter_runs() { + // The Filter phase span should only be recorded when the registry + // actually has a request filter registered, so unconfigured + // deployments omit ts-filter from the Server-Timing header entirely. + let state = state_with_request_filters(vec![Arc::new(RecordingRequestFilter)]); + let services = RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(ClientInfo::default()) + .build(); + let mut req = empty_request(Method::GET, "/some-page"); + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + let _ = block_on(super::run_pre_route_filters( + &state, &services, &mut req, None, + )); + + assert!( + timings.snapshot().filter_ms.is_some(), + "should record the Filter phase span when a request filter is registered and runs" + ); + } + + #[test] + fn filter_span_not_recorded_when_no_request_filters_registered() { + // Mirror test: an empty registry must never record the Filter span, + // even though run_pre_route_filters still runs (as a no-op loop). + let state = state_with_request_filters(Vec::new()); + let services = RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(ClientInfo::default()) + .build(); + let mut req = empty_request(Method::GET, "/some-page"); + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + let _ = block_on(super::run_pre_route_filters( + &state, &services, &mut req, None, + )); + + assert!( + timings.snapshot().filter_ms.is_none(), + "should omit the Filter phase span when no request filters are registered" + ); + } + #[test] fn dispatch_runs_request_filter_and_threads_response_effects() { // Regression guard for the EdgeZero request-filter bypass: the publisher diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index b04b84fe0..ae9c600fb 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -24,6 +24,7 @@ use trusted_server_core::ec::pull_sync::{ }; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::TrustedServerError; +use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; @@ -209,14 +210,30 @@ fn edgezero_main(mut req: FastlyRequest) { let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); + let geo_lookup_state = response + .extensions_mut() + .remove::() + .unwrap_or(GeoLookupState::NotAttempted); if !take_finalize_sentinel(&mut response) { if let Some(settings) = settings_snapshot.as_deref() { - apply_entry_point_finalize_headers(settings, &mut response, client_ip); + apply_entry_point_finalize_headers( + settings, + &mut response, + client_ip, + &geo_lookup_state, + &timings, + ); } else { match load_settings_from_config_store() { Ok(settings) => { - apply_entry_point_finalize_headers(&settings, &mut response, client_ip); + apply_entry_point_finalize_headers( + &settings, + &mut response, + client_ip, + &geo_lookup_state, + &timings, + ); } Err(e) => { log::warn!("entry-point finalize skipped: failed to reload settings: {e:?}"); @@ -319,8 +336,11 @@ fn apply_entry_point_finalize_headers( settings: &Settings, response: &mut HttpResponse, client_ip: Option, + geo_state: &GeoLookupState, + timings: &RequestTimings, ) { - let geo_info = resolve_geo_for_response(response, client_ip, |client_ip| { + let geo_info = resolve_geo_for_response(response, geo_state, client_ip, |client_ip| { + let _span = timings.span(Phase::Geo); FastlyPlatformGeo.lookup(client_ip).unwrap_or_else(|e| { log::warn!("entry-point geo lookup failed: {e}"); None @@ -891,9 +911,10 @@ mod tests { .body(EdgeBody::empty()) .expect("should build response"); - let geo_info = resolve_geo_for_response(&response, None, |_| { - panic!("should skip entry-point geo lookup for 401 responses"); - }); + let geo_info = + resolve_geo_for_response(&response, &GeoLookupState::NotAttempted, None, |_| { + panic!("should skip entry-point geo lookup for 401 responses"); + }); apply_finalize_headers(&settings, geo_info.as_ref(), &mut response); assert_eq!( diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 18f309c68..17ecc13a4 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -24,8 +24,9 @@ use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_SERVICE_VERSION, HEADER_X_GEO_INFO_AVAILABLE, HEADER_X_TS_ENV, HEADER_X_TS_VERSION, }; -use trusted_server_core::geo::GeoInfo; +use trusted_server_core::geo::{GeoInfo, GeoLookupState}; use trusted_server_core::platform::PlatformGeo; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::settings::Settings; pub(crate) const HEADER_X_TS_FINALIZED: &str = "x-ts-finalized"; @@ -68,6 +69,12 @@ impl FinalizeResponseMiddleware { impl Middleware for FinalizeResponseMiddleware { async fn handle(&self, ctx: RequestContext, next: Next<'_>) -> Result { let client_ip = FastlyRequestContext::get(ctx.request()).and_then(|c| c.client_ip); + let timings = ctx + .request() + .extensions() + .get::() + .cloned() + .unwrap_or_default(); let mut response = match next.run(ctx).await { Ok(r) => r, @@ -77,7 +84,13 @@ impl Middleware for FinalizeResponseMiddleware { } }; - let geo_info = resolve_geo_for_response(&response, client_ip, |ip| { + let carried = response + .extensions() + .get::() + .cloned() + .unwrap_or(GeoLookupState::NotAttempted); + let geo_info = resolve_geo_for_response(&response, &carried, client_ip, |ip| { + let _span = timings.span(Phase::Geo); self.geo.lookup(ip).unwrap_or_else(|e| { log::warn!("geo lookup failed: {e}"); None @@ -142,14 +155,20 @@ impl Middleware for AuthMiddleware { // Shared geo resolution helper // --------------------------------------------------------------------------- -/// Resolves geo for a response, skipping the lookup for 401 responses. +/// Resolves geo for a response, skipping the lookup for 401 responses and +/// reusing a request-phase lookup when one was already carried. /// -/// Returns `None` for authentication rejections (401) without calling `lookup_geo` -/// to avoid unnecessary work and exposing geo data to unauthenticated callers. -/// All other responses call `lookup_geo` and return its result. +/// Returns `None` for authentication rejections (401) without consulting +/// `carried` or calling `lookup_geo`, to avoid unnecessary work and exposing +/// geo data to unauthenticated callers. Otherwise dispatches on `carried`: +/// a [`GeoLookupState::Resolved`] value is reused as-is, a +/// [`GeoLookupState::Attempted`] value is treated as no geo info without +/// retrying the lookup, and [`GeoLookupState::NotAttempted`] falls back to +/// calling `lookup_geo`. /// /// Used by both [`FinalizeResponseMiddleware`] and the entry-point finalization -/// in `main.rs` so the 401-skip rule is defined in one place. +/// in `main.rs` so the 401-skip rule and the dedupe rule are each defined in +/// one place. /// /// # Parity note /// @@ -161,6 +180,7 @@ impl Middleware for AuthMiddleware { /// server or the upstream origin. pub(crate) fn resolve_geo_for_response( response: &Response, + carried: &GeoLookupState, client_ip: Option, lookup_geo: F, ) -> Option @@ -168,9 +188,12 @@ where F: FnOnce(Option) -> Option, { if response.status() == StatusCode::UNAUTHORIZED { - None - } else { - lookup_geo(client_ip) + return None; + } + match carried { + GeoLookupState::Resolved(geo) => Some(geo.clone()), + GeoLookupState::Attempted => None, + GeoLookupState::NotAttempted => lookup_geo(client_ip), } } @@ -277,6 +300,19 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn sample_geo_info() -> GeoInfo { + GeoInfo { + city: "Testville".to_string(), + country: "US".to_string(), + continent: "NorthAmerica".to_string(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + } + } + struct FixedGeo(Option); impl PlatformGeo for FixedGeo { @@ -639,6 +675,30 @@ mod tests { ); } + #[test] + #[allow(clippy::panic)] + fn geo_lookup_skipped_for_unauthorized_responses() { + // The 401 short-circuit in resolve_geo_for_response must win + // regardless of what state the request phase carried in, and must + // never invoke the fallback lookup closure. + let mut response = empty_response(); + *response.status_mut() = StatusCode::UNAUTHORIZED; + + for carried in [ + GeoLookupState::NotAttempted, + GeoLookupState::Attempted, + GeoLookupState::Resolved(sample_geo_info()), + ] { + let geo_info = resolve_geo_for_response(&response, &carried, None, |_| { + panic!("401 responses must never trigger a geo lookup"); + }); + assert!( + geo_info.is_none(), + "401 responses should never resolve geo info, regardless of carried state" + ); + } + } + // --------------------------------------------------------------------------- // AuthMiddleware::handle tests // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-core/src/geo.rs b/crates/trusted-server-core/src/geo.rs index 63f7907f5..fe5785d26 100644 --- a/crates/trusted-server-core/src/geo.rs +++ b/crates/trusted-server-core/src/geo.rs @@ -48,6 +48,24 @@ impl GeoInfo { } } +/// Carries the outcome of a request-phase geo lookup across to +/// response-phase finalization, so a finalize consumer can reuse it instead +/// of performing a second lookup for the same request. +/// +/// Attached as a response extension on every exit path that attempted a +/// lookup, including the asset-route fallback (which does not carry an EC +/// finalize state). +#[derive(Debug, Clone)] +pub enum GeoLookupState { + /// No lookup has been attempted for this request. + NotAttempted, + /// A lookup ran and failed (or returned no result). This must not be + /// retried: finalize treats it the same as no geo info being available. + Attempted, + /// A lookup ran and resolved geo info. + Resolved(GeoInfo), +} + fn insert_geo_header(headers: &mut http::HeaderMap, name: http::header::HeaderName, value: &str) { match HeaderValue::from_str(value) { Ok(header_value) => { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 280eae847..376f5e492 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -893,6 +893,17 @@ impl IntegrationRegistry { self.find_route(method, path).is_some() } + /// Return true when at least one integration request filter is + /// registered. + /// + /// Adapters use this to decide whether to record a request-filter phase + /// timing span, so unconfigured deployments (no request filters) omit + /// that entry from observability output entirely. + #[must_use] + pub fn has_request_filters(&self) -> bool { + !self.inner.request_filters.is_empty() + } + /// Run pre-routing request filters. /// /// Request header mutations are applied immediately so later filters and From ce295f1405a0f5be2ee7363bd4f6ba7f3fc94587 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 22:37:53 -0600 Subject: [PATCH 007/104] Record origin, template cache, and KV phase spans in core --- .../trusted-server-adapter-fastly/src/app.rs | 132 +++++++++++-- .../trusted-server-adapter-fastly/src/main.rs | 184 +++++++++++++++++- crates/trusted-server-core/src/ec/kv.rs | 22 +++ .../trusted-server-core/src/platform/mod.rs | 2 + .../src/platform/timed_kv.rs | 180 +++++++++++++++++ crates/trusted-server-core/src/publisher.rs | 99 +++++++++- 6 files changed, 595 insertions(+), 24 deletions(-) create mode 100644 crates/trusted-server-core/src/platform/timed_kv.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 0cf80fb74..5cf7f9709 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -120,7 +120,9 @@ use trusted_server_core::integrations::{ IntegrationRegistry, ProxyDispatchInput, RequestFilterEffects, RequestFilterRegistryInput, RequestFilterRegistryOutcome, }; -use trusted_server_core::platform::{ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices}; +use trusted_server_core::platform::{ + ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices, TimedKvStore, +}; use trusted_server_core::proxy::{ AssetProxyCachePolicy, handle_asset_proxy_request, handle_first_party_click, handle_first_party_proxy, handle_first_party_proxy_rebuild, handle_first_party_proxy_sign, @@ -222,13 +224,18 @@ fn warn_if_certificate_check_disabled(settings: &Settings) { pub(crate) fn runtime_services_for_consent_route( settings: &Settings, runtime_services: &RuntimeServices, + timings: &RequestTimings, ) -> Result> { let Some(store_name) = settings.consent.consent_store.as_deref() else { return Ok(runtime_services.clone()); }; open_kv_store(store_name) - .map(|store| runtime_services.clone().with_kv_store(store)) + .map(|store| { + let timed_store = + Arc::new(TimedKvStore::new(store, timings.clone())) as Arc; + runtime_services.clone().with_kv_store(timed_store) + }) .map_err(|e| { Report::new(TrustedServerError::KvStore { store_name: store_name.to_string(), @@ -451,7 +458,7 @@ fn build_ec_request_state( // Bot gate: suppress KV-backed EC writes for unrecognized clients, except // consent withdrawals. Revocations keep the write path so tombstones stay // authoritative even for privacy-extension-heavy clients. - let kv_graph = crate::maybe_identity_graph(settings); + let kv_graph = crate::identity_graph_with_timing(settings, &timings); let finalize_kv_graph = if setup_error.is_none() && (is_real_browser || ec_consent_withdrawn(ec_context.consent())) { @@ -585,7 +592,12 @@ async fn execute_named( // Deliberately do not use an EC request-state graph: that // copy is bot-gated, while operators use curl for this // authenticated diagnostic. - let kv = crate::maybe_identity_graph(&state.settings); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let kv = crate::identity_graph_with_timing(&state.settings, &timings); handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) } NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), @@ -656,7 +668,12 @@ async fn run_named_route( if req.method() == Method::OPTIONS { cors_preflight_identify(&state.settings, &req) } else { - let kv = crate::require_identity_graph(&state.settings)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let kv = crate::require_identity_graph_with_timing(&state.settings, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; handle_identify( &state.settings, @@ -673,7 +690,13 @@ async fn run_named_route( // The auction reads consent data, so the consent KV store must be // available — fail closed with 503 when it is configured but // cannot be opened, matching legacy behavior. - let consent_services = runtime_services_for_consent_route(&state.settings, services)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let consent_services = + runtime_services_for_consent_route(&state.settings, services, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let registry_ref = if partner_registry.is_empty() { None @@ -701,7 +724,13 @@ async fn run_named_route( // Like the auction, page-bids reads consent data, so the consent KV // store must be available — fail closed with 503 when configured but // unopenable, matching legacy. - let consent_services = runtime_services_for_consent_route(&state.settings, services)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let consent_services = + runtime_services_for_consent_route(&state.settings, services, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let registry_ref = if partner_registry.is_empty() { None @@ -746,12 +775,18 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> let is_real_browser = device_signals.looks_like_browser(); let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); - let result = crate::require_identity_graph(&state.settings).and_then(|kv| { - let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; - let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - handle_batch_sync(&kv, &partner_registry, &limiter, req) - }); + let result = + crate::require_identity_graph_with_timing(&state.settings, &timings).and_then(|kv| { + let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; + let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); + handle_batch_sync(&kv, &partner_registry, &limiter, req) + }); let mut response = result.unwrap_or_else(|e| http_error(&e)); // Legacy parity: batch-sync responses still pass through @@ -870,7 +905,12 @@ async fn dispatch_fallback( // Publisher pages read consent data, so the consent KV store must be // available — fail closed with 503 when it is configured but cannot // be opened, matching legacy behavior. - match runtime_services_for_consent_route(&state.settings, services) { + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + match runtime_services_for_consent_route(&state.settings, services, &timings) { Ok(publisher_services) => { // Run the server-side auction with the configured creative- // opportunity slots and collect dispatched bids from the lazy @@ -2995,6 +3035,72 @@ mod tests { ); } + fn settings_with_consent_and_ec_store() -> Settings { + Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + ec_store = "ec_identity_store" + + [consent] + consent_store = "consent_store" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + "#, + ) + .expect("should parse settings with consent and EC KV stores configured") + } + + #[test] + fn consent_store_reads_are_timed_and_pull_sync_is_not() { + // Consent-store access threaded through RuntimeServices uses the same + // TimedKvStore decorator as request-path KvIdentityGraph + // construction, so a read through it records Phase::EcKv. + let settings = settings_with_consent_and_ec_store(); + let services = streaming_runtime_services(); + let timings = RequestTimings::new(); + + let consent_services = + super::runtime_services_for_consent_route(&settings, &services, &timings) + .expect("should open the configured consent store"); + let _ = block_on(consent_services.kv_store().get_bytes("consent-read-key")); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "a consent-store read through the decorated RuntimeServices store should record Phase::EcKv" + ); + + // Pull-sync's identity graph is built by `require_identity_graph`, + // which takes no `timings` parameter at all — the untimed store it + // constructs cannot record into any handle, including a fresh one. + let graph = crate::require_identity_graph(&settings) + .expect("should construct the pull-sync identity graph"); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let _ = graph.get(ec_id); + + let pull_sync_timings = RequestTimings::new(); + pull_sync_timings.mark_headers_ready(); + assert!( + pull_sync_timings.snapshot().kv_ms.is_none(), + "pull-sync's untimed graph construction has no timings handle to record into" + ); + } + #[test] fn dispatch_runs_request_filter_and_threads_response_effects() { // Regression guard for the EdgeZero request-filter bypass: the publisher diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ae9c600fb..ca97f6475 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -27,7 +27,7 @@ use trusted_server_core::error::TrustedServerError; use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; -use trusted_server_core::platform::RuntimeServices; +use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; @@ -248,7 +248,7 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(ec_state) = ec_state { if let Some(settings) = settings_snapshot.as_deref() { - match apply_edgezero_ec_finalize(settings, &ec_state, &mut response) { + match apply_edgezero_ec_finalize(settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { send_edgezero_response( response, @@ -270,7 +270,8 @@ fn edgezero_main(mut req: FastlyRequest) { } else { match load_settings_from_config_store() { Ok(settings) => { - match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response) { + match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) + { Ok(partner_registry) => { send_edgezero_response( response, @@ -353,10 +354,11 @@ fn apply_edgezero_ec_finalize( settings: &Settings, ec_state: &EcFinalizeState, response: &mut HttpResponse, + timings: &RequestTimings, ) -> Result> { let partner_registry = PartnerRegistry::from_config(&settings.ec.partners)?; let finalize_kv_graph = if ec_state.use_finalize_kv { - maybe_identity_graph(settings) + identity_graph_with_timing(settings, timings) } else { None }; @@ -575,12 +577,21 @@ fn build_ja4_debug_response(req: &FastlyRequest) -> FastlyResponse { .with_body(body) } -pub(crate) fn maybe_identity_graph(settings: &Settings) -> Option { - settings - .ec - .ec_store - .as_ref() - .map(|store_name| KvIdentityGraph::new(FastlyEcKvStore::new(store_name))) +/// Constructs a `KvIdentityGraph` wrapped in the [`Phase::EcKv`] timing +/// decorator, for request-path callers with a `RequestTimings` handle. +/// +/// Returns `None` when `ec.ec_store` is not configured, matching +/// [`require_identity_graph_with_timing`]'s contract on every other axis. +pub(crate) fn identity_graph_with_timing( + settings: &Settings, + timings: &RequestTimings, +) -> Option { + settings.ec.ec_store.as_ref().map(|store_name| { + KvIdentityGraph::new(TimedKvStore::new( + FastlyEcKvStore::new(store_name), + timings.clone(), + )) + }) } fn run_pull_sync_after_send( @@ -603,6 +614,12 @@ fn run_pull_sync_after_send( /// Constructs a `KvIdentityGraph` from settings, or returns an error if the /// `ec_store` config is not set. +/// +/// Deliberately untimed: pull-sync (this function's only caller) runs after +/// `send_edgezero_response`'s Server-Timing freeze point, so a decorated +/// store here would record into a handle nothing ever renders. +/// Request-path callers with a `RequestTimings` handle use +/// [`require_identity_graph_with_timing`] instead. pub(crate) fn require_identity_graph( settings: &Settings, ) -> Result> { @@ -615,6 +632,27 @@ pub(crate) fn require_identity_graph( Ok(KvIdentityGraph::new(FastlyEcKvStore::new(store_name))) } +/// Constructs a `KvIdentityGraph` wrapped in the [`Phase::EcKv`] timing +/// decorator, or returns an error if the `ec_store` config is not set. +/// +/// Request-path sibling of [`require_identity_graph`], which pull-sync uses +/// unwrapped because pull-sync runs after the Server-Timing freeze point. +pub(crate) fn require_identity_graph_with_timing( + settings: &Settings, + timings: &RequestTimings, +) -> Result> { + let store_name = settings.ec.ec_store.as_deref().ok_or_else(|| { + Report::new(TrustedServerError::KvStore { + store_name: "ec.ec_store".to_owned(), + message: "ec.ec_store is not configured".to_owned(), + }) + })?; + Ok(KvIdentityGraph::new(TimedKvStore::new( + FastlyEcKvStore::new(store_name), + timings.clone(), + ))) +} + /// Extracts a named cookie value from the request's `Cookie` header. pub(crate) fn extract_cookie_value(req: &HttpRequest, name: &str) -> Option { let cookie_header = req.headers().get("cookie").and_then(|v| v.to_str().ok())?; @@ -644,6 +682,7 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { use super::*; + use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; @@ -980,4 +1019,129 @@ mod tests { "should include sec-ch-ua-platform fallback" ); } + + fn ec_finalize_settings() -> Settings { + Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + ec_store = "ec_identity_store" + + [[ec.partners]] + name = "Example Partner" + source_domain = "example.com" + api_token = "test-vendor-token-32-bytes-minimum" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + "#, + ) + .expect("should parse EC finalize test settings") + } + + /// Minimal `RuntimeServices` for `EcFinalizeState.services`. Real + /// `FastlyPlatform*` handles are used as inert placeholders: EC + /// finalization never calls through them, it only satisfies the field. + fn inert_runtime_services() -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore) + as Arc) + .backend(Arc::new(crate::platform::FastlyPlatformBackend)) + .http_client(Arc::new(crate::platform::FastlyPlatformHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(trusted_server_core::platform::ClientInfo::default()) + .build() + } + + #[test] + fn ec_finalize_kv_lands_before_freeze() { + // A pre-seeded EC entry (see fastly.toml's ec_identity_store fixture) + // for a returning user carrying an eids cookie that matches the + // configured partner. This drives ec_finalize_response into + // ingest_eid_cookies, which reads and writes the KV identity graph. + let settings = ec_finalize_settings(); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let eids = serde_json::json!([{ + "source": "example.com", + "uids": [{ "id": "example-uid", "atype": 1 }] + }]); + let eids_cookie = base64::engine::general_purpose::STANDARD.encode(eids.to_string()); + let request = edgezero_core::http::request_builder() + .method(fastly::http::Method::GET) + .uri("https://test-publisher.com/article") + .header("cookie", format!("ts-ec={ec_id}; ts-eids={eids_cookie}")) + .body(EdgeBody::empty()) + .expect("should build EC finalize test request"); + + let services = inert_runtime_services(); + let geo_info = trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: "NorthAmerica".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }; + let ec_context = trusted_server_core::ec::EcContext::read_from_request_with_geo( + &settings, + &request, + &services, + Some(&geo_info), + ) + .expect("should read EC context from a non-regulated request"); + assert!( + ec_context.ec_was_present(), + "the pre-seeded ts-ec cookie should be recognized" + ); + + let ec_state = EcFinalizeState { + ec_context, + use_finalize_kv: true, + eids_cookie: Some(eids_cookie), + sharedid_cookie: None, + is_real_browser: true, + services, + }; + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(EdgeBody::empty()) + .expect("should build EC finalize response fixture"); + let timings = RequestTimings::new(); + + // Mirrors edgezero_main's ordering: EC finalize runs, then the freeze + // point (apply_server_timing_header, called just before + // response.into_parts() inside send_edgezero_response) renders the + // header. Calling both directly exercises exactly this order without + // requiring a live Fastly client connection. + apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) + .expect("should finalize EC response"); + apply_server_timing_header(&mut response, &timings, true); + + let header = response + .headers() + .get("server-timing") + .and_then(|v| v.to_str().ok()) + .expect("should emit a Server-Timing header"); + assert!( + header.contains("ts-kv"), + "the freeze point must run after EC finalization recorded KV time: {header}" + ); + } } diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 3572581ce..dc9885fde 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -810,6 +810,28 @@ mod tests { assert!(ts > 0, "should return a nonzero timestamp"); } + #[test] + fn kv_span_accumulates_across_graph_operations() { + let timings = crate::request_timing::RequestTimings::new(); + let graph = KvIdentityGraph::new(crate::platform::TimedKvStore::new( + crate::ec::kv_backend::test_support::InMemoryEcKv::new("test-store"), + timings.clone(), + )); + + graph + .create("ec-1", &live_entry()) + .expect("should create entry through the timed store"); + graph + .get("ec-1") + .expect("should read the entry back through the timed store"); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should accumulate Phase::EcKv across both graph operations, not just the last write" + ); + } + #[test] fn serialize_entry_produces_valid_json() { let entry = KvEntry::tombstone(1000); diff --git a/crates/trusted-server-core/src/platform/mod.rs b/crates/trusted-server-core/src/platform/mod.rs index 1c5bf4c2a..e6ed3ef90 100644 --- a/crates/trusted-server-core/src/platform/mod.rs +++ b/crates/trusted-server-core/src/platform/mod.rs @@ -42,6 +42,7 @@ mod template_assembly; mod template_cache; #[cfg(test)] pub(crate) mod test_support; +mod timed_kv; mod traits; mod types; @@ -67,6 +68,7 @@ pub use template_cache::{ TemplateEntry, TemplateMetadata, TemplateMetadataEncodeError, UnavailableTemplateCache, VaryHeaderValues, VarySpec, }; +pub use timed_kv::TimedKvStore; pub use traits::{PlatformBackend, PlatformConfigStore, PlatformGeo, PlatformSecretStore}; pub use types::{ ClientInfo, GeoInfo, PlatformBackendSpec, RuntimeServices, RuntimeServicesBuilder, StoreId, diff --git a/crates/trusted-server-core/src/platform/timed_kv.rs b/crates/trusted-server-core/src/platform/timed_kv.rs new file mode 100644 index 000000000..0594b7cef --- /dev/null +++ b/crates/trusted-server-core/src/platform/timed_kv.rs @@ -0,0 +1,180 @@ +//! Latency-only timing decorator for KV store handles. +//! +//! [`TimedKvStore`] wraps an inner store plus a [`RequestTimings`] handle and +//! records [`Phase::EcKv`] around every call. It implements both +//! [`PlatformKvStore`] (for consent-store access obtained through +//! [`RuntimeServices`](super::RuntimeServices)) and [`EcKvStore`] (for +//! [`KvIdentityGraph`](crate::ec::kv::KvIdentityGraph) construction sites), +//! because no single existing abstraction covers the whole `ts-kv` taxonomy: +//! EC graph operations go through [`EcKvStore`] while consent persistence +//! uses [`PlatformKvStore`] directly. +//! +//! The decorator measures store-call latency only: it never reads, parses, +//! or logs any value passing through it. + +use std::sync::Arc; +use std::time::Duration; + +use async_trait::async_trait; +use bytes::Bytes; +use edgezero_core::key_value_store::{KvError, KvPage, KvStore as PlatformKvStore}; +use error_stack::Report; + +use crate::ec::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteOutcome}; +use crate::error::TrustedServerError; +use crate::request_timing::{Phase, RequestTimings}; + +/// Wraps `inner` plus a [`RequestTimings`] handle, recording [`Phase::EcKv`] +/// around every store call made through it. +pub struct TimedKvStore { + /// The wrapped store handle. + inner: S, + /// The request's phase-timing collector. + timings: RequestTimings, +} + +impl TimedKvStore { + /// Creates a decorator around `inner` that records into `timings`. + #[must_use] + pub fn new(inner: S, timings: RequestTimings) -> Self { + Self { inner, timings } + } +} + +#[async_trait(?Send)] +impl PlatformKvStore for TimedKvStore> { + async fn get_bytes(&self, key: &str) -> Result, KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.get_bytes(key).await + } + + async fn put_bytes(&self, key: &str, value: Bytes) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.put_bytes(key, value).await + } + + async fn put_bytes_with_ttl( + &self, + key: &str, + value: Bytes, + ttl: Duration, + ) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.put_bytes_with_ttl(key, value, ttl).await + } + + async fn delete(&self, key: &str) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.delete(key).await + } + + async fn list_keys_page( + &self, + prefix: &str, + cursor: Option<&str>, + limit: usize, + ) -> Result { + let _span = self.timings.span(Phase::EcKv); + self.inner.list_keys_page(prefix, cursor, limit).await + } +} + +impl EcKvStore for TimedKvStore { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + let _span = self.timings.span(Phase::EcKv); + self.inner.lookup(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + let _span = self.timings.span(Phase::EcKv); + self.inner.insert(key, write) + } + + fn count_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result> { + let _span = self.timings.span(Phase::EcKv); + self.inner.count_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + let _span = self.timings.span(Phase::EcKv); + self.inner.delete(key) + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration as StdDuration; + + use super::*; + use crate::ec::kv_backend::test_support::InMemoryEcKv; + + #[test] + fn ec_kv_store_operations_accumulate_into_ec_kv_phase() { + let timings = RequestTimings::new(); + let store = TimedKvStore::new(InMemoryEcKv::new("test-store"), timings.clone()); + + store + .insert( + "key-a", + EcKvWrite { + body: "{}", + metadata: "{}", + ttl: StdDuration::from_secs(60), + mode: crate::ec::kv_backend::EcKvWriteMode::Add, + }, + ) + .expect("should insert into the in-memory store"); + store.lookup("key-a").expect("should read back the entry"); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should record Phase::EcKv across both store calls" + ); + } + + #[test] + fn store_name_is_not_timed() { + let timings = RequestTimings::new(); + let store = TimedKvStore::new(InMemoryEcKv::new("test-store"), timings.clone()); + + assert_eq!(store.store_name(), "test-store"); + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_none(), + "store_name is a metadata accessor, not a store operation" + ); + } + + #[test] + fn platform_kv_store_operations_accumulate_into_ec_kv_phase() { + let timings = RequestTimings::new(); + let inner: Arc = Arc::new(crate::platform::UnavailableKvStore); + let store = TimedKvStore::new(inner, timings.clone()); + + // UnavailableKvStore errors on every call; the decorator still times + // the attempt regardless of outcome. + futures::executor::block_on(async { + let _ = store.get_bytes("key").await; + let _ = store.put_bytes("key", Bytes::from_static(b"value")).await; + }); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should record Phase::EcKv even when the inner store errors" + ); + } +} diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index b0d63b82a..07d6704a7 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -70,6 +70,7 @@ use crate::platform::{ contains_publisher_esi_directive, }; use crate::price_bucket::{PriceGranularity, price_bucket}; +use crate::request_timing::{Phase, RequestTimings}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -4044,6 +4045,14 @@ pub async fn handle_publisher_request( ) -> Result> { log::debug!("Proxying request to publisher_origin"); + // A defaulted handle records into nothing that ever renders, so tests + // that don't populate the request extension are unaffected. + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + // Adapter fallbacks prepare this before EC/cookie handling. Keep this // idempotent call as a direct-handler safety net and for focused tests. let gpt_diagnostics = @@ -4476,7 +4485,10 @@ pub async fn handle_publisher_request( // not be served a shared template even if that template is perfectly cacheable. let mut template_cache_reservation = None; if let Some(key) = template_cache_key.as_ref() { - match services.template_cache().lookup_or_reserve(key).await { + let template_cache_span = timings.span(Phase::TemplateCacheLookup); + let template_cache_lookup = services.template_cache().lookup_or_reserve(key).await; + drop(template_cache_span); + match template_cache_lookup { Ok(crate::platform::TemplateCacheLookup::Hit(entry)) => { log::debug!("template_cache hit: {} bytes", entry.body.len()); @@ -4577,6 +4589,7 @@ pub async fn handle_publisher_request( platform_request = platform_request.with_cache_bypass(); } + let origin_span = timings.span(Phase::Origin); let mut response = match services.http_client().send(platform_request).await { Ok(platform_response) => platform_response.response, Err(err) => { @@ -4594,6 +4607,7 @@ pub async fn handle_publisher_request( })); } }; + drop(origin_span); log::debug!( "Publisher origin response received: status={}, header_count={}", @@ -7737,6 +7751,35 @@ mod tests { .expect("should proxy publisher request") } + #[tokio::test] + async fn origin_span_covers_the_publisher_fetch() { + let settings = create_test_settings(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let services = + build_services_with_http_client(stub as Arc); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let _response = run_publisher_proxy(&settings, &services, request).await; + + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.is_some(), + "should record the Origin phase span around the publisher fetch" + ); + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! @@ -8973,6 +9016,60 @@ mod tests { ); } + #[tokio::test] + async fn template_cache_span_recorded_only_when_lookup_runs() { + // Inline mode: no shared-cache key is ever computed, so the lookup + // never runs and the span is never recorded. + let inline_settings = create_test_settings(); + let inline_stub = Arc::new(StubHttpClient::new()); + inline_stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let inline_services = build_services_with_http_client( + inline_stub as Arc, + ); + let mut inline_request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let inline_timings = RequestTimings::new(); + inline_request + .extensions_mut() + .insert(inline_timings.clone()); + + let _inline_response = + run_publisher_proxy(&inline_settings, &inline_services, inline_request).await; + + inline_timings.mark_headers_ready(); + assert!( + inline_timings.snapshot().template_cache_ms.is_none(), + "inline mode should never run the template-cache lookup" + ); + + // Shared-mode eligible: a matched slot plus a template-cache-eligible + // assembly mode compute a cache key, so the lookup runs and is timed. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + let mut request = navigation_request(); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let _response = run(&settings, &services, request).await; + + timings.mark_headers_ready(); + assert!( + timings.snapshot().template_cache_ms.is_some(), + "a shared-cache-eligible request should record the TemplateCacheLookup span" + ); + } + #[tokio::test] async fn only_the_cold_miss_uses_the_platform_assembler() { let stub = Arc::new(StubHttpClient::new()); From c222ae280931999fdd822473581ac22a632cab27 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 00:57:06 -0700 Subject: [PATCH 008/104] Capture stream duration, auction wait placement, and response bytes --- .../trusted-server-adapter-fastly/src/main.rs | 269 ++++++++++++++++-- crates/trusted-server-core/src/publisher.rs | 210 +++++++++++++- 2 files changed, 462 insertions(+), 17 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ca97f6475..5747dc8fd 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,4 +1,5 @@ use std::sync::Arc; +use std::time::Instant; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; use edgezero_adapter_fastly::request::into_core_request; @@ -405,10 +406,15 @@ pub(crate) struct DeliveryOutcome { } /// Whether [`send_edgezero_response`] completed delivery or failed partway. +#[derive(Debug, PartialEq, Eq)] pub(crate) enum DeliveryResult { /// The response was handed to the client in full. Complete, - /// Delivery failed partway through. + /// Delivery started but did not finish cleanly: some bytes reached the + /// client's transport before a stream error, or the transport could not + /// be closed cleanly after every byte was written. + Partial, + /// Delivery failed before any bytes reached the client. Error, } @@ -448,6 +454,93 @@ pub(crate) fn apply_server_timing_header( } } +/// A [`Write`](std::io::Write) wrapper that tallies bytes successfully written +/// to the inner writer. +/// +/// Wraps the client transport during a streaming drive so a truncated or +/// failed drive still reports how many bytes actually reached it, instead of +/// the placeholder `0` a failed/aborted drive would otherwise report. +struct CountingWriter { + inner: W, + bytes: u64, +} + +impl CountingWriter { + fn new(inner: W) -> Self { + Self { inner, bytes: 0 } + } + + /// Bytes successfully written to the inner writer so far. + fn bytes(&self) -> u64 { + self.bytes + } + + fn into_inner(self) -> W { + self.inner + } +} + +impl std::io::Write for CountingWriter { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + let written = self.inner.write(buf)?; + self.bytes = self.bytes.saturating_add(written as u64); + Ok(written) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush() + } +} + +/// Drives a streaming `EdgeZero` body through `output`, tallying bytes written +/// and timing the drive into `timings`. +/// +/// Stamps `resp_bytes` and `request_elapsed` immediately once the drive +/// returns — before the caller does anything transport-specific (finishing +/// the streaming body, logging) — so `request_elapsed` never includes that +/// work. Returns the counting writer (so the caller can recover both the +/// tallied byte count and the wrapped transport) alongside the drive's +/// result. +fn drive_streaming_body( + body: EdgeBody, + output: W, + timings: &RequestTimings, +) -> (CountingWriter, Result<(), Report>) { + let mut counting = CountingWriter::new(output); + let drive_started = Instant::now(); + let result = futures::executor::block_on(stream_asset_body(body, &mut counting)); + timings.record(Phase::Stream, drive_started.elapsed()); + timings.set_resp_bytes(counting.bytes()); + timings.mark_request_elapsed(); + (counting, result) +} + +/// Classifies a completed streaming drive into a [`DeliveryResult`]. +/// +/// A drive that failed after writing at least one byte delivered a truncated +/// response rather than nothing at all, so it is [`DeliveryResult::Partial`], +/// not [`DeliveryResult::Error`]. +fn classify_stream_delivery( + drive_result: &Result<(), Report>, + bytes: u64, +) -> DeliveryResult { + match drive_result { + Ok(()) => DeliveryResult::Complete, + Err(_) if bytes > 0 => DeliveryResult::Partial, + Err(_) => DeliveryResult::Error, + } +} + +/// Stamps `resp_bytes`/`request_elapsed` for an already-materialized body, +/// immediately before it is handed to the Fastly client transport, and +/// returns its byte length. +fn record_buffered_delivery(body: &EdgeBody, timings: &RequestTimings) -> u64 { + let bytes = u64::try_from(body.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); + timings.set_resp_bytes(bytes); + timings.mark_request_elapsed(); + bytes +} + /// Sends a finalized `EdgeZero` response to the client. /// /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's @@ -473,30 +566,40 @@ fn send_edgezero_response( parts, EdgeBody::empty(), )); - let mut streaming_body = skeleton.stream_to_client(); - match futures::executor::block_on(stream_asset_body(body, &mut streaming_body)) { - Ok(()) => { - if let Err(e) = streaming_body.finish() { - log::error!("failed to finish EdgeZero streaming body: {e}"); - } - DeliveryOutcome { - bytes: 0, + let (counting, drive_result) = + drive_streaming_body(body, skeleton.stream_to_client(), &context.timings); + let bytes = counting.bytes(); + let streaming_body = counting.into_inner(); + // Computed before `drive_result` is matched by value below, since + // the `Err` arm there moves its `Report` out. + let result = classify_stream_delivery(&drive_result, bytes); + match drive_result { + Ok(()) => match streaming_body.finish() { + Ok(()) => DeliveryOutcome { + bytes, result: DeliveryResult::Complete, + }, + Err(e) => { + // Every byte was handed to the transport (the drive + // above returned Ok), but the transport itself could + // not close cleanly — the client may still see a + // truncated response. + log::error!("failed to finish EdgeZero streaming body: {e}"); + DeliveryOutcome { + bytes, + result: DeliveryResult::Partial, + } } - } + }, Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); - DeliveryOutcome { - bytes: 0, - result: DeliveryResult::Error, - } + DeliveryOutcome { bytes, result } } } } once => { - let bytes = - u64::try_from(once.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); + let bytes = record_buffered_delivery(&once, &context.timings); compat::to_fastly_response(HttpResponse::from_parts(parts, once)).send_to_client(); DeliveryOutcome { bytes, @@ -687,7 +790,9 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; + use trusted_server_core::request_timing::AuctionWaitPlacement; fn test_settings() -> Settings { Settings::from_toml( @@ -1144,4 +1249,136 @@ mod tests { "the freeze point must run after EC finalization recorded KV time: {header}" ); } + + #[test] + fn delivery_outcome_reports_bytes_and_request_elapsed_set() { + let timings = RequestTimings::new(); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello "), + bytes::Bytes::from_static(b"world"), + ])); + + let (counting, drive_result) = drive_streaming_body(body, Vec::new(), &timings); + drive_result.expect("streaming a well-formed body should not fail"); + let bytes = counting.bytes(); + let outcome = DeliveryOutcome { + bytes, + result: DeliveryResult::Complete, + }; + + assert_eq!( + counting.into_inner(), + b"hello world", + "should write every byte to the underlying transport" + ); + assert_eq!( + outcome.bytes, + "hello world".len() as u64, + "DeliveryOutcome.bytes should equal the streamed body length" + ); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.resp_bytes, + Some("hello world".len() as u64), + "should stamp resp_bytes to the tallied byte count" + ); + assert!( + snapshot.request_elapsed_ms.is_some(), + "should stamp request_elapsed once the drive returns" + ); + } + + #[test] + fn buffered_delivery_stamps_bytes_and_request_elapsed() { + let timings = RequestTimings::new(); + let body = EdgeBody::from(b"a buffered body".to_vec()); + + let bytes = record_buffered_delivery(&body, &timings); + + assert_eq!( + bytes, + "a buffered body".len() as u64, + "should report the buffered body length" + ); + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.resp_bytes, + Some("a buffered body".len() as u64), + "should stamp resp_bytes for the buffered path too" + ); + assert!( + snapshot.request_elapsed_ms.is_some(), + "should stamp request_elapsed for the buffered path too" + ); + } + + #[test] + fn stream_drive_records_stream_ms_covering_the_in_stream_auction_wait() { + // A streaming seam wait (Task 6, publisher.rs) records into the same + // `RequestTimings` handle the adapter drives with. `Phase::Stream` + // wraps the entire drive, so it must cover — and therefore be at + // least as large as — any `AuctionWait` recorded while the body was + // being polled. + let timings = RequestTimings::new(); + let wait_timings = timings.clone(); + let stream = futures::stream::once(async move { + let waited = Duration::from_millis(5); + std::thread::sleep(waited); + wait_timings.record_auction_wait(AuctionWaitPlacement::InStream, waited); + bytes::Bytes::from_static(b"") + }); + let body = EdgeBody::stream(stream); + + let (_counting, drive_result) = drive_streaming_body(body, Vec::new(), &timings); + drive_result.expect("streaming a well-formed body should not fail"); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::InStream), + "should preserve the placement recorded from inside the polled body" + ); + let auction_wait_ms = snapshot + .auction_wait_ms + .expect("should record the auction wait"); + let stream_ms = snapshot.stream_ms.expect("should record the stream drive"); + assert!( + stream_ms >= auction_wait_ms, + "the drive's Phase::Stream span must cover the in-stream auction wait: \ + stream_ms={stream_ms} auction_wait_ms={auction_wait_ms}" + ); + } + + #[test] + fn classify_stream_delivery_treats_bytes_written_before_an_error_as_partial() { + let err = Report::new(TrustedServerError::Proxy { + message: "boom".to_string(), + }); + assert_eq!( + classify_stream_delivery(&Err(err), 42), + DeliveryResult::Partial, + "bytes already on the wire before a stream error is a truncated delivery" + ); + } + + #[test] + fn classify_stream_delivery_treats_an_error_with_no_bytes_as_error() { + let err = Report::new(TrustedServerError::Proxy { + message: "boom".to_string(), + }); + assert_eq!( + classify_stream_delivery(&Err(err), 0), + DeliveryResult::Error, + "a failure before any byte reached the client is a clean failure, not a truncation" + ); + } + + #[test] + fn classify_stream_delivery_treats_ok_as_complete() { + assert_eq!( + classify_stream_delivery(&Ok(()), 123), + DeliveryResult::Complete + ); + } } diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 07d6704a7..1943d9b3c 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -70,7 +70,7 @@ use crate::platform::{ contains_publisher_esi_directive, }; use crate::price_bucket::{PriceGranularity, price_bucket}; -use crate::request_timing::{Phase, RequestTimings}; +use crate::request_timing::{AuctionWaitPlacement, Phase, RequestTimings}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -1617,6 +1617,12 @@ pub struct OwnedProcessResponseParams { /// rescanned from the output, which cannot tell a `nonce` attribute from the same /// word inside a script. pub(crate) csp_nonce_observed: Option>, + /// Per-request phase-timing handle, carried into the streaming/buffered + /// finalizers so the `` seam wait can be recorded with the right + /// [`AuctionWaitPlacement`]. Cheap to clone (an `Arc` handle); a request that + /// never attached one to its extensions gets a fresh, unattached collector + /// that nothing ever renders. + pub(crate) timings: RequestTimings, } /// Response-authorized template cache insert inputs. The key is built before origin lookup; the @@ -1860,6 +1866,8 @@ pub async fn buffer_publisher_response_async( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::PreHeader, }, ) .await; @@ -2019,6 +2027,7 @@ fn build_template_assembly_params( request_scheme: &str, price_granularity: PriceGranularity, ad_bids_state: AdBidsState, + timings: RequestTimings, ) -> OwnedProcessResponseParams { OwnedProcessResponseParams { csp_nonce_observed: None, @@ -2041,6 +2050,7 @@ fn build_template_assembly_params( price_granularity, gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings, } } @@ -2383,6 +2393,8 @@ pub async fn publisher_response_into_streaming_response( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::InStream, }, ) .await; @@ -2501,6 +2513,7 @@ pub async fn publisher_response_into_streaming_response( &orchestrator, &services, &settings, + AuctionWaitPlacement::InStream, ) .await; // Collection reached a terminal result; disarm only now @@ -2522,6 +2535,8 @@ pub async fn publisher_response_into_streaming_response( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::InStream, }; while let Some(step) = hold_step_next_chunk( @@ -2857,6 +2872,7 @@ pub async fn stream_publisher_body_async( orchestrator, services, settings, + AuctionWaitPlacement::PreHeader, ) .await; if body.is_stream() { @@ -2923,6 +2939,8 @@ pub async fn stream_publisher_body_async( services, settings, request_origin: request_origin(¶ms.request_scheme, ¶ms.request_host), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::PreHeader, }, }, ) @@ -3490,6 +3508,12 @@ struct AuctionCollectDeps<'a> { settings: &'a Settings, /// Trusted request origin (`scheme://host`) for absolute inline creative URLs. request_origin: String, + /// Phase-timing handle the collect step records the auction wait into. + timings: RequestTimings, + /// Where this collect call sits relative to response headers: streaming + /// callers await inside the body already handed to the client, buffered + /// callers await before anything has been sent. + placement: AuctionWaitPlacement, } /// Run the close-body hold loop for HTML bodies, collecting the auction before @@ -3870,6 +3894,11 @@ async fn emit_abandoned_auction( /// Collect a dispatched auction before a non-HTML body streams: there is no /// `` to inject into, so bids are written to state up front and the /// auction telemetry completes immediately. +/// +/// `placement` records where this wait sits relative to response headers — the +/// caller decides, since this collector runs from both the buffered finalizer +/// (headers not yet committed) and the true streaming path (headers already +/// sent, this body only just started being polled). async fn collect_non_html_auction( dispatched: DispatchedAuction, telemetry: AuctionTelemetryCarry, @@ -3877,12 +3906,14 @@ async fn collect_non_html_auction( orchestrator: &AuctionOrchestrator, services: &RuntimeServices, settings: &Settings, + placement: AuctionWaitPlacement, ) { let auction_id = telemetry .auction_request .as_ref() .and_then(|_| diagnostics_auction_id(settings)); let placeholder = mediator_placeholder_request(); + let wait_started = Instant::now(); let result = orchestrator .collect_dispatched_auction( dispatched, @@ -3890,6 +3921,9 @@ async fn collect_non_html_auction( &make_collect_context(settings, services, &placeholder), ) .await; + params + .timings + .record_auction_wait(placement, wait_started.elapsed()); let delivered_winner_slots = write_bids_to_state( &result.winning_bids, params.price_granularity, @@ -3931,6 +3965,8 @@ async fn collect_stream_auction( services, settings, request_origin, + timings, + placement, } = deps; let auction_id = telemetry .auction_request @@ -3939,9 +3975,11 @@ async fn collect_stream_auction( log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); + let wait_started = Instant::now(); let result = orchestrator .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; + timings.record_auction_wait(*placement, wait_started.elapsed()); log::info!( "body_close_hold_loop: collect complete - {} winning bid(s)", result.winning_bids.len() @@ -4538,6 +4576,7 @@ pub async fn handle_publisher_request( request_scheme, price_granularity, ad_bids_state.clone(), + timings.clone(), ); params.seam_ad_slots = seam_ad_slots.clone(); params.dispatched_auction = dispatched_auction.take(); @@ -4910,6 +4949,7 @@ pub async fn handle_publisher_request( dispatched_auction, price_granularity, gpt_diagnostics: Some(gpt_diagnostics), + timings: timings.clone(), }), }) } @@ -7546,6 +7586,7 @@ mod tests { price_granularity: Default::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -14592,6 +14633,8 @@ mod tests { services: &services, settings: &settings, request_origin: String::new(), + timings: RequestTimings::new(), + placement: AuctionWaitPlacement::PreHeader, }, }; let mut output = Vec::new(); @@ -14641,6 +14684,8 @@ mod tests { services: &services, settings: &settings, request_origin: String::new(), + timings: RequestTimings::new(), + placement: AuctionWaitPlacement::PreHeader, }; // Passthrough processor: the ordering contract is about collection, not // HTML rewriting, so keep the emitted bytes verbatim. @@ -15517,6 +15562,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -15570,6 +15616,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -15612,6 +15659,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::from_stream(futures::stream::iter(vec![Ok::<_, io::Error>( bytes::Bytes::from_static(b"live"), @@ -15732,6 +15780,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![ bytes::Bytes::from_static(b"body{background:url('https://origin.example.com/"), @@ -15790,6 +15839,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = gzip_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15851,6 +15901,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = deflate_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15912,6 +15963,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = brotli_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15973,6 +16025,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = brotli_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -16022,6 +16075,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -16221,6 +16275,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![ bytes::Bytes::from_static(b"hello"), @@ -16290,6 +16345,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; // The `` that triggers bid injection lives in the SECOND gzip // member. `flate2::read::GzDecoder` decodes only the first member, so @@ -16358,6 +16414,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![bytes::Bytes::from_static( b"body{background:url('https://origin.example.com/asset.png')}", @@ -16419,6 +16476,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let publisher_response = PublisherResponse::Stream { response, @@ -16568,6 +16626,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -16958,6 +17017,7 @@ mod tests { price_granularity: PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } }; let make_stream_response = || PublisherResponse::Stream { @@ -17020,6 +17080,149 @@ mod tests { assert_bodiless_abandoned(&buffered_sink); } + #[test] + fn streaming_seam_wait_records_in_stream_placement() { + // The true Fastly streaming path: `publisher_response_into_streaming_response` + // hands back a lazy body after headers have already been committed by + // `stream_to_client()`. The `` seam wait polled from inside that body + // must therefore be attributed `InStream`, never `PreHeader`. + let settings = Arc::new(create_test_settings()); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new(settings.auction.clone())); + let timings = RequestTimings::new(); + + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html; charset=utf-8".to_string(); + params.dispatched_auction = Some(DispatchedAuction::empty_for_test( + test_auction_request(), + 500, + )); + params.auction_request = Some(test_auction_request()); + params.timings = timings.clone(); + + let response = Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build response"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello"), + bytes::Bytes::from_static(b""), + ])); + + let response = futures::executor::block_on(publisher_response_into_streaming_response( + PublisherResponse::Stream { + response, + body, + params: Box::new(params), + }, + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + noop_services(), + )) + .expect("streaming finalize should succeed"); + + // The wait is only recorded once the lazy body is actually polled — the + // finalizer call above only constructs it. + let drained = futures::executor::block_on( + response + .into_body() + .into_bytes_bounded(settings.publisher.max_buffered_body_bytes), + ) + .expect("body should drain"); + assert!( + String::from_utf8_lossy(&drained).contains("hello"), + "should still stream the document" + ); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::InStream), + "the streaming seam wait must be attributed InStream" + ); + assert!( + snapshot.auction_wait_ms.is_some(), + "should record an auction wait duration" + ); + } + + #[test] + fn buffered_template_miss_records_pre_header_placement() { + // The buffered finalizer materializes the entire response — headers and + // body — before any of it reaches the client. Even though the wait runs + // through the same `` seam code path as the streaming finalizer + // above, headers have not committed here, so it must be attributed + // `PreHeader`. (This exercises the same collect step a shared-template + // authorized miss rides through: `template_cache_key`'s presence only + // changes what happens *after* collection — whether the transformed bytes + // are stored — not where the wait itself is measured.) + let settings = create_test_settings(); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let services = noop_services(); + let timings = RequestTimings::new(); + + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html; charset=utf-8".to_string(); + params.dispatched_auction = Some(DispatchedAuction::empty_for_test( + test_auction_request(), + 500, + )); + params.auction_request = Some(test_auction_request()); + params.timings = timings.clone(); + + let response = Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build response"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello"), + bytes::Bytes::from_static(b""), + ])); + + let response = futures::executor::block_on(buffer_publisher_response_async( + PublisherResponse::Stream { + response, + body, + params: Box::new(params), + }, + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + )) + .expect("buffered finalize should succeed"); + + let html = String::from_utf8( + response + .into_body() + .into_bytes() + .unwrap_or_default() + .to_vec(), + ) + .expect("should be valid UTF-8"); + assert!(html.contains("hello"), "should still assemble the document"); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "the buffered finalizer's wait must be attributed PreHeader even though \ + it shares the seam code path with the streaming finalizer" + ); + assert!( + snapshot.auction_wait_ms.is_some(), + "should record an auction wait duration" + ); + } + #[test] fn publisher_response_streaming_finalize_processes_gzip_stream() { let compressed = @@ -17142,6 +17345,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let publisher_response = PublisherResponse::Stream { response, @@ -17214,6 +17418,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -17269,6 +17474,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let bogus_body = EdgeBody::from(b"not gzip".to_vec()); @@ -17382,6 +17588,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); stream_publisher_body(body, &mut output, ¶ms, &settings, ®istry) @@ -17444,6 +17651,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); From 8b028bb90aca365eddd4f106e3bc2f5ec818735b Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 09:27:44 -0700 Subject: [PATCH 009/104] Add access telemetry snapshot, route classes, and coarse route templates --- .../trusted-server-adapter-fastly/src/app.rs | 196 +++++++- .../trusted-server-adapter-fastly/src/main.rs | 283 +++++++++++- .../src/access_telemetry.rs | 418 ++++++++++++++++++ crates/trusted-server-core/src/constants.rs | 2 + crates/trusted-server-core/src/lib.rs | 1 + crates/trusted-server-core/src/publisher.rs | 78 +++- 6 files changed, 967 insertions(+), 11 deletions(-) create mode 100644 crates/trusted-server-core/src/access_telemetry.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 5cf7f9709..0a0cc8c5a 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -98,6 +98,7 @@ use edgezero_core::http::{ }; use edgezero_core::router::RouterService; use error_stack::Report; +use trusted_server_core::access_telemetry::{RouteClass, RouteMetadata, publisher_route_template}; use trusted_server_core::auction::AuctionTelemetrySink; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; @@ -303,6 +304,12 @@ fn uses_dynamic_tsjs_fallback(method: &Method, path: &str) -> bool { *method == Method::GET && path.starts_with("/static/tsjs=") } +/// Coarse route template for every `tsjs` bundle request, used as the +/// `route_template` in the [`RouteMetadata`] attached by the tsjs branch of +/// [`dispatch_fallback`]. Actual filenames vary by module/hash; the prefix +/// alone is the route identity that matters for access telemetry. +const TSJS_ROUTE_TEMPLATE: &str = "/static/tsjs=*"; + // --------------------------------------------------------------------------- // EC request state // --------------------------------------------------------------------------- @@ -846,12 +853,28 @@ async fn dispatch_fallback( PreRoute::Continue { effects } => effects, }; + // Assigned exactly once, per branch below, alongside the routing + // decision itself, so the access-telemetry route identity always + // reflects which branch actually dispatched the request — including + // when that branch's handler errors. The asset-route sub-branch is an + // early return handled separately by `dispatch_asset_fallback`, so it + // never reaches (or needs to assign) this binding. + let route_metadata: Option; + let result = if uses_dynamic_tsjs_fallback(&method, &path) { + route_metadata = Some(RouteMetadata { + route_class: RouteClass::Tsjs, + route_template: TSJS_ROUTE_TEMPLATE.to_owned(), + }); handle_tsjs_dynamic(&req, &state.registry, EdgeCacheHeader::SurrogateControl) } else if state.registry.has_route(&method, &path) { // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + route_metadata = Some(RouteMetadata { + route_class: RouteClass::IntegrationProxy, + route_template: publisher_route_template(&path), + }); state .registry .handle_proxy(ProxyDispatchInput { @@ -890,6 +913,11 @@ async fn dispatch_fallback( .await; } + route_metadata = Some(RouteMetadata { + route_class: RouteClass::PublisherHtml, + route_template: publisher_route_template(&path), + }); + // Generate an EC ID if needed — mirrors the legacy catch-all arm. // Only for document navigations by recognised browsers; subresource // requests may lack consent signals such as Sec-GPC. @@ -958,7 +986,10 @@ async fn dispatch_fallback( } }; - let response = result.unwrap_or_else(|e| http_error(&e)); + let mut response = result.unwrap_or_else(|e| http_error(&e)); + if let Some(metadata) = route_metadata { + response.extensions_mut().insert(metadata); + } attach_dispatch_extensions(response, ec, effects) } @@ -1154,6 +1185,10 @@ struct NamedRoute { path: &'static str, primary_methods: &'static [Method], handler: NamedRouteHandler, + /// Access-telemetry traffic category for this row. Attached verbatim + /// alongside `path` (the route-table pattern) to every response this + /// route produces — see [`named_route_handler`]. + route_class: RouteClass, } const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ @@ -1171,21 +1206,25 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/.well-known/trusted-server.json", primary_methods: &[Method::GET], handler: NamedRouteHandler::TrustedServerDiscovery, + route_class: RouteClass::Other, }, NamedRoute { path: "/verify-signature", primary_methods: &[Method::POST], handler: NamedRouteHandler::VerifySignature, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/keys/rotate", primary_methods: &[Method::POST], handler: NamedRouteHandler::RotateKey, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/keys/deactivate", primary_methods: &[Method::POST], handler: NamedRouteHandler::DeactivateKey, + route_class: RouteClass::Ec, }, // Admin EC lookup: the bare route reads the EC ID from the caller's // `ts-ec` cookie; the parameterized route takes an explicit EC ID. @@ -1193,11 +1232,13 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/_ts/admin/ec", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/ec/{id}", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, + route_class: RouteClass::Ec, }, // Admin EIDs echo: decodes the request's ts-eids/sharedId cookies with // an ingestion preview. Pure request inspection — no KV access. @@ -1205,6 +1246,7 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/_ts/admin/eids", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEidsLookup, + route_class: RouteClass::Ec, }, // The legacy non-`/_ts` aliases (`/admin/keys/*`) are denied locally with a // 404 instead of executing key operations: the production basic-auth handler @@ -1216,36 +1258,43 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/admin/keys/rotate", primary_methods: LEGACY_ADMIN_DENY_METHODS, handler: NamedRouteHandler::LegacyAdminDenied, + route_class: RouteClass::Other, }, NamedRoute { path: "/admin/keys/deactivate", primary_methods: LEGACY_ADMIN_DENY_METHODS, handler: NamedRouteHandler::LegacyAdminDenied, + route_class: RouteClass::Other, }, NamedRoute { path: "/_ts/api/v1/batch-sync", primary_methods: &[Method::POST], handler: NamedRouteHandler::BatchSync, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/api/v1/identify", primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::Identify, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/set-tester", primary_methods: &[Method::GET], handler: NamedRouteHandler::SetTester, + route_class: RouteClass::Other, }, NamedRoute { path: "/_ts/clear-tester", primary_methods: &[Method::GET], handler: NamedRouteHandler::ClearTester, + route_class: RouteClass::Other, }, NamedRoute { path: "/auction", primary_methods: &[Method::POST], handler: NamedRouteHandler::Auction, + route_class: RouteClass::AuctionApi, }, // GET runs the SPA re-auction; OPTIONS is denied in-handler as a CORS // preflight guard for this side-effecting endpoint. @@ -1253,6 +1302,7 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: PAGE_BIDS_PATH, primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::PageBids, + route_class: RouteClass::AuctionApi, }, // Deprecated double-underscore alias. tsjs bundles served before the // `/_ts/page-bids` rename keep requesting this path from already-loaded @@ -1263,21 +1313,29 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: PAGE_BIDS_LEGACY_PATH, primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::PageBids, + route_class: RouteClass::AuctionApi, }, + // Classified `Other` rather than `IntegrationProxy`: that class is + // reserved for `state.registry.handle_proxy` (the js-integration proxy + // dispatch in `dispatch_fallback`), which these first-party proxy routes + // do not go through. NamedRoute { path: "/first-party/proxy", primary_methods: &[Method::GET], handler: NamedRouteHandler::FirstPartyProxy, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/click", primary_methods: &[Method::GET], handler: NamedRouteHandler::FirstPartyClick, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/sign", primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartySign, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/proxy-rebuild", @@ -1286,16 +1344,35 @@ const NAMED_ROUTES: &[NamedRoute] = &[ // POST is blocked by CORS and the guard navigates here for a 302 instead. primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartyProxyRebuild, + route_class: RouteClass::Other, }, ]; +/// Wraps [`execute_named`], attaching a [`RouteMetadata`] extension carrying +/// `route_class` and the route-table pattern (`route_template`, verbatim, +/// with parameters left as placeholders) to every response the handler +/// produces — including its early-return diagnostic and setup-error arms, +/// since the attachment happens once around the whole future rather than in +/// each branch. fn named_route_handler( state: Arc, handler: NamedRouteHandler, + route_class: RouteClass, + route_template: &'static str, ) -> impl Fn(RequestContext) -> HandlerFuture + Clone + Send + Sync + 'static { move |ctx: RequestContext| { let state = Arc::clone(&state); - Box::pin(execute_named(state, ctx, handler)) + Box::pin(async move { + execute_named(state, ctx, handler) + .await + .map(|mut response| { + response.extensions_mut().insert(RouteMetadata { + route_class, + route_template: route_template.to_owned(), + }); + response + }) + }) } } @@ -1354,7 +1431,12 @@ impl TrustedServerApp { router = router.route( route.path, method.clone(), - named_route_handler(Arc::clone(state), route.handler), + named_route_handler( + Arc::clone(state), + route.handler, + route.route_class, + route.path, + ), ); } @@ -1392,7 +1474,8 @@ mod tests { use super::{ AppState, NAMED_ROUTES, NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, - TrustedServerApp, build_per_request_services, build_state_from_settings, + RouteClass, RouteMetadata, TSJS_ROUTE_TEMPLATE, TrustedServerApp, + build_per_request_services, build_state_from_settings, publisher_route_template, startup_error_router, }; use base64::Engine as _; @@ -2223,6 +2306,111 @@ mod tests { ); } + /// `Authorization: Basic` header value for `test_settings()`'s + /// `^/_ts/admin` handler (`admin` / `admin-pass`). + fn admin_basic_auth_header() -> edgezero_core::http::HeaderValue { + let credentials = base64::engine::general_purpose::STANDARD.encode("admin:admin-pass"); + format!("Basic {credentials}") + .parse() + .expect("should parse basic-auth header value") + } + + #[test] + fn named_route_attaches_the_table_pattern_verbatim_even_with_a_real_id_in_the_path() { + // A named-route response must carry the route-TABLE pattern + // (`{id}` left as a placeholder), never the caller's actual matched + // path segment — this is what keeps a real EC identifier out of + // access telemetry, independent of anything the row-serialization + // layer does. + let router = test_router(); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let mut req = empty_request(Method::GET, &format!("/_ts/admin/ec/{ec_id}")); + req.headers_mut() + .insert(header::AUTHORIZATION, admin_basic_auth_header()); + let response = route(&router, req); + + let metadata = response + .extensions() + .get::() + .expect("named-route responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Ec); + assert_eq!(metadata.route_template, "/_ts/admin/ec/{id}"); + assert!( + !metadata.route_template.contains(ec_id), + "the attached template must never contain the matched id" + ); + } + + #[test] + fn named_route_attaches_metadata_even_on_a_read_only_diagnostic_early_return() { + // AdminEidsLookup is handled by an early-return arm inside + // execute_named, before the normal EC lifecycle runs (see the + // "read-only diagnostics" comment there). named_route_handler wraps + // the whole future, so the attachment must still happen here too. + let router = test_router(); + let mut req = empty_request(Method::GET, "/_ts/admin/eids"); + req.headers_mut() + .insert(header::AUTHORIZATION, admin_basic_auth_header()); + let response = route(&router, req); + + let metadata = response + .extensions() + .get::() + .expect("even a read-only diagnostic early-return response should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Ec); + assert_eq!(metadata.route_template, "/_ts/admin/eids"); + } + + #[test] + fn tsjs_fallback_attaches_tsjs_route_metadata() { + let router = test_router(); + let response = route( + &router, + empty_request(Method::GET, "/static/tsjs=tsjs-unified.min.js"), + ); + + let metadata = response + .extensions() + .get::() + .expect("tsjs fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Tsjs); + assert_eq!(metadata.route_template, TSJS_ROUTE_TEMPLATE); + } + + #[test] + fn integration_proxy_fallback_attaches_integration_proxy_route_metadata() { + // test_settings() enables the prebid integration, which registers a + // proxy route at /integrations/prebid/bundle.js. + let router = test_router(); + let response = route( + &router, + empty_request(Method::GET, "/integrations/prebid/bundle.js"), + ); + + let metadata = response + .extensions() + .get::() + .expect("integration-proxy fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::IntegrationProxy); + assert_eq!( + metadata.route_template, + publisher_route_template("/integrations/prebid/bundle.js") + ); + } + + #[test] + fn publisher_fallback_attaches_publisher_html_route_metadata() { + let router = test_router(); + let response = route(&router, empty_request(Method::GET, "/news/some-article")); + + let metadata = response + .extensions() + .get::() + .expect("publisher fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::PublisherHtml); + assert_eq!(metadata.route_template, "/news/*"); + } + #[test] fn browser_device_signals_from_extension_reach_ec_finalize_state() { // Regression guard for the EdgeZero JA4/H2 signal loss: `edgezero_main` diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 5747dc8fd..c09663d13 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -14,9 +14,13 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; +use trusted_server_core::access_telemetry::{AccessTelemetrySnapshot, RouteClass, RouteMetadata}; use trusted_server_core::cache_policy::{ EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; +use trusted_server_core::constants::{ + ENV_FASTLY_IS_STAGING, ENV_FASTLY_POP, ENV_FASTLY_SERVICE_ID, ENV_FASTLY_SERVICE_VERSION, +}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -30,6 +34,7 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::publisher::TemplateCacheResponseState; use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -145,6 +150,18 @@ fn edgezero_main(mut req: FastlyRequest) { let server_timing_enabled = settings_snapshot .as_deref() .is_some_and(|settings| settings.observability.server_timing_enabled); + // Both read once here rather than at each `send_edgezero_response` call + // site: if `app_state` failed to build, there is no settings snapshot to + // read them from at all, so every call site would need the same + // degraded-mode fallback. `access_sample_rate` defaults to `0.0` (never + // sampled in) and `publisher_domain` to `"unknown"` in that case. + let access_sample_rate = settings_snapshot + .as_deref() + .map_or(0.0, |settings| settings.tinybird.access_sample_rate); + let publisher_domain = settings_snapshot.as_deref().map_or_else( + || "unknown".to_owned(), + |settings| settings.publisher.domain.clone(), + ); // Strip client-spoofable forwarded headers before dispatch. compat::sanitize_fastly_forwarded_headers(&mut req); @@ -159,8 +176,11 @@ fn edgezero_main(mut req: FastlyRequest) { req.set_header("fastly-ssl", "1"); } - // Capture client IP before the request is consumed by dispatch. + // Capture client IP and method before the request is consumed by + // dispatch: nothing else survives to the freeze point in + // `send_edgezero_response`, which only receives the response. let client_ip = req.get_client_ip_addr(); + let request_method = req.get_method_str().to_owned(); // Strip any client-supplied x-ts-tls-* headers before injecting the trusted // values from the Fastly SDK. Must run after sanitize_fastly_forwarded_headers. @@ -211,9 +231,13 @@ fn edgezero_main(mut req: FastlyRequest) { let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); + // Read rather than pop: the access-telemetry snapshot built later in + // `send_edgezero_response` reads this same extension, so it must still + // be attached to `response` at that point. let geo_lookup_state = response - .extensions_mut() - .remove::() + .extensions() + .get::() + .cloned() .unwrap_or(GeoLookupState::NotAttempted); if !take_finalize_sentinel(&mut response) { @@ -257,6 +281,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings: timings.clone(), server_timing_enabled, + method: request_method.clone(), + publisher_domain: publisher_domain.clone(), + access_sample_rate, }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); @@ -280,6 +307,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings: timings.clone(), server_timing_enabled, + method: request_method.clone(), + publisher_domain: publisher_domain.clone(), + access_sample_rate, }, ); run_edgezero_pull_sync_after_send( @@ -309,6 +339,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings, server_timing_enabled, + method: request_method, + publisher_domain, + access_sample_rate, }, ); } @@ -349,6 +382,18 @@ fn apply_entry_point_finalize_headers( }) }); apply_finalize_headers(settings, geo_info.as_ref(), response); + + // This path runs only when the middleware chain was bypassed (e.g. a + // router-level 404/405 for an unregistered method), so `geo_state` may + // still be `NotAttempted` even after a fresh lookup just ran above. + // Write the resolved outcome back so the access-telemetry snapshot built + // later in `send_edgezero_response` sees what was actually looked up, + // not the stale carried-in state. + let resolved_state = match &geo_info { + Some(info) => GeoLookupState::Resolved(info.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); } fn apply_edgezero_ec_finalize( @@ -394,6 +439,13 @@ struct SendContext { timings: RequestTimings, /// Whether `observability.server_timing_enabled` is set. server_timing_enabled: bool, + /// The request's HTTP method, captured before the request was consumed + /// by dispatch. + method: String, + /// The configured publisher domain. + publisher_domain: String, + /// The configured access-telemetry sample rate. + access_sample_rate: f64, } /// Outcome of handing a finalized response to the client. @@ -403,6 +455,9 @@ pub(crate) struct DeliveryOutcome { pub bytes: u64, /// Whether delivery completed or failed partway. pub result: DeliveryResult, + /// Access-telemetry dimensions captured for this response at the + /// freeze point. + pub snapshot: AccessTelemetrySnapshot, } /// Whether [`send_edgezero_response`] completed delivery or failed partway. @@ -558,6 +613,12 @@ fn send_edgezero_response( context.server_timing_enabled, ); + // Built unconditionally, right after the freeze point and before + // `into_parts()` consumes `response`: nothing else survives to + // post-send on every path (the request was consumed by dispatch, and + // `EcFinalizeState` is absent on asset, admin, and error paths). + let snapshot = build_access_telemetry_snapshot(&response, context); + let (parts, body) = response.into_parts(); match body { @@ -578,6 +639,7 @@ fn send_edgezero_response( Ok(()) => DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot, }, Err(e) => { // Every byte was handed to the transport (the drive @@ -588,13 +650,18 @@ fn send_edgezero_response( DeliveryOutcome { bytes, result: DeliveryResult::Partial, + snapshot, } } }, Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); - DeliveryOutcome { bytes, result } + DeliveryOutcome { + bytes, + result, + snapshot, + } } } } @@ -604,11 +671,89 @@ fn send_edgezero_response( DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot, } } } } +/// Builds the [`AccessTelemetrySnapshot`] for `response` at the +/// `Server-Timing` freeze point. +/// +/// Reads route identity, geo country, and template-cache state from typed +/// response extensions rather than the headers those extensions back — +/// operator-configured response headers can override a managed header, so +/// reading a header here could silently drift from what actually happened. +/// Falls back to `"unknown"`/[`RouteClass::Other`] sentinels when an +/// extension was never attached (router-generated, asset, and other +/// responses that never passed through a `RouteMetadata`-attaching +/// wrapper). +fn build_access_telemetry_snapshot( + response: &HttpResponse, + context: &SendContext, +) -> AccessTelemetrySnapshot { + let (route_class, route_template) = match response.extensions().get::() { + Some(metadata) => (metadata.route_class, metadata.route_template.clone()), + None => (RouteClass::Other, "unknown".to_owned()), + }; + + let country = match response.extensions().get::() { + Some(GeoLookupState::Resolved(info)) => info.country.clone(), + Some(GeoLookupState::Attempted | GeoLookupState::NotAttempted) | None => { + "unknown".to_owned() + } + }; + + let template_cache_state = response + .extensions() + .get::() + .map_or_else(|| "unknown".to_owned(), |state| state.as_str().to_owned()); + + let body_mode = if matches!(response.body(), EdgeBody::Stream(_)) { + "streamed" + } else { + "buffered" + }; + + AccessTelemetrySnapshot { + method: context.method.clone(), + status: response.status().as_u16(), + route_class, + route_template, + publisher_domain: context.publisher_domain.clone(), + env: resolve_env_dimension(), + service_id: env_var_or_unknown(ENV_FASTLY_SERVICE_ID), + pop: env_var_or_unknown(ENV_FASTLY_POP), + ts_version: env_var_or_unknown(ENV_FASTLY_SERVICE_VERSION), + country, + template_cache_state, + body_mode, + sample_rate: context.access_sample_rate, + } +} + +/// Derives the `env` access-telemetry dimension from the same +/// `FASTLY_IS_STAGING` input that drives the `x-ts-env` response header +/// (see [`apply_finalize_headers`]), never from [`Settings`] — `Settings` +/// has no environment field and does not gain one for this. +/// +/// `"unknown"` covers contexts where the variable is entirely absent (for +/// example native unit tests run outside Fastly Compute); on the Fastly +/// platform the variable is always present, as either `"1"` or not. +fn resolve_env_dimension() -> String { + match std::env::var(ENV_FASTLY_IS_STAGING) { + Ok(value) if value == "1" => "staging".to_owned(), + Ok(_) => "production".to_owned(), + Err(_) => "unknown".to_owned(), + } +} + +/// Reads a Fastly-provided environment variable, defaulting to `"unknown"` +/// when unset. +fn env_var_or_unknown(name: &str) -> String { + std::env::var(name).unwrap_or_else(|_| "unknown".to_owned()) +} + /// Apply every late response mutation, then restore privacy invariants before headers commit. fn apply_terminal_response_effects( response: &mut HttpResponse, @@ -820,6 +965,26 @@ mod tests { .expect("should parse test settings") } + /// A minimal [`AccessTelemetrySnapshot`] fixture for tests that only + /// need a `DeliveryOutcome` to exist, not its telemetry content. + fn sample_access_snapshot() -> AccessTelemetrySnapshot { + AccessTelemetrySnapshot { + method: "GET".to_owned(), + status: 200, + route_class: RouteClass::Other, + route_template: "/other/*".to_owned(), + publisher_domain: "unknown".to_owned(), + env: "unknown".to_owned(), + service_id: "unknown".to_owned(), + pop: "unknown".to_owned(), + ts_version: "unknown".to_owned(), + country: "unknown".to_owned(), + template_cache_state: "unknown".to_owned(), + body_mode: "buffered", + sample_rate: 0.0, + } + } + #[test] fn health_response_short_circuits_get_health() { let req = FastlyRequest::get("https://example.com/health"); @@ -1264,6 +1429,7 @@ mod tests { let outcome = DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot: sample_access_snapshot(), }; assert_eq!( @@ -1313,6 +1479,115 @@ mod tests { ); } + fn send_context_fixture() -> SendContext { + SendContext { + timings: RequestTimings::new(), + server_timing_enabled: false, + method: "GET".to_owned(), + publisher_domain: "test-publisher.com".to_owned(), + access_sample_rate: 0.25, + } + } + + #[test] + fn access_snapshot_defaults_when_no_extensions_are_attached() { + // Router-generated 404/405 responses and other paths that never pass + // through a RouteMetadata-attaching wrapper must still produce a + // usable snapshot: RouteClass::Other and "unknown" sentinels, never + // a missing/panicking build. + let response = response_builder() + .status(404) + .body(EdgeBody::empty()) + .expect("should build response"); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert_eq!(snapshot.status, 404); + assert_eq!(snapshot.method, "GET"); + assert!(matches!(snapshot.route_class, RouteClass::Other)); + assert_eq!(snapshot.route_template, "unknown"); + assert_eq!(snapshot.country, "unknown"); + assert_eq!(snapshot.template_cache_state, "unknown"); + assert_eq!(snapshot.body_mode, "buffered"); + assert_eq!(snapshot.publisher_domain, "test-publisher.com"); + assert_eq!(snapshot.sample_rate, 0.25); + } + + #[test] + fn access_snapshot_reads_route_geo_and_template_cache_extensions() { + let mut response = response_builder() + .status(200) + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(RouteMetadata { + route_class: RouteClass::AuctionApi, + route_template: "/auction".to_owned(), + }); + response.extensions_mut().insert(GeoLookupState::Resolved( + trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: "NorthAmerica".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }, + )); + response + .extensions_mut() + .insert(TemplateCacheResponseState::Hit); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert!(matches!(snapshot.route_class, RouteClass::AuctionApi)); + assert_eq!(snapshot.route_template, "/auction"); + assert_eq!(snapshot.country, "US"); + assert_eq!(snapshot.template_cache_state, "hit"); + } + + #[test] + fn access_snapshot_treats_attempted_geo_lookup_as_unknown_country() { + let mut response = response_builder() + .status(200) + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(GeoLookupState::Attempted); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert_eq!( + snapshot.country, "unknown", + "an attempted-but-unresolved lookup must not surface a stale country" + ); + } + + #[test] + fn access_snapshot_body_mode_reflects_the_response_body_variant() { + let streamed = response_builder() + .status(200) + .body(EdgeBody::stream(futures::stream::empty())) + .expect("should build streaming response"); + let buffered = response_builder() + .status(200) + .body(EdgeBody::from(b"hi".to_vec())) + .expect("should build buffered response"); + let context = send_context_fixture(); + + assert_eq!( + build_access_telemetry_snapshot(&streamed, &context).body_mode, + "streamed" + ); + assert_eq!( + build_access_telemetry_snapshot(&buffered, &context).body_mode, + "buffered" + ); + } + #[test] fn stream_drive_records_stream_ms_covering_the_in_stream_auction_wait() { // A streaming seam wait (Task 6, publisher.rs) records into the same diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs new file mode 100644 index 000000000..9daf77abc --- /dev/null +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -0,0 +1,418 @@ +//! Access telemetry: route classification and the per-request access log row. +//! +//! Extends the reserved `access_logs_raw` Tinybird datasource with bounded, +//! content-free route identity (see [`RouteClass`] and +//! [`publisher_route_template`]) instead of the raw request path, which would +//! otherwise carry identifiers, search terms, and other user-generated +//! content into a 30-day dataset. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` +//! section 9. + +use serde_json::json; + +use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; + +/// Maximum number of characters kept from a publisher path's first segment +/// by [`publisher_route_template`]. +const MAX_SEGMENT_LEN: usize = 32; + +/// Coarse traffic category for one response, used as a `LowCardinality` +/// dimension in the access telemetry row. +/// +/// Assigned per route by the adapter at dispatch time (see +/// [`RouteMetadata`]) rather than reconstructed from a handler enum or a +/// path regex at emission time, so the mapping lives in exactly one place. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RouteClass { + /// A publisher-origin page served through the assembly/auction pipeline. + PublisherHtml, + /// The unified `tsjs` static bundle. + Tsjs, + /// A request served by a registered [`crate::integrations::IntegrationProxy`]. + IntegrationProxy, + /// An Edge Cookie identity endpoint (verify, rotate, identify, admin + /// lookups, batch sync). + Ec, + /// The server-side auction or SPA re-auction (`page-bids`) endpoint. + AuctionApi, + /// Everything else: discovery, tester-cookie toggles, denied legacy + /// aliases, and any response with no attached [`RouteMetadata`]. + Other, +} + +impl RouteClass { + /// Renders this variant as the `snake_case` string stored in the + /// `route_class` column. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::PublisherHtml => "publisher_html", + Self::Tsjs => "tsjs", + Self::IntegrationProxy => "integration_proxy", + Self::Ec => "ec", + Self::AuctionApi => "auction_api", + Self::Other => "other", + } + } +} + +/// Route identity for one response, carried from dispatch to the freeze +/// point as a response extension. +/// +/// The matched route pattern does not otherwise survive dispatch: the +/// request is consumed by the router, and nothing else records which +/// route-table row (or coarse fallback bucket) produced the response. Each +/// named-route handler wrapper attaches its matched route-table pattern +/// verbatim; the publisher fallback and `tsjs` handlers attach their class +/// plus a coarse template ([`publisher_route_template`] for the former). +#[derive(Debug, Clone)] +pub struct RouteMetadata { + /// Coarse traffic category for this response. + pub route_class: RouteClass, + /// Bounded, content-free route identifier. For named routes this is the + /// route-table pattern verbatim (e.g. `/_ts/admin/ec/{id}`); for + /// publisher-fallback traffic it is the output of + /// [`publisher_route_template`]. + pub route_template: String, +} + +/// Normalizes a publisher-fallback request path into a bounded, +/// content-free route template. +/// +/// Returns `/` plus the first path segment, lowercased and restricted to +/// `[a-z0-9_-]`, truncated to [`MAX_SEGMENT_LEN`] characters, with a +/// trailing `/*` appended when the path has additional segments beyond the +/// first. The root path `/` maps to itself. An empty first segment, or one +/// containing any character outside the allowlist (after lowercasing), +/// maps to `/other/*` — the segment is rejected outright rather than +/// filtered, so no fragment of a disallowed segment (an email address, a +/// search phrase) ever reaches the row. +/// +/// This is deliberately coarser than the auction-telemetry path +/// normalizer, which redacts long tokens but preserves short identifiers +/// and arbitrary slugs; that normalizer is not sufficient for a dataset +/// this broad. +/// +/// # Examples +/// +/// ``` +/// use trusted_server_core::access_telemetry::publisher_route_template; +/// +/// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); +/// assert_eq!(publisher_route_template("/"), "/"); +/// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// ``` +#[must_use] +pub fn publisher_route_template(path: &str) -> String { + if path == "/" { + return "/".to_owned(); + } + + let trimmed = path.strip_prefix('/').unwrap_or(path); + let (first_segment, rest) = match trimmed.split_once('/') { + Some((first, rest)) => (first, rest), + None => (trimmed, ""), + }; + let has_more_depth = !rest.is_empty(); + + let lowered = first_segment.to_ascii_lowercase(); + let is_allowlisted = !lowered.is_empty() + && lowered + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); + + if !is_allowlisted { + return "/other/*".to_owned(); + } + + let truncated: String = lowered.chars().take(MAX_SEGMENT_LEN).collect(); + if has_more_depth { + format!("/{truncated}/*") + } else { + format!("/{truncated}") + } +} + +/// A point-in-time view of the access-log dimensions for one response, +/// captured unconditionally at the `Server-Timing` freeze point. +/// +/// Built from typed response extensions ([`RouteMetadata`], the geo +/// lookup state, and the template-cache response state) plus adapter-owned +/// environment values, never from the public response headers those +/// extensions back — operator-configured response headers can override a +/// managed header, so reading the header instead of the extension would let +/// the row silently drift from the extension of record. +#[derive(Debug, Clone)] +pub struct AccessTelemetrySnapshot { + /// The request's HTTP method (e.g. `GET`). + pub method: String, + /// The response's HTTP status code. + pub status: u16, + /// Coarse traffic category (see [`RouteClass`]). + pub route_class: RouteClass, + /// Bounded, content-free route identifier (see + /// [`publisher_route_template`]). + pub route_template: String, + /// The configured publisher domain. + pub publisher_domain: String, + /// Adapter-derived deployment environment: `production`, `staging`, or + /// `unknown`. + pub env: String, + /// Fastly service ID, or `unknown` when unavailable. + pub service_id: String, + /// Fastly POP code, or `unknown` when unavailable. + pub pop: String, + /// Trusted Server build/version identifier, or `unknown` when + /// unavailable. + pub ts_version: String, + /// Two-letter geo country code, or `unknown` when no geo lookup + /// resolved one. + pub country: String, + /// Template-cache outcome for this response, or `unknown` when the + /// response never passed through the assembly pipeline. + pub template_cache_state: String, + /// Whether the response body was streamed or buffered to the client. + pub body_mode: &'static str, + /// The configured access-telemetry sample rate at the time this + /// response was handled. + pub sample_rate: f64, +} + +/// Renders one NDJSON access-log row for the Tinybird Events API. +/// +/// Column names match spec section 9 exactly. Phase columns come from +/// `timings` and serialize as JSON `null` for phases that were never +/// recorded; every dimension column comes from `snapshot` and is a +/// non-nullable string (callers are expected to substitute an `unknown` +/// sentinel rather than leave a dimension empty). `event_date` is omitted: +/// the datasource derives it from `event_ts` by default. +#[must_use] +pub fn access_event_row( + snapshot: &AccessTelemetrySnapshot, + timings: &TimingSnapshot, + event_ts_epoch_ms: u64, +) -> String { + let auction_wait_placement = match timings.auction_wait_placement { + Some(AuctionWaitPlacement::PreHeader) => "pre_header", + Some(AuctionWaitPlacement::InStream) => "in_stream", + None => "none", + }; + + let row = json!({ + "event_ts": format_event_timestamp(event_ts_epoch_ms), + "method": snapshot.method, + "status": snapshot.status, + "time_elapsed_ms": timings.time_elapsed_ms, + "sample_rate": snapshot.sample_rate, + "service_id": snapshot.service_id, + "publisher_domain": snapshot.publisher_domain, + "env": snapshot.env, + "route_class": snapshot.route_class.as_str(), + "route_template": snapshot.route_template, + "body_mode": snapshot.body_mode, + "auction_wait_placement": auction_wait_placement, + "appbuild_ms": timings.appbuild_ms, + "filter_ms": timings.filter_ms, + "geo_ms": timings.geo_ms, + "kv_ms": timings.kv_ms, + "origin_ms": timings.origin_ms, + "template_cache_ms": timings.template_cache_ms, + "auction_wait_ms": timings.auction_wait_ms, + "stream_ms": timings.stream_ms, + "request_elapsed_ms": timings.request_elapsed_ms, + "resp_bytes": timings.resp_bytes, + "template_cache_state": snapshot.template_cache_state, + "country": snapshot.country, + "ts_version": snapshot.ts_version, + "pop": snapshot.pop, + }); + row.to_string() +} + +/// Formats `epoch_ms` as a `ClickHouse` `DateTime64(3)`-compatible string +/// (`%Y-%m-%d %H:%M:%S%.3f`), matching the format the auction telemetry +/// sink already uses for `event_ts`. +/// +/// Falls back to the Unix epoch when `epoch_ms` cannot be represented as a +/// valid timestamp, which never happens for a real wall-clock reading. +fn format_event_timestamp(epoch_ms: u64) -> String { + let epoch_ms = i64::try_from(epoch_ms).unwrap_or(i64::MAX); + let timestamp = chrono::DateTime::::from_timestamp_millis(epoch_ms) + .unwrap_or(chrono::DateTime::::UNIX_EPOCH); + timestamp.format("%Y-%m-%d %H:%M:%S%.3f").to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A 64-hex-plus-suffix EC identifier, matching the format the admin + /// EC lookup route (`/_ts/admin/ec/{id}`) accepts as a path parameter. + const SYNTHETIC_EC_ID: &str = + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + + fn unknown_snapshot(route_class: RouteClass, route_template: &str) -> AccessTelemetrySnapshot { + AccessTelemetrySnapshot { + method: "GET".to_owned(), + status: 200, + route_class, + route_template: route_template.to_owned(), + publisher_domain: "unknown".to_owned(), + env: "unknown".to_owned(), + service_id: "unknown".to_owned(), + pop: "unknown".to_owned(), + ts_version: "unknown".to_owned(), + country: "unknown".to_owned(), + template_cache_state: "unknown".to_owned(), + body_mode: "buffered", + sample_rate: 0.0, + } + } + + #[test] + fn admin_ec_route_template_never_contains_the_identifier() { + // Named-route templates come from the route table verbatim, never + // from the matched request path, so a row built for this route can + // never carry the caller's actual EC id — even when a caller + // supplies one shaped exactly like a real one. + let snapshot = unknown_snapshot(RouteClass::Ec, "/_ts/admin/ec/{id}"); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + + assert!( + !row.contains(SYNTHETIC_EC_ID), + "row must never contain a literal EC identifier: {row}" + ); + assert!( + row.contains("/_ts/admin/ec/{id}"), + "row should still carry the route-table pattern: {row}" + ); + } + + #[test] + fn publisher_paths_normalize_to_coarse_templates() { + assert_eq!( + publisher_route_template("/news/some-article-slug"), + "/news/*" + ); + assert_eq!(publisher_route_template("/"), "/"); + assert_eq!( + publisher_route_template("/user@example.com/profile"), + "/other/*", + "should reject non-allowlisted characters" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a".repeat(500))), + format!("/{}", "a".repeat(32)), + "should bound segment length" + ); + assert_eq!(publisher_route_template("/search terms here"), "/other/*"); + } + + #[test] + fn publisher_route_template_rejects_empty_first_segment() { + assert_eq!( + publisher_route_template("//double-slash"), + "/other/*", + "an empty first segment should not be treated as allowlisted" + ); + } + + #[test] + fn publisher_route_template_uppercases_lowercase_before_allowlisting() { + assert_eq!( + publisher_route_template("/News/Article"), + "/news/*", + "should lowercase before validating and truncating" + ); + } + + #[test] + fn row_serializes_nulls_for_missing_phases() { + let snapshot = unknown_snapshot(RouteClass::Other, "/other/*"); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + for field in [ + "time_elapsed_ms", + "appbuild_ms", + "filter_ms", + "geo_ms", + "kv_ms", + "origin_ms", + "template_cache_ms", + "auction_wait_ms", + "stream_ms", + "request_elapsed_ms", + "resp_bytes", + ] { + assert!( + parsed[field].is_null(), + "unrecorded phase `{field}` should serialize as null: {row}" + ); + } + + for field in [ + "service_id", + "publisher_domain", + "env", + "route_class", + "route_template", + "body_mode", + "template_cache_state", + "country", + "ts_version", + "pop", + ] { + assert!( + parsed[field].is_string(), + "dimension `{field}` must never be null: {row}" + ); + } + assert_eq!(parsed["auction_wait_placement"], "none"); + } + + #[test] + fn row_serializes_recorded_phases_as_numbers() { + let snapshot = unknown_snapshot(RouteClass::AuctionApi, "/auction"); + let timings = TimingSnapshot { + time_elapsed_ms: Some(12), + request_elapsed_ms: Some(15), + appbuild_ms: Some(1), + filter_ms: Some(2), + geo_ms: Some(3), + kv_ms: Some(4), + origin_ms: Some(5), + template_cache_ms: Some(6), + auction_wait_ms: Some(7), + stream_ms: Some(8), + auction_wait_placement: Some(AuctionWaitPlacement::InStream), + resp_bytes: Some(1024), + }; + let row = access_event_row(&snapshot, &timings, 1_700_000_000_000); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + assert_eq!(parsed["appbuild_ms"], 1); + assert_eq!(parsed["stream_ms"], 8); + assert_eq!(parsed["resp_bytes"], 1024); + assert_eq!(parsed["auction_wait_placement"], "in_stream"); + } + + #[test] + fn route_class_renders_snake_case() { + assert_eq!(RouteClass::PublisherHtml.as_str(), "publisher_html"); + assert_eq!(RouteClass::Tsjs.as_str(), "tsjs"); + assert_eq!(RouteClass::IntegrationProxy.as_str(), "integration_proxy"); + assert_eq!(RouteClass::Ec.as_str(), "ec"); + assert_eq!(RouteClass::AuctionApi.as_str(), "auction_api"); + assert_eq!(RouteClass::Other.as_str(), "other"); + } + + #[test] + fn format_event_timestamp_matches_clickhouse_datetime64_shape() { + // 2023-11-14T22:13:20.000Z + let rendered = format_event_timestamp(1_700_000_000_000); + assert_eq!(rendered, "2023-11-14 22:13:20.000"); + } +} diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index e1152b1e7..1f85facb8 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -34,6 +34,8 @@ pub const HEADER_X_TS_ENV: HeaderName = HeaderName::from_static("x-ts-env"); // Fastly environment variables pub const ENV_FASTLY_SERVICE_VERSION: &str = "FASTLY_SERVICE_VERSION"; pub const ENV_FASTLY_IS_STAGING: &str = "FASTLY_IS_STAGING"; +pub const ENV_FASTLY_SERVICE_ID: &str = "FASTLY_SERVICE_ID"; +pub const ENV_FASTLY_POP: &str = "FASTLY_POP"; // Common standard header names used across modules pub const HEADER_USER_AGENT: HeaderName = HeaderName::from_static("user-agent"); diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index b8a1a5718..de1bbad21 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -31,6 +31,7 @@ ) )] +pub mod access_telemetry; pub(crate) mod asset_image_optimizer; pub mod auction; pub mod auction_config_types; diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 1943d9b3c..6a30611c8 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -91,21 +91,44 @@ const DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(15); const HEADER_X_TS_TEMPLATE_CACHE: &str = "x-ts-template-cache"; const HEADER_X_TS_ASSEMBLY: &str = "x-ts-assembly"; -#[derive(Clone, Copy, PartialEq, Eq)] -enum TemplateCacheResponseState { +/// Outcome of a template-cache lookup/store attempt for one response. +/// +/// Set on every response that passes through the assembly pipeline via +/// [`set_template_cache_response_state`], which writes both the +/// `x-ts-template-cache` response header and this same value as a typed +/// response extension, so the two can never drift. Access telemetry reads +/// the extension rather than the header, since operator-configured response +/// headers can override a managed header but cannot touch extensions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TemplateCacheResponseState { + /// The cached template was found and reused. Hit, + /// No cached template existed; the cache store is reserved for this + /// content type. MissReserved, + /// No cached template existed; one was stored after assembly. MissStored, + /// No cached template existed; storing the freshly assembled template + /// failed. MissStoreError, + /// The request bypassed the cache lookup. BypassRequest, + /// The response bypassed the cache store. BypassResponse, + /// The response's content type is not supported by the template cache. Unsupported, + /// The cached template entry was invalid and could not be reused. Invalid, + /// A backend error prevented the cache lookup or store. BackendError, } impl TemplateCacheResponseState { - const fn as_str(self) -> &'static str { + /// Renders this variant as the string written to the + /// `x-ts-template-cache` header and the `template_cache_state` access + /// telemetry column. + #[must_use] + pub const fn as_str(self) -> &'static str { match self { Self::Hit => "hit", Self::MissReserved => "miss-reserved", @@ -128,6 +151,7 @@ fn set_template_cache_response_state( HEADER_X_TS_TEMPLATE_CACHE, HeaderValue::from_static(state.as_str()), ); + response.extensions_mut().insert(state); } #[derive(Clone, Copy, PartialEq, Eq)] @@ -9057,6 +9081,54 @@ mod tests { ); } + #[tokio::test] + async fn template_cache_response_extension_matches_the_header_on_every_transition() { + // The typed extension and the `x-ts-template-cache` header are written + // together by a single setter, so they must always agree — access + // telemetry reads the extension precisely because it cannot drift from + // what an operator-configured header override might otherwise show. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + + queue_shareable_html(&stub); + + let cold = run(&settings, &services, navigation_request()).await; + let cold_header = cold + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let cold_extension = cold + .extensions() + .get::() + .map(|state| state.as_str()); + assert_eq!( + cold_extension, + cold_header.as_deref(), + "the cold-fill extension must match the header" + ); + assert_eq!(cold_extension, Some("miss-stored")); + + let warm = run(&settings, &services, navigation_request()).await; + let warm_header = warm + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let warm_extension = warm + .extensions() + .get::() + .map(|state| state.as_str()); + assert_eq!( + warm_extension, + warm_header.as_deref(), + "the warm-hit extension must match the header" + ); + assert_eq!(warm_extension, Some("hit")); + } + #[tokio::test] async fn template_cache_span_recorded_only_when_lookup_runs() { // Inline mode: no shared-cache key is ever computed, so the lookup From 48acd816d7044c5a6d77d6780e0afdf1473bd214 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:00:30 -0700 Subject: [PATCH 010/104] Emit confirmed access telemetry rows after pull-sync post-send --- .../trusted-server-adapter-fastly/src/main.rs | 207 ++++++++++- .../src/tinybird.rs | 336 +++++++++++++++++- 2 files changed, 524 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index c09663d13..65e89cd1c 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,5 +1,5 @@ use std::sync::Arc; -use std::time::Instant; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; use edgezero_adapter_fastly::request::into_core_request; @@ -14,7 +14,9 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; -use trusted_server_core::access_telemetry::{AccessTelemetrySnapshot, RouteClass, RouteMetadata}; +use trusted_server_core::access_telemetry::{ + AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, +}; use trusted_server_core::cache_policy::{ EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; @@ -275,7 +277,7 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { @@ -287,6 +289,7 @@ fn edgezero_main(mut req: FastlyRequest) { }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); + emit_access_telemetry_after_send(settings, &outcome, &timings); return; } Err(e) => { @@ -301,7 +304,7 @@ fn edgezero_main(mut req: FastlyRequest) { match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { @@ -317,6 +320,7 @@ fn edgezero_main(mut req: FastlyRequest) { &partner_registry, &ec_state, ); + emit_access_telemetry_after_send(&settings, &outcome, &timings); return; } Err(e) => { @@ -333,17 +337,31 @@ fn edgezero_main(mut req: FastlyRequest) { } } - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { - timings, + timings: timings.clone(), server_timing_enabled, method: request_method, publisher_domain, access_sample_rate, }, ); + // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec + // finalize branch above failed), so there is no pull-sync dispatch here + // at all — telemetry is the only post-send step. Reload settings when + // `app_state` never built, matching the fallback used earlier in this + // function for entry-point finalize headers. + match settings_snapshot.as_deref() { + Some(settings) => emit_access_telemetry_after_send(settings, &outcome, &timings), + None => match load_settings_from_config_store() { + Ok(settings) => emit_access_telemetry_after_send(&settings, &outcome, &timings), + Err(e) => { + log::warn!("access telemetry emission skipped: failed to reload settings: {e:?}"); + } + }, + } } fn edge_error_response(error: EdgeError) -> HttpResponse { @@ -432,6 +450,59 @@ fn run_edgezero_pull_sync_after_send( } } +/// Builds and emits the access-telemetry row for one delivered response, +/// when access telemetry is enabled and this request is sampled in. +/// +/// Called last at every `send_edgezero_response` call site in +/// [`edgezero_main`] — after `run_edgezero_pull_sync_after_send` on the two +/// EC-finalized paths, and directly after send on the asset/admin/error +/// fallback path, which never builds an [`EcFinalizeState`] or route-scoped +/// `RuntimeServices` at all. The Tinybird transport context is therefore +/// constructed fresh from `settings` here rather than threaded through +/// either of those per-route types, so every response class can emit. +/// +/// Sampled-out requests return silently — that is the expected, high-volume +/// case and not worth a log line. Every other drop (row build, token load, +/// send, or non-2xx status — all folded into `emit_access_event`'s `Result`) +/// logs exactly one warning naming the reason. +fn emit_access_telemetry_after_send( + settings: &Settings, + outcome: &DeliveryOutcome, + timings: &RequestTimings, +) { + if !settings.tinybird.enabled || !settings.tinybird.access_enabled { + return; + } + + let since_epoch = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default(); + let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); + // Entropy for the sampling decision: the timestamp's nanosecond + // resolution XORed with a per-request value already on hand + // (`outcome.bytes`), so two requests handled in the same instance never + // collide on sampling decisions purely because they read the same + // millisecond. There is no `rand` crate dependency here — see + // `tinybird::sampled_in`. + let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); + let entropy = entropy_nanos ^ outcome.bytes; + + if !tinybird::sampled_in(settings.tinybird.access_sample_rate, entropy) { + return; + } + + let row = access_event_row(&outcome.snapshot, &timings.snapshot(), epoch_ms); + let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); + let result = futures::executor::block_on(tinybird::emit_access_event( + &platform::FastlyPlatformHttpClient, + &target, + row, + )); + if let Err(error) = result { + log::warn!("access telemetry emission dropped: {error:?}"); + } +} + /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -935,6 +1006,7 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use std::sync::Mutex; use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; use trusted_server_core::request_timing::AuctionWaitPlacement; @@ -1656,4 +1728,127 @@ mod tests { DeliveryResult::Complete ); } + + /// Records `"telemetry"` into a shared order log instead of sending a + /// real request, standing in for the adapter's platform HTTP client in + /// [`post_send_order_is_elapsed_then_pull_sync_then_telemetry`]. + struct OrderingHttpClient { + log: Arc>>, + } + + #[async_trait::async_trait(?Send)] + impl trusted_server_core::platform::PlatformHttpClient for OrderingHttpClient { + async fn send( + &self, + _request: trusted_server_core::platform::PlatformHttpRequest, + ) -> Result< + trusted_server_core::platform::PlatformResponse, + Report, + > { + self.log + .lock() + .expect("should lock order log") + .push("telemetry"); + let response = response_builder() + .status(edgezero_core::http::StatusCode::ACCEPTED) + .body(EdgeBody::empty()) + .expect("should build ordering test response"); + Ok(trusted_server_core::platform::PlatformResponse::new( + response, + )) + } + + async fn send_async( + &self, + _request: trusted_server_core::platform::PlatformHttpRequest, + ) -> Result< + trusted_server_core::platform::PlatformPendingRequest, + Report, + > { + Err(Report::new( + trusted_server_core::platform::PlatformError::Unsupported, + )) + } + + async fn select( + &self, + _pending_requests: Vec, + ) -> Result< + trusted_server_core::platform::PlatformSelectResult, + Report, + > { + Err(Report::new( + trusted_server_core::platform::PlatformError::Unsupported, + )) + } + } + + #[test] + fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // `edgezero_main` cannot be driven directly in a unit test (it + // consumes a live `fastly::Request::from_client()`), and + // `run_edgezero_pull_sync_after_send` has no injectable seam of its + // own — it dispatches through the real identity-graph pull-sync + // path, which needs a configured EC KV store, partner registry, and + // rate limiter wired together. This test instead exercises the two + // REAL functions `edgezero_main` calls that DO have a testable seam + // — `send_edgezero_response` (which stamps `request_elapsed` before + // returning, per Task 6/7) and `tinybird::emit_access_event` (the + // telemetry send added by this task) — around an instrumented + // stand-in for the pull-sync dispatch call, in the exact order + // `edgezero_main` places them. + // + // This proves the elapsed-before-telemetry leg from real production + // code (the assertion below reads the real `timings` snapshot + // between the two calls). The pull-sync-before-telemetry leg is a + // source-order invariant in `edgezero_main`'s three call sites + // (verified by code review, not by this test) because + // `run_edgezero_pull_sync_after_send` itself has no seam to + // instrument — see task-8-report.md for this residual. + let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + let timings = RequestTimings::new(); + let response = response_builder() + .body(EdgeBody::from("ok")) + .expect("should build response"); + + let outcome = send_edgezero_response( + response, + None, + &SendContext { + timings: timings.clone(), + server_timing_enabled: false, + method: "GET".to_owned(), + publisher_domain: "test-publisher.com".to_owned(), + access_sample_rate: 1.0, + }, + ); + assert!( + timings.snapshot().request_elapsed_ms.is_some(), + "request_elapsed should already be stamped before pull-sync/telemetry run" + ); + + // Stand-in for `run_edgezero_pull_sync_after_send`, which has no + // injectable seam (see the test doc comment above). + log.lock().expect("should lock order log").push("pull_sync"); + + let http_client = OrderingHttpClient { + log: Arc::clone(&log), + }; + let target = tinybird::TinybirdEventsTarget::from_access_config( + trusted_server_core::settings::TinybirdSettings { + api_host: "api.us-east.aws.tinybird.co".to_owned(), + ..trusted_server_core::settings::TinybirdSettings::default() + }, + ); + let row = access_event_row(&outcome.snapshot, &timings.snapshot(), 0); + + futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) + .expect("should send access telemetry"); + + assert_eq!( + *log.lock().expect("should lock order log"), + vec!["pull_sync", "telemetry"], + "pull-sync must dispatch before telemetry emits" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 08b5811fc..5025f9a73 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -11,10 +11,13 @@ use trusted_server_core::auction::telemetry::{ }; use trusted_server_core::error::TrustedServerError; use trusted_server_core::platform::{ - PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, StoreName, + PlatformBackend as _, PlatformBackendSpec, PlatformHttpClient, PlatformHttpRequest, + PlatformSecretStore as _, RuntimeServices, StoreName, }; use trusted_server_core::settings::{Settings, TinybirdSettings}; +use crate::platform::{FastlyPlatformBackend, FastlyPlatformSecretStore}; + const TINYBIRD_EVENTS_PATH: &str = "/v0/events"; const TINYBIRD_NDJSON_CONTENT_TYPE: &str = "application/x-ndjson"; const TINYBIRD_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(2); @@ -45,7 +48,7 @@ struct FastlyTinybirdAuctionTelemetrySink { } #[derive(Debug, Clone)] -struct TinybirdEventsTarget { +pub(crate) struct TinybirdEventsTarget { api_host: String, dataset: String, secret_store: StoreName, @@ -69,6 +72,27 @@ impl TinybirdEventsTarget { max_body_bytes: config.max_body_bytes, } } + + /// Builds the Events API target for the access-log datasource. + /// + /// Shares [`from_config`](Self::from_config)'s host/secret-store/ + /// body-size-limit derivation, but points at `access_dataset` and + /// `access_token_secret` instead of the auction pair, so access-log + /// emission never shares a datasource or token with auction telemetry + /// even though both configs come from the same [`TinybirdSettings`]. + pub(crate) fn from_access_config(config: TinybirdSettings) -> Self { + let uri = tinybird_events_uri(&config.api_host, &config.access_dataset); + let backend_spec = tinybird_backend_spec(&config.api_host); + Self { + api_host: config.api_host, + dataset: config.access_dataset, + secret_store: StoreName::from(config.secret_store), + token_secret: config.access_token_secret, + uri, + backend_spec, + max_body_bytes: config.max_body_bytes, + } + } } impl FastlyTinybirdAuctionTelemetrySink { @@ -208,6 +232,149 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { } } +// --------------------------------------------------------------------------- +// Access telemetry: confirmed-delivery emitter +// --------------------------------------------------------------------------- + +/// Bucket count [`sampled_in`] maps `entropy` into. +/// +/// Large enough that `rate` values with several significant digits (e.g. +/// `0.015`) still land in a distinct bucket instead of rounding away, while +/// staying well inside `u64` range once multiplied by `rate`. +const ACCESS_SAMPLE_BUCKETS: u64 = 1_000_000; + +/// Decides whether one request's access-telemetry row should be emitted. +/// +/// `entropy` should vary from request to request — callers derive it from +/// the wall-clock event timestamp `XORed` with a cheap per-request value (see +/// the call site in `main.rs`). There is no `rand` crate dependency here: +/// the wasm32-wasip1 guest has no equivalent to `Math.random()`. Mapping +/// `entropy % ACCESS_SAMPLE_BUCKETS` into `[0, 1)` and comparing against +/// `rate` is not cryptographically uniform (the low bits of a timestamp are +/// not perfectly evenly distributed), but access-telemetry sampling only +/// needs an approximately even sample, not a provably unbiased one. +/// +/// `rate <= 0.0` always returns `false` and `rate >= 1.0` always returns +/// `true`, independent of `entropy`, so both boundary configurations behave +/// predictably. `0.0` cannot actually occur while `access_enabled` is `true` +/// (`Settings` validation requires `access_sample_rate > 0.0` in that case), +/// but this function stays total rather than leaning on that invariant. +#[must_use] +pub(crate) fn sampled_in(rate: f64, entropy: u64) -> bool { + if rate >= 1.0 { + return true; + } + if rate <= 0.0 { + return false; + } + let threshold = (rate * ACCESS_SAMPLE_BUCKETS as f64) as u64; + entropy % ACCESS_SAMPLE_BUCKETS < threshold +} + +/// Loads and validates the access-log APPEND token from the Fastly secret store. +/// +/// Constructs [`FastlyPlatformSecretStore`] directly instead of routing +/// through [`RuntimeServices`]: access-telemetry emission runs post-delivery +/// for every response class — including asset, admin, and error responses +/// that never build a route-scoped `RuntimeServices` — so the transport +/// context here must be adapter-owned and route-independent rather than +/// threaded from wherever the route happened to construct one. +fn load_access_token(target: &TinybirdEventsTarget) -> Result> { + let token = FastlyPlatformSecretStore + .get_string(&target.secret_store, &target.token_secret) + .change_context(TrustedServerError::Proxy { + message: "Tinybird access append token unavailable".to_owned(), + })?; + let token = token.trim().to_owned(); + if token.is_empty() { + return Err(Report::new(TrustedServerError::Proxy { + message: "Tinybird access append token is empty".to_owned(), + })); + } + Ok(token) +} + +/// Builds the Events API POST request for one access-log row. +fn build_access_events_request( + target: &TinybirdEventsTarget, + body: String, + auth_header: HeaderValue, +) -> Result> { + request_builder() + .method(Method::POST) + .uri(target.uri.as_str()) + .header(header::AUTHORIZATION, auth_header) + .header(header::CONTENT_TYPE, TINYBIRD_NDJSON_CONTENT_TYPE) + .body(Body::from(body)) + .change_context(TrustedServerError::Proxy { + message: "failed to build Tinybird Events API request".to_owned(), + }) +} + +/// Sends one confirmed access-log row to the Tinybird Events API and waits +/// for the response. +/// +/// Unlike [`FastlyTinybirdAuctionTelemetrySink::emit_auction_events`] (fire- +/// and-forget, dispatched mid-request so it never adds latency to the +/// response), this runs post-delivery: the response has already reached the +/// client, so there is no latency budget left to protect, and the send can +/// afford to wait for — and validate — the reply. `client` is the adapter's +/// stateless platform HTTP client in production +/// ([`crate::platform::FastlyPlatformHttpClient`]); accepting it as `&dyn +/// PlatformHttpClient` here (rather than that concrete type) is what lets +/// tests substitute a recording double instead of performing a real network +/// send, matching how [`RuntimeServices::http_client`] is consumed +/// elsewhere. `target` is derived from settings once at the post-send call +/// site rather than threaded through any per-route state. +/// +/// A non-2xx status is reported as `Err` naming the status; there is no +/// retry — the caller logs exactly one warning and moves on. +/// +/// # Errors +/// +/// Returns `Err` when the access-log APPEND token cannot be loaded, the +/// backend cannot be registered, the request cannot be built or sent, or the +/// Tinybird Events API responds with a non-2xx status. +pub(crate) async fn emit_access_event( + client: &dyn PlatformHttpClient, + target: &TinybirdEventsTarget, + row: String, +) -> Result<(), Report> { + let token = load_access_token(target)?; + let auth_header = FastlyTinybirdAuctionTelemetrySink::authorization_header(&token)?; + let backend_name = FastlyPlatformBackend + .ensure(&target.backend_spec) + .change_context(TrustedServerError::Proxy { + message: "Tinybird backend registration failed".to_owned(), + })?; + let request = build_access_events_request(target, row, auth_header)?; + + log::info!( + "sending access telemetry to Tinybird dataset={} host={} backend={}", + target.dataset, + target.api_host, + backend_name + ); + + let response = client + .send(PlatformHttpRequest::new(request, backend_name)) + .await + .change_context(TrustedServerError::Proxy { + message: "failed to send Tinybird access telemetry request".to_owned(), + })?; + + if response.response.status().is_success() { + Ok(()) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: format!( + "Tinybird access telemetry request failed with status {}", + response.response.status() + ), + })) + } +} + fn tinybird_backend_spec(api_host: &str) -> PlatformBackendSpec { PlatformBackendSpec { scheme: "https".to_owned(), @@ -327,25 +494,28 @@ mod tests { body: Vec, } + /// Records outbound requests and, for [`PlatformHttpClient::send`] (the + /// blocking variant `emit_access_event` uses), returns a synthetic + /// response carrying `respond_status` instead of performing a real + /// network send. #[derive(Default)] struct RecordingHttpClient { requests: Mutex>, select_calls: Mutex, + respond_status: Mutex, } - #[async_trait::async_trait(?Send)] - impl PlatformHttpClient for RecordingHttpClient { - async fn send( - &self, - _request: PlatformHttpRequest, - ) -> Result> { - Err(Report::new(PlatformError::Unsupported)) + impl RecordingHttpClient { + /// Status [`PlatformHttpClient::send`] should reply with. Irrelevant + /// to auction-sink tests, which only exercise `send_async`. + fn respond_with(status: u16) -> Self { + Self { + respond_status: Mutex::new(status), + ..Self::default() + } } - async fn send_async( - &self, - request: PlatformHttpRequest, - ) -> Result> { + fn record(&self, request: PlatformHttpRequest) { let backend_name = request.backend_name; let (parts, body) = request.request.into_parts(); let headers = parts @@ -369,6 +539,35 @@ mod tests { .lock() .expect("should lock recorded requests") .push(recorded); + } + } + + #[async_trait::async_trait(?Send)] + impl PlatformHttpClient for RecordingHttpClient { + async fn send( + &self, + request: PlatformHttpRequest, + ) -> Result> { + self.record(request); + let status = *self + .respond_status + .lock() + .expect("should lock configured response status"); + let response = edgezero_core::http::response_builder() + .status( + edgezero_core::http::StatusCode::from_u16(status) + .expect("should build a valid test status code"), + ) + .body(edgezero_core::body::Body::empty()) + .expect("should build test response"); + Ok(PlatformResponse::new(response)) + } + + async fn send_async( + &self, + request: PlatformHttpRequest, + ) -> Result> { + self.record(request); Ok(PlatformPendingRequest::new(()).with_backend_name("tinybird-backend")) } @@ -698,6 +897,117 @@ mod tests { ); } + #[test] + fn access_emitter_posts_ndjson_and_validates_2xx() { + // `ts_secrets`/`tinybird_access_append_token` is seeded in + // fastly.toml's `[local_server.secret_stores]` fixture (value + // "test-tinybird-access-append-token"), so `emit_access_event` can + // load a real token through Viceroy without a secret-store test + // double — the same fixture backs the auction-token secret used + // above. + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let http_client = RecordingHttpClient::respond_with(202); + let row = r#"{"status":200}"#.to_owned(); + + futures::executor::block_on(emit_access_event(&http_client, &target, row.clone())) + .expect("should accept a 202 response"); + + let requests = http_client + .requests + .lock() + .expect("should lock recorded requests"); + assert_eq!(requests.len(), 1, "should send exactly one request"); + assert_eq!( + requests[0].uri, + "https://api.us-east.aws.tinybird.co/v0/events?name=access_logs_raw" + ); + assert_eq!(requests[0].method, Method::POST.to_string()); + assert_eq!( + header_value(&requests[0].headers, header::AUTHORIZATION.as_str()), + Some("Bearer test-tinybird-access-append-token") + ); + assert_eq!( + std::str::from_utf8(&requests[0].body).expect("should record utf8 body"), + row, + "should send the row verbatim as the request body" + ); + } + + #[test] + fn access_emitter_warns_and_drops_on_non_2xx() { + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let http_client = RecordingHttpClient::respond_with(422); + + let result = futures::executor::block_on(emit_access_event( + &http_client, + &target, + r#"{"status":422}"#.to_owned(), + )); + + let error = result.expect_err("a 422 response should be reported as an error"); + assert!( + error.to_string().contains("422"), + "error should name the failing status: {error}" + ); + assert_eq!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .len(), + 1, + "should not retry after a non-2xx response" + ); + } + + #[test] + fn sampled_out_requests_emit_nothing() { + // Mirrors main.rs's post-send gate exactly (`if sampled_in(rate, + // entropy) { emit_access_event(...) }`): `emit_access_event` is only + // reached when `sampled_in` returns `true`. With a `0.0` rate it + // never does, for any entropy, so the http client should never see + // a request. + let http_client = RecordingHttpClient::respond_with(202); + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let rate = 0.0; + let entropy = 123_456_789_u64; + + if sampled_in(rate, entropy) { + futures::executor::block_on(emit_access_event(&http_client, &target, "{}".to_owned())) + .expect("should send when sampled in"); + } + + assert_eq!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .len(), + 0, + "sampled-out requests must never reach emit_access_event" + ); + } + + #[test] + fn sampled_in_boundary_rates_are_unconditional() { + assert!( + sampled_in(1.0, 0), + "a 1.0 sample rate should always sample in" + ); + assert!( + sampled_in(1.0, u64::MAX), + "a 1.0 sample rate should always sample in regardless of entropy" + ); + assert!( + !sampled_in(0.0, 0), + "a 0.0 sample rate should never sample in" + ); + assert!( + !sampled_in(0.0, u64::MAX), + "a 0.0 sample rate should never sample in regardless of entropy" + ); + } + fn header_value<'a>(headers: &'a [(String, String)], name: &str) -> Option<&'a str> { headers .iter() From 72d57557827453d18409494e4301cc7fd6fe3734 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:10:07 -0700 Subject: [PATCH 011/104] Extend access_logs_raw with phase columns and a non-null sorting key --- .../datasources/access_logs_raw.datasource | 28 +++++++++++++++---- tinybird/fixtures/access_logs_raw.ndjson | 1 + 2 files changed, 24 insertions(+), 5 deletions(-) create mode 100644 tinybird/fixtures/access_logs_raw.ndjson diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 42f214e07..a484964db 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -1,19 +1,37 @@ DESCRIPTION > - Optional sampled Trusted Server access telemetry rows. Disabled by default in Fastly config. + Per-request phase-timing telemetry rows, sampled and emitted post-send by the edge service. SCHEMA > `event_ts` DateTime64(3), `method` LowCardinality(String), - `path` String, `status` UInt16, `time_elapsed_ms` UInt32, - `cache_state` LowCardinality(Nullable(String)), - `country` LowCardinality(String), `sample_rate` Float64, + `service_id` LowCardinality(String), + `publisher_domain` LowCardinality(String), + `env` LowCardinality(String), + `route_class` LowCardinality(String), + `route_template` String, + `body_mode` LowCardinality(String), + `auction_wait_placement` LowCardinality(String), + `appbuild_ms` Nullable(UInt32), + `filter_ms` Nullable(UInt32), + `geo_ms` Nullable(UInt32), + `kv_ms` Nullable(UInt32), + `origin_ms` Nullable(UInt32), + `template_cache_ms` Nullable(UInt32), + `auction_wait_ms` Nullable(UInt32), + `stream_ms` Nullable(UInt32), + `request_elapsed_ms` Nullable(UInt32), + `resp_bytes` Nullable(UInt64), + `template_cache_state` LowCardinality(String), + `country` LowCardinality(String), + `ts_version` LowCardinality(String), + `pop` LowCardinality(String), `event_date` Date DEFAULT toDate(event_ts) ENGINE "MergeTree" -ENGINE_SORTING_KEY "event_date, path, status, method" +ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status" TTL "event_date + INTERVAL 30 DAY" TOKEN ts_access_ingest APPEND diff --git a/tinybird/fixtures/access_logs_raw.ndjson b/tinybird/fixtures/access_logs_raw.ndjson new file mode 100644 index 000000000..3c82c5ca6 --- /dev/null +++ b/tinybird/fixtures/access_logs_raw.ndjson @@ -0,0 +1 @@ +{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} From c50be0323d258cae2b449dd4cb7ea98e232e115d Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:17:02 -0700 Subject: [PATCH 012/104] Widen time_elapsed_ms to nullable so dropped snapshots cannot quarantine rows --- .../specs/2026-08-24-request-phase-timing-design.md | 3 ++- tinybird/datasources/access_logs_raw.datasource | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 06f27c1c2..d9bbd4bdd 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -275,7 +275,8 @@ Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the -`mark_headers_ready()` snapshot), `sample_rate`, `event_date`, 30-day TTL. +`mark_headers_ready()` snapshot; nullable because a contended lock drop can lose the +snapshot), `sample_rate`, `event_date`, 30-day TTL. Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index a484964db..918f3d5fd 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -5,7 +5,7 @@ SCHEMA > `event_ts` DateTime64(3), `method` LowCardinality(String), `status` UInt16, - `time_elapsed_ms` UInt32, + `time_elapsed_ms` Nullable(UInt32), `sample_rate` Float64, `service_id` LowCardinality(String), `publisher_domain` LowCardinality(String), From 0aead79727d81cde132f03be94a022db9ef18cb8 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 12:24:22 -0700 Subject: [PATCH 013/104] Emit Server-Timing from the Axum terminal layer with adapter-specific semantics --- crates/trusted-server-adapter-axum/Cargo.toml | 6 +- crates/trusted-server-adapter-axum/src/app.rs | 32 ++- crates/trusted-server-adapter-axum/src/lib.rs | 3 + .../trusted-server-adapter-axum/src/main.rs | 71 ++++- .../trusted-server-adapter-axum/src/timing.rs | 242 ++++++++++++++++++ .../trusted-server-adapter-fastly/src/main.rs | 51 +--- .../trusted-server-core/src/request_timing.rs | 91 +++++++ 7 files changed, 438 insertions(+), 58 deletions(-) create mode 100644 crates/trusted-server-adapter-axum/src/timing.rs diff --git a/crates/trusted-server-adapter-axum/Cargo.toml b/crates/trusted-server-adapter-axum/Cargo.toml index 15b6ee59d..09e8c77d2 100644 --- a/crates/trusted-server-adapter-axum/Cargo.toml +++ b/crates/trusted-server-adapter-axum/Cargo.toml @@ -20,6 +20,7 @@ path = "src/main.rs" [dependencies] async-trait = { workspace = true } +axum = { workspace = true } edgezero-adapter-axum = { workspace = true, features = ["axum"] } edgezero-core = { workspace = true } error-stack = { workspace = true } @@ -27,12 +28,11 @@ futures = { workspace = true } log = { workspace = true } reqwest = { workspace = true } simple_logger = { workspace = true } -tokio = { workspace = true, features = ["rt-multi-thread", "macros", "sync", "time"] } +tokio = { workspace = true, features = ["rt-multi-thread", "macros", "net", "signal", "sync", "time"] } +tower = { workspace = true, features = ["util"] } trusted-server-core = { workspace = true } [dev-dependencies] -axum = { workspace = true } base64 = { workspace = true } temp-env = { workspace = true } tokio = { workspace = true, features = ["rt-multi-thread", "macros"] } -tower = { workspace = true, features = ["util"] } diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 4b71d07ce..4bdf27d01 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -565,15 +565,7 @@ impl Hooks for TrustedServerApp { } fn routes() -> RouterService { - let state = match build_state() { - Ok(s) => s, - Err(ref e) => { - log::error!("failed to build application state: {:?}", e); - return startup_error_router(e); - } - }; - - build_router(&state) + Self::routes_with_server_timing_flag().0 } } @@ -594,6 +586,28 @@ impl TrustedServerApp { let state = build_state_with_settings(settings)?; Ok(build_router(&state)) } + + /// Build the router alongside whether `Server-Timing` emission is + /// enabled, read from the same settings snapshot used to build the + /// router. + /// + /// The Axum dev server's terminal timing layer ([`crate::timing`]) needs + /// this flag once at startup: unlike the Fastly adapter, which rebuilds + /// `Settings` per request, the Axum dev server builds its application + /// state once and reuses the same [`RouterService`] for every request. + #[must_use] + pub fn routes_with_server_timing_flag() -> (RouterService, bool) { + let state = match build_state() { + Ok(s) => s, + Err(ref e) => { + log::error!("failed to build application state: {:?}", e); + return (startup_error_router(e), false); + } + }; + + let server_timing_enabled = state.settings.observability.server_timing_enabled; + (build_router(&state), server_timing_enabled) + } } fn build_router(state: &Arc) -> RouterService { diff --git a/crates/trusted-server-adapter-axum/src/lib.rs b/crates/trusted-server-adapter-axum/src/lib.rs index 2f15e566d..b1d4c3dd8 100644 --- a/crates/trusted-server-adapter-axum/src/lib.rs +++ b/crates/trusted-server-adapter-axum/src/lib.rs @@ -10,3 +10,6 @@ pub mod app; pub mod middleware; /// Platform-trait implementations backed by env vars and `reqwest`. pub mod platform; +/// Terminal timing layer wrapping the Axum dev server's tower `Service` +/// boundary with the request-phase `Server-Timing` freeze point. +pub mod timing; diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index 960982176..b8bc28ae2 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -1,6 +1,16 @@ -use edgezero_adapter_axum::dev_server::{AxumDevServer, AxumDevServerConfig}; -use edgezero_core::app::Hooks as _; +use std::net::SocketAddr; + +use axum::Router; +use edgezero_adapter_axum::dev_server::AxumDevServerConfig; +use edgezero_adapter_axum::service::EdgeZeroAxumService; +use edgezero_core::router::RouterService; +use tokio::net::TcpListener; +use tokio::runtime::Builder as RuntimeBuilder; +use tokio::signal; +use tower::Service as _; +use tower::service_fn; use trusted_server_adapter_axum::app::TrustedServerApp; +use trusted_server_adapter_axum::timing::TimingService; #[allow(clippy::print_stderr)] fn main() { @@ -20,13 +30,66 @@ fn main() { }; log::info!("Listening on http://{}", config.addr); - let router = TrustedServerApp::routes(); - if let Err(err) = AxumDevServer::with_config(router, config).run() { + let (router, server_timing_enabled) = TrustedServerApp::routes_with_server_timing_flag(); + if let Err(err) = run(router, server_timing_enabled, config) { log::error!("trusted-server-adapter-axum failed: {err}"); std::process::exit(1); } } +/// Runs the Axum dev server with the request-phase timing terminal layer +/// ([`trusted_server_adapter_axum::timing::TimingService`]) wrapped around +/// `EdgeZeroAxumService`, ahead of `axum::serve`. +/// +/// This does not use `edgezero_adapter_axum::dev_server::AxumDevServer::run`: +/// that helper only accepts a bare [`RouterService`] and builds its own +/// `EdgeZeroAxumService` and `axum::Router` internally, with no seam for an +/// outer service wrapper. Router-generated 404/405 responses bypass +/// `RouterBuilder::middleware` (see `trusted_server_adapter_axum::timing`), +/// so the freeze point has to wrap the tower `Service` boundary itself. +/// Driving `axum::serve` directly here mirrors that helper's own internal +/// bind/wrap/serve/shutdown sequence closely enough to keep behavior +/// identical for callers (`PORT` env var, ctrl-c graceful shutdown). +/// +/// # Errors +/// +/// Returns an error if the Tokio runtime fails to start, the listener fails +/// to bind, or the underlying serve loop errors. +fn run( + router: RouterService, + server_timing_enabled: bool, + config: AxumDevServerConfig, +) -> std::io::Result<()> { + let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; + runtime.block_on(serve(router, server_timing_enabled, config)) +} + +async fn serve( + router: RouterService, + server_timing_enabled: bool, + config: AxumDevServerConfig, +) -> std::io::Result<()> { + let listener = TcpListener::bind(config.addr).await?; + + let service = TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled); + let axum_router = Router::new().fallback_service(service_fn(move |req| { + let mut svc = service.clone(); + async move { svc.call(req).await } + })); + let make_service = axum_router.into_make_service_with_connect_info::(); + + let server = axum::serve(listener, make_service); + if config.enable_ctrl_c { + server + .with_graceful_shutdown(async { + let _ctrl_c = signal::ctrl_c().await; + }) + .await + } else { + server.await + } +} + /// Read a port number from the `PORT` environment variable. /// /// Returns `None` when the variable is unset. Exits non-zero if the value diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs new file mode 100644 index 000000000..832823c15 --- /dev/null +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -0,0 +1,242 @@ +//! Terminal timing layer for the Axum dev server. +//! +//! [`TimingService`](crate::timing::TimingService) wraps the tower `Service` +//! boundary the Axum dev server's router sits behind: it creates a +//! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) +//! collector per request, threads it through request extensions so +//! downstream core handlers can record into it, and on the way back stamps +//! `mark_headers_ready` and appends the `Server-Timing` header via +//! [`append_server_timing_if_private`](trusted_server_core::request_timing::append_server_timing_if_private). +//! +//! This wraps *outside* `RouterService` rather than registering as +//! `RouterBuilder::middleware`. A router-generated 404/405 short-circuits +//! `RouterInner::dispatch` before its middleware chain ever runs, so +//! middleware never sees those responses. By the time a response reaches +//! this layer -- after `RouterService::oneshot` inside +//! `EdgeZeroAxumService::call` has already converted any dispatch error into +//! a plain response -- every response is covered uniformly, router-generated +//! or not. +//! +//! `/health` is excluded by path match before a +//! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) +//! collector is even created: health checks never carry timing data on any +//! adapter. +//! +//! Unlike the Fastly adapter (state built per request, adding +//! `Phase::AppBuild` to the rendered header), the Axum dev server builds its +//! application state once at startup. There is no per-request app-build +//! interval to measure, so `ts-appbuild` never appears in the header here. + +use std::convert::Infallible; +use std::future::Future; +use std::pin::Pin; +use std::task::{Context, Poll}; + +use axum::body::Body as AxumBody; +use axum::http::{Request, Response}; +use tower::Service; +use trusted_server_core::request_timing::{RequestTimings, append_server_timing_if_private}; + +/// Path excluded from timing collection and `Server-Timing` emission: health +/// checks never carry timing data on any adapter. +const HEALTH_PATH: &str = "/health"; + +/// Wraps an inner Axum tower service with the request-phase timing freeze +/// point described in the module docs. +#[derive(Clone)] +pub struct TimingService { + inner: S, + server_timing_enabled: bool, +} + +impl TimingService { + /// Wraps `inner`, appending `Server-Timing` when `server_timing_enabled` + /// is set and the response is conclusively private. + #[must_use] + pub fn new(inner: S, server_timing_enabled: bool) -> Self { + Self { + inner, + server_timing_enabled, + } + } +} + +impl Service> for TimingService +where + S: Service, Response = Response, Error = Infallible> + + Clone + + Send + + 'static, + S::Future: Send + 'static, +{ + type Error = Infallible; + type Future = Pin> + Send>>; + type Response = Response; + + fn call(&mut self, mut req: Request) -> Self::Future { + let mut inner = self.inner.clone(); + + // Excluded before a collector is even created: `/health` never + // carries timing data, on any adapter. + if req.uri().path() == HEALTH_PATH { + return Box::pin(async move { inner.call(req).await }); + } + + let server_timing_enabled = self.server_timing_enabled; + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + Box::pin(async move { + let mut response = inner.call(req).await?; + append_server_timing_if_private(&mut response, &timings, server_timing_enabled); + Ok(response) + }) + } + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + self.inner.poll_ready(cx) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use axum::http::header::CACHE_CONTROL; + use axum::http::{HeaderValue, StatusCode}; + use edgezero_adapter_axum::service::EdgeZeroAxumService; + use edgezero_core::body::Body as EdgeBody; + use edgezero_core::context::RequestContext; + use edgezero_core::error::EdgeError; + use edgezero_core::http::response_builder; + use edgezero_core::router::RouterService; + use tower::{ServiceExt as _, service_fn}; + + /// Builds a private (`cache-control: private, no-store`) response for a + /// handler under test. + fn private_ok_response() -> Result { + Ok(response_builder() + .status(StatusCode::OK) + .header("cache-control", "private, no-store") + .body(EdgeBody::from("ok")) + .expect("should build a private response fixture")) + } + + /// Reads a response header as a UTF-8 string, or `None` if absent. + fn header(response: &Response, name: &str) -> Option { + response + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_emits_header_on_private_response() { + let router = RouterService::builder() + .get("/private", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + let server_timing = header(&response, "server-timing").expect("should emit header"); + assert!( + server_timing.contains("ts-total;dur="), + "should carry the collected total: {server_timing}" + ); + assert!( + !server_timing.contains("ts-appbuild"), + "the Axum dev server builds state once at startup, so there is no \ + per-request app-build interval to render: {server_timing}" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_404_carries_header_when_private() { + // An empty router has no routes at all, so any path dispatches + // through `RouterInner::dispatch`'s `NotFound` branch -- exactly the + // path that bypasses `RouterBuilder::middleware`. The router's own + // `EdgeError::into_response` does not attach `Cache-Control`, so a + // small wrapping service forces the response private here, standing + // in for whatever upstream layer would normally mark a genuinely + // private 404. This proves the freeze point still runs for a + // router-generated response without weakening + // `append_server_timing_if_private`'s real gating logic. + let empty_router = RouterService::builder().build(); + let inner = EdgeZeroAxumService::new(empty_router); + let force_private = service_fn(move |req: Request| { + let mut svc = inner.clone(); + async move { + let mut response = svc.call(req).await?; + response + .headers_mut() + .insert(CACHE_CONTROL, HeaderValue::from_static("private, no-store")); + Ok::<_, Infallible>(response) + } + }); + let mut service = TimingService::new(force_private, true); + + let request = Request::builder() + .uri("/does-not-exist") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should still be the router's own not-found response" + ); + let server_timing = header(&response, "server-timing") + .expect("a router-generated 404 must still carry the header when private"); + assert!( + server_timing.contains("ts-total;dur="), + "should carry the collected total: {server_timing}" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_health_is_excluded() { + let router = RouterService::builder() + .get("/health", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/health") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert!( + header(&response, "server-timing").is_none(), + "/health must never carry a server-timing header" + ); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 65e89cd1c..07c15b8b8 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -6,9 +6,7 @@ use edgezero_adapter_fastly::request::into_core_request; use edgezero_core::body::Body as EdgeBody; use edgezero_core::config_store::ConfigStoreHandle; use edgezero_core::error::EdgeError; -use edgezero_core::http::{ - HeaderName, HeaderValue, Request as HttpRequest, Response as HttpResponse, -}; +use edgezero_core::http::{Request as HttpRequest, Response as HttpResponse}; use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; @@ -17,9 +15,7 @@ use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::access_telemetry::{ AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, }; -use trusted_server_core::cache_policy::{ - EdgeCacheHeader, cache_control_headers_are_private_or_no_store, -}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_POP, ENV_FASTLY_SERVICE_ID, ENV_FASTLY_SERVICE_VERSION, }; @@ -37,7 +33,7 @@ use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::publisher::TemplateCacheResponseState; -use trusted_server_core::request_timing::{Phase, RequestTimings}; +use trusted_server_core::request_timing::{Phase, RequestTimings, append_server_timing_if_private}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -62,12 +58,6 @@ use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; const TRUSTED_SERVER_CONFIG_STORE: &str = "trusted_server_config"; -/// `Server-Timing` header name. Not present in the `http` crate's `header` -/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the -/// same `HeaderName::from_static` pattern used in -/// `trusted_server_core::constants`. -const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); - /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. /// /// # Errors @@ -544,40 +534,17 @@ pub(crate) enum DeliveryResult { Error, } -/// Stamps [`RequestTimings::mark_headers_ready`] and, when observability is -/// enabled and the response is conclusively private, appends the rendered -/// `Server-Timing` header. -/// -/// Always stamps `mark_headers_ready` regardless of whether the header is -/// rendered, so the collector's `ts-total` reflects the moment headers -/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` -/// value set upstream survives alongside the TS-owned set. A response is -/// never promoted to shared-cacheable just because the header would -/// otherwise be omitted: this only gates emission, it does not touch -/// `Cache-Control`. +/// Thin Fastly-adapter wrapper around +/// [`append_server_timing_if_private`], the freeze point shared with the +/// Axum adapter's terminal timing layer. See that function's doc for the +/// emission rules (always stamps `mark_headers_ready`; appends rather than +/// overwrites; never promotes a response to shared-cacheable). pub(crate) fn apply_server_timing_header( response: &mut HttpResponse, timings: &RequestTimings, server_timing_enabled: bool, ) { - timings.mark_headers_ready(); - - let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); - if !server_timing_enabled || !conclusively_private { - return; - } - - let Some(value) = timings.server_timing_value() else { - return; - }; - match HeaderValue::from_str(&value) { - Ok(header_value) => { - response - .headers_mut() - .append(HEADER_SERVER_TIMING, header_value); - } - Err(error) => log::warn!("skipping server-timing header: {error}"), - } + append_server_timing_if_private(response, timings, server_timing_enabled); } /// A [`Write`](std::io::Write) wrapper that tallies bytes successfully written diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index b0cb7b0dd..ccd393555 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -7,6 +7,16 @@ use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; +use http::{HeaderName, HeaderValue, Response}; + +use crate::cache_policy::cache_control_headers_are_private_or_no_store; + +/// `Server-Timing` header name. Not present in the `http` crate's `header` +/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the +/// same `HeaderName::from_static` pattern used in +/// `trusted_server_core::constants`. +const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); + /// Number of [`Phase`] variants; sizes the fixed-slot duration array in /// [`Inner`]. const PHASE_COUNT: usize = 8; @@ -253,6 +263,46 @@ impl Default for RequestTimings { } } +/// Stamps [`RequestTimings::mark_headers_ready`] and, when `enabled` and the +/// response is conclusively private, appends the rendered `Server-Timing` +/// header. +/// +/// Always stamps `mark_headers_ready` regardless of whether the header is +/// rendered, so the collector's `ts-total` reflects the moment headers +/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` +/// value set upstream survives alongside the TS-owned set. A response is +/// never promoted to shared-cacheable just because the header would +/// otherwise be omitted: this only gates emission, it does not touch +/// `Cache-Control`. +/// +/// Generic over the response body type so every adapter's terminal layer can +/// call the same emission logic regardless of which body type its HTTP stack +/// uses. +pub fn append_server_timing_if_private( + response: &mut Response, + timings: &RequestTimings, + enabled: bool, +) { + timings.mark_headers_ready(); + + let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); + if !enabled || !conclusively_private { + return; + } + + let Some(value) = timings.server_timing_value() else { + return; + }; + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response + .headers_mut() + .append(HEADER_SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } +} + /// The header-bearing phases (see [`Phase::header_name`]), in the enum /// declaration order [`RequestTimings::server_timing_value`] renders them in. const HEADER_PHASES: [Phase; 6] = [ @@ -434,4 +484,45 @@ mod tests { "should mask vendors" ); } + + #[test] + fn append_server_timing_emits_on_private_response_when_enabled() { + let mut response = Response::builder() + .header("cache-control", "private, no-store") + .body(()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + append_server_timing_if_private(&mut response, &timings, true); + + let header = response + .headers() + .get("server-timing") + .and_then(|value| value.to_str().ok()) + .expect("should emit a Server-Timing header"); + assert!( + header.starts_with("ts-total;dur="), + "should lead with the stored total: {header}" + ); + } + + #[test] + fn append_server_timing_marks_headers_ready_even_when_not_emitted() { + let mut response = Response::builder() + .header("cache-control", "max-age=60") + .body(()) + .expect("should build a shared-cacheable response fixture"); + let timings = RequestTimings::new(); + + append_server_timing_if_private(&mut response, &timings, true); + + assert!( + response.headers().get("server-timing").is_none(), + "should not emit on a shared-cacheable response" + ); + assert!( + timings.server_timing_value().is_some(), + "should still stamp mark_headers_ready so ts-total reflects the freeze point" + ); + } } From e9cf5b97a3677e734184b2604abc7df325e37ec5 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 15:50:12 -0700 Subject: [PATCH 014/104] Document the observability and access telemetry configuration surface --- docs/guide/configuration.md | 126 ++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a1f172429..675aaf40a 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -77,6 +77,8 @@ fail and the service will return its startup-error response. | `[request_signing]` | Ed25519 request signing | | `[auction]` | Auction orchestration | | `[integrations.*]` | Partner integrations (Prebid, Next.js, etc.) | +| `[observability]` | Server-Timing header emission | +| `[tinybird]` | Auction and access telemetry transport | ## Example: Production Setup @@ -1805,6 +1807,130 @@ Rollback to the legacy entry point is no longer controlled by runtime config keys. Use the normal deployment rollback path to restore a pre-cleanup service version if that is required. +## Observability and Access Telemetry Configuration + +Settings for the `Server-Timing` response header and the sampled +access-telemetry sink. Both are off by default and are independent switches: +enabling one does not enable the other. + +### `[observability]` + +| Field | Type | Required | Default | Description | +| ----------------------- | ------- | -------- | ------- | ------------------------------------------------------------------- | +| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | + +**Purpose**: Surfaces per-phase request timing (`ts-total` plus recorded +phases such as `ts-appbuild`, `ts-filter`, `ts-geo`, `ts-kv`, `ts-origin`, and +`ts-template-cache`) as a standard `Server-Timing` header, in milliseconds +with one decimal place. An unrecorded phase is omitted from the header +rather than rendered as zero. + +**Emission is conservative**: the header is appended only on responses that +are conclusively private, meaning `Cache-Control` contains `private` or +`no-store`. A response that is heuristically cacheable, carries a bare +`max-age`, or has no cache header at all never receives the header, because a +shared-cache object would otherwise replay one request's timings for its +entire stored lifetime. The long-lived, shared-cacheable `tsjs` asset route is +the concrete case this excludes. The header is appended, never inserted, so +an origin-supplied `Server-Timing` value and any entries the fronting +delivery layer adds are preserved alongside the TS entries. + +The Axum adapter applies the same private-response rule at its own terminal +point before serializing the response, and emits the header only; it does not +send access-telemetry rows. + +**Example**: + +```toml +[observability] +server_timing_enabled = true +``` + +**Environment Override**: + +```bash +TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED=true +``` + +::: tip Present-but-false by default +`server_timing_enabled` ships as `false` in the base operator config rather +than being left out, even though `false` is also its default. The +environment-variable overlay can only override a leaf that already exists in +the parsed TOML; it cannot create a missing one. Keeping the leaf present lets +`TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED` take effect without an +extra edit to add the table first. +::: + +### `[tinybird]` access telemetry keys + +`[tinybird]` configures a shared Events API transport (`enabled`, `api_host`, +`secret_store`, and per-sink dataset and token fields) used by two +independent emitters: auction telemetry (`auction_dataset`, +`auction_token_secret`) and access telemetry. The keys below cover the +access-telemetry sink and the shared enable flags. + +| Field | Type | Required | Default | Description | +| -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | +| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | +| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | +| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | +| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | + +**Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird +sinks separately so that turning on one does not silently turn on (or leave +off) the other; a settings test locks this decoupling in both directions. +Setting `access_enabled = true` with `access_sample_rate = 0.0` is rejected at +config load as an armed-but-silent configuration; use `access_enabled` itself +to turn the sink off, not the sample rate. Enabling `access_enabled` also +requires the shared transport fields (`enabled`, non-empty `api_host`, +`secret_store`, `access_dataset`, `access_token_secret`, and a positive +`max_body_bytes`) to already be set. + +**Example**: + +```toml +[tinybird] +enabled = true +api_host = "api.tinybird.example.com" +secret_store = "ts_secrets" +auction_enabled = true + +# Access-log telemetry, decoupled from auction emission. +access_enabled = true +access_dataset = "access_logs_raw" +access_token_secret = "tinybird_access_append_token" +access_sample_rate = 0.05 +max_body_bytes = 1048576 +``` + +**Environment Override**: + +```bash +TRUSTED_SERVER__TINYBIRD__ACCESS_ENABLED=true +TRUSTED_SERVER__TINYBIRD__ACCESS_SAMPLE_RATE=0.05 +TRUSTED_SERVER__TINYBIRD__AUCTION_ENABLED=true +``` + +A sampled request emits one access-telemetry row to `access_dataset` after +the response has already been delivered to the client, so ingest never delays +the response the reader sees. + +### Deploy and rollback ordering + +::: warning `Settings` rejects unknown fields; order matters +Both `[observability].server_timing_enabled` and the new `[tinybird]` access +keys are new fields on a config schema that uses `deny_unknown_fields`, so an +older binary fails to load a config that carries them. + +**Deploying**: upgrade the binary first, then push a config containing the +new fields second. Never push a config with these fields while a +pre-observability binary can still receive it. + +**Rolling back**: reverse the order. Remove the `[observability]` table and +any new `[tinybird]` access keys from the config and push that first, then +roll back the binary second. +::: + ## Validation ### Automatic Validation From 7942c74aa6c2a298fed80dfade8ee3059c34cc7c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 18:45:28 -0700 Subject: [PATCH 015/104] Normalize telemetry method, guard zero sample rate, and mirror geo write-back in middleware Three final-review fixes for access telemetry correctness: - Normalize the HTTP method to an allowlist (GET/HEAD/POST/PUT/DELETE/ PATCH/OPTIONS, else "other") inside access_event_row, so a client- controlled extension-method token can never inflate the LowCardinality method column, regardless of which adapter builds the row. - Guard emit_access_telemetry_after_send against snapshots carrying a degraded sample_rate of 0.0 (captured on the app-state-build-failure fallback path), which could otherwise be sampled in by freshly reloaded settings and corrupt the sum(1.0/sample_rate) volume estimator. - Mirror the geo lookup write-back from apply_entry_point_finalize_headers into FinalizeResponseMiddleware::handle, so a middleware-finalized response that resolved geo via fallback carries the resolved GeoLookupState for the access-telemetry snapshot instead of showing country "unknown". --- .../trusted-server-adapter-fastly/src/main.rs | 88 ++++++++++++++++++- .../src/middleware.rs | 73 +++++++++++++++ .../src/access_telemetry.rs | 79 ++++++++++++++++- 3 files changed, 235 insertions(+), 5 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 07c15b8b8..ec2370272 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -452,9 +452,12 @@ fn run_edgezero_pull_sync_after_send( /// either of those per-route types, so every response class can emit. /// /// Sampled-out requests return silently — that is the expected, high-volume -/// case and not worth a log line. Every other drop (row build, token load, -/// send, or non-2xx status — all folded into `emit_access_event`'s `Result`) -/// logs exactly one warning naming the reason. +/// case and not worth a log line. A snapshot carrying a degraded +/// `sample_rate` (see [`should_sample_access_row`]) also returns silently, +/// since it only occurs on an already-degraded path. Every other drop (row +/// build, token load, send, or non-2xx status — all folded into +/// `emit_access_event`'s `Result`) logs exactly one warning naming the +/// reason. fn emit_access_telemetry_after_send( settings: &Settings, outcome: &DeliveryOutcome, @@ -477,7 +480,11 @@ fn emit_access_telemetry_after_send( let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); let entropy = entropy_nanos ^ outcome.bytes; - if !tinybird::sampled_in(settings.tinybird.access_sample_rate, entropy) { + if !should_sample_access_row( + outcome.snapshot.sample_rate, + settings.tinybird.access_sample_rate, + entropy, + ) { return; } @@ -493,6 +500,39 @@ fn emit_access_telemetry_after_send( } } +/// Whether one response's access-telemetry row should be emitted, combining +/// the degraded-snapshot guard with the sampling roll. +/// +/// `snapshot_sample_rate` is the rate recorded on the [`AccessTelemetrySnapshot`] +/// itself (the value serialized into the row's `sample_rate` column, which +/// the documented volume estimator divides by as `1.0 / sample_rate`). +/// `settings_sample_rate` is the rate used for the sampling decision at +/// call time. The two can diverge: when `app_state` fails to build, +/// [`edgezero_main`] captures a snapshot with `sample_rate` defaulted to +/// `0.0` before any settings ever load, but the two settings-reload +/// emission sites still gate and sample using the *reloaded* settings' +/// (nonzero) rate. Without this guard, such a row could be sampled in and +/// emitted while carrying `sample_rate: 0.0`, corrupting the volume +/// estimator. Dropping these rows is acceptable: they only occur on an +/// already-degraded path, consistent with this pipeline's fail-quiet +/// telemetry policy. Split out of [`emit_access_telemetry_after_send`] so +/// the guard is unit-testable without a network seam. +/// +/// Callers must already have applied the coarse +/// `tinybird.enabled`/`access_enabled` gate. +#[must_use] +fn should_sample_access_row( + snapshot_sample_rate: f64, + settings_sample_rate: f64, + entropy: u64, +) -> bool { + if snapshot_sample_rate <= 0.0 { + return false; + } + + tinybird::sampled_in(settings_sample_rate, entropy) +} + /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -1818,4 +1858,44 @@ mod tests { "pull-sync must dispatch before telemetry emits" ); } + + #[test] + fn should_sample_access_row_rejects_a_degraded_zero_sample_rate() { + // A snapshot captured on the app-state-build-failure fallback path + // carries `sample_rate: 0.0`. Even when the reloaded settings' rate + // would sample every request in (1.0), the row must not emit — + // otherwise it would claim `sample_rate: 0.0` and corrupt the + // `sum(1.0 / sample_rate)` volume estimator. + assert!( + !should_sample_access_row(0.0, 1.0, 0), + "a snapshot with sample_rate 0.0 must never emit, regardless of entropy or settings' rate" + ); + assert!( + !should_sample_access_row(0.0, 1.0, u64::MAX), + "the degraded-rate guard must not depend on the entropy value" + ); + } + + #[test] + fn should_sample_access_row_rejects_a_negative_sample_rate() { + assert!( + !should_sample_access_row(-1.0, 1.0, 0), + "a negative snapshot sample_rate is equally degraded and must not emit" + ); + } + + #[test] + fn should_sample_access_row_defers_to_the_settings_sampling_roll_when_not_degraded() { + // With a healthy (nonzero) snapshot sample_rate, the outcome should + // match `tinybird::sampled_in` exactly, since that is the only + // remaining decision. + assert!( + should_sample_access_row(0.25, 1.0, 0), + "a settings rate of 1.0 always samples in, independent of entropy" + ); + assert!( + !should_sample_access_row(0.25, 0.0, 0), + "a settings rate of 0.0 always samples out, independent of the snapshot's rate" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 17ecc13a4..ae1efe7c3 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -97,6 +97,18 @@ impl Middleware for FinalizeResponseMiddleware { }) }); + // Write the resolved outcome back so a downstream access-telemetry + // snapshot (built from response extensions after finalize) sees + // what was actually looked up here rather than the stale carried-in + // state — mirrors the entry-point finalize site in `main.rs` + // (`apply_entry_point_finalize_headers`), which writes back for the + // same reason. + let resolved_state = match &geo_info { + Some(geo) => GeoLookupState::Resolved(geo.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); + apply_finalize_headers(&self.settings, geo_info.as_ref(), &mut response); response .headers_mut() @@ -594,6 +606,67 @@ mod tests { ); } + #[test] + fn finalize_handle_writes_back_resolved_geo_state_after_fallback_lookup() { + // The request phase never attempted a geo lookup (no GeoLookupState + // extension on the handler's response), so the middleware resolves + // one via the fallback closure. That resolved outcome must be + // written back into response extensions -- mirroring + // apply_entry_point_finalize_headers in main.rs -- so a downstream + // access-telemetry snapshot sees the freshly resolved country + // instead of a stale/missing GeoLookupState. + let settings = settings_with_response_headers(vec![]); + let middleware = FinalizeResponseMiddleware::new( + Arc::new(settings), + Arc::new(FixedGeo(Some(sample_geo_info()))), + ); + let handler = + Arc::new( + |_ctx: RequestContext| async move { Ok::(empty_response()) }, + ); + + let response = block_on(middleware.handle(empty_ctx(), Next::new(&[], &*handler))) + .expect("should succeed"); + + match response.extensions().get::() { + Some(GeoLookupState::Resolved(info)) => { + assert_eq!( + info.country, "US", + "should carry the fallback-resolved geo info" + ); + } + other => { + panic!("expected GeoLookupState::Resolved after a fallback lookup, got {other:?}") + } + } + } + + #[test] + fn finalize_handle_writes_back_attempted_geo_state_when_fallback_finds_nothing() { + // The fallback lookup ran but resolved no geo info. The middleware + // must still record that the lookup was attempted, so a later + // consumer of the extension does not mistake this for + // GeoLookupState::NotAttempted and retry the lookup. + let settings = settings_with_response_headers(vec![]); + let middleware = + FinalizeResponseMiddleware::new(Arc::new(settings), Arc::new(FixedGeo(None))); + let handler = + Arc::new( + |_ctx: RequestContext| async move { Ok::(empty_response()) }, + ); + + let response = block_on(middleware.handle(empty_ctx(), Next::new(&[], &*handler))) + .expect("should succeed"); + + assert!( + matches!( + response.extensions().get::(), + Some(GeoLookupState::Attempted) + ), + "should write back Attempted when the fallback lookup finds no geo info" + ); + } + #[test] fn finalize_handle_marks_response_as_finalized() { let settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 9daf77abc..1642a1334 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -16,6 +16,42 @@ use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; /// by [`publisher_route_template`]. const MAX_SEGMENT_LEN: usize = 32; +/// Normalizes an HTTP method token into the bounded set of values stored in +/// the `method` `LowCardinality` column. +/// +/// HTTP permits arbitrary extension-method tokens (`PROPFIND`, `MKCOL`, or +/// any client-supplied garbage), and the token on an inbound request is +/// entirely client controlled. Capturing one verbatim into a 30-day +/// `LowCardinality(String)` column would let a single caller inflate that +/// column's cardinality without bound and would violate this dataset's +/// bounded-dimension privacy rule (see the module doc). Every standard +/// method maps to its uppercase form; anything else maps to `"other"`. Runs +/// inside [`access_event_row`] rather than at each capture site, so every +/// row-building path is covered regardless of how `method` was populated. +/// +/// # Examples +/// +/// ``` +/// use trusted_server_core::access_telemetry::normalize_method; +/// +/// assert_eq!(normalize_method("get"), "GET"); +/// assert_eq!(normalize_method("PROPFIND"), "other"); +/// assert_eq!(normalize_method(""), + "other", + "an unbounded client-controlled token must not reach the row verbatim" + ); + } + + #[test] + fn row_normalizes_method_even_when_snapshot_carries_a_raw_token() { + // The normalizer runs inside `access_event_row` so every row-building + // path is covered, regardless of what the snapshot's `method` field + // holds — a caller-controlled extension method must never leak into + // the row unnormalized. + let mut snapshot = unknown_snapshot(RouteClass::Other, "/other/*"); + snapshot.method = "PROPFIND".to_owned(); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + assert_eq!(parsed["method"], "other"); + } } From d8fcb5e933e0cc8f0df15e98613dfce43636bd0f Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 19:51:06 -0700 Subject: [PATCH 016/104] Add a local dev config envelope generator example --- .../examples/local_dev_config.rs | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 crates/trusted-server-core/examples/local_dev_config.rs diff --git a/crates/trusted-server-core/examples/local_dev_config.rs b/crates/trusted-server-core/examples/local_dev_config.rs new file mode 100644 index 000000000..19826231d --- /dev/null +++ b/crates/trusted-server-core/examples/local_dev_config.rs @@ -0,0 +1,126 @@ +//! Generate a ready-to-use local dev config envelope for the Axum adapter. +//! +//! Reads `trusted-server.example.toml`, replaces the placeholder secrets with +//! random values, flips the flags a local smoke test needs, validates the +//! result through [`trusted_server_core::settings::Settings::from_toml`], and +//! prints the blob envelope JSON that +//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG` expects. +//! +//! The random values are time-and-pid seeded, not cryptographic. This tool +//! exists for throwaway local test instances only; never use its output for a +//! deployed service. +//! +//! Usage: +//! +//! ```text +//! cargo run -p trusted-server-core --example local_dev_config \ +//! --target -- [origin-url] [--realistic] +//! ``` +//! +//! `origin-url` defaults to `https://www.example.com`. By default every +//! response is forced `Cache-Control: private, no-store` so the Server-Timing +//! header is visible on all routes; pass `--realistic` to keep the origin's +//! own cache policy instead. + +use std::time::{SystemTime, UNIX_EPOCH}; + +/// Deliberately non-cryptographic generator for local placeholder secrets. +struct WeakRandom(u64); + +impl WeakRandom { + fn from_environment() -> Self { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .subsec_nanos() as u64; + let secs = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .as_secs(); + let pid = std::process::id() as u64; + Self(nanos ^ (secs << 20) ^ (pid << 40) ^ 0x9e37_79b9_7f4a_7c15) + } + + fn next(&mut self) -> u64 { + let mut x = self.0; + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + self.0 = x; + x + } + + fn hex(&mut self, chars: usize) -> String { + let mut out = String::with_capacity(chars); + while out.len() < chars { + out.push_str(&format!("{:016x}", self.next())); + } + out.truncate(chars); + out + } +} + +#[allow(clippy::print_stdout, clippy::print_stderr)] +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + let realistic = args.iter().any(|a| a == "--realistic"); + let origin = args + .iter() + .find(|a| !a.starts_with("--")) + .cloned() + .unwrap_or_else(|| "https://www.example.com".to_string()); + + let template = std::fs::read_to_string("trusted-server.example.toml") + .expect("should read trusted-server.example.toml from the repo root"); + + let mut random = WeakRandom::from_environment(); + let mut config = template + .replace( + "password = \"replace-with-admin-password-32-bytes\"", + &format!("password = \"{}\"", random.hex(48)), + ) + .replace( + "proxy_secret = \"change-me-proxy-secret\"", + &format!("proxy_secret = \"{}\"", random.hex(48)), + ) + .replace( + "passphrase = \"trusted-server-placeholder-secret\"", + &format!("passphrase = \"{}\"", random.hex(48)), + ) + .replace( + "server_timing_enabled = false", + "server_timing_enabled = true", + ); + + let origin_line = config + .lines() + .find(|line| line.starts_with("origin_url = ")) + .expect("should find the origin_url line in the template") + .to_string(); + config = config.replace(&origin_line, &format!("origin_url = \"{origin}\"")); + + if !realistic { + config = config.replace( + "# [response_headers]", + "[response_headers]\n\"Cache-Control\" = \"private, no-store\"", + ); + } + + let settings = trusted_server_core::settings::Settings::from_toml(&config) + .expect("should validate the generated local config"); + let data = serde_json::to_value(&settings).expect("should serialize settings"); + let generated_at = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .as_secs() + .to_string(); + let envelope = edgezero_core::blob_envelope::BlobEnvelope::new(data, generated_at); + println!( + "{}", + serde_json::to_string(&envelope).expect("should serialize the envelope") + ); + eprintln!( + "local dev envelope generated: origin={origin} force_private={}", + !realistic + ); +} From 7cf7d86712c996e80ce1cd4ae436df7974a69732 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 20:41:15 -0700 Subject: [PATCH 017/104] Add JSONPaths and expression sorting key to the access datasource --- .../2026-08-24-request-phase-timing-design.md | 7 ++- .../datasources/access_logs_raw.datasource | 60 ++++++++++--------- 2 files changed, 36 insertions(+), 31 deletions(-) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index d9bbd4bdd..d8791a799 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -340,8 +340,11 @@ guest-visible cache behavior is already carried by `template_cache_state` and `origin_ms`. Rows exist only for guest-handled requests; fronting-cache hits are invisible by construction and the dashboard documentation says so. -Sorting key: `(event_date, service_id, publisher_domain, env, route_class, pop, -status)`. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query +Sorting key: `(toDate(event_ts), service_id, publisher_domain, env, route_class, +pop, status)`. Every column carries a `json:$.` path (the Events API rejects +NDJSON into a datasource without JSONPaths, discovered live); `event_date` was +dropped in favor of the sorting-key expression because a DEFAULT column cannot +carry a JSONPath the producer never sends. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query also carries an `event_date` predicate so the primary index prunes; rollout validates the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 918f3d5fd..062b884ba 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -2,36 +2,38 @@ DESCRIPTION > Per-request phase-timing telemetry rows, sampled and emitted post-send by the edge service. SCHEMA > - `event_ts` DateTime64(3), - `method` LowCardinality(String), - `status` UInt16, - `time_elapsed_ms` Nullable(UInt32), - `sample_rate` Float64, - `service_id` LowCardinality(String), - `publisher_domain` LowCardinality(String), - `env` LowCardinality(String), - `route_class` LowCardinality(String), - `route_template` String, - `body_mode` LowCardinality(String), - `auction_wait_placement` LowCardinality(String), - `appbuild_ms` Nullable(UInt32), - `filter_ms` Nullable(UInt32), - `geo_ms` Nullable(UInt32), - `kv_ms` Nullable(UInt32), - `origin_ms` Nullable(UInt32), - `template_cache_ms` Nullable(UInt32), - `auction_wait_ms` Nullable(UInt32), - `stream_ms` Nullable(UInt32), - `request_elapsed_ms` Nullable(UInt32), - `resp_bytes` Nullable(UInt64), - `template_cache_state` LowCardinality(String), - `country` LowCardinality(String), - `ts_version` LowCardinality(String), - `pop` LowCardinality(String), - `event_date` Date DEFAULT toDate(event_ts) + `event_ts` DateTime64(3) `json:$.event_ts`, + `method` LowCardinality(String) `json:$.method`, + `status` UInt16 `json:$.status`, + `time_elapsed_ms` Nullable(UInt32) `json:$.time_elapsed_ms`, + `sample_rate` Float64 `json:$.sample_rate`, + `service_id` LowCardinality(String) `json:$.service_id`, + `publisher_domain` LowCardinality(String) `json:$.publisher_domain`, + `env` LowCardinality(String) `json:$.env`, + `route_class` LowCardinality(String) `json:$.route_class`, + `route_template` String `json:$.route_template`, + `body_mode` LowCardinality(String) `json:$.body_mode`, + `auction_wait_placement` LowCardinality(String) `json:$.auction_wait_placement`, + `appbuild_ms` Nullable(UInt32) `json:$.appbuild_ms`, + `filter_ms` Nullable(UInt32) `json:$.filter_ms`, + `geo_ms` Nullable(UInt32) `json:$.geo_ms`, + `kv_ms` Nullable(UInt32) `json:$.kv_ms`, + `origin_ms` Nullable(UInt32) `json:$.origin_ms`, + `template_cache_ms` Nullable(UInt32) `json:$.template_cache_ms`, + `auction_wait_ms` Nullable(UInt32) `json:$.auction_wait_ms`, + `stream_ms` Nullable(UInt32) `json:$.stream_ms`, + `request_elapsed_ms` Nullable(UInt32) `json:$.request_elapsed_ms`, + `resp_bytes` Nullable(UInt64) `json:$.resp_bytes`, + `template_cache_state` LowCardinality(String) `json:$.template_cache_state`, + `country` LowCardinality(String) `json:$.country`, + `ts_version` LowCardinality(String) `json:$.ts_version`, + `pop` LowCardinality(String) `json:$.pop` ENGINE "MergeTree" -ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status" -TTL "event_date + INTERVAL 30 DAY" +ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status" +TTL "toDate(event_ts) + INTERVAL 30 DAY" + +FORWARD_QUERY > + SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop TOKEN ts_access_ingest APPEND From 600746f957f2a92c49fca93cea4d7d1a9c10de7c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:16 -0700 Subject: [PATCH 018/104] Use web_time Instant on request timing paths std::time::Instant::now() panics on wasm32-unknown-unknown, so every publisher request on the Cloudflare adapter trapped when the timing collector was constructed, and the two auction-wait sites would trap once an auction dispatched. web_time re-exports std's Instant on every other target, so Fastly, Axum, and Spin behavior is unchanged. The publisher.rs sites are qualified locally because that module's std Instant import still serves the pre-existing template-cache sites, which are out of scope here. --- crates/trusted-server-core/src/publisher.rs | 10 ++++++++-- crates/trusted-server-core/src/request_timing.rs | 6 +++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 6a30611c8..53d09bfc6 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3937,7 +3937,10 @@ async fn collect_non_html_auction( .as_ref() .and_then(|_| diagnostics_auction_id(settings)); let placeholder = mediator_placeholder_request(); - let wait_started = Instant::now(); + // Qualified: `std::time::Instant::now()` panics on Cloudflare's + // `wasm32-unknown-unknown` target; this module's `Instant` import stays + // std for the pre-existing template-cache sites. + let wait_started = web_time::Instant::now(); let result = orchestrator .collect_dispatched_auction( dispatched, @@ -3999,7 +4002,10 @@ async fn collect_stream_auction( log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); - let wait_started = Instant::now(); + // Qualified: `std::time::Instant::now()` panics on Cloudflare's + // `wasm32-unknown-unknown` target; this module's `Instant` import stays + // std for the pre-existing template-cache sites. + let wait_started = web_time::Instant::now(); let result = orchestrator .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index ccd393555..6ac9a9692 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -5,9 +5,13 @@ //! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. use std::sync::{Arc, Mutex}; -use std::time::{Duration, Instant}; +use std::time::Duration; use http::{HeaderName, HeaderValue, Response}; +// `std::time::Instant::now()` panics on `wasm32-unknown-unknown` (the +// Cloudflare adapter's target); `web_time` re-exports std's `Instant` on +// every other target. +use web_time::Instant; use crate::cache_policy::cache_control_headers_are_private_or_no_store; From 3d7e697cad8d1fb6b3206249d66a54e117d40527 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:33 -0700 Subject: [PATCH 019/104] Reject opaque identifier segments in publisher route templates The character allowlist alone does not bound identity: [a-z0-9_-] is exactly the alphabet UUIDs, hex ids, reset tokens, and article slugs are built from, and truncating to 32 characters still leaves a globally unique prefix. A first segment now rejects whole to /other/* when it exceeds 32 characters or carries more than 7 ASCII digits, alongside the existing charset rejection. Year archives and hyphenated section names still pass. Extends the adversarial tests to the publisher-fallback path with UUID, hex-id, token, and slug shapes, and fixes the stale event_date reference in the row-builder doc. --- .../src/access_telemetry.rs | 114 +++++++++++++++--- .../2026-08-24-request-phase-timing-design.md | 11 +- 2 files changed, 105 insertions(+), 20 deletions(-) diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 1642a1334..28e391f7e 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -12,10 +12,20 @@ use serde_json::json; use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; -/// Maximum number of characters kept from a publisher path's first segment -/// by [`publisher_route_template`]. +/// Maximum length of a publisher path's first segment before +/// [`publisher_route_template`] rejects it to `/other/*`. Longer segments +/// are opaque-identifier or slug shaped (a UUID is 36 characters), and a +/// truncated prefix of either would still be identifying, so the segment +/// is rejected whole rather than truncated. const MAX_SEGMENT_LEN: usize = 32; +/// Maximum number of ASCII digits in a publisher path's first segment +/// before [`publisher_route_template`] rejects it to `/other/*`. Hex ids, +/// base36 ids, and reset tokens are digit-heavy; real section names carry +/// at most a year (`2026`) or a small version number, so a segment with +/// more digits than this is treated as an identifier, not a name. +const MAX_SEGMENT_DIGITS: usize = 7; + /// Normalizes an HTTP method token into the bounded set of values stored in /// the `method` `LowCardinality` column. /// @@ -116,18 +126,27 @@ pub struct RouteMetadata { /// content-free route template. /// /// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, truncated to [`MAX_SEGMENT_LEN`] characters, with a -/// trailing `/*` appended when the path has additional segments beyond the -/// first. The root path `/` maps to itself. An empty first segment, or one -/// containing any character outside the allowlist (after lowercasing), -/// maps to `/other/*` — the segment is rejected outright rather than -/// filtered, so no fragment of a disallowed segment (an email address, a -/// search phrase) ever reaches the row. +/// `[a-z0-9_-]`, with a trailing `/*` appended when the path has +/// additional segments beyond the first. The root path `/` maps to itself. +/// A first segment is rejected to `/other/*` — outright, never filtered or +/// truncated, so no fragment of it ever reaches the row — when it: +/// +/// - is empty, or contains any character outside the allowlist after +/// lowercasing (an email address, a search phrase); +/// - is longer than [`MAX_SEGMENT_LEN`] characters (UUIDs, long hex +/// tokens, and full article slugs all exceed it — a truncated prefix of +/// any of these would still be identifying); or +/// - contains more than [`MAX_SEGMENT_DIGITS`] ASCII digits. Opaque +/// identifiers (hex ids, base36 ids, reset tokens) are digit-heavy; +/// publisher section names are words, at most a year or a version +/// number. /// /// This is deliberately coarser than the auction-telemetry path /// normalizer, which redacts long tokens but preserves short identifiers /// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. +/// this broad. Short all-alpha slugs on single-segment paths are +/// indistinguishable from section names and still pass; the bound here is +/// shape-based, not semantic. /// /// # Examples /// @@ -137,6 +156,10 @@ pub struct RouteMetadata { /// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); /// assert_eq!(publisher_route_template("/"), "/"); /// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// assert_eq!( +/// publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), +/// "/other/*" +/// ); /// ``` #[must_use] pub fn publisher_route_template(path: &str) -> String { @@ -156,16 +179,17 @@ pub fn publisher_route_template(path: &str) -> String { && lowered .chars() .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); + let within_length = lowered.chars().count() <= MAX_SEGMENT_LEN; + let digit_count = lowered.chars().filter(char::is_ascii_digit).count(); - if !is_allowlisted { + if !is_allowlisted || !within_length || digit_count > MAX_SEGMENT_DIGITS { return "/other/*".to_owned(); } - let truncated: String = lowered.chars().take(MAX_SEGMENT_LEN).collect(); if has_more_depth { - format!("/{truncated}/*") + format!("/{lowered}/*") } else { - format!("/{truncated}") + format!("/{lowered}") } } @@ -220,8 +244,9 @@ pub struct AccessTelemetrySnapshot { /// `timings` and serialize as JSON `null` for phases that were never /// recorded; every dimension column comes from `snapshot` and is a /// non-nullable string (callers are expected to substitute an `unknown` -/// sentinel rather than leave a dimension empty). `event_date` is omitted: -/// the datasource derives it from `event_ts` by default. +/// sentinel rather than leave a dimension empty). There is no `event_date` +/// column: the datasource's sorting key derives the date via +/// `toDate(event_ts)`. #[must_use] pub fn access_event_row( snapshot: &AccessTelemetrySnapshot, @@ -338,12 +363,65 @@ mod tests { ); assert_eq!( publisher_route_template(&format!("/{}", "a".repeat(500))), - format!("/{}", "a".repeat(32)), - "should bound segment length" + "/other/*", + "should reject overlong segments whole rather than truncate" ); assert_eq!(publisher_route_template("/search terms here"), "/other/*"); } + #[test] + fn publisher_route_template_rejects_opaque_identifier_segments() { + // Every row here passes the character allowlist (`[a-z0-9_-]` is + // exactly what UUIDs, hex ids, and tokens are built from) and must + // be caught by the length and digit-count bounds instead. A + // truncated prefix of any of these would still be identifying, so + // rejection must be whole-segment. + assert_eq!( + publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), + "/other/*", + "should reject a UUID (36 chars) by length" + ); + assert_eq!( + publisher_route_template("/550e8400-e29b-41d4-a716-446655440000/profile"), + "/other/*", + "should reject a UUID first segment on deeper paths too" + ); + assert_eq!( + publisher_route_template("/8f3a9c2b1d4e5f6a7b8c9d0e1f2a3b4c"), + "/other/*", + "should reject a 32-char hex id by digit count" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a1".repeat(32))), + "/other/*", + "should reject a 64-char token by length" + ); + assert_eq!( + publisher_route_template("/reset-password-token-9f2b1c7d4e8a"), + "/other/*", + "should reject a reset token by length" + ); + assert_eq!( + publisher_route_template("/how-to-treat-my-recent-hiv-diagnosis"), + "/other/*", + "should reject a full article slug by length" + ); + } + + #[test] + fn publisher_route_template_keeps_digit_light_section_names() { + assert_eq!( + publisher_route_template("/2026/08/some-article"), + "/2026/*", + "a year archive segment should pass the digit bound" + ); + assert_eq!( + publisher_route_template("/wp-content/themes/site/app.css"), + "/wp-content/*", + "a hyphenated section name should pass" + ); + } + #[test] fn publisher_route_template_rejects_empty_first_segment() { assert_eq!( diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index d8791a799..214d146a1 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -289,9 +289,16 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced restricted to a bounded allowlisted charset, plus `/*` when deeper (for example `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. +- Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` + when it fails the charset allowlist, exceeds 32 characters, or carries more than 7 + ASCII digits. The character allowlist alone does not bound identity (`[a-z0-9_-]` + is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated + prefix of any of those is still identifying, so the length and digit bounds reject + the segment outright. - Tests are adversarial, not just the happy path: a literal EC identifier on the admin - route, an email address in a path segment, search-term-shaped segments, and - overlong segments must all normalize to bounded, content-free templates. + route, an email address in a path segment, search-term-shaped segments, overlong + segments, UUIDs, hex ids, reset tokens, and full article slugs must all normalize + to bounded, content-free templates. Added columns (all dimension columns non-nullable with an `unknown` sentinel, because ClickHouse sorting keys cannot contain nullable columns): From 38043d7464362d44519153a09fe850bacc256b58 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:41 -0700 Subject: [PATCH 020/104] Address access telemetry review feedback - Gate building the access snapshot on tinybird.enabled and access_enabled, threaded through SendContext: a disabled deployment (the default) no longer pays env reads and String allocations on the pre-send path. DeliveryOutcome.snapshot becomes Option and the emitter treats None as nothing to send. - Classify asset-fallback responses as route_class asset with the operator-configured route prefix as the template, instead of landing in the other/unknown bucket alongside 404s. - Pin Phase::index() to PHASE_COUNT with a uniqueness-and-bounds test so a future variant fails the suite instead of panicking at runtime. - Drop the tautological sampled-out emission test; the 0.0-rate behavior is covered by sampled_in_boundary_rates_are_unconditional. - Clarify that the local dev config env var name genuinely triples trusted_server_config (prefix, store, key) rather than reading as a find/replace mistake. --- .../trusted-server-adapter-fastly/src/app.rs | 11 +++- .../trusted-server-adapter-fastly/src/main.rs | 51 +++++++++++++++---- .../src/tinybird.rs | 28 ---------- .../examples/local_dev_config.rs | 7 ++- .../src/access_telemetry.rs | 4 ++ .../trusted-server-core/src/request_timing.rs | 34 +++++++++++++ 6 files changed, 93 insertions(+), 42 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 0a0cc8c5a..3a4ceda8f 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -902,7 +902,14 @@ async fn dispatch_fallback( .then(|| state.settings.asset_route_for_path(&path)) .flatten(); if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback( + // The template is the operator-configured route prefix, so it + // is bounded and content-free by construction (unlike request + // paths, which need `publisher_route_template`). + let asset_metadata = RouteMetadata { + route_class: RouteClass::Asset, + route_template: format!("{}/*", asset_route.prefix.trim_end_matches('/')), + }; + let mut response = dispatch_asset_fallback( state, services, req, @@ -911,6 +918,8 @@ async fn dispatch_fallback( ec.geo_lookup_state(), ) .await; + response.extensions_mut().insert(asset_metadata); + return response; } route_metadata = Some(RouteMetadata { diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ec2370272..934f98402 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -150,6 +150,9 @@ fn edgezero_main(mut req: FastlyRequest) { let access_sample_rate = settings_snapshot .as_deref() .map_or(0.0, |settings| settings.tinybird.access_sample_rate); + let access_telemetry_enabled = settings_snapshot + .as_deref() + .is_some_and(|settings| settings.tinybird.enabled && settings.tinybird.access_enabled); let publisher_domain = settings_snapshot.as_deref().map_or_else( || "unknown".to_owned(), |settings| settings.publisher.domain.clone(), @@ -276,6 +279,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method.clone(), publisher_domain: publisher_domain.clone(), access_sample_rate, + access_telemetry_enabled, }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); @@ -303,6 +307,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method.clone(), publisher_domain: publisher_domain.clone(), access_sample_rate, + access_telemetry_enabled, }, ); run_edgezero_pull_sync_after_send( @@ -336,6 +341,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method, publisher_domain, access_sample_rate, + access_telemetry_enabled, }, ); // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec @@ -467,6 +473,12 @@ fn emit_access_telemetry_after_send( return; } + // No snapshot means access telemetry was disabled when the response + // was sent (the flag is read once, before dispatch); nothing to emit. + let Some(snapshot) = &outcome.snapshot else { + return; + }; + let since_epoch = SystemTime::now() .duration_since(UNIX_EPOCH) .unwrap_or_default(); @@ -481,14 +493,14 @@ fn emit_access_telemetry_after_send( let entropy = entropy_nanos ^ outcome.bytes; if !should_sample_access_row( - outcome.snapshot.sample_rate, + snapshot.sample_rate, settings.tinybird.access_sample_rate, entropy, ) { return; } - let row = access_event_row(&outcome.snapshot, &timings.snapshot(), epoch_ms); + let row = access_event_row(snapshot, &timings.snapshot(), epoch_ms); let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); let result = futures::executor::block_on(tinybird::emit_access_event( &platform::FastlyPlatformHttpClient, @@ -547,6 +559,12 @@ struct SendContext { publisher_domain: String, /// The configured access-telemetry sample rate. access_sample_rate: f64, + /// Whether `tinybird.enabled` and `tinybird.access_enabled` were both + /// set when settings were first read. Gates building the + /// [`AccessTelemetrySnapshot`] at all: the snapshot costs env reads and + /// `String` allocations on the pre-send path, which a disabled + /// deployment (the default) should not pay. + access_telemetry_enabled: bool, } /// Outcome of handing a finalized response to the client. @@ -557,8 +575,9 @@ pub(crate) struct DeliveryOutcome { /// Whether delivery completed or failed partway. pub result: DeliveryResult, /// Access-telemetry dimensions captured for this response at the - /// freeze point. - pub snapshot: AccessTelemetrySnapshot, + /// freeze point. `None` when access telemetry was disabled at snapshot + /// time; the emitter treats that as nothing to send. + pub snapshot: Option, } /// Whether [`send_edgezero_response`] completed delivery or failed partway. @@ -691,11 +710,15 @@ fn send_edgezero_response( context.server_timing_enabled, ); - // Built unconditionally, right after the freeze point and before - // `into_parts()` consumes `response`: nothing else survives to - // post-send on every path (the request was consumed by dispatch, and - // `EcFinalizeState` is absent on asset, admin, and error paths). - let snapshot = build_access_telemetry_snapshot(&response, context); + // Built right after the freeze point and before `into_parts()` + // consumes `response`: nothing else survives to post-send on every + // path (the request was consumed by dispatch, and `EcFinalizeState` + // is absent on asset, admin, and error paths). Skipped entirely when + // access telemetry is disabled, so the default configuration pays no + // env reads or allocations here. + let snapshot = context + .access_telemetry_enabled + .then(|| build_access_telemetry_snapshot(&response, context)); let (parts, body) = response.into_parts(); @@ -1508,7 +1531,7 @@ mod tests { let outcome = DeliveryOutcome { bytes, result: DeliveryResult::Complete, - snapshot: sample_access_snapshot(), + snapshot: Some(sample_access_snapshot()), }; assert_eq!( @@ -1565,6 +1588,7 @@ mod tests { method: "GET".to_owned(), publisher_domain: "test-publisher.com".to_owned(), access_sample_rate: 0.25, + access_telemetry_enabled: true, } } @@ -1827,6 +1851,7 @@ mod tests { method: "GET".to_owned(), publisher_domain: "test-publisher.com".to_owned(), access_sample_rate: 1.0, + access_telemetry_enabled: true, }, ); assert!( @@ -1847,7 +1872,11 @@ mod tests { ..trusted_server_core::settings::TinybirdSettings::default() }, ); - let row = access_event_row(&outcome.snapshot, &timings.snapshot(), 0); + let snapshot = outcome + .snapshot + .as_ref() + .expect("should build a snapshot when access telemetry is enabled"); + let row = access_event_row(snapshot, &timings.snapshot(), 0); futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) .expect("should send access telemetry"); diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 5025f9a73..1158ada68 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -960,34 +960,6 @@ mod tests { ); } - #[test] - fn sampled_out_requests_emit_nothing() { - // Mirrors main.rs's post-send gate exactly (`if sampled_in(rate, - // entropy) { emit_access_event(...) }`): `emit_access_event` is only - // reached when `sampled_in` returns `true`. With a `0.0` rate it - // never does, for any entropy, so the http client should never see - // a request. - let http_client = RecordingHttpClient::respond_with(202); - let target = TinybirdEventsTarget::from_access_config(enabled_config()); - let rate = 0.0; - let entropy = 123_456_789_u64; - - if sampled_in(rate, entropy) { - futures::executor::block_on(emit_access_event(&http_client, &target, "{}".to_owned())) - .expect("should send when sampled in"); - } - - assert_eq!( - http_client - .requests - .lock() - .expect("should lock recorded requests") - .len(), - 0, - "sampled-out requests must never reach emit_access_event" - ); - } - #[test] fn sampled_in_boundary_rates_are_unconditional() { assert!( diff --git a/crates/trusted-server-core/examples/local_dev_config.rs b/crates/trusted-server-core/examples/local_dev_config.rs index 19826231d..acbb61b6c 100644 --- a/crates/trusted-server-core/examples/local_dev_config.rs +++ b/crates/trusted-server-core/examples/local_dev_config.rs @@ -3,8 +3,11 @@ //! Reads `trusted-server.example.toml`, replaces the placeholder secrets with //! random values, flips the flags a local smoke test needs, validates the //! result through [`trusted_server_core::settings::Settings::from_toml`], and -//! prints the blob envelope JSON that -//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG` expects. +//! prints the blob envelope JSON that the Axum adapter's +//! `TRUSTED_SERVER_CONFIG_{STORE}_{KEY}` environment variable expects. With +//! the default store and key both named `trusted_server_config`, the +//! concrete variable resolves (not a typo) to +//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG`. //! //! The random values are time-and-pid seeded, not cryptographic. This tool //! exists for throwaway local test instances only; never use its output for a diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 28e391f7e..2aac6d7d3 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -81,6 +81,9 @@ pub enum RouteClass { Ec, /// The server-side auction or SPA re-auction (`page-bids`) endpoint. AuctionApi, + /// A response proxied through a configured asset route (the + /// non-document fallback for scripts, styles, images, and fonts). + Asset, /// Everything else: discovery, tester-cookie toggles, denied legacy /// aliases, and any response with no attached [`RouteMetadata`]. Other, @@ -97,6 +100,7 @@ impl RouteClass { Self::IntegrationProxy => "integration_proxy", Self::Ec => "ec", Self::AuctionApi => "auction_api", + Self::Asset => "asset", Self::Other => "other", } } diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 6ac9a9692..8d1b92cc1 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -385,6 +385,40 @@ pub struct TimingSnapshot { mod tests { use super::*; + #[test] + fn every_phase_index_is_unique_and_in_bounds() { + // `PHASE_COUNT` and `Phase::index()` are hand-synced; nothing at + // compile time ties them together. A new variant whose `index()` + // returns `PHASE_COUNT` would panic at runtime on first `record`, + // contradicting the module's no-panics claim, so this test fails + // first instead. (A variant missing from this list is a compile + // error via the exhaustive `match` in `index()` once added there.) + let phases = [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, + Phase::AuctionWait, + Phase::Stream, + ]; + let mut seen = [false; PHASE_COUNT]; + for phase in phases { + let index = phase.index(); + assert!( + index < PHASE_COUNT, + "should be in bounds: {phase:?} -> {index}" + ); + assert!(!seen[index], "should be unique: {phase:?} -> {index}"); + seen[index] = true; + } + assert!( + seen.iter().all(|slot| *slot), + "should cover every phases-array slot" + ); + } + #[test] fn render_omits_unrecorded_phases_and_orders_total_first() { let timings = RequestTimings::new(); From 274241b9ebb0f9bcabff420bc56a3a82ab1920a0 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:00:29 -0700 Subject: [PATCH 021/104] Add auction timeline offsets spec amendment Adds section 18 to the request phase timing spec: three first-call-wins T0 offsets (auction dispatched, resolved, committed) on RequestTimings, emitted as additive nullable columns on access_logs_raw with auction_id as the join key to the per-bidder auction dataset. Answers the overlap-proof questions the two existing clocks cannot: when the auction started relative to request entry, when the final bid landed, and when targeting was committed toward GAM. --- .../2026-08-24-request-phase-timing-design.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 214d146a1..56c2685b3 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -562,3 +562,108 @@ streamed. No allocation in the hot path beyond the one `Arc` at entry, the - The stall window itself remains unattributed until this ships. If it recurs first, the bisection runbook from 2026-08-21 (cookie-free curl UA request, static-asset path versus HTML path) is the fallback. + +## 18. Auction timeline offsets (follow-up increment) + +Status: spec amendment for a follow-up PR; not part of the initial implementation +(#1074). Builds only on machinery that spec sections 5, 9, and 10 already define. + +### Problem + +The pipeline has two clocks that never meet. The auction dataset +(`auction_events_raw`, PR #813) measures the auction internally: `total_time_ms` +from auction start to terminal, `provider_response_time_ms` per bidder call. Its +clock starts when the auction observation is created, so nothing places those +numbers on the request timeline. The access row is T0-anchored but records only +`auction_wait_ms`: time the handler was blocked at collect, deliberately not the +auction's own timeline. + +That leaves three questions unanswerable today: + +1. At what request-relative time did the auction start (dispatch leave the edge)? +2. At what request-relative time did the auction resolve (final bid or timeout)? +3. At what request-relative time were the results committed toward GAM? + +These are the overlap-proof questions. A client-side wrapper cannot dispatch until +the browser boots (t≈3000ms on measured prospect pages); the server-side auction +dispatches while the origin fetch is in flight. Proving that requires all +milestones on one clock. + +### Design + +Three first-call-wins marks on `RequestTimings`, in the style of +`mark_headers_ready()`, each storing `Option` since T0: + +| Mark | Recorded at | Meaning | +| --------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| `mark_auction_dispatched()` | immediately before `orchestrator.dispatch_auction` returns control to the caller (`publisher.rs` dispatch site) | bid requests have left the edge | +| `mark_auction_resolved()` | immediately after `collect_dispatched_auction` returns, both collect sites | final bid returned or auction timed out; terminal either way | +| `mark_auction_committed()` | immediately after `write_bids_to_state` returns, both call sites | winning bids are in page state, available to the response pipeline | + +Notes on the definitions: + +- "Committed toward GAM" is defined as `write_bids_to_state` returning: the common + point in buffered and streaming modes where targeting becomes part of the + response. TS never calls GAM server-side; the browser's GPT call carries the + targeting, and that half of the timeline belongs to client-side measurement. + The edge proves when targeting was available; the client proves when GAM saw it. +- First-call-wins on all three marks. A request produces at most one publisher-path + auction today; if a second auction ever occurs in one request, the row describes + the first and the auction dataset still carries both in full. +- Same locking and failure model as every other `RequestTimings` write: `try_lock`, + drop on contention, saturating conversion at serialization. + +### Row changes + +Four additive columns on `access_logs_raw`, all populated from the +`TimingSnapshot` at the existing freeze/emission points (no new emission path): + +``` +`auction_dispatched_ms` Nullable(UInt32), `json:$.auction_dispatched_ms` +`auction_resolved_ms` Nullable(UInt32), `json:$.auction_resolved_ms` +`auction_committed_ms` Nullable(UInt32), `json:$.auction_committed_ms` +`auction_id` String, `json:$.auction_id` +``` + +- The three offsets are null when no auction ran (the common case: assets, EC + endpoints, auction-disabled deployments). Null means "no auction", never "zero". +- `auction_id` is the telemetry auction UUID already present on every + `auction_events_raw` row, carried onto the access row as the join key between + the T0 timeline and per-bidder detail. Sentinel `none` when no auction ran, + matching the non-nullable-dimension convention of section 9. It is a random + UUID, not identity-bearing; unbounded cardinality is accepted for the same + reason it is accepted in the auction dataset. +- Schema evolution is additive with JSONPaths on every new column and + `FORWARD_QUERY` carrying the existing columns, per the deployed datasource's + established evolution path. Verified with `tb --cloud deploy --check` before + deploy. + +### Interpretation model + +Combined with existing columns, one access row now reads as a timeline: + +``` +t=0 ......... request entry +t=D ......... auction_dispatched_ms (bids out; origin fetch typically in flight) +t=R ......... auction_resolved_ms (R - D ~ auction duration; join auction_id + for the per-bidder long pole) +t=C ......... auction_committed_ms (targeting in page state) +t=H ......... time_elapsed_ms (headers committed) +``` + +Derivations the dashboard can add without schema help: auction duration on the +request clock (`R - D`), commit latency (`C - R`), and overlap ratio (share of +`R - D` that ran concurrently with `ts-origin`). `auction_wait_ms` keeps its +existing meaning (blocked time only) and is now interpretable next to the +timeline: `R - D` minus `auction_wait_ms` approximates how much of the auction +was absorbed by work the request needed anyway. + +### Scope + +- Fastly emits; Axum, Cloudflare, and Spin collect the marks but do not emit, + matching section 8a adapter semantics. +- No header emission for any of these values: they are post-hoc analysis fields, + and two of the three are typically unknown at the header freeze point in + streaming mode. +- No config surface: the marks are always-on collection like every other phase, + gated at emission by the existing `tinybird.access_enabled`. From 29d45e8c8087173ef8daa221898892b58a399989 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:09 -0700 Subject: [PATCH 022/104] Add auction timeline offsets implementation plan --- .../2026-08-26-auction-timeline-offsets.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md diff --git a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md new file mode 100644 index 000000000..08e14dabc --- /dev/null +++ b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md @@ -0,0 +1,59 @@ +# Auction Timeline Offsets Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Record three T0-anchored auction milestones (dispatched, resolved, committed) plus the auction id on `RequestTimings`, and emit them as four additive columns on the `access_logs_raw` row. + +**Architecture:** Follows spec section 18 exactly. All state lives in the existing `RequestTimings` inner (same `try_lock`/first-call-wins/saturating model as `mark_headers_ready`); the row builder reads the values from `TimingSnapshot`, so no new emission path and no adapter changes. + +**Tech Stack:** Rust (core crate only), Tinybird datasource file. + +**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` section 18. + +## Global Constraints + +- Marks are first-call-wins; `try_lock` only; a contended lock drops the sample. +- Null offsets mean "no auction ran", never zero. `auction_id` sentinel is `none`. +- Column names: `auction_dispatched_ms`, `auction_resolved_ms`, `auction_committed_ms`, `auction_id`; JSONPaths `json:$.`; FORWARD_QUERY extended in the same order. +- Dispatch mark records only on `DispatchAuctionOutcome::Dispatched`; a failed dispatch leaves all three offsets null (the auction dataset still records the failure). +- No header emission, no config surface, no changes outside `trusted-server-core` and `tinybird/`. + +--- + +### Task 1: RequestTimings marks and snapshot fields + +**Files:** +- Modify: `crates/trusted-server-core/src/request_timing.rs` + +**Interfaces:** +- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option, auction_id: Option, .. }` + +- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option` and `auction_id: Option` to `Inner`; initialize `None`. +- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins. +- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone. +- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`. +- [ ] `cargo test-fastly request_timing`, commit. + +### Task 2: Publisher call sites + +**Files:** +- Modify: `crates/trusted-server-core/src/publisher.rs` + +**Interfaces:** +- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site. + +- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());` +- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`. +- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`. +- [ ] `cargo test-fastly`, commit. + +### Task 3: Row columns and datasource + +**Files:** +- Modify: `crates/trusted-server-core/src/access_telemetry.rs` +- Modify: `tinybird/datasources/access_logs_raw.datasource` + +- [ ] `access_event_row`: add the three offset keys (nullable) and `auction_id` with `none` sentinel, after the existing phase keys. +- [ ] Extend `row_serializes_nulls_for_missing_phases` and `row_serializes_recorded_phases_as_numbers` for the new keys. +- [ ] Datasource: four schema columns with JSONPaths (`Nullable(UInt32)` ×3, `String`), appended at the end of SCHEMA and FORWARD_QUERY so existing column order stays stable. +- [ ] Full gates: fmt, clippy (all six), test-fastly/axum/cloudflare/spin, parity. Commit. From 46911a647347adc13791171995839e43a92c56fb Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:09 -0700 Subject: [PATCH 023/104] Record T0-anchored auction timeline offsets in the access row Implements spec section 18: three first-call-wins marks on RequestTimings (dispatched at the DispatchAuctionOutcome::Dispatched arm, resolved after collect at both sites, committed after write_bids_to_state at both sites), carried through TimingSnapshot into four additive access_logs_raw columns: auction_dispatched_ms, auction_resolved_ms, auction_committed_ms, and auction_id as the join key to the per-bidder auction dataset. Null offsets mean no auction ran; a failed dispatch records nothing. FORWARD_QUERY fills the new columns with typed defaults for pre-existing rows. No header emission, no config surface, no adapter changes: the values ride the existing snapshot and the tinybird.access_enabled gate. --- .../wrangler.integration.generated.toml | 16 +++ .../src/access_telemetry.rs | 19 +++ crates/trusted-server-core/src/publisher.rs | 14 ++ .../trusted-server-core/src/request_timing.rs | 134 ++++++++++++++++++ .../datasources/access_logs_raw.datasource | 8 +- 5 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml new file mode 100644 index 000000000..263403193 --- /dev/null +++ b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml @@ -0,0 +1,16 @@ +name = "trusted-server" +main = "build/index.js" +compatibility_date = "2024-09-23" +# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the +# outbound `CacheMode::NoStore` cache bypass under this compatibility date. +compatibility_flags = ["nodejs_compat", "cache_option_enabled"] +# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. + +[[kv_namespaces]] +binding = "TRUSTED_SERVER_KV" +id = "ci-local-kv" + +[vars] +# Placeholder replaced by the integration test harness with a JSON object that +# contains the runtime Trusted Server app-config blob envelope. +TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 2aac6d7d3..083adbe8e 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -286,6 +286,10 @@ pub fn access_event_row( "stream_ms": timings.stream_ms, "request_elapsed_ms": timings.request_elapsed_ms, "resp_bytes": timings.resp_bytes, + "auction_dispatched_ms": timings.auction_dispatched_ms, + "auction_resolved_ms": timings.auction_resolved_ms, + "auction_committed_ms": timings.auction_committed_ms, + "auction_id": timings.auction_id.as_deref().unwrap_or("none"), "template_cache_state": snapshot.template_cache_state, "country": snapshot.country, "ts_version": snapshot.ts_version, @@ -463,6 +467,9 @@ mod tests { "stream_ms", "request_elapsed_ms", "resp_bytes", + "auction_dispatched_ms", + "auction_resolved_ms", + "auction_committed_ms", ] { assert!( parsed[field].is_null(), @@ -488,6 +495,10 @@ mod tests { ); } assert_eq!(parsed["auction_wait_placement"], "none"); + assert_eq!( + parsed["auction_id"], "none", + "auction_id should carry the none sentinel when no auction ran" + ); } #[test] @@ -506,6 +517,10 @@ mod tests { stream_ms: Some(8), auction_wait_placement: Some(AuctionWaitPlacement::InStream), resp_bytes: Some(1024), + auction_dispatched_ms: Some(9), + auction_resolved_ms: Some(10), + auction_committed_ms: Some(11), + auction_id: Some("33333333-3333-3333-3333-333333333333".to_owned()), }; let row = access_event_row(&snapshot, &timings, 1_700_000_000_000); let parsed: serde_json::Value = @@ -515,6 +530,10 @@ mod tests { assert_eq!(parsed["stream_ms"], 8); assert_eq!(parsed["resp_bytes"], 1024); assert_eq!(parsed["auction_wait_placement"], "in_stream"); + assert_eq!(parsed["auction_dispatched_ms"], 9); + assert_eq!(parsed["auction_resolved_ms"], 10); + assert_eq!(parsed["auction_committed_ms"], 11); + assert_eq!(parsed["auction_id"], "33333333-3333-3333-3333-333333333333"); } #[test] diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 53d09bfc6..3981c386b 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3951,6 +3951,8 @@ async fn collect_non_html_auction( params .timings .record_auction_wait(placement, wait_started.elapsed()); + // T0-anchored timeline mark (spec section 18): final bid or timeout. + params.timings.mark_auction_resolved(); let delivered_winner_slots = write_bids_to_state( &result.winning_bids, params.price_granularity, @@ -3960,6 +3962,9 @@ async fn collect_non_html_auction( settings.debug.inject_adm_for_testing, auction_id.as_deref(), ); + // T0-anchored timeline mark (spec section 18): winning bids are in page + // state, available to the response pipeline. + params.timings.mark_auction_committed(); if let (Some(observation), Some(auction_request)) = (telemetry.observation, telemetry.auction_request.as_ref()) { @@ -4010,6 +4015,8 @@ async fn collect_stream_auction( .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; timings.record_auction_wait(*placement, wait_started.elapsed()); + // T0-anchored timeline mark (spec section 18): final bid or timeout. + timings.mark_auction_resolved(); log::info!( "body_close_hold_loop: collect complete - {} winning bid(s)", result.winning_bids.len() @@ -4023,6 +4030,9 @@ async fn collect_stream_auction( settings.debug.inject_adm_for_testing, auction_id.as_deref(), ); + // T0-anchored timeline mark (spec section 18): winning bids are in page + // state, available to the response pipeline. + timings.mark_auction_committed(); if let (Some(observation), Some(auction_request)) = (telemetry.observation, telemetry.auction_request.as_ref()) { @@ -4340,6 +4350,10 @@ pub async fn handle_publisher_request( .await { DispatchAuctionOutcome::Dispatched(dispatched) => { + // T0-anchored timeline mark (spec section 18): bid + // requests have left the edge. A failed dispatch never + // marks, so all three auction offsets stay null for it. + timings.mark_auction_dispatched(observation.auction_id.to_string()); auction_request_for_telemetry = Some(auction_request); auction_observation = Some(observation); Some(dispatched) diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 8d1b92cc1..edcd9bfc2 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -106,6 +106,19 @@ struct Inner { /// Response body size in bytes, set via /// [`RequestTimings::set_resp_bytes`]. resp_bytes: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_dispatched`] call. + auction_dispatched: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_resolved`] call. + auction_resolved: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_committed`] call. + auction_committed: Option, + /// Telemetry auction UUID recorded by the first + /// [`RequestTimings::mark_auction_dispatched`] call; joins the access + /// row to the per-bidder auction dataset. + auction_id: Option, } /// Per-request phase timing collector. @@ -128,6 +141,10 @@ impl RequestTimings { request_elapsed: None, auction_wait_placement: None, resp_bytes: None, + auction_dispatched: None, + auction_resolved: None, + auction_committed: None, + auction_id: None, }))) } @@ -200,6 +217,53 @@ impl RequestTimings { } } + /// Stamps the elapsed time since `t0` as the auction dispatch offset and + /// records the telemetry auction id, the first time this is called. + /// + /// Called when `dispatch_auction` reports the bid requests dispatched; + /// a failed dispatch never records, so all three auction offsets stay + /// `None` for it. Subsequent calls are no-ops (first call wins). Drops + /// the sample silently on lock contention or poisoning. + pub fn mark_auction_dispatched(&self, auction_id: String) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_dispatched.is_none() { + inner.auction_dispatched = Some(inner.t0.elapsed()); + inner.auction_id = Some(auction_id); + } + } + + /// Stamps the elapsed time since `t0` as the auction resolve offset (the + /// final bid returned or the auction timed out), the first time this is + /// called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_auction_resolved(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_resolved.is_none() { + inner.auction_resolved = Some(inner.t0.elapsed()); + } + } + + /// Stamps the elapsed time since `t0` as the auction commit offset + /// (winning bids written into page state), the first time this is + /// called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_auction_committed(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_committed.is_none() { + inner.auction_committed = Some(inner.t0.elapsed()); + } + } + /// Records the response body size in bytes. /// /// Drops the sample silently on lock contention or poisoning. @@ -257,6 +321,10 @@ impl RequestTimings { stream_ms: duration_ms(inner.phases[Phase::Stream.index()]), auction_wait_placement: inner.auction_wait_placement, resp_bytes: inner.resp_bytes, + auction_dispatched_ms: duration_ms(inner.auction_dispatched), + auction_resolved_ms: duration_ms(inner.auction_resolved), + auction_committed_ms: duration_ms(inner.auction_committed), + auction_id: inner.auction_id.clone(), } } } @@ -379,6 +447,18 @@ pub struct TimingSnapshot { /// Response body size in bytes, set via /// [`RequestTimings::set_resp_bytes`]. pub resp_bytes: Option, + /// T0 offset at which the auction dispatched (bid requests left the + /// edge), or `None` when no auction ran. + pub auction_dispatched_ms: Option, + /// T0 offset at which the auction resolved (final bid or timeout), or + /// `None` when no auction ran. + pub auction_resolved_ms: Option, + /// T0 offset at which winning bids were committed into page state, or + /// `None` when no auction ran. + pub auction_committed_ms: Option, + /// Telemetry auction UUID joining this row to the auction dataset, or + /// `None` when no auction ran. + pub auction_id: Option, } #[cfg(test)] @@ -419,6 +499,60 @@ mod tests { ); } + #[test] + fn auction_marks_are_first_call_wins_and_snapshot_maps_them() { + let timings = RequestTimings::new(); + timings.mark_auction_dispatched("11111111-1111-1111-1111-111111111111".to_owned()); + timings.mark_auction_resolved(); + timings.mark_auction_committed(); + // Second calls must not overwrite the first-recorded values. + timings.mark_auction_dispatched("22222222-2222-2222-2222-222222222222".to_owned()); + timings.mark_auction_resolved(); + timings.mark_auction_committed(); + + let snapshot = timings.snapshot(); + assert!( + snapshot.auction_dispatched_ms.is_some(), + "should record the dispatch offset" + ); + assert!( + snapshot.auction_resolved_ms.is_some(), + "should record the resolve offset" + ); + assert!( + snapshot.auction_committed_ms.is_some(), + "should record the commit offset" + ); + assert_eq!( + snapshot.auction_id.as_deref(), + Some("11111111-1111-1111-1111-111111111111"), + "should keep the first-recorded auction id" + ); + } + + #[test] + fn snapshot_without_auction_marks_yields_none_for_all_offsets() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_dispatched_ms, None, + "should stay None when no auction dispatched" + ); + assert_eq!( + snapshot.auction_resolved_ms, None, + "should stay None when no auction resolved" + ); + assert_eq!( + snapshot.auction_committed_ms, None, + "should stay None when no auction committed" + ); + assert_eq!( + snapshot.auction_id, None, + "should carry no auction id when no auction ran" + ); + } + #[test] fn render_omits_unrecorded_phases_and_orders_total_first() { let timings = RequestTimings::new(); diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 062b884ba..4441de384 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -27,13 +27,17 @@ SCHEMA > `template_cache_state` LowCardinality(String) `json:$.template_cache_state`, `country` LowCardinality(String) `json:$.country`, `ts_version` LowCardinality(String) `json:$.ts_version`, - `pop` LowCardinality(String) `json:$.pop` + `pop` LowCardinality(String) `json:$.pop`, + `auction_dispatched_ms` Nullable(UInt32) `json:$.auction_dispatched_ms`, + `auction_resolved_ms` Nullable(UInt32) `json:$.auction_resolved_ms`, + `auction_committed_ms` Nullable(UInt32) `json:$.auction_committed_ms`, + `auction_id` String `json:$.auction_id` ENGINE "MergeTree" ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status" TTL "toDate(event_ts) + INTERVAL 30 DAY" FORWARD_QUERY > - SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop + SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop, CAST(NULL AS Nullable(UInt32)) AS auction_dispatched_ms, CAST(NULL AS Nullable(UInt32)) AS auction_resolved_ms, CAST(NULL AS Nullable(UInt32)) AS auction_committed_ms, 'none' AS auction_id TOKEN ts_access_ingest APPEND From 1fa9f8cc09df889aec42039b13b7a108d1df9d47 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:29 -0700 Subject: [PATCH 024/104] Remove generated integration wrangler config from tracking The Cloudflare integration harness writes wrangler.integration.generated.toml at test time; it was swept into the previous commit by accident. Ignore it so local CI=1 runs cannot commit it again. --- .gitignore | 1 + .../wrangler.integration.generated.toml | 16 ---------------- 2 files changed, 1 insertion(+), 16 deletions(-) delete mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/.gitignore b/.gitignore index 24b9e06aa..96ffa2a5c 100644 --- a/.gitignore +++ b/.gitignore @@ -63,3 +63,4 @@ src/*.html # leftover local build artifacts (node_modules, target, dist) that remain on disk. /crates/js/ /crates/integration-tests/ +wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml deleted file mode 100644 index 263403193..000000000 --- a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml +++ /dev/null @@ -1,16 +0,0 @@ -name = "trusted-server" -main = "build/index.js" -compatibility_date = "2024-09-23" -# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the -# outbound `CacheMode::NoStore` cache bypass under this compatibility date. -compatibility_flags = ["nodejs_compat", "cache_option_enabled"] -# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. - -[[kv_namespaces]] -binding = "TRUSTED_SERVER_KV" -id = "ci-local-kv" - -[vars] -# Placeholder replaced by the integration test harness with a JSON object that -# contains the runtime Trusted Server app-config blob envelope. -TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' From 2ef7635437d50b1c4ec3eac57321aeb168970c0c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 025/104] Reject single-segment publisher paths in route templates The first path segment is only a section name when the path has depth: under a /%postname%/ permalink structure every article is a single-segment path, so keeping those segments verbatim put full article slugs into the 30-day dataset, against spec section 9. Depth is now required for a named template; single-segment paths, root landing pages included, bucket to /other/*. Route slicing keeps route_class and multi-segment templates like /news/*. --- .../src/access_telemetry.rs | 69 ++++++++++++++----- 1 file changed, 50 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 2aac6d7d3..aaad46633 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -130,27 +130,27 @@ pub struct RouteMetadata { /// content-free route template. /// /// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, with a trailing `/*` appended when the path has -/// additional segments beyond the first. The root path `/` maps to itself. -/// A first segment is rejected to `/other/*` — outright, never filtered or -/// truncated, so no fragment of it ever reaches the row — when it: +/// `[a-z0-9_-]`, plus `/*`, only when the path has at least two segments: +/// depth is what makes the first segment a section name (`/news/*`) rather +/// than the document itself. The root path `/` maps to itself. Everything +/// else is rejected to `/other/*` — outright, never filtered or truncated, +/// so no fragment of a rejected path ever reaches the row: /// -/// - is empty, or contains any character outside the allowlist after -/// lowercasing (an email address, a search phrase); -/// - is longer than [`MAX_SEGMENT_LEN`] characters (UUIDs, long hex -/// tokens, and full article slugs all exceed it — a truncated prefix of -/// any of these would still be identifying); or -/// - contains more than [`MAX_SEGMENT_DIGITS`] ASCII digits. Opaque -/// identifiers (hex ids, base36 ids, reset tokens) are digit-heavy; -/// publisher section names are words, at most a year or a version -/// number. +/// - single-segment paths (`/my-post-title` under a `/%postname%/` +/// permalink structure is a per-article slug that no shape heuristic +/// can separate from a section name); +/// - an empty first segment, or one containing any character outside the +/// allowlist after lowercasing (an email address, a search phrase); +/// - a first segment longer than [`MAX_SEGMENT_LEN`] characters (a +/// truncated prefix of a UUID or token would still be identifying); or +/// - a first segment with more than [`MAX_SEGMENT_DIGITS`] ASCII digits +/// (hex ids, base36 ids, and reset tokens are digit-heavy; section +/// names carry at most a year or a version number). /// /// This is deliberately coarser than the auction-telemetry path /// normalizer, which redacts long tokens but preserves short identifiers /// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. Short all-alpha slugs on single-segment paths are -/// indistinguishable from section names and still pass; the bound here is -/// shape-based, not semantic. +/// this broad. /// /// # Examples /// @@ -193,7 +193,11 @@ pub fn publisher_route_template(path: &str) -> String { if has_more_depth { format!("/{lowered}/*") } else { - format!("/{lowered}") + // A single-segment path's first segment is the document, not a + // section: `/my-post-title` under WordPress `/%postname%/` is a + // per-article slug, and no shape heuristic can separate it from a + // section name. Only depth >= 2 makes the first segment a section. + "/other/*".to_owned() } } @@ -436,11 +440,37 @@ mod tests { } #[test] - fn publisher_route_template_uppercases_lowercase_before_allowlisting() { + fn publisher_route_template_rejects_single_segment_paths() { + // A single-segment path's first segment is the document itself + // (WordPress `/%postname%/` puts every article at depth 1), so no + // shape heuristic can separate a slug from a section name; depth + // is the only safe signal. Root-level landing pages pay for this + // deliberately. + assert_eq!(publisher_route_template("/my-post-title"), "/other/*"); + assert_eq!( + publisher_route_template("/my-post-title/"), + "/other/*", + "a trailing slash should not count as depth" + ); + assert_eq!( + publisher_route_template("/1234567"), + "/other/*", + "a numeric post id at the digit boundary should still reject" + ); + assert_eq!(publisher_route_template("/user-8f3a9c2b"), "/other/*"); + assert_eq!( + publisher_route_template("/about"), + "/other/*", + "root-level landing pages reject too; only depth makes a section" + ); + } + + #[test] + fn publisher_route_template_lowercases_before_allowlisting() { assert_eq!( publisher_route_template("/News/Article"), "/news/*", - "should lowercase before validating and truncating" + "should lowercase before validating" ); } @@ -524,6 +554,7 @@ mod tests { assert_eq!(RouteClass::IntegrationProxy.as_str(), "integration_proxy"); assert_eq!(RouteClass::Ec.as_str(), "ec"); assert_eq!(RouteClass::AuctionApi.as_str(), "auction_api"); + assert_eq!(RouteClass::Asset.as_str(), "asset"); assert_eq!(RouteClass::Other.as_str(), "other"); } From 082461d10eeabf1e5658dd27ebd89f4b6068af3d Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 026/104] Sample access rows with real randomness at the snapshot rate The bucket-quantized sampler truncated rates below one in a million to a zero threshold (silently emitting nothing) and quantized other low rates downward while rows still carried the configured rate, biasing the sum(1.0 / sample_rate) volume estimator. Its no-rand premise was also wrong: rand::thread_rng() is WASI-backed on this target and the EC generation path already relies on it. The sampler is now a direct uniform-roll comparison, and the roll gates on the rate stored in the snapshot itself, so emission probability and the row's sample_rate column cannot diverge; the divergence guard and its tests are removed. Also per review: the settings-reload fallback in the post-send path could never emit (no snapshot exists when settings were absent) and is removed; the dead_code allow on DeliveryOutcome narrows to the one collected-but-unemitted field; and the post-send ordering test is narrowed to the leg it actually proves, that request_elapsed is stamped when send returns. --- Cargo.lock | 1 + .../trusted-server-adapter-fastly/Cargo.toml | 1 + .../trusted-server-adapter-fastly/src/main.rs | 243 +++--------------- .../src/tinybird.rs | 83 +++--- 4 files changed, 85 insertions(+), 243 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e29380b77..201060a52 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5372,6 +5372,7 @@ dependencies = [ "futures", "log", "log-fastly", + "rand 0.8.6", "serde", "serde_json", "sha2 0.10.9", diff --git a/crates/trusted-server-adapter-fastly/Cargo.toml b/crates/trusted-server-adapter-fastly/Cargo.toml index 47cc609b2..3835d6469 100644 --- a/crates/trusted-server-adapter-fastly/Cargo.toml +++ b/crates/trusted-server-adapter-fastly/Cargo.toml @@ -29,6 +29,7 @@ serde = { workspace = true } serde_json = { workspace = true } sha2 = { workspace = true } trusted-server-core = { workspace = true } +rand = { workspace = true } url = { workspace = true } urlencoding = { workspace = true } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 934f98402..b5b08ab2e 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,4 +1,6 @@ use std::sync::Arc; + +use rand::Rng as _; use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; @@ -346,17 +348,12 @@ fn edgezero_main(mut req: FastlyRequest) { ); // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec // finalize branch above failed), so there is no pull-sync dispatch here - // at all — telemetry is the only post-send step. Reload settings when - // `app_state` never built, matching the fallback used earlier in this - // function for entry-point finalize headers. - match settings_snapshot.as_deref() { - Some(settings) => emit_access_telemetry_after_send(settings, &outcome, &timings), - None => match load_settings_from_config_store() { - Ok(settings) => emit_access_telemetry_after_send(&settings, &outcome, &timings), - Err(e) => { - log::warn!("access telemetry emission skipped: failed to reload settings: {e:?}"); - } - }, + // at all — telemetry is the only post-send step. When `app_state` never + // built there is nothing to emit either: `access_telemetry_enabled` was + // necessarily false without a settings snapshot, so the outcome carries + // no access snapshot, and reloading settings here could not change that. + if let Some(settings) = settings_snapshot.as_deref() { + emit_access_telemetry_after_send(settings, &outcome, &timings); } } @@ -458,9 +455,9 @@ fn run_edgezero_pull_sync_after_send( /// either of those per-route types, so every response class can emit. /// /// Sampled-out requests return silently — that is the expected, high-volume -/// case and not worth a log line. A snapshot carrying a degraded -/// `sample_rate` (see [`should_sample_access_row`]) also returns silently, -/// since it only occurs on an already-degraded path. Every other drop (row +/// case and not worth a log line. The sampling roll uses the rate stored on +/// the snapshot itself, so the emission probability always matches the +/// row's `sample_rate` column by construction. Every other drop (row /// build, token load, send, or non-2xx status — all folded into /// `emit_access_event`'s `Result`) logs exactly one warning naming the /// reason. @@ -483,20 +480,12 @@ fn emit_access_telemetry_after_send( .duration_since(UNIX_EPOCH) .unwrap_or_default(); let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); - // Entropy for the sampling decision: the timestamp's nanosecond - // resolution XORed with a per-request value already on hand - // (`outcome.bytes`), so two requests handled in the same instance never - // collide on sampling decisions purely because they read the same - // millisecond. There is no `rand` crate dependency here — see - // `tinybird::sampled_in`. - let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); - let entropy = entropy_nanos ^ outcome.bytes; - - if !should_sample_access_row( - snapshot.sample_rate, - settings.tinybird.access_sample_rate, - entropy, - ) { + // Sample with the rate stored on the snapshot itself — the same value + // serialized into the row's `sample_rate` column — so the emission + // probability and the row's claimed rate cannot diverge, which the + // documented `sum(1.0 / sample_rate)` volume estimator depends on. + let roll = rand::thread_rng().r#gen::(); + if !tinybird::sampled_in(snapshot.sample_rate, roll) { return; } @@ -512,39 +501,6 @@ fn emit_access_telemetry_after_send( } } -/// Whether one response's access-telemetry row should be emitted, combining -/// the degraded-snapshot guard with the sampling roll. -/// -/// `snapshot_sample_rate` is the rate recorded on the [`AccessTelemetrySnapshot`] -/// itself (the value serialized into the row's `sample_rate` column, which -/// the documented volume estimator divides by as `1.0 / sample_rate`). -/// `settings_sample_rate` is the rate used for the sampling decision at -/// call time. The two can diverge: when `app_state` fails to build, -/// [`edgezero_main`] captures a snapshot with `sample_rate` defaulted to -/// `0.0` before any settings ever load, but the two settings-reload -/// emission sites still gate and sample using the *reloaded* settings' -/// (nonzero) rate. Without this guard, such a row could be sampled in and -/// emitted while carrying `sample_rate: 0.0`, corrupting the volume -/// estimator. Dropping these rows is acceptable: they only occur on an -/// already-degraded path, consistent with this pipeline's fail-quiet -/// telemetry policy. Split out of [`emit_access_telemetry_after_send`] so -/// the guard is unit-testable without a network seam. -/// -/// Callers must already have applied the coarse -/// `tinybird.enabled`/`access_enabled` gate. -#[must_use] -fn should_sample_access_row( - snapshot_sample_rate: f64, - settings_sample_rate: f64, - entropy: u64, -) -> bool { - if snapshot_sample_rate <= 0.0 { - return false; - } - - tinybird::sampled_in(settings_sample_rate, entropy) -} - /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -568,11 +524,12 @@ struct SendContext { } /// Outcome of handing a finalized response to the client. -#[allow(dead_code)] pub(crate) struct DeliveryOutcome { /// Response body size in bytes. pub bytes: u64, - /// Whether delivery completed or failed partway. + /// Whether delivery completed or failed partway. Collected as + /// groundwork; not yet emitted on any surface. + #[allow(dead_code)] pub result: DeliveryResult, /// Access-telemetry dimensions captured for this response at the /// freeze point. `None` when access telemetry was disabled at snapshot @@ -672,6 +629,11 @@ fn drive_streaming_body( /// A drive that failed after writing at least one byte delivered a truncated /// response rather than nothing at all, so it is [`DeliveryResult::Partial`], /// not [`DeliveryResult::Error`]. +/// +/// The `Ok(())` arm exists for the classifier's totality, not for the +/// production caller: `send_edgezero_response` consumes this value only in +/// its `Err` branch and re-derives the success outcome from +/// `streaming_body.finish()`. fn classify_stream_delivery( drive_result: &Result<(), Report>, bytes: u64, @@ -1036,7 +998,6 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; - use std::sync::Mutex; use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; use trusted_server_core::request_timing::AuctionWaitPlacement; @@ -1760,83 +1721,14 @@ mod tests { ); } - /// Records `"telemetry"` into a shared order log instead of sending a - /// real request, standing in for the adapter's platform HTTP client in - /// [`post_send_order_is_elapsed_then_pull_sync_then_telemetry`]. - struct OrderingHttpClient { - log: Arc>>, - } - - #[async_trait::async_trait(?Send)] - impl trusted_server_core::platform::PlatformHttpClient for OrderingHttpClient { - async fn send( - &self, - _request: trusted_server_core::platform::PlatformHttpRequest, - ) -> Result< - trusted_server_core::platform::PlatformResponse, - Report, - > { - self.log - .lock() - .expect("should lock order log") - .push("telemetry"); - let response = response_builder() - .status(edgezero_core::http::StatusCode::ACCEPTED) - .body(EdgeBody::empty()) - .expect("should build ordering test response"); - Ok(trusted_server_core::platform::PlatformResponse::new( - response, - )) - } - - async fn send_async( - &self, - _request: trusted_server_core::platform::PlatformHttpRequest, - ) -> Result< - trusted_server_core::platform::PlatformPendingRequest, - Report, - > { - Err(Report::new( - trusted_server_core::platform::PlatformError::Unsupported, - )) - } - - async fn select( - &self, - _pending_requests: Vec, - ) -> Result< - trusted_server_core::platform::PlatformSelectResult, - Report, - > { - Err(Report::new( - trusted_server_core::platform::PlatformError::Unsupported, - )) - } - } - #[test] - fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { - // `edgezero_main` cannot be driven directly in a unit test (it - // consumes a live `fastly::Request::from_client()`), and - // `run_edgezero_pull_sync_after_send` has no injectable seam of its - // own — it dispatches through the real identity-graph pull-sync - // path, which needs a configured EC KV store, partner registry, and - // rate limiter wired together. This test instead exercises the two - // REAL functions `edgezero_main` calls that DO have a testable seam - // — `send_edgezero_response` (which stamps `request_elapsed` before - // returning, per Task 6/7) and `tinybird::emit_access_event` (the - // telemetry send added by this task) — around an instrumented - // stand-in for the pull-sync dispatch call, in the exact order - // `edgezero_main` places them. - // - // This proves the elapsed-before-telemetry leg from real production - // code (the assertion below reads the real `timings` snapshot - // between the two calls). The pull-sync-before-telemetry leg is a - // source-order invariant in `edgezero_main`'s three call sites - // (verified by code review, not by this test) because - // `run_edgezero_pull_sync_after_send` itself has no seam to - // instrument — see task-8-report.md for this residual. - let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + fn request_elapsed_is_stamped_when_send_returns() { + // `edgezero_main`'s post-send ordering (pull-sync before telemetry) + // is a source-order invariant with no injectable seam, so this test + // deliberately proves only the leg that has one: by the time + // `send_edgezero_response` returns, `request_elapsed` is already + // stamped, so everything `edgezero_main` runs afterwards (pull-sync, + // telemetry emission) is excluded from `request_elapsed_ms`. let timings = RequestTimings::new(); let response = response_builder() .body(EdgeBody::from("ok")) @@ -1854,77 +1746,14 @@ mod tests { access_telemetry_enabled: true, }, ); - assert!( - timings.snapshot().request_elapsed_ms.is_some(), - "request_elapsed should already be stamped before pull-sync/telemetry run" - ); - - // Stand-in for `run_edgezero_pull_sync_after_send`, which has no - // injectable seam (see the test doc comment above). - log.lock().expect("should lock order log").push("pull_sync"); - let http_client = OrderingHttpClient { - log: Arc::clone(&log), - }; - let target = tinybird::TinybirdEventsTarget::from_access_config( - trusted_server_core::settings::TinybirdSettings { - api_host: "api.us-east.aws.tinybird.co".to_owned(), - ..trusted_server_core::settings::TinybirdSettings::default() - }, - ); - let snapshot = outcome - .snapshot - .as_ref() - .expect("should build a snapshot when access telemetry is enabled"); - let row = access_event_row(snapshot, &timings.snapshot(), 0); - - futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) - .expect("should send access telemetry"); - - assert_eq!( - *log.lock().expect("should lock order log"), - vec!["pull_sync", "telemetry"], - "pull-sync must dispatch before telemetry emits" - ); - } - - #[test] - fn should_sample_access_row_rejects_a_degraded_zero_sample_rate() { - // A snapshot captured on the app-state-build-failure fallback path - // carries `sample_rate: 0.0`. Even when the reloaded settings' rate - // would sample every request in (1.0), the row must not emit — - // otherwise it would claim `sample_rate: 0.0` and corrupt the - // `sum(1.0 / sample_rate)` volume estimator. - assert!( - !should_sample_access_row(0.0, 1.0, 0), - "a snapshot with sample_rate 0.0 must never emit, regardless of entropy or settings' rate" - ); assert!( - !should_sample_access_row(0.0, 1.0, u64::MAX), - "the degraded-rate guard must not depend on the entropy value" - ); - } - - #[test] - fn should_sample_access_row_rejects_a_negative_sample_rate() { - assert!( - !should_sample_access_row(-1.0, 1.0, 0), - "a negative snapshot sample_rate is equally degraded and must not emit" - ); - } - - #[test] - fn should_sample_access_row_defers_to_the_settings_sampling_roll_when_not_degraded() { - // With a healthy (nonzero) snapshot sample_rate, the outcome should - // match `tinybird::sampled_in` exactly, since that is the only - // remaining decision. - assert!( - should_sample_access_row(0.25, 1.0, 0), - "a settings rate of 1.0 always samples in, independent of entropy" + timings.snapshot().request_elapsed_ms.is_some(), + "request_elapsed should be stamped by the time send returns" ); assert!( - !should_sample_access_row(0.25, 0.0, 0), - "a settings rate of 0.0 always samples out, independent of the snapshot's rate" + outcome.snapshot.is_some(), + "the access snapshot should exist for the enabled context" ); } } diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 1158ada68..97a1ffd3e 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -236,39 +236,26 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { // Access telemetry: confirmed-delivery emitter // --------------------------------------------------------------------------- -/// Bucket count [`sampled_in`] maps `entropy` into. -/// -/// Large enough that `rate` values with several significant digits (e.g. -/// `0.015`) still land in a distinct bucket instead of rounding away, while -/// staying well inside `u64` range once multiplied by `rate`. -const ACCESS_SAMPLE_BUCKETS: u64 = 1_000_000; - /// Decides whether one request's access-telemetry row should be emitted. /// -/// `entropy` should vary from request to request — callers derive it from -/// the wall-clock event timestamp `XORed` with a cheap per-request value (see -/// the call site in `main.rs`). There is no `rand` crate dependency here: -/// the wasm32-wasip1 guest has no equivalent to `Math.random()`. Mapping -/// `entropy % ACCESS_SAMPLE_BUCKETS` into `[0, 1)` and comparing against -/// `rate` is not cryptographically uniform (the low bits of a timestamp are -/// not perfectly evenly distributed), but access-telemetry sampling only -/// needs an approximately even sample, not a provably unbiased one. +/// `roll` is a uniform draw from `[0, 1)`; callers pass +/// `rand::thread_rng().r#gen::()`, which the wasm32-wasip1 guest backs with +/// real WASI randomness (the EC generation path already relies on this and +/// the CI wasm release build verifies it). Comparing the draw directly +/// against `rate` keeps the sampling probability exactly `rate` for every +/// positive value: there is no bucket quantization, so rates below one in a +/// million sample proportionally instead of never, and emitted rows' +/// `sample_rate` matches the probability they were sampled at, which the +/// `sum(1.0 / sample_rate)` volume estimator depends on. /// -/// `rate <= 0.0` always returns `false` and `rate >= 1.0` always returns -/// `true`, independent of `entropy`, so both boundary configurations behave -/// predictably. `0.0` cannot actually occur while `access_enabled` is `true` -/// (`Settings` validation requires `access_sample_rate > 0.0` in that case), -/// but this function stays total rather than leaning on that invariant. +/// `rate <= 0.0` never samples and `rate >= 1.0` always samples, for any +/// `roll` in `[0, 1)`. `0.0` cannot actually occur while `access_enabled` +/// is `true` (`Settings` validation requires `access_sample_rate > 0.0` in +/// that case), but this function stays total rather than leaning on that +/// invariant. #[must_use] -pub(crate) fn sampled_in(rate: f64, entropy: u64) -> bool { - if rate >= 1.0 { - return true; - } - if rate <= 0.0 { - return false; - } - let threshold = (rate * ACCESS_SAMPLE_BUCKETS as f64) as u64; - entropy % ACCESS_SAMPLE_BUCKETS < threshold +pub(crate) fn sampled_in(rate: f64, roll: f64) -> bool { + roll < rate } /// Loads and validates the access-log APPEND token from the Fastly secret store. @@ -963,20 +950,44 @@ mod tests { #[test] fn sampled_in_boundary_rates_are_unconditional() { assert!( - sampled_in(1.0, 0), + sampled_in(1.0, 0.0), "a 1.0 sample rate should always sample in" ); assert!( - sampled_in(1.0, u64::MAX), - "a 1.0 sample rate should always sample in regardless of entropy" + sampled_in(1.0, 0.999_999), + "a 1.0 sample rate should sample in for the largest roll" + ); + assert!( + !sampled_in(0.0, 0.0), + "a 0.0 sample rate should never sample in, even on a zero roll" + ); + assert!( + !sampled_in(-1.0, 0.0), + "a negative rate should never sample in" + ); + } + + #[test] + fn sampled_in_keeps_exact_probability_for_tiny_rates() { + // The previous bucket-quantized sampler truncated rates below one + // in a million to a zero threshold, silently emitting nothing. + // Direct comparison keeps every positive rate proportional. + let rate = 0.000_000_1; + assert!( + sampled_in(rate, rate / 2.0), + "a roll below a tiny positive rate should sample in" + ); + assert!( + !sampled_in(rate, rate * 2.0), + "a roll above a tiny positive rate should sample out" ); assert!( - !sampled_in(0.0, 0), - "a 0.0 sample rate should never sample in" + !sampled_in(0.000_001_9, 0.000_001_95), + "no downward quantization: the boundary sits exactly at the rate" ); assert!( - !sampled_in(0.0, u64::MAX), - "a 0.0 sample rate should never sample in regardless of entropy" + sampled_in(0.000_001_9, 0.000_001_85), + "rolls just under the rate should sample in" ); } From f3ee47381858e2030ee3e040f773d565f39be0ca Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 027/104] Drop the origin span before error-path auction telemetry On origin failure with a dispatched auction, the origin span guard stayed alive through the emit_abandoned_auction await, so ts-origin and origin_ms absorbed Tinybird emission time. The span now closes when the send resolves, before either branch, with an error-path regression test. --- crates/trusted-server-core/src/publisher.rs | 55 ++++++++++++++++++++- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 53d09bfc6..4c7933795 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -4658,8 +4658,13 @@ pub async fn handle_publisher_request( platform_request = platform_request.with_cache_bypass(); } + // The span must end when the origin responds (or fails), before any + // abandonment-telemetry await below — otherwise `ts-origin` would + // absorb Tinybird emission time on the error path. let origin_span = timings.span(Phase::Origin); - let mut response = match services.http_client().send(platform_request).await { + let origin_send_result = services.http_client().send(platform_request).await; + drop(origin_span); + let mut response = match origin_send_result { Ok(platform_response) => platform_response.response, Err(err) => { if let Some(dispatched) = dispatched_auction.take() { @@ -4676,7 +4681,6 @@ pub async fn handle_publisher_request( })); } }; - drop(origin_span); log::debug!( "Publisher origin response received: status={}, header_count={}", @@ -7851,6 +7855,53 @@ mod tests { ); } + #[tokio::test] + async fn origin_span_is_recorded_when_the_origin_send_fails() { + // Regression guard for the review finding that the origin span + // guard stayed alive through the error branch (and its + // abandonment-telemetry await): the span must close when the send + // resolves, so a failed fetch still records `origin_ms` and the + // error branch's own work is excluded from it. + let settings = create_test_settings(); + // No queued response: the stub client fails the origin send. + let stub = Arc::new(StubHttpClient::new()); + let services = + build_services_with_http_client(stub as Arc); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let mut ec_context = EcContext::read_from_request(&settings, &request, &services) + .expect("should read EC context"); + let result = handle_publisher_request( + &settings, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[], + registry: None, + }, + request, + EdgeCacheHeader::SurrogateControl, + ) + .await; + + assert!(result.is_err(), "should surface the origin failure"); + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.is_some(), + "should record the Origin span even when the origin send fails" + ); + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! From 46b3c7f99ab6bf9335d20e17922f0ddeb5866690 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 028/104] Address remaining review feedback on timing surfaces and docs - Pin HEADER_PHASES against Phase::header_name() in the phase-index test, closing the second hand-synced list. - Add RouteClass::Asset to the snake_case rendering test; rename the lowercasing test to say what it does. - Give the Axum adapter a named, fully configured construction path (TrustedServerApp::dev_server_service) so server_timing_enabled is never silently discarded; the tuple API is private now. - Document that Server-Timing is client-visible when enabled, in the configuration guide's observability section. - Replace stale event_date references in the spec, plan, and dashboard guidance with the toDate(event_ts) sorting-key expression, and state the single-segment rejection rule in spec section 9. --- crates/trusted-server-adapter-axum/src/app.rs | 19 ++++++++++++++++++- .../trusted-server-adapter-axum/src/main.rs | 14 +++++--------- .../trusted-server-core/src/request_timing.rs | 13 +++++++++++++ docs/guide/configuration.md | 13 +++++++++++++ .../plans/2026-08-24-request-phase-timing.md | 5 +++-- .../2026-08-24-request-phase-timing-design.md | 16 ++++++++++------ 6 files changed, 62 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 4bdf27d01..aeda26a84 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -1,6 +1,7 @@ use core::future::Future; use std::sync::Arc; +use edgezero_adapter_axum::service::EdgeZeroAxumService; use edgezero_core::app::Hooks; use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; @@ -587,6 +588,22 @@ impl TrustedServerApp { Ok(build_router(&state)) } + /// The dev server's fully configured tower service: the application + /// router wrapped in the terminal timing layer + /// ([`crate::timing::TimingService`]), with `server_timing_enabled` + /// read from the same settings snapshot that built the router. + /// + /// This is the standard construction path for serving this adapter. + /// [`Hooks::routes`] satisfies the `Hooks` trait contract and returns + /// the bare router without the timing layer; callers who serve traffic + /// should use this instead so `server_timing_enabled` is never + /// silently discarded. + #[must_use] + pub fn dev_server_service() -> crate::timing::TimingService { + let (router, server_timing_enabled) = Self::routes_with_server_timing_flag(); + crate::timing::TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled) + } + /// Build the router alongside whether `Server-Timing` emission is /// enabled, read from the same settings snapshot used to build the /// router. @@ -596,7 +613,7 @@ impl TrustedServerApp { /// `Settings` per request, the Axum dev server builds its application /// state once and reuses the same [`RouterService`] for every request. #[must_use] - pub fn routes_with_server_timing_flag() -> (RouterService, bool) { + fn routes_with_server_timing_flag() -> (RouterService, bool) { let state = match build_state() { Ok(s) => s, Err(ref e) => { diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index b8bc28ae2..4e360ea41 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -3,7 +3,6 @@ use std::net::SocketAddr; use axum::Router; use edgezero_adapter_axum::dev_server::AxumDevServerConfig; use edgezero_adapter_axum::service::EdgeZeroAxumService; -use edgezero_core::router::RouterService; use tokio::net::TcpListener; use tokio::runtime::Builder as RuntimeBuilder; use tokio::signal; @@ -30,8 +29,8 @@ fn main() { }; log::info!("Listening on http://{}", config.addr); - let (router, server_timing_enabled) = TrustedServerApp::routes_with_server_timing_flag(); - if let Err(err) = run(router, server_timing_enabled, config) { + let service = TrustedServerApp::dev_server_service(); + if let Err(err) = run(service, config) { log::error!("trusted-server-adapter-axum failed: {err}"); std::process::exit(1); } @@ -56,22 +55,19 @@ fn main() { /// Returns an error if the Tokio runtime fails to start, the listener fails /// to bind, or the underlying serve loop errors. fn run( - router: RouterService, - server_timing_enabled: bool, + service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; - runtime.block_on(serve(router, server_timing_enabled, config)) + runtime.block_on(serve(service, config)) } async fn serve( - router: RouterService, - server_timing_enabled: bool, + service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { let listener = TcpListener::bind(config.addr).await?; - let service = TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled); let axum_router = Router::new().fallback_service(service_fn(move |req| { let mut svc = service.clone(); async move { svc.call(req).await } diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 8d1b92cc1..cbb269b1e 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -417,6 +417,19 @@ mod tests { seen.iter().all(|slot| *slot), "should cover every phases-array slot" ); + + // `HEADER_PHASES` is a second hand-synced list: a phase that gains + // a `header_name()` but is never added there silently renders + // nothing in the `Server-Timing` value. + let header_bearing: Vec = phases + .into_iter() + .filter(|phase| phase.header_name().is_some()) + .collect(); + assert_eq!( + header_bearing, + HEADER_PHASES.to_vec(), + "should render every header-bearing phase, in declaration order" + ); } #[test] diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 675aaf40a..729801ecb 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1846,6 +1846,19 @@ send access-telemetry rows. server_timing_enabled = true ``` +::: warning Client-visible latency disclosure +The `Server-Timing` header is sent to every client on eligible responses, +not only to operators: browsers expose the values to same-origin JavaScript +via `PerformanceResourceTiming.serverTiming`, and any caller can read the +raw header. Enabling it publishes measured per-phase server latency, +including KV read timing on the public identity endpoints (`ts-kv`) and +origin/cache behaviour on publisher pages (`ts-origin`, +`ts-template-cache`). This is standard `Server-Timing` practice and the +values are durations only, but treat the flag as a diagnostic aid to enable +deliberately, not a general always-on toggle, unless disclosing those +timings to all clients is acceptable for the deployment. +::: + **Environment Override**: ```bash diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md index 9ff3905f4..02cda64d8 100644 --- a/docs/superpowers/plans/2026-08-24-request-phase-timing.md +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -928,10 +928,11 @@ git commit -m "Emit confirmed access telemetry rows after pull-sync post-send" - [ ] **Step 1: Rewrite the schema** per spec section 9: keep `event_ts DateTime64(3)`, `method`, `status UInt16`, `time_elapsed_ms UInt32`, - `sample_rate Float64`, `event_date` + 30-day TTL; add the columns from spec 9 with + `sample_rate Float64` + 30-day TTL; add the columns from spec 9 with dimension columns non-nullable `LowCardinality(String)` and phase columns `Nullable(UInt32)`; drop `path` and `cache_state`; set - `ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status"`. + `ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status"` + (`event_date` was dropped for the sorting-key expression; see spec section 9). - [ ] **Step 2: Validate** with the tinybird toolchain if available locally (`tb check` / project tests under `tinybird/tests`); otherwise assert the file diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 214d146a1..2892386f4 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -276,7 +276,8 @@ Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the `mark_headers_ready()` snapshot; nullable because a contended lock drop can lose the -snapshot), `sample_rate`, `event_date`, 30-day TTL. +snapshot), `sample_rate`, 30-day TTL. (`event_date` was later dropped for the +`toDate(event_ts)` sorting-key expression; see section 9's schema note.) Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality @@ -290,8 +291,11 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. - Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` - when it fails the charset allowlist, exceeds 32 characters, or carries more than 7 - ASCII digits. The character allowlist alone does not bound identity (`[a-z0-9_-]` + when it fails the charset allowlist, exceeds 32 characters, carries more than 7 + ASCII digits, or is the only segment in the path. Depth is what makes a first + segment a section name: single-segment paths are documents (WordPress + `/%postname%/` puts every article at depth 1), so they reject wholesale, root + landing pages included. The character allowlist alone does not bound identity (`[a-z0-9_-]` is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated prefix of any of those is still identifying, so the length and digit bounds reject the segment outright. @@ -352,7 +356,7 @@ pop, status)`. Every column carries a `json:$.` path (the Events API rejec NDJSON into a datasource without JSONPaths, discovered live); `event_date` was dropped in favor of the sorting-key expression because a DEFAULT column cannot carry a JSONPath the producer never sends. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query -also carries an `event_date` predicate so the primary index prunes; rollout validates +also carries a `toDate(event_ts)` predicate so the primary index prunes; rollout validates the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the reserved datasource was ever deployed to the remote workspace; if it was, this schema @@ -409,8 +413,8 @@ ships as a versioned replacement datasource with a cutover, not an in-place edit ## 11. Dashboard and query model No endpoint pipe in v1. Grafana queries `access_logs_raw` directly through the -ClickHouse connector with `$__timeFilter(event_ts)` plus an `event_date` predicate, -matching the auction dashboards. +ClickHouse connector with `$__timeFilter(event_ts)` plus a `toDate(event_ts)` +predicate, matching the auction dashboards. Dashboard: a new standalone `grafana/dashboards/edge-performance.json` in the telemetry repo (`trusted-server-tinybird`), performance only, no panels shared with From c6235b91fcd247a57d27fa8703734340d85d0c22 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 31 Aug 2026 14:09:26 -0500 Subject: [PATCH 029/104] Enforce access telemetry body limit The access emitter carried the configured body limit without enforcing it, allowing oversized rows to bypass the intended transport safeguard. --- .../src/tinybird.rs | 42 +++++++++++++++++-- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 97a1ffd3e..84dfc37d8 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -319,14 +319,25 @@ fn build_access_events_request( /// /// # Errors /// -/// Returns `Err` when the access-log APPEND token cannot be loaded, the -/// backend cannot be registered, the request cannot be built or sent, or the -/// Tinybird Events API responds with a non-2xx status. +/// Returns `Err` when the row exceeds the configured request-body limit, the +/// access-log APPEND token cannot be loaded, the backend cannot be registered, +/// the request cannot be built or sent, or the Tinybird Events API responds +/// with a non-2xx status. pub(crate) async fn emit_access_event( client: &dyn PlatformHttpClient, target: &TinybirdEventsTarget, row: String, ) -> Result<(), Report> { + let body_len = row.len(); + if body_len > target.max_body_bytes { + return Err(Report::new(TrustedServerError::Proxy { + message: format!( + "Tinybird access telemetry request body has {body_len} bytes, exceeding {} byte limit", + target.max_body_bytes + ), + })); + } + let token = load_access_token(target)?; let auth_header = FastlyTinybirdAuctionTelemetrySink::authorization_header(&token)?; let backend_name = FastlyPlatformBackend @@ -884,6 +895,31 @@ mod tests { ); } + #[test] + fn access_emitter_rejects_oversized_row_before_sending() { + let mut config = enabled_config(); + config.max_body_bytes = 1024; + let target = TinybirdEventsTarget::from_access_config(config); + let http_client = RecordingHttpClient::respond_with(202); + let row = "x".repeat(1025); + + let result = futures::executor::block_on(emit_access_event(&http_client, &target, row)); + + let error = result.expect_err("should reject a row above the configured body limit"); + assert!( + error.to_string().contains("1024"), + "error should name the configured body limit: {error}" + ); + assert!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .is_empty(), + "should not send an oversized access row" + ); + } + #[test] fn access_emitter_posts_ndjson_and_validates_2xx() { // `ts_secrets`/`tinybird_access_append_token` is seeded in From fde55b1d828037ab5f437aa70df83ca663e607fe Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 1 Sep 2026 14:45:53 +0530 Subject: [PATCH 030/104] docs: design mobile ad-render trace endpoint --- ...-mobile-ad-render-trace-endpoint-design.md | 733 ++++++++++++++++++ 1 file changed, 733 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md new file mode 100644 index 000000000..e46f9892b --- /dev/null +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -0,0 +1,733 @@ +# Mobile Ad-Rendering Trace Endpoint Design + +**Status:** Proposed + +**Issue:** [#1050 — Create debug endpoint for mobile user to trace ad rendering](https://github.com/IABTechLab/trusted-server/issues/1050) + +**Related work:** + +- [#1081 — Improvements to TS_CONSOLE for ad observability](https://github.com/IABTechLab/trusted-server/issues/1081) +- [#1074 — Request-phase timing](https://github.com/IABTechLab/trusted-server/pull/1074) +- [#1076 — Auction timing milestones](https://github.com/IABTechLab/trusted-server/pull/1076) +- [#974 — GPT runtime diagnostics](https://github.com/IABTechLab/trusted-server/pull/974) +- [#997 — Trusted Server delivery attribution](https://github.com/IABTechLab/trusted-server/pull/997) + +## 1. Summary + +Add a deployment-controlled, public, privacy-safe `GET /_ts/trace` page for a +mobile end user who needs to reproduce an ad-rendering problem and give support +an exportable diagnostic report. + +The endpoint is both a setup page and a report viewer. On the first visit it +enables the existing GPT diagnostics browser session and explains how to +reproduce the problem. The user then returns to the real publisher page and +reloads it. Trusted Server supplies redacted request context, while the existing +TS Console records GPT, auction, and render evidence. A `View trace results` +action creates one bounded, allowlisted snapshot in same-tab `sessionStorage` +and navigates to `/_ts/trace`. The endpoint reads that snapshot, presents a +mobile-first HTML report, and offers JSON export, copy, and progressive Web +Share actions. + +The design introduces no report database, server-side trace store, report ID, +target URL parameter, telemetry query, or publisher-origin change. It cannot +recover an event that happened before tracing was enabled; the user must +reproduce the problem. + +## 2. Problem and product interpretation + +Issue #1050 names three required data groups: + +1. Network information inspired by `fastly-debug.com`. +2. Auction information per ad slot, similar to the TS tracer and auction + telemetry. +3. End-user cookie information. + +The title additionally establishes two product constraints: the experience is +for a mobile user, and it is reached through an endpoint. A mobile user should +not need browser developer tools, Basic Authentication, a copied trace ID, or a +second copy of the affected page URL. + +A standalone request cannot know what occurred in a previous document. Exact +render evidence exists only while the publisher page is running in the browser. +The design therefore separates two responsibilities without separating the user +experience: + +- `/_ts/trace` owns setup, consolidated presentation, and export. +- TS Console owns observation of the real publisher page. + +The browser-local handoff joins them without introducing a backend report +service. + +## 3. Goals + +- Give a non-technical mobile user one memorable URL: + `https:///_ts/trace`. +- Capture evidence from a real publisher-page reproduction, not a synthetic + auction. +- Display a Fastly-inspired network summary for the traced publisher request. +- Report health for an explicit allowlist of Trusted Server cookies without + exposing their values. +- Present the versioned, allowlisted TS Console evidence for every retained GPT + slot and request cycle. +- Support a full report in a narrow mobile viewport without developer tools. +- Export the same allowlisted model as formatted JSON. +- Keep capture bounded, same-tab, temporary, and inactive by default. +- Preserve normal auction, GPT, rendering, origin, and caching behavior whenever + diagnostics is inactive. +- Keep core behavior platform-neutral while allowing Fastly to provide richer + optional request fields. + +## 4. Non-goals + +- Recovering a failure that occurred before tracing was enabled. +- Permanent history, cross-device retrieval, server upload, or support-ticket + integration. +- A database, distributed trace store, report token, or telemetry lookup. +- A target URL such as `/_ts/trace?target=/article`. +- Replaying an auction or treating a synthetic auction as evidence about a + publisher page. +- Exact parity with every field or active measurement on `fastly-debug.com`. +- Reading the browser's complete cookie jar, third-party cookies, cookie + attributes, or cookies withheld from the request. +- Exposing raw cookie values, EC IDs, EIDs, consent strings, unmasked IP + addresses, internal auction request IDs, creative markup, targeting maps, + cache URLs, or stack traces. +- Reimplementing TS Console auction and creative observability requested by + #1081. +- Querying Tinybird to build an interactive report. +- Adding exact provider-by-slot no-bid explanations before the auction model can + observe those dispositions. +- Direct `POST /auction` browser diagnostics in the first release. + +## 5. Decisions + +### 5.1 Public, redacted endpoint + +`/_ts/trace` is public when explicitly enabled by deployment configuration. It +is not placed under `/_ts/admin`, because the intended user is a layperson on a +phone and the existing Basic Authentication flow is unsuitable for that +journey. + +Public access is safe only because both the page and export use a strict +allowlist. The activation cookie is a feature toggle, not authentication. No +field becomes eligible merely because tracing is active. + +### 5.2 Reuse the existing diagnostics session + +The endpoint reuses `__Host-ts-console` and the existing GPT diagnostics +activation semantics rather than creating a second `ts-trace` session. The +cookie remains host-only, `Secure`, `HttpOnly`, `SameSite=Lax`, and +browser-session scoped. + +`/_ts/trace?enabled=false` clears the activation cookie and browser snapshot. +Other values, duplicate `enabled` parameters, and malformed directives fail +closed and do not mutate session state. + +### 5.3 Browser-local, explicit handoff + +TS Console remains memory-only during observation. It writes a report to +`sessionStorage` only after the user selects `View trace results`. The action: + +1. Builds the same versioned allowlisted snapshot used by export. +2. Adds the redacted request-context envelope. +3. Serializes and validates the size. +4. Stores it under one versioned key in the current tab. +5. Navigates the same tab to `/_ts/trace`. + +Continuous persistence is prohibited. Opening the endpoint in another tab does +not retrieve the snapshot. Closing the tab deletes it according to browser +session-storage semantics. + +### 5.4 Forward reproduction, not historical diagnosis + +The first endpoint visit enables tracing for subsequent eligible document +navigations. The setup page must say plainly that the user needs to return to +the affected page, reload it, and reproduce the problem. + +If the user replaced the affected URL in the address bar with `/_ts/trace`, the +page offers a `Return to previous page` action backed by browser history and +then instructs the user to reload once. The design does not claim that +back-forward-cache restoration caused a new server request. + +Support should preferably give the user the trace URL before reproduction. The +product does not attempt to discover the previous URL through `Referer`, because +address-bar navigation commonly omits it and relying on it would create +inconsistent behavior. + +### 5.5 Separate issue ownership + +#1050 defines the report shell, request context, mobile flow, browser-local +handoff, and export. #1081 remains the owner of creative numbering, auction +classification, bidder/price policy, terminology, and normalized auction/render +timing. + +The trace report consumes TS Console's public versioned export contract. It +does not read TS Console internals or create an alternate slot correlation +engine. + +## 6. User experience + +### 6.1 First visit: no captured report + +`GET /_ts/trace` returns a mobile-first HTML page with: + +- Title: `Trusted Server ad diagnostics`. +- State: `Trace ready` after the response establishes the session cookie. +- A short explanation that no previous ad failure can be recovered. +- Network and cookie health for the setup request, labeled `Setup request`. +- Primary action: `Return to previous page` when browser history permits. +- Secondary instructions: return to the affected page, reload once, reproduce + the problem, then select `View trace results`. +- Action to disable tracing. + +The page must not imply that setup-request network facts or an empty auction +section describe the affected page. + +### 6.2 Active publisher page + +The existing TS Console remains available. On mobile it gains a prominent +`View trace results` action. Selecting it never changes ad behavior; it only +snapshots retained observations and navigates after serialization succeeds. + +If the snapshot cannot be stored, the page remains in place, announces the +failure, and keeps the existing direct JSON export available. + +### 6.3 Report visit + +When a valid snapshot exists, `/_ts/trace` renders: + +1. Report summary and capture time. +2. Network and request section for the traced publisher document. +3. Trusted Server cookie-health section. +4. Auction and rendering section grouped by numbered slot. +5. Coverage and ambiguity section. +6. Export actions. +7. `Clear report and end tracing` action. + +The setup request's facts are not merged into or substituted for missing traced +page facts. Missing fields display `Unavailable`; missing evidence displays +`Not observed` or `Unknown`, following TS Console terminology. + +### 6.4 Mobile and accessibility requirements + +- Support viewport widths down to 320 CSS pixels without horizontal page + scrolling. +- Use a full-document report rather than the current floating 460-pixel panel. +- Use at least 44-by-44 CSS pixel primary touch targets. +- Keep export/end-trace actions reachable without covering report content. +- Use semantic headings, lists, tables only where they remain readable on a + narrow viewport, visible focus styles, and an `aria-live` status region. +- Do not rely on hover, color alone, badges alone, or precise pointer input. +- Preserve browser zoom and safe-area insets. +- Prefer native text and controls over a framework or new UI dependency. + +## 7. Architecture + +```text +First GET /_ts/trace + | + |-- core route builds setup request context + |-- response sets __Host-ts-console + |-- HTML explains forward reproduction + v +Real publisher document reload + | + |-- adapter supplies optional network facts + |-- core computes allowlisted cookie health + |-- core injects redacted TraceRequestContextV1 + |-- existing TS Console observes GPT and TS delivery + v +User selects "View trace results" + | + |-- JS builds bounded TraceReportV1 + |-- same-tab sessionStorage write + |-- location.assign('/_ts/trace') + v +Second GET /_ts/trace + | + |-- static report shell reads and validates TraceReportV1 + |-- mobile HTML renders sections + |-- local JSON/copy/share actions +``` + +### 7.1 Core responsibilities + +- Define configuration and route behavior. +- Register the route before publisher fallback on every supported adapter. +- Define the platform-neutral request-context and report-envelope schemas. +- Build cookie-health facts through read-only parsing. +- Convert `ClientInfo` and available geo data into the public network allowlist. +- Inject request context only into an active private diagnostics document. +- Apply response privacy and security headers. +- Ensure trace requests never reach the publisher origin. + +### 7.2 Adapter responsibilities + +- Register the named route with exact method handling. +- Populate optional `ClientInfo` fields available on the platform. +- Fastly may supply POP, HTTP version, TLS, JA4, H2 fingerprint, and edge + server data when the SDK exposes them. +- Other adapters return the same schema with unsupported fields absent. +- Adapter-specific errors omit optional facts rather than failing publisher + delivery. + +### 7.3 JavaScript responsibilities + +- Accept the immutable redacted request context at initialization. +- Preserve the existing bounded TS Console observation store. +- Build and validate `TraceReportV1` on explicit user action. +- Store only one report in same-tab `sessionStorage`. +- Render the report shell from the validated model. +- Implement download, copy, progressive Web Share, clearing, expiry, and + accessible status reporting. +- Never upload diagnostic data or issue a telemetry query. + +## 8. Route and configuration contract + +Add an explicit default-off option to the existing integration: + +```toml +[integrations.gpt_diagnostics] +enabled = true +trace_page_enabled = false +``` + +Rules: + +- `trace_page_enabled = true` requires `enabled = true`; invalid combinations + fail configuration validation. +- `GET /_ts/trace` returns the setup/report HTML and establishes the session. +- `GET /_ts/trace?enabled=false` returns the shell, clears the cookie, and asks + the client to clear the stored snapshot. +- `HEAD /_ts/trace` returns the same status and headers without a body but does + not mutate the cookie. +- All other methods return a local `405 Method Not Allowed` with `Allow: GET, +HEAD`. +- Disabled deployments return a local `404` for the exact route and never fall + through to the publisher origin. +- Extra path segments, encoded separators, duplicate parameters, and lookalike + paths do not match. +- The route never creates or refreshes an EC, ingests EIDs, runs an auction, + fetches the publisher origin, or emits auction telemetry. + +The current `?ts_console=1` and `?ts_console=0` activation flow remains +supported for technical users. Both activation surfaces drive the same cookie +and runtime; they must not create two concurrent diagnostic modes. + +## 9. Data contracts + +### 9.1 Request context + +The server injects one immutable `TraceRequestContextV1` into active diagnostic +documents: + +```text +TraceRequestContextV1 + schema_version: 1 + captured_at: RFC 3339 UTC timestamp + page: + origin: publisher origin + path: normalized path + network: + masked_client_ip?: string + country?: string + region?: string + asn?: u32 + http_version?: string + tls_protocol?: string + tls_cipher?: string + tls_ja4?: string + h2_fingerprint?: string + edge_hostname?: string + edge_region?: string + edge_pop?: string + cookies: + ts_ec: CookieHealth + ts_eids: CookieHealth + ts_tester: CookieHealth + diagnostics_session: CookieHealth +``` + +The page field omits query and fragment data. It does not contain origin-facing +URLs, referrers, or arbitrary headers. + +`masked_client_ip` uses a deterministic display-only mask for the current +request: IPv4 keeps at most the first 24 bits and IPv6 keeps at most the first +48 bits. The full address never enters HTML, JavaScript, browser storage, or +export. + +JA4 and H2 fingerprints are optional probabilistic identifiers. They are +included only when the deployment has separately enabled the existing +fingerprint diagnostic capability. Their absence is not an error. + +### 9.2 Cookie health + +```text +CookieHealth + state: + absent | present_valid | present_invalid | duplicate | unavailable + source: request + detail?: allowlisted enum +``` + +Allowed details describe shape, not value, for example `valid_ec_format`, +`malformed`, `oversized`, or `activation_pending_response`. + +The parser must inspect the incoming request before any diagnostics-cookie +sanitization, while preserving existing authoritative-cookie and consent +semantics. Inspection is read-only: it must not generate an EC, touch the +identity graph, sync partner IDs, or extend any cookie lifetime. + +Only Trusted Server-owned cookie names are reported. Arbitrary cookie names and +values are excluded. The endpoint cannot claim knowledge of browser attributes, +expiry, or cookies the browser withheld from the request. + +### 9.3 Report envelope + +```text +TraceReportV1 + schema_version: 1 + captured_at: RFC 3339 UTC timestamp + request_context: TraceRequestContextV1 + gpt_diagnostics: GptDiagnosticsExportV1-or-successor +``` + +The trace envelope owns request context and transport. TS Console continues to +own its nested schema. Compatibility is explicit: the viewer supports a small +documented set of TS Console schema versions and rejects unknown versions with +an actionable message rather than guessing. + +### 9.4 Storage limits and expiry + +- Storage key: a namespaced, versioned constant owned by the diagnostics + module. +- Maximum encoded report size: 512 KiB. +- Maximum report age: 15 minutes from `captured_at`. +- One report per tab; a new explicit snapshot replaces the old report. +- Invalid, oversized, expired, or unsupported reports are removed immediately. +- `Clear report and end tracing` removes the storage entry and clears the + activation cookie. + +These are product limits, not assumptions about browser quota. A storage write +failure is handled even when the report is below the application limit. + +## 10. Auction and rendering evidence + +The report uses TS Console's evidence model. It must preserve the distinction +between: + +- A Trusted Server opportunity. +- A provider response. +- A selected Trusted Server candidate. +- A GPT request and response. +- A non-empty GPT render. +- Trusted Server creative-bridge evidence. +- Creative load and viewability. +- A publisher or client-side refresh. + +The viewer must not infer that Trusted Server rendered an ad merely because GPT +reported a filled slot. Ambiguous and unattributed cycles remain explicit. + +Current diagnostics tokens exist only on delivered winning bids. No-bid, +failed, skipped, hidden, unresolved, and direct `/auction` paths can lack server +correlation. The report displays the available observed facts and `Unknown` +rather than manufacturing a correlation. + +Provider-call telemetry is auction-wide, while bid rows exist only for returned +bids. Exact `provider X was asked for slot Y` and exact no-bid causality require +a future provider-impression disposition model. That instrumentation is not +silently assumed by this design. + +Timing fields introduced by #1074/#1076 are consumed only after they merge and +are propagated through the live diagnostics contract. The report never queries +Tinybird, and it does not combine browser `performance.now()` values with +server-relative timing as though they were one clock. + +Bidder and winning price are included only if #1081 approves them in the public +TS Console export contract. #1050 does not independently weaken the existing +privacy policy. + +## 11. Network scope + +The report is inspired by Fastly Debug, not a clone. Version one uses facts +already present or reasonably addable to the platform request abstraction. + +Supported categories: + +- Masked client address. +- Country, region, and ASN when available. +- HTTP version. +- TLS protocol and cipher. +- Optional JA4 and H2 fingerprints. +- Edge hostname, region, and POP. +- Capture time. + +Explicitly excluded: + +- DNS resolver address and resolver ASN. +- Active bandwidth or speed tests. +- TCP congestion window, next hop, RTT, and retransmit counters. +- DDoS/internal Fastly classifications. +- Arbitrary request headers. +- Full client IP in HTML or export. + +Unsupported optional fields are omitted rather than populated with fabricated +fallbacks. + +## 12. Security and privacy + +### 12.1 Allowlist boundary + +The report serializer constructs a new public model field by field. It never +serializes request structs, cookie parsers, auction requests, telemetry rows, or +browser objects wholesale. + +Forbidden data includes: + +- Raw `Cookie` and `Set-Cookie` headers. +- EC IDs, EIDs, bidder user IDs, and consent strings. +- Unmasked client IP. +- Query strings and fragments. +- Fastly or internal request identifiers that can join to user-bearing logs. +- Internal `AuctionRequest.id`. +- Bid requests/responses, losing-bid payloads, targeting, creative markup, + cache URLs, and stack traces. + +### 12.2 Same-origin script visibility + +Publisher and third-party scripts running on the publisher origin can access +`sessionStorage`. Therefore the stored model must be safe even if read by any +same-origin script. A random storage key, closed shadow root, or public endpoint +does not change this requirement. + +### 12.3 Response hardening + +Both the endpoint and every active diagnostic publisher response are terminally +`private, no-store`. The endpoint also sends: + +- `Content-Type: text/html; charset=utf-8` +- `X-Content-Type-Options: nosniff` +- `Referrer-Policy: no-referrer` +- `Content-Security-Policy` restricting content to the endpoint's own static + assets and prohibiting framing +- A restrictive `Permissions-Policy` + +The endpoint makes no third-party requests. Dynamic JSON embedded in HTML uses +the repository's script-safe serializer and is never concatenated into +executable JavaScript. + +### 12.4 Shared templates and ESI + +Per-request trace context must never enter a shared template or ESI fragment. +The existing diagnostics private/no-store decision remains a load-bearing gate. +Tests must prove that late response-header handlers cannot make traced content +publicly cacheable. + +## 13. Failure handling + +- Disabled route: local privacy-safe `404`. +- Unsupported method: local `405`; never publisher fallback. +- Optional platform fact unavailable: omit the field and continue. +- Cookie parser failure: report `present_invalid` without the value. +- Diagnostics context serialization failure: omit the context, log a bounded + server error, and preserve publisher delivery. +- TS Console capture failure: fail open for advertising and show incomplete + coverage in diagnostics. +- Storage unavailable, quota exceeded, or serialization oversized: remain on + the publisher page, announce the error, and offer direct download. +- Missing snapshot on endpoint: show setup state, not an empty successful + report. +- Expired, malformed, or unknown report schema: clear it and explain that the + user must reproduce again. +- Clipboard or Web Share unavailable: keep JSON download available. +- Export failure: retain the on-screen report and show an accessible error. + +Diagnostic failures must never suppress, delay, add, remove, or reorder GPT +requests, auctions, targeting, or creative rendering. + +## 14. Testing strategy + +### 14.1 Core unit tests + +- Configuration defaults off and rejects trace-page enablement without GPT + diagnostics. +- Exact route, query, method, encoded-path, and fallback behavior. +- Session cookie set/clear attributes and duplicate-directive fail-closed + behavior. +- Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, + and origin fetch. +- Cookie-health parser covers absent, valid, malformed, duplicate, non-UTF-8, + and oversized inputs without retaining values. +- Request-context serializer masks IPv4/IPv6 and omits query, raw headers, IDs, + and unsupported fields. +- Active responses remain terminally private/no-store under hostile late header + overrides. +- Dynamic HTML/JSON values cannot close elements or create executable script. + +### 14.2 Adapter parity tests + +- Fastly route registration and optional field mapping. +- Axum, Cloudflare, and Spin return the common route/schema with unavailable + fields omitted. +- Named route failures never fall through to publisher origin. +- HEAD and unsupported methods behave identically across adapters. +- Fastly fingerprint fields respect the existing fingerprint-debug gate. + +### 14.3 JavaScript unit tests + +- Explicit snapshot only; no continuous `sessionStorage` writes. +- Size limit, schema validation, expiry, replacement, clearing, and storage + exceptions. +- Same-tab navigation occurs only after a successful write. +- Viewer handles absent optional network facts and every cookie-health state. +- Forbidden fields never enter storage or export fixtures. +- Download filename and MIME type are deterministic. +- Copy and Web Share success, rejection, absence, and fallback behavior. +- 320-pixel layout, keyboard navigation, focus handling, and accessible status + announcements. + +### 14.4 Browser integration tests + +- First endpoint visit sets the session and shows setup state. +- A real fixture reload activates diagnostics and captures multiple slots. +- `View trace results` navigates in the same tab and renders the captured + request context and slot evidence. +- Empty, filled, ambiguous, no-candidate, and unattributed slot states remain + distinct. +- Reloading the trace page retains an unexpired same-tab report. +- A new tab cannot access the original tab's report. +- Disabling clears both cookie and report. +- Back-forward-cache restoration is not described as a fresh traced request; + the setup page tells the user to reload. +- Export JSON matches the displayed versioned model. +- Inactive publisher traffic has no trace assets, storage access, listeners, or + cache-policy change. + +### 14.5 Manual acceptance + +Verify on current iOS Safari and Android Chrome using a representative publisher +fixture: + +- A layperson can follow the page instructions without developer tools. +- Touch targets, scrolling, zoom, safe areas, download, copy, and native share + behavior are usable. +- The user can distinguish setup information from captured-page information. +- A failed share or download does not lose the visible report. + +## 15. Rollout and observability + +- Ship default-off. +- Enable first in a controlled staging publisher configuration. +- Validate response cache headers and CDN behavior before production use. +- Validate with redacted fixtures before real publisher traffic. +- Log only route outcome, schema version, report-present boolean, and bounded + error category. Never log report contents or cookie/network values. +- Roll back by disabling `trace_page_enabled`; existing `?ts_console=1` + diagnostics remain independently configurable. + +## 16. Acceptance criteria + +1. With the feature disabled, exact trace routes return local `404` and ordinary + traffic is unchanged. +2. A mobile user can enable tracing by opening only `/_ts/trace`; no target URL, + credentials, or trace ID is required. +3. The setup page accurately explains that the problem must be reproduced after + activation. +4. A subsequent real publisher-page reload captures redacted request context + and existing TS Console evidence without altering ad behavior. +5. `View trace results` transfers one bounded snapshot in the same tab and opens + the report page without server-side storage. +6. The report separates network, cookie health, auction/render evidence, and + coverage/unknowns. +7. JSON export contains the same versioned allowlisted information shown on the + page. +8. No raw cookies, user IDs, full IPs, consent strings, query strings, internal + auction IDs, targeting, or creative payloads appear in HTML, browser storage, + logs, or export. +9. Trace HTML and active publisher pages remain terminally private/no-store. +10. Missing platform fields, incomplete auction correlation, storage failure, + and unavailable share APIs degrade honestly without affecting advertising. +11. The full report is usable at 320 CSS pixels and with keyboard/screen-reader + navigation. +12. Auction fields owned by #1081 are consumed through its versioned public + contract rather than duplicated in #1050. + +## 17. Implementation sequencing + +This design is one product flow but should be implemented in dependency order: + +1. Core request-context schema, cookie-health classification, configuration, + and endpoint shell. +2. Adapter route parity and Fastly optional network enrichment. +3. TS Console request-context envelope and explicit same-tab snapshot handoff. +4. Mobile viewer, export/copy/share, expiry, and clearing. +5. Integration with the current TS Console schema. +6. Additive adoption of #1081 and #1074/#1076 fields after their contracts + merge. +7. Browser, privacy, cache, and real-device acceptance. + +The implementation plan must not claim completion of #1081 or the open timing +PRs as part of #1050. If those dependencies are unavailable, the report ships +only with current observed auction/render evidence and labels unavailable fields +honestly. + +## 18. Rejected alternatives + +### `/_ts/admin/trace?target=/article` + +Rejected because it requires the user to supply the affected URL twice, adds +target validation and open-redirect risk, and is unsuitable for a layperson. + +### Basic Authentication + +Rejected for the mobile end-user workflow. Authentication also would not make +it safe to inject raw secrets into a publisher page containing third-party +JavaScript. + +### Server-managed trace sessions + +Rejected because they require shared storage, report authorization, expiry, +deletion, and operational infrastructure beyond the issue's needs. + +### Synthetic auction on the endpoint + +Rejected because it does not reproduce the real page's DOM, GPT lifecycle, +consent context, refresh path, or auction timing and could produce misleading +results. + +### Endpoint-only report with no publisher-page integration + +Rejected because a request to `/_ts/trace` cannot observe rendering that +occurred in another document. + +### Query-only in-page console + +The existing `?ts_console=1` flow remains supported, but it is not the complete +answer to #1050: the issue asks for a memorable mobile endpoint and a +Fastly-style consolidated HTML report. The endpoint/viewer builds on rather +than replaces the console. + +### Cross-page URL payload + +Rejected because fragments or query strings containing the report create URL +length, history, logging, referrer, and accidental-sharing risks. + +## 19. Known limitations + +- The user must reproduce the problem after enabling tracing. +- Same-tab storage prevents cross-device and cross-tab sharing; JSON export is + the handoff artifact. +- Publisher-origin scripts can read the stored public-safe report. +- Browser privacy settings may disable storage, clipboard, download, or share + capabilities. +- Current server/browser correlation does not cover every no-bid, skipped, + failed, hidden, unresolved, or direct-auction path. +- Fastly-only transport details do not exist on every adapter. +- The current 30-pixel TS Console controls are not sufficient for this mobile + report; the endpoint uses independent 44-pixel touch targets. +- `fastly-debug.com` fields that require resolver, TCP, or active speed probes + remain out of scope. + +These limitations are displayed in operator documentation and, where relevant, +in the report itself. They are not hidden behind apparently successful empty +states. From 8899dcadc89d1c4db3c5902078fa3a9e2cbf0c42 Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 1 Sep 2026 15:33:05 +0530 Subject: [PATCH 031/104] docs: harden mobile ad trace design --- ...-mobile-ad-render-trace-endpoint-design.md | 754 ++++++++++++++---- 1 file changed, 591 insertions(+), 163 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index e46f9892b..6ed9f8c48 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -16,16 +16,17 @@ Add a deployment-controlled, public, privacy-safe `GET /_ts/trace` page for a mobile end user who needs to reproduce an ad-rendering problem and give support -an exportable diagnostic report. +an exportable diagnostic report. Visiting the page is read-only. The user +intentionally enables or ends tracing with a same-origin POST action. The endpoint is both a setup page and a report viewer. On the first visit it -enables the existing GPT diagnostics browser session and explains how to -reproduce the problem. The user then returns to the real publisher page and -reloads it. Trusted Server supplies redacted request context, while the existing -TS Console records GPT, auction, and render evidence. A `View trace results` -action creates one bounded, allowlisted snapshot in same-tab `sessionStorage` -and navigates to `/_ts/trace`. The endpoint reads that snapshot, presents a -mobile-first HTML report, and offers JSON export, copy, and progressive Web +offers a large `Enable tracing` action and explains how to reproduce the +problem. The user then returns to the real publisher page and reloads it. +Trusted Server supplies redacted request context, while the existing TS Console +records GPT, auction, and render evidence. A `View trace results` action creates +one bounded, allowlisted snapshot in same-tab `sessionStorage` and navigates to +`/_ts/trace`. The endpoint reads that untrusted snapshot, validates it, presents +a mobile-first HTML report, and offers JSON export, copy, and progressive Web Share actions. The design introduces no report database, server-side trace store, report ID, @@ -68,10 +69,12 @@ service. - Report health for an explicit allowlist of Trusted Server cookies without exposing their values. - Present the versioned, allowlisted TS Console evidence for every retained GPT - slot and request cycle. + slot and request cycle that fits the public report bounds, with explicit + omission counts when deterministic size truncation is required. - Support a full report in a narrow mobile viewport without developer tools. - Export the same allowlisted model as formatted JSON. -- Keep capture bounded, same-tab, temporary, and inactive by default. +- Keep the supported capture journey bounded, same-tab, temporary, and inactive + by default without treating browser storage as a security boundary. - Preserve normal auction, GPT, rendering, origin, and caching behavior whenever diagnostics is inactive. - Keep core behavior platform-neutral while allowing Fastly to provide richer @@ -101,7 +104,7 @@ service. ## 5. Decisions -### 5.1 Public, redacted endpoint +### 5.1 Public, redacted endpoint with intentional activation `/_ts/trace` is public when explicitly enabled by deployment configuration. It is not placed under `/_ts/admin`, because the intended user is a layperson on a @@ -112,16 +115,27 @@ Public access is safe only because both the page and export use a strict allowlist. The activation cookie is a feature toggle, not authentication. No field becomes eligible merely because tracing is active. +`GET /_ts/trace` is read-only and never activates or ends tracing. Activation +and deactivation use an in-page same-origin `fetch` POST accepted only when its +fixed custom action header, `Origin`, and Fetch Metadata identify the publisher +origin. Requests with a conflicting or missing signal fail closed. The POST +updates the existing page rather than adding a history entry, so browser Back +can still reach the article. This prevents an unrelated site from silently +toggling diagnostics through a top-level GET while preserving a one-URL, +one-tap mobile workflow. This control does not defend against code already +executing on the publisher origin. + ### 5.2 Reuse the existing diagnostics session The endpoint reuses `__Host-ts-console` and the existing GPT diagnostics activation semantics rather than creating a second `ts-trace` session. The -cookie remains host-only, `Secure`, `HttpOnly`, `SameSite=Lax`, and -browser-session scoped. - -`/_ts/trace?enabled=false` clears the activation cookie and browser snapshot. -Other values, duplicate `enabled` parameters, and malformed directives fail -closed and do not mutate session state. +cookie remains host-only, `Secure`, `HttpOnly`, and `SameSite=Lax`. +`POST /_ts/trace/enable` sets it with a fixed 30-minute `Max-Age` and does not +refresh that lifetime on publisher requests; `POST /_ts/trace/end` clears it. +Neither action accepts state-changing query parameters. The shorter endpoint +lifetime bounds accidental private/no-store operation if a user forgets to end +tracing; the existing technical query flow keeps its existing session-cookie +semantics. ### 5.3 Browser-local, explicit handoff @@ -134,20 +148,26 @@ TS Console remains memory-only during observation. It writes a report to 4. Stores it under one versioned key in the current tab. 5. Navigates the same tab to `/_ts/trace`. -Continuous persistence is prohibited. Opening the endpoint in another tab does -not retrieve the snapshot. Closing the tab deletes it according to browser -session-storage semantics. +Continuous persistence is prohibited. Same-tab navigation is the supported +handoff, not an isolation guarantee: a browser may copy session storage into an +opener-created tab or preserve it during session restore. Same-origin scripts +and service workers can read, replace, or forge the snapshot. The viewer +therefore treats it as untrusted, applies an application-level expiry, and +labels it browser-observed rather than authoritative. ### 5.4 Forward reproduction, not historical diagnosis -The first endpoint visit enables tracing for subsequent eligible document +The user's explicit activation enables tracing for subsequent eligible document navigations. The setup page must say plainly that the user needs to return to the affected page, reload it, and reproduce the problem. If the user replaced the affected URL in the address bar with `/_ts/trace`, the page offers a `Return to previous page` action backed by browser history and -then instructs the user to reload once. The design does not claim that -back-forward-cache restoration caused a new server request. +then instructs the user to reload once. History is only a convenience: it may +lead to a messaging app, search page, or unrelated site. The page includes a +fallback instruction to reopen the affected article on the same hostname and +in the same tab. The design does not claim that back-forward-cache restoration +caused a new server request. Support should preferably give the user the trace URL before reproduction. The product does not attempt to discover the previous URL through `Referer`, because @@ -161,9 +181,11 @@ handoff, and export. #1081 remains the owner of creative numbering, auction classification, bidder/price policy, terminology, and normalized auction/render timing. -The trace report consumes TS Console's public versioned export contract. It +Version one consumes `GptDiagnosticsExportV1` through TS Console's public export +contract and projects it into a distinct redacted `TraceGptDiagnosticsV1`. It does not read TS Console internals or create an alternate slot correlation -engine. +engine. #1081 and #1074/#1076 are additive follow-up work and are not release +gates for this version. ## 6. User experience @@ -172,13 +194,22 @@ engine. `GET /_ts/trace` returns a mobile-first HTML page with: - Title: `Trusted Server ad diagnostics`. -- State: `Trace ready` after the response establishes the session cookie. +- State derived from the setup request: `Tracing is off` unless the server + observed a valid existing diagnostics cookie, including one activated through + the technical query flow. - A short explanation that no previous ad failure can be recovered. - Network and cookie health for the setup request, labeled `Setup request`. -- Primary action: `Return to previous page` when browser history permits. +- Primary action: `Enable tracing`, implemented as an in-page same-origin fetch + POST that does not add a history entry. +- After activation and a successful state-verification request, state: `Tracing +is on — cookie observed by server`. +- After activation, primary action: `Return to previous page` when browser + history permits. - Secondary instructions: return to the affected page, reload once, reproduce the problem, then select `View trace results`. -- Action to disable tracing. +- A recovery instruction to reopen the affected article on the exact same + hostname and in the same tab if browser history is not useful. +- Action to end tracing when it is active. The page must not imply that setup-request network facts or an empty auction section describe the affected page. @@ -189,8 +220,12 @@ The existing TS Console remains available. On mobile it gains a prominent `View trace results` action. Selecting it never changes ad behavior; it only snapshots retained observations and navigates after serialization succeeds. -If the snapshot cannot be stored, the page remains in place, announces the -failure, and keeps the existing direct JSON export available. +If a valid bounded snapshot is built but browser storage rejects it, the page +remains in place, announces the storage failure, and offers a direct download +of that same combined `TraceReportV1` envelope. If projection, validation, or +size bounding fails before a valid report exists, the page reports capture +failure and does not mislabel the existing GPT-only export as an equivalent +fallback. ### 6.3 Report visit @@ -208,6 +243,16 @@ The setup request's facts are not merged into or substituted for missing traced page facts. Missing fields display `Unavailable`; missing evidence displays `Not observed` or `Unknown`, following TS Console terminology. +The report begins with `Browser-observed, unverified diagnostic data`. It does +not claim that the snapshot is authentic or suitable as forensic or security +evidence. + +`Copy` copies formatted JSON. `Share` supplies the same JSON file to the native +Web Share sheet only after an explicit tap and tells the user that the selected +app will receive it. If file sharing is unsupported or rejected, the viewer +keeps Copy and Download available; it does not silently share a URL or upload +the report. + ### 6.4 Mobile and accessibility requirements - Support viewport widths down to 320 CSS pixels without horizontal page @@ -224,11 +269,17 @@ page facts. Missing fields display `Unavailable`; missing evidence displays ## 7. Architecture ```text -First GET /_ts/trace +GET /_ts/trace | - |-- core route builds setup request context + |-- early reserved-route classifier terminates locally + |-- HTML explains forward reproduction; no state mutation + v +POST /_ts/trace/enable after explicit user action + | + |-- validates same-origin request signals |-- response sets __Host-ts-console - |-- HTML explains forward reproduction + |-- client requests /_ts/trace/state + |-- server reports whether the new request carried a valid cookie v Real publisher document reload | @@ -243,7 +294,7 @@ User selects "View trace results" |-- same-tab sessionStorage write |-- location.assign('/_ts/trace') v -Second GET /_ts/trace +Report GET /_ts/trace | |-- static report shell reads and validates TraceReportV1 |-- mobile HTML renders sections @@ -253,7 +304,8 @@ Second GET /_ts/trace ### 7.1 Core responsibilities - Define configuration and route behavior. -- Register the route before publisher fallback on every supported adapter. +- Provide a shared exact-path reserved-route classifier that runs before event + context, filters, auctions, named routes, or publisher fallback. - Define the platform-neutral request-context and report-envelope schemas. - Build cookie-health facts through read-only parsing. - Convert `ClientInfo` and available geo data into the public network allowlist. @@ -263,10 +315,11 @@ Second GET /_ts/trace ### 7.2 Adapter responsibilities -- Register the named route with exact method handling. +- Invoke the reserved-route classifier at the earliest adapter dispatch point + with exact path and method handling. - Populate optional `ClientInfo` fields available on the platform. -- Fastly may supply POP, HTTP version, TLS, JA4, H2 fingerprint, and edge - server data when the SDK exposes them. +- Fastly may supply bounded POP, HTTP version, TLS, and edge-server data when + the SDK exposes them. JA4 and H2 fingerprints are excluded from version one. - Other adapters return the same schema with unsupported fields absent. - Adapter-specific errors omit optional facts rather than failing publisher delivery. @@ -275,11 +328,14 @@ Second GET /_ts/trace - Accept the immutable redacted request context at initialization. - Preserve the existing bounded TS Console observation store. -- Build and validate `TraceReportV1` on explicit user action. -- Store only one report in same-tab `sessionStorage`. +- Build and validate `TraceReportV1` with a redacted + `TraceGptDiagnosticsV1` projection on explicit user action. +- Store only one supported report for the same-tab workflow in + `sessionStorage`, while treating its contents as untrusted. - Render the report shell from the validated model. -- Implement download, copy, progressive Web Share, clearing, expiry, and - accessible status reporting. +- Implement equivalent combined-report download, formatted-JSON copy, + progressive JSON-file Web Share, clearing, expiry, and accessible status + reporting. - Never upload diagnostic data or issue a telemetry query. ## 8. Route and configuration contract @@ -296,23 +352,102 @@ Rules: - `trace_page_enabled = true` requires `enabled = true`; invalid combinations fail configuration validation. -- `GET /_ts/trace` returns the setup/report HTML and establishes the session. -- `GET /_ts/trace?enabled=false` returns the shell, clears the cookie, and asks - the client to clear the stored snapshot. -- `HEAD /_ts/trace` returns the same status and headers without a body but does - not mutate the cookie. -- All other methods return a local `405 Method Not Allowed` with `Allow: GET, -HEAD`. -- Disabled deployments return a local `404` for the exact route and never fall - through to the publisher origin. -- Extra path segments, encoded separators, duplicate parameters, and lookalike - paths do not match. -- The route never creates or refreshes an EC, ingests EIDs, runs an auction, - fetches the publisher origin, or emits auction telemetry. +- Operator documentation beside this option states that the public page makes + the allowlisted presence/validity of four HttpOnly Trusted Server cookies + visible to same-origin JavaScript whenever the feature is enabled. It also + states that masked IP prefixes and coarse geo remain potentially personal or + pseudonymous network data. Enabling the option is the deployment's explicit + acceptance of those bounded disclosures. +- `GET /_ts/trace` returns the setup/report shell without changing cookies or + browser storage at the HTTP layer. After load, the explicitly included viewer + script may remove a rejected or expired local entry. Unrelated query + parameters do not activate or deactivate tracing and are not reflected into + the page or export. +- `HEAD /_ts/trace` returns the GET status and headers without a body or state + mutation. +- `GET /_ts/trace/state` returns private/no-store JSON containing only + `observed_active: true|false`, determined from whether that request carried + exactly one valid diagnostics cookie. `HEAD` returns the same status and + headers without a body. Other methods return local 405 responses. +- `GET` and `HEAD` on exactly `/_ts/trace/assets/v1.js` and + `/_ts/trace/assets/v1.css` return fixed versioned assets. They contain no + request or report data and may use immutable public caching. Other methods + return local 405 responses. These v1 URLs are immutable byte contracts: any + JS or CSS byte change requires a new asset-set URL such as `v2.js`/`v2.css` + and an updated shell reference; a release never replaces bytes at a published + immutable URL. +- `POST /_ts/trace/enable` accepts no query parameters, validates an empty body, + the exact `X-TS-Trace-Action: enable` header, and same-origin request signals; + sets the diagnostics cookie; and returns a small local JSON result. A success + response means only that the server requested the cookie change. +- `POST /_ts/trace/end` applies the same validation, clears the diagnostics + cookie using `X-TS-Trace-Action: end`, and returns a small local JSON result. + After explicit user confirmation, client JavaScript independently attempts + local report deletion and the end POST. Neither result gates the other. +- For both POST paths, absent or exactly-zero `Content-Length` is accepted, + `Transfer-Encoding` is rejected, and the adapter reads at most one byte when + it must verify an absent length. Any body byte or positive/invalid length + returns local `413 Payload Too Large` without draining or processing an + unbounded body. The one-byte read inherits a maximum two-second adapter + request-body deadline; timeout returns local `408 Request Timeout` with no + mutation. +- State-changing POSTs require an `Origin` exactly matching the canonical + request origin and `Sec-Fetch-Site: same-origin`. Missing, conflicting, + malformed, cross-site, or duplicate control values return local `403` without + mutation. This deliberately targets current supported mobile browsers rather + than weakening the check for legacy clients. +- The canonical request origin comes from adapter-owned inbound URL/scheme and + validated authority data, never an arbitrary forwarded header. Both it and + the single parsed `Origin` header are serialized with lowercase host and + default ports removed before exact comparison. Invalid or multi-valued host, + authority, scheme, or origin input fails closed. +- Unsupported methods on a shell or state-changing path return a local 405 + Method Not Allowed response with the path-specific `Allow` header. +- Disabled deployments return a local `404` for the complete trace route set, + including assets, and never fall through to the publisher origin. +- The `/_ts/trace` namespace is reserved. A trailing slash, extra path segment, + unsupported asset name, repeated separator, or lookalike beneath that + namespace returns a local `404`; an encoded separator or ambiguous dot + segment returns a local `400`. None falls through to the publisher origin. + The adapter classifies from its canonical parsed path while retaining enough + raw-path information to reject ambiguous encodings consistently. + +Every adapter implements the following order: + +1. Parse the method, canonical host/origin, path, query, and bounded headers + required for route safety. +2. Classify an exact Trusted Server reserved path. +3. For a trace path, terminate locally after only trace-specific validation and + bounded request-context inspection, including an optional read-only platform + geo lookup used solely for the displayed setup request. +4. For all other paths, continue through the adapter's ordinary event context, + authentication, request filters, geo enrichment, EC/EID processing, named + routes, auction handling, telemetry, and publisher fallback. + +Consequently a trace route never creates or finalizes an ordinary event +context, invokes publisher-configured filters, creates or refreshes an EC, +ingests EIDs, runs an auction, fetches the publisher origin, or emits auction +telemetry. Tests must verify ordering in Fastly, Axum, Cloudflare, and Spin; +ordinary named-route registration alone does not satisfy this contract. + +After an enable or end POST succeeds, the client performs a no-store state GET. +It claims `Tracing is on — cookie observed by server` only when that separate +request reports active, and `Tracing is off — cookie absent on server request` +only when it reports inactive. A mismatch or failed verification is +`Activation unconfirmed` or `Deactivation unconfirmed` and offers an idempotent +retry. These are server-observation statements, not proof that browser state is +authentic: same-origin service workers can forge or suppress the whole exchange. + +Versioned assets may remain in a CDN or browser cache after the feature is +disabled. Cache misses return the configured local 404, but rollback relies on +the uncached shell, state, and action routes being disabled; inert cached assets +alone cannot activate tracing or access a report page. The current `?ts_console=1` and `?ts_console=0` activation flow remains supported for technical users. Both activation surfaces drive the same cookie -and runtime; they must not create two concurrent diagnostic modes. +and runtime; they must not create two concurrent diagnostic modes. That +pre-existing query flow has its existing top-level-navigation activation risk; +#1050 neither expands it to the new trace GET nor claims to remediate it. ## 9. Data contracts @@ -325,9 +460,6 @@ documents: TraceRequestContextV1 schema_version: 1 captured_at: RFC 3339 UTC timestamp - page: - origin: publisher origin - path: normalized path network: masked_client_ip?: string country?: string @@ -336,8 +468,6 @@ TraceRequestContextV1 http_version?: string tls_protocol?: string tls_cipher?: string - tls_ja4?: string - h2_fingerprint?: string edge_hostname?: string edge_region?: string edge_pop?: string @@ -348,17 +478,27 @@ TraceRequestContextV1 diagnostics_session: CookieHealth ``` -The page field omits query and fragment data. It does not contain origin-facing -URLs, referrers, or arbitrary headers. +The request-context envelope intentionally contains no page URL, path, +referrer, query, or fragment. During the field-by-field trace projection, +`GptDiagnosticsExportV1.page.origin` is retained after validation and its +`pathname` is replaced with the literal `/[redacted]`. The trace viewer accepts +only that literal. Version one therefore does not store or export an exact page +path. Any future route-template policy requires a new schema and privacy review +because paths can contain accounts, emails, preview tokens, and other secrets. `masked_client_ip` uses a deterministic display-only mask for the current request: IPv4 keeps at most the first 24 bits and IPv6 keeps at most the first 48 bits. The full address never enters HTML, JavaScript, browser storage, or -export. +export. These prefixes can still be personal or pseudonymous network data; the +report labels them as approximate network identifiers and the operator privacy +decision covers them explicitly. -JA4 and H2 fingerprints are optional probabilistic identifiers. They are -included only when the deployment has separately enabled the existing -fingerprint diagnostic capability. Their absence is not an error. +All platform strings are normalized to printable characters and bounded before +they enter logs, HTML, storage, or export. Country uses at most 2 ASCII +characters; region and POP 32 UTF-8 bytes; HTTP/TLS enumerations 32 bytes; and +edge hostname/region 128 bytes. Values that fail their field contract are +omitted and produce only a bounded error category. JA4 and H2 fingerprints are +not members of `TraceRequestContextV1`. ### 9.2 Cookie health @@ -367,20 +507,62 @@ CookieHealth state: absent | present_valid | present_invalid | duplicate | unavailable source: request - detail?: allowlisted enum + detail?: + valid_ec_format | valid_eids_format | valid_tester_value + | valid_diagnostics_value | malformed | oversized + | unsupported_value | multiple_values + | header_too_large | header_not_utf8 ``` -Allowed details describe shape, not value, for example `valid_ec_format`, -`malformed`, `oversized`, or `activation_pending_response`. - -The parser must inspect the incoming request before any diagnostics-cookie -sanitization, while preserving existing authoritative-cookie and consent -semantics. Inspection is read-only: it must not generate an EC, touch the -identity graph, sync partner IDs, or extend any cookie lifetime. - -Only Trusted Server-owned cookie names are reported. Arbitrary cookie names and -values are excluded. The endpoint cannot claim knowledge of browser attributes, -expiry, or cookies the browser withheld from the request. +Details describe shape, never value. `absent` has no detail; `duplicate` uses +`multiple_values`; `unavailable` uses `header_too_large` or `header_not_utf8`; +and a valid state uses its cookie-specific valid detail. + +The classifier uses this deterministic contract: + +- Inspect all `Cookie` header fields in wire order, up to a combined 16 KiB. + Exceeding the cap or encountering any non-UTF-8 header makes all four states + `unavailable`; no partial result is presented as authoritative. +- Split each readable header on semicolons and trim optional ASCII whitespace. + A valid pair contains a non-empty RFC 6265 token name, one `=`, and the + remaining bytes as its value; additional `=` bytes belong to the value. Empty + segments and malformed pairs with an unrelated name are ignored. A segment + with no `=` counts as one malformed reserved occurrence only when its first + whitespace-delimited token is exactly a reserved name; a name such as + `ts-ec-extra` remains unrelated. No malformed unrelated pair poisons a + reserved-cookie result. +- Count exact, case-sensitive reserved names before passing values to existing + parsers. Zero occurrences is `absent`; more than one is `duplicate`, + regardless of whether one value would otherwise be valid. Duplicate + precedence is therefore diagnostic rather than first- or last-value + selection. +- Per-value limits are 512 bytes for `ts-ec`, 8 KiB for `ts-eids`, and 16 bytes + each for `ts-tester` and `__Host-ts-console`. A single value beyond its limit + is `present_invalid/oversized`; it does not change the other three states. +- One `ts-ec` occurrence is valid only when the canonical EC cookie validator + accepts its complete value. +- One `ts-eids` occurrence is valid only when the existing bounded Base64/JSON + EID parser accepts its complete value, including its current 8 KiB value cap. +- One `ts-tester` occurrence is valid only when its value is exactly `true`. +- One `__Host-ts-console` occurrence is valid only when its value is exactly + `1`. +- A single rejected value is `present_invalid` with exactly one of the public + details `malformed`, `oversized`, or `unsupported_value`. Parser error text + and the value itself never enter the report or logs. + +The parser inspects the incoming request before diagnostics-cookie sanitation, +while preserving existing authoritative-cookie and consent semantics. It must +scan without using the current lossy `CookieJar` representation, which skips +malformed pairs and cannot preserve duplicate evidence. Inspection is +read-only: it must not generate an EC, touch the identity graph, sync partner +IDs, or extend any cookie lifetime. + +Only those four Trusted Server-owned cookie names are reported. Arbitrary +cookie names and values are excluded. The endpoint cannot claim knowledge of +browser attributes, expiry, or cookies the browser withheld from the request. +Because the result reveals presence and validity of HttpOnly cookies to +same-origin JavaScript, enabling this public feature requires an explicit +operator privacy decision documented beside `trace_page_enabled`. ### 9.3 Report envelope @@ -389,28 +571,132 @@ TraceReportV1 schema_version: 1 captured_at: RFC 3339 UTC timestamp request_context: TraceRequestContextV1 - gpt_diagnostics: GptDiagnosticsExportV1-or-successor + gpt_diagnostics: TraceGptDiagnosticsV1 + truncation: + omitted_request_cycles: u16 + omitted_callback_issues: u16 + omitted_attribution_issues: u16 + omitted_nested_values: u16 ``` -The trace envelope owns request context and transport. TS Console continues to -own its nested schema. Compatibility is explicit: the viewer supports a small -documented set of TS Console schema versions and rejects unknown versions with -an actionable message rather than guessing. +`TraceGptDiagnosticsV1` is a trace-owned projection sourced only from +`GptDiagnosticsExportV1`. It contains: + +- `schema_version: 1` and `source_schema_version: 1`; +- the source `capturedAt` value; +- `page.origin` after validation and `page.pathname` fixed to `/[redacted]`; +- field-for-field allowlisted copies of the current v1 slots, requests, + callback issues, attribution issues, coverage, and metadata, subject to the + bounds and truncation below. + +It is deliberately not named or represented as `GptDiagnosticsExportV1`, +because the fixed pathname and trace-level bounds change the source field +semantics. TS Console continues to own the source schema; the trace envelope +owns its public projection and transport. The initial compatibility matrix is +exactly `TraceReportV1` plus `TraceGptDiagnosticsV1`, sourced from +`GptDiagnosticsExportV1`. The viewer rejects every unknown outer, trace-auction, +or source version with an actionable message rather than guessing. A future TS +Console successor requires an additive source compatibility change and, if the +public projection changes, a new trace-envelope version. + +Origin validation requires a parseable HTTP(S) origin whose canonical +serialization exactly equals `window.location.origin`; credentials, paths, +queries, and fragments are rejected. Outer and source capture times require +strict RFC 3339 UTC strings and must be within 60 seconds of `stored_at_ms`. +`TraceRequestContextV1.captured_at` requires strict RFC 3339 UTC but may be older +because it represents the publisher document request rather than snapshot time. ### 9.4 Storage limits and expiry - Storage key: a namespaced, versioned constant owned by the diagnostics module. -- Maximum encoded report size: 512 KiB. -- Maximum report age: 15 minutes from `captured_at`. -- One report per tab; a new explicit snapshot replaces the old report. -- Invalid, oversized, expired, or unsupported reports are removed immediately. -- `Clear report and end tracing` removes the storage entry and clears the - activation cookie. +- Stored value: `{ stored_at_ms, report }`, where `stored_at_ms` is generated by + the capture code and is not taken from report content. +- Maximum encoded size: 512 KiB, defined as the byte length of the complete + compact UTF-8 `{ stored_at_ms, report }` JSON measured with `TextEncoder` + before storage. Formatted download size and JavaScript UTF-16 string length + are not used for enforcement. +- Maximum age: 15 minutes from `stored_at_ms`. Non-finite, negative, malformed, + more than 60 seconds in the future, or older values are rejected. A backward + wall-clock jump that places the timestamp beyond the tolerated future skew + also invalidates the entry. Expiry is exposure reduction, not a security + guarantee. +- One supported report for the current browsing context; a new explicit + snapshot replaces the old report. Browser opener cloning and session restore + may copy or retain it. +- The runtime validator accepts only the exact outer and trace-auction v1 + schemas, rejects unknown fields, applies the limits below, and checks compact + UTF-8 size before rendering. +- Invalid, oversized, expired, unsupported, or hostile reports are removed when + possible and otherwise ignored. Rendering uses DOM properties and + `textContent`, never report-derived HTML. +- After confirmation, `Clear report and end tracing` always attempts local + deletion, the validated end POST, and state verification as independent + retry-safe steps. Offline or server failure cannot prevent local deletion. + The UI reports server-observed cookie state and local-report state separately. + A distinct `Delete local report` action remains available whenever a report + is displayed, including after an earlier local-deletion failure. These are product limits, not assumptions about browser quota. A storage write failure is handled even when the report is below the application limit. +Runtime limits are part of the v1 contract: + +| Value | Limit | +| --------------------------------------------- | ------------------------------------------------------------ | +| Container nesting | 8 levels | +| Slots | 64 | +| Request cycles | 10 per slot before total-size truncation | +| Callback issues | 128 | +| Attribution issues | 128 | +| Requested slot sizes | 16 per cycle | +| Ad Manager yield-group or company IDs | 8 of each per cycle | +| Creative-failure enums | 16 per cycle | +| Origin | 255 UTF-8 bytes | +| GPT pathname in trace projection | Exact literal `/[redacted]` | +| Slot element ID and ad-unit path | 512 UTF-8 bytes each | +| Trusted Server auction ID and callback reason | 256 UTF-8 bytes each | +| Any other string | 128 UTF-8 bytes | +| Enum | Exact documented value only | +| Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | +| Browser-relative timestamp or duration | Finite number from 0 through `Number.MAX_SAFE_INTEGER` | +| Visibility percentage | Finite number from 0 through 100 | +| Slot dimension | Finite integer from 1 through 100,000 | + +Every accepted string must be valid Unicode and must not contain C0/C1 control +characters or bidirectional override/isolate controls. This applies to browser +source fields as well as platform fields and precedes rendering or export. + +The snapshot builder creates a new field-by-field projection and rejects an +invalid source value rather than stringifying it. It retains only the first +documented number of requested sizes, yield-group IDs, company IDs, and creative +failure enums, recording discarded entries in `omitted_nested_values`; strings +are never silently shortened. It then measures the complete compact UTF-8 +storage wrapper. If it exceeds 512 KiB, it removes the globally oldest request +cycles first while retaining the newest cycle for each slot, then the oldest +callback issues, then the oldest attribution issues, and finally the oldest +remaining request cycles until the report fits. It records every removal in +`truncation`. +A report that still cannot fit after this bounded procedure fails snapshot +creation. The implementation must include a worst-case fixture proving the +result is bounded. + +All omission counters use checked addition. If any source collection would +make a counter exceed `u16::MAX`, projection rejects the source instead of +wrapping or saturating the count. + +For depth accounting, the `TraceReportV1` object—not its storage wrapper—is +level 1; entering either an object or an array increments the level by one; +primitives do not. No accepted report value may enter a ninth container level. +The storage wrapper is validated separately as the exact two-field object +`{ stored_at_ms, report }`. + +For deterministic ordering, a request cycle with no `requestedAtMs` sorts +before a cycle with a timestamp; otherwise cycles sort by `requestedAtMs`, then +`runtimeSlotNumber`, then `requestNumber`. Callback and attribution issues sort +by `timestampMs`, then their original array index. The builder preserves the +relative order of all retained records. + ## 10. Auction and rendering evidence The report uses TS Console's evidence model. It must preserve the distinction @@ -438,19 +724,21 @@ bids. Exact `provider X was asked for slot Y` and exact no-bid causality require a future provider-impression disposition model. That instrumentation is not silently assumed by this design. -Timing fields introduced by #1074/#1076 are consumed only after they merge and -are propagated through the live diagnostics contract. The report never queries -Tinybird, and it does not combine browser `performance.now()` values with -server-relative timing as though they were one clock. +Timing fields introduced by #1074/#1076 are outside the v1 compatibility +matrix. They may be consumed in a later version only after they merge and are +propagated through the public live-diagnostics contract. The report never +queries Tinybird, and it does not combine browser `performance.now()` values +with server-relative timing as though they were one clock. -Bidder and winning price are included only if #1081 approves them in the public -TS Console export contract. #1050 does not independently weaken the existing -privacy policy. +Bidder and winning price are not added by version one. A later version may +consume them only if #1081 approves them in the public TS Console export +contract. #1050 does not independently weaken the existing privacy policy. ## 11. Network scope -The report is inspired by Fastly Debug, not a clone. Version one uses facts -already present or reasonably addable to the platform request abstraction. +The report is inspired by Fastly Debug, not a clone. Version one exposes only +facts with a defined source and privacy boundary. Every field is optional; an +adapter must omit a value it cannot obtain directly and safely. Supported categories: @@ -458,10 +746,28 @@ Supported categories: - Country, region, and ASN when available. - HTTP version. - TLS protocol and cipher. -- Optional JA4 and H2 fingerprints. - Edge hostname, region, and POP. - Capture time. +Initial provenance and adapter support are: + +| Public field | Source | Fastly | Axum | Cloudflare | Spin | +| ------------------------------ | ---------------------------------------------------------------------------------------------- | --------------------------- | ---------------------- | ------------------------------------ | ----------- | +| `masked_client_ip` | `RuntimeServices.client_info.client_ip`, after trusted-client-IP resolution, then core masking | expected | expected | expected | expected | +| `country`, `region` | `RuntimeServices.geo.lookup(client_info.client_ip)` projected to `GeoInfo.country/region` | expected | unavailable by default | country expected, region unavailable | unavailable | +| `asn` | `GeoInfo.asn` | unavailable until populated | unavailable | unavailable until populated | unavailable | +| `http_version` | new bounded adapter mapping from inbound protocol metadata | optional | expected | optional | optional | +| `tls_protocol`, `tls_cipher` | `ClientInfo.tls_protocol/tls_cipher` | expected | unavailable | unavailable | unavailable | +| `edge_hostname`, `edge_region` | `ClientInfo.server_hostname/server_region` | expected | unavailable | unavailable | unavailable | +| `edge_pop` | new bounded adapter mapping from documented runtime metadata | optional | unavailable | optional | unavailable | + +`expected` means the implementation plan must map and test an existing source; +`optional` means the adapter includes it only when its supported SDK exposes a +stable value; `unavailable` means v1 intentionally omits it. In particular, +ASN is currently not populated by the Fastly or Cloudflare geo adapters and +must not be claimed until a concrete source is implemented. New HTTP-version or +POP mappings must be confirmed against the pinned adapter SDK before addition. + Explicitly excluded: - DNS resolver address and resolver ASN. @@ -470,6 +776,7 @@ Explicitly excluded: - DDoS/internal Fastly classifications. - Arbitrary request headers. - Full client IP in HTML or export. +- JA4, H2, or other probabilistic client fingerprints. Unsupported optional fields are omitted rather than populated with fabricated fallbacks. @@ -482,6 +789,12 @@ The report serializer constructs a new public model field by field. It never serializes request structs, cookie parsers, auction requests, telemetry rows, or browser objects wholesale. +The deserializer is an equally strict boundary. It validates the complete +outer and nested schema at runtime before any display, export, copy, or share +operation. Unknown properties, overlong strings, non-finite numbers, excessive +arrays, excessive depth, unsupported versions, and invalid timestamps reject +the report. Validation errors expose only bounded categories. + Forbidden data includes: - Raw `Cookie` and `Set-Cookie` headers. @@ -496,25 +809,52 @@ Forbidden data includes: ### 12.2 Same-origin script visibility Publisher and third-party scripts running on the publisher origin can access -`sessionStorage`. Therefore the stored model must be safe even if read by any -same-origin script. A random storage key, closed shadow root, or public endpoint -does not change this requirement. +`sessionStorage`. They can also replace it, opener-created tabs may receive a +copy, browser session restore may preserve it, and a same-origin service worker +may intercept navigation. Therefore the stored model must be safe even if read +or forged by any same-origin code. A random storage key, closed shadow root, or +public endpoint does not change this requirement. + +Every report view and exported artifact is labeled `Browser-observed, +unverified diagnostic data`. Support documentation says that it helps +troubleshoot rendering but is not proof of a server event, user identity, or +security incident. ### 12.3 Response hardening -Both the endpoint and every active diagnostic publisher response are terminally -`private, no-store`. The endpoint also sends: +The HTML shell, enable/end responses, and every active diagnostic publisher +response are terminally `private, no-store`. The fixed versioned JS/CSS assets +are the sole exception and may be publicly cached because they contain no +request or report data. HTML and JSON endpoint responses also send: -- `Content-Type: text/html; charset=utf-8` +- Path-appropriate `Content-Type`: `text/html; charset=utf-8` for the shell and + `application/json; charset=utf-8` for enable, end, and state results. - `X-Content-Type-Options: nosniff` - `Referrer-Policy: no-referrer` -- `Content-Security-Policy` restricting content to the endpoint's own static - assets and prohibiting framing -- A restrictive `Permissions-Policy` +- The Content Security Policy specified below. +- `Permissions-Policy: camera=(), microphone=(), geolocation=(), payment=(), +usb=()` + +```text +default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; +object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self' +``` -The endpoint makes no third-party requests. Dynamic JSON embedded in HTML uses -the repository's script-safe serializer and is never concatenated into -executable JavaScript. +The endpoint makes no third-party requests. Its script and stylesheet are fixed +same-origin static assets. Setup-request values are server-rendered as escaped +text nodes, and the viewer obtains the report only from browser storage. Active +publisher-page context continues to use the repository's script-safe serializer +and is never concatenated into executable JavaScript. If inline executable +assets become necessary, they require a per-response nonce or fixed build-time +hash and a corresponding CSP change. Validated report strings enter the +document through `textContent` or equivalent DOM properties, never `innerHTML`. + +The JS asset uses `application/javascript; charset=utf-8`; the CSS asset uses +`text/css; charset=utf-8`. Both send `X-Content-Type-Options: nosniff`, +`Cache-Control: public, max-age=31536000, immutable`, and a strong ETag derived +from their build bytes. They accept no dynamic input. `script-src 'self'` is an +origin-level CSP permission, not a path restriction; same-origin script +interference remains inside the stated trust limitation. ### 12.4 Shared templates and ESI @@ -527,20 +867,34 @@ publicly cacheable. - Disabled route: local privacy-safe `404`. - Unsupported method: local `405`; never publisher fallback. +- Rejected activation/end POST: local `403` with no state mutation. - Optional platform fact unavailable: omit the field and continue. -- Cookie parser failure: report `present_invalid` without the value. +- Bounded cookie inspection failure: report the contract-defined invalid or + unavailable state without a value or parser message. - Diagnostics context serialization failure: omit the context, log a bounded server error, and preserve publisher delivery. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. -- Storage unavailable, quota exceeded, or serialization oversized: remain on - the publisher page, announce the error, and offer direct download. +- Storage unavailable or quota exceeded after a valid bounded report exists: + remain on the publisher page, announce the error, and offer direct download + of that same combined `TraceReportV1`. +- Invalid projection or a report that remains oversized after deterministic + truncation: remain on the publisher page, show a bounded capture-failure + category, and do not claim that a combined trace report exists. - Missing snapshot on endpoint: show setup state, not an empty successful report. - Expired, malformed, or unknown report schema: clear it and explain that the user must reproduce again. - Clipboard or Web Share unavailable: keep JSON download available. - Export failure: retain the on-screen report and show an accessible error. +- End POST failure: report that tracing may remain active and offer an + idempotent server retry; do not undo or block the independent local-deletion + attempt. +- State verification failure or mismatch: use `unconfirmed` wording and offer + an idempotent server retry independently of local report state. +- Local deletion failure: report separately that saved browser data could not be + removed and retain the always-available deletion retry, regardless of the + server end result. Diagnostic failures must never suppress, delay, add, remove, or reorder GPT requests, auctions, targeting, or creative rendering. @@ -551,55 +905,96 @@ requests, auctions, targeting, or creative rendering. - Configuration defaults off and rejects trace-page enablement without GPT diagnostics. -- Exact route, query, method, encoded-path, and fallback behavior. -- Session cookie set/clear attributes and duplicate-directive fail-closed - behavior. +- Exact reserved-route classification, canonical-path, query, method, encoded + path, and fallback behavior. +- Exact versioned asset routes are local and contain no dynamic data; lookalike + asset paths never reach the publisher origin. +- Same-origin POST validation, cross-site/missing signal rejection, cookie + set/clear attributes, fixed 30-minute endpoint activation without request + refresh, and idempotent enable/end behavior. +- Enable/end success requires a separate state request to observe the resulting + cookie; failed and mismatched verification never displays confirmed state. +- Empty-body enforcement rejects positive/invalid lengths, transfer encoding, + the first unexpected body byte, and the two-second deadline without an + unbounded read. - Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, - and origin fetch. -- Cookie-health parser covers absent, valid, malformed, duplicate, non-UTF-8, - and oversized inputs without retaining values. -- Request-context serializer masks IPv4/IPv6 and omits query, raw headers, IDs, - and unsupported fields. + configured filters, ordinary event context, and origin fetch. +- Cookie-health scanner covers multiple header fields; zero, one, and duplicate + occurrences; mixed valid/invalid duplicates; non-UTF-8; malformed pairs; and + per-value and total-header limits without retaining values. +- Request-context serializer masks IPv4/IPv6; enforces every string bound; and + omits page paths, fingerprints, query, raw headers, IDs, and unsupported + fields. - Active responses remain terminally private/no-store under hostile late header overrides. - Dynamic HTML/JSON values cannot close elements or create executable script. ### 14.2 Adapter parity tests -- Fastly route registration and optional field mapping. +- Fastly early-route ordering and optional field mapping from documented + sources. - Axum, Cloudflare, and Spin return the common route/schema with unavailable fields omitted. -- Named route failures never fall through to publisher origin. -- HEAD and unsupported methods behave identically across adapters. -- Fastly fingerprint fields respect the existing fingerprint-debug gate. +- Trace-route failures never fall through to publisher origin. +- GET, HEAD, state-changing POST, and unsupported methods obey the same + lifecycle contract across adapters. +- Every adapter omits JA4/H2 and rejects control characters or overlong platform + strings. ### 14.3 JavaScript unit tests - Explicit snapshot only; no continuous `sessionStorage` writes. -- Size limit, schema validation, expiry, replacement, clearing, and storage +- Compact UTF-8 size measurement; exact outer/nested schema validation; unknown + fields; per-string/array/numeric/depth caps; hostile mutation; expiry; + future-clock skew; wall-clock rollback; replacement; clearing; and storage exceptions. +- Omission counters use checked arithmetic and reject overflow. +- Trace projection replaces the nested GPT pathname with `/[redacted]`, rejects + any other stored value, emits `TraceGptDiagnosticsV1`, applies deterministic + ordering/truncation, and records exact omission counts in a worst-case 512 KiB + fixture. - Same-tab navigation occurs only after a successful write. - Viewer handles absent optional network facts and every cookie-health state. - Forbidden fields never enter storage or export fixtures. - Download filename and MIME type are deterministic. -- Copy and Web Share success, rejection, absence, and fallback behavior. +- Formatted-JSON copy and JSON-file Web Share success, rejection, absence, and + download/copy fallback behavior. - 320-pixel layout, keyboard navigation, focus handling, and accessible status announcements. ### 14.4 Browser integration tests -- First endpoint visit sets the session and shows setup state. +- First endpoint GET is read-only and shows setup state; a user-initiated, + same-origin enable POST sets the session. +- Successful in-page activation adds no history entry, so Back can return to + the article when it was the prior same-tab page. +- Cross-site top-level GET, form POST, and fetch attempts cannot enable or end + tracing. - A real fixture reload activates diagnostics and captures multiple slots. - `View trace results` navigates in the same tab and renders the captured request context and slot evidence. - Empty, filled, ambiguous, no-candidate, and unattributed slot states remain distinct. - Reloading the trace page retains an unexpired same-tab report. -- A new tab cannot access the original tab's report. -- Disabling clears both cookie and report. +- Opener-cloned tabs and browser session restore never bypass validation or + application expiry; the UI does not promise tab isolation. +- Ending tracing covers successful clearing, offline POST failure, idempotent + retry, verification mismatch, successful local deletion while offline, and + local-storage deletion failure without false success messaging. - Back-forward-cache restoration is not described as a fresh traced request; the setup page tells the user to reload. - Export JSON matches the displayed versioned model. +- A storage-failure direct export matches the combined displayed model rather + than the GPT-only export. +- Hostname changes between apex, `www`, or another subdomain show the recovery + guidance rather than claiming the session followed the user. +- A fixture service worker interception is recognized as a same-origin trust + limitation, and the server endpoint remains correct when the request reaches + it. +- The delivered CSP blocks inline injection, framing, third-party connections, + and report-derived executable HTML. +- Immutable asset fixtures prove published v1 bytes never change; changed bytes + require a new URL referenced by the shell. - Inactive publisher traffic has no trace assets, storage access, listeners, or cache-policy change. @@ -620,58 +1015,84 @@ fixture: - Enable first in a controlled staging publisher configuration. - Validate response cache headers and CDN behavior before production use. - Validate with redacted fixtures before real publisher traffic. -- Log only route outcome, schema version, report-present boolean, and bounded - error category. Never log report contents or cookie/network values. +- Log only server-observable route outcome, served shell/schema version, and + bounded error category. The server cannot know whether a browser-local report + exists and must not add an upload or beacon merely to learn that fact. Never + log report contents or cookie/network values. - Roll back by disabling `trace_page_enabled`; existing `?ts_console=1` diagnostics remain independently configurable. ## 16. Acceptance criteria -1. With the feature disabled, exact trace routes return local `404` and ordinary - traffic is unchanged. -2. A mobile user can enable tracing by opening only `/_ts/trace`; no target URL, - credentials, or trace ID is required. +1. With the feature disabled, trace-route origin requests return local `404` + and ordinary traffic is unchanged. Previously cached inert versioned assets + may remain until cache eviction, but cannot activate tracing or load a shell. +2. A mobile user can enable tracing by opening only `/_ts/trace` and selecting + one prominent action; no target URL, credentials, or trace ID is required, + and a cross-site GET cannot activate tracing. 3. The setup page accurately explains that the problem must be reproduced after activation. 4. A subsequent real publisher-page reload captures redacted request context and existing TS Console evidence without altering ad behavior. -5. `View trace results` transfers one bounded snapshot in the same tab and opens - the report page without server-side storage. +5. `View trace results` transfers one bounded, runtime-validated snapshot in + the supported same-tab journey and opens the report page without server-side + storage or claims of browser-storage isolation. 6. The report separates network, cookie health, auction/render evidence, and coverage/unknowns. 7. JSON export contains the same versioned allowlisted information shown on the page. -8. No raw cookies, user IDs, full IPs, consent strings, query strings, internal - auction IDs, targeting, or creative payloads appear in HTML, browser storage, - logs, or export. +8. No raw cookies, user IDs, full IPs, consent strings, exact page paths, query + strings, fingerprints, internal auction IDs, targeting, or creative payloads + appear in trace HTML, browser storage, logs, or export. 9. Trace HTML and active publisher pages remain terminally private/no-store. 10. Missing platform fields, incomplete auction correlation, storage failure, and unavailable share APIs degrade honestly without affecting advertising. 11. The full report is usable at 320 CSS pixels and with keyboard/screen-reader navigation. -12. Auction fields owned by #1081 are consumed through its versioned public - contract rather than duplicated in #1050. +12. Version one accepts exactly `TraceReportV1` with + `TraceGptDiagnosticsV1`, sourced only from `GptDiagnosticsExportV1`; #1081 + and #1074/#1076 are optional additive follow-ups rather than release gates. +13. Every rendered and exported report is identified as browser-observed and + unverified, and hostile storage content cannot create executable HTML or + unbounded DOM output. +14. Enable/end operations expose partial failure honestly and are safe to + retry; the UI does not claim server-observed cookie state without the + follow-up state request or claim that local data was cleared when deletion + fails. ## 17. Implementation sequencing -This design is one product flow but should be implemented in dependency order: +This design is one product flow, but its implementation is split into three +independently reviewable plans and preferably three PRs: + +1. **Reserved route and privacy foundation:** configuration, shared early-route + classification, same-origin enable/end lifecycle, bounded cookie-health + inspection, base request-context schema, projection of already populated + `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not + add speculative new platform fields in this change. +2. **Browser handoff and viewer:** integrate current + `GptDiagnosticsExportV1`, project `TraceGptDiagnosticsV1`, construct and + strictly validate `TraceReportV1`, implement the same-tab workflow, combined + direct/storage exports, mobile viewer, copy/share, expiry, clearing, and + browser/accessibility tests. +3. **Optional network enrichment and future schemas:** add HTTP-version, POP, + ASN, or other fields only from SDK-verified platform sources with explicit + bounds. Adopt #1081 or #1074/#1076 later through a separately reviewed + versioned compatibility change. + +Each plan must include its own adapter, privacy, cache, and failure tests. The +implementation must not claim completion of #1081 or the timing work as part of +#1050. Version one ships with current observed auction/render evidence and +labels unavailable fields honestly. -1. Core request-context schema, cookie-health classification, configuration, - and endpoint shell. -2. Adapter route parity and Fastly optional network enrichment. -3. TS Console request-context envelope and explicit same-tab snapshot handoff. -4. Mobile viewer, export/copy/share, expiry, and clearing. -5. Integration with the current TS Console schema. -6. Additive adoption of #1081 and #1074/#1076 fields after their contracts - merge. -7. Browser, privacy, cache, and real-device acceptance. +## 18. Rejected alternatives -The implementation plan must not claim completion of #1081 or the open timing -PRs as part of #1050. If those dependencies are unavailable, the report ships -only with current observed auction/render evidence and labels unavailable fields -honestly. +### Automatic activation on `GET /_ts/trace` -## 18. Rejected alternatives +Rejected because a cross-site top-level navigation can trigger a public GET and +`SameSite=Lax` does not make that activation intentional. A same-origin fetch +POST after one explicit button press preserves the simple mobile journey and +the useful Back history entry without requiring server-side session storage. ### `/_ts/admin/trace?target=/article` @@ -715,9 +1136,16 @@ length, history, logging, referrer, and accidental-sharing risks. ## 19. Known limitations - The user must reproduce the problem after enabling tracing. -- Same-tab storage prevents cross-device and cross-tab sharing; JSON export is - the handoff artifact. -- Publisher-origin scripts can read the stored public-safe report. +- Same-tab navigation is the supported workflow, but opener-created tabs and + browser session restore may copy or retain session storage. JSON export is + the intentional support handoff artifact. +- Publisher-origin scripts and service workers can read or forge the stored + public-safe report. A same-origin service worker can also intercept or fake + the shell, enable/end/state requests, assets, and report navigation. The + experience is diagnostic evidence, not an authenticity boundary. +- The activation cookie is host-only and session storage is origin-scoped, so + the workflow does not follow the user across apex, `www`, or other + subdomains. - Browser privacy settings may disable storage, clipboard, download, or share capabilities. - Current server/browser correlation does not cover every no-bid, skipped, From 534a691e2e64852c28728be80fb90e44167c22ce Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 1 Sep 2026 16:04:45 +0530 Subject: [PATCH 032/104] Add server auction evidence to mobile trace design --- ...-mobile-ad-render-trace-endpoint-design.md | 726 +++++++++++++++--- 1 file changed, 616 insertions(+), 110 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index 6ed9f8c48..092466a6d 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -22,12 +22,12 @@ intentionally enables or ends tracing with a same-origin POST action. The endpoint is both a setup page and a report viewer. On the first visit it offers a large `Enable tracing` action and explains how to reproduce the problem. The user then returns to the real publisher page and reloads it. -Trusted Server supplies redacted request context, while the existing TS Console -records GPT, auction, and render evidence. A `View trace results` action creates -one bounded, allowlisted snapshot in same-tab `sessionStorage` and navigates to -`/_ts/trace`. The endpoint reads that untrusted snapshot, validates it, presents -a mobile-first HTML report, and offers JSON export, copy, and progressive Web -Share actions. +Trusted Server supplies redacted request context and a minimal live summary of +each server auction, while the existing TS Console records GPT and render +evidence. A `View trace results` action creates one bounded, allowlisted +snapshot in same-tab `sessionStorage` and navigates to `/_ts/trace`. The +endpoint reads that untrusted snapshot, validates it, presents a mobile-first +HTML report, and offers JSON export, copy, and progressive Web Share actions. The design introduces no report database, server-side trace store, report ID, target URL parameter, telemetry query, or publisher-origin change. It cannot @@ -57,7 +57,10 @@ experience: - TS Console owns observation of the real publisher page. The browser-local handoff joins them without introducing a backend report -service. +service. The report keeps three evidence layers separate: what the server +auction observed, what GPT observed in the page, and what the creative bridge +observed during rendering. Correlation is displayed only when an ephemeral +diagnostic token connects those layers. ## 3. Goals @@ -68,6 +71,9 @@ service. - Display a Fastly-inspired network summary for the traced publisher request. - Report health for an explicit allowlist of Trusted Server cookies without exposing their values. +- Report each observed server auction as initial-navigation SSAT, SPA page-bids, + or the Trusted Server `/auction` API, including bounded auction-local timing, + provider-call outcomes, and per-slot candidate outcomes. - Present the versioned, allowlisted TS Console evidence for every retained GPT slot and request cycle that fits the public report bounds, with explicit omission counts when deterministic size truncation is required. @@ -97,10 +103,15 @@ service. cache URLs, or stack traces. - Reimplementing TS Console auction and creative observability requested by #1081. +- Naming bidders, exposing winning price, assigning final creative numbers, or + asserting which demand path ultimately won when the available evidence does + not prove it; those remain #1081 concerns. - Querying Tinybird to build an interactive report. - Adding exact provider-by-slot no-bid explanations before the auction model can observe those dispositions. -- Direct `POST /auction` browser diagnostics in the first release. +- Introspection into third-party client-side auction internals. Version one can + report that a publisher/Prebid refresh was observed, but cannot identify its + participants or winner unless Trusted Server itself handled that auction. ## 5. Decisions @@ -137,6 +148,13 @@ lifetime bounds accidental private/no-store operation if a user forgets to end tracing; the existing technical query flow keeps its existing session-cookie semantics. +The new trace capture is active only when both `trace_page_enabled = true` and +the request carries exactly one valid diagnostics cookie. The shared cookie by +itself continues to activate the existing TS Console but does not authorize +trace tokens, server-auction projection, trace response extensions, or +correlation sidecars on a deployment whose trace page is disabled. This makes +the configuration flag the disclosure and rollback boundary. + ### 5.3 Browser-local, explicit handoff TS Console remains memory-only during observation. It writes a report to @@ -177,15 +195,25 @@ inconsistent behavior. ### 5.5 Separate issue ownership #1050 defines the report shell, request context, mobile flow, browser-local -handoff, and export. #1081 remains the owner of creative numbering, auction -classification, bidder/price policy, terminology, and normalized auction/render -timing. - -Version one consumes `GptDiagnosticsExportV1` through TS Console's public export -contract and projects it into a distinct redacted `TraceGptDiagnosticsV1`. It -does not read TS Console internals or create an alternate slot correlation -engine. #1081 and #1074/#1076 are additive follow-up work and are not release -gates for this version. +handoff, export, and the minimum server-auction facts needed to answer whether +SSAT or the Trusted Server auction API ran. Core produces a new redacted +`TraceAuctionEvidenceV1`; TS Console consumes it as immutable evidence rather +than reconstructing server behavior from GPT callbacks. + +#1081 remains the owner of bidder/price disclosure policy, creative numbering, +final user-facing terminology, and any richer cross-demand winner +classification. #1074/#1076 remain the owners of request-phase and +request-relative auction milestones. Version one may ship without those +follow-ups because it exposes already-available auction-local total and provider +durations and labels each clock explicitly. + +Version one also consumes `GptDiagnosticsExportV1` through TS Console's public +export contract and projects it into a distinct redacted +`TraceGptDiagnosticsV1`. It does not read TS Console internals or create a +second GPT attribution engine. The existing recorder emits an exact-token +`TraceSlotCorrelationV1` sidecar when it binds an opportunity to a request +cycle. The server-auction, correlation, and GPT projections are separate sibling +contracts in `TraceReportV1`; none is treated as a substitute for another. ## 6. User experience @@ -234,18 +262,43 @@ When a valid snapshot exists, `/_ts/trace` renders: 1. Report summary and capture time. 2. Network and request section for the traced publisher document. 3. Trusted Server cookie-health section. -4. Auction and rendering section grouped by numbered slot. -5. Coverage and ambiguity section. -6. Export actions. -7. `Clear report and end tracing` action. +4. Server-auction section grouped by auction and numbered slot. +5. GPT delivery and creative-rendering section grouped by numbered slot. +6. Coverage and ambiguity section. +7. Export actions. +8. `Clear report and end tracing` action. The setup request's facts are not merged into or substituted for missing traced page facts. Missing fields display `Unavailable`; missing evidence displays `Not observed` or `Unknown`, following TS Console terminology. -The report begins with `Browser-observed, unverified diagnostic data`. It does -not claim that the snapshot is authentic or suitable as forensic or security -evidence. +The viewer presents an evidence chain rather than one overloaded status: + +```text +Server auction -> GPT request/response -> creative render/load/viewability +``` + +For each step it shows the source, observed outcome, and correlation state. +`Client-side refresh observed` describes browser intent only; it must never be +rendered as `client-side auction won`. Likewise, a filled GPT slot does not +prove that the Trusted Server candidate rendered. + +Internal enums are exported for machines, but the page uses plain labels: + +| Evidence enum | Mobile label | +| ------------------------------------------ | ------------------------------------------------- | +| `initial_navigation_ssat` | `Initial-page server auction (SSAT)` | +| `spa_page_bids` | `Trusted Server page-refresh auction` | +| `auction_api` | `Trusted Server auction API` | +| GPT `prebid_refresh`/`publisher_refresh` | `Browser refresh observed; winner not determined` | +| GPT `competing`/`unattributed` | `Multiple or unknown delivery paths` | +| matched non-empty creative-bridge evidence | `Trusted Server creative rendered` | + +The report begins with `Browser-carried, unverified diagnostic data`. Server +auction entries are labeled `Produced by Trusted Server; copied through an +untrusted browser snapshot`, while GPT and creative entries are labeled +`Browser observed`. The viewer does not claim that the stored snapshot is +authentic or suitable as forensic or security evidence. `Copy` copies formatted JSON. `Share` supplies the same JSON file to the native Web Share sheet only after an explicit tap and tells the user that the selected @@ -286,7 +339,8 @@ Real publisher document reload |-- adapter supplies optional network facts |-- core computes allowlisted cookie health |-- core injects redacted TraceRequestContextV1 - |-- existing TS Console observes GPT and TS delivery + |-- core observes live server auctions and emits TraceAuctionEvidenceV1 + |-- existing TS Console observes GPT and creative delivery v User selects "View trace results" | @@ -307,6 +361,10 @@ Report GET /_ts/trace - Provide a shared exact-path reserved-route classifier that runs before event context, filters, auctions, named routes, or publisher fallback. - Define the platform-neutral request-context and report-envelope schemas. +- Build `TraceAuctionEvidenceV1` directly from the live auction observation and + orchestration result; never query telemetry or serialize telemetry rows. +- Mint and thread public diagnostic auction/slot tokens independently of + internal request IDs, including zero-bid and failed auctions. - Build cookie-health facts through read-only parsing. - Convert `ClientInfo` and available geo data into the public network allowlist. - Inject request context only into an active private diagnostics document. @@ -327,9 +385,14 @@ Report GET /_ts/trace ### 7.3 JavaScript responsibilities - Accept the immutable redacted request context at initialization. +- Accept immutable server-auction evidence delivered through the supported + initial-navigation, page-bids, and `/auction` transports. +- Emit the bounded `TraceSlotCorrelationV1` sidecar at the existing GPT + recorder's opportunity-to-cycle binding point. - Preserve the existing bounded TS Console observation store. -- Build and validate `TraceReportV1` with a redacted - `TraceGptDiagnosticsV1` projection on explicit user action. +- Build and validate `TraceReportV1` with redacted + `TraceAuctionEvidenceV1` and `TraceGptDiagnosticsV1` projections on explicit + user action. - Store only one supported report for the same-tab workflow in `sessionStorage`, while treating its contents as untrusted. - Render the report shell from the validated model. @@ -356,8 +419,9 @@ Rules: the allowlisted presence/validity of four HttpOnly Trusted Server cookies visible to same-origin JavaScript whenever the feature is enabled. It also states that masked IP prefixes and coarse geo remain potentially personal or - pseudonymous network data. Enabling the option is the deployment's explicit - acceptance of those bounded disclosures. + pseudonymous network data, and that opaque server-auction outcomes become + visible to same-origin JavaScript. Enabling the option is the deployment's + explicit acceptance of those bounded disclosures. - `GET /_ts/trace` returns the setup/report shell without changing cookies or browser storage at the HTTP layer. After load, the explicitly included viewer script may remove a rejected or expired local entry. Unrelated query @@ -571,16 +635,28 @@ TraceReportV1 schema_version: 1 captured_at: RFC 3339 UTC timestamp request_context: TraceRequestContextV1 + server_auctions: TraceAuctionEvidenceV1[] + slot_correlations: TraceSlotCorrelationV1[] gpt_diagnostics: TraceGptDiagnosticsV1 + auction_coverage: + capture_status: complete | partial | unavailable | not_observed + issues: + evidence_projection_failed | evidence_transport_failed + | evidence_validation_failed | record_evicted + | correlation_unavailable | external_client_side_unobservable truncation: + omitted_server_auctions: u16 + omitted_slot_correlations: u16 omitted_request_cycles: u16 omitted_callback_issues: u16 omitted_attribution_issues: u16 omitted_nested_values: u16 ``` -`TraceGptDiagnosticsV1` is a trace-owned projection sourced only from -`GptDiagnosticsExportV1`. It contains: +`TraceAuctionEvidenceV1` is a server-produced public model defined in section +9.4. `TraceSlotCorrelationV1` is the browser-produced exact-token sidecar +defined in section 9.4.1. `TraceGptDiagnosticsV1` is a separate trace-owned +projection sourced only from `GptDiagnosticsExportV1`. It contains: - `schema_version: 1` and `source_schema_version: 1`; - the source `capturedAt` value; @@ -593,11 +669,24 @@ It is deliberately not named or represented as `GptDiagnosticsExportV1`, because the fixed pathname and trace-level bounds change the source field semantics. TS Console continues to own the source schema; the trace envelope owns its public projection and transport. The initial compatibility matrix is -exactly `TraceReportV1` plus `TraceGptDiagnosticsV1`, sourced from -`GptDiagnosticsExportV1`. The viewer rejects every unknown outer, trace-auction, -or source version with an actionable message rather than guessing. A future TS -Console successor requires an additive source compatibility change and, if the -public projection changes, a new trace-envelope version. +exactly `TraceReportV1`, `TraceAuctionEvidenceV1`, `TraceSlotCorrelationV1`, and +`TraceGptDiagnosticsV1`, with the GPT projection sourced from +`GptDiagnosticsExportV1`. The viewer rejects every unknown outer, +server-auction, correlation, GPT-projection, or GPT-source version with an +actionable message rather than guessing. A future TS Console successor requires +an additive source compatibility change and, if the public projection changes, +a new trace-envelope version. + +`auction_coverage.capture_status` describes only what reached the browser +collector: `not_observed` means no valid server-auction record arrived, not that +no server auction ran. `partial` requires at least one retained record plus a +projection, transport, validation, or eviction issue; `unavailable` requires no +retained records plus a known projection, transport, or validation issue; and +`complete` requires at least one retained record without those capture issues. +`correlation_unavailable` and `external_client_side_unobservable` describe +interpretation limits and do not change an otherwise complete capture status. +The issue array is deduplicated, sorted in enum order, bounded to 16 values, and +contains no error text. Origin validation requires a parseable HTTP(S) origin whose canonical serialization exactly equals `window.location.origin`; credentials, paths, @@ -606,7 +695,268 @@ strict RFC 3339 UTC strings and must be within 60 seconds of `stored_at_ms`. `TraceRequestContextV1.captured_at` requires strict RFC 3339 UTC but may be older because it represents the publisher document request rather than snapshot time. -### 9.4 Storage limits and expiry +### 9.4 Server-auction evidence + +Core creates one `TraceAuctionEvidenceV1` at the live auction boundary. It is +not derived from the browser, reconstructed from winning-bid targeting, or +loaded from auction telemetry: + +```text +TraceAuctionEvidenceV1 + schema_version: 1 + diagnostic_auction_id: string + source: + initial_navigation_ssat | spa_page_bids | auction_api + terminal_status: + completed | execution_failed | dispatch_failed | abandoned | skipped + terminal_reason?: + policy_skipped | no_eligible_slots | no_provider_launched + | provider_execution_failed | collection_failed | unknown + total_time_ms?: u32 + provider_calls: + - provider_number: u16 + role: bidder | mediator | unknown + status: success | no_bid | error | pending | abandoned | unknown + response_time_ms?: u32 + returned_bid_count: u16 + slots: + - slot_number: u16 + slot_ref: string + requested_sizes: [u32, u32][] + returned_bid_count: u16 + candidate: + selected | no_candidate | selected_unrenderable | unknown + selected_creative_size?: [u32, u32] + truncation: + omitted_provider_calls: u16 + omitted_slots: u16 + omitted_nested_values: u16 + coverage: + provider_to_slot_no_bid: unavailable +``` + +The model has deliberately lower cardinality and sensitivity than the existing +telemetry and OpenRTB objects: + +- `diagnostic_auction_id` is a fresh opaque `ts-auc-...` correlation token. When + trace capture is active under the two-part gate in section 5.2, it is minted + once when an eligible auction is observed, before dispatch, and is retained + for zero-bid, skipped, dispatch-failed, execution-failed, and abandoned + outcomes. It is never `AuctionRequest.id`, the telemetry UUID, a provider + request ID, or an identifier joinable to user-bearing logs. +- Auction and slot tokens are the fixed prefixes `ts-auc-` and `ts-slot-` + followed by a canonical lowercase hyphenated UUID v4. Validators reject every + other shape; tokens are not silently shortened or normalized. +- `slot_number` is a one-based ordinal over the exact post-conversion + `AuctionRequest.slots` sequence observed by orchestration. It is display-only + and is never used to map a response back to pre-conversion client input. + `slot_ref` is a fresh auction-local opaque token carried with that slot. Core + creates it for initial-navigation and SPA auctions. For a TSJS `/auction` + request, TSJS creates it only after `buildAdRequest` has finished grouping and + deduplicating the final `adUnits` array, attaches it to that exact outgoing + unit as `adUnits[].ext.trusted_server.trace_slot_ref`, and retains the + request-scoped token-to-unit mapping. Core accepts that member only under the + two-part trace gate, validates and echoes the token for accepted converted + slots, and strips it before every provider or mediator request. A missing or + invalid client token causes core to mint a server token with no browser + correlation; it never changes ordinary auction acceptance. TSJS uses + `crypto.randomUUID()` and, if unavailable, omits the client token rather than + using weak randomness. Raw publisher slot IDs, ad-unit paths, and internal + impression IDs are not copied into this model. Neither the ordinal nor token + is a creative number. +- `source` is assigned by the server call site: initial document auction is + `initial_navigation_ssat`, `/_ts/page-bids` is `spa_page_bids`, and + `POST /auction` is `auction_api`. Browser `requestPath` does not determine or + override this value. +- `provider_number` is assigned deterministically in provider dispatch order + and is stable only within one auction. Provider names, bidder/seat names, and + provider metadata are omitted. `returned_bid_count` is a count, not a bid + payload. +- `terminal_reason` is mapped to the allowlisted category at the observation + boundary. Raw error messages, parser errors, URLs, and provider text never + enter the model. +- Slot candidate state is computed from the requested slots, returned bids, + winner selection, and final response-conversion disposition. A winner that + cannot safely enter the bid map/OpenRTB response is + `selected_unrenderable`, not `selected` and not `no_candidate`. +- Provider calls are auction-wide. The model does not claim that a provider was + called, timed out, or returned no bid for a particular slot. Only returned + bids can contribute to a slot's `returned_bid_count`; the fixed coverage + value makes the missing provider-to-slot no-bid relation explicit. +- `total_time_ms` and `response_time_ms` use the server's auction-local monotonic + durations. They are not request-relative milestones and are never + arithmetically combined with browser timestamps. +- Core applies the provider, slot, size, string, and numeric limits before the + model crosses into HTML or JSON. It retains request/dispatch order and records + every discarded nested entry in the auction-local `truncation` object. An + omission-counter overflow rejects that auction evidence rather than wrapping + or saturating. A duration that cannot convert to its optional public integer + type is omitted and counted; a required count or ordinal conversion failure + rejects that auction evidence. Values are never clamped to a plausible value. + +The server evidence and GPT evidence retain separate meanings: + +| Question | Authoritative v1 source | +| --------------------------------------------------- | ------------------------------------------------------------- | +| Did initial-navigation SSAT run? | `server_auctions[].source = initial_navigation_ssat` | +| Did the Trusted Server auction API run? | `server_auctions[].source = auction_api` | +| Was a publisher/Prebid refresh observed? | GPT `requestPath`, labeled as browser intent | +| Did a server auction select a slot candidate? | matching server-auction slot `candidate` | +| Did GPT request, fill, and render the slot? | `TraceGptDiagnosticsV1` request-cycle evidence | +| Did the Trusted Server creative bridge participate? | `TraceGptDiagnosticsV1` creative-delivery evidence | +| Which path ultimately won? | only when existing correlation proves it; otherwise `Unknown` | + +Absence is not converted into a negative assertion. If transport or +correlation failed, the viewer shows `Server auction evidence unavailable` or +`Correlation unknown`, not `SSAT did not run`. + +#### 9.4.1 Live transport and correlation + +Evidence is transported only while `trace_page_enabled` is true and the +diagnostics cookie is valid. Every response carrying it is terminally +`private, no-store`: + +```text +TraceAuctionTransportV1 + schema_version: 1 + evidence?: TraceAuctionEvidenceV1 + unavailable_reason?: evidence_projection_failed +``` + +Exactly one of `evidence` and `unavailable_reason` is present. This small +transport envelope lets core report a safe projection failure without exposing +the raw error. A malformed envelope is rejected as a whole. A network failure +before an envelope arrives is recorded separately by the browser as +`evidence_transport_failed`. + +Initial-navigation and SPA slot definitions carry their token on the exact slot +object TSJS already consumes: + +```text +AuctionSlot.ext.trusted_server.trace_slot_ref: string +``` + +Core adds that optional nested member only under the two-part trace gate. It +assigns the token while constructing the request-scoped slot definitions and +threads the same token into the corresponding `AuctionRequest` observation, so +neither side needs to recover the relationship from an ordinal or raw slot ID. +The ordinary `AuctionSlot.id`, `gam_unit_path`, `div_id`, formats, targeting, +ordering, and bid-map keys remain unchanged. The extension is absent when the +gate is false and is never copied into `TraceAuctionEvidenceV1` except as its +already-allowlisted opaque `slot_ref`. + +TSJS accepts a slot extension only when its canonical token occurs exactly once +in both the delivered slot list and the matching auction evidence. A missing, +malformed, duplicate, or conflicting token prevents only that sidecar join, +adds `evidence_validation_failed` and `correlation_unavailable`, and does not +drop, reorder, or mutate the ordinary slot or bid. TSJS reads no other extension +property. This validation occurs before the slot is handed to the existing GPT +initialization path. + +The three transport call shapes are exact v1 contracts: + +```text +Initial seam: + scheduleInitialAdInit(bids, slots?, traceAuctionTransport?) + +SPA JSON: + { slots, bids, trace_auction?: TraceAuctionTransportV1 } + +/auction request unit: + adUnits[].ext.trusted_server.trace_slot_ref?: string + +/auction OpenRTB response: + ext.trusted_server.trace_auction?: TraceAuctionTransportV1 +``` + +The initial scheduler validates/records the optional third argument before it +runs `adInit`; cached older bundles may ignore the extra argument without +affecting ads, in which case evidence remains `not_observed`. The SPA parser +accepts only the exact optional top-level member and preserves its existing +`slots` and `bids` behavior. These trace members never become required for a +successful advertising response. + +When the existing GPT recorder consumes the matching Trusted Server opportunity +for a concrete request cycle, it emits this trace-owned sidecar: + +```text +TraceSlotCorrelationV1 + schema_version: 1 + diagnostic_auction_id: string + slot_ref: string + runtime_slot_number: safe positive integer + request_number: safe positive integer +``` + +The recorder already decides which pending opportunity belongs to which GPT +slot/request cycle. The sidecar records that exact decision; it does not rerun +attribution or read a private store during export. It is emitted only when both +opaque server tokens and the concrete GPT cycle are present. The current +`GptDiagnosticsExportV1` remains unchanged and the sidecar contains no slot +element ID or ad-unit path. + +- **Initial-navigation SSAT:** core builds the evidence when the split auction + is collected at the held body tail. It injects the script-safe public model + beside the winning-bid map before initial ad initialization; the corresponding + request-scoped slot definitions already carry + `ext.trusted_server.trace_slot_ref`. Failed, abandoned, skipped, and zero-bid + outcomes still inject their bounded evidence when the publisher document can + be delivered. +- **SPA page-bids:** `/_ts/page-bids` adds an optional, namespaced + `trace_auction` transport envelope beside its existing bid result. TSJS + validates and records it and consumes each returned slot's + `ext.trusted_server.trace_slot_ref` before triggering ad initialization. Both + the envelope and slot extensions are absent when the trace gate is inactive. +- **Trusted Server `/auction` API:** the existing OpenRTB response adds a + namespaced `ext.trusted_server.trace_auction` transport envelope only for an + active diagnostics request. After producing the final grouped `AdRequest`, + both TSJS callers assign one fresh token to each outgoing unit and retain that + exact request-scoped mapping. They validate the echoed evidence and record it + before parsing bids. A converted or skipped unit therefore cannot shift + another slot's correlation. The response member does not replace or expose + the existing orchestrator extension, and the trace projection must not copy + that extension's provider names, bidder names, metadata, price, creative IDs, + domains, or markup. HTTP/transport failures with no readable response are + browser-observed failures only; no successful server evidence is + manufactured. + +The direct TSJS caller records `evidence_transport_failed` from its existing +non-OK, unreadable-JSON, and rejected-`fetch` paths. The Prebid adapter +creates one bounded pending transport record after `buildRequests`, keyed by the +request's original bid IDs and its normalized unit tokens. `interpretResponse` +consumes it on a readable response; the pinned Prebid `onTimeout` and +`onBidderError` bidder-spec hooks consume it and record +`evidence_transport_failed` otherwise. Repeated hooks are idempotent. Pending +records are capped at 128 and expire after the configured bid timeout plus five +seconds. Expiry without any supported success/error/timeout hook proves no +transport outcome, so it removes the marker and leaves evidence `not_observed` +rather than inventing a failure. These hooks collect only bounded categories and +opaque tokens, never XHR error text or response bodies. + +The diagnostic auction token is also attached to the existing GPT opportunity +marker, and the opaque slot token is carried through the corresponding +winning-bid/slot initialization path. The numeric ordinal is never a +correlation key. The viewer joins a server slot to a GPT cycle only when one validated +`TraceSlotCorrelationV1` exactly matches both tokens and the exported +runtime/request numbers. It displays unmatched, duplicate, or conflicting +records independently, preserves competing paths, and never joins by +timestamps, implicit array position, ad-unit path, or a best-effort heuristic. + +TSJS retains at most the newest 16 validated server-auction records and 128 +correlation sidecars in memory. It increments checked eviction counters for +older records; the snapshot adds those counts to the matching truncation fields +and emits `record_evicted` with `partial`. It performs no storage write until +the explicit snapshot action. +Requests for which either side of the trace-capture gate is false do not mint +trace tokens, build trace evidence, add response members, emit sidecars, or +install auction-evidence listeners. + +No database, server-side report store, Tinybird query, beacon, or follow-up +network request is required. Evidence already available at the live auction +boundary is projected and carried forward in the response that the browser is +already receiving. + +### 9.5 Storage limits and expiry - Storage key: a namespaced, versioned constant owned by the diagnostics module. @@ -624,9 +974,10 @@ because it represents the publisher document request rather than snapshot time. - One supported report for the current browsing context; a new explicit snapshot replaces the old report. Browser opener cloning and session restore may copy or retain it. -- The runtime validator accepts only the exact outer and trace-auction v1 - schemas, rejects unknown fields, applies the limits below, and checks compact - UTF-8 size before rendering. +- The runtime validator accepts only the exact outer, server-auction, + slot-correlation, auction-coverage, and GPT-projection v1 schemas, rejects + unknown fields, applies the limits below, and checks compact UTF-8 size before + rendering. - Invalid, oversized, expired, unsupported, or hostile reports are removed when possible and otherwise ignored. Rendering uses DOM properties and `textContent`, never report-derived HTML. @@ -645,6 +996,11 @@ Runtime limits are part of the v1 contract: | Value | Limit | | --------------------------------------------- | ------------------------------------------------------------ | | Container nesting | 8 levels | +| Server auctions | 16 | +| Slot correlations | 128 | +| Provider calls | 16 per server auction | +| Auction slots | 64 per server auction | +| Auction coverage issues | 16 | | Slots | 64 | | Request cycles | 10 per slot before total-size truncation | | Callback issues | 128 | @@ -656,6 +1012,7 @@ Runtime limits are part of the v1 contract: | GPT pathname in trace projection | Exact literal `/[redacted]` | | Slot element ID and ad-unit path | 512 UTF-8 bytes each | | Trusted Server auction ID and callback reason | 256 UTF-8 bytes each | +| Diagnostic auction ID and opaque slot ref | 128 UTF-8 bytes each | | Any other string | 128 UTF-8 bytes | | Enum | Exact documented value only | | Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | @@ -668,15 +1025,24 @@ characters or bidirectional override/isolate controls. This applies to browser source fields as well as platform fields and precedes rendering or export. The snapshot builder creates a new field-by-field projection and rejects an -invalid source value rather than stringifying it. It retains only the first -documented number of requested sizes, yield-group IDs, company IDs, and creative -failure enums, recording discarded entries in `omitted_nested_values`; strings -are never silently shortened. It then measures the complete compact UTF-8 -storage wrapper. If it exceeds 512 KiB, it removes the globally oldest request -cycles first while retaining the newest cycle for each slot, then the oldest -callback issues, then the oldest attribution issues, and finally the oldest -remaining request cycles until the report fits. It records every removal in -`truncation`. +invalid source value rather than stringifying it. Server auctions are already +bounded by core; the browser rejects an invalid inner model rather than +truncating it. The builder retains the newest 16 server auctions in observation +order, the newest 128 correlations in recorder emission order, and only the +first documented number of GPT requested sizes, yield-group IDs, company IDs, +and creative failure enums. It records each discard in +`omitted_server_auctions`, `omitted_slot_correlations`, or +`omitted_nested_values`; strings are never silently shortened. It then measures +the complete compact UTF-8 storage wrapper. If it exceeds 512 KiB, it removes +the globally oldest GPT +request cycles first while retaining the newest cycle for each GPT slot, then +the oldest callback issues, then the oldest attribution issues, then the oldest +uncorrelated server auctions, and finally the oldest remaining GPT cycles and +server auctions until the report fits. It records every removal in +`truncation`. A correlated auction and the newest GPT cycle that references it +are retained or removed together once the algorithm reaches correlated server +auctions; every sidecar referencing a removed auction or cycle is removed and +counted. The report must not retain a dangling token while claiming a join. A report that still cannot fit after this bounded procedure fails snapshot creation. The implementation must include a worst-case fixture proving the result is bounded. @@ -691,48 +1057,61 @@ primitives do not. No accepted report value may enter a ninth container level. The storage wrapper is validated separately as the exact two-field object `{ stored_at_ms, report }`. -For deterministic ordering, a request cycle with no `requestedAtMs` sorts -before a cycle with a timestamp; otherwise cycles sort by `requestedAtMs`, then -`runtimeSlotNumber`, then `requestNumber`. Callback and attribution issues sort -by `timestampMs`, then their original array index. The builder preserves the -relative order of all retained records. +For deterministic ordering, retained server auctions remain in observation +order, provider calls and auction slots retain their server-assigned numeric +order, and correlations retain recorder emission order. A request cycle with no +`requestedAtMs` sorts before a cycle with a timestamp; otherwise cycles sort by +`requestedAtMs`, then `runtimeSlotNumber`, then `requestNumber`. Callback and +attribution issues sort by `timestampMs`, then their original array index. The +builder preserves the relative order of all retained records. ## 10. Auction and rendering evidence -The report uses TS Console's evidence model. It must preserve the distinction -between: +The report combines, but never conflates, the server model from section 9.4 and +TS Console's browser model. It must preserve the distinction between: -- A Trusted Server opportunity. -- A provider response. -- A selected Trusted Server candidate. +- A server-observed auction and its source. +- An auction-wide provider call and response. +- A server-selected candidate for one opaque slot reference. +- A browser-observed Trusted Server opportunity. - A GPT request and response. - A non-empty GPT render. - Trusted Server creative-bridge evidence. - Creative load and viewability. - A publisher or client-side refresh. -The viewer must not infer that Trusted Server rendered an ad merely because GPT -reported a filled slot. Ambiguous and unattributed cycles remain explicit. - -Current diagnostics tokens exist only on delivered winning bids. No-bid, -failed, skipped, hidden, unresolved, and direct `/auction` paths can lack server -correlation. The report displays the available observed facts and `Unknown` -rather than manufacturing a correlation. +The diagnostic auction token is created before dispatch rather than only on a +delivered winner, so zero-bid and terminal failure states have an identity when +a response can carry evidence. The viewer still must not infer that Trusted +Server rendered an ad merely because the server selected a candidate or GPT +reported a filled slot. Ambiguous, competing, unmatched, and unattributed +cycles remain explicit. Provider-call telemetry is auction-wide, while bid rows exist only for returned bids. Exact `provider X was asked for slot Y` and exact no-bid causality require a future provider-impression disposition model. That instrumentation is not silently assumed by this design. -Timing fields introduced by #1074/#1076 are outside the v1 compatibility -matrix. They may be consumed in a later version only after they merge and are -propagated through the public live-diagnostics contract. The report never -queries Tinybird, and it does not combine browser `performance.now()` values -with server-relative timing as though they were one clock. +The UI groups timing into three labeled clocks: + +1. **Server auction-local:** v1 `total_time_ms` and provider + `response_time_ms`, measured from the live orchestration result. +2. **Request-relative server milestones:** dispatched, resolved, and committed + milestones from #1076, unavailable in this schema and adoptable later. +3. **Browser/GPT:** TS Console request, response, render, load, and viewability + timings. -Bidder and winning price are not added by version one. A later version may -consume them only if #1081 approves them in the public TS Console export -contract. #1050 does not independently weaken the existing privacy policy. +The report never queries Tinybird and never subtracts or combines values from +different clocks. #1074/#1076 may add request-relative fields only through a +separately reviewed compatibility change. + +Bidder identity, provider identity, winning price, currency, creative numbering, +and a final `SSAT/TS/client-side winner` label are not added by version one. A +later version may consume them only if #1081 approves their meaning and public +disclosure policy. #1050 does not independently weaken the existing privacy +policy. Version one can nevertheless answer the narrower, evidence-based +questions: which Trusted Server entry point ran, what bounded server outcome it +reported, what GPT did afterward, and where correlation is missing. ## 11. Network scope @@ -798,13 +1177,15 @@ the report. Validation errors expose only bounded categories. Forbidden data includes: - Raw `Cookie` and `Set-Cookie` headers. -- EC IDs, EIDs, bidder user IDs, and consent strings. +- EC IDs, EIDs, bidder user IDs, provider/bidder/seat names, and consent + strings. - Unmasked client IP. - Query strings and fragments. - Fastly or internal request identifiers that can join to user-bearing logs. - Internal `AuctionRequest.id`. -- Bid requests/responses, losing-bid payloads, targeting, creative markup, - cache URLs, and stack traces. +- Bid requests/responses, bid prices/currency, losing-bid payloads, provider + metadata, targeting, creative IDs/domains/markup, cache URLs, and stack + traces. ### 12.2 Same-origin script visibility @@ -815,15 +1196,17 @@ may intercept navigation. Therefore the stored model must be safe even if read or forged by any same-origin code. A random storage key, closed shadow root, or public endpoint does not change this requirement. -Every report view and exported artifact is labeled `Browser-observed, -unverified diagnostic data`. Support documentation says that it helps -troubleshoot rendering but is not proof of a server event, user identity, or -security incident. +Every report view and exported artifact is labeled `Browser-carried, +unverified diagnostic data`. Individual server entries retain their +server-produced provenance, but support documentation says the browser-carried +copy helps troubleshoot rendering and is not cryptographic proof of a server +event, user identity, or security incident. ### 12.3 Response hardening -The HTML shell, enable/end responses, and every active diagnostic publisher -response are terminally `private, no-store`. The fixed versioned JS/CSS assets +The HTML shell, enable/end responses, every active diagnostic publisher +response, and every dynamic page-bids or `/auction` response carrying trace +evidence are terminally `private, no-store`. The fixed versioned JS/CSS assets are the sole exception and may be publicly cached because they contain no request or report data. HTML and JSON endpoint responses also send: @@ -859,6 +1242,9 @@ interference remains inside the stated trust limitation. ### 12.4 Shared templates and ESI Per-request trace context must never enter a shared template or ESI fragment. +This includes diagnostic auction/slot tokens and the optional `AuctionSlot` +extension; active responses add them only in request-scoped injection or the +request-scoped body seam. The existing diagnostics private/no-store decision remains a load-bearing gate. Tests must prove that late response-header handlers cannot make traced content publicly cacheable. @@ -873,6 +1259,15 @@ publicly cacheable. unavailable state without a value or parser message. - Diagnostics context serialization failure: omit the context, log a bounded server error, and preserve publisher delivery. +- Server-auction evidence construction or serialization failure: omit only the + affected evidence, retain a bounded `unavailable` coverage marker when safe, + log no report content, and preserve the normal auction/result path. +- Initial-navigation evidence cannot be injected because the body tail is not + reached: preserve publisher delivery. Because the browser received no safe + marker, display `not_observed` rather than claiming a known server failure. +- Page-bids or `/auction` evidence is absent or rejected by its strict client + validator: parse the ordinary bid response exactly as before, discard the + diagnostic member, and show an unmatched/invalid-evidence coverage category. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. - Storage unavailable or quota exceeded after a valid bounded report exists: @@ -896,8 +1291,10 @@ publicly cacheable. removed and retain the always-available deletion retry, regardless of the server end result. -Diagnostic failures must never suppress, delay, add, remove, or reorder GPT -requests, auctions, targeting, or creative rendering. +Diagnostic failures must never suppress, delay, add, remove, reorder, or change +the success status of GPT requests, auctions, bid responses, targeting, or +creative rendering. The evidence projection is a side effect of already-known +results, never a prerequisite for returning them. ## 14. Testing strategy @@ -905,6 +1302,10 @@ requests, auctions, targeting, or creative rendering. - Configuration defaults off and rejects trace-page enablement without GPT diagnostics. +- With GPT diagnostics enabled but `trace_page_enabled = false`, a valid + console cookie still enables the existing console but never mints trace + tokens, builds auction evidence, adds response extensions, or emits + correlation sidecars. - Exact reserved-route classification, canonical-path, query, method, encoded path, and fallback behavior. - Exact versioned asset routes are local and contain no dynamic data; lookalike @@ -925,6 +1326,38 @@ requests, auctions, targeting, or creative rendering. - Request-context serializer masks IPv4/IPv6; enforces every string bound; and omits page paths, fingerprints, query, raw headers, IDs, and unsupported fields. +- Server-auction projection maps initial navigation, SPA page-bids, and auction + API call sites to the exact public source enums without using a browser hint. +- The diagnostic auction token is minted before dispatch and remains identical + across completed, zero-bid, skipped, failed, and abandoned evidence and the + corresponding browser opportunity marker. It never equals or contains the + internal auction ID or telemetry UUID. +- Server-auction projection covers every terminal status/reason mapping, + auction-local total duration, provider role/status/duration/count, per-slot + requested sizes/bid count/candidate disposition, and checked numeric + conversion. +- Provider numbering and opaque slot references are deterministic and bounded; + provider names, bidder/seat names, prices, currency, publisher slot IDs, + creative identifiers/domains/markup, metadata, raw errors, and internal IDs + are absent from serialized fixtures. +- Auction-wide provider no-bid evidence is never projected as a per-slot + disposition, and `provider_to_slot_no_bid` remains `unavailable`. +- Evidence projection/serialization failure leaves the ordinary bid map or + OpenRTB response unchanged apart from the bounded unavailable transport + envelope. +- `/auction` request tokens survive the exact AdRequest-to-AuctionRequest slot + conversion, are stripped before provider dispatch, and remain correctly + associated across grouped multi-bidder units, duplicate codes, skipped + non-banner units, and mixed accepted/filtered inputs. Numeric ordinals are + never used for client correlation. +- Token tests cover canonical UUID-v4 shape, missing Web Crypto, malformed or + duplicate request extensions, the disabled trace gate, and proof that invalid + tokens neither fail nor otherwise alter the ordinary auction. +- Initial and SPA slot JSON attaches the exact + `ext.trusted_server.trace_slot_ref` token that appears in server evidence; + inactive responses omit it. Missing, duplicate, conflicting, malformed, and + evidence-mismatched slot tokens suppress only correlation and produce the + specified coverage issues without changing slot/bid order or contents. - Active responses remain terminally private/no-store under hostile late header overrides. - Dynamic HTML/JSON values cannot close elements or create executable script. @@ -949,6 +1382,31 @@ requests, auctions, targeting, or creative rendering. future-clock skew; wall-clock rollback; replacement; clearing; and storage exceptions. - Omission counters use checked arithmetic and reject overflow. +- Strict validation of every server-auction enum, token, numeric bound, array + bound, nesting level, and unknown property; invalid evidence is discarded + without changing ordinary bid parsing. +- Transport envelopes require exactly one of evidence/unavailable reason; + projection, transport, validation, eviction, correlation, and external-client + coverage states produce the specified complete/partial/unavailable/not-observed + result without treating absence as proof that no auction ran. +- Direct fetch failures and Prebid `interpretResponse`, `onTimeout`, and + `onBidderError` paths consume their pending transport record exactly once; + capped/expired records and absent hooks follow the specified `not_observed` + behavior without retaining error text or bodies. +- Exact-token correlation joins matching server auctions, GPT opportunities, + and slot references; unmatched, duplicated, conflicting, missing, and + forged tokens stay separate and produce explicit coverage states. No + timestamp, index, or ad-unit-path heuristic is used. +- `TraceSlotCorrelationV1` is emitted only at the existing recorder's exact + opportunity-to-cycle binding, is capped and evicted deterministically, + contains only opaque tokens plus runtime/request numbers, and does not alter + `GptDiagnosticsExportV1`. +- Source presentation distinguishes server-owned SSAT/page-bids/auction API + facts from browser-observed publisher refresh, Prebid refresh, competing, and + unattributed request paths. No fixture turns intent or a GPT fill into a + winner assertion. +- Server auction-local, request-relative unavailable, and browser/GPT timings + render in separate labeled groups and are never combined arithmetically. - Trace projection replaces the nested GPT pathname with `/[redacted]`, rejects any other stored value, emits `TraceGptDiagnosticsV1`, applies deterministic ordering/truncation, and records exact omission counts in a worst-case 512 KiB @@ -971,8 +1429,20 @@ requests, auctions, targeting, or creative rendering. - Cross-site top-level GET, form POST, and fetch attempts cannot enable or end tracing. - A real fixture reload activates diagnostics and captures multiple slots. +- Initial-navigation SSAT fixtures cover selected, no-candidate, + selected-unrenderable, skipped, dispatch-failed, execution-failed, and + abandoned outcomes without requiring a winning bid. +- SPA page-bids and both TSJS `/auction` callers consume the optional + `trace_auction` member before ad initialization/bid parsing; inactive + responses have no member, and malformed members do not affect bids. +- Initial seams pass the optional transport as the scheduler's third argument, + SPA JSON uses the exact optional top-level member, and old-scheduler/absent + transport fixtures preserve ad initialization while reporting `not_observed`. - `View trace results` navigates in the same tab and renders the captured request context and slot evidence. +- A correlated fixture renders the chain `server auction -> GPT -> creative`, + while unmatched server, client-side refresh, competing, and transport-failure + fixtures show honest independent evidence and `Unknown` where appropriate. - Empty, filled, ambiguous, no-candidate, and unattributed slot states remain distinct. - Reloading the trace page retains an unexpired same-tab report. @@ -996,7 +1466,10 @@ requests, auctions, targeting, or creative rendering. - Immutable asset fixtures prove published v1 bytes never change; changed bytes require a new URL referenced by the shell. - Inactive publisher traffic has no trace assets, storage access, listeners, or - cache-policy change. + cache-policy change, diagnostic token generation, or trace-auction response + extension. +- Disabling `trace_page_enabled` removes every new capture behavior even when a + technical `?ts_console=1` session leaves a valid diagnostics cookie present. ### 14.5 Manual acceptance @@ -1007,6 +1480,9 @@ fixture: - Touch targets, scrolling, zoom, safe areas, download, copy, and native share behavior are usable. - The user can distinguish setup information from captured-page information. +- The user can distinguish `SSAT/Trusted Server auction ran`, `browser refresh +observed`, `GPT filled/rendered`, and `Unknown` without understanding internal + request-path names. - A failed share or download does not lose the visible report. ## 15. Rollout and observability @@ -1032,58 +1508,74 @@ fixture: and a cross-site GET cannot activate tracing. 3. The setup page accurately explains that the problem must be reproduced after activation. -4. A subsequent real publisher-page reload captures redacted request context - and existing TS Console evidence without altering ad behavior. +4. A subsequent real publisher-page reload captures redacted request context, + live server-auction evidence, and existing TS Console evidence without + altering ad behavior. 5. `View trace results` transfers one bounded, runtime-validated snapshot in the supported same-tab journey and opens the report page without server-side storage or claims of browser-storage isolation. -6. The report separates network, cookie health, auction/render evidence, and - coverage/unknowns. +6. The report separates network, cookie health, server auction, GPT delivery, + creative rendering, and coverage/unknowns. 7. JSON export contains the same versioned allowlisted information shown on the page. 8. No raw cookies, user IDs, full IPs, consent strings, exact page paths, query - strings, fingerprints, internal auction IDs, targeting, or creative payloads - appear in trace HTML, browser storage, logs, or export. + strings, fingerprints, internal auction IDs, provider/bidder/seat names, + prices, targeting, or creative identifiers/payloads appear in trace HTML, + browser storage, trace-specific logs, or export. 9. Trace HTML and active publisher pages remain terminally private/no-store. -10. Missing platform fields, incomplete auction correlation, storage failure, - and unavailable share APIs degrade honestly without affecting advertising. +10. Missing platform fields, failed evidence projection/transport, incomplete + auction correlation, storage failure, and unavailable share APIs degrade + honestly without affecting advertising. 11. The full report is usable at 320 CSS pixels and with keyboard/screen-reader navigation. -12. Version one accepts exactly `TraceReportV1` with - `TraceGptDiagnosticsV1`, sourced only from `GptDiagnosticsExportV1`; #1081 - and #1074/#1076 are optional additive follow-ups rather than release gates. -13. Every rendered and exported report is identified as browser-observed and - unverified, and hostile storage content cannot create executable HTML or +12. Version one accepts exactly `TraceReportV1` with server-produced + `TraceAuctionEvidenceV1`, browser-produced `TraceSlotCorrelationV1`, and + `TraceGptDiagnosticsV1`; only the latter is sourced from + `GptDiagnosticsExportV1`. #1081 and #1074/#1076 are optional additive + follow-ups rather than release gates. +13. Every rendered and exported report is identified as browser-carried and + unverified; server-produced and browser-observed entries retain distinct + provenance, and hostile storage content cannot create executable HTML or unbounded DOM output. 14. Enable/end operations expose partial failure honestly and are safe to retry; the UI does not claim server-observed cookie state without the follow-up state request or claim that local data was cleared when deletion fails. +15. A report can show that initial-navigation SSAT, SPA page-bids, or the + Trusted Server auction API ran; show its bounded provider and per-slot + outcome; show subsequent GPT/creative evidence; and display `Unknown` + rather than inventing a client-side winner or an unsupported correlation. ## 17. Implementation sequencing -This design is one product flow, but its implementation is split into three -independently reviewable plans and preferably three PRs: +This design is one product flow, but its implementation is split into four +independently reviewable plans and preferably four PRs: 1. **Reserved route and privacy foundation:** configuration, shared early-route classification, same-origin enable/end lifecycle, bounded cookie-health inspection, base request-context schema, projection of already populated `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not add speculative new platform fields in this change. -2. **Browser handoff and viewer:** integrate current +2. **Live server-auction evidence:** introduce the public diagnostic auction and + slot tokens, project `TraceAuctionEvidenceV1` at the live observation + boundary, transport it through initial navigation, page-bids, and both + TSJS `/auction` callers, emit `TraceSlotCorrelationV1` from the existing GPT + recorder binding, and prove ordinary bid behavior is unchanged on + absent/invalid evidence. +3. **Browser handoff and viewer:** integrate current `GptDiagnosticsExportV1`, project `TraceGptDiagnosticsV1`, construct and strictly validate `TraceReportV1`, implement the same-tab workflow, combined - direct/storage exports, mobile viewer, copy/share, expiry, clearing, and - browser/accessibility tests. -3. **Optional network enrichment and future schemas:** add HTTP-version, POP, + server/GPT/creative correlation, direct/storage exports, mobile viewer, + copy/share, expiry, clearing, and browser/accessibility tests. +4. **Optional network enrichment and future schemas:** add HTTP-version, POP, ASN, or other fields only from SDK-verified platform sources with explicit bounds. Adopt #1081 or #1074/#1076 later through a separately reviewed versioned compatibility change. Each plan must include its own adapter, privacy, cache, and failure tests. The -implementation must not claim completion of #1081 or the timing work as part of -#1050. Version one ships with current observed auction/render evidence and -labels unavailable fields honestly. +implementation must not claim completion of #1081 or request-relative timing as +part of #1050. Version one ships with minimal live server-auction evidence plus +current GPT/render evidence and labels unavailable fields honestly. ## 18. Rejected alternatives @@ -1148,8 +1640,22 @@ length, history, logging, referrer, and accidental-sharing risks. subdomains. - Browser privacy settings may disable storage, clipboard, download, or share capabilities. -- Current server/browser correlation does not cover every no-bid, skipped, - failed, hidden, unresolved, or direct-auction path. +- A server can report an auction failure only when it can still deliver a + response containing evidence. Network termination or an unreadable `/auction` + response remains a browser-observed transport failure with no server outcome. +- Provider calls are auction-wide. Version one cannot attribute a provider + no-bid, timeout, or error to a specific slot. +- Third-party client-side auction participants, bids, and winners are not + observable. Publisher/Prebid refresh is browser intent, not proof that a + client-side bidder won. +- `/auction` evidence requires the TSJS request to reach the same Trusted Server + host with the active diagnostics cookie. A custom cross-origin auction + endpoint does not inherit this trace session and is shown as unavailable. +- Exact-token correlation can remain unavailable for hidden, unresolved, + competing, or independently initiated GPT cycles. The report preserves both + sides instead of guessing. +- Version one has auction-local server durations and browser/GPT timings, but + request-relative dispatched/resolved/committed milestones await #1076. - Fastly-only transport details do not exist on every adapter. - The current 30-pixel TS Console controls are not sufficient for this mobile report; the endpoint uses independent 44-pixel touch targets. From dc34278c8552d2782b22ab23b4bdae09bdea19e9 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Thu, 20 Aug 2026 10:26:06 -0500 Subject: [PATCH 033/104] Report a reason when an APS creative render fails An APS bid that wins Prebid targeting is served by Ad Manager as a 1x1 universal creative that resizes itself only after the creative draws. Every guard on that render path returned silently, so a slot that never drew was indistinguishable from one that did: Ad Manager reports a non-empty 1x1 render either way, and the tester framework reports "filled". Name the guard that stopped the render instead. The sandboxed renderer document now reports bad_hash, source_mismatch, nonce_mismatch, descriptor_keys, descriptor_fields, descriptor_envelope, and amazon_script_error on the existing failure message. Reporting is one-shot and answers through the parent, never the sender, so an unrelated sender cannot consume the frame's single report or learn anything from it. Traffic that is not shaped like the render handshake stays silent as before. The Universal Creative source labels its own frame_timeout and frame_load_error, and relays whichever reason it holds to the top window. That relay crosses an origin boundary, so reasons resolve through a null-prototype allowlist that drops anything unlisted and leaves a hostile __proto__ or constructor as undefined. Reasons are fixed categories. A descriptor is never echoed back. --- .../src/integrations/aps.rs | 89 ++++++++++---- .../trusted-server-js/lib/src/core/types.ts | 29 ++++- .../lib/src/integrations/aps/render.ts | 67 ++++++++++- .../lib/test/integrations/aps/render.test.ts | 109 ++++++++++++++++++ 4 files changed, 266 insertions(+), 28 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/aps.rs b/crates/trusted-server-core/src/integrations/aps.rs index 581e5c200..23f092b48 100644 --- a/crates/trusted-server-core/src/integrations/aps.rs +++ b/crates/trusted-server-core/src/integrations/aps.rs @@ -57,42 +57,54 @@ const APS_RENDERER_DOCUMENT: &str = r#" var match=/^#tsaps=([A-Za-z0-9_-]{22,128})$/.exec(location.hash); var expected=match&&match[1]; try{history.replaceState(null,'',location.pathname+location.search);}catch(_error){} -if(!expected)return; +var reported=false; +function report(reason,nonce){ + if(reported)return; + reported=true; + try{parent.postMessage({message:'trusted-server/aps/renderer-failed',nonce:nonce,reason:reason},'*');}catch(_error){} +} +if(!expected){report('bad_hash');return;} function keys(value,expectedKeys){ if(!value||typeof value!=='object'||Array.isArray(value))return false; var actual=Object.keys(value).sort(); return actual.length===expectedKeys.length&&actual.every(function(key,index){return key===expectedKeys[index];}); } -function validRenderer(renderer){ +function rendererProblem(renderer){ if(!keys(renderer,['aaxResponse','accountId','bidId','creativeId','creativeUrl','height','tagType','type','version','width'])&& - !keys(renderer,['aaxResponse','accountId','bidId','creativeUrl','height','tagType','type','version','width']))return false; - if(renderer.type!=='aps'||renderer.version!==1||typeof renderer.accountId!=='string'||!renderer.accountId||new TextEncoder().encode(renderer.accountId).length>1024)return false; - if(typeof renderer.bidId!=='string'||!renderer.bidId||!Number.isInteger(renderer.width)||renderer.width<=0||!Number.isInteger(renderer.height)||renderer.height<=0)return false; - if(Object.prototype.hasOwnProperty.call(renderer,'creativeId')&&(typeof renderer.creativeId!=='string'||!renderer.creativeId||new TextEncoder().encode(renderer.creativeId).length>1024))return false; - if(renderer.tagType!=='iframe'&&renderer.tagType!=='script')return false; - if(typeof renderer.creativeUrl!=='string'||new TextEncoder().encode(renderer.creativeUrl).length>4096)return false; - if(typeof renderer.aaxResponse!=='string'||!renderer.aaxResponse||renderer.aaxResponse.length>349528)return false; + !keys(renderer,['aaxResponse','accountId','bidId','creativeUrl','height','tagType','type','version','width']))return 'descriptor_keys'; + if(renderer.type!=='aps'||renderer.version!==1||typeof renderer.accountId!=='string'||!renderer.accountId||new TextEncoder().encode(renderer.accountId).length>1024)return 'descriptor_fields'; + if(typeof renderer.bidId!=='string'||!renderer.bidId||!Number.isInteger(renderer.width)||renderer.width<=0||!Number.isInteger(renderer.height)||renderer.height<=0)return 'descriptor_fields'; + if(Object.prototype.hasOwnProperty.call(renderer,'creativeId')&&(typeof renderer.creativeId!=='string'||!renderer.creativeId||new TextEncoder().encode(renderer.creativeId).length>1024))return 'descriptor_fields'; + if(renderer.tagType!=='iframe'&&renderer.tagType!=='script')return 'descriptor_fields'; + if(typeof renderer.creativeUrl!=='string'||new TextEncoder().encode(renderer.creativeUrl).length>4096)return 'descriptor_fields'; + if(typeof renderer.aaxResponse!=='string'||!renderer.aaxResponse||renderer.aaxResponse.length>349528)return 'descriptor_fields'; try{ var url=new URL(renderer.creativeUrl); - if(url.protocol!=='https:'||url.username||url.password)return false; + if(url.protocol!=='https:'||url.username||url.password)return 'descriptor_envelope'; var binary=atob(renderer.aaxResponse); - if(binary.length>262144||btoa(binary)!==renderer.aaxResponse)return false; + if(binary.length>262144||btoa(binary)!==renderer.aaxResponse)return 'descriptor_envelope'; var bytes=Uint8Array.from(binary,function(character){return character.charCodeAt(0);}); var decoded=JSON.parse(new TextDecoder('utf-8',{fatal:true}).decode(bytes)); - if(!keys(decoded,['seatbid'])||!Array.isArray(decoded.seatbid)||decoded.seatbid.length!==1)return false; + if(!keys(decoded,['seatbid'])||!Array.isArray(decoded.seatbid)||decoded.seatbid.length!==1)return 'descriptor_envelope'; var seat=decoded.seatbid[0]; - if(!keys(seat,['bid'])||!Array.isArray(seat.bid)||seat.bid.length!==1)return false; + if(!keys(seat,['bid'])||!Array.isArray(seat.bid)||seat.bid.length!==1)return 'descriptor_envelope'; var bid=seat.bid[0]; - if(!keys(bid,['ext','h','id','price','w'])||!keys(bid.ext,['creativeurl','tagtype']))return false; - return bid.id===renderer.bidId&&bid.w===renderer.width&&bid.h===renderer.height&& + if(!keys(bid,['ext','h','id','price','w'])||!keys(bid.ext,['creativeurl','tagtype']))return 'descriptor_envelope'; + if(bid.id===renderer.bidId&&bid.w===renderer.width&&bid.h===renderer.height&& bid.ext.creativeurl===renderer.creativeUrl&&bid.ext.tagtype===renderer.tagType&& - typeof bid.price==='number'&&Number.isFinite(bid.price)&&bid.price>=0; - }catch(_error){return false;} + typeof bid.price==='number'&&Number.isFinite(bid.price)&&bid.price>=0)return undefined; + return 'descriptor_envelope'; + }catch(_error){return 'descriptor_envelope';} } function receive(event){ - if(event.source!==parent)return; var message=event.data; - if(!keys(message,['nonce','renderer'])||message.nonce!==expected||!validRenderer(message.renderer))return; + // Stay silent for traffic that is not shaped like the render handshake, so an + // unrelated sender cannot consume this frame's single report. + if(!keys(message,['nonce','renderer']))return; + if(event.source!==parent){report('source_mismatch');return;} + if(message.nonce!==expected){report('nonce_mismatch');return;} + var problem=rendererProblem(message.renderer); + if(problem){report(problem,message.nonce);return;} removeEventListener('message',receive); var acceptedNonce=expected; expected=''; @@ -107,7 +119,7 @@ function receive(event){ var script=document.createElement('script'); script.src='https://client.aps.amazon-adsystem.com/prebid-creative.js'; script.onload=function(){parent.postMessage({message:'trusted-server/aps/renderer-ready',nonce:acceptedNonce},'*');}; - script.onerror=function(){parent.postMessage({message:'trusted-server/aps/renderer-failed',nonce:acceptedNonce},'*');}; + script.onerror=function(){report('amazon_script_error',acceptedNonce);}; document.head.appendChild(script); } addEventListener('message',receive); @@ -2613,4 +2625,41 @@ mod tests { assert!(APS_RENDERER_CSP.contains("sandbox allow-forms")); assert!(!APS_RENDERER_CSP.contains("allow-same-origin")); } + + #[test] + fn renderer_document_reports_a_reason_for_every_silent_guard() { + for reason in [ + "bad_hash", + "source_mismatch", + "nonce_mismatch", + "descriptor_keys", + "descriptor_fields", + "descriptor_envelope", + "amazon_script_error", + ] { + assert!( + APS_RENDERER_DOCUMENT.contains(reason), + "renderer document should report a `{reason}` reason instead of returning silently" + ); + } + + // Reasons travel on the existing failure message rather than a new channel. + assert!( + APS_RENDERER_DOCUMENT.contains("reason:reason"), + "should attach the reason to the failure message" + ); + + // A reason is a fixed category, never a copy of the rejected descriptor. + assert!(!APS_RENDERER_DOCUMENT.contains("JSON.stringify(renderer)")); + assert!(!APS_RENDERER_DOCUMENT.contains("reason:message")); + + // Reporting is one-shot so a hostile sender cannot flood the parent. + assert!( + APS_RENDERER_DOCUMENT.contains("if(reported)return"), + "should report at most one reason per frame" + ); + + // A foreign sender is answered through the parent, never the sender. + assert!(!APS_RENDERER_DOCUMENT.contains("event.source.postMessage")); + } } diff --git a/crates/trusted-server-js/lib/src/core/types.ts b/crates/trusted-server-js/lib/src/core/types.ts index 03ff0aca2..05af3fff6 100644 --- a/crates/trusted-server-js/lib/src/core/types.ts +++ b/crates/trusted-server-js/lib/src/core/types.ts @@ -181,12 +181,37 @@ export type GptDiagnosticsTrustedServerOpportunity = | 'unrenderable_candidate' | 'no_candidate'; -/** A safe failure category observed while obtaining or posting creative markup. */ +/** + * A safe failure category observed while obtaining or posting creative markup. + * + * The `aps_` members cover the APS Universal Creative render path, where a + * blank slot is otherwise indistinguishable from a filled one: Ad Manager + * reports a non-empty 1x1 render whether or not the creative ever drew. Each + * member names the exact guard that stopped the render. + */ export type GptDiagnosticsCreativeFailure = | 'missing_render_source' | 'cache_fetch_failed' | 'invalid_cache_payload' - | 'response_post_failed'; + | 'response_post_failed' + // Reported by the sandboxed renderer document and relayed by the creative. + | 'aps_bad_hash' + | 'aps_nonce_mismatch' + | 'aps_source_mismatch' + | 'aps_descriptor_keys' + | 'aps_descriptor_fields' + | 'aps_descriptor_envelope' + | 'aps_runner_script_error' + // Observed by the Universal Creative source around its renderer frame. + | 'aps_frame_timeout' + | 'aps_frame_load_error' + | 'aps_frame_reported_failure' + | 'aps_unknown' + // Observed on the Trusted Server side of the capability handshake. + | 'aps_consumed_tombstone' + | 'aps_source_not_in_ad_unit' + | 'aps_missing_renderer_url' + | 'aps_tombstone_capacity'; /** Delivery evidence derived for a GPT request cycle. */ export type GptDiagnosticsDelivery = diff --git a/crates/trusted-server-js/lib/src/integrations/aps/render.ts b/crates/trusted-server-js/lib/src/integrations/aps/render.ts index adec0b036..4d20d23a7 100644 --- a/crates/trusted-server-js/lib/src/integrations/aps/render.ts +++ b/crates/trusted-server-js/lib/src/integrations/aps/render.ts @@ -1,6 +1,11 @@ import { log } from '../../core/log'; import { findSlot } from '../../core/render'; -import type { ApsPrebidRendererEntry, ApsRendererV1, TsjsApi } from '../../core/types'; +import type { + ApsPrebidRendererEntry, + ApsRendererV1, + GptDiagnosticsCreativeFailure, + TsjsApi, +} from '../../core/types'; export const APS_RENDERER_PATH = '/integrations/aps/renderer'; export const APS_RENDERING_MODE_ATTRIBUTE_NAME = 'data-ts-aps-rendering-mode'; @@ -32,6 +37,55 @@ const activeFrames = new WeakMap(); const pendingFrameCancels = new WeakMap void>(); const RENDERER_READY_MESSAGE = 'trusted-server/aps/renderer-ready'; const RENDERER_FAILED_MESSAGE = 'trusted-server/aps/renderer-failed'; +/** + * Message the Universal Creative frame relays to the top window when an APS + * render never completes. + * + * The creative frame is cross-origin, so the top-window listener treats every + * field as untrusted and validates the reason against + * [`APS_RENDER_FAILURE_REASONS`] before recording it. The relay is + * diagnostics-only and never influences creative delivery. + */ +export const APS_RENDER_FAILED_MESSAGE = 'trusted-server/aps/render-failed'; + +/** + * Wire reasons the render path can emit, mapped onto safe diagnostic categories. + * + * Built on a null prototype so a hostile `__proto__`, `constructor`, or + * `toString` relayed by the cross-origin creative frame resolves to `undefined` + * rather than an inherited member. + */ +const APS_RENDER_FAILURE_REASONS: Readonly> = + Object.freeze( + Object.assign(Object.create(null) as Record, { + bad_hash: 'aps_bad_hash', + nonce_mismatch: 'aps_nonce_mismatch', + source_mismatch: 'aps_source_mismatch', + descriptor_keys: 'aps_descriptor_keys', + descriptor_fields: 'aps_descriptor_fields', + descriptor_envelope: 'aps_descriptor_envelope', + amazon_script_error: 'aps_runner_script_error', + frame_timeout: 'aps_frame_timeout', + frame_load_error: 'aps_frame_load_error', + frame_reported_failure: 'aps_frame_reported_failure', + unknown: 'aps_unknown', + } as const) + ); + +/** + * Resolve a relayed render failure reason to a safe diagnostic category. + * + * Returns `undefined` for anything not on the allowlist, so an unrecognized or + * hostile value from the cross-origin creative frame is dropped instead of + * being recorded. + * + * @example + * apsRenderFailureReason('frame_timeout'); // 'aps_frame_timeout' + * apsRenderFailureReason('__proto__'); // undefined + */ +export function apsRenderFailureReason(value: unknown): GptDiagnosticsCreativeFailure | undefined { + return typeof value === 'string' ? APS_RENDER_FAILURE_REASONS[value] : undefined; +} const RENDERER_READY_TIMEOUT_MS = 10_000; const MAX_PREBID_RENDERER_ENTRIES = 256; const DEFAULT_PREBID_RENDERER_TTL_SECONDS = 300; @@ -711,12 +765,13 @@ var b=new Uint8Array(16);c.getRandomValues(b);var s="";for(var i=0;i { }); }); +describe('APS render failure reason allowlist', () => { + it('maps every reason the renderer frame and creative source can emit', () => { + expect(apsRenderFailureReason('descriptor_envelope')).toBe('aps_descriptor_envelope'); + expect(apsRenderFailureReason('source_mismatch')).toBe('aps_source_mismatch'); + expect(apsRenderFailureReason('bad_hash')).toBe('aps_bad_hash'); + expect(apsRenderFailureReason('frame_timeout')).toBe('aps_frame_timeout'); + expect(apsRenderFailureReason('amazon_script_error')).toBe('aps_runner_script_error'); + }); + + it('rejects unlisted, inherited, and non-string reasons from the cross-origin frame', () => { + expect(apsRenderFailureReason('not_a_real_reason')).toBeUndefined(); + expect(apsRenderFailureReason('__proto__')).toBeUndefined(); + expect(apsRenderFailureReason('constructor')).toBeUndefined(); + expect(apsRenderFailureReason('toString')).toBeUndefined(); + expect(apsRenderFailureReason(42)).toBeUndefined(); + expect(apsRenderFailureReason(undefined)).toBeUndefined(); + expect(apsRenderFailureReason({ toString: () => 'frame_timeout' })).toBeUndefined(); + }); +}); + describe('Universal Creative APS source', () => { it('uses the deployed dynamic renderer protocol and only creates the opaque route frame', () => { expect(APS_UNIVERSAL_CREATIVE_RENDERER_VERSION).toBeGreaterThanOrEqual(4); @@ -803,4 +825,91 @@ describe('Universal Creative APS source', () => { document.body.innerHTML = ''; } }); + + it('relays the frame failure reason to the top window for diagnostics', async () => { + const dynamicWindow = window as unknown as { + render?: (data: Record, helper: unknown, target: Window) => Promise; + }; + const relayed: unknown[] = []; + const capture = (event: MessageEvent): void => { + const data = event.data as { message?: unknown } | undefined; + if (data && data.message === APS_RENDER_FAILED_MESSAGE) relayed.push(data); + }; + window.addEventListener('message', capture); + window.eval(APS_UNIVERSAL_CREATIVE_RENDERER); + + try { + const rendered = dynamicWindow.render!( + { adId: 'example-ad-id', apsRenderer: descriptor(), rendererUrl: apsRendererUrl() }, + undefined, + window + ); + const iframe = document.body.querySelector('iframe')!; + const postMessage = vi.spyOn(iframe.contentWindow!, 'postMessage'); + iframe.dispatchEvent(new Event('load')); + const sent = postMessage.mock.calls[0][0] as { nonce: string }; + + window.dispatchEvent( + new MessageEvent('message', { + data: { + message: 'trusted-server/aps/renderer-failed', + nonce: sent.nonce, + reason: 'descriptor_envelope', + }, + source: iframe.contentWindow, + }) + ); + + await expect(rendered).rejects.toThrow(); + // `postMessage` is delivered on a later task than the rejection microtask. + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(relayed).toEqual([ + { + message: APS_RENDER_FAILED_MESSAGE, + adId: 'example-ad-id', + reason: 'descriptor_envelope', + }, + ]); + } finally { + window.removeEventListener('message', capture); + delete dynamicWindow.render; + document.body.innerHTML = ''; + } + }); + + it('reports a frame timeout when the renderer frame never acknowledges', async () => { + const dynamicWindow = window as unknown as { + render?: (data: Record, helper: unknown, target: Window) => Promise; + }; + const relayed: unknown[] = []; + const capture = (event: MessageEvent): void => { + const data = event.data as { message?: unknown } | undefined; + if (data && data.message === APS_RENDER_FAILED_MESSAGE) relayed.push(data); + }; + window.addEventListener('message', capture); + vi.useFakeTimers(); + window.eval(APS_UNIVERSAL_CREATIVE_RENDERER); + + try { + const rendered = dynamicWindow.render!( + { adId: 'timeout-ad-id', apsRenderer: descriptor(), rendererUrl: apsRendererUrl() }, + undefined, + window + ); + rendered.catch(() => {}); + document.body.querySelector('iframe')!.dispatchEvent(new Event('load')); + + // Async advance so the queued `postMessage` dispatch task also runs. + await vi.advanceTimersByTimeAsync(10_001); + + expect(relayed).toEqual([ + { message: APS_RENDER_FAILED_MESSAGE, adId: 'timeout-ad-id', reason: 'frame_timeout' }, + ]); + } finally { + vi.useRealTimers(); + window.removeEventListener('message', capture); + delete dynamicWindow.render; + document.body.innerHTML = ''; + } + }); }); From fae9e357ce88cf7de8140f3fc442482f41807103 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Thu, 20 Aug 2026 10:32:12 -0500 Subject: [PATCH 034/104] Attribute APS creative delivery in GPT diagnostics The APS capability handshake never told diagnostics anything, so every request cycle on that path reported `delivery: unknown` and no creative failures at all. On a live page that meant 24 of 24 cycles were unattributed while APS bids were winning and rendering blank, which is the state that made this hard to diagnose from the outside. Record the attempt around the handshake. The path runs on the publisher's own Prebid ad units, which never pass through Trusted Server slot mapping, so no creative opportunity exists for them and the store would reject the attempt as `creative_request_without_slot`. Resolve the GPT slot by element ID and record the opportunity first. Each silent return that ends in a blank now names itself: aps_consumed_tombstone, aps_source_not_in_ad_unit, aps_descriptor_fields, aps_tombstone_capacity, and aps_missing_renderer_url. A successful post records a response. Consumed ad IDs carry the attempt they were served under, so a replay, or a failure the creative frame relays after the fact, is attributed to the render it belongs to rather than guessed at. The relay listener treats the creative as untrusted: the reason must resolve through the allowlist, the attempt comes from our own tombstone rather than the message, and it never answers the sender. --- .../lib/src/integrations/aps/render.ts | 29 ++--- .../lib/src/integrations/gpt/index.ts | 97 +++++++++++++++-- .../lib/test/integrations/gpt/ad_init.test.ts | 102 ++++++++++++++++++ 3 files changed, 205 insertions(+), 23 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/aps/render.ts b/crates/trusted-server-js/lib/src/integrations/aps/render.ts index 4d20d23a7..cfe93502c 100644 --- a/crates/trusted-server-js/lib/src/integrations/aps/render.ts +++ b/crates/trusted-server-js/lib/src/integrations/aps/render.ts @@ -57,19 +57,22 @@ export const APS_RENDER_FAILED_MESSAGE = 'trusted-server/aps/render-failed'; */ const APS_RENDER_FAILURE_REASONS: Readonly> = Object.freeze( - Object.assign(Object.create(null) as Record, { - bad_hash: 'aps_bad_hash', - nonce_mismatch: 'aps_nonce_mismatch', - source_mismatch: 'aps_source_mismatch', - descriptor_keys: 'aps_descriptor_keys', - descriptor_fields: 'aps_descriptor_fields', - descriptor_envelope: 'aps_descriptor_envelope', - amazon_script_error: 'aps_runner_script_error', - frame_timeout: 'aps_frame_timeout', - frame_load_error: 'aps_frame_load_error', - frame_reported_failure: 'aps_frame_reported_failure', - unknown: 'aps_unknown', - } as const) + Object.assign( + Object.create(null) as Record, + { + bad_hash: 'aps_bad_hash', + nonce_mismatch: 'aps_nonce_mismatch', + source_mismatch: 'aps_source_mismatch', + descriptor_keys: 'aps_descriptor_keys', + descriptor_fields: 'aps_descriptor_fields', + descriptor_envelope: 'aps_descriptor_envelope', + amazon_script_error: 'aps_runner_script_error', + frame_timeout: 'aps_frame_timeout', + frame_load_error: 'aps_frame_load_error', + frame_reported_failure: 'aps_frame_reported_failure', + unknown: 'aps_unknown', + } as const + ) ); /** diff --git a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts index 89b480c6f..cf49d654d 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts @@ -8,8 +8,10 @@ import type { TsjsApi, } from '../../core/types'; import { + APS_RENDER_FAILED_MESSAGE, APS_UNIVERSAL_CREATIVE_RENDERER, APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, + apsRenderFailureReason, apsRendererUrl, dispatchApsRendering, consumeApsPrebidRenderer, @@ -1597,11 +1599,49 @@ function safelyRecordCreativeFailure( } } +/** + * Open a diagnostics attempt for an APS capability handshake. + * + * The APS path runs on the publisher's own Prebid ad units, which never pass + * through Trusted Server slot mapping, so no creative opportunity has been + * recorded for them. Without one the store rejects the attempt as + * `creative_request_without_slot` and the request cycle stays `unknown`, + * leaving a blank APS render indistinguishable from a delivered one. + */ +function beginApsCreativeAttempt(adUnitCode: string): number | undefined { + try { + const pubads = window.googletag?.pubads?.(); + const slot = pubads ? findGptSlotByElementId(pubads, adUnitCode) : undefined; + if (slot) { + window.tsjs?.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( + slot, + adUnitCode, + 'renderable_candidate' + ); + } + } catch { + // Diagnostics must not alter creative delivery. + } + return safelyRecordCreativeRequest(adUnitCode); +} + +/** + * A consumed APS ad ID, retained as a security tombstone. + * + * `attemptId` carries the diagnostics attempt the capability was served under so + * a later replay, or a failure relayed by the creative frame, is attributed to + * the render it belongs to. + */ +interface ApsConsumedTombstone { + expiresAt: number; + attemptId?: number; +} + /** Maximum number of consumed APS Prebid IDs retained as security tombstones. */ const MAX_CONSUMED_PREBID_APS_IDS = 256; function pruneConsumedPrebidApsIds( - consumedIds: Map, + consumedIds: Map, now: number ): void { for (const [adId, consumed] of consumedIds) { @@ -1610,7 +1650,7 @@ function pruneConsumedPrebidApsIds( } function hasConsumedPrebidApsIdCapacity( - consumedIds: Map, + consumedIds: Map, adId: string ): boolean { if (consumedIds.has(adId) || consumedIds.size < MAX_CONSUMED_PREBID_APS_IDS) return true; @@ -1620,11 +1660,12 @@ function hasConsumedPrebidApsIdCapacity( } function recordConsumedPrebidApsId( - consumedIds: Map, + consumedIds: Map, adId: string, - expiresAt: number + expiresAt: number, + attemptId: number | undefined ): void { - consumedIds.set(adId, { expiresAt }); + consumedIds.set(adId, { expiresAt, attemptId }); } /** @@ -1658,7 +1699,7 @@ export function installTsRenderBridge(): void { // is scoped to the slot, not the bare adId: hb_adid is not unique per bid, so // keying on it alone would let one slot block a distinct slot's render. const renderingKeys = new Set(); - const consumedPrebidApsIds = new Map(); + const consumedPrebidApsIds = new Map(); // One consumed APS ad ID per slot is sufficient: a newer bid replaces the // slot's old ad ID in `window.tsjs.bids`, so the ownership guard rejects it. const consumedServerApsBySlot = new Map(); @@ -1674,6 +1715,20 @@ export function installTsRenderBridge(): void { return; } + // Diagnostics relayed by the APS Universal Creative frame. The creative is + // cross-origin, so every field is untrusted: the reason must resolve through + // the allowlist and the attempt comes from our own tombstone, never the + // message. Recording only, and it never answers the sender. + if (data['message'] === APS_RENDER_FAILED_MESSAGE) { + const failedAdId = data['adId']; + const reason = apsRenderFailureReason(data['reason']); + if (typeof failedAdId === 'string' && reason !== undefined) { + pruneConsumedPrebidApsIds(consumedPrebidApsIds, Date.now()); + safelyRecordCreativeFailure(consumedPrebidApsIds.get(failedAdId)?.attemptId, reason); + } + return; + } + if (data['message'] !== 'Prebid Request') return; const adId = data['adId'] as string | undefined; if (!adId) return; @@ -1689,6 +1744,7 @@ export function installTsRenderBridge(): void { // other iframe. Letting Prebid's global handler answer a foreign source // would expose the creative despite the slot-bound capability check. e.stopImmediatePropagation(); + safelyRecordCreativeFailure(consumedPrebidAps.attemptId, 'aps_consumed_tombstone'); return; } @@ -1698,11 +1754,27 @@ export function installTsRenderBridge(): void { // Prebid handles ad IDs globally and would otherwise answer a request from // an unrelated iframe when this slot-bound capability rejects it. e.stopImmediatePropagation(); - if (!messageSourceBelongsToAdUnit(e.source, prebidRendererEntry.adUnitCode)) return; + const attemptId = beginApsCreativeAttempt(prebidRendererEntry.adUnitCode); + if (!messageSourceBelongsToAdUnit(e.source, prebidRendererEntry.adUnitCode)) { + safelyRecordCreativeFailure(attemptId, 'aps_source_not_in_ad_unit'); + return; + } const renderer = validateApsRenderer(prebidRendererEntry.renderer); - if (!renderer || !hasConsumedPrebidApsIdCapacity(consumedPrebidApsIds, adId)) return; + if (!renderer) { + safelyRecordCreativeFailure(attemptId, 'aps_descriptor_fields'); + return; + } + if (!hasConsumedPrebidApsIdCapacity(consumedPrebidApsIds, adId)) { + safelyRecordCreativeFailure(attemptId, 'aps_tombstone_capacity'); + return; + } if (!consumeApsPrebidRenderer(adId, prebidRendererEntry)) return; - recordConsumedPrebidApsId(consumedPrebidApsIds, adId, prebidRendererEntry.expiresAt); + recordConsumedPrebidApsId( + consumedPrebidApsIds, + adId, + prebidRendererEntry.expiresAt, + attemptId + ); const markUsed = (): void => { try { @@ -1717,7 +1789,10 @@ export function installTsRenderBridge(): void { source: e.source, trustedServer: (validatedRenderer) => { const rendererUrl = apsRendererUrl(); - if (!rendererUrl) return false; + if (!rendererUrl) { + safelyRecordCreativeFailure(attemptId, 'aps_missing_renderer_url'); + return false; + } try { port.postMessage( JSON.stringify({ @@ -1731,9 +1806,11 @@ export function installTsRenderBridge(): void { height: validatedRenderer.height, }) ); + safelyRecordCreativeResponse(attemptId); return true; } catch (err) { log.warn(`[tsjs-gpt] APS Prebid response post failed for '${adId}'`, err); + safelyRecordCreativeFailure(attemptId, 'response_post_failed'); return false; } }, diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts index b7186518b..c04409faf 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts @@ -3034,6 +3034,13 @@ describe('installTsRenderBridge', () => { return iframe.contentWindow!; } + // The APS capability path runs on publisher Prebid ad units, so diagnostics + // resolve the GPT slot by element ID rather than through TS slot mapping. + function stubGoogletagSlot(elementId: string): void { + const slot = { getSlotElementId: () => elementId }; + vi.stubGlobal('googletag', { pubads: () => ({ getSlots: () => [slot] }) }); + } + async function captureBridgeListener(): Promise<(e: MessageEvent) => unknown> { let bridgeListener: ((e: MessageEvent) => unknown) | undefined; const origAdd = window.addEventListener.bind(window); @@ -3339,6 +3346,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -3378,6 +3386,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -3457,6 +3466,96 @@ describe('installTsRenderBridge', () => { foreignIframe.remove(); }); + it('records a creative attempt for a registered APS renderer so delivery is attributable', async () => { + const renderer = apsRenderer(); + const prebidAdId = 'prebid-diagnostics-ad-id'; + const recordTrustedServerOpportunity = vi.fn(); + const recordTrustedServerCreativeRequest = vi.fn(() => 7); + const recordTrustedServerCreativeResponse = vi.fn(); + const recordTrustedServerCreativeFailure = vi.fn(); + (window as TestWindow).tsjs.gptDiagnosticsRecorder = { + recordTrustedServerOpportunity, + recordTrustedServerCreativeRequest, + recordTrustedServerCreativeResponse, + recordTrustedServerCreativeFailure, + } as never; + (window as TestWindow).tsjs.apsPrebidRenderers = { + [prebidAdId]: { + adUnitCode: 'div-header', + renderer, + registeredAt: Date.now(), + expiresAt: Date.now() + 60_000, + markUsed: vi.fn(), + }, + }; + + try { + stubGoogletagSlot('div-header'); + const bridgeListener = await captureBridgeListener(); + const portMessages: string[] = []; + bridgeListener( + Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: prebidAdId }), + ports: [{ postMessage: (message: string) => portMessages.push(message) }], + source: createTrustedSlotIframe(), + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent + ); + + expect(portMessages).toHaveLength(1); + expect(recordTrustedServerOpportunity.mock.calls[0]?.slice(1, 3)).toEqual([ + 'div-header', + 'renderable_candidate', + ]); + expect(recordTrustedServerCreativeRequest).toHaveBeenCalledWith('div-header'); + expect(recordTrustedServerCreativeResponse).toHaveBeenCalledWith(7); + expect(recordTrustedServerCreativeFailure).not.toHaveBeenCalled(); + } finally { + delete (window as TestWindow).tsjs.gptDiagnosticsRecorder; + } + }); + + it('records the tombstone reason when a consumed APS ad ID is replayed', async () => { + const renderer = apsRenderer(); + const prebidAdId = 'prebid-replayed-ad-id'; + const recordTrustedServerCreativeFailure = vi.fn(); + (window as TestWindow).tsjs.gptDiagnosticsRecorder = { + recordTrustedServerOpportunity: vi.fn(), + recordTrustedServerCreativeRequest: vi.fn(() => 11), + recordTrustedServerCreativeResponse: vi.fn(), + recordTrustedServerCreativeFailure, + } as never; + (window as TestWindow).tsjs.apsPrebidRenderers = { + [prebidAdId]: { + adUnitCode: 'div-header', + renderer, + registeredAt: Date.now(), + expiresAt: Date.now() + 60_000, + markUsed: vi.fn(), + }, + }; + + try { + stubGoogletagSlot('div-header'); + const bridgeListener = await captureBridgeListener(); + const request = (): MessageEvent => + Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: prebidAdId }), + ports: [{ postMessage: () => {} }], + source: createTrustedSlotIframe(), + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent; + + bridgeListener(request()); + recordTrustedServerCreativeFailure.mockClear(); + bridgeListener(request()); + + expect(recordTrustedServerCreativeFailure).toHaveBeenCalledWith(11, 'aps_consumed_tombstone'); + } finally { + delete (window as TestWindow).tsjs.gptDiagnosticsRecorder; + } + }); + it('contract test: fails a registered APS runner without a Universal Creative response or markUsed', async () => { const renderer = apsRenderer(); const prebidAdId = 'native-prebid-decline-ad-id'; @@ -3473,6 +3572,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -3515,6 +3615,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -4033,6 +4134,7 @@ describe('installTsRenderBridge', () => { }); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const postMessage = vi.fn(); From ca46e4f7393826f0e33725299bb08f8bc57ad998 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 4 Sep 2026 12:38:55 -0500 Subject: [PATCH 035/104] Format auction timeline implementation plan --- .../superpowers/plans/2026-08-26-auction-timeline-offsets.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md index 08e14dabc..6dfff6c8c 100644 --- a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md +++ b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md @@ -23,9 +23,11 @@ ### Task 1: RequestTimings marks and snapshot fields **Files:** + - Modify: `crates/trusted-server-core/src/request_timing.rs` **Interfaces:** + - Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option, auction_id: Option, .. }` - [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option` and `auction_id: Option` to `Inner`; initialize `None`. @@ -37,9 +39,11 @@ ### Task 2: Publisher call sites **Files:** + - Modify: `crates/trusted-server-core/src/publisher.rs` **Interfaces:** + - Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site. - [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());` @@ -50,6 +54,7 @@ ### Task 3: Row columns and datasource **Files:** + - Modify: `crates/trusted-server-core/src/access_telemetry.rs` - Modify: `tinybird/datasources/access_logs_raw.datasource` From c59611ec086b8f64782c16e271f9e4637c0a8720 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 7 Sep 2026 16:56:55 -0400 Subject: [PATCH 036/104] Persist the tester cookie for 30 days so it survives browser restarts The set-tester endpoint minted the ts-tester cookie without Max-Age or Expires, making it a session cookie. Safari deletes session cookies when the browser quits, so Safari testers silently fell back to the baseline arm on every restart. Add a 30-day Max-Age; /_ts/clear-tester already expires the cookie explicitly and is unchanged. --- .../trusted-server-adapter-fastly/src/app.rs | 5 +++-- .../trusted-server-core/src/tester_cookie.rs | 19 ++++++++++++++----- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index cf19518ff..000f7f06e 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -2108,8 +2108,9 @@ mod tests { .to_str() .expect("should render set-cookie as utf-8"); assert_eq!( - set_cookie, "ts-tester=true; Domain=.test-publisher.com; Path=/; Secure; SameSite=Lax", - "tester cookie should use publisher.cookie_domain" + set_cookie, + "ts-tester=true; Domain=.test-publisher.com; Path=/; Secure; SameSite=Lax; Max-Age=2592000", + "tester cookie should use publisher.cookie_domain and persist across browser restarts" ); } diff --git a/crates/trusted-server-core/src/tester_cookie.rs b/crates/trusted-server-core/src/tester_cookie.rs index 58de3d2de..18f2a896e 100644 --- a/crates/trusted-server-core/src/tester_cookie.rs +++ b/crates/trusted-server-core/src/tester_cookie.rs @@ -14,11 +14,18 @@ use crate::constants::COOKIE_TS_TESTER; use crate::error::TrustedServerError; use crate::settings::Settings; +/// Lifetime of the tester cookie in seconds (30 days). +/// +/// Without an explicit lifetime the cookie is session-scoped, and Safari +/// deletes session cookies when the browser quits, so testers silently fall +/// back to the baseline arm on their next visit. +const TESTER_COOKIE_MAX_AGE_SECONDS: u32 = 2_592_000; + /// Formats the tester cookie `Set-Cookie` header value. fn format_tester_cookie(domain: &str) -> String { format!( - "{}=true; Domain={}; Path=/; Secure; SameSite=Lax", - COOKIE_TS_TESTER, domain, + "{}=true; Domain={}; Path=/; Secure; SameSite=Lax; Max-Age={}", + COOKIE_TS_TESTER, domain, TESTER_COOKIE_MAX_AGE_SECONDS, ) } @@ -34,7 +41,8 @@ fn format_clear_tester_cookie(domain: &str) -> String { /// /// Returns `404 Not Found` while `[tester_cookie].enabled` is false. When the /// feature is enabled, returns `204 No Content` with `Set-Cookie: ts-tester=true` -/// scoped to `publisher.cookie_domain`. +/// scoped to `publisher.cookie_domain` and persisted for +/// [`TESTER_COOKIE_MAX_AGE_SECONDS`]. /// /// # Errors /// @@ -138,8 +146,9 @@ mod tests { .to_str() .expect("should render set-cookie as utf-8"); assert_eq!( - set_cookie, "ts-tester=true; Domain=.tester.example; Path=/; Secure; SameSite=Lax", - "tester cookie should use publisher.cookie_domain" + set_cookie, + "ts-tester=true; Domain=.tester.example; Path=/; Secure; SameSite=Lax; Max-Age=2592000", + "tester cookie should use publisher.cookie_domain and persist across browser restarts" ); } From bcc1475f7d458addd2c21745f85456302a9c8ed3 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 8 Sep 2026 13:55:24 -0400 Subject: [PATCH 037/104] Bound route templates by allowlist and harden the telemetry config surface Review round 3, the blocking route finding plus the config items: - Publisher route templates now come from an operator allowlist (observability.route_sections, default empty): a first segment that matches an entry and has further depth emits the lowercased allowlist entry as /{section}/*; everything else emits /other/*. The shape heuristics (charset, length, digit bounds) are gone because they could not bound identity: depth-2 first segments are usernames under /{username}/posts shapes and single-segment paths are documents. The emitted value set is now fixed by configuration, so no request-derived byte reaches the row. - Integration-proxy responses carry the registered route pattern verbatim (bounded, integration-defined) instead of a classifier output; the registry stores the pattern at registration. - auction_enabled serializes only when false, so a pushed config cannot silently re-enable auction telemetry on rollback; with a serialization test alongside the observability one. - The secret-store validator is renamed to validate_secret_store_key_name with a key_name parameter: it validates an identifier, never a credential, and the old name tainted the key name as a secret value in CodeQL, lighting up eleven pre-existing log sites. - Docs: the tinybird table gains its three missing rows, max_body_bytes states the 1024 floor the code enforces, the rollback guidance now describes the real compatibility boundary (push a compat config first: drop [observability], access_enabled = false, and enabled = false for access-only deployments), and the fixture uses the example-domain convention. Co-Authored-By: Claude Fable 5 --- .../trusted-server-adapter-fastly/src/app.rs | 41 ++- .../src/access_telemetry.rs | 248 +++++++----------- .../src/integrations/registry.rs | 35 ++- crates/trusted-server-core/src/settings.rs | 83 +++++- docs/guide/configuration.md | 51 ++-- .../2026-08-24-request-phase-timing-design.md | 33 ++- tinybird/fixtures/access_logs_raw.ndjson | 2 +- 7 files changed, 289 insertions(+), 204 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index a12484ec5..b2221de8d 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -881,9 +881,16 @@ async fn dispatch_fallback( // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + // The matched route pattern is an integration-defined literal + // (bounded and content-free by construction), so telemetry keeps it + // verbatim instead of running the request path through the lossy + // publisher classifier. route_metadata = Some(RouteMetadata { route_class: RouteClass::IntegrationProxy, - route_template: publisher_route_template(&path), + route_template: state + .registry + .matched_route_pattern(&method, &path) + .map_or_else(|| "/other/*".to_owned(), str::to_owned), }); state .registry @@ -934,7 +941,10 @@ async fn dispatch_fallback( route_metadata = Some(RouteMetadata { route_class: RouteClass::PublisherHtml, - route_template: publisher_route_template(&path), + route_template: publisher_route_template( + &path, + &state.settings.observability.route_sections, + ), }); // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -1517,7 +1527,7 @@ mod tests { NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, RouteClass, RouteMetadata, TSJS_ROUTE_TEMPLATE, TrustedServerApp, build_per_request_services, build_state_from_settings, handle_publisher_request, - publisher_response_into_streaming_response, publisher_route_template, startup_error_router, + publisher_response_into_streaming_response, startup_error_router, }; use base64::Engine as _; use bytes::Bytes; @@ -2451,8 +2461,8 @@ mod tests { .expect("integration-proxy fallback responses should carry RouteMetadata"); assert_eq!(metadata.route_class, RouteClass::IntegrationProxy); assert_eq!( - metadata.route_template, - publisher_route_template("/integrations/prebid/bundle.js") + metadata.route_template, "/integrations/prebid/bundle.js", + "should carry the registered route pattern verbatim, not a classifier output" ); } @@ -2466,7 +2476,10 @@ mod tests { .get::() .expect("publisher fallback responses should carry RouteMetadata"); assert_eq!(metadata.route_class, RouteClass::PublisherHtml); - assert_eq!(metadata.route_template, "/news/*"); + assert_eq!( + metadata.route_template, "/other/*", + "the default empty section allowlist should collapse publisher paths" + ); } #[test] @@ -3853,6 +3866,22 @@ mod tests { ); } + #[test] + fn server_timing_absent_when_no_cache_control_header_exists() { + // The fail-closed case: absence of Cache-Control is not evidence of + // privacy, so emission must be suppressed rather than defaulted on. + let mut response = response_builder() + .body(Body::empty()) + .expect("should build a response with no cache-control header"); + + crate::apply_server_timing_header(&mut response, &RequestTimings::new(), true); + + assert!( + response_header(&response, "server-timing").is_none(), + "should not emit when the response carries no Cache-Control at all" + ); + } + #[test] fn preexisting_server_timing_values_survive() { let mut response = response_builder() diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index aaad46633..e468b109c 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -12,20 +12,6 @@ use serde_json::json; use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; -/// Maximum length of a publisher path's first segment before -/// [`publisher_route_template`] rejects it to `/other/*`. Longer segments -/// are opaque-identifier or slug shaped (a UUID is 36 characters), and a -/// truncated prefix of either would still be identifying, so the segment -/// is rejected whole rather than truncated. -const MAX_SEGMENT_LEN: usize = 32; - -/// Maximum number of ASCII digits in a publisher path's first segment -/// before [`publisher_route_template`] rejects it to `/other/*`. Hex ids, -/// base36 ids, and reset tokens are digit-heavy; real section names carry -/// at most a year (`2026`) or a small version number, so a segment with -/// more digits than this is treated as an identifier, not a name. -const MAX_SEGMENT_DIGITS: usize = 7; - /// Normalizes an HTTP method token into the bounded set of values stored in /// the `method` `LowCardinality` column. /// @@ -127,46 +113,43 @@ pub struct RouteMetadata { } /// Normalizes a publisher-fallback request path into a bounded, -/// content-free route template. -/// -/// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, plus `/*`, only when the path has at least two segments: -/// depth is what makes the first segment a section name (`/news/*`) rather -/// than the document itself. The root path `/` maps to itself. Everything -/// else is rejected to `/other/*` — outright, never filtered or truncated, -/// so no fragment of a rejected path ever reaches the row: +/// content-free route template using an operator-configured allowlist of +/// section names. /// -/// - single-segment paths (`/my-post-title` under a `/%postname%/` -/// permalink structure is a per-article slug that no shape heuristic -/// can separate from a section name); -/// - an empty first segment, or one containing any character outside the -/// allowlist after lowercasing (an email address, a search phrase); -/// - a first segment longer than [`MAX_SEGMENT_LEN`] characters (a -/// truncated prefix of a UUID or token would still be identifying); or -/// - a first segment with more than [`MAX_SEGMENT_DIGITS`] ASCII digits -/// (hex ids, base36 ids, and reset tokens are digit-heavy; section -/// names carry at most a year or a version number). +/// The root path `/` maps to itself. A path whose first segment matches an +/// entry in `sections` (ASCII case-insensitive) and that has at least one +/// further segment maps to `/{section}/*`, emitting the lowercased +/// allowlist entry rather than anything taken from the request. Every +/// other path maps to `/other/*`. /// -/// This is deliberately coarser than the auction-telemetry path -/// normalizer, which redacts long tokens but preserves short identifiers -/// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. +/// This construction makes the template content-free by definition: the +/// set of emitted values is exactly `{"/", "/other/*"}` plus one +/// `/{section}/*` per configured entry, so no request-derived byte ever +/// reaches the row and cardinality is bounded by operator configuration. +/// With the default empty allowlist, every publisher path collapses to +/// `/other/*`. Depth alone was rejected as a signal in review: under +/// `/{username}/posts` shapes the first segment is user data, and under +/// single-segment permalink structures it is the document itself. /// /// # Examples /// /// ``` /// use trusted_server_core::access_telemetry::publisher_route_template; /// -/// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); -/// assert_eq!(publisher_route_template("/"), "/"); -/// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// let sections = vec!["news".to_owned()]; +/// assert_eq!( +/// publisher_route_template("/news/some-article-slug", §ions), +/// "/news/*" +/// ); +/// assert_eq!(publisher_route_template("/", §ions), "/"); /// assert_eq!( -/// publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), +/// publisher_route_template("/alice/orders", §ions), /// "/other/*" /// ); +/// assert_eq!(publisher_route_template("/news/x", &[]), "/other/*"); /// ``` #[must_use] -pub fn publisher_route_template(path: &str) -> String { +pub fn publisher_route_template(path: &str, sections: &[String]) -> String { if path == "/" { return "/".to_owned(); } @@ -178,31 +161,18 @@ pub fn publisher_route_template(path: &str) -> String { }; let has_more_depth = !rest.is_empty(); - let lowered = first_segment.to_ascii_lowercase(); - let is_allowlisted = !lowered.is_empty() - && lowered - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); - let within_length = lowered.chars().count() <= MAX_SEGMENT_LEN; - let digit_count = lowered.chars().filter(char::is_ascii_digit).count(); - - if !is_allowlisted || !within_length || digit_count > MAX_SEGMENT_DIGITS { - return "/other/*".to_owned(); - } - - if has_more_depth { - format!("/{lowered}/*") - } else { - // A single-segment path's first segment is the document, not a - // section: `/my-post-title` under WordPress `/%postname%/` is a - // per-article slug, and no shape heuristic can separate it from a - // section name. Only depth >= 2 makes the first segment a section. - "/other/*".to_owned() + let matched_section = sections + .iter() + .find(|section| section.eq_ignore_ascii_case(first_segment)); + match matched_section { + Some(section) if has_more_depth => format!("/{}/*", section.to_ascii_lowercase()), + _ => "/other/*".to_owned(), } } /// A point-in-time view of the access-log dimensions for one response, -/// captured unconditionally at the `Server-Timing` freeze point. +/// captured at the `Server-Timing` freeze point when access telemetry is +/// enabled (a disabled deployment skips the build entirely). /// /// Built from typed response extensions ([`RouteMetadata`], the geo /// lookup state, and the template-cache response state) plus adapter-owned @@ -358,120 +328,84 @@ mod tests { } #[test] - fn publisher_paths_normalize_to_coarse_templates() { + fn publisher_route_template_emits_only_allowlisted_sections() { + let sections = vec!["news".to_owned(), "Sports".to_owned()]; assert_eq!( - publisher_route_template("/news/some-article-slug"), + publisher_route_template("/news/some-article-slug", §ions), "/news/*" ); - assert_eq!(publisher_route_template("/"), "/"); - assert_eq!( - publisher_route_template("/user@example.com/profile"), - "/other/*", - "should reject non-allowlisted characters" - ); - assert_eq!( - publisher_route_template(&format!("/{}", "a".repeat(500))), - "/other/*", - "should reject overlong segments whole rather than truncate" - ); - assert_eq!(publisher_route_template("/search terms here"), "/other/*"); - } - - #[test] - fn publisher_route_template_rejects_opaque_identifier_segments() { - // Every row here passes the character allowlist (`[a-z0-9_-]` is - // exactly what UUIDs, hex ids, and tokens are built from) and must - // be caught by the length and digit-count bounds instead. A - // truncated prefix of any of these would still be identifying, so - // rejection must be whole-segment. - assert_eq!( - publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), - "/other/*", - "should reject a UUID (36 chars) by length" - ); - assert_eq!( - publisher_route_template("/550e8400-e29b-41d4-a716-446655440000/profile"), - "/other/*", - "should reject a UUID first segment on deeper paths too" - ); - assert_eq!( - publisher_route_template("/8f3a9c2b1d4e5f6a7b8c9d0e1f2a3b4c"), - "/other/*", - "should reject a 32-char hex id by digit count" - ); assert_eq!( - publisher_route_template(&format!("/{}", "a1".repeat(32))), - "/other/*", - "should reject a 64-char token by length" + publisher_route_template("/NEWS/some-article-slug", §ions), + "/news/*", + "matching should be case-insensitive on the request side" ); assert_eq!( - publisher_route_template("/reset-password-token-9f2b1c7d4e8a"), - "/other/*", - "should reject a reset token by length" + publisher_route_template("/sports/scores/today", §ions), + "/sports/*", + "the emitted value should be the lowercased allowlist entry" ); + assert_eq!(publisher_route_template("/", §ions), "/"); assert_eq!( - publisher_route_template("/how-to-treat-my-recent-hiv-diagnosis"), + publisher_route_template("/news", §ions), "/other/*", - "should reject a full article slug by length" + "an allowlisted section with no further depth is the document, not a section" ); - } - - #[test] - fn publisher_route_template_keeps_digit_light_section_names() { - assert_eq!( - publisher_route_template("/2026/08/some-article"), - "/2026/*", - "a year archive segment should pass the digit bound" - ); - assert_eq!( - publisher_route_template("/wp-content/themes/site/app.css"), - "/wp-content/*", - "a hyphenated section name should pass" - ); - } - - #[test] - fn publisher_route_template_rejects_empty_first_segment() { assert_eq!( - publisher_route_template("//double-slash"), + publisher_route_template("/opinion/some-slug", §ions), "/other/*", - "an empty first segment should not be treated as allowlisted" + "a section absent from the allowlist should collapse" ); } #[test] - fn publisher_route_template_rejects_single_segment_paths() { - // A single-segment path's first segment is the document itself - // (WordPress `/%postname%/` puts every article at depth 1), so no - // shape heuristic can separate a slug from a section name; depth - // is the only safe signal. Root-level landing pages pay for this - // deliberately. - assert_eq!(publisher_route_template("/my-post-title"), "/other/*"); - assert_eq!( - publisher_route_template("/my-post-title/"), - "/other/*", - "a trailing slash should not count as depth" - ); - assert_eq!( - publisher_route_template("/1234567"), - "/other/*", - "a numeric post id at the digit boundary should still reject" - ); - assert_eq!(publisher_route_template("/user-8f3a9c2b"), "/other/*"); - assert_eq!( - publisher_route_template("/about"), - "/other/*", - "root-level landing pages reject too; only depth makes a section" - ); + fn publisher_route_template_collapses_everything_by_default() { + // The default allowlist is empty, so no request-derived byte can + // reach the row: the only emitted values are `/` and `/other/*`. + let empty: Vec = Vec::new(); + for path in [ + "/news/some-article-slug", + "/alice/orders", + "/jane-doe/posts", + "/johnsmith1985/settings", + "/how-to-treat-my-hiv-today/comments", + "/abcdefabcdefabcdefabcdefabcdefab/x", + "/8f3a9c2b/x", + "/my-post-title", + "/user@example.com/profile", + "/search terms here", + "//double-slash", + "/550e8400-e29b-41d4-a716-446655440000/profile", + ] { + assert_eq!( + publisher_route_template(path, &empty), + "/other/*", + "should collapse with an empty allowlist: {path}" + ); + } + assert_eq!(publisher_route_template("/", &empty), "/"); } #[test] - fn publisher_route_template_lowercases_before_allowlisting() { - assert_eq!( - publisher_route_template("/News/Article"), - "/news/*", - "should lowercase before validating" - ); + fn publisher_route_template_never_emits_user_data_even_when_allowlisted() { + // Adversarial depth-2 shapes from review: even with sections + // configured, a first segment that is not an exact allowlist match + // collapses, and the emitted value on a match is the allowlist + // entry itself, never request bytes. + let sections = vec!["news".to_owned()]; + for path in [ + "/jane-doe/posts", + "/johnsmith1985/settings", + "/how-to-treat-my-hiv-today/comments", + "/abcdefabcdefabcdefabcdefabcdefab/x", + "/8f3a9c2b/x", + "/newsy/article", + ] { + assert_eq!( + publisher_route_template(path, §ions), + "/other/*", + "should collapse non-allowlisted first segments: {path}" + ); + } } #[test] diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 1acb6fdb6..6d65afef3 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -688,7 +688,17 @@ impl IntegrationRegistrationBuilder { } } -type RouteValue = (Arc, &'static str); +/// Proxy handler, integration id, and the registered route pattern (kept so +/// telemetry can label responses with the integration-defined literal, e.g. +/// `/integrations/prebid/*`, instead of deriving anything from the request +/// path). +type RouteValue = (Arc, &'static str, String); + +/// A test-constructor route entry: method, path pattern, and the proxy with +/// its integration id ([`IntegrationRegistry::from_routes`] fills the +/// pattern into [`RouteValue`] itself). +#[cfg(test)] +type RouteEntry<'a> = (Method, &'a str, (Arc, &'static str)); struct IntegrationRegistryInner { // Method-specific routers for O(log n) lookups @@ -805,7 +815,11 @@ impl IntegrationRegistry { for proxy in registration.proxies { for route in proxy.routes() { - let value = (proxy.clone(), registration.integration_id); + let value = ( + proxy.clone(), + registration.integration_id, + route.path.clone(), + ); // Convert /* wildcard to matchit's {*rest} syntax let matchit_path = if route.path.ends_with("/*") { @@ -894,6 +908,16 @@ impl IntegrationRegistry { self.find_route(method, path).is_some() } + /// The registered route pattern matched by `method` and `path`, if any. + /// + /// Patterns are integration-defined literals (for example + /// `/integrations/prebid/*`), so they are bounded and content-free and + /// safe to store as a telemetry dimension, unlike the request path. + #[must_use] + pub fn matched_route_pattern(&self, method: &Method, path: &str) -> Option<&str> { + self.find_route(method, path).map(|value| value.2.as_str()) + } + /// Return true when at least one integration request filter is /// registered. /// @@ -980,7 +1004,7 @@ impl IntegrationRegistry { services, mut req, } = input; - if let Some((proxy, _)) = self.find_route(method, path) { + if let Some((proxy, _, _)) = self.find_route(method, path) { // Organic proxy handler: generate if needed (best effort). // Only generate for document navigations — subresource requests // may lack consent signals such as the Sec-GPC header. @@ -1296,7 +1320,7 @@ impl IntegrationRegistry { /// # Panics /// /// Panics if route registration fails due to duplicate or invalid paths. - pub fn from_routes(routes: Vec<(Method, &str, RouteValue)>) -> Self { + pub fn from_routes(routes: Vec>) -> Self { let mut get_router = Router::new(); let mut post_router = Router::new(); let mut put_router = Router::new(); @@ -1305,7 +1329,8 @@ impl IntegrationRegistry { let mut head_router = Router::new(); let mut options_router = Router::new(); - for (method, path, value) in routes { + for (method, path, (proxy, integration_id)) in routes { + let value: RouteValue = (proxy, integration_id, path.to_owned()); // Convert /* wildcard to matchit's {*rest} syntax let matchit_path = if path.ends_with("/*") { format!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 216a7fd7a..19951b8d4 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1795,7 +1795,11 @@ pub struct TinybirdSettings { /// configs preserve their current auction-emission behavior after /// upgrading; set `false` to silence auction events while keeping /// `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). - #[serde(default = "default_true")] + /// Serialized only when `false`: older binaries ignore the key rather + /// than reject it, so writing the default `true` into every pushed + /// config would let a rollback silently resume auction telemetry after + /// an operator disabled it. + #[serde(default = "default_true", skip_serializing_if = "is_true")] pub auction_enabled: bool, /// Regional Tinybird API host, without scheme or path. #[serde(default)] @@ -1924,11 +1928,17 @@ impl TinybirdSettings { } if self.auction_enabled { validate_tinybird_dataset(&self.auction_dataset, "tinybird.auction_dataset")?; - validate_tinybird_secret(&self.auction_token_secret, "tinybird.auction_token_secret")?; + validate_secret_store_key_name( + &self.auction_token_secret, + "tinybird.auction_token_secret", + )?; } if self.access_enabled { validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; - validate_tinybird_secret(&self.access_token_secret, "tinybird.access_token_secret")?; + validate_secret_store_key_name( + &self.access_token_secret, + "tinybird.access_token_secret", + )?; if self.access_sample_rate <= 0.0 { return Err(Report::new(TrustedServerError::Configuration { message: "tinybird.access_sample_rate must be > 0 when tinybird.access_enabled is true".to_owned(), @@ -1973,8 +1983,15 @@ fn validate_tinybird_dataset(value: &str, setting: &str) -> Result<(), Report Result<(), Report> { - if value.is_empty() || value.chars().any(char::is_control) { +// Named to make the key-name-vs-secret-value distinction legible to static +// analysis: the argument is a Secret Store KEY NAME (an identifier such as +// `tinybird_access_append_token`), never a credential value, so formatting +// it into an error message discloses nothing. +fn validate_secret_store_key_name( + key_name: &str, + setting: &str, +) -> Result<(), Report> { + if key_name.is_empty() || key_name.chars().any(char::is_control) { return Err(Report::new(TrustedServerError::Configuration { message: format!("{setting} must be a non-empty Secret Store key"), })); @@ -2554,6 +2571,14 @@ pub(crate) const AUCTION_DEBUG_UPSTREAM_METADATA_KEYS: &[&str] = &[ "upstream_message_truncated", ]; +/// `skip_serializing_if` helper: true is the serde default for the fields +/// that use it, so serializing it would only widen the pushed config's +/// rollback surface. +#[allow(clippy::trivially_copy_pass_by_ref)] +fn is_true(value: &bool) -> bool { + *value +} + fn default_true() -> bool { true } @@ -2714,6 +2739,13 @@ pub struct ObservabilitySettings { /// timing. Defaults to `false` (off). #[serde(default)] pub server_timing_enabled: bool, + /// Section names whose publisher paths keep a named route template in + /// access telemetry (`/{section}/*`); everything else collapses to + /// `/other/*`. Matching is ASCII case-insensitive on the first path + /// segment, and a match requires at least one further segment. Defaults + /// to empty, which collapses every publisher path. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub route_sections: Vec, } impl ObservabilitySettings { @@ -4448,6 +4480,47 @@ mod tests { ); } + #[test] + fn auction_enabled_serializes_only_when_disabled() { + let mut settings = create_test_settings(); + assert!(settings.tinybird.auction_enabled, "should default on"); + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("auction_enabled"), + "should omit the default-true key: a rollback must not silently \ + re-enable auction telemetry an operator disabled" + ); + + settings.tinybird.auction_enabled = false; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + toml.contains("auction_enabled = false"), + "should serialize the operator's explicit disable" + ); + } + + #[test] + fn route_sections_serialize_only_when_configured() { + let mut settings = create_test_settings(); + assert!( + settings.observability.route_sections.is_empty(), + "should default to the collapse-everything allowlist" + ); + settings.observability.server_timing_enabled = true; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("route_sections"), + "should omit the empty allowlist so a prior binary can parse the config" + ); + + settings.observability.route_sections = vec!["news".to_owned()]; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + toml.contains("route_sections"), + "should serialize a configured allowlist" + ); + } + #[test] fn test_settings_from_valid_toml() { let toml_str = crate_test_settings_str(); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 2b154b8a3..8cc3f7324 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1926,9 +1926,10 @@ enabling one does not enable the other. ### `[observability]` -| Field | Type | Required | Default | Description | -| ----------------------- | ------- | -------- | ------- | ------------------------------------------------------------------- | -| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | +| Field | Type | Required | Default | Description | +| ----------------------- | -------- | -------- | ------- | --------------------------------------------------------------------------------------------------------- | +| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | +| `route_sections` | String[] | No | `[]` | Section names kept as publisher route templates (`/{section}/*`) in access telemetry; empty collapses all | **Purpose**: Surfaces per-phase request timing (`ts-total` plus recorded phases such as `ts-appbuild`, `ts-filter`, `ts-geo`, `ts-kv`, `ts-origin`, and @@ -1957,6 +1958,12 @@ send access-telemetry rows. server_timing_enabled = true ``` +::: tip The Axum dev server reads this flag once at startup +Unlike the Fastly adapter, which reads settings per request, the Axum dev +server bakes `server_timing_enabled` into its service when it starts. +Flipping the flag there requires a restart to take effect. +::: + ::: warning Client-visible latency disclosure The `Server-Timing` header is sent to every client on eligible responses, not only to operators: browsers expose the values to same-origin JavaScript @@ -1993,12 +2000,15 @@ independent emitters: auction telemetry (`auction_dataset`, `auction_token_secret`) and access telemetry. The keys below cover the access-telemetry sink and the shared enable flags. -| Field | Type | Required | Default | Description | -| -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | -| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | -| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | -| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | -| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | +| Field | Type | Required | Default | Description | +| --------------------- | ------- | ------------------------------------ | ------------------------------ | ------------------------------------------------------------------------------- | +| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | +| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | +| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | +| `access_dataset` | String | Yes, when `access_enabled` | `access_logs_raw` | Access-log Events API datasource name | +| `access_token_secret` | String | Yes, when `access_enabled` | `tinybird_access_append_token` | Secret Store key holding the access APPEND token | +| `max_body_bytes` | Integer | No | `1048576` | Maximum NDJSON request body size; must be at least 1024 | +| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | **Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird sinks separately so that turning on one does not silently turn on (or leave @@ -2007,8 +2017,8 @@ Setting `access_enabled = true` with `access_sample_rate = 0.0` is rejected at config load as an armed-but-silent configuration; use `access_enabled` itself to turn the sink off, not the sample rate. Enabling `access_enabled` also requires the shared transport fields (`enabled`, non-empty `api_host`, -`secret_store`, `access_dataset`, `access_token_secret`, and a positive -`max_body_bytes`) to already be set. +`secret_store`, `access_dataset`, `access_token_secret`, and a +`max_body_bytes` of at least 1024) to already be set. **Example**: @@ -2041,18 +2051,23 @@ the response the reader sees. ### Deploy and rollback ordering -::: warning `Settings` rejects unknown fields; order matters -Both `[observability].server_timing_enabled` and the new `[tinybird]` access -keys are new fields on a config schema that uses `deny_unknown_fields`, so an -older binary fails to load a config that carries them. +::: warning Push a compatibility config before rolling back +The compatibility boundary is uneven. The top-level `Settings` schema uses +`deny_unknown_fields`, so an older binary rejects a config carrying the +`[observability]` table. The nested `[tinybird]` table does not: an older +binary accepts unknown keys there, rejects `access_enabled = true` through +validation, ignores `auction_enabled` entirely, and reads `enabled = true` +as "auction telemetry on". **Deploying**: upgrade the binary first, then push a config containing the new fields second. Never push a config with these fields while a pre-observability binary can still receive it. -**Rolling back**: reverse the order. Remove the `[observability]` table and -any new `[tinybird]` access keys from the config and push that first, then -roll back the binary second. +**Rolling back**: push a compatibility config first, then roll the binary +back. The compatibility config removes the `[observability]` table, sets +`access_enabled = false`, and, for a deployment that only used the access +sink, sets `enabled = false` as well; otherwise the older binary would +interpret the leftover `enabled = true` as enabling auction telemetry. ::: ## Validation diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 2892386f4..3833e81eb 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -85,7 +85,7 @@ auction wait, buffered mode ..... auction_wait_ms (row only; pre-header in th | send_edgezero_response, immediately before into_parts(): mark_headers_ready() snapshot (unconditional) - build AccessTelemetrySnapshot (unconditional) + build AccessTelemetrySnapshot (only when tinybird.access_enabled) append Server-Timing header (flag-gated, only on conclusively private responses) | headers committed; body streams @@ -125,9 +125,11 @@ New module `crates/trusted-server-core/src/request_timing.rs`. - Sharing: `RequestTimings` is a cheap-clone handle, `Arc>`. It crosses three boundaries: adapter entry to core handlers, the streaming body closure (records body-phase spans after the response object has been handed off), and the adapter's - post-send emission read. Access is exclusively `try_lock()`: a contended or - poisoned lock drops the sample immediately rather than waiting, so recording can - never delay a request. + post-send emission read. Access is exclusively `try_lock()`: a contended lock + drops the one sample rather than waiting, so recording can never delay a + request, and a poisoned lock is recovered (the guarded data are plain counters + with no invariant a panic can break) so one panic cannot silence the rest of + the request's timing. - Recording API: `timings.record(Phase::Geo, dur)` and a scope guard `timings.span(Phase::Origin)` that records on drop. Guards use saturating duration math; a non-monotonic reading records zero rather than panicking. The auction-wait @@ -214,7 +216,8 @@ marks middleware finalization, not header commitment, and must not be treated as timing boundary. At the freeze point, in order: `mark_headers_ready()` (unconditional), the -`AccessTelemetrySnapshot` build (unconditional, section 10), then, gated on +`AccessTelemetrySnapshot` build (gated on `tinybird.access_enabled`, +section 10), then, gated on `observability.server_timing_enabled`, append one `Server-Timing` header from `server_timing_value()`. Append semantics, never insert: an origin-supplied Server-Timing survives, and the fronting delivery layer's own entries (`time-elapsed`, @@ -290,12 +293,16 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced restricted to a bounded allowlisted charset, plus `/*` when deeper (for example `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. -- Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` - when it fails the charset allowlist, exceeds 32 characters, carries more than 7 - ASCII digits, or is the only segment in the path. Depth is what makes a first - segment a section name: single-segment paths are documents (WordPress - `/%postname%/` puts every article at depth 1), so they reject wholesale, root - landing pages included. The character allowlist alone does not bound identity (`[a-z0-9_-]` +- Publisher templates come from an operator-configured allowlist of section names + (`observability.route_sections`, default empty), not from the request: a path + whose first segment matches an allowlist entry and that has at least one further + segment emits `/{section}/*` (the lowercased allowlist entry itself); everything + else emits `/other/*`, and the root path emits `/`. This replaces the earlier + shape heuristics (charset, length, digit bounds), which review showed cannot + bound identity: depth-2 first segments are usernames on `/{username}/posts` + shapes, and single-segment paths are documents under `/%postname%/` permalinks. + With the allowlist, the emitted value set is fixed by configuration, so no + request-derived byte ever reaches the row. The character allowlist alone does not bound identity (`[a-z0-9_-]` is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated prefix of any of those is still identifying, so the length and digit bounds reject the segment outright. @@ -364,7 +371,9 @@ ships as a versioned replacement datasource with a cutover, not an in-place edit ## 10. Emission mechanics -- `AccessTelemetrySnapshot`: built unconditionally at the freeze point, before +- `AccessTelemetrySnapshot`: built at the freeze point when + `tinybird.access_enabled` is set (revised in review from the original + unconditional build, so a disabled deployment pays nothing here), before `into_parts()` consumes the response. It captures method, status, route metadata (from the `RouteMetadata` extension), and typed dimension states (`env`, `template_cache_state`, geo country). It exists because nothing else survives to diff --git a/tinybird/fixtures/access_logs_raw.ndjson b/tinybird/fixtures/access_logs_raw.ndjson index 3c82c5ca6..983b3acd1 100644 --- a/tinybird/fixtures/access_logs_raw.ndjson +++ b/tinybird/fixtures/access_logs_raw.ndjson @@ -1 +1 @@ -{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} +{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.example","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} From a1523675a21f5bbd3a327de45a95385d26716fc2 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 8 Sep 2026 13:55:24 -0400 Subject: [PATCH 038/104] Address round-3 telemetry robustness findings - The access sink streams the Tinybird response (the body is never consumed; buffered conversion materializes chunked bodies before the limit check) and newline-terminates rows to match the auction sink's NDJSON framing, with the recording client now asserting both. - Poisoned RequestTimings locks recover via into_inner instead of silently dropping every subsequent sample: the guarded data are plain counters, so one panic cannot blank the header and row for the rest of the request. Module and spec docs updated to stop conflating poisoning with contention. - The geo write-back skips 401 responses through a shared helper: resolve_geo_for_response short-circuits on 401 before consulting the carried state, so the old unconditional write downgraded a carried Resolved to Attempted and cost the row its country. - TimedKvStore forwards exists, so decorating a store with a cheap metadata probe (Spin) no longer downgrades it to the get-and-discard default body; with a contradiction-stub delegation test. - Post-send ordering is owned by run_post_send_steps, which both production sites route through, and the instrumented sequence test drives the real seam: elapsed stamped by send, then pull-sync, then telemetry. - Axum: dev_server_service remains the standard path; new tests pin flag-off suppression and the extension round trip (a phase recorded in the handler must surface as ts-filter in the header); the outer-wrapper rationale is reworded to the terminal-freeze-point argument; the configuration guide notes the flag is read once at startup. - The no-Cache-Control fail-closed case is pinned by a test, and t0's boundary (constructed after the adapter prologue) is documented. Co-Authored-By: Claude Fable 5 --- .../trusted-server-adapter-axum/src/timing.rs | 82 +++++++++++++++-- .../wrangler.integration.generated.toml | 16 ++++ .../trusted-server-adapter-fastly/src/main.rs | 91 ++++++++++++++----- .../src/middleware.rs | 80 +++++++++++++--- .../src/tinybird.rs | 26 +++++- .../src/platform/timed_kv.rs | 68 ++++++++++++++ .../trusted-server-core/src/request_timing.rs | 89 +++++++++++++----- .../plans/2026-08-24-request-phase-timing.md | 3 +- 8 files changed, 385 insertions(+), 70 deletions(-) create mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs index 832823c15..315ebb5ef 100644 --- a/crates/trusted-server-adapter-axum/src/timing.rs +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -9,13 +9,15 @@ //! [`append_server_timing_if_private`](trusted_server_core::request_timing::append_server_timing_if_private). //! //! This wraps *outside* `RouterService` rather than registering as -//! `RouterBuilder::middleware`. A router-generated 404/405 short-circuits -//! `RouterInner::dispatch` before its middleware chain ever runs, so -//! middleware never sees those responses. By the time a response reaches -//! this layer -- after `RouterService::oneshot` inside -//! `EdgeZeroAxumService::call` has already converted any dispatch error into -//! a plain response -- every response is covered uniformly, router-generated -//! or not. +//! `RouterBuilder::middleware` because the tower boundary is the terminal +//! freeze point: by the time a response reaches this layer -- after +//! `RouterService::oneshot` inside `EdgeZeroAxumService::call` has +//! converted any dispatch error into a plain response -- every response is +//! covered uniformly regardless of how routing produced it, and the +//! position survives future routing changes. (In this application's router +//! a catch-all fallback spans every path and publisher method, so +//! router-generated 404/405s that bypass middleware are close to +//! unreachable today; the outer position does not depend on them.) //! //! `/health` is excluded by path match before a //! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) @@ -163,6 +165,72 @@ mod tests { ); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_suppresses_header_when_flag_is_off() { + let router = RouterService::builder() + .get("/private", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), false); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert!( + header(&response, "server-timing").is_none(), + "should not emit server-timing when server_timing_enabled is false" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_round_trips_phase_timings_recorded_in_the_handler() { + // The collector crosses the adapter boundary as a request extension; + // this pins the round trip end to end: a phase recorded inside a + // core-style handler must come back out in the rendered header, so + // a future adapter conversion that drops request extensions fails + // here instead of silently losing every phase. + let router = RouterService::builder() + .get("/private", |ctx: RequestContext| async move { + if let Some(timings) = ctx.request().extensions().get::() { + timings.record( + trusted_server_core::request_timing::Phase::Filter, + std::time::Duration::from_millis(7), + ); + } + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + let server_timing = header(&response, "server-timing").expect("should emit header"); + assert!( + server_timing.contains("ts-filter;dur=7.0"), + "a phase recorded in the handler should survive the adapter round trip: {server_timing}" + ); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn axum_404_carries_header_when_private() { // An empty router has no routes at all, so any path dispatches diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml new file mode 100644 index 000000000..263403193 --- /dev/null +++ b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml @@ -0,0 +1,16 @@ +name = "trusted-server" +main = "build/index.js" +compatibility_date = "2024-09-23" +# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the +# outbound `CacheMode::NoStore` cache bypass under this compatibility date. +compatibility_flags = ["nodejs_compat", "cache_option_enabled"] +# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. + +[[kv_namespaces]] +binding = "TRUSTED_SERVER_KV" +id = "ci-local-kv" + +[vars] +# Placeholder replaced by the integration test harness with a JSON object that +# contains the runtime Trusted Server app-config blob envelope. +TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 333df736a..8cd2b39d0 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -290,8 +290,16 @@ fn edgezero_main(mut req: FastlyRequest, env: &EnvConfig) { access_telemetry_enabled, }, ); - run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); - emit_access_telemetry_after_send(settings, &outcome, &timings); + run_post_send_steps( + || { + run_edgezero_pull_sync_after_send( + settings, + &partner_registry, + &ec_state, + ) + }, + || emit_access_telemetry_after_send(settings, &outcome, &timings), + ); return; } Err(e) => { @@ -322,12 +330,16 @@ fn edgezero_main(mut req: FastlyRequest, env: &EnvConfig) { access_telemetry_enabled, }, ); - run_edgezero_pull_sync_after_send( - &settings, - &partner_registry, - &ec_state, + run_post_send_steps( + || { + run_edgezero_pull_sync_after_send( + &settings, + &partner_registry, + &ec_state, + ); + }, + || emit_access_telemetry_after_send(&settings, &outcome, &timings), ); - emit_access_telemetry_after_send(&settings, &outcome, &timings); return; } Err(e) => { @@ -408,13 +420,9 @@ fn apply_entry_point_finalize_headers( // router-level 404/405 for an unregistered method), so `geo_state` may // still be `NotAttempted` even after a fresh lookup just ran above. // Write the resolved outcome back so the access-telemetry snapshot built - // later in `send_edgezero_response` sees what was actually looked up, - // not the stale carried-in state. - let resolved_state = match &geo_info { - Some(info) => GeoLookupState::Resolved(info.clone()), - None => GeoLookupState::Attempted, - }; - response.extensions_mut().insert(resolved_state); + // later in `send_edgezero_response` sees what was actually looked up; + // 401 handling lives in the shared helper. + middleware::write_back_geo_lookup_state(response, geo_info.as_ref()); } fn apply_edgezero_ec_finalize( @@ -453,6 +461,19 @@ fn run_edgezero_pull_sync_after_send( } } +/// Runs the post-send steps in their contract order: EC identity pull-sync +/// first, then access-telemetry emission. +/// +/// Every `edgezero_main` site that has both steps routes through this +/// function, so the ordering is owned in exactly one place and the +/// sequence test can instrument it; `request_elapsed` is already stamped +/// before either step because `send_edgezero_response` stamps it before +/// returning. +fn run_post_send_steps(pull_sync: impl FnOnce(), emit_access_telemetry: impl FnOnce()) { + pull_sync(); + emit_access_telemetry(); +} + /// Builds and emits the access-telemetry row for one delivered response, /// when access telemetry is enabled and this request is sampled in. /// @@ -1003,6 +1024,8 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { + use std::sync::Mutex; + use super::*; use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; @@ -1733,13 +1756,13 @@ mod tests { } #[test] - fn request_elapsed_is_stamped_when_send_returns() { - // `edgezero_main`'s post-send ordering (pull-sync before telemetry) - // is a source-order invariant with no injectable seam, so this test - // deliberately proves only the leg that has one: by the time - // `send_edgezero_response` returns, `request_elapsed` is already - // stamped, so everything `edgezero_main` runs afterwards (pull-sync, - // telemetry emission) is excluded from `request_elapsed_ms`. + fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // The full contract sequence, instrumented through the real seams: + // `send_edgezero_response` stamps `request_elapsed` before + // returning, and `run_post_send_steps` (which every production + // site with both steps routes through) owns pull-sync-then- + // telemetry ordering. + let log: Arc>> = Arc::new(Mutex::new(Vec::new())); let timings = RequestTimings::new(); let response = response_builder() .body(EdgeBody::from("ok")) @@ -1757,14 +1780,36 @@ mod tests { access_telemetry_enabled: true, }, ); - assert!( timings.snapshot().request_elapsed_ms.is_some(), - "request_elapsed should be stamped by the time send returns" + "request_elapsed should be stamped before any post-send step runs" ); assert!( outcome.snapshot.is_some(), "the access snapshot should exist for the enabled context" ); + + let pull_log = Arc::clone(&log); + let emit_log = Arc::clone(&log); + run_post_send_steps( + move || { + pull_log + .lock() + .expect("should lock order log") + .push("pull_sync") + }, + move || { + emit_log + .lock() + .expect("should lock order log") + .push("telemetry") + }, + ); + + assert_eq!( + *log.lock().expect("should lock order log"), + vec!["pull_sync", "telemetry"], + "pull-sync must dispatch before telemetry emits" + ); } } diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 79ded8362..7bd577363 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -100,17 +100,10 @@ impl Middleware for FinalizeResponseMiddleware { }) }); - // Write the resolved outcome back so a downstream access-telemetry - // snapshot (built from response extensions after finalize) sees - // what was actually looked up here rather than the stale carried-in - // state — mirrors the entry-point finalize site in `main.rs` - // (`apply_entry_point_finalize_headers`), which writes back for the - // same reason. - let resolved_state = match &geo_info { - Some(geo) => GeoLookupState::Resolved(geo.clone()), - None => GeoLookupState::Attempted, - }; - response.extensions_mut().insert(resolved_state); + // Mirrors the entry-point finalize site in `main.rs` + // (`apply_entry_point_finalize_headers`); 401 handling lives in the + // shared helper. + write_back_geo_lookup_state(&mut response, geo_info.as_ref()); apply_finalize_headers(&self.settings, geo_info.as_ref(), &mut response); response @@ -193,6 +186,26 @@ impl Middleware for AuthMiddleware { /// is intentionally more conservative: geo data is not sent to any /// unauthenticated caller regardless of whether the 401 originated from this /// server or the upstream origin. +/// Writes the resolved geo outcome back onto the response as a +/// [`GeoLookupState`] extension, so a downstream access-telemetry snapshot +/// sees what was actually looked up rather than the stale carried-in state. +/// +/// Skips the write on a 401: [`resolve_geo_for_response`] returns `None` +/// for unauthorized responses before consulting the carried state, so +/// writing `Attempted` there would overwrite a carried `Resolved` with a +/// value that was never looked up, and the row would lose a country it +/// legitimately had. +pub(crate) fn write_back_geo_lookup_state(response: &mut Response, geo_info: Option<&GeoInfo>) { + if response.status() == StatusCode::UNAUTHORIZED { + return; + } + let resolved_state = match geo_info { + Some(geo) => GeoLookupState::Resolved(geo.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); +} + pub(crate) fn resolve_geo_for_response( response: &Response, carried: &GeoLookupState, @@ -285,6 +298,51 @@ pub(crate) use trusted_server_core::response_privacy::{ mod tests { use super::*; + #[test] + fn geo_write_back_preserves_resolved_state_on_401() { + // A 401 short-circuits geo resolution before the carried state is + // consulted, so the write-back must not downgrade a carried + // Resolved to Attempted (which would cost the row its country). + let mut response = response_builder() + .status(StatusCode::UNAUTHORIZED) + .body(Body::empty()) + .expect("should build a 401 response"); + response + .extensions_mut() + .insert(GeoLookupState::Resolved(sample_geo_info())); + + write_back_geo_lookup_state(&mut response, None); + + match response.extensions().get::() { + Some(GeoLookupState::Resolved(geo)) => { + assert_eq!( + geo.country, + sample_geo_info().country, + "should keep the carried country" + ); + } + other => panic!("should keep the Resolved state on a 401, got {other:?}"), + } + } + + #[test] + fn geo_write_back_records_attempted_on_non_401_miss() { + let mut response = response_builder() + .status(StatusCode::OK) + .body(Body::empty()) + .expect("should build a 200 response"); + + write_back_geo_lookup_state(&mut response, None); + + assert!( + matches!( + response.extensions().get::(), + Some(GeoLookupState::Attempted) + ), + "should record an attempted-but-missed lookup on ordinary responses" + ); + } + use std::collections::HashMap; use std::net::IpAddr; use std::sync::{Arc, Mutex}; diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 84dfc37d8..dfcf65f98 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -326,8 +326,13 @@ fn build_access_events_request( pub(crate) async fn emit_access_event( client: &dyn PlatformHttpClient, target: &TinybirdEventsTarget, - row: String, + mut row: String, ) -> Result<(), Report> { + // Match the auction sink's NDJSON framing: every row is + // newline-terminated, and the terminator counts toward the body limit. + if !row.ends_with('\n') { + row.push('\n'); + } let body_len = row.len(); if body_len > target.max_body_bytes { return Err(Report::new(TrustedServerError::Proxy { @@ -354,8 +359,11 @@ pub(crate) async fn emit_access_event( backend_name ); + // The response body is never consumed, so stream it: buffered + // conversion on Fastly materializes the body before the size limit is + // enforced, which a chunked response could abuse. let response = client - .send(PlatformHttpRequest::new(request, backend_name)) + .send(PlatformHttpRequest::new(request, backend_name).with_stream_response()) .await .change_context(TrustedServerError::Proxy { message: "failed to send Tinybird access telemetry request".to_owned(), @@ -490,6 +498,7 @@ mod tests { uri: String, headers: Vec<(String, String)>, body: Vec, + stream_response: bool, } /// Records outbound requests and, for [`PlatformHttpClient::send`] (the @@ -515,6 +524,7 @@ mod tests { fn record(&self, request: PlatformHttpRequest) { let backend_name = request.backend_name; + let stream_response = request.stream_response; let (parts, body) = request.request.into_parts(); let headers = parts .headers @@ -532,6 +542,7 @@ mod tests { uri: parts.uri.to_string(), headers, body: body.into_bytes().unwrap_or_default().to_vec(), + stream_response, }; self.requests .lock() @@ -949,10 +960,15 @@ mod tests { header_value(&requests[0].headers, header::AUTHORIZATION.as_str()), Some("Bearer test-tinybird-access-append-token") ); + let body = std::str::from_utf8(&requests[0].body).expect("should record utf8 body"); assert_eq!( - std::str::from_utf8(&requests[0].body).expect("should record utf8 body"), - row, - "should send the row verbatim as the request body" + body, + format!("{row}\n"), + "should send newline-delimited JSON, matching the auction sink's framing" + ); + assert!( + requests[0].stream_response, + "should stream the Tinybird response: the body is never consumed" ); } diff --git a/crates/trusted-server-core/src/platform/timed_kv.rs b/crates/trusted-server-core/src/platform/timed_kv.rs index 0594b7cef..faac0ad2c 100644 --- a/crates/trusted-server-core/src/platform/timed_kv.rs +++ b/crates/trusted-server-core/src/platform/timed_kv.rs @@ -53,6 +53,15 @@ impl PlatformKvStore for TimedKvStore> { self.inner.put_bytes(key, value).await } + // Forwarded explicitly: the trait's default body falls back to + // `get_bytes`, which would silently downgrade a backend's cheap + // metadata-only existence probe (the Spin adapter has one) into a full + // value transfer just because the store was decorated. + async fn exists(&self, key: &str) -> Result { + let _span = self.timings.span(Phase::EcKv); + self.inner.exists(key).await + } + async fn put_bytes_with_ttl( &self, key: &str, @@ -145,6 +154,65 @@ mod tests { ); } + #[test] + fn exists_delegates_to_the_inner_store_not_get_bytes() { + // A stub whose `exists` answer contradicts its `get_bytes` answer: + // if the decorator fell back to the trait's get-and-discard default + // body, this would return `false`. + struct ExistsOnlyStore; + + #[async_trait::async_trait(?Send)] + impl PlatformKvStore for ExistsOnlyStore { + async fn get_bytes(&self, _key: &str) -> Result, KvError> { + Ok(None) + } + async fn put_bytes(&self, _key: &str, _value: Bytes) -> Result<(), KvError> { + Ok(()) + } + async fn put_bytes_with_ttl( + &self, + _key: &str, + _value: Bytes, + _ttl: StdDuration, + ) -> Result<(), KvError> { + Ok(()) + } + async fn delete(&self, _key: &str) -> Result<(), KvError> { + Ok(()) + } + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result { + Ok(KvPage { + keys: Vec::new(), + cursor: None, + }) + } + async fn exists(&self, _key: &str) -> Result { + Ok(true) + } + } + + let timings = RequestTimings::new(); + let inner: Arc = Arc::new(ExistsOnlyStore); + let store = TimedKvStore::new(inner, timings.clone()); + + let exists = futures::executor::block_on(store.exists("key")) + .expect("should forward the existence probe"); + assert!( + exists, + "should delegate to the inner exists, not the get_bytes default body" + ); + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should time the existence probe like any other store operation" + ); + } + #[test] fn store_name_is_not_timed() { let timings = RequestTimings::new(); diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index cbb269b1e..3b26c3ca2 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -1,10 +1,12 @@ //! Per-request phase timing collection and Server-Timing rendering. //! -//! Collection is always-on and infallible: saturating math, lock failure -//! drops the sample, no panics. See the design spec +//! Collection is always-on and infallible: saturating math, no panics. A +//! contended lock drops the one sample; a poisoned lock recovers (the +//! guarded data are plain counters), so a panic elsewhere cannot silence +//! the rest of the request's timing. See the design spec //! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, TryLockError}; use std::time::Duration; use http::{HeaderName, HeaderValue, Response}; @@ -91,7 +93,12 @@ pub enum AuctionWaitPlacement { /// Mutable state behind [`RequestTimings`], guarded by a [`Mutex`]. struct Inner { - /// Instant the request started; the reference point for elapsed marks. + /// Instant the collector was constructed; the reference point for + /// elapsed marks. Adapters construct the collector after their own + /// prologue (client-request acquisition, logger init, and any + /// short-circuit routes on Fastly), so `ts-total` and + /// `time_elapsed_ms` exclude that prologue rather than measuring + /// wall-clock-from-accept. t0: Instant, /// Accumulated duration per [`Phase`], indexed by [`Phase::index`]. phases: [Option; PHASE_COUNT], @@ -111,9 +118,9 @@ struct Inner { /// Per-request phase timing collector. /// /// Cheap to clone (an [`Arc`] handle) and safe to share across threads and -/// async tasks handling the same request. Every method is infallible: lock -/// contention or poisoning silently drops the sample rather than blocking or -/// panicking. +/// async tasks handling the same request. Every method is infallible: +/// contention silently drops the one sample rather than blocking, and a +/// poisoned lock is recovered rather than treated as permanent loss. #[derive(Clone)] pub struct RequestTimings(Arc>); @@ -134,10 +141,16 @@ impl RequestTimings { /// Accumulates `dur` into `phase`'s running total. /// /// Repeated calls for the same phase saturate-add rather than overwrite. - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn record(&self, phase: Phase, dur: Duration) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; let index = phase.index(); let accumulated = inner.phases[index] @@ -149,10 +162,16 @@ impl RequestTimings { /// Records an auction wait duration under [`Phase::AuctionWait`] and /// stores its placement. /// - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; let index = Phase::AuctionWait.index(); let accumulated = inner.phases[index] @@ -178,8 +197,14 @@ impl RequestTimings { /// Subsequent calls are no-ops (first call wins). Drops the sample /// silently on lock contention or poisoning. pub fn mark_headers_ready(&self) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; if inner.headers_ready_total.is_none() { inner.headers_ready_total = Some(inner.t0.elapsed()); @@ -192,8 +217,14 @@ impl RequestTimings { /// Subsequent calls are no-ops (first call wins). Drops the sample /// silently on lock contention or poisoning. pub fn mark_request_elapsed(&self) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; if inner.request_elapsed.is_none() { inner.request_elapsed = Some(inner.t0.elapsed()); @@ -202,10 +233,16 @@ impl RequestTimings { /// Records the response body size in bytes. /// - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn set_resp_bytes(&self, bytes: u64) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; inner.resp_bytes = Some(bytes); } @@ -221,7 +258,11 @@ impl RequestTimings { /// silently (returning `None`) on lock contention or poisoning. #[must_use] pub fn server_timing_value(&self) -> Option { - let inner = self.0.try_lock().ok()?; + let inner = match self.0.try_lock() { + Ok(guard) => guard, + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return None, + }; let total = inner.headers_ready_total?; let mut entries = vec![format_entry("ts-total", total)]; for phase in HEADER_PHASES { @@ -241,8 +282,10 @@ impl RequestTimings { /// consistent with the infallibility of every other method. #[must_use] pub fn snapshot(&self) -> TimingSnapshot { - let Ok(inner) = self.0.try_lock() else { - return TimingSnapshot::default(); + let inner = match self.0.try_lock() { + Ok(guard) => guard, + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return TimingSnapshot::default(), }; TimingSnapshot { time_elapsed_ms: duration_ms(inner.headers_ready_total), diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md index 02cda64d8..1b25b70d1 100644 --- a/docs/superpowers/plans/2026-08-24-request-phase-timing.md +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -819,7 +819,8 @@ plus the coarse template; the freeze point consumes the extension (no `RouteClas column in `NAMED_ROUTES`, no reconstruction from a handler enum). Also in this task: make `TemplateCacheResponseState` a typed response extension in `publisher.rs`, set at every point that writes `x-ts-template-cache` so header and extension cannot -drift; the row reads the extension. The snapshot is built unconditionally in +drift; the row reads the extension. The snapshot is built (when access +telemetry is enabled) in `send_edgezero_response` right after `mark_headers_ready()` and returned inside `DeliveryOutcome` (add field `pub snapshot: AccessTelemetrySnapshot`). From 1476137e794e95f43d36a7531d59a082d640ca8f Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 15 Sep 2026 17:29:49 +0530 Subject: [PATCH 039/104] Show help for a bare ts dev proxy invocation Signed-off-by: dhruv8sh --- .../src/commands/dev/mod.rs | 2 +- .../src/commands/dev/proxy/config.rs | 11 ++-------- .../src/commands/dev/proxy/mod.rs | 1 + crates/trusted-server-cli/src/run.rs | 21 +++++++++++++++++++ 4 files changed, 25 insertions(+), 10 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/mod.rs b/crates/trusted-server-cli/src/commands/dev/mod.rs index 7a61d769b..822d99fd1 100644 --- a/crates/trusted-server-cli/src/commands/dev/mod.rs +++ b/crates/trusted-server-cli/src/commands/dev/mod.rs @@ -31,6 +31,6 @@ pub enum DevCommand { pub fn run(command: DevCommand) -> Result<(), String> { match command { #[cfg(target_os = "macos")] - DevCommand::Proxy(args) => proxy::run(&args).map_err(|report| format!("{report:?}")), + DevCommand::Proxy(args) => proxy::run(&args).map_err(|report| format!("{report:#}")), } } diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs index d831735c2..9d0ef2380 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs @@ -359,14 +359,7 @@ mod tests { }; fn base_args() -> crate::commands::dev::proxy::ProxyArgs { - // Construct via clap so defaults match the real surface. - use clap::Parser; - #[derive(clap::Parser)] - struct W { - #[command(flatten)] - a: crate::commands::dev::proxy::ProxyArgs, - } - W::parse_from(["ts"]).a + parse_args(&["ts", "--listen", "127.0.0.1:18080"]) } fn parse_args(argv: &[&str]) -> crate::commands::dev::proxy::ProxyArgs { @@ -382,7 +375,7 @@ mod tests { #[test] fn clap_parses_rewrite_host_as_a_bool() { assert!( - !parse_args(&["ts"]).rewrite_host, + !parse_args(&["ts", "--listen", "127.0.0.1:18080"]).rewrite_host, "absent --rewrite-host is false" ); assert!( diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs index 1bebd872d..6b6d2b748 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs @@ -78,6 +78,7 @@ async fn finish_interrupted_run( /// `ts dev proxy [OPTIONS]` — see the design spec §4. #[derive(Debug, clap::Args)] +#[command(arg_required_else_help = true)] pub struct ProxyArgs { /// Rewrite rule `FROM=TO` (repeatable). #[arg(long = "map", value_name = "FROM=TO")] diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 13009d448..e5079ebb8 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -678,4 +678,25 @@ mod tests { "error should explain unsupported option" ); } + + #[test] + #[cfg(target_os = "macos")] + fn dev_proxy_bare_invocation_shows_help_before_running() { + let error = Args::try_parse_from(["ts", "dev", "proxy"]) + .expect_err("a bare `ts dev proxy` should short-circuit to help, not run"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::DisplayHelpOnMissingArgumentOrSubcommand, + "should print help instead of touching system proxy state or attempting sudo" + ); + parse(&["ts", "dev", "proxy", "ca", "path"]); + } + + #[test] + #[cfg(target_os = "macos")] + fn dev_proxy_partial_rule_parses_instead_of_showing_help() { + // An explicit but incomplete rule (`--from` with no `--to`) must reach + // `run` and surface the concise no-rule error there, not clap help. + parse(&["ts", "dev", "proxy", "--from", "a.example.com"]); + } } From ecc962fc550001f2b6d4363f8e0d49f816c164f6 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 15 Sep 2026 18:27:28 +0530 Subject: [PATCH 040/104] Enforce formatting on root level markdown files Signed-off-by: dhruv8sh --- .github/workflows/format.yml | 4 ++++ AGENTS.md | 38 ++++++++++++++++++------------------ FAQ_POC.md | 18 ++++++++--------- ProjectGovernance.md | 12 ++++++------ 4 files changed, 38 insertions(+), 34 deletions(-) diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 7253b5ea2..c844e95d3 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -149,5 +149,9 @@ jobs: - name: Run Prettier (check) run: npm run format + - name: Run Prettier (check) — root Markdown + working-directory: . + run: docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" + - name: Build with VitePress (fails on dead links) run: npm run build diff --git a/AGENTS.md b/AGENTS.md index 3b7189204..88bd7595a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -323,14 +323,14 @@ IntegrationRegistration::builder(ID) ## Configuration Files -| File | Purpose | -| --------------------- | ---------------------------------------------------------- | -| `edgezero.toml` | EdgeZero app/platform manifest and logical stores | -| `fastly.toml` | Fastly service configuration and build settings | -| `trusted-server.example.toml` | Source-controlled Trusted Server app-config template | -| `trusted-server.toml` | Operator-owned app config; gitignored; `ts config push` publishes it as an EdgeZero blob envelope | -| `rust-toolchain.toml` | Pins Rust version to 1.95.0 | -| `.env.dev` | Local development environment variables | +| File | Purpose | +| ----------------------------- | ------------------------------------------------------------------------------------------------- | +| `edgezero.toml` | EdgeZero app/platform manifest and logical stores | +| `fastly.toml` | Fastly service configuration and build settings | +| `trusted-server.example.toml` | Source-controlled Trusted Server app-config template | +| `trusted-server.toml` | Operator-owned app config; gitignored; `ts config push` publishes it as an EdgeZero blob envelope | +| `rust-toolchain.toml` | Pins Rust version to 1.95.0 | +| `.env.dev` | Local development environment variables | --- @@ -433,18 +433,18 @@ both runtime behavior and build/tooling changes. ## Key Files -| File | Purpose | -| -------------------------------------------- | ------------------------------------------------- | -| `crates/trusted-server-core/src/integrations/registry.rs` | IntegrationRegistry, `js_module_ids()` | -| `crates/trusted-server-core/src/tsjs.rs` | Script tag generation with module IDs | -| `crates/trusted-server-core/src/html_processor.rs` | Injects `` sequences inside the string. pub(crate) fn build_bids_script(bid_map: &serde_json::Map) -> String { + build_bids_script_with_diagnostics(bid_map, None) +} + +fn build_bids_script_with_diagnostics( + bid_map: &serde_json::Map, + auction_diagnostics: Option<&BrowserAuctionDiagnostics>, +) -> String { let json = serde_json::to_string(bid_map) .expect("serde_json::to_string of Map should be infallible"); let escaped = html_escape_for_script(&json); @@ -5614,6 +5706,23 @@ pub(crate) fn build_bids_script(bid_map: &serde_json::Map(function(){{\ +var t=window.tsjs=window.tsjs||{{}};\ +var b=JSON.parse(\"{}\");\ +var d=JSON.parse(\"{}\");\ +var s=t.scheduleInitialAdInit;\ +if(typeof s===\"function\")s(b,void 0,d);\ +else{{t.bids=b;t.auctionDiagnostics=d;}}\ +}})();", + escaped, + html_escape_for_script(&diagnostics) + ); + } + format!( "", + html_escape_for_script(slots_json), + html_escape_for_script(&bids), + html_escape_for_script(&diagnostics) + ); + } + format!( ""#.to_vec(), + vec![("content-type", "text/html"), ("cache-control", "no-store")], + ); + let services = build_services_with_http_client(client.clone()); + + let response = integration + .handle( + &settings, + &services, + make_req(Method::GET, "https://publisher.example.com/integrations/sourcepoint/cdn/us_pm/index.html"), + ) + .await + .expect("should proxy HTML without Content-Length"); + + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve upstream HTML cache policy" + ); + assert_eq!( + response.into_body().into_bytes_bounded(1024).await.expect("should collect HTML response").as_ref(), + br#""#, + "should rewrite streamed privacy-manager assets without Content-Length" + ); + }); + } + + #[test] + fn handle_accepts_exact_rewrite_limit_and_stops_reading_on_overflow() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + let limit = MAX_REWRITE_BODY_SIZE as usize; + for content_type in ["application/javascript", "text/html"] { + for declared_length in [None, Some("1")] { + for extra_bytes in [0, 1, TEST_CHUNK_SIZE * 2] { + let client = Arc::new(StreamingHttpClient::new()); + let mut headers = vec![("content-type", content_type)]; + if let Some(length) = declared_length { + headers.push(("content-length", length)); + } + client.stub.push_response_with_headers( + 200, + vec![b' '; limit + extra_bytes], + headers, + ); + let services = build_services_with_http_client(client.clone()); + + let result = integration.handle( + &settings, + &services, + make_req(Method::GET, "https://publisher.example.com/integrations/sourcepoint/cdn/asset"), + ).await; + + if extra_bytes == 0 { + let response = result.expect("should accept exactly 5 MiB"); + assert!( + response.headers().get(header::CONTENT_LENGTH).is_none(), + "should remove the advisory length after rewriting" + ); + assert_eq!( + take_body_bytes(response).len(), + limit, + "should retain the entire body at the limit" + ); + } else { + let error = + result.expect_err("should reject an oversized streamed body"); + assert_eq!( + error.current_context().status_code(), + StatusCode::BAD_GATEWAY, + "should report upstream overflow as 502" + ); + assert!( + matches!(error.current_context(), TrustedServerError::Integration { integration, message } if integration == SOURCEPOINT_INTEGRATION_ID && message.contains("exceeds")), + "should identify Sourcepoint response overflow" + ); + } + assert_eq!( + client.reads.load(Ordering::Relaxed), + limit / TEST_CHUNK_SIZE + 1, + "should stop at EOF or the first overflowing chunk without draining the stream" + ); + } + } + } + }); + } + + #[test] + fn handle_passes_through_declared_oversize_without_reading() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + for content_type in ["application/javascript", "text/html"] { + let client = Arc::new(StreamingHttpClient::new()); + let length = (MAX_REWRITE_BODY_SIZE + 1).to_string(); + client.stub.push_response_with_headers( + 200, + vec![b' '; MAX_REWRITE_BODY_SIZE as usize + 1], + vec![ + ("content-type", content_type), + ("content-length", &length), + ("cache-control", "no-store"), + ], + ); + let services = build_services_with_http_client(client.clone()); + + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/asset", + ), + ) + .await + .expect("should pass through a declared oversized response"); + + assert!( + matches!(response.body(), EdgeBody::Stream(_)), + "should retain the original stream" + ); + assert_eq!( + client.reads.load(Ordering::Relaxed), + 0, + "should not poll a declared oversized body" + ); + assert_eq!( + get_header_str(&response, header::CONTENT_LENGTH), + Some(length.as_str()), + "should preserve the pass-through length" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve upstream cache policy" + ); + } + }); + } + + #[test] + fn handle_keeps_ineligible_responses_streaming() { + futures::executor::block_on(async { + let settings = create_test_settings(); + for (content_type, method, status, rewrite_sdk) in [ + ("application/json", Method::GET, 200, true), + ("text/css", Method::GET, 200, true), + ("image/png", Method::GET, 200, true), + ("application/javascript", Method::GET, 200, false), + ("text/html", Method::GET, 200, false), + ("application/javascript", Method::HEAD, 200, true), + ("text/html", Method::POST, 200, true), + ("application/javascript", Method::GET, 206, true), + ("text/html", Method::GET, 404, true), + ] { + let mut cfg = config(true); + cfg.rewrite_sdk = rewrite_sdk; + let integration = SourcepointIntegration::new(Arc::new(cfg)); + let client = Arc::new(StreamingHttpClient::new()); + client.stub.push_response_with_headers( + status, + b"unchanged".to_vec(), + vec![ + ("content-type", content_type), + ("content-encoding", "gzip"), + ("cache-control", "no-store"), + ], + ); + let services = build_services_with_http_client(client.clone()); + let mut request = make_req( + method, + "https://publisher.example.com/integrations/sourcepoint/cdn/asset", + ); + set_req_header(&mut request, header::ACCEPT_ENCODING, "gzip, br"); + + let response = integration + .handle(&settings, &services, request) + .await + .expect("should pass through an ineligible response"); + + assert!( + matches!(response.body(), EdgeBody::Stream(_)), + "should leave ineligible bodies streaming" + ); + assert_eq!( + client.reads.load(Ordering::Relaxed), + 0, + "should not poll an ineligible body" + ); + assert_eq!( + get_header_str(&response, header::CONTENT_ENCODING), + Some("gzip"), + "should preserve pass-through encoding" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve pass-through cache policy" + ); + assert_eq!( + response + .into_body() + .into_bytes_bounded(1024) + .await + .expect("should collect pass-through body") + .as_ref(), + b"unchanged", + "should preserve pass-through bytes" + ); + assert!( + client.stub.recorded_request_headers()[0] + .iter() + .any(|(name, value)| name == "accept-encoding" && value == "gzip, br"), + "should forward the client's encoding for non-script paths" + ); + } + }); + } + + #[test] + fn handle_rewrites_on_buffered_adapters_with_or_without_content_length() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + for has_length in [false, true] { + let client = Arc::new(StubHttpClient::new()); + let input = format!(r#"var api="https://{SOURCEPOINT_CDN_HOST}/consent/tcfv2";"#); + let length = input.len().to_string(); + let mut headers = vec![ + ("content-type", "application/javascript"), + ("content-encoding", "identity"), + ("vary", "Accept-Encoding, Origin"), + ]; + if has_length { + headers.push(("content-length", &length)); + } + client.push_response_with_headers(200, input.into_bytes(), headers); + let services = build_services_with_http_client(client.clone()); + + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/wrapper.js", + ), + ) + .await + .expect("should rewrite a buffered response"); + + assert_eq!( + client.recorded_stream_response_flags(), + vec![false], + "should respect adapters without streaming support" + ); + assert!( + response.headers().get(header::CONTENT_LENGTH).is_none(), + "should not forward a stale upstream length" + ); + assert!( + response.headers().get(header::CONTENT_ENCODING).is_none(), + "should remove upstream encoding after rewriting" + ); + assert_eq!( + get_header_str(&response, header::VARY), + Some("Origin"), + "should remove only Accept-Encoding from Vary" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("public, max-age=3600"), + "should retain the static JavaScript cache policy" + ); + assert_eq!( + get_header_str(&response, header::CONTENT_TYPE), + Some("application/javascript; charset=utf-8"), + "should identify rewritten JavaScript" + ); + assert_eq!( + take_body_bytes(response), + br#"var api="/integrations/sourcepoint/cdn/consent/tcfv2";"#, + "should rewrite with either header state" + ); + } + }); + } + + #[test] + fn handle_site_data_requests_identity_and_preserves_dynamic_cache_policy() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + for (upstream_cache, forwarded_cookies, sets_cookie, expected_cache) in [ + (Some("no-store"), false, false, "no-store"), + ( + Some("private, max-age=60"), + true, + false, + "private, max-age=60", + ), + (None, true, false, "private, max-age=0"), + (None, false, false, "public, max-age=3600"), + ( + Some("public, max-age=3600"), + false, + true, + "private, no-store", + ), + ] { + let client = Arc::new(StreamingHttpClient::new()); + let input = format!(r#"var api="https://{SOURCEPOINT_CDN_HOST}/consent/tcfv2";"#); + let mut headers = vec![("content-type", "application/javascript")]; + if let Some(cache) = upstream_cache { + headers.push(("cache-control", cache)); + } + if sets_cookie { + headers.push(("set-cookie", "consentUUID=example; Path=/")); + } + client + .stub + .push_response_with_headers(200, input.into_bytes(), headers); + let services = build_services_with_http_client(client.clone()); + let mut request = make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/mms/v2/get_site_data?account_id=123", + ); + set_req_header(&mut request, header::ACCEPT_ENCODING, "gzip, br"); + if forwarded_cookies { + set_req_header(&mut request, header::COOKIE, "consentUUID=example"); + } + + let response = integration + .handle(&settings, &services, request) + .await + .expect("should rewrite dynamic site data"); + + assert!( + client.stub.recorded_request_headers()[0] + .iter() + .any(|(name, value)| name == "accept-encoding" && value == "identity"), + "should request uncompressed site data despite its extensionless path" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some(expected_cache), + "should use the dynamic endpoint's cache policy" + ); + assert_eq!( + take_body_bytes(response), + br#"var api="/integrations/sourcepoint/cdn/consent/tcfv2";"#, + "should rewrite unknown-length site data" + ); + } + }); + } + + #[test] + fn handle_preserves_invalid_utf8_bytes_and_headers() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + let client = Arc::new(StreamingHttpClient::new()); + let bytes = vec![0x1f, 0x8b, 0xff]; + client.stub.push_response_with_headers( + 200, + bytes.clone(), + vec![ + ("content-type", "application/javascript"), + ("content-encoding", "gzip"), + ("cache-control", "no-store"), + ], + ); + let services = build_services_with_http_client(client); + + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/wrapper.js", + ), + ) + .await + .expect("should retain non-UTF-8 content unchanged"); + + assert_eq!( + get_header_str(&response, header::CONTENT_ENCODING), + Some("gzip"), + "should preserve encoding when no rewrite occurs" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve cache policy when no rewrite occurs" + ); + assert_eq!( + take_body_bytes(response), + bytes, + "should preserve invalid UTF-8 bytes" + ); + }); + } fn config(enabled: bool) -> SourcepointConfig { SourcepointConfig { @@ -1437,9 +1984,12 @@ mod tests { assert!(SourcepointIntegration::is_likely_javascript_path( "/module/sourcepoint.mjs" )); - assert!(!SourcepointIntegration::is_likely_javascript_path( + assert!(SourcepointIntegration::is_likely_javascript_path( "/mms/v2/get_site_data" )); + assert!(!SourcepointIntegration::is_likely_javascript_path( + "/mms/v2/get_site_data/other" + )); assert!(!SourcepointIntegration::is_likely_javascript_path( "/consent/tcfv2" )); @@ -1943,7 +2493,12 @@ mod tests { set_header(&mut response, header::CACHE_CONTROL, "no-store"); *response.body_mut() = EdgeBody::from(b"payload".to_vec()); - integration.rewrite_javascript_response(&mut response, "rewritten".to_string()); + integration.rewrite_javascript_response( + &mut response, + "rewritten".to_string(), + "/wrapper.js", + false, + ); assert_eq!(response.status(), StatusCode::OK); assert_eq!( @@ -1982,7 +2537,12 @@ mod tests { set_header(&mut response, header::CACHE_CONTROL, "public, max-age=3600"); *response.body_mut() = EdgeBody::from(b"payload".to_vec()); - integration.rewrite_javascript_response(&mut response, "rewritten".to_string()); + integration.rewrite_javascript_response( + &mut response, + "rewritten".to_string(), + "/wrapper.js", + false, + ); assert_eq!( get_header_str(&response, header::CACHE_CONTROL), @@ -2002,7 +2562,12 @@ mod tests { set_header(&mut response, header::VARY, "Accept-Encoding"); *response.body_mut() = EdgeBody::from(b"payload".to_vec()); - integration.rewrite_javascript_response(&mut response, "rewritten".to_string()); + integration.rewrite_javascript_response( + &mut response, + "rewritten".to_string(), + "/wrapper.js", + false, + ); assert!( response.headers().get(header::VARY).is_none(), diff --git a/docs/guide/integrations/sourcepoint.md b/docs/guide/integrations/sourcepoint.md index 7b9ce9435..22437dd2b 100644 --- a/docs/guide/integrations/sourcepoint.md +++ b/docs/guide/integrations/sourcepoint.md @@ -60,6 +60,21 @@ When `rewrite_sdk = true`, Trusted Server rewrites matching Sourcepoint URLs in If a publisher uses a Content Security Policy, `script-src` must allow the first-party Trusted Server host after rewriting. A policy that only allows `https://cdn.privacy-mgmt.com` can block the rewritten first-party script URLs. +## Response body rewriting + +With `rewrite_sdk = true`, successful `GET` responses with status 200 and a JavaScript or HTML content type are eligible for body rewriting. JavaScript CDN URLs and privacy-manager HTML root-relative `src` / `href` assets are routed through the first-party proxy. + +The input body limit is 5 MiB: + +- Bodies at or below the limit are rewritten, including chunked or HTTP/2 responses without `Content-Length`. +- A declared `Content-Length` above the limit skips rewriting and leaves the body unchanged. +- If collection exceeds the limit, Trusted Server stops reading and returns `502 Bad Gateway`. This also applies when `Content-Length` understates the size. The partial body is discarded, not returned to the browser. +- Other content types, disabled rewriting, and ineligible methods or statuses do not collect the body for rewriting. + +On Fastly, upstream responses remain streaming until the bounded collector reads them. It retains at most 5 MiB of input plus the current transport chunk, with additional bounded allocations for rewriting. Non-rewritten responses remain streaming. Adapters without streaming support still buffer upstream responses before this check; this limit is not an adapter-level memory guarantee on Cloudflare or Spin. + +Likely JavaScript and HTML paths, including `/mms/v2/get_site_data`, request `Accept-Encoding: identity`. Rewritten bodies discard the upstream `Content-Length` and `Content-Encoding`, and remove `Accept-Encoding` from `Vary`. Non-UTF-8 bodies pass through unchanged after bounded collection. + ## Runtime Config Rewriting Trusted Server also injects a `window._sp_` property trap that rewrites known Sourcepoint URL-bearing config fields (`baseEndpoint`, `mmsDomain`, `wrapperAPIOrigin`, `cmpOrigin`, and `metricUrl`). If Sourcepoint introduces another URL-bearing field and third-party `cdn.privacy-mgmt.com` requests remain after enabling this integration, report the missed field so it can be added to the trap. @@ -74,6 +89,8 @@ Trusted Server forwards only Sourcepoint's documented cookie names upstream, plu Responses that include `Set-Cookie` are forced to `Cache-Control: private, no-store` so cookie-bearing Sourcepoint traffic is never marked as publicly cacheable content by the proxy. +Rewritten `/mms/v2/get_site_data` responses preserve upstream cache policy rather than receiving the static JavaScript cache policy. When upstream omits `Cache-Control`, they use `private, max-age=0` if Sourcepoint cookies were forwarded, or the configured public TTL otherwise. Rewritten HTML uses the same policy. Other rewritten JavaScript retains the existing configured public TTL, except for responses that set cookies. + ## Notes - This version scopes the integration to `cdn.privacy-mgmt.com`. Additional Sourcepoint domains (e.g., `geo.privacymanager.io`) can be added later if publishers require them. From 713b27134d13262ee5728dff6ca92634f89d1305 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:02:14 -0700 Subject: [PATCH 069/104] Move the EdgeZero pin to the PR 381 branch head 4531aeec Upstream now parses the live Fastly resource-link types (`config`, `kv-store`, `secret-store`), documents the custom entry point migration, and moves the release verifier into edgezero-adapter, which adds serde, serde_json, sha2, and walkdir edges under its cli feature. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 285104697..e5cb56241 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,15 +1427,19 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ + "serde", + "serde_json", + "sha2 0.10.9", "toml", + "walkdir", ] [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-trait", @@ -1463,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-trait", @@ -1486,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-stream", @@ -1515,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-trait", @@ -1542,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "chrono", "clap", @@ -1567,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-compression", @@ -1598,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "log", "proc-macro2", From 3c9d8c05a30d01ab194f6f76840ce3f76f66d850 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:16:51 -0700 Subject: [PATCH 070/104] Move the EdgeZero pin to the PR 381 branch head 7162b7e2 The two upstream commits since 4531aeec touch only the deploy action script and a demo lockfile. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e5cb56241..c9b545ca6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "log", "proc-macro2", From 9226cfbba9672d52d0c3d63fe0b754b08511f7d9 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:41:13 -0700 Subject: [PATCH 071/104] Move the EdgeZero pin to the PR 381 branch head 8b506782 The two upstream commits since 7162b7e2 only reorganize the EdgeZero CI test workflows. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index c9b545ca6..283e99dc0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "log", "proc-macro2", From 7d212b7d3182162e881e7208ac37abf92f945188 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 15:31:21 -0700 Subject: [PATCH 072/104] Move the EdgeZero pin to the PR 381 branch head fd45db1f Upstream replaces the Compute-unsupported version diff snapshot with per-collection reads and drops the parent service ID comparison from the staged-source guard and staging rollback. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 283e99dc0..742351842 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "log", "proc-macro2", From c7483aa9245e3dea2bf16d197f40d2ca7b08b8a3 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 16:21:33 -0700 Subject: [PATCH 073/104] Move the EdgeZero pin to the PR 381 branch head 6258b6b2 The only upstream change since fd45db1f corrects the Google Pub/Sub logging snapshot path to the Fastly API's `logging/pubsub` and adds `logentries` to the swept endpoint kinds. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 742351842..5c1e1a412 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "log", "proc-macro2", From 9380c12aff3b22be401a41d13302506c00f34747 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 17:03:58 -0700 Subject: [PATCH 074/104] Move the EdgeZero pin to the PR 381 branch head bb4e0040 The only upstream change since 6258b6b2 switches the Fastly deploy to the current `service resource-link` and `service version` command spellings, removing the CLI deprecation notices. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 5c1e1a412..d655fc4ed 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "log", "proc-macro2", From b1d41c541e6e089764ef4b9bce404f56b3fd5e3b Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:01:20 -0700 Subject: [PATCH 075/104] Move the EdgeZero pin to the PR 381 branch head cf9a96a0 The only crate change since bb4e0040 canonicalizes the Fastly configuration snapshot by sorting object keys before ordering rows, so the drift check no longer fails on Fastly's random JSON key order. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d655fc4ed..229bd7191 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "log", "proc-macro2", From 429503979dc81a16e19e1b27b48eae79cf8c6c17 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:41:47 -0700 Subject: [PATCH 076/104] Move the EdgeZero pin to the PR 381 branch head 119fbf80 The only upstream commit since cf9a96a0 adds a Cargo target cache to EdgeZero's CI. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 229bd7191..c2d6c9f17 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "log", "proc-macro2", From 1e6fc674f27e4110866776469d9d197c9e49550d Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:39:33 -0700 Subject: [PATCH 077/104] Watch only bundle inputs in the trusted-server-js build script The build script asked Cargo to rerun on every file under lib/, including node_modules, which emitted 33,932 rerun-if-changed directives in a 2.8 MB output that Cargo re-checked on every build and invalidated the crate after any npm install. Watch the TypeScript sources, package manifests, and bundler configuration explicitly instead. Measured on an Apple Silicon host with a warm cache: a no-op `cargo check -p trusted-server-js` drops from 1.1s to 0.15s, the script emits 7 directives, touching a node_modules file no longer reruns it, and touching lib/src or package-lock.json still does. --- crates/trusted-server-js/build.rs | 40 ++++++++++++------------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/crates/trusted-server-js/build.rs b/crates/trusted-server-js/build.rs index ba6cd88f2..519f0bd71 100644 --- a/crates/trusted-server-js/build.rs +++ b/crates/trusted-server-js/build.rs @@ -15,9 +15,21 @@ use build_print::{info, warn}; use sha2::{Digest as _, Sha256}; fn main() { - // Rebuild if TS sources change (belt-and-suspenders): enumerate every file under lib/ - println!("cargo:rerun-if-changed=lib"); - watch_dir_recursively(Path::new("lib")); + // Rebuild when the TypeScript sources or anything that shapes the bundle + // output changes. `lib` as a whole is deliberately not watched: it holds + // `node_modules`, and enumerating that tree emitted tens of thousands of + // directives that Cargo re-checked on every build. + for watched in [ + "lib/src", + "lib/package.json", + "lib/package-lock.json", + "lib/build-all.mjs", + "lib/build-prebid-external.mjs", + "lib/tsconfig.json", + "lib/vite.config.ts", + ] { + println!("cargo:rerun-if-changed={watched}"); + } // Allow opt-out or force via env let skip = env::var("TSJS_SKIP_BUILD").is_ok_and(|value| value == "1"); @@ -197,25 +209,3 @@ fn copy_bundle(filename: &str, required: bool, dist_dir: &Path, out_dir: &Path) fs::write(&target, "").expect("should write optional empty bundle placeholder"); } - -fn watch_dir_recursively(root: &Path) { - if !root.exists() { - return; - } - let mut stack = vec![root.to_path_buf()]; - while let Some(dir) = stack.pop() { - let Ok(read) = fs::read_dir(&dir) else { - continue; - }; - for entry in read.flatten() { - let path = entry.path(); - // Always ask Cargo to rerun if this path changes - if let Some(path_str) = path.to_str() { - println!("cargo:rerun-if-changed={path_str}"); - } - if path.is_dir() { - stack.push(path); - } - } - } -} From ae47b4350b0297be935f258c9e9d81d22426bf4c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:50:52 -0700 Subject: [PATCH 078/104] Move the EdgeZero pin to the PR 381 branch head 5878eb74 The only upstream commit since 119fbf80 fixes a shellcheck finding in EdgeZero's CI cache action. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index c2d6c9f17..25197f4e6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "log", "proc-macro2", From e3f371f8cb102d4b7f713d375f8587a34c6f01fd Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Sat, 19 Sep 2026 11:34:08 +0530 Subject: [PATCH 079/104] Reconcile mobile trace design contracts --- ...-mobile-ad-render-trace-endpoint-design.md | 419 +++++++++++++----- 1 file changed, 302 insertions(+), 117 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index 092466a6d..5d86b9092 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -14,7 +14,8 @@ ## 1. Summary -Add a deployment-controlled, public, privacy-safe `GET /_ts/trace` page for a +Add a deployment-controlled, privacy-safe `GET /_ts/trace` page, public subject +to operator authentication rules, for a mobile end user who needs to reproduce an ad-rendering problem and give support an exportable diagnostic report. Visiting the page is read-only. The user intentionally enables or ends tracing with a same-origin POST action. @@ -44,9 +45,10 @@ Issue #1050 names three required data groups: 3. End-user cookie information. The title additionally establishes two product constraints: the experience is -for a mobile user, and it is reached through an endpoint. A mobile user should -not need browser developer tools, Basic Authentication, a copied trace ID, or a -second copy of the affected page URL. +for a mobile user, and it is reached through an endpoint. On a deployment whose +authentication rules leave trace routes public, a mobile +user should not need browser developer tools, credentials, a copied trace ID, +or a second copy of the affected page URL. A standalone request cannot know what occurred in a previous document. Exact render evidence exists only while the publisher page is running in the browser. @@ -117,10 +119,13 @@ diagnostic token connects those layers. ### 5.1 Public, redacted endpoint with intentional activation -`/_ts/trace` is public when explicitly enabled by deployment configuration. It -is not placed under `/_ts/admin`, because the intended user is a layperson on a -phone and the existing Basic Authentication flow is unsuitable for that -journey. +`/_ts/trace` is available when explicitly enabled by deployment configuration +and is public only when no operator authentication rule covers it. It is not +placed under `/_ts/admin`, because the intended user is a layperson on a phone. +Existing Basic Authentication rules still apply to every trace path, including +assets and actions; the early dispatcher must not carve out an exemption. +Operators offering the credential-free journey must scope authentication to +the paths they intend to protect, for example `^/_ts/admin` (see section 12.5). Public access is safe only because both the page and export use a strict allowlist. The activation cookie is a feature toggle, not authentication. No @@ -141,19 +146,51 @@ executing on the publisher origin. The endpoint reuses `__Host-ts-console` and the existing GPT diagnostics activation semantics rather than creating a second `ts-trace` session. The cookie remains host-only, `Secure`, `HttpOnly`, and `SameSite=Lax`. -`POST /_ts/trace/enable` sets it with a fixed 30-minute `Max-Age` and does not -refresh that lifetime on publisher requests; `POST /_ts/trace/end` clears it. -Neither action accepts state-changing query parameters. The shorter endpoint -lifetime bounds accidental private/no-store operation if a user forgets to end -tracing; the existing technical query flow keeps its existing session-cookie -semantics. - -The new trace capture is active only when both `trace_page_enabled = true` and -the request carries exactly one valid diagnostics cookie. The shared cookie by +Both `POST /_ts/trace/enable` and the existing `?ts_console=1` writer must +set `Max-Age=1800` through one shared cookie policy, including when +`trace_page_enabled` is false. This explicitly changes the technical query flow +from a browser-session cookie to a 30-minute cookie. An explicit activation +restarts the 30-minute lifetime; ordinary publisher requests do not refresh it. +`POST /_ts/trace/end` and `?ts_console=0` clear the same cookie. Neither POST +action accepts state-changing query parameters. A later query activation must +never replace the bounded cookie with a session cookie. + +This bounds browser-managed persistence from the most recent activation by an +updated writer, not total session duration or server-enforced authorization. +Pre-existing session cookies carry no expiry in the request and cannot be +retroactively aged; users with those cookies must end or re-enable diagnostics +to adopt the new lifetime. Cookie expiry also does not unload a running page +or clear an existing report; report expiry is defined separately in section 9.5. + +The base trace gate requires both `trace_page_enabled = true` and exactly one +valid incoming diagnostics cookie, inspected before cookie sanitation. +Publisher-document tracing additionally requires the existing effective +`GptDiagnosticsRequestDecision.active` decision. Query disable or invalid +directives, prefetches, bots, and other ineligible navigations therefore +suppress document trace context, auction evidence, and browser activation even +when an incoming cookie is valid. A query activation without an incoming valid +cookie enables the existing console; trace capture starts on the next eligible +reload carrying that cookie. Page-bids and `/auction` requests use the base +configuration-plus-cookie gate without requiring a document-navigation +decision. References below to the trace gate include the additional effective +diagnostics decision for publisher documents. The shared cookie by itself continues to activate the existing TS Console but does not authorize trace tokens, server-auction projection, trace response extensions, or correlation sidecars on a deployment whose trace page is disabled. This makes -the configuration flag the disclosure and rollback boundary. +the configuration flag the disclosure and rollback boundary for new requests. + +Core exposes the evaluated document trace gate as the literal boolean +`window.__tsjs_trace_active` before TSJS initializes on the publisher document; +it is true only when the base gate and effective document diagnostics decision +are both active. TSJS requires +`window.__tsjs_trace_active === true` before minting slot tokens, retaining +request mappings, installing trace listeners, collecting transport evidence, +emitting sidecars, or offering the trace handoff action. Missing or non-boolean +values mean inactive; `window.__tsjs_gpt_diagnostics_active` alone is +insufficient. Core independently rechecks the gate on each request and never +trusts the browser flag as authorization. Configuration changes or cookie expiry +cannot revoke code already loaded in a document; rollback takes effect on its +next reload, and subsequent server requests stop returning evidence immediately. ### 5.3 Browser-local, explicit handoff @@ -324,7 +361,7 @@ the report. ```text GET /_ts/trace | - |-- early reserved-route classifier terminates locally + |-- early reserved-route classifier applies configured auth, then terminates locally |-- HTML explains forward reproduction; no state mutation v POST /_ts/trace/enable after explicit user action @@ -411,7 +448,7 @@ enabled = true trace_page_enabled = false ``` -Rules: +Rules (route responses below apply after configured authentication): - `trace_page_enabled = true` requires `enabled = true`; invalid combinations fail configuration validation. @@ -481,9 +518,15 @@ Every adapter implements the following order: 1. Parse the method, canonical host/origin, path, query, and bounded headers required for route safety. 2. Classify an exact Trusted Server reserved path. -3. For a trace path, terminate locally after only trace-specific validation and - bounded request-context inspection, including an optional read-only platform - geo lookup used solely for the displayed setup request. +3. For a trace path, enforce the existing configured Basic Authentication + rules before any trace response, cookie mutation, body inspection, or setup + context projection. A matching rule challenges missing/invalid credentials + locally with the ordinary `401` and `WWW-Authenticate` behavior. After + authentication succeeds or no rule matches, apply the trace-specific + validation and bounded request-context inspection, including an optional + read-only platform geo lookup, and terminate locally. This applies to the + entire reserved namespace, assets, unsupported methods, and disabled routes; + the route statuses below authentication are never an auth exemption. 4. For all other paths, continue through the adapter's ordinary event context, authentication, request filters, geo enrichment, EC/EID processing, named routes, auction handling, telemetry, and publisher fallback. @@ -496,8 +539,9 @@ ordinary named-route registration alone does not satisfy this contract. After an enable or end POST succeeds, the client performs a no-store state GET. It claims `Tracing is on — cookie observed by server` only when that separate -request reports active, and `Tracing is off — cookie absent on server request` -only when it reports inactive. A mismatch or failed verification is +request reports active, and `Tracing is off — no valid diagnostics session +observed` only when it reports inactive. An inactive result covers absent, +invalid, duplicate, or uninspectable cookies; it does not prove cookie absence. A mismatch or failed verification is `Activation unconfirmed` or `Deactivation unconfirmed` and offers an idempotent retry. These are server-observation statements, not proof that browser state is authentic: same-origin service workers can forge or suppress the whole exchange. @@ -508,7 +552,8 @@ the uncached shell, state, and action routes being disabled; inert cached assets alone cannot activate tracing or access a report page. The current `?ts_console=1` and `?ts_console=0` activation flow remains -supported for technical users. Both activation surfaces drive the same cookie +supported for technical users with the shared 30-minute cookie policy in +section 5.2. Both activation surfaces drive the same cookie and runtime; they must not create two concurrent diagnostic modes. That pre-existing query flow has its existing top-level-navigation activation risk; #1050 neither expands it to the new trace GET nor claims to remediate it. @@ -517,8 +562,8 @@ pre-existing query flow has its existing top-level-navigation activation risk; ### 9.1 Request context -The server injects one immutable `TraceRequestContextV1` into active diagnostic -documents: +The server injects one immutable `TraceRequestContextV1` only into documents +that satisfy the applicable trace gate in section 5.2: ```text TraceRequestContextV1 @@ -546,8 +591,12 @@ The request-context envelope intentionally contains no page URL, path, referrer, query, or fragment. During the field-by-field trace projection, `GptDiagnosticsExportV1.page.origin` is retained after validation and its `pathname` is replaced with the literal `/[redacted]`. The trace viewer accepts -only that literal. Version one therefore does not store or export an exact page -path. Any future route-template policy requires a new schema and privacy review +only that literal. The projection also omits `slotElementId` and `adUnitPath` +from slots and `slotElementId` from callback and attribution issues, because +these values can embed the same page path. They are not hashed or truncated. +Numbered slots and request cycles retain grouping and exact-token correlation. +Version one therefore does not intentionally retain an exact page path in any +of these source fields. Any future route-template policy requires a new schema and privacy review because paths can contain accounts, emails, preview tokens, and other secrets. `masked_client_ip` uses a deterministic display-only mask for the current @@ -661,13 +710,26 @@ projection sourced only from `GptDiagnosticsExportV1`. It contains: - `schema_version: 1` and `source_schema_version: 1`; - the source `capturedAt` value; - `page.origin` after validation and `page.pathname` fixed to `/[redacted]`; -- field-for-field allowlisted copies of the current v1 slots, requests, - callback issues, attribution issues, coverage, and metadata, subject to the - bounds and truncation below. +- explicit field-by-field projections of current v1 slots, requests, callback + issues, attribution issues, coverage, and metadata, subject to the following + exclusions and the bounds/truncation below. + +The trace schema omits the entire request-cycle `adManager` object, including +`lineItemId`, `creativeId`, `campaignId`, `advertiserId`, +`sourceAgnosticLineItemId`, `sourceAgnosticCreativeId`, `yieldGroupIds`, and +`companyIds`, and omits `previousCreativeId`. Derived `responseClass` and +`creativeChanged` facts remain eligible without their underlying identifiers. +It also omits slot `slotElementId` and `adUnitPath` and callback/attribution +issue `slotElementId`. These properties are forbidden in the trace schema, +not optional passthrough fields: the builder never copies them and the viewer +rejects a stored report that supplies them. The remaining current v1 fields +retain their source meaning and require explicit allowlisting; future source +fields are not inherited automatically. `trustedServerAuctionId`, when +present, must satisfy the diagnostic auction-token contract in section 9.4. It is deliberately not named or represented as `GptDiagnosticsExportV1`, -because the fixed pathname and trace-level bounds change the source field -semantics. TS Console continues to own the source schema; the trace envelope +because redaction, excluded identifiers, and trace-level bounds change the +source field semantics. TS Console continues to own the source schema; the trace envelope owns its public projection and transport. The initial compatibility matrix is exactly `TraceReportV1`, `TraceAuctionEvidenceV1`, `TraceSlotCorrelationV1`, and `TraceGptDiagnosticsV1`, with the GPT projection sourced from @@ -678,13 +740,17 @@ an additive source compatibility change and, if the public projection changes, a new trace-envelope version. `auction_coverage.capture_status` describes only what reached the browser -collector: `not_observed` means no valid server-auction record arrived, not that -no server auction ran. `partial` requires at least one retained record plus a -projection, transport, validation, or eviction issue; `unavailable` requires no -retained records plus a known projection, transport, or validation issue; and +collector: `not_observed` means no valid server-auction record arrived and no +capture issue is known, not that no server auction ran. `partial` requires at +least one retained record plus a projection, transport, validation, or eviction +issue; `unavailable` requires no retained records plus a known projection, +transport, validation, or eviction issue; and `complete` requires at least one retained record without those capture issues. `correlation_unavailable` and `external_client_side_unobservable` describe interpretation limits and do not change an otherwise complete capture status. +Evicting all received records therefore yields `unavailable` with +`record_evicted`, never `not_observed`. Recompute status after both in-memory +eviction and snapshot size truncation. The issue array is deduplicated, sorted in enum order, bounded to 16 values, and contains no error text. @@ -739,24 +805,29 @@ The model has deliberately lower cardinality and sensitivity than the existing telemetry and OpenRTB objects: - `diagnostic_auction_id` is a fresh opaque `ts-auc-...` correlation token. When - trace capture is active under the two-part gate in section 5.2, it is minted + trace capture is active under the applicable trace gate in section 5.2, it is minted once when an eligible auction is observed, before dispatch, and is retained for zero-bid, skipped, dispatch-failed, execution-failed, and abandoned outcomes. It is never `AuctionRequest.id`, the telemetry UUID, a provider request ID, or an identifier joinable to user-bearing logs. -- Auction and slot tokens are the fixed prefixes `ts-auc-` and `ts-slot-` - followed by a canonical lowercase hyphenated UUID v4. Validators reject every - other shape; tokens are not silently shortened or normalized. +- Auction tokens are `ts-auc-` followed by a lowercase UUID v4 in the existing + producer's 32-hex-digit unhyphenated form (`Uuid::new_v4().simple()`). Slot + tokens are `ts-slot-` followed by a canonical lowercase hyphenated UUID v4, + matching `crypto.randomUUID()`. Both validators enforce UUID version 4 and + the RFC variant and reject every other shape. Tokens are compared verbatim, + never normalized during validation or correlation; no producer format change + is required for the existing GPT auction opportunity marker. - `slot_number` is a one-based ordinal over the exact post-conversion `AuctionRequest.slots` sequence observed by orchestration. It is display-only and is never used to map a response back to pre-conversion client input. `slot_ref` is a fresh auction-local opaque token carried with that slot. Core creates it for initial-navigation and SPA auctions. For a TSJS `/auction` - request, TSJS creates it only after `buildAdRequest` has finished grouping and + request, TSJS creates it only when `window.__tsjs_trace_active === true` and + only after `buildAdRequest` has finished grouping and deduplicating the final `adUnits` array, attaches it to that exact outgoing unit as `adUnits[].ext.trusted_server.trace_slot_ref`, and retains the request-scoped token-to-unit mapping. Core accepts that member only under the - two-part trace gate, validates and echoes the token for accepted converted + applicable trace gate, validates and echoes the token for accepted converted slots, and strips it before every provider or mediator request. A missing or invalid client token causes core to mint a server token with no browser correlation; it never changes ordinary auction acceptance. TSJS uses @@ -812,8 +883,8 @@ correlation failed, the viewer shows `Server auction evidence unavailable` or #### 9.4.1 Live transport and correlation -Evidence is transported only while `trace_page_enabled` is true and the -diagnostics cookie is valid. Every response carrying it is terminally +Evidence is transported only when the applicable trace gate in section 5.2 +is active, including the effective diagnostics decision for publisher documents. Every response carrying it is terminally `private, no-store`: ```text @@ -836,7 +907,7 @@ object TSJS already consumes: AuctionSlot.ext.trusted_server.trace_slot_ref: string ``` -Core adds that optional nested member only under the two-part trace gate. It +Core adds that optional nested member only under the applicable trace gate. It assigns the token while constructing the request-scoped slot definitions and threads the same token into the corresponding `AuctionRequest` observation, so neither side needs to recover the relationship from an ordinal or raw slot ID. @@ -845,13 +916,15 @@ ordering, and bid-map keys remain unchanged. The extension is absent when the gate is false and is never copied into `TraceAuctionEvidenceV1` except as its already-allowlisted opaque `slot_ref`. -TSJS accepts a slot extension only when its canonical token occurs exactly once +When valid auction evidence is supplied, TSJS accepts a slot extension only +when its canonical token occurs exactly once in both the delivered slot list and the matching auction evidence. A missing, malformed, duplicate, or conflicting token prevents only that sidecar join, adds `evidence_validation_failed` and `correlation_unavailable`, and does not drop, reorder, or mutate the ordinary slot or bid. TSJS reads no other extension property. This validation occurs before the slot is handed to the existing GPT -initialization path. +initialization path. An absent optional transport member does not trigger +missing-token validation; no sidecar is emitted and no capture issue is added. The three transport call shapes are exact v1 contracts: @@ -876,8 +949,9 @@ accepts only the exact optional top-level member and preserves its existing `slots` and `bids` behavior. These trace members never become required for a successful advertising response. -When the existing GPT recorder consumes the matching Trusted Server opportunity -for a concrete request cycle, it emits this trace-owned sidecar: +For initial-navigation SSAT and SPA page-bids only, when the existing GPT +recorder consumes the matching Trusted Server opportunity for a concrete request +cycle, it emits this trace-owned sidecar: ```text TraceSlotCorrelationV1 @@ -895,6 +969,19 @@ opaque server tokens and the concrete GPT cycle are present. The current `GptDiagnosticsExportV1` remains unchanged and the sidecar contains no slot element ID or ad-unit path. +Version one does not correlate either `/auction` caller to a GPT cycle. Prebid +refresh records only `prebid_refresh` intent and has no existing token-bearing +opportunity binding; passing its tokens through `recordTrustedServerOpportunity` +would incorrectly introduce `trusted_server_direct` attribution. The direct +`requestAds` caller renders outside GPT and supplies no GPT request cycle. Both +callers retain server evidence and the request-unit mapping, but emit no +`TraceSlotCorrelationV1`. The viewer displays API evidence independently with +`correlation_unavailable`; this interpretation limit does not downgrade an +otherwise complete server capture. A sidecar referencing an `auction_api` +record is invalid in v1. A future Prebid join requires a separately designed +association between request-unit tokens and exact GPT slot/request identities +that preserves existing request-path attribution. + - **Initial-navigation SSAT:** core builds the evidence when the split auction is collected at the held body tail. It injects the script-safe public model beside the winning-bid map before initial ad initialization; the corresponding @@ -908,12 +995,17 @@ element ID or ad-unit path. `ext.trusted_server.trace_slot_ref` before triggering ad initialization. Both the envelope and slot extensions are absent when the trace gate is inactive. - **Trusted Server `/auction` API:** the existing OpenRTB response adds a - namespaced `ext.trusted_server.trace_auction` transport envelope only for an - active diagnostics request. After producing the final grouped `AdRequest`, - both TSJS callers assign one fresh token to each outgoing unit and retain that - exact request-scoped mapping. They validate the echoed evidence and record it + namespaced `ext.trusted_server.trace_auction` transport envelope only for a + request satisfying the base trace gate in section 5.2. After producing the + final grouped `AdRequest`, + both TSJS callers check `window.__tsjs_trace_active === true` before + assigning one fresh token to each outgoing unit and retaining that exact + request-scoped mapping. An inactive caller creates neither tokens nor pending + transport records and installs no trace-specific timeout/error hooks. Active + callers validate the echoed evidence and record it before parsing bids. A converted or skipped unit therefore cannot shift - another slot's correlation. The response member does not replace or expose + another request unit's server-evidence association. This mapping does not + imply a GPT-cycle join. The response member does not replace or expose the existing orchestrator extension, and the trace projection must not copy that extension's provider names, bidder names, metadata, price, creative IDs, domains, or markup. HTTP/transport failures with no readable response are @@ -933,8 +1025,9 @@ transport outcome, so it removes the marker and leaves evidence `not_observed` rather than inventing a failure. These hooks collect only bounded categories and opaque tokens, never XHR error text or response bodies. -The diagnostic auction token is also attached to the existing GPT opportunity -marker, and the opaque slot token is carried through the corresponding +For SSAT and SPA page-bids, the diagnostic auction token is also attached to +the existing GPT opportunity marker, and the opaque slot token is carried +through the corresponding winning-bid/slot initialization path. The numeric ordinal is never a correlation key. The viewer joins a server slot to a GPT cycle only when one validated `TraceSlotCorrelationV1` exactly matches both tokens and the exported @@ -945,9 +1038,10 @@ timestamps, implicit array position, ad-unit path, or a best-effort heuristic. TSJS retains at most the newest 16 validated server-auction records and 128 correlation sidecars in memory. It increments checked eviction counters for older records; the snapshot adds those counts to the matching truncation fields -and emits `record_evicted` with `partial`. It performs no storage write until +and emits `record_evicted`, with `partial` if server records remain or +`unavailable` if none remain after snapshot truncation. It performs no storage write until the explicit snapshot action. -Requests for which either side of the trace-capture gate is false do not mint +Requests that fail the applicable trace gate do not mint trace tokens, build trace evidence, add response members, emit sidecars, or install auction-evidence listeners. @@ -993,32 +1087,40 @@ failure is handled even when the report is below the application limit. Runtime limits are part of the v1 contract: -| Value | Limit | -| --------------------------------------------- | ------------------------------------------------------------ | -| Container nesting | 8 levels | -| Server auctions | 16 | -| Slot correlations | 128 | -| Provider calls | 16 per server auction | -| Auction slots | 64 per server auction | -| Auction coverage issues | 16 | -| Slots | 64 | -| Request cycles | 10 per slot before total-size truncation | -| Callback issues | 128 | -| Attribution issues | 128 | -| Requested slot sizes | 16 per cycle | -| Ad Manager yield-group or company IDs | 8 of each per cycle | -| Creative-failure enums | 16 per cycle | -| Origin | 255 UTF-8 bytes | -| GPT pathname in trace projection | Exact literal `/[redacted]` | -| Slot element ID and ad-unit path | 512 UTF-8 bytes each | -| Trusted Server auction ID and callback reason | 256 UTF-8 bytes each | -| Diagnostic auction ID and opaque slot ref | 128 UTF-8 bytes each | -| Any other string | 128 UTF-8 bytes | -| Enum | Exact documented value only | -| Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | -| Browser-relative timestamp or duration | Finite number from 0 through `Number.MAX_SAFE_INTEGER` | -| Visibility percentage | Finite number from 0 through 100 | -| Slot dimension | Finite integer from 1 through 100,000 | +| Value | Limit | +| ----------------------------------------------- | ------------------------------------------------------------ | +| Container nesting | 8 levels | +| Server auctions | 16 | +| Slot correlations | 128 | +| Provider calls | 16 per server auction | +| Auction slots | 64 per server auction | +| Auction coverage issues | 16 | +| Slots | 64 | +| Request cycles | 10 per slot before total-size truncation | +| Callback issues | 128 | +| Attribution issues | 128 | +| Requested slot sizes | 16 per cycle | +| Creative-failure enums | 16 per cycle | +| Origin | 255 UTF-8 bytes | +| GPT pathname in trace projection | Exact literal `/[redacted]` | +| Slot element ID and ad-unit path | Forbidden, including callback/attribution issue copies | +| Trusted Server auction ID | Exact diagnostic auction-token shape | +| Callback reason | Exact documented value only | +| Diagnostic auction ID and opaque slot ref | 128 UTF-8 bytes each | +| Any other string | 128 UTF-8 bytes | +| Enum | Exact documented value only | +| Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | +| Browser-relative timestamp or duration | Finite number from 0 through `Number.MAX_SAFE_INTEGER` | +| Visibility percentage | Finite number from 0 through 100 | +| Requested or selected creative dimension | Finite integer from 1 through 100,000 | +| GPT-reported fill dimension (`size`) | Finite integer from 1 through 100,000 | +| Observed CSS box dimension (`observedSlotSize`) | Finite integer from 0 through 100,000 | + +`observedSlotSize` preserves zero dimensions, including `[0, 0]` for a +hidden or collapsed element after a filled GPT render. Zero is a measured box +size, not missing data or evidence of an empty GPT response; do not omit it or +change `isEmpty`. Positive dimensions remain required for requested and +selected creative sizes and GPT-reported fill sizes. Every accepted string must be valid Unicode and must not contain C0/C1 control characters or bidirectional override/isolate controls. This applies to browser @@ -1029,8 +1131,7 @@ invalid source value rather than stringifying it. Server auctions are already bounded by core; the browser rejects an invalid inner model rather than truncating it. The builder retains the newest 16 server auctions in observation order, the newest 128 correlations in recorder emission order, and only the -first documented number of GPT requested sizes, yield-group IDs, company IDs, -and creative failure enums. It records each discard in +first documented number of GPT requested sizes and creative failure enums. It records each discard in `omitted_server_auctions`, `omitted_slot_correlations`, or `omitted_nested_values`; strings are never silently shortened. It then measures the complete compact UTF-8 storage wrapper. If it exceeds 512 KiB, it removes @@ -1180,7 +1281,9 @@ Forbidden data includes: - EC IDs, EIDs, bidder user IDs, provider/bidder/seat names, and consent strings. - Unmasked client IP. -- Query strings and fragments. +- Exact page paths, slot element IDs, ad-unit paths (including issue copies), + query strings, and fragments. +- The entire GPT `adManager` identity object and `previousCreativeId`. - Fastly or internal request identifiers that can join to user-bearing logs. - Internal `AuctionRequest.id`. - Bid requests/responses, bid prices/currency, losing-bid payloads, provider @@ -1233,9 +1336,11 @@ hash and a corresponding CSP change. Validated report strings enter the document through `textContent` or equivalent DOM properties, never `innerHTML`. The JS asset uses `application/javascript; charset=utf-8`; the CSS asset uses -`text/css; charset=utf-8`. Both send `X-Content-Type-Options: nosniff`, -`Cache-Control: public, max-age=31536000, immutable`, and a strong ETag derived -from their build bytes. They accept no dynamic input. `script-src 'self'` is an +`text/css; charset=utf-8`. Both send `X-Content-Type-Options: nosniff` and a strong ETag derived +from their build bytes. They use +`Cache-Control: public, max-age=31536000, immutable` only when no configured +authentication rule covers the asset; otherwise they use `private, no-store` +as specified in section 12.5. They accept no dynamic input. `script-src 'self'` is an origin-level CSP permission, not a path restriction; same-origin script interference remains inside the stated trust limitation. @@ -1249,9 +1354,29 @@ The existing diagnostics private/no-store decision remains a load-bearing gate. Tests must prove that late response-header handlers cannot make traced content publicly cacheable. +### 12.5 Operator authentication policy + +Trace routing preserves the existing `auth.rs` namespace contract: every +matching operator Basic Authentication rule is enforced, including `^/_ts` or +`^/`. Classification may run early, but does not authorize a request. Only +authentication is factored ahead of trace handling; ordinary event, identity, +filter, and auction processing stays outside trace routes. A challenge exposes +no setup context, changes no cookie, and is terminally private/no-store. + +The credential-free mobile journey requires deployment rules that leave the +trace namespace public. Document this beside `trace_page_enabled`; do not +silently narrow an operator rule. On origin requests, authentication precedes +disabled-route `404` and unsupported-method `405` responses as well. Asset +responses covered by an authentication rule must use `private, no-store` +instead of public immutable caching. Previously public cached inert assets may +remain available, but contain no report data and cannot bypass the uncached +shell or actions. + ## 13. Failure handling -- Disabled route: local privacy-safe `404`. +- Configured authentication failure: local private/no-store `401` challenge + before trace handling, including on disabled routes. +- Disabled route after authentication: local privacy-safe `404`. - Unsupported method: local `405`; never publisher fallback. - Rejected activation/end POST: local `403` with no state mutation. - Optional platform fact unavailable: omit the field and continue. @@ -1265,9 +1390,14 @@ publicly cacheable. - Initial-navigation evidence cannot be injected because the body tail is not reached: preserve publisher delivery. Because the browser received no safe marker, display `not_observed` rather than claiming a known server failure. -- Page-bids or `/auction` evidence is absent or rejected by its strict client - validator: parse the ordinary bid response exactly as before, discard the - diagnostic member, and show an unmatched/invalid-evidence coverage category. +- A successful page-bids or `/auction` response omits the optional evidence + member: parse ordinary bids unchanged and add no capture issue. With no other + records or known capture issues, coverage is `not_observed`; absence does not + reset previously collected evidence or failures. +- A supplied evidence member fails strict validation: discard that member, + parse ordinary bids unchanged, and add `evidence_validation_failed`. Coverage + is `partial` when valid records remain or `unavailable` when none remain, + following section 9.3. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. - Storage unavailable or quota exceeded after a valid bounded report exists: @@ -1306,15 +1436,24 @@ results, never a prerequisite for returning them. console cookie still enables the existing console but never mints trace tokens, builds auction evidence, adds response extensions, or emits correlation sidecars. +- With a valid incoming cookie, `?ts_console=0`, invalid/duplicate directives, + prefetches, bots, and other ineligible navigations suppress all document trace + capture through the effective diagnostics decision. Query enable without a + cookie activates only the console until the next eligible cookie-bearing + reload. Page-bids and `/auction` use the base gate without navigation checks. - Exact reserved-route classification, canonical-path, query, method, encoded path, and fallback behavior. - Exact versioned asset routes are local and contain no dynamic data; lookalike asset paths never reach the publisher origin. - Same-origin POST validation, cross-site/missing signal rejection, cookie - set/clear attributes, fixed 30-minute endpoint activation without request - refresh, and idempotent enable/end behavior. + set/clear attributes, shared `Max-Age=1800` for endpoint and query activation + without ordinary-request refresh, and idempotent enable/end behavior. Cover + enable then query activation, query then enable, repeated explicit activation, + clearing through either surface, and the pre-existing session-cookie caveat. - Enable/end success requires a separate state request to observe the resulting cookie; failed and mismatched verification never displays confirmed state. + Absent, invalid, duplicate, and uninspectable cookies all report inactive, + with no claim that inactive proves the cookie is absent. - Empty-body enforcement rejects positive/invalid lengths, transfer encoding, the first unexpected body byte, and the two-second deadline without an unbounded read. @@ -1330,7 +1469,8 @@ results, never a prerequisite for returning them. API call sites to the exact public source enums without using a browser hint. - The diagnostic auction token is minted before dispatch and remains identical across completed, zero-bid, skipped, failed, and abandoned evidence and the - corresponding browser opportunity marker. It never equals or contains the + corresponding SSAT/SPA browser opportunity marker. API tokens remain + server-evidence identifiers without GPT sidecars. No token equals or contains the internal auction ID or telemetry UUID. - Server-auction projection covers every terminal status/reason mapping, auction-local total duration, provider role/status/duration/count, per-slot @@ -1350,13 +1490,15 @@ results, never a prerequisite for returning them. associated across grouped multi-bidder units, duplicate codes, skipped non-banner units, and mixed accepted/filtered inputs. Numeric ordinals are never used for client correlation. -- Token tests cover canonical UUID-v4 shape, missing Web Crypto, malformed or +- Token tests cover the existing simple auction UUID-v4 shape and hyphenated + slot UUID-v4 shape, exact producer-to-sidecar joins, missing Web Crypto, malformed or duplicate request extensions, the disabled trace gate, and proof that invalid tokens neither fail nor otherwise alter the ordinary auction. - Initial and SPA slot JSON attaches the exact `ext.trusted_server.trace_slot_ref` token that appears in server evidence; - inactive responses omit it. Missing, duplicate, conflicting, malformed, and - evidence-mismatched slot tokens suppress only correlation and produce the + inactive responses omit it. With valid matching auction evidence supplied, + missing, duplicate, conflicting, malformed, and evidence-mismatched slot + tokens suppress only correlation and produce the specified coverage issues without changing slot/bid order or contents. - Active responses remain terminally private/no-store under hostile late header overrides. @@ -1369,6 +1511,11 @@ results, never a prerequisite for returning them. - Axum, Cloudflare, and Spin return the common route/schema with unavailable fields omitted. - Trace-route failures never fall through to publisher origin. +- Broad `^/_ts` and `^/` authentication rules challenge every trace path + (shell/state/actions/assets, disabled routes, and unsupported methods); valid + credentials proceed to trace handling, while `^/_ts/admin` leaves trace + routes public. Challenges expose no context or cookie mutation, and + protected assets remain private/no-store. - GET, HEAD, state-changing POST, and unsupported methods obey the same lifecycle contract across adapters. - Every adapter omits JA4/H2 and rejects control characters or overlong platform @@ -1377,10 +1524,19 @@ results, never a prerequisite for returning them. ### 14.3 JavaScript unit tests - Explicit snapshot only; no continuous `sessionStorage` writes. +- Both TSJS auction callers require the literal `__tsjs_trace_active === true` + before token generation or trace collection. Cover missing/false/non-boolean + flags with GPT diagnostics active: no tokens, mappings, listeners, pending + transport records, sidecars, or handoff action are created. - Compact UTF-8 size measurement; exact outer/nested schema validation; unknown fields; per-string/array/numeric/depth caps; hostile mutation; expiry; future-clock skew; wall-clock rollback; replacement; clearing; and storage exceptions. +- A filled GPT cycle with `observedSlotSize: [0, 0]`, `[0, 250]`, or `[300, 0]` + survives projection, storage validation, rendering, and export unchanged. + Negative, fractional, non-finite, and over-limit observed dimensions fail + validation; requested/selected creative and GPT fill dimensions retain their + positive bounds. - Omission counters use checked arithmetic and reject overflow. - Strict validation of every server-auction enum, token, numeric bound, array bound, nesting level, and unknown property; invalid evidence is discarded @@ -1388,12 +1544,18 @@ results, never a prerequisite for returning them. - Transport envelopes require exactly one of evidence/unavailable reason; projection, transport, validation, eviction, correlation, and external-client coverage states produce the specified complete/partial/unavailable/not-observed - result without treating absence as proof that no auction ran. + result without treating absence as proof that no auction ran. A fixture that + receives valid evidence and then evicts every server record during size + truncation must yield `unavailable` with `record_evicted` and exact omission + counts; partial eviction remains `partial`. +- Successful responses without an optional transport member add no validation + issue; malformed supplied members add `evidence_validation_failed`. Cover + both an empty collector and one with retained evidence or earlier failures. - Direct fetch failures and Prebid `interpretResponse`, `onTimeout`, and `onBidderError` paths consume their pending transport record exactly once; capped/expired records and absent hooks follow the specified `not_observed` behavior without retaining error text or bodies. -- Exact-token correlation joins matching server auctions, GPT opportunities, +- Exact-token correlation joins matching SSAT/SPA server auctions, GPT opportunities, and slot references; unmatched, duplicated, conflicting, missing, and forged tokens stay separate and produce explicit coverage states. No timestamp, index, or ad-unit-path heuristic is used. @@ -1401,6 +1563,11 @@ results, never a prerequisite for returning them. opportunity-to-cycle binding, is capped and evicted deterministically, contains only opaque tokens plus runtime/request numbers, and does not alter `GptDiagnosticsExportV1`. +- Prebid `/auction` evidence remains independent of its `prebid_refresh` GPT + cycle, and direct `requestAds` evidence remains independent of GPT. Neither + path emits a sidecar or introduces `trusted_server_direct`/`competing` + attribution. Both show `correlation_unavailable` without downgrading complete + server capture; supplied API sidecars are rejected by the viewer. - Source presentation distinguishes server-owned SSAT/page-bids/auction API facts from browser-observed publisher refresh, Prebid refresh, competing, and unattributed request paths. No fixture turns intent or a GPT fill into a @@ -1413,7 +1580,12 @@ results, never a prerequisite for returning them. fixture. - Same-tab navigation occurs only after a successful write. - Viewer handles absent optional network facts and every cookie-health state. -- Forbidden fields never enter storage or export fixtures. +- Populate every excluded `adManager` field, `previousCreativeId`, slot + `slotElementId`/`adUnitPath`, and callback/attribution issue `slotElementId` + with distinct sentinel values, including a synthetic secret-bearing path. + Verify none enter trace HTML, storage, copy/share, or direct export; hostile + stored copies containing any excluded property are rejected. Numbered + slot/cycle correlation still joins after redaction. - Download filename and MIME type are deterministic. - Formatted-JSON copy and JSON-file Web Share success, rejection, absence, and download/copy fallback behavior. @@ -1440,7 +1612,7 @@ results, never a prerequisite for returning them. transport fixtures preserve ad initialization while reporting `not_observed`. - `View trace results` navigates in the same tab and renders the captured request context and slot evidence. -- A correlated fixture renders the chain `server auction -> GPT -> creative`, +- A correlated SSAT/SPA fixture renders the chain `server auction -> GPT -> creative`, while unmatched server, client-side refresh, competing, and transport-failure fixtures show honest independent evidence and `Unknown` where appropriate. - Empty, filled, ambiguous, no-candidate, and unattributed slot states remain @@ -1468,8 +1640,10 @@ results, never a prerequisite for returning them. - Inactive publisher traffic has no trace assets, storage access, listeners, or cache-policy change, diagnostic token generation, or trace-auction response extension. -- Disabling `trace_page_enabled` removes every new capture behavior even when a - technical `?ts_console=1` session leaves a valid diagnostics cookie present. +- After disabling `trace_page_enabled`, a fresh publisher load removes every + new capture behavior even when a technical `?ts_console=1` session leaves a + valid diagnostics cookie present. Already-loaded documents cannot be remotely + deactivated, but subsequent server responses contain no trace evidence. ### 14.5 Manual acceptance @@ -1500,12 +1674,17 @@ observed`, `GPT filled/rendered`, and `Unknown` without understanding internal ## 16. Acceptance criteria -1. With the feature disabled, trace-route origin requests return local `404` - and ordinary traffic is unchanged. Previously cached inert versioned assets +1. With the feature disabled, trace-route origin requests that pass configured + authentication return local `404` + and ordinary requests without explicit diagnostics activation are unchanged. + Query activation adopts the shared cookie lifetime in section 5.2. + Previously cached inert versioned assets may remain until cache eviction, but cannot activate tracing or load a shell. -2. A mobile user can enable tracing by opening only `/_ts/trace` and selecting +2. On a deployment whose authentication rules leave trace routes public, a + mobile user can enable tracing by opening only `/_ts/trace` and selecting one prominent action; no target URL, credentials, or trace ID is required, - and a cross-site GET cannot activate tracing. + and a cross-site GET cannot activate tracing. Matching operator auth rules + remain enforced on all trace paths. 3. The setup page accurately explains that the problem must be reproduced after activation. 4. A subsequent real publisher-page reload captures redacted request context, @@ -1552,7 +1731,8 @@ This design is one product flow, but its implementation is split into four independently reviewable plans and preferably four PRs: 1. **Reserved route and privacy foundation:** configuration, shared early-route - classification, same-origin enable/end lifecycle, bounded cookie-health + classification with operator authentication, same-origin enable/end lifecycle, + shared 30-minute endpoint/query cookie policy, bounded cookie-health inspection, base request-context schema, projection of already populated `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not add speculative new platform fields in this change. @@ -1593,9 +1773,11 @@ target validation and open-redirect risk, and is unsuitable for a layperson. ### Basic Authentication -Rejected for the mobile end-user workflow. Authentication also would not make -it safe to inject raw secrets into a publisher page containing third-party -JavaScript. +Rejected as a new mandatory prerequisite for the mobile end-user workflow. +Existing operator authentication rules remain enforced; deployment configuration +must leave trace routes public to offer the credential-free journey. +Authentication also would not make it safe to inject raw secrets into a +publisher page containing third-party JavaScript. ### Server-managed trace sessions @@ -1651,6 +1833,9 @@ length, history, logging, referrer, and accidental-sharing risks. - `/auction` evidence requires the TSJS request to reach the same Trusted Server host with the active diagnostics cookie. A custom cross-origin auction endpoint does not inherit this trace session and is shown as unavailable. +- Version one has no GPT correlation for `/auction`: the Prebid caller lacks + a token-bearing GPT binding, and direct `requestAds` renders outside GPT. + Their server evidence is displayed independently with correlation unavailable. - Exact-token correlation can remain unavailable for hidden, unresolved, competing, or independently initiated GPT cycles. The report preserves both sides instead of guessing. From 6e9c50ca51d8fd3a9295f134d95e92fddc128e7a Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 20 Sep 2026 00:32:31 -0700 Subject: [PATCH 080/104] Select the release manifest in the CLI deploy examples `ts deploy --application-release` does not choose the release's manifest. The CLI loads `edgezero.toml` from the working directory unless `EDGEZERO_MANIFEST` names another file, and it rejects a manifest outside the release root, so both documented commands failed from the checkout root with "loaded application manifest ... is outside application release root" before reaching Fastly. Set `EDGEZERO_MANIFEST` to the release's manifest in both examples, as the EdgeZero deploy action does, and say where the release root comes from. --- docs/guide/cli.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 04f8ef3f0..7231e3d82 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -122,13 +122,24 @@ adapter flags directly need the `--` added. Trusted Server declares Config, KV, and Secret Stores, so EdgeZero treats every Fastly deploy as managed and requires a verified application release root via `--application-release`; a bare `ts deploy --adapter fastly` without a release is accepted only for -store-free applications: +store-free applications. The CLI loads `edgezero.toml` from the working +directory unless `EDGEZERO_MANIFEST` names another file, and it rejects a +manifest outside the release root, so select the release's own manifest: ```bash -ts deploy --adapter fastly --service-id --application-release --staging -ts deploy --adapter fastly --service-id --application-release -- --comment "release" +EDGEZERO_MANIFEST="/edgezero.toml" \ + ts deploy --adapter fastly --service-id --application-release "" --staging +EDGEZERO_MANIFEST="/edgezero.toml" \ + ts deploy --adapter fastly --service-id --application-release "" -- --comment "release" ``` +`` is the extracted immutable application release that EdgeZero's +`package-fastly-application-release` action produces from a `ts build`. The +EdgeZero `deploy-fastly` action performs both steps, selects the manifest, and +supplies the release root; see EdgeZero's GitHub Actions deployment guide +(`docs/guide/deploy-github-actions.md` in the EdgeZero repository) for the +producer and consumer workflow. + A staged deploy selects the physical Config Store from the staging environment and links it to the staged version under the logical store ID. The staged runtime reads the `` key from that store. It does not copy From ed82795d1d4989a61e467473f57d423a70d1c81d Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Sun, 20 Sep 2026 13:53:32 +0530 Subject: [PATCH 081/104] Derive ALL_MODULE_IDS from TSJS_MODULES to prevent the two lists drifting Fix a compile error in the const-derived array and add a regression test pinning all_module_ids() to the generated TSJS_MODULES table. Signed-off-by: dhruv8sh --- crates/trusted-server-js/build.rs | 7 +------ crates/trusted-server-js/src/bundle.rs | 10 ++++++++++ 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-js/build.rs b/crates/trusted-server-js/build.rs index 784d87494..8fbd34c9e 100644 --- a/crates/trusted-server-js/build.rs +++ b/crates/trusted-server-js/build.rs @@ -131,7 +131,6 @@ fn main() { // Generate tsjs_modules.rs with include_str!() for each module let mut codegen = String::new(); - let mut ids_block = String::new(); codegen.push_str("// Auto-generated by build.rs - DO NOT EDIT\n\n"); writeln!( @@ -141,8 +140,6 @@ fn main() { ) .expect("should write generated module header"); for (id, filename) in &modules { - writeln!(ids_block, " \"{id}\",").expect("should write generated module ID"); - let sha256 = bundle_sha256(&out_dir.join(filename)); writeln!( codegen, @@ -153,12 +150,10 @@ fn main() { codegen.push_str("];\n"); writeln!( codegen, - "pub(crate) const ALL_MODULE_IDS: [&str; {}] = [", + "pub(crate) const ALL_MODULE_IDS: [&str; {0}] = {{\n let mut ids = [\"\"; {0}];\n let mut index = 0;\n while index < {0} {{\n ids[index] = TSJS_MODULES[index].id;\n index += 1;\n }}\n ids\n}};", modules.len() ) .expect("should write generated module IDs"); - codegen.push_str(&ids_block); - codegen.push_str("];\n"); codegen.push_str("\npub(crate) struct TsjsModuleMeta {\n"); codegen.push_str(" pub bundle: &'static str,\n"); codegen.push_str(" pub id: &'static str,\n"); diff --git a/crates/trusted-server-js/src/bundle.rs b/crates/trusted-server-js/src/bundle.rs index 762d51cf4..267f74495 100644 --- a/crates/trusted-server-js/src/bundle.rs +++ b/crates/trusted-server-js/src/bundle.rs @@ -138,6 +138,16 @@ mod tests { encode(Sha256::digest(bytes)) } + #[test] + fn all_module_ids_matches_generated_module_list() { + let from_modules: Vec<&str> = TSJS_MODULES.iter().map(|module| module.id).collect(); + assert_eq!( + all_module_ids(), + from_modules.as_slice(), + "the generated ID list should match the generated module table" + ); + } + #[test] fn generated_single_module_hashes_match_bundle_contents() { for id in all_module_ids() { From 95bd289427cf161442fbda484b160e6127f93d71 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Sun, 20 Sep 2026 14:05:06 +0530 Subject: [PATCH 082/104] Reorder root Markdown check, document npm ci prereq Signed-off-by: dhruv8sh --- .github/workflows/format.yml | 6 +++--- AGENTS.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 4df61cf25..c844e95d3 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -149,9 +149,9 @@ jobs: - name: Run Prettier (check) run: npm run format - - name: Build with VitePress (fails on dead links) - run: npm run build - - name: Run Prettier (check) — root Markdown working-directory: . run: docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" + + - name: Build with VitePress (fails on dead links) + run: npm run build diff --git a/AGENTS.md b/AGENTS.md index ac9e3c80c..aab22cae5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -345,7 +345,7 @@ Every PR must pass: 5. JS build and test (`cd crates/trusted-server-js/lib && npx vitest run`) 6. JS format (`cd crates/trusted-server-js/lib && npm run format`) 7. Docs format (`cd docs && npm run format`) -8. Root Markdown format (`docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md"`; fix with `--write` in place of `--check`) +8. Root Markdown format (requires `cd docs && npm ci` first): `docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md"`; fix with `--write` in place of `--check` --- From 3e8cde2b74e5328330c71f8b45a54ad5f0014241 Mon Sep 17 00:00:00 2001 From: Christian Pavilonis Date: Sun, 20 Sep 2026 11:09:09 -0500 Subject: [PATCH 083/104] Update crates/trusted-server-js/lib/src/integrations/prebid/index.ts Co-authored-by: prk-Jr <49094961+prk-Jr@users.noreply.github.com> --- .../trusted-server-js/lib/src/integrations/prebid/index.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 51be7d665..df589150d 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -518,7 +518,12 @@ function installPrebidWinDiagnostics(): void { const bid = rawBid as Record; const auctionId = typeof bid.auctionId === 'string' ? bid.auctionId : undefined; const adUnitCode = typeof bid.adUnitCode === 'string' ? bid.adUnitCode : undefined; - if (!auctionId || !adUnitCode) return; + if ( + !auctionId || + !adUnitCode || + (bid.latestTargetedAuctionId !== undefined && bid.latestTargetedAuctionId !== auctionId) + ) + return; const key = prebidDiagnosticKey(auctionId, adUnitCode); const attempt = prebidDiagnosticAttempts.get(key); prebidDiagnosticAttempts.delete(key); From d6fecdf8a24fba7a5a9b916cd0f3dfc6a6876877 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 20 Sep 2026 10:56:57 -0700 Subject: [PATCH 084/104] Move the EdgeZero pin to the PR 381 branch head 12c3215c Upstream merged main into the branch, which renames the staged-deploy identifiers to the staging naming. Trusted Server references none of the renamed variants, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 25197f4e6..62bca5610 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "log", "proc-macro2", From 14c274cbae3e33b338fe6119e98ec4470d4b075d Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Mon, 21 Sep 2026 10:50:53 +0530 Subject: [PATCH 085/104] Clarify mobile trace design contracts --- ...-mobile-ad-render-trace-endpoint-design.md | 190 +++++++++++++++--- 1 file changed, 160 insertions(+), 30 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index 5d86b9092..e3aaca4c6 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -451,7 +451,17 @@ trace_page_enabled = false Rules (route responses below apply after configured authentication): - `trace_page_enabled = true` requires `enabled = true`; invalid combinations - fail configuration validation. + fail configuration validation, including when `enabled` is omitted (defaults + to false). Add a raw-config validation hook in both deploy and runtime + validation, following `validate_js_asset_proxy_config` in + `crates/trusted-server-core/src/config.rs`. It must deserialize and validate + this combination outside the enabled gate: `IntegrationSettings::get_typed` + returns `Ok(None)` before `validate()` for disabled integrations, so a schema + validator alone cannot enforce this rule. Preserve unknown-field rejection + on disabled configurations. This spec explicitly requires invalid enabled + configuration to fail rather than be logged and disabled; the prior rationale + is the HIGH-severity finding in + `2026-03-11-production-readiness-report-design.md`, not a contributor-doc rule. - Operator documentation beside this option states that the public page makes the allowlisted presence/validity of four HttpOnly Trusted Server cookies visible to same-origin JavaScript whenever the feature is enabled. It also @@ -465,7 +475,12 @@ Rules (route responses below apply after configured authentication): parameters do not activate or deactivate tracing and are not reflected into the page or export. - `HEAD /_ts/trace` returns the GET status and headers without a body or state - mutation. + mutation. Explicitly handle `HEAD` on every trace path in all four adapters, + returning a bodyless local 405 on POST-only paths. Fastly's + `publisher_fallback_methods()` includes `HEAD`; a GET registration alone + sends HEAD to the publisher fallback, as pinned by + `dispatch_head_on_named_get_route_falls_through_to_publisher_fallback` in + `crates/trusted-server-adapter-fastly/src/app.rs`. - `GET /_ts/trace/state` returns private/no-store JSON containing only `observed_active: true|false`, determined from whether that request carried exactly one valid diagnostics cookie. `HEAD` returns the same status and @@ -485,13 +500,20 @@ Rules (route responses below apply after configured authentication): cookie using `X-TS-Trace-Action: end`, and returns a small local JSON result. After explicit user confirmation, client JavaScript independently attempts local report deletion and the end POST. Neither result gates the other. -- For both POST paths, absent or exactly-zero `Content-Length` is accepted, - `Transfer-Encoding` is rejected, and the adapter reads at most one byte when - it must verify an absent length. Any body byte or positive/invalid length - returns local `413 Payload Too Large` without draining or processing an - unbounded body. The one-byte read inherits a maximum two-second adapter - request-body deadline; timeout returns local `408 Request Timeout` with no - mutation. +- For both POST paths, accept absent or exactly-zero `Content-Length` only + when the body is empty; reject `Transfer-Encoding`, positive/invalid lengths, + and any actual body bytes with local `413 Payload Too Large` and no mutation. + Precheck headers, then use `Body::into_bytes_bounded(0)` to check emptiness, + following the header-precheck/body-size-check pattern in + `crates/trusted-server-core/src/auction/endpoints.rs`. This is an application + acceptance limit, not a transport read or allocation limit. Pinned EdgeZero + v0.0.8 buffers the Fastly body with blocking `read_to_end` and the Cloudflare + body with `req.bytes().await` before core handling. Spin also buffers the + body; Axum buffers JSON bodies but can stream other content types. `Body` + exposes no read deadline, and Fastly uses `futures::executor::block_on` without a timer. + Therefore v1 promises neither a one-byte transport read, a two-second timeout, + nor a local 408. Transport-level size/deadline protection requires a separate + adapter/upstream change; document the pre-buffering limitation at deployment. - State-changing POSTs require an `Origin` exactly matching the canonical request origin and `Sec-Fetch-Site: same-origin`. Missing, conflicting, malformed, cross-site, or duplicate control values return local `403` without @@ -503,7 +525,16 @@ Rules (route responses below apply after configured authentication): default ports removed before exact comparison. Invalid or multi-valued host, authority, scheme, or origin input fails closed. - Unsupported methods on a shell or state-changing path return a local 405 - Method Not Allowed response with the path-specific `Allow` header. + Method Not Allowed response with the path-specific `Allow` header: + `GET, HEAD` for shell/state/assets and `POST` for enable/end. Classification + must intercept unsupported methods before router dispatch. The router's + `MethodNotAllowed` response has no `Allow` header and bypasses + `FinalizeResponseMiddleware`, as pinned by + `dispatch_unregistered_method_returns_405_at_router_level` in the Fastly + adapter. The trace responder itself supplies `Allow` and all section 12.3 + error-response hardening; it must not rely on router-generated errors. + Fastly entry-point finalization can add ordinary headers later, but does not + establish this trace-specific contract on behalf of the router. - Disabled deployments return a local `404` for the complete trace route set, including assets, and never fall through to the publisher origin. - The `/_ts/trace` namespace is reserved. A trailing slash, extra path segment, @@ -513,6 +544,13 @@ Rules (route responses below apply after configured authentication): The adapter classifies from its canonical parsed path while retaining enough raw-path information to reject ambiguous encodings consistently. +Reuse the bounded percent-decode-to-fixed-point classification pattern from +`deny_admin_diagnostic_fallback` in `crates/trusted-server-core/src/ec/admin.rs` +(`MAX_PERCENT_DECODE_ROUNDS = 4`), moving trace classification before dispatch +rather than relying on fallback. Register and intercept trace paths on Fastly, +Axum, Cloudflare, and Spin from the first implementation PR; existing Fastly-only +`/_ts/*` routes are not a parity precedent. + Every adapter implements the following order: 1. Parse the method, canonical host/origin, path, query, and bounded headers @@ -650,7 +688,11 @@ The classifier uses this deterministic contract: precedence is therefore diagnostic rather than first- or last-value selection. - Per-value limits are 512 bytes for `ts-ec`, 8 KiB for `ts-eids`, and 16 bytes - each for `ts-tester` and `__Host-ts-console`. A single value beyond its limit + each for `ts-tester` and `__Host-ts-console`. Only the 8 KiB EID limit is + inherited (`MAX_EIDS_COOKIE_BYTES` in `ec/prebid_eids.rs`); the other limits + are new trace-inspection bounds. The 512-byte EC limit is an outer guard, + above the exact 71-character format accepted by `is_valid_ec_id` in + `ec/generation.rs`, and does not broaden EC validity. A single value beyond its limit is `present_invalid/oversized`; it does not change the other three states. - One `ts-ec` occurrence is valid only when the canonical EC cookie validator accepts its complete value. @@ -727,6 +769,40 @@ retain their source meaning and require explicit allowlisting; future source fields are not inherited automatically. `trustedServerAuctionId`, when present, must satisfy the diagnostic auction-token contract in section 9.4. +The following is the exhaustive v1 property allowlist derived from the current +interfaces in `crates/trusted-server-js/lib/src/core/types.ts`, after the above +exclusions. Preserve source optionality and validate each source enum against its +explicit current members; do not spread source objects or dynamically inherit +later fields. Section 9.5 supplies numeric, string, array, and depth bounds. + +| Object | Allowed properties | +| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| GPT projection | `schema_version`, `source_schema_version`, `capturedAt`, `page`, `slots`, `callbackIssues`, `attributionIssues`, `coverage`, `metadata` | +| `page` | `origin`, `pathname` (fixed redacted literal) | +| Slot | `runtimeSlotNumber`, `binding`, `currentVisibilityPercentage`, `maximumVisibilityPercentage`, `requests` | +| `binding` | `status`, `reason` (current `GptDiagnosticsBindingReason` enum) | +| Request cycle | `requestNumber`, `requestedAtMs`, `responseAtMs`, `renderAtMs`, `loadAtMs`, `viewableAtMs`, `durations`, `isEmpty`, `requestedSlotSizes`, `size`, `observedSlotSize`, `isBackfill`, `slotContentChanged`, `incompleteSequence`, `responseClass`, `requestPath`, `requestIntentId`, `trustedServerAuctionId`, `opportunityToRequestMs`, `replacedRequestNumber`, `previousRenderToRequestMs`, `creativeChanged`, `loadObservedBeforeRender`, `trustedServerOpportunity`, `trustedServerCreativeRequestAtMs`, `trustedServerCreativeResponseAtMs`, `trustedServerCreativeFailures`, `delivery` | +| `durations` | `requestToResponseMs`, `responseToRenderMs`, `requestToRenderMs`, `renderToLoadMs`, `renderToViewableMs` | +| Callback issue | `kind`, `runtimeSlotNumber`, `timestampMs`, `disposition`, `reason` (only the documented values below) | +| Attribution issue | `reason` (current `GptDiagnosticsAttributionIssueReason` enum), `timestampMs`, `runtimeSlotNumber` | +| `coverage` | Exactly the six current callback-kind keys: `slotRequested`, `slotResponseReceived`, `slotRenderEnded`, `slotOnload`, `impressionViewable`, `slotVisibilityChanged` | +| Each coverage counter | `observed`, `matched`, `unmatched`, `ambiguous` | +| `metadata` | `droppedCallbacks`, `droppedAttributionIssues`, `evictedSlots`, `evictedRequestCycles` | + +Callback `reason` accepts only the current store's six emitted literals: +`invalid_event_order`, `missing_response_before_render`, +`invalid_visibility_percentage`, `evicted_slot`, `no_compatible_request_cycle`, +and `overlapping_request_cycles`. +Although the source interface types it as `string`, the trace validator rejects +all other values rather than copying arbitrary text. + +`requestPath` is the bounded `GptDiagnosticsRequestPath` enum, not a URL or +pathname; include it as browser intent only. `requestIntentId` is a local numeric +sequence, not an external request identifier. Source `version` becomes +`source_schema_version`; no other source properties pass through. Tests must +classify every current source member as copied, transformed, or excluded, and +reject extra keys at every object boundary. + It is deliberately not named or represented as `GptDiagnosticsExportV1`, because redaction, excluded identifiers, and trace-level bounds change the source field semantics. TS Console continues to own the source schema; the trace envelope @@ -812,11 +888,16 @@ telemetry and OpenRTB objects: request ID, or an identifier joinable to user-bearing logs. - Auction tokens are `ts-auc-` followed by a lowercase UUID v4 in the existing producer's 32-hex-digit unhyphenated form (`Uuid::new_v4().simple()`). Slot - tokens are `ts-slot-` followed by a canonical lowercase hyphenated UUID v4, + tokens are new: neither a `ts-slot-` producer nor a TSJS `crypto.randomUUID()` + call exists today. This design introduces tokens that are `ts-slot-` followed by a canonical lowercase hyphenated UUID v4, matching `crypto.randomUUID()`. Both validators enforce UUID version 4 and the RFC variant and reject every other shape. Tokens are compared verbatim, never normalized during validation or correlation; no producer format change - is required for the existing GPT auction opportunity marker. + is required for the existing GPT auction opportunity marker. For the GPT + projection, validation and comparison start from the marker returned by + `normalizedAuctionId` in `gpt_diagnostics/store.ts`, which already trims + whitespace and caps the stored value. Trace validation performs no further + normalization; invalid shapes are not repaired into valid tokens. - `slot_number` is a one-based ordinal over the exact post-conversion `AuctionRequest.slots` sequence observed by orchestration. It is display-only and is never used to map a response back to pre-conversion client input. @@ -838,7 +919,10 @@ telemetry and OpenRTB objects: - `source` is assigned by the server call site: initial document auction is `initial_navigation_ssat`, `/_ts/page-bids` is `spa_page_bids`, and `POST /auction` is `auction_api`. Browser `requestPath` does not determine or - override this value. + override this value. These are intentionally trace-owned public names: + existing `AuctionSource` in `auction/telemetry.rs` uses `initial_navigation`, + `spa_navigation`, and `auction_api`, respectively. Map these explicitly; + do not change telemetry vocabulary or serialize it directly into the report. - `provider_number` is assigned deterministically in provider dispatch order and is stable only within one auction. Provider names, bidder/seat names, and provider metadata are omitted. `returned_bid_count` is a count, not a bid @@ -901,7 +985,10 @@ before an envelope arrives is recorded separately by the browser as `evidence_transport_failed`. Initial-navigation and SPA slot definitions carry their token on the exact slot -object TSJS already consumes: +object TSJS already consumes. This is a two-sided addition: add optional `ext?` +to the TypeScript `AuctionSlot` interface in `core/types.ts` and the matching +nested key to Rust's `build_slot_json` in `publisher.rs`. Neither has this member +today; Rust builds free-form JSON, and TSJS currently retains the slot objects: ```text AuctionSlot.ext.trusted_server.trace_slot_ref: string @@ -989,7 +1076,8 @@ that preserves existing request-path attribution. `ext.trusted_server.trace_slot_ref`. Failed, abandoned, skipped, and zero-bid outcomes still inject their bounded evidence when the publisher document can be delivered. -- **SPA page-bids:** `/_ts/page-bids` adds an optional, namespaced +- **SPA page-bids:** `/_ts/page-bids` and its still-live deprecated alias + `/__ts/page-bids` both add an optional, namespaced `trace_auction` transport envelope beside its existing bid result. TSJS validates and records it and consumes each returned slot's `ext.trusted_server.trace_slot_ref` before triggering ad initialization. Both @@ -1089,7 +1177,7 @@ Runtime limits are part of the v1 contract: | Value | Limit | | ----------------------------------------------- | ------------------------------------------------------------ | -| Container nesting | 8 levels | +| Container nesting | 10 levels | | Server auctions | 16 | | Slot correlations | 128 | | Provider calls | 16 per server auction | @@ -1116,6 +1204,12 @@ Runtime limits are part of the v1 contract: | GPT-reported fill dimension (`size`) | Finite integer from 1 through 100,000 | | Observed CSS box dimension (`observedSlotSize`) | Finite integer from 0 through 100,000 | +The depth cap includes two levels of headroom above the current deepest valid +GPT path: report (1), GPT projection (2), slots array (3), slot (4), requests +array (5), cycle (6), `requestedSlotSizes` array (7), size tuple (8). Headroom +does not permit unknown fields; future schema additions still require explicit +compatibility review. Test the complete current projection and over-depth input. + `observedSlotSize` preserves zero dimensions, including `[0, 0]` for a hidden or collapsed element after a filled GPT render. Zero is a measured box size, not missing data or evidence of an empty GPT response; do not omit it or @@ -1154,7 +1248,7 @@ wrapping or saturating the count. For depth accounting, the `TraceReportV1` object—not its storage wrapper—is level 1; entering either an object or an array increments the level by one; -primitives do not. No accepted report value may enter a ninth container level. +primitives do not. No accepted report value may enter an eleventh container level. The storage wrapper is validated separately as the exact two-field object `{ stored_at_ms, report }`. @@ -1311,7 +1405,10 @@ The HTML shell, enable/end responses, every active diagnostic publisher response, and every dynamic page-bids or `/auction` response carrying trace evidence are terminally `private, no-store`. The fixed versioned JS/CSS assets are the sole exception and may be publicly cached because they contain no -request or report data. HTML and JSON endpoint responses also send: +request or report data. HTML and JSON endpoint responses, including state +results and all local authentication, routing, validation, and disabled-route errors, also send the +following headers. Every such error is `private, no-store`; only successful +fixed-asset responses qualify for the cache exception: - Path-appropriate `Content-Type`: `text/html; charset=utf-8` for the shell and `application/json; charset=utf-8` for enable, end, and state results. @@ -1323,9 +1420,19 @@ usb=()` ```text default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; -object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self' +object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self'; +img-src data: ``` +The shell supplies a fixed data-URL favicon; `img-src data:` allows it without +an automatic publisher `/favicon.ico` fetch. All styles live in the fixed CSS +asset: toggle classes or the `hidden` attribute, with no inline style attributes, +style blocks, or JavaScript style-property writes. JSON download uses a Blob +object URL assigned directly to an `` followed by a click, then +revokes the URL after the download has started. Do not fetch the blob URL or +embed it in a frame. This fixes the download mechanism without widening +`connect-src` or enabling frames; browser tests exercise it under this exact CSP. + The endpoint makes no third-party requests. Its script and stylesheet are fixed same-origin static assets. Setup-request values are server-rendered as escaped text nodes, and the viewer obtains the report only from browser storage. Active @@ -1352,13 +1459,26 @@ extension; active responses add them only in request-scoped injection or the request-scoped body seam. The existing diagnostics private/no-store decision remains a load-bearing gate. Tests must prove that late response-header handlers cannot make traced content -publicly cacheable. +publicly cacheable. Reuse +`apply_response_headers_with_cache_privacy` in `response_privacy.rs`, which +skips operator cache-header overrides on already uncacheable responses. Fastly +also re-runs privacy guards in `apply_terminal_response_effects` after late EC +and filter effects; retain its +`late_filter_effects_cannot_make_an_assembled_response_public` regression. +`apply_finalize_headers` is terminal on Axum, Cloudflare, and Spin, but not on +Fastly; trace handling must preserve the appropriate terminal protection. ### 12.5 Operator authentication policy -Trace routing preserves the existing `auth.rs` namespace contract: every -matching operator Basic Authentication rule is enforced, including `^/_ts` or -`^/`. Classification may run early, but does not authorize a request. Only +Trace routing preserves existing first-match-wins operator authentication: +`Settings::handler_for_path` selects one handler, and trace handling enforces +that handler's Basic Authentication policy. Rules do not compose; a preceding +narrow rule can shadow `^/_ts` or `^/`. With no matching handler, trace remains +public: the fail-closed unmatched-path backstop in `enforce_basic_auth` applies +only to `/_ts/admin`, not trace. Call the synchronous `enforce_basic_auth` with +settings and the request before serving any trace response. Do not copy the +Fastly `/_ts/debug/ja4` early return, which bypasses `AuthMiddleware`. +Classification may run early, but does not authorize a request. Only authentication is factored ahead of trace handling; ordinary event, identity, filter, and auction processing stays outside trace routes. A challenge exposes no setup context, changes no cookie, and is terminally private/no-store. @@ -1431,7 +1551,8 @@ results, never a prerequisite for returning them. ### 14.1 Core unit tests - Configuration defaults off and rejects trace-page enablement without GPT - diagnostics. + diagnostics, with `enabled` both explicitly false and omitted, on both + deploy and runtime validation paths. - With GPT diagnostics enabled but `trace_page_enabled = false`, a valid console cookie still enables the existing console but never mints trace tokens, builds auction evidence, adds response extensions, or emits @@ -1455,8 +1576,9 @@ results, never a prerequisite for returning them. Absent, invalid, duplicate, and uninspectable cookies all report inactive, with no claim that inactive proves the cookie is absent. - Empty-body enforcement rejects positive/invalid lengths, transfer encoding, - the first unexpected body byte, and the two-second deadline without an - unbounded read. + nonempty bodies even with absent/zero lengths, and verifies no mutation on + rejection. Tests must not claim a transport bound or timeout that the pinned + adapters cannot enforce. - Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, configured filters, ordinary event context, and origin fetch. - Cookie-health scanner covers multiple header fields; zero, one, and duplicate @@ -1467,6 +1589,8 @@ results, never a prerequisite for returning them. fields. - Server-auction projection maps initial navigation, SPA page-bids, and auction API call sites to the exact public source enums without using a browser hint. + Canonical and legacy page-bids routes produce equivalent gated evidence and + slot extensions, including the TSJS retry against the legacy alias. - The diagnostic auction token is minted before dispatch and remains identical across completed, zero-bid, skipped, failed, and abandoned evidence and the corresponding SSAT/SPA browser opportunity marker. API tokens remain @@ -1510,12 +1634,16 @@ results, never a prerequisite for returning them. sources. - Axum, Cloudflare, and Spin return the common route/schema with unavailable fields omitted. -- Trace-route failures never fall through to publisher origin. +- Trace-route failures never fall through to publisher origin, including HEAD + on each exact path, malformed reserved paths, and arbitrary unsupported + methods intercepted before router dispatch. Assert path-specific `Allow`, + bodyless HEAD errors, and hardening headers on local errors. - Broad `^/_ts` and `^/` authentication rules challenge every trace path (shell/state/actions/assets, disabled routes, and unsupported methods); valid credentials proceed to trace handling, while `^/_ts/admin` leaves trace routes public. Challenges expose no context or cookie mutation, and - protected assets remain private/no-store. + protected assets remain private/no-store. Also test an earlier narrow handler + shadowing a broad rule to pin first-match-wins behavior. - GET, HEAD, state-changing POST, and unsupported methods obey the same lifecycle contract across adapters. - Every adapter omits JA4/H2 and rejects control characters or overlong platform @@ -1735,7 +1863,9 @@ independently reviewable plans and preferably four PRs: shared 30-minute endpoint/query cookie policy, bounded cookie-health inspection, base request-context schema, projection of already populated `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not - add speculative new platform fields in this change. + add speculative new platform fields in this change. Include the raw-config + validation hook and method-independent dispatch on all four adapters in + this first PR; Fastly-only route registration does not satisfy the contract. 2. **Live server-auction evidence:** introduce the public diagnostic auction and slot tokens, project `TraceAuctionEvidenceV1` at the live observation boundary, transport it through initial navigation, page-bids, and both From a7b19ca2881eef16bcf04a78ce75ec839f89307e Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 21 Sep 2026 09:50:54 -0500 Subject: [PATCH 086/104] Address remaining GPT diagnostics review feedback --- .../integrations/gpt_diagnostics/overlay.ts | 2 + .../src/integrations/gpt_diagnostics/store.ts | 10 +-- .../lib/src/integrations/prebid/index.ts | 3 + .../gpt_diagnostics/overlay.test.ts | 62 +++++++++++++++++++ .../test/integrations/prebid/index.test.ts | 11 ++++ .../gpt-diagnostics-dictionary.md | 32 +++++----- 6 files changed, 99 insertions(+), 21 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index b69b40194..fe064064e 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -761,6 +761,8 @@ export class GptDiagnosticsOverlay { note.setAttribute('role', 'status'); note.textContent = `Ad #${this.selectedRequest.runtimeSlotNumber}, Request #${this.selectedRequest.requestNumber} is no longer retained.`; panel.append(note); + this.selectedRequest = undefined; + this.selectedRequestHasFocus = false; } const content = this.document.createElement('div'); diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts index 344e20221..fd32b9c26 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts @@ -730,11 +730,11 @@ export class GptDiagnosticsStore { const requestPath = this.requestPath(intent); const auctionType = this.auctionType(intent, trustedServerEvidence); const serverAuctionTimingOrigin: GptDiagnosticsServerAuctionTimingOrigin | undefined = - trustedServerEvidence?.serverAuctionTimings === undefined - ? undefined - : trustedServerEvidence.auctionType === 'trusted_server' - ? 'spa_auction' - : 'navigation'; + trustedServerEvidence?.auctionType === 'trusted_server' + ? 'spa_auction' + : trustedServerEvidence?.auctionType === 'ssat' + ? 'navigation' + : undefined; record.requests.push({ requestNumber, requestedAtMs: timestampMs, diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index df589150d..9764dcb97 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -484,6 +484,9 @@ function recordCompletedPrebidAuction( const counts = new Map(); for (const code of adUnitCodes) counts.set(code, (counts.get(code) ?? 0) + 1); const nowMs = performance.now(); + for (const [key, attempt] of prebidDiagnosticAttempts) { + if (nowMs > attempt.expiresAtMs) prebidDiagnosticAttempts.delete(key); + } const generation = window.tsjs?.navGeneration ?? 0; for (let index = 0; index < auctionSlots.length; index += 1) { const slot = auctionSlots[index]; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index a5c4bf812..6f81692bc 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -575,6 +575,68 @@ describe('GptDiagnosticsOverlay', () => { overlay.destroy(); }); + it('labels unavailable SPA auction timing with the SPA request clock', () => { + const frames: Array<() => void> = []; + const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); + const diagnosticSlot = slot('spa-without-timings'); + store.recordTrustedServerOpportunity( + diagnosticSlot, + 'auction-spa-without-timings', + 'renderable_candidate', + undefined, + undefined, + { + auctionType: 'trusted_server', + winner: { bidder: 'example-bidder', priceBucket: '1.20' }, + } + ); + store.recordSlotRequested(diagnosticSlot); + let root: ShadowRoot | undefined; + const overlay = new GptDiagnosticsOverlay(store, new FakeBindings(), { + scheduleFrame: (callback) => frames.push(callback), + onShadowRoot: (createdRoot) => { + root = createdRoot; + }, + }); + runNextFrame(frames); + runNextFrame(frames); + + expect(root?.textContent).toContain('SPA page-bids T0 → auction dispatched Unavailable'); + expect(root?.textContent).not.toContain('Edge request T0 → auction dispatched Unavailable'); + overlay.destroy(); + }); + + it('announces an evicted request selection once and then clears it', () => { + const frames: Array<() => void> = []; + const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); + const diagnosticSlot = slot('evicted-selection'); + store.recordSlotRequested(diagnosticSlot); + let root: ShadowRoot | undefined; + const overlay = new GptDiagnosticsOverlay(store, new FakeBindings(), { + scheduleFrame: (callback) => frames.push(callback), + onShadowRoot: (createdRoot) => { + root = createdRoot; + }, + }); + runNextFrame(frames); + runNextFrame(frames); + + overlay.selectRequest(1, 1); + runNextFrame(frames); + let requestsToRecord = 10; + while (requestsToRecord > 0) { + store.recordSlotRequested(diagnosticSlot); + requestsToRecord -= 1; + } + runNextFrame(frames); + + expect(root?.textContent).toContain('Ad #1, Request #1 is no longer retained.'); + store.recordSlotResponseReceived(diagnosticSlot); + runNextFrame(frames); + expect(root?.textContent).not.toContain('Ad #1, Request #1 is no longer retained.'); + overlay.destroy(); + }); + it('reveals an exact request and locates it without mutating publisher markup', () => { const frames: Array<() => void> = []; const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); diff --git a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts index a5b9a756c..08ff55085 100644 --- a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts @@ -2619,6 +2619,9 @@ describe('prebid publisher snapshots and delivery refreshes', () => { bidder: 'example-client', priceBucket: '2.40', }); + expect(recordPrebidRefresh.mock.invocationCallOrder[0]).toBeLessThan( + recordPrebidAuction.mock.invocationCallOrder[0] + ); expect(getTargeting).not.toHaveBeenCalledWith('hb_cur'); const bidWon = mockOnEvent.mock.calls.find(([event]) => event === 'bidWon')?.[1]; expect(bidWon).toBeTypeOf('function'); @@ -2630,6 +2633,14 @@ describe('prebid publisher snapshots and delivery refreshes', () => { expect(recordPrebidWin).not.toHaveBeenCalled(); bidWon?.({ auctionId: 'example-client-auction', + latestTargetedAuctionId: 'later-client-auction', + adUnitCode: 'example-client-slot', + adserverTargeting: { hb_bidder: 'stale-client', hb_pb: '9.99' }, + }); + expect(recordPrebidWin).not.toHaveBeenCalled(); + bidWon?.({ + auctionId: 'example-client-auction', + latestTargetedAuctionId: 'example-client-auction', adUnitCode: 'example-client-slot', adserverTargeting: { hb_bidder: 'example-client', hb_pb: '2.40' }, }); diff --git a/docs/guide/integrations/gpt-diagnostics-dictionary.md b/docs/guide/integrations/gpt-diagnostics-dictionary.md index 896fc0cce..e0dc433a6 100644 --- a/docs/guide/integrations/gpt-diagnostics-dictionary.md +++ b/docs/guide/integrations/gpt-diagnostics-dictionary.md @@ -86,21 +86,21 @@ Path markers live for five seconds, are consumed once, and are keyed by GPT slot ## Timing -All values are milliseconds. Missing timing that should apply is `Unavailable`, never zero. Server timing is `Not applicable` when no completed server auction was observed. A displayed zero is a valid immediate observation. - -| Label | Origin and boundaries | Raw field | -| ------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------ | -| `Server request start → auction dispatched` | Server request `RequestTimings` T0 to successful `dispatch_auction` outcome | `serverAuctionTimings.auctionDispatchedMs` | -| `Server request start → auction collected` | Same T0 to completion of `collect_dispatched_auction` | `auctionResolvedMs` | -| `Server request start → bids ready` | Same T0 to winning-bid map commit | `auctionCommittedMs` | -| `Auction collection wait` | Actual duration blocked in collect; placement is `pre-header` or `in stream` | `auctionWaitMs`, `auctionWaitPlacement` | -| `Opportunity → request` | Browser `performance.now()`: recorder observation to matched GPT `slotRequested` | `opportunityToRequestMs` | -| `GAM request → response` | Browser `slotRequested` to `slotResponseReceived` | `durations.requestToResponseMs` | -| `GAM response → render` | Browser `slotResponseReceived` to `slotRenderEnded` | `responseToRenderMs` | -| `GAM request → render` | Browser `slotRequested` to `slotRenderEnded` | `requestToRenderMs` | -| `Render → load` | Browser `slotRenderEnded` to `slotOnload` | `renderToLoadMs` | -| `Render → viewable` | Browser `slotRenderEnded` to `impressionViewable` | `renderToViewableMs` | -| `Replaced rendered request` | Earlier browser render callback to later request callback | `previousRenderToRequestMs` | +All values are milliseconds. Missing timing that should apply is `Unavailable`, never zero. Server timing is `Not applicable` when no completed server auction was observed. A displayed zero is a valid immediate observation. The three server rows below are rendered with their T0 anchor substituted: `Edge request T0` for a navigation-origin auction and `SPA page-bids T0` for an SPA-origin one. + +| Label | Origin and boundaries | Raw field | +| ---------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------ | +| ` → auction dispatched` | Server request `RequestTimings` T0 to successful `dispatch_auction` outcome | `serverAuctionTimings.auctionDispatchedMs` | +| ` → auction collected` | Same T0 to completion of `collect_dispatched_auction` | `auctionResolvedMs` | +| ` → bids ready` | Same T0 to winning-bid map commit | `auctionCommittedMs` | +| `Auction collection wait` | Actual duration blocked in collect; placement is `pre-header` or `in stream` | `auctionWaitMs`, `auctionWaitPlacement` | +| `Opportunity → request` | Browser `performance.now()`: recorder observation to matched GPT `slotRequested` | `opportunityToRequestMs` | +| `GAM request → response` | Browser `slotRequested` to `slotResponseReceived` | `durations.requestToResponseMs` | +| `GAM response → render` | Browser `slotResponseReceived` to `slotRenderEnded` | `responseToRenderMs` | +| `GAM request → render` | Browser `slotRequested` to `slotRenderEnded` | `requestToRenderMs` | +| `Render → load` | Browser `slotRenderEnded` to `slotOnload` | `renderToLoadMs` | +| `Render → viewable` | Browser `slotRenderEnded` to `impressionViewable` | `renderToViewableMs` | +| `Replaced rendered request` | Earlier browser render callback to later request callback | `previousRenderToRequestMs` | Server offsets are not browser timestamps. `auctionResolvedMs` means collection completed (including timeout handling), not that a network byte arrived at that exact instant. @@ -109,7 +109,7 @@ Server offsets are not browser timestamps. `auctionResolvedMs` means collection | Label | Raw field / source | Meaning and limits | | ------------------------------------------------------------------------------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `Requested sizes` | `requestedSlotSizes` | Configured sizes supplied to GPT; ordinary sizes such as 300×250 remain visible. Missing data is shown as `Requested sizes: Not observed`. | -| `GPT-reported size` | `size` | Exact `slotRenderEnded.size`. A 1×1 placeholder is shown as `GPT-reported size: 1×1 placeholder hidden` and retained unchanged in V1 JSON. | +| `GPT-reported size` | `size` | Exact `slotRenderEnded.size`. A 1×1 placeholder is shown as `GPT-reported size: placeholder hidden` and retained unchanged in V1 JSON. | | `Size filled` / `Measured outer slot size` | `observedSlotSize` | CSS outer box of the exact uniquely bound slot after fill. Missing data is shown as `Size filled: Not observed · Measured outer slot size`. | | `GPT visibility` | current/maximum visibility percentage | Values from GPT visibility callbacks; absence is `GPT visibility: Not observed`. | | `Binding: Bound` / `Bound` | `binding.status=bound` | Exactly one connected publisher element matched. | From f951955f537b89392dcb863fca7a01a5f6fa4846 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 21 Sep 2026 10:21:17 -0500 Subject: [PATCH 087/104] Fix cross-adapter GPT diagnostic timing origins --- .../src/app.rs | 5 +- .../src/middleware.rs | 77 +++++++++++++++++-- crates/trusted-server-adapter-spin/src/app.rs | 4 +- .../src/middleware.rs | 77 +++++++++++++++++-- crates/trusted-server-core/src/publisher.rs | 76 +++++++++++++++--- .../integrations/gpt_diagnostics/overlay.ts | 3 +- .../gpt_diagnostics/overlay.test.ts | 10 +-- docs/guide/integrations/gpt-diagnostics.md | 11 +-- .../2026-08-24-request-phase-timing-design.md | 13 ++-- 9 files changed, 238 insertions(+), 38 deletions(-) diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index a3457ef7c..9aba604b0 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -36,7 +36,9 @@ use trusted_server_core::request_signing::{ }; use trusted_server_core::settings::Settings; -use crate::middleware::{AuthMiddleware, FinalizeResponseMiddleware, SanitizeRequestMiddleware}; +use crate::middleware::{ + AuthMiddleware, FinalizeResponseMiddleware, RequestTimingMiddleware, SanitizeRequestMiddleware, +}; use crate::platform::build_runtime_services; // --------------------------------------------------------------------------- @@ -467,6 +469,7 @@ fn build_router(state: &Arc) -> RouterService { // any middleware registered ahead of it would observe the // shared-secret authentication header. .middleware(SanitizeRequestMiddleware::new(Arc::clone(&state.settings))) + .middleware(RequestTimingMiddleware::new()) .middleware(FinalizeResponseMiddleware::new(Arc::clone(&state.settings))) .middleware(AuthMiddleware::new(Arc::clone(&state.settings))) .get( diff --git a/crates/trusted-server-adapter-cloudflare/src/middleware.rs b/crates/trusted-server-adapter-cloudflare/src/middleware.rs index 14efed56a..6b418688b 100644 --- a/crates/trusted-server-adapter-cloudflare/src/middleware.rs +++ b/crates/trusted-server-adapter-cloudflare/src/middleware.rs @@ -8,6 +8,7 @@ use edgezero_core::middleware::{Middleware, Next}; use trusted_server_core::auth::enforce_basic_auth; use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; use trusted_server_core::http_util::sanitize_trusted_client_ip_headers; +use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; // --------------------------------------------------------------------------- @@ -46,6 +47,40 @@ impl Middleware for SanitizeRequestMiddleware { } } +// --------------------------------------------------------------------------- +// RequestTimingMiddleware +// --------------------------------------------------------------------------- + +/// Attaches the server request clock consumed by core timing instrumentation. +/// +/// This adapter does not emit `Server-Timing`; the collector keeps timing +/// origins consistent for request-scoped consumers such as GPT diagnostics. +/// Health checks remain outside timing collection on every adapter. +#[derive(Default)] +pub struct RequestTimingMiddleware; + +impl RequestTimingMiddleware { + /// Creates a new [`RequestTimingMiddleware`]. + #[must_use] + pub fn new() -> Self { + Self + } +} + +#[async_trait(?Send)] +impl Middleware for RequestTimingMiddleware { + async fn handle(&self, mut ctx: RequestContext, next: Next<'_>) -> Result { + if ctx.request().uri().path() != "/health" + && ctx.request().extensions().get::().is_none() + { + ctx.request_mut() + .extensions_mut() + .insert(RequestTimings::new()); + } + next.run(ctx).await + } +} + // --------------------------------------------------------------------------- // FinalizeResponseMiddleware // --------------------------------------------------------------------------- @@ -56,9 +91,9 @@ impl Middleware for SanitizeRequestMiddleware { /// (injected by the Cloudflare Workers runtime). On the native host target the /// header is absent, so `X-Geo-Info-Available: false` is emitted. /// -/// Registered directly inside [`SanitizeRequestMiddleware`] and ahead of -/// [`AuthMiddleware`] so that every outgoing response — including auth-rejected -/// ones — carries a consistent set of headers. +/// Registered inside [`RequestTimingMiddleware`] and ahead of [`AuthMiddleware`] +/// so that every outgoing response — including auth-rejected ones — carries a +/// consistent set of headers. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -180,10 +215,10 @@ mod tests { .expect("should build empty test response") } - fn empty_ctx() -> RequestContext { + fn ctx_for_path(path: &str) -> RequestContext { let req = request_builder() .method(Method::GET) - .uri("/test") + .uri(path) .header("x-reader-ip", "198.51.100.7") .header("x-reader-ip-auth", "fictional-shared-secret-0123456789") .body(Body::empty()) @@ -191,6 +226,10 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn empty_ctx() -> RequestContext { + ctx_for_path("/test") + } + fn settings_with_response_headers(headers: Vec<(&str, &str)>) -> Settings { // Build from explicit test settings: the settings baked into the // binary contain placeholder secrets that `get_settings()` rejects @@ -289,6 +328,34 @@ mod tests { ); } + #[test] + fn request_timing_middleware_attaches_a_collector_except_for_health() { + for (path, expected) in [("/test", true), ("/health", false)] { + let observed = Arc::new(Mutex::new(None)); + let handler_observed = Arc::clone(&observed); + let handler = Arc::new(move |ctx: RequestContext| { + let handler_observed = Arc::clone(&handler_observed); + async move { + *handler_observed.lock().expect("should lock observation") = + Some(ctx.request().extensions().get::().is_some()); + Ok::(empty_response()) + } + }); + + block_on( + RequestTimingMiddleware::new() + .handle(ctx_for_path(path), Next::new(&[], &*handler)), + ) + .expect("should run timing middleware"); + + assert_eq!( + *observed.lock().expect("should lock observation"), + Some(expected), + "collector presence should match timing policy for {path}" + ); + } + } + #[test] fn sanitize_middleware_strips_configured_trust_headers_before_routing() { let mut settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 8917f2a65..59e7734f2 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -36,7 +36,8 @@ use trusted_server_core::request_signing::{ use trusted_server_core::settings::Settings; use crate::middleware::{ - AuthMiddleware, FinalizeResponseMiddleware, NormalizeMiddleware, SanitizeRequestMiddleware, + AuthMiddleware, FinalizeResponseMiddleware, NormalizeMiddleware, RequestTimingMiddleware, + SanitizeRequestMiddleware, }; use crate::platform::build_runtime_services; @@ -773,6 +774,7 @@ fn build_router(state: &Arc) -> RouterService { // any middleware registered ahead of it would observe the // shared-secret authentication header. .middleware(SanitizeRequestMiddleware::new(Arc::clone(&state.settings))) + .middleware(RequestTimingMiddleware::new()) .middleware(FinalizeResponseMiddleware::new(Arc::clone(&state.settings))) .middleware(AuthMiddleware::new(Arc::clone(&state.settings))) // Innermost middleware: normalize every routed request (strip diff --git a/crates/trusted-server-adapter-spin/src/middleware.rs b/crates/trusted-server-adapter-spin/src/middleware.rs index d7a09987a..d31c6ad7e 100644 --- a/crates/trusted-server-adapter-spin/src/middleware.rs +++ b/crates/trusted-server-adapter-spin/src/middleware.rs @@ -8,6 +8,7 @@ use edgezero_core::middleware::{Middleware, Next}; use trusted_server_core::auth::enforce_basic_auth; use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; use trusted_server_core::http_util::sanitize_trusted_client_ip_headers; +use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; // --------------------------------------------------------------------------- @@ -46,6 +47,40 @@ impl Middleware for SanitizeRequestMiddleware { } } +// --------------------------------------------------------------------------- +// RequestTimingMiddleware +// --------------------------------------------------------------------------- + +/// Attaches the server request clock consumed by core timing instrumentation. +/// +/// This adapter does not emit `Server-Timing`; the collector keeps timing +/// origins consistent for request-scoped consumers such as GPT diagnostics. +/// Health checks remain outside timing collection on every adapter. +#[derive(Default)] +pub struct RequestTimingMiddleware; + +impl RequestTimingMiddleware { + /// Creates a new [`RequestTimingMiddleware`]. + #[must_use] + pub fn new() -> Self { + Self + } +} + +#[async_trait(?Send)] +impl Middleware for RequestTimingMiddleware { + async fn handle(&self, mut ctx: RequestContext, next: Next<'_>) -> Result { + if ctx.request().uri().path() != "/health" + && ctx.request().extensions().get::().is_none() + { + ctx.request_mut() + .extensions_mut() + .insert(RequestTimings::new()); + } + next.run(ctx).await + } +} + // --------------------------------------------------------------------------- // FinalizeResponseMiddleware // --------------------------------------------------------------------------- @@ -55,9 +90,9 @@ impl Middleware for SanitizeRequestMiddleware { /// Spin does not expose geo headers to the application, so /// `X-Geo-Info-Available: false` is emitted for every response. /// -/// Registered directly inside [`SanitizeRequestMiddleware`] and ahead of -/// [`AuthMiddleware`] so that every outgoing response — including auth-rejected -/// ones — carries a consistent set of headers. +/// Registered inside [`RequestTimingMiddleware`] and ahead of [`AuthMiddleware`] +/// so that every outgoing response — including auth-rejected ones — carries a +/// consistent set of headers. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -207,10 +242,10 @@ mod tests { .expect("should build empty test response") } - fn empty_ctx() -> RequestContext { + fn ctx_for_path(path: &str) -> RequestContext { let req = request_builder() .method(Method::GET) - .uri("/test") + .uri(path) .header("x-reader-ip", "198.51.100.7") .header("x-reader-ip-auth", "fictional-shared-secret-0123456789") .body(Body::empty()) @@ -218,6 +253,10 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn empty_ctx() -> RequestContext { + ctx_for_path("/test") + } + fn settings_with_response_headers(headers: Vec<(&str, &str)>) -> Settings { // Build from explicit test settings: the settings baked into the // binary contain placeholder secrets that `get_settings()` rejects @@ -316,6 +355,34 @@ mod tests { ); } + #[test] + fn request_timing_middleware_attaches_a_collector_except_for_health() { + for (path, expected) in [("/test", true), ("/health", false)] { + let observed = Arc::new(Mutex::new(None)); + let handler_observed = Arc::clone(&observed); + let handler = Arc::new(move |ctx: RequestContext| { + let handler_observed = Arc::clone(&handler_observed); + async move { + *handler_observed.lock().expect("should lock observation") = + Some(ctx.request().extensions().get::().is_some()); + Ok::(empty_response()) + } + }); + + block_on( + RequestTimingMiddleware::new() + .handle(ctx_for_path(path), Next::new(&[], &*handler)), + ) + .expect("should run timing middleware"); + + assert_eq!( + *observed.lock().expect("should lock observation"), + Some(expected), + "collector presence should match timing policy for {path}" + ); + } + } + #[test] fn sanitize_middleware_strips_configured_trust_headers_before_routing() { let mut settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index c2b6bc6be..52cbc39a8 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -6681,13 +6681,13 @@ pub async fn handle_page_bids( ec_context: &mut EcContext, mut req: Request, ) -> Result, Report> { - // Same defaulted-handle rule as `handle_publisher_request`: a request - // without the extension records into a collector nothing reads. - let timings = req - .extensions() - .get::() - .cloned() - .unwrap_or_default(); + // Keep a local collector for direct-handler callers, but only expose + // browser timings from an adapter-attached request clock. Otherwise an + // adapter that forgets the extension would silently report a different + // handler-entry timing origin. + let request_timings = req.extensions().get::().cloned(); + let has_request_timings = request_timings.is_some(); + let timings = request_timings.unwrap_or_default(); // Adapter fallbacks prepare this before routing. Keep this idempotent call as // a direct-handler safety net and retain the session decision after the @@ -6925,7 +6925,7 @@ pub async fn handle_page_bids( // A successful result proves at least one pending or immediate // provider outcome. Failures can occur before any request leaves // the edge, so they must not fabricate dispatch timing evidence. - if gpt_diagnostics.browser_session_active() { + if gpt_diagnostics.browser_session_active() && has_request_timings { let timing_snapshot = timings.snapshot(); auction_diagnostics = Some(BrowserAuctionDiagnostics { auction_dispatched_ms: timing_snapshot.auction_dispatched_ms, @@ -21087,6 +21087,52 @@ mod tests { ); } + #[tokio::test] + async fn active_page_bids_omits_timings_without_adapter_collector() { + let mut settings = settings_with_co(); + settings.auction.providers = vec![AUCTION_ID_TEST_PROVIDER.to_string()]; + settings + .integrations + .insert_config("gpt_diagnostics", &serde_json::json!({ "enabled": true })) + .expect("should enable diagnostics"); + let slots = article_slot(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"winner".to_vec()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let orchestrator = + auction_id_test_orchestrator(&settings, Arc::new(Mutex::new(None)), true); + let mut ec_context = consent_allowing_ec_context(); + + let response = handle_page_bids( + &settings, + &services, + None, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &slots, + registry: None, + }, + &mut ec_context, + make_active_page_bids_request("/2024/01/my-article/"), + ) + .await + .expect("should return page-bids response"); + let body: serde_json::Value = serde_json::from_slice( + &response + .into_body() + .into_bytes() + .expect("should read page-bids response body"), + ) + .expect("should serialize page-bids response as JSON"); + + assert!( + body.get("auctionDiagnostics").is_none(), + "missing adapter timing context must not emit handler-local timing facts" + ); + } + #[tokio::test] async fn page_bids_response_includes_auction_id_only_for_winning_bids() { let mut settings = settings_with_co(); @@ -21113,6 +21159,14 @@ mod tests { }, ); + let mut winning_page_bids_request = + make_active_page_bids_request("/2024/01/my-article/"); + let request_timings = RequestTimings::new(); + winning_page_bids_request + .extensions_mut() + .insert(request_timings); + std::thread::sleep(std::time::Duration::from_millis(2)); + let winning_response = handle_page_bids( &settings, &winning_services, @@ -21123,7 +21177,7 @@ mod tests { registry: None, }, &mut ec_context, - make_active_page_bids_request("/2024/01/my-article/"), + winning_page_bids_request, ) .await .expect("should return winning page-bids response"); @@ -21161,6 +21215,10 @@ mod tests { let resolved_ms = timing("auctionResolvedMs"); let committed_ms = timing("auctionCommittedMs"); let wait_ms = timing("auctionWaitMs"); + assert!( + dispatched_ms > 0, + "page-bids dispatch should include time since adapter request entry" + ); assert!( dispatched_ms <= resolved_ms && resolved_ms <= committed_ms, "page-bids auction milestones should be monotonic" diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index 127317035..35aed65a7 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -243,7 +243,8 @@ function cycleFacts(cycle: GptDiagnosticsRequestCycle): string[] { const timingOrigin = cycle.serverAuctionTimingOrigin ?? (cycle.auctionType === 'trusted_server' ? 'spa_auction' : 'navigation'); - const timingAnchor = timingOrigin === 'spa_auction' ? 'SPA page-bids T0' : 'Edge request T0'; + const timingAnchor = + timingOrigin === 'spa_auction' ? 'SPA page-bids request T0' : 'Initial document request T0'; const serverTimings = [ [`${timingAnchor} → auction dispatched`, cycle.serverAuctionTimings.auctionDispatchedMs], [`${timingAnchor} → auction resolved`, cycle.serverAuctionTimings.auctionResolvedMs], diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index cdc4e4aef..c20d6d7f5 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -275,9 +275,9 @@ describe('GptDiagnosticsOverlay', () => { expect(responseSentArticle).toContain('Auction type: SSAT'); expect(responseSentArticle).toContain('Winning bidder: example-bidder'); expect(responseSentArticle).toContain('Winning bid price bucket: 1.20'); - expect(responseSentArticle).toContain('Edge request T0 → auction dispatched 4 ms'); - expect(responseSentArticle).toContain('Edge request T0 → auction resolved 84 ms'); - expect(responseSentArticle).toContain('Edge request T0 → bids committed 85 ms'); + expect(responseSentArticle).toContain('Initial document request T0 → auction dispatched 4 ms'); + expect(responseSentArticle).toContain('Initial document request T0 → auction resolved 84 ms'); + expect(responseSentArticle).toContain('Initial document request T0 → bids committed 85 ms'); expect(responseSentArticle).toContain('Auction wait (in stream) 80 ms'); expect(responseSentArticle).toContain('Opportunity → request 0 ms'); expect(responseSentArticle).toContain('Direct opportunity: Renderable candidate'); @@ -303,8 +303,8 @@ describe('GptDiagnosticsOverlay', () => { const selectedArticle = slotArticle(root!, 'selected-slot').textContent ?? ''; expect(selectedArticle).toContain('Request path: Competing paths'); expect(selectedArticle).toContain('Auction type: Competing auctions'); - expect(selectedArticle).toContain('SPA page-bids T0 → auction dispatched 0 ms'); - expect(selectedArticle).toContain('SPA page-bids T0 → auction resolved 40 ms'); + expect(selectedArticle).toContain('SPA page-bids request T0 → auction dispatched 0 ms'); + expect(selectedArticle).toContain('SPA page-bids request T0 → auction resolved 40 ms'); expect(selectedArticle).toContain('Direct opportunity: Unrenderable candidate'); expect(selectedArticle).toContain('Trusted Server creative request observed at 23 ms'); expect(selectedArticle).not.toContain('Trusted Server markup response sent'); diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md index 0f6c81ce8..d8273174e 100644 --- a/docs/guide/integrations/gpt-diagnostics.md +++ b/docs/guide/integrations/gpt-diagnostics.md @@ -153,11 +153,12 @@ unattributed requests have no auction label because the available evidence does establish an auction implementation. Server auction timing and browser GPT timing use separate clocks and are never -subtracted from each other. Initial SSAT offsets use edge-request T0. SPA TS auction -offsets use a local server clock started when the page-bids handler begins, not the -browser's navigation clock or the edge's request-receipt time. Diagnostics retain that -timing origin separately from the aggregate auction classification, so a request marked -`competing` still labels SPA offsets from SPA page-bids T0. The server facts are: +subtracted from each other. Both initial SSAT and SPA TS auction offsets use the +adapter's server-request clock. Initial offsets belong to the document request, while +SPA offsets belong to the later `/_ts/page-bids` request. Diagnostics retain that +request origin separately from the aggregate auction classification, so a request marked +`competing` still labels SPA offsets from the SPA page-bids request T0. Neither server +clock is the browser's navigation clock. The server facts are: - `auctionDispatchedMs`: bid dispatch offset from that timing origin. - `auctionResolvedMs`: final bid or timeout offset from the same timing origin. diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 61d43e273..0d510971c 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -271,7 +271,8 @@ structurally and its emissions are defined accordingly rather than pretending pa by path match inside the wrapper. - Axum emits the header only; no Tinybird rows in v1 (unchanged). -Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). +Cloudflare and Spin attach the per-request collector so core request-scoped consumers +share the adapter clock, but have no header or telemetry emission wiring in v1. ## 9. Access telemetry row @@ -750,11 +751,11 @@ the auction dataset as the authority on per-bidder duration. ### Scope -- Fastly emits. Axum attaches a collector and records the marks but does not - emit them. Cloudflare and Spin attach no collector at all: `handle_publisher_request` - falls back to `RequestTimings::default()`, so the marks land in a throwaway - handle and are dropped with it. That is pre-existing for every phase, not new - to these marks, and it matches section 8a adapter semantics. +- Fastly emits. Axum, Cloudflare, and Spin attach a collector and record the marks but + do not emit them. Cloudflare and Spin added collection when GPT diagnostics began + exposing request-scoped auction offsets; without an adapter collector those offsets + silently used a different handler-entry clock. This matches section 8a adapter + semantics: collection is portable, while emission remains adapter-specific. - No header emission for any of these values: they are post-hoc analysis fields, and two of the three are typically unknown at the header freeze point in streaming mode. From 58963a80e3e5cad029294c52409e9e4e4b4fbfd1 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 21 Sep 2026 21:26:02 +0530 Subject: [PATCH 088/104] Strengthen the oversized-dimension regression test The fixture used u32::MAX + 1, which truncates to zero under the original as u32 cast and is already caught by the separate zero-dimension check, so the test passed even without the fix. Use u32::MAX + 101 so the test actually fails against the old truncating cast, and log the offending raw value when a width or height is rejected. Signed-off-by: dhruv8sh --- .../src/integrations/adserver_mock.rs | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 96a2654d2..a2a47c9b1 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -294,13 +294,15 @@ impl AdServerMockProvider { let Some(width) = bid["w"].as_u64().and_then(|v| u32::try_from(v).ok()) else { log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid width, skipping" + "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", + bid["w"] ); continue; }; let Some(height) = bid["h"].as_u64().and_then(|v| u32::try_from(v).ok()) else { log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid height, skipping" + "adserver_mock: bid for slot '{slot_id}' has invalid height {:?}, skipping", + bid["h"] ); continue; }; @@ -1336,11 +1338,13 @@ mod tests { #[test] fn test_parse_mediation_response_skips_oversized_dimensions() { // A dimension above u32::MAX must be rejected rather than silently - // wrapped into a small, plausible-looking value. + // wrapped into a small, plausible-looking value. The offset of 101 is + // load-bearing: u32::MAX + 1 truncates to 0, which the zero-check + // below would already have caught, so it would not pin this fix. let config = AdServerMockConfig::default(); let provider = AdServerMockProvider::new(config); - let oversized_width = u64::from(u32::MAX) + 1; + let oversized_width = u64::from(u32::MAX) + 101; let mediation_response = json!({ "id": "test-auction-123", "seatbid": [ From b70cb57a0af89cd9e6f8400a19a17e965bb3825c Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 22 Sep 2026 00:42:52 +0530 Subject: [PATCH 089/104] Guard the IntegrationSettings debug boundary and correct the test comment claims Adds a canary that pins IntegrationSettings hand-written Debug impl, which is the only thing keeping resolved DataDome credentials out of Settings debug output. Rewrites the leading comment to state what the canary list actually guarantees instead of an enforcement it cannot provide, and adds a positive assertion that a non-secret field stays visible so a blanket-redacting Debug impl would not pass unnoticed. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/settings.rs | 46 +++++++++++++++++----- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index d6581f466..dc0504398 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3807,16 +3807,20 @@ mod tests { } // One distinctive canary per `Redacted` field reachable from - // `Settings`'s derived `Debug` impl. A new secret field added without the - // `Redacted` wrapper should be caught here by adding its own canary; - // re-run `rg 'Redacted<' crates/trusted-server-core/src` when touching - // this test to check the field list is still complete. + // `Settings`'s derived `Debug` impl. This is a regression guard over the + // field list below, not a completeness guarantee: a new secret field + // added without the `Redacted` wrapper has no canary here and will pass + // this test while leaking. Adding the canary is a manual step. // - // Do not use `..Struct::default()` anywhere in this function. This test's - // entire purpose is exhaustive field coverage, and a default spread would - // silently swallow any field added to `Handler`, `TinybirdSettings`, or - // any other struct built here, defeating that coverage. List every field - // explicitly instead. + // Integration configs are deliberately out of scope. They reach + // `Settings` as opaque JSON under `IntegrationSettings`, whose + // hand-written `Debug` impl prints only integration IDs, never values. + // + // Do not use `..Struct::default()` anywhere in this function. A default + // spread would let a new secret field be added to `Handler`, + // `TinybirdSettings`, or any other struct built here without forcing + // anyone to consider it. The compile break is the prompt; the canary + // list below is still maintained by hand. List every field explicitly. #[test] fn settings_debug_output_redacts_every_secret_field() { const CANARY_PROXY_SECRET: &str = "CANARY-PROXY-SECRET-0123456789"; @@ -3832,6 +3836,7 @@ mod tests { const CANARY_S3_SESSION_TOKEN: &str = "CANARY-S3-SESSION-TOKEN-0123456789"; const CANARY_TINYBIRD_AUCTION_TOKEN: &str = "CANARY-TINYBIRD-AUCTION-TOKEN-0123456789"; const CANARY_TINYBIRD_ACCESS_TOKEN: &str = "CANARY-TINYBIRD-ACCESS-TOKEN-0123456789"; + const CANARY_DATADOME_SERVER_SIDE_KEY: &str = "CANARY-DATADOME-SERVER-SIDE-KEY-0123456789"; let mut settings = create_test_settings(); @@ -3890,12 +3895,31 @@ mod tests { max_body_bytes: 0, }; + // `IntegrationSettings` stores integration configs as opaque JSON and + // relies on a hand-written `Debug` impl to suppress their values. That + // impl is the only thing keeping resolved DataDome credentials out of + // this output, so pin it here. + settings + .integrations + .insert_config( + "datadome", + &json!({ + "enabled": true, + "server_side_key_secret_name": CANARY_DATADOME_SERVER_SIDE_KEY, + }), + ) + .expect("should insert datadome integration config"); + let debug = format!("{settings:?}"); assert!( debug.contains("[REDACTED]"), "should redact secret fields in Settings debug output" ); + assert!( + debug.contains("^/secure"), + "should leave non-secret handler path visible in debug output" + ); let canaries = [ ("publisher.proxy_secret", CANARY_PROXY_SECRET), @@ -3928,6 +3952,10 @@ mod tests { CANARY_TINYBIRD_AUCTION_TOKEN, ), ("tinybird.access_token_secret", CANARY_TINYBIRD_ACCESS_TOKEN), + ( + "integrations.datadome.server_side_key_secret_name", + CANARY_DATADOME_SERVER_SIDE_KEY, + ), ]; for (field, canary) in canaries { From 57eab4aa28d886736714975f2899952e0b9d04f3 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 22 Sep 2026 01:22:04 +0530 Subject: [PATCH 090/104] Fix placeholder partner secret check to compare resolved values, not key names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit partner_api_token and partner_ts_pull_token are secret-store key names shipped as examples in trusted-server.example.toml, not resolved secret values — reject_placeholder_secrets runs after secret resolution and only ever sees resolved values, matching every other secret field in the template. Removes the two key-name entries from EcPartner::API_TOKEN_PLACEHOLDERS, retargets the affected test at an existing value placeholder, and drops the drift-guard test that asserted the incorrect invariant. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/settings.rs | 39 +++------------------- 1 file changed, 5 insertions(+), 34 deletions(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 48560bc57..47f614387 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -443,15 +443,14 @@ pub struct EcPartner { } impl EcPartner { - /// Known partner API token placeholders that must not be used in deployments. + /// Known partner secret placeholders (`api_token` and `ts_pull_token`) that + /// must not be used in deployments. pub const API_TOKEN_PLACEHOLDERS: &[&str] = &[ "partner-api-token-32-bytes-minimum", "replace-with-partner-api-token-32-bytes-minimum", "sharedid-internal-token-32-bytes", "inttest-api-key-1-32-bytes-minimum", "inttest2-api-key-2-32-bytes-minimum", - "partner_api_token", - "partner_ts_pull_token", ]; /// Returns `true` if `api_token` matches a known placeholder value @@ -5441,7 +5440,9 @@ source_domain = "partner.example.com" Settings::from_toml(&crate_test_settings_str()).expect("should parse test settings"); settings.publisher.proxy_secret = Redacted::new("unit-test-proxy-secret".to_owned()); settings.ec.passphrase = Redacted::new("test-secret-key-32-bytes-minimum".to_owned()); - settings.ec.partners = vec![test_partner_with_pull_token("partner_ts_pull_token")]; + settings.ec.partners = vec![test_partner_with_pull_token( + "partner-api-token-32-bytes-minimum", + )]; let err = settings .reject_placeholder_secrets() @@ -5467,36 +5468,6 @@ source_domain = "partner.example.com" .expect("should accept a realistic partner pull token"); } - /// Guards against the placeholder lists drifting away from the example config - /// Example value shipped in `trusted-server.example.toml` must be recognized - /// as a placeholder - #[test] - fn example_toml_partner_secret_examples_are_recognized_placeholders() { - const EXAMPLE_TOML: &str = include_str!("../../../trusted-server.example.toml"); - const EXAMPLE_PARTNER_KEYS: &[&str] = &["api_token", "ts_pull_token"]; - let pattern = format!( - r#"(?m)^\s*#?\s*(?:{})\s*=\s*"([^"]+)""#, - EXAMPLE_PARTNER_KEYS.join("|") - ); - let example_value = - Regex::new(&pattern).expect("should compile example partner secret regex"); - - let mut checked_any = false; - for capture in example_value.captures_iter(EXAMPLE_TOML) { - checked_any = true; - let value = &capture[1]; - assert!( - EcPartner::is_placeholder_api_token(value), - "example config partner secret '{value}' should be a recognized placeholder \ - — add it to EcPartner::API_TOKEN_PLACEHOLDERS" - ); - } - assert!( - checked_any, - "should have found at least one partner secret example value in the template" - ); - } - #[test] fn is_unusable_store_id_rejects_placeholders_empty_and_padded_values() { for placeholder in RequestSigning::STORE_ID_PLACEHOLDERS { From ffd5bf8ffa8d9e1170362f646f7f94637b368139 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 22 Sep 2026 09:46:25 +0530 Subject: [PATCH 091/104] Rework proxy CLI arg-parsing tests to cover the real clap defaults Removes base_args, the shared test helper that let every test skip the real --listen default; each test now passes its args explicitly through parse_args, so nothing can mask default_value drifting from DEFAULT_LISTEN. Splits the former no_rule_passed_is_a_no_rule_error into bare_invocation_is_rejected_at_parse_time (proves arg_required_else_help rejects a fully-bare ts before resolve runs) and a fixed no_rule_passed_is_a_no_rule_error that uses --insecure so it still exercises the NoRule path, restoring coverage the rewrite had silently dropped. Also fixes two invalid --rewrite-host true/false assertions (that flag takes no value) and corrects the dev-proxy guide, which still described the pre-fix bare-invocation error. Signed-off-by: dhruv8sh --- .../src/commands/dev/proxy/config.rs | 178 ++++++++++++------ docs/guide/ts-dev-proxy.md | 3 +- 2 files changed, 122 insertions(+), 59 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs index 6f4ada02d..f097c8b71 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs @@ -359,14 +359,6 @@ mod tests { AddressPolicy, OriginKey, ReferenceIdentity, Transport, VerifyMode, }; - fn base_args() -> crate::commands::dev::proxy::ProxyArgs { - parse_args(&[ - "ts", - "--listen", - crate::commands::dev::proxy::DEFAULT_LISTEN, - ]) - } - fn parse_args(argv: &[&str]) -> crate::commands::dev::proxy::ProxyArgs { #[derive(clap::Parser)] struct W { @@ -378,18 +370,35 @@ mod tests { #[test] fn clap_parses_rewrite_host_as_a_bool() { - assert!(!base_args().rewrite_host, "absent --rewrite-host is false"); + assert!( + !parse_args(&["ts", "--from", "a.example.com", "--to", "b.example.com"]).rewrite_host, + "absent --rewrite-host is false" + ); assert!( parse_args(&["ts", "--rewrite-host"]).rewrite_host, "present --rewrite-host is true" ); } + #[test] + fn clap_applies_the_real_listen_default() { + let args = parse_args(&["ts", "--rewrite-host"]); + assert_eq!( + args.listen, + crate::commands::dev::proxy::DEFAULT_LISTEN, + "should apply the real clap --listen default" + ); + } + #[test] fn single_rule_from_to_keeps_from_host_by_default() { - let mut args = base_args(); - args.from = Some("www.example-publisher.com".into()); - args.to = Some("to.edgecompute.app".into()); + let args = parse_args(&[ + "ts", + "--from", + "www.example-publisher.com", + "--to", + "to.edgecompute.app", + ]); let cfg = resolve(&args).expect("should resolve"); let rule = cfg .rules @@ -405,9 +414,12 @@ mod tests { #[test] fn rewrite_host_uses_to() { - let mut args = base_args(); - args.map = vec!["www.example-publisher.com=to.edgecompute.app".into()]; - args.rewrite_host = true; + let args = parse_args(&[ + "ts", + "--map", + "www.example-publisher.com=to.edgecompute.app", + "--rewrite-host", + ]); let cfg = resolve(&args).expect("should resolve"); assert_eq!( rewrite_for( @@ -423,10 +435,13 @@ mod tests { #[test] fn resolve_pins_host_to_ip() { - let mut args = base_args(); - args.map = vec!["www.example-publisher.com=ts.edgecompute.app".into()]; - // Mixed case to confirm the host key is lowercased. - args.resolve = vec!["TS.EdgeCompute.app:192.0.2.10".into()]; + let args = parse_args(&[ + "ts", + "--map", + "www.example-publisher.com=ts.edgecompute.app", + "--resolve", + "TS.EdgeCompute.app:192.0.2.10", // Mixed case to confirm the host key is lowercased. + ]); let cfg = resolve(&args).expect("should resolve"); assert_eq!( cfg.resolve.get("ts.edgecompute.app"), @@ -437,10 +452,13 @@ mod tests { #[test] fn resolve_accepts_ipv6_target() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - // Split-on-first-colon must keep the colon-bearing IPv6 address intact. - args.resolve = vec!["b.edgecompute.app:::1".into()]; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--resolve", + "b.edgecompute.app:::1", // Split-on-first-colon must keep the colon-bearing IPv6 address intact. + ]); let cfg = resolve(&args).expect("should resolve"); assert_eq!( cfg.resolve.get("b.edgecompute.app"), @@ -451,11 +469,15 @@ mod tests { #[test] fn resolve_host_not_matching_any_rule_warns_but_succeeds() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - // A pin for a host that is no rule's TO is a likely typo: it should warn - // (not error) and still be recorded. - args.resolve = vec!["typo.edgecompute.app:192.0.2.10".into()]; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + // A pin for a host that is no rule's TO is a likely typo: it should warn + // (not error) and still be recorded. + "--resolve", + "typo.edgecompute.app:192.0.2.10", + ]); let cfg = resolve(&args).expect("an unmatched --resolve host should warn, not error"); assert!( cfg.resolve.contains_key("typo.edgecompute.app"), @@ -465,9 +487,13 @@ mod tests { #[test] fn resolve_rejects_malformed_value() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.resolve = vec!["b.edgecompute.app:not-an-ip".into()]; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--resolve", + "b.edgecompute.app:not-an-ip", + ]); let err = resolve(&args).expect_err("a non-IP --resolve target should error"); assert!( matches!(err.current_context(), ConfigError::Resolve { .. }), @@ -477,8 +503,7 @@ mod tests { #[test] fn map_value_must_be_from_equals_to() { - let mut args = base_args(); - args.map = vec!["not-a-map".into()]; + let args = parse_args(&["ts", "--map", "not-a-map"]); assert!(resolve(&args).is_err(), "malformed --map errors"); } @@ -486,11 +511,16 @@ mod tests { fn basic_auth_on_non_loopback_listen_is_rejected() { // Injected Basic auth on a non-loopback bind would expose the upstream // credentials to any reachable network client. - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.listen = "0.0.0.0:18080".into(); - args.allow_non_loopback = true; - args.basic_auth = Some("dev:secret".into()); + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + "--allow-non-loopback", + "--basic-auth", + "dev:secret", + ]); let err = resolve(&args).expect_err("non-loopback listen with --basic-auth should be rejected"); assert!( @@ -502,7 +532,14 @@ mod tests { ); // The same non-loopback bind without credentials is allowed. - args.basic_auth = None; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + "--allow-non-loopback", + ]); assert!( resolve(&args).is_ok(), "non-loopback without --basic-auth is allowed" @@ -512,8 +549,7 @@ mod tests { #[test] fn invalid_from_host_is_rejected() { // A FROM with characters that would break the PAC JS / Host header. - let mut args = base_args(); - args.map = vec!["bad\"host=to.edgecompute.app".into()]; + let args = parse_args(&["ts", "--map", "bad\"host=to.edgecompute.app"]); let err = resolve(&args).expect_err("a malformed FROM host should error"); assert!( matches!(err.current_context(), ConfigError::InvalidFrom { .. }), @@ -523,14 +559,25 @@ mod tests { #[test] fn non_loopback_listen_requires_flag() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.listen = "0.0.0.0:18080".into(); + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + ]); assert!( resolve(&args).is_err(), "non-loopback without flag is rejected" ); - args.allow_non_loopback = true; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + "--allow-non-loopback", + ]); assert!(resolve(&args).is_ok(), "non-loopback allowed with flag"); } @@ -553,11 +600,15 @@ mod tests { #[test] fn resolve_precomputes_typed_rule_identity_and_headers() { - let mut args = base_args(); - args.map = vec!["www.example.com=TO.Example.com:8443".into()]; - args.rewrite_host = true; - args.insecure = true; - args.resolve = vec!["to.example.com:192.0.2.10".into()]; + let args = parse_args(&[ + "ts", + "--map", + "www.example.com=TO.Example.com:8443", + "--rewrite-host", + "--insecure", + "--resolve", + "to.example.com:192.0.2.10", + ]); let cfg = resolve(&args).expect("should resolve"); let rule = cfg @@ -596,10 +647,7 @@ mod tests { #[test] fn resolve_keeps_ip_reference_identities_http1_only() { - let mut args = base_args(); - args.map = vec!["www.example.com=127.0.0.1".into()]; - args.rewrite_host = true; - + let args = parse_args(&["ts", "--map", "www.example.com=127.0.0.1", "--rewrite-host"]); let cfg = resolve(&args).expect("should resolve"); let rule = cfg .rules @@ -649,9 +697,13 @@ mod tests { let dir = tempfile::tempdir().expect("should create temp dir"); let missing = dir.path().join("no-such-file.txt"); - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.basic_auth_file = Some(missing.to_string_lossy().into_owned()); + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--basic-auth-file", + &missing.to_string_lossy(), + ]); let err = resolve(&args).expect_err("should fail when file is missing"); assert!( @@ -660,9 +712,19 @@ mod tests { ); } + #[test] + #[should_panic(expected = "DisplayHelpOnMissingArgumentOrSubcommand")] + fn bare_invocation_is_rejected_at_parse_time() { + // `arg_required_else_help` makes a fully-bare `ts` fail to parse at all, + // before `resolve` (and its `NoRule` check) ever runs. + parse_args(&["ts"]); + } + #[test] fn no_rule_passed_is_a_no_rule_error() { - let args = base_args(); + // An invocation with some other flag but no rule still reaches + // `resolve`: `arg_required_else_help` only rejects a fully-bare `ts`. + let args = parse_args(&["ts", "--insecure"]); let err = resolve(&args).expect_err("should error when no rule is passed"); assert!( matches!(err.current_context(), ConfigError::NoRule), diff --git a/docs/guide/ts-dev-proxy.md b/docs/guide/ts-dev-proxy.md index b99603b9e..234385f3d 100644 --- a/docs/guide/ts-dev-proxy.md +++ b/docs/guide/ts-dev-proxy.md @@ -77,7 +77,8 @@ shorthand, or one or more `--map FROM=TO` rules: ts dev proxy -f www.example-publisher.com -t trusted-server-example.edgecompute.app ``` -With no `--map`/`-f`/`-t`, the proxy exits with +A bare `ts dev proxy` prints help and exits before proxy startup. An +invocation with explicit options but no complete rewrite rule reports `no rewrite rule: pass --map FROM=TO (or -f/--from with -t/--to)`. Connection options — `--rewrite-host`, `--basic-auth`/`--basic-auth-file`, From 155d3b09ffb0fe637cf59f6972cf327be5ce7cc9 Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Thu, 24 Sep 2026 10:34:10 +0530 Subject: [PATCH 092/104] Clarify trace body validation and browser cleanup --- ...-mobile-ad-render-trace-endpoint-design.md | 45 +++++++++++++++---- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index e3aaca4c6..e12d79760 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -503,9 +503,20 @@ Rules (route responses below apply after configured authentication): - For both POST paths, accept absent or exactly-zero `Content-Length` only when the body is empty; reject `Transfer-Encoding`, positive/invalid lengths, and any actual body bytes with local `413 Payload Too Large` and no mutation. - Precheck headers, then use `Body::into_bytes_bounded(0)` to check emptiness, - following the header-precheck/body-size-check pattern in - `crates/trusted-server-core/src/auction/endpoints.rs`. This is an application + Precheck headers, then explicitly match both body variants: accept + `Body::Once` only when its bytes are empty; for `Body::Stream`, consume chunks + until EOF, skipping empty chunks and rejecting the first non-empty chunk + with a local `413`. A stream read error returns local `400 Bad Request` + without mutation; only a clean EOF proves a streamed body empty. Build the + `StatusCode::PAYLOAD_TOO_LARGE` response explicitly, as in the + header-precheck/body-size-check pattern in + `crates/trusted-server-core/src/auction/endpoints.rs`. Do not copy that + handler's `into_bytes().unwrap_or_default()`: `Body::into_bytes` returns + `None` for streams, which would incorrectly treat a non-empty streamed body + as empty. Nor should `Body::into_bytes_bounded(0)` errors be propagated as + the response: overflow is `EdgeError::bad_request`, which maps to `400`, + and `EdgeError` has no `413` variant. All local body-validation errors use + the section 12.3 response hardening. This is an application acceptance limit, not a transport read or allocation limit. Pinned EdgeZero v0.0.8 buffers the Fastly body with blocking `read_to_end` and the Cloudflare body with `req.bytes().await` before core handling. Spin also buffers the @@ -1424,12 +1435,16 @@ object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self img-src data: ``` -The shell supplies a fixed data-URL favicon; `img-src data:` allows it without -an automatic publisher `/favicon.ico` fetch. All styles live in the fixed CSS -asset: toggle classes or the `hidden` attribute, with no inline style attributes, +The shell declares a fixed data-URL favicon with `` to suppress +the implicit publisher `/favicon.ico` fallback; `img-src data:` permits that +declared data URL to load despite `default-src 'none'`. All styles live in the +fixed CSS asset: toggle classes or the `hidden` attribute, with no inline style attributes, style blocks, or JavaScript style-property writes. JSON download uses a Blob object URL assigned directly to an `` followed by a click, then -revokes the URL after the download has started. Do not fetch the blob URL or +schedules `URL.revokeObjectURL` with `setTimeout` for 1000 ms after the click. +Never revoke synchronously after `click()`; the delay gives the browser time +to acquire the Blob and is not a download-completion signal. Each download +schedules cleanup of its own object URL. Do not fetch the blob URL or embed it in a frame. This fixes the download mechanism without widening `connect-src` or enabling frames; browser tests exercise it under this exact CSP. @@ -1577,8 +1592,11 @@ results, never a prerequisite for returning them. with no claim that inactive proves the cookie is absent. - Empty-body enforcement rejects positive/invalid lengths, transfer encoding, nonempty bodies even with absent/zero lengths, and verifies no mutation on - rejection. Tests must not claim a transport bound or timeout that the pinned - adapters cannot enforce. + rejection. Cover both `Body::Once` and `Body::Stream`, including clean EOF, + empty chunks before EOF or a non-empty chunk, and stream read errors. Assert + local `413` for body bytes and local `400` for read errors, with section 12.3 + hardening and no cookie mutation. Tests must not claim a transport bound or + timeout that the pinned adapters cannot enforce. - Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, configured filters, ordinary event context, and origin fetch. - Cookie-health scanner covers multiple header fields; zero, one, and duplicate @@ -1646,6 +1664,8 @@ results, never a prerequisite for returning them. shadowing a broad rule to pin first-match-wins behavior. - GET, HEAD, state-changing POST, and unsupported methods obey the same lifecycle contract across adapters. +- Enable/end POSTs without `Content-Type` accept empty bodies and reject actual + body bytes without cookie mutation, including Axum's streaming path. - Every adapter omits JA4/H2 and rejects control characters or overlong platform strings. @@ -1715,6 +1735,9 @@ results, never a prerequisite for returning them. stored copies containing any excluded property are rejected. Numbered slot/cycle correlation still joins after redaction. - Download filename and MIME type are deterministic. +- Fake timers verify that each download clicks its Blob-backed anchor before + scheduling cleanup, never revokes synchronously, and revokes its own URL + when the 1000 ms timer fires, including repeated downloads. - Formatted-JSON copy and JSON-file Web Share success, rejection, absence, and download/copy fallback behavior. - 320-pixel layout, keyboard navigation, focus handling, and accessible status @@ -1722,6 +1745,10 @@ results, never a prerequisite for returning them. ### 14.4 Browser integration tests +- Under the exact section 12.3 CSP, JSON downloads contain the complete report + with deferred URL cleanup, including the storage-failure recovery path and + clipboard/Web Share fallback. The declared data-URL favicon loads without + an implicit `/favicon.ico` request. - First endpoint GET is read-only and shows setup state; a user-initiated, same-origin enable POST sets the session. - Successful in-page activation adds no history entry, so Back can return to From a1710a059c0401447f4ac7e033799d07cbcf0081 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 23 Sep 2026 23:04:58 -0700 Subject: [PATCH 093/104] Pin EdgeZero to an immutable rev and close the review findings Replace the mutable `branch =` pin on all six EdgeZero dependencies with `rev = "12c3215c2637d961a27eefa12e235033fadb4c4a"`, the commit the lockfile already resolved, so `main` stays reproducible and no `cargo update` or upstream force-push can move it. Issue #1195 tracks replacing the rev with the release tag once the upstream PR merges. Assert the legacy physical `ts_secrets` store is absent from both Viceroy configurations, so a squash merge that resurrects the block fails instead of writing entries into a store nothing opens. Correct the `RuntimeStoreConfig::logical` doc comment: the crate does read the process environment elsewhere, so state the real invariant, that no store selector reaches the runtime. Document the Config Store selector in the Fastly guide alongside the secrets one, and drop the leftover instruction not to create a `trusted_server_secrets` link, which the deployment now creates itself. --- Cargo.lock | 16 ++++++++-------- Cargo.toml | 12 ++++++------ crates/trusted-server-adapter-fastly/src/app.rs | 11 ++++++----- .../tests/common/config.rs | 4 ++++ docs/guide/fastly.md | 14 ++++++++++++-- 5 files changed, 36 insertions(+), 21 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d5b777858..0aadf3761 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "log", "proc-macro2", diff --git a/Cargo.toml b/Cargo.toml index 5476d0760..75060c4c7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -55,12 +55,12 @@ cssparser = "0.36" derive_more = { version = "2.0", features = ["display", "error"] } directories = "5" ed25519-dalek = { version = "2.2", features = ["rand_core"] } -edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-cli = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors" } -edgezero-core = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } +edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a" } +edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } env_logger = "0.11" error-stack = "0.6" esi = "0.7.2" diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 9f538ad1c..0829045ff 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -162,11 +162,12 @@ pub(crate) struct RuntimeStoreConfig { impl RuntimeStoreConfig { /// Store bindings for the Fastly runtime. /// - /// Fastly Compute has no process environment. `EdgeZero` links each selected - /// physical store to the service version under its logical ID, so the - /// runtime opens stores by logical ID and reads the config entry under - /// that same ID for every publication target. Staging isolation comes from - /// linking a different physical store, never from a different key. + /// Store selection is a deploy-time input; no store selector reaches the + /// Fastly runtime. `EdgeZero` links each selected physical store to the + /// service version under its logical ID, so the runtime opens stores by + /// logical ID and reads the config entry under that same ID for every + /// publication target. Staging isolation comes from linking a different + /// physical store, never from a different key. pub(crate) fn logical() -> Self { Self { config_store_name: StoreName::from(DEFAULT_CONFIG_STORE_ID), diff --git a/crates/trusted-server-integration-tests/tests/common/config.rs b/crates/trusted-server-integration-tests/tests/common/config.rs index 1e83da363..b8eeb0783 100644 --- a/crates/trusted-server-integration-tests/tests/common/config.rs +++ b/crates/trusted-server-integration-tests/tests/common/config.rs @@ -86,6 +86,10 @@ mod tests { local_server["secret_stores"][LOGICAL_SECRET_STORE_ID].is_array(), "{name} should expose the secret store under its logical ID" ); + assert!( + local_server["secret_stores"].get("ts_secrets").is_none(), + "{name} should not define the legacy physical secret store" + ); assert!( local_server["config_stores"] .get("edgezero_runtime_env") diff --git a/docs/guide/fastly.md b/docs/guide/fastly.md index cfd3df15a..4ec64ebef 100644 --- a/docs/guide/fastly.md +++ b/docs/guide/fastly.md @@ -271,6 +271,16 @@ Used for storing public configuration (e.g., public keys, key metadata): fastly config-store create --name jwks_store ``` +Trusted Server's app config lives under logical store ID +`trusted_server_config`. Select its physical store in each deployment +environment, the same way as the secret store below. Production and staging +isolate their app config by selecting different physical stores, because the +config entry key is the logical ID on every target: + +```bash +export EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME= +``` + ### Secret Stores Trusted Server keeps static app-config credentials under logical store ID @@ -312,8 +322,8 @@ Create the separate request-signing store when that feature is enabled: fastly secret-store create --name signing_keys ``` -Do not copy the same app credential store under a second hardcoded -`trusted_server_secrets` Fastly link. Configure the mapping instead. +The deployment creates the `trusted_server_secrets` link itself from the +selected physical store. Do not create that link by hand. ## Create EC KV Store From bf96305b9d327b7927eb918361f9eb30018e0d1e Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 24 Sep 2026 13:50:42 +0530 Subject: [PATCH 094/104] Pin adserver mock bid dimension boundaries in tests Cover zero height separately from missing dimensions, accept u32::MAX exactly, skip negative dimensions, rename the shared oversized dimension variable, and document why the raw upstream value is logged with Debug formatting. Signed-off-by: dhruv8sh --- .../src/integrations/adserver_mock.rs | 130 +++++++++++++++++- 1 file changed, 124 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index a2a47c9b1..066e32e28 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -292,6 +292,8 @@ impl AdServerMockProvider { let restored_bidder = original.map_or_else(|| seat_name.to_string(), |b| b.bidder.clone()); + // Keep `{:?}` for the raw upstream values below: Debug escapes + // newlines and quotes, which prevents log injection. let Some(width) = bid["w"].as_u64().and_then(|v| u32::try_from(v).ok()) else { log::debug!( "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", @@ -1344,7 +1346,7 @@ mod tests { let config = AdServerMockConfig::default(); let provider = AdServerMockProvider::new(config); - let oversized_width = u64::from(u32::MAX) + 101; + let oversized_dimension = u64::from(u32::MAX) + 101; let mediation_response = json!({ "id": "test-auction-123", "seatbid": [ @@ -1356,7 +1358,7 @@ mod tests { "impid": "header-banner", "price": 3.50, "adm": "
Oversized width
", - "w": oversized_width, + "w": oversized_dimension, "h": 90, }, { @@ -1365,7 +1367,7 @@ mod tests { "price": 1.25, "adm": "
Oversized height
", "w": 300, - "h": oversized_width, + "h": oversized_dimension, }, { "id": "bid-valid", @@ -1415,9 +1417,17 @@ mod tests { "h": 90, }, { - "id": "bid-missing-dimensions", + "id": "bid-zero-height", "impid": "sidebar", "price": 1.25, + "adm": "
Zero height
", + "w": 300, + "h": 0, + }, + { + "id": "bid-missing-dimensions", + "impid": "skyscraper", + "price": 1.10, "adm": "
Missing dimensions
", }, { @@ -1434,15 +1444,123 @@ mod tests { "cur": "USD" }); + let auction_response = + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + + let slots: Vec<&str> = auction_response + .bids + .iter() + .map(|bid| bid.slot_id.as_str()) + .collect(); + assert_eq!( + slots, + ["footer"], + "should drop the zero-width, zero-height, and missing-dimension bids" + ); + } + + #[test] + fn test_parse_mediation_response_accepts_u32_max_dimensions() { + // u32::MAX is the largest representable dimension and must be accepted, + // pinning the upper boundary of the oversized-dimension check. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-max-dimensions", + "impid": "header-banner", + "price": 3.50, + "adm": "
Max dimensions
", + "w": u32::MAX, + "h": u32::MAX, + } + ] + } + ], + "cur": "USD" + }); + let auction_response = provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); assert_eq!( auction_response.bids.len(), 1, - "Bids with zero or missing w/h should be skipped, only the valid bid should remain" + "should accept a bid whose w/h equal u32::MAX" + ); + assert_eq!( + auction_response.bids[0].width, + u32::MAX, + "should keep width at u32::MAX" + ); + assert_eq!( + auction_response.bids[0].height, + u32::MAX, + "should keep height at u32::MAX" + ); + } + + #[test] + fn test_parse_mediation_response_skips_negative_dimensions() { + // Negative dimensions are not valid u64 values and must be skipped. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-negative-width", + "impid": "header-banner", + "price": 3.50, + "adm": "
Negative width
", + "w": -1, + "h": 90, + }, + { + "id": "bid-negative-height", + "impid": "sidebar", + "price": 1.25, + "adm": "
Negative height
", + "w": 300, + "h": -1, + }, + { + "id": "bid-valid", + "impid": "footer", + "price": 2.00, + "adm": "
Valid Ad
", + "w": 728, + "h": 90, + } + ] + } + ], + "cur": "USD" + }); + + let auction_response = + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + + let slots: Vec<&str> = auction_response + .bids + .iter() + .map(|bid| bid.slot_id.as_str()) + .collect(); + assert_eq!( + slots, + ["footer"], + "should drop the negative-width and negative-height bids" ); - assert_eq!(auction_response.bids[0].slot_id, "footer"); } #[test] From 8dd9713b3fcf30cba7b9615e6486b8a9c199e8a0 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 24 Sep 2026 14:01:15 +0530 Subject: [PATCH 095/104] Assert bare proxy invocation via clap's typed error kind Add a fallible try_parse_args test helper and check error.kind() against DisplayHelpOnMissingArgumentOrSubcommand instead of matching a should_panic substring from clap's private ErrorInner Debug output. Signed-off-by: dhruv8sh --- .../src/commands/dev/proxy/config.rs | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs index f097c8b71..86ffc9d0f 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs @@ -360,12 +360,18 @@ mod tests { }; fn parse_args(argv: &[&str]) -> crate::commands::dev::proxy::ProxyArgs { + try_parse_args(argv).expect("should parse proxy args") + } + + fn try_parse_args( + argv: &[&str], + ) -> Result { #[derive(clap::Parser)] struct W { #[command(flatten)] a: crate::commands::dev::proxy::ProxyArgs, } - W::try_parse_from(argv).expect("should parse proxy args").a + W::try_parse_from(argv).map(|w| w.a) } #[test] @@ -713,11 +719,16 @@ mod tests { } #[test] - #[should_panic(expected = "DisplayHelpOnMissingArgumentOrSubcommand")] fn bare_invocation_is_rejected_at_parse_time() { // `arg_required_else_help` makes a fully-bare `ts` fail to parse at all, // before `resolve` (and its `NoRule` check) ever runs. - parse_args(&["ts"]); + let error = try_parse_args(&["ts"]) + .expect_err("a fully-bare invocation should short-circuit to help"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::DisplayHelpOnMissingArgumentOrSubcommand, + "should short-circuit to help rather than reaching resolve" + ); } #[test] From d100d1b3c87755d5af42ab106a904ebe565e2698 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 24 Sep 2026 14:01:15 +0530 Subject: [PATCH 096/104] Resolve proxy rules before restoring pending system proxy state An invocation with flags but no usable rewrite rule (for example ts dev proxy --insecure) previously ran the Safari system-proxy restore, and could attempt sudo, before failing on the missing rule. Resolve the config first so it fails without touching system proxy state; a proxy stranded by a hard-killed run is still restored on the next valid run. Add a regression test using a malformed restore file, which the restore path deletes without running networksetup, and soften the guide so it no longer promises the concise one-line error output. Signed-off-by: dhruv8sh --- .../src/commands/dev/proxy/browser.rs | 2 +- .../src/commands/dev/proxy/mod.rs | 50 ++++++++++++++++--- docs/guide/ts-dev-proxy.md | 4 +- 3 files changed, 47 insertions(+), 9 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs b/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs index 4af9b2d3d..e61f92a01 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs @@ -18,7 +18,7 @@ use crate::output; /// is `on` or `off`. A missing third line is tolerated when reading (treated as /// `on` if a URL is present, else `off`) for forward-compatibility with the /// earlier two-line format. -const SAFARI_RESTORE_FILE: &str = "safari-proxy-restore"; +pub(super) const SAFARI_RESTORE_FILE: &str = "safari-proxy-restore"; /// Generates a PAC script that proxies only `https://` requests for matched FROM hosts. /// diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs index 6ec9e497d..8bb28118a 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs @@ -252,14 +252,19 @@ pub fn run(args: &ProxyArgs) -> core::result::Result<(), error_stack::Report::try_parse_from(["ts", "--insecure", "--ca-dir", &ca_dir]) + .expect("should parse proxy args") + .a; + + let err = run(&args).expect_err("should fail without a rewrite rule"); + + assert!( + matches!(err.current_context(), ProxyError::Config), + "should fail with a config error" + ); + assert!( + restore_path.exists(), + "should fail before attempting to restore the system proxy" + ); + } } diff --git a/docs/guide/ts-dev-proxy.md b/docs/guide/ts-dev-proxy.md index 234385f3d..789393e92 100644 --- a/docs/guide/ts-dev-proxy.md +++ b/docs/guide/ts-dev-proxy.md @@ -78,8 +78,8 @@ ts dev proxy -f www.example-publisher.com -t trusted-server-example.edgecompute. ``` A bare `ts dev proxy` prints help and exits before proxy startup. An -invocation with explicit options but no complete rewrite rule reports -`no rewrite rule: pass --map FROM=TO (or -f/--from with -t/--to)`. +invocation with explicit options but no complete rewrite rule fails with a +`no rewrite rule` error before touching system proxy state. Connection options — `--rewrite-host`, `--basic-auth`/`--basic-auth-file`, `--insecure`, and `--upstream-plaintext` — apply to every mapping, not per-rule. From 0fa7dfb649da04050b9185bde34137a1e0696b2e Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 24 Sep 2026 19:38:49 -0500 Subject: [PATCH 097/104] Clarify Prebid diagnostics export boundaries --- .../src/integrations/gpt_diagnostics/store.ts | 2 ++ .../gpt_diagnostics/store.test.ts | 24 +++++++++++++++++++ .../gpt-diagnostics-dictionary.md | 10 ++++++++ docs/guide/integrations/gpt-diagnostics.md | 22 +++++++++++++---- 4 files changed, 54 insertions(+), 4 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts index fd32b9c26..69c87abea 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts @@ -1101,6 +1101,8 @@ export class GptDiagnosticsStore { this.nextRequestIntentId += 1; this.pendingRequestIntents.set(slot, intent); } + // Replace rather than merge: a later bare refresh must not carry an + // earlier auction's evidence into the next GPT request. intent.sources.set(source, { observedAtMs, ...facts }); } diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts index 22c9760c1..64b4ecfa4 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts @@ -710,6 +710,30 @@ describe('GptDiagnosticsStore', () => { }); }); + it('does not carry an earlier auction into a later bare Prebid refresh', () => { + let now = 10; + const store = new GptDiagnosticsStore({ now: () => now, defer: () => undefined }); + const slot = fakeSlot('prebid-replaced-intent'); + store.recordPrebidRefresh([slot]); + store.recordPrebidAuction(slot, 'auction-client-1', { + bidder: 'example-client', + priceBucket: '2.40', + }); + + now = 20; + store.recordPrebidRefresh([slot]); + store.recordSlotRequested(slot); + store.recordPrebidWin(slot, 'auction-client-1', { + bidder: 'example-client', + priceBucket: '2.40', + }); + + const cycle = store.snapshot().slots[0].requests[0]; + expect(cycle.requestPath).toBe('prebid_refresh'); + expect(cycle.auctionType).toBeUndefined(); + expect(cycle.prebidAuction).toBeUndefined(); + }); + it('retains bounded winner and server timing facts for a Trusted Server auction', () => { const store = new GptDiagnosticsStore({ now: () => 10, defer: () => undefined }); const slot = fakeSlot('auction-facts'); diff --git a/docs/guide/integrations/gpt-diagnostics-dictionary.md b/docs/guide/integrations/gpt-diagnostics-dictionary.md index e0dc433a6..2901cbb64 100644 --- a/docs/guide/integrations/gpt-diagnostics-dictionary.md +++ b/docs/guide/integrations/gpt-diagnostics-dictionary.md @@ -65,6 +65,16 @@ Panel status is `GPT observed` or `Waiting for GPT`. Filter values are `All`, `V Bidder names are limited to 128 UTF-8 bytes, numeric bucket strings to 64 bytes, currencies to three ASCII letters, and auction IDs to 256 UTF-8 bytes. Prebid supplies its own auction ID through `bidsBackHandler`; diagnostics does not override Prebid auction identity. Duplicate, ambiguous, expired, late, prior-navigation, and malformed observations are rejected. Only an active diagnostics recorder installs the `bidWon` listener or retains candidate/win state, which is bounded to 128 pending attempts and 30 seconds. No raw CPM, creative markup, targeting dump, or losing bids are retained. +First-party GPT and Prebid integrations never populate `currency`. The optional field +accepts currency supplied through the internal `gptDiagnosticsRecorder` channel; that +channel is not a supported operator API. Diagnostics does not infer a currency or read +`hb_cur`, and missing currency adds no suffix to the price bucket. + +`Auction not observed` also covers the Prebid watchdog fallback: targeting can be +applied even when Prebid never invokes `bidsBackHandler`, so no completed-auction ID +is available to correlate with the GPT request. Targeting alone is not proof of a +completed auction. + ## Request paths and opportunities | Label | Raw value | Meaning | diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md index 7b8345400..1928c1d6c 100644 --- a/docs/guide/integrations/gpt-diagnostics.md +++ b/docs/guide/integrations/gpt-diagnostics.md @@ -494,6 +494,13 @@ The allowlisted export contains: timestamps, and safe failure enums. - The bounded winning bidder and bucketed price plus server auction timing fields and their `navigation` or `spa_auction` origin when direct auction evidence was observed. +- Completed client-side Prebid evidence (`prebidAuction`): Prebid's opaque `auctionId`, + plus bounded bidder and bucketed price fields in `targetingCandidate` and `win`, + when a completed attempt and any documented `bidWon` observation were correlated + to that exact request. Neither candidate nor win evidence proves the final served bidder. +- Optional `currency` on server winner, Prebid candidate, and Prebid win evidence, + when supplied through the internal recorder. Values are normalized to three uppercase + ASCII letters; first-party integrations do not populate this field. - The per-auction diagnostics token (`trustedServerAuctionId`) and the opportunity-to-request duration, when a direct opportunity was observed. - Replacement facts for a re-rendered slot: `replacedRequestNumber`, @@ -504,10 +511,11 @@ The allowlisted export contains: - Separate callback issues, attribution issues, coverage counters, and retention counters. -It does not contain raw targeting, bid IDs, exact unbucketed bid prices, losing bidder -identity, creative markup, cache URLs, cache payloads, cache or bridge error details, -cookies, user identifiers, query strings, or URL fragments. It does contain the winning -bidder and bucketed `hb_pb` value described above. The exported `trustedServerAuctionId` +It does not contain raw targeting, bid IDs, exact unbucketed bid prices, losing bid +lists, creative markup, cache URLs, cache payloads, cache or bridge error details, +cookies, user identifiers, query strings, or URL fragments. It does contain the server +winner and the Prebid candidate and win bidder names with their bucketed `hb_pb` values +as described above, even when the final served bidder is unconfirmed. The exported `trustedServerAuctionId` is the `hb_auction_id` value described in [Auction correlation token](#auction-correlation-token): minted fresh for each server-side auction, not derived from the Edge Cookie ID or any other visitor @@ -515,6 +523,12 @@ identifier, and never repeated across auctions, so it cannot be joined back to a visitor. Diagnostics retain it only after trimming to a non-empty value of at most 256 UTF-8 bytes. +The exported `prebidAuction.auctionId` is supplied by Prebid for auction correlation, +not visitor identity. Diagnostics does not generate or replace it and retains only +non-empty values of at most 256 UTF-8 bytes. Bidder names are limited to 128 UTF-8 +bytes and numeric price bucket strings to 64 bytes. These bounds limit retained data; +they do not establish how a publisher's Prebid configuration generated its auction ID. + Captured records are memory-only. Diagnostics do not add an upload, diagnostics network request, `localStorage`, `sessionStorage`, IndexedDB, or other persistence. `export()` creates only the user-requested local JSON download; it sends nothing to a From 819ac714f3c2f16e267ce91f4e8c8f06ebd99af7 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 13:13:15 +0530 Subject: [PATCH 098/104] Parse mock bid dimensions with the shared OpenRTB helper Reuse parse_optional_bid_dimension so integral float dimensions are accepted like in the other auction providers, and pin that behavior with a test. Signed-off-by: dhruv8sh --- .../src/integrations/adserver_mock.rs | 79 ++++++++++++++++--- 1 file changed, 67 insertions(+), 12 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 066e32e28..21cb41341 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -16,6 +16,7 @@ use std::time::Duration; use validator::Validate; use crate::auction::context::{ContextQueryParams, build_url_with_context_params}; +use crate::auction::openrtb::parse_optional_bid_dimension; use crate::auction::provider::{AuctionProvider, ProviderRequestOutcome}; use crate::auction::types::{ AuctionContext, AuctionRequest, AuctionResponse, Bid, BidStatus, MediaType, @@ -292,28 +293,25 @@ impl AdServerMockProvider { let restored_bidder = original.map_or_else(|| seat_name.to_string(), |b| b.bidder.clone()); - // Keep `{:?}` for the raw upstream values below: Debug escapes - // newlines and quotes, which prevents log injection. - let Some(width) = bid["w"].as_u64().and_then(|v| u32::try_from(v).ok()) else { + // Reuse the shared `OpenRTB` dimension parser so integral floats + // (`300.0`) are accepted and zero, negative, or oversized values + // are rejected, matching the other auction providers. Keep `{:?}` + // for the raw upstream values below: Debug escapes newlines and + // quotes, which prevents log injection. + let Ok(Some(width)) = parse_optional_bid_dimension(bid, "w") else { log::debug!( "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", bid["w"] ); continue; }; - let Some(height) = bid["h"].as_u64().and_then(|v| u32::try_from(v).ok()) else { + let Ok(Some(height)) = parse_optional_bid_dimension(bid, "h") else { log::debug!( "adserver_mock: bid for slot '{slot_id}' has invalid height {:?}, skipping", bid["h"] ); continue; }; - if width == 0 || height == 0 { - log::debug!( - "adserver_mock: bid for slot '{slot_id}' has zero dimension ({width}×{height}), skipping" - ); - continue; - } all_bids.push(Bid { slot_id, @@ -1341,8 +1339,8 @@ mod tests { fn test_parse_mediation_response_skips_oversized_dimensions() { // A dimension above u32::MAX must be rejected rather than silently // wrapped into a small, plausible-looking value. The offset of 101 is - // load-bearing: u32::MAX + 1 truncates to 0, which the zero-check - // below would already have caught, so it would not pin this fix. + // load-bearing: u32::MAX + 1 truncates to 0, which is already rejected + // as a zero dimension, so it would not pin this fix. let config = AdServerMockConfig::default(); let provider = AdServerMockProvider::new(config); @@ -1563,6 +1561,63 @@ mod tests { ); } + #[test] + fn test_parse_mediation_response_accepts_integral_float_dimensions() { + // Integral floats (`300.0`) are a legitimate dimension encoding, matching + // the shared `OpenRTB` parser; fractional floats are still skipped. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-float-dimensions", + "impid": "header-banner", + "price": 3.50, + "adm": "
Float dimensions
", + "w": 300.0, + "h": 250.0, + }, + { + "id": "bid-fractional-width", + "impid": "sidebar", + "price": 1.25, + "adm": "
Fractional width
", + "w": 300.5, + "h": 250, + } + ] + } + ], + "cur": "USD" + }); + + let auction_response = + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + + assert_eq!( + auction_response.bids.len(), + 1, + "should accept the integral-float bid and skip the fractional one" + ); + assert_eq!( + auction_response.bids[0].slot_id, "header-banner", + "should keep the integral-float bid" + ); + assert_eq!( + ( + auction_response.bids[0].width, + auction_response.bids[0].height + ), + (300, 250), + "should convert integral floats to u32 dimensions" + ); + } + #[test] fn test_build_endpoint_url_with_context_query_params() { let config = AdServerMockConfig { From 7eed6a3f2a4bfa0a25abd7c738087162f16f738d Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 13:39:04 +0530 Subject: [PATCH 099/104] Validate mock mediated bid dimensions against requested slots Build the requested-dimension index from the auction request and run mediated bids through resolve_bid_dimensions, so the mock provider rejects unrequested impressions and mismatched sizes and infers missing dimensions like the other auction providers. Signed-off-by: dhruv8sh --- .../src/auction/openrtb.rs | 17 +- .../src/integrations/adserver_mock.rs | 164 ++++++++++++++---- 2 files changed, 142 insertions(+), 39 deletions(-) diff --git a/crates/trusted-server-core/src/auction/openrtb.rs b/crates/trusted-server-core/src/auction/openrtb.rs index cf657a1ec..d2a988db3 100644 --- a/crates/trusted-server-core/src/auction/openrtb.rs +++ b/crates/trusted-server-core/src/auction/openrtb.rs @@ -16,7 +16,7 @@ use super::profile::{ use super::routing::{ PrebidTransportHeaders, ProviderAuctionInput, ProviderSlotInput, RoutedAuction, }; -use super::types::{AdFormat, AuctionResponse, Bid}; +use super::types::{AdFormat, AdSlot, AuctionResponse, Bid}; use crate::consent::ConsentSource; use crate::error::TrustedServerError; use crate::openrtb::{ @@ -154,11 +154,20 @@ pub(crate) type BidDimensionIndex = BTreeMap; /// Build the requested-dimension index once for one provider response. pub(crate) fn build_bid_dimension_index(input: &ProviderAuctionInput) -> BidDimensionIndex { + build_bid_dimension_index_from_slots(input.slots().iter().map(ProviderSlotInput::slot)) +} + +/// Build the requested-dimension index from plain [`AdSlot`]s. +/// +/// The first slot wins when several share an ID. +pub(crate) fn build_bid_dimension_index_from_slots<'a>( + slots: impl IntoIterator, +) -> BidDimensionIndex { let mut index = BidDimensionIndex::new(); - for slot in input.slots() { + for slot in slots { index - .entry(slot.slot().id.clone()) - .or_insert_with(|| SlotBidDimensions::from_formats(slot.slot().formats.as_slice())); + .entry(slot.id.clone()) + .or_insert_with(|| SlotBidDimensions::from_formats(slot.formats.as_slice())); } index } diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 21cb41341..640574e03 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -16,7 +16,10 @@ use std::time::Duration; use validator::Validate; use crate::auction::context::{ContextQueryParams, build_url_with_context_params}; -use crate::auction::openrtb::parse_optional_bid_dimension; +use crate::auction::openrtb::{ + BidDimensionIndex, BidRejectionReason, build_bid_dimension_index_from_slots, + parse_optional_bid_dimension, resolve_bid_dimensions, +}; use crate::auction::provider::{AuctionProvider, ProviderRequestOutcome}; use crate::auction::types::{ AuctionContext, AuctionRequest, AuctionResponse, Bid, BidStatus, MediaType, @@ -265,6 +268,7 @@ impl AdServerMockProvider { json: &Json, response_time_ms: u64, bid_index: &BidIndex, + dimensions_by_slot: Option<&BidDimensionIndex>, ) -> AuctionResponse { let empty_array = vec![]; let seatbid = json["seatbid"].as_array().unwrap_or(&empty_array); @@ -293,24 +297,31 @@ impl AdServerMockProvider { let restored_bidder = original.map_or_else(|| seat_name.to_string(), |b| b.bidder.clone()); - // Reuse the shared `OpenRTB` dimension parser so integral floats - // (`300.0`) are accepted and zero, negative, or oversized values - // are rejected, matching the other auction providers. Keep `{:?}` - // for the raw upstream values below: Debug escapes newlines and - // quotes, which prevents log injection. - let Ok(Some(width)) = parse_optional_bid_dimension(bid, "w") else { - log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", - bid["w"] - ); - continue; + // Reuse the shared `OpenRTB` dimension parser and slot-format + // validation so this provider admits the same dimensions as the + // other auction providers. Keep `{:?}` for the raw upstream + // values below: Debug escapes newlines and quotes, which + // prevents log injection. + let dimensions = match ( + parse_optional_bid_dimension(bid, "w"), + parse_optional_bid_dimension(bid, "h"), + ) { + (Ok(width), Ok(height)) => match dimensions_by_slot { + Some(index) => resolve_bid_dimensions(index, &slot_id, width, height), + None => width.zip(height).ok_or(BidRejectionReason::InvalidBid), + }, + _ => Err(BidRejectionReason::InvalidBid), }; - let Ok(Some(height)) = parse_optional_bid_dimension(bid, "h") else { - log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid height {:?}, skipping", - bid["h"] - ); - continue; + let (width, height) = match dimensions { + Ok(dimensions) => dimensions, + Err(reason) => { + log::debug!( + "adserver_mock: bid for slot '{slot_id}' has unusable dimensions {:?}×{:?} ({reason:?}), skipping", + bid["w"], + bid["h"] + ); + continue; + } }; all_bids.push(Bid { @@ -372,6 +383,7 @@ impl AdServerMockProvider { response: PlatformResponse, response_time_ms: u64, bid_index: &BidIndex, + dimensions_by_slot: Option<&BidDimensionIndex>, ) -> Result> { let response = response.response; @@ -397,8 +409,12 @@ impl AdServerMockProvider { log::trace!("AdServer Mock response: {:?}", response_json); - let auction_response = - self.parse_mediation_response(&response_json, response_time_ms, bid_index); + let auction_response = self.parse_mediation_response( + &response_json, + response_time_ms, + bid_index, + dimensions_by_slot, + ); log::info!( "AdServer Mock returned {} bids in {}ms", @@ -517,7 +533,7 @@ impl AuctionProvider for AdServerMockProvider { // [`parse_response_with_context`], so this path only serves callers // outside the orchestration flow. log::debug!("adserver_mock: parsing without context — SSP bid metadata unavailable"); - self.parse_response_inner(response, response_time_ms, &BidIndex::new()) + self.parse_response_inner(response, response_time_ms, &BidIndex::new(), None) .await } @@ -525,15 +541,21 @@ impl AuctionProvider for AdServerMockProvider { &self, response: PlatformResponse, response_time_ms: u64, - _request: &AuctionRequest, + request: &AuctionRequest, context: &AuctionContext<'_>, ) -> Result> { // Rebuild the SSP-bid lookup from the orchestrator-provided bidder // responses so nurl/burl/ad_id survive mediation. Request-scoped data // travels on the context instead of provider-instance state. let bid_index = build_bid_index(context.provider_responses.unwrap_or(&[])); - self.parse_response_inner(response, response_time_ms, &bid_index) - .await + let dimensions_by_slot = build_bid_dimension_index_from_slots(&request.slots); + self.parse_response_inner( + response, + response_time_ms, + &bid_index, + Some(&dimensions_by_slot), + ) + .await } fn supports_media_type(&self, media_type: &MediaType) -> bool { @@ -801,7 +823,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!(auction_response.provider, "adserver_mock"); assert_eq!(auction_response.status, BidStatus::Success); @@ -848,7 +870,8 @@ mod tests { ] }); - let response = provider.parse_mediation_response(&mediation_response, 10, &BidIndex::new()); + let response = + provider.parse_mediation_response(&mediation_response, 10, &BidIndex::new(), None); assert_eq!(response.bids.len(), 2); assert_eq!(response.bids[0].bidder, "provider-instance"); @@ -924,7 +947,7 @@ mod tests { ); let auction_response = - provider.parse_mediation_response(&mediation_response, 42, &bid_index); + provider.parse_mediation_response(&mediation_response, 42, &bid_index, None); assert_eq!(auction_response.status, BidStatus::Success); assert_eq!(auction_response.bids.len(), 1); @@ -1028,7 +1051,7 @@ mod tests { ); let auction_response = - provider.parse_mediation_response(&mediation_response, 42, &bid_index); + provider.parse_mediation_response(&mediation_response, 42, &bid_index, None); assert_eq!( auction_response.bids[0].bid_id.as_deref(), @@ -1092,6 +1115,7 @@ mod tests { }), 2, &reduced_index, + None, ); let winner = mediated .bids @@ -1123,7 +1147,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 100, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 100, &BidIndex::new(), None); assert_eq!(auction_response.status, BidStatus::NoBid); assert_eq!(auction_response.bids.len(), 0); @@ -1316,7 +1340,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!(auction_response.status, BidStatus::Success); assert_eq!(auction_response.bids.len(), 2); @@ -1382,7 +1406,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!( auction_response.bids.len(), @@ -1443,7 +1467,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); let slots: Vec<&str> = auction_response .bids @@ -1485,7 +1509,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!( auction_response.bids.len(), @@ -1547,7 +1571,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); let slots: Vec<&str> = auction_response .bids @@ -1597,7 +1621,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!( auction_response.bids.len(), @@ -1618,6 +1642,76 @@ mod tests { ); } + #[test] + fn test_parse_mediation_response_validates_dimensions_against_slots() { + // With the request slots available, mediated bids must match a + // requested format, and missing dimensions are inferred from the + // slot's single format, matching the other auction providers. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + let request = create_test_auction_request(); + let dimensions_by_slot = build_bid_dimension_index_from_slots(&request.slots); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-exact", + "impid": "header-banner", + "price": 3.50, + "w": 728, + "h": 90, + }, + { + "id": "bid-mismatch", + "impid": "header-banner", + "price": 3.00, + "w": 300, + "h": 250, + }, + { + "id": "bid-inferred", + "impid": "header-banner", + "price": 2.50, + }, + { + "id": "bid-unrequested", + "impid": "sidebar", + "price": 1.25, + "w": 728, + "h": 90, + } + ] + } + ], + "cur": "USD" + }); + + let auction_response = provider.parse_mediation_response( + &mediation_response, + 200, + &BidIndex::new(), + Some(&dimensions_by_slot), + ); + + let admitted: Vec<(Option<&str>, u32, u32)> = auction_response + .bids + .iter() + .map(|bid| (bid.bid_id.as_deref(), bid.width, bid.height)) + .collect(); + assert_eq!( + admitted, + [ + (Some("bid-exact"), 728, 90), + (Some("bid-inferred"), 728, 90) + ], + "should keep the exact and inferred bids and drop the mismatched and unrequested ones" + ); + } + #[test] fn test_build_endpoint_url_with_context_query_params() { let config = AdServerMockConfig { From 44fb9c2dd4c8644c5e21e9bbb9436307b5a3eca4 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 13:40:47 +0530 Subject: [PATCH 100/104] Extend the Markdown format check beyond the repo root Check every tracked Markdown file outside docs/ with Prettier, and reformat the two files that failed. In pr-reviewer.md, escape BRANCH-* so Prettier does not read it as emphasis, and keep long code spans on one line so they are not split across list-item wraps. Signed-off-by: dhruv8sh --- .claude/agents/pr-reviewer.md | 74 ++++++++++--------- .../SKILL.md | 6 +- .github/workflows/format.yml | 6 +- AGENTS.md | 2 +- 4 files changed, 46 insertions(+), 42 deletions(-) diff --git a/.claude/agents/pr-reviewer.md b/.claude/agents/pr-reviewer.md index 4f65a426f..66bc88c10 100644 --- a/.claude/agents/pr-reviewer.md +++ b/.claude/agents/pr-reviewer.md @@ -27,13 +27,13 @@ The agent resolves the input into exactly one of three **modes** before starting any fetch. Modes determine the per-invocation variables defined in step 1; later steps reference those variables and don't restate mode logic. -| Input | Mode (after resolution) | -|---|---| -| A PR number (e.g. `#165`) | **PR** | -| A branch name, and PR lookup returns exactly one matching PR | **PR** (via lookup) | -| A branch name, no PR exists | **BRANCH-REMOTE** — always. No probe on the current checkout. | -| User explicitly says "review my local working tree" | **BRANCH-LOCAL** — current checkout only. If the user also names a branch, the agent verifies it matches `git branch --show-current`; otherwise it stops and asks the user to either check out that branch first or drop the name. The agent does **not** silently review whatever HEAD happens to be. | -| No input | Run the PR lookup probe with `$(git branch --show-current)`. If it returns a PR → PR mode. Otherwise apply the no-input rule below. | +| Input | Mode (after resolution) | +| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| A PR number (e.g. `#165`) | **PR** | +| A branch name, and PR lookup returns exactly one matching PR | **PR** (via lookup) | +| A branch name, no PR exists | **BRANCH-REMOTE** — always. No probe on the current checkout. | +| User explicitly says "review my local working tree" | **BRANCH-LOCAL** — current checkout only. If the user also names a branch, the agent verifies it matches `git branch --show-current`; otherwise it stops and asks the user to either check out that branch first or drop the name. The agent does **not** silently review whatever HEAD happens to be. | +| No input | Run the PR lookup probe with `$(git branch --show-current)`. If it returns a PR → PR mode. Otherwise apply the no-input rule below. | **Branch-to-PR lookup rule.** `gh pr list --head ` does not support `:` syntax, so fork PRs with the same branch name can collide. @@ -52,7 +52,7 @@ matches=$(gh pr list --head "$REQUESTED_HEAD" \ - `>1` matches → stop and ask the user which PR number to review; do not pick the first row. -**No-input / no-PR rule** (the only place an inferred BRANCH-* mode happens +**No-input / no-PR rule** (the only place an inferred BRANCH-\* mode happens — a named branch never triggers this probe; the user said the name, the agent honours it): @@ -60,7 +60,7 @@ agent honours it): configured **and** `git rev-list --left-right --count "@{upstream}...HEAD"` returns `0 0` → resolve to **BRANCH-REMOTE**, with `` bound to the branch name the upstream points at, not to the local branch name. The - probe approves the *upstream* state; the agent must fetch that exact + probe approves the _upstream_ state; the agent must fetch that exact branch, not `origin/$(git branch --show-current)` which could be a different ref (the local branch might track `origin/main-fork`). Because BRANCH-REMOTE fetches from `origin`, this inference only applies when the @@ -87,6 +87,7 @@ agent honours it): When `$REQUESTED_HEAD` is bound, BRANCH-REMOTE proceeds with that name. When it's not bound (upstream on a non-`origin` remote), the agent asks the user instead. + - Anything else → **ask the user** which mode they want. The probe uses only `git status` / `git rev-parse` on existing local refs — it does **not** fetch, so the choice is made before any network or worktree side effect. @@ -283,7 +284,7 @@ fi This block adds the worktree **at most once** per invocation, and reuses the one from a prior invocation rather than adding a second. The decision keys off -git's worktree *registry* — the source of truth — not just the directory's +git's worktree _registry_ — the source of truth — not just the directory's existence, because the two can disagree (a registered worktree whose directory was manually `rm -rf`'d, or a leftover directory git never registered). Keying off the directory alone would send a dangling-but-registered path into @@ -345,9 +346,9 @@ After this step every later step uses `${WT:-.}` for cwd (so BRANCH-LOCAL implicitly runs from the project root) and `$DIFF_RANGE` for diffs. There are no more per-mode forks until step 7e (which checks `[ -n "$WT" ]` for scratch verification) and step 8, where only **step 8b (the GitHub review -submission)** is skipped for BRANCH-* modes — no PR to submit a review to. +submission)** is skipped for BRANCH-\* modes — no PR to submit a review to. Step 8a still runs in every mode to compose the review artifact and, in -BRANCH-* modes, render the would-have-been verdict and findings into chat. +BRANCH-\* modes, render the would-have-been verdict and findings into chat. Stash the `HEAD_OID_EXPECTED` value — step 8 re-checks it immediately before submission and pins it into the review payload as `commit_id`. @@ -449,7 +450,7 @@ fi If the PR has passing CI checks, report them as PASS in the review. Only run CI locally if checks haven't run yet or if you need to verify a specific failure. Note any CI failures in the review but continue with the code review -regardless. (This governs CI *status reporting* only — the suggestion +regardless. (This governs CI _status reporting_ only — the suggestion scratch-verification in step 7e is independent and runs regardless of what GitHub's checks say.) @@ -647,7 +648,7 @@ numbers it covers; pick `line` (and `start_line` when multi-line) from inside the same hunk; don't span hunk boundaries. **Deleted lines and base-side context.** Findings about something the PR -*removed* don't live on the RIGHT side at all — there are no new-file lines +_removed_ don't live on the RIGHT side at all — there are no new-file lines to anchor to. Two options: - Anchor the inline comment on the LEFT side: `"side": "LEFT"` (and @@ -662,7 +663,7 @@ to anchor to. Two options: see step 1's "base-side reads" note.) Same rule for renamed files: the file's new path can carry a `suggestion` -block normally; comments about content the rename *also dropped* anchor on +block normally; comments about content the rename _also dropped_ anchor on the old path with `side: "LEFT"`. Use a `suggestion` block when: @@ -693,12 +694,12 @@ fenced code block when: cleanly revised. In all of those cases, give the proposed code in a plain fenced block (e.g. -```` ```rust ````) and end with a short "Apply manually — can't be auto-applied +` ```rust `) and end with a short "Apply manually — can't be auto-applied as a suggestion because …" sentence. ##### Fence length when the replacement itself contains backticks -The default `` ```suggestion `` fence is three backticks. If the replacement +The default ` ```suggestion ` fence is three backticks. If the replacement bytes contain a line that is itself a run of three-or-more backticks — common for Markdown/docs suggestions that include a nested code fence — that inner run closes the outer `suggestion` block early, and the rendered one-click @@ -706,14 +707,14 @@ suggestion is truncated or malformed rather than matching the bytes the user approved. Before displaying or submitting any suggestion: 1. Scan the replacement for the longest run of consecutive backticks, `N`. -2. If `N < 3`, use the normal three-backtick `` ```suggestion `` fence. +2. If `N < 3`, use the normal three-backtick ` ```suggestion ` fence. 3. If `N >= 3`, open and close the block with a fence of `N + 1` backticks - (e.g. ` ````suggestion ` for an inner ```` ``` ````), so the outer fence is + (e.g. ` ````suggestion ` for an inner ` ``` `), so the outer fence is strictly longer than any inner run — GitHub follows the CommonMark rule that a fence closes only on a run of **at least** as many backticks. This rule is deterministic; whether GitHub renders the widened fence as a one-click suggestion is a server-side property that cannot be checked locally (7e's - scratch pass verifies replacement *bytes*, not rendering). When in doubt — + scratch pass verifies replacement _bytes_, not rendering). When in doubt — e.g. an unusually exotic replacement — **demote the finding to prose-only** (a plain fenced block plus the "Apply manually …" sentence) rather than risk posting a malformed suggestion. @@ -753,7 +754,7 @@ For a multi-line suggestion, add `start_line` and `start_side`: **Indentation matters**: the block replaces the original lines verbatim, so leading whitespace must match exactly what the file expects after the fix. -**Fence length matters too**: the `` ```suggestion `` fences above use three +**Fence length matters too**: the ` ```suggestion ` fences above use three backticks, which only holds when the replacement contains no three-or-more backtick run of its own. When it does (e.g. a docs suggestion with a nested code fence), widen the outer fence per step 7a's fence-length rule or demote @@ -776,7 +777,7 @@ the comment body and tell the author it has to be applied manually: #### 7c-bis. Inline comment on a removed (LEFT-side) line -A finding about a line the PR *removed* has no RIGHT-side anchor — pin it +A finding about a line the PR _removed_ has no RIGHT-side anchor — pin it on the LEFT (base) side instead. `suggestion` blocks aren't applicable (GitHub only commits suggestions from the RIGHT side), so the body uses a plain code block: @@ -805,8 +806,8 @@ can't straddle sides. - The total number of inline comments has a soft cap of ~30. If you would exceed that, consolidate the lowest-severity findings into the review body with file/line references but no inline comment. -- A given inline comment may contain at most one ```` ```suggestion ```` block. - Prose context blocks (e.g. ```` ```rust ````) are fine alongside it. +- A given inline comment may contain at most one ` ```suggestion ` block. + Prose context blocks (e.g. ` ```rust `) are fine alongside it. - If the user changed an emoji tag during triage, the comment uses the new tag. - Don't post suggestions on lines outside the RIGHT side of the diff — they'll fail GitHub's "position could not be resolved" check. Comments about @@ -838,7 +839,7 @@ suggestion A only compiles because suggestion B was also applied, the agent has labelled A as verified but A-alone can break the build. So the inner loop tests each suggestion against a clean worktree first; a final batch pass (all approved suggestions applied together) is a nice-to-have to -catch *interactions*, but the per-suggestion runs are the real gate: +catch _interactions_, but the per-suggestion runs are the real gate: ```bash # Confirm clean starting state at $WT (HEAD = $HEAD_REF, status empty). @@ -931,8 +932,9 @@ gate when **any** of these is true: - The suggestion touches a `#[cfg(test)]` module, a test, or a feature gate. - The finding is 🔧 wrench (blocking) — release-blocking fixes must clear the release gate. -- The touched code is shared (`crates/trusted-server-core/src/{auction,ec, - http_util,publisher,html_processor,settings,constants}` and similar). +- The touched code is shared + (`crates/trusted-server-core/src/{auction,ec,http_util,publisher,html_processor,settings,constants}` + and similar). - The suggestion changes program behaviour and the agent prefers to ship it **without** the compile-verified-only disclaimer. @@ -967,11 +969,11 @@ at build time. Run the build whenever the suggestion touches files under **Post-verify drift check (snapshot the approved patch, hard-fail on any deviation).** Filename-level comparison isn't enough — a formatter or -codegen step can change a different range *inside* an approved file, while +codegen step can change a different range _inside_ an approved file, while the posted GitHub suggestion still contains only the originally-approved range. The correct check is byte-exact: snapshot the full patch immediately after applying the approved suggestions but **before** running any -verification command, then compare with the patch *after* verification. Any +verification command, then compare with the patch _after_ verification. Any delta — different range in the same file, an extra tracked file, a whitespace change in `Cargo.toml` from a build script — means verification mutated the tree beyond what the agent approved, and the suggestion as @@ -1060,7 +1062,7 @@ After cleanup, the worktree's HEAD must be at `$HEAD_REF` Step 8 splits into two halves. **8a** is mode-agnostic: determine the verdict and compose the review body + inline comments. **8b** is PR-only: post the -review to GitHub. BRANCH-* modes still produce the artifact in 8a (so step 10 +review to GitHub. BRANCH-\* modes still produce the artifact in 8a (so step 10 has a "would-have-been verdict" to report) but skip 8b — the artifact is rendered into the chat instead. @@ -1164,14 +1166,14 @@ inline comment", not "this finding is out of scope". Omit any section that has no findings — don't include empty headings. -In BRANCH-* modes (`[ -z "$NUMBER" ]`) the artifact is now complete: render it +In BRANCH-\* modes (`[ -z "$NUMBER" ]`) the artifact is now complete: render it into the chat exactly as the GitHub UI would have shown it (body markdown followed by each inline comment, labelled with the file:line it would have anchored to). Stop after rendering — there is no review to submit. #### 8b. Submit the GitHub review (PR mode only — `[ -n "$NUMBER" ]`) -Skip this entire sub-step in BRANCH-* modes. +Skip this entire sub-step in BRANCH-\* modes. ##### Re-check the PR head before submission @@ -1197,7 +1199,7 @@ fi If `SKIP_SUBMISSION` is set, the agent **must skip every command in the "Submit the review" sub-section below**, render the artifact in chat (the -same way BRANCH-* mode would in 8a), and report `submission skipped: +same way BRANCH-\* mode would in 8a), and report `submission skipped: $SKIP_REASON` as the stop reason in step 10. The agent does not restart inside the same invocation — single-invocation = single pass; the user re-invokes if they want another pass against the new head. @@ -1218,7 +1220,7 @@ fi ``` When `STOP_SUBMISSION=1`, render the composed artifact into chat exactly the -way BRANCH-* mode does in 8a, report `submission skipped: $SKIP_REASON` per +way BRANCH-\* mode does in 8a, report `submission skipped: $SKIP_REASON` per step 10, and do not run any of the `gh api` submit/delete commands below. Use the GitHub API to submit. Handle these known issues: @@ -1307,7 +1309,7 @@ Use the GitHub API to submit. Handle these known issues: fallthrough, no `exit 1`. **Re-gate after the pending-review check.** The `SKIP_SUBMISSION` guard - at the top of "Submit the review" runs *before* the pending-review check, + at the top of "Submit the review" runs _before_ the pending-review check, so a `keep` decision must be caught by a second gate immediately after the case block — otherwise the JSON-assembly / `gh api … -X POST` below would still run: @@ -1349,7 +1351,7 @@ head-recheck above. **Invariant — never heredoc-interpolate user content into the payload.** Inline-comment bodies routinely contain `$`, backticks, backslashes, -quoted code, and the literal `` ```suggestion `` fence. A `cat <` (binary name `ts`), or install it with - `cargo install --path crates/trusted-server-cli`. +- The `ts` CLI runs from source without installing: + `cargo run -p trusted-server-cli -- config ` (binary name `ts`), or + install it with `cargo install --path crates/trusted-server-cli`. - The app-config store's logical id and blob key are `trusted_server_config` (`settings_data.rs` `DEFAULT_CONFIG_STORE_ID`, `config_payload.rs` `CONFIG_BLOB_KEY`); older builds used `app_config`. The override env-var key diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index c844e95d3..93bc99229 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -149,9 +149,11 @@ jobs: - name: Run Prettier (check) run: npm run format - - name: Run Prettier (check) — root Markdown + - name: Run Prettier (check) — Markdown outside docs/ working-directory: . - run: docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" + run: >- + docs/node_modules/.bin/prettier --config docs/.prettierrc --check + "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md" - name: Build with VitePress (fails on dead links) run: npm run build diff --git a/AGENTS.md b/AGENTS.md index 9641e79fc..738fcab34 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -351,7 +351,7 @@ Every PR must pass: 5. JS build and test (`cd crates/trusted-server-js/lib && npx vitest run`) 6. JS format (`cd crates/trusted-server-js/lib && npm run format`) 7. Docs format (`cd docs && npm run format`) -8. Root Markdown format (requires `cd docs && npm ci` first): `docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md"`; fix with `--write` in place of `--check` +8. Markdown format outside `docs/` (requires `cd docs && npm ci` first): `docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md"`; fix with `--write` in place of `--check` --- From d7dcb183a70e4d51379d0ff942e9785cbc16cbf4 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 14:13:08 +0530 Subject: [PATCH 101/104] Test that both partner token placeholders are reported together Let the partner test helper take an api_token, and assert that one partner with placeholder api_token and ts_pull_token values reports both fields in a single error. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/settings.rs | 31 +++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 47f614387..5eb2e070b 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -5418,12 +5418,16 @@ source_domain = "partner.example.com" } fn test_partner_with_pull_token(ts_pull_token: &str) -> EcPartner { + test_partner_with_tokens(None, ts_pull_token) + } + + fn test_partner_with_tokens(api_token: Option<&str>, ts_pull_token: &str) -> EcPartner { EcPartner { name: "Test Partner".to_owned(), source_domain: "partner.example.com".to_owned(), openrtb_atype: EcPartner::default_openrtb_atype(), bidstream_enabled: false, - api_token: None, + api_token: api_token.map(|token| Redacted::new(token.to_owned())), batch_rate_limit: EcPartner::default_batch_rate_limit(), pull_sync_enabled: true, pull_sync_url: Some("https://partner.example.com/sync".to_owned()), @@ -5468,6 +5472,31 @@ source_domain = "partner.example.com" .expect("should accept a realistic partner pull token"); } + #[test] + fn reject_placeholder_secrets_reports_both_partner_tokens() { + let mut settings = + Settings::from_toml(&crate_test_settings_str()).expect("should parse test settings"); + settings.publisher.proxy_secret = Redacted::new("unit-test-proxy-secret".to_owned()); + settings.ec.passphrase = Redacted::new("test-secret-key-32-bytes-minimum".to_owned()); + settings.ec.partners = vec![test_partner_with_tokens( + Some("partner-api-token-32-bytes-minimum"), + "replace-with-partner-api-token-32-bytes-minimum", + )]; + + let err = settings + .reject_placeholder_secrets() + .expect_err("should reject placeholder partner tokens"); + let message = format!("{err:?}"); + assert!( + message.contains("ec.partners[partner.example.com].api_token"), + "error should mention the partner API token field" + ); + assert!( + message.contains("ec.partners[partner.example.com].ts_pull_token"), + "error should also mention the partner pull token field on the same partner" + ); + } + #[test] fn is_unusable_store_id_rejects_placeholders_empty_and_padded_values() { for placeholder in RequestSigning::STORE_ID_PLACEHOLDERS { From 4552fcfa621231789bb0830518091995ada52ace Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 28 Sep 2026 17:51:32 -0500 Subject: [PATCH 102/104] Fix Sourcepoint HEAD pass-through test --- .../trusted-server-core/src/integrations/sourcepoint.rs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/sourcepoint.rs b/crates/trusted-server-core/src/integrations/sourcepoint.rs index 1bdc6aa6f..d22cc4ffe 100644 --- a/crates/trusted-server-core/src/integrations/sourcepoint.rs +++ b/crates/trusted-server-core/src/integrations/sourcepoint.rs @@ -1400,6 +1400,11 @@ mod tests { ], ); let services = build_services_with_http_client(client.clone()); + let expected_body: &[u8] = if method == Method::HEAD { + b"" + } else { + b"unchanged" + }; let mut request = make_req( method, "https://publisher.example.com/integrations/sourcepoint/cdn/asset", @@ -1437,8 +1442,8 @@ mod tests { .await .expect("should collect pass-through body") .as_ref(), - b"unchanged", - "should preserve pass-through bytes" + expected_body, + "should preserve pass-through bytes or omit them for HEAD" ); assert!( client.stub.recorded_request_headers()[0] From 4811d98edd316294f7e87cfd909119cf4434c87f Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Thu, 1 Oct 2026 09:47:25 +0530 Subject: [PATCH 103/104] Clarify trace failure and export cleanup contracts --- ...9-01-mobile-ad-render-trace-endpoint-design.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index e12d79760..bc6387914 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -252,6 +252,15 @@ second GPT attribution engine. The existing recorder emits an exact-token cycle. The server-auction, correlation, and GPT projections are separate sibling contracts in `TraceReportV1`; none is treated as a substitute for another. +The deferred object-URL cleanup in section 12.3 deliberately diverges from TS +Console's existing export, which revokes synchronously in a `finally` block +immediately after `anchor.click()` +(`crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts`, +reachable through the console's `export()` action). That older path keeps the +truncation risk this design avoids. Version one does not change it; correcting +it belongs to TS Console under #1081, and until then the two download paths +intentionally differ. + ## 6. User experience ### 6.1 First visit: no captured report @@ -1513,6 +1522,12 @@ shell or actions. before trace handling, including on disabled routes. - Disabled route after authentication: local privacy-safe `404`. - Unsupported method: local `405`; never publisher fallback. +- Reserved-namespace path that is a trailing slash, extra segment, unsupported + asset name, repeated separator, or lookalike: local `404`; an encoded + separator or ambiguous dot segment: local `400`. Never publisher fallback. +- Non-empty or unreadable activation/end body: local `413` for any body bytes, + positive/invalid `Content-Length`, or `Transfer-Encoding`, and local `400` + for a stream read error, both with no cookie mutation. - Rejected activation/end POST: local `403` with no state mutation. - Optional platform fact unavailable: omit the field and continue. - Bounded cookie inspection failure: report the contract-defined invalid or From 217dd4342ebfbac7caea5b415c8390bd6091270e Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 1 Oct 2026 12:12:20 -0500 Subject: [PATCH 104/104] Allow bounded cold browser launches in audits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI timed out before the first browser fixture reached any page assertions. Set the shared browser startup budget to 60 seconds instead of chromiumoxide’s 20-second default. Navigation, CDP, settling, teardown bounds and all fixture assertions remain unchanged. The existing GPT fixture fails with a 22-second Chrome startup delay before this change and passes afterward. Full CLI, adapter, lint, parity, JS and formatting gates pass locally. --- crates/trusted-server-cli/src/commands/audit/browser.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-cli/src/commands/audit/browser.rs b/crates/trusted-server-cli/src/commands/audit/browser.rs index fd79155f0..50bda2c5f 100644 --- a/crates/trusted-server-cli/src/commands/audit/browser.rs +++ b/crates/trusted-server-cli/src/commands/audit/browser.rs @@ -48,6 +48,8 @@ const NAVIGATION_TIMEOUT: Duration = Duration::from_secs(30); const MAX_EVIDENCE_ENTRIES: usize = 128; /// Hard cap on the UTF-8 JSON payload before CDP transfers it back to Rust. const MAX_EVIDENCE_PAYLOAD_BYTES: usize = 1024 * 1024; +/// Hard cap on browser startup, allowing cold launches on shared CI runners. +const BROWSER_LAUNCH_TIMEOUT: Duration = Duration::from_secs(60); /// Hard cap on browser teardown so a wedged Chrome cannot hang the audit. const BROWSER_CLOSE_TIMEOUT: Duration = Duration::from_secs(5); @@ -151,7 +153,8 @@ pub(crate) fn build_browser_config( ) -> Result { let mut builder = BrowserConfig::builder() .chrome_executable(options.chrome) - .user_data_dir(options.profile_dir); + .user_data_dir(options.profile_dir) + .launch_timeout(BROWSER_LAUNCH_TIMEOUT); if !options.accept_invalid_certs { builder = builder.respect_https_errors(); }