From d9d4925469c668965bcb6276df945efff200b020 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 14 Sep 2026 17:06:45 -0500 Subject: [PATCH 01/14] Add AWS planning skill for Prebid Server Go Gather deployment requirements before choosing AWS services and generating Terraform or runtime files. Keep cloud execution behind separate approval and document safe testing, state ownership, and credential handling. Refs #1163 --- .claude/skills/planning-prebid-aws/SKILL.md | 78 +++++++++++++++++++ .../examples/two-region-pilot.md | 58 ++++++++++++++ .../references/architecture.md | 73 +++++++++++++++++ .../references/file-generation.md | 77 ++++++++++++++++++ .../references/prebid-go.md | 54 +++++++++++++ .../references/terraform.md | 74 ++++++++++++++++++ 6 files changed, 414 insertions(+) create mode 100644 .claude/skills/planning-prebid-aws/SKILL.md create mode 100644 .claude/skills/planning-prebid-aws/examples/two-region-pilot.md create mode 100644 .claude/skills/planning-prebid-aws/references/architecture.md create mode 100644 .claude/skills/planning-prebid-aws/references/file-generation.md create mode 100644 .claude/skills/planning-prebid-aws/references/prebid-go.md create mode 100644 .claude/skills/planning-prebid-aws/references/terraform.md diff --git a/.claude/skills/planning-prebid-aws/SKILL.md b/.claude/skills/planning-prebid-aws/SKILL.md new file mode 100644 index 000000000..6b4c12afd --- /dev/null +++ b/.claude/skills/planning-prebid-aws/SKILL.md @@ -0,0 +1,78 @@ +--- +name: planning-prebid-aws +disable-model-invocation: true +description: Plan Prebid Server Go deployments on AWS through an interactive requirements interview. Use when choosing deployment architecture or AWS services, or generating Terraform, runtime configuration, deployment scripts, and runbooks for a new or revised PBS deployment. Stops before cloud changes and live traffic. +--- + +# Planning Prebid Server Go on AWS + +Produce an approved deployment design and locally checked implementation files. Prebid Server Go is fixed; ask for its release and image source, not its implementation language. + +## Authority + +This workflow permits planning and, after design approval, file generation and safe local checks. Cloud provisioning, credential writes, application deployment, production traffic changes, and teardown require a separate execution workflow with explicit authorization. Generated scripts and CI jobs must remain inactive. Ask before authenticated AWS inspection or state access, naming the account, role, regions, and read scope. Keep real credentials out of the conversation and generated files. + +## 1. Establish the baseline + +Inspect repository instructions, Git status, existing infrastructure, runtime files, and supplied documents. Preserve unrelated changes. Identify the target repository and deployment directory before proposing edits. + +Record requirements in one decision record, initially in the conversation, then in the approved deployment plan: + +| Requirement | Value | Status | Evidence or decision owner | Blocks | +| -------------------- | ----------------------- | ---------------------------------- | -------------------------- | ----------------------------------------- | +| One row per decision | Answer or open question | Confirmed, proposed, or unresolved | Source or person | Design, generation, live rollout, or none | + +Treat supplied examples as evidence of intent, not accepted requirements for this deployment. Reconcile conflicting inputs with the user. Inspect existing answers before asking again. + +Done when the current deployment, requested outcome, reusable resources, and unanswered decisions are identified. + +## 2. Interview by decision + +Ask two to four related questions per turn. Start with questions that change the architecture; explain the consequence of unfamiliar choices. Offer a recommendation the user can accept rather than requiring AWS expertise. + +Cover these topics, skipping already confirmed answers: + +- Purpose: disposable test, live pilot, or ongoing production; retained provider and migration scope. +- Availability: acceptable interruption, host/AZ/region failure tolerance, maintenance windows, recovery time, and loss tolerance for required data. +- Workload: absolute peak eligible auction QPS, allocation, regional mix, bidder fan-out, payload sizes, burst duration, and caller latency budget. +- Operations: fixed headroom versus automatic scaling, budget ceiling, owner and backup, existing CI and AWS platform, regions, DNS, network restrictions, and bidder IP allowlists. +- Security: permitted callers/publishers, public or private access, data residency, retention, and the privacy policy owner. +- Auction behavior: caller, bidder set, formats, consent and identity, account settings, stored requests, and cache dependencies. Read [Prebid Go requirements](references/prebid-go.md) before resolving these inputs. + +Translate "production ready" into measurable availability, security, capacity, and recovery requirements. Scaling and availability are separate decisions. Offer a measurement plan for unknown traffic rather than inventing capacity. + +Done when every topic is confirmed, explicitly inapplicable, or recorded as an unresolved blocker. Continue a provisional design around unknowns, but pause affected file generation until architecture-changing decisions are approved. + +## 3. Recommend and obtain approval + +Read [architecture decisions](references/architecture.md). For Terraform state/authentication decisions and HCL generation, read [Terraform guidance](references/terraform.md). For a two-region standalone-host pilot, consult [the worked example](examples/two-region-pilot.md); its values remain conditional. + +Present one recommended design and only alternatives that resolve a real tradeoff. Include: + +- A top-to-bottom Mermaid diagram and the failure boundaries. +- Each selected AWS service, its requirement, and whether to reuse or create it. +- Capacity assumptions, cost drivers and estimate date, accepted limitations, and blockers. +- Runtime, infrastructure, secret, and traffic-control ownership. +- Target files and checks, with cloud-dependent checks separated from local checks. + +Ask the user to approve the architecture, assumptions, target files, and accepted limitations. Approval to generate files is not approval to execute them. Reopen approval if later findings change topology, cost commitments, or ownership. + +Done when the user explicitly approves the design and file scope. If blockers remain, agree on a bounded draft and label it incomplete rather than producing apparently deployable infrastructure. + +## 4. Generate the approved files + +Read [file generation and validation](references/file-generation.md). Follow existing repository conventions and generate only artifacts used by the selected design. Keep the decision record in the deployment plan; reference it from the runbook rather than repeating it. + +Verify version-specific PBS fields and adapter bindings against the selected release. Verify AWS/Terraform behavior and pricing against current primary documentation. Record source links, versions, and verification dates in the deployment plan. Unavailable evidence remains a named blocker; do not invent image digests, configuration keys, prices, or benchmark results. + +Done when every approved artifact exists, has a named owner and check, and every unresolved input is visible and prevents unsafe use where applicable. + +## 5. Validate and hand off + +Run the applicable safe checks in the loaded generation references and inspect the final diff. Report changed paths, exact commands, results, checks not run, and remaining blockers. Separate these states: + +- Draft: unresolved generation inputs or incomplete artifacts. +- Locally checked: applicable local checks passed; cloud and integration behavior remain unverified. +- Ready for deployment review: file scope complete, blockers for generation cleared, local evidence recorded, and cloud plan/live checks listed for the authorized operator. + +This skill never establishes production readiness from generated files alone. End with the next approval or evidence needed, not a provisioning command executed on the user's behalf. diff --git a/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md new file mode 100644 index 000000000..e072d1af7 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md @@ -0,0 +1,58 @@ +# Worked example: two-region pilot + +This is a conditional design derived from a supplied pilot plan, not a default architecture or evidence of provisioned capacity. Use it only when the user accepts its limitations. Operational rules live in the skill references. + +## Example requirements + +Assume the user has confirmed: + +- Prebid Server Go on AWS with Terraform. +- One deployment in `us-east-1` and one in `us-west-2`. +- A retained provider and a caller-controlled trial targeting 5% of eligible traffic. +- A fixed approved bidder set, versioned configuration, and credential rotation. +- Host maintenance interruptions, manual host replacement, and no autoscaling for this bounded pilot. + +Still resolve absolute peak traffic, regional distribution, bidder fan-out, caller type, domain ownership, inventory/cache dependencies, secret permissions, budget, recovery expectations, and numeric rollout gates. The 5% value supplies no instance-size evidence. + +## Conditional recommendation + +```mermaid +flowchart TD + A["Existing caller"] --> S["Caller experiment allocation"] + S -->|"95% of eligible traffic"| P["Existing provider"] + S -->|"5% of eligible traffic"| R["Selected regional routing"] + R --> E["East EC2: Caddy and PBS Go"] + R --> W["West EC2: Caddy and PBS Go"] + E --> B["Approved bidder endpoints"] + W --> B +``` + +Per region, propose a public subnet, internet gateway, EC2 host with encrypted storage and Elastic IP, Compose, Caddy, SSM access, regional secret access, release artifacts, and telemetry. Select actual instance sizes and images only after workload and compatibility evidence. + +Prefer existing caller-side regional routing when suitable. Otherwise evaluate shared-hostname Route 53 latency routing with independent regional health checks and a verified DNS-challenge certificate setup. Implement only the selected branch. + +Use one pilot Terraform root with two provider aliases and explicit regional module calls, unless ownership requires separate roots. Keep backend bootstrap independent. Generate Compose/Caddy and host deployment tools only after approving this profile. + +No ALB, NAT gateway, ECS, distributed database, or cache is justified solely by the pilot's two regions. Add or replace components when a confirmed requirement demands it. Exclude inventory with unmet dependencies explicitly rather than presenting a reduced-function trial as equivalent. + +## Approval and evidence gates + +The user must accept one-host-per-region outages and maintenance behavior. If regional failover concentrates the pilot on one node, prove that capacity or agree on reducing allocation. Automatic replacement, uninterrupted deployment, or tighter recovery requirements invalidate this profile and require another architecture decision. + +File generation starts after architecture-changing questions and target paths are approved. Live-traffic gates remain deferred to authorized execution: zero-allocation deployment checks, internal inventory validation, then agreed 1% and 5% observation windows. A tested caller kill switch and measured refresh bound are required before the live pilot. + +## Walkthrough assertions + +Use these contrasts when reviewing the skill. They are expected behavior, not executed deployment tests. Terraform cases exercise the rules in `references/terraform.md`. + +| Input | Expected planning behavior | +| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| This pilot, with outage acceptance but unknown QPS | Preserve the supplied answers; ask about workload and other blockers; propose this profile provisionally without inventing instance capacity | +| Production requires AZ survival, uninterrupted releases, and automatic scaling | Reopen topology; evaluate multi-AZ ECS/ALB or EC2 Auto Scaling/ALB against the existing platform; define minimum/failover capacity and scaling evidence; generate only the selected runtime's files | +| Pilot adds video with independent regional caches | Resolve cache write/retrieval and failure behavior before generating affected infrastructure, or obtain approval to exclude that inventory | +| User approves file generation but has supplied no AWS execution authorization | Generate inactive tools and run safe local checks; leave cloud actions and traffic changes unexecuted | +| A requirement is unknown or a validation tool is unavailable | Record a blocker or not-run check; preserve the distinction between a draft and locally checked files | +| A proposed local test omits `command` and uses a real AWS provider | Do not run it: the default is apply. Generate an explicitly commanded, isolated mocked test or defer it as an authorized cloud integration test | +| A plan-mode suite mocks East but retains a real West provider alias | Reject the credential-free claim; inspect all provider mappings and setup dependencies, mock the remaining external provider, and select exact reviewed test files | +| The existing S3 backend uses DynamoDB locking | Preserve it while proposing a version/permissions/recovery-aware migration; no backend replacement, lock removal, or state migration during generation | +| An upstream CI example uses stored AWS keys and applies after merge | Generate the approved short-lived identity and protected saved-plan review procedure as inactive tooling; upstream examples grant no execution authority | diff --git a/.claude/skills/planning-prebid-aws/references/architecture.md b/.claude/skills/planning-prebid-aws/references/architecture.md new file mode 100644 index 000000000..d16aa9878 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/architecture.md @@ -0,0 +1,73 @@ +# Architecture decisions + +Select the smallest design that meets the approved requirements. Reuse an established AWS platform when it meets them; avoid introducing a second operations model just for PBS. + +## Runtime and availability + +| Requirement | Candidate | Decision to resolve | +| ------------------------------------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | +| Bounded pilot accepting host outages and maintenance interruptions | One EC2 host per selected region, Compose, Caddy, SSM | Explicit outage acceptance, replacement procedure, and measured fixed headroom | +| Automatic host replacement with an existing EC2 operating model | Launch template, Auto Scaling group, regional ALB | Multi-AZ placement, minimum capacity, health checks, draining, and instance refresh policy | +| Managed container lifecycle and rolling deployments | ECS service with Fargate or EC2 capacity, regional ALB | Existing platform, resource fit, sustained cost, capacity provider, and rollout headroom | +| Variable demand requiring elastic capacity | Scaling policy on the chosen runtime | Minimum/maximum capacity, signal, target, startup time, cooldown, quotas, and behavior at the cap | + +A region containing one host is still a regional single point of failure. Host replacement, multi-AZ availability, regional failover, and zero-downtime releases each need their own evidence. ECS alone establishes none of these without the corresponding service configuration and capacity. + +Use EKS only when the user's existing platform or an explicit requirement justifies Kubernetes ownership. Do not infer a need for orchestration from the word "production" alone. + +## Network, ingress, and egress + +| Need | Candidate services or mechanism | Required evidence | +| --------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| Public HTTPS on standalone hosts | Public subnet, internet gateway, Elastic IP, Caddy | Accepted exposure, durable certificate storage, renewal and replacement path | +| Managed HTTPS with multiple backends | ALB and ACM | Certificate ownership/validation, target health, draining, timeout budget, private management endpoints | +| Private compute reaching public bidders | NAT-based egress or an approved existing egress service | AZ failure behavior, routing, hourly/data charges, and outbound address stability | +| Bidder source-IP allowlists | Stable egress IPs | Every normal, scaling, and failover path uses partner-approved addresses | +| Existing caller selects regions | Regional hostnames | Caller routing and failure policy, TLS, identity behavior | +| DNS-based regional selection | Route 53 latency records with per-region health checks | Independent health targets, cached-answer behavior, all-unhealthy behavior, and failover capacity | +| Public auctions from untrusted callers | Existing controls or WAF with compatible ingress | Auction payload compatibility, rate limits, false-positive testing, cost, and owner | + +Private subnets do not provide internet egress by themselves. VPC endpoints may serve supported AWS APIs but do not replace bidder internet access. Public addressing on replaceable compute does not by itself provide stable egress for allowlists. + +For a shared hostname on standalone Caddy hosts, specify a workable multi-host certificate issuance method. Route 53 DNS challenges require the matching Caddy module, a pinned custom image, and scoped DNS permissions. Choose one routing/TLS design rather than generating all alternatives. + +Define trusted proxy hops and client-IP handling, ingress ports, metrics isolation, outbound needs, and management access. Specify publisher/account admission and request limits for public auctions; CORS alone is not authorization. For EC2, use SSM without inbound SSH, require IMDSv2, encrypt disks, and separate host AWS access from application access. For ECS, distinguish the execution role from the application task role. + +## Supporting services + +For each item, record reuse, create, or not needed, with the reason: + +- State storage and locking, independently owned from the deployment it records. +- Secrets Manager or an approved existing secret service, with regional access, rotation, and least-privilege permissions. +- Artifact storage, typically versioned S3 releases and an approved image registry such as ECR. Include checksums, retention, and regional startup independence where required. +- Logs, metrics, alarms, notification delivery, and budget alerts through CloudWatch or existing systems. Resolve metrics collection and dashboards, not just a log group. +- Stored-request/account storage and Prebid Cache only where the selected auction workload needs them. A generic Redis or database deployment is not automatically a PBS integration. + +Alert on caller failures and latency, bidder behavior, capacity, resource exhaustion, and cost. Budget alerts are notifications, not a hard spending cap. Define data retention and redaction before recording auction diagnostics. + +## Capacity and cost + +Calculate from measured workload: + +```text +pilot_peak_qps = eligible_peak_qps * allocation_fraction +regional_peak_qps = pilot_peak_qps * measured_regional_share +outbound_qps = pilot_peak_qps * measured_bidder_calls_per_auction +``` + +Use representative payloads and controlled bidder responses to establish throughput, tail latency, resource limits, and slow-bidder behavior. Include deployment overlap and the approved failure scenario in required headroom. If failover concentrates traffic, either prove surviving capacity or define how allocation is reduced. + +Scaling needs a load-tested signal that predicts auction saturation, plus minimum capacity for bursts during startup. Consider CPU, memory, concurrency, connection limits, and outbound bandwidth. Never equate an instance count or CPU percentage with a measured QPS guarantee. + +Estimate compute, storage, public IPv4, load balancing, NAT processing, internet/inter-AZ/inter-region transfer, DNS/health checks, secret retrieval, artifacts, and telemetry. State region, prices checked on, traffic assumptions, and uncertainty. Unknown volume means a conditional estimate, not a quoted total. + +## Sources to verify for the selected branch + +Consult current AWS documentation when recommending concrete settings: + +- [ECS service autoscaling](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html) +- [EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html) +- [NAT gateway basics](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html) +- [Route 53 latency routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html) +- [Caddy Route 53 module](https://github.com/caddy-dns/route53) +- [AWS Pricing Calculator](https://calculator.aws/) diff --git a/.claude/skills/planning-prebid-aws/references/file-generation.md b/.claude/skills/planning-prebid-aws/references/file-generation.md new file mode 100644 index 000000000..89e365848 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/file-generation.md @@ -0,0 +1,77 @@ +# File generation and validation + +Generate files only after the design and target paths are approved. Use the repository's existing layout; the paths below are suggestions for a new deployment, not a template to copy wholesale. The authority boundary in `SKILL.md` applies to every check and generated tool. + +## Artifact contract + +| Artifact | Suggested location | Contents and owner | +| -------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Deployment plan | `docs/pbs-deployment-plan.md` | Requirements record, approved architecture, sources, costs, blockers, file/check inventory; user approves | +| State bootstrap, if needed | `infra/bootstrap/` | Independently managed backend, access and locking setup; infrastructure owner | +| Deployment root | `infra//` | Providers, variables/validation, explicit regional modules, outputs, selected DNS configuration, nonsecret examples, provider lock file; Terraform owns AWS resources | +| Reusable infrastructure | `infra/modules/` | Modules justified by repeated topology, not one module per AWS service | +| Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | +| Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | +| Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | +| Operational tools | `scripts/` or existing CI layout | Release preparation, deployment, secret loading where needed, smoke checks; deployment owner | +| Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | + +Each generated artifact must have a consumer. Add ignore rules for local credentials, runtime secret files, `.terraform`, state files, saved plans, and generated sensitive output. Track nonsecret examples. Use `example.com` hostnames and visibly fictional identifiers in examples. + +For Terraform artifacts, follow [Terraform generation and review](terraform.md), including module tests under `tests/` and the saved-plan procedure in the runbook. + +## Runtime and secrets + +Pin PBS and ancillary images by verified digest with CPU compatibility. Render structured regional changes deterministically. Include release identity, checksums, required secret identifiers/keys, and dependencies in the manifest, never secret values. + +Choose one secret injection path for the approved runtime: + +- EC2/Compose: a host-side role retrieves secrets, validates supported bindings, and atomically writes restricted ephemeral runtime files. Rebuild them at boot. Encode through a parser with tested Compose behavior for quotes, dollars, newlines, and empty values; never use shell evaluation. Keep instance credentials away from application containers. +- ECS: use verified task-definition secret references or an explicitly justified retrieval mechanism, with appropriate execution/task-role permissions. Identify which changes require task replacement. + +A secret update does not automatically refresh process environment. Specify how replacement containers consume new values and how regional replicas become ready. Coordinate partner-side rotation and overlap; application rollback cannot restore a revoked credential. Treat debug output and container inspection as privileged. + +## Generated deployment tools + +Generated scripts must require an explicit environment, target, and release, with identity/preflight checks and no production defaults. New CI deployment jobs must be manual and approval-gated. File generation must not trigger existing auto-apply or deployment jobs; inspect those triggers before editing their watched paths. + +For the selected runtime, implement or explicitly defer each release stage: + +1. Serialize competing deployments to the same target. +2. Retrieve and verify an immutable release, image availability, and required secrets before replacing working capacity. +3. Record previous release and secret version identifiers without values. +4. Deploy with the approved outage/draining policy and bounded health deadlines. +5. Check HTTPS, application health, and an agreed auction fixture. +6. Restore the previous release on failure only when dependencies and credentials remain compatible; otherwise stop and report the recovery action. +7. Record target, release, timestamps, and outcome. + +Define retry limits, interrupted-run handling, and behavior when the same release is requested twice. On standalone hosts, provide boot recovery, deployment locking, bounded logs, certificate persistence, and an explicit response to hung-but-running containers. A Compose unhealthy status alone is not a restart policy. On managed runtimes, express equivalent rollout/rollback controls through the platform rather than adding host scripts. + +## Runbook and deferred evidence + +Reference the approved decision record. Include operator prerequisites and explicit account/region targeting, initial deployment, configuration updates, secret rotation, failed-release recovery, host/task replacement, alert response, and teardown with retention rules. Clearly mark these as procedures not yet executed. + +For a pilot, include the caller kill switch, its refresh deadline, staged allocation gates, and verification that traffic drained before teardown. DNS changes and host shutdown are not substitutes for caller rollback. For ongoing production, use the approved traffic and recovery policy rather than inventing an old-provider fallback. + +List deferred checks with expected results and an owner: authenticated Terraform plan review, TLS/DNS, IAM and exposed ports, regional dependency availability, missing secrets, invalid configuration, reboot/replacement, interrupted deployment, credential rotation, representative load, failover, alert delivery, and applicable caller/auction checks. Every approved availability and scaling claim needs a matching test. + +## Safe local checks + +Run applicable checks on the generated paths, recording exact commands and results. Inspect project scripts before executing them. Use isolated dummy credentials, fake bidder endpoints, and fixtures; local startup must not contact production bidders or fetch real AWS secrets. + +| Area | Check | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Terraform | Follow [Terraform local checks](terraform.md#tests-and-safe-local-checks), including inspected, explicitly selected mocked tests | +| Runtime structure | Parse YAML/JSON, verify manifest paths/checksums, and render deterministic overrides twice | +| Compose branch | `docker compose --env-file -f config --quiet` and dummy-value round-trip checks | +| Scripts | Language syntax/lint and focused tests for missing/malformed inputs, special-character dummy secrets, failed release, interruption, and repeated invocation | +| PBS behavior | Approved isolated container startup and smoke fixture against controlled bidder responses, if a suitable local runtime is available | +| Final files | Diff review for scope, secret exposure, unresolved placeholders, inactive deployment triggers, and one writer per mutable resource | + +If a tool, image, network permission, or fixture is unavailable, mark that check not run with the next action. Never substitute a checklist for executed evidence or call a no-bid response proof of bidder success. + +## Sources to verify during generation + +- [ECS Secrets Manager injection](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html) +- [Secrets Manager regional replication](https://docs.aws.amazon.com/secretsmanager/latest/userguide/replicate-secrets.html) +- [SSM Run Command](https://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html) diff --git a/.claude/skills/planning-prebid-aws/references/prebid-go.md b/.claude/skills/planning-prebid-aws/references/prebid-go.md new file mode 100644 index 000000000..00c8212c1 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/prebid-go.md @@ -0,0 +1,54 @@ +# Prebid Go requirements + +Resolve these requirements before claiming the infrastructure can serve the selected inventory. Go is fixed; select an approved release, image source, digest, and compatible CPU architecture. + +## Caller and bidders + +Identify the caller as browser Prebid.js, backend, or edge service. Capture a sanitized baseline containing eligible inventory, enabled bidders, account/placement parameters, timeouts, privacy signals, and host-specific endpoints. + +For each bidder, confirm server-side authorization, host credentials, supported formats, approved endpoints, adapter support in the pinned release, and source-IP allowlists. Existing provider credentials and account IDs may not transfer to a self-hosted PBS. + +Distinguish publisher request parameters from host-level adapter secrets. Map each secret to a real field or environment binding supported by the pinned adapter. A generic API-key environment variable does not configure arbitrary bidders. Ask for secret identifiers and required keys, never credential values. + +For browsers, resolve CORS/OPTIONS, cookie scope, user-sync and callback URLs, consent-dependent endpoints, and identity behavior across regions. For backend/edge callers, define trusted forwarding hops and preserve device IP, privacy signals, request identifiers, and the caller deadline. Verify outgoing bidder requests use the intended device context rather than the proxy's identity. + +## Configuration + +PBS Go supports environment variables and configuration files. Resolve one nonsecret configuration file and verify precedence, search paths, required fields, and environment-name mapping against the selected release. Use a parser for structured overrides; PBS is not an arbitrary multi-file YAML merger. + +Account for external URL, listener, enabled adapters, bidder endpoints, auction timeout, privacy defaults, account policy, user sync, stored requests, cache, metrics, and logs. Privacy choices need an approved policy owner; do not weaken them to make a smoke test pass. + +Preserve required upstream static assets when assembling or mounting the runtime. Check startup output and adapter debug paths with dummy values for unintended disclosure, even if PBS documents secret redaction. + +## Stored requests, accounts, and cache + +Ask whether eligible auctions reference stored-request IDs or depend on dynamically updated account settings. For a small fixed set, local versioned definitions may suffice. Otherwise reproduce the update and lookup behavior or explicitly restrict eligibility. Validate every referenced ID used in fixtures. + +Video and AMP can introduce Prebid Cache requirements. Determine the actual request and response behavior rather than assuming all formats share the same dependencies. Specify write location, public retrieval URL, expiration, and failure behavior. + +With independent regional caches, a key written in East must remain retrievable from the URL returned in that auction. A shared latency-routed lookup hostname can send retrieval to the wrong cache. Choose region-specific retrieval URLs with accepted availability limits, or a storage/routing design that guarantees retrieval through the approved failure scenario. Do not substitute a database product for a verified cache integration. + +## Pilot allocation and rollback + +Load this section when comparing with an existing provider or migrating traffic. + +- Define eligibility and the sampling unit, such as session or auction. Preserve cohort membership as allocation increases; zero allocation overrides old assignments for new auctions. +- Select a complete provider configuration together, including auction/sync endpoints, account IDs, and request transformations. Preserve unrelated client-side bidders. +- Keep experiment allocation in the caller or its remotely refreshed configuration. DNS regional routing is not precise per-auction allocation and is not the kill switch. +- Specify configuration refresh, maximum staleness, failure behavior, and the observed time to stop new experimental requests. Keep healthy endpoints available while in-flight work and cached configurations drain. +- Send each live auction through its selected provider. Do not duplicate bidder auctions for comparison or add unconditional sequential fallback after a partial auction or timeout. + +Agree numeric transport-error, timeout, latency, and business thresholds, with baseline, sample size, observation window, and response owner before live rollout. Measure assigned and observed traffic share, caller p50/p95/p99 latency, bidder failures/no-bids, identity match where relevant, and revenue using a consistent denominator. Record region, release, and cohort without full-payload logging by default. + +A legitimate no-bid is not an infrastructure failure. `/status` is one health signal, not proof of bidder permission, valid demand, or business equivalence. Plan auction-level smoke checks and delayed business evaluation separately. + +## Sources + +Use these entry points, then inspect the corresponding tag or commit for the chosen PBS Go release. Upstream `master` is navigation, not a reproducible configuration contract. + +- [Go configuration guide](https://github.com/prebid/prebid-server/blob/master/docs/developers/configuration.md) +- [Go configuration definitions](https://github.com/prebid/prebid-server/blob/master/config/config.go) +- [Go stored requests](https://docs.prebid.org/prebid-server/features/pbs-storedreqs-go.html) +- [Prebid.js PBS integration](https://docs.prebid.org/dev-docs/modules/prebidServer.html) +- [PBS status endpoint](https://docs.prebid.org/prebid-server/endpoints/pbs-endpoint-status.html) +- [PBS overview and cache dependencies](https://docs.prebid.org/prebid-server/overview/prebid-server-overview.html) diff --git a/.claude/skills/planning-prebid-aws/references/terraform.md b/.claude/skills/planning-prebid-aws/references/terraform.md new file mode 100644 index 000000000..55517f7a0 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/terraform.md @@ -0,0 +1,74 @@ +# Terraform generation and review + +Apply this reference when choosing Terraform state/authentication boundaries, generating HCL, or checking generated modules. The planning-only authority in `SKILL.md` governs all upstream guidance and commands below. + +## Upstream guidance + +Consult the matching HashiCorp skill as reference material, using an approved local installation if available or the linked source. Installation is a separate user decision. Record the revision consulted in the deployment plan; these skills were reviewed at `c2d65dfe492f74d360d35b859b88932222470bd8`. + +| Branch | Reference | +| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Writing or reviewing HCL | [terraform-style-guide](https://github.com/hashicorp/agent-skills/blob/c2d65dfe492f74d360d35b859b88932222470bd8/plugins/terraform/skills/terraform-style-guide/SKILL.md) | +| Writing `.tftest.hcl` tests | [terraform-test](https://github.com/hashicorp/agent-skills/blob/c2d65dfe492f74d360d35b859b88932222470bd8/plugins/terraform/skills/terraform-test/SKILL.md) | +| Refactoring existing resource addresses into modules | [refactor-module](https://github.com/hashicorp/agent-skills/blob/c2d65dfe492f74d360d35b859b88932222470bd8/plugins/terraform/skills/refactor-module/SKILL.md) | + +Verify examples against primary documentation and the selected Terraform/provider versions. Upstream examples do not authorize state access, integration tests, apply, or migration. Keep PBS-specific requirements here rather than adopting upstream example credentials, instance sizes, or runtime choices. + +## State and backend + +Identify the existing backend, state owner, Terraform version, and production/nonproduction boundaries before choosing storage. Reuse an approved backend. Keep bootstrap ownership separate from resources whose teardown it records. + +For a newly selected S3 backend, require bucket versioning, encryption, blocked public access, restricted state access, and native locking with `use_lockfile = true` on a compatible Terraform version. Include the required read/write/delete permissions for the `.tflock` object; lock deletion does not require permission to delete the state object. Verify these details against the [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3). + +DynamoDB-based locking is deprecated. For an existing backend using it, propose a migration that accounts for every Terraform client, lock permissions, state recovery, and the cutover owner. Preserve current locking until the separately authorized migration establishes the replacement. File generation must not migrate state, disable locking, or replace the existing backend automatically. + +Separate production and nonproduction state and access. Split regional state only when ownership, failure isolation, or infrastructure change lifecycles justify it. One pilot root with explicit regional provider mappings remains a valid approved choice; backend selection does not select ECS or impose East/West regions. + +## Versions and module contracts + +- Select a reproducible Terraform CLI version in the repository's toolchain and compatible `required_version` constraints. Verify feature support before selecting a version; sample version numbers are not defaults. +- Declare compatible provider constraints and generate `.terraform.lock.hcl` in each root. Commit the locks and review upgrades deliberately. The lock file records provider selections and checksums, not the CLI or remote module versions. +- Pin external registry modules to explicit versions for reproducible roots; pin Git sources with immutable commit references. A module's `version` argument applies only to registry sources. Local modules share their caller's repository revision. See [dependency locking](https://developer.hashicorp.com/terraform/language/files/dependency-lock) and [module sources](https://developer.hashicorp.com/terraform/language/modules/sources). +- Keep provider configurations in roots; reusable modules declare requirements and receive provider mappings. Test every selected regional mapping, including aliased providers. +- Expose settings expected to vary. Give every input a type and description, and every output a description. Validate real restrictions, resource bounds, and conditional requirements. Required unknowns must fail validation or an explicit preflight instead of silently choosing a deployment target. +- Use stable keys for independently named resource collections. Avoid index-driven address churn when membership changes. Keep module boundaries tied to responsibility and shared lifecycle rather than wrapping each AWS resource. +- Apply common ownership/cost tags through each AWS provider configuration where supported, including aliases. Check resource-specific exceptions and override behavior; `default_tags` is not proof every resource is tagged. + +When refactoring existing infrastructure, obtain authorization for scoped state inspection and map old/new addresses before proposing `moved` blocks. Require a reviewed migration plan showing intended address moves without unintended resource destruction or replacement. Leave state mutation and migration execution to the authorized operator. + +## Credentials and resource ownership + +Prefer short-lived authentication through the approved AWS identity system: [Identity Center/SSO](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) for operators and [OIDC federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) for CI where supported. Scope CI trust to the intended repository, branch or protected environment, and audience. Separate plan/apply permissions where practical; the plan role still needs only the state/lock and read permissions its work requires. Keep cloud credentials unavailable to untrusted pull-request code. Record any platform exception for approval instead of generating permanent access-key secrets by default. + +Terraform owns AWS resources, secret metadata, and access policy. An authorized external workflow owns credential values. Keep secret values out of user data, Terraform variables/data sources, plans, and state; `sensitive` masks display but does not exclude ordinary values from state. Output identifiers and endpoints only. Native secret references preserve this boundary for the selected runtime; ephemeral/write-only support is not a reason to route bidder credentials through Terraform. See [sensitive data](https://developer.hashicorp.com/terraform/language/manage-sensitive-data). + +Prefer provider-native resources, image builds, and explicit deployment tools over provisioner side effects. Keep application releases outside `local-exec`, `remote-exec`, and shell-command wrappers such as `null_resource`. For ECS, assign task-definition revisions and the service's selected revision to one writer. If a deployer selects revisions outside Terraform, document and test drift handling so a later apply cannot silently revert a release. + +## Tests and safe local checks + +Generate module tests for input restrictions, conditional resources, capacity bounds, provider mappings, tags/security settings, secret references, and meaningful outputs as applicable to the approved topology. Include negative cases using `expect_failures` for custom validation. Use explicit mock values when assertions depend on computed attributes; plan-time values may otherwise remain unknown. + +Before running tests: + +1. Enumerate the exact test files and their run blocks, setup modules, providers, aliases, and external data sources. Inspect downloaded modules and executable hooks. Terraform can mix real and mocked providers in one suite. +2. Prefer explicit `command = plan` with mocked external providers. Plan mode alone is not credential-free: real providers and data sources can contact AWS. Mock every external provider used by the selected tests, including aliases and setup dependencies; exclude command-executing data sources and other unapproved side effects. +3. Fully mocked apply-mode tests may check computed results after the same inspection. Mocks work with both plan and apply. An omitted command defaults to apply, so require explicit commands; a real-provider apply test is a cloud integration test outside this workflow. +4. Select individual reviewed files with `terraform test -filter=tests/.tftest.hcl`, repeating `-filter` for additional files. Filters take file paths, not run names or naming substrings. `-test-directory` alone is not isolation because Terraform also discovers tests in the root directory. +5. Run without cloud credentials or metadata-credential access and with network restrictions appropriate to the test environment. Confirm the expected tests actually ran; zero tests is not a pass. Record commands, counts, results, and remaining cloud-validation gaps. + +Use the selected CLI's help and [test command documentation](https://developer.hashicorp.com/terraform/cli/commands/test) to verify flags. Consult [test syntax](https://developer.hashicorp.com/terraform/language/tests) and [mocking](https://developer.hashicorp.com/terraform/language/tests/mocking) rather than copying upstream CLI examples blindly. + +Run `terraform fmt -check -recursive `. In each root, inspect dependency sources before `terraform init -backend=false`, then run `terraform validate`. Backend-disabled init still downloads providers/modules; follow dependency-download policy. Once dependencies are installed, keep the test run separate from downloads. Use existing configured lint/security scanners when present, or propose their addition rather than silently installing tools. + +Validation and mocks prove configuration behavior, not IAM permissions, quotas, real bidder connectivity, or deployment capacity. If tooling or safe isolation is unavailable, mark checks not run. Live integration tests need a separately authorized test account, cost limits, and cleanup verification; automatic teardown is not a guarantee that all resources were removed. + +## Saved-plan handoff + +Generate the following procedure in the runbook or inactive, approval-gated CI. Do not execute it as a local validation step: + +1. An authorized operator verifies the account/role, regions, backend/state target, source revision, toolchain, dependency locks, and input set before producing `terraform plan -out=`. +2. Store the plan as a protected, short-lived artifact with its checksum and provenance. Saved plans and JSON renderings can contain cleartext secrets. Keep them out of Git and public PR logs; restrict review access and retention. +3. Review additions, changes, deletions/replacements, IAM/network exposure, cost implications, and target identity. Tie approval to that artifact and source revision, not just a PR's earlier speculative plan. +4. The separately authorized apply stage verifies the artifact identity and uses `terraform apply ` rather than silently generating another plan. Re-plan and re-review after relevant code/input/state changes, an expired approval window, or known drift. A saved plan is not a lock on AWS and does not guarantee success against out-of-band changes. + +Authenticated plan/refresh, backend migration, state mutation, and real-resource apply/destroy remain execution tasks. See [saved-plan behavior and sensitivity](https://developer.hashicorp.com/terraform/cli/commands/plan). From 6763b0b41135a6f56c543ffb2603752e927c741a Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 15 Sep 2026 17:49:43 -0500 Subject: [PATCH 02/14] Add experimental PBS operator commands and configuration guidance --- .claude/skills/planning-prebid-aws/SKILL.md | 9 +- .../examples/two-region-pilot.md | 16 + .../references/configuration-and-secrets.md | 110 ++++ .../references/file-generation.md | 57 +- .../references/prebid-go.md | 11 +- .tool-versions | 9 +- Cargo.lock | 43 ++ crates/trusted-server-cli/Cargo.toml | 5 +- crates/trusted-server-cli/README.md | 107 ++++ .../examples/pbs/bindings.json | 21 + .../examples/pbs/deployment.yaml | 16 + .../trusted-server-cli/examples/pbs/east.yaml | 3 + .../trusted-server-cli/examples/pbs/pbs.yaml | 6 + crates/trusted-server-cli/src/commands/mod.rs | 1 + .../src/commands/pbs/aws.rs | 179 ++++++ .../src/commands/pbs/config.rs | 509 ++++++++++++++++++ .../src/commands/pbs/inspect.rs | 299 ++++++++++ .../src/commands/pbs/mod.rs | 237 ++++++++ .../src/commands/pbs/secrets.rs | 393 ++++++++++++++ .../src/commands/pbs/status.rs | 183 +++++++ crates/trusted-server-cli/src/run.rs | 77 +++ crates/trusted-server-cli/tests/pbs_cli.rs | 254 +++++++++ 22 files changed, 2494 insertions(+), 51 deletions(-) create mode 100644 .claude/skills/planning-prebid-aws/references/configuration-and-secrets.md create mode 100644 crates/trusted-server-cli/README.md create mode 100644 crates/trusted-server-cli/examples/pbs/bindings.json create mode 100644 crates/trusted-server-cli/examples/pbs/deployment.yaml create mode 100644 crates/trusted-server-cli/examples/pbs/east.yaml create mode 100644 crates/trusted-server-cli/examples/pbs/pbs.yaml create mode 100644 crates/trusted-server-cli/src/commands/pbs/aws.rs create mode 100644 crates/trusted-server-cli/src/commands/pbs/config.rs create mode 100644 crates/trusted-server-cli/src/commands/pbs/inspect.rs create mode 100644 crates/trusted-server-cli/src/commands/pbs/mod.rs create mode 100644 crates/trusted-server-cli/src/commands/pbs/secrets.rs create mode 100644 crates/trusted-server-cli/src/commands/pbs/status.rs create mode 100644 crates/trusted-server-cli/tests/pbs_cli.rs diff --git a/.claude/skills/planning-prebid-aws/SKILL.md b/.claude/skills/planning-prebid-aws/SKILL.md index 6b4c12afd..fce347753 100644 --- a/.claude/skills/planning-prebid-aws/SKILL.md +++ b/.claude/skills/planning-prebid-aws/SKILL.md @@ -16,6 +16,8 @@ This workflow permits planning and, after design approval, file generation and s Inspect repository instructions, Git status, existing infrastructure, runtime files, and supplied documents. Preserve unrelated changes. Identify the target repository and deployment directory before proposing edits. +For read-only discovery from any existing `trusted-server.toml`, and for designing the operator commands and PBS config/secret delivery, read [configuration and secrets](references/configuration-and-secrets.md). Select the authoritative config source before inferring requirements; examples and disabled integrations are not active deployment inputs. + Record requirements in one decision record, initially in the conversation, then in the approved deployment plan: | Requirement | Value | Status | Evidence or decision owner | Blocks | @@ -24,7 +26,7 @@ Record requirements in one decision record, initially in the conversation, then Treat supplied examples as evidence of intent, not accepted requirements for this deployment. Reconcile conflicting inputs with the user. Inspect existing answers before asking again. -Done when the current deployment, requested outcome, reusable resources, and unanswered decisions are identified. +Done when the current deployment, requested outcome, reusable resources, selected Trusted Server config source or its absence, and unanswered decisions are identified. ## 2. Interview by decision @@ -53,6 +55,7 @@ Present one recommended design and only alternatives that resolve a real tradeof - Each selected AWS service, its requirement, and whether to reuse or create it. - Capacity assumptions, cost drivers and estimate date, accepted limitations, and blockers. - Runtime, infrastructure, secret, and traffic-control ownership. +- Use the experimental `ts pbs` commands for supported local checks, secret writes, and EC2 status. Record the selected YAML/secret delivery path and unsupported operations explicitly. Deployment, rollback, and other runtime support need a separately approved implementation; avoid competing wrappers for implemented commands. - Target files and checks, with cloud-dependent checks separated from local checks. Ask the user to approve the architecture, assumptions, target files, and accepted limitations. Approval to generate files is not approval to execute them. Reopen approval if later findings change topology, cost commitments, or ownership. @@ -63,9 +66,11 @@ Done when the user explicitly approves the design and file scope. If blockers re Read [file generation and validation](references/file-generation.md). Follow existing repository conventions and generate only artifacts used by the selected design. Keep the decision record in the deployment plan; reference it from the runbook rather than repeating it. +Read the [PBS CLI usage and descriptor schema](../../../crates/trusted-server-cli/README.md) before generating inputs consumed by `ts pbs`. Its current descriptor supports EC2/Compose only. Keep other architecture choices available, but mark their CLI integration deferred rather than generating unsupported fields. + Verify version-specific PBS fields and adapter bindings against the selected release. Verify AWS/Terraform behavior and pricing against current primary documentation. Record source links, versions, and verification dates in the deployment plan. Unavailable evidence remains a named blocker; do not invent image digests, configuration keys, prices, or benchmark results. -Done when every approved artifact exists, has a named owner and check, and every unresolved input is visible and prevents unsafe use where applicable. +Done when every approved artifact exists, has a named owner and check, and every unresolved input is visible and prevents unsafe use where applicable. Every generated operator command must have documented inputs, access requirements, output, failure behavior, and a recovery action; proposing command names alone is not implementation. ## 5. Validate and hand off diff --git a/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md index e072d1af7..137ea2e1b 100644 --- a/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md +++ b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md @@ -41,6 +41,22 @@ The user must accept one-host-per-region outages and maintenance behavior. If re File generation starts after architecture-changing questions and target paths are approved. Live-traffic gates remain deferred to authorized execution: zero-allocation deployment checks, internal inventory validation, then agreed 1% and 5% observation windows. A tested caller kill switch and measured refresh bound are required before the live pilot. +## Configuration and operator walkthroughs + +The experimental `ts pbs` CLI implements local inspection/checks, secret value writes, and EC2 infrastructure status. The release, runtime delivery, and rollback scenarios below remain acceptance criteria for a future approved implementation, not executed deployment evidence. Consult the [current command contract](../references/configuration-and-secrets.md#operator-command-contract) before documenting an invocation. + +| Input or task | Expected behavior | +| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Several `trusted-server.toml` files, including a disabled example | Ask which source/environment is authoritative; preserve every source file and report uncertainty about remote overrides | +| The config lists server bidders, client-side bidders, and bundle adapters | Classify them separately; verify candidate server adapters and report host-secret needs as required, not needed, or unresolved without activating bidders | +| An operator changes a PBS timeout | Check and render regional YAML, prepare an immutable release, preview and approve deployment; no manual environment-file edits or Terraform apply | +| A binding conflicts with YAML or a required secret key is missing | Reject conflicting local inputs or stop the authorized deployment preflight; preserve working capacity and never print credential values | +| A credential contains quotes, dollars, or newlines | Prompt without echo or accept file/stdin, validate and safely encode it; no secret-value command argument or log output | +| East consumes a rotated credential while West still runs the old version | Report actual regional versions and pending work, verify replication and replacement, then coordinate partner revocation; never report global success from East alone | +| An operator requests rollback after the old credential was revoked | Refuse known-incompatible recovery and explain the credential action needed; a Git release rollback cannot restore bidder validity | +| A command resolves the wrong AWS account or an ambiguous deployment | Stop before mutation and require corrected target selection; local inspect/check stay credential-free | +| ECS is selected instead of Compose | Generate one concrete YAML delivery mechanism and task secret references behind the same operator commands; do not generate unused host loaders | + ## Walkthrough assertions Use these contrasts when reviewing the skill. They are expected behavior, not executed deployment tests. Terraform cases exercise the rules in `references/terraform.md`. diff --git a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md new file mode 100644 index 000000000..723e22509 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md @@ -0,0 +1,110 @@ +# Configuration, secrets, and operator workflow + +Use this reference for Trusted Server configuration discovery, the experimental `ts pbs` operator commands, and proposed runtime delivery. The current commands are implemented in `crates/trusted-server-cli`; deployment and rollback remain design work. Their existence does not authorize cloud operations. Read the [CLI usage and schema](../../../../crates/trusted-server-cli/README.md) before generating its input files or documenting invocations. + +## Discover requirements from Trusted Server + +Before asking questions the repository can answer: + +1. Locate `trusted-server.toml` in the selected project, including ignored operator-owned files and any path supplied by the user. Limit discovery to that project. When several files or environments could apply, ask which is authoritative; never select one by modification time. If none exists, continue the interview and label example files as examples. +2. Parse the selected file with a TOML parser and inspect only relevant fields. Check their semantics against the repository's Trusted Server schema. Comments, disabled integrations, and browser build inputs do not establish active server-side demand. +3. Ask whether environment overrides, remote configuration, or request-time parameters make the file incomplete or stale. Record the effective source as unresolved when it cannot be established locally. Cloud inspection still requires authorization. +4. Add observations to the existing deployment decision record with source path/key, candidate requirement, confidence/status, and the remaining question. Report credential identifiers or presence only; keep raw TOML, credential values, and commercially sensitive publisher values out of reports and generated examples. +5. Verify candidate adapters against the selected PBS Go release and confirm partner authorization. Record each host-secret requirement as required, not needed, or unresolved. A bidder name alone proves neither a secret requirement nor permission to use the bidder. + +| Trusted Server input | Discovery use | +| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| `integrations.prebid.enabled` | Determine active versus disabled intent using the schema's defaults | +| `server_url`, `account_id` | Identify existing provider/account compatibility questions, not automatically portable settings | +| `bidders` | Candidate server-side adapter set, subject to effective runtime configuration | +| `client_side_bidders` | Browser-side participation; do not automatically enable these adapters in PBS | +| `timeout_ms` | Caller budget; leave room for network/proxy work when proposing PBS auction timeouts | +| `test_mode`, `debug` | Testing/diagnostic intent, not authorization to contact bidders or proof that no real auction occurs | +| `bid_param_override_rules` | Conditional publisher/placement inputs and inventory behavior, not generic host credentials | +| `bundle.adapters`, identity modules | Browser bundle capabilities and identity questions, not proof of server-side use | +| Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller/request path | + +Use `ts pbs inspect --config ` for redacted local discovery. It reports explicit values without expanding defaults or proving host-secret requirements. Resolve schema defaults and adapter metadata separately, recording unsupported details as unresolved rather than fetching credentials or inventing mappings. Read config without invoking commands that publish or rewrite it. Leave `trusted-server.toml` unchanged. Propose caller endpoint/account changes separately after integration approval; discovery must not activate bidders, publish configuration, or change traffic. + +Done when the selected source and its limits are recorded, every observed bidder is classified, and missing inputs remain visible rather than filled from examples. + +## One home for each input + +Use existing repository paths where available. For a new deployment, propose: + +| Input | Owner and destination | +| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| `runtime/pbs.yaml` | Git-owned nonsecret PBS baseline | +| `runtime/regions/` | Optional nonsecret regional overrides, rendered into each region's YAML | +| `runtime/secret-bindings.json` | Git-owned secret identifiers, JSON key names, supported PBS destinations, types, required/optional status, and regional scope; no values | +| `deployment.yaml` | Nonsecret deployment descriptor: environment, AWS account/profile or role, regions, selected runtime, resource identifiers, and artifact locations | +| Credential values | Secrets Manager through an authorized operator or automation workflow | +| Publisher/site/placement parameters | The approved caller configuration or stored-request system, passed with auctions | + +PBS Go supports files and environment variables, with environment values taking precedence. Default to YAML for normal PBS settings and environment bindings for supported host secrets. Keep unrelated process/AWS environment variables separate. Any nonsecret PBS environment override must be explicit, generated, and justified; never forward arbitrary operator `PBS_*` variables into the container. + +Resolve one YAML file per region using a real parser. Define precedence as baseline then regional overrides; merge mapping keys and replace sequences as whole values. Validate the resolved fields and binding types against the pinned PBS release. A secret-bound field has one source: reject conflicting YAML values, duplicate environment destinations, and unknown bindings instead of silently choosing a winner. Keep unresolved required settings blocking release preparation. + +Cover external URL, listener, adapters/endpoints, timeout, privacy/account policy, identity, stored requests/cache, and telemetry. Preserve required upstream static assets. The privacy policy owner approves privacy defaults; a smoke test does not justify weakening them. + +## Release and runtime delivery + +This is the design contract for a separately approved runtime integration, not a capability of the current CLI. If Terraform user data currently owns runtime files, agree on the ownership transition before implementing independent release updates. A Secrets Manager write alone does not install a secret loader or replace consumers. + +```mermaid +flowchart TD + T["Selected Trusted Server config"] --> Q["Read-only discovery and approved requirements"] + Q --> Y["PBS YAML, regional overrides, secret bindings"] + Y --> R["Validated immutable release, no secret values"] + R --> D["Separately authorized deployment"] + S["Secrets Manager versions"] --> D + D --> P["PBS: resolved YAML and supported secret environment"] + P --> V["Health, auction checks, and deployment record"] +``` + +An immutable release records resolved nonsecret configuration checksums, PBS and ancillary image digests, required bindings, and optional stored-request assets. Pin CPU-compatible images. Resolve intended regional secret versions for an authorized deployment and record their identifiers with the release and target. Make containers consume those versions; recording a version while injecting an unconstrained latest value is not reproducible. + +Generate only the approved runtime's mechanism: + +- EC2/Compose: a versioned release supplies `/etc/config/pbs.yaml` through a read-only mount. A host-side role retrieves the selected secret versions and validates required keys before replacement. A parser/encoder writes restricted runtime environment files atomically on ephemeral storage; recreate them on boot. Test Compose quoting, dollars, newlines, and empty values. Never construct environment files through shell evaluation. Keep instance-role credentials inaccessible to the application. +- ECS: prefer an immutable configuration-bearing image containing the resolved nonsecret YAML and required assets. If using artifact retrieval instead, specify the downloader, permissions, checksums, startup ordering, and failure handling; an S3 object alone is not configuration delivery. Use supported task-definition secret references with the selected versions and execution/task-role permissions. Verify replacement tasks can load configuration and credentials before retiring healthy capacity according to the approved availability policy. + +Never package or log secret values. Treat container inspection and debug output as privileged; test redaction with dummy secrets. Follow [Terraform ownership](terraform.md#credentials-and-resource-ownership) for metadata and access policy instead of recreating that responsibility in runtime scripts. + +## Operator command contract + +Use `ts pbs` instead of generating deployment-local wrappers for these implemented operations. Keep existing `ts config`, `ts deploy`, and `ts prebid bundle` behavior unchanged. + +| Command | Current result and boundary | +| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ts pbs inspect --config ` | Local discovery with unresolved requirements; no source changes or secret retrieval | +| `ts pbs check --deployment ` | Local schema, binding, and regional merge checks; no AWS calls, upstream PBS schema validation, or startup proof | +| `ts pbs secrets set --deployment --region ` | Authorized complete-value write to an existing declared secret; hidden terminal or file/stdin input, account/confirmation checks, and retry UUID; no deployment | +| `ts pbs status --deployment ` | Authorized EC2 infrastructure reads for declared instances; PBS health, installed release, and consumed secret versions stay unknown | + +There are no `ts pbs deploy` or `ts pbs rollback` subcommands. Versioned release delivery, runtime injection, and release-aware status need separate implementation and approval. Do not present the future release contract as a working command. Current descriptors accept only `ec2-compose`; an ECS design requires another approved implementation rather than a fabricated descriptor. + +Read targets from the deployment descriptor. Allow an explicit deployment selection; if selection is ambiguous, stop. Before cloud operations, display environment/account/regions and verify authenticated account identity against the descriptor. Require deliberate regional scope for mutations rather than silently changing every region or defaulting to production. Local commands do not authenticate. Cloud read commands must not retrieve credential values merely to report status. + +Interactive mutations require confirmation of the concrete target and proposed change. Automation uses explicit target, release or secret-write inputs and an external approval gate; missing approval is an error, not a prompt fallback. Release preparation may render/build locally, but publication and cloud changes occur only after approval. If the preview changes, obtain approval again. A skill run that only authorizes file generation must not invoke these cloud commands. + +Secret input supports a hidden interactive prompt in the operator's terminal, not the agent conversation, and mutually exclusive file/stdin inputs. Accept no credential values as ordinary arguments. Validate a complete payload against the binding definition; use restricted temporary storage only when needed and clean up tool-created files on success or failure. Leave operator-supplied input files unchanged. Write the declared primary secret for replicated credentials or the selected independently owned regional secret, not a replica as if it were writable independently. Preserve retry identity for a logical write so retrying it does not create unintended versions. Report changed version identifiers and regions requiring deployment, never the payload. + +A future approved deployment implementation must own the render/package/inject/restart sequence; operators should not manually edit generated environment files or ECS task definitions. Reuse the [release lifecycle](file-generation.md#generated-deployment-tools) for locking, health deadlines, draining, and failure handling. Routine configuration or credential-value changes do not require Terraform apply. New secret metadata, permissions, or infrastructure changes follow the separate reviewed Terraform workflow. None of these commands modifies caller traffic allocation implicitly. + +For supported CLI commands, reference the CLI's input and failure contract in the runbook. For any separately approved deployment tool, document inputs, local versus cloud access, output, nonzero failure behavior, and the recovery action. Distinguish desired configuration from observed running state in status output; provide machine-readable nonsecret output where automation consumes it. + +## Rotation and recovery + +Updating Secrets Manager does not refresh a running container's environment. Coordinate credential issuance and overlap with the bidder, write the new value, verify regional availability, replace consumers in the approved order, and verify each region before retiring the old credential. Replication alone proves neither successful injection nor bidder authorization. Keep partial regional success visible with the release and secret versions actually running. + +On required-secret retrieval or validation failure, preserve working capacity and report the blocker. On failed deployment, use the recorded prior release and compatible credential versions; refuse a known-incompatible rollback. If partner validity is unknown, report that uncertainty before replacement rather than promising recovery. An application rollback cannot restore a credential revoked by the bidder. + +Done when the runbook gives each supported routine task one documented command, expected evidence, and failure/recovery behavior, with unimplemented release tasks clearly deferred. Prove discovery preserves the source file and redacts values; test rendering determinism, override conflicts, missing/optional keys, special-character dummy secrets, wrong-account/ambiguous-target refusal, retries, partial regional rollout, and revoked-credential recovery. Separate local fixture results from deferred live integration evidence. + +## Sources + +- [PBS Go configuration guide](https://github.com/prebid/prebid-server/blob/master/docs/developers/configuration.md), then the configuration definitions and adapter schema at the pinned release +- [ECS Secrets Manager injection](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html) +- [Secrets Manager regional replication](https://docs.aws.amazon.com/secretsmanager/latest/userguide/replicate-secrets.html) +- [Secrets Manager value updates](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/put-secret-value.html) diff --git a/.claude/skills/planning-prebid-aws/references/file-generation.md b/.claude/skills/planning-prebid-aws/references/file-generation.md index 89e365848..78c8c1f9b 100644 --- a/.claude/skills/planning-prebid-aws/references/file-generation.md +++ b/.claude/skills/planning-prebid-aws/references/file-generation.md @@ -4,45 +4,36 @@ Generate files only after the design and target paths are approved. Use the repo ## Artifact contract -| Artifact | Suggested location | Contents and owner | -| -------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Deployment plan | `docs/pbs-deployment-plan.md` | Requirements record, approved architecture, sources, costs, blockers, file/check inventory; user approves | -| State bootstrap, if needed | `infra/bootstrap/` | Independently managed backend, access and locking setup; infrastructure owner | -| Deployment root | `infra//` | Providers, variables/validation, explicit regional modules, outputs, selected DNS configuration, nonsecret examples, provider lock file; Terraform owns AWS resources | -| Reusable infrastructure | `infra/modules/` | Modules justified by repeated topology, not one module per AWS service | -| Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | -| Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | -| Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | -| Operational tools | `scripts/` or existing CI layout | Release preparation, deployment, secret loading where needed, smoke checks; deployment owner | -| Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | +| Artifact | Suggested location | Contents and owner | +| -------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Deployment plan | `docs/pbs-deployment-plan.md` | Requirements record, approved architecture, sources, costs, blockers, file/check inventory; user approves | +| State bootstrap, if needed | `infra/bootstrap/` | Independently managed backend, access and locking setup; infrastructure owner | +| Deployment root | `infra//` | Providers, variables/validation, explicit regional modules, outputs, selected DNS configuration, nonsecret examples, provider lock file; Terraform owns AWS resources | +| Reusable infrastructure | `infra/modules/` | Modules justified by repeated topology, not one module per AWS service | +| Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | +| Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | +| Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | +| Operator interface | Existing `ts pbs` CLI and approved CI layout | CLI descriptor and documented invocations; any missing release stages need separate implementation approval | +| Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | Each generated artifact must have a consumer. Add ignore rules for local credentials, runtime secret files, `.terraform`, state files, saved plans, and generated sensitive output. Track nonsecret examples. Use `example.com` hostnames and visibly fictional identifiers in examples. For Terraform artifacts, follow [Terraform generation and review](terraform.md), including module tests under `tests/` and the saved-plan procedure in the runbook. -## Runtime and secrets - -Pin PBS and ancillary images by verified digest with CPU compatibility. Render structured regional changes deterministically. Include release identity, checksums, required secret identifiers/keys, and dependencies in the manifest, never secret values. - -Choose one secret injection path for the approved runtime: - -- EC2/Compose: a host-side role retrieves secrets, validates supported bindings, and atomically writes restricted ephemeral runtime files. Rebuild them at boot. Encode through a parser with tested Compose behavior for quotes, dollars, newlines, and empty values; never use shell evaluation. Keep instance credentials away from application containers. -- ECS: use verified task-definition secret references or an explicitly justified retrieval mechanism, with appropriate execution/task-role permissions. Identify which changes require task replacement. - -A secret update does not automatically refresh process environment. Specify how replacement containers consume new values and how regional replicas become ready. Coordinate partner-side rotation and overlap; application rollback cannot restore a revoked credential. Treat debug output and container inspection as privileged. +For the deployment descriptor, resolved PBS YAML, secret bindings, runtime delivery, and command behavior, follow [configuration and secrets](configuration-and-secrets.md). Generate inputs for the existing CLI rather than replacement wrappers. Document unimplemented release stages as deferred unless their implementation is separately approved. ## Generated deployment tools -Generated scripts must require an explicit environment, target, and release, with identity/preflight checks and no production defaults. New CI deployment jobs must be manual and approval-gated. File generation must not trigger existing auto-apply or deployment jobs; inspect those triggers before editing their watched paths. +Use the target-selection and approval contract in [operator commands](configuration-and-secrets.md#operator-command-contract), including descriptor-based interactive use and explicit automation inputs. New CI deployment jobs must remain inactive and approval-gated. File generation must not trigger existing auto-apply or deployment jobs; inspect those triggers before editing their watched paths. -For the selected runtime, implement or explicitly defer each release stage: +The current `ts pbs` CLI does not implement deployment or rollback. For a separately approved deployment implementation, implement or explicitly defer each release stage: 1. Serialize competing deployments to the same target. 2. Retrieve and verify an immutable release, image availability, and required secrets before replacing working capacity. 3. Record previous release and secret version identifiers without values. 4. Deploy with the approved outage/draining policy and bounded health deadlines. 5. Check HTTPS, application health, and an agreed auction fixture. -6. Restore the previous release on failure only when dependencies and credentials remain compatible; otherwise stop and report the recovery action. +6. Follow [rotation and recovery](configuration-and-secrets.md#rotation-and-recovery) before restoring the previous release or reporting a recovery blocker. 7. Record target, release, timestamps, and outcome. Define retry limits, interrupted-run handling, and behavior when the same release is requested twice. On standalone hosts, provide boot recovery, deployment locking, bounded logs, certificate persistence, and an explicit response to hung-but-running containers. A Compose unhealthy status alone is not a restart policy. On managed runtimes, express equivalent rollout/rollback controls through the platform rather than adding host scripts. @@ -59,19 +50,17 @@ List deferred checks with expected results and an owner: authenticated Terraform Run applicable checks on the generated paths, recording exact commands and results. Inspect project scripts before executing them. Use isolated dummy credentials, fake bidder endpoints, and fixtures; local startup must not contact production bidders or fetch real AWS secrets. -| Area | Check | -| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Terraform | Follow [Terraform local checks](terraform.md#tests-and-safe-local-checks), including inspected, explicitly selected mocked tests | -| Runtime structure | Parse YAML/JSON, verify manifest paths/checksums, and render deterministic overrides twice | -| Compose branch | `docker compose --env-file -f config --quiet` and dummy-value round-trip checks | -| Scripts | Language syntax/lint and focused tests for missing/malformed inputs, special-character dummy secrets, failed release, interruption, and repeated invocation | -| PBS behavior | Approved isolated container startup and smoke fixture against controlled bidder responses, if a suitable local runtime is available | -| Final files | Diff review for scope, secret exposure, unresolved placeholders, inactive deployment triggers, and one writer per mutable resource | +| Area | Check | +| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Terraform | Follow [Terraform local checks](terraform.md#tests-and-safe-local-checks), including inspected, explicitly selected mocked tests | +| Runtime structure | Parse YAML/JSON, verify manifest paths/checksums, and render deterministic overrides twice | +| Compose branch | `docker compose --env-file -f config --quiet` and dummy-value round-trip checks | +| Operator tool | Language syntax/lint and [configuration/secrets contract tests](configuration-and-secrets.md#rotation-and-recovery), plus failed-release and interrupted/repeated invocation tests | +| PBS behavior | Approved isolated container startup and smoke fixture against controlled bidder responses, if a suitable local runtime is available | +| Final files | Diff review for scope, secret exposure, unresolved placeholders, inactive deployment triggers, and one writer per mutable resource | If a tool, image, network permission, or fixture is unavailable, mark that check not run with the next action. Never substitute a checklist for executed evidence or call a no-bid response proof of bidder success. ## Sources to verify during generation -- [ECS Secrets Manager injection](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html) -- [Secrets Manager regional replication](https://docs.aws.amazon.com/secretsmanager/latest/userguide/replicate-secrets.html) - [SSM Run Command](https://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html) diff --git a/.claude/skills/planning-prebid-aws/references/prebid-go.md b/.claude/skills/planning-prebid-aws/references/prebid-go.md index 00c8212c1..b90892ffb 100644 --- a/.claude/skills/planning-prebid-aws/references/prebid-go.md +++ b/.claude/skills/planning-prebid-aws/references/prebid-go.md @@ -8,18 +8,10 @@ Identify the caller as browser Prebid.js, backend, or edge service. Capture a sa For each bidder, confirm server-side authorization, host credentials, supported formats, approved endpoints, adapter support in the pinned release, and source-IP allowlists. Existing provider credentials and account IDs may not transfer to a self-hosted PBS. -Distinguish publisher request parameters from host-level adapter secrets. Map each secret to a real field or environment binding supported by the pinned adapter. A generic API-key environment variable does not configure arbitrary bidders. Ask for secret identifiers and required keys, never credential values. +Use [configuration and secrets](configuration-and-secrets.md) to discover candidate requirements from the selected `trusted-server.toml`, distinguish caller parameters from host credentials, and define supported runtime bindings. Discovery is evidence for the interview, not bidder activation. For browsers, resolve CORS/OPTIONS, cookie scope, user-sync and callback URLs, consent-dependent endpoints, and identity behavior across regions. For backend/edge callers, define trusted forwarding hops and preserve device IP, privacy signals, request identifiers, and the caller deadline. Verify outgoing bidder requests use the intended device context rather than the proxy's identity. -## Configuration - -PBS Go supports environment variables and configuration files. Resolve one nonsecret configuration file and verify precedence, search paths, required fields, and environment-name mapping against the selected release. Use a parser for structured overrides; PBS is not an arbitrary multi-file YAML merger. - -Account for external URL, listener, enabled adapters, bidder endpoints, auction timeout, privacy defaults, account policy, user sync, stored requests, cache, metrics, and logs. Privacy choices need an approved policy owner; do not weaken them to make a smoke test pass. - -Preserve required upstream static assets when assembling or mounting the runtime. Check startup output and adapter debug paths with dummy values for unintended disclosure, even if PBS documents secret redaction. - ## Stored requests, accounts, and cache Ask whether eligible auctions reference stored-request IDs or depend on dynamically updated account settings. For a small fixed set, local versioned definitions may suffice. Otherwise reproduce the update and lookup behavior or explicitly restrict eligibility. Validate every referenced ID used in fixtures. @@ -46,7 +38,6 @@ A legitimate no-bid is not an infrastructure failure. `/status` is one health si Use these entry points, then inspect the corresponding tag or commit for the chosen PBS Go release. Upstream `master` is navigation, not a reproducible configuration contract. -- [Go configuration guide](https://github.com/prebid/prebid-server/blob/master/docs/developers/configuration.md) - [Go configuration definitions](https://github.com/prebid/prebid-server/blob/master/config/config.go) - [Go stored requests](https://docs.prebid.org/prebid-server/features/pbs-storedreqs-go.html) - [Prebid.js PBS integration](https://docs.prebid.org/dev-docs/modules/prebidServer.html) diff --git a/.tool-versions b/.tool-versions index 758146800..8b805f271 100644 --- a/.tool-versions +++ b/.tool-versions @@ -1,5 +1,6 @@ -fastly 15.1.0 -rust 1.95.0 -nodejs 24.12.0 -viceroy 0.17.0 +fastly 15.1.0 +rust 1.95.0 +nodejs 24.12.0 +viceroy 0.17.0 wasmtime 44.0.1 +aws latest diff --git a/Cargo.lock b/Cargo.lock index 4f8d87c29..6fcdd78ee 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4103,6 +4103,17 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "rpassword" +version = "7.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2da316a15f47e3d053de9cb2c439650bd8fa4aaeb9365f2e5f27f492ff73c196" +dependencies = [ + "libc", + "rtoolbox", + "windows-sys 0.61.2", +] + [[package]] name = "rsa" version = "0.9.10" @@ -4123,6 +4134,16 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rtoolbox" +version = "0.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a1efe12a1469752d0e6ff5ebec0b6ef4924cc5c4c71046b0ec730040535819d" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + [[package]] name = "rusqlite" version = "0.32.1" @@ -4567,6 +4588,19 @@ dependencies = [ "syn 2.0.118", ] +[[package]] +name = "serde_yaml_ng" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4db627b98b36d4203a7b458cf3573730f2bb591b28871d916dfa9efabfd41f" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "servo_arc" version = "0.4.3" @@ -5449,11 +5483,13 @@ dependencies = [ "rand 0.8.6", "rcgen", "regex", + "rpassword", "rustls", "rustls-pemfile", "scraper", "serde", "serde_json", + "serde_yaml_ng", "similar", "temp-env", "tempfile", @@ -5465,6 +5501,7 @@ dependencies = [ "tracing", "trusted-server-core", "url", + "uuid", "webpki-roots", "which", "x509-parser", @@ -5661,6 +5698,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.9.0" diff --git a/crates/trusted-server-cli/Cargo.toml b/crates/trusted-server-cli/Cargo.toml index 0f93651c5..10f2ebd00 100644 --- a/crates/trusted-server-cli/Cargo.toml +++ b/crates/trusted-server-cli/Cargo.toml @@ -18,6 +18,7 @@ workspace = true chromiumoxide = { workspace = true } clap = { workspace = true } derive_more = { workspace = true } +error-stack = { workspace = true } edgezero-cli = { workspace = true } edgezero-core = { workspace = true } futures = { workspace = true } @@ -26,9 +27,11 @@ http = { workspace = true } log = { workspace = true } rand = { workspace = true } regex = { workspace = true } +rpassword = "7.5" scraper = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +serde_yaml_ng = "0.10" similar = { workspace = true } tempfile = { workspace = true } tokio = { workspace = true } @@ -37,6 +40,7 @@ toml_edit = { workspace = true } tracing = { workspace = true } trusted-server-core = { workspace = true } url = { workspace = true } +uuid = { workspace = true } which = { workspace = true } # `ts dev proxy` is macOS-only — CA trust via the login keychain, Safari @@ -49,7 +53,6 @@ which = { workspace = true } base64 = { workspace = true } bytes = { workspace = true } directories = { workspace = true } -error-stack = { workspace = true } http-body-util = { workspace = true } hyper = { workspace = true, features = ["http1", "server", "client"] } hyper-util = { workspace = true, features = ["tokio"] } diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md new file mode 100644 index 000000000..7f9445190 --- /dev/null +++ b/crates/trusted-server-cli/README.md @@ -0,0 +1,107 @@ +# Trusted Server CLI: experimental PBS commands + +`ts pbs` manages local configuration inputs and a small set of AWS operations for self-hosted Prebid Server Go. It is separate from `ts prebid bundle`, which builds browser JavaScript, and from the existing Trusted Server `ts config` and `ts deploy` commands. + +The namespace is experimental and is being evaluated in PR review. There is no separate binary or crate. + +## Build and try locally + +Use this branch's executable, not an older installed `ts`: + +```bash +cargo build_cli_linux +cargo run_cli_linux pbs --help +cargo run_cli_linux pbs inspect --config trusted-server.example.toml --json +cargo run_cli_linux pbs check --deployment crates/trusted-server-cli/examples/pbs/deployment.yaml +``` + +On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fictional resource identifiers, a fictional image digest, and a fictional adapter binding. They exercise local checks only and must not be used as real deployment settings. + +## Commands and current limits + +| Command | What it does | Access | +| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------- | +| `ts pbs inspect --config ` | Reports selected local Prebid fields, classifies bidder lists, and marks host-secret requirements unresolved | Local read-only | +| `ts pbs check --deployment ` | Validates schema, targets, binding metadata, and deterministic regional YAML merging | Local read-only | +| `ts pbs secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | +| `ts pbs status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | + +Add `--json` anywhere under `ts pbs` for a machine-readable report. Errors and operator notices go to stderr; failures exit with code 2. A partial status report still appears on stdout, with `complete: false` and exit code 2. + +Not implemented: container deployment, rollback, runtime secret injection, caller updates, Terraform execution, ECS status, or PBS application health checks. `status` always reports the installed release and consumed secret versions as unknown. EC2 health is not PBS readiness. + +`check` does not validate every PBS configuration field against the upstream Go schema, independently verify adapter documentation, retrieve secrets, pull images, start PBS, or prove a deployment ready. Bindings are operator-supplied metadata, not a bundled bidder catalog. Validate actual adapter mappings and startup against the pinned PBS release before use. + +## Discovering requirements + +`inspect` reads exactly the chosen file and never rewrites or publishes it. It accepts the runtime's array, indexed-map, and string bidder-list encodings, array-form bundle/module lists, and reports explicitly supplied values only. It does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source/environment is authoritative before relying on the report. + +Account identifiers, endpoint values, and bid-parameter values are withheld. Parser errors also withhold source snippets. Server-side bidders, client-side bidders, and browser bundle adapters remain separate lists; listing a bidder does not establish partner authorization or a host-secret requirement. Disabled integrations remain disabled. + +## Deployment descriptor + +See [deployment.yaml](examples/pbs/deployment.yaml), [PBS YAML](examples/pbs/pbs.yaml), [regional overrides](examples/pbs/east.yaml), and [bindings.json](examples/pbs/bindings.json). + +Version 1 requires: + +- `schema_version: 1`, an explicit environment, and `runtime: ec2-compose`. +- An explicit 12-digit AWS account ID and AWS CLI profile. Environment/profile/bidder identifiers use letters, digits, underscores, and hyphens. +- A digest-pinned PBS image and a baseline YAML path. +- A nonempty region map with optional override paths and explicit EC2 instance IDs for `status`. +- An optional binding-file path. Omit it when no host secrets are needed. + +Paths resolve relative to the descriptor, not the working directory. No automatic descriptor discovery or production default exists. Unknown descriptor fields, unsupported runtime types, duplicate YAML keys, tags, and implicit YAML merge keys are rejected. + +Mappings merge recursively with regional values taking precedence; sequences and scalars replace whole values. Rendering happens in memory and writes no generated files. The command does not read arbitrary operator `PBS_*` environment overrides. + +### Binding schema + +The binding file is a JSON or YAML mapping keyed by bidder identifier. Each entry declares: + +- `verified_image`: the exact PBS image string matching the descriptor. +- `source`: an HTTPS reference used by the operator to verify the mapping. +- `secrets`: one complete Secrets Manager ARN for every descriptor region, matching its account and region. +- `keys`: credential key names mapped to `env`, `pbs_path`, and optional `required`, which defaults to true. + +Version 1 supports string-valued credentials and one binding set across all selected regions. Each secret ARN must belong to exactly one bidder binding because updates replace the complete JSON object. A PBS destination cannot overlap another binding or a value already present in the resolved YAML, including a non-mapping parent. Names alone do not prove that PBS supports the mapping; the metadata records an operator decision. + +## Setting a secret + +Prerequisites: AWS CLI v2 on PATH, a trusted local AWS profile using short-lived credentials, and a previously provisioned secret with approved permissions. AWS CLI command history must be disabled; the tool checks both the selected profile and default history settings before submitting a value. It suppresses AWS stderr and disables configured endpoint URL overrides for API calls. + +The command verifies the account through STS, describes the exact declared secret, and refuses replica writes or secrets scheduled for deletion. Interactive use reads a complete JSON object through a hidden terminal prompt, shows the target and retry UUID, and requires typing `yes` before writing. The prompt belongs in the operator's terminal, not an agent conversation. + +For approved automation, supply a file or stdin, `--yes`, and a stable UUID identifying the logical write: + +```bash +ts pbs secrets set examplebidder \ + --deployment /secure/path/deployment.yaml \ + --region us-east-1 \ + --file /secure/path/credential.json \ + --request-token 11111111-2222-4333-8444-555555555555 \ + --yes --json +``` + +The command above is an interface example, not authorization to run it. Generate a new UUID for each logical update and reuse that UUID with identical values for retries. Reusing it with different values fails in Secrets Manager. `--yes` confirms only this value write; it does not authorize deployment, partner rotation, or future writes. Use protected CI approval gates before invoking it. + +File and stdin inputs are mutually exclusive. `--stdin` requires `--yes` and `--request-token`. Input must contain only declared keys, include all required nonempty string values, and fit the Secrets Manager size limit. Optional keys may be omitted. Duplicate keys and non-string values are rejected. Quotes, dollar signs, newlines, and backslashes are encoded as JSON, never shell expressions. + +Secret values never enter command arguments or reports. The AWS CLI receives JSON through a tool-owned temporary file, owner-only on Unix, which is removed on normal success and error paths. Operator-provided input files are not changed or deleted. Run on a trusted host with protected temporary storage; abrupt process termination can leave temporary files requiring cleanup. Windows temporary-file ACL behavior has not been validated. + +A successful write reports its version identifier. It does not create secret metadata, change infrastructure, replace containers, or rotate the bidder's credential. Check regional replication, separately replace consumers, and verify them before revoking old partner credentials. A lost/malformed response leaves the write outcome uncertain; retain the displayed request token and retry identical input rather than creating another logical update. + +## Ownership and sandbox compatibility + +These commands do not adopt the existing sandbox's Terraform state, edit its files, or change its IAM roles. A descriptor must reference resources the operator has explicitly approved. The sandbox currently bootstraps runtime files through Terraform user data and has no runtime secret loader. Writing a secret therefore does not make that sandbox consume it. + +Deployment and rollback require a separately approved move to versioned runtime releases. Until that exists, `ts pbs` has no deployment or rollback subcommands and the skill must not promise them. + +## Verification + +```bash +./scripts/test-cli.sh +cargo fmt --all -- --check +cargo clippy --package trusted-server-cli --all-targets --target x86_64-unknown-linux-gnu -- -D warnings +``` + +Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require Python 3. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. diff --git a/crates/trusted-server-cli/examples/pbs/bindings.json b/crates/trusted-server-cli/examples/pbs/bindings.json new file mode 100644 index 000000000..24bd8a03a --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/bindings.json @@ -0,0 +1,21 @@ +{ + "examplebidder": { + "verified_image": "registry.example.com/pbs@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "source": "https://example.com/adapter-reference", + "secrets": { + "us-east-1": "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf" + }, + "keys": { + "api_key": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", + "pbs_path": ["adapters", "examplebidder", "api_key"], + "required": true + }, + "optional_token": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN", + "pbs_path": ["adapters", "examplebidder", "optional_token"], + "required": false + } + } + } +} diff --git a/crates/trusted-server-cli/examples/pbs/deployment.yaml b/crates/trusted-server-cli/examples/pbs/deployment.yaml new file mode 100644 index 000000000..cf0031c69 --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/deployment.yaml @@ -0,0 +1,16 @@ +# Fictional targets and image. Local check fixture, not deployable infrastructure. +schema_version: 1 +environment: sandbox +runtime: ec2-compose +aws: + account_id: '123456789012' + profile: pbs-sandbox +pbs: + config: pbs.yaml + image: registry.example.com/pbs@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bindings: bindings.json +regions: + us-east-1: + instance_ids: + - i-0123456789abcdef0 + overrides: east.yaml diff --git a/crates/trusted-server-cli/examples/pbs/east.yaml b/crates/trusted-server-cli/examples/pbs/east.yaml new file mode 100644 index 000000000..9c7f78ddd --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/east.yaml @@ -0,0 +1,3 @@ +external_url: https://pbs-east.example.com +auction_timeouts_ms: + default: 900 diff --git a/crates/trusted-server-cli/examples/pbs/pbs.yaml b/crates/trusted-server-cli/examples/pbs/pbs.yaml new file mode 100644 index 000000000..346cf1e9b --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/pbs.yaml @@ -0,0 +1,6 @@ +# Illustrates local merging only; validate actual PBS fields at the selected release. +port: 8000 +external_url: https://pbs.example.com +auction_timeouts_ms: + default: 1000 + max: 1200 diff --git a/crates/trusted-server-cli/src/commands/mod.rs b/crates/trusted-server-cli/src/commands/mod.rs index a01559fd4..f1d08a7ee 100644 --- a/crates/trusted-server-cli/src/commands/mod.rs +++ b/crates/trusted-server-cli/src/commands/mod.rs @@ -1,5 +1,6 @@ pub(crate) mod audit; pub(crate) mod config; +pub(crate) mod pbs; // `dev` is `pub` so the macOS-gated `tests/proxy_e2e.rs` suite can reach // `commands::dev::proxy`; the other command modules are crate-internal. pub mod dev; diff --git a/crates/trusted-server-cli/src/commands/pbs/aws.rs b/crates/trusted-server-cli/src/commands/pbs/aws.rs new file mode 100644 index 000000000..e4d3047ec --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/aws.rs @@ -0,0 +1,179 @@ +use std::io::Write; +use std::process::{Command, Stdio}; + +use error_stack::Report; +use serde_json::{Value, json}; +use tempfile::NamedTempFile; + +use super::config::{AwsTarget, Deployment, invalid}; +use super::{PbsError, Result}; + +/// An AWS call carries data separately from arguments; payloads never enter shell history or argv. +pub(super) trait Aws { + /// Invoke one AWS API with explicit identity and region. + /// + /// # Errors + /// Returns sanitized process/API failures; implementations must not expose raw stderr/payloads. + fn call( + &self, + target: &AwsTarget, + region: &str, + service: &'static str, + operation: &'static str, + input: &Value, + ) -> Result; +} + +pub(super) struct AwsCli; + +impl Aws for AwsCli { + fn call( + &self, + target: &AwsTarget, + region: &str, + service: &'static str, + operation: &'static str, + input: &Value, + ) -> Result { + if operation == "put-secret-value" { + ensure_history_disabled(target)?; + } + // tempfile creates an owner-only file on Unix. No command uses a shell or logs this payload. + let mut payload = NamedTempFile::new() + .map_err(|_| Report::new(PbsError::Io("cannot create private AWS request file")))?; + serde_json::to_writer(payload.as_file_mut(), input) + .map_err(|_| Report::new(PbsError::Io("cannot encode AWS request")))?; + payload + .flush() + .map_err(|_| Report::new(PbsError::Io("cannot flush AWS request")))?; + let output = Command::new("aws") + .args([ + "--profile", + &target.profile, + "--region", + region, + "--output", + "json", + "--no-cli-pager", + "--no-cli-auto-prompt", + "--cli-connect-timeout", + "10", + "--cli-read-timeout", + "20", + service, + operation, + "--cli-input-json", + ]) + .arg(format!("file://{}", payload.path().display())) + .env("AWS_CLI_AUTO_PROMPT", "off") + .env("AWS_PAGER", "") + .env("AWS_MAX_ATTEMPTS", "2") + .env("AWS_IGNORE_CONFIGURED_ENDPOINT_URLS", "true") + .stdin(Stdio::null()) + .output() + .map_err(|_| Report::new(PbsError::Aws("could not execute AWS CLI v2")))?; + // Closing removes the temporary payload, including on every earlier error path via Drop. + drop(payload); + if !output.status.success() { + return Err(Report::new(PbsError::Aws(operation))); + } + serde_json::from_slice(&output.stdout) + .map_err(|_| Report::new(PbsError::Aws("invalid JSON response"))) + } +} + +/// Refuse credential writes when AWS CLI history could retain the request payload. +/// +/// # Errors +/// Rejects enabled/unknown history settings and failures to inspect configuration. +fn ensure_history_disabled(target: &AwsTarget) -> Result<()> { + for key in ["cli_history", "default.cli_history"] { + let output = Command::new("aws") + .args([ + "--profile", + &target.profile, + "--no-cli-pager", + "--no-cli-auto-prompt", + "configure", + "get", + key, + ]) + .env("AWS_PAGER", "") + .env("AWS_CLI_AUTO_PROMPT", "off") + .stdin(Stdio::null()) + .output() + .map_err(|_| Report::new(PbsError::Aws("cannot verify CLI history is disabled")))?; + let value = String::from_utf8_lossy(&output.stdout); + if !matches!(output.status.code(), Some(0 | 1)) || !matches!(value.trim(), "" | "disabled") + { + return Err(Report::new(PbsError::Aws( + "disable AWS CLI history before writing credentials", + ))); + } + } + Ok(()) +} + +/// Verify account identity before any deployment resource lookup or mutation. +/// +/// # Errors +/// Rejects undeclared regions, failed identity reads, and mismatched accounts. +pub(super) fn verify_identity(deployment: &Deployment, region: &str, aws: &dyn Aws) -> Result<()> { + if !deployment.regions.contains_key(region) { + return Err(invalid( + "select a region declared in the deployment descriptor", + )); + } + let identity = aws.call( + &deployment.aws, + region, + "sts", + "get-caller-identity", + &json!({}), + )?; + if identity.get("Account").and_then(Value::as_str) != Some(deployment.aws.account_id.as_str()) { + return Err(Report::new(PbsError::AccountMismatch)); + } + Ok(()) +} + +#[cfg(test)] +pub(super) mod tests { + use std::cell::RefCell; + use std::collections::VecDeque; + + use super::*; + + pub(crate) struct FakeAws { + pub replies: RefCell>, + pub calls: RefCell>, + } + + impl FakeAws { + pub(crate) fn new(replies: Vec) -> Self { + Self { + replies: RefCell::new(replies.into()), + calls: RefCell::new(Vec::new()), + } + } + } + + impl Aws for FakeAws { + fn call( + &self, + _target: &AwsTarget, + region: &str, + _service: &'static str, + operation: &'static str, + input: &Value, + ) -> Result { + self.calls + .borrow_mut() + .push((region.to_owned(), operation.to_owned(), input.clone())); + self.replies + .borrow_mut() + .pop_front() + .ok_or_else(|| Report::new(PbsError::Aws("simulated failure"))) + } + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/config.rs b/crates/trusted-server-cli/src/commands/pbs/config.rs new file mode 100644 index 000000000..6a8fe37da --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/config.rs @@ -0,0 +1,509 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Path, PathBuf}; + +use error_stack::Report; +use serde::Deserialize; +use serde_json::json; +use serde_yaml_ng::Value; +use url::Url; + +use super::{Output, PbsError, Result, identifier, read_text}; + +const MAX_CONFIG_BYTES: usize = 2 * 1024 * 1024; + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Descriptor { + schema_version: u32, + environment: String, + runtime: Runtime, + aws: AwsTarget, + pbs: PbsInputs, + regions: BTreeMap, +} + +#[derive(Deserialize)] +#[serde(rename_all = "kebab-case")] +enum Runtime { + Ec2Compose, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct AwsTarget { + pub account_id: String, + pub profile: String, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct PbsInputs { + config: PathBuf, + image: String, + bindings: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Region { + #[serde(default)] + pub instance_ids: Vec, + overrides: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Binding { + pub verified_image: String, + pub source: String, + pub secrets: BTreeMap, + pub keys: BTreeMap, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct KeyBinding { + pub env: String, + pub pbs_path: Vec, + #[serde(default = "required")] + pub required: bool, +} + +fn required() -> bool { + true +} + +pub(super) struct Deployment { + pub environment: String, + pub aws: AwsTarget, + pub regions: BTreeMap, + pub bindings: BTreeMap, + pub rendered: BTreeMap, + image: String, +} + +impl Deployment { + /// Load and validate every local deployment input without calling AWS or writing files. + /// + /// # Errors + /// Rejects malformed schemas, unsupported runtimes, invalid targets, and binding conflicts. + pub fn load(path: &Path) -> Result { + let base = path.parent().unwrap_or_else(|| Path::new(".")); + let descriptor: Descriptor = decode_yaml(path)?; + let Descriptor { + schema_version, + environment, + runtime: Runtime::Ec2Compose, + aws, + pbs, + regions, + } = descriptor; + if schema_version != 1 + || !identifier(&environment) + || !valid_account(&aws.account_id) + || !identifier(&aws.profile) + || regions.is_empty() + || !pinned_image(&pbs.image) + { + return Err(invalid( + "descriptor requires version 1, explicit account/profile, regions, and digest-pinned PBS image", + )); + } + let baseline = read_yaml(&resolve(base, &pbs.config)?)?; + if !baseline.is_mapping() { + return Err(invalid("PBS configuration must be a YAML mapping")); + } + validate_yaml(&baseline)?; + let bindings: BTreeMap = match &pbs.bindings { + Some(path) => decode_yaml(&resolve(base, path)?)?, + None => BTreeMap::new(), + }; + let mut rendered = BTreeMap::new(); + for (name, region) in ®ions { + if !valid_region(name) + || region.instance_ids.len() > 100 + || region.instance_ids.iter().any(|id| !valid_instance(id)) + || region.instance_ids.iter().collect::>().len() + != region.instance_ids.len() + { + return Err(invalid("invalid region or EC2 instance identifiers")); + } + let mut config = baseline.clone(); + if let Some(path) = ®ion.overrides { + let overrides = read_yaml(&resolve(base, path)?)?; + if !overrides.is_mapping() { + return Err(invalid("regional overrides must be a YAML mapping")); + } + validate_yaml(&overrides)?; + merge(&mut config, overrides); + } + rendered.insert(name.clone(), config); + } + let deployment = Self { + environment, + aws, + regions, + bindings, + rendered, + image: pbs.image, + }; + deployment.validate_bindings()?; + Ok(deployment) + } + + /// Validate declared binding evidence and reject conflicting destinations in every region. + /// + /// # Errors + /// Rejects incomplete metadata, wrong-account ARNs, duplicate destinations and YAML conflicts. + fn validate_bindings(&self) -> Result<()> { + let mut envs = BTreeSet::new(); + let mut paths = Vec::>::new(); + let mut secret_arns = BTreeSet::new(); + for (bidder, binding) in &self.bindings { + let source = Url::parse(&binding.source) + .map_err(|_| invalid("binding source must be an HTTPS documentation URL"))?; + if !identifier(bidder) + || binding.verified_image != self.image + || binding.keys.is_empty() + || binding.secrets.is_empty() + || source.scheme() != "https" + || source.host_str().is_none() + || !source.username().is_empty() + || source.password().is_some() + { + return Err(invalid( + "binding requires matching image, documentation source, keys, and regional secret ARNs", + )); + } + for (region, arn) in &binding.secrets { + if !secret_arns.insert(arn) { + return Err(invalid( + "each secret ARN must belong to only one bidder binding", + )); + } + if !self.regions.contains_key(region) + || !secret_arn(arn, region, &self.aws.account_id) + { + return Err(invalid( + "secret ARN must match a declared region and AWS account", + )); + } + } + // Version 1 uses one binding set across regions; require explicit coverage. + if binding.secrets.len() != self.regions.len() { + return Err(invalid( + "every binding must declare a secret ARN for every deployment region", + )); + } + for (key, target) in &binding.keys { + if !identifier(key) + || !target.env.starts_with("PBS_") + || target.env.len() > 256 + || !target.env.bytes().all(|byte| { + byte.is_ascii_uppercase() || byte.is_ascii_digit() || byte == b'_' + }) + || target.pbs_path.is_empty() + || target.pbs_path.iter().any(|part| !identifier(part)) + { + return Err(invalid("invalid secret key or PBS binding destination")); + } + if !envs.insert(&target.env) + || paths.iter().any(|path| { + path.starts_with(&target.pbs_path) || target.pbs_path.starts_with(path) + }) + { + return Err(invalid( + "secret bindings must have distinct non-overlapping PBS destinations", + )); + } + paths.push(target.pbs_path.clone()); + for config in self.rendered.values() { + if occupied(config, &target.pbs_path) { + return Err(invalid( + "secret destination conflicts with a YAML value or non-mapping parent", + )); + } + } + } + } + Ok(()) + } +} + +pub(super) fn valid_account(value: &str) -> bool { + value.len() == 12 && value.bytes().all(|byte| byte.is_ascii_digit()) +} + +fn valid_region(value: &str) -> bool { + let parts: Vec<_> = value.split('-').collect(); + parts.len() >= 3 + && value.len() < 40 + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-') + && parts + .last() + .is_some_and(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn valid_instance(value: &str) -> bool { + value.strip_prefix("i-").is_some_and(|id| { + matches!(id.len(), 8 | 17) && id.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) +} + +fn pinned_image(value: &str) -> bool { + value + .rsplit_once("@sha256:") + .is_some_and(|(image, digest)| { + !image.is_empty() + && !image.contains('@') + && !image.chars().any(char::is_whitespace) + && digest.len() == 64 + && digest.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) +} + +fn secret_arn(value: &str, region: &str, account: &str) -> bool { + let parts: Vec<_> = value.splitn(7, ':').collect(); + parts.len() == 7 + && parts[0] == "arn" + && matches!(parts[1], "aws" | "aws-us-gov" | "aws-cn") + && parts[2] == "secretsmanager" + && parts[3] == region + && parts[4] == account + && parts[5] == "secret" + && !parts[6].is_empty() + && parts[6] + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"/_+=.@-".contains(&byte)) +} + +/// Resolve descriptor-relative paths, retaining support for explicit operator-owned absolute files. +/// +/// # Errors +/// Rejects empty paths. +fn resolve(base: &Path, path: &Path) -> Result { + if path.as_os_str().is_empty() { + return Err(invalid("configuration path cannot be empty")); + } + Ok(base.join(path)) +} + +/// Parse YAML into a value first to detect duplicate mapping keys before typed conversion. +/// +/// # Errors +/// Rejects invalid YAML without exposing parser snippets. +fn read_yaml(path: &Path) -> Result { + serde_yaml_ng::from_str(&read_text(path, MAX_CONFIG_BYTES)?) + .map_err(|_| invalid("cannot parse YAML/JSON input; source details withheld")) +} + +/// Decode a schema with unknown fields rejected by each descriptor type. +/// +/// # Errors +/// Rejects unsupported schema fields and types without echoing values. +fn decode_yaml(path: &Path) -> Result { + let value = read_yaml(path)?; + validate_yaml(&value)?; + serde_yaml_ng::from_value(value) + .map_err(|_| invalid("unsupported deployment/binding schema; see ts pbs documentation")) +} + +/// Require plain string-keyed data: YAML tags and merge keys obscure the effective configuration. +/// +/// # Errors +/// Rejects tagged values, merge keys, and non-string mapping keys. +fn validate_yaml(value: &Value) -> Result<()> { + match value { + Value::Mapping(map) => { + for (key, value) in map { + if key.as_str().is_none_or(|key| key == "<<") { + return Err(invalid( + "YAML mappings require string keys and explicit overrides, not merge keys", + )); + } + validate_yaml(value)?; + } + } + Value::Sequence(items) => { + for item in items { + validate_yaml(item)?; + } + } + Value::Tagged(_) => return Err(invalid("YAML tags are not supported")), + _ => {} + } + Ok(()) +} + +/// Recursively merge maps; replace arrays and scalar values as whole values. +fn merge(base: &mut Value, overrides: Value) { + match (base, overrides) { + (Value::Mapping(base), Value::Mapping(overrides)) => { + for (key, value) in overrides { + if let Some(previous) = base.get_mut(&key) { + merge(previous, value); + } else { + base.insert(key, value); + } + } + } + (base, overrides) => *base = overrides, + } +} + +fn occupied(value: &Value, path: &[String]) -> bool { + if path.is_empty() { + return true; + } + match value.as_mapping() { + Some(map) => map + .get(Value::String(path[0].clone())) + .is_some_and(|value| occupied(value, &path[1..])), + None => true, + } +} + +pub(super) fn invalid(message: &'static str) -> Report { + Report::new(PbsError::Input(message)) +} + +/// Report structural validation only, withholding resolved YAML values. +/// +/// # Errors +/// Returns an error if deterministic rendering fails. +pub(super) fn check(deployment: &Deployment) -> Result { + for value in deployment.rendered.values() { + let rendered = serde_yaml_ng::to_string(value) + .map_err(|_| invalid("cannot serialize resolved YAML"))?; + let second = serde_yaml_ng::to_string(value) + .map_err(|_| invalid("cannot serialize resolved YAML"))?; + if rendered != second { + return Err(invalid("regional rendering was not deterministic")); + } + } + Ok(Output { + failure: None, + summary: format!("PBS local checks passed for {}", deployment.environment), + details: vec![ + format!("Regions: {}", deployment.regions.keys().cloned().collect::>().join(", ")), + format!("Declared bidder secret bindings: {}", deployment.bindings.len()), + "Structural checks only: PBS startup, adapter mappings, credential availability and AWS behavior remain unverified.".to_owned(), + "No configuration files were written or AWS calls made.".to_owned(), + ], + data: json!({"environment": deployment.environment, "regions": deployment.regions.keys().collect::>(), + "binding_count": deployment.bindings.len(), "local_checks": "passed", "pbs_runtime_validation": "not_run", + "aws_validation": "not_run", "binding_evidence": "operator_declared_not_independently_verified"}), + }) +} + +#[cfg(test)] +pub(super) mod tests { + use std::fs; + + use super::*; + + pub(crate) fn fixture() -> (tempfile::TempDir, PathBuf) { + let dir = tempfile::tempdir().expect("should create temp directory"); + fs::write( + dir.path().join("pbs.yaml"), + "port: 8000\nauction_timeouts_ms:\n default: 1000\n max: 1200\nlist: [a, b]\n", + ) + .expect("should write PBS config"); + fs::write( + dir.path().join("east.yaml"), + "auction_timeouts_ms:\n default: 900\nlist: [c]\n", + ) + .expect("should write overrides"); + let image = format!("registry.example.com/pbs@sha256:{}", "a".repeat(64)); + let path = dir.path().join("deployment.yaml"); + fs::write(&path, format!("schema_version: 1\nenvironment: sandbox\nruntime: ec2-compose\naws:\n account_id: '123456789012'\n profile: pbs-sandbox\npbs:\n config: pbs.yaml\n image: {image}\n bindings: bindings.json\nregions:\n us-east-1:\n instance_ids: [i-0123456789abcdef0]\n overrides: east.yaml\n")).expect("should write descriptor"); + let bindings = json!({"examplebidder": { + "verified_image": image, "source": "https://example.com/adapter-reference", + "secrets": {"us-east-1": "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf"}, + "keys": {"api_key": {"env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", "pbs_path": ["adapters", "examplebidder", "api_key"]}} + }}); + fs::write(dir.path().join("bindings.json"), bindings.to_string()) + .expect("should write bindings"); + (dir, path) + } + + #[test] + fn resolves_relative_paths_merges_maps_and_replaces_arrays() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load descriptor"); + let east = &deployment.rendered["us-east-1"]; + assert_eq!(east["auction_timeouts_ms"]["default"].as_u64(), Some(900)); + assert_eq!(east["auction_timeouts_ms"]["max"].as_u64(), Some(1200)); + assert_eq!( + east["list"].as_sequence().expect("should have list").len(), + 1 + ); + let again = Deployment::load(&path).expect("should load again"); + assert_eq!(deployment.rendered, again.rendered); + assert_eq!( + check(&deployment).expect("should validate").data["aws_validation"], + "not_run" + ); + } + + #[test] + fn yaml_and_secret_destinations_cannot_conflict() { + let (dir, path) = fixture(); + fs::write( + dir.path().join("east.yaml"), + "adapters:\n examplebidder:\n api_key: NEVER_PRINT_ME\n", + ) + .expect("should write conflicting override"); + let error = Deployment::load(&path) + .err() + .expect("should reject conflicting source"); + assert!(error.to_string().contains("conflicts")); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + + #[test] + fn duplicate_yaml_keys_and_unsupported_runtime_fail_closed() { + let (dir, path) = fixture(); + fs::write(dir.path().join("pbs.yaml"), "port: 8000\nport: 9000\n") + .expect("should write duplicates"); + assert!(Deployment::load(&path).is_err()); + fs::write(dir.path().join("pbs.yaml"), "port: 8000\n").expect("should restore config"); + let text = fs::read_to_string(&path).expect("should read descriptor"); + fs::write(&path, text.replace("ec2-compose", "ecs")).expect("should change runtime"); + assert!(Deployment::load(&path).is_err()); + } + + #[test] + fn bidders_cannot_overwrite_each_others_secret_object() { + let (dir, path) = fixture(); + let file = dir.path().join("bindings.json"); + let mut bindings: serde_json::Value = + serde_json::from_str(&fs::read_to_string(&file).expect("should read bindings")) + .expect("should parse bindings"); + let mut other = bindings["examplebidder"].clone(); + other["keys"]["api_key"]["env"] = json!("PBS_ADAPTERS_OTHERBIDDER_API_KEY"); + other["keys"]["api_key"]["pbs_path"] = json!(["adapters", "otherbidder", "api_key"]); + bindings["otherbidder"] = other; + fs::write(file, bindings.to_string()).expect("should write bindings"); + assert!( + Deployment::load(&path).is_err(), + "complete-value writes require distinct secret ownership per bidder" + ); + } + + #[test] + fn unknown_schema_fields_and_wrong_account_secret_arns_are_rejected() { + let (dir, path) = fixture(); + let original = fs::read_to_string(&path).expect("should read descriptor"); + fs::write(&path, format!("{original}typo: value\n")).expect("should append unknown field"); + assert!(Deployment::load(&path).is_err()); + fs::write(&path, original).expect("should restore descriptor"); + let bindings = dir.path().join("bindings.json"); + let original = fs::read_to_string(&bindings).expect("should read bindings"); + fs::write(bindings, original.replace("123456789012", "999999999999")) + .expect("should change ARN"); + assert!(Deployment::load(&path).is_err()); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs new file mode 100644 index 000000000..69adfa04a --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -0,0 +1,299 @@ +use std::path::Path; + +use error_stack::Report; +use serde::{Deserialize, Deserializer}; +use serde_json::{Value, json}; + +use super::{Output, PbsError, Result, identifier, read_text}; + +/// Deliberately partial: inspecting PBS requirements must not require unrelated TS settings. +#[derive(Default, Deserialize)] +struct Source { + #[serde(default)] + integrations: Integrations, +} + +#[derive(Default, Deserialize)] +struct Integrations { + prebid: Option, +} + +#[derive(Default, Deserialize)] +struct Prebid { + enabled: Option, + server_url: Option, + account_id: Option, + timeout_ms: Option, + test_mode: Option, + debug: Option, + #[serde(default, deserialize_with = "bidder_list")] + bidders: Vec, + #[serde(default, deserialize_with = "bidder_list")] + client_side_bidders: Vec, + #[serde(default)] + bundle: Bundle, + #[serde(default)] + bid_param_override_rules: Vec, +} + +#[derive(Default, Deserialize)] +struct Bundle { + #[serde(default)] + adapters: Vec, + #[serde(default)] + user_id_modules: Vec, +} + +/// Mirror the private core list deserializer without expanding runtime defaults. +/// Parity tests compare these accepted representations with `PrebidIntegrationConfig`. +/// +/// # Errors +/// Rejects malformed lists, invalid numeric indexes, and non-string items. +fn bidder_list<'de, D: Deserializer<'de>>( + deserializer: D, +) -> std::result::Result, D::Error> { + match Value::deserialize(deserializer)? { + Value::Array(values) => { + serde_json::from_value(Value::Array(values)).map_err(serde::de::Error::custom) + } + Value::Object(values) => { + let mut indexed = Vec::with_capacity(values.len()); + for (index, value) in values { + let index = index.parse::().map_err(serde::de::Error::custom)?; + let value: String = + serde_json::from_value(value).map_err(serde::de::Error::custom)?; + indexed.push((index, value)); + } + indexed.sort_by_key(|(index, _)| *index); + Ok(indexed.into_iter().map(|(_, value)| value).collect()) + } + Value::String(value) => { + let text = value.trim(); + let bracketed = text.starts_with('[') && text.ends_with(']'); + if bracketed && let Ok(values) = serde_json::from_str::>(text) { + return Ok(values); + } + let parts = if bracketed { + text[1..text.len() - 1] + .trim() + .split(',') + .map(str::trim) + .filter(|part| !part.is_empty()) + .collect::>() + } else if text.contains(',') { + text.split(',') + .map(str::trim) + .filter(|part| !part.is_empty()) + .collect() + } else { + vec![text] + }; + parts + .into_iter() + .map(|part| { + serde_json::from_str(&format!("\"{}\"", part.replace('"', "\\\""))) + .map_err(serde::de::Error::custom) + }) + .collect() + } + _ => Err(serde::de::Error::custom( + "expected bidder list, indexed map, or list string", + )), + } +} + +/// Inspect selected local fields, preserving the source and redacting account/parameter values. +/// +/// # Errors +/// Rejects invalid TOML or malformed identifiers with sanitized messages. +pub(super) fn inspect(path: &Path) -> Result { + let text = read_text(path, 2 * 1024 * 1024)?; + let source: Source = toml::from_str(&text).map_err(|_| { + Report::new(PbsError::Input( + "cannot parse Trusted Server TOML; source details withheld", + )) + })?; + let present = source.integrations.prebid.is_some(); + let prebid = source.integrations.prebid.unwrap_or_default(); + for name in prebid + .bidders + .iter() + .chain(&prebid.client_side_bidders) + .chain(&prebid.bundle.adapters) + .chain(&prebid.bundle.user_id_modules) + { + if !identifier(name) { + return Err(Report::new(PbsError::Input( + "invalid bidder or identity-module identifier", + ))); + } + } + let requirements: Vec<_> = prebid + .bidders + .iter() + .map(|bidder| { + json!({ + "bidder": bidder, + "source_key": "integrations.prebid.bidders", + "host_secret_requirement": "unresolved", + "partner_authorization": "unresolved" + }) + }) + .collect(); + let warnings = [ + "Local file only: confirm environment, remote configuration, and request-time overrides.", + "Omitted fields/defaults are not expanded; empty candidate lists are not proof of no demand.", + "Disabled integrations and browser bundle adapters do not authorize PBS activation.", + "Host secret requirements need adapter metadata verified against the selected PBS release.", + ]; + let mut details = vec![ + format!( + "Prebid section present: {present}; enabled explicitly: {:?}", + prebid.enabled + ), + format!("Server bidder candidates: {}", prebid.bidders.join(", ")), + format!( + "Client-side bidders: {}", + prebid.client_side_bidders.join(", ") + ), + format!( + "Browser bundle adapters: {}", + prebid.bundle.adapters.join(", ") + ), + format!( + "Bid-parameter rules: {}; values withheld", + prebid.bid_param_override_rules.len() + ), + ]; + details.extend(warnings.iter().map(|warning| (*warning).to_owned())); + Ok(Output { + failure: None, + summary: "Local PBS requirements discovery; no files changed or AWS calls made".to_owned(), + details, + data: json!({ + "source": path, + "source_section": "integrations.prebid", + "section_present": present, + "enabled_explicit": prebid.enabled, + "server_url_configured": prebid.server_url.is_some(), + "account_id_configured": prebid.account_id.is_some(), + "timeout_ms_explicit": prebid.timeout_ms, + "test_mode_explicit": prebid.test_mode, + "debug_explicit": prebid.debug, + "server_bidder_candidates": requirements, + "client_side_bidders": prebid.client_side_bidders, + "bundle_adapters": prebid.bundle.adapters, + "identity_modules": prebid.bundle.user_id_modules, + "bid_param_override_rule_count": prebid.bid_param_override_rules.len(), + "warnings": warnings + }), + }) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + + #[test] + fn discovery_classifies_bidders_redacts_values_and_preserves_source() { + let dir = tempfile::tempdir().expect("should create temp directory"); + let path = dir.path().join("trusted-server.toml"); + let source = r#" +[integrations.prebid] +enabled = false +server_url = "https://user:NEVER_PRINT_ME@pbs.example.com/path?token=NEVER_PRINT_ME" +account_id = "NEVER_PRINT_ME" +bidders = ["serverbidder"] +client_side_bidders = ["browserbidder"] +[[integrations.prebid.bid_param_override_rules]] +set = { placementId = "NEVER_PRINT_ME" } +[integrations.prebid.bundle] +adapters = ["bundlebidder"] +user_id_modules = ["sharedIdSystem"] +"#; + fs::write(&path, source).expect("should write fixture"); + let report = inspect(&path).expect("should inspect config"); + assert_eq!(report.data["enabled_explicit"], false); + assert_eq!( + report.data["server_bidder_candidates"][0]["bidder"], + "serverbidder" + ); + assert_eq!(report.data["client_side_bidders"][0], "browserbidder"); + assert_eq!(report.data["bundle_adapters"][0], "bundlebidder"); + assert_eq!( + report.data["server_bidder_candidates"][0]["host_secret_requirement"], + "unresolved" + ); + for json in [false, true] { + let mut output = Vec::new(); + report + .write(json, &mut output) + .expect("should render report"); + assert!(!String::from_utf8_lossy(&output).contains("NEVER_PRINT_ME")); + } + assert_eq!( + fs::read_to_string(path).expect("should read fixture"), + source + ); + } + + #[test] + fn invalid_toml_never_exposes_parser_snippets() { + let dir = tempfile::tempdir().expect("should create temp directory"); + let path = dir.path().join("invalid.toml"); + fs::write(&path, "secret = NEVER_PRINT_ME").expect("should write fixture"); + let error = inspect(&path).err().expect("should reject invalid TOML"); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + + #[test] + fn accepts_the_runtime_bidder_list_encodings() { + for input in [ + "['examplebidder', 'otherbidder']", + "'examplebidder,otherbidder'", + "'[examplebidder, otherbidder]'", + "'[\"examplebidder\", \"otherbidder\"]'", + "{ '10' = 'otherbidder', '2' = 'examplebidder' }", + ] { + let text = format!( + "[integrations.prebid]\nserver_url='https://pbs.example.com'\nbidders={input}\nclient_side_bidders={input}\n" + ); + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write(file.path(), &text).expect("should write config"); + let source: toml::Value = toml::from_str(&text).expect("should parse source"); + let runtime: trusted_server_core::integrations::prebid::PrebidIntegrationConfig = + source["integrations"]["prebid"] + .clone() + .try_into() + .expect("runtime should accept encoding"); + let output = inspect(file.path()).expect("inspect should accept runtime encoding"); + let candidates: Vec<_> = output.data["server_bidder_candidates"] + .as_array() + .expect("should report candidates") + .iter() + .map(|candidate| { + candidate["bidder"] + .as_str() + .expect("should identify bidder") + }) + .collect(); + assert_eq!(candidates, runtime.bidders); + assert_eq!( + output.data["client_side_bidders"], + json!(runtime.client_side_bidders) + ); + } + } + + #[test] + fn missing_section_stays_unresolved_instead_of_enabling_defaults() { + let dir = tempfile::tempdir().expect("should create temp directory"); + let path = dir.path().join("empty.toml"); + fs::write(&path, "").expect("should write fixture"); + let output = inspect(&path).expect("should inspect empty file"); + assert_eq!(output.data["section_present"], false); + assert!(output.data["enabled_explicit"].is_null()); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/mod.rs b/crates/trusted-server-cli/src/commands/pbs/mod.rs new file mode 100644 index 000000000..548fa1be9 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/mod.rs @@ -0,0 +1,237 @@ +//! Experimental PBS operator commands, independent of Trusted Server deployment. + +mod aws; +mod config; +mod inspect; +mod secrets; +mod status; + +use std::fs::File; +use std::io::{self, BufRead, IsTerminal, Read, Write}; +use std::path::{Path, PathBuf}; + +use clap::{Args, Subcommand}; +use derive_more::Display; +use error_stack::Report; +use serde_json::Value; + +use aws::AwsCli; +use config::Deployment; + +/// Arguments for the experimental `ts pbs` namespace. +#[derive(Debug, Args)] +pub(crate) struct PbsArgs { + /// Emit a nonsecret JSON report instead of a human summary. + #[arg(long, global = true)] + json: bool, + #[command(subcommand)] + command: PbsCommand, +} + +#[derive(Debug, Subcommand)] +enum PbsCommand { + /// Inspect local Trusted Server configuration without modifying it or contacting AWS. + Inspect { + /// Explicit source file; omitted fields/defaults and remote overrides remain unresolved. + #[arg(long, default_value = "trusted-server.toml")] + config: PathBuf, + }, + /// Validate declared PBS inputs and regional YAML merges locally, without AWS access. + Check(TargetArgs), + /// Read EC2 infrastructure status, not PBS health or the installed release. + Status(TargetArgs), + /// Manage values for existing, explicitly declared Secrets Manager secrets. + #[command(subcommand)] + Secrets(SecretCommand), +} + +#[derive(Debug, Args)] +struct TargetArgs { + /// Deployment descriptor; paths inside it are relative to this file. + #[arg(long)] + deployment: PathBuf, +} + +#[derive(Debug, Subcommand)] +enum SecretCommand { + /// Write a complete JSON credential payload; does not deploy or rotate partner credentials. + Set(secrets::SetArgs), +} + +/// Sanitized errors: source parser output and AWS stderr can contain credential values. +#[derive(Debug, Display)] +pub(crate) enum PbsError { + #[display("invalid PBS input: {_0}")] + Input(&'static str), + #[display("PBS I/O failed: {_0}")] + Io(&'static str), + #[display("AWS operation failed: {_0}; provider output withheld")] + Aws(&'static str), + #[display("AWS account does not match the deployment descriptor; no further calls made")] + AccountMismatch, + #[display("operation was not confirmed; no secret value was written")] + NotConfirmed, +} + +impl std::error::Error for PbsError {} + +pub(super) type Result = std::result::Result>; + +pub(super) struct Output { + summary: String, + details: Vec, + data: Value, + failure: Option, +} + +impl Output { + /// Write only explicitly constructed nonsecret report fields. + /// + /// # Errors + /// Returns a sanitized error if output cannot be written or serialized. + fn write(&self, json: bool, out: &mut dyn Write) -> Result<()> { + if json { + serde_json::to_writer_pretty(&mut *out, &self.data) + .map_err(|_| Report::new(PbsError::Io("cannot write JSON report")))?; + writeln!(out).map_err(|_| Report::new(PbsError::Io("cannot write report")))?; + } else { + writeln!(out, "{}", self.summary) + .map_err(|_| Report::new(PbsError::Io("cannot write report")))?; + for detail in &self.details { + writeln!(out, " {detail}") + .map_err(|_| Report::new(PbsError::Io("cannot write report")))?; + } + } + Ok(()) + } +} + +/// Execute PBS commands. Only explicitly selected cloud commands construct an AWS client. +/// +/// # Errors +/// Returns sanitized validation, I/O, identity, confirmation, or AWS errors. +pub(crate) fn run(args: &PbsArgs) -> Result<()> { + let output = match &args.command { + PbsCommand::Inspect { config } => inspect::inspect(config)?, + PbsCommand::Check(target) => config::check(&Deployment::load(&target.deployment)?)?, + PbsCommand::Status(target) => { + let deployment = Deployment::load(&target.deployment)?; + Terminal.notice(&format!( + "Read EC2 status: environment={}; account={}; profile={}; regions={}", + deployment.environment, + deployment.aws.account_id, + deployment.aws.profile, + deployment + .regions + .keys() + .cloned() + .collect::>() + .join(", ") + ))?; + status::status(&deployment, &AwsCli)? + } + PbsCommand::Secrets(SecretCommand::Set(set)) => { + let deployment = Deployment::load(&set.deployment)?; + secrets::set(set, &deployment, &AwsCli, &mut Terminal)? + } + }; + output.write(args.json, &mut io::stdout().lock())?; + match output.failure { + Some(error) => Err(Report::new(error)), + None => Ok(()), + } +} + +/// Read bounded local input without including file contents in error reports. +/// +/// # Errors +/// Returns an I/O error or rejects oversized/non-UTF-8 input. +pub(super) fn read_text(path: &Path, limit: usize) -> Result { + let file = File::open(path).map_err(|_| Report::new(PbsError::Io("cannot open input file")))?; + read_bounded(file, limit) +} + +/// Read bounded input, including piped credential JSON. +/// +/// # Errors +/// Rejects oversized/non-UTF-8 input and read failures without disclosing contents. +pub(super) fn read_bounded(input: impl Read, limit: usize) -> Result { + let mut text = String::new(); + input + .take(limit as u64 + 1) + .read_to_string(&mut text) + .map_err(|_| Report::new(PbsError::Io("cannot read UTF-8 input")))?; + if text.len() > limit { + return Err(Report::new(PbsError::Input("input exceeds size limit"))); + } + Ok(text) +} + +/// Keep report identifiers bounded and free of terminal control sequences. +pub(super) fn identifier(value: &str) -> bool { + !value.is_empty() + && value.len() <= 128 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-')) +} + +/// Operator interaction is injectable so tests cannot accidentally read a terminal. +pub(super) trait Interaction { + /// Read a complete secret JSON object without echoing it. + /// + /// # Errors + /// Returns an error when an interactive terminal is unavailable. + fn secret(&mut self) -> Result; + /// Request approval of a nonsecret target description. + /// + /// # Errors + /// Returns an error on unavailable terminal or failed I/O. + fn confirm(&mut self, target: &str) -> Result; + /// Display nonsecret target and retry information separately from JSON stdout. + /// + /// # Errors + /// Returns an error if the notice cannot be written. + fn notice(&mut self, message: &str) -> Result<()>; +} + +struct Terminal; + +impl Interaction for Terminal { + fn secret(&mut self) -> Result { + if !io::stdin().is_terminal() { + return Err(Report::new(PbsError::Input( + "use --file or --stdin for noninteractive secret input", + ))); + } + rpassword::prompt_password("Secret JSON (hidden): ") + .map_err(|_| Report::new(PbsError::Io("cannot read hidden secret input"))) + } + + fn confirm(&mut self, target: &str) -> Result { + if !io::stdin().is_terminal() { + return Err(Report::new(PbsError::Input( + "automation requires --yes and --request-token", + ))); + } + self.notice(target)?; + self.notice("Write this secret version? Type yes to confirm:")?; + let line = read_bounded( + io::stdin() + .lock() + .lines() + .next() + .transpose() + .map_err(|_| Report::new(PbsError::Io("cannot read confirmation")))? + .unwrap_or_default() + .as_bytes(), + 16, + )?; + Ok(line.trim() == "yes") + } + + fn notice(&mut self, message: &str) -> Result<()> { + writeln!(io::stderr().lock(), "{message}") + .map_err(|_| Report::new(PbsError::Io("cannot write operator notice"))) + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/secrets.rs b/crates/trusted-server-cli/src/commands/pbs/secrets.rs new file mode 100644 index 000000000..d7c473a7b --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/secrets.rs @@ -0,0 +1,393 @@ +use std::collections::BTreeMap; +use std::io; +use std::path::PathBuf; + +use clap::Args; +use error_stack::Report; +use serde::de::{MapAccess, Visitor}; +use serde::{Deserialize, Deserializer}; +use serde_json::{Value, json}; +use uuid::Uuid; + +use super::aws::{Aws, verify_identity}; +use super::config::{Binding, Deployment, invalid}; +use super::{Interaction, Output, PbsError, Result, read_bounded, read_text}; + +const MAX_SECRET_BYTES: usize = 65_536; + +/// No argument accepts a secret value. File/stdin and terminal input carry the payload. +#[derive(Debug, Args)] +pub(super) struct SetArgs { + /// Bidder identifier declared in the binding file. + bidder: String, + #[arg(long)] + pub deployment: PathBuf, + /// Exactly one declared region; replicas cannot be written independently. + #[arg(long)] + region: String, + /// Read a complete JSON string-valued object from this file; never changes the file. + #[arg(long, conflicts_with = "stdin")] + file: Option, + /// Read the JSON object from stdin; requires --yes and --request-token. + #[arg(long)] + stdin: bool, + /// Approve the declared target noninteractively, for an externally approved automation job. + #[arg(long, requires = "request_token")] + yes: bool, + /// Stable UUID for retries of this logical write. Reuse with identical values only. + #[arg(long)] + request_token: Option, +} + +struct Payload(BTreeMap); + +impl<'de> Deserialize<'de> for Payload { + fn deserialize>(deserializer: D) -> std::result::Result { + struct UniqueKeys; + impl<'de> Visitor<'de> for UniqueKeys { + type Value = Payload; + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a JSON object with unique keys and string values") + } + fn visit_map>( + self, + mut map: M, + ) -> std::result::Result { + let mut values = BTreeMap::new(); + while let Some((key, value)) = map.next_entry::()? { + if values.insert(key, value).is_some() { + return Err(serde::de::Error::custom("duplicate credential key")); + } + } + Ok(Payload(values)) + } + } + deserializer.deserialize_map(UniqueKeys) + } +} + +/// Validate a complete payload without returning secret-bearing parser diagnostics. +/// +/// # Errors +/// Rejects duplicate/unknown/missing keys, non-string values, empty required values and oversized JSON. +fn validate_payload(text: &str, binding: &Binding) -> Result { + if text.len() > MAX_SECRET_BYTES { + return Err(invalid("secret JSON exceeds Secrets Manager size limit")); + } + let payload: Payload = serde_json::from_str(text) + .map_err(|_| invalid("secret input must be a JSON object with unique keys and string values; contents withheld"))?; + if payload.0.keys().any(|key| !binding.keys.contains_key(key)) { + return Err(invalid("secret JSON includes undeclared keys")); + } + for (key, rule) in &binding.keys { + if rule.required && payload.0.get(key).is_none_or(String::is_empty) { + return Err(invalid("secret JSON is missing a required nonempty string")); + } + } + if payload.0.is_empty() { + return Err(invalid( + "secret JSON must contain at least one declared key", + )); + } + let encoded = + serde_json::to_string(&payload.0).map_err(|_| invalid("cannot encode credential JSON"))?; + if encoded.len() > MAX_SECRET_BYTES { + return Err(invalid( + "encoded credential JSON exceeds Secrets Manager size limit", + )); + } + Ok(encoded) +} + +/// Set a declared credential version after identity, metadata, input and approval checks. +/// +/// # Errors +/// Rejects unapproved input, account/secret mismatches, replicas, deletion, malformed values and AWS errors. +pub(super) fn set( + args: &SetArgs, + deployment: &Deployment, + aws: &dyn Aws, + interaction: &mut dyn Interaction, +) -> Result { + if args.yes && args.request_token.is_none() || args.stdin && !args.yes { + return Err(invalid( + "noninteractive writes require --yes and --request-token", + )); + } + let binding = deployment + .bindings + .get(&args.bidder) + .ok_or_else(|| invalid("bidder has no declared secret binding"))?; + let arn = binding + .secrets + .get(&args.region) + .ok_or_else(|| invalid("bidder has no secret in the selected region"))?; + let target = format!( + "Environment: {}; account: {}; profile: {}; region: {}; bidder: {}; secret: {arn}", + deployment.environment, + deployment.aws.account_id, + deployment.aws.profile, + args.region, + args.bidder + ); + interaction.notice(&target)?; + verify_identity(deployment, &args.region, aws)?; + let metadata = aws.call( + &deployment.aws, + &args.region, + "secretsmanager", + "describe-secret", + &json!({"SecretId": arn}), + )?; + if metadata.get("ARN").and_then(Value::as_str) != Some(arn.as_str()) + || metadata + .get("DeletedDate") + .is_some_and(|value| !value.is_null()) + { + return Err(invalid( + "declared secret identity differs or is scheduled for deletion", + )); + } + if metadata + .get("PrimaryRegion") + .and_then(Value::as_str) + .is_some_and(|region| region != args.region) + { + return Err(invalid( + "selected secret is a replica; write to its approved primary region instead", + )); + } + let text = if let Some(path) = &args.file { + read_text(path, MAX_SECRET_BYTES)? + } else if args.stdin { + read_bounded(io::stdin().lock(), MAX_SECRET_BYTES)? + } else { + interaction.secret()? + }; + let payload = validate_payload(&text, binding)?; + let token = args.request_token.unwrap_or_else(Uuid::new_v4).to_string(); + interaction.notice(&format!("Request token for identical retries: {token}"))?; + if !args.yes && !interaction.confirm(&target)? { + return Err(Report::new(PbsError::NotConfirmed)); + } + let response = aws.call( + &deployment.aws, + &args.region, + "secretsmanager", + "put-secret-value", + &json!({"SecretId": arn, "ClientRequestToken": token, "SecretString": payload}), + )?; + if response.get("ARN").and_then(Value::as_str) != Some(arn.as_str()) + || response.get("VersionId").and_then(Value::as_str) != Some(token.as_str()) + { + return Err(Report::new(PbsError::Aws( + "write response could not be verified; outcome uncertain, retain the request token", + ))); + } + Ok(Output { + failure: None, + summary: format!("Stored credential version for {} in {}; no containers changed", args.bidder, args.region), + details: vec![format!("Version / retry token: {token}"), + "Confirm replica readiness and separately deploy consuming containers before revoking old partner credentials.".to_owned()], + data: json!({"environment": deployment.environment, "account_id": deployment.aws.account_id, + "region": args.region, "bidder": args.bidder, "secret_arn": arn, "version_id": token, + "deployed": false, "partner_credential_rotated": false, + "regions_requiring_deployment_review": binding.secrets.keys().collect::>()}), + }) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + use crate::commands::pbs::aws::tests::FakeAws; + use crate::commands::pbs::config::tests::fixture; + + const ARN: &str = "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf"; + const TOKEN: &str = "11111111-2222-4333-8444-555555555555"; + + struct Prompt { + approve: bool, + notices: Vec, + } + + impl Interaction for Prompt { + fn secret(&mut self) -> Result { + Ok("{\"api_key\":\"dummy\"}".to_owned()) + } + fn confirm(&mut self, _target: &str) -> Result { + Ok(self.approve) + } + fn notice(&mut self, text: &str) -> Result<()> { + self.notices.push(text.to_owned()); + Ok(()) + } + } + + fn args(path: &std::path::Path) -> SetArgs { + SetArgs { + bidder: "examplebidder".to_owned(), + deployment: path.to_path_buf(), + region: "us-east-1".to_owned(), + file: None, + stdin: false, + yes: false, + request_token: Some(Uuid::parse_str(TOKEN).expect("should parse token")), + } + } + + fn replies() -> Vec { + vec![ + json!({"Account": "123456789012"}), + json!({"ARN": ARN}), + json!({"ARN": ARN, "VersionId": TOKEN}), + ] + } + + #[test] + fn wrong_account_and_declined_confirmation_never_write() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![json!({"Account": "999999999999"})]); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 1); + let aws = FakeAws::new(replies()); + prompt.approve = false; + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 2); + } + + #[test] + fn special_characters_round_trip_without_output_or_source_file_changes() { + let (dir, path) = fixture(); + let secret = "dummy-$\"\nline\\end"; + let original = json!({"api_key": secret}).to_string(); + let input = dir.path().join("secret.json"); + fs::write(&input, &original).expect("should write secret fixture"); + let deployment = Deployment::load(&path).expect("should load deployment"); + let mut arguments = args(&path); + arguments.file = Some(input.clone()); + arguments.yes = true; + let aws = FakeAws::new(replies()); + let mut prompt = Prompt { + approve: false, + notices: vec![], + }; + let report = + set(&arguments, &deployment, &aws, &mut prompt).expect("should write approved secret"); + let calls = aws.calls.borrow(); + assert_eq!(calls[2].1, "put-secret-value"); + let stored: Value = serde_json::from_str( + calls[2].2["SecretString"] + .as_str() + .expect("should have encoded JSON"), + ) + .expect("should parse payload"); + assert_eq!(stored["api_key"], secret); + assert_eq!(calls[2].2["ClientRequestToken"], TOKEN); + for json in [false, true] { + let mut output = Vec::new(); + report + .write(json, &mut output) + .expect("should render report"); + assert!(!String::from_utf8_lossy(&output).contains("dummy-")); + } + assert!(!prompt.notices.join("\n").contains("dummy-")); + assert_eq!( + fs::read_to_string(input).expect("should read input"), + original + ); + assert_eq!(report.data["deployed"], false); + } + + #[test] + fn replicas_and_missing_confirmation_inputs_fail_closed() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![ + json!({"Account": "123456789012"}), + json!({"ARN": ARN, "PrimaryRegion": "us-west-2"}), + ]); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 2); + let mut arguments = args(&path); + arguments.yes = true; + arguments.request_token = None; + let aws = FakeAws::new(vec![]); + assert!(set(&arguments, &deployment, &aws, &mut prompt).is_err()); + assert!(aws.calls.borrow().is_empty()); + } + + #[test] + fn undeclared_region_and_mismatched_secret_metadata_never_write() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let mut arguments = args(&path); + arguments.region = "us-west-2".to_owned(); + let aws = FakeAws::new(vec![]); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + assert!(set(&arguments, &deployment, &aws, &mut prompt).is_err()); + assert!(aws.calls.borrow().is_empty()); + for metadata in [ + json!({"ARN": "arn:aws:secretsmanager:us-east-1:123456789012:secret:other-AbCdEf"}), + json!({"ARN": ARN, "DeletedDate": 1234567890}), + ] { + let aws = FakeAws::new(vec![json!({"Account": "123456789012"}), metadata]); + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 2); + } + } + + #[test] + fn payload_rejects_missing_extra_duplicate_and_nonstring_keys_without_leaks() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let binding = &deployment.bindings["examplebidder"]; + for payload in [ + "{}", + "{\"api_key\":null}", + "{\"api_key\":\"\"}", + "{\"api_key\":\"dummy\",\"extra\":\"NEVER_PRINT_ME\"}", + "{\"api_key\":\"NEVER_PRINT_ME\",\"api_key\":\"second\"}", + "NEVER_PRINT_ME", + ] { + let error = validate_payload(payload, binding) + .expect_err("should reject invalid credential payload"); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + } + + #[test] + fn retries_reuse_the_same_token_and_payload_and_failures_are_reported() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let mut all = replies(); + all.extend(replies()); + let aws = FakeAws::new(all); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + for _ in 0..2 { + set(&args(&path), &deployment, &aws, &mut prompt).expect("should submit retry"); + } + assert_eq!(aws.calls.borrow()[2].2, aws.calls.borrow()[5].2); + let aws = FakeAws::new(vec![ + json!({"Account": "123456789012"}), + json!({"ARN": ARN}), + ]); + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/status.rs b/crates/trusted-server-cli/src/commands/pbs/status.rs new file mode 100644 index 000000000..9812a8152 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/status.rs @@ -0,0 +1,183 @@ +use serde_json::{Value, json}; + +use super::aws::{Aws, verify_identity}; +use super::config::{Deployment, invalid}; +use super::{Output, PbsError, Result}; + +/// Read only the explicitly declared EC2 instances. Never infer PBS health from EC2 health. +/// +/// # Errors +/// Rejects missing instance targets, wrong accounts, and failed identity verification. +/// Resource-query failures are included as unknown in a partial report with a failing exit status. +pub(super) fn status(deployment: &Deployment, aws: &dyn Aws) -> Result { + if deployment + .regions + .values() + .any(|region| region.instance_ids.is_empty()) + { + return Err(invalid( + "status requires explicit instance_ids in every selected descriptor region", + )); + } + let mut regions = Vec::new(); + let mut details = Vec::new(); + let mut failed = false; + for (region, target) in &deployment.regions { + verify_identity(deployment, region, aws)?; + let response = aws.call( + &deployment.aws, + region, + "ec2", + "describe-instance-status", + &json!({"InstanceIds": target.instance_ids, "IncludeAllInstances": true}), + ); + let statuses = response + .as_ref() + .ok() + .and_then(|value| value.get("InstanceStatuses")) + .and_then(Value::as_array); + let mut instances = Vec::new(); + for id in &target.instance_ids { + let matching: Vec<_> = statuses + .into_iter() + .flatten() + .filter(|value| { + value.get("InstanceId").and_then(Value::as_str) == Some(id.as_str()) + }) + .collect(); + let state = if matching.len() == 1 { + Some(matching[0]) + } else { + None + }; + let instance_state = allowed( + state, + "/InstanceState/Name", + &[ + "pending", + "running", + "shutting-down", + "terminated", + "stopping", + "stopped", + ], + ); + let instance_health = allowed( + state, + "/InstanceStatus/Status", + &[ + "ok", + "impaired", + "initializing", + "insufficient-data", + "not-applicable", + ], + ); + let system_health = allowed( + state, + "/SystemStatus/Status", + &[ + "ok", + "impaired", + "initializing", + "insufficient-data", + "not-applicable", + ], + ); + failed |= state.is_none() + || instance_state == "unknown" + || instance_health == "unknown" + || system_health == "unknown"; + details.push(format!("{region} {id}: state={instance_state}, EC2 instance={instance_health}, system={system_health}; PBS health/release unknown")); + instances.push(json!({"instance_id": id, "state": instance_state, "instance_health": instance_health, + "system_health": system_health, "pbs_health": "unknown", "release_id": null, "secret_versions": null})); + } + regions.push(json!({"region": region, "query": if statuses.is_some() { "completed" } else { "failed_or_invalid" }, "instances": instances})); + } + Ok(Output { + summary: format!( + "EC2 infrastructure status for {}; PBS runtime status is not implemented", + deployment.environment + ), + details, + data: json!({"environment": deployment.environment, "account_id": deployment.aws.account_id, + "complete": !failed, "regions": regions}), + failure: failed.then_some(PbsError::Aws("status report is incomplete")), + }) +} + +fn allowed(record: Option<&Value>, pointer: &str, choices: &[&'static str]) -> &'static str { + let value = record + .and_then(|record| record.pointer(pointer)) + .and_then(Value::as_str); + choices + .iter() + .copied() + .find(|choice| Some(*choice) == value) + .unwrap_or("unknown") +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::commands::pbs::aws::tests::FakeAws; + use crate::commands::pbs::config::tests::fixture; + + #[test] + fn queries_only_declared_instances_and_does_not_claim_pbs_health() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![ + json!({"Account": "123456789012"}), + json!({"InstanceStatuses": [{ + "InstanceId": "i-0123456789abcdef0", "InstanceState": {"Name": "running"}, + "InstanceStatus": {"Status": "ok"}, "SystemStatus": {"Status": "ok"} + }]}), + ]); + let output = status(&deployment, &aws).expect("should read status"); + assert!(output.failure.is_none()); + assert_eq!( + output.data["regions"][0]["instances"][0]["pbs_health"], + "unknown" + ); + assert!(output.data["regions"][0]["instances"][0]["release_id"].is_null()); + assert_eq!( + aws.calls.borrow()[1].2["InstanceIds"], + json!(["i-0123456789abcdef0"]) + ); + } + + #[test] + fn absent_and_failed_queries_produce_incomplete_unknown_reports() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + for replies in [ + vec![json!({"Account": "123456789012"})], + vec![ + json!({"Account": "123456789012"}), + json!({"InstanceStatuses": []}), + ], + ] { + let aws = FakeAws::new(replies); + let output = status(&deployment, &aws).expect("should preserve partial report"); + assert!(output.failure.is_some()); + assert_eq!(output.data["complete"], false); + assert_eq!( + output.data["regions"][0]["instances"][0]["state"], + "unknown" + ); + } + } + + #[test] + fn wrong_account_prevents_resource_queries() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![json!({"Account": "999999999999"})]); + let error = status(&deployment, &aws) + .err() + .expect("should reject account mismatch"); + assert!(matches!(error.current_context(), PbsError::AccountMismatch)); + assert_eq!(aws.calls.borrow().len(), 1); + } +} diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 44ddbafe7..7fe1be374 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -10,6 +10,7 @@ use trusted_server_core::config::TrustedServerAppConfig; use crate::commands::audit::{AuditArgs, run_audit}; use crate::commands::config::ad_templates::{AdTemplatesCommand, run_ad_templates}; use crate::commands::config::init::{ConfigInitArgs, run_config_init}; +use crate::commands::pbs::{self, PbsArgs}; use crate::prebid_bundle::{NpmPrebidBundleGenerator, PrebidBundleArgs, run_bundle}; #[derive(Debug, Parser)] @@ -36,6 +37,8 @@ enum Command { Deploy(DeployArgs), /// Probe a deployed version until it reports healthy. Healthcheck(HealthcheckArgs), + /// Experimental self-hosted Prebid Server configuration and AWS operations. + Pbs(PbsArgs), /// Trusted Server Prebid commands. Prebid(PrebidArgs), /// Provision platform resources through a target adapter. @@ -143,6 +146,9 @@ fn dispatch(args: Args) -> Result { Command::Healthcheck(args) => { edgezero_cli::run_healthcheck(&args).map(|()| RunOutcome::Success) } + Command::Pbs(args) => pbs::run(&args) + .map(|()| RunOutcome::Success) + .map_err(|error| error.current_context().to_string()), Command::Prebid(prebid) => { let mut generator = NpmPrebidBundleGenerator; let mut stdout = std::io::stdout(); @@ -187,6 +193,77 @@ mod tests { ); } + #[test] + fn parses_pbs_inspect() { + assert!( + Args::try_parse_from([ + "ts", + "pbs", + "inspect", + "--config", + "trusted-server.toml", + "--json" + ]) + .is_ok(), + "should accept the standalone PBS command namespace" + ); + } + + #[test] + fn pbs_rejects_ambiguous_secret_inputs_and_unimplemented_commands() { + for arguments in [ + vec!["ts", "pbs", "deploy"], + vec!["ts", "pbs", "rollback", "--release", "example"], + vec!["ts", "pbs", "check"], + vec![ + "ts", + "pbs", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--yes", + ], + vec![ + "ts", + "pbs", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--file", + "secret.json", + "--stdin", + ], + ] { + assert!( + Args::try_parse_from(arguments).is_err(), + "should reject unsafe or unsupported command shape" + ); + } + assert!( + Args::try_parse_from([ + "ts", + "pbs", + "check", + "--deployment", + "deployment.yaml", + "--json" + ]) + .is_ok() + ); + assert!( + Args::try_parse_from(["ts", "prebid", "bundle", "--config", "trusted-server.toml"]) + .is_ok() + ); + } + #[test] fn parses_active_version() { let args = parse(&[ diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs new file mode 100644 index 000000000..38c7d4e23 --- /dev/null +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -0,0 +1,254 @@ +//! Exercise the actual CLI and process adapter against a local fake AWS executable. +#![cfg(unix)] + +use std::fs; +use std::os::unix::fs::PermissionsExt; +use std::path::Path; +use std::process::{Command, Output}; + +use serde_json::{Value, json}; +use tempfile::TempDir; + +const TOKEN: &str = "11111111-2222-4333-8444-555555555555"; + +fn fixture() -> TempDir { + let dir = tempfile::tempdir().expect("should create fixture directory"); + let source = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/pbs"); + for name in ["deployment.yaml", "pbs.yaml", "east.yaml", "bindings.json"] { + fs::copy(source.join(name), dir.path().join(name)).expect("should copy fixture"); + } + let fake = dir.path().join("aws"); + fs::write( + &fake, + r#"#!/usr/bin/env python3 +import json, os, pathlib, stat, sys +args = sys.argv[1:] +root = pathlib.Path(os.environ['PBS_FAKE_ROOT']) +with (root / 'calls').open('a') as log: + log.write(json.dumps(args) + '\n') +assert '--profile' in args and args[args.index('--profile')+1] == 'pbs-sandbox' +if 'configure' in args: + if os.environ.get('PBS_FAKE_HISTORY') == 'enabled': + print('enabled') + else: + print('disabled') + sys.exit(0) +assert args[args.index('--region')+1] == 'us-east-1' +assert os.environ.get('AWS_IGNORE_CONFIGURED_ENDPOINT_URLS') == 'true' +path = pathlib.Path(args[args.index('--cli-input-json')+1].removeprefix('file://')) +assert stat.S_IMODE(path.stat().st_mode) == 0o600 +with (root / 'payload_paths').open('a') as paths: + paths.write(str(path) + '\n') +request = json.loads(path.read_text()) +arn = 'arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf' +if 'get-caller-identity' in args: + print(json.dumps({'Account': os.environ.get('PBS_FAKE_ACCOUNT', '123456789012')})) +elif 'describe-secret' in args: + print(json.dumps({'ARN': arn})) +elif 'put-secret-value' in args: + if os.environ.get('PBS_FAKE_FAILURE') == 'yes': + print(request['SecretString'], file=sys.stderr) + sys.exit(1) + (root / 'captured_request.json').write_text(json.dumps(request)) + print(json.dumps({'ARN': arn, 'VersionId': request['ClientRequestToken']})) +elif 'describe-instance-status' in args: + print(json.dumps({'InstanceStatuses': []})) +else: + sys.exit(2) +"#, + ) + .expect("should write fake AWS executable"); + fs::set_permissions(fake, fs::Permissions::from_mode(0o700)) + .expect("should set executable mode"); + dir +} + +fn command(dir: &Path) -> Command { + let mut command = Command::new(env!("CARGO_BIN_EXE_ts")); + command + .current_dir(dir) + .env("PBS_FAKE_ROOT", dir) + .env( + "PATH", + format!( + "{}:{}", + dir.display(), + std::env::var("PATH").unwrap_or_default() + ), + ) + .env_remove("AWS_ACCESS_KEY_ID") + .env_remove("AWS_SECRET_ACCESS_KEY") + .env_remove("AWS_SESSION_TOKEN") + .env("AWS_EC2_METADATA_DISABLED", "true"); + command +} + +fn secret_command(dir: &Path) -> Command { + let mut command = command(dir); + command.args([ + "pbs", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--file", + "secret.json", + "--yes", + "--request-token", + TOKEN, + "--json", + ]); + command +} + +fn assert_no_secret(output: &Output) { + assert!(!String::from_utf8_lossy(&output.stdout).contains("DUMMY_SECRET")); + assert!(!String::from_utf8_lossy(&output.stderr).contains("DUMMY_SECRET")); +} + +fn assert_payload_cleanup(dir: &Path) { + let paths = + fs::read_to_string(dir.join("payload_paths")).expect("should capture payload paths"); + for path in paths.lines() { + assert!( + !Path::new(path).exists(), + "should remove tool-created request payloads" + ); + } +} + +#[test] +fn local_commands_never_execute_aws_and_preserve_the_source() { + let dir = fixture(); + let toml = "[integrations.prebid]\nenabled=false\nbidders=['examplebidder']\naccount_id='DUMMY_SECRET'\n"; + fs::write(dir.path().join("trusted-server.toml"), toml).expect("should write TOML"); + let output = command(dir.path()) + .args(["pbs", "inspect", "--json"]) + .output() + .expect("should run CLI"); + assert!(output.status.success()); + assert_no_secret(&output); + let report: Value = serde_json::from_slice(&output.stdout).expect("should emit JSON"); + assert_eq!(report["enabled_explicit"], false); + assert_eq!( + fs::read_to_string(dir.path().join("trusted-server.toml")).expect("should read TOML"), + toml + ); + let output = command(dir.path()) + .args(["pbs", "check", "--deployment", "deployment.yaml", "--json"]) + .output() + .expect("should run checks"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert!( + !dir.path().join("calls").exists(), + "local commands must not invoke AWS" + ); +} + +#[test] +fn secret_payload_is_private_not_in_argv_and_deleted_after_use() { + let dir = fixture(); + let input = json!({"api_key": "DUMMY_SECRET-$\"\nvalue"}).to_string(); + fs::write(dir.path().join("secret.json"), &input).expect("should write dummy credential"); + let output = secret_command(dir.path()) + .output() + .expect("should execute CLI"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert_no_secret(&output); + let calls = + fs::read_to_string(dir.path().join("calls")).expect("should read captured arguments"); + assert!( + !calls.contains("DUMMY_SECRET"), + "secret values must not appear in arguments" + ); + let request: Value = serde_json::from_str( + &fs::read_to_string(dir.path().join("captured_request.json")) + .expect("should read dummy request"), + ) + .expect("should parse request"); + assert_eq!(request["ClientRequestToken"], TOKEN); + let secret: Value = serde_json::from_str( + request["SecretString"] + .as_str() + .expect("should have string payload"), + ) + .expect("should parse payload"); + assert_eq!( + secret, + serde_json::from_str::(&input).expect("should parse fixture") + ); + assert_eq!( + fs::read_to_string(dir.path().join("secret.json")).expect("should preserve input"), + input + ); + assert_payload_cleanup(dir.path()); +} + +#[test] +fn aws_errors_never_forward_provider_stderr_and_cleanup_payloads() { + let dir = fixture(); + fs::write( + dir.path().join("secret.json"), + "{\"api_key\":\"DUMMY_SECRET\"}", + ) + .expect("should write fixture"); + let output = secret_command(dir.path()) + .env("PBS_FAKE_FAILURE", "yes") + .output() + .expect("should run CLI"); + assert_eq!(output.status.code(), Some(2)); + assert_no_secret(&output); + assert_payload_cleanup(dir.path()); +} + +#[test] +fn enabled_cli_history_and_wrong_accounts_block_writes() { + for (setting, value) in [ + ("PBS_FAKE_HISTORY", "enabled"), + ("PBS_FAKE_ACCOUNT", "999999999999"), + ] { + let dir = fixture(); + fs::write( + dir.path().join("secret.json"), + "{\"api_key\":\"DUMMY_SECRET\"}", + ) + .expect("should write fixture"); + let output = secret_command(dir.path()) + .env(setting, value) + .output() + .expect("should run CLI"); + assert_eq!(output.status.code(), Some(2)); + assert_no_secret(&output); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + assert!(!calls.contains("put-secret-value")); + assert!(!dir.path().join("captured_request.json").exists()); + assert_payload_cleanup(dir.path()); + } +} + +#[test] +fn incomplete_status_returns_json_and_nonzero_exit() { + let dir = fixture(); + let output = command(dir.path()) + .args(["pbs", "status", "--deployment", "deployment.yaml", "--json"]) + .output() + .expect("should run status"); + assert_eq!(output.status.code(), Some(2)); + let report: Value = serde_json::from_slice(&output.stdout).expect("should emit partial JSON"); + assert_eq!(report["complete"], false); + assert_eq!( + report["regions"][0]["instances"][0]["pbs_health"], + "unknown" + ); +} From ecfb0b38e37145c1421ab96def8beca71c0afabf Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 15 Sep 2026 18:17:11 -0500 Subject: [PATCH 03/14] Fix PBS inspect CLI test for current config --- crates/trusted-server-cli/src/commands/pbs/inspect.rs | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs index 69adfa04a..9114a3d86 100644 --- a/crates/trusted-server-cli/src/commands/pbs/inspect.rs +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -262,11 +262,8 @@ user_id_modules = ["sharedIdSystem"] ); let file = tempfile::NamedTempFile::new().expect("should create config"); fs::write(file.path(), &text).expect("should write config"); - let source: toml::Value = toml::from_str(&text).expect("should parse source"); let runtime: trusted_server_core::integrations::prebid::PrebidIntegrationConfig = - source["integrations"]["prebid"] - .clone() - .try_into() + toml::from_str(&format!("client_side_bidders={input}")) .expect("runtime should accept encoding"); let output = inspect(file.path()).expect("inspect should accept runtime encoding"); let candidates: Vec<_> = output.data["server_bidder_candidates"] @@ -279,7 +276,7 @@ user_id_modules = ["sharedIdSystem"] .expect("should identify bidder") }) .collect(); - assert_eq!(candidates, runtime.bidders); + assert_eq!(candidates, ["examplebidder", "otherbidder"]); assert_eq!( output.data["client_side_bidders"], json!(runtime.client_side_bidders) From 18d39104a19893d2c7b5eda2c28730d1f3db3e73 Mon Sep 17 00:00:00 2001 From: Christian Date: Wed, 16 Sep 2026 07:26:06 -0500 Subject: [PATCH 04/14] Nest Prebid client and server CLI commands --- .claude/skills/planning-prebid-aws/SKILL.md | 4 +- .../examples/two-region-pilot.md | 2 +- .../references/configuration-and-secrets.md | 16 ++-- .../references/file-generation.md | 4 +- crates/trusted-server-cli/README.md | 22 ++--- .../src/commands/pbs/config.rs | 5 +- .../src/commands/pbs/mod.rs | 2 +- .../trusted-server-cli/src/prebid_bundle.rs | 4 +- crates/trusted-server-cli/src/run.rs | 88 ++++++++++--------- crates/trusted-server-cli/tests/pbs_cli.rs | 23 ++++- .../src/integrations/prebid.rs | 4 +- .../lib/src/integrations/prebid/index.ts | 2 +- docs/guide/cli.md | 8 +- docs/guide/configuration.md | 6 +- docs/guide/integrations/prebid.md | 10 +-- ...6-07-24-prebid-refresh-gam-path-opt-out.md | 2 +- .../2026-06-17-prebid-bundle-cli-design.md | 40 ++++----- trusted-server.example.toml | 4 +- 18 files changed, 134 insertions(+), 112 deletions(-) diff --git a/.claude/skills/planning-prebid-aws/SKILL.md b/.claude/skills/planning-prebid-aws/SKILL.md index fce347753..bcd021846 100644 --- a/.claude/skills/planning-prebid-aws/SKILL.md +++ b/.claude/skills/planning-prebid-aws/SKILL.md @@ -55,7 +55,7 @@ Present one recommended design and only alternatives that resolve a real tradeof - Each selected AWS service, its requirement, and whether to reuse or create it. - Capacity assumptions, cost drivers and estimate date, accepted limitations, and blockers. - Runtime, infrastructure, secret, and traffic-control ownership. -- Use the experimental `ts pbs` commands for supported local checks, secret writes, and EC2 status. Record the selected YAML/secret delivery path and unsupported operations explicitly. Deployment, rollback, and other runtime support need a separately approved implementation; avoid competing wrappers for implemented commands. +- Use the experimental `ts prebid server` commands for supported local checks, secret writes, and EC2 status. Record the selected YAML/secret delivery path and unsupported operations explicitly. Deployment, rollback, and other runtime support need a separately approved implementation; avoid competing wrappers for implemented commands. - Target files and checks, with cloud-dependent checks separated from local checks. Ask the user to approve the architecture, assumptions, target files, and accepted limitations. Approval to generate files is not approval to execute them. Reopen approval if later findings change topology, cost commitments, or ownership. @@ -66,7 +66,7 @@ Done when the user explicitly approves the design and file scope. If blockers re Read [file generation and validation](references/file-generation.md). Follow existing repository conventions and generate only artifacts used by the selected design. Keep the decision record in the deployment plan; reference it from the runbook rather than repeating it. -Read the [PBS CLI usage and descriptor schema](../../../crates/trusted-server-cli/README.md) before generating inputs consumed by `ts pbs`. Its current descriptor supports EC2/Compose only. Keep other architecture choices available, but mark their CLI integration deferred rather than generating unsupported fields. +Read the [PBS CLI usage and descriptor schema](../../../crates/trusted-server-cli/README.md) before generating inputs consumed by `ts prebid server`. Its current descriptor supports EC2/Compose only. Keep other architecture choices available, but mark their CLI integration deferred rather than generating unsupported fields. Verify version-specific PBS fields and adapter bindings against the selected release. Verify AWS/Terraform behavior and pricing against current primary documentation. Record source links, versions, and verification dates in the deployment plan. Unavailable evidence remains a named blocker; do not invent image digests, configuration keys, prices, or benchmark results. diff --git a/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md index 137ea2e1b..d942c708e 100644 --- a/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md +++ b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md @@ -43,7 +43,7 @@ File generation starts after architecture-changing questions and target paths ar ## Configuration and operator walkthroughs -The experimental `ts pbs` CLI implements local inspection/checks, secret value writes, and EC2 infrastructure status. The release, runtime delivery, and rollback scenarios below remain acceptance criteria for a future approved implementation, not executed deployment evidence. Consult the [current command contract](../references/configuration-and-secrets.md#operator-command-contract) before documenting an invocation. +The experimental `ts prebid server` CLI implements local inspection/checks, secret value writes, and EC2 infrastructure status. The release, runtime delivery, and rollback scenarios below remain acceptance criteria for a future approved implementation, not executed deployment evidence. Consult the [current command contract](../references/configuration-and-secrets.md#operator-command-contract) before documenting an invocation. | Input or task | Expected behavior | | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md index 723e22509..34acd0f1d 100644 --- a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md +++ b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md @@ -1,6 +1,6 @@ # Configuration, secrets, and operator workflow -Use this reference for Trusted Server configuration discovery, the experimental `ts pbs` operator commands, and proposed runtime delivery. The current commands are implemented in `crates/trusted-server-cli`; deployment and rollback remain design work. Their existence does not authorize cloud operations. Read the [CLI usage and schema](../../../../crates/trusted-server-cli/README.md) before generating its input files or documenting invocations. +Use this reference for Trusted Server configuration discovery, the experimental `ts prebid server` operator commands, and proposed runtime delivery. The current commands are implemented in `crates/trusted-server-cli`; deployment and rollback remain design work. Their existence does not authorize cloud operations. Read the [CLI usage and schema](../../../../crates/trusted-server-cli/README.md) before generating its input files or documenting invocations. ## Discover requirements from Trusted Server @@ -24,7 +24,7 @@ Before asking questions the repository can answer: | `bundle.adapters`, identity modules | Browser bundle capabilities and identity questions, not proof of server-side use | | Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller/request path | -Use `ts pbs inspect --config ` for redacted local discovery. It reports explicit values without expanding defaults or proving host-secret requirements. Resolve schema defaults and adapter metadata separately, recording unsupported details as unresolved rather than fetching credentials or inventing mappings. Read config without invoking commands that publish or rewrite it. Leave `trusted-server.toml` unchanged. Propose caller endpoint/account changes separately after integration approval; discovery must not activate bidders, publish configuration, or change traffic. +Use `ts prebid server inspect --config ` for redacted local discovery. It reports explicit values without expanding defaults or proving host-secret requirements. Resolve schema defaults and adapter metadata separately, recording unsupported details as unresolved rather than fetching credentials or inventing mappings. Read config without invoking commands that publish or rewrite it. Leave `trusted-server.toml` unchanged. Propose caller endpoint/account changes separately after integration approval; discovery must not activate bidders, publish configuration, or change traffic. Done when the selected source and its limits are recorded, every observed bidder is classified, and missing inputs remain visible rather than filled from examples. @@ -73,16 +73,16 @@ Never package or log secret values. Treat container inspection and debug output ## Operator command contract -Use `ts pbs` instead of generating deployment-local wrappers for these implemented operations. Keep existing `ts config`, `ts deploy`, and `ts prebid bundle` behavior unchanged. +Use `ts prebid server` instead of generating deployment-local wrappers for these implemented operations. Keep existing `ts config`, `ts deploy`, and `ts prebid client` behavior unchanged. | Command | Current result and boundary | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `ts pbs inspect --config ` | Local discovery with unresolved requirements; no source changes or secret retrieval | -| `ts pbs check --deployment ` | Local schema, binding, and regional merge checks; no AWS calls, upstream PBS schema validation, or startup proof | -| `ts pbs secrets set --deployment --region ` | Authorized complete-value write to an existing declared secret; hidden terminal or file/stdin input, account/confirmation checks, and retry UUID; no deployment | -| `ts pbs status --deployment ` | Authorized EC2 infrastructure reads for declared instances; PBS health, installed release, and consumed secret versions stay unknown | +| `ts prebid server inspect --config ` | Local discovery with unresolved requirements; no source changes or secret retrieval | +| `ts prebid server check --deployment ` | Local schema, binding, and regional merge checks; no AWS calls, upstream PBS schema validation, or startup proof | +| `ts prebid server secrets set --deployment --region ` | Authorized complete-value write to an existing declared secret; hidden terminal or file/stdin input, account/confirmation checks, and retry UUID; no deployment | +| `ts prebid server status --deployment ` | Authorized EC2 infrastructure reads for declared instances; PBS health, installed release, and consumed secret versions stay unknown | -There are no `ts pbs deploy` or `ts pbs rollback` subcommands. Versioned release delivery, runtime injection, and release-aware status need separate implementation and approval. Do not present the future release contract as a working command. Current descriptors accept only `ec2-compose`; an ECS design requires another approved implementation rather than a fabricated descriptor. +There are no `ts prebid server deploy` or `ts prebid server rollback` subcommands. Versioned release delivery, runtime injection, and release-aware status need separate implementation and approval. Do not present the future release contract as a working command. Current descriptors accept only `ec2-compose`; an ECS design requires another approved implementation rather than a fabricated descriptor. Read targets from the deployment descriptor. Allow an explicit deployment selection; if selection is ambiguous, stop. Before cloud operations, display environment/account/regions and verify authenticated account identity against the descriptor. Require deliberate regional scope for mutations rather than silently changing every region or defaulting to production. Local commands do not authenticate. Cloud read commands must not retrieve credential values merely to report status. diff --git a/.claude/skills/planning-prebid-aws/references/file-generation.md b/.claude/skills/planning-prebid-aws/references/file-generation.md index 78c8c1f9b..3cda5cceb 100644 --- a/.claude/skills/planning-prebid-aws/references/file-generation.md +++ b/.claude/skills/planning-prebid-aws/references/file-generation.md @@ -13,7 +13,7 @@ Generate files only after the design and target paths are approved. Use the repo | Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | | Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | | Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | -| Operator interface | Existing `ts pbs` CLI and approved CI layout | CLI descriptor and documented invocations; any missing release stages need separate implementation approval | +| Operator interface | Existing `ts prebid server` CLI and approved CI layout | CLI descriptor and documented invocations; any missing release stages need separate implementation approval | | Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | Each generated artifact must have a consumer. Add ignore rules for local credentials, runtime secret files, `.terraform`, state files, saved plans, and generated sensitive output. Track nonsecret examples. Use `example.com` hostnames and visibly fictional identifiers in examples. @@ -26,7 +26,7 @@ For the deployment descriptor, resolved PBS YAML, secret bindings, runtime deliv Use the target-selection and approval contract in [operator commands](configuration-and-secrets.md#operator-command-contract), including descriptor-based interactive use and explicit automation inputs. New CI deployment jobs must remain inactive and approval-gated. File generation must not trigger existing auto-apply or deployment jobs; inspect those triggers before editing their watched paths. -The current `ts pbs` CLI does not implement deployment or rollback. For a separately approved deployment implementation, implement or explicitly defer each release stage: +The current `ts prebid server` CLI does not implement deployment or rollback. For a separately approved deployment implementation, implement or explicitly defer each release stage: 1. Serialize competing deployments to the same target. 2. Retrieve and verify an immutable release, image availability, and required secrets before replacing working capacity. diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 7f9445190..f0cc348db 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -1,6 +1,6 @@ # Trusted Server CLI: experimental PBS commands -`ts pbs` manages local configuration inputs and a small set of AWS operations for self-hosted Prebid Server Go. It is separate from `ts prebid bundle`, which builds browser JavaScript, and from the existing Trusted Server `ts config` and `ts deploy` commands. +`ts prebid server` manages local configuration inputs and a small set of AWS operations for self-hosted Prebid Server Go. It sits beside `ts prebid client`, which builds browser JavaScript, and remains separate from the existing Trusted Server `ts config` and `ts deploy` commands. The namespace is experimental and is being evaluated in PR review. There is no separate binary or crate. @@ -10,9 +10,9 @@ Use this branch's executable, not an older installed `ts`: ```bash cargo build_cli_linux -cargo run_cli_linux pbs --help -cargo run_cli_linux pbs inspect --config trusted-server.example.toml --json -cargo run_cli_linux pbs check --deployment crates/trusted-server-cli/examples/pbs/deployment.yaml +cargo run_cli_linux prebid server --help +cargo run_cli_linux prebid server inspect --config trusted-server.example.toml --json +cargo run_cli_linux prebid server check --deployment crates/trusted-server-cli/examples/pbs/deployment.yaml ``` On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fictional resource identifiers, a fictional image digest, and a fictional adapter binding. They exercise local checks only and must not be used as real deployment settings. @@ -21,12 +21,12 @@ On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fiction | Command | What it does | Access | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------- | -| `ts pbs inspect --config ` | Reports selected local Prebid fields, classifies bidder lists, and marks host-secret requirements unresolved | Local read-only | -| `ts pbs check --deployment ` | Validates schema, targets, binding metadata, and deterministic regional YAML merging | Local read-only | -| `ts pbs secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | -| `ts pbs status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | +| `ts prebid server inspect --config ` | Reports selected local Prebid fields, classifies bidder lists, and marks host-secret requirements unresolved | Local read-only | +| `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and deterministic regional YAML merging | Local read-only | +| `ts prebid server secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | +| `ts prebid server status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | -Add `--json` anywhere under `ts pbs` for a machine-readable report. Errors and operator notices go to stderr; failures exit with code 2. A partial status report still appears on stdout, with `complete: false` and exit code 2. +Add `--json` anywhere under `ts prebid server` for a machine-readable report. Errors and operator notices go to stderr; failures exit with code 2. A partial status report still appears on stdout, with `complete: false` and exit code 2. Not implemented: container deployment, rollback, runtime secret injection, caller updates, Terraform execution, ECS status, or PBS application health checks. `status` always reports the installed release and consumed secret versions as unknown. EC2 health is not PBS readiness. @@ -74,7 +74,7 @@ The command verifies the account through STS, describes the exact declared secre For approved automation, supply a file or stdin, `--yes`, and a stable UUID identifying the logical write: ```bash -ts pbs secrets set examplebidder \ +ts prebid server secrets set examplebidder \ --deployment /secure/path/deployment.yaml \ --region us-east-1 \ --file /secure/path/credential.json \ @@ -94,7 +94,7 @@ A successful write reports its version identifier. It does not create secret met These commands do not adopt the existing sandbox's Terraform state, edit its files, or change its IAM roles. A descriptor must reference resources the operator has explicitly approved. The sandbox currently bootstraps runtime files through Terraform user data and has no runtime secret loader. Writing a secret therefore does not make that sandbox consume it. -Deployment and rollback require a separately approved move to versioned runtime releases. Until that exists, `ts pbs` has no deployment or rollback subcommands and the skill must not promise them. +Deployment and rollback require a separately approved move to versioned runtime releases. Until that exists, `ts prebid server` has no deployment or rollback subcommands and the skill must not promise them. ## Verification diff --git a/crates/trusted-server-cli/src/commands/pbs/config.rs b/crates/trusted-server-cli/src/commands/pbs/config.rs index 6a8fe37da..f899e3eb4 100644 --- a/crates/trusted-server-cli/src/commands/pbs/config.rs +++ b/crates/trusted-server-cli/src/commands/pbs/config.rs @@ -306,8 +306,9 @@ fn read_yaml(path: &Path) -> Result { fn decode_yaml(path: &Path) -> Result { let value = read_yaml(path)?; validate_yaml(&value)?; - serde_yaml_ng::from_value(value) - .map_err(|_| invalid("unsupported deployment/binding schema; see ts pbs documentation")) + serde_yaml_ng::from_value(value).map_err(|_| { + invalid("unsupported deployment/binding schema; see ts prebid server documentation") + }) } /// Require plain string-keyed data: YAML tags and merge keys obscure the effective configuration. diff --git a/crates/trusted-server-cli/src/commands/pbs/mod.rs b/crates/trusted-server-cli/src/commands/pbs/mod.rs index 548fa1be9..6a10e2ef9 100644 --- a/crates/trusted-server-cli/src/commands/pbs/mod.rs +++ b/crates/trusted-server-cli/src/commands/pbs/mod.rs @@ -18,7 +18,7 @@ use serde_json::Value; use aws::AwsCli; use config::Deployment; -/// Arguments for the experimental `ts pbs` namespace. +/// Arguments for the experimental `ts prebid server` namespace. #[derive(Debug, Args)] pub(crate) struct PbsArgs { /// Emit a nonsecret JSON report instead of a human summary. diff --git a/crates/trusted-server-cli/src/prebid_bundle.rs b/crates/trusted-server-cli/src/prebid_bundle.rs index bdf449e65..446826464 100644 --- a/crates/trusted-server-cli/src/prebid_bundle.rs +++ b/crates/trusted-server-cli/src/prebid_bundle.rs @@ -9,7 +9,7 @@ use toml_edit::{DocumentMut, Item, table, value}; pub(crate) type CliResult = Result; -const NODE_MODULES_MISSING_HELP: &str = "Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid bundle`."; +const NODE_MODULES_MISSING_HELP: &str = "Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid client`."; #[derive(Debug, clap::Args)] pub(crate) struct PrebidBundleArgs { @@ -412,7 +412,7 @@ fn find_js_lib_dir(start: &Path) -> CliResult { } cli_error( - "failed to locate crates/trusted-server-js/lib; run `ts prebid bundle` from the Trusted Server repository", + "failed to locate crates/trusted-server-js/lib; run `ts prebid client` from the Trusted Server repository", ) } diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 7fe1be374..27129dc20 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -37,8 +37,6 @@ enum Command { Deploy(DeployArgs), /// Probe a deployed version until it reports healthy. Healthcheck(HealthcheckArgs), - /// Experimental self-hosted Prebid Server configuration and AWS operations. - Pbs(PbsArgs), /// Trusted Server Prebid commands. Prebid(PrebidArgs), /// Provision platform resources through a target adapter. @@ -77,8 +75,10 @@ struct PrebidArgs { #[derive(Debug, Subcommand)] enum PrebidCommand { - /// Generate a local external Prebid bundle and update config metadata. - Bundle(PrebidBundleArgs), + /// Generate a local external Prebid client bundle and update config metadata. + Client(PrebidBundleArgs), + /// Configure and operate a self-hosted Prebid Server deployment. + Server(PbsArgs), } /// Process-level outcome for commands that distinguish drift from tool errors. @@ -146,19 +146,17 @@ fn dispatch(args: Args) -> Result { Command::Healthcheck(args) => { edgezero_cli::run_healthcheck(&args).map(|()| RunOutcome::Success) } - Command::Pbs(args) => pbs::run(&args) - .map(|()| RunOutcome::Success) - .map_err(|error| error.current_context().to_string()), - Command::Prebid(prebid) => { - let mut generator = NpmPrebidBundleGenerator; - let mut stdout = std::io::stdout(); - let mut stderr = std::io::stderr(); - match prebid.command { - PrebidCommand::Bundle(args) => { - run_bundle(&args, &mut generator, &mut stdout, &mut stderr) - .map(|()| RunOutcome::Success) - } + Command::Prebid(prebid) => match prebid.command { + PrebidCommand::Client(args) => { + let mut generator = NpmPrebidBundleGenerator; + let mut stdout = std::io::stdout(); + let mut stderr = std::io::stderr(); + run_bundle(&args, &mut generator, &mut stdout, &mut stderr) + .map(|()| RunOutcome::Success) } + PrebidCommand::Server(args) => pbs::run(&args) + .map(|()| RunOutcome::Success) + .map_err(|error| error.current_context().to_string()), } Command::Provision(args) => { edgezero_cli::run_provision(&args).map(|()| RunOutcome::Success) @@ -194,30 +192,32 @@ mod tests { } #[test] - fn parses_pbs_inspect() { + fn parses_prebid_server_inspect() { assert!( Args::try_parse_from([ "ts", - "pbs", + "prebid", + "server", "inspect", "--config", "trusted-server.toml", - "--json" + "--json", ]) .is_ok(), - "should accept the standalone PBS command namespace" + "should accept PBS commands under the Prebid namespace" ); } #[test] - fn pbs_rejects_ambiguous_secret_inputs_and_unimplemented_commands() { + fn prebid_server_rejects_ambiguous_secret_inputs_and_unimplemented_commands() { for arguments in [ - vec!["ts", "pbs", "deploy"], - vec!["ts", "pbs", "rollback", "--release", "example"], - vec!["ts", "pbs", "check"], + vec!["ts", "prebid", "server", "deploy"], + vec!["ts", "prebid", "server", "rollback", "--release", "example"], + vec!["ts", "prebid", "server", "check"], vec![ "ts", - "pbs", + "prebid", + "server", "secrets", "set", "examplebidder", @@ -229,7 +229,8 @@ mod tests { ], vec![ "ts", - "pbs", + "prebid", + "server", "secrets", "set", "examplebidder", @@ -250,16 +251,17 @@ mod tests { assert!( Args::try_parse_from([ "ts", - "pbs", + "prebid", + "server", "check", "--deployment", "deployment.yaml", - "--json" + "--json", ]) .is_ok() ); assert!( - Args::try_parse_from(["ts", "prebid", "bundle", "--config", "trusted-server.toml"]) + Args::try_parse_from(["ts", "prebid", "client", "--config", "trusted-server.toml"]) .is_ok() ); } @@ -1111,22 +1113,24 @@ mod tests { } #[test] - fn prebid_bundle_defaults_match_spec() { - let args = parse(&["ts", "prebid", "bundle"]); + fn prebid_client_defaults_match_spec() { + let args = parse(&["ts", "prebid", "client"]); let Command::Prebid(prebid) = args.command else { panic!("expected prebid command"); }; - let PrebidCommand::Bundle(bundle) = prebid.command; - assert_eq!(bundle.config, PathBuf::from("trusted-server.toml")); - assert_eq!(bundle.out, PathBuf::from("dist/prebid")); + let PrebidCommand::Client(client) = prebid.command else { + panic!("expected prebid client command"); + }; + assert_eq!(client.config, PathBuf::from("trusted-server.toml")); + assert_eq!(client.out, PathBuf::from("dist/prebid")); } #[test] - fn prebid_bundle_accepts_custom_paths() { + fn prebid_client_accepts_custom_paths() { let args = parse(&[ "ts", "prebid", - "bundle", + "client", "--config", "publisher.toml", "--out", @@ -1135,14 +1139,16 @@ mod tests { let Command::Prebid(prebid) = args.command else { panic!("expected prebid command"); }; - let PrebidCommand::Bundle(bundle) = prebid.command; - assert_eq!(bundle.config, PathBuf::from("publisher.toml")); - assert_eq!(bundle.out, PathBuf::from("build/prebid")); + let PrebidCommand::Client(client) = prebid.command else { + panic!("expected prebid client command"); + }; + assert_eq!(client.config, PathBuf::from("publisher.toml")); + assert_eq!(client.out, PathBuf::from("build/prebid")); } #[test] - fn prebid_bundle_does_not_accept_adapter_option() { - let error = Args::try_parse_from(["ts", "prebid", "bundle", "--adapter", "fastly"]) + fn prebid_client_does_not_accept_adapter_option() { + let error = Args::try_parse_from(["ts", "prebid", "client", "--adapter", "fastly"]) .expect_err("should reject prebid adapter option"); assert!( error.to_string().contains("unexpected argument") diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index 38c7d4e23..c88bf0dba 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -86,7 +86,8 @@ fn command(dir: &Path) -> Command { fn secret_command(dir: &Path) -> Command { let mut command = command(dir); command.args([ - "pbs", + "prebid", + "server", "secrets", "set", "examplebidder", @@ -126,7 +127,7 @@ fn local_commands_never_execute_aws_and_preserve_the_source() { let toml = "[integrations.prebid]\nenabled=false\nbidders=['examplebidder']\naccount_id='DUMMY_SECRET'\n"; fs::write(dir.path().join("trusted-server.toml"), toml).expect("should write TOML"); let output = command(dir.path()) - .args(["pbs", "inspect", "--json"]) + .args(["prebid", "server", "inspect", "--json"]) .output() .expect("should run CLI"); assert!(output.status.success()); @@ -138,7 +139,14 @@ fn local_commands_never_execute_aws_and_preserve_the_source() { toml ); let output = command(dir.path()) - .args(["pbs", "check", "--deployment", "deployment.yaml", "--json"]) + .args([ + "prebid", + "server", + "check", + "--deployment", + "deployment.yaml", + "--json", + ]) .output() .expect("should run checks"); assert!( @@ -241,7 +249,14 @@ fn enabled_cli_history_and_wrong_accounts_block_writes() { fn incomplete_status_returns_json_and_nonzero_exit() { let dir = fixture(); let output = command(dir.path()) - .args(["pbs", "status", "--deployment", "deployment.yaml", "--json"]) + .args([ + "prebid", + "server", + "status", + "--deployment", + "deployment.yaml", + "--json", + ]) .output() .expect("should run status"); assert_eq!(output.status.code(), Some(2)); diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index 095dc6eb6..9201b9ac8 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -377,12 +377,12 @@ impl IntegrationConfig for LegacyPrebidServerConfig { #[derive(Debug, Clone, Default, Deserialize, Serialize)] #[serde(deny_unknown_fields)] pub struct PrebidBundleBuildConfig { - /// Typed Prebid.js module selections consumed by `ts prebid bundle`. + /// Typed Prebid.js module selections consumed by `ts prebid client`. #[serde(default)] pub modules: PrebidBundleModulesConfig, } -/// Exact Prebid.js module stems selected by `ts prebid bundle`. +/// Exact Prebid.js module stems selected by `ts prebid client`. /// /// The CLI validates these values. The runtime only parses them so app config /// carrying build inputs remains loadable. diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index e8162767f..d998ce4a1 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -1669,7 +1669,7 @@ export function installPrebidNpm(config?: Partial): typeof pbjs `[tsjs-prebid] client-side bidder "${bidder}" has no adapter in the external ` + 'Prebid bundle. Add its exact Prebid module stem to ' + '[integrations.prebid.bundle.modules].bidder in trusted-server.toml and ' + - 'rebuild it with `ts prebid bundle`.' + 'rebuild it with `ts prebid client`.' ); } } diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 8b2666176..4c9d343eb 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -608,7 +608,7 @@ APIs. ## Generate an external Prebid bundle -`ts prebid bundle` builds the local external Prebid browser bundle configured in +`ts prebid client` builds the local external Prebid browser bundle configured in `trusted-server.toml`. ```toml @@ -628,7 +628,7 @@ Run the command after installing JS dependencies: ```bash cd crates/trusted-server-js/lib && npm ci cd ../../.. -ts prebid bundle +ts prebid client ``` By default, generated artifacts are written to `dist/prebid/`. The versioned @@ -644,8 +644,8 @@ HTTPS asset URL, and include that host plus any redirect targets in Use custom paths when needed: ```bash -ts prebid bundle --config publisher-a.toml --out build/prebid +ts prebid client --config publisher-a.toml --out build/prebid ``` -`ts prebid bundle` is local-only. It has no `--adapter` option and does not +`ts prebid client` is local-only. It has no `--adapter` option and does not upload, provision, deploy, or push config. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 18d29fd9b..9ebef1171 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1348,9 +1348,9 @@ overrides, and notification suppression belong under `[auction]`. | `script_patterns` | Array[String] | `["/prebid.js", "/prebid.min.js", "/prebidjs.js", "/prebidjs.min.js"]` | Publisher Prebid script paths intercepted by Trusted Server | | `external_bundle_url` | String | Required when enabled | HTTPS publisher-specific Prebid.js bundle URL | | `external_bundle_sha256` / `*_sri` | String | `None` | Optional bundle integrity and cache metadata | -| `bundle.modules.bidder` | Array[String] | Required and non-empty | Exact bidder module stems used by `ts prebid bundle` | -| `bundle.modules.user_id` | Array[String] | Curated preset when omitted | Exact User ID module stems used by `ts prebid bundle` | -| `bundle.modules.analytics` | Array[String] | `[]` | Exact analytics module stems used by `ts prebid bundle` | +| `bundle.modules.bidder` | Array[String] | Required and non-empty | Exact bidder module stems used by `ts prebid client` | +| `bundle.modules.user_id` | Array[String] | Curated preset when omitted | Exact User ID module stems used by `ts prebid client` | +| `bundle.modules.analytics` | Array[String] | `[]` | Exact analytics module stems used by `ts prebid client` | Server-side bidder codes are derived from validated `[auction.bidders.*]` routes and injected into the browser. There is no second server bidder list in diff --git a/docs/guide/integrations/prebid.md b/docs/guide/integrations/prebid.md index 3dd12bae7..928585fe8 100644 --- a/docs/guide/integrations/prebid.md +++ b/docs/guide/integrations/prebid.md @@ -34,7 +34,7 @@ external_bundle_url = "https://assets.example.com/prebid/trusted-prebid.js" # external_bundle_sha256 = "" # external_bundle_sri = "sha384-" -# External bundle generation inputs used by `ts prebid bundle`. +# External bundle generation inputs used by `ts prebid client`. # Values are exact Prebid module stems without `.js`. [integrations.prebid.bundle.modules] bidder = ["rubiconBidAdapter"] @@ -144,11 +144,11 @@ and a `prebid-server` provider can exist independently from browser injection. ## External Bundle Generation -Use `ts prebid bundle` to build the publisher-specific browser bundle from +Use `ts prebid client` to build the publisher-specific browser bundle from `[integrations.prebid.bundle.modules]` selections: ```bash -ts prebid bundle +ts prebid client ``` The command writes generated artifacts to `dist/prebid/` by default and updates @@ -194,7 +194,7 @@ upstream stems. For example, `adapters = ["rubicon"]` becomes `bidder = ["rubiconBidAdapter"]`; `client_side_bidders` continues to use the runtime code `rubicon`. -`ts prebid bundle` rejects the removed `adapters`, `user_id_modules`, and +`ts prebid client` rejects the removed `adapters`, `user_id_modules`, and `analytics_adapters` fields with the replacement path. Runtime config validation, `ts config push`, and server startup also reject the old bundle fields. @@ -500,7 +500,7 @@ bidder = ["rubiconBidAdapter", "appnexusBidAdapter", "openxBidAdapter"] user_id = ["sharedIdSystem", "uid2IdSystem"] ``` -Run `ts prebid bundle` after changing the module list. The generator resolves +Run `ts prebid client` after changing the module list. The generator resolves `prebid.js/modules/.js` through the pinned package and records both stems and registered bidder codes in `manifest.json`. At runtime, TSJS checks each `client_side_bidders` runtime code against that manifest. diff --git a/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md b/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md index edb0e27db..a111b4228 100644 --- a/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md +++ b/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md @@ -9,7 +9,7 @@ Server's Prebid refresh auctions without suppressing the corresponding GAM refre - Do not change publisher source, slot div IDs, or GAM configuration. - Do not edit `dist` output, minified assets, or an externally hosted Prebid bundle - by hand. `build-prebid-external.mjs`/`ts prebid bundle` are the supported build + by hand. `build-prebid-external.mjs`/`ts prebid client` are the supported build path. - The mechanism is literal, case-sensitive GAM-path suffix matching; it is not a size-based rule and does not add a div-ID fallback. diff --git a/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md b/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md index 69a8f35de..3d60679b5 100644 --- a/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md +++ b/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md @@ -2,7 +2,7 @@ **Date:** 2026-06-17 **Status:** Implemented -**Scope:** `ts prebid bundle` local external Prebid bundle generation +**Scope:** `ts prebid client` local external Prebid bundle generation **Related context:** - `docs/superpowers/specs/2026-05-28-external-prebid-first-party-proxy-design.md` @@ -18,7 +18,7 @@ Add a Trusted Server-specific CLI command for generating the external Prebid browser bundle used by the first-party Prebid proxy flow: ```bash -ts prebid bundle +ts prebid client ``` The command should make the existing external bundle generation path ergonomic for @@ -43,7 +43,7 @@ proxy spec: ## 2. Non-goals -The initial `ts prebid bundle` command does **not** do any of the following: +The initial `ts prebid client` command does **not** do any of the following: - upload generated bundles to an asset host or CDN; - infer or construct the public `external_bundle_url`; @@ -63,7 +63,7 @@ The initial `ts prebid bundle` command does **not** do any of the following: ## 3. Command surface ```bash -ts prebid bundle [--config ] [--out ] +ts prebid client [--config ] [--out ] ``` Defaults: @@ -77,13 +77,13 @@ Examples: ```bash # Generate from trusted-server.toml into dist/prebid -ts prebid bundle +ts prebid client # Generate from a draft config -ts prebid bundle --config ./publisher-a.trusted-server.toml +ts prebid client --config ./publisher-a.trusted-server.toml # Generate into a custom local directory -ts prebid bundle --out ./build/prebid +ts prebid client --out ./build/prebid ``` Successful output should be concise and actionable, for example: @@ -159,7 +159,7 @@ bundle URL's host and any HTTPS redirect targets. ## 5. Config update behavior -After a successful local bundle build, `ts prebid bundle` must read the generated +After a successful local bundle build, `ts prebid client` must read the generated `manifest.json` and update the same `trusted-server.toml` file with: ```toml @@ -235,7 +235,7 @@ The manifest schema remains unchanged: ## 7. Dependency and environment handling -`ts prebid bundle` should fail fast with actionable diagnostics when local JS +`ts prebid client` should fail fast with actionable diagnostics when local JS build prerequisites are missing. Minimum checks before shelling out: @@ -249,7 +249,7 @@ If `node_modules` is missing, the command must not run dependency installation. It should fail with an instruction like: ```text -Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid bundle`. +Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid client`. ``` Errors from the JS generator, including unknown adapter names or unknown User ID @@ -261,7 +261,7 @@ and stderr enough for debugging. ## 8. Config loading and validation -`ts prebid bundle` should not require full production config validity. It is a +`ts prebid client` should not require full production config validity. It is a local artifact-generation command, and operators may run it before the config is ready for `ts config validate` or `ts config push`. @@ -289,7 +289,7 @@ This spec extends the `ts` product CLI command surface with a new Trusted Server-specific command group: ```text -ts prebid bundle +ts prebid client ``` The resulting CLI command enum should conceptually become: @@ -297,7 +297,7 @@ The resulting CLI command enum should conceptually become: ```text ts audit ... ts config ... -ts prebid bundle ... +ts prebid client ... ts auth ... ts provision ... ts serve ... @@ -305,7 +305,7 @@ ts build ... ts deploy ... ``` -`ts prebid bundle` is similar to `ts audit` and `ts config` in that it owns +`ts prebid client` is similar to `ts audit` and `ts config` in that it owns Trusted Server behavior directly. It is unlike `ts build` / `ts deploy`, which are EdgeZero lifecycle delegates. @@ -316,12 +316,12 @@ are EdgeZero lifecycle delegates. ### CLI argument parsing - Add `Command::Prebid(PrebidArgs)`. -- Add `PrebidCommand::Bundle(PrebidBundleArgs)`. +- Add `PrebidCommand::Client(PrebidBundleArgs)`. - Add options: - `--config ` defaulting to `trusted-server.toml`; - `--out ` defaulting to `dist/prebid`. - Add parser tests for defaults and custom paths. -- Reject `--adapter` for `ts prebid bundle`. +- Reject `--adapter` for `ts prebid client`. ### CLI implementation @@ -353,11 +353,11 @@ User ID module names, hashing bundle bytes, and writing `manifest.json`. ### CLI parser tests -- `ts prebid bundle` parses with defaults: +- `ts prebid client` parses with defaults: - config: `trusted-server.toml` - out: `dist/prebid` -- `ts prebid bundle --config publisher.toml --out build/prebid` parses custom paths. -- `ts prebid bundle --adapter fastly` is rejected. +- `ts prebid client --config publisher.toml --out build/prebid` parses custom paths. +- `ts prebid client --adapter fastly` is rejected. ### Unit tests @@ -389,7 +389,7 @@ cd crates/trusted-server-js/lib npm ci cd ../../.. -ts prebid bundle +ts prebid client ls dist/prebid rg 'external_bundle_sha256|external_bundle_sri' trusted-server.toml ``` diff --git a/trusted-server.example.toml b/trusted-server.example.toml index b6a0496f4..9859f10da 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -477,11 +477,11 @@ client_side_bidders = [] # bidders running via native Prebid.js adapter # Keep selected GAM inventory out of Trusted Server's Prebid refresh auctions. # Matching slots still refresh through GAM. # excluded_gam_ad_unit_path_suffixes = ["/trackingonly"] -# Runtime bundle metadata — set after running `ts prebid bundle` and uploading: +# Runtime bundle metadata — set after running `ts prebid client` and uploading: # external_bundle_url = "https://assets.example.com/prebid/trusted-prebid-.js" # external_bundle_sha256 = "" # external_bundle_sri = "" -# Bundle build inputs consumed by `ts prebid bundle`, not by the edge runtime. +# Bundle build inputs consumed by `ts prebid client`, not by the edge runtime. # Values are exact upstream module stems without `.js`. # [integrations.prebid.bundle.modules] # bidder = ["rubiconBidAdapter"] From 98376af9d31b6c43feb7cd34aae5a8fa94f32e36 Mon Sep 17 00:00:00 2001 From: Christian Date: Wed, 16 Sep 2026 14:43:00 -0500 Subject: [PATCH 05/14] Add production-shaped PBS AWS example --- deploy/pbs-example/.gitignore | 14 ++ deploy/pbs-example/.terraform.lock.hcl | 26 +++ deploy/pbs-example/DEPLOYMENT_PLAN.md | 113 +++++++++++++ deploy/pbs-example/RUNBOOK.md | 106 +++++++++++++ deploy/pbs-example/deployment.yaml | 22 +++ deploy/pbs-example/dns.tf | 35 ++++ deploy/pbs-example/locals.tf | 8 + deploy/pbs-example/main.tf | 41 +++++ deploy/pbs-example/modules/regional/README.md | 5 + .../pbs-example/modules/regional/compute.tf | 31 ++++ .../modules/regional/load_balancing.tf | 59 +++++++ deploy/pbs-example/modules/regional/locals.tf | 9 ++ .../modules/regional/monitoring.tf | 74 +++++++++ .../pbs-example/modules/regional/network.tf | 109 +++++++++++++ .../pbs-example/modules/regional/outputs.tf | 29 ++++ .../pbs-example/modules/regional/secrets.tf | 80 ++++++++++ .../pbs-example/modules/regional/security.tf | 55 +++++++ .../pbs-example/modules/regional/terraform.tf | 7 + .../pbs-example/modules/regional/variables.tf | 76 +++++++++ deploy/pbs-example/outputs.tf | 29 ++++ deploy/pbs-example/providers.tf | 20 +++ deploy/pbs-example/runtime/README.md | 7 + deploy/pbs-example/runtime/compose.yaml | 15 ++ deploy/pbs-example/runtime/examples/README.md | 1 + .../pbs-example/runtime/examples/compose.env | 3 + .../runtime/examples/pbs-secrets.env | 3 + deploy/pbs-example/runtime/pbs.yaml | 11 ++ deploy/pbs-example/runtime/regions/east.yaml | 2 + deploy/pbs-example/runtime/regions/west.yaml | 2 + .../pbs-example/runtime/secret-bindings.json | 22 +++ deploy/pbs-example/terraform.tf | 14 ++ deploy/pbs-example/terraform.tfvars.example | 14 ++ deploy/pbs-example/variables.tf | 150 ++++++++++++++++++ 33 files changed, 1192 insertions(+) create mode 100644 deploy/pbs-example/.gitignore create mode 100644 deploy/pbs-example/.terraform.lock.hcl create mode 100644 deploy/pbs-example/DEPLOYMENT_PLAN.md create mode 100644 deploy/pbs-example/RUNBOOK.md create mode 100644 deploy/pbs-example/deployment.yaml create mode 100644 deploy/pbs-example/dns.tf create mode 100644 deploy/pbs-example/locals.tf create mode 100644 deploy/pbs-example/main.tf create mode 100644 deploy/pbs-example/modules/regional/README.md create mode 100644 deploy/pbs-example/modules/regional/compute.tf create mode 100644 deploy/pbs-example/modules/regional/load_balancing.tf create mode 100644 deploy/pbs-example/modules/regional/locals.tf create mode 100644 deploy/pbs-example/modules/regional/monitoring.tf create mode 100644 deploy/pbs-example/modules/regional/network.tf create mode 100644 deploy/pbs-example/modules/regional/outputs.tf create mode 100644 deploy/pbs-example/modules/regional/secrets.tf create mode 100644 deploy/pbs-example/modules/regional/security.tf create mode 100644 deploy/pbs-example/modules/regional/terraform.tf create mode 100644 deploy/pbs-example/modules/regional/variables.tf create mode 100644 deploy/pbs-example/outputs.tf create mode 100644 deploy/pbs-example/providers.tf create mode 100644 deploy/pbs-example/runtime/README.md create mode 100644 deploy/pbs-example/runtime/compose.yaml create mode 100644 deploy/pbs-example/runtime/examples/README.md create mode 100644 deploy/pbs-example/runtime/examples/compose.env create mode 100644 deploy/pbs-example/runtime/examples/pbs-secrets.env create mode 100644 deploy/pbs-example/runtime/pbs.yaml create mode 100644 deploy/pbs-example/runtime/regions/east.yaml create mode 100644 deploy/pbs-example/runtime/regions/west.yaml create mode 100644 deploy/pbs-example/runtime/secret-bindings.json create mode 100644 deploy/pbs-example/terraform.tf create mode 100644 deploy/pbs-example/terraform.tfvars.example create mode 100644 deploy/pbs-example/variables.tf diff --git a/deploy/pbs-example/.gitignore b/deploy/pbs-example/.gitignore new file mode 100644 index 000000000..018d39b61 --- /dev/null +++ b/deploy/pbs-example/.gitignore @@ -0,0 +1,14 @@ +# Terraform working data and local state +.terraform/ +*.tfstate +*.tfstate.* +.terraform.tfstate.lock.info + +# Saved plans can contain provider-returned metadata +*.tfplan + +# Local inputs and crash output +*.auto.tfvars +*.tfvars +crash.log +crash.*.log diff --git a/deploy/pbs-example/.terraform.lock.hcl b/deploy/pbs-example/.terraform.lock.hcl new file mode 100644 index 000000000..38e6eaafe --- /dev/null +++ b/deploy/pbs-example/.terraform.lock.hcl @@ -0,0 +1,26 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "6.64.0" + constraints = "6.64.0" + hashes = [ + "h1:2fTLxzUDmp/KVIHbIeLTB4bIzWHx8E6Dw+1ALLUi+Yw=", + "zh:07172315d67bc9781240272759cdfc7bd32b7e72384a56862c2c1da3cca99a81", + "zh:154ce7d2659de9a59ddfe96d7cab41a9ddc2cb267a7d4bcdf4e737ff2ffdec06", + "zh:17324d4335a7a7ac01cc23eded530775606680ff53b47cb74a3cb95d1121f836", + "zh:307ab92324ec5a61b124881ab8cac1d9e316f4527dfd0e1b59794c229407eb4e", + "zh:31e25f1903661332e36a95283042dd3ec50b47c186db00663fbd976a11e6a6b2", + "zh:3311d9f3bd12a24886027dbe73859dcd1e67bd0e3046227a338cf2c7ca04d18e", + "zh:37916156a3aac3b29be3acebd15d53145ea4ab5d4aaa825eaebe75481fa00500", + "zh:4158cb8c38b3ac6aa98eb15935ec6bd7c30838d85d2b00acc9812df8382ae908", + "zh:5bfb9499c66d9db5b34dc5c60f426a1ab1baa5457ce2aefebca826a9c3f92fb0", + "zh:6eb29ead5a4aca3b1f35812e7e8c75419180e1928e479b458f206861277736db", + "zh:7a82b6dd0c0cdef8045a4adfbddd36acb86b6b23fcbed8e189c2d71f7dc4a502", + "zh:9556bd792032c3f7e73ea4dd08cec88dc1327f5a4a57d79c30ba844ae2b9a3c0", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:c5234180464cb800c83a41f57462742b802c150ad7d4417626fcd9cb511c01d2", + "zh:cd776b83b1f7b36635957350afe7ce28ba4e4ea3a5e2deb00d13dbd3b35d9d40", + "zh:fb583a7b791c6f915b86573d04f05ddbf7f1a5e4120c5d8a7450a3086c1225c4", + ] +} diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md new file mode 100644 index 000000000..70f5b3368 --- /dev/null +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -0,0 +1,113 @@ +# Prebid Server AWS example deployment plan + +Status: draft generated files, locally checked after validation. This directory is an example only. It has not been planned against a live AWS account, applied, deployed, or load tested. + +## Decision record + +| Requirement | Value | Status | Evidence or decision owner | Blocks | +| --- | --- | --- | --- | --- | +| Purpose | Production-shaped architecture example, not a live deployment | Confirmed | User approval | None | +| Caller | Trusted Server | Confirmed | User | Caller egress ranges remain required | +| Regions | `us-east-1` and `us-west-2` | Confirmed | User | Account-specific AZ selection remains unresolved | +| Regional topology | Two AZs, one PBS EC2 host per AZ | Confirmed | User | AMIs and subnets remain required | +| Total PBS hosts | Four | Confirmed | Derived from topology | Capacity remains unmeasured | +| Runtime | EC2 with Docker Compose | Confirmed | User | Release and secret injection are deferred implementation work | +| Regional ingress | Public HTTPS ALB | Confirmed | User | ACM certificate ARNs and caller CIDRs required | +| Global routing | Route 53 latency aliases with ALB health evaluation | Proposed | Architecture decision | Existing hosted-zone ID required | +| WAF | Not included in this demo | Confirmed | User | Abuse controls remain outside this example | +| Terraform state | Local backend | Confirmed | User | No team locking or remote recovery | +| PBS release | Go v4.7.0, digest pinned | Proposed and verified | PBS release and Docker metadata checked during generation | Recheck before any future use | +| Workload | 200 global peak auctions/s, 4 bidders, 1.5 s caller timeout, 1 s PBS timeout | Proposed example assumption | Planning assumption | No capacity claim until load tested | +| Secrets | Secrets Manager metadata and EC2 read policy; values written separately | Confirmed | User and repository CLI contract | Real bidder mapping and authorization required | +| Trusted Server config | No authoritative `trusted-server.toml` exists in this checkout | Confirmed | Repository inspection | Caller behavior remains an external input | +| Deployment directory | `deploy/pbs-example/` | Confirmed | User | None | + +## Architecture + +```mermaid +flowchart TD + T["Trusted Server"] --> R["Route 53 latency aliases"] + R --> E["us-east-1 ALB"] + R --> W["us-west-2 ALB"] + E --> E1["East private PBS host, AZ 1"] + E --> E2["East private PBS host, AZ 2"] + W --> W1["West private PBS host, AZ 1"] + W --> W2["West private PBS host, AZ 2"] + E1 --> B["Bidder endpoints"] + E2 --> B + W1 --> B + W2 --> B +``` + +Each region has a public ALB in two public subnets and one private PBS host in each AZ. Each private subnet has its own NAT gateway so normal outbound traffic remains in the AZ and can use a stable Elastic IP for bidder allowlists. Route 53 returns the lowest-latency healthy regional ALB. DNS caching means failover is not immediate. + +The EC2 AMIs are inputs rather than built by Terraform. They must contain the approved OS, SSM agent, Docker, and Compose. The generated Terraform does not install packages or deploy PBS through user data. Runtime release delivery, configuration rendering, secret injection, restart, and rollback need a separately approved implementation. + +## Services and ownership + +| Service or artifact | Action | Owner | Purpose | +| --- | --- | --- | --- | +| VPC, subnets, routes, IGW, NAT, security groups | Create | Terraform | Regional network and egress | +| ALB, target group, HTTPS listener | Create | Terraform | Regional HTTPS ingress and health routing | +| Route 53 records | Reuse zone, create records | Terraform | Latency-based regional selection | +| EC2 instances and IAM profiles | Create | Terraform | Compose hosts and SSM access | +| Secrets Manager secret metadata | Create | Terraform | One regional secret for the example bidder | +| Secret values | External write | Authorized operator or automation | Credential lifecycle; never Terraform | +| PBS YAML and Compose definition | Git-owned | Runtime owner | Nonsecret runtime contract | +| Regional YAML merge and rendered file | Deferred runtime release owner | Deployment implementation | One resolved config per region | +| PBS process lifecycle | Deferred runtime release owner | Deployment implementation | Start, replace, health, rollback | +| Terraform state | Local operator | Terraform | Demo-only state; no shared locking | + +## Capacity assumptions + +The example uses these fictional planning values: + +- 200 peak eligible auctions per second globally. +- 100 QPS per region during normal routing. +- Either region must be able to receive 200 QPS during regional failover. +- Four bidder requests per auction, or 800 outbound requests per second globally. +- 16 KiB average request and 32 KiB average response. +- Trusted Server timeout of 1.5 seconds and PBS auction timeout of 1 second. +- `c7i.large` is an initial instance-type example, not a throughput guarantee. + +A controlled load test must measure CPU, memory, connection reuse, outbound bandwidth, bidder tail latency, ALB target health, and PBS queueing before changing these values. The four-host count does not prove capacity or availability. + +## Failure boundaries and limitations + +- Loss of one host leaves the other regional target serving traffic. The ALB health check must drain the failed target. +- Loss of one AZ removes that region's one target and leaves the region degraded. A later scale-out would be required for full AZ redundancy. +- Loss of a whole region depends on Route 53 resolver behavior and surviving-region capacity. +- NAT, ALB, and EC2 resources have separate regional failure and cost boundaries. +- Local state has no remote locking or shared recovery. Only one operator may use it. +- No WAF or rate-based abuse control is included. +- The example uses a fictional bidder binding. The adapter name, credential keys, source authorization, and PBS mapping must be replaced and verified against v4.7.0 before use. +- A Secrets Manager write does not refresh a running Compose container. The deferred runtime implementation must retrieve a selected version, render an environment file atomically, replace the consumer, and verify health before retiring the old version. + +## Cost drivers + +No price estimate is claimed. The main drivers are four EC2 instances, four NAT gateways and their Elastic IPs, two ALBs, public IPv4 addresses, cross-AZ traffic if routing changes, CloudWatch logs and alarms, Secrets Manager, Route 53 records, and bidder internet traffic. A current estimate requires selected regions, traffic volume, log retention, and current AWS pricing verification. + +## Generated files and checks + +| Path | Consumer | Local check | +| --- | --- | --- | +| `terraform.tf`, `providers.tf`, `variables.tf` | Terraform | Format and validate | +| `main.tf`, `modules/regional/` | Terraform | Format and validate | +| `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | +| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose config with dummy values | +| `runtime/secret-bindings.json` | CLI and runtime secret loader | JSON parse and deployment check | +| `deployment.yaml` | Existing PBS CLI | `ts prebid server check` only | +| `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | +| `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | + +## Sources and verification + +- PBS Go v4.7.0 release: https://github.com/prebid/prebid-server/releases +- PBS v4.7.0 configuration guide: https://github.com/prebid/prebid-server/blob/v4.7.0/docs/developers/configuration.md +- PBS v4.7.0 configuration definitions: https://github.com/prebid/prebid-server/blob/v4.7.0/config/config.go +- PBS v4.7.0 Docker metadata: https://hub.docker.com/v2/repositories/prebid/prebid-server/tags/v4.7.0 +- ALB requirements: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html +- Route 53 latency routing: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html +- Secrets Manager value updates: https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/put-secret-value.html +- Terraform S3 backend guidance, not selected for this demo: https://developer.hashicorp.com/terraform/language/backend/s3 +- HashiCorp Terraform style guide revision consulted: `c2d65dfe492f74d360d35b859b88932222470bd8` diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md new file mode 100644 index 000000000..c080b4a96 --- /dev/null +++ b/deploy/pbs-example/RUNBOOK.md @@ -0,0 +1,106 @@ +# PBS example runbook + +This runbook describes local checks and the separately authorized operations that would be needed for a real deployment. It does not authorize AWS access, Terraform apply, secret writes, traffic changes, or runtime deployment. + +## Preconditions + +- Work only in `deploy/pbs-example`. +- Use Terraform `1.16.2` and the committed AWS provider lock file. +- Use fictional values from `terraform.tfvars.example` only for local validation. +- Replace the fictional AWS account, profile, certificate, hosted-zone, AMI, CIDR, instance, and secret identifiers before any authorized cloud plan. +- Confirm the approved Trusted Server egress CIDRs. Do not use the documentation CIDR as a real allowlist. +- Confirm the PBS v4.7.0 image digest and every adapter binding again before release preparation. +- Keep state, saved plans, credentials, and AWS provider output out of Git and public logs. + +## Local validation + +From the repository root: + +```bash +terraform fmt -recursive deploy/pbs-example +terraform -chdir=deploy/pbs-example init -backend=false -input=false +terraform -chdir=deploy/pbs-example validate +``` + +`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. `validate` checks configuration structure, not AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. + +Check the PBS descriptor without AWS access: + +```bash +cargo run_cli_linux prebid server check \ + --deployment deploy/pbs-example/deployment.yaml \ + --json +``` + +Expected result: the descriptor, image, bindings, regions, and regional YAML merge pass local validation. This command does not retrieve secrets, contact AWS, pull the PBS image, or start PBS. + +Check Compose with dummy values: + +```bash +docker compose \ + --env-file deploy/pbs-example/runtime/examples/compose.env \ + -f deploy/pbs-example/runtime/compose.yaml \ + config --quiet +``` + +Expected result: Compose renders successfully without pulling or starting the image. The dummy environment file is not a credential. + +Validate the JSON input: + +```bash +python3 -m json.tool deploy/pbs-example/runtime/secret-bindings.json >/dev/null +``` + +## Authorized Terraform workflow + +These steps remain deferred. They require an approved AWS account, role, region scope, cost limit, and operator authorization. + +1. Replace fictional variables with approved values in a protected local tfvars file. +2. Recheck the AWS identity and account against the variables. +3. Run `terraform init` with the local backend and review any lock-file change. +4. Run `terraform plan -out=pbs-example.tfplan`. +5. Review ALB exposure, CIDR rules, NAT gateways, IAM secret access, Route 53 records, instance placement, and expected cost. +6. Store the saved plan privately with its checksum. Do not place it in Git or a public CI log. +7. Apply only the reviewed saved plan after a separate approval. + +Failure behavior: stop on wrong-account identity, an unexpected resource action, a certificate or zone mismatch, broad ingress, missing egress approval, or a secret policy broader than the declared bidder bindings. Reconcile the inputs and create a new plan. Never reuse a stale saved plan after source, state, credentials, or assumptions change. + +## Secret value workflow + +Terraform creates regional secret metadata and grants the EC2 role read access. It never writes credential values. + +After an authorized operator has created the infrastructure and verified the declared secret ARNs, write a complete value through the existing CLI: + +```bash +ts prebid server secrets set examplebidder \ + --deployment /secure/path/deployment.yaml \ + --region us-east-1 \ + --file /secure/path/examplebidder.json \ + --request-token 11111111-2222-4333-8444-555555555555 \ + --yes --json +``` + +Inputs: an approved profile, the exact descriptor, one complete JSON payload, a fresh retry UUID, and operator approval. The payload must contain only the declared keys and must not appear in command arguments or logs. + +Output: a nonsecret Secrets Manager version identifier and the target region. The command does not deploy PBS, refresh Compose, or prove bidder authorization. + +Failure behavior: preserve the retry UUID and input file. If the result is uncertain, retry the identical logical write rather than creating a new version. On partial regional success, record each region separately. + +## Runtime release and rollback + +The current CLI has no deploy or rollback command. A future approved runtime implementation must: + +1. Resolve baseline plus regional overrides into one YAML file per region. +2. Retrieve and validate the selected immutable secret version. +3. Render `/run/pbs/secrets/examplebidder.env` with restrictive permissions, without shell evaluation. +4. Start or replace Compose with the pinned PBS image and read-only YAML. +5. Check ALB target health, `/status`, logs, and an isolated auction fixture. +6. Drain the old consumer before retiring it. +7. Record the release digest, resolved-config checksum, secret version, target, and result. +8. Use the recorded prior release and compatible secret version for rollback. + +Do not manually edit generated runtime environment files. A secret update alone does not change a running container. + +## Deferred evidence + +An authorized operator still needs to verify TLS and DNS, IAM permissions, public ingress, NAT and bidder egress, real adapter credentials, AMI contents, reboot and host replacement, interrupted runtime replacement, secret rotation, representative load, regional failover, alarm delivery, and the Trusted Server auction path. diff --git a/deploy/pbs-example/deployment.yaml b/deploy/pbs-example/deployment.yaml new file mode 100644 index 000000000..ee496a570 --- /dev/null +++ b/deploy/pbs-example/deployment.yaml @@ -0,0 +1,22 @@ +# Fictional local-check descriptor. Replace all account, profile, secret, and instance IDs before authorized use. +schema_version: 1 +environment: example +runtime: ec2-compose +aws: + account_id: '123456789012' + profile: pbs-example +pbs: + config: runtime/pbs.yaml + image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 + bindings: runtime/secret-bindings.json +regions: + us-east-1: + instance_ids: + - i-0123456789abcde01 + - i-0123456789abcde02 + overrides: runtime/regions/east.yaml + us-west-2: + instance_ids: + - i-0fedcba9876543210 + - i-0fedcba9876543211 + overrides: runtime/regions/west.yaml diff --git a/deploy/pbs-example/dns.tf b/deploy/pbs-example/dns.tf new file mode 100644 index 000000000..902c22a18 --- /dev/null +++ b/deploy/pbs-example/dns.tf @@ -0,0 +1,35 @@ +resource "aws_route53_record" "east" { + zone_id = var.route53_zone_id + name = var.pbs_hostname + type = "A" + + set_identifier = "us-east-1" + + alias { + name = module.east.alb_dns_name + zone_id = module.east.alb_zone_id + evaluate_target_health = true + } + + latency_routing_policy { + region = "us-east-1" + } +} + +resource "aws_route53_record" "west" { + zone_id = var.route53_zone_id + name = var.pbs_hostname + type = "A" + + set_identifier = "us-west-2" + + alias { + name = module.west.alb_dns_name + zone_id = module.west.alb_zone_id + evaluate_target_health = true + } + + latency_routing_policy { + region = "us-west-2" + } +} diff --git a/deploy/pbs-example/locals.tf b/deploy/pbs-example/locals.tf new file mode 100644 index 000000000..2e19de5dc --- /dev/null +++ b/deploy/pbs-example/locals.tf @@ -0,0 +1,8 @@ +locals { + common_tags = merge({ + Environment = "example" + ManagedBy = "Terraform" + Project = "prebid-server-example" + Owner = "example-team" + }, var.tags) +} diff --git a/deploy/pbs-example/main.tf b/deploy/pbs-example/main.tf new file mode 100644 index 000000000..930c4c605 --- /dev/null +++ b/deploy/pbs-example/main.tf @@ -0,0 +1,41 @@ +module "east" { + source = "./modules/regional" + + providers = { + aws = aws + } + + alarm_actions = var.alarm_actions + availability_zones = var.east_availability_zones + ami_id = var.east_ami_id + certificate_arn = var.east_certificate_arn + instance_type = var.instance_type + log_retention_days = var.log_retention_days + name = "pbs-example-us-east-1" + region = "us-east-1" + secrets_kms_key_arn = var.secrets_kms_key_arn + tags = local.common_tags + trusted_server_cidr_blocks = var.trusted_server_cidr_blocks + vpc_cidr = var.east_vpc_cidr +} + +module "west" { + source = "./modules/regional" + + providers = { + aws = aws.west + } + + alarm_actions = var.alarm_actions + availability_zones = var.west_availability_zones + ami_id = var.west_ami_id + certificate_arn = var.west_certificate_arn + instance_type = var.instance_type + log_retention_days = var.log_retention_days + name = "pbs-example-us-west-2" + region = "us-west-2" + secrets_kms_key_arn = var.secrets_kms_key_arn + tags = local.common_tags + trusted_server_cidr_blocks = var.trusted_server_cidr_blocks + vpc_cidr = var.west_vpc_cidr +} diff --git a/deploy/pbs-example/modules/regional/README.md b/deploy/pbs-example/modules/regional/README.md new file mode 100644 index 000000000..b1f3a2868 --- /dev/null +++ b/deploy/pbs-example/modules/regional/README.md @@ -0,0 +1,5 @@ +# Regional PBS infrastructure module + +This local module creates one regional ALB, two public and two private subnets, one NAT gateway per AZ, one private EC2 Compose host per AZ, Secrets Manager metadata, IAM access, and CloudWatch alarms. + +It does not install software, write secret values, deploy PBS, or perform runtime replacement. diff --git a/deploy/pbs-example/modules/regional/compute.tf b/deploy/pbs-example/modules/regional/compute.tf new file mode 100644 index 000000000..9b9e2b076 --- /dev/null +++ b/deploy/pbs-example/modules/regional/compute.tf @@ -0,0 +1,31 @@ +resource "aws_instance" "pbs" { + for_each = local.availability_zone_index + + ami = var.ami_id + instance_type = var.instance_type + iam_instance_profile = aws_iam_instance_profile.pbs.name + subnet_id = aws_subnet.private[each.key].id + vpc_security_group_ids = [aws_security_group.pbs.id] + associate_public_ip_address = false + + metadata_options { + http_endpoint = "enabled" + http_protocol_ipv6 = "disabled" + http_put_response_hop_limit = 1 + http_tokens = "required" + instance_metadata_tags = "disabled" + } + + root_block_device { + delete_on_termination = true + encrypted = true + volume_size = 30 + volume_type = "gp3" + } + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-pbs" + }) + + depends_on = [aws_iam_role_policy.runtime] +} diff --git a/deploy/pbs-example/modules/regional/load_balancing.tf b/deploy/pbs-example/modules/regional/load_balancing.tf new file mode 100644 index 000000000..20cf58d64 --- /dev/null +++ b/deploy/pbs-example/modules/regional/load_balancing.tf @@ -0,0 +1,59 @@ +resource "aws_lb" "main" { + name = substr("${var.name}-alb", 0, 32) + internal = false + load_balancer_type = "application" + security_groups = [aws_security_group.alb.id] + subnets = [for availability_zone in var.availability_zones : aws_subnet.public[availability_zone].id] + drop_invalid_header_fields = true + idle_timeout = 30 + + tags = merge(var.tags, { + Name = "${var.name}-alb" + }) +} + +resource "aws_lb_target_group" "pbs" { + name = substr("${var.name}-pbs", 0, 32) + port = var.pbs_port + protocol = "HTTP" + target_type = "instance" + vpc_id = aws_vpc.main.id + deregistration_delay = 30 + + health_check { + enabled = true + path = "/status" + port = "traffic-port" + protocol = "HTTP" + matcher = "200-399" + interval = 15 + timeout = 5 + healthy_threshold = 2 + unhealthy_threshold = 3 + } + + tags = merge(var.tags, { + Name = "${var.name}-pbs" + }) +} + +resource "aws_lb_target_group_attachment" "pbs" { + for_each = aws_instance.pbs + + target_group_arn = aws_lb_target_group.pbs.arn + target_id = each.value.id + port = var.pbs_port +} + +resource "aws_lb_listener" "https" { + load_balancer_arn = aws_lb.main.arn + port = 443 + protocol = "HTTPS" + ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06" + certificate_arn = var.certificate_arn + + default_action { + type = "forward" + target_group_arn = aws_lb_target_group.pbs.arn + } +} diff --git a/deploy/pbs-example/modules/regional/locals.tf b/deploy/pbs-example/modules/regional/locals.tf new file mode 100644 index 000000000..92722f50b --- /dev/null +++ b/deploy/pbs-example/modules/regional/locals.tf @@ -0,0 +1,9 @@ +locals { + availability_zone_index = { + for index, availability_zone in var.availability_zones : availability_zone => index + } + + bidder_secret_names = { + examplebidder = "${var.name}/examplebidder" + } +} diff --git a/deploy/pbs-example/modules/regional/monitoring.tf b/deploy/pbs-example/modules/regional/monitoring.tf new file mode 100644 index 000000000..c170015a0 --- /dev/null +++ b/deploy/pbs-example/modules/regional/monitoring.tf @@ -0,0 +1,74 @@ +resource "aws_cloudwatch_log_group" "runtime" { + name = "/pbs-example/${var.region}" + retention_in_days = var.log_retention_days + + tags = merge(var.tags, { + Name = "${var.name}-runtime" + }) +} + +resource "aws_cloudwatch_metric_alarm" "instance_cpu" { + for_each = aws_instance.pbs + + alarm_name = "${var.name}-${each.key}-high-cpu" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = 3 + metric_name = "CPUUtilization" + namespace = "AWS/EC2" + period = 60 + statistic = "Average" + threshold = 80 + alarm_actions = var.alarm_actions + treat_missing_data = "breaching" + + dimensions = { + InstanceId = each.value.id + } + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-high-cpu" + }) +} + +resource "aws_cloudwatch_metric_alarm" "unhealthy_hosts" { + alarm_name = "${var.name}-unhealthy-hosts" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = 2 + metric_name = "UnHealthyHostCount" + namespace = "AWS/ApplicationELB" + period = 60 + statistic = "Maximum" + threshold = 0 + alarm_actions = var.alarm_actions + treat_missing_data = "breaching" + + dimensions = { + LoadBalancer = aws_lb.main.arn_suffix + TargetGroup = aws_lb_target_group.pbs.arn_suffix + } + + tags = merge(var.tags, { + Name = "${var.name}-unhealthy-hosts" + }) +} + +resource "aws_cloudwatch_metric_alarm" "elb_5xx" { + alarm_name = "${var.name}-elb-5xx" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = 3 + metric_name = "HTTPCode_ELB_5XX_Count" + namespace = "AWS/ApplicationELB" + period = 60 + statistic = "Sum" + threshold = 1 + alarm_actions = var.alarm_actions + treat_missing_data = "notBreaching" + + dimensions = { + LoadBalancer = aws_lb.main.arn_suffix + } + + tags = merge(var.tags, { + Name = "${var.name}-elb-5xx" + }) +} diff --git a/deploy/pbs-example/modules/regional/network.tf b/deploy/pbs-example/modules/regional/network.tf new file mode 100644 index 000000000..a1f251012 --- /dev/null +++ b/deploy/pbs-example/modules/regional/network.tf @@ -0,0 +1,109 @@ +resource "aws_vpc" "main" { + cidr_block = var.vpc_cidr + enable_dns_hostnames = true + enable_dns_support = true + + tags = merge(var.tags, { + Name = "${var.name}-vpc" + }) +} + +resource "aws_internet_gateway" "main" { + vpc_id = aws_vpc.main.id + + tags = merge(var.tags, { + Name = "${var.name}-igw" + }) +} + +resource "aws_subnet" "public" { + for_each = local.availability_zone_index + + vpc_id = aws_vpc.main.id + availability_zone = each.key + cidr_block = cidrsubnet(var.vpc_cidr, 4, each.value) + map_public_ip_on_launch = false + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-public" + Tier = "public" + }) +} + +resource "aws_subnet" "private" { + for_each = local.availability_zone_index + + vpc_id = aws_vpc.main.id + availability_zone = each.key + cidr_block = cidrsubnet(var.vpc_cidr, 4, each.value + 8) + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-private" + Tier = "private" + }) +} + +resource "aws_route_table" "public" { + vpc_id = aws_vpc.main.id + + route { + cidr_block = "0.0.0.0/0" + gateway_id = aws_internet_gateway.main.id + } + + tags = merge(var.tags, { + Name = "${var.name}-public" + }) +} + +resource "aws_route_table_association" "public" { + for_each = local.availability_zone_index + + route_table_id = aws_route_table.public.id + subnet_id = aws_subnet.public[each.key].id +} + +resource "aws_eip" "nat" { + for_each = local.availability_zone_index + + domain = "vpc" + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-nat" + }) +} + +resource "aws_nat_gateway" "main" { + for_each = local.availability_zone_index + + allocation_id = aws_eip.nat[each.key].id + subnet_id = aws_subnet.public[each.key].id + + depends_on = [aws_internet_gateway.main] + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-nat" + }) +} + +resource "aws_route_table" "private" { + for_each = local.availability_zone_index + + vpc_id = aws_vpc.main.id + + route { + cidr_block = "0.0.0.0/0" + nat_gateway_id = aws_nat_gateway.main[each.key].id + } + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-private" + }) +} + +resource "aws_route_table_association" "private" { + for_each = local.availability_zone_index + + route_table_id = aws_route_table.private[each.key].id + subnet_id = aws_subnet.private[each.key].id +} diff --git a/deploy/pbs-example/modules/regional/outputs.tf b/deploy/pbs-example/modules/regional/outputs.tf new file mode 100644 index 000000000..1b3b47a4f --- /dev/null +++ b/deploy/pbs-example/modules/regional/outputs.tf @@ -0,0 +1,29 @@ +output "alb_dns_name" { + description = "Regional ALB DNS name." + value = aws_lb.main.dns_name +} + +output "alb_zone_id" { + description = "Route 53 hosted-zone ID for the regional ALB alias." + value = aws_lb.main.zone_id +} + +output "instance_ids" { + description = "PBS EC2 instance IDs keyed by Availability Zone." + value = { for availability_zone, instance in aws_instance.pbs : availability_zone => instance.id } +} + +output "private_subnet_ids" { + description = "Private subnet IDs keyed by Availability Zone." + value = { for availability_zone, subnet in aws_subnet.private : availability_zone => subnet.id } +} + +output "secret_arns" { + description = "Bidder Secrets Manager ARNs keyed by bidder identifier." + value = { for bidder, secret in aws_secretsmanager_secret.bidder : bidder => secret.arn } +} + +output "vpc_id" { + description = "Regional VPC ID." + value = aws_vpc.main.id +} diff --git a/deploy/pbs-example/modules/regional/secrets.tf b/deploy/pbs-example/modules/regional/secrets.tf new file mode 100644 index 000000000..7e8e89120 --- /dev/null +++ b/deploy/pbs-example/modules/regional/secrets.tf @@ -0,0 +1,80 @@ +resource "aws_secretsmanager_secret" "bidder" { + for_each = local.bidder_secret_names + + description = "Credential payload for ${each.key}; values are written outside Terraform." + name = each.value + kms_key_id = var.secrets_kms_key_arn + recovery_window_in_days = 7 + + tags = merge(var.tags, { + Name = each.value + Secret = each.key + }) +} + +resource "aws_iam_role" "pbs" { + name = "${var.name}-runtime" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ec2.amazonaws.com" + } + }] + }) + + tags = merge(var.tags, { + Name = "${var.name}-runtime" + }) +} + +resource "aws_iam_instance_profile" "pbs" { + name = "${var.name}-runtime" + role = aws_iam_role.pbs.name + + tags = merge(var.tags, { + Name = "${var.name}-runtime" + }) +} + +resource "aws_iam_role_policy_attachment" "ssm" { + role = aws_iam_role.pbs.name + policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" +} + +resource "aws_iam_role_policy" "runtime" { + name = "runtime" + role = aws_iam_role.pbs.name + + policy = jsonencode({ + Version = "2012-10-17" + Statement = concat([ + { + Sid = "ReadBidderSecrets" + Effect = "Allow" + Action = [ + "secretsmanager:DescribeSecret", + "secretsmanager:GetSecretValue", + ] + Resource = [for secret in aws_secretsmanager_secret.bidder : secret.arn] + }, + { + Sid = "WriteRuntimeLogs" + Effect = "Allow" + Action = [ + "logs:CreateLogStream", + "logs:DescribeLogStreams", + "logs:PutLogEvents", + ] + Resource = "${aws_cloudwatch_log_group.runtime.arn}:*" + }, + ], var.secrets_kms_key_arn == null ? [] : [{ + Sid = "DecryptBidderSecrets" + Effect = "Allow" + Action = ["kms:Decrypt"] + Resource = var.secrets_kms_key_arn + }]) + }) +} diff --git a/deploy/pbs-example/modules/regional/security.tf b/deploy/pbs-example/modules/regional/security.tf new file mode 100644 index 000000000..40257be3a --- /dev/null +++ b/deploy/pbs-example/modules/regional/security.tf @@ -0,0 +1,55 @@ +resource "aws_security_group" "alb" { + name = "${var.name}-alb" + description = "HTTPS ingress for the regional PBS ALB" + vpc_id = aws_vpc.main.id + + dynamic "ingress" { + for_each = var.trusted_server_cidr_blocks + + content { + description = "Trusted Server HTTPS" + from_port = 443 + to_port = 443 + protocol = "tcp" + cidr_blocks = [ingress.value] + } + } + + egress { + description = "Forward requests to private PBS hosts" + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } + + tags = merge(var.tags, { + Name = "${var.name}-alb" + }) +} + +resource "aws_security_group" "pbs" { + name = "${var.name}-pbs" + description = "PBS host traffic from the regional ALB and outbound bidder access" + vpc_id = aws_vpc.main.id + + ingress { + description = "PBS HTTP from the regional ALB" + from_port = var.pbs_port + to_port = var.pbs_port + protocol = "tcp" + security_groups = [aws_security_group.alb.id] + } + + egress { + description = "Bidder and AWS API access through the per-AZ NAT gateway" + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } + + tags = merge(var.tags, { + Name = "${var.name}-pbs" + }) +} diff --git a/deploy/pbs-example/modules/regional/terraform.tf b/deploy/pbs-example/modules/regional/terraform.tf new file mode 100644 index 000000000..f2702bf6e --- /dev/null +++ b/deploy/pbs-example/modules/regional/terraform.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + } + } +} diff --git a/deploy/pbs-example/modules/regional/variables.tf b/deploy/pbs-example/modules/regional/variables.tf new file mode 100644 index 000000000..da5c33804 --- /dev/null +++ b/deploy/pbs-example/modules/regional/variables.tf @@ -0,0 +1,76 @@ +variable "alarm_actions" { + description = "CloudWatch alarm action ARNs." + type = set(string) +} + +variable "ami_id" { + description = "Pre-baked regional AMI containing the approved host runtime." + type = string +} + +variable "availability_zones" { + description = "Exactly two Availability Zones for the regional ALB and PBS hosts." + type = list(string) + + validation { + condition = length(var.availability_zones) == 2 && length(distinct(var.availability_zones)) == 2 + error_message = "availability_zones must contain exactly two distinct Availability Zones." + } +} + +variable "certificate_arn" { + description = "Existing ACM certificate ARN for the regional ALB." + type = string +} + +variable "instance_type" { + description = "EC2 instance type for the regional PBS hosts." + type = string +} + +variable "log_retention_days" { + description = "CloudWatch Logs retention period." + type = number +} + +variable "name" { + description = "Stable name prefix for regional resources." + type = string +} + +variable "pbs_port" { + description = "Host port used by the PBS Compose service." + type = number + default = 8000 + + validation { + condition = var.pbs_port >= 1 && var.pbs_port <= 65535 + error_message = "pbs_port must be a valid TCP port." + } +} + +variable "region" { + description = "AWS region represented by this module instance." + type = string +} + +variable "secrets_kms_key_arn" { + description = "Optional customer-managed KMS key ARN for regional bidder secrets." + type = string + default = null +} + +variable "tags" { + description = "Common resource tags." + type = map(string) +} + +variable "trusted_server_cidr_blocks" { + description = "CIDR blocks permitted to reach the regional ALB." + type = set(string) +} + +variable "vpc_cidr" { + description = "Regional VPC CIDR block." + type = string +} diff --git a/deploy/pbs-example/outputs.tf b/deploy/pbs-example/outputs.tf new file mode 100644 index 000000000..efd88b222 --- /dev/null +++ b/deploy/pbs-example/outputs.tf @@ -0,0 +1,29 @@ +output "east_alb_dns_name" { + description = "DNS name of the us-east-1 regional ALB." + value = module.east.alb_dns_name +} + +output "east_instance_ids" { + description = "PBS EC2 instance IDs in us-east-1." + value = module.east.instance_ids +} + +output "east_secret_arns" { + description = "Bidder Secrets Manager ARNs in us-east-1. Values are never managed by Terraform." + value = module.east.secret_arns +} + +output "west_alb_dns_name" { + description = "DNS name of the us-west-2 regional ALB." + value = module.west.alb_dns_name +} + +output "west_instance_ids" { + description = "PBS EC2 instance IDs in us-west-2." + value = module.west.instance_ids +} + +output "west_secret_arns" { + description = "Bidder Secrets Manager ARNs in us-west-2. Values are never managed by Terraform." + value = module.west.secret_arns +} diff --git a/deploy/pbs-example/providers.tf b/deploy/pbs-example/providers.tf new file mode 100644 index 000000000..875201637 --- /dev/null +++ b/deploy/pbs-example/providers.tf @@ -0,0 +1,20 @@ +provider "aws" { + profile = var.aws_profile + region = "us-east-1" + allowed_account_ids = [var.aws_account_id] + + default_tags { + tags = local.common_tags + } +} + +provider "aws" { + alias = "west" + profile = var.aws_profile + region = "us-west-2" + allowed_account_ids = [var.aws_account_id] + + default_tags { + tags = local.common_tags + } +} diff --git a/deploy/pbs-example/runtime/README.md b/deploy/pbs-example/runtime/README.md new file mode 100644 index 000000000..88d9eebd7 --- /dev/null +++ b/deploy/pbs-example/runtime/README.md @@ -0,0 +1,7 @@ +# PBS runtime example + +This directory owns nonsecret PBS configuration and the Compose shape. The image is pinned to Prebid Server Go v4.7.0 by digest. + +The `/run/pbs/secrets/examplebidder.env` file is a runtime contract, not a checked-in credential file. A separately approved host-side loader must retrieve the selected regional Secrets Manager version, validate the complete JSON payload, render this restricted environment file atomically, and start or replace Compose. The current `ts prebid server` CLI does not implement that release or injection workflow. + +`examples/pbs-secrets.env` contains dummy values for local Compose parsing only. diff --git a/deploy/pbs-example/runtime/compose.yaml b/deploy/pbs-example/runtime/compose.yaml new file mode 100644 index 000000000..38a2167d8 --- /dev/null +++ b/deploy/pbs-example/runtime/compose.yaml @@ -0,0 +1,15 @@ +services: + pbs: + image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 + restart: unless-stopped + ports: + - "8000:8000" + env_file: + - ${PBS_SECRET_ENV_FILE:-/run/pbs/secrets/examplebidder.env} + volumes: + - ${PBS_CONFIG_FILE:-./pbs.yaml}:/etc/config/pbs.yaml:ro + read_only: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL diff --git a/deploy/pbs-example/runtime/examples/README.md b/deploy/pbs-example/runtime/examples/README.md new file mode 100644 index 000000000..fa0017b87 --- /dev/null +++ b/deploy/pbs-example/runtime/examples/README.md @@ -0,0 +1 @@ +These files contain fictional values for local structure checks only. They are not bidder credentials and must never be copied into an AWS secret. \ No newline at end of file diff --git a/deploy/pbs-example/runtime/examples/compose.env b/deploy/pbs-example/runtime/examples/compose.env new file mode 100644 index 000000000..a1727c63e --- /dev/null +++ b/deploy/pbs-example/runtime/examples/compose.env @@ -0,0 +1,3 @@ +# Local-only Compose validation selectors. +PBS_CONFIG_FILE=./pbs.yaml +PBS_SECRET_ENV_FILE=./examples/pbs-secrets.env diff --git a/deploy/pbs-example/runtime/examples/pbs-secrets.env b/deploy/pbs-example/runtime/examples/pbs-secrets.env new file mode 100644 index 000000000..ff5644bab --- /dev/null +++ b/deploy/pbs-example/runtime/examples/pbs-secrets.env @@ -0,0 +1,3 @@ +# Fictional dummy values for local Compose parsing only. +PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY=example-only-api-key +PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN=example-only-optional-token diff --git a/deploy/pbs-example/runtime/pbs.yaml b/deploy/pbs-example/runtime/pbs.yaml new file mode 100644 index 000000000..858d694e5 --- /dev/null +++ b/deploy/pbs-example/runtime/pbs.yaml @@ -0,0 +1,11 @@ +# PBS Go v4.7.0 baseline. Verify every field against the pinned release before deployment. +external_url: https://pbs.example.com +host: 0.0.0.0 +port: 8000 +status_response: ok +auction_timeouts_ms: + default: 1000 + max: 1200 +stored_requests_timeout_ms: 100 +gdpr: + default_value: "1" diff --git a/deploy/pbs-example/runtime/regions/east.yaml b/deploy/pbs-example/runtime/regions/east.yaml new file mode 100644 index 000000000..58cb7ae81 --- /dev/null +++ b/deploy/pbs-example/runtime/regions/east.yaml @@ -0,0 +1,2 @@ +# Regional nonsecret override. The shared external URL remains the Route 53 hostname. +datacenter: us-east-1 diff --git a/deploy/pbs-example/runtime/regions/west.yaml b/deploy/pbs-example/runtime/regions/west.yaml new file mode 100644 index 000000000..84227de00 --- /dev/null +++ b/deploy/pbs-example/runtime/regions/west.yaml @@ -0,0 +1,2 @@ +# Regional nonsecret override. The shared external URL remains the Route 53 hostname. +datacenter: us-west-2 diff --git a/deploy/pbs-example/runtime/secret-bindings.json b/deploy/pbs-example/runtime/secret-bindings.json new file mode 100644 index 000000000..dc6340844 --- /dev/null +++ b/deploy/pbs-example/runtime/secret-bindings.json @@ -0,0 +1,22 @@ +{ + "examplebidder": { + "verified_image": "prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7", + "source": "https://example.com/pbs-adapter-reference", + "secrets": { + "us-east-1": "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs-example-us-east-1/examplebidder-AbCdEf", + "us-west-2": "arn:aws:secretsmanager:us-west-2:123456789012:secret:pbs-example-us-west-2/examplebidder-GhIjKl" + }, + "keys": { + "api_key": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", + "pbs_path": ["adapters", "examplebidder", "api_key"], + "required": true + }, + "optional_token": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN", + "pbs_path": ["adapters", "examplebidder", "optional_token"], + "required": false + } + } + } +} diff --git a/deploy/pbs-example/terraform.tf b/deploy/pbs-example/terraform.tf new file mode 100644 index 000000000..b857d386f --- /dev/null +++ b/deploy/pbs-example/terraform.tf @@ -0,0 +1,14 @@ +terraform { + required_version = ">= 1.16.2, < 1.17.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "= 6.64.0" + } + } + + backend "local" { + path = "terraform.tfstate" + } +} diff --git a/deploy/pbs-example/terraform.tfvars.example b/deploy/pbs-example/terraform.tfvars.example new file mode 100644 index 000000000..733d0347a --- /dev/null +++ b/deploy/pbs-example/terraform.tfvars.example @@ -0,0 +1,14 @@ +# Fictional local example values. Do not use these identifiers for AWS access. +aws_account_id = "123456789012" +aws_profile = "pbs-example" + +east_ami_id = "ami-0123456789abcdef0" +east_certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-1111-4111-8111-111111111111" +west_ami_id = "ami-0fedcba9876543210" +west_certificate_arn = "arn:aws:acm:us-west-2:123456789012:certificate/22222222-2222-4222-8222-222222222222" + +# Existing hosted-zone ID in the fictional example account. +route53_zone_id = "Z00000000000000000000" + +# Replace with the actual egress ranges approved for Trusted Server. +trusted_server_cidr_blocks = ["203.0.113.0/24"] diff --git a/deploy/pbs-example/variables.tf b/deploy/pbs-example/variables.tf new file mode 100644 index 000000000..fdace5889 --- /dev/null +++ b/deploy/pbs-example/variables.tf @@ -0,0 +1,150 @@ +variable "alarm_actions" { + description = "SNS topic ARNs that should receive CloudWatch alarm notifications. Leave empty for this local example." + type = set(string) + default = [] +} + +variable "aws_account_id" { + description = "The 12-digit AWS account expected by Terraform. Replace the fictional example value before any authorized plan." + type = string + + validation { + condition = can(regex("^[0-9]{12}$", var.aws_account_id)) + error_message = "aws_account_id must be a 12-digit AWS account ID." + } +} + +variable "aws_profile" { + description = "The short-lived AWS CLI profile used by an authorized operator." + type = string +} + +variable "east_ami_id" { + description = "Pre-baked us-east-1 AMI containing the approved host runtime, SSM agent, Docker, and Compose." + type = string + + validation { + condition = can(regex("^ami-[0-9a-f]+$", var.east_ami_id)) + error_message = "east_ami_id must be an AMI ID." + } +} + +variable "east_certificate_arn" { + description = "Existing ACM certificate ARN for the us-east-1 regional ALB." + type = string + + validation { + condition = can(regex("^arn:aws[a-z-]*:acm:us-east-1:[0-9]{12}:certificate/.+$", var.east_certificate_arn)) + error_message = "east_certificate_arn must be an ACM certificate ARN in us-east-1." + } +} + +variable "east_vpc_cidr" { + description = "CIDR block for the us-east-1 VPC." + type = string + default = "10.80.0.0/16" +} + +variable "east_availability_zones" { + description = "Two us-east-1 Availability Zones used by the ALB and PBS host." + type = list(string) + default = ["us-east-1a", "us-east-1b"] + + validation { + condition = length(var.east_availability_zones) == 2 && length(distinct(var.east_availability_zones)) == 2 + error_message = "east_availability_zones must contain two distinct Availability Zones." + } +} + +variable "instance_type" { + description = "EC2 instance type for each PBS host. This is a starting assumption, not a capacity guarantee." + type = string + default = "c7i.large" +} + +variable "log_retention_days" { + description = "CloudWatch log retention for each regional runtime log group." + type = number + default = 14 + + validation { + condition = contains([1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653], var.log_retention_days) + error_message = "log_retention_days must be a supported CloudWatch Logs retention period." + } +} + +variable "pbs_hostname" { + description = "Shared HTTPS hostname returned by Route 53 latency records." + type = string + default = "pbs.example.com" + + validation { + condition = !can(regex("[/\\s]", var.pbs_hostname)) + error_message = "pbs_hostname must be a hostname without a path or whitespace." + } +} + +variable "route53_zone_id" { + description = "Existing Route 53 public hosted zone ID that owns pbs_hostname." + type = string +} + +variable "secrets_kms_key_arn" { + description = "Optional customer-managed KMS key ARN for bidder secrets. Null uses the Secrets Manager service key." + type = string + default = null +} + +variable "tags" { + description = "Additional tags applied to supported AWS resources." + type = map(string) + default = {} +} + +variable "trusted_server_cidr_blocks" { + description = "Approved Trusted Server egress CIDR blocks allowed to reach the public ALBs. The default is documentation-only." + type = set(string) + default = ["203.0.113.0/24"] + + validation { + condition = length(var.trusted_server_cidr_blocks) > 0 && alltrue([for cidr in var.trusted_server_cidr_blocks : can(cidrhost(cidr, 0))]) + error_message = "trusted_server_cidr_blocks must contain at least one valid CIDR block." + } +} + +variable "west_ami_id" { + description = "Pre-baked us-west-2 AMI containing the approved host runtime, SSM agent, Docker, and Compose." + type = string + + validation { + condition = can(regex("^ami-[0-9a-f]+$", var.west_ami_id)) + error_message = "west_ami_id must be an AMI ID." + } +} + +variable "west_certificate_arn" { + description = "Existing ACM certificate ARN for the us-west-2 regional ALB." + type = string + + validation { + condition = can(regex("^arn:aws[a-z-]*:acm:us-west-2:[0-9]{12}:certificate/.+$", var.west_certificate_arn)) + error_message = "west_certificate_arn must be an ACM certificate ARN in us-west-2." + } +} + +variable "west_vpc_cidr" { + description = "CIDR block for the us-west-2 VPC." + type = string + default = "10.81.0.0/16" +} + +variable "west_availability_zones" { + description = "Two us-west-2 Availability Zones used by the ALB and PBS host." + type = list(string) + default = ["us-west-2a", "us-west-2b"] + + validation { + condition = length(var.west_availability_zones) == 2 && length(distinct(var.west_availability_zones)) == 2 + error_message = "west_availability_zones must contain two distinct Availability Zones." + } +} From a36cc8b38fcdc4b5bc8dd30490360b85381cfad2 Mon Sep 17 00:00:00 2001 From: Christian Date: Wed, 16 Sep 2026 16:51:32 -0500 Subject: [PATCH 06/14] Clarify PBS example operator workflow --- deploy/pbs-example/.gitignore | 5 + deploy/pbs-example/DEPLOYMENT_PLAN.md | 18 ++- deploy/pbs-example/README.md | 79 ++++++++++ deploy/pbs-example/RUNBOOK.md | 41 ++++- ...eployment.yaml => deployment.example.yaml} | 2 +- deploy/pbs-example/locals.tf | 50 ++++++ deploy/pbs-example/main.tf | 8 +- .../modules/regional/monitoring.tf | 9 -- .../pbs-example/modules/regional/outputs.tf | 5 + .../pbs-example/modules/regional/secrets.tf | 10 -- .../tests/security_unit_test.tftest.hcl | 83 ++++++++++ .../pbs-example/modules/regional/variables.tf | 10 -- deploy/pbs-example/outputs.tf | 20 +++ deploy/pbs-example/runtime/README.md | 4 +- deploy/pbs-example/runtime/compose.yaml | 7 +- ...ings.json => secret-bindings.example.json} | 0 deploy/pbs-example/scripts/smoke-runtime.sh | 43 +++++ deploy/pbs-example/terraform.tfvars.example | 5 + .../tests/root_unit_test.tftest.hcl | 147 ++++++++++++++++++ deploy/pbs-example/variables.tf | 39 +++-- 20 files changed, 515 insertions(+), 70 deletions(-) create mode 100644 deploy/pbs-example/README.md rename deploy/pbs-example/{deployment.yaml => deployment.example.yaml} (92%) create mode 100644 deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl rename deploy/pbs-example/runtime/{secret-bindings.json => secret-bindings.example.json} (100%) create mode 100755 deploy/pbs-example/scripts/smoke-runtime.sh create mode 100644 deploy/pbs-example/tests/root_unit_test.tftest.hcl diff --git a/deploy/pbs-example/.gitignore b/deploy/pbs-example/.gitignore index 018d39b61..d73176818 100644 --- a/deploy/pbs-example/.gitignore +++ b/deploy/pbs-example/.gitignore @@ -1,5 +1,6 @@ # Terraform working data and local state .terraform/ +/modules/**/.terraform.lock.hcl *.tfstate *.tfstate.* .terraform.tfstate.lock.info @@ -7,6 +8,10 @@ # Saved plans can contain provider-returned metadata *.tfplan +# Generated operator inputs +/deployment.generated.yaml +/runtime/secret-bindings.generated.json + # Local inputs and crash output *.auto.tfvars *.tfvars diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md index 70f5b3368..68df5a052 100644 --- a/deploy/pbs-example/DEPLOYMENT_PLAN.md +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -18,7 +18,7 @@ Status: draft generated files, locally checked after validation. This directory | Terraform state | Local backend | Confirmed | User | No team locking or remote recovery | | PBS release | Go v4.7.0, digest pinned | Proposed and verified | PBS release and Docker metadata checked during generation | Recheck before any future use | | Workload | 200 global peak auctions/s, 4 bidders, 1.5 s caller timeout, 1 s PBS timeout | Proposed example assumption | Planning assumption | No capacity claim until load tested | -| Secrets | Secrets Manager metadata and EC2 read policy; values written separately | Confirmed | User and repository CLI contract | Real bidder mapping and authorization required | +| Secrets | Independent regional Secrets Manager metadata and EC2 read policy; values written separately | Confirmed | User and repository CLI contract | Real bidder mapping, authorization, and runtime injection required | | Trusted Server config | No authoritative `trusted-server.toml` exists in this checkout | Confirmed | Repository inspection | Caller behavior remains an external input | | Deployment directory | `deploy/pbs-example/` | Confirmed | User | None | @@ -51,7 +51,7 @@ The EC2 AMIs are inputs rather than built by Terraform. They must contain the ap | ALB, target group, HTTPS listener | Create | Terraform | Regional HTTPS ingress and health routing | | Route 53 records | Reuse zone, create records | Terraform | Latency-based regional selection | | EC2 instances and IAM profiles | Create | Terraform | Compose hosts and SSM access | -| Secrets Manager secret metadata | Create | Terraform | One regional secret for the example bidder | +| Secrets Manager secret metadata | Create | Terraform | One independently encrypted secret per region for the example bidder | | Secret values | External write | Authorized operator or automation | Credential lifecycle; never Terraform | | PBS YAML and Compose definition | Git-owned | Runtime owner | Nonsecret runtime contract | | Regional YAML merge and rendered file | Deferred runtime release owner | Deployment implementation | One resolved config per region | @@ -82,21 +82,25 @@ A controlled load test must measure CPU, memory, connection reuse, outbound band - No WAF or rate-based abuse control is included. - The example uses a fictional bidder binding. The adapter name, credential keys, source authorization, and PBS mapping must be replaced and verified against v4.7.0 before use. - A Secrets Manager write does not refresh a running Compose container. The deferred runtime implementation must retrieve a selected version, render an environment file atomically, replace the consumer, and verify health before retiring the old version. +- Customer-managed KMS keys are regional inputs. East and West accept separate ARNs; null uses each region's Secrets Manager service key. +- The committed descriptor and binding files are fictional local fixtures. After an authorized apply, Terraform renders ignored operator files containing the actual instance IDs and secret ARNs. ## Cost drivers -No price estimate is claimed. The main drivers are four EC2 instances, four NAT gateways and their Elastic IPs, two ALBs, public IPv4 addresses, cross-AZ traffic if routing changes, CloudWatch logs and alarms, Secrets Manager, Route 53 records, and bidder internet traffic. A current estimate requires selected regions, traffic volume, log retention, and current AWS pricing verification. +No price estimate is claimed. The main drivers are four EC2 instances, four NAT gateways and their Elastic IPs, two ALBs, public IPv4 addresses, cross-AZ traffic if routing changes, CloudWatch alarms, Secrets Manager, Route 53 records, and bidder internet traffic. A current estimate requires selected regions, traffic volume, and current AWS pricing verification. ## Generated files and checks | Path | Consumer | Local check | | --- | --- | --- | +| `README.md` | Example user | Safe walkthrough and stop boundary review | | `terraform.tf`, `providers.tf`, `variables.tf` | Terraform | Format and validate | -| `main.tf`, `modules/regional/` | Terraform | Format and validate | +| `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, ingress, and secret access | | `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | -| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose config with dummy values | -| `runtime/secret-bindings.json` | CLI and runtime secret loader | JSON parse and deployment check | -| `deployment.yaml` | Existing PBS CLI | `ts prebid server check` only | +| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and pinned-image startup smoke | +| `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | +| `deployment.example.yaml` | Existing PBS CLI | Fictional local `ts prebid server check` only | +| Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | | `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | | `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md new file mode 100644 index 000000000..219d329a9 --- /dev/null +++ b/deploy/pbs-example/README.md @@ -0,0 +1,79 @@ +# PBS AWS example + +This directory is the locally checked output of the Prebid Server AWS planning workflow. It is a reference for preparing infrastructure and operator inputs. It does not deploy PBS, inject runtime secrets, or change traffic. + +## What it models + +```mermaid +flowchart TD + T["Trusted Server"] --> R["Route 53 latency aliases"] + R --> E["us-east-1 ALB"] + R --> W["us-west-2 ALB"] + E --> E1["Private PBS host, AZ 1"] + E --> E2["Private PBS host, AZ 2"] + W --> W1["Private PBS host, AZ 1"] + W --> W2["Private PBS host, AZ 2"] + E1 --> B["Public bidder endpoints"] + E2 --> B + W1 --> B + W2 --> B +``` + +Each region has two AZs, one EC2 host per AZ, a public ALB, and one NAT gateway per AZ. Terraform creates regional Secrets Manager metadata but never stores credential values. + +## Safe local walkthrough + +These commands do not contact AWS: + +```bash +terraform fmt -check -recursive deploy/pbs-example +terraform -chdir=deploy/pbs-example init -backend=false -input=false +terraform -chdir=deploy/pbs-example validate +terraform -chdir=deploy/pbs-example test -filter=tests/root_unit_test.tftest.hcl +terraform -chdir=deploy/pbs-example/modules/regional init -backend=false -input=false +terraform -chdir=deploy/pbs-example/modules/regional test -filter=tests/security_unit_test.tftest.hcl +cargo run_cli_linux prebid server check \ + --deployment deploy/pbs-example/deployment.example.yaml \ + --json +python3 -m json.tool \ + deploy/pbs-example/runtime/secret-bindings.example.json >/dev/null +docker compose \ + --env-file deploy/pbs-example/runtime/examples/compose.env \ + -f deploy/pbs-example/runtime/compose.yaml \ + config --quiet +deploy/pbs-example/scripts/smoke-runtime.sh +``` + +The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The smoke check pulls and starts the pinned image with dummy values, checks `/status` and startup-log redaction, and sends no auction request. + +## Adapting the example + +1. Copy this directory into an environment-specific deployment root. +2. Replace the fictional values from `terraform.tfvars.example` in a protected, ignored `.tfvars` file. +3. Replace `examplebidder` with adapters and credential mappings verified against the pinned PBS release and approved by each bidder. +4. Supply approved regional AMIs. Each AMI must contain the chosen OS, SSM agent, Docker, and Compose. This repository does not build the AMIs. +5. Run the local checks above. +6. Obtain separate authorization for AWS identity checks, Terraform plan, apply, and secret writes. +7. Follow `RUNBOOK.md` for the reviewed plan and apply process. + +## Generating real CLI inputs after apply + +A successful apply produces actual instance IDs and regional secret ARNs. Render ignored operator files from those outputs instead of editing the fictional fixtures: + +```bash +terraform output -raw deployment_descriptor_yaml > deployment.generated.yaml +terraform output -raw secret_bindings_json > runtime/secret-bindings.generated.json + +ts prebid server check --deployment deployment.generated.yaml +ts prebid server status --deployment deployment.generated.yaml +``` + +Run these commands from this directory. `status` performs authenticated EC2 reads and requires separate authorization. Review both generated files before using them. They contain resource identifiers but no credential values. + +An authorized operator can then write a complete regional secret value with `ts prebid server secrets set`, as documented in `RUNBOOK.md`. + +## Stop boundary + +The current workflow stops after infrastructure preparation and secret-value writes. It does not install a secret loader, resolve regional PBS YAML onto hosts, start or replace Compose, verify application health, or roll back a release. Those operations require a separately approved runtime implementation. A secret write alone does not update a running container. + +See `DEPLOYMENT_PLAN.md` for decisions and limitations, and `RUNBOOK.md` for operator contracts. diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md index c080b4a96..cc76d1b2c 100644 --- a/deploy/pbs-example/RUNBOOK.md +++ b/deploy/pbs-example/RUNBOOK.md @@ -6,8 +6,8 @@ This runbook describes local checks and the separately authorized operations tha - Work only in `deploy/pbs-example`. - Use Terraform `1.16.2` and the committed AWS provider lock file. -- Use fictional values from `terraform.tfvars.example` only for local validation. -- Replace the fictional AWS account, profile, certificate, hosted-zone, AMI, CIDR, instance, and secret identifiers before any authorized cloud plan. +- Use fictional values from `terraform.tfvars.example`, `deployment.example.yaml`, and `runtime/secret-bindings.example.json` only for local validation. +- Replace the fictional AWS account, profile, certificate, hosted-zone, AMI, and CIDR inputs before any authorized cloud plan. Terraform renders actual instance and secret identifiers after apply. - Confirm the approved Trusted Server egress CIDRs. Do not use the documentation CIDR as a real allowlist. - Confirm the PBS v4.7.0 image digest and every adapter binding again before release preparation. - Keep state, saved plans, credentials, and AWS provider output out of Git and public logs. @@ -20,15 +20,21 @@ From the repository root: terraform fmt -recursive deploy/pbs-example terraform -chdir=deploy/pbs-example init -backend=false -input=false terraform -chdir=deploy/pbs-example validate +terraform -chdir=deploy/pbs-example test \ + -filter=tests/root_unit_test.tftest.hcl +terraform -chdir=deploy/pbs-example/modules/regional init \ + -backend=false -input=false +terraform -chdir=deploy/pbs-example/modules/regional test \ + -filter=tests/security_unit_test.tftest.hcl ``` -`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. `validate` checks configuration structure, not AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. +`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs two tests. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. Check the PBS descriptor without AWS access: ```bash cargo run_cli_linux prebid server check \ - --deployment deploy/pbs-example/deployment.yaml \ + --deployment deploy/pbs-example/deployment.example.yaml \ --json ``` @@ -45,12 +51,16 @@ docker compose \ Expected result: Compose renders successfully without pulling or starting the image. The dummy environment file is not a credential. -Validate the JSON input: +Validate the JSON input and start the pinned PBS image against the nonsecret baseline: ```bash -python3 -m json.tool deploy/pbs-example/runtime/secret-bindings.json >/dev/null +python3 -m json.tool \ + deploy/pbs-example/runtime/secret-bindings.example.json >/dev/null +deploy/pbs-example/scripts/smoke-runtime.sh ``` +The smoke script pulls the pinned image if needed, starts it with dummy values on local port `18080`, requires `/status` to return `ok`, rejects either dummy credential appearing in startup logs, and removes its container and network on exit. It sends no auction request and contacts no bidder. Set `PBS_SMOKE_PORT` only when port `18080` is unavailable. + ## Authorized Terraform workflow These steps remain deferred. They require an approved AWS account, role, region scope, cost limit, and operator authorization. @@ -65,6 +75,23 @@ These steps remain deferred. They require an approved AWS account, role, region Failure behavior: stop on wrong-account identity, an unexpected resource action, a certificate or zone mismatch, broad ingress, missing egress approval, or a secret policy broader than the declared bidder bindings. Reconcile the inputs and create a new plan. Never reuse a stale saved plan after source, state, credentials, or assumptions change. +## Generate operator inputs + +After an authorized apply, run these commands from `deploy/pbs-example`: + +```bash +terraform output -raw deployment_descriptor_yaml > deployment.generated.yaml +terraform output -raw secret_bindings_json > runtime/secret-bindings.generated.json + +ts prebid server check --deployment deployment.generated.yaml --json +``` + +Inputs: the applied local state and unchanged Terraform source. Access: local state reads only; the CLI check makes no AWS call. Outputs: ignored, nonsecret files containing actual instance IDs and secret ARNs. + +Failure behavior: stop if either output is empty, references the wrong account or region, or fails the CLI check. Do not repair generated identifiers manually. Reconcile Terraform source and state, then render both files again. + +`ts prebid server status --deployment deployment.generated.yaml` is a separate authenticated EC2 read. Run it only after checking the account, role, and regions and obtaining cloud-read authorization. + ## Secret value workflow Terraform creates regional secret metadata and grants the EC2 role read access. It never writes credential values. @@ -73,7 +100,7 @@ After an authorized operator has created the infrastructure and verified the dec ```bash ts prebid server secrets set examplebidder \ - --deployment /secure/path/deployment.yaml \ + --deployment deployment.generated.yaml \ --region us-east-1 \ --file /secure/path/examplebidder.json \ --request-token 11111111-2222-4333-8444-555555555555 \ diff --git a/deploy/pbs-example/deployment.yaml b/deploy/pbs-example/deployment.example.yaml similarity index 92% rename from deploy/pbs-example/deployment.yaml rename to deploy/pbs-example/deployment.example.yaml index ee496a570..8df93a10e 100644 --- a/deploy/pbs-example/deployment.yaml +++ b/deploy/pbs-example/deployment.example.yaml @@ -8,7 +8,7 @@ aws: pbs: config: runtime/pbs.yaml image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 - bindings: runtime/secret-bindings.json + bindings: runtime/secret-bindings.example.json regions: us-east-1: instance_ids: diff --git a/deploy/pbs-example/locals.tf b/deploy/pbs-example/locals.tf index 2e19de5dc..db3d60304 100644 --- a/deploy/pbs-example/locals.tf +++ b/deploy/pbs-example/locals.tf @@ -5,4 +5,54 @@ locals { Project = "prebid-server-example" Owner = "example-team" }, var.tags) + + pbs_image = "prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7" + + deployment_descriptor = { + schema_version = 1 + environment = "example" + runtime = "ec2-compose" + aws = { + account_id = var.aws_account_id + profile = var.aws_profile + } + pbs = { + config = "runtime/pbs.yaml" + image = local.pbs_image + bindings = "runtime/secret-bindings.generated.json" + } + regions = { + us-east-1 = { + instance_ids = values(module.east.instance_ids) + overrides = "runtime/regions/east.yaml" + } + us-west-2 = { + instance_ids = values(module.west.instance_ids) + overrides = "runtime/regions/west.yaml" + } + } + } + + secret_bindings = { + examplebidder = { + verified_image = local.pbs_image + source = "https://example.com/pbs-adapter-reference" + secrets = { + us-east-1 = module.east.secret_arns["examplebidder"] + us-west-2 = module.west.secret_arns["examplebidder"] + } + keys = { + api_key = { + env = "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY" + pbs_path = ["adapters", "examplebidder", "api_key"] + required = true + } + optional_token = { + env = "PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN" + pbs_path = ["adapters", "examplebidder", "optional_token"] + required = false + } + } + } + } } diff --git a/deploy/pbs-example/main.tf b/deploy/pbs-example/main.tf index 930c4c605..35c901fab 100644 --- a/deploy/pbs-example/main.tf +++ b/deploy/pbs-example/main.tf @@ -10,10 +10,8 @@ module "east" { ami_id = var.east_ami_id certificate_arn = var.east_certificate_arn instance_type = var.instance_type - log_retention_days = var.log_retention_days name = "pbs-example-us-east-1" - region = "us-east-1" - secrets_kms_key_arn = var.secrets_kms_key_arn + secrets_kms_key_arn = var.east_secrets_kms_key_arn tags = local.common_tags trusted_server_cidr_blocks = var.trusted_server_cidr_blocks vpc_cidr = var.east_vpc_cidr @@ -31,10 +29,8 @@ module "west" { ami_id = var.west_ami_id certificate_arn = var.west_certificate_arn instance_type = var.instance_type - log_retention_days = var.log_retention_days name = "pbs-example-us-west-2" - region = "us-west-2" - secrets_kms_key_arn = var.secrets_kms_key_arn + secrets_kms_key_arn = var.west_secrets_kms_key_arn tags = local.common_tags trusted_server_cidr_blocks = var.trusted_server_cidr_blocks vpc_cidr = var.west_vpc_cidr diff --git a/deploy/pbs-example/modules/regional/monitoring.tf b/deploy/pbs-example/modules/regional/monitoring.tf index c170015a0..ddfc81566 100644 --- a/deploy/pbs-example/modules/regional/monitoring.tf +++ b/deploy/pbs-example/modules/regional/monitoring.tf @@ -1,12 +1,3 @@ -resource "aws_cloudwatch_log_group" "runtime" { - name = "/pbs-example/${var.region}" - retention_in_days = var.log_retention_days - - tags = merge(var.tags, { - Name = "${var.name}-runtime" - }) -} - resource "aws_cloudwatch_metric_alarm" "instance_cpu" { for_each = aws_instance.pbs diff --git a/deploy/pbs-example/modules/regional/outputs.tf b/deploy/pbs-example/modules/regional/outputs.tf index 1b3b47a4f..2d5268e28 100644 --- a/deploy/pbs-example/modules/regional/outputs.tf +++ b/deploy/pbs-example/modules/regional/outputs.tf @@ -13,6 +13,11 @@ output "instance_ids" { value = { for availability_zone, instance in aws_instance.pbs : availability_zone => instance.id } } +output "nat_eip_addresses" { + description = "Stable NAT gateway egress addresses keyed by Availability Zone for bidder allowlists." + value = { for availability_zone, eip in aws_eip.nat : availability_zone => eip.public_ip } +} + output "private_subnet_ids" { description = "Private subnet IDs keyed by Availability Zone." value = { for availability_zone, subnet in aws_subnet.private : availability_zone => subnet.id } diff --git a/deploy/pbs-example/modules/regional/secrets.tf b/deploy/pbs-example/modules/regional/secrets.tf index 7e8e89120..0296045ba 100644 --- a/deploy/pbs-example/modules/regional/secrets.tf +++ b/deploy/pbs-example/modules/regional/secrets.tf @@ -60,16 +60,6 @@ resource "aws_iam_role_policy" "runtime" { ] Resource = [for secret in aws_secretsmanager_secret.bidder : secret.arn] }, - { - Sid = "WriteRuntimeLogs" - Effect = "Allow" - Action = [ - "logs:CreateLogStream", - "logs:DescribeLogStreams", - "logs:PutLogEvents", - ] - Resource = "${aws_cloudwatch_log_group.runtime.arn}:*" - }, ], var.secrets_kms_key_arn == null ? [] : [{ Sid = "DecryptBidderSecrets" Effect = "Allow" diff --git a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl new file mode 100644 index 000000000..7b9455229 --- /dev/null +++ b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl @@ -0,0 +1,83 @@ +mock_provider "aws" { + mock_resource "aws_instance" { + defaults = { + id = "i-0123456789abcdef0" + } + } + + mock_resource "aws_eip" { + defaults = { + public_ip = "192.0.2.10" + } + } + + mock_resource "aws_secretsmanager_secret" { + override_during = plan + + defaults = { + arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-AbCdEf" + } + } +} + +variables { + alarm_actions = [] + ami_id = "ami-0123456789abcdef0" + availability_zones = ["us-east-1a", "us-east-1b"] + certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-1111-4111-8111-111111111111" + instance_type = "c7i.large" + name = "pbs-example-us-east-1" + tags = { ManagedBy = "Terraform" } + trusted_server_cidr_blocks = ["192.0.2.0/24", "198.51.100.0/24"] + vpc_cidr = "10.80.0.0/16" +} + +run "plans_private_hosts_and_scoped_ingress" { + command = plan + + assert { + condition = length(aws_instance.pbs) == 2 + error_message = "The regional module should plan one PBS host in each of two AZs." + } + + assert { + condition = alltrue([for instance in aws_instance.pbs : instance.associate_public_ip_address == false]) + error_message = "PBS hosts should not receive public IP addresses." + } + + assert { + condition = length(aws_security_group.alb.ingress) == length(var.trusted_server_cidr_blocks) + error_message = "The ALB should have one ingress rule for each approved caller CIDR." + } + + assert { + condition = alltrue([ + for rule in aws_security_group.alb.ingress : + length(rule.cidr_blocks) == 1 && contains(var.trusted_server_cidr_blocks, one(rule.cidr_blocks)) + ]) + error_message = "Every ALB ingress rule should use an approved caller CIDR." + } + + assert { + condition = length(aws_secretsmanager_secret.bidder) == 1 + error_message = "The example should create metadata for only the declared example bidder secret." + } + + assert { + condition = length(jsondecode(aws_iam_role_policy.runtime.policy).Statement[0].Resource) == length(aws_secretsmanager_secret.bidder) + error_message = "The runtime role should read only the declared bidder secrets." + } +} + +run "scopes_optional_kms_decryption" { + command = plan + + variables { + secrets_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/11111111-1111-4111-8111-111111111111" + } + + assert { + condition = jsondecode(aws_iam_role_policy.runtime.policy).Statement[1].Resource == var.secrets_kms_key_arn + error_message = "KMS decryption should be scoped to the selected regional key." + } +} diff --git a/deploy/pbs-example/modules/regional/variables.tf b/deploy/pbs-example/modules/regional/variables.tf index da5c33804..835ae19ca 100644 --- a/deploy/pbs-example/modules/regional/variables.tf +++ b/deploy/pbs-example/modules/regional/variables.tf @@ -28,11 +28,6 @@ variable "instance_type" { type = string } -variable "log_retention_days" { - description = "CloudWatch Logs retention period." - type = number -} - variable "name" { description = "Stable name prefix for regional resources." type = string @@ -49,11 +44,6 @@ variable "pbs_port" { } } -variable "region" { - description = "AWS region represented by this module instance." - type = string -} - variable "secrets_kms_key_arn" { description = "Optional customer-managed KMS key ARN for regional bidder secrets." type = string diff --git a/deploy/pbs-example/outputs.tf b/deploy/pbs-example/outputs.tf index efd88b222..117489ace 100644 --- a/deploy/pbs-example/outputs.tf +++ b/deploy/pbs-example/outputs.tf @@ -8,11 +8,26 @@ output "east_instance_ids" { value = module.east.instance_ids } +output "east_nat_eip_addresses" { + description = "Stable us-east-1 NAT egress addresses for bidder allowlists." + value = module.east.nat_eip_addresses +} + output "east_secret_arns" { description = "Bidder Secrets Manager ARNs in us-east-1. Values are never managed by Terraform." value = module.east.secret_arns } +output "deployment_descriptor_yaml" { + description = "Rendered nonsecret CLI descriptor. Write it to the ignored deployment.generated.yaml after apply." + value = yamlencode(local.deployment_descriptor) +} + +output "secret_bindings_json" { + description = "Rendered nonsecret binding metadata. Write it to the ignored runtime/secret-bindings.generated.json after apply." + value = jsonencode(local.secret_bindings) +} + output "west_alb_dns_name" { description = "DNS name of the us-west-2 regional ALB." value = module.west.alb_dns_name @@ -23,6 +38,11 @@ output "west_instance_ids" { value = module.west.instance_ids } +output "west_nat_eip_addresses" { + description = "Stable us-west-2 NAT egress addresses for bidder allowlists." + value = module.west.nat_eip_addresses +} + output "west_secret_arns" { description = "Bidder Secrets Manager ARNs in us-west-2. Values are never managed by Terraform." value = module.west.secret_arns diff --git a/deploy/pbs-example/runtime/README.md b/deploy/pbs-example/runtime/README.md index 88d9eebd7..5113030b0 100644 --- a/deploy/pbs-example/runtime/README.md +++ b/deploy/pbs-example/runtime/README.md @@ -1,7 +1,7 @@ # PBS runtime example -This directory owns nonsecret PBS configuration and the Compose shape. The image is pinned to Prebid Server Go v4.7.0 by digest. +This directory owns nonsecret PBS configuration and the Compose shape. The image is pinned to Prebid Server Go v4.7.0 by digest. Compose uses bounded local logs rather than implying that a CloudWatch log shipper is installed. The `/run/pbs/secrets/examplebidder.env` file is a runtime contract, not a checked-in credential file. A separately approved host-side loader must retrieve the selected regional Secrets Manager version, validate the complete JSON payload, render this restricted environment file atomically, and start or replace Compose. The current `ts prebid server` CLI does not implement that release or injection workflow. -`examples/pbs-secrets.env` contains dummy values for local Compose parsing only. +`secret-bindings.example.json` and `examples/pbs-secrets.env` contain fictional values for local checks only. An authorized Terraform apply can render the ignored `secret-bindings.generated.json` with actual regional secret ARNs, but never secret values. diff --git a/deploy/pbs-example/runtime/compose.yaml b/deploy/pbs-example/runtime/compose.yaml index 38a2167d8..4db566901 100644 --- a/deploy/pbs-example/runtime/compose.yaml +++ b/deploy/pbs-example/runtime/compose.yaml @@ -3,7 +3,7 @@ services: image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 restart: unless-stopped ports: - - "8000:8000" + - "${PBS_HOST_PORT:-8000}:8000" env_file: - ${PBS_SECRET_ENV_FILE:-/run/pbs/secrets/examplebidder.env} volumes: @@ -13,3 +13,8 @@ services: - no-new-privileges:true cap_drop: - ALL + logging: + driver: local + options: + max-file: "3" + max-size: 10m diff --git a/deploy/pbs-example/runtime/secret-bindings.json b/deploy/pbs-example/runtime/secret-bindings.example.json similarity index 100% rename from deploy/pbs-example/runtime/secret-bindings.json rename to deploy/pbs-example/runtime/secret-bindings.example.json diff --git a/deploy/pbs-example/scripts/smoke-runtime.sh b/deploy/pbs-example/scripts/smoke-runtime.sh new file mode 100755 index 000000000..a765773ca --- /dev/null +++ b/deploy/pbs-example/scripts/smoke-runtime.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -euo pipefail + +example_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +project_name="pbs-example-smoke" +host_port="${PBS_SMOKE_PORT:-18080}" +compose=( + docker compose + --project-name "$project_name" + --env-file "$example_dir/runtime/examples/compose.env" + -f "$example_dir/runtime/compose.yaml" +) + +cleanup() { + "${compose[@]}" down --volumes --remove-orphans >/dev/null 2>&1 || true +} +trap cleanup EXIT + +export PBS_HOST_PORT="$host_port" +"${compose[@]}" up --detach + +for _ in $(seq 1 30); do + if response="$(curl --fail --silent --show-error "http://127.0.0.1:${host_port}/status" 2>/dev/null)"; then + if [[ "$response" != "ok" ]]; then + printf 'Unexpected PBS status response: %s\n' "$response" >&2 + exit 1 + fi + + logs="$("${compose[@]}" logs --no-color)" + if grep -Fq "example-only-api-key" <<<"$logs" || grep -Fq "example-only-optional-token" <<<"$logs"; then + printf 'PBS startup logs exposed a dummy secret.\n' >&2 + exit 1 + fi + + printf 'PBS runtime and startup-log redaction smoke passed on port %s.\n' "$host_port" + exit 0 + fi + sleep 1 +done + +"${compose[@]}" logs --no-color >&2 +printf 'PBS did not become healthy within 30 seconds.\n' >&2 +exit 1 diff --git a/deploy/pbs-example/terraform.tfvars.example b/deploy/pbs-example/terraform.tfvars.example index 733d0347a..4c544dbbf 100644 --- a/deploy/pbs-example/terraform.tfvars.example +++ b/deploy/pbs-example/terraform.tfvars.example @@ -10,5 +10,10 @@ west_certificate_arn = "arn:aws:acm:us-west-2:123456789012:certificate/22222222- # Existing hosted-zone ID in the fictional example account. route53_zone_id = "Z00000000000000000000" +# Optional regional customer-managed keys. Keep null to use each region's +# Secrets Manager service key. +# east_secrets_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/11111111-1111-4111-8111-111111111111" +# west_secrets_kms_key_arn = "arn:aws:kms:us-west-2:123456789012:key/22222222-2222-4222-8222-222222222222" + # Replace with the actual egress ranges approved for Trusted Server. trusted_server_cidr_blocks = ["203.0.113.0/24"] diff --git a/deploy/pbs-example/tests/root_unit_test.tftest.hcl b/deploy/pbs-example/tests/root_unit_test.tftest.hcl new file mode 100644 index 000000000..618b03c19 --- /dev/null +++ b/deploy/pbs-example/tests/root_unit_test.tftest.hcl @@ -0,0 +1,147 @@ +mock_provider "aws" { + mock_resource "aws_instance" { + override_during = plan + + defaults = { + id = "i-0123456789abcdef0" + } + } + + mock_resource "aws_eip" { + defaults = { + public_ip = "192.0.2.10" + } + } + + mock_resource "aws_lb" { + defaults = { + dns_name = "east-alb.example.com" + zone_id = "Z00000000000000000001" + } + } + + mock_resource "aws_secretsmanager_secret" { + override_during = plan + + defaults = { + arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-AbCdEf" + } + } +} + +mock_provider "aws" { + alias = "west" + + mock_resource "aws_instance" { + override_during = plan + + defaults = { + id = "i-0fedcba9876543210" + } + } + + mock_resource "aws_eip" { + defaults = { + public_ip = "192.0.2.20" + } + } + + mock_resource "aws_lb" { + defaults = { + dns_name = "west-alb.example.com" + zone_id = "Z00000000000000000002" + } + } + + mock_resource "aws_secretsmanager_secret" { + override_during = plan + + defaults = { + arn = "arn:aws:secretsmanager:us-west-2:123456789012:secret:example-GhIjKl" + } + } +} + +variables { + aws_account_id = "123456789012" + aws_profile = "pbs-example" + east_ami_id = "ami-0123456789abcdef0" + east_certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-1111-4111-8111-111111111111" + route53_zone_id = "Z00000000000000000000" + west_ami_id = "ami-0fedcba9876543210" + west_certificate_arn = "arn:aws:acm:us-west-2:123456789012:certificate/22222222-2222-4222-8222-222222222222" +} + +run "plans_both_regional_provider_mappings" { + command = plan + + assert { + condition = length(module.east.instance_ids) == 2 + error_message = "The east provider mapping should plan two PBS instances." + } + + assert { + condition = length(module.west.instance_ids) == 2 + error_message = "The west provider mapping should plan two PBS instances." + } + + assert { + condition = length(local.deployment_descriptor.regions) == 2 + error_message = "The generated deployment descriptor should include both regions." + } + + assert { + condition = toset(keys(local.secret_bindings.examplebidder.secrets)) == toset(["us-east-1", "us-west-2"]) + error_message = "The generated binding should name an independent secret in each region." + } + + assert { + condition = yamldecode(output.deployment_descriptor_yaml).pbs.bindings == "runtime/secret-bindings.generated.json" + error_message = "The generated descriptor should reference the ignored generated binding file." + } + + assert { + condition = jsondecode(output.secret_bindings_json).examplebidder.secrets.us-east-1 != jsondecode(output.secret_bindings_json).examplebidder.secrets.us-west-2 + error_message = "The rendered binding should contain distinct regional secret ARNs." + } +} + +run "rejects_duplicate_east_availability_zones" { + command = plan + + variables { + east_availability_zones = ["us-east-1a", "us-east-1a"] + } + + expect_failures = [var.east_availability_zones] +} + +run "rejects_empty_caller_allowlist" { + command = plan + + variables { + trusted_server_cidr_blocks = [] + } + + expect_failures = [var.trusted_server_cidr_blocks] +} + +run "rejects_east_key_from_wrong_region" { + command = plan + + variables { + east_secrets_kms_key_arn = "arn:aws:kms:us-west-2:123456789012:key/11111111-1111-4111-8111-111111111111" + } + + expect_failures = [var.east_secrets_kms_key_arn] +} + +run "rejects_west_key_from_wrong_account" { + command = plan + + variables { + west_secrets_kms_key_arn = "arn:aws:kms:us-west-2:999999999999:key/22222222-2222-4222-8222-222222222222" + } + + expect_failures = [var.west_secrets_kms_key_arn] +} diff --git a/deploy/pbs-example/variables.tf b/deploy/pbs-example/variables.tf index fdace5889..2b3775cc6 100644 --- a/deploy/pbs-example/variables.tf +++ b/deploy/pbs-example/variables.tf @@ -39,6 +39,17 @@ variable "east_certificate_arn" { } } +variable "east_secrets_kms_key_arn" { + description = "Optional us-east-1 customer-managed KMS key ARN for bidder secrets. Null uses the regional Secrets Manager service key." + type = string + default = null + + validation { + condition = var.east_secrets_kms_key_arn == null || can(regex("^arn:aws[a-z-]*:kms:us-east-1:${var.aws_account_id}:key/.+$", var.east_secrets_kms_key_arn)) + error_message = "east_secrets_kms_key_arn must be a KMS key ARN in us-east-1 and the declared AWS account." + } +} + variable "east_vpc_cidr" { description = "CIDR block for the us-east-1 VPC." type = string @@ -62,17 +73,6 @@ variable "instance_type" { default = "c7i.large" } -variable "log_retention_days" { - description = "CloudWatch log retention for each regional runtime log group." - type = number - default = 14 - - validation { - condition = contains([1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653], var.log_retention_days) - error_message = "log_retention_days must be a supported CloudWatch Logs retention period." - } -} - variable "pbs_hostname" { description = "Shared HTTPS hostname returned by Route 53 latency records." type = string @@ -89,12 +89,6 @@ variable "route53_zone_id" { type = string } -variable "secrets_kms_key_arn" { - description = "Optional customer-managed KMS key ARN for bidder secrets. Null uses the Secrets Manager service key." - type = string - default = null -} - variable "tags" { description = "Additional tags applied to supported AWS resources." type = map(string) @@ -132,6 +126,17 @@ variable "west_certificate_arn" { } } +variable "west_secrets_kms_key_arn" { + description = "Optional us-west-2 customer-managed KMS key ARN for bidder secrets. Null uses the regional Secrets Manager service key." + type = string + default = null + + validation { + condition = var.west_secrets_kms_key_arn == null || can(regex("^arn:aws[a-z-]*:kms:us-west-2:${var.aws_account_id}:key/.+$", var.west_secrets_kms_key_arn)) + error_message = "west_secrets_kms_key_arn must be a KMS key ARN in us-west-2 and the declared AWS account." + } +} + variable "west_vpc_cidr" { description = "CIDR block for the us-west-2 VPC." type = string From ff30d9e797dc76c3598640ac8ff0f5cbafdc00de Mon Sep 17 00:00:00 2001 From: Christian Date: Wed, 16 Sep 2026 17:09:15 -0500 Subject: [PATCH 07/14] Address PBS CLI review feedback --- .../references/file-generation.md | 2 +- .tool-versions | 2 +- AGENTS.md | 1 + Cargo.toml | 2 + crates/trusted-server-cli/Cargo.toml | 4 +- crates/trusted-server-cli/README.md | 4 +- .../src/commands/pbs/aws.rs | 17 +++++-- .../src/commands/pbs/config.rs | 49 +++++++++++++++---- .../src/commands/pbs/inspect.rs | 36 ++++++++++++-- .../src/commands/pbs/secrets.rs | 2 +- .../src/commands/pbs/status.rs | 3 +- crates/trusted-server-cli/tests/pbs_cli.rs | 8 +++ docs/guide/cli.md | 19 +++++++ ...6-07-24-prebid-refresh-gam-path-opt-out.md | 5 +- .../2026-06-17-prebid-bundle-cli-design.md | 44 +++++++++-------- 15 files changed, 153 insertions(+), 45 deletions(-) diff --git a/.claude/skills/planning-prebid-aws/references/file-generation.md b/.claude/skills/planning-prebid-aws/references/file-generation.md index 3cda5cceb..72abe045d 100644 --- a/.claude/skills/planning-prebid-aws/references/file-generation.md +++ b/.claude/skills/planning-prebid-aws/references/file-generation.md @@ -53,7 +53,7 @@ Run applicable checks on the generated paths, recording exact commands and resul | Area | Check | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Terraform | Follow [Terraform local checks](terraform.md#tests-and-safe-local-checks), including inspected, explicitly selected mocked tests | -| Runtime structure | Parse YAML/JSON, verify manifest paths/checksums, and render deterministic overrides twice | +| Runtime structure | Parse YAML/JSON, verify manifest paths/checksums, and validate regional override structure | | Compose branch | `docker compose --env-file -f config --quiet` and dummy-value round-trip checks | | Operator tool | Language syntax/lint and [configuration/secrets contract tests](configuration-and-secrets.md#rotation-and-recovery), plus failed-release and interrupted/repeated invocation tests | | PBS behavior | Approved isolated container startup and smoke fixture against controlled bidder responses, if a suitable local runtime is available | diff --git a/.tool-versions b/.tool-versions index 8b805f271..34ab119d8 100644 --- a/.tool-versions +++ b/.tool-versions @@ -3,4 +3,4 @@ rust 1.95.0 nodejs 24.12.0 viceroy 0.17.0 wasmtime 44.0.1 -aws latest +aws 2.36.45 diff --git a/AGENTS.md b/AGENTS.md index 3b7189204..cb73c1dc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,6 +39,7 @@ Supporting files: `edgezero.toml`, `fastly.toml`, | Fastly CLI | 15.1.0 (from `.tool-versions`) | | Viceroy | 0.17.0 (from `.tool-versions`) | | Wasmtime | 44.0.1 (from `.tool-versions`) | +| AWS CLI | 2.36.45 (from `.tool-versions`) | --- diff --git a/Cargo.toml b/Cargo.toml index ac0cac621..51e3038ea 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -91,11 +91,13 @@ rand = "0.8" rcgen = { version = "0.13", features = ["x509-parser"] } regex = "1.12.3" reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } +rpassword = "7.5" rustls = "0.23" rustls-pemfile = "2" scraper = "0.24.0" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0.149" +serde_yaml_ng = "0.10" sha2 = "0.10.9" similar = "2.7" simple_logger = "5" diff --git a/crates/trusted-server-cli/Cargo.toml b/crates/trusted-server-cli/Cargo.toml index 10f2ebd00..522bf2304 100644 --- a/crates/trusted-server-cli/Cargo.toml +++ b/crates/trusted-server-cli/Cargo.toml @@ -27,11 +27,11 @@ http = { workspace = true } log = { workspace = true } rand = { workspace = true } regex = { workspace = true } -rpassword = "7.5" +rpassword = { workspace = true } scraper = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } -serde_yaml_ng = "0.10" +serde_yaml_ng = { workspace = true } similar = { workspace = true } tempfile = { workspace = true } tokio = { workspace = true } diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index f0cc348db..4cdabecc0 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -22,7 +22,7 @@ On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fiction | Command | What it does | Access | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------- | | `ts prebid server inspect --config ` | Reports selected local Prebid fields, classifies bidder lists, and marks host-secret requirements unresolved | Local read-only | -| `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and deterministic regional YAML merging | Local read-only | +| `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and regional YAML merging | Local read-only | | `ts prebid server secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | | `ts prebid server status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | @@ -45,7 +45,7 @@ See [deployment.yaml](examples/pbs/deployment.yaml), [PBS YAML](examples/pbs/pbs Version 1 requires: - `schema_version: 1`, an explicit environment, and `runtime: ec2-compose`. -- An explicit 12-digit AWS account ID and AWS CLI profile. Environment/profile/bidder identifiers use letters, digits, underscores, and hyphens. +- An explicit 12-digit AWS account ID and AWS CLI profile. Environment and bidder identifiers use letters, digits, underscores, and hyphens. Profile names may also use periods. - A digest-pinned PBS image and a baseline YAML path. - A nonempty region map with optional override paths and explicit EC2 instance IDs for `status`. - An optional binding-file path. Omit it when no host secrets are needed. diff --git a/crates/trusted-server-cli/src/commands/pbs/aws.rs b/crates/trusted-server-cli/src/commands/pbs/aws.rs index e4d3047ec..d49bae99e 100644 --- a/crates/trusted-server-cli/src/commands/pbs/aws.rs +++ b/crates/trusted-server-cli/src/commands/pbs/aws.rs @@ -75,10 +75,21 @@ impl Aws for AwsCli { // Closing removes the temporary payload, including on every earlier error path via Drop. drop(payload); if !output.status.success() { - return Err(Report::new(PbsError::Aws(operation))); + let message = if operation == "put-secret-value" { + "write outcome uncertain; retain the request token and retry identical input" + } else { + operation + }; + return Err(Report::new(PbsError::Aws(message))); } - serde_json::from_slice(&output.stdout) - .map_err(|_| Report::new(PbsError::Aws("invalid JSON response"))) + serde_json::from_slice(&output.stdout).map_err(|_| { + let message = if operation == "put-secret-value" { + "write response invalid; outcome uncertain, retain the request token" + } else { + "invalid JSON response" + }; + Report::new(PbsError::Aws(message)) + }) } } diff --git a/crates/trusted-server-cli/src/commands/pbs/config.rs b/crates/trusted-server-cli/src/commands/pbs/config.rs index f899e3eb4..1e00c09ff 100644 --- a/crates/trusted-server-cli/src/commands/pbs/config.rs +++ b/crates/trusted-server-cli/src/commands/pbs/config.rs @@ -98,10 +98,14 @@ impl Deployment { pbs, regions, } = descriptor; + if !valid_profile(&aws.profile) { + return Err(invalid( + "AWS profile must use 1-128 letters, digits, periods, underscores, or hyphens", + )); + } if schema_version != 1 || !identifier(&environment) || !valid_account(&aws.account_id) - || !identifier(&aws.profile) || regions.is_empty() || !pinned_image(&pbs.image) { @@ -234,6 +238,14 @@ pub(super) fn valid_account(value: &str) -> bool { value.len() == 12 && value.bytes().all(|byte| byte.is_ascii_digit()) } +fn valid_profile(value: &str) -> bool { + !value.is_empty() + && value.len() <= 128 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-')) +} + fn valid_region(value: &str) -> bool { let parts: Vec<_> = value.split('-').collect(); parts.len() >= 3 @@ -373,16 +385,10 @@ pub(super) fn invalid(message: &'static str) -> Report { /// Report structural validation only, withholding resolved YAML values. /// /// # Errors -/// Returns an error if deterministic rendering fails. +/// Returns an error if resolved YAML cannot be serialized. pub(super) fn check(deployment: &Deployment) -> Result { for value in deployment.rendered.values() { - let rendered = serde_yaml_ng::to_string(value) - .map_err(|_| invalid("cannot serialize resolved YAML"))?; - let second = serde_yaml_ng::to_string(value) - .map_err(|_| invalid("cannot serialize resolved YAML"))?; - if rendered != second { - return Err(invalid("regional rendering was not deterministic")); - } + serde_yaml_ng::to_string(value).map_err(|_| invalid("cannot serialize resolved YAML"))?; } Ok(Output { failure: None, @@ -449,6 +455,31 @@ pub(super) mod tests { ); } + #[test] + fn accepts_dotted_aws_profiles_and_reports_invalid_profiles() { + let (_dir, path) = fixture(); + let descriptor = fs::read_to_string(&path).expect("should read descriptor"); + fs::write( + &path, + descriptor.replace("profile: pbs-sandbox", "profile: pbs.sandbox"), + ) + .expect("should write dotted profile"); + Deployment::load(&path).expect("should accept dotted AWS profile"); + + fs::write( + &path, + descriptor.replace("profile: pbs-sandbox", "profile: pbs sandbox"), + ) + .expect("should write invalid profile"); + let error = Deployment::load(&path) + .err() + .expect("should reject invalid AWS profile"); + assert!( + error.to_string().contains("AWS profile"), + "should identify the invalid field: {error}" + ); + } + #[test] fn yaml_and_secret_destinations_cannot_conflict() { let (dir, path) = fixture(); diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs index 9114a3d86..c0337c884 100644 --- a/crates/trusted-server-cli/src/commands/pbs/inspect.rs +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -88,13 +88,16 @@ fn bidder_list<'de, D: Deserializer<'de>>( } else { vec![text] }; - parts + Ok(parts .into_iter() .map(|part| { - serde_json::from_str(&format!("\"{}\"", part.replace('"', "\\\""))) - .map_err(serde::de::Error::custom) + let json = format!("\"{}\"", part.replace('"', "\\\"")); + match serde_json::from_str(&json) { + Ok(value) => value, + Err(_) => part.to_owned(), + } }) - .collect() + .collect()) } _ => Err(serde::de::Error::custom( "expected bidder list, indexed map, or list string", @@ -248,6 +251,30 @@ user_id_modules = ["sharedIdSystem"] assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); } + #[test] + fn invalid_list_identifier_reports_identifier_error() { + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write( + file.path(), + r#" +[integrations.prebid] +bidders = 'examplebidder\' +"#, + ) + .expect("should write config"); + + let error = inspect(file.path()) + .err() + .expect("should reject invalid identifier"); + + assert!( + error + .to_string() + .contains("invalid bidder or identity-module identifier"), + "should report identifier validation: {error}" + ); + } + #[test] fn accepts_the_runtime_bidder_list_encodings() { for input in [ @@ -255,6 +282,7 @@ user_id_modules = ["sharedIdSystem"] "'examplebidder,otherbidder'", "'[examplebidder, otherbidder]'", "'[\"examplebidder\", \"otherbidder\"]'", + "'example\\u0062idder,otherbidder'", "{ '10' = 'otherbidder', '2' = 'examplebidder' }", ] { let text = format!( diff --git a/crates/trusted-server-cli/src/commands/pbs/secrets.rs b/crates/trusted-server-cli/src/commands/pbs/secrets.rs index d7c473a7b..4c6177cb1 100644 --- a/crates/trusted-server-cli/src/commands/pbs/secrets.rs +++ b/crates/trusted-server-cli/src/commands/pbs/secrets.rs @@ -109,7 +109,7 @@ pub(super) fn set( aws: &dyn Aws, interaction: &mut dyn Interaction, ) -> Result { - if args.yes && args.request_token.is_none() || args.stdin && !args.yes { + if (args.yes && args.request_token.is_none()) || (args.stdin && !args.yes) { return Err(invalid( "noninteractive writes require --yes and --request-token", )); diff --git a/crates/trusted-server-cli/src/commands/pbs/status.rs b/crates/trusted-server-cli/src/commands/pbs/status.rs index 9812a8152..8656f2fb9 100644 --- a/crates/trusted-server-cli/src/commands/pbs/status.rs +++ b/crates/trusted-server-cli/src/commands/pbs/status.rs @@ -8,7 +8,8 @@ use super::{Output, PbsError, Result}; /// /// # Errors /// Rejects missing instance targets, wrong accounts, and failed identity verification. -/// Resource-query failures are included as unknown in a partial report with a failing exit status. +/// Identity failures abort the entire report. Resource-query failures are included as unknown in a +/// partial report with a failing exit status. pub(super) fn status(deployment: &Deployment, aws: &dyn Aws) -> Result { if deployment .regions diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index c88bf0dba..d2eb15e44 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -12,6 +12,10 @@ use tempfile::TempDir; const TOKEN: &str = "11111111-2222-4333-8444-555555555555"; fn fixture() -> TempDir { + assert!( + which::which("python3").is_ok(), + "pbs_cli tests need python3 on PATH for the fake AWS executable" + ); let dir = tempfile::tempdir().expect("should create fixture directory"); let source = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/pbs"); for name in ["deployment.yaml", "pbs.yaml", "east.yaml", "bindings.json"] { @@ -217,6 +221,10 @@ fn aws_errors_never_forward_provider_stderr_and_cleanup_payloads() { .expect("should run CLI"); assert_eq!(output.status.code(), Some(2)); assert_no_secret(&output); + assert!( + String::from_utf8_lossy(&output.stderr).contains("outcome uncertain"), + "should warn that the write may have succeeded" + ); assert_payload_cleanup(dir.path()); } diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 4c9d343eb..c446fd766 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -649,3 +649,22 @@ ts prebid client --config publisher-a.toml --out build/prebid `ts prebid client` is local-only. It has no `--adapter` option and does not upload, provision, deploy, or push config. + +## Operate a self-hosted Prebid Server + +The experimental `ts prebid server` namespace supports four bounded operations: + +- `inspect` reads selected local Trusted Server configuration. +- `check` validates a deployment descriptor and regional configuration locally. +- `secrets set` writes one approved value to an existing AWS Secrets Manager secret. +- `status` reads declared EC2 infrastructure state, not PBS health or readiness. + +AWS operations require AWS CLI v2 on `PATH`; `inspect` and `check` do not contact +AWS. Add `--json` anywhere under `ts prebid server` for machine-readable output. +Failures use exit code 2, including incomplete `status` reports that still write +partial JSON to stdout. + +See the +[experimental PBS command reference](https://github.com/IABTechLab/trusted-server/blob/main/crates/trusted-server-cli/README.md) +for the deployment descriptor schema, secret-write safeguards, and current +limitations. diff --git a/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md b/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md index a111b4228..46f12cf79 100644 --- a/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md +++ b/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md @@ -2,6 +2,9 @@ **Design:** `docs/superpowers/specs/2026-07-24-prebid-refresh-gam-path-opt-out-design.md` +> **Update (2026-09-16):** The `ts prebid bundle` command referenced in this +> plan was later renamed to `ts prebid client`. + **Goal:** Let operators exclude selected GAM ad-unit-path suffixes from Trusted Server's Prebid refresh auctions without suppressing the corresponding GAM refresh. @@ -9,7 +12,7 @@ Server's Prebid refresh auctions without suppressing the corresponding GAM refre - Do not change publisher source, slot div IDs, or GAM configuration. - Do not edit `dist` output, minified assets, or an externally hosted Prebid bundle - by hand. `build-prebid-external.mjs`/`ts prebid client` are the supported build + by hand. `build-prebid-external.mjs`/`ts prebid bundle` are the supported build path. - The mechanism is literal, case-sensitive GAM-path suffix matching; it is not a size-based rule and does not add a div-ID fallback. diff --git a/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md b/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md index 3d60679b5..8e8701579 100644 --- a/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md +++ b/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md @@ -2,7 +2,11 @@ **Date:** 2026-06-17 **Status:** Implemented -**Scope:** `ts prebid client` local external Prebid bundle generation +**Scope:** `ts prebid bundle` local external Prebid bundle generation + +> **Update (2026-09-16):** The command described here was later renamed to +> `ts prebid client` when the `ts prebid server` namespace was introduced. + **Related context:** - `docs/superpowers/specs/2026-05-28-external-prebid-first-party-proxy-design.md` @@ -18,7 +22,7 @@ Add a Trusted Server-specific CLI command for generating the external Prebid browser bundle used by the first-party Prebid proxy flow: ```bash -ts prebid client +ts prebid bundle ``` The command should make the existing external bundle generation path ergonomic for @@ -43,7 +47,7 @@ proxy spec: ## 2. Non-goals -The initial `ts prebid client` command does **not** do any of the following: +The initial `ts prebid bundle` command does **not** do any of the following: - upload generated bundles to an asset host or CDN; - infer or construct the public `external_bundle_url`; @@ -63,7 +67,7 @@ The initial `ts prebid client` command does **not** do any of the following: ## 3. Command surface ```bash -ts prebid client [--config ] [--out ] +ts prebid bundle [--config ] [--out ] ``` Defaults: @@ -77,13 +81,13 @@ Examples: ```bash # Generate from trusted-server.toml into dist/prebid -ts prebid client +ts prebid bundle # Generate from a draft config -ts prebid client --config ./publisher-a.trusted-server.toml +ts prebid bundle --config ./publisher-a.trusted-server.toml # Generate into a custom local directory -ts prebid client --out ./build/prebid +ts prebid bundle --out ./build/prebid ``` Successful output should be concise and actionable, for example: @@ -159,7 +163,7 @@ bundle URL's host and any HTTPS redirect targets. ## 5. Config update behavior -After a successful local bundle build, `ts prebid client` must read the generated +After a successful local bundle build, `ts prebid bundle` must read the generated `manifest.json` and update the same `trusted-server.toml` file with: ```toml @@ -235,7 +239,7 @@ The manifest schema remains unchanged: ## 7. Dependency and environment handling -`ts prebid client` should fail fast with actionable diagnostics when local JS +`ts prebid bundle` should fail fast with actionable diagnostics when local JS build prerequisites are missing. Minimum checks before shelling out: @@ -249,7 +253,7 @@ If `node_modules` is missing, the command must not run dependency installation. It should fail with an instruction like: ```text -Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid client`. +Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid bundle`. ``` Errors from the JS generator, including unknown adapter names or unknown User ID @@ -261,7 +265,7 @@ and stderr enough for debugging. ## 8. Config loading and validation -`ts prebid client` should not require full production config validity. It is a +`ts prebid bundle` should not require full production config validity. It is a local artifact-generation command, and operators may run it before the config is ready for `ts config validate` or `ts config push`. @@ -289,7 +293,7 @@ This spec extends the `ts` product CLI command surface with a new Trusted Server-specific command group: ```text -ts prebid client +ts prebid bundle ``` The resulting CLI command enum should conceptually become: @@ -297,7 +301,7 @@ The resulting CLI command enum should conceptually become: ```text ts audit ... ts config ... -ts prebid client ... +ts prebid bundle ... ts auth ... ts provision ... ts serve ... @@ -305,7 +309,7 @@ ts build ... ts deploy ... ``` -`ts prebid client` is similar to `ts audit` and `ts config` in that it owns +`ts prebid bundle` is similar to `ts audit` and `ts config` in that it owns Trusted Server behavior directly. It is unlike `ts build` / `ts deploy`, which are EdgeZero lifecycle delegates. @@ -316,12 +320,12 @@ are EdgeZero lifecycle delegates. ### CLI argument parsing - Add `Command::Prebid(PrebidArgs)`. -- Add `PrebidCommand::Client(PrebidBundleArgs)`. +- Add `PrebidCommand::Bundle(PrebidBundleArgs)`. - Add options: - `--config ` defaulting to `trusted-server.toml`; - `--out ` defaulting to `dist/prebid`. - Add parser tests for defaults and custom paths. -- Reject `--adapter` for `ts prebid client`. +- Reject `--adapter` for `ts prebid bundle`. ### CLI implementation @@ -353,11 +357,11 @@ User ID module names, hashing bundle bytes, and writing `manifest.json`. ### CLI parser tests -- `ts prebid client` parses with defaults: +- `ts prebid bundle` parses with defaults: - config: `trusted-server.toml` - out: `dist/prebid` -- `ts prebid client --config publisher.toml --out build/prebid` parses custom paths. -- `ts prebid client --adapter fastly` is rejected. +- `ts prebid bundle --config publisher.toml --out build/prebid` parses custom paths. +- `ts prebid bundle --adapter fastly` is rejected. ### Unit tests @@ -389,7 +393,7 @@ cd crates/trusted-server-js/lib npm ci cd ../../.. -ts prebid client +ts prebid bundle ls dist/prebid rg 'external_bundle_sha256|external_bundle_sri' trusted-server.toml ``` From 0e5000afa3d92876e5a58b60d350544274845daa Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 17 Sep 2026 08:33:38 -0500 Subject: [PATCH 08/14] Expand PBS AWS planning interview for RTB Fabric --- .claude/skills/planning-prebid-aws/SKILL.md | 14 +++- .../references/architecture.md | 1 + .../references/rtb-fabric.md | 69 +++++++++++++++++++ 3 files changed, 81 insertions(+), 3 deletions(-) create mode 100644 .claude/skills/planning-prebid-aws/references/rtb-fabric.md diff --git a/.claude/skills/planning-prebid-aws/SKILL.md b/.claude/skills/planning-prebid-aws/SKILL.md index bcd021846..27134562a 100644 --- a/.claude/skills/planning-prebid-aws/SKILL.md +++ b/.claude/skills/planning-prebid-aws/SKILL.md @@ -40,14 +40,22 @@ Cover these topics, skipping already confirmed answers: - Operations: fixed headroom versus automatic scaling, budget ceiling, owner and backup, existing CI and AWS platform, regions, DNS, network restrictions, and bidder IP allowlists. - Security: permitted callers/publishers, public or private access, data residency, retention, and the privacy policy owner. - Auction behavior: caller, bidder set, formats, consent and identity, account settings, stored requests, and cache dependencies. Read [Prebid Go requirements](references/prebid-go.md) before resolving these inputs. +- Outbound bidder connectivity: for each bidder and region, choose public/NAT egress, an internal RTB Fabric link, or an RTB Fabric outbound external link. If RTB Fabric is a candidate, read [RTB Fabric connectivity](references/rtb-fabric.md) before asking the conditional questions. Record partner participation, gateway and link ownership, PBS endpoint mapping, regional support, quotas, timeout, cost, fallback, and monitoring. + +When RTB Fabric is a candidate, ask these questions in related groups: + +1. Which bidders participate in RTB Fabric, in which regions, and which partner provides each responder gateway ID? Who accepts and owns each link? +2. For each participating bidder, should PBS use an internal Fabric link or an outbound external link? What remains on NAT egress, and what is the explicit behavior when a Fabric link is pending, unavailable, or over quota? +3. Which pinned PBS Go release and adapter configuration own the endpoint mapping? Who approves link creation, partner acceptance, configuration rollout, and endpoint changes? +4. What peak transactions per second, payload sizes, bidder deadlines, regional failover load, and monthly volume should size each link and compare Fabric cost with NAT? Translate "production ready" into measurable availability, security, capacity, and recovery requirements. Scaling and availability are separate decisions. Offer a measurement plan for unknown traffic rather than inventing capacity. -Done when every topic is confirmed, explicitly inapplicable, or recorded as an unresolved blocker. Continue a provisional design around unknowns, but pause affected file generation until architecture-changing decisions are approved. +Done when every topic is confirmed, explicitly inapplicable, or recorded as an unresolved blocker. For an RTB Fabric branch, every participating bidder and region has a selected path, partner owner, endpoint mapping, quota and timeout check, cost assumption, fallback, and monitoring owner. Continue a provisional design around unknowns, but pause affected file generation until architecture-changing decisions are approved. ## 3. Recommend and obtain approval -Read [architecture decisions](references/architecture.md). For Terraform state/authentication decisions and HCL generation, read [Terraform guidance](references/terraform.md). For a two-region standalone-host pilot, consult [the worked example](examples/two-region-pilot.md); its values remain conditional. +Read [architecture decisions](references/architecture.md). If the interview selected RTB Fabric, also read [RTB Fabric connectivity](references/rtb-fabric.md). For Terraform state/authentication decisions and HCL generation, read [Terraform guidance](references/terraform.md). For a two-region standalone-host pilot, consult [the worked example](examples/two-region-pilot.md); its values remain conditional. Present one recommended design and only alternatives that resolve a real tradeoff. Include: @@ -68,7 +76,7 @@ Read [file generation and validation](references/file-generation.md). Follow exi Read the [PBS CLI usage and descriptor schema](../../../crates/trusted-server-cli/README.md) before generating inputs consumed by `ts prebid server`. Its current descriptor supports EC2/Compose only. Keep other architecture choices available, but mark their CLI integration deferred rather than generating unsupported fields. -Verify version-specific PBS fields and adapter bindings against the selected release. Verify AWS/Terraform behavior and pricing against current primary documentation. Record source links, versions, and verification dates in the deployment plan. Unavailable evidence remains a named blocker; do not invent image digests, configuration keys, prices, or benchmark results. +Verify version-specific PBS fields and adapter bindings against the selected release. For an approved RTB Fabric branch, reread [RTB Fabric connectivity](references/rtb-fabric.md) and keep partner acceptance, link activation, and any unsupported CLI integration visible as separate work. Verify AWS/Terraform behavior and pricing against current primary documentation. Record source links, versions, and verification dates in the deployment plan. Unavailable evidence remains a named blocker; do not invent image digests, configuration keys, prices, or benchmark results. Done when every approved artifact exists, has a named owner and check, and every unresolved input is visible and prevents unsafe use where applicable. Every generated operator command must have documented inputs, access requirements, output, failure behavior, and a recovery action; proposing command names alone is not implementation. diff --git a/.claude/skills/planning-prebid-aws/references/architecture.md b/.claude/skills/planning-prebid-aws/references/architecture.md index d16aa9878..a00909f5c 100644 --- a/.claude/skills/planning-prebid-aws/references/architecture.md +++ b/.claude/skills/planning-prebid-aws/references/architecture.md @@ -22,6 +22,7 @@ Use EKS only when the user's existing platform or an explicit requirement justif | Public HTTPS on standalone hosts | Public subnet, internet gateway, Elastic IP, Caddy | Accepted exposure, durable certificate storage, renewal and replacement path | | Managed HTTPS with multiple backends | ALB and ACM | Certificate ownership/validation, target health, draining, timeout budget, private management endpoints | | Private compute reaching public bidders | NAT-based egress or an approved existing egress service | AZ failure behavior, routing, hourly/data charges, and outbound address stability | +| Private PBS-to-bidder connectivity | AWS RTB Fabric requester gateway and standard or outbound external links | Partner participation, gateway and link ownership, regional support, quotas, bidder endpoint mapping, timeout, fallback, and cost | | Bidder source-IP allowlists | Stable egress IPs | Every normal, scaling, and failover path uses partner-approved addresses | | Existing caller selects regions | Regional hostnames | Caller routing and failure policy, TLS, identity behavior | | DNS-based regional selection | Route 53 latency records with per-region health checks | Independent health targets, cached-answer behavior, all-unhealthy behavior, and failover capacity | diff --git a/.claude/skills/planning-prebid-aws/references/rtb-fabric.md b/.claude/skills/planning-prebid-aws/references/rtb-fabric.md new file mode 100644 index 000000000..98f49714a --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/rtb-fabric.md @@ -0,0 +1,69 @@ +# AWS RTB Fabric connectivity + +Load this reference when a bidder partner offers AWS RTB Fabric connectivity or the deployment is considering it as an alternative to public/NAT egress. + +## Decision boundary + +For this PBS workflow, RTB Fabric is an optional outbound path from PBS to selected bidders. PBS is the requester. A partner's bidder is the responder. + +- A requester gateway is colocated with the PBS VPC. +- A standard Fabric link connects that gateway to a partner responder gateway. The partner must provide its gateway ID and accept the request before traffic can use the link. +- An outbound external link targets a partner's public HTTP or HTTPS endpoint. AWS may use its Global Network or public Internet routing, so treat it as a different security, latency, and cost profile from a standard link. +- PBS still fans out auctions to its configured bidders. Fabric does not multiplex traffic to multiple partners. Plan one endpoint path per participating bidder. +- Keep the regional ALB or other approved PBS ingress path separate. Replacing it with Fabric requires a different responder or inbound design and separate approval. + +Route only partner-approved bidder endpoints through Fabric. Keep the approved NAT or external path for bidders without Fabric participation until a reviewed migration removes that dependency. + +## Interview record + +For each bidder and PBS region, record: + +| Decision | Required evidence | +| --- | --- | +| Participation | Partner name, responder gateway ID or public endpoint, AWS account owner, and acceptance owner | +| Link type | Standard internal link or outbound external link, with the selected region | +| PBS binding | Pinned PBS Go release, adapter endpoint field, config source, rollout owner, and link URL update procedure | +| Capacity | Peak transactions per second, bidder timeout, payload sizes, burst duration, and regional failover load | +| Fallback | Behavior while a link is requested, inactive, over quota, timing out, or unavailable | +| Operations | Link creation, partner acceptance, status checks, endpoint changes, deletion order, alerts, and recovery owner | +| Cost | Region, monthly sent transactions, payload distribution, no-bid volume, link count, external traffic, and estimate date | +| Privacy and security | OpenRTB fields, TLS mode, source-IP requirements, data residency, logging, and retention | + +Gateway IDs, link state, quota increases, and partner acceptance are external facts. Ask before authenticated AWS inspection, naming the account, role, regions, and read scope. Do not treat a partner's willingness to participate as proof that its gateway, endpoint, adapter, or regional capacity is ready. + +## Constraints to verify + +AWS currently lists RTB Fabric in US East (N. Virginia), US West (Oregon), Singapore, Tokyo, Frankfurt, and Ireland. Verify regional availability again during design. Gateways are regional, so a multi-region PBS deployment needs a regional connectivity decision for every participating bidder. + +The current default quotas include two gateways per account and region, two standard links per gateway, two outbound external links per gateway, one Availability Zone per gateway, 1,000 transactions per second per link, and a 1.5-second HTTP request timeout. Gateway and throughput quotas may be adjustable. The control-plane quota is 10 API requests per second per account and region and is not adjustable. Use the selected bidder deadline and measured traffic, not the defaults, as the capacity requirement. + +RTB Fabric supports OpenRTB versions including 2.6 and does not support OpenRTB 3.0. Verify the selected PBS Go release, bidder adapter, headers, custom extensions, and TLS behavior against the partner contract. + +AWS charges for sent RTB requests and responses according to region, traffic type, volume, and payload. No-bid traffic has its own pricing dimension, received traffic is not charged, and there is no Free Tier. Use the current pricing page or calculator. Do not copy rates from an example or assume that Fabric eliminates all NAT cost. + +CloudWatch exposes RTB Fabric request count, success and failure count, HTTP status, forwarding and total latency, target counts, filtered transactions, and internal or external no-bid metrics. Use Sum for volume metrics and P90, P95, or P99 for latency metrics where the metric supports those statistics. Define alarms and retention with the rest of the PBS observability plan. + +## Terraform and lifecycle + +The AWS Labs Terraform module creates requester and responder gateways, links, and inbound external links through the AWS Cloud Control provider. Pin a reviewed module release. Its documentation currently shows `v0.3.0` as the pinned example. + +Treat a standard partner link as a two-party lifecycle, not a normal single-account Terraform resource. The requester can create the request, but the partner must accept it. Keep acceptance, activation, and endpoint publication as separate operator-owned steps unless a separately approved automation owns both accounts. + +The module documents these limitations: + +- `http_responder_allowed` is immutable and may be absent from Cloud Control responses, which can cause Terraform state drift. The documented recovery is a one-time state cleanup and re-import. Changing the value requires link replacement. +- Link module configuration is currently supported from the requester side, not the responder side. +- Managed responder endpoints use EKS or EC2 Auto Scaling groups. The current Trusted Server descriptor supports `ec2-compose` with explicit instance IDs, so it does not establish RTB Fabric responder support. + +Keep secret values out of link configuration, Terraform variables, state, and reports. Link IDs, gateway IDs, and endpoint URLs are deployment metadata, but still require the selected access controls and change owner. + +## Sources + +- [AWS RTB Fabric FAQ](https://aws.amazon.com/rtb-fabric/faqs/) +- [RTB Fabric concepts](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/what-is-rtb-fabric.html) +- [Create links](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/creating-rtb-links.html) +- [Create outbound external links](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/creating-outbound-external-links.html) +- [RTB Fabric quotas](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/rtb-fabric-quotas.html) +- [RTB Fabric metrics](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/monitoring-cloudwatch-metrics.html) +- [AWS Prebid Server partner onboarding](https://docs.aws.amazon.com/solutions/latest/prebid-server-deployment-on-aws/use-the-solution.html) +- [AWS Labs Terraform module](https://github.com/awslabs/rtb-fabric-terraform-module) From 2aa065de5bab482ff20a63d1a15d4c5000227425 Mon Sep 17 00:00:00 2001 From: Christian Date: Sun, 20 Sep 2026 13:27:05 -0500 Subject: [PATCH 09/14] Address current PBS deployment review feedback --- .../references/configuration-and-secrets.md | 22 +- .tool-versions | 2 +- crates/trusted-server-cli/README.md | 6 +- .../src/commands/pbs/inspect.rs | 267 ++++++++++++++---- .../src/commands/pbs/mod.rs | 36 ++- crates/trusted-server-cli/tests/pbs_cli.rs | 7 +- .../modules/regional/monitoring.tf | 2 +- .../pbs-example/modules/regional/terraform.tf | 3 +- .../tests/security_unit_test.tftest.hcl | 8 + 9 files changed, 261 insertions(+), 92 deletions(-) diff --git a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md index 34acd0f1d..3b49f66a4 100644 --- a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md +++ b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md @@ -12,17 +12,17 @@ Before asking questions the repository can answer: 4. Add observations to the existing deployment decision record with source path/key, candidate requirement, confidence/status, and the remaining question. Report credential identifiers or presence only; keep raw TOML, credential values, and commercially sensitive publisher values out of reports and generated examples. 5. Verify candidate adapters against the selected PBS Go release and confirm partner authorization. Record each host-secret requirement as required, not needed, or unresolved. A bidder name alone proves neither a secret requirement nor permission to use the bidder. -| Trusted Server input | Discovery use | -| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| `integrations.prebid.enabled` | Determine active versus disabled intent using the schema's defaults | -| `server_url`, `account_id` | Identify existing provider/account compatibility questions, not automatically portable settings | -| `bidders` | Candidate server-side adapter set, subject to effective runtime configuration | -| `client_side_bidders` | Browser-side participation; do not automatically enable these adapters in PBS | -| `timeout_ms` | Caller budget; leave room for network/proxy work when proposing PBS auction timeouts | -| `test_mode`, `debug` | Testing/diagnostic intent, not authorization to contact bidders or proof that no real auction occurs | -| `bid_param_override_rules` | Conditional publisher/placement inputs and inventory behavior, not generic host credentials | -| `bundle.adapters`, identity modules | Browser bundle capabilities and identity questions, not proof of server-side use | -| Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller/request path | +| Trusted Server input | Discovery use | +| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| `auction.enabled` | Determine active versus disabled server-auction intent without expanding defaults | +| `auction.providers.` with `profile = "prebid-server"` | Identify each configured PBS provider; withhold endpoint values | +| Provider `timeout_ms` and `profile_config.test_mode` or `debug` | Record explicit timeout and diagnostic intent, not permission to contact bidders | +| Provider `profile_config.bid_param_override_rules` | Count conditional publisher or placement inputs without reporting their values | +| `auction.bidders..provider` | Group candidate server-side adapters under the referenced PBS provider | +| `integrations.prebid.enabled`, `account_id`, `timeout_ms`, `debug` | Record browser integration intent and compatibility questions separately from server demand | +| `integrations.prebid.client_side_bidders` | Browser-side participation; do not automatically enable these adapters in PBS | +| `integrations.prebid.bundle` | Browser bundle and identity capabilities, not proof of server-side use | +| Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller or request path | Use `ts prebid server inspect --config ` for redacted local discovery. It reports explicit values without expanding defaults or proving host-secret requirements. Resolve schema defaults and adapter metadata separately, recording unsupported details as unresolved rather than fetching credentials or inventing mappings. Read config without invoking commands that publish or rewrite it. Leave `trusted-server.toml` unchanged. Propose caller endpoint/account changes separately after integration approval; discovery must not activate bidders, publish configuration, or change traffic. diff --git a/.tool-versions b/.tool-versions index 34ab119d8..388787c3b 100644 --- a/.tool-versions +++ b/.tool-versions @@ -3,4 +3,4 @@ rust 1.95.0 nodejs 24.12.0 viceroy 0.17.0 wasmtime 44.0.1 -aws 2.36.45 +awscli 2.36.45 diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 4cdabecc0..18222f9c3 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -21,7 +21,7 @@ On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fiction | Command | What it does | Access | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------- | -| `ts prebid server inspect --config ` | Reports selected local Prebid fields, classifies bidder lists, and marks host-secret requirements unresolved | Local read-only | +| `ts prebid server inspect --config ` | Reports routed Prebid Server providers and bidders separately from browser Prebid fields; host-secret requirements remain unresolved | Local read-only | | `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and regional YAML merging | Local read-only | | `ts prebid server secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | | `ts prebid server status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | @@ -34,9 +34,9 @@ Not implemented: container deployment, rollback, runtime secret injection, calle ## Discovering requirements -`inspect` reads exactly the chosen file and never rewrites or publishes it. It accepts the runtime's array, indexed-map, and string bidder-list encodings, array-form bundle/module lists, and reports explicitly supplied values only. It does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source/environment is authoritative before relying on the report. +`inspect` reads exactly the chosen file and never rewrites or publishes it. It discovers server demand from `[auction.providers.*]` entries using the `prebid-server` profile and bidders routed through `[auction.bidders.*]`. The JSON report groups routed bidders under each provider. Browser settings still come from `[integrations.prebid]`, including the runtime's array, indexed-map, and string encodings for `client_side_bidders`. The command reports explicitly supplied values only; it does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source and environment are authoritative before relying on the report. -Account identifiers, endpoint values, and bid-parameter values are withheld. Parser errors also withhold source snippets. Server-side bidders, client-side bidders, and browser bundle adapters remain separate lists; listing a bidder does not establish partner authorization or a host-secret requirement. Disabled integrations remain disabled. +Account identifiers, endpoint values, and bid-parameter values are withheld. Parser errors also withhold source snippets. Server-side bidders, client-side bidders, and browser bundle adapters remain separate; listing a bidder does not establish partner authorization or a host-secret requirement. Disabled auctions and integrations remain disabled. ## Deployment descriptor diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs index c0337c884..da009fad1 100644 --- a/crates/trusted-server-cli/src/commands/pbs/inspect.rs +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -1,18 +1,76 @@ +use std::collections::BTreeMap; use std::path::Path; use error_stack::Report; -use serde::{Deserialize, Deserializer}; +use serde::{Deserialize, Deserializer, Serialize}; use serde_json::{Value, json}; +use trusted_server_core::auction_config_types::{BidderId, ProviderId}; use super::{Output, PbsError, Result, identifier, read_text}; /// Deliberately partial: inspecting PBS requirements must not require unrelated TS settings. #[derive(Default, Deserialize)] struct Source { + auction: Option, #[serde(default)] integrations: Integrations, } +#[derive(Default, Deserialize)] +struct Auction { + enabled: Option, + #[serde(default)] + providers: BTreeMap, + #[serde(default)] + bidders: BTreeMap, +} + +#[derive(Deserialize)] +struct AuctionProvider { + profile: Option, + endpoint: Option, + timeout_ms: Option, + profile_config: Option, +} + +impl AuctionProvider { + fn profile_bool(&self, key: &str) -> Option { + self.profile_config.as_ref()?.get(key)?.as_bool() + } + + fn override_rule_count(&self) -> usize { + self.profile_config + .as_ref() + .and_then(|config| config.get("bid_param_override_rules")) + .and_then(toml::Value::as_array) + .map_or(0, Vec::len) + } +} + +#[derive(Deserialize)] +struct AuctionBidder { + provider: ProviderId, +} + +#[derive(Serialize)] +struct ServerBidderCandidate { + bidder: String, + source_key: String, + host_secret_requirement: &'static str, + partner_authorization: &'static str, +} + +#[derive(Serialize)] +struct ServerProviderReport { + provider: String, + endpoint_configured: bool, + timeout_ms_explicit: Option, + test_mode_explicit: Option, + debug_explicit: Option, + bid_param_override_rule_count: usize, + server_bidder_candidates: Vec, +} + #[derive(Default, Deserialize)] struct Integrations { prebid: Option, @@ -21,19 +79,13 @@ struct Integrations { #[derive(Default, Deserialize)] struct Prebid { enabled: Option, - server_url: Option, account_id: Option, timeout_ms: Option, - test_mode: Option, debug: Option, #[serde(default, deserialize_with = "bidder_list")] - bidders: Vec, - #[serde(default, deserialize_with = "bidder_list")] client_side_bidders: Vec, #[serde(default)] bundle: Bundle, - #[serde(default)] - bid_param_override_rules: Vec, } #[derive(Default, Deserialize)] @@ -116,12 +168,13 @@ pub(super) fn inspect(path: &Path) -> Result { "cannot parse Trusted Server TOML; source details withheld", )) })?; - let present = source.integrations.prebid.is_some(); + let auction_present = source.auction.is_some(); + let auction = source.auction.unwrap_or_default(); + let prebid_present = source.integrations.prebid.is_some(); let prebid = source.integrations.prebid.unwrap_or_default(); for name in prebid - .bidders + .client_side_bidders .iter() - .chain(&prebid.client_side_bidders) .chain(&prebid.bundle.adapters) .chain(&prebid.bundle.user_id_modules) { @@ -131,30 +184,69 @@ pub(super) fn inspect(path: &Path) -> Result { ))); } } - let requirements: Vec<_> = prebid - .bidders + let server_providers: Vec<_> = auction + .providers .iter() - .map(|bidder| { - json!({ - "bidder": bidder, - "source_key": "integrations.prebid.bidders", - "host_secret_requirement": "unresolved", - "partner_authorization": "unresolved" - }) + .filter(|(_, provider)| provider.profile.as_deref() == Some("prebid-server")) + .map(|(provider_id, provider)| { + let requirements: Vec<_> = auction + .bidders + .iter() + .filter(|(_, bidder)| &bidder.provider == provider_id) + .map(|(bidder_id, _)| ServerBidderCandidate { + bidder: bidder_id.as_str().to_owned(), + source_key: format!("auction.bidders.{}.provider", bidder_id.as_str()), + host_secret_requirement: "unresolved", + partner_authorization: "unresolved", + }) + .collect(); + ServerProviderReport { + provider: provider_id.as_str().to_owned(), + endpoint_configured: provider.endpoint.is_some(), + timeout_ms_explicit: provider.timeout_ms, + test_mode_explicit: provider.profile_bool("test_mode"), + debug_explicit: provider.profile_bool("debug"), + bid_param_override_rule_count: provider.override_rule_count(), + server_bidder_candidates: requirements, + } }) .collect(); let warnings = [ "Local file only: confirm environment, remote configuration, and request-time overrides.", "Omitted fields/defaults are not expanded; empty candidate lists are not proof of no demand.", - "Disabled integrations and browser bundle adapters do not authorize PBS activation.", + "Disabled auctions, providers, and browser bundle adapters do not authorize PBS activation.", "Host secret requirements need adapter metadata verified against the selected PBS release.", ]; let mut details = vec![ format!( - "Prebid section present: {present}; enabled explicitly: {:?}", + "Auction section present: {auction_present}; enabled explicitly: {:?}", + auction.enabled + ), + format!( + "Prebid browser section present: {prebid_present}; enabled explicitly: {:?}", prebid.enabled ), - format!("Server bidder candidates: {}", prebid.bidders.join(", ")), + format!("Prebid Server providers: {}", server_providers.len()), + ]; + details.extend(server_providers.iter().map(|provider| { + let bidders = provider + .server_bidder_candidates + .iter() + .map(|candidate| candidate.bidder.as_str()) + .collect::>() + .join(", "); + format!( + "Server provider {}: bidders: {}; endpoint configured: {}; timeout explicit: {:?}; test mode explicit: {:?}; debug explicit: {:?}; bid-parameter rules: {}; values withheld", + provider.provider, + bidders, + provider.endpoint_configured, + provider.timeout_ms_explicit, + provider.test_mode_explicit, + provider.debug_explicit, + provider.bid_param_override_rule_count + ) + })); + details.extend([ format!( "Client-side bidders: {}", prebid.client_side_bidders.join(", ") @@ -163,11 +255,7 @@ pub(super) fn inspect(path: &Path) -> Result { "Browser bundle adapters: {}", prebid.bundle.adapters.join(", ") ), - format!( - "Bid-parameter rules: {}; values withheld", - prebid.bid_param_override_rules.len() - ), - ]; + ]); details.extend(warnings.iter().map(|warning| (*warning).to_owned())); Ok(Output { failure: None, @@ -175,19 +263,21 @@ pub(super) fn inspect(path: &Path) -> Result { details, data: json!({ "source": path, - "source_section": "integrations.prebid", - "section_present": present, + "source_sections": { + "server": ["auction.providers", "auction.bidders"], + "browser": "integrations.prebid" + }, + "auction_section_present": auction_present, + "auction_enabled_explicit": auction.enabled, + "server_providers": server_providers, + "section_present": prebid_present, "enabled_explicit": prebid.enabled, - "server_url_configured": prebid.server_url.is_some(), "account_id_configured": prebid.account_id.is_some(), "timeout_ms_explicit": prebid.timeout_ms, - "test_mode_explicit": prebid.test_mode, "debug_explicit": prebid.debug, - "server_bidder_candidates": requirements, "client_side_bidders": prebid.client_side_bidders, "bundle_adapters": prebid.bundle.adapters, "identity_modules": prebid.bundle.user_id_modules, - "bid_param_override_rule_count": prebid.bid_param_override_rules.len(), "warnings": warnings }), }) @@ -204,38 +294,104 @@ mod tests { let dir = tempfile::tempdir().expect("should create temp directory"); let path = dir.path().join("trusted-server.toml"); let source = r#" +[auction] +enabled = true + +[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://user:NEVER_PRINT_ME@pbs.example.com/path?token=NEVER_PRINT_ME" +timeout_ms = 900 +routing = "explicit" + +[auction.providers.pbs-main.profile_config] +debug = false +test_mode = true +bid_param_override_rules = [{ when = { bidder = "serverbidder" }, set = { placementId = "NEVER_PRINT_ME" } }] + +[auction.bidders.serverbidder] +provider = "pbs-main" + +[auction.providers.pbs-secondary] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://NEVER_PRINT_ME@secondary.example.com/openrtb2/auction" +routing = "explicit" + +[auction.bidders.otherbidder] +provider = "pbs-secondary" + [integrations.prebid] enabled = false -server_url = "https://user:NEVER_PRINT_ME@pbs.example.com/path?token=NEVER_PRINT_ME" account_id = "NEVER_PRINT_ME" -bidders = ["serverbidder"] client_side_bidders = ["browserbidder"] -[[integrations.prebid.bid_param_override_rules]] -set = { placementId = "NEVER_PRINT_ME" } + [integrations.prebid.bundle] adapters = ["bundlebidder"] user_id_modules = ["sharedIdSystem"] "#; fs::write(&path, source).expect("should write fixture"); let report = inspect(&path).expect("should inspect config"); + assert_eq!(report.data["auction_section_present"], true); + assert_eq!(report.data["auction_enabled_explicit"], true); assert_eq!(report.data["enabled_explicit"], false); assert_eq!( - report.data["server_bidder_candidates"][0]["bidder"], + report.data["server_providers"].as_array().map(Vec::len), + Some(2) + ); + assert_eq!(report.data["server_providers"][0]["provider"], "pbs-main"); + assert_eq!( + report.data["server_providers"][0]["server_bidder_candidates"][0]["bidder"], "serverbidder" ); + assert_eq!( + report.data["server_providers"][1]["provider"], + "pbs-secondary" + ); + assert_eq!( + report.data["server_providers"][1]["server_bidder_candidates"][0]["bidder"], + "otherbidder" + ); + assert_eq!( + report.data["server_providers"][0]["server_bidder_candidates"][0]["source_key"], + "auction.bidders.serverbidder.provider" + ); + assert_eq!( + report.data["server_providers"][0]["endpoint_configured"], + true + ); + assert_eq!( + report.data["server_providers"][0]["timeout_ms_explicit"], + 900 + ); + assert_eq!( + report.data["server_providers"][0]["test_mode_explicit"], + true + ); + assert_eq!(report.data["server_providers"][0]["debug_explicit"], false); + assert_eq!( + report.data["server_providers"][0]["bid_param_override_rule_count"], + 1 + ); assert_eq!(report.data["client_side_bidders"][0], "browserbidder"); assert_eq!(report.data["bundle_adapters"][0], "bundlebidder"); assert_eq!( - report.data["server_bidder_candidates"][0]["host_secret_requirement"], + report.data["server_providers"][0]["server_bidder_candidates"][0]["host_secret_requirement"], "unresolved" ); - for json in [false, true] { - let mut output = Vec::new(); - report - .write(json, &mut output) - .expect("should render report"); - assert!(!String::from_utf8_lossy(&output).contains("NEVER_PRINT_ME")); - } + let mut human = Vec::new(); + report + .write(false, &mut human) + .expect("should render human report"); + let human = String::from_utf8(human).expect("should emit UTF-8"); + assert!(human.contains("pbs-main")); + assert!(human.contains("serverbidder")); + assert!(!human.contains("NEVER_PRINT_ME")); + let mut json = Vec::new(); + report + .write(true, &mut json) + .expect("should render JSON report"); + assert!(!String::from_utf8_lossy(&json).contains("NEVER_PRINT_ME")); assert_eq!( fs::read_to_string(path).expect("should read fixture"), source @@ -258,7 +414,7 @@ user_id_modules = ["sharedIdSystem"] file.path(), r#" [integrations.prebid] -bidders = 'examplebidder\' +client_side_bidders = 'examplebidder\' "#, ) .expect("should write config"); @@ -276,7 +432,7 @@ bidders = 'examplebidder\' } #[test] - fn accepts_the_runtime_bidder_list_encodings() { + fn accepts_the_runtime_browser_bidder_list_encodings() { for input in [ "['examplebidder', 'otherbidder']", "'examplebidder,otherbidder'", @@ -285,26 +441,13 @@ bidders = 'examplebidder\' "'example\\u0062idder,otherbidder'", "{ '10' = 'otherbidder', '2' = 'examplebidder' }", ] { - let text = format!( - "[integrations.prebid]\nserver_url='https://pbs.example.com'\nbidders={input}\nclient_side_bidders={input}\n" - ); + let text = format!("[integrations.prebid]\nclient_side_bidders={input}\n"); let file = tempfile::NamedTempFile::new().expect("should create config"); fs::write(file.path(), &text).expect("should write config"); let runtime: trusted_server_core::integrations::prebid::PrebidIntegrationConfig = toml::from_str(&format!("client_side_bidders={input}")) .expect("runtime should accept encoding"); let output = inspect(file.path()).expect("inspect should accept runtime encoding"); - let candidates: Vec<_> = output.data["server_bidder_candidates"] - .as_array() - .expect("should report candidates") - .iter() - .map(|candidate| { - candidate["bidder"] - .as_str() - .expect("should identify bidder") - }) - .collect(); - assert_eq!(candidates, ["examplebidder", "otherbidder"]); assert_eq!( output.data["client_side_bidders"], json!(runtime.client_side_bidders) diff --git a/crates/trusted-server-cli/src/commands/pbs/mod.rs b/crates/trusted-server-cli/src/commands/pbs/mod.rs index 6a10e2ef9..ce11434f7 100644 --- a/crates/trusted-server-cli/src/commands/pbs/mod.rs +++ b/crates/trusted-server-cli/src/commands/pbs/mod.rs @@ -176,6 +176,10 @@ pub(super) fn identifier(value: &str) -> bool { .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-')) } +fn is_confirmation(input: &str) -> bool { + input.len() <= 16 && input.trim() == "yes" +} + /// Operator interaction is injectable so tests cannot accidentally read a terminal. pub(super) trait Interaction { /// Read a complete secret JSON object without echoing it. @@ -216,18 +220,14 @@ impl Interaction for Terminal { } self.notice(target)?; self.notice("Write this secret version? Type yes to confirm:")?; - let line = read_bounded( - io::stdin() - .lock() - .lines() - .next() - .transpose() - .map_err(|_| Report::new(PbsError::Io("cannot read confirmation")))? - .unwrap_or_default() - .as_bytes(), - 16, - )?; - Ok(line.trim() == "yes") + let line = io::stdin() + .lock() + .lines() + .next() + .transpose() + .map_err(|_| Report::new(PbsError::Io("cannot read confirmation")))? + .unwrap_or_default(); + Ok(is_confirmation(&line)) } fn notice(&mut self, message: &str) -> Result<()> { @@ -235,3 +235,15 @@ impl Interaction for Terminal { .map_err(|_| Report::new(PbsError::Io("cannot write operator notice"))) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn confirmation_accepts_only_bounded_yes() { + assert!(is_confirmation("yes")); + assert!(!is_confirmation("no")); + assert!(!is_confirmation("yes-but-with-more-than-sixteen-bytes")); + } +} diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index d2eb15e44..ca316eed0 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -128,7 +128,7 @@ fn assert_payload_cleanup(dir: &Path) { #[test] fn local_commands_never_execute_aws_and_preserve_the_source() { let dir = fixture(); - let toml = "[integrations.prebid]\nenabled=false\nbidders=['examplebidder']\naccount_id='DUMMY_SECRET'\n"; + let toml = "[auction]\nenabled=true\n[auction.providers.pbs-main]\nprofile='prebid-server'\nendpoint='https://DUMMY_SECRET@pbs.example.com/openrtb2/auction'\n[auction.bidders.examplebidder]\nprovider='pbs-main'\n[integrations.prebid]\nenabled=false\naccount_id='DUMMY_SECRET'\n"; fs::write(dir.path().join("trusted-server.toml"), toml).expect("should write TOML"); let output = command(dir.path()) .args(["prebid", "server", "inspect", "--json"]) @@ -138,6 +138,11 @@ fn local_commands_never_execute_aws_and_preserve_the_source() { assert_no_secret(&output); let report: Value = serde_json::from_slice(&output.stdout).expect("should emit JSON"); assert_eq!(report["enabled_explicit"], false); + assert_eq!(report["server_providers"][0]["provider"], "pbs-main"); + assert_eq!( + report["server_providers"][0]["server_bidder_candidates"][0]["bidder"], + "examplebidder" + ); assert_eq!( fs::read_to_string(dir.path().join("trusted-server.toml")).expect("should read TOML"), toml diff --git a/deploy/pbs-example/modules/regional/monitoring.tf b/deploy/pbs-example/modules/regional/monitoring.tf index ddfc81566..8c5a56766 100644 --- a/deploy/pbs-example/modules/regional/monitoring.tf +++ b/deploy/pbs-example/modules/regional/monitoring.tf @@ -10,7 +10,7 @@ resource "aws_cloudwatch_metric_alarm" "instance_cpu" { statistic = "Average" threshold = 80 alarm_actions = var.alarm_actions - treat_missing_data = "breaching" + treat_missing_data = "missing" dimensions = { InstanceId = each.value.id diff --git a/deploy/pbs-example/modules/regional/terraform.tf b/deploy/pbs-example/modules/regional/terraform.tf index f2702bf6e..1e8efd5f5 100644 --- a/deploy/pbs-example/modules/regional/terraform.tf +++ b/deploy/pbs-example/modules/regional/terraform.tf @@ -1,7 +1,8 @@ terraform { required_providers { aws = { - source = "hashicorp/aws" + source = "hashicorp/aws" + version = "= 6.64.0" } } } diff --git a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl index 7b9455229..eae45a4e6 100644 --- a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl +++ b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl @@ -63,6 +63,14 @@ run "plans_private_hosts_and_scoped_ingress" { error_message = "The example should create metadata for only the declared example bidder secret." } + assert { + condition = alltrue([ + for alarm in aws_cloudwatch_metric_alarm.instance_cpu : + alarm.treat_missing_data == "missing" + ]) + error_message = "Missing CPU samples should not trigger high-CPU alarms." + } + assert { condition = length(jsondecode(aws_iam_role_policy.runtime.policy).Statement[0].Resource) == length(aws_secretsmanager_secret.bidder) error_message = "The runtime role should read only the declared bidder secrets." From fbb18bd7532cb9a56c02b564d20297cd51746357 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 21 Sep 2026 15:08:41 -0500 Subject: [PATCH 10/14] Harden PBS validation and deployment examples Make security tests catch broken host and network invariants, and keep local smoke checks separate from deployment bindings. Improve operator errors without exposing credential values or changing write authority. Keep the intentionally retired bundle command rejected. Record deferred cloud evidence and use reproducible Terraform tooling and provider locks. --- .../references/architecture.md | 18 +-- .../references/configuration-and-secrets.md | 26 ++-- .../references/file-generation.md | 20 +-- .../references/rtb-fabric.md | 20 +-- .tool-versions | 1 + AGENTS.md | 1 + crates/trusted-server-cli/README.md | 14 +- .../src/commands/pbs/aws.rs | 7 +- .../src/commands/pbs/config.rs | 19 ++- .../src/commands/pbs/mod.rs | 28 +++- .../src/commands/pbs/secrets.rs | 9 +- crates/trusted-server-cli/src/run.rs | 21 ++- crates/trusted-server-cli/tests/pbs_cli.rs | 141 +++++++++++++++--- deploy/pbs-example/.terraform.lock.hcl | 3 + deploy/pbs-example/DEPLOYMENT_PLAN.md | 90 ++++++----- deploy/pbs-example/README.md | 7 +- deploy/pbs-example/RUNBOOK.md | 17 ++- .../pbs-example/modules/regional/security.tf | 40 ++--- .../tests/security_unit_test.tftest.hcl | 103 ++++++++++++- deploy/pbs-example/runtime/README.md | 2 + deploy/pbs-example/runtime/compose.yaml | 2 +- deploy/pbs-example/runtime/examples/README.md | 2 +- deploy/pbs-example/scripts/smoke-runtime.sh | 4 + .../pbs-example/scripts/test-smoke-runtime.py | 90 +++++++++++ 24 files changed, 533 insertions(+), 152 deletions(-) create mode 100755 deploy/pbs-example/scripts/test-smoke-runtime.py diff --git a/.claude/skills/planning-prebid-aws/references/architecture.md b/.claude/skills/planning-prebid-aws/references/architecture.md index a00909f5c..df3783e65 100644 --- a/.claude/skills/planning-prebid-aws/references/architecture.md +++ b/.claude/skills/planning-prebid-aws/references/architecture.md @@ -17,16 +17,16 @@ Use EKS only when the user's existing platform or an explicit requirement justif ## Network, ingress, and egress -| Need | Candidate services or mechanism | Required evidence | -| --------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -| Public HTTPS on standalone hosts | Public subnet, internet gateway, Elastic IP, Caddy | Accepted exposure, durable certificate storage, renewal and replacement path | -| Managed HTTPS with multiple backends | ALB and ACM | Certificate ownership/validation, target health, draining, timeout budget, private management endpoints | -| Private compute reaching public bidders | NAT-based egress or an approved existing egress service | AZ failure behavior, routing, hourly/data charges, and outbound address stability | +| Need | Candidate services or mechanism | Required evidence | +| --------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | +| Public HTTPS on standalone hosts | Public subnet, internet gateway, Elastic IP, Caddy | Accepted exposure, durable certificate storage, renewal and replacement path | +| Managed HTTPS with multiple backends | ALB and ACM | Certificate ownership/validation, target health, draining, timeout budget, private management endpoints | +| Private compute reaching public bidders | NAT-based egress or an approved existing egress service | AZ failure behavior, routing, hourly/data charges, and outbound address stability | | Private PBS-to-bidder connectivity | AWS RTB Fabric requester gateway and standard or outbound external links | Partner participation, gateway and link ownership, regional support, quotas, bidder endpoint mapping, timeout, fallback, and cost | -| Bidder source-IP allowlists | Stable egress IPs | Every normal, scaling, and failover path uses partner-approved addresses | -| Existing caller selects regions | Regional hostnames | Caller routing and failure policy, TLS, identity behavior | -| DNS-based regional selection | Route 53 latency records with per-region health checks | Independent health targets, cached-answer behavior, all-unhealthy behavior, and failover capacity | -| Public auctions from untrusted callers | Existing controls or WAF with compatible ingress | Auction payload compatibility, rate limits, false-positive testing, cost, and owner | +| Bidder source-IP allowlists | Stable egress IPs | Every normal, scaling, and failover path uses partner-approved addresses | +| Existing caller selects regions | Regional hostnames | Caller routing and failure policy, TLS, identity behavior | +| DNS-based regional selection | Route 53 latency records with per-region health checks | Independent health targets, cached-answer behavior, all-unhealthy behavior, and failover capacity | +| Public auctions from untrusted callers | Existing controls or WAF with compatible ingress | Auction payload compatibility, rate limits, false-positive testing, cost, and owner | Private subnets do not provide internet egress by themselves. VPC endpoints may serve supported AWS APIs but do not replace bidder internet access. Public addressing on replaceable compute does not by itself provide stable egress for allowlists. diff --git a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md index 3b49f66a4..f0ff876d6 100644 --- a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md +++ b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md @@ -12,17 +12,17 @@ Before asking questions the repository can answer: 4. Add observations to the existing deployment decision record with source path/key, candidate requirement, confidence/status, and the remaining question. Report credential identifiers or presence only; keep raw TOML, credential values, and commercially sensitive publisher values out of reports and generated examples. 5. Verify candidate adapters against the selected PBS Go release and confirm partner authorization. Record each host-secret requirement as required, not needed, or unresolved. A bidder name alone proves neither a secret requirement nor permission to use the bidder. -| Trusted Server input | Discovery use | -| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| `auction.enabled` | Determine active versus disabled server-auction intent without expanding defaults | -| `auction.providers.` with `profile = "prebid-server"` | Identify each configured PBS provider; withhold endpoint values | -| Provider `timeout_ms` and `profile_config.test_mode` or `debug` | Record explicit timeout and diagnostic intent, not permission to contact bidders | -| Provider `profile_config.bid_param_override_rules` | Count conditional publisher or placement inputs without reporting their values | -| `auction.bidders..provider` | Group candidate server-side adapters under the referenced PBS provider | -| `integrations.prebid.enabled`, `account_id`, `timeout_ms`, `debug` | Record browser integration intent and compatibility questions separately from server demand | -| `integrations.prebid.client_side_bidders` | Browser-side participation; do not automatically enable these adapters in PBS | -| `integrations.prebid.bundle` | Browser bundle and identity capabilities, not proof of server-side use | -| Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller or request path | +| Trusted Server input | Discovery use | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| `auction.enabled` | Determine active versus disabled server-auction intent without expanding defaults | +| `auction.providers.` with `profile = "prebid-server"` | Identify each configured PBS provider; withhold endpoint values | +| Provider `timeout_ms` and `profile_config.test_mode` or `debug` | Record explicit timeout and diagnostic intent, not permission to contact bidders | +| Provider `profile_config.bid_param_override_rules` | Count conditional publisher or placement inputs without reporting their values | +| `auction.bidders..provider` | Group candidate server-side adapters under the referenced PBS provider | +| `integrations.prebid.enabled`, `account_id`, `timeout_ms`, `debug` | Record browser integration intent and compatibility questions separately from server demand | +| `integrations.prebid.client_side_bidders` | Browser-side participation; do not automatically enable these adapters in PBS | +| `integrations.prebid.bundle` | Browser bundle and identity capabilities, not proof of server-side use | +| Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller or request path | Use `ts prebid server inspect --config ` for redacted local discovery. It reports explicit values without expanding defaults or proving host-secret requirements. Resolve schema defaults and adapter metadata separately, recording unsupported details as unresolved rather than fetching credentials or inventing mappings. Read config without invoking commands that publish or rewrite it. Leave `trusted-server.toml` unchanged. Propose caller endpoint/account changes separately after integration approval; discovery must not activate bidders, publish configuration, or change traffic. @@ -75,8 +75,8 @@ Never package or log secret values. Treat container inspection and debug output Use `ts prebid server` instead of generating deployment-local wrappers for these implemented operations. Keep existing `ts config`, `ts deploy`, and `ts prebid client` behavior unchanged. -| Command | Current result and boundary | -| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Command | Current result and boundary | +| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ts prebid server inspect --config ` | Local discovery with unresolved requirements; no source changes or secret retrieval | | `ts prebid server check --deployment ` | Local schema, binding, and regional merge checks; no AWS calls, upstream PBS schema validation, or startup proof | | `ts prebid server secrets set --deployment --region ` | Authorized complete-value write to an existing declared secret; hidden terminal or file/stdin input, account/confirmation checks, and retry UUID; no deployment | diff --git a/.claude/skills/planning-prebid-aws/references/file-generation.md b/.claude/skills/planning-prebid-aws/references/file-generation.md index 72abe045d..fb27e3fbe 100644 --- a/.claude/skills/planning-prebid-aws/references/file-generation.md +++ b/.claude/skills/planning-prebid-aws/references/file-generation.md @@ -4,17 +4,17 @@ Generate files only after the design and target paths are approved. Use the repo ## Artifact contract -| Artifact | Suggested location | Contents and owner | -| -------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Deployment plan | `docs/pbs-deployment-plan.md` | Requirements record, approved architecture, sources, costs, blockers, file/check inventory; user approves | -| State bootstrap, if needed | `infra/bootstrap/` | Independently managed backend, access and locking setup; infrastructure owner | -| Deployment root | `infra//` | Providers, variables/validation, explicit regional modules, outputs, selected DNS configuration, nonsecret examples, provider lock file; Terraform owns AWS resources | -| Reusable infrastructure | `infra/modules/` | Modules justified by repeated topology, not one module per AWS service | -| Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | -| Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | -| Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | +| Artifact | Suggested location | Contents and owner | +| -------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Deployment plan | `docs/pbs-deployment-plan.md` | Requirements record, approved architecture, sources, costs, blockers, file/check inventory; user approves | +| State bootstrap, if needed | `infra/bootstrap/` | Independently managed backend, access and locking setup; infrastructure owner | +| Deployment root | `infra//` | Providers, variables/validation, explicit regional modules, outputs, selected DNS configuration, nonsecret examples, provider lock file; Terraform owns AWS resources | +| Reusable infrastructure | `infra/modules/` | Modules justified by repeated topology, not one module per AWS service | +| Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | +| Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | +| Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | | Operator interface | Existing `ts prebid server` CLI and approved CI layout | CLI descriptor and documented invocations; any missing release stages need separate implementation approval | -| Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | +| Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | Each generated artifact must have a consumer. Add ignore rules for local credentials, runtime secret files, `.terraform`, state files, saved plans, and generated sensitive output. Track nonsecret examples. Use `example.com` hostnames and visibly fictional identifiers in examples. diff --git a/.claude/skills/planning-prebid-aws/references/rtb-fabric.md b/.claude/skills/planning-prebid-aws/references/rtb-fabric.md index 98f49714a..c325f1593 100644 --- a/.claude/skills/planning-prebid-aws/references/rtb-fabric.md +++ b/.claude/skills/planning-prebid-aws/references/rtb-fabric.md @@ -18,16 +18,16 @@ Route only partner-approved bidder endpoints through Fabric. Keep the approved N For each bidder and PBS region, record: -| Decision | Required evidence | -| --- | --- | -| Participation | Partner name, responder gateway ID or public endpoint, AWS account owner, and acceptance owner | -| Link type | Standard internal link or outbound external link, with the selected region | -| PBS binding | Pinned PBS Go release, adapter endpoint field, config source, rollout owner, and link URL update procedure | -| Capacity | Peak transactions per second, bidder timeout, payload sizes, burst duration, and regional failover load | -| Fallback | Behavior while a link is requested, inactive, over quota, timing out, or unavailable | -| Operations | Link creation, partner acceptance, status checks, endpoint changes, deletion order, alerts, and recovery owner | -| Cost | Region, monthly sent transactions, payload distribution, no-bid volume, link count, external traffic, and estimate date | -| Privacy and security | OpenRTB fields, TLS mode, source-IP requirements, data residency, logging, and retention | +| Decision | Required evidence | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| Participation | Partner name, responder gateway ID or public endpoint, AWS account owner, and acceptance owner | +| Link type | Standard internal link or outbound external link, with the selected region | +| PBS binding | Pinned PBS Go release, adapter endpoint field, config source, rollout owner, and link URL update procedure | +| Capacity | Peak transactions per second, bidder timeout, payload sizes, burst duration, and regional failover load | +| Fallback | Behavior while a link is requested, inactive, over quota, timing out, or unavailable | +| Operations | Link creation, partner acceptance, status checks, endpoint changes, deletion order, alerts, and recovery owner | +| Cost | Region, monthly sent transactions, payload distribution, no-bid volume, link count, external traffic, and estimate date | +| Privacy and security | OpenRTB fields, TLS mode, source-IP requirements, data residency, logging, and retention | Gateway IDs, link state, quota increases, and partner acceptance are external facts. Ask before authenticated AWS inspection, naming the account, role, regions, and read scope. Do not treat a partner's willingness to participate as proof that its gateway, endpoint, adapter, or regional capacity is ready. diff --git a/.tool-versions b/.tool-versions index 388787c3b..6a44ca668 100644 --- a/.tool-versions +++ b/.tool-versions @@ -4,3 +4,4 @@ nodejs 24.12.0 viceroy 0.17.0 wasmtime 44.0.1 awscli 2.36.45 +terraform 1.16.2 diff --git a/AGENTS.md b/AGENTS.md index cb73c1dc5..419a07956 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,6 +40,7 @@ Supporting files: `edgezero.toml`, `fastly.toml`, | Viceroy | 0.17.0 (from `.tool-versions`) | | Wasmtime | 44.0.1 (from `.tool-versions`) | | AWS CLI | 2.36.45 (from `.tool-versions`) | +| Terraform | 1.16.2 (from `.tool-versions`) | --- diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 18222f9c3..91d128829 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -2,7 +2,7 @@ `ts prebid server` manages local configuration inputs and a small set of AWS operations for self-hosted Prebid Server Go. It sits beside `ts prebid client`, which builds browser JavaScript, and remains separate from the existing Trusted Server `ts config` and `ts deploy` commands. -The namespace is experimental and is being evaluated in PR review. There is no separate binary or crate. +The namespace is experimental; its interface may change without a deprecation cycle. There is no separate binary or crate. ## Build and try locally @@ -19,12 +19,12 @@ On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fiction ## Commands and current limits -| Command | What it does | Access | -| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------- | +| Command | What it does | Access | +| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | | `ts prebid server inspect --config ` | Reports routed Prebid Server providers and bidders separately from browser Prebid fields; host-secret requirements remain unresolved | Local read-only | -| `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and regional YAML merging | Local read-only | -| `ts prebid server secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | -| `ts prebid server status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | +| `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and regional YAML merging | Local read-only | +| `ts prebid server secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | +| `ts prebid server status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | Add `--json` anywhere under `ts prebid server` for a machine-readable report. Errors and operator notices go to stderr; failures exit with code 2. A partial status report still appears on stdout, with `complete: false` and exit code 2. @@ -88,7 +88,7 @@ File and stdin inputs are mutually exclusive. `--stdin` requires `--yes` and `-- Secret values never enter command arguments or reports. The AWS CLI receives JSON through a tool-owned temporary file, owner-only on Unix, which is removed on normal success and error paths. Operator-provided input files are not changed or deleted. Run on a trusted host with protected temporary storage; abrupt process termination can leave temporary files requiring cleanup. Windows temporary-file ACL behavior has not been validated. -A successful write reports its version identifier. It does not create secret metadata, change infrastructure, replace containers, or rotate the bidder's credential. Check regional replication, separately replace consumers, and verify them before revoking old partner credentials. A lost/malformed response leaves the write outcome uncertain; retain the displayed request token and retry identical input rather than creating another logical update. +A successful write reports its version identifier. It does not create secret metadata, change infrastructure, replace containers, or rotate the bidder's credential. Check regional replication, separately replace consumers, and verify them before revoking old partner credentials. A failed or unverifiable response means the write is not confirmed; the outcome may be uncertain. Retain the displayed request token and reuse it only for the original identical payload. Use a new token only for separately intended changed values. Provider error details are withheld, so the CLI cannot distinguish a rejected write from a lost response. ## Ownership and sandbox compatibility diff --git a/crates/trusted-server-cli/src/commands/pbs/aws.rs b/crates/trusted-server-cli/src/commands/pbs/aws.rs index d49bae99e..70a29e9b8 100644 --- a/crates/trusted-server-cli/src/commands/pbs/aws.rs +++ b/crates/trusted-server-cli/src/commands/pbs/aws.rs @@ -26,6 +26,9 @@ pub(super) trait Aws { pub(super) struct AwsCli; +/// Provider output is withheld, so a rejected write cannot be distinguished from a lost response. +pub(super) const WRITE_NOT_CONFIRMED: &str = "write not confirmed; outcome may be uncertain; retain the request token and reuse it only for the original identical payload; use a new token for separately intended changed values"; + impl Aws for AwsCli { fn call( &self, @@ -76,7 +79,7 @@ impl Aws for AwsCli { drop(payload); if !output.status.success() { let message = if operation == "put-secret-value" { - "write outcome uncertain; retain the request token and retry identical input" + WRITE_NOT_CONFIRMED } else { operation }; @@ -84,7 +87,7 @@ impl Aws for AwsCli { } serde_json::from_slice(&output.stdout).map_err(|_| { let message = if operation == "put-secret-value" { - "write response invalid; outcome uncertain, retain the request token" + WRITE_NOT_CONFIRMED } else { "invalid JSON response" }; diff --git a/crates/trusted-server-cli/src/commands/pbs/config.rs b/crates/trusted-server-cli/src/commands/pbs/config.rs index 1e00c09ff..671972462 100644 --- a/crates/trusted-server-cli/src/commands/pbs/config.rs +++ b/crates/trusted-server-cli/src/commands/pbs/config.rs @@ -272,7 +272,9 @@ fn pinned_image(value: &str) -> bool { && !image.contains('@') && !image.chars().any(char::is_whitespace) && digest.len() == 64 - && digest.bytes().all(|byte| byte.is_ascii_hexdigit()) + && digest + .bytes() + .all(|byte| byte.is_ascii_digit() || matches!(byte, b'a'..=b'f')) }) } @@ -436,6 +438,21 @@ pub(super) mod tests { (dir, path) } + #[test] + fn pinned_image_requires_lowercase_sha256_hex() { + let image = |digest: &str| format!("registry.example.com/pbs@sha256:{digest}"); + assert!(pinned_image(&image(&"0123456789abcdef".repeat(4)))); + for uppercase in 'A'..='F' { + assert!(!pinned_image(&image(&format!( + "{}{uppercase}", + "a".repeat(63) + )))); + } + for digest in ["a".repeat(63), "a".repeat(65), "g".repeat(64)] { + assert!(!pinned_image(&image(&digest))); + } + } + #[test] fn resolves_relative_paths_merges_maps_and_replaces_arrays() { let (_dir, path) = fixture(); diff --git a/crates/trusted-server-cli/src/commands/pbs/mod.rs b/crates/trusted-server-cli/src/commands/pbs/mod.rs index ce11434f7..521f7c6ed 100644 --- a/crates/trusted-server-cli/src/commands/pbs/mod.rs +++ b/crates/trusted-server-cli/src/commands/pbs/mod.rs @@ -65,6 +65,8 @@ pub(crate) enum PbsError { Input(&'static str), #[display("PBS I/O failed: {_0}")] Io(&'static str), + #[display("PBS I/O failed: cannot open input file {path:?}")] + InputFile { path: PathBuf }, #[display("AWS operation failed: {_0}; provider output withheld")] Aws(&'static str), #[display("AWS account does not match the deployment descriptor; no further calls made")] @@ -147,7 +149,11 @@ pub(crate) fn run(args: &PbsArgs) -> Result<()> { /// # Errors /// Returns an I/O error or rejects oversized/non-UTF-8 input. pub(super) fn read_text(path: &Path, limit: usize) -> Result { - let file = File::open(path).map_err(|_| Report::new(PbsError::Io("cannot open input file")))?; + let file = File::open(path).map_err(|_| { + Report::new(PbsError::InputFile { + path: path.to_path_buf(), + }) + })?; read_bounded(file, limit) } @@ -240,6 +246,26 @@ impl Interaction for Terminal { mod tests { use super::*; + #[test] + fn read_bounded_preserves_size_utf8_and_error_redaction_guards() { + assert_eq!( + read_bounded(&b"abcd"[..], 4).expect("should accept limit"), + "abcd" + ); + for (input, limit) in [(&b"DUMMY_SECRET"[..], 4), (&b"\xffDUMMY_SECRET"[..], 64)] { + let error = read_bounded(input, limit).expect_err("should reject invalid input"); + assert!(!format!("{error:?}").contains("DUMMY_SECRET")); + } + struct FailedRead; + impl Read for FailedRead { + fn read(&mut self, _buffer: &mut [u8]) -> io::Result { + Err(io::Error::other("DUMMY_SECRET")) + } + } + let error = read_bounded(FailedRead, 4).expect_err("should reject read failure"); + assert!(!format!("{error:?}").contains("DUMMY_SECRET")); + } + #[test] fn confirmation_accepts_only_bounded_yes() { assert!(is_confirmation("yes")); diff --git a/crates/trusted-server-cli/src/commands/pbs/secrets.rs b/crates/trusted-server-cli/src/commands/pbs/secrets.rs index 4c6177cb1..3aa885187 100644 --- a/crates/trusted-server-cli/src/commands/pbs/secrets.rs +++ b/crates/trusted-server-cli/src/commands/pbs/secrets.rs @@ -9,7 +9,7 @@ use serde::{Deserialize, Deserializer}; use serde_json::{Value, json}; use uuid::Uuid; -use super::aws::{Aws, verify_identity}; +use super::aws::{Aws, WRITE_NOT_CONFIRMED, verify_identity}; use super::config::{Binding, Deployment, invalid}; use super::{Interaction, Output, PbsError, Result, read_bounded, read_text}; @@ -20,8 +20,9 @@ const MAX_SECRET_BYTES: usize = 65_536; pub(super) struct SetArgs { /// Bidder identifier declared in the binding file. bidder: String, + /// Deployment descriptor; paths inside it are relative to this file. #[arg(long)] - pub deployment: PathBuf, + pub(super) deployment: PathBuf, /// Exactly one declared region; replicas cannot be written independently. #[arg(long)] region: String, @@ -180,9 +181,7 @@ pub(super) fn set( if response.get("ARN").and_then(Value::as_str) != Some(arn.as_str()) || response.get("VersionId").and_then(Value::as_str) != Some(token.as_str()) { - return Err(Report::new(PbsError::Aws( - "write response could not be verified; outcome uncertain, retain the request token", - ))); + return Err(Report::new(PbsError::Aws(WRITE_NOT_CONFIRMED))); } Ok(Output { failure: None, diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 27129dc20..27137fd1b 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -157,7 +157,7 @@ fn dispatch(args: Args) -> Result { PrebidCommand::Server(args) => pbs::run(&args) .map(|()| RunOutcome::Success) .map_err(|error| error.current_context().to_string()), - } + }, Command::Provision(args) => { edgezero_cli::run_provision(&args).map(|()| RunOutcome::Success) } @@ -191,6 +191,25 @@ mod tests { ); } + #[test] + fn prebid_rejects_retired_bundle_command() { + let error = Args::try_parse_from(["ts", "prebid", "bundle"]) + .expect_err("should reject intentionally retired bundle spelling"); + assert_eq!(error.kind(), clap::error::ErrorKind::InvalidSubcommand); + } + + #[test] + fn prebid_server_secret_help_explains_descriptor_relative_paths() { + let error = Args::try_parse_from(["ts", "prebid", "server", "secrets", "set", "--help"]) + .expect_err("should display secret help"); + assert_eq!(error.kind(), clap::error::ErrorKind::DisplayHelp); + assert!( + error + .to_string() + .contains("Deployment descriptor; paths inside it are relative to this file") + ); + } + #[test] fn parses_prebid_server_inspect() { assert!( diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index ca316eed0..25b6ca1e8 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -50,11 +50,18 @@ if 'get-caller-identity' in args: elif 'describe-secret' in args: print(json.dumps({'ARN': arn})) elif 'put-secret-value' in args: - if os.environ.get('PBS_FAKE_FAILURE') == 'yes': - print(request['SecretString'], file=sys.stderr) + failure = os.environ.get('PBS_FAKE_FAILURE') + if failure in ('collision', 'transport'): + error = 'ResourceExistsException' if failure == 'collision' else 'Connection reset by peer' + print(error + ': ' + request['SecretString'], file=sys.stderr) sys.exit(1) (root / 'captured_request.json').write_text(json.dumps(request)) - print(json.dumps({'ARN': arn, 'VersionId': request['ClientRequestToken']})) + if failure == 'invalid-json': + print('invalid response ' + request['SecretString']) + elif failure == 'unverified': + print(json.dumps({'ARN': arn, 'VersionId': 'unexpected', 'SecretString': request['SecretString']})) + else: + print(json.dumps({'ARN': arn, 'VersionId': request['ClientRequestToken']})) elif 'describe-instance-status' in args: print(json.dumps({'InstanceStatuses': []})) else: @@ -169,6 +176,75 @@ fn local_commands_never_execute_aws_and_preserve_the_source() { ); } +#[test] +fn missing_descriptor_inputs_report_escaped_paths() { + for missing in ["deployment.yaml", "pbs.yaml", "bindings.json", "east.yaml"] { + let dir = fixture(); + let inputs = dir.path().join("inputs\n\t\u{1b}[31m\"\\"); + fs::create_dir(&inputs).expect("should create nested input directory"); + for name in ["deployment.yaml", "pbs.yaml", "bindings.json", "east.yaml"] { + if name != missing { + fs::copy(dir.path().join(name), inputs.join(name)).expect("should copy input"); + } + } + let path = inputs.join(missing); + let output = command(dir.path()) + .args(["prebid", "server", "check", "--deployment"]) + .arg(inputs.join("deployment.yaml")) + .output() + .expect("should run CLI"); + assert_missing_path(&output, &path); + assert!(!dir.path().join("calls").exists(), "should not call AWS"); + } +} + +#[test] +fn missing_secret_input_reports_escaped_path_without_writing() { + let dir = fixture(); + let path = dir.path().join("missing\n\t\u{1b}[31m\"\\secret.json"); + let output = command(dir.path()) + .args([ + "prebid", + "server", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--yes", + "--request-token", + TOKEN, + "--file", + ]) + .arg(&path) + .output() + .expect("should run CLI"); + assert_missing_path(&output, &path); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + assert!(!calls.contains("put-secret-value")); + assert_payload_cleanup(dir.path()); +} + +fn assert_missing_path(output: &Output, path: &Path) { + assert_eq!(output.status.code(), Some(2)); + assert!(output.stdout.is_empty()); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&format!("cannot open input file {path:?}")), + "{stderr}" + ); + assert!( + !stderr.contains(&*path.to_string_lossy()), + "should escape control characters" + ); + assert!(!stderr.contains("os error")); + assert!(!stderr.contains('\u{1b}')); + assert!(!stderr.contains('\t')); + assert_no_secret(output); +} + #[test] fn secret_payload_is_private_not_in_argv_and_deleted_after_use() { let dir = fixture(); @@ -214,23 +290,48 @@ fn secret_payload_is_private_not_in_argv_and_deleted_after_use() { #[test] fn aws_errors_never_forward_provider_stderr_and_cleanup_payloads() { - let dir = fixture(); - fs::write( - dir.path().join("secret.json"), - "{\"api_key\":\"DUMMY_SECRET\"}", - ) - .expect("should write fixture"); - let output = secret_command(dir.path()) - .env("PBS_FAKE_FAILURE", "yes") - .output() - .expect("should run CLI"); - assert_eq!(output.status.code(), Some(2)); - assert_no_secret(&output); - assert!( - String::from_utf8_lossy(&output.stderr).contains("outcome uncertain"), - "should warn that the write may have succeeded" - ); - assert_payload_cleanup(dir.path()); + for failure in ["collision", "transport", "invalid-json", "unverified"] { + let dir = fixture(); + let input = "{\"api_key\":\"DUMMY_SECRET\"}"; + fs::write(dir.path().join("secret.json"), input).expect("should write fixture"); + let output = secret_command(dir.path()) + .env("PBS_FAKE_FAILURE", failure) + .output() + .expect("should run CLI"); + assert_eq!(output.status.code(), Some(2)); + assert_no_secret(&output); + let stderr = String::from_utf8_lossy(&output.stderr); + for guidance in [ + "write not confirmed", + "uncertain", + "retain the request token", + "reuse it only for the original identical payload", + "new token for separately intended changed values", + "provider output withheld", + TOKEN, + ] { + assert!( + stderr.contains(guidance), + "{failure}: missing {guidance}: {stderr}" + ); + } + assert!(!stderr.contains("ResourceExistsException")); + assert!(!stderr.contains("Connection reset by peer")); + assert!(!stderr.contains("unexpected")); + assert!(output.stdout.is_empty()); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + assert_eq!( + calls.matches("put-secret-value").count(), + 1, + "should not retry" + ); + assert!(!calls.contains("DUMMY_SECRET")); + assert_eq!( + fs::read_to_string(dir.path().join("secret.json")).expect("should read input"), + input + ); + assert_payload_cleanup(dir.path()); + } } #[test] diff --git a/deploy/pbs-example/.terraform.lock.hcl b/deploy/pbs-example/.terraform.lock.hcl index 38e6eaafe..4f85daae2 100644 --- a/deploy/pbs-example/.terraform.lock.hcl +++ b/deploy/pbs-example/.terraform.lock.hcl @@ -6,6 +6,9 @@ provider "registry.terraform.io/hashicorp/aws" { constraints = "6.64.0" hashes = [ "h1:2fTLxzUDmp/KVIHbIeLTB4bIzWHx8E6Dw+1ALLUi+Yw=", + "h1:4siTahLyzGh4BoMQcL7VXeL/mn8iR/zKv93NyhQob+0=", + "h1:EEWCXlg69fty/Qi+kehrREnVEaWdKnLlVon542b6nxQ=", + "h1:wXARLY+IeQ7ufYxCLTPCwToWGMRvOpiOTfJS97iwUzI=", "zh:07172315d67bc9781240272759cdfc7bd32b7e72384a56862c2c1da3cca99a81", "zh:154ce7d2659de9a59ddfe96d7cab41a9ddc2cb267a7d4bcdf4e737ff2ffdec06", "zh:17324d4335a7a7ac01cc23eded530775606680ff53b47cb74a3cb95d1121f836", diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md index 68df5a052..9c12a6c7b 100644 --- a/deploy/pbs-example/DEPLOYMENT_PLAN.md +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -4,23 +4,23 @@ Status: draft generated files, locally checked after validation. This directory ## Decision record -| Requirement | Value | Status | Evidence or decision owner | Blocks | -| --- | --- | --- | --- | --- | -| Purpose | Production-shaped architecture example, not a live deployment | Confirmed | User approval | None | -| Caller | Trusted Server | Confirmed | User | Caller egress ranges remain required | -| Regions | `us-east-1` and `us-west-2` | Confirmed | User | Account-specific AZ selection remains unresolved | -| Regional topology | Two AZs, one PBS EC2 host per AZ | Confirmed | User | AMIs and subnets remain required | -| Total PBS hosts | Four | Confirmed | Derived from topology | Capacity remains unmeasured | -| Runtime | EC2 with Docker Compose | Confirmed | User | Release and secret injection are deferred implementation work | -| Regional ingress | Public HTTPS ALB | Confirmed | User | ACM certificate ARNs and caller CIDRs required | -| Global routing | Route 53 latency aliases with ALB health evaluation | Proposed | Architecture decision | Existing hosted-zone ID required | -| WAF | Not included in this demo | Confirmed | User | Abuse controls remain outside this example | -| Terraform state | Local backend | Confirmed | User | No team locking or remote recovery | -| PBS release | Go v4.7.0, digest pinned | Proposed and verified | PBS release and Docker metadata checked during generation | Recheck before any future use | -| Workload | 200 global peak auctions/s, 4 bidders, 1.5 s caller timeout, 1 s PBS timeout | Proposed example assumption | Planning assumption | No capacity claim until load tested | -| Secrets | Independent regional Secrets Manager metadata and EC2 read policy; values written separately | Confirmed | User and repository CLI contract | Real bidder mapping, authorization, and runtime injection required | -| Trusted Server config | No authoritative `trusted-server.toml` exists in this checkout | Confirmed | Repository inspection | Caller behavior remains an external input | -| Deployment directory | `deploy/pbs-example/` | Confirmed | User | None | +| Requirement | Value | Status | Evidence or decision owner | Blocks | +| --------------------- | -------------------------------------------------------------------------------------------- | --------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------ | +| Purpose | Production-shaped architecture example, not a live deployment | Confirmed | User approval | None | +| Caller | Trusted Server | Confirmed | User | Caller egress ranges remain required | +| Regions | `us-east-1` and `us-west-2` | Confirmed | User | Account-specific AZ selection remains unresolved | +| Regional topology | Two AZs, one PBS EC2 host per AZ | Confirmed | User | AMIs and subnets remain required | +| Total PBS hosts | Four | Confirmed | Derived from topology | Capacity remains unmeasured | +| Runtime | EC2 with Docker Compose | Confirmed | User | Release and secret injection are deferred implementation work | +| Regional ingress | Public HTTPS ALB | Confirmed | User | ACM certificate ARNs and caller CIDRs required | +| Global routing | Route 53 latency aliases with ALB health evaluation | Proposed | Architecture decision | Existing hosted-zone ID required | +| WAF | Not included in this demo | Confirmed | User | Abuse controls remain outside this example | +| Terraform state | Local backend | Confirmed | User | No team locking or remote recovery | +| PBS release | Go v4.7.0, digest pinned | Proposed and verified | PBS release and Docker metadata checked during generation | Recheck before any future use | +| Workload | 200 global peak auctions/s, 4 bidders, 1.5 s caller timeout, 1 s PBS timeout | Proposed example assumption | Planning assumption | No capacity claim until load tested | +| Secrets | Independent regional Secrets Manager metadata and EC2 read policy; values written separately | Confirmed | User and repository CLI contract | Real bidder mapping, authorization, and runtime injection required | +| Trusted Server config | No authoritative `trusted-server.toml` exists in this checkout | Confirmed | Repository inspection | Caller behavior remains an external input | +| Deployment directory | `deploy/pbs-example/` | Confirmed | User | None | ## Architecture @@ -45,18 +45,18 @@ The EC2 AMIs are inputs rather than built by Terraform. They must contain the ap ## Services and ownership -| Service or artifact | Action | Owner | Purpose | -| --- | --- | --- | --- | -| VPC, subnets, routes, IGW, NAT, security groups | Create | Terraform | Regional network and egress | -| ALB, target group, HTTPS listener | Create | Terraform | Regional HTTPS ingress and health routing | -| Route 53 records | Reuse zone, create records | Terraform | Latency-based regional selection | -| EC2 instances and IAM profiles | Create | Terraform | Compose hosts and SSM access | -| Secrets Manager secret metadata | Create | Terraform | One independently encrypted secret per region for the example bidder | -| Secret values | External write | Authorized operator or automation | Credential lifecycle; never Terraform | -| PBS YAML and Compose definition | Git-owned | Runtime owner | Nonsecret runtime contract | -| Regional YAML merge and rendered file | Deferred runtime release owner | Deployment implementation | One resolved config per region | -| PBS process lifecycle | Deferred runtime release owner | Deployment implementation | Start, replace, health, rollback | -| Terraform state | Local operator | Terraform | Demo-only state; no shared locking | +| Service or artifact | Action | Owner | Purpose | +| ----------------------------------------------- | ------------------------------ | --------------------------------- | -------------------------------------------------------------------- | +| VPC, subnets, routes, IGW, NAT, security groups | Create | Terraform | Regional network and egress | +| ALB, target group, HTTPS listener | Create | Terraform | Regional HTTPS ingress and health routing | +| Route 53 records | Reuse zone, create records | Terraform | Latency-based regional selection | +| EC2 instances and IAM profiles | Create | Terraform | Compose hosts and SSM access | +| Secrets Manager secret metadata | Create | Terraform | One independently encrypted secret per region for the example bidder | +| Secret values | External write | Authorized operator or automation | Credential lifecycle; never Terraform | +| PBS YAML and Compose definition | Git-owned | Runtime owner | Nonsecret runtime contract | +| Regional YAML merge and rendered file | Deferred runtime release owner | Deployment implementation | One resolved config per region | +| PBS process lifecycle | Deferred runtime release owner | Deployment implementation | Start, replace, health, rollback | +| Terraform state | Local operator | Terraform | Demo-only state; no shared locking | ## Capacity assumptions @@ -85,24 +85,32 @@ A controlled load test must measure CPU, memory, connection reuse, outbound band - Customer-managed KMS keys are regional inputs. East and West accept separate ARNs; null uses each region's Secrets Manager service key. - The committed descriptor and binding files are fictional local fixtures. After an authorized apply, Terraform renders ignored operator files containing the actual instance IDs and secret ARNs. +## Deferred security and observability decisions + +ALB access logging is deliberately deferred. This example has no request-level ALB forensic record. Before production use, the platform owner must propose the log bucket, delivery permissions, encryption, access controls, and deletion lifecycle. The privacy owner must approve which request metadata may be retained, its retention period, and who may query it. The runtime owner must define incident retrieval and verify delivery. No log bucket or access-log policy is provisioned here. + +AWS API traffic, including SSM and Secrets Manager requests and any direct KMS requests, uses public service endpoints through each AZ's NAT gateway over TLS. Private hosts do not imply a private AWS API path. The platform and security owners must decide whether regional interface VPC endpoints are required, including endpoint policies, security groups, and private DNS. Compare endpoint hourly and data-processing charges against NAT charges and the required security boundary. Endpoints for AWS APIs would not replace public bidder egress; this example adds none. + +The AWS-managed `AmazonSSMManagedInstanceCore` attachment is the deliberate broad IAM exception. Its SSM agent permissions include `Resource: "*"`, unlike the inline policy scoped to declared bidder secret ARNs and the optional regional KMS key. The security owner must review the managed policy and its future updates before deployment. Do not describe the entire instance role as resource-scoped. + ## Cost drivers No price estimate is claimed. The main drivers are four EC2 instances, four NAT gateways and their Elastic IPs, two ALBs, public IPv4 addresses, cross-AZ traffic if routing changes, CloudWatch alarms, Secrets Manager, Route 53 records, and bidder internet traffic. A current estimate requires selected regions, traffic volume, and current AWS pricing verification. ## Generated files and checks -| Path | Consumer | Local check | -| --- | --- | --- | -| `README.md` | Example user | Safe walkthrough and stop boundary review | -| `terraform.tf`, `providers.tf`, `variables.tf` | Terraform | Format and validate | -| `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, ingress, and secret access | -| `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | -| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and pinned-image startup smoke | -| `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | -| `deployment.example.yaml` | Existing PBS CLI | Fictional local `ts prebid server check` only | -| Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | -| `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | -| `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | +| Path | Consumer | Local check | +| ---------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `README.md` | Example user | Safe walkthrough and stop boundary review | +| `terraform.tf`, `providers.tf`, `variables.tf` | Terraform | Format and validate | +| `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, IMDSv2, encrypted root volumes, private subnet/AZ placement, TLS policy, scoped ALB ingress/egress, and secret access | +| `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | +| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and fake-command smoke wiring; separately approved pinned-image startup smoke | +| `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | +| `deployment.example.yaml` | Existing PBS CLI | Fictional local `ts prebid server check` only | +| Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | +| `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | +| `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | ## Sources and verification diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md index 219d329a9..05baaf207 100644 --- a/deploy/pbs-example/README.md +++ b/deploy/pbs-example/README.md @@ -41,10 +41,13 @@ docker compose \ --env-file deploy/pbs-example/runtime/examples/compose.env \ -f deploy/pbs-example/runtime/compose.yaml \ config --quiet -deploy/pbs-example/scripts/smoke-runtime.sh +bash -n deploy/pbs-example/scripts/smoke-runtime.sh +deploy/pbs-example/scripts/test-smoke-runtime.py ``` -The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The smoke check pulls and starts the pinned image with dummy values, checks `/status` and startup-log redaction, and sends no auction request. +The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. + +A separately approved `deploy/pbs-example/scripts/smoke-runtime.sh` run pulls and starts the pinned image with dummy values on loopback, checks `/status` and startup-log redaction, and sends no auction request. Static rendering and fake-command checks are not runtime startup evidence. ## Adapting the example diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md index cc76d1b2c..07f7037d9 100644 --- a/deploy/pbs-example/RUNBOOK.md +++ b/deploy/pbs-example/RUNBOOK.md @@ -28,7 +28,7 @@ terraform -chdir=deploy/pbs-example/modules/regional test \ -filter=tests/security_unit_test.tftest.hcl ``` -`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs two tests. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. +`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs two tests. Regional assertions cover IMDSv2, encrypted root volumes, private subnet/AZ placement, the TLS policy, scoped ALB ingress/egress, alarms, and secret access. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. Check the PBS descriptor without AWS access: @@ -51,15 +51,24 @@ docker compose \ Expected result: Compose renders successfully without pulling or starting the image. The dummy environment file is not a credential. -Validate the JSON input and start the pinned PBS image against the nonsecret baseline: +Validate the JSON input and smoke command wiring without starting containers: ```bash python3 -m json.tool \ deploy/pbs-example/runtime/secret-bindings.example.json >/dev/null +bash -n deploy/pbs-example/scripts/smoke-runtime.sh +deploy/pbs-example/scripts/test-smoke-runtime.py +``` + +The wiring test requires Python 3 and Docker Compose. It renders production and smoke configurations, uses fake Docker lifecycle and curl commands, and checks that inherited selectors cannot replace the dummy inputs. It does not pull images, start PBS, or prove runtime health. + +### Separately approved local runtime smoke + +```bash deploy/pbs-example/scripts/smoke-runtime.sh ``` -The smoke script pulls the pinned image if needed, starts it with dummy values on local port `18080`, requires `/status` to return `ok`, rejects either dummy credential appearing in startup logs, and removes its container and network on exit. It sends no auction request and contacts no bidder. Set `PBS_SMOKE_PORT` only when port `18080` is unavailable. +The smoke script pulls the pinned image if needed, starts it with dummy values bound only to `127.0.0.1:18080`, requires `/status` to return `ok`, rejects either dummy credential appearing in startup logs, and removes its container and network on exit. It sends no auction request and contacts no bidder. Set `PBS_SMOKE_PORT` only when port `18080` is unavailable. The script forces the checked-in baseline and dummy secret file regardless of inherited `PBS_CONFIG_FILE` or `PBS_SECRET_ENV_FILE` values. Shared deployment Compose still binds all host interfaces so the ALB can reach PBS; the PBS security group restricts ingress. ## Authorized Terraform workflow @@ -111,7 +120,7 @@ Inputs: an approved profile, the exact descriptor, one complete JSON payload, a Output: a nonsecret Secrets Manager version identifier and the target region. The command does not deploy PBS, refresh Compose, or prove bidder authorization. -Failure behavior: preserve the retry UUID and input file. If the result is uncertain, retry the identical logical write rather than creating a new version. On partial regional success, record each region separately. +Failure behavior: preserve the retry UUID and input file. A failed or unverifiable response means the write is not confirmed and its outcome may be uncertain. Reuse the token only for the original identical payload; use a new token only for separately intended changed values. Withheld provider errors prevent distinguishing a rejected write from a lost response. On partial regional success, record each region separately. ## Runtime release and rollback diff --git a/deploy/pbs-example/modules/regional/security.tf b/deploy/pbs-example/modules/regional/security.tf index 40257be3a..9f7e62976 100644 --- a/deploy/pbs-example/modules/regional/security.tf +++ b/deploy/pbs-example/modules/regional/security.tf @@ -3,31 +3,31 @@ resource "aws_security_group" "alb" { description = "HTTPS ingress for the regional PBS ALB" vpc_id = aws_vpc.main.id - dynamic "ingress" { - for_each = var.trusted_server_cidr_blocks - - content { - description = "Trusted Server HTTPS" - from_port = 443 - to_port = 443 - protocol = "tcp" - cidr_blocks = [ingress.value] - } - } - - egress { - description = "Forward requests to private PBS hosts" - from_port = 0 - to_port = 0 - protocol = "-1" - cidr_blocks = ["0.0.0.0/0"] - } - tags = merge(var.tags, { Name = "${var.name}-alb" }) } +resource "aws_vpc_security_group_ingress_rule" "trusted_server_https" { + for_each = var.trusted_server_cidr_blocks + + description = "Trusted Server HTTPS" + security_group_id = aws_security_group.alb.id + cidr_ipv4 = each.value + from_port = 443 + to_port = 443 + ip_protocol = "tcp" +} + +resource "aws_vpc_security_group_egress_rule" "alb_to_pbs" { + description = "Forward requests to private PBS hosts" + security_group_id = aws_security_group.alb.id + referenced_security_group_id = aws_security_group.pbs.id + from_port = var.pbs_port + to_port = var.pbs_port + ip_protocol = "tcp" +} + resource "aws_security_group" "pbs" { name = "${var.name}-pbs" description = "PBS host traffic from the regional ALB and outbound bidder access" diff --git a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl index eae45a4e6..9087f7382 100644 --- a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl +++ b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl @@ -20,6 +20,54 @@ mock_provider "aws" { } } +override_resource { + target = aws_subnet.public["us-east-1a"] + override_during = plan + values = { + id = "subnet-00000000000000001" + } +} + +override_resource { + target = aws_subnet.public["us-east-1b"] + override_during = plan + values = { + id = "subnet-00000000000000002" + } +} + +override_resource { + target = aws_subnet.private["us-east-1a"] + override_during = plan + values = { + id = "subnet-00000000000000003" + } +} + +override_resource { + target = aws_subnet.private["us-east-1b"] + override_during = plan + values = { + id = "subnet-00000000000000004" + } +} + +override_resource { + target = aws_security_group.alb + override_during = plan + values = { + id = "sg-00000000000000001" + } +} + +override_resource { + target = aws_security_group.pbs + override_during = plan + values = { + id = "sg-00000000000000002" + } +} + variables { alarm_actions = [] ami_id = "ami-0123456789abcdef0" @@ -45,17 +93,64 @@ run "plans_private_hosts_and_scoped_ingress" { error_message = "PBS hosts should not receive public IP addresses." } + # Explicit instance paths keep Terraform 1.16.2 failure diagnostics from serializing + # the whole instance map, which contains provider-marked sensitive attributes. + assert { + condition = alltrue([ + aws_instance.pbs["us-east-1a"].metadata_options[0].http_tokens == "required", + aws_instance.pbs["us-east-1b"].metadata_options[0].http_tokens == "required", + ]) + error_message = "PBS hosts should require IMDSv2 session tokens." + } + + assert { + condition = alltrue([ + aws_instance.pbs["us-east-1a"].root_block_device[0].encrypted, + aws_instance.pbs["us-east-1b"].root_block_device[0].encrypted, + ]) + error_message = "PBS host root volumes should be encrypted." + } + + assert { + condition = alltrue([ + aws_instance.pbs["us-east-1a"].subnet_id == aws_subnet.private["us-east-1a"].id, + aws_instance.pbs["us-east-1b"].subnet_id == aws_subnet.private["us-east-1b"].id, + ]) + error_message = "PBS hosts should launch in the private subnet of their own AZ." + } + + assert { + condition = aws_lb_listener.https.ssl_policy == "ELBSecurityPolicy-TLS13-1-2-2021-06" + error_message = "The ALB listener should pin the TLS 1.3/1.2 policy." + } + + assert { + condition = ( + aws_vpc_security_group_egress_rule.alb_to_pbs.security_group_id == aws_security_group.alb.id && + aws_vpc_security_group_egress_rule.alb_to_pbs.referenced_security_group_id == aws_security_group.pbs.id && + aws_vpc_security_group_egress_rule.alb_to_pbs.ip_protocol == "tcp" && + aws_vpc_security_group_egress_rule.alb_to_pbs.from_port == var.pbs_port && + aws_vpc_security_group_egress_rule.alb_to_pbs.to_port == var.pbs_port && + aws_vpc_security_group_egress_rule.alb_to_pbs.cidr_ipv4 == null && + aws_vpc_security_group_egress_rule.alb_to_pbs.cidr_ipv6 == null && + aws_vpc_security_group_egress_rule.alb_to_pbs.prefix_list_id == null + ) + error_message = "ALB outbound traffic should reach only the PBS security group on the PBS TCP port." + } + assert { - condition = length(aws_security_group.alb.ingress) == length(var.trusted_server_cidr_blocks) + condition = length(aws_vpc_security_group_ingress_rule.trusted_server_https) == length(var.trusted_server_cidr_blocks) error_message = "The ALB should have one ingress rule for each approved caller CIDR." } assert { condition = alltrue([ - for rule in aws_security_group.alb.ingress : - length(rule.cidr_blocks) == 1 && contains(var.trusted_server_cidr_blocks, one(rule.cidr_blocks)) + for cidr, rule in aws_vpc_security_group_ingress_rule.trusted_server_https : + rule.security_group_id == aws_security_group.alb.id && + rule.cidr_ipv4 == cidr && contains(var.trusted_server_cidr_blocks, cidr) && + rule.ip_protocol == "tcp" && rule.from_port == 443 && rule.to_port == 443 ]) - error_message = "Every ALB ingress rule should use an approved caller CIDR." + error_message = "Every ALB ingress rule should permit only HTTPS from its approved caller CIDR." } assert { diff --git a/deploy/pbs-example/runtime/README.md b/deploy/pbs-example/runtime/README.md index 5113030b0..fe0527cc0 100644 --- a/deploy/pbs-example/runtime/README.md +++ b/deploy/pbs-example/runtime/README.md @@ -5,3 +5,5 @@ This directory owns nonsecret PBS configuration and the Compose shape. The image The `/run/pbs/secrets/examplebidder.env` file is a runtime contract, not a checked-in credential file. A separately approved host-side loader must retrieve the selected regional Secrets Manager version, validate the complete JSON payload, render this restricted environment file atomically, and start or replace Compose. The current `ts prebid server` CLI does not implement that release or injection workflow. `secret-bindings.example.json` and `examples/pbs-secrets.env` contain fictional values for local checks only. An authorized Terraform apply can render the ignored `secret-bindings.generated.json` with actual regional secret ARNs, but never secret values. + +Deployment Compose binds `0.0.0.0:8000` by default so the ALB can reach PBS through the host security group. `scripts/smoke-runtime.sh` forces `PBS_BIND_ADDRESS=127.0.0.1` and checked-in dummy inputs for local startup. `scripts/test-smoke-runtime.py` verifies both bindings and the smoke selectors with Compose rendering and fake commands only. Run both scripts from the parent example directory or use their repository-relative paths. diff --git a/deploy/pbs-example/runtime/compose.yaml b/deploy/pbs-example/runtime/compose.yaml index 4db566901..2269a765a 100644 --- a/deploy/pbs-example/runtime/compose.yaml +++ b/deploy/pbs-example/runtime/compose.yaml @@ -3,7 +3,7 @@ services: image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 restart: unless-stopped ports: - - "${PBS_HOST_PORT:-8000}:8000" + - "${PBS_BIND_ADDRESS:-0.0.0.0}:${PBS_HOST_PORT:-8000}:8000" env_file: - ${PBS_SECRET_ENV_FILE:-/run/pbs/secrets/examplebidder.env} volumes: diff --git a/deploy/pbs-example/runtime/examples/README.md b/deploy/pbs-example/runtime/examples/README.md index fa0017b87..a8e9ca3e4 100644 --- a/deploy/pbs-example/runtime/examples/README.md +++ b/deploy/pbs-example/runtime/examples/README.md @@ -1 +1 @@ -These files contain fictional values for local structure checks only. They are not bidder credentials and must never be copied into an AWS secret. \ No newline at end of file +These files contain fictional values for local structure checks only. They are not bidder credentials and must never be copied into an AWS secret. diff --git a/deploy/pbs-example/scripts/smoke-runtime.sh b/deploy/pbs-example/scripts/smoke-runtime.sh index a765773ca..7a5d89a9b 100755 --- a/deploy/pbs-example/scripts/smoke-runtime.sh +++ b/deploy/pbs-example/scripts/smoke-runtime.sh @@ -16,7 +16,11 @@ cleanup() { } trap cleanup EXIT +# Smoke always uses loopback and checked-in dummy inputs, never inherited selectors. +export PBS_BIND_ADDRESS="127.0.0.1" export PBS_HOST_PORT="$host_port" +export PBS_CONFIG_FILE="$example_dir/runtime/pbs.yaml" +export PBS_SECRET_ENV_FILE="$example_dir/runtime/examples/pbs-secrets.env" "${compose[@]}" up --detach for _ in $(seq 1 30); do diff --git a/deploy/pbs-example/scripts/test-smoke-runtime.py b/deploy/pbs-example/scripts/test-smoke-runtime.py new file mode 100755 index 000000000..f49078687 --- /dev/null +++ b/deploy/pbs-example/scripts/test-smoke-runtime.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +"""Render Compose and exercise smoke command wiring without starting containers.""" + +import json +import os +from pathlib import Path +import shutil +import subprocess +import tempfile + + +EXAMPLE = Path(__file__).resolve().parent.parent +DOCKER = shutil.which("docker") +assert DOCKER, "docker with the Compose plugin must be on PATH" +ENV = {key: os.environ[key] for key in ("PATH", "HOME") if key in os.environ} + + +def render_production(): + result = subprocess.run( + [DOCKER, "compose", "-f", str(EXAMPLE / "runtime/compose.yaml"), + "config", "--format", "json", "--no-env-resolution"], + env=ENV, check=True, capture_output=True, text=True, + ) + service = json.loads(result.stdout)["services"]["pbs"] + port = service["ports"][0] + assert port.get("host_ip", "0.0.0.0") == "0.0.0.0", port + assert port["published"] == "8000" and port["target"] == 8000, port + assert service["env_file"][0]["path"] == "/run/pbs/secrets/examplebidder.env" + + +def check_smoke(): + with tempfile.TemporaryDirectory(prefix="pbs-smoke-wiring-") as scratch: + directory = Path(scratch) + docker = directory / "docker" + docker.write_text('''#!/usr/bin/env python3 +import json, os, pathlib, subprocess, sys +args = sys.argv[1:] +assert args[0] == "compose", args +operation = args[7] +assert operation in ("up", "logs", "down"), args +with open(os.environ["CALLS"], "a") as log: + log.write(operation + "\\n") +if operation == "up": + result = subprocess.run( + [os.environ["REAL_DOCKER"], *args[:7], "config", "--format", "json"], + capture_output=True, text=True, + ) + if result.returncode: + sys.stderr.write(result.stderr) + sys.exit(result.returncode) + pathlib.Path(os.environ["RENDERED"]).write_text(result.stdout) +''') + curl = directory / "curl" + curl.write_text('''#!/usr/bin/env python3 +import sys +assert sys.argv[-1] == "http://127.0.0.1:18081/status", sys.argv +print("ok") +''') + docker.chmod(0o755) + curl.chmod(0o755) + env = { + **ENV, + "PATH": f"{directory}:{ENV['PATH']}", + "REAL_DOCKER": DOCKER, + "CALLS": str(directory / "calls"), + "RENDERED": str(directory / "rendered.json"), + "PBS_SMOKE_PORT": "18081", + "PBS_HOST_PORT": "19000", + "PBS_BIND_ADDRESS": "0.0.0.0", + "PBS_CONFIG_FILE": "/nonexistent/inherited-pbs.yaml", + "PBS_SECRET_ENV_FILE": "/nonexistent/inherited-secrets.env", + } + subprocess.run( + [str(EXAMPLE / "scripts/smoke-runtime.sh")], + env=env, check=True, capture_output=True, text=True, + ) + assert (directory / "calls").read_text().splitlines() == ["up", "logs", "down"] + service = json.loads((directory / "rendered.json").read_text())["services"]["pbs"] + port = service["ports"][0] + assert port.get("host_ip") == "127.0.0.1", port + assert port["published"] == "18081" and port["target"] == 8000, port + assert service["volumes"][0]["source"] == str(EXAMPLE / "runtime/pbs.yaml") + assert service["environment"]["PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY"] == "example-only-api-key" + assert service["environment"]["PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN"] == "example-only-optional-token" + + +if __name__ == "__main__": + render_production() + check_smoke() + print("Compose production/smoke bindings and dummy selectors passed; no containers started.") From f969cf9931ad62515021daffa45689180bfb86b1 Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 24 Sep 2026 20:03:05 -0500 Subject: [PATCH 11/14] Address PBS review follow-ups and add example CI --- .github/workflows/pbs-example.yml | 68 ++++++++++++ .tool-versions | 12 +- CHANGELOG.md | 1 + crates/trusted-server-cli/README.md | 8 +- .../src/commands/pbs/inspect.rs | 12 +- crates/trusted-server-cli/tests/pbs_cli.rs | 88 +++++++++++++++ deploy/pbs-example/.gitignore | 1 - deploy/pbs-example/DEPLOYMENT_PLAN.md | 6 + deploy/pbs-example/README.md | 9 +- deploy/pbs-example/RUNBOOK.md | 20 ++-- .../modules/regional/.terraform.lock.hcl | 29 +++++ .../pbs-example/modules/regional/compute.tf | 4 + .../modules/regional/load_balancing.tf | 8 +- deploy/pbs-example/modules/regional/locals.tf | 4 + .../pbs-example/modules/regional/secrets.tf | 4 +- .../pbs-example/modules/regional/security.tf | 8 +- .../tests/security_unit_test.tftest.hcl | 92 +++++++++++++++- .../pbs-example/modules/regional/variables.tf | 12 +- .../pbs-example/scripts/test-smoke-runtime.py | 6 +- .../tests/root_unit_test.tftest.hcl | 51 ++++++--- docs/guide/cli.md | 103 +++++++++++++++++- 21 files changed, 481 insertions(+), 65 deletions(-) create mode 100644 .github/workflows/pbs-example.yml create mode 100644 deploy/pbs-example/modules/regional/.terraform.lock.hcl diff --git a/.github/workflows/pbs-example.yml b/.github/workflows/pbs-example.yml new file mode 100644 index 000000000..fbd5dc3a2 --- /dev/null +++ b/.github/workflows/pbs-example.yml @@ -0,0 +1,68 @@ +name: PBS example checks + +on: + push: + branches: [main] + paths: + - 'deploy/pbs-example/**' + - '.tool-versions' + - '.github/workflows/pbs-example.yml' + pull_request: + paths: + - 'deploy/pbs-example/**' + - '.tool-versions' + - '.github/workflows/pbs-example.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + local-checks: + name: PBS Terraform and Compose checks + runs-on: ubuntu-latest + timeout-minutes: 20 + env: + TF_IN_AUTOMATION: 'true' + AWS_EC2_METADATA_DISABLED: 'true' + steps: + - uses: actions/checkout@v4 + + - name: Read pinned Terraform version + id: terraform + shell: bash + run: echo "version=$(awk '$1 == "terraform" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: ${{ steps.terraform.outputs.version }} + terraform_wrapper: false + + - uses: actions/setup-python@v5 + with: + python-version: '3.13' + + - name: Check Docker Compose prerequisite + run: docker compose version + + # These suites use mocked providers and plan mode. No AWS credentials, + # Terraform apply, image pulls, or container startup are needed. + - name: Format and test Terraform root + run: | + terraform fmt -check -recursive deploy/pbs-example + terraform -chdir=deploy/pbs-example init -backend=false -input=false -lockfile=readonly + terraform -chdir=deploy/pbs-example validate + terraform -chdir=deploy/pbs-example test -filter=tests/root_unit_test.tftest.hcl + + - name: Validate and test regional module + run: | + terraform -chdir=deploy/pbs-example/modules/regional init -backend=false -input=false -lockfile=readonly + terraform -chdir=deploy/pbs-example/modules/regional validate + terraform -chdir=deploy/pbs-example/modules/regional test -filter=tests/security_unit_test.tftest.hcl + git diff --exit-code -- deploy/pbs-example/.terraform.lock.hcl deploy/pbs-example/modules/regional/.terraform.lock.hcl + + - name: Check runtime inputs and smoke wiring without starting containers + run: | + python3 -m json.tool deploy/pbs-example/runtime/secret-bindings.example.json > /dev/null + bash -n deploy/pbs-example/scripts/smoke-runtime.sh + python3 -I deploy/pbs-example/scripts/test-smoke-runtime.py diff --git a/.tool-versions b/.tool-versions index 6a44ca668..1b29972c2 100644 --- a/.tool-versions +++ b/.tool-versions @@ -1,7 +1,7 @@ -fastly 15.1.0 -rust 1.95.0 -nodejs 24.12.0 -viceroy 0.17.0 -wasmtime 44.0.1 -awscli 2.36.45 +fastly 15.1.0 +rust 1.95.0 +nodejs 24.12.0 +viceroy 0.17.0 +wasmtime 44.0.1 +awscli 2.36.45 terraform 1.16.2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 083d6e746..9e46fdf85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. - **Breaking:** Auction providers and bidder routes now use the configuration-first `[auction.providers.]` and `[auction.bidders.]` maps. The removed `[auction].providers = [...]` list and removed server fields under `[integrations.prebid]` and `[integrations.aps]` are rejected even when those integrations are disabled, and `ts config push` rejects the old shape before publication. Move PBS `server_url` to provider `endpoint`, server timeout to provider `timeout_ms`, request controls and bidder-parameter overrides to the `prebid-server` `profile_config`, notification suppression to `notifications`, and each former server bidder to an `[auction.bidders.]` route. Move APS endpoint, timeout, account, inventory, debug, and creative controls to an `aps` provider and its `profile_config`. Browser Prebid settings remain under `[integrations.prebid]`; values such as timeout and debug that previously affected both browser and server behavior must now be configured for each owner. Provider endpoints must be absolute HTTPS URLs. Only bidder codes present in `[auction.bidders]` are folded into Trusted Server requests; unlisted publisher bids remain native browser demand. Provider response names now use the configured provider ID, such as `pbs-main`, instead of the legacy literal `prebid`; audit consumers that match `AuctionResponse.provider`. This schema has no mixed-version-safe deployment order: old binaries reject the maps and new binaries reject the retired fields, so activate the new binary and config blob together. Rollbacks must restore an old-schema blob together with the old binary. - **Breaking** — Admin Basic-auth coverage now includes `GET /_ts/admin/ec`, `GET /_ts/admin/ec/{id}`, and `GET /_ts/admin/eids`. Existing configurations whose `[[handlers]]` patterns protect only the key-management endpoints now fail startup; broaden coverage before deploying, preferably with a namespace-boundary pattern such as `^/_ts/admin(?:/|$)`. Coverage of the dynamic `/_ts/admin/ec/{id}` route is no longer inferred from ID-shaped samples: the router accepts any segment after `/_ts/admin/ec/` and Basic Auth runs on the raw path before routing, so patterns anchored to the EC ID grammar (for example `^/_ts/admin/ec/[a-f0-9]{64}[.][A-Za-z0-9]{6}$`) are rejected in favor of a prefix-level matcher. Placeholder and well-known weak handler passwords (`changeme`, `password`, `admin`, `replace-with-…`) now fail startup on every handler rather than only on handlers inferred to cover an admin endpoint, because first-match-wins handler selection lets a narrow handler shadow the admin namespace. - Prebid Server provider endpoints now normalize origin-only legacy `server_url` values to `/openrtb2/auction`. Query parameters are preserved, the canonical path loses a trailing slash, and configured non-root custom paths remain exact. diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 91d128829..3e97436a4 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -15,7 +15,7 @@ cargo run_cli_linux prebid server inspect --config trusted-server.example.toml - cargo run_cli_linux prebid server check --deployment crates/trusted-server-cli/examples/pbs/deployment.yaml ``` -On macOS use `build_cli_macos` and `run_cli_macos`. The examples contain fictional resource identifiers, a fictional image digest, and a fictional adapter binding. They exercise local checks only and must not be used as real deployment settings. +On Apple Silicon macOS use `build_cli_macos` and `run_cli_macos`. Other hosts can run `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" -- prebid server --help`. The examples contain fictional resource identifiers, a fictional image digest, and a fictional adapter binding. They exercise local checks only and must not be used as real deployment settings. ## Commands and current limits @@ -86,7 +86,7 @@ The command above is an interface example, not authorization to run it. Generate File and stdin inputs are mutually exclusive. `--stdin` requires `--yes` and `--request-token`. Input must contain only declared keys, include all required nonempty string values, and fit the Secrets Manager size limit. Optional keys may be omitted. Duplicate keys and non-string values are rejected. Quotes, dollar signs, newlines, and backslashes are encoded as JSON, never shell expressions. -Secret values never enter command arguments or reports. The AWS CLI receives JSON through a tool-owned temporary file, owner-only on Unix, which is removed on normal success and error paths. Operator-provided input files are not changed or deleted. Run on a trusted host with protected temporary storage; abrupt process termination can leave temporary files requiring cleanup. Windows temporary-file ACL behavior has not been validated. +Secret values never enter command arguments or reports. The AWS CLI receives JSON through a tool-owned temporary file, owner-only on Unix, which is removed on normal success and error paths. Operator-provided input files are not changed or deleted. Input-file errors print the full escaped path to stderr, including under `--json`; directory layouts and partner names in paths can enter logs even though credential contents are withheld. Run on a trusted host with protected temporary storage; abrupt process termination can leave temporary files requiring cleanup. Windows temporary-file ACL behavior has not been validated. A successful write reports its version identifier. It does not create secret metadata, change infrastructure, replace containers, or rotate the bidder's credential. Check regional replication, separately replace consumers, and verify them before revoking old partner credentials. A failed or unverifiable response means the write is not confirmed; the outcome may be uncertain. Retain the displayed request token and reuse it only for the original identical payload. Use a new token only for separately intended changed values. Provider error details are withheld, so the CLI cannot distinguish a rejected write from a lost response. @@ -104,4 +104,6 @@ cargo fmt --all -- --check cargo clippy --package trusted-server-cli --all-targets --target x86_64-unknown-linux-gnu -- -D warnings ``` -Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require Python 3. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. +Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require Python 3. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. The unset-history case covers exit 1 with empty stdout, but this fake response does not establish the real AWS CLI contract. An authorized runtime owner must verify real secret-write version IDs and retry behavior against a throwaway secret before operational use. + +Process tests also send deeply nested flow sequences, flow mappings, and block mappings through `check`. The pinned `serde_yaml_ng` parser rejects them with sanitized errors before the recursive configuration walkers run. These tests guard the parser's depth-limit behavior; they are not proof against every possible YAML resource-exhaustion input. diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs index da009fad1..71bd34c78 100644 --- a/crates/trusted-server-cli/src/commands/pbs/inspect.rs +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -96,8 +96,9 @@ struct Bundle { user_id_modules: Vec, } -/// Mirror the private core list deserializer without expanding runtime defaults. -/// Parity tests compare these accepted representations with `PrebidIntegrationConfig`. +/// Accept the same encodings as the private core list deserializer without expanding defaults. +/// Unparseable escapes instead reach identifier validation for a sanitized error. +/// Parity tests compare accepted representations with `PrebidIntegrationConfig`. /// /// # Errors /// Rejects malformed lists, invalid numeric indexes, and non-string items. @@ -144,10 +145,9 @@ fn bidder_list<'de, D: Deserializer<'de>>( .into_iter() .map(|part| { let json = format!("\"{}\"", part.replace('"', "\\\"")); - match serde_json::from_str(&json) { - Ok(value) => value, - Err(_) => part.to_owned(), - } + // Core propagates this error; keep raw text here so identifier validation + // rejects it without a serde snippet that could echo source contents. + serde_json::from_str(&json).unwrap_or_else(|_| part.to_owned()) }) .collect()) } diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index 25b6ca1e8..c0edd1d0d 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -32,6 +32,8 @@ with (root / 'calls').open('a') as log: log.write(json.dumps(args) + '\n') assert '--profile' in args and args[args.index('--profile')+1] == 'pbs-sandbox' if 'configure' in args: + if os.environ.get('PBS_FAKE_HISTORY') == 'unset': + sys.exit(1) if os.environ.get('PBS_FAKE_HISTORY') == 'enabled': print('enabled') else: @@ -176,6 +178,49 @@ fn local_commands_never_execute_aws_and_preserve_the_source() { ); } +#[test] +fn deeply_nested_yaml_is_rejected_without_aborting_or_echoing_input() { + let flow_sequence = format!( + "nested: {}NEVER_PRINT_ME{}", + "[".repeat(1000), + "]".repeat(1000) + ); + let flow_mapping = format!( + "{}\"NEVER_PRINT_ME\"{}", + "{\"nested\":".repeat(1000), + "}".repeat(1000) + ); + let mut block_mapping = (0..256) + .map(|depth| format!("{}nested:\n", " ".repeat(depth))) + .collect::(); + block_mapping.push_str(&format!("{}value: NEVER_PRINT_ME\n", " ".repeat(256))); + for yaml in [flow_sequence, flow_mapping, block_mapping] { + let dir = fixture(); + fs::write(dir.path().join("pbs.yaml"), yaml).expect("should write nested input"); + let output = command(dir.path()) + .args([ + "prebid", + "server", + "check", + "--deployment", + "deployment.yaml", + "--json", + ]) + .output() + .expect("should run CLI"); + assert_eq!( + output.status.code(), + Some(2), + "should reject input without a signal" + ); + assert!(output.stdout.is_empty()); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("cannot parse YAML/JSON input; source details withheld")); + assert!(!stderr.contains("NEVER_PRINT_ME")); + assert!(!dir.path().join("calls").exists(), "should not call AWS"); + } +} + #[test] fn missing_descriptor_inputs_report_escaped_paths() { for missing in ["deployment.yaml", "pbs.yaml", "bindings.json", "east.yaml"] { @@ -288,6 +333,49 @@ fn secret_payload_is_private_not_in_argv_and_deleted_after_use() { assert_payload_cleanup(dir.path()); } +#[test] +fn unset_cli_history_allows_write_after_both_history_probes() { + let dir = fixture(); + fs::write( + dir.path().join("secret.json"), + "{\"api_key\":\"DUMMY_SECRET\"}", + ) + .expect("should write dummy credential"); + let output = secret_command(dir.path()) + .env("PBS_FAKE_HISTORY", "unset") + .output() + .expect("should run CLI"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert_no_secret(&output); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + let calls = calls + .lines() + .map(|line| serde_json::from_str::>(line).expect("should parse arguments")) + .collect::>(); + let probes = calls + .iter() + .filter(|args| args.iter().any(|arg| arg == "configure")) + .map(|args| { + args.last() + .expect("should have a configuration key") + .as_str() + }) + .collect::>(); + assert_eq!(probes, ["cli_history", "default.cli_history"]); + assert_eq!( + calls + .iter() + .filter(|args| args.iter().any(|arg| arg == "put-secret-value")) + .count(), + 1 + ); + assert_payload_cleanup(dir.path()); +} + #[test] fn aws_errors_never_forward_provider_stderr_and_cleanup_payloads() { for failure in ["collision", "transport", "invalid-json", "unverified"] { diff --git a/deploy/pbs-example/.gitignore b/deploy/pbs-example/.gitignore index d73176818..bf80e0040 100644 --- a/deploy/pbs-example/.gitignore +++ b/deploy/pbs-example/.gitignore @@ -1,6 +1,5 @@ # Terraform working data and local state .terraform/ -/modules/**/.terraform.lock.hcl *.tfstate *.tfstate.* .terraform.tfstate.lock.info diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md index 9c12a6c7b..9390d431a 100644 --- a/deploy/pbs-example/DEPLOYMENT_PLAN.md +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -83,6 +83,7 @@ A controlled load test must measure CPU, memory, connection reuse, outbound band - The example uses a fictional bidder binding. The adapter name, credential keys, source authorization, and PBS mapping must be replaced and verified against v4.7.0 before use. - A Secrets Manager write does not refresh a running Compose container. The deferred runtime implementation must retrieve a selected version, render an environment file atomically, replace the consumer, and verify health before retiring the old version. - Customer-managed KMS keys are regional inputs. East and West accept separate ARNs; null uses each region's Secrets Manager service key. +- The regional module fixes the PBS host port at `8000`, matching `runtime/compose.yaml` and `runtime/pbs.yaml`. A port change requires coordinated Terraform and runtime edits; it is not a deployment input. - The committed descriptor and binding files are fictional local fixtures. After an authorized apply, Terraform renders ignored operator files containing the actual instance IDs and secret ARNs. ## Deferred security and observability decisions @@ -99,6 +100,8 @@ No price estimate is claimed. The main drivers are four EC2 instances, four NAT ## Generated files and checks +The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI requires Python 3.13 and Docker Compose, but no AWS credentials or container startup. The example maintainer owns these checks. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. + | Path | Consumer | Local check | | ---------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `README.md` | Example user | Safe walkthrough and stop boundary review | @@ -106,6 +109,9 @@ No price estimate is claimed. The main drivers are four EC2 instances, four NAT | `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, IMDSv2, encrypted root volumes, private subnet/AZ placement, TLS policy, scoped ALB ingress/egress, and secret access | | `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | | `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and fake-command smoke wiring; separately approved pinned-image startup smoke | +| `scripts/smoke-runtime.sh` | Runtime owner | CI shell syntax and fake-command wiring; real startup smoke requires separate authorization | +| `scripts/test-smoke-runtime.py` | Example maintainer | CI Python 3.13 and Docker Compose rendering; no image pull or containers | +| Root and regional `.terraform.lock.hcl` | Example maintainer | Readonly initialization and tests; regenerate both locks for the four supported platforms on provider upgrades | | `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | | `deployment.example.yaml` | Existing PBS CLI | Fictional local `ts prebid server check` only | | Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md index 05baaf207..c7c136cde 100644 --- a/deploy/pbs-example/README.md +++ b/deploy/pbs-example/README.md @@ -23,14 +23,15 @@ Each region has two AZs, one EC2 host per AZ, a public ALB, and one NAT gateway ## Safe local walkthrough -These commands do not contact AWS: +These commands do not contact AWS. They require the pinned Terraform and Rust toolchains, Python 3, and Docker with the Compose plugin. On Apple Silicon macOS, replace `cargo run_cli_linux` with `cargo run_cli_macos`. Other hosts can use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`. ```bash terraform fmt -check -recursive deploy/pbs-example -terraform -chdir=deploy/pbs-example init -backend=false -input=false +terraform -chdir=deploy/pbs-example init -backend=false -input=false -lockfile=readonly terraform -chdir=deploy/pbs-example validate terraform -chdir=deploy/pbs-example test -filter=tests/root_unit_test.tftest.hcl -terraform -chdir=deploy/pbs-example/modules/regional init -backend=false -input=false +terraform -chdir=deploy/pbs-example/modules/regional init -backend=false -input=false -lockfile=readonly +terraform -chdir=deploy/pbs-example/modules/regional validate terraform -chdir=deploy/pbs-example/modules/regional test -filter=tests/security_unit_test.tftest.hcl cargo run_cli_linux prebid server check \ --deployment deploy/pbs-example/deployment.example.yaml \ @@ -45,6 +46,8 @@ bash -n deploy/pbs-example/scripts/smoke-runtime.sh deploy/pbs-example/scripts/test-smoke-runtime.py ``` +The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI sets up Python 3.13 and checks Docker Compose availability. Run the wiring test without Python optimization, which disables assertions. The CLI descriptor command above is a separate local check; the example owner must rerun it when changing or adapting inputs. + The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. A separately approved `deploy/pbs-example/scripts/smoke-runtime.sh` run pulls and starts the pinned image with dummy values on loopback, checks `/status` and startup-log redaction, and sends no auction request. Static rendering and fake-command checks are not runtime startup evidence. diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md index 07f7037d9..99f229c8d 100644 --- a/deploy/pbs-example/RUNBOOK.md +++ b/deploy/pbs-example/RUNBOOK.md @@ -5,32 +5,36 @@ This runbook describes local checks and the separately authorized operations tha ## Preconditions - Work only in `deploy/pbs-example`. -- Use Terraform `1.16.2` and the committed AWS provider lock file. +- Use Terraform `1.16.2` and the committed AWS provider lock files in both the root and regional module. Both lock AWS `6.64.0` for Linux and macOS on AMD64 and ARM64. +- Local wiring checks require Python 3 and Docker with the Compose plugin. CI uses Python 3.13. Do not enable Python optimization; the wiring test refuses to run without assertions. - Use fictional values from `terraform.tfvars.example`, `deployment.example.yaml`, and `runtime/secret-bindings.example.json` only for local validation. - Replace the fictional AWS account, profile, certificate, hosted-zone, AMI, and CIDR inputs before any authorized cloud plan. Terraform renders actual instance and secret identifiers after apply. - Confirm the approved Trusted Server egress CIDRs. Do not use the documentation CIDR as a real allowlist. - Confirm the PBS v4.7.0 image digest and every adapter binding again before release preparation. -- Keep state, saved plans, credentials, and AWS provider output out of Git and public logs. +- Keep state, saved plans, credentials, and AWS provider output out of Git and public logs. Input-file errors echo the full escaped path to stderr, including under `--json`; directory layouts and partner names in paths can therefore enter logs even though credential contents are withheld. ## Local validation From the repository root: ```bash -terraform fmt -recursive deploy/pbs-example -terraform -chdir=deploy/pbs-example init -backend=false -input=false +terraform fmt -check -recursive deploy/pbs-example +terraform -chdir=deploy/pbs-example init -backend=false -input=false -lockfile=readonly terraform -chdir=deploy/pbs-example validate terraform -chdir=deploy/pbs-example test \ -filter=tests/root_unit_test.tftest.hcl terraform -chdir=deploy/pbs-example/modules/regional init \ - -backend=false -input=false + -backend=false -input=false -lockfile=readonly +terraform -chdir=deploy/pbs-example/modules/regional validate terraform -chdir=deploy/pbs-example/modules/regional test \ -filter=tests/security_unit_test.tftest.hcl ``` -`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs two tests. Regional assertions cover IMDSv2, encrypted root volumes, private subnet/AZ placement, the TLS policy, scoped ALB ingress/egress, alarms, and secret access. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. +`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs eight tests. Root assertions require distinct instance IDs in each regional descriptor. Module tests also cover resource-name boundaries and partition-specific SSM policy ARNs. Regional assertions cover IMDSv2, encrypted root volumes, private subnet/AZ placement, the TLS policy, scoped ALB ingress/egress, alarms, and secret access. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. -Check the PBS descriptor without AWS access: +The `PBS example checks` workflow runs these Terraform checks and the JSON, shell, and Compose wiring checks below. CLI descriptor validation remains a local responsibility of the example owner. Provider upgrades must refresh both locks with `terraform providers lock -platform=darwin_arm64 -platform=darwin_amd64 -platform=linux_amd64 -platform=linux_arm64` from each directory and rerun both suites. + +Check the PBS descriptor without AWS access. On Apple Silicon macOS, use `cargo run_cli_macos` instead of `cargo run_cli_linux`. On another host, use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`: ```bash cargo run_cli_linux prebid server check \ @@ -139,4 +143,6 @@ Do not manually edit generated runtime environment files. A secret update alone ## Deferred evidence +Before relying on secret writes, the runtime owner must arrange an authorized sandbox check of the real AWS CLI and Secrets Manager contract. Use a throwaway secret and dummy values to confirm that `VersionId` equals `ClientRequestToken`, an identical retry is idempotent, and changed values with the same token are rejected. The pinned AWS CLI `2.36.45` was checked with isolated local configuration files: `aws configure get cli_history` and `default.cli_history` both returned exit 1 with empty stdout when unset, without API calls. Repeat that local check on CLI upgrades. Process tests also cover this shape with a fake AWS executable. Neither check establishes real Secrets Manager behavior; no live verification is implied by the local checks. + An authorized operator still needs to verify TLS and DNS, IAM permissions, public ingress, NAT and bidder egress, real adapter credentials, AMI contents, reboot and host replacement, interrupted runtime replacement, secret rotation, representative load, regional failover, alarm delivery, and the Trusted Server auction path. diff --git a/deploy/pbs-example/modules/regional/.terraform.lock.hcl b/deploy/pbs-example/modules/regional/.terraform.lock.hcl new file mode 100644 index 000000000..4f85daae2 --- /dev/null +++ b/deploy/pbs-example/modules/regional/.terraform.lock.hcl @@ -0,0 +1,29 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "6.64.0" + constraints = "6.64.0" + hashes = [ + "h1:2fTLxzUDmp/KVIHbIeLTB4bIzWHx8E6Dw+1ALLUi+Yw=", + "h1:4siTahLyzGh4BoMQcL7VXeL/mn8iR/zKv93NyhQob+0=", + "h1:EEWCXlg69fty/Qi+kehrREnVEaWdKnLlVon542b6nxQ=", + "h1:wXARLY+IeQ7ufYxCLTPCwToWGMRvOpiOTfJS97iwUzI=", + "zh:07172315d67bc9781240272759cdfc7bd32b7e72384a56862c2c1da3cca99a81", + "zh:154ce7d2659de9a59ddfe96d7cab41a9ddc2cb267a7d4bcdf4e737ff2ffdec06", + "zh:17324d4335a7a7ac01cc23eded530775606680ff53b47cb74a3cb95d1121f836", + "zh:307ab92324ec5a61b124881ab8cac1d9e316f4527dfd0e1b59794c229407eb4e", + "zh:31e25f1903661332e36a95283042dd3ec50b47c186db00663fbd976a11e6a6b2", + "zh:3311d9f3bd12a24886027dbe73859dcd1e67bd0e3046227a338cf2c7ca04d18e", + "zh:37916156a3aac3b29be3acebd15d53145ea4ab5d4aaa825eaebe75481fa00500", + "zh:4158cb8c38b3ac6aa98eb15935ec6bd7c30838d85d2b00acc9812df8382ae908", + "zh:5bfb9499c66d9db5b34dc5c60f426a1ab1baa5457ce2aefebca826a9c3f92fb0", + "zh:6eb29ead5a4aca3b1f35812e7e8c75419180e1928e479b458f206861277736db", + "zh:7a82b6dd0c0cdef8045a4adfbddd36acb86b6b23fcbed8e189c2d71f7dc4a502", + "zh:9556bd792032c3f7e73ea4dd08cec88dc1327f5a4a57d79c30ba844ae2b9a3c0", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:c5234180464cb800c83a41f57462742b802c150ad7d4417626fcd9cb511c01d2", + "zh:cd776b83b1f7b36635957350afe7ce28ba4e4ea3a5e2deb00d13dbd3b35d9d40", + "zh:fb583a7b791c6f915b86573d04f05ddbf7f1a5e4120c5d8a7450a3086c1225c4", + ] +} diff --git a/deploy/pbs-example/modules/regional/compute.tf b/deploy/pbs-example/modules/regional/compute.tf index 9b9e2b076..d6772b89d 100644 --- a/deploy/pbs-example/modules/regional/compute.tf +++ b/deploy/pbs-example/modules/regional/compute.tf @@ -1,3 +1,7 @@ +# No user_data by design. Host bootstrap, PBS YAML rendering, secret-file +# materialization, and Compose startup require a separately approved runtime +# loader. Applied hosts stay ALB-unhealthy until it runs. See ../../README.md, +# "Stop boundary". resource "aws_instance" "pbs" { for_each = local.availability_zone_index diff --git a/deploy/pbs-example/modules/regional/load_balancing.tf b/deploy/pbs-example/modules/regional/load_balancing.tf index 20cf58d64..481b1998d 100644 --- a/deploy/pbs-example/modules/regional/load_balancing.tf +++ b/deploy/pbs-example/modules/regional/load_balancing.tf @@ -1,5 +1,5 @@ resource "aws_lb" "main" { - name = substr("${var.name}-alb", 0, 32) + name = "${var.name}-alb" internal = false load_balancer_type = "application" security_groups = [aws_security_group.alb.id] @@ -13,8 +13,8 @@ resource "aws_lb" "main" { } resource "aws_lb_target_group" "pbs" { - name = substr("${var.name}-pbs", 0, 32) - port = var.pbs_port + name = "${var.name}-pbs" + port = local.pbs_port protocol = "HTTP" target_type = "instance" vpc_id = aws_vpc.main.id @@ -42,7 +42,7 @@ resource "aws_lb_target_group_attachment" "pbs" { target_group_arn = aws_lb_target_group.pbs.arn target_id = each.value.id - port = var.pbs_port + port = local.pbs_port } resource "aws_lb_listener" "https" { diff --git a/deploy/pbs-example/modules/regional/locals.tf b/deploy/pbs-example/modules/regional/locals.tf index 92722f50b..57c182868 100644 --- a/deploy/pbs-example/modules/regional/locals.tf +++ b/deploy/pbs-example/modules/regional/locals.tf @@ -1,4 +1,8 @@ locals { + # Coupled to runtime/compose.yaml's host/container ports and runtime/pbs.yaml's + # port. Changing this requires matching runtime edits and wiring checks. + pbs_port = 8000 + availability_zone_index = { for index, availability_zone in var.availability_zones : availability_zone => index } diff --git a/deploy/pbs-example/modules/regional/secrets.tf b/deploy/pbs-example/modules/regional/secrets.tf index 0296045ba..360b41709 100644 --- a/deploy/pbs-example/modules/regional/secrets.tf +++ b/deploy/pbs-example/modules/regional/secrets.tf @@ -39,9 +39,11 @@ resource "aws_iam_instance_profile" "pbs" { }) } +data "aws_partition" "current" {} + resource "aws_iam_role_policy_attachment" "ssm" { role = aws_iam_role.pbs.name - policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" + policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" } resource "aws_iam_role_policy" "runtime" { diff --git a/deploy/pbs-example/modules/regional/security.tf b/deploy/pbs-example/modules/regional/security.tf index 9f7e62976..9e85e62d0 100644 --- a/deploy/pbs-example/modules/regional/security.tf +++ b/deploy/pbs-example/modules/regional/security.tf @@ -23,8 +23,8 @@ resource "aws_vpc_security_group_egress_rule" "alb_to_pbs" { description = "Forward requests to private PBS hosts" security_group_id = aws_security_group.alb.id referenced_security_group_id = aws_security_group.pbs.id - from_port = var.pbs_port - to_port = var.pbs_port + from_port = local.pbs_port + to_port = local.pbs_port ip_protocol = "tcp" } @@ -35,8 +35,8 @@ resource "aws_security_group" "pbs" { ingress { description = "PBS HTTP from the regional ALB" - from_port = var.pbs_port - to_port = var.pbs_port + from_port = local.pbs_port + to_port = local.pbs_port protocol = "tcp" security_groups = [aws_security_group.alb.id] } diff --git a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl index 9087f7382..b15b4bab4 100644 --- a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl +++ b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl @@ -1,4 +1,8 @@ mock_provider "aws" { + mock_data "aws_partition" { + defaults = { partition = "aws" } + } + mock_resource "aws_instance" { defaults = { id = "i-0123456789abcdef0" @@ -129,8 +133,8 @@ run "plans_private_hosts_and_scoped_ingress" { aws_vpc_security_group_egress_rule.alb_to_pbs.security_group_id == aws_security_group.alb.id && aws_vpc_security_group_egress_rule.alb_to_pbs.referenced_security_group_id == aws_security_group.pbs.id && aws_vpc_security_group_egress_rule.alb_to_pbs.ip_protocol == "tcp" && - aws_vpc_security_group_egress_rule.alb_to_pbs.from_port == var.pbs_port && - aws_vpc_security_group_egress_rule.alb_to_pbs.to_port == var.pbs_port && + aws_vpc_security_group_egress_rule.alb_to_pbs.from_port == 8000 && + aws_vpc_security_group_egress_rule.alb_to_pbs.to_port == 8000 && aws_vpc_security_group_egress_rule.alb_to_pbs.cidr_ipv4 == null && aws_vpc_security_group_egress_rule.alb_to_pbs.cidr_ipv6 == null && aws_vpc_security_group_egress_rule.alb_to_pbs.prefix_list_id == null @@ -153,6 +157,20 @@ run "plans_private_hosts_and_scoped_ingress" { error_message = "Every ALB ingress rule should permit only HTTPS from its approved caller CIDR." } + assert { + condition = ( + aws_lb_target_group.pbs.port == 8000 && + alltrue([for attachment in aws_lb_target_group_attachment.pbs : attachment.port == 8000]) && + alltrue([for rule in aws_security_group.pbs.ingress : rule.from_port == 8000 && rule.to_port == 8000]) + ) + error_message = "ALB targets and PBS ingress must match the runtime's fixed port 8000." + } + + assert { + condition = aws_iam_role_policy_attachment.ssm.policy_arn == "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" + error_message = "Commercial-region hosts should attach the commercial SSM managed policy." + } + assert { condition = length(aws_secretsmanager_secret.bidder) == 1 error_message = "The example should create metadata for only the declared example bidder secret." @@ -172,6 +190,76 @@ run "plans_private_hosts_and_scoped_ingress" { } } +run "rejects_overlong_resource_names" { + command = plan + + variables { + name = "abcdefghijklmnopqrstuvwxyzabc" + } + + expect_failures = [var.name] +} + +run "rejects_invalid_resource_names" { + command = plan + + variables { + name = "-pbs-example" + } + + expect_failures = [var.name] +} + +run "rejects_reserved_alb_prefix" { + command = plan + + variables { + name = "internal-pbs" + } + + expect_failures = [var.name] +} + +run "accepts_maximum_resource_name" { + command = plan + + variables { + name = "abcdefghijklmnopqrstuvwxyzab" + } + + assert { + condition = aws_lb.main.name == "${var.name}-alb" && aws_lb_target_group.pbs.name == "${var.name}-pbs" + error_message = "Names should preserve the full prefix and resource suffix." + } +} + +run "accepts_single_character_resource_name" { + command = plan + + variables { + name = "a" + } + + assert { + condition = aws_lb.main.name == "a-alb" && aws_lb_target_group.pbs.name == "a-pbs" + error_message = "A single alphanumeric character is a valid name prefix." + } +} + +run "uses_current_partition_for_ssm_policy" { + command = plan + + override_data { + target = data.aws_partition.current + values = { partition = "aws-us-gov" } + } + + assert { + condition = aws_iam_role_policy_attachment.ssm.policy_arn == "arn:aws-us-gov:iam::aws:policy/AmazonSSMManagedInstanceCore" + error_message = "The SSM managed policy ARN should follow the provider's AWS partition." + } +} + run "scopes_optional_kms_decryption" { command = plan diff --git a/deploy/pbs-example/modules/regional/variables.tf b/deploy/pbs-example/modules/regional/variables.tf index 835ae19ca..46fe9383b 100644 --- a/deploy/pbs-example/modules/regional/variables.tf +++ b/deploy/pbs-example/modules/regional/variables.tf @@ -29,18 +29,12 @@ variable "instance_type" { } variable "name" { - description = "Stable name prefix for regional resources." + description = "Stable name prefix for regional resources, leaving room for the ALB and target-group suffixes." type = string -} - -variable "pbs_port" { - description = "Host port used by the PBS Compose service." - type = number - default = 8000 validation { - condition = var.pbs_port >= 1 && var.pbs_port <= 65535 - error_message = "pbs_port must be a valid TCP port." + condition = can(regex("^[a-zA-Z0-9]([a-zA-Z0-9-]{0,26}[a-zA-Z0-9])?$", var.name)) && !startswith(var.name, "internal-") + error_message = "name must be 1-28 alphanumeric/hyphen characters, must not start or end with a hyphen, and must not use the reserved ALB prefix internal-." } } diff --git a/deploy/pbs-example/scripts/test-smoke-runtime.py b/deploy/pbs-example/scripts/test-smoke-runtime.py index f49078687..16fc1f6dc 100755 --- a/deploy/pbs-example/scripts/test-smoke-runtime.py +++ b/deploy/pbs-example/scripts/test-smoke-runtime.py @@ -6,9 +6,13 @@ from pathlib import Path import shutil import subprocess +import sys import tempfile +if not __debug__: + sys.exit("Run this check without python -O or PYTHONOPTIMIZE; assertions are required.") + EXAMPLE = Path(__file__).resolve().parent.parent DOCKER = shutil.which("docker") assert DOCKER, "docker with the Compose plugin must be on PATH" @@ -23,7 +27,7 @@ def render_production(): ) service = json.loads(result.stdout)["services"]["pbs"] port = service["ports"][0] - assert port.get("host_ip", "0.0.0.0") == "0.0.0.0", port + assert port["host_ip"] == "0.0.0.0", port assert port["published"] == "8000" and port["target"] == 8000, port assert service["env_file"][0]["path"] == "/run/pbs/secrets/examplebidder.env" diff --git a/deploy/pbs-example/tests/root_unit_test.tftest.hcl b/deploy/pbs-example/tests/root_unit_test.tftest.hcl index 618b03c19..459352eb6 100644 --- a/deploy/pbs-example/tests/root_unit_test.tftest.hcl +++ b/deploy/pbs-example/tests/root_unit_test.tftest.hcl @@ -1,10 +1,6 @@ mock_provider "aws" { - mock_resource "aws_instance" { - override_during = plan - - defaults = { - id = "i-0123456789abcdef0" - } + mock_data "aws_partition" { + defaults = { partition = "aws" } } mock_resource "aws_eip" { @@ -32,12 +28,8 @@ mock_provider "aws" { mock_provider "aws" { alias = "west" - mock_resource "aws_instance" { - override_during = plan - - defaults = { - id = "i-0fedcba9876543210" - } + mock_data "aws_partition" { + defaults = { partition = "aws" } } mock_resource "aws_eip" { @@ -62,6 +54,30 @@ mock_provider "aws" { } } +override_resource { + target = module.east.aws_instance.pbs["us-east-1a"] + override_during = plan + values = { id = "i-00000000000000001" } +} + +override_resource { + target = module.east.aws_instance.pbs["us-east-1b"] + override_during = plan + values = { id = "i-00000000000000002" } +} + +override_resource { + target = module.west.aws_instance.pbs["us-west-2a"] + override_during = plan + values = { id = "i-00000000000000003" } +} + +override_resource { + target = module.west.aws_instance.pbs["us-west-2b"] + override_during = plan + values = { id = "i-00000000000000004" } +} + variables { aws_account_id = "123456789012" aws_profile = "pbs-example" @@ -76,18 +92,21 @@ run "plans_both_regional_provider_mappings" { command = plan assert { - condition = length(module.east.instance_ids) == 2 + condition = length(distinct(values(module.east.instance_ids))) == 2 error_message = "The east provider mapping should plan two PBS instances." } assert { - condition = length(module.west.instance_ids) == 2 + condition = length(distinct(values(module.west.instance_ids))) == 2 error_message = "The west provider mapping should plan two PBS instances." } assert { - condition = length(local.deployment_descriptor.regions) == 2 - error_message = "The generated deployment descriptor should include both regions." + condition = ( + length(local.deployment_descriptor.regions) == 2 && + alltrue([for region in local.deployment_descriptor.regions : length(distinct(region.instance_ids)) == 2]) + ) + error_message = "The generated deployment descriptor should include two distinct instance IDs in each region." } assert { diff --git a/docs/guide/cli.md b/docs/guide/cli.md index c446fd766..e0739dcea 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -611,6 +611,10 @@ APIs. `ts prebid client` builds the local external Prebid browser bundle configured in `trusted-server.toml`. +> This command was previously `ts prebid bundle`. Update scripts and runbooks to +> use `ts prebid client` with the same arguments. The old spelling is retired and +> is not accepted as an alias. + ```toml [integrations.prebid.bundle.modules] bidder = ["rubiconBidAdapter", "kargoBidAdapter"] @@ -664,7 +668,102 @@ AWS. Add `--json` anywhere under `ts prebid server` for machine-readable output. Failures use exit code 2, including incomplete `status` reports that still write partial JSON to stdout. +### Deployment descriptor and bindings + +Pass an explicit `--deployment `; there is no automatic discovery or +production default. This fictional descriptor is suitable only for local checks: + +```yaml +schema_version: 1 +environment: sandbox +runtime: ec2-compose +aws: + account_id: '123456789012' + profile: pbs-sandbox +pbs: + config: pbs.yaml + image: registry.example.com/pbs@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bindings: bindings.json +regions: + us-east-1: + instance_ids: + - i-0123456789abcdef0 + overrides: east.yaml +``` + +Version 1 requires an explicit environment, `ec2-compose` runtime, 12-digit AWS +account ID, AWS CLI profile, digest-pinned image, baseline YAML path, and nonempty +region map. Environment and bidder identifiers use letters, digits, underscores, +and hyphens; profile names may also contain periods. Instance IDs are optional, +but `status` needs explicitly listed instances to report infrastructure health. +Regional override paths and `pbs.bindings` are optional. Omit bindings when no +host secrets are needed. + +Paths resolve relative to the descriptor. Unknown schema fields, duplicate YAML +keys, tags, and implicit YAML merge keys are rejected. Regional mappings merge +recursively over the baseline; sequences and scalars replace whole values. The +CLI validates in memory and writes no rendered files. + +The binding file is a JSON or YAML mapping keyed by bidder identifier. Each +entry requires: + +- `verified_image`, exactly matching the descriptor's image. +- `source`, an HTTPS reference used by the operator to verify the mapping. +- `secrets`, a complete Secrets Manager ARN for each descriptor region, matching + its account and region. Each secret must belong to only one bidder binding. +- `keys`, mapping credential names to an environment variable `env`, a YAML path + array `pbs_path`, and optional `required`, which defaults to true. + +For example, a key mapping can be: + +```json +{ + "api_key": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", + "pbs_path": ["adapters", "examplebidder", "api_key"], + "required": true + } +} +``` + +Bindings support string-valued credentials. Environment variables and YAML +destinations must not conflict across bindings or with values already present in +the resolved YAML. These are operator-supplied mappings, not a verified adapter +catalog. `check` does not validate the complete upstream PBS schema, verify +bidder authorization, retrieve secrets, pull images, or prove application health. + +### Secret-write safeguards + +`secrets set` replaces the complete JSON value of an existing declared secret. +It does not create secret metadata, deploy PBS, refresh a container, or rotate a +bidder's credential. Use a trusted host and short-lived AWS credentials. Obtain +separate approval for each target and value write. + +Before writing, the CLI verifies the account through STS, describes the declared +secret, rejects replicas and secrets scheduled for deletion, and checks both the +selected profile's and default AWS CLI history settings. History must be disabled +or unset. Provider stderr is withheld and configured API endpoint overrides are +disabled. + +Interactive use reads JSON in a hidden terminal prompt and requires typing +`yes`. For approved automation, use `--file ` or `--stdin` with `--yes` and +an explicit `--request-token `. The payload must contain only declared keys, +include required nonempty string values, and fit the Secrets Manager size limit. +Duplicate keys and non-string values are rejected. + +Secret contents do not enter arguments or reports. The AWS request goes through a +tool-owned temporary file, owner-only on Unix, removed on normal success and +error paths. Abrupt termination can leave that file behind; Windows temporary-file +ACLs have not been verified. Operator-provided input files are never deleted. +Input-file errors print the full escaped path to stderr, including under `--json`. +Keep sensitive directory and partner names out of public logs. + +A successful write reports its version identifier, not runtime readiness. A failed +or unverifiable response means the outcome may be uncertain. Preserve the request +UUID and input. Reuse that UUID only with the original identical payload; use a +new UUID only for a separately intended update. Do not retry changed values under +the same UUID. `--yes` authorizes only this write, not deployment or future writes. + See the [experimental PBS command reference](https://github.com/IABTechLab/trusted-server/blob/main/crates/trusted-server-cli/README.md) -for the deployment descriptor schema, secret-write safeguards, and current -limitations. +for complete binding fixtures, command examples, and remaining runtime limits. From 6516a676a682136d161112a26feabd59c439435e Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 2 Oct 2026 12:02:05 -0500 Subject: [PATCH 12/14] Replace PBS Python test helpers with Node.js --- .github/workflows/pbs-example.yml | 12 +- .github/workflows/test.yml | 18 +++ crates/trusted-server-cli/README.md | 2 +- .../tests/fixtures/pbs-aws.cjs | 72 ++++++++++ crates/trusted-server-cli/tests/pbs_cli.rs | 57 +------- deploy/pbs-example/DEPLOYMENT_PLAN.md | 4 +- deploy/pbs-example/README.md | 8 +- deploy/pbs-example/RUNBOOK.md | 8 +- deploy/pbs-example/runtime/README.md | 2 +- .../scripts/test-smoke-runtime.mjs | 125 ++++++++++++++++++ .../pbs-example/scripts/test-smoke-runtime.py | 94 ------------- 11 files changed, 239 insertions(+), 163 deletions(-) create mode 100644 crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs create mode 100644 deploy/pbs-example/scripts/test-smoke-runtime.mjs delete mode 100755 deploy/pbs-example/scripts/test-smoke-runtime.py diff --git a/.github/workflows/pbs-example.yml b/.github/workflows/pbs-example.yml index fbd5dc3a2..dced0c7b6 100644 --- a/.github/workflows/pbs-example.yml +++ b/.github/workflows/pbs-example.yml @@ -38,9 +38,14 @@ jobs: terraform_version: ${{ steps.terraform.outputs.version }} terraform_wrapper: false - - uses: actions/setup-python@v5 + - name: Read pinned Node.js version + id: node + shell: bash + run: echo "version=$(awk '$1 == "nodejs" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: actions/setup-node@v4 with: - python-version: '3.13' + node-version: ${{ steps.node.outputs.version }} - name: Check Docker Compose prerequisite run: docker compose version @@ -63,6 +68,5 @@ jobs: - name: Check runtime inputs and smoke wiring without starting containers run: | - python3 -m json.tool deploy/pbs-example/runtime/secret-bindings.example.json > /dev/null bash -n deploy/pbs-example/scripts/smoke-runtime.sh - python3 -I deploy/pbs-example/scripts/test-smoke-runtime.py + env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a7fc78d08..a122fc83a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -88,6 +88,15 @@ jobs: target: wasm32-wasip1 cache-shared-key: cargo-${{ runner.os }} + - name: Read pinned Node.js version for CLI fixtures + id: node-version + shell: bash + run: echo "version=$(awk '$1 == "nodejs" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.version }} + - name: Build Axum adapter run: cargo build -p trusted-server-adapter-axum @@ -226,6 +235,15 @@ jobs: components: "clippy, rustfmt" cache-shared-key: cargo-cli-${{ runner.os }} + - name: Read pinned Node.js version for CLI fixtures + id: node-version + shell: bash + run: echo "version=$(awk '$1 == "nodejs" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.version }} + # No separate `cargo fmt` here: `trusted-server-cli` is a workspace member, # so the main `cargo fmt --all` job already formats it. Clippy still needs a # native run — the workspace clippy jobs are wasm/adapter-scoped and never diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index d8c7dd7cf..1455de8f7 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -128,6 +128,6 @@ cargo fmt --all -- --check cargo clippy --package trusted-server-cli --all-targets --target x86_64-unknown-linux-gnu -- -D warnings ``` -Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require Python 3. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. The unset-history case covers exit 1 with empty stdout, but this fake response does not establish the real AWS CLI contract. An authorized runtime owner must verify real secret-write version IDs and retry behavior against a throwaway secret before operational use. +Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require the Node.js version pinned in `.tool-versions`. The fake uses only built-in modules and needs no npm install. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. The unset-history case covers exit 1 with empty stdout, but this fake response does not establish the real AWS CLI contract. An authorized runtime owner must verify real secret-write version IDs and retry behavior against a throwaway secret before operational use. Process tests also send deeply nested flow sequences, flow mappings, and block mappings through `check`. The pinned `serde_yaml_ng` parser rejects them with sanitized errors before the recursive configuration walkers run. These tests guard the parser's depth-limit behavior; they are not proof against every possible YAML resource-exhaustion input. diff --git a/crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs b/crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs new file mode 100644 index 000000000..24a52f294 --- /dev/null +++ b/crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs @@ -0,0 +1,72 @@ +#!/usr/bin/env node +// Fake AWS process for PBS CLI tests. Never contacts AWS. +const assert = require('node:assert/strict') +const { + appendFileSync, + readFileSync, + statSync, + writeFileSync, +} = require('node:fs') +const { join } = require('node:path') + +const args = process.argv.slice(2) +const root = process.env.PBS_FAKE_ROOT +appendFileSync(join(root, 'calls'), `${JSON.stringify(args)}\n`) +assert(args.includes('--profile')) +assert.equal(args[args.indexOf('--profile') + 1], 'pbs-sandbox') +if (args.includes('configure')) { + if (process.env.PBS_FAKE_HISTORY === 'unset') process.exit(1) + console.log( + process.env.PBS_FAKE_HISTORY === 'enabled' ? 'enabled' : 'disabled' + ) + process.exit(0) +} +assert(args.includes('--region')) +assert.equal(args[args.indexOf('--region') + 1], 'us-east-1') +assert.equal(process.env.AWS_IGNORE_CONFIGURED_ENDPOINT_URLS, 'true') +assert(args.includes('--cli-input-json')) +const input = args[args.indexOf('--cli-input-json') + 1] +assert(input.startsWith('file://')) +const path = input.slice('file://'.length) +assert.equal(statSync(path).mode & 0o7777, 0o600) +appendFileSync(join(root, 'payload_paths'), `${path}\n`) +const request = JSON.parse(readFileSync(path, 'utf8')) +const arn = + 'arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf' +if (args.includes('get-caller-identity')) { + console.log( + JSON.stringify({ Account: process.env.PBS_FAKE_ACCOUNT ?? '123456789012' }) + ) +} else if (args.includes('describe-secret')) { + console.log(JSON.stringify({ ARN: arn })) +} else if (args.includes('put-secret-value')) { + const failure = process.env.PBS_FAKE_FAILURE + if (failure === 'collision' || failure === 'transport') { + const error = + failure === 'collision' + ? 'ResourceExistsException' + : 'Connection reset by peer' + console.error(`${error}: ${request.SecretString}`) + process.exit(1) + } + writeFileSync(join(root, 'captured_request.json'), JSON.stringify(request)) + if (failure === 'invalid-json') { + console.log(`invalid response ${request.SecretString}`) + } else if (failure === 'unverified') { + console.log( + JSON.stringify({ + ARN: arn, + VersionId: 'unexpected', + SecretString: request.SecretString, + }) + ) + } else { + console.log( + JSON.stringify({ ARN: arn, VersionId: request.ClientRequestToken }) + ) + } +} else if (args.includes('describe-instance-status')) { + console.log(JSON.stringify({ InstanceStatuses: [] })) +} else { + process.exit(2) +} diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index c0edd1d0d..26d48abb1 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -13,8 +13,8 @@ const TOKEN: &str = "11111111-2222-4333-8444-555555555555"; fn fixture() -> TempDir { assert!( - which::which("python3").is_ok(), - "pbs_cli tests need python3 on PATH for the fake AWS executable" + which::which("node").is_ok(), + "pbs_cli tests need the pinned Node.js on PATH for the fake AWS executable" ); let dir = tempfile::tempdir().expect("should create fixture directory"); let source = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/pbs"); @@ -22,55 +22,8 @@ fn fixture() -> TempDir { fs::copy(source.join(name), dir.path().join(name)).expect("should copy fixture"); } let fake = dir.path().join("aws"); - fs::write( - &fake, - r#"#!/usr/bin/env python3 -import json, os, pathlib, stat, sys -args = sys.argv[1:] -root = pathlib.Path(os.environ['PBS_FAKE_ROOT']) -with (root / 'calls').open('a') as log: - log.write(json.dumps(args) + '\n') -assert '--profile' in args and args[args.index('--profile')+1] == 'pbs-sandbox' -if 'configure' in args: - if os.environ.get('PBS_FAKE_HISTORY') == 'unset': - sys.exit(1) - if os.environ.get('PBS_FAKE_HISTORY') == 'enabled': - print('enabled') - else: - print('disabled') - sys.exit(0) -assert args[args.index('--region')+1] == 'us-east-1' -assert os.environ.get('AWS_IGNORE_CONFIGURED_ENDPOINT_URLS') == 'true' -path = pathlib.Path(args[args.index('--cli-input-json')+1].removeprefix('file://')) -assert stat.S_IMODE(path.stat().st_mode) == 0o600 -with (root / 'payload_paths').open('a') as paths: - paths.write(str(path) + '\n') -request = json.loads(path.read_text()) -arn = 'arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf' -if 'get-caller-identity' in args: - print(json.dumps({'Account': os.environ.get('PBS_FAKE_ACCOUNT', '123456789012')})) -elif 'describe-secret' in args: - print(json.dumps({'ARN': arn})) -elif 'put-secret-value' in args: - failure = os.environ.get('PBS_FAKE_FAILURE') - if failure in ('collision', 'transport'): - error = 'ResourceExistsException' if failure == 'collision' else 'Connection reset by peer' - print(error + ': ' + request['SecretString'], file=sys.stderr) - sys.exit(1) - (root / 'captured_request.json').write_text(json.dumps(request)) - if failure == 'invalid-json': - print('invalid response ' + request['SecretString']) - elif failure == 'unverified': - print(json.dumps({'ARN': arn, 'VersionId': 'unexpected', 'SecretString': request['SecretString']})) - else: - print(json.dumps({'ARN': arn, 'VersionId': request['ClientRequestToken']})) -elif 'describe-instance-status' in args: - print(json.dumps({'InstanceStatuses': []})) -else: - sys.exit(2) -"#, - ) - .expect("should write fake AWS executable"); + fs::write(&fake, include_str!("fixtures/pbs-aws.cjs")) + .expect("should write fake AWS executable"); fs::set_permissions(fake, fs::Permissions::from_mode(0o700)) .expect("should set executable mode"); dir @@ -92,6 +45,8 @@ fn command(dir: &Path) -> Command { .env_remove("AWS_ACCESS_KEY_ID") .env_remove("AWS_SECRET_ACCESS_KEY") .env_remove("AWS_SESSION_TOKEN") + .env_remove("NODE_OPTIONS") + .env_remove("NODE_PATH") .env("AWS_EC2_METADATA_DISABLED", "true"); command } diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md index 9390d431a..474f78c85 100644 --- a/deploy/pbs-example/DEPLOYMENT_PLAN.md +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -100,7 +100,7 @@ No price estimate is claimed. The main drivers are four EC2 instances, four NAT ## Generated files and checks -The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI requires Python 3.13 and Docker Compose, but no AWS credentials or container startup. The example maintainer owns these checks. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. +The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI uses the Node.js version in `.tool-versions` and Docker Compose, but no npm dependencies, AWS credentials, or container startup. The example maintainer owns these checks. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. | Path | Consumer | Local check | | ---------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -110,7 +110,7 @@ The `PBS example checks` workflow runs Terraform format/validate and both mocked | `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | | `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and fake-command smoke wiring; separately approved pinned-image startup smoke | | `scripts/smoke-runtime.sh` | Runtime owner | CI shell syntax and fake-command wiring; real startup smoke requires separate authorization | -| `scripts/test-smoke-runtime.py` | Example maintainer | CI Python 3.13 and Docker Compose rendering; no image pull or containers | +| `scripts/test-smoke-runtime.mjs` | Example maintainer | Pinned Node.js, JSON parsing, and Docker Compose rendering; no image pull or containers | | Root and regional `.terraform.lock.hcl` | Example maintainer | Readonly initialization and tests; regenerate both locks for the four supported platforms on provider upgrades | | `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | | `deployment.example.yaml` | Existing PBS CLI | Fictional local `ts prebid server check` only | diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md index c7c136cde..1c4b3bdef 100644 --- a/deploy/pbs-example/README.md +++ b/deploy/pbs-example/README.md @@ -23,7 +23,7 @@ Each region has two AZs, one EC2 host per AZ, a public ALB, and one NAT gateway ## Safe local walkthrough -These commands do not contact AWS. They require the pinned Terraform and Rust toolchains, Python 3, and Docker with the Compose plugin. On Apple Silicon macOS, replace `cargo run_cli_linux` with `cargo run_cli_macos`. Other hosts can use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`. +These commands do not contact AWS. They require the pinned Terraform, Rust, and Node.js toolchains and Docker with the Compose plugin. The Node.js check uses only built-in modules; no npm install is needed. On Apple Silicon macOS, replace `cargo run_cli_linux` with `cargo run_cli_macos`. Other hosts can use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`. ```bash terraform fmt -check -recursive deploy/pbs-example @@ -36,17 +36,15 @@ terraform -chdir=deploy/pbs-example/modules/regional test -filter=tests/security cargo run_cli_linux prebid server check \ --deployment deploy/pbs-example/deployment.example.yaml \ --json -python3 -m json.tool \ - deploy/pbs-example/runtime/secret-bindings.example.json >/dev/null docker compose \ --env-file deploy/pbs-example/runtime/examples/compose.env \ -f deploy/pbs-example/runtime/compose.yaml \ config --quiet bash -n deploy/pbs-example/scripts/smoke-runtime.sh -deploy/pbs-example/scripts/test-smoke-runtime.py +env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs ``` -The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI sets up Python 3.13 and checks Docker Compose availability. Run the wiring test without Python optimization, which disables assertions. The CLI descriptor command above is a separate local check; the example owner must rerun it when changing or adapting inputs. +The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI reads the Node.js version from `.tool-versions` and checks Docker Compose availability. The wiring command clears inherited Node.js preload options and module paths, and the script passes only `PATH` and `HOME` plus its explicit test inputs to subprocesses. It also parses `runtime/secret-bindings.example.json`. The CLI descriptor command above is a separate local check; the example owner must rerun it when changing or adapting inputs. The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md index 99f229c8d..e539c1fad 100644 --- a/deploy/pbs-example/RUNBOOK.md +++ b/deploy/pbs-example/RUNBOOK.md @@ -6,7 +6,7 @@ This runbook describes local checks and the separately authorized operations tha - Work only in `deploy/pbs-example`. - Use Terraform `1.16.2` and the committed AWS provider lock files in both the root and regional module. Both lock AWS `6.64.0` for Linux and macOS on AMD64 and ARM64. -- Local wiring checks require Python 3 and Docker with the Compose plugin. CI uses Python 3.13. Do not enable Python optimization; the wiring test refuses to run without assertions. +- Local wiring checks require the Node.js version in `.tool-versions` and Docker with the Compose plugin. The check uses only built-in Node.js modules; no npm install is needed. - Use fictional values from `terraform.tfvars.example`, `deployment.example.yaml`, and `runtime/secret-bindings.example.json` only for local validation. - Replace the fictional AWS account, profile, certificate, hosted-zone, AMI, and CIDR inputs before any authorized cloud plan. Terraform renders actual instance and secret identifiers after apply. - Confirm the approved Trusted Server egress CIDRs. Do not use the documentation CIDR as a real allowlist. @@ -58,13 +58,11 @@ Expected result: Compose renders successfully without pulling or starting the im Validate the JSON input and smoke command wiring without starting containers: ```bash -python3 -m json.tool \ - deploy/pbs-example/runtime/secret-bindings.example.json >/dev/null bash -n deploy/pbs-example/scripts/smoke-runtime.sh -deploy/pbs-example/scripts/test-smoke-runtime.py +env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs ``` -The wiring test requires Python 3 and Docker Compose. It renders production and smoke configurations, uses fake Docker lifecycle and curl commands, and checks that inherited selectors cannot replace the dummy inputs. It does not pull images, start PBS, or prove runtime health. +The wiring command clears inherited Node.js preload options and module paths. The script passes only `PATH` and `HOME` plus explicit test inputs to subprocesses. It parses the binding JSON, renders production and smoke configurations, uses fake Docker lifecycle and curl commands, and checks that inherited selectors cannot replace the dummy inputs. It does not pull images, start PBS, or prove runtime health. ### Separately approved local runtime smoke diff --git a/deploy/pbs-example/runtime/README.md b/deploy/pbs-example/runtime/README.md index fe0527cc0..9c795880a 100644 --- a/deploy/pbs-example/runtime/README.md +++ b/deploy/pbs-example/runtime/README.md @@ -6,4 +6,4 @@ The `/run/pbs/secrets/examplebidder.env` file is a runtime contract, not a check `secret-bindings.example.json` and `examples/pbs-secrets.env` contain fictional values for local checks only. An authorized Terraform apply can render the ignored `secret-bindings.generated.json` with actual regional secret ARNs, but never secret values. -Deployment Compose binds `0.0.0.0:8000` by default so the ALB can reach PBS through the host security group. `scripts/smoke-runtime.sh` forces `PBS_BIND_ADDRESS=127.0.0.1` and checked-in dummy inputs for local startup. `scripts/test-smoke-runtime.py` verifies both bindings and the smoke selectors with Compose rendering and fake commands only. Run both scripts from the parent example directory or use their repository-relative paths. +Deployment Compose binds `0.0.0.0:8000` by default so the ALB can reach PBS through the host security group. `scripts/smoke-runtime.sh` forces `PBS_BIND_ADDRESS=127.0.0.1` and checked-in dummy inputs for local startup. `scripts/test-smoke-runtime.mjs` verifies both bindings and the smoke selectors with Compose rendering and fake commands only. Use the Node.js wiring command in the parent runbook. Run the shell smoke from the parent example directory or use its repository-relative path. diff --git a/deploy/pbs-example/scripts/test-smoke-runtime.mjs b/deploy/pbs-example/scripts/test-smoke-runtime.mjs new file mode 100644 index 000000000..17aec86bd --- /dev/null +++ b/deploy/pbs-example/scripts/test-smoke-runtime.mjs @@ -0,0 +1,125 @@ +#!/usr/bin/env node +// Render Compose and exercise smoke command wiring without starting containers. +import assert from 'node:assert/strict' +import { execFileSync } from 'node:child_process' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { delimiter, join } from 'node:path' +import { fileURLToPath } from 'node:url' + +const example = fileURLToPath(new URL('../', import.meta.url)) +const env = Object.fromEntries( + ['PATH', 'HOME'] + .filter((key) => process.env[key] !== undefined) + .map((key) => [key, process.env[key]]) +) + +// Inherit stderr so a failed Compose command explains why it failed. +function run(command, args, environment = env) { + return execFileSync(command, args, { + env: environment, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'inherit'], + }) +} + +function renderProduction() { + const compose = join(example, 'runtime/compose.yaml') + // Older Compose releases stat env_file even with --no-env-resolution. + // Check the deployment default separately and render with dummy values. + assert( + readFileSync(compose, 'utf8').includes( + '${PBS_SECRET_ENV_FILE:-/run/pbs/secrets/examplebidder.env}' + ) + ) + const service = JSON.parse( + run('docker', ['compose', '-f', compose, 'config', '--format', 'json'], { + ...env, + PBS_SECRET_ENV_FILE: join(example, 'runtime/examples/pbs-secrets.env'), + }) + ).services.pbs + const port = service.ports[0] + assert.equal(port.host_ip, '0.0.0.0') + assert.equal(port.published, '8000') + assert.equal(port.target, 8000) +} + +function checkSmoke() { + const directory = mkdtempSync(join(tmpdir(), 'pbs-smoke-wiring-')) + try { + writeFileSync( + join(directory, 'docker'), + `#!/usr/bin/env node +const assert = require('node:assert/strict') +const { execFileSync } = require('node:child_process') +const { appendFileSync, writeFileSync } = require('node:fs') +const args = process.argv.slice(2) +assert.equal(args[0], 'compose') +const operation = args[7] +assert(['up', 'logs', 'down'].includes(operation)) +appendFileSync(process.env.CALLS, operation + '\\n') +if (operation === 'up') { + const rendered = execFileSync('docker', [...args.slice(0, 7), 'config', '--format', 'json'], { + env: { ...process.env, PATH: process.env.REAL_PATH }, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'inherit'], + }) + writeFileSync(process.env.RENDERED, rendered) +} +`, + { mode: 0o700 } + ) + writeFileSync( + join(directory, 'curl'), + `#!/usr/bin/env node +const assert = require('node:assert/strict') +assert.equal(process.argv.at(-1), 'http://127.0.0.1:18081/status') +console.log('ok') +`, + { mode: 0o700 } + ) + run(join(example, 'scripts/smoke-runtime.sh'), [], { + ...env, + PATH: `${directory}${delimiter}${env.PATH}`, + REAL_PATH: env.PATH, + CALLS: join(directory, 'calls'), + RENDERED: join(directory, 'rendered.json'), + PBS_SMOKE_PORT: '18081', + PBS_HOST_PORT: '19000', + PBS_BIND_ADDRESS: '0.0.0.0', + PBS_CONFIG_FILE: '/nonexistent/inherited-pbs.yaml', + PBS_SECRET_ENV_FILE: '/nonexistent/inherited-secrets.env', + }) + assert.deepEqual( + readFileSync(join(directory, 'calls'), 'utf8').trim().split('\n'), + ['up', 'logs', 'down'] + ) + const service = JSON.parse( + readFileSync(join(directory, 'rendered.json'), 'utf8') + ).services.pbs + const port = service.ports[0] + assert.equal(port.host_ip, '127.0.0.1') + assert.equal(port.published, '18081') + assert.equal(port.target, 8000) + assert.equal(service.volumes[0].source, join(example, 'runtime/pbs.yaml')) + assert.equal( + service.environment.PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY, + 'example-only-api-key' + ) + assert.equal( + service.environment.PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN, + 'example-only-optional-token' + ) + } finally { + rmSync(directory, { recursive: true, force: true }) + } +} + +JSON.parse( + readFileSync(join(example, 'runtime/secret-bindings.example.json'), 'utf8') +) +renderProduction() +checkSmoke() +console.log( + 'Runtime JSON, Compose production/smoke bindings and dummy selectors passed; no containers started.' +) diff --git a/deploy/pbs-example/scripts/test-smoke-runtime.py b/deploy/pbs-example/scripts/test-smoke-runtime.py deleted file mode 100755 index 16fc1f6dc..000000000 --- a/deploy/pbs-example/scripts/test-smoke-runtime.py +++ /dev/null @@ -1,94 +0,0 @@ -#!/usr/bin/env python3 -"""Render Compose and exercise smoke command wiring without starting containers.""" - -import json -import os -from pathlib import Path -import shutil -import subprocess -import sys -import tempfile - - -if not __debug__: - sys.exit("Run this check without python -O or PYTHONOPTIMIZE; assertions are required.") - -EXAMPLE = Path(__file__).resolve().parent.parent -DOCKER = shutil.which("docker") -assert DOCKER, "docker with the Compose plugin must be on PATH" -ENV = {key: os.environ[key] for key in ("PATH", "HOME") if key in os.environ} - - -def render_production(): - result = subprocess.run( - [DOCKER, "compose", "-f", str(EXAMPLE / "runtime/compose.yaml"), - "config", "--format", "json", "--no-env-resolution"], - env=ENV, check=True, capture_output=True, text=True, - ) - service = json.loads(result.stdout)["services"]["pbs"] - port = service["ports"][0] - assert port["host_ip"] == "0.0.0.0", port - assert port["published"] == "8000" and port["target"] == 8000, port - assert service["env_file"][0]["path"] == "/run/pbs/secrets/examplebidder.env" - - -def check_smoke(): - with tempfile.TemporaryDirectory(prefix="pbs-smoke-wiring-") as scratch: - directory = Path(scratch) - docker = directory / "docker" - docker.write_text('''#!/usr/bin/env python3 -import json, os, pathlib, subprocess, sys -args = sys.argv[1:] -assert args[0] == "compose", args -operation = args[7] -assert operation in ("up", "logs", "down"), args -with open(os.environ["CALLS"], "a") as log: - log.write(operation + "\\n") -if operation == "up": - result = subprocess.run( - [os.environ["REAL_DOCKER"], *args[:7], "config", "--format", "json"], - capture_output=True, text=True, - ) - if result.returncode: - sys.stderr.write(result.stderr) - sys.exit(result.returncode) - pathlib.Path(os.environ["RENDERED"]).write_text(result.stdout) -''') - curl = directory / "curl" - curl.write_text('''#!/usr/bin/env python3 -import sys -assert sys.argv[-1] == "http://127.0.0.1:18081/status", sys.argv -print("ok") -''') - docker.chmod(0o755) - curl.chmod(0o755) - env = { - **ENV, - "PATH": f"{directory}:{ENV['PATH']}", - "REAL_DOCKER": DOCKER, - "CALLS": str(directory / "calls"), - "RENDERED": str(directory / "rendered.json"), - "PBS_SMOKE_PORT": "18081", - "PBS_HOST_PORT": "19000", - "PBS_BIND_ADDRESS": "0.0.0.0", - "PBS_CONFIG_FILE": "/nonexistent/inherited-pbs.yaml", - "PBS_SECRET_ENV_FILE": "/nonexistent/inherited-secrets.env", - } - subprocess.run( - [str(EXAMPLE / "scripts/smoke-runtime.sh")], - env=env, check=True, capture_output=True, text=True, - ) - assert (directory / "calls").read_text().splitlines() == ["up", "logs", "down"] - service = json.loads((directory / "rendered.json").read_text())["services"]["pbs"] - port = service["ports"][0] - assert port.get("host_ip") == "127.0.0.1", port - assert port["published"] == "18081" and port["target"] == 8000, port - assert service["volumes"][0]["source"] == str(EXAMPLE / "runtime/pbs.yaml") - assert service["environment"]["PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY"] == "example-only-api-key" - assert service["environment"]["PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN"] == "example-only-optional-token" - - -if __name__ == "__main__": - render_production() - check_smoke() - print("Compose production/smoke bindings and dummy selectors passed; no containers started.") From ae3cbacc416306402591ada41838caf17559fd62 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 2 Oct 2026 14:56:42 -0500 Subject: [PATCH 13/14] Fix PBS bundle discovery and test committed deployment inputs --- crates/trusted-server-cli/README.md | 2 +- .../src/commands/pbs/config.rs | 10 ++ .../src/commands/pbs/inspect.rs | 132 ++++++++++++++++-- crates/trusted-server-cli/tests/pbs_cli.rs | 8 +- deploy/pbs-example/DEPLOYMENT_PLAN.md | 4 +- deploy/pbs-example/README.md | 2 +- deploy/pbs-example/RUNBOOK.md | 2 +- trusted-server.example.toml | 2 +- 8 files changed, 142 insertions(+), 20 deletions(-) diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 1455de8f7..6a24cae57 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -58,7 +58,7 @@ Not implemented: container deployment, rollback, runtime secret injection, calle ### Discovering requirements -`inspect` reads exactly the chosen file and never rewrites or publishes it. It discovers server demand from `[auction.providers.*]` entries using the `prebid-server` profile and bidders routed through `[auction.bidders.*]`. The JSON report groups routed bidders under each provider. Browser settings still come from `[integrations.prebid]`, including the runtime's array, indexed-map, and string encodings for `client_side_bidders`. The command reports explicitly supplied values only; it does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source and environment are authoritative before relying on the report. +`inspect` reads exactly the chosen file and never rewrites or publishes it. It discovers server demand from `[auction.providers.*]` entries using the `prebid-server` profile and bidders routed through `[auction.bidders.*]`. The JSON report groups routed bidders under each provider. Browser settings still come from `[integrations.prebid]`, including the runtime's array, indexed-map, and string encodings for `client_side_bidders`. Bundle selections come from `[integrations.prebid.bundle.modules]`: `bidder`, `user_id`, and `analytics` appear as `bundle_adapters`, `identity_modules`, and `analytics_modules` in JSON and as separate lists in human output. The command reports explicitly supplied values only; it does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source and environment are authoritative before relying on the report. Account identifiers, endpoint values, and bid-parameter values are withheld. Parser errors also withhold source snippets. Server-side bidders, client-side bidders, and browser bundle adapters remain separate; listing a bidder does not establish partner authorization or a host-secret requirement. Disabled auctions and integrations remain disabled. diff --git a/crates/trusted-server-cli/src/commands/pbs/config.rs b/crates/trusted-server-cli/src/commands/pbs/config.rs index 671972462..805722c06 100644 --- a/crates/trusted-server-cli/src/commands/pbs/config.rs +++ b/crates/trusted-server-cli/src/commands/pbs/config.rs @@ -438,6 +438,16 @@ pub(super) mod tests { (dir, path) } + #[test] + fn validates_the_committed_pbs_example_descriptor() { + let path = Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../../deploy/pbs-example/deployment.example.yaml"); + let deployment = Deployment::load(&path) + .expect("should load the committed PBS example descriptor and its inputs"); + let output = check(&deployment).expect("should check the committed PBS example"); + assert_eq!(output.data["local_checks"], "passed"); + } + #[test] fn pinned_image_requires_lowercase_sha256_hex() { let image = |digest: &str| format!("registry.example.com/pbs@sha256:{digest}"); diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs index 71bd34c78..0bfd39df1 100644 --- a/crates/trusted-server-cli/src/commands/pbs/inspect.rs +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -91,9 +91,18 @@ struct Prebid { #[derive(Default, Deserialize)] struct Bundle { #[serde(default)] - adapters: Vec, + modules: BundleModules, +} + +/// Explicit selections from core's bundle module schema; never expand generator presets. +#[derive(Default, Deserialize)] +struct BundleModules { #[serde(default)] - user_id_modules: Vec, + bidder: Vec, + #[serde(default)] + user_id: Vec, + #[serde(default)] + analytics: Vec, } /// Accept the same encodings as the private core list deserializer without expanding defaults. @@ -175,12 +184,13 @@ pub(super) fn inspect(path: &Path) -> Result { for name in prebid .client_side_bidders .iter() - .chain(&prebid.bundle.adapters) - .chain(&prebid.bundle.user_id_modules) + .chain(&prebid.bundle.modules.bidder) + .chain(&prebid.bundle.modules.user_id) + .chain(&prebid.bundle.modules.analytics) { if !identifier(name) { return Err(Report::new(PbsError::Input( - "invalid bidder or identity-module identifier", + "invalid bidder or bundle-module identifier", ))); } } @@ -253,7 +263,15 @@ pub(super) fn inspect(path: &Path) -> Result { ), format!( "Browser bundle adapters: {}", - prebid.bundle.adapters.join(", ") + prebid.bundle.modules.bidder.join(", ") + ), + format!( + "Browser identity modules: {}", + prebid.bundle.modules.user_id.join(", ") + ), + format!( + "Browser analytics modules: {}", + prebid.bundle.modules.analytics.join(", ") ), ]); details.extend(warnings.iter().map(|warning| (*warning).to_owned())); @@ -276,8 +294,9 @@ pub(super) fn inspect(path: &Path) -> Result { "timeout_ms_explicit": prebid.timeout_ms, "debug_explicit": prebid.debug, "client_side_bidders": prebid.client_side_bidders, - "bundle_adapters": prebid.bundle.adapters, - "identity_modules": prebid.bundle.user_id_modules, + "bundle_adapters": prebid.bundle.modules.bidder, + "identity_modules": prebid.bundle.modules.user_id, + "analytics_modules": prebid.bundle.modules.analytics, "warnings": warnings }), }) @@ -326,9 +345,10 @@ enabled = false account_id = "NEVER_PRINT_ME" client_side_bidders = ["browserbidder"] -[integrations.prebid.bundle] -adapters = ["bundlebidder"] -user_id_modules = ["sharedIdSystem"] +[integrations.prebid.bundle.modules] +bidder = ["exampleBidAdapter"] +user_id = ["sharedIdSystem"] +analytics = ["exampleAnalyticsAdapter"] "#; fs::write(&path, source).expect("should write fixture"); let report = inspect(&path).expect("should inspect config"); @@ -374,7 +394,12 @@ user_id_modules = ["sharedIdSystem"] 1 ); assert_eq!(report.data["client_side_bidders"][0], "browserbidder"); - assert_eq!(report.data["bundle_adapters"][0], "bundlebidder"); + assert_eq!(report.data["bundle_adapters"], json!(["exampleBidAdapter"])); + assert_eq!(report.data["identity_modules"], json!(["sharedIdSystem"])); + assert_eq!( + report.data["analytics_modules"], + json!(["exampleAnalyticsAdapter"]) + ); assert_eq!( report.data["server_providers"][0]["server_bidder_candidates"][0]["host_secret_requirement"], "unresolved" @@ -386,6 +411,9 @@ user_id_modules = ["sharedIdSystem"] let human = String::from_utf8(human).expect("should emit UTF-8"); assert!(human.contains("pbs-main")); assert!(human.contains("serverbidder")); + assert!(human.contains("Browser bundle adapters: exampleBidAdapter")); + assert!(human.contains("Browser identity modules: sharedIdSystem")); + assert!(human.contains("Browser analytics modules: exampleAnalyticsAdapter")); assert!(!human.contains("NEVER_PRINT_ME")); let mut json = Vec::new(); report @@ -426,7 +454,7 @@ client_side_bidders = 'examplebidder\' assert!( error .to_string() - .contains("invalid bidder or identity-module identifier"), + .contains("invalid bidder or bundle-module identifier"), "should report identifier validation: {error}" ); } @@ -455,6 +483,84 @@ client_side_bidders = 'examplebidder\' } } + #[test] + fn bundle_selections_match_core_without_expanding_defaults() { + for input in [ + "", + "[bundle]", + "[bundle.modules]", + "[bundle.modules]\nbidder = []\nuser_id = []\nanalytics = []", + "[bundle.modules]\nbidder = ['exampleBidAdapter', 'otherBidAdapter']\nuser_id = ['sharedIdSystem']\nanalytics = ['exampleAnalyticsAdapter']", + "[bundle.modules]\nuser_id = ['sharedIdSystem']", + "[bundle.modules]\nanalytics = ['exampleAnalyticsAdapter']", + ] { + let runtime: trusted_server_core::integrations::prebid::PrebidIntegrationConfig = + toml::from_str(input).expect("should parse current core bundle schema"); + let text = format!( + "[integrations.prebid]\n{}", + input.replace("[bundle", "[integrations.prebid.bundle") + ); + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write(file.path(), &text).expect("should write config"); + let output = inspect(file.path()).expect("should inspect current core bundle schema"); + assert_eq!( + output.data["bundle_adapters"], + json!(runtime.bundle.modules.bidder) + ); + assert_eq!( + output.data["identity_modules"], + json!(runtime.bundle.modules.user_id.unwrap_or_default()) + ); + assert_eq!( + output.data["analytics_modules"], + json!(runtime.bundle.modules.analytics.unwrap_or_default()) + ); + } + } + + #[test] + fn retired_bundle_fields_are_not_reported_as_current_selections() { + let input = + "[bundle]\nadapters = ['exampleBidAdapter']\nuser_id_modules = ['sharedIdSystem']"; + assert!( + toml::from_str::( + input + ) + .is_err(), + "core should reject the retired bundle schema" + ); + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write( + file.path(), + input.replace("[bundle]", "[integrations.prebid.bundle]"), + ) + .expect("should write config"); + // Inspection is deliberately partial, not full runtime validation. + let output = + inspect(file.path()).expect("should ignore fields outside the inspected schema"); + for field in ["bundle_adapters", "identity_modules", "analytics_modules"] { + assert_eq!(output.data[field], json!([]), "should not infer {field}"); + } + } + + #[test] + fn invalid_bundle_selections_never_echo_source_values() { + for field in ["bidder", "user_id", "analytics"] { + for value in ["['NEVER_PRINT_ME invalid']", "'NEVER_PRINT_ME'"] { + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write( + file.path(), + format!("[integrations.prebid.bundle.modules]\n{field} = {value}\n"), + ) + .expect("should write config"); + let error = inspect(file.path()) + .err() + .expect("should reject invalid module selection"); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + } + } + #[test] fn missing_section_stays_unresolved_instead_of_enabling_defaults() { let dir = tempfile::tempdir().expect("should create temp directory"); diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs index 26d48abb1..07536965a 100644 --- a/crates/trusted-server-cli/tests/pbs_cli.rs +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -92,7 +92,7 @@ fn assert_payload_cleanup(dir: &Path) { #[test] fn local_commands_never_execute_aws_and_preserve_the_source() { let dir = fixture(); - let toml = "[auction]\nenabled=true\n[auction.providers.pbs-main]\nprofile='prebid-server'\nendpoint='https://DUMMY_SECRET@pbs.example.com/openrtb2/auction'\n[auction.bidders.examplebidder]\nprovider='pbs-main'\n[integrations.prebid]\nenabled=false\naccount_id='DUMMY_SECRET'\n"; + let toml = "[auction]\nenabled=true\n[auction.providers.pbs-main]\nprofile='prebid-server'\nendpoint='https://DUMMY_SECRET@pbs.example.com/openrtb2/auction'\n[auction.bidders.examplebidder]\nprovider='pbs-main'\n[integrations.prebid]\nenabled=false\naccount_id='DUMMY_SECRET'\n[integrations.prebid.bundle.modules]\nbidder=['exampleBidAdapter']\nuser_id=['sharedIdSystem']\nanalytics=['exampleAnalyticsAdapter']\n"; fs::write(dir.path().join("trusted-server.toml"), toml).expect("should write TOML"); let output = command(dir.path()) .args(["prebid", "server", "inspect", "--json"]) @@ -102,6 +102,12 @@ fn local_commands_never_execute_aws_and_preserve_the_source() { assert_no_secret(&output); let report: Value = serde_json::from_slice(&output.stdout).expect("should emit JSON"); assert_eq!(report["enabled_explicit"], false); + assert_eq!(report["bundle_adapters"], json!(["exampleBidAdapter"])); + assert_eq!(report["identity_modules"], json!(["sharedIdSystem"])); + assert_eq!( + report["analytics_modules"], + json!(["exampleAnalyticsAdapter"]) + ); assert_eq!(report["server_providers"][0]["provider"], "pbs-main"); assert_eq!( report["server_providers"][0]["server_bidder_candidates"][0]["bidder"], diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md index 474f78c85..6df010bc2 100644 --- a/deploy/pbs-example/DEPLOYMENT_PLAN.md +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -100,7 +100,7 @@ No price estimate is claimed. The main drivers are four EC2 instances, four NAT ## Generated files and checks -The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI uses the Node.js version in `.tool-versions` and Docker Compose, but no npm dependencies, AWS credentials, or container startup. The example maintainer owns these checks. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. +The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI uses the Node.js version in `.tool-versions` and Docker Compose, but no npm dependencies, AWS credentials, or container startup. The example maintainer owns these checks. The CLI test suite separately validates the committed deployment descriptor and its referenced inputs on every PR. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. | Path | Consumer | Local check | | ---------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -113,7 +113,7 @@ The `PBS example checks` workflow runs Terraform format/validate and both mocked | `scripts/test-smoke-runtime.mjs` | Example maintainer | Pinned Node.js, JSON parsing, and Docker Compose rendering; no image pull or containers | | Root and regional `.terraform.lock.hcl` | Example maintainer | Readonly initialization and tests; regenerate both locks for the four supported platforms on provider upgrades | | `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | -| `deployment.example.yaml` | Existing PBS CLI | Fictional local `ts prebid server check` only | +| `deployment.example.yaml` | Existing PBS CLI | CLI regression against committed inputs and local `ts prebid server check` | | Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | | `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | | `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md index 1c4b3bdef..f9243b7ad 100644 --- a/deploy/pbs-example/README.md +++ b/deploy/pbs-example/README.md @@ -44,7 +44,7 @@ bash -n deploy/pbs-example/scripts/smoke-runtime.sh env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs ``` -The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI reads the Node.js version from `.tool-versions` and checks Docker Compose availability. The wiring command clears inherited Node.js preload options and module paths, and the script passes only `PATH` and `HOME` plus its explicit test inputs to subprocesses. It also parses `runtime/secret-bindings.example.json`. The CLI descriptor command above is a separate local check; the example owner must rerun it when changing or adapting inputs. +The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI reads the Node.js version from `.tool-versions` and checks Docker Compose availability. The wiring command clears inherited Node.js preload options and module paths, and the script passes only `PATH` and `HOME` plus its explicit test inputs to subprocesses. It also parses `runtime/secret-bindings.example.json`. The CLI test suite separately loads and checks the committed deployment descriptor and its referenced inputs on every PR. The example owner must still run the descriptor command above when adapting inputs. The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md index e539c1fad..b759f4c45 100644 --- a/deploy/pbs-example/RUNBOOK.md +++ b/deploy/pbs-example/RUNBOOK.md @@ -32,7 +32,7 @@ terraform -chdir=deploy/pbs-example/modules/regional test \ `init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs eight tests. Root assertions require distinct instance IDs in each regional descriptor. Module tests also cover resource-name boundaries and partition-specific SSM policy ARNs. Regional assertions cover IMDSv2, encrypted root volumes, private subnet/AZ placement, the TLS policy, scoped ALB ingress/egress, alarms, and secret access. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. -The `PBS example checks` workflow runs these Terraform checks and the JSON, shell, and Compose wiring checks below. CLI descriptor validation remains a local responsibility of the example owner. Provider upgrades must refresh both locks with `terraform providers lock -platform=darwin_arm64 -platform=darwin_amd64 -platform=linux_amd64 -platform=linux_arm64` from each directory and rerun both suites. +The `PBS example checks` workflow runs these Terraform checks and the JSON, shell, and Compose wiring checks below. The CLI test suite separately validates the committed deployment descriptor and its referenced inputs on every PR. The example owner remains responsible for local validation of adapted inputs. Provider upgrades must refresh both locks with `terraform providers lock -platform=darwin_arm64 -platform=darwin_amd64 -platform=linux_amd64 -platform=linux_arm64` from each directory and rerun both suites. Check the PBS descriptor without AWS access. On Apple Silicon macOS, use `cargo run_cli_macos` instead of `cargo run_cli_linux`. On another host, use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`: diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 10d28f084..e841515ae 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -512,7 +512,7 @@ client_side_bidders = [] # bidders running via native Prebid.js adapter # the checked-in User ID registry. Each `name` must appear only once. # # The module must be present in the built bundle: name it under -# [integrations.prebid.bundle].user_id_modules, or omit that list to take the +# [integrations.prebid.bundle.modules].user_id, or omit that list to take the # generator's default preset. `ts prebid client` resolves every managed name # through the checked-in User ID registry and fails if the generated manifest # omits its required module without updating the configured hash or SRI. From aa698f3949091c87c64b5684b2815a2d82649be9 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 5 Oct 2026 11:45:34 -0500 Subject: [PATCH 14/14] Clarify PBS operator docs and artifact checks --- crates/trusted-server-cli/README.md | 4 ++-- deploy/pbs-example/DEPLOYMENT_PLAN.md | 33 +++++++++++++++------------ deploy/pbs-example/README.md | 2 +- 3 files changed, 21 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 6a24cae57..e73c64628 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -58,7 +58,7 @@ Not implemented: container deployment, rollback, runtime secret injection, calle ### Discovering requirements -`inspect` reads exactly the chosen file and never rewrites or publishes it. It discovers server demand from `[auction.providers.*]` entries using the `prebid-server` profile and bidders routed through `[auction.bidders.*]`. The JSON report groups routed bidders under each provider. Browser settings still come from `[integrations.prebid]`, including the runtime's array, indexed-map, and string encodings for `client_side_bidders`. Bundle selections come from `[integrations.prebid.bundle.modules]`: `bidder`, `user_id`, and `analytics` appear as `bundle_adapters`, `identity_modules`, and `analytics_modules` in JSON and as separate lists in human output. The command reports explicitly supplied values only; it does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source and environment are authoritative before relying on the report. +`inspect` reads exactly the chosen file and never rewrites or publishes it. Omitting `--config` reads `trusted-server.toml` in the working directory. It discovers server demand from `[auction.providers.*]` entries using the `prebid-server` profile and bidders routed through `[auction.bidders.*]`. The JSON report groups routed bidders under each provider. Browser settings still come from `[integrations.prebid]`, including the runtime's array, indexed-map, and string encodings for `client_side_bidders`. Bundle selections come from `[integrations.prebid.bundle.modules]`: `bidder`, `user_id`, and `analytics` appear as `bundle_adapters`, `identity_modules`, and `analytics_modules` in JSON and as separate lists in human output. The command reports explicitly supplied values only; it does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source and environment are authoritative before relying on the report. Account identifiers, endpoint values, and bid-parameter values are withheld. Parser errors also withhold source snippets. Server-side bidders, client-side bidders, and browser bundle adapters remain separate; listing a bidder does not establish partner authorization or a host-secret requirement. Disabled auctions and integrations remain disabled. @@ -125,7 +125,7 @@ Deployment and rollback require a separately approved move to versioned runtime ```bash ./scripts/test-cli.sh cargo fmt --all -- --check -cargo clippy --package trusted-server-cli --all-targets --target x86_64-unknown-linux-gnu -- -D warnings +cargo clippy-cli ``` Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require the Node.js version pinned in `.tool-versions`. The fake uses only built-in modules and needs no npm install. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. The unset-history case covers exit 1 with empty stdout, but this fake response does not establish the real AWS CLI contract. An authorized runtime owner must verify real secret-write version IDs and retry behavior against a throwaway secret before operational use. diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md index 6df010bc2..839b5836e 100644 --- a/deploy/pbs-example/DEPLOYMENT_PLAN.md +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -102,21 +102,24 @@ No price estimate is claimed. The main drivers are four EC2 instances, four NAT The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI uses the Node.js version in `.tool-versions` and Docker Compose, but no npm dependencies, AWS credentials, or container startup. The example maintainer owns these checks. The CLI test suite separately validates the committed deployment descriptor and its referenced inputs on every PR. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. -| Path | Consumer | Local check | -| ---------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `README.md` | Example user | Safe walkthrough and stop boundary review | -| `terraform.tf`, `providers.tf`, `variables.tf` | Terraform | Format and validate | -| `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, IMDSv2, encrypted root volumes, private subnet/AZ placement, TLS policy, scoped ALB ingress/egress, and secret access | -| `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | -| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and fake-command smoke wiring; separately approved pinned-image startup smoke | -| `scripts/smoke-runtime.sh` | Runtime owner | CI shell syntax and fake-command wiring; real startup smoke requires separate authorization | -| `scripts/test-smoke-runtime.mjs` | Example maintainer | Pinned Node.js, JSON parsing, and Docker Compose rendering; no image pull or containers | -| Root and regional `.terraform.lock.hcl` | Example maintainer | Readonly initialization and tests; regenerate both locks for the four supported platforms on provider upgrades | -| `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | -| `deployment.example.yaml` | Existing PBS CLI | CLI regression against committed inputs and local `ts prebid server check` | -| Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | -| `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | -| `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | +| Path | Consumer | Local check | +| ----------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `README.md` | Example user | Safe walkthrough and stop boundary review | +| `terraform.tf`, `providers.tf`, `variables.tf`, `dns.tf`, `locals.tf`, `outputs.tf` | Terraform | Format and validate; review generated descriptor and binding outputs | +| `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, IMDSv2, encrypted root volumes, private subnet/AZ placement, TLS policy, scoped ALB ingress/egress, and secret access | +| `tests/root_unit_test.tftest.hcl` | Example maintainer | CI mocked root plan tests for regional wiring, generated CLI inputs, and invalid deployment inputs | +| `terraform.tfvars.example` | Authorized operator | Local mocked plan tests; review fictional inputs before adapting | +| `.gitignore` | Example maintainer | Verify state, saved plans, local inputs, and generated operator files remain ignored | +| `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | +| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and fake-command smoke wiring; separately approved pinned-image startup smoke | +| `scripts/smoke-runtime.sh` | Runtime owner | CI shell syntax and fake-command wiring; real startup smoke requires separate authorization | +| `scripts/test-smoke-runtime.mjs` | Example maintainer | Pinned Node.js, JSON parsing, and Docker Compose rendering; no image pull or containers | +| Root and regional `.terraform.lock.hcl` | Example maintainer | Readonly initialization and tests; regenerate both locks for the four supported platforms on provider upgrades | +| `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | +| `deployment.example.yaml` | Existing PBS CLI | CLI regression against committed inputs and local `ts prebid server check` | +| Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | +| `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | +| `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | ## Sources and verification diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md index f9243b7ad..1fe071669 100644 --- a/deploy/pbs-example/README.md +++ b/deploy/pbs-example/README.md @@ -46,7 +46,7 @@ env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runt The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI reads the Node.js version from `.tool-versions` and checks Docker Compose availability. The wiring command clears inherited Node.js preload options and module paths, and the script passes only `PATH` and `HOME` plus its explicit test inputs to subprocesses. It also parses `runtime/secret-bindings.example.json`. The CLI test suite separately loads and checks the committed deployment descriptor and its referenced inputs on every PR. The example owner must still run the descriptor command above when adapting inputs. -The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional identifiers for local validation only. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. +The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional account, profile, secret, instance, and adapter identifiers for local validation only. Their pinned image digest identifies the upstream PBS Go v4.7.0 release, not a fictional image; preserve it unless deliberately selecting and verifying another release, and keep both files' image references identical. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. A separately approved `deploy/pbs-example/scripts/smoke-runtime.sh` run pulls and starts the pinned image with dummy values on loopback, checks `/status` and startup-log redaction, and sends no auction request. Static rendering and fake-command checks are not runtime startup evidence.